A Problem with Testing

Chance in Biology,
by Mark Denny and Steven Gaines.

With a pandemic raging, I hear a lot about testing. One problem is we don’t have an adequate supply of test kits to screen for Covid-19. But another problem would arise even if we had enough kits to test everyone. To explain, I’ll describe an example presented in Chance in Biology: Using Probability to Explore Nature, by Mark Denny and Steven Gaines. Their analysis was based on testing for HIV, but I’ll recast the story in terms of the coronavirus.

Suppose we have an accurate test for Covid-19. No test is perfect, so let’s assume it’s correct 99.5% of the time. In other words, its error rate is 0.5% (one error for every two hundred tests). We’ll assume this error rate is the same for false positives and false negatives. Furthermore, we’ll assume Covid-19 is not prevalent, infecting only 0.1% of the population. I doubt this is a good assumption right now, when the virus seems to be infecting everyone, but I can imagine a time not too far in the future (a few weeks maybe, a few months probably) when the fraction of people having the virus is small.

In a population of a million people, 1000 would have Covid-19 and 999,000 would not. First, consider what happens when you test the thousand that are infected. The test would come back positive for Covid-19 in 99.5% of the cases, so it would produce 995 true positives. The test would be in error and give a negative result 0.5% of the time, giving 5 false negatives (the test would say you don’t have the disease when in fact you do). Next, consider the results from testing the 999,000 people who are not infected. Again, the test accuracy is 99.5%, so you would get a negative result (true negatives) for 994,005 people (0.995 times 999,000). You’d make a mistake 0.5% of the time, so you get false positives in 4995 cases (the test would say you have the disease when in fact you don’t). Let’s summarize:

True Positives	995
False Positives	4,995
True Negatives	994,005
False Negatives	5

Now, suppose Michigan Governor Gretchen Whitmer (I’m a big fan of the governor) decides that—to prevent the virus from flaring up again—everyone will be tested; anyone who tests positive for Covid-19 must be quarantined, and anyone who tests negative is free to go wherever they please (restaurants, sporting events, movies... oh how I miss them!). Out of a million people, 5990 will test positive (995 + 4995). Of those, 4995 are mistakes (false positives). In other words, 83% of the people who are forced into quarantine are false positives; they don’t have the disease, but the blasted test said they do and they must suffer for it. Only 17% of the quarantined people are infected.

Is this acceptable? Maybe. We might decide as a state that it’s worth it; we insist on quarantines, even if five out of six people forced into isolation are actually healthy. The needs of the many outweigh the needs of the few. Or, we might decide this is too high a price to pay; people are innocent until proven guilty, so to speak. This decision is not simple. Whitmer will have people mad at her regardless of what she does. But we must decide based on the facts. If the incidence of Covid-19 is 0.1%, and the accuracy of the test is 99.5%, then five out of six quarantined people will be false positives. That’s how the math works. Denny and Gaines summarize it this way:

Although individual tests have a low chance of error, most individuals who are tested are not infected with [Covid-19]. Therefore, we are multiplying a small probability of false positives by a large number of uninfected individuals. Even a minute probability of false positives for individual tests can in this circumstance produce many more false positives than true positives. As long as the disease is rare, even a very accurate test of infection will not be able to accurately identify infected individuals in a random test.

I don’t know the prevalence of Covid-19 or the accuracy of Covid-19 tests. The numbers for the coronavirus may be different than what I used in this example. My point is that even an accurate test can produce many false positives for a rare disease. That’s an important insight, whether or not the numbers are accurate for our current plague.

Chance in Biology is full of examples like this one. It’s a good book (although I like Denny’s Air and Water better). It’s a useful supplement to Intermediate Physics for Medicine and Biology, providing all the probability you need to understand biological physics.