I used my cell phone last night to go into the problem of faulty testing for COVID-19, based on a NYT article. As a result, I couldn’t make any nice tables. Let me remedy that and also look at a few more assumptions.
This table summarizes the testing results on a theoretical group of a million Americans tested, assuming that 5% of the population actually has coronavirus antibodies, and that the tests being given have a false negative rate of 10% and a false positive rate of 3%. Reminder: a ‘false negative’ result means that you are told that you don’t have any coronavirus antibodies but you actually do have them, and a ‘false positive’ result means that you are told that you DO have those antibodies, but you really do NOT. I have tried to highlight the numbers of people who get incorrect results in the color red.
Table A
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 50,000 | 10% | 45,000 | 5,000 |
| Actually Negative | 950,000 | 3% | 28,500 | 921,500 |
| Totals | 1,000,000 | 73,500 | 926,500 | |
| Percent we assume are actually positive | 5% | Accuracy Rating | 61.2% | 99.5% |
As you can see, using those assumptions, if you get a lab test result that says you are positive, that will only be correct in about 61% of the time. Which means that you need to take another test, or perhaps two more tests, to see whether they agree.
The next table assumes again a true 5% positive result for the population and a false negative rate of 10%, but a false positive rate of 14%.
Table B
| Assume 5% really exposed, 14% false positive rate, 10% false negative | ||||
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 50,000 | 10% | 45,000 | 5,000 |
| Actually Negative | 950,000 | 14% | 133,000 | 817,000 |
| Totals | 1,000,000 | 178,000 | 822,000 | |
| Percent we assume are actually positive | 5% | Accuracy Rating | 25.3% | 99.4% |
Note that in this scenario, if you get a test result that says you are positive, that is only going to be correct one-quarter of the time (25.3%)! That is useless!
Now, let’s assume a lower percentage of the population actually has the COVID-19 antibodies, say, two percent. Here are the results if we assume a 3% false positive rate:
Table C
| Assume 2% really exposed, 3% false positive rate, 10% false negative | ||||
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 20,000 | 10% | 18,000 | 2,000 |
| Actually Negative | 980,000 | 3% | 29,400 | 950,600 |
| Totals | 1,000,000 | 47,400 | 952,600 | |
| Percent we assume are actually positive | 2% | Accuracy Rating | 38.0% | 99.8% |
Notice that in this scenario, if you get a ‘positive’ result, it is likely to be correct only a little better than one-third of the time (38.0%).
And now let’s assume 2% actual exposure, 14% false positive, 10% false negative:
Table D
| Assume 2% really exposed, 14% false positive rate, 10% false negative | ||||
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 20,000 | 10% | 45,000 | 2,000 |
| Actually Negative | 980,000 | 14% | 137,200 | 842,800 |
| Totals | 1,000,000 | 182,200 | 844,800 | |
| Percent we assume are actually positive | 2% | Accuracy Rating | 24.7% | 99.8% |
Once again, the chances of a ‘positive’ test result being accurate is only about one in four (24.7%), which means that this level of accuracy is not going to be useful to the public at large.
Final set of assumptions: 3% actual positive rate, and excellent tests with only 3% false positive and false negative rates:
Table E
| Assume 3% really exposed, 3% false positive rate, 3% false negative | ||||
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 30,000 | 3% | 45,000 | 900 |
| Actually Negative | 970,000 | 3% | 29,100 | 940,900 |
| Totals | 1,000,000 | 74,100 | 941,800 | |
| Percent we assume are actually positive | 3% | Accuracy Rating | 60.7% | 99.9% |
Once again, if you test positive in this scenario, that result is only going to be correct about 3/5 of the time (60.7%).
All is not lost, however. Suppose we re-test all the people who tested positive in this last group (that’s a bit over seventy-four thousand people, in Table E). Here are the results:
Table F
| Assume 60.7% really exposed, 3% false positive rate, 3% false negative | ||||
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 45,000 | 3% | 43,650 | 1,350 |
| Actually Negative | 29,100 | 3% | 873 | 28,227 |
| Totals | 74,100 | 44,523 | 29,577 | |
| Percent we assume are actually positive | 60.7% | Accuracy Rating | 98.0% | 95.4% |
Notice that 98% accuracy rating for positive results! Much better!
What about our earlier scenario, in table B, with a 5% overall exposure rating, 14% false positives, and 10% false negatives — what if we re-test all the folks who tested positive? Here are the results:
Table G
| Assume 25.3% really exposed, 14% false positive rate, 10% false negative | ||||
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 45,000 | 14% | 38,700 | 6,300 |
| Actually Negative | 133,000 | 10% | 13,300 | 119,700 |
| Totals | 178,000 | 52,000 | 126,000 | |
| Percent we assume are really positive | 25.3% | Accuracy Rating | 74.4% | 95.0% |
This is still not very good: the re-test is going to be accurate only about three-quarters of the time (74.4%) that it says you really have been exposed, and would only clear you 95% of the time. So we would need to run yet another test on those who again tested positive in Table G. If we do it, the results are here:
Table H
| Assume 74.4% really exposed, 14% false positive rate, 10% false negative | ||||
| Group | Total | Error rate | Test says they are Positive | Test says they are Negative |
| Actually Positive | 38,700 | 14% | 33,282 | 5,418 |
| Actually Negative | 13,300 | 10% | 1,330 | 11,970 |
| Totals | 52,000 | 34,612 | 17,388 | |
| Percent we assume are really positive | 74.4% | Accuracy Rating | 96.2% | 68.8% |
This result is much better, but note that this requires THREE TESTS on each of these supposedly positive people to see if they are in fact positive. It also means that if they get a ‘negative’ result, that’s likely to be correct only about 2/3 of the time (68.8%).
So, no wonder that a lot of the testing results we are seeing are difficult to interpret! This is why science requires repeated measurements to separate the truth from fiction! And it also explains some of the snafus committed by our current federal leadership in insisting on not using tests offered from abroad.
============
EDIT at 10:30 pm on 4/25/2020: I found a few minor mistakes and corrected them, and tried to format things more clearly.
GFBrandenburg's Blog 
Leave a Reply