More on the “false positive” COVID-19 testing problem

I used my cell phone last night to go into the problem of faulty testing for COVID-19, based on a NYT article. As a result, I couldn’t make any nice tables. Let me remedy that and also look at a few more assumptions.

This table summarizes the testing results on a theoretical group of a million Americans tested, assuming that 5% of the population actually has coronavirus antibodies, and that the tests being given have a false negative rate of 10% and a false positive rate of 3%. Reminder: a ‘false negative’ result means that you are told that you don’t have any coronavirus antibodies but you actually do have them, and a ‘false positive’ result means that you are told that you DO have those antibodies, but you really do NOT. I have tried to highlight the numbers of people who get incorrect results in the color red.

Table A

Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 50,000 10% 45,000 5,000
Actually Negative 950,000 3% 28,500 921,500
Totals 1,000,000 73,500 926,500
Percent we assume are actually positive 5% Accuracy Rating 61.2% 99.5%

As you can see, using those assumptions, if you get a lab test result that says you are positive, that will only be correct in about 61% of the time. Which means that you need to take another test, or perhaps two more tests, to see whether they agree.

The next table assumes again a true 5% positive result for the population and a false negative rate of 10%, but a false positive rate of 14%.

Table B

Assume 5% really exposed, 14% false positive rate, 10% false negative
Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 50,000 10% 45,000 5,000
Actually Negative 950,000 14% 133,000 817,000
Totals 1,000,000 178,000 822,000
Percent we assume are actually positive 5% Accuracy Rating 25.3% 99.4%

Note that in this scenario, if you get a test result that says you are positive, that is only going to be correct one-quarter of the time (25.3%)! That is useless!

Now, let’s assume a lower percentage of the population actually has the COVID-19 antibodies, say, two percent. Here are the results if we assume a 3% false positive rate:

Table C

Assume 2% really exposed, 3% false positive rate, 10% false negative
Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 20,000 10% 18,000 2,000
Actually Negative 980,000 3% 29,400 950,600
Totals 1,000,000 47,400 952,600
Percent we assume are actually positive 2% Accuracy Rating 38.0% 99.8%

Notice that in this scenario, if you get a ‘positive’ result, it is likely to be correct only a little better than one-third of the time (38.0%).

And now let’s assume 2% actual exposure, 14% false positive, 10% false negative:

Table D

Assume 2% really exposed, 14% false positive rate, 10% false negative
Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 20,000 10% 45,000 2,000
Actually Negative 980,000 14% 137,200 842,800
Totals 1,000,000 182,200 844,800
Percent we assume are actually positive 2% Accuracy Rating 24.7% 99.8%

Once again, the chances of a ‘positive’ test result being accurate is only about one in four (24.7%), which means that this level of accuracy is not going to be useful to the public at large.

Final set of assumptions: 3% actual positive rate, and excellent tests with only 3% false positive and false negative rates:

Table E

Assume 3% really exposed, 3% false positive rate, 3% false negative
Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 30,000 3% 45,000 900
Actually Negative 970,000 3% 29,100 940,900
Totals 1,000,000 74,100 941,800
Percent we assume are actually positive 3% Accuracy Rating 60.7% 99.9%

Once again, if you test positive in this scenario, that result is only going to be correct about 3/5 of the time (60.7%).

All is not lost, however. Suppose we re-test all the people who tested positive in this last group (that’s a bit over seventy-four thousand people, in Table E). Here are the results:

Table F

Assume 60.7% really exposed, 3% false positive rate, 3% false negative
Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 45,000 3% 43,650 1,350
Actually Negative 29,100 3% 873 28,227
Totals 74,100 44,523 29,577
Percent we assume are actually positive 60.7% Accuracy Rating 98.0% 95.4%

Notice that 98% accuracy rating for positive results! Much better!

What about our earlier scenario, in table B, with a 5% overall exposure rating, 14% false positives, and 10% false negatives — what if we re-test all the folks who tested positive? Here are the results:

Table G

Assume 25.3% really exposed, 14% false positive rate, 10% false negative
Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 45,000 14% 38,700 6,300
Actually Negative 133,000 10% 13,300 119,700
Totals 178,000 52,000 126,000
Percent we assume are really positive 25.3% Accuracy Rating 74.4% 95.0%

This is still not very good: the re-test is going to be accurate only about three-quarters of the time (74.4%) that it says you really have been exposed, and would only clear you 95% of the time. So we would need to run yet another test on those who again tested positive in Table G. If we do it, the results are here:

Table H

Assume 74.4% really exposed, 14% false positive rate, 10% false negative
Group Total Error rate Test says they are Positive Test says they are Negative
Actually Positive 38,700 14% 33,282 5,418
Actually Negative 13,300 10% 1,330 11,970
Totals 52,000 34,612 17,388
Percent we assume are really positive 74.4% Accuracy Rating 96.2% 68.8%

This result is much better, but note that this requires THREE TESTS on each of these supposedly positive people to see if they are in fact positive. It also means that if they get a ‘negative’ result, that’s likely to be correct only about 2/3 of the time (68.8%).

So, no wonder that a lot of the testing results we are seeing are difficult to interpret! This is why science requires repeated measurements to separate the truth from fiction! And it also explains some of the snafus committed by our current federal leadership in insisting on not using tests offered from abroad.

 

============

EDIT at 10:30 pm on 4/25/2020: I found a few minor mistakes and corrected them, and tried to format things more clearly.

PISA shows great US education progress under Common Core, charter proliferation, reforms. (JUST KIDDING!)

If there is anything that the recent PISA results show, it’s that the promises by David Coleman, Bill Gates, Michelle Rhee, Betsy Devos, Arne Duncan, Barack Obama, and others of tremendous achievement increases and closing socioeconomic gaps with their ‘reforms’ were completely unfilled. I am copying and pasting here how American students have done on the PISA, a test given in many, many countries, since 2006. There have been tiny changes over the past dozen years in the scores of American students in reading, math, and science, but virtually none have been statistically significant, according to the statisticians who compiled and published the data.

Then again, nearly any classroom teacher you talked to over the past decade or two of educational ‘reforms’ in American classrooms could have told you why and how it was bound to fail.

Look for yourself:

PISA results through 2018

 

Source: https://www.oecd.org/pisa/publications/PISA2018_CN_USA.pdf

 

EDIT: I meant David Coleman the educational reform huckster, not Gary Coleman the actor!

 

More Educational Miracles (Not!)

I have prepared charts and graphs for 8th grade NAEP average scale scores for black, hispanic, and white students in various jurisdictions: the entire nation; all large cities; Washington DC; Florida, Michigan; and Mississippi.

You will see that there was a general upwards trend in math from about 1992 to roughly 2007 or 2009, but the scores have mostly leveled off during the last decade. I included Michigan, since that is the state where current Education Secretary Betsy DeVos has had the mo$t per$sonal influence, but that influence doesn’t look to be positive.

While it’s good that DC’s black students no longer score the lowest in the nation (that would be Michigan – see the first graph), there is another feature of my fair city: very high-performing white students (generally with affluent, well-educated parents) in its unfortunately rather segregated public schools, as you can see in the last graph. Naep 8th grade math, black students, various placesnaep math, hispanic, 8th grade, various places

naep 8th grade math, white students, various places

Not So Fast, Betsy DeVos!

I attended the official roll-out of the results of the 2019 National Assessment of Educational Progress (NAEP) a couple of days ago at the National Press Club here in DC on 14th Street NW, and listened to the current education secretary, Betsy Devos, slam public schools and their administrators as having accomplished nothing while spending tons of money. She and other speakers held up DC, Mississippi, and Florida as examples to follow. Devos basically advocated abandoning public schools altogether, in favor of giving each parent a “backpack full of cash” to do whatever they want with.

Some other education activists I know here in DC shared their thoughts with me, and I decided to look at the results for DC’s white, black, and Hispanic students over time as reported on the NAEP’s official site. (You can find them here, but be prepared to do quite a bit of work to get them and make sense out of them!)

I found that it is true that DC’s recent increases in scores on the NAEP for all students, and for black and Hispanic students, are higher than in other jurisdictions.

However, I also found that those increases were happening at a HIGHER rate BEFORE DC’s mayor was given total control of DC’s public schools; BEFORE the appointment of Michelle Rhee; and BEFORE the massive DC expansion of charter schools.

Here are two graphs (which I think show a lot more than a table does) which give ‘average scale scores’ for black students in math at grades 4 and 8 in DC, in all large US cities, and in the nation as a whole. I have drawn a vertical red line at the year 2008, separating the era before mayoral control of schools (when we had an elected school board) and the era afterwards (starting with appointed chancellor Michelle Rhee and including a massive expansion of the charter school sector). These results include both regular DC Public School students and the charter school sector, but not the private schools.

I asked Excel to produce linear correlations of the average scale scores for black students in DC starting in 1996 through 2007, and also for 2009 through 2019. It wasn’t obvious to my naked eye, but the improvement rates, or slopes of those lines, were TWICE AS HIGH before mayoral control. At the 4th grade level, the improvement rate was 2.69 points per year BEFORE mayoral control, but only 1.34 points per year afterwards.

Yes, that is a two-to-one ratio AGAINST mayoral control & massive charter expansion.

At the 8th grade level, same time span, the slope was 1.53 points per year before mayoral control, but 0.77 points per year afterwards.

Again, just about exactly a two-to-one ratio AGAINST the status quo that we have today.

pre and post Rhee, 4th grade NAEP, black students in DC, nation, large cities

pre and post Rhee, 8th grade NAEP, black students in DC, large cities, and nation

Charter schools do NOT get better NAEP test results than regular public schools

It is not easy to find comparisons between charter schools and regular public schools, partly because the charter schools are not required to be nearly as transparent or accountable as regular public schools. (Not in their finances, nor in requests for public records, nor for student or teacher disciplinary data, and much more.) At the state or district level, it has in the past been hard or impossible to find comparative data on the NAEP (National Assessment of Educational Progress).

We all have heard the propaganda that charter and voucher schools are so much better than regular public schools, because they supposedly get superior test scores and aren’t under the thumb of  those imaginary ‘teacher union thugs’.

However, NCES has released results where they actually do this comparison. Guess what: there is next to no difference between the scores of all US charter schools on the NAEP in both reading and math at either the 4th grade or 8th grade level! In fact, at the 12th grade, regular public schools seem to outscore the charter schools by a significant margin.

Take a look at the two graphs below, which I copied and pasted from the NCES website. The only change I made was to paint orange for the bar representing the charter schools. Note that there is no data available for private schools as a whole.

public vs charter vs catholic, naep, math

If you aren’t good at reading graphs, the one above says that on a 500-point scale, in 2017 (which was the last year for which we have results), at the 4th grade, regular public school students scored an average of 239 points in math, three points higher than charter school students (probably not a significant difference). At the 8th grade level, the two groups scored identically: 282 points. At the 12th grade, in 2015, regular public school students outscored charter school students by a score of 150 to 133 on a 300-point scale (I suspect that difference IS statistically significant). We have no results from private schools, but Catholic schools do have higher scores than the public or charter schools.

The next graph is for reading. At the 4th grade, charter school students in 2017 outscored regular public school students by a totally-insignificant 1 point (222 to 221 on a 500 point scale) and the same thing happened at the 8th grade level (266 to 265 on a 500 point scale). However, at the 12th grade, the regular public school students outscore their charter school counterparts by a score of 285 to 269, which I bet is significant.

charter vs public vs catholic, naep, reading, 2017

 

 

“And forgive our debts, as we forgive those who owe us!”

The title of this post might remind you of part of the so-called Lord’s Prayer, which in English is usually rendered “And forgive us our trespasses, as we forgive those who have trespassed against us.”

This sounds like forgiving sins, but in Latin, which I studied for about six years, the prayer is really about forgiving debts:

“et dimitte nobis debita nostra sicut et nos dimittimus debitoribus nostris”

I don’t know enough Greek to be able to comment on the original meaning of the words as apparently written down in the New Testament in that language, but it is generally accepted that Jesus (if he really existed) spoke Aramaic – but only a few of his (alleged) words were recorded in that language, since the entire NT was written in Greek, not in Hebrew or Latin, and definitely not in English!

The following book makes the argument that forgiving debts, wholesale, was essential if you wanted to avoid stratification of society into a class of oligarchs and a class of everybody else, who were essentially little better than slaves. They make the point that compounded interest grows exponentially and without limit, but economic growth does NOT: it follows a logistic curve at best, which means that there are certain limits.

For example, while bacteria growing in a petri dish appear to grow exponentially for some hours, perhaps for a few days, eventually, there is no more uncontaminated agar for the bacteria to eat, and they start drowning in their own waste products. So despite what one learns in most Algebra classes (including my own), bacterial growth is in actually logistic, not exponential. However, unless debt is periodically forgiven – which seldom if ever happens these days – the debtors end up drowning in debt, as you might be able to discern from this little graph I made:

logistic versus exponential growth

I haven’t read the book, but the review is most interesting. Here is a quote:

Nowhere, Hudson shows, is it more evident that we are blinded by a deracinated, by a decontextualizedunderstanding of our history than in our ignorance of the career of Jesus. Hence the title of the book: And Forgive Them Their Debts and the cover illustration of Jesus flogging the moneylenders — the creditors who do not forgive debts — in the Temple. For centuries English-speakers have recited the Lord’s Prayer with the assumption that they were merely asking for the forgiveness of their trespasses, their theological sins: “… and forgive us our trespasses, as we forgive those who trespass against us….” is the translation presented in the Revised Standard Version of the Bible. What is lost in translation is the fact that Jesus came “to preach the gospel to the poor … to preach the acceptable Year of the Lord”: He came, that is, to proclaim a Jubilee Year, a restoration of deror for debtors: He came to institute a Clean Slate Amnesty (which is what Hebrew דְּרוֹר connotes in this context).

So consider the passage from the Lord’s Prayer literally: … καὶ ἄφες ἡμῖν τὰ ὀφειλήματα ἡμῶν: “… and send away (ἄφες) for us our debts (ὀφειλήματα).” The Latin translation is not only grammatically identical to the Greek, but also shows the Greek word ὀφειλήματα revealingly translated as debita: … et dimitte nobis debita nostra: “… and discharge (dimitte) for us our debts (debita).” There was consequently, on the part of the creditor class, a most pressing and practical reason to have Jesus put to death: He was demanding that they restore the property they had rapaciously taken from their debtors. And after His death there was likewise a most pressing and practical reason to have His Jubilee proclamation of a Clean Slate Amnesty made toothless, that is to say, made merely theological: So the rich could continue to oppress the poor, forever and ever. Amen.

The Math Teacher’s Job is Neither to Teach the Lesson, Nor to Help Individual Students Who are Struggling!

….but rather, to prepare a lesson from which ALL the students can learn!

… according to the way that Japanese math teachers are taught their craft, as described below. You will find that these methods, which include Lesson Study, are pretty much the exact opposite of American “Direct Instruction” or “Teaching Like A Champion.”  Given that nobody claims that Japanese students lag behind American ones in math or science, perhaps we in the US could profit from examining how other nations’ teachers do it. Note also that this description is of mathematics lessons in elementary school, not middle or high school.

Please read the following description and leave comments on what you think.

*************************************
From Tom McDougal. Lesson Study Alliance, Chicago [and brought to my attention by Jerry Becker. – GFB]
*************************************
It’s not the teacher’s job to teach the students!

By Tom McDougal

What?? You might be thinking. What else could the teacher’s job be but to teach?

The teacher’s job is to ensure that students learn, all of them, we hope, though we know we will usually fall short.

In Japan, most (elementary) math lessons are designed as  “teaching through problem solving” lessons (TtP). A teaching through problem solving lesson typically includes the following parts:

 
1.  introduce the problem
2.  explicitly pose the task for students
3.  students work on the task (5-10 minutes)
4.  share student ideas
5.  compare and discuss the ideas for the purpose of learning new mathematics
6.  summarize major points from the lesson
7.  student reflections

(There is sometimes overlap, and a back-and-forth between some of these, e.g. #4 & #5 may be combined.)

While students are working on the task (#3), the teacher walks around the room, monitoring their progress. Japanese educators have a term for this, kikkan shido, or  “providing] guidance between the desks.” They recognize that there are different ways to do kikkan shido, and it is often a subject of discussion in Lesson Study. During planning, for example, a team will usually discuss how – or whether  – the teacher should respond to a student who exhibits a particular misconception; during the post-lesson discussion, there may be argument about whether the kikkan shido was effective. And, it is considered a skill that new teachers need to develop.

Teachers who are inexperienced with TtP lessons often make an unfortunate error while doing kikkan shido: they see a student who is struggling, or who has done something wrong, and they stop and help that student. After several minutes the teacher moves on, encounters another student who is having trouble, helps that student, and so on. Then, suddenly, time is up, and the lesson ends.

There are at least four important drawbacks to this type of kikkan shido. First, as my description suggests, it uses up a lot of time. The teacher may never get around to all of the students, and other students who need help may never get it. Second, by addressing misconceptions privately rather than publicly, the teacher deprives other students of the opportunity to analyze those misconceptions and learn why they are incorrect. Any experienced teacher knows that certain misconceptions are very common, so when one student makes an error that stems from a common misconception, that offers an opportunity to “inoculate” other students against making the same error sometime later.

The third problem with tutoring students individually is that it conflicts with the whole premise of teaching through problem solving. You expect that some, or even all, of the students will have difficulty with the task; that’s why it’s called “problem solving” and not “practice.” Teaching through problem solving involves an expectation that students will have difficulty, but that the comparison and discussion phase will address their difficulties and that, by the end of the lesson, all (or almost all) of the students will have learned what they need to know.

And fourth, we want to help students learn to give viable arguments and to critique the reasoning of others, the third Standard for Mathematical Practice in the Common Core State Standards. To accomplish this, we need for students to share and discuss different, perhaps conflicting solutions. Students need to do the critiquing, not the teacher.

 
Of course, some errors are simply the result of sloppiness, or otherwise unrelated to the main learning goals of the lesson. So when the teacher sees an error while conducting kikkan shido, he or she has to decide: should this be addressed privately or publicly? What should I say to this student? Do I expect that, by the end of the lesson, this student will understand what he or she has done wrong? This is a tricky decision, and an important part of lesson planning is anticipating different student responses, correct and incorrect, and deciding ahead of time how to handle them.

Caring teachers naturally feel drawn to help struggling students: they feel like it is their duty to help those students right now. To counteract that impulse, I say, bluntly:

It is not the teacher’s job to teach the students. It’s the teacher’s job to create a lesson that teaches the students.

 

DC’s Black, Hispanic and White Students Progress on the NAEP Under Mayoral Control and Before – 8th Grade Reading

8th grade naep reading, DC, B + W + H

We are looking at the average scale scores for 8th grade black, Hispanic, and white students in DC on the NAEP reading tests over the past two decades. Ten years ago, Washington DC made the transition from a popularly-elected school board to direct mayoral control of the school system. Michelle Rhee and Kaya Henderson, our first and second Chancellors under the new system, promised some pretty amazing gains if they were given all that power and many millions of dollars from the Walton, Arnold, and Broad foundations, and I showed that almost none of their promises worked out.

In the graph above, the vertical, dashed, green line shows when mayoral control was imposed, shortly after the end of school in 2007, so it marks a convenient end-point for school board control and a baseline for measuring the effects of mayoral control.

For 8th grade black students in reading in DC, their average scale scores went from 233 in 1998 to 238 in 2007, under the elected school board, which is a (very small) rise of 5 points in 9 years, or about 0.6 points per year. Under mayoral control, their scores went from 238 to 240, which is an even tinier increase of 2 points in 10 years, or 0.2 points per year.

Worse, not better.

For the Hispanic students, scores only increased from 246 to 249 before we had chancellors, or 3 points in 9 years, or about 0.3 points per year. After mayoral control, their scores went DOWN from 249 to 242 in 10 years, or a decrease of 0.7 points per year.

Again, worse, not better: going in the wrong direction entirely.

For white DC 8th graders, it’s not possible to make the same types of comparisons, because there were not sufficient numbers of white eighth-grade students in DC taking the test during five of the last ten test administrations for the NCES statisticians to give reliable results. However, we do know that in 2005 (pre-mayoral control) white 8th graders in DC scored 301 points. And since the mayors and the chancellors took over direct control of education in DC, not once have white students scored that high.

Again, worse, not better.

Why do we keep doing the same things that keep making things worse?

==============================================

My previous posts on this topic:

  1. https://gfbrandenburg.wordpress.com/2018/04/20/comparing-dcs-4th-grade-white-black-and-hispanic-students-in-the-math-naep/
  2. https://gfbrandenburg.wordpress.com/2018/04/17/the-one-area-where-some-dc-students-improved-under-mayoral-control-of-education/
  3. https://gfbrandenburg.wordpress.com/2018/04/17/more-flat-lines-4th-grade-reading-for-hispanic-and-white-students-dc-and-nationwide/
  4. https://gfbrandenburg.wordpress.com/2018/04/17/one-area-with-a-bit-of-improvement-4th-grade-math-for-black-students-on-the-naep/
  5. https://gfbrandenburg.wordpress.com/2018/04/16/was-there-any-progress-in-8th-grade-math-on-the-naep-in-dc-or-elsewhere/
  6. https://gfbrandenburg.wordpress.com/2018/04/16/progress-perhaps-with-8th-grade-white-students-in-dc-on-naep-after-mayoral-control/
  7. https://gfbrandenburg.wordpress.com/2018/04/16/maybe-there-was-progress-with-hispanic-students-in-dc-and-elsewhere/
  8. https://gfbrandenburg.wordpress.com/2018/04/16/just-how-much-success-has-there-been-with-the-reformista-drive-to-improve-scores-over-the-past-20-years/

 

One Area With A Bit of Improvement: 4th Grade Reading for Black Students on the NAEP

If you’ve been following this series, you’ve noticed that pretty much all of the trends for all subgroups in 8th grade math and reading have been flat — essentially no change after 20 years and billions of dollars of ill-spent money on ‘reforms’ that most teachers have found to be utterly worthless.

This graph, however, for 4th grade students in reading in DC and elsewhere, does show a little bit of improvement in DC only for the period 2013-now. However, there have also been drops. Mostly, there is not much change at all. But if you look very carefully you can see a small increase during that time period – just as there were small increases for DC’s hispanic students BEFORE mayoral control.

How good are your eyes?

4th grade math naep black students dc + elsewhere 1998-2017

 

EDIT: My bad: this was reading, not math. It’s fixed now, and I edited some of the text. GFB, 4/17/2018, 2:12 pm

Was There Any Progress in 8th Grade Math on the NAEP in DC or Elsewhere?

The answer is, basically, no.

You can see for yourself. This time I am posting a graph and table for average math NAEP scale scores for 8th graders who were black, Hispanic, or white. Honestly, they show that billions of dollars spent in dubious schemes such as having students spend an enormous fraction of the school year doing test prep, firing teachers based on students’ test scores (either their own students or those whom they’ve never met) and turning over much of our public educational system over to billionaires and profiteers — it’s all been a failure. Based on their own yardstick — test scores.

Here are the graphs. Read them and either weep or get determined to do something better about this ‘reformster’ charade.

8th grade math naep black students 1996-2017

8th grade math naep hispanic students 1996-2917

8th grade math Naep scores, white students, 1996-2017

%d bloggers like this: