Regression to the mean?

It was suggested to me that the fact that very few teachers remained in the 90th percentile for two years in a row in NYC’s value-added madness (my description, not his) is simply yet another case of regression to the mean. That’s a phenomenon where very tall parents tend to have kids that are a bit shorter than they are, and very short parents have children that are taller than them.

Perhaps. But we definitely from ordinary observation that that height, hair color and skin color (but not tattoos or most illnesses) are rather inheritable: tall parents tend to have kids that are taller than most, and short parents have kids that are shorter than most, and so on.

But when the data is a blob showing almost no correlation at all, then regression to the mean doesn’t really mean the same thing. I mean, it starts to become just random variation. Yaknow whatI mean?

OK, let’s look at NYC again.

I just figured out how to get Excel to count some stuff for me in a neat and efficient manner. It counted for me all of the NYC public school teachers who were at or above the 80th percentile in the value-added measurement scheme they'[ve been using there for sy 0506; (That would be considered excellent.) I also had it check to see whether they were also in the 80th percentile during SY 0708, two years later. Or not.

If we trust my programming of Excel, there were exactly 161 such teachers who were in the 80th percentile rank or higher during both years.

But I also had it count how many teachers “dropped”, so to speak. I found there were 545 who were below the 80th percentile the second measured year. Oh, well, that’s nothing – that’s just regression to the mean, you say.

Well, what about those who go BELOW the mean the second year? That’s more than just regression to the mean. After all, the children of congolese pygmies do not get tall enough to play for the NBA. It would take a lot of intermarriage for several generations for that to happen (sorry about that, Bugsy Malone).

In New York city, I found that 146 teachers who were high-flyers in 2005-6 (at or above the 80th percentile) were distincly sub-par, scoring at or below the 50th percentile, two years later.

That’s close to the number of high flyers, nand is about 1/3 of those who “dropped.”

 

My conclusion:

This ‘value added’ stuff is worthless. It has no real predictive value. It doesn’t tell us anything we really want to know, even on its own terms. Plus, it’s measuring the wrong things — but that’s the subject for many more columns to come, and not just by me.