Regression to the mean?

It was suggested to me that the fact that very few teachers remained in the 90th percentile for two years in a row in NYC’s value-added madness (my description, not his) is simply yet another case of regression to the mean. That’s a phenomenon where very tall parents tend to have kids that are a bit shorter than they are, and very short parents have children that are taller than them.

Perhaps. But we definitely from ordinary observation that that height, hair color and skin color (but not tattoos or most illnesses) are rather inheritable: tall parents tend to have kids that are taller than most, and short parents have kids that are shorter than most, and so on.

But when the data is a blob showing almost no correlation at all, then regression to the mean doesn’t really mean the same thing. I mean, it starts to become just random variation. Yaknow whatI mean?

OK, let’s look at NYC again.

I just figured out how to get Excel to count some stuff for me in a neat and efficient manner. It counted for me all of the NYC public school teachers who were at or above the 80th percentile in the value-added measurement scheme they'[ve been using there for sy 0506; (That would be considered excellent.) I also had it check to see whether they were also in the 80th percentile during SY 0708, two years later. Or not.

If we trust my programming of Excel, there were exactly 161 such teachers who were in the 80th percentile rank or higher during both years.

But I also had it count how many teachers “dropped”, so to speak. I found there were 545 who were below the 80th percentile the second measured year. Oh, well, that’s nothing – that’s just regression to the mean, you say.

Well, what about those who go BELOW the mean the second year? That’s more than just regression to the mean. After all, the children of congolese pygmies do not get tall enough to play for the NBA. It would take a lot of intermarriage for several generations for that to happen (sorry about that, Bugsy Malone).

In New York city, I found that 146 teachers who were high-flyers in 2005-6 (at or above the 80th percentile) were distincly sub-par, scoring at or below the 50th percentile, two years later.

That’s close to the number of high flyers, nand is about 1/3 of those who “dropped.”

 

My conclusion:

This ‘value added’ stuff is worthless. It has no real predictive value. It doesn’t tell us anything we really want to know, even on its own terms. Plus, it’s measuring the wrong things — but that’s the subject for many more columns to come, and not just by me.

 

Published in: on March 5, 2012 at 10:27 pm  Comments (3)  
Tags: , ,

The URI to TrackBack this entry is: https://gfbrandenburg.wordpress.com/2012/03/05/regression-to-the-mean/trackback/

RSS feed for comments on this post.

3 CommentsLeave a comment

  1. Do you think they’re worthless in their present state of implementation, or worthless overall?

    Like

    • Worthless and harmful as they currently are being implemented.

      If we had simply let econometricians and psychologists do a number of carefully controlled experiments using a variety of measurement methods over a number of years in different locations with different tests and student bodies, then we could have gathered some data that might have been useful for informing public policy-making decisions. After careful analysis of what works and what doesn’t.

      Instead, we take wild-eyed exaggerated extrapolations by Erik Hanushek and a few others to form public policy. We are making something so totally unreliable as what we have in VAM being used to comprise 50% of the year-end evaluation of some teachers, and then firing many of those teachers summarily in the middle of the summer, with no recourse or appeal, based on what are clearly utterly unreliable numbers that are close to being random (not quite, but close). It’s kind of like having a principal who, every week, reads the latest fad educational article in some EdBiz publication and decides immediately to try to implement it in every classroom in her school.

      Oh, wait. I had a principal like that. I am SOOO glad I retired, and also am glad that she apparently got forced out not long thereafter.

      And we have teachers and local administrators all over the country being forced to cut out nearly all of the curriculum except for Multiple-Choice Math For Idiots and Brief Constructed Responses for Dummies.

      Like

      • Amen.

        Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: