It all makes sense now.
At first I was a bit surprised that Bill Gates and Michelle Rhee were opposed to publicizing the value-added data from New York City and other cities.
Could they be experiencing twinges of a bad conscience?
No way.
That’s not it. Nor do these educational Deformers think that value-added mysticism is nonsense. They think it’s wonderful and that teachers’ ability to retain their jobs and earn bonuses or warnings should largely depend on it.
The problem, for them, is that they don’t want the public to see for themselves that it’s a complete and utter crock. Nor to see the little man behind the curtain.
I present evidence of the fallacy of depending on “value-added” measurements in yet another graph — this time using what NYCPS says is the actual value-added scores of all of the many thousands of elementary school teachers for whom they have such value-added scores in the school years that ended in 2006 and in 2007.
I was afraid that by using the percentile ranks as I did in my previous post, I might have exaggerated or distorted how bad “value added” really was.
No worries, mate – it’s even more embarrassing for the educational deformers this way.
In any introductory statistics course, you learn that a graph like the one below is a textbook case of “no correlation”. I had Excel draw a line of best fit anyway, and calculate an r-squared correlation coefficient. Its value? 0.057 — once again, just about as close to zero correlation as you are ever going to find in the real world.
In plain English, what that means is that there is essentially no such thing as a teacher who is consistently wonderful (or awful) on this extremely complicated measurement scheme. How teacher X does one year in “value-added” in no way allows anybody to predict how teacher X will do the next year. They could do much worse, they could do much better, they could do about the same.
Even I find this to be an amazing revelation. What about you?
And to think that I’m not making any of this up. (unlike Michelle Rhee, who loves to invent statistics and “facts”.)
PS:
I neglected to give the links to where you can find the raw data. (Warning: some of these spreadsheets are enormous); Here they are:
GFBrandenburg's Blog 
Reblogged this on educatorvoices and commented:
If this data analysis by a New York teacher is based on real data as he claims then it suggests that rating teachers by the VAM scores is no more valid that raning/ rating teachers by their numerology score. perhaps we could campaign for this.
LikeLike
I’m stealing your post –giving you credit, but stealing it. I can’t guarantee people will link over here.
This is rather devastating to the claims of value-added, isn’t it?
Wow.
LikeLike
Reblogged this on Millard Fillmore's Bathtub and commented:
You wanted evidence that Michelle Rhee’s plans in Washington, D.C., were not coming to fruition, that the entire scheme was just one more exercise in “the daily flogging of teachers will continue until morale improves?”
G. G. Brandenburg ran the numbers. It isn’t pretty.
LikeLike
This is intriguing. Can you share your raw data sources?
LikeLike
Yes, sorry for not doing so earlier. Here is the URL link to the data:
In case you didn’t know, the raw data can be found here:
http://www.ny1.com/content/top_stories/156599/now-available–2007-2010-nyc-teacher-performance-data#doereports
or, if you prefer a shorter URL, try this one:
http://tinyurl.com/836a8cj
LikeLike
[…] Reblogged from GFBrandenburg's Blog: […]
LikeLike
Really good research, and I really like your hypothesis about Gates and Rhee. Maybe teachers should shift our tactics and release more of this data to show just how flawed it really is- kind of like the Chevy Volt- sounds great, but no once it is out everyone sees its limited value.
LikeLike
[…] ego invested to turn back now, which to me is insane. But check out Brandenburg’s post here and his analysis of the value added data. Share this:Like this:LikeBe the first to like this post. Tags: brandenburgh, data, edreform, […]
LikeLike
[…] GFBrandenburg’s blog explains why Bill Gates didn’t want the VAM scores made public. They’re total bullshit and blow the cover off of his reform […]
LikeLike
What Rhee and Gates will never figure out is the ultimate variables….students, their desires, their backgrounds, and their issues.
LikeLike
Here, here! (or is that hear, hear!) (Or should it be See! See!)
LikeLike
the perfect comment. game over.
LikeLike
What a mess. It looks like somebody threw up.
LikeLike
I interpret this result to mean that 6% of “Value Added” (whatever that is) would be associated with the combination of “teacher&school”. That means 94% would be attributed to everything else like “students”. Logic does not dictate blaming the teachers.
Personally, I think it’s poverty, and the data I’ve plotted on NC school performance vs. free lunch fraction bears this out.
I’ve also plotted SAT scores versus family income across North Carolina counties and posted it at:
Across the range of county average income, the SAT scores increase more than 150 points from low to high income. I also show data on school performance vs free lunch fraction, and high-free-lunch schools show just 60% of the performance of those with low-free-lunch fractions.
Any discussion of “standardized testing” has to include a discussion of “socioeconomic status”.
LikeLiked by 1 person
I don’t think Rhee and Gates were against LAT publication of teacher ratings; do you have sources for that? Now they are b/c of all the bad blow-back but I don’t recall them speaking out either way at the time.
LikeLike
You may be right. I have definitely read comments by them on the NYC data, and not on the LA data.
I’ll fix that.
LikeLike
As I said on Strauss’ blog on 2/26:
“I do believe this type of exposure is why people like Michelle Rhee were against publicizing the NY scores. She knows just how shaky they are, thus just how bogus the IMPACT system is that she and current Chancellor Henderson instituted here in DC.
They don’t want to have to defend actual data, because they know they can’t. Supposing some teachers were fired here for poor performance based on questionable VAM scores. Imagine all the bad publicity, just as reformers are trying to have these measures used across the country.”
http://www.washingtonpost.com/blogs/answer-sheet/post/a-simple-question-teachers-should-now-ask-about-their-profession/2012/02/26/gIQAwMplcR_blog.html#comments
LikeLike
These values do not tell the true story. There will be a different group of students from year to year. Culture, economic conditions, previous education or training will invalidate the data.
LikeLike
The kids in a given school generally do not change very much from one year to the next.
LikeLike
It is true that the overall population of a school won’t likely change a great deal from year to year. However, the population and dynamics of an individual class within that school can, and does, change dramatically. The mix of personalities, abilities and exceptionalities can vary widely.
LikeLike
Kids do change dramatically from year to the next. This is what is disturbing when we compare value-added data from year to year. Moreover, by so doing, we may be playing into Gates’ program. Pointing out the absurdity of the data, so obvious to anyone who has taught, does acknowledge, at least tacitly, that the data exists. What we should be saying (you are) is such data is meaningless. Good teachers do get best results, but the circumstances accompanying the kids makes any evaluation based on test scores dead on arrival.
I don’t know if Gates and company know this (anyone who thinks increasing class size helps educate probably does not) but the fact remains that he has way too much influence on American schools. You should also consider that the deck could be stacked against teachers who fail the sycophant biopsy. Their rolls could be stacked in a manner that insures teacher failure. I know of principals that do this. Proving it is another matter.
LikeLike
[…] found a blog from a retired math teacher and followed it. Share this:TwitterFacebookLike this:LikeBe the first […]
LikeLike
2007 scores were inflated. Thus (through no fault of your own) I think this graph is essentially meaningless.
Year to year value-added correlation is about .35 (range .30-.50). Year to career value added correlation is .6.
Not a meaningless tool. A misused one.
LikeLike
But these inflated scores are being used to falsely rate teachers, right?
Thus, the graph is incredibly meaningful to teachers.
In DC, we have some teachers being rated on scores inflated by administrator cheating and cover-up of the cheating. This type of deception can have a huge effect on teachers, especially when the scores plummet the next year when the cheating stops and teachers get blamed for for the declining achievement.
LikeLike
Meaningless for the purposes of abstract statistical analysis. Deeply harmful to publish for the reasons you state and many others.
LikeLike
Not sure if I’m convinced that VA is not a useful indicator (albeit as with every number trying to illustrate something complex it is flawed). You’re trying to find a trend over two years, but may be VA is like the weather: erratic over the short term but consistent over the long-term.
LikeLike
If we had some sort of medical test that was as unreliable as this VA stuff, we would say it was utterly useless. What good is a test for condition X if it has an error rate of 57% or more? None, and any doctor who put faith in it ought to have his/her head examined.
LikeLike
That’s not an accurate analogy – we’re talking about performance measures.
In medicine (btw, I’m from London, UK, so that’s my ref point), there are performance measures such as ‘waiting times’, or ‘surgery deaths per thousand’. These are also deeply flawed, but tell a story of some kind.
I think that VA has purpose, but probably not, as this debate is right to highlight, as an accountability measure; probably as a self-reflective tool and a school development measure (among many others). Eg a teacher might want to know the difference s/he has made compared to the previous year and reflect accordingly.
And if VA is going to be used for public reporting, first it shouldn’t be published teacher-by-teacher – it’ll just kills motivation and is flawed – but it could be collated school-by-school or district-by-district. VA often shows that schools in poorer areas are doing a better job than well funded private schools who get their kudos from the educationally rich home environment.
It is also worth diving into VA more. In the UK, until recently, we had Contextual Value-Added, which factored into the VA figure a metric for poverty and ESL. As you can imagine, the tough urban school often were found to perform the best in terms of the impact teachers were making on children.
There is another massive problem with all this data though and that’s how it effects teaching and learning. Publishing data based on test scores tells a very partial picture, but will influence the direction of teaching. In UK we’ve seen it in practices such as teaching to the test and targeting particular groups of children to ensure certain benchmarks are met.
LikeLike
I would love to hear more about what is happening in the UK with education. Sounds like some of the same misguided ideas on how to deform education are crossing the north atlantic in both directions. Do you know of any thoughtful, insightful blogs or websites on what is happening with schooling or social policy in the UK or elsewhere in Europe? I just started reading Pasi Stahlberg’s “Finnish Lessons” yesterday.
LikeLike
Glad to see this, the problem is that children are NOT a product, they cannot be treated with one treatment and are not meant to turn out the same way.
And if the poster above did think about this more, this is CHILDREN, not medical issues, how would you want your children to be educated?? Step into a school and see what it’s like, where is the creativity, the projects, the children’s voice etc. Why are these business people forcing our children through the schools to think the same way, respond the same way???
And when does society think about the poverty issue?? It is not just the fault of the school system, until we all take responsibility and face it, we will always have children who are facing this huge issue.
LikeLike
Sean — the Value-Added results are being used in DC to rate teachers, in some cases to ineffective or minimally effective, based on faulty data.
I hasn’t been published, but it has ruined some careers.
LikeLike
[…] these performance evaluations systems themselves do not perform as expected: Here it is, “Now I understand why Bill Gates didn’t want the value-added data made public“: It all makes sense […]
LikeLike
What would a chart look like, were a measure like this actually to work in distinguishing good from bad teachers?
LikeLike
It would look like a cigar- shaped blob that forms a 46-degree angle with the two axes, going thru the origin, from upper right to lower left.
LikeLike
[…] year essentially explains nothing about what’s going to happen next year. Or look here, or here, or here. Or here or here or here (by Gary Rubenstein) Would you trust a medical test of some […]
LikeLike
[…] on how this “value-added” model of teacher evaluations from these test scores is a “complete and utter crock” when analyzing the […]
LikeLike
How are you matching the teachers for that graph?
The xls files that I just downloaded don’t have unique teacher ids. In fact there is no teacher id in the Teacher 1 and Teacher 2 columns as there seems to be in the simple report forms (the pdf format reports).
thanks
LikeLike
From what I understand, each line represents a unique teacher.
If you have a PDF version of much the same data, I’d love to see it!!
LikeLike
Oh, never thought of something that simple. I now see that teacher names are included, just not where I expected them to be I would assume that one needs both first name, susrname and probably school to allow matching acros the years.
If I am correct the simple reports are just above the Raw data xls fijles assuming I have not gotten lost . In any case try http://www.ny1.com/content/top_stories/156599/now-available–2007-2010-nyc-teacher-performance-data#doereports
LikeLike
Couldn’t reply to the chain above, but in response to your request for UK based blogs on social policy and education:
This is a really good journalist’s blog: http://www.educationbynumbers.org.uk/blog/
This is a great researcher – best follow him on twitter:
http://web.me.com/dylanwiliam/Dylan_Wiliams_website/Welcome.html
This is good think tank that looks at a broad range of issues (I’ve filtered their research page to only show ed publications):
http://www.ippr.org/research-projects/12?megafilter=education
And this is these are the two main UK ed newspaper:
http://www.guardian.co.uk/education
http://www.tes.co.uk/
Also, if you’re on Twitter you can check me our here (https://twitter.com/#!/ShirazC) and see who from the UK I follow.
Hope that’s helpful!
LikeLike
[…] Cuomo, Arne Duncan, and Bill Gates all agree that it may not be best to make all teacher evaluations […]
LikeLike
[…] would Cuomo, Arne Duncan, and Bill Gates all agree that it may not be best to make all teacher evaluations […]
LikeLike
I was wondering if you ever considered changing
the structure of your website? Its very well written; I love
what youve got to say. But maybe you could a little more in the way of content so people could
connect with it better. Youve got an awful lot of text for only having one or two
pictures. Maybe you could space it out better?
LikeLike
[…] "It all makes sense now. At first I was a bit surprised that Bill Gates and Michelle Rhee were opposed to publicizing the value-added data from New York City and other cities. The problem, for them, is that they don’t want the public to see for themselves that it’s a complete and utter crock. Nor to see the little man behind the curtain." | by Scott Mcleod […]
LikeLike
[…] Looking at the same exact teachers, in the same exact schools, teaching the same subjects there was no correlation between teachers’ 2005-06 scores and their 2007-08 scores. There was no correlation between a […]
LikeLike
[…] Looking at the same exact teachers, in the same exact schools, teaching the same subjects there was no correlation between teachers’ 2005-06 scores and their 2007-08 scores. There was no correlation between a […]
LikeLike
Reblogged this on ΕΝΙΑΙΟ ΜΕΤΩΠΟ ΠΑΙΔΕΙΑΣ.
LikeLike
I just retired after 41 years as a chemistry/physics/math teacher. The first thing I thought of when I saw this graph is the probability diagram in quantum mechanics for finding an s orbital. It’s a sphere, and as we all know, a sphere is symmetrical, so it goes no place. And that’s where “value added” data should end up going.
LikeLike
Or into the “circular file”?
LikeLike
Statistics? Are they using statistics? Are they using research based theories to make sweeping changes? If so, I haven’t seen the studies.
LikeLike
[…] had no consistency from one subject to the next. See my blog (not all on NY City) here, here, here, here, here, here, here, here, here, here, and here. See Gary R’s six part series on […]
LikeLike
[…] Rubinstein and I have dissected the value-added scores for New York City public school teachers that were computed and released by […]
LikeLike