Teacher VAM scores aren’t stable over time in Florida, either

I quote from a paper studying whether value-added scores for the same teachers tend to be consistent. In other words, does VAM allow us a chance to pick out the crappy teachers and give bonuses to the good one?

The answer, in complicated language, is essentially NO, but here is how they word it:

“Recently, a number of school districts have begun using measures of teachers’ contributions to student test scores or teacher “value added” to determine salaries and other monetary rewards.

In this paper we investigate the precision of valueadded measures by analyzing their inter-temporal stability.

We find that these measures of teacher productivity are only moderately stable over time, with year-to-year correlations in the range of 0.2-0.3.”

Or in plain English, and if you know anything at all about scatter plots and linear correlation, those scores wander all over the place and should never be used to provide any serious evidence about anything. Speculation, perhaps, but not policy or hiring or firing decisions of any sort.

They do say that they have some statistical tricks that allow them to make the correlation look better, but I don’t trust that sort of thing.  It’s not real.

Here’s a table from the paper. Look at those R values, and note that if you squared those correlation constants (go ahead, use your calculator on your cell phone) you get numbers that are way, way smaller – like what I and Gary Rubenstein reported concerning DCPS and NYCPS.

For your convenience, I circled the highest R value, 0.61, in middle schools on something called the normed FCAT-SSS, whatever that is (go ahead and look it up if it interests you) in  Duval county, Florida, one of the places where they had data. I also circled the lowest R value, 0.07, in Palm Beach  county, on the FCAT-NRT, whatever that is.

I couldn’t resist, so 0.56^2 is about 0.31 as an r-squared, which is moderate. There is only one score anywhere near that high 0.56, out of 24 such correlation calculations. The lowest value is 0.07 and if we square that and round it off we get an r-squared value of 0.005, shockingly low — essentially none at all.

The median correlation constant is about 0.285, which I indicated by circling two adjacent values of 0.28 and 0.29 in green. If you square that value you get r^2=0.08, which is nearly useless. Again.

I’m really sorry, but even though this paper was published four years ago, it’s still under wraps, or so it says?!?! I’m not supposed to quote from it? Well, to hell with that. it’s important data, for keereissake!

The title and authors are as follows, and perhaps they can forgive me. I don’t know how to contact them anyway. Does anybody have their contact information? Here is the title, credits, and warning:

THE INTERTEMPORAL STABILITY OF TEACHER EFFECT ESTIMATES *
by
Daniel F. McCaffrey;         Tim R. Sass;                         J. R. Lockwood
The RAND Corporation; Florida State University; The RAND Corporation
Original Version: April 9, 2008
This Version: June 27, 2008

*This paper has not been formally reviewed and should not be cited, quoted, reproduced, or retransmitted without the authors’ permission. This material is based on work supported by a supplemental grant to the National Center for Performance Initiatives funded by the United States Department of Education, Institute of Education Sciences. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of these organizations.

Published in: on March 30, 2012 at 3:58 pm  Comments (7)  
Tags: , ,

The URI to TrackBack this entry is: https://gfbrandenburg.wordpress.com/2012/03/30/teacher-vam-scores-arent-stable-over-time-in-florida-either/trackback/

RSS feed for comments on this post.

7 CommentsLeave a comment

  1. Thanks again for another informative post and keep up the good work.

    Like

  2. Products of educational evaluators never cease to amaze. Hook them up to the RAND corporation and stand back ! The Edushit is bound to fly !

    Like

  3. Actually a correlation of .56 isn’t that bad in the social sciences, which shows that the teacher accounts for about 30 percent of the variance in scores, if I recall my statistics. It would be reasonable to predict that the teacher does play a role here. The .61 in Palm Beach gives about a 36 percent variance, even better. I would need to see how the variables were measured. What is the mean VAM? What other measure is being correlated with VAM?

    Like

    • Yes, 0.56 isn’t that bad. But notice that there was only ONE correlation that came out that high, out of 24 correlations that they calculated. I notice with embarrassment that the highest correlation was actually 0.61, not 0.56.

      In any case, the median VAM correlation in this Florida data was 0.285, which means that the variance was about 0.08, which (IMO) is uselessly low.

      This study was primarily correlating VAM in year N with VAM in year N+1, for the same teachers, if I understand correctly.

      Like

  4. Have you looked at multiple years?

    Like

  5. […] 80% of ALL teachers should be fired after their first year of teaching) Plus RAND researchers found much the same thing in North Carolina. Also see this. And […]

    Like

  6. […] blog  (not all on NY City) here, here, here,  here, here, here, here, here,  here, here, and here. See Gary R’s six part series on his blog here, here, here, here, here, and here. As well as […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: