The 2010 DCPS VAM Mathematica Final Report, Leaked

An anonymous reader gave me a copy of the document that Sarah Bax (and others) have been looking for.

The one which Jason Kamras et al in DCPS Central Administration has been saying was so wonderful, but wasn’t available.

It was actually published by Mathematica, Inc. in the Spring of 2010.

It’s pretty long, and it’s in PDF format. I saved it here as a Google document so that anybody can attempt to read it and make sense of it:

I will warn you, it’s extremely heavy going, and it probably won’t help you understand VAM in DC any more (or less) than you already do (or don’t)!

Published in: on March 8, 2011 at 2:22 pm  Comments (7)  

The URI to TrackBack this entry is:

RSS feed for comments on this post.

7 CommentsLeave a comment

  1. You’re not supposed to understand it: you’re supposed to kneel before it, eyes averted.


  2. Michelle Rhee would have hated this model if she understood it. Why? Because it includes control variables for student attributes like reduced/free lunch eligibility. In other words, it assumes that low-income students will make less progress than better off ones, and consequently does not hold teachers of poor students accountable for achieving the same amount of “growth” as teachers of upper/middle class students.

    Of course, that’s a reasonable assumption — but poor Michelle would have had a royal hissy fit if she had known.


  3. @Reality Check, yes it includes free and reduced lunch, but there are some questionable choices made that could change the model significantly and they did not discuss what happens if you include or exclude race. While they didn’t want to label kids (I appreciate this as a black woman), they also ignore a substantive research base. Without knowing that their test has no racial bias when controlling for SES, I think it is important to include race.

    Also, I can’t believe they included students who had not been in a teacher’s class for the whole year. In my work, we exclude those students, and we exclude teachers whose classes have 80% or more students who have not been in their class for the whole year. Only someone who has never taught thinks that you can really measure a teacher’s “dose” for a student he or she has not had all year. They also may have never moved schools themselves during a school year. Mid-year school and class transitions are tough on kids and teachers and I would not include those students who moved around. My final concern is for using 2-3 year old pretest data as a control for the 10th grade teachers. Not to mention the equating issues for those tests, but that is crazy when you are talking about the age ranges 12-15 and not think there might be some huge, non-teacher effects going on there is naive. Spend some time with middle school and early high school students and then think you can have that kind of lag. They are practically different every week, much less from 2-3 years before.

    Overall, the model is OK, but the use is terrible. I know most of the teachers leave before you have multiple years of student data, but we all know the times that we have gotten a “great class” or a “bad class.” the research also demonstrates the instability of the measures.

    They basically acknowledge this by accepting that they are sure to 99.9% that their model predicts well at the extremes (top and bottom ~4%). But, that is where “sorting” of teachers is least needed. Everyone in the building knows which teachers really suck. Anyone who has ever taught knows they dread getting kids from Mr. X or Ms. Y because they are not good teachers and their kids come in not knowing things. Teachers generally don’t need this kind of fancy model to tell you that information. It is the 90% of teachers in the middle that they don’t report any confidence intervals on that worry me.


  4. There has to be some statistician or economist out there who can make some sense out of this business for the rest of us… ?


    • Actually, there is a good new book with a forward by Randi Weingarten that I highly recommend. The author is Douglas N. Harris and the book is Value-Added Measures in Education: What Every Educator Needs to Know. He really does a great job of explaining value-added measures in understandable terms, and then explains the trade-offs in some of the choices. It is fair and balanced (no relationship to the Fox version of such).


  5. Muy buen post, me ha gustado, gracias. Good Post. Thank you.


    • Gracias. Pero, has leido Mis otros postos?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: