An Article by the Late Gerald Bracey on Value-Added Measurement

This was published in The Huffington Post on May 1, 2007 06:20 PM. Gerald Bracey wrote it, not me. William Sanders is the statistician who has done the most to publicize the idea of Value-Added Assessment, the idea that if you look at a few test scores, you can tell exactly what a teacher has been accomplishing and what he/she is worth. (If you’ve been looking at my critique of the DC-CAS, you can tell that the DC-CAS isn’t worth much of anything….)

====================================================

I knew in advance that things would be weird at this debate, symposium, whatever with William Sanders. About three years ago in my Phi Delta Kappanresearch column, I summarized an article critical of Sanders’ value-added assessment model. Sanders visited upon me a two-hour telephone explanation-harangue of why I was wrong. He visited upon the editors a lonnnggg letter. On learning that I would be the other principal speaker, Sanders insisted that he have the last word. Otherwise, he wouldn’t come. Can you accept these terms, my host at North Carolina State wanted to know. I said I could.

Sanders made a presentation which had virtually nothing to do with anything that he is known for, which is using value-added assessment (VAA) to determine which teachers are effective. I had been less than enamored with this from the beginning since his first model used off-the-shelf items from McGraw-Hill’s CTBS. Wait, you’re using norm-referenced test items to pass judgments on teachers? Oh, please. In this talk, though, he did not consider the background knowledge of the listeners, most of whom were teachers hearing about value-added for the first time and one could almost see the bullets of jargon zipping past their ears.

A value-added model tests students at the beginning of the year and at the end. The change in test scores over the year is the “value” that has been added. The question then becomes: how much of this added value does the teacher account for (as opposed to what is added by parents, community, etc.)?

My points were these:

VAA makes more sense than the current successive-cohorts system for determining AYP. It makes more sense to follow kids over time, although if the goal remains 100% proficiency the whole operation remains nuts.

VAA is circular: it defines effective teachers as those who raise test scores, then uses test score gains to determine who’s an effective teacher.

Aside from Sanders, those working in VAA (Henry Braun, Howard Wainer, Dan McCaffrey, Dale Ballou, J. R. Lockwood, Haggai Kupermintz, from all of whom I had quotes) acknowledge that it cannot permit causal inferences about individual teachers. At best, it is a beginning step to identify teachers who might need additional professional development.

It is regressive in that it reinforces the idea that schools have teachers in boxes with 25 kids. Sanders claims his technique can deal with team-taught classes, but even if that is true, and he offered no data, it misses the dynamic of schools. As Kupermintz put it, “The TVAAS model represents teacher effects as independent, additive and linear. Educational communities that value collaborations, team teaching, interdisciplinary curricula and promote student autonomy and active participation may find [it of little use]. It regards teachers as independent actors and students as passive recipients of teacher ‘effects’…” In fact, as class size gets smaller, the TVAAS makes it harder for a teacher to look outstanding or ineffectual.

Sanders’ model improperly assumes that educational tests form equal-interval scales, but they do not and no amount of finagling with item response theory will fix that. On a thermometer, a true equal interval scale, the amount of heat needed to go from 10 degrees to 11 is the same as that needed to go from 110 to 111. On a test, it might require very different amounts of “achievement” to get from one point to another on different parts of the scale. Sanders believes that using NCE’s cures this (ha). It presumes that the teacher “effect” persists — like a diamond, it lasts undiminished forever. I’d like to run that by a few cognitive psychologists. It presumes that academic achievement is unidimensional.

And, perhaps most crucially, it presumes that students and teachers are randomly assigned to classes and overlooks that they are not. Many people choose a school by choosing where to live and within districts they sometimes choose a school other than the neighborhood school. Teachers with seniority get to choose what school or what classes they teach. They don’t usually choose hard-to-teach kids. And parents exert pressure–here, parents kill to get their kids into Pat Welsh’s high school writing classes. Big changes in test scores might well reflect these deviations from randomness as much as anything teachers do in their classrooms. Value-added models typically act as if this isn’t important. It is.

Worst, even ignoring its failures, value-added might not give stable results. An article by J. R. Lockwood and others in the Spring, 2007 issue of the Journal of Educational Measurement finds that, using a test that tests mathematical procedures, they could generate a list of effective teachers. Using a test of math problem solving they could generate a list of effective teachers. But they weren’t the same lists!

Value-added is currently being oversold. At the Battelle for Children website, one read, “Combining value-added analysis and improved high school assessments will lead to improved high school graduation rates, increased rigor in academic content, high college going rates less college remediation and increased teacher accountability.” And how many validity studies support these assertions?

Sanders’ 15 minutes of last word was a rambling, illogical lecture of the type a father might visit on a prodigal. The sponsors were embarrassed, the audience was pissed. At the reception that followed, for a while Sanders sorta took over a group I was talking with and I concluded that Sanders has an extremely limited yet extremely rigid idea of how schools work (his doctorate is in biostatistics and he worked with the Atomic Energy Commission and in agriculture until the late 80’s), rejects any conclusion counter to his own and, in spite of his age, somewhere around 75, is as defensive as any novice.

Published in: on August 15, 2010 at 11:37 pm  Comments (3)  

The URI to TrackBack this entry is: https://gfbrandenburg.wordpress.com/2010/08/15/an-article-by-the-late-gerald-bracey-on-value-added-measurement/trackback/

RSS feed for comments on this post.

3 CommentsLeave a comment

  1. Here are some additional references and a few followup questions:

    From Valerie Strauss’ Answer Sheet:
    Don’t rush to link teacher evaluation to student achievement My guest is Susan H. Fuhrman
    http://voices.washingtonpost.com/answer-sheet/national-standards/lets-not-rush-to-link-teacher.html#more

    (see Fuhrman’s bio at: http://www.tc.columbia.edu/faculty/index.htm?facid=shf2111 )

    Information about the design of DCPS’ IMPACT “value add” growth model:
    “Mathematica Designs Value-Added Model for the DC Public Schools”
    http://www.mathematica-mpr.com/Newsroom/Releases/2010/DCPS_VAM_7_10.asp

    http://www.mathematica-mpr.com/Education/value_added.asp

    Dr. Pallas and IMPACT was reasonably summarized at: Catalyst Ohio:
    http://www.catalyst-ohio.org/news/index.php?item=981&cat=38

    Original blog at Valerie Straus’ Answer Sheet:
    Were some DC teachers fired based on flawed calculations?
    http://voices.washingtonpost.com/answer-sheet/dc-schools/were-some-dc-teacher-dismissal.html

    Follow up at Eye on Education (blog posting by Prof. Pallas)
    http://eyeoned.org/content/accountability-in-d-c-public-schools-details-from-a-teacher’s-impact-report_71/

    also posted on The Answer Sheet here:
    http://voices.washingtonpost.com/answer-sheet/dc-schools/accountability-in-dcps-details.html#more

    Blog posting by Rick Hess at Ed Week:
    “Professor Pallas’s Inept, Irresponsible Attack on DCPS” (by Rick Hess)
    http://blogs.edweek.org/edweek/rick_hess_straight_up/2010/08/professor_pallass_inept_irresponsible_attack_on_dcps.html

    Follow up by Rick Hess at Ed Week: “Two Cheers for Professor Pallas”
    http://blogs.edweek.org/edweek/rick_hess_straight_up/2010/08/two_cheers_for_professor_pallas.html

    =====
    Some of the open questions about IMPACT’s value add growth model using DC CAS scores would include the following:

    o> Where is the ‘technical reference’ (or “white paper”) that shows it to be clearly documented and carefully and accurately communicated to teachers?

    o> Has this work by Mathematica been closely reviewed by other experts in the field and, if so, where is their technical write-up and critique?

    o> Where is it explained how unequally spaced scale scores from the DC CAS can be so cavalierly “subtracted” to measure growth?

    o> Where is the field testing of this new instrument that shows that it actually works?

    o> What other school systems are using the Mathematica statistical growth models and how accurate have they been shown to be? How consistent have the results been over time?

    o> Is there really sufficient research with real world data experience to show that a mathematical “value added growth model” performs well enough to base 50% (or 55%) of one’s performance rating and associated pay-scale adjustment on it? Especially with scores from last year versus scores from this year as compared to “an average class composed just like your class is composed this year” and account for all the non-teacher controlled factors (such as absences, emotional distress, illness, etc.) Many are not convinced of this nor have we seen ANY research or detailed description that even begin to attempt such an explanation emanate from DCPS (who clearly don’t understand it either). The “just trust us” mantra is no longer acceptable!

    Like

  2. Here are some additional references and a few followup questions:

    A relevantposting in Valerie Strauss’ Answer Sheet:
    “Don’t rush to link teacher evaluation to student achievement” My guest is Susan H. Fuhrman
    http://voices.washingtonpost.com/answer-sheet/national-standards/lets-not-rush-to-link-teacher.html#more

    (see Fuhrman’s bio at: http://www.tc.columbia.edu/faculty/index.htm?facid=shf2111 )

    Information about the design of DCPS’ IMPACT “value add” growth model:
    “Mathematica Designs Value-Added Model for the DC Public Schools”
    http://www.mathematica-mpr.com/Newsroom/Releases/2010/DCPS_VAM_7_10.asp

    http://www.mathematica-mpr.com/Education/value_added.asp

    “Because student growth models are used to judge the relative effectiveness of schools and teachers, their statistical foundations must be solid, clearly documented, and hold up to public scrutiny. Further, they must be communicated clearly to educators so that the findings are perceived as accurate and fair. Mathematica’s extensive statistical and analytic expertise can help school leaders develop student growth models and communicate their technical underpinnings through language and graphics that resonate with educators.”

    “To help educators interpret measures, we present information about estimation sample sizes, standard errors, and limitations of the data in a user-friendly format.”

    Dr. Pallas and IMPACT was reasonably summarized at: Catalyst Ohio:
    http://www.catalyst-ohio.org/news/index.php?item=981&cat=38

    Original blog at Valerie Straus’ Answer Sheet:
    “Were some DC teachers fired based on flawed calculations?”
    http://voices.washingtonpost.com/answer-sheet/dc-schools/were-some-dc-teacher-dismissal.html

    Follow up at Eye on Education (blog posting by Prof. Pallas)
    http://eyeoned.org/content/accountability-in-d-c-public-schools-details-from-a-teacher’s-impact-report_71/

    also posted on The Answer Sheet here:
    http://voices.washingtonpost.com/answer-sheet/dc-schools/accountability-in-dcps-details.html#more

    Blog posting by Rick Hess at Ed Week:
    “Professor Pallas’s Inept, Irresponsible Attack on DCPS” (by Rick Hess)
    http://blogs.edweek.org/edweek/rick_hess_straight_up/2010/08/professor_pallass_inept_irresponsible_attack_on_dcps.html

    Follow up by Rick Hess at Ed Week: “Two Cheers for Professor Pallas”
    http://blogs.edweek.org/edweek/rick_hess_straight_up/2010/08/two_cheers_for_professor_pallas.html

    ====
    Some of the open questions about IMPACT’s value add growth model using DC CAS scores would include the following:

    o> Where is the ‘technical reference’ (or “white paper”) that shows it to be clearly documented and carefully and accurately communicated to teachers?

    o> Has this work by Mathematica been closely reviewed by other experts in the field and, if so, where is their technical write-up and critique?

    o> Where is it explained how unequally spaced scale scores from the DC CAS can be so cavalierly “subtracted” to measure growth?

    o> Where is the field testing of this new instrument that shows that it actually works?

    o> What other school systems are using the Mathematica statistical growth models and how accurate have they been shown to be? How consistent have the results been over time?

    o> Is there really sufficient research with real world data experience to show that a mathematical “value added growth model” performs well enough to base 50% (or 55%) of one’s performance rating and associated pay-scale adjustment on it? Especially with scores from last year versus scores from this year as compared to “an average class composed just like your class is composed this year” and account for all the non-teacher controlled factors (such as absences, emotional distress, illness, etc.) Many are not convinced of this nor have we seen ANY research or detailed description that even begin to attempt such an explanation emanate from DCPS (who clearly don’t understand it either). The “just trust us” mantra is no longer acceptable!

    Like

  3. […] This post was mentioned on Twitter by Joe Bower, Joe Bower. Joe Bower said: An Article by the Late Gerald Bracey on Value-Added Measurement http://bit.ly/9BvCYI <<va […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: