I just discovered another weird feature of New York City’s value-added process for teachers.
According to NYC’s own data, a teacher’s percentile ranks for “effectiveness” vary ON AVEARAGE, and for the SAME YEAR, SAME STUDENTS, SAME SUBJECT, SAME CLASS by over 50 percentile ranks.
In other words, a teacher at the median for this variability, about 57 points, could be scoring anywhere from the 20th percentile (very low) on up to the 77th percentile (rather high).
You probably think I’m making this up.
Let me give you some raw data and names to chew on.
The following teachers, who are named in the spreadsheets that were obtained by the NYC media, have scores as follows. All of these are NYC PS mathematics teachers. I give you the grade level, followed by what NYCPS says is their lowest value-added percentile rank for 2009/2010, and then their highest possible value-added percentile rank for the same year. In other words, they can’t tell how “good” these teachers really are, even by their own murky methodology.
These are not the exceptional, weird cases. The MEDIAN range of scores for the entire city is 57 points, and if you do a little subtraction, you will notice that in every single one of these cases, their top and bottom scores are 57 percentile points apart.
RHONDA DUFF BAPTISTE 5th Grade 7 64
KRISTIN DUNBAR 5th Grade 25 82
DANIELLE DUNNE 4th Grade 30 87
TONIA EDWARDS 4th Grade 28 85
ELAINE ELFOND 8th Grade 36 93
KATHLEEN ESTES MILANO 4th Grade 3 60
STEPHANIE FAIELLA 5th Grade 8 65
CORDELIA FAULKNER 5th Grade 7 64
GLORIA FEIERSTEIN 8th Grade 31 88
SCOTT FLATOW 4th Grade 29 86
MORGAN FLUSSER 8th Grade 23 80
DONNA FOSTER 6th Grade 8 65
PATRICK FOY 4th Grade 33 90
JENNIFER FRANCKLIN 4th Grade 4 61
ALIZA FUENTES 8th Grade 17 74
MARYANN GANCI 4th Grade 32 89
(By the way, I don’t know any of these folks, how old they are, what they look like, whether they are strict or lenient, give lots of homework, are tough graders, coach basketball, or anything else about them. But I know that they are real people, real teachers, college grads, and probably a lot like my and my fellow DC teachers except that many of them probably talk funny because they have Noo Yawk accents. Instead of talking normal like y’all do here in DC. ;=) They don’t need to be treated as if their life’s work revolves around a single number — one that nobody seems to be able to pin down very well, at that. Same thing with their students!)
OK, you might be thinking that this only applies to math teachers.
Same deal with English Language Arts teachers, as I show you here below, just like the list up above, which was for math teachers. Again, these teachers have the median (normal) differences between their highest and lowest possible value-added numbers, so they are not exceptional cases. THESE ARE THE TYPICAL CASES.
JULIE BOLAND 4th Grade 38 95
SHARON BOONE 4th Grade 3 60
JENNIFER BRANDES 8th Grade 0 57
SHARON CANNELLA 4th Grade 38 95
MONIQUE CARMICHAEL 4th Grade 40 97
ANATEA CARPENTER 7th Grade 41 98
REGINA CARROLL 5th Grade 0 57
CHRISTINA CASSASE 5th Grade 0 57
CHRISTOPHE CECIL 5th Grade 39 96
CHEZ DAVIS 4th Grade 1 58
JUANITA DAWSON 4th Grade 6 63
KAREN DOHERTY 4th Grade 2 59
MICHAEL DONOGHUE 8th Grade 41 98
JAIME DRAGOON 4th Grade 2 59
Let me emphasize that these are typical New York teachers. There are OVER ONE THOUSAND, FIVE HUNDRED TEACHERS WHOSE VALUE ADDED SCORES VARY BY 80 PERCENTAGE POINTS OR MORE.
If you’ve forgotten what a percentile rank is, it goes like this: if you are at the 10th percentile for height, that means you are only taller than 10% of your peers, and about 90% of them are taller than you. I.e., you are kinda short. If you at the 86th percentile for height, that means that about 86% of your peers are shorter than you, and you are only shorter than roughly 14% of your peers. In other words, you are rather tall.
If no-one can pinpoint your height any better than by saying you are somewhere between the 41st and 98th percentile, then they haven’t said diddly.
Still don’t believe me? Look at the exact same spreadsheet that I did, I posted it as a google doc at the following URL:
This range of values is probably their confidence interval, most likely one standard deviation on either side of the theoretical value. However, I don’t see where they actually state that, so I didn’t, either.