How well does Rhee’s “Broom” work?

So how well does Rhee’s broom actually work? You know, the one she is supposedly using to get rid of incompetent, veteran* administrators and teachers? To hear her say it, her policy has been a smashing success. But the truth is quite different.

You might remember the 2008 TIME magazine cover with a photo of Michelle Rhee, frowning, dressed in black, holding a broom in a classroom. Some people simply thought it meant that Rhee was an old-fashioned witch, about to get on her broom and ride around. I understand that sentiment, but I suspect that the author of the article meant that Rhee planned to clean out the school system by getting rid of veteran teachers and administrators.

According to Rhee’s numerous public statements, the only thing that really matters is standardized test scores. (Though they are probably the LEAST important things in the lives of most students, and, it is very ironic that Rhee herself is unable to produce any test scores whatsoever backing up her mythical “Baltimore Miracle.”)

Taking this at face value (even though many people have shown that most standardized tests produce little data that is actually valuable), I have done a little bit of analysis to see whether the schools where Rhee fired or replaced the principals actually did better on the DC-CAS than the schools where the principals were not replaced. Recall that Rhee has claimed many times that her approach has been extremely successful. (I was about to say, “a sweeping victory.”)

What I found is that both groups of schools included ones where the scores went up a lot from SY 2007-8 to SY 2008-9, and both groups of schools had members where the scores dropped about as much. To me, using Rhee’s own yardstick, it’s hard to find any big difference between the two groups of schools .

Don’t believe me? Look at the graphs, below.

This first graph, with a light green background, shows the changes in the percentages of students scoring “proficient” or better on the MATH portion of the DC-CAS from SY 2007-8 to SY 2008-9 (that is, from the year before last to last year), at schools that had the same principals both years. Each bar represents a single school, but there was no room to label them all. This is for ALL DC public schools for which there is data for both school years – elementary and secondary, but not charter schools, since Rhee is not in charge of them. I have no way of tracking individual students’ scores. These graphs just give the changes in percentages of students “passing” the DC-CAS.

As you can see, one school increased its percentage of ‘passing’ students by about 43%. In fact, something like 2/3 of these schools showed an increase in the proportion of students passing in math. A couple of schools had declines of around 24% in the percentages of students ‘passing’ the math test.

Now let’s compare that graph with one that shows the progress (or lack thereof) of the counterparts to these schools —  the ones where Rhee put in a new principal for the second year (SY 2008-09, i.e. last school year).

I see three main differences between these two graphs:

• The second graph has a background that is light orange or the color of canned salmon.  The first one has a light green background. (I changed the colors so that you and I can tell them apart; otherwise, it’s pretty tough.)
• The second, light orange graph has fewer bars. That’s simply because more schools kept their principals than had them replaced by Rhee for SY 2008-9.
• One of the bars in the first graph is a lot longer than any of the other ones. That was Birney ES. (Questions were raised by the DC State Board of Education about those scores.)

Now, for the similarities:

• In both graphs, which means in both groups of schools,  somewhere near  1/3 of the schools showed declines from the first year to the second year, because the bars point downwards into negative territory. (Counting and using a calculator, I find that 12 out of 33, or 36% of the schools with NEW principals, had drops in scores; and 24 out of 70, or 34% of the schools that did NOT change principals, had drops in scores. Not a significant difference in my opinion, but if you think it maters, then it’s one that favors the schools that did NOT change principals.)
• The bars are generally very similar in length in the two graphs.
• In both graphs, the scale on the vertical axis is the same, so you can compare the graphs without bias.

So, not really much difference at all in the two groups of schools in terms of changes in math scores. Maybe reading tells a different story? Take a look.

First, changes in reading scores over the same period of time as the other two graphs, in schools that kept the SAME principal for both years:

That graph has sort of a blue-gray background. Notice that about half the schools improved, and a little more than half the schools did worse during the second year. I also estimate that at about 25% to 30% of the schools (the ones near the center), there was very, very little change.

Let’s now look at the changes in reading scores at schools where the principal was REPLACED:

Differences?

• This last graph has a bright yellow background, not a pale blue-gray.
• The bars on this second graph are significantly shorter, both in the positive and in negative directions.
• One bar on the gray graph is a lot taller than any of the other ones. (Guess which school that was!**)
• In the yellow graph, the fraction of schools that did worse the second year (i.e. the ones that have bars pointing downwards) is a bit under one-half, not a bit over one-half. (17 out of 45, or 38% of the schools with new principals had drops in reading scores, versus 37 out of 70, or 53%, for the schools which kept their principals.)
• The fraction of schools that made essentially no change at al, one way or the other, appears to be smaller in the yellow graph.

So, all in all, even on Rhee’s own terms (where the only important thing is standardized test scores) her broom has so far made very, very little difference.

Which is not so surprising. After all, a broom is rather a blunt instrument.

=========================

* To some people, the terms “veteran” and “incompetent” are synonymous.  I disagree. It takes several years for a teacher to begin to become competent. (I’ve had my differences with a lot of local school administrators, but they also need a lot of experience to begin to become competent.)

* *Yes, Birney again.

========================

My data came from two sources.

• Mary Levy had very carefully compiled a spreadsheet with all of the names of all of the principals in all of the DC public schools going back to at least 2002, and emailed that list to me. I spot-checked it to make sure it was accurate, ad found that it was. Mary’s compilation saved me many hours of labor.
  
• For the scores, I used the data that you can find for yourself for each school on the OSSE-NCLB-DCPS website, which gives, among other things, the percentages of students scoring at various levels on the DC-CAS and its predecessors at all of the various publicly-supported schools in DC. I cut-and-pasted the data on AYP scores for the past 8 years into the previous spreadsheet. Then I had the spreadsheet do a bit of subtraction (scores in 2009 minus the scores in 2008), some sorting (by whether the same principal was kept or not, followed by the change in scores), and finally a bit of bar-graphing. That’s all.