What About Matthew Ladner’s Closer Look at DCPS NAEP scores?

Ladner says, essentially, that increased numbers of illegally excluded ESL students brought heroic DC chancellor Rhee to grief – that is, when NAEP no longer allowed them to be excluded. Making their scores count made his heroine look bad, he concludes.

His solution? Exclude them all and look what wonders appear! The poor little fourth-graders in DCPS made great strides  — nay, unprecedented, unequalled strides — under the great Michelle Rhee and Kaya Henderson, after all, and despite the nay-sayers.

My response?

Meh.

Despite all his statistical legerdemain – which I have not attempted to replicate, at least not  yet — I still don’t see any miracle occurring in years 2009 and 2011, not even using his own graph and data. What I see is a continuation of past trends. No miracle.

For your edification, I attach both his graph and the original graph. Ladner has clearly learned his lesson from the great “How To Lie With Statistics” — he has stretched the vertical axis to make it appear that the gains were big. He uses about twice as much room going from 180 to 215 as NAEP does in going from 170 to 220.

Look for yourself. And wonder why he didn’t also do this for 4th grade math, 8th grade math, and 8th grade reading.

The top one is from Matthew Ladner at Jay P. Greene’s blog; I added the vertical and horizontal lines.

The bottom one is from the NAEP TUDA site.

 

According to the data produced by Ladner, there was a rise of 27 points in 13 years from 1998 to 2011, a shade more than 2 points per year, or 4 points every 2 years. Some years a bit more (like from 1998 to 2002, when they rose by 12 points in 4 years), some times a bit less. What happened in 2009 and 2011 fits a pure-and-simple straight-line test for all the data quite well.

And, by the way, the folks at NAEP, who are way better statisticians than you, me, or Ladner, I bet, don’t think much of his argument. Although the official NAEP TUDA 4th grade scores for 2011 and 2009 are different from each other by 2 points, they don’t believe that they are significantly different, statistically speaking.

Published in: on January 25, 2012 at 2:44 pm  Comments (1)  
Tags: , , , ,

8th Grade TUDA NAEP Reading Results Also Show No Miracles Under Rhee, Henderson et al.

This post concerns the 8th grade NAEP TUDA reading scores for DC and other large cities.  As you may recall, according to Michelle Rhee’s authorized biographer and friend, here is how Rhee claims she could tell if an applicant for a principal position was good enough:

“What’s good to Rhee? If they arrived at their previous school with 20 percent of students reading on grade level and when they left, the number was 70 percent.” (page 132 of The Bee Eater)

Also recall what Rhee claimed to have done in Baltimore: “Over a two-year period, moved students scoring at the 13th percentile on national standardized tests to 90 percent of students scoring at the 90th percentile or higher.”

Keep that standard in mind when you see these graphs and tables.

First, where are DCPS students right now, after more than 4 years of Rhee/Henderson’s “radical reformist” regime? Way down near the bottom of the pack as measured by percentages of students “below basic”, according to NAEP:

But, you say, surely that pitiful figure of 54% of 8th graders reading on a “below basic” level has been getting better under Rhee and Henderson? Guess again, or else look at this graph for DCPS 8th grade reading results over time, and you will see that just the opposite is true, since it’s now worse than in 2002, 2003, 2007, or even 2009:

Now let’s look at those trends in DC again. As you can see, NAEP TUDA 8th grade reading scores are, in fact, slightly LOWER in 2011 than they have been AT ANY TIME while the TUDA study has been going on:

You can see also that while the scores for higher-income kids dropped in 2011, the gap between the richer and poorer students widened by quite a lot in 2009, and is unchanged in 2011 (in both years, it’s 31 points, as opposed to 15 to 21 points during the earlier years).

And let’s look at the gaps between scores of various ethnic groups in DC over time at the 8th grade reading level:

In the graph above, we lack a lot of scores for white students for many years, simply because there were not enough white DCPS students tested at those points in time to satisfy statistical reporting requirements. However, it is clear that the arrival of the protegees of Joel Klein and Michael Bloomberg has NOT benefited any of the 8th grade groups shown: not Hispanic students, not Black students, and not White students.

You might be thinking, “Well, perhaps these gaps are bad in DC, but they are worse elsewhere?”

Pretty much, NO. There are only two cities with larger 8th grade reading achievement gaps between the rich and the poor on the NAEP: Austin and Fresno.

Now let’s look at the gap between scores of white and black 8th graders in various cities, below. As you can see, DC’s gap (circled) is by far the widest such gap: 58 points.

And for Hispanics and whites?

(/sarcasm on) We beat Fresno and Austin! Yippee! DC has the biggest gap! (/sarcasm off)

Published in: on December 17, 2011 at 2:54 pm  Comments (9)  
Tags: , , , , , , ,

More on Gaps – This Time, 8th Grade Math NAEP TUDA

This year, the 8th grade math NAEP TUDA results are the only place where the scores in DCPS appear to be going up, but the gap between the top students and bottom students is getting significantly larger.

(Wait until we look at the reading results…)

If you need a little background information, take a look at yesterday’s post for a bit of explanation of what the percentiles mean.

I begin with a graph and table of the gaps in 8th grade math for all US public school students from 2003 to 2011.

As you can see, these scores, which are for all US 8th grade students in public schools, have been slowly but steadily rising since 2003. The gaps between the highest-achieving and lowest-achieving students have been fairly steady or else getting a bit smaller, if you look at the gap between those at the 75th percentile and those at the 25th percentile.

Now let’s look at a similar graph for students in large cities:

While these scores are also rising, I notice one thing that’s different from the previous graph: the kids at the 10th percentile in US large cities score extremely low. That’s the dark blue line at the bottom of the graph, separated from the other lines by a much larger gap than any I’ve noticed before. This phenomenon also shows up because the “90-10 gap” is much larger in this graph than in the previous one.

Now let’s look at the graph and table for Washington DC public schools. Recall that charter school students are excluded from this data starting in 2009.

Quite a different-looking graph, don’t you think? It’s almost like it’s beginning to open up to the right like a fan, because the top teal-colored line, which represents scores of the students at the top of the achievement scale (those at the 90th percentile), and the purple one just below it, are going up faster and faster, leaving the others behind. In 2009, after two years of Rhee, students at the 25th and 10th percentile dropped. To be fair, we don’t know if that drop is real, or is a result of the fact that so many students migrated to charter schools.

We do see that students at the 10th and 25th percentiles had their scores rise significantly in 2011 over 2009, so now they are a couple of points higher than they were in 2007, but that change is not statistically significant.

However, it is clear that overall, the gap in DCPS between the top scorers and the lowest scorers is widening. The 90-10 gap used to be 90 points back in 2003, but has risen to a new high of 107 points in 2011. The 75-25 gap has also reached a new high of 55 points, climbing from 48 points.

Now let’s look at the large city 200 miles to our northeast. I am referring to New York City, of course.

Unlike in DC, where we saw the lines getting farther apart as we move towards the present, in NYC the lines get closer together, which means that the gap between the high-achievers and low-achievers is getting smaller.

But not in a good way.

What is happening in New York’s public schools is that the scores of the students at the top and middle actually dropped over the past 2 years, while the scores of the students at the very bottom (10th percentile) continued to rise. That’s one way to narrow the gap, but it’s the wrong way.

So much for educational miracles happening under undemocratic control of the schools by Mayor Bloomberg! (Remember please that Bloomberg was the person who recommended Michelle Rhee to Adrian Fenty as another miracle-maker.)

Finally, let’s look at Atlanta.

Here we see a pattern that’s different from what we saw elsewhere. The various colored lines seem to be moving just about in lockstep with each other. The gap between the highest and lowest achievers hasn’t changed by much.

—————

Now go back and look at all of those scores.

Have the students in DC miraculously overtaken students in other large cities, or the nation, as essentially promised by the education DEform crowd? No.

Are they on the road to doing so? Heck, no.

Have Rhee, Fenty, Henderson, Bloomberg, Joel Klein, and Vincent Gray delivered any educational miracles in DC or New York? Definitely not.

Published in: on December 9, 2011 at 11:02 am  Comments (1)  
Tags: , , , , , , ,

Gaps between high- and low-achieving students in 4th grade math in DC and elsewhere

Using the NAEP TUDA data, we can see how various groups of students are doing in various cities and here in Washington, DC. In some places, the gap between the highest-achieving students and the lowest-achieving ones is remaining stable. In DCPS, the gap has been getting remarkably wider under the Deformist leadership of Michelle Rhee and Kaya Henderson.

I begin with several graphs and tables that I made, using data from the appendices to the 2011 TUDA report, for 4th grade mathematics over the time period from 2003 to 2011. NAEP gives the average scale score for students who are in the 90th percentile (which is nearly at the top of the charts); those at the 75th percentile (students who score better than about three-fourths of their peers); those at the 50th percentile (kids who are right at the median); those at the 25th percentile (3/4 of the students scoring BETTER than they do); and those at the 10th percentile (only 1 kid in 10 scores LOWER than they do).

We have this data for all 26 TUDA cities, for the nation’s public schools as a whole, and for all large cities added together. And of course for Washington, DC.

I decided to show you also graphs/tables for two other Eastern cities — ones for which we have data going back to 2003: Atlanta and New York.

First, let’s look at national public school data for 4th grade NAEP math scale score, by percentiles:

The top, light blue line represents average scores of kids at the 90th percentile. The bottom, darker blue line represents average scores of students at the 10th percentile.

As you can see, these scores have all been rising very slowly over time, but only by a few points since 2003. The gaps are tabulated in the bottom two rows of the tables by subtracting scores of those in the top ranks minus the scores of the students in the bottom ranks.

The “90-10 gap” means the differences in scores between those at the 90th percentile and those at the 10th percentile. Looking at the bottom two rows of the table, and at the relative distances between the lines in the graph, you can see that the gaps between the various groups have also essentially stayed the same on a national basis.

Now let’s look at a similar graph that’s for all large US cities (places with more than 250,000 population):

Here we see a very similar story, with two differences:

(1) the scores in all US cities seem to be rising faster than the scores in the nation as a whole

(2) the scores in all US cities are lower than those of the nation as a whole.

The gap between the highest-scoring and lowest-scoring students in the nation as a whole hasn’t really changed much over the past 8 years.

——————————————————

Now let’s look at Washington, DC. What differences do you notice?

I don’t know about you, but I notice that the scores for the lowest-scoring students in DCPS seem to be flat or going down over the past few years.

I also notice that the gap between the lowest-scoring and highest-scoring students in DCPS is getting significantly wider. This is not an illusion, but it is troubling.

While the top-scoring kids in DCPS score almost as well as the top-scoring kids nationally or in other large cities, our lowest-achieving students score much lower than their counterparts elsewhere.

—————————————————————-

Is this true in Atlanta and New York? Let’s look at Atlanta first.

In Atlanta, while the scores are generally increasing in all groups, the gap between the top-scorers and the bottom scorers seems to be getting a little wider.  But not nearly so much as in DC under Rhee and Henderson.

(Yes, we know that Atlanta had a major cheating scandal — like here in DC — on their NCLB tests, but this is a different test that is not administered directly by the schools or the teachers. Plus, students in any one classroom who take the test are taking several different forms of the test that gauge student knowledge on entirely different topics. So we can discount entirely the chance that these results are caused by cheating.)

Finally, let’s look at New York:

Isn’t this remarkable? New York has direct mayoral control of the schools, just like DC. Mayor Bloomberg and his current puppet school chancellor have seen scores go DOWN across the board over the past two years as they enact whatever crazy DEforms they are putting into practice. And the gaps between the high and low scorers seem to be getting a bit wider, too.

So, Billionaire-led educvational Deforms cause no more miracles in the Big Apple than they do in the nation’s capital.

Published in: on December 8, 2011 at 9:22 pm  Leave a Comment  
Tags: , , , ,

Possible Causes for DC’s Widening Achievement Gap

In my opinion, there are two possible causes. I would like to pose and discuss both of them.

(1) The group of students being tested, scored, and counted in DCPS on the NAEP has changed in nature.

or

(2) There have been major changes in curriculum and/or instructional practices in DCPS.

Let’s consider #1.

We know that this is definitely part of the story,  case, since the NAEP TUDA report (which you can find at http://nationsreportcard.gov/math_2009/math_2009_tudareport/ ) states that for 2009, TUDA no longer counted DC’s charter schools when giving detailed breakdowns of population groups, percentile ranks, and so on for DCPS. (I’ve tried to make this clearer by putting a note to this effect on every graph and table. )  They also point out in their technical notes that if they had also excluded the charter schools in 2007, then the average 8th grade math NAEP score for DCPS in 2007 would have been lower: 244, not 248. However, they do not provide us with enough information for us  to figure out what else would have changed. We are left with a good bit of speculation.

In spring 2009, according to my calculations, there were more 8th graders in regular DC public schools than in the DC charter schools ( 2531 to 1940). However,  higher percentages of students in the 8th grade charter schools scored “proficient” or “advanced” on the DC-CAS  in reading (56% as compared to 40%). So the actual numbers of students making AYP in the public and charter schools in reading in the 8th grade, according to my calculations, were 1,019 in the public and 1,093 in the charter schools – very close. We can infer from the NAEP technical note that in 2007, the 8th grade charter school students had again performed a bit better in math than the regular DCPS students. However, I don’t know off-hand the enrollment figures at that time for the two groups, and I also don’t know if NAEP does some sort of weighted average or not with different students or groups of students. So I really can’t extrapolate to say exactly what the average for regular DCPS students would have been in math for the 8th grade in 2007.

In the 4th grade, the situation is quite different. For one thing, NAEP does not inform us what the true 2007 4th grade regular-DCPS average math score would have been. However, I calculate that there were 3,303 students in the 4th grade  regular DC public schools in April 2009, of which about 48%, or 1,573  students, made AYP in reading on the DC-CAS (not the NAEP). In the charter schools, there were 1,254 students, of whom 492, or about 39%, made AYP in reading. So , unlike in the 8th grade, the 4th grade regular DCPS students did better than the 4th grade charter school students in reading. I think we can safely conclude that in 2007, the regular 4th grade DCPS students probably also did better in math.  If this conclusion is correct, then the regular DC public school students‘ average score in math for 4th grade in 2007 would have been somewhat higher than the 214 that is recorded for year 2007 on page 48 of the NAEP TUDA report.

Unfortunately, I have not done similar detailed breakdown calculations for every single grade level and subgroup at every single public or charter school for the math scores for 2009’s DC-CAS.  I only did them for reading. (And that, alone, took a huge amount of work.) I have done neither math nor reading for 2008, nor 2007, nor any other prior year’s DC-CAS scores.

As you probably know large percentages of students in DC are now attending charter schools, rather than regular public schools. I imagine that every time that DCPS administration does something stupid, then more parents probably decide to move their children to a charter school, hoping that they will do better there. (Despite the statistical evidence to the contrary.) And since there has been so much movement (and with record numbers under Rhee’s administration), the population of DCPS has been generally shrinking.

But why, and how, would this, all by itself, possibly make the achievement gap get larger? Let me try to explain by making up an example that has to do with space rocks.

Suppose a museum has a very large collection of meteorites (rocks from space). These rocks are quite expensive, and they are weighed (and priced) by the gram. One of the curators decides to put into a case a sub-collection, consisting of exactly one 1-gram space rock, one 2-gram space rock, one 3-gram rock, one 4-gram rock, and so on, all the way up to a single  100-gram meteorite. There are 100 rocks in the collection, as you can see below. The average weight of all of these rocks is the sum of all the weights, divided by the number of weights, or 5,050 grams divided by 100 rocks, or 50.5 grams per rock.

I have also shown where the 10th, 25th, 75th, and 90th percentile rocks are, which I will explain later.

Now, if you take out the ten smallest meteorites, the average will increase. Why? Because the total weight of all of the meteorites is now only 4995 grams, and the number of rocks is 90, and when you divide those you get an average of 55.5 grams per rock.  So, dropping the lowest ten rocks increases the average by 5 grams, and the locations of the 10th and 90th percentile rocks have moved as well, as you can see below.

What if the curator had taken out the 10 heaviest meteorites? The sum of all the rocks’ weights would now be 4,095 grams, and when we divide that by 90, we get an average of 45.5 grams per rock. So the average weight has dropped by 5 grams, as you see below, and the locations of the 10th and 90th percentiles have moved, also.

If we take out the 10 middle-sized rocks, that is, the ones from 46 through 55, then you would think that the average weight shouldn’t change. And you would be correct.  The sum of all the rocks’ weights is now 4,545 grams, and when you divide that by 90, you get 50.5 grams again.

But something definitely HAS changed – the middle has disappeared.

You might remember those graphs I made about percentile ranks in DCPS, the nation, and in other large cities. If not, you can look at some prior posts in this blog. Let’s agree that a thing is at the 90th percentile if it is greater than 90 percent of the other things in the group. If we are talking about heights of people, being at the 90th percentile probably doesn’t mean they are 90 inches tall! In fact, men who are about 6 feet, 2 inches are taller than 90% of all other men, so they are at about the 90th percentile.

However, in our example with the 100 original space rocks, the rock that weighs 91 grams has exactly 90 rocks that are lighter than it is, so I will say it’s at the 90th percentile. The rock that weighs 11 grams only has 10 rocks lighter than it is, so let’s agree that it’s at the 1oth percentile. The rock weighing 76 grams is at the 75th percentile, and the rock with a mass of 26 grams is at the 25th percentile.

Now let’s now take out the middle 40 rocks, much like charter schools have taken nearly 40% of the students in DCPS. So the remaining 60 rocks have masses of  1 gram through 30 grams, and then 71 grams through 100 grams. To find the rock that is at the 10th percentile, take 10% of 60. That’s 6. So for a rock to be at the 10th percentile, it only has to outweigh six rocks; those are the ones with masses of 1 g through 6 g. So the rock at the 10th percentile now weighs 7 grams. A rock at the 90th percentile has to have a mass greater than 90% of 60 rocks, which is 54 rocks. I get that  the rock weighing 95 grams is now the one at the 90th percentile. Similarly, I get that the rock weighing 16 grams is now the one at the 25th percentile, and that the rock at the 75th percentile is now the rock with a mass of 86 grams, as you see below.

In the original meteorite collection, as you can see in the first diagram I made, the gap in masses (or weights) between the 10th and 90th percentile was 91 grams minus 11 grams, or 80 grams. The gap between the 25th and 75th percentile is 76 minus 26, or 50 grams.

When we take out the middle 40% of the rocks, the average weight won’t change. But the gaps between the 10th and 90th percentiles HAVE changed: it’s 95 minus 7, or 88 grams, which is 8 grams larger than it used to be. And the gap between the 25th and 75th percentile is now 86  g minus 16 g, or 70 grams, which is 20 grams wider than it used to be.

However, if we just took out the lightest 40% of the rocks, the average mass of the remaining rocks would go up, but the intervals between the 10th and 90th percentiles, or etween the 25th and 75th percentiles, would get smaller.

My conclusion from all of this is that the charter schools are probably *not* really taking the very highest-performing students from DCPS. Nor are they taking the very worst-performing students from DCPS. I think they are taking the ones that are in the middle of the road, so to speak.

Next time: changes in instructional practices…

Published in: on December 22, 2009 at 4:15 pm  Comments (2)  
Tags: , , , , , , ,

Why, and how, has Michelle Rhee enlarged the Achievement Gap in DC?

Bill Turque’s recent article in the Washington Post (12-13-09) points out that under Michelle Rhee (but not previously), the gaps between white and black; between poor and non-poor; and between high-achievers and low-achievers among DCPS students have gotten noticeably larger. Please permit me to show how striking Rhee’s enlarged gap is, and to compare it to the rest of the country.

In fact, I conclude that the enlargement of this gap is Michelle Rhee’s singular achievement so far as Chancellor of the DC public school system, since she has so far not succeeded in breaking the Washington Teachers’ Union, which was probably her main goal, and since scores on the NAEP and DC-CAS were going up pretty steadily before she arrived.

First of all, let’s compare 4th grade students’ scores in DCPS in math on the NAEP, or National Assessment of Educational Progress, over the time period 2003 – 2009, for various percentile ranks. (Students at the 90th percentile do better than 90% of their fellow students, or cohort, so they are probably very good at math, compared to their DCPS peers. If you are at the 75th percentile, you do better than 75% of your cohort. And so on. Naturally, the students in the 25th percentile  in 2003 are not the same students who were at the 25th percentile in 2009; however, both students were right in the middle of the bottom half of their class. The scores are whatever the scaled score was that they got on the NAEP. And no, I don’t quite know what that means either. But a higher score on the NAEP is most likely better than a lower score. It appears that NAEP scores can range from 0 to 500. All my data comes from the lengfthy PDF file on the NAEP TUDA report, which is at http://nationsreportcard.gov/math_2009/math_2009_tudareport/ .

Notice that although all of the groups’ scores are generally increasing, the scores for the students at the 75th and 90th percentiles are going up a lot faster during the last 2 years. So the gap between the top and bottom students is widening in the 4th grade in DC, under Michelle Rhee’s watch.

Rhee’s defenders will probably object that this is happening everywhere in the country. But that is not true. Here is a table and graph of the exact same percentile ranks for the nation as a whole:

To make this a bit clearer, I also have made a table showing the gaps between the 90th and 10th percentiles in the nation as a whole, in all large cities, and in DCPS. Take a look at how much it has grown in DC in the past few years:

Next, let’s look at the 8th grade. The story is very similar, but there are differences. In 2009, the students in DCPS at the 25th and 10th percentiles actually did worse than their counterparts in 2007. And the gap widened.

What about the nation as a whole? Much like in 4th grade:

Now let’s compare gaps for 8th grade math in all big cities, the nation, and DC Public Schools:

Quite a disturbing pattern. Tomorrow: even more graphs and data, including the white-black gap and the poor-nonpoor gap!

Bill Turque’s article can be found at http://www.washingtonpost.com/wp-dyn/content/article/2009/12/12/AR2009121201276.html?nav=emailpage or http://tinyurl.com/turque12-13 .

Published in: on December 15, 2009 at 6:18 pm  Comments (8)  
Tags: , , , , ,
%d bloggers like this: