Just how flat ARE those 12th grade NAEP scores?

Perhaps you read or heard that the 12th grade NAEP reading and math scores, which just got reported, were “flat“.

Did you wonder what that meant?

The short answer is: those scores have essentially not changed since they began giving the tests! Not for the kids at the top of the testing heap, not for those at the bottom, not for blacks, not for whites, not for hispanics.

No change, nada, zip.

Not even after a full dozen years of Bush’s looney No Child Left Behind Act, nor its twisted Obama-style descendant, Race to the Trough. Top.

I took a look at the official reports and I’ve plotted them here you can see how little effect all those billions spent on testing;  firing veteran teachers; writing and publishing new tests and standards; and opening thousands of charter schools has had.

Here are the tables:

naep 12th grade reading by percentiles over time

This first graph shows that other than a slight widening of the gap between the kids at the top (at the 90th percentile) and those at the bottom (at the 10th percentile) back in the early 1990s, there has been essentially no change in the average scores over the past two full decades.

I think we can assume that the test makers, who are professional psychometricians and not political appointees, tried their very best to make the test of equal difficulty every year. So those flat lines mean that there has been no change, despite all the efforts of the education secretaries of Clinton, Bush 2, and Obama. And despite the wholesale replacement of an enormous fraction of the nation’s teachers, and the handing over of public education resources to charter school operators.

naep 12th grade reading by group over time

 

This next graph shows much the same thing, but the data is broken down into ethnic/racial groups. Again, these lines are about as flat (horizontal) as you will ever see in the social sciences,

However, I think it’s instructive to note that the gap between, say, Hispanic and Black students on the one hand, and White and Asian students on the other, is much smaller than the gap between the 10th and 90th percentiles we saw in the very first graph: about 30 points as opposed to almost 100 points.
naep 12th grade math by percentiles over time

 

The third graph shows the  NAEP math scores for 12th graders since 2005, since that was the first time that the test was given. The psychometricians atNAEP claim there has been a :statistically significant” change since 2005 in some of those scores, but I don’t really see it. Being “statistically significant’ and being REALLY significant are two different things.

*Note: the 12th grade Math NAEP was given for the first time in 2005, unlike the 12th grade reading test.

naep 12th grade math by group over time

 

And here we have the same data broken down by ethnic/racial groups. Since 2009 there has been essentially no change, and there was precious little before that, except for Asian students.

Diane Ravitch correctly dismissed all of this as a sign that everything that Rod Paige, Margaret Spellings and Arne Duncan have done, is a complete and utter failure. Her conclusion, which I agree with, is that NCLB and RTTT need to be thrown out.

 

Trends for DC & Charters & Nation in 8th grade NAEP reading scores, black students

Here we have yet another surprising graph showing how the scores for black 8th graders on the NAEP reading tests have been bouncing around for students in DC public schools, DC charter schools, DC as a whole, large US cities as a whole, and the nation’s public schools as a whole.

Tell me what you see:

dc, dcps, charters, national, black 8th grade reading, naep to 2013

What I see is that under the ‘leadership’ of Rhee and Henderson, African-american 8th graders enrolled in DC public schools (blue and purple line) are actually doing a bit worse than they did before mayoral control. However, the average scores for the their counterparts in DC’s charter schools (dotted orange line)  are rising quite rapidly and are now higher than the national averages for black 8th graders.

However, on the average, the scores for all 8th-grade black students in publicly-funded DC schools (black dashed line) on the NAEP since 2008 (when Rhee was installed – purple vertical line) seem to be following the trends that were in place before that date.

No wonder Henderson recently admitted that her administration had no real idea on how to make DCPS middle schools attractive to families. One might conclude that the DC African-American families and students who were motivated to do well in school have in many cases migrated to the charter schools, leaving the less-motivated ones behind.

As in my previous three posts, I had to do have my spreadsheet do some computation to calculate the scores for the charter schools. You can find the formula in my first two posts. I used the overall DCPS and charter school and DC total enrollments rather than the specific 8th-grade-level enrollments for each institution because the latter was too difficult to find and I suspected that it wouldn’t make a big difference. If anybody finds any errors, please let me know.

Trends on NAEP for 8th grade math, black students in DC, DCPS, DC charters, and nation

Yet another graph, this one showing how this year’s group of African-American 8th grade students did on the NAEP math tests in the regular DC public schools, in all DC publicly-funded schools, in the DC charter schools, in large cities across the nation, and in all US public school systems, going back to the early 1990s.

dc, dcps, charters, national - black 8th graders, math to 2013

As usual, I had to do a bit of algebra to calculate what the average charter school scores were in the post-Rhee era, since those are not explicitly given anywhere. I give the explanation in my previous two posts.

My previous results seem to disagree a bit with those produced by NCES (by a couple of points). Therefore I used their data instead of what I calculated; the graph above is new as of 1/6/2014.

I still make these conclusions:

(1) Since the establishment of mayoral control of the schools, as a whole, the overall average for DC students in publicly-supported schools is following just about the exact same trends that were established from 2000 through 2007.  As a result, math scores for DC’s African-American 8th graders are now equal to those in large cities across the nation, which is a positive development.

(2) The DC charter schools seemed to have siphoned off the more motivated black 8th grade students and their families; as a result, scores for students in the regular DC public schools at that level in math lag significantly behind those of their counterparts in the charter schools, whose scores now surpass those of black 8th graders n the nation’s public schools as a whole and also those in large urban school systems as well.

As usual, if anybody finds any errors in my work, please let me know by leaving a comment.

Trends in DC on the NAEP for 4th grade reading, black students only: regular DCPS, charter schools, and pre- and post-Rhee

Here is a graph showing how African-American 4th students have been doing over time in Washington DC public schools and charter schools. I have drawn a clear dividing line at year 2008, because the scores before that were under the influence of DC’s former school board and superintendents. After that time, DC has been under a chancellor answerable only to the mayor.

dc, dcps, dc charter, and national naep trends, 4th grade reading to 2013You may notice that the blue, black and purple lines separate after 2007. That’s because NAEP began reporting separate scores for DC’s regular public schools and for all publicly-supported schools, though not for the charter schools as a bloc. As a result, you have to do a little bit of linear algebra to calculate what the average scales were for the charter schools from 2009 onwards. (I used essentially the same equation that I did in the previous post. Please write me a note if you think I made an error.)

As usual, we can see that since the late 1990s and up until Rhee took over, the overall trend in all large cities, in the nation’s public schools, and in DC’s publicly-supported schools was upwards on this test. (Yes, I know, these are not scores that follow the same kids year after year, but for whatever reason, the group of kids answering these tests are in general getting more answers right every two years.) Before that, i.e. from 1992 to 1998, scores bounced around or went down.

After Rhee took over, those scores seem to have entered another bouncy period. In fact, in DCPS, the scores on this test in 2013 were only back up to the level of 2007. There is a clear demarcation between the scores in the charter schools (blue line) and the regular public schools. The line for the charter schools seems to follow the trend from 1998 to 2007.

If I knew nothing about the politics of EduDeform, I would wonder why the WaPo editorial board is claiming victory.

 

 

And another look at those NAEP TUDA scores for DCPS

For those who don’t like looking at graphs, this time I will let the NAEP TUDA authors speak for themselves. I copied, and paste here, what I think were their most important conclusions.

If you would like the short version, here it is: Gaps between ‘haves’ and ‘have-nots’ (i.e. between whites and blacks, whites and hispanics, and the poor and non-poor) either grew or stayed the same.

That’s not good. And it’s completely at odds to the stated goals and claims of the educational “reformers” like Michelle Rhee, Kaya Henderson, Arne Duncan, and all the rest of the billionaires who line their pockets.

All of the rest, except for my notes in black italics, is taken directly from the NAEP website.

Score Gaps for Student Groups:  Fourth-Grade Math, NAEP TUDA, DC Public Schools

  • In 2013, students who were eligible for free/reduced-price school lunch, an indicator of low family income, had an average score that was 49 points lower than students who were not eligible. This performance gap was wider than that in 2003 (21 points). [emphasis added]
  • In 2013, Black students had an average score that was 59 points lower than White students. This performance gap was not significantly different from that in 2003 (60 points).
  • In 2013, Hispanic students had an average score that was 51 points lower than White students. This performance gap was not significantly different from that in 2003 (57 points).

Score Gaps for Student Groups: Eighth-Grade Math, NAEP TUDA, DC Public Schools

 [There were not enough 8th-grade white students in DCPS in 2003 for NAEP to  be able to make a measurement. Now there are.]

  • In 2013, students who were eligible for free/reduced-price school lunch, an indicator of low family income, had an average score that was 42 points lower than students who were not eligible. This performance gap was wider than that in 2003 (18 points).
  • In 2013, Black students had an average score that was 62 points lower than White students. Data are not reported for White students in 2003, because reporting standards were not met.
  •  In 2013, Hispanic students had an average score that was 53 points lower than White students. Data are not reported for White students in 2003, because reporting standards were not met.

 

Score Gaps for Student Groups Fourth-Grade Reading, NAEP TUDA, DC Public Schools

  • In 2013, students who were eligible for free/reduced-price school lunch, an indicator of low family income, had an average score that was 58 points lower than students who were not eligible. This performance gap was wider than that in 2002 (25 points).
  • In 2013, Black students had an average score that was 68 points lower than White students. This performance gap was not significantly different from that in 2002 (60 points).
  • In 2013, Hispanic students had an average score that was 50 points lower than White students. This performance gap was not significantly different from that in 2002 (55 points).

 

Score Gaps for Student Groups, Eighth-Grade Reading

  • In 2013, students who were eligible for free/reduced-price school lunch, an indicator of low family income, had an average score that was 40 points lower than students who were not eligible. This performance gap was wider than that in 2002 (17 points).

More on those supposedly wonderful DCPS NAEP TUDA scores…

In this post, let us look at how the District of Columbia Public Schools fared on the Trial Urban District Assessment sub-set of the National Assessment of Educational Progress. I think you will agree that there has been no significant change in trends if you compare the pre-Rhee era and the post-Rhee era, which we are in now. None of these graphs were made by me: I merely copied and pasted them from the NAEP website, and added a little color and a few labels.

The next graph shows the average scores on the NAEP for 8th-grade math for DC Public Schools and for all large-city public school systems in the US. You will have to look very hard to notice any change in slope for the lower, blue line, which represents DCPS, on either side of the orange vertical line, which separates the pre-Rhee era from the post-Rhee era.

dcps and large urban public schools math 8th grade

The next graph shows the average scores on the NAEP for 8th-grade reading in DCPS and all other large urban school systems. There has been no large change in either the national scores or the local DCPS scores since 2002, but I guess the best we can say that after two periods of small declines after mayoral control was imposed in 2007, the scores actually went up a bit in 2013 in DC. However, DCPS students on the whole are a little farther behind other urban kids now, under Chancellor Henderson, than they were at any time in the era before Rhee. But the changes are not very large or significant.

dcps and large urban public schools 8th grade reading

dcps and large urban public schools math 4th grade

The previous graph shows average Math NAEP scores for fourth-graders in DCPS and all other urban districts. Do you really see any big changes in the trends for DCPS scores? They have been going up rather steadily since 2003… It’s nice to see that DCPS kids seem to be catching up with those in other cities, but that was happening anyway.

My last graph in this post is for fourth-grade reading. It looks like I forgot to draw the vertical line separating the pre-Rhee and post-Rhee eras. Draw it in yourself. Do you see evidence of the supposed miracles that getting rid of 90% of the veteran teachers and school administrators, and hiring enormous numbers of inexperienced, highly-paid central-office administrators, has caused?

I surely don’t.

dcps and large urban public schools reading 4th grade

In a future post, I will actually dive a little deeper and ask how much of these changes (or lack thereof) are due to changing demographics….

I will also attempt to tease out how the privately-run charter schools in DC compare..

Published in: on December 19, 2013 at 1:37 pm  Comments (6)  
Tags: , , , , , ,

What About those NAEP TUDA Scores for DC?

You may have read the article in today’s Washington Post where Education DEformer-in-chief Arne Duncan claimed that the DC NAEP TUDA scores were “great examples for the rest of the country of what can happen when schools embrace innovative reforms and do the hard work necessary to ensure that all students graduate ready for college and careers.”

Oh, really?

Let’s remember that those “innovative reforms” started with the 2009-10 school year, though Chancellor Rhee took over at the beginning of the 2007-8 school year and fired a few hundred teachers the next school year.

Whichever date you use, a casual glance at the graphs published by the National Assessment of Educational Progress (NAEP) in their Trial Urban District Assessment (TUDA) shows that all those expensive billionaire-directed reforms have had nearly no effect in the District of Columbia, except for one: gap between the haves and have-nots is growing wider, not smaller. Otherwise, trends that began in 2002 or 2003 have pretty much continued.

It makes me wonder: Is Arne Duncan merely stupid and can’t even read simple graphs, or is he just a highly-paid liar and shill for the billionaires who have succeeded in hijacking our school system and in eliminating tenure for teachers, eliminating art, recess, PE and music for millions of inner-city students?

You decide, after looking at these graphs which I lifted from the NAEP TUDA website. The “Pre-Rhee” and “Post-Rhee” markers and some color and labels for clarity were added by me. Otherwise, I didn’t change a thing, and I didn’t have to do any complicated digging or perform any statistical tricks whatsoever to find these graphs.

First, let’s look at how students in DC Public Schools fared at the fourth and eighth grade, in reading and math, as compared with each other. Meaning, how did kids at the 75th percentile (top quartile) do, compared to the kids at the median (50th percentile), and compared to the kids at the 25th quartile (bottom quartile), over the past decade or so.

4th grade naep dcps math tuda 2003-2013 by quartile

That was for fourth-grade math. All three of the green lines slant mostly up to the right, meaning their scores are improving, which is generally a good thing. But do you honestly see any big difference between the pre-Rhee years and the post-Rhee years? The only real difference I see is that the gap between the top scorers is getting gradually wider, which is NOT a good thing. The gap used to be about 39 points but is now 52 points.

The next one is for fourth-grade reading.

4th grade naep dcps reading tuda 2003-13 by quartile

I’m not even going to complain that the bottom-quartile students are now scoring slightly lower than they were in 2009, since I know there is a lot of small random variation from one year to year because of the small sample sizes. However, NAEP themselves claim that the reading scores for the 25h- and 50th-percentile kids this year are NOT significantly different from what they were going back 6 to 8 years. And we can see that the gap between the top scorers and  bottom scorers seems to be a lot wider now.

Some great progress, huh? Definitely worth subjecting teachers to a random-number-generator called IVA in order to fire them randomly for that!

Now let’s look at 8th graders:

8th grade naep dcps math tuda 2003-2013 by quartile

 

That previous graph was for 8th grade math students in DC public schools. Do you see any great changes in trends from the pre-Rhee era to the post-Rhee era. I surely don’t. Was this “change” worth getting rid of democratic local control of the school system?

Lastly, in this post, let’s look at the same sort of graph for 8th grade reading:

8th grade naep dcps reading tuda 2003-13 by quartile

Here, the big trend seems to have been a fairly large drop-off in scores for the bottom quartile right after Rhee was anointed Chancellor, but those scores have almost reached the levels of 2002. Otherwise, no significant changes.

So, let me repeat the question:

Is Arne Duncan merely stupid, or just a liar?

 

Published in: on December 19, 2013 at 11:30 am  Comments (4)  
Tags: , , , , , ,

More on the 2013 NAEP

I would like to present some more results from the latest batch of released scores from the 2013 National Assessment of Educational Progress, or NAEP, so you can judge for yourselves.

As usual, the charlatans and quacks who are guiding US educational policy today claim that the results are clear proof that their ill-considered policies are working miracles, especially in the District of Columbia, my home town.

I claim that there has been no miracle. Yes, scores on the NAEP reading and math scores in the 4th grade and 8th grade are gradually but unevenly increasing — as has been the case for the past twenty years or so. But there has been no Rhee/Kamras/Henderson miracle in DC, or at least not one we can see on these graphs — no huge, enormous jump that trumps all growth prior to their mayoral takeover of the DC public schools.

Plus, we don’t yet know what weight the NAEP statisticians give to the scores of the kids in the regular public schools, those in the private or religious schools, or those in the charter schools. We do know that the proportion of white students counted in DC has increased substantially since the 1990’s, and that the proportion of black kids has shrunk, but we can only guess just what that means.

For each graph, I have drawn a thick, red, vertical line to distinguish the pre-“Rhee-form” era from the Era of Excellence and Data. See if you honestly see significant differences.

First, average NAEP math scores by states for 8th grade kids, 1990-2013. Remember, please, this is public AND private schools. I chose the states because they were the highest- or lowest-scoring ones in the nation (MA & MS) or because they were located near DC.

Fixed average 8th grade naep MATH scores by jurisdiction 1990-2013

Next, average NAEP reading scores for 4th graders:

fixed aveage 4th grade reaqding naep by states 1993-2013And lastly, average NAEP reading scores for 8th graders:

fixed average 8th grade naep reading scores by jurisdiction 1990-2013Remember: Mississippi, in an ugly shade of green on these graphs, is the lowest-performing state on both math and reading, and DC is still behind it.

 

 

 

Published in: on November 11, 2013 at 8:41 pm  Comments (8)  
Tags: , , , , , , ,

The 2012 Long Term NAEP Report Is In. Can You Spot the Miracle?

The National Assessment for Educational Progress has been giving assessments to 9-, 13-, and 17-year-old American students in reading math since 1971, about every four years, more or less. That’s over four decades worth of data.

They just released their latest report, which includes test data from 2012.

You know how the Michelle Rhees and Arne Duncans and Joel Kleins and Paul Vallases and Bill Gates and the Koch Brothers have taken nearly full control of our nation’s public schools in the past four years or so. They have been firing teachers and evaluating them on strict rubrics and arcane algorithms with test scores that no one understands, eliminating any union protections, hiring untrained and unqualified but “excellent” low-paid temporary teachers, closing regular public schools, opening charter schools, promoting vouchers, and generally bringing on a Brave New World where all the children are above average. And scores have supposedly been rising through the roof, to hear them tell it.

So all those wonderful attacks on veteran teachers, added to enormous cuts in educational funding in many districts, must have caused huge increases in test scores, right?

Uh, I said, Right?

No?

Well, I’ll let you see for yourself.

Actually, there has been no miracle whatsoever.

I reproduce a whole bunch of graphs from the NAEP report, and made a couple of charts from their data. I think you will be very hard pressed to find any big positive trends in the past four years.

naep ltt reading 9- 13- 17- year olds 1971-2012(You can click on any of these graphs to make them larger.)

The graph above shows the long-term trends in scores on the various math assessments given to 9-year olds, 13-year-olds, and 17-year olds (mostly but not always kids in grades 4, 8, and 11 or 12). There has not been a lot of change, frankly, but the tests have also changed in content and format, so I don’t really know how comparable they are over 40 years. With the current format, there was a bit of improvement from 2004 to 2008, but very little from 2008 to 2012. Certainly no “Michelle Rhee Miracle.”

The next one is the same thing, only for reading:

naep ltt reading 9- 13- 17- year olds 1971-2012Do you see this miracle spike we were promised in 2012? Me neither.

The next graph shows the gap in average reading scores between white and black 9 year olds. Here, the overall picture is that the gap is getting considerably narrower. Or, it WAS, until the DEformistas took over. It looks like the gap essentially stagnated in the past four years, under their enlightened despotism.

black-white reading naep ltt gaps 9 year olds

The next graph shows the gap in average reading scores for black and white 13 year olds:

black-white ach gaps 13 year olds reading naep ltt

Hmmm… looks like the gap’s getting wider now!?!

The next one is for reading again, black and white kids again, but for 17 year olds:

black-white ach gaps 17yo reading naep ltt

Finally a little bit of narrowing in the past four years — but NAEP statisticians carefully note that it’s not statistically significant, since the scores for neither group for 2008 had asterisks. Actually, that’s the case for the 13 year olds and the 9 year olds, too. No miracles.

And we gave Michelle Rhee and all the rest of her ilk all that money and fame — and they gave us NOTHING.

Published in: on July 1, 2013 at 8:21 pm  Comments (1)  
Tags: , ,

Details on those ‘Dozens and Dozens’ of Schools

As promised, I am posting the charts that I laboriously put together from data published by OSSE and a spreadsheet leaked to me, showing how unconvincing was the progress at the only 13 schools that WaPo reporter Jay Mathews could find that even vaguely resembled the ones that Michelle Rhee bragged about.

I am not going to look at the schools with large percentages of white students or at McKinley Tech.

First, here are my charts for Payne ES, Plummer ES and Prospect LC. I color-coded the chart much the way that Erich Martel does. That is, each diagonal sloping up to the right represents an individual cohort of students as they move from grade to grade, from year to year. Obviously I have no way of telling how many students transferred into our out of each cohort, but my experience in several DC public schools in all four quadrants indicates that students don’t move around all that much.

Thus, at Payne, under “Reading”, in the column for 3rd grade, in the row for 2010, you see the number 27. That means that at Payne, in the 3rd grade, in school year 2009-2010, about 27% of the students were ‘proficient’ or ‘advanced’ in reading according to their scores on the DC-CAS. The next year, most of those kids returned as fourth graders in the same school, and about 23% of them ‘passed’ the DC-CAS in reading because their answer sheets had enough correct answers for them to score ‘proficient’ or ‘advanced’. Note, that cell is also blue. But the next year, 2011-12, the percentage of students ‘passing’ the DC-CAS doubled, to 46%. I find that jump worthy of a red flag. Either that teacher did something astounding, or the students are an entirely different group of kids, or else someone cheated (most likely not the students).

payne plummer + Prospect

 

Any time I saw a drop or rise of 10% or more from the year before for a single cohort, I noted my “red flag” by writing the  percentage of passing scores in bold, red.

Notice that at Plummer, the cohort in blue, in reading, went fomr 40% passing to 18% passing to 46% passing in the course of three years. The cohort in green in math went from 60% passing to 18% passing to  29% passing.

At Prospect, the cohort in yellow goes from 25% passing to 0% passing to 5% passing to 5% passing in reading. In math, the same group goes from 13% to 31% to 0% to 24% passing.

You see anything weird with that? I sure do.

Next come Thomson, Tubman, Hart, and Sousa:

thomson tubman hart + sousa

 

The only one of these schools with a chart not covered with ‘red flags’ is Hart.

Your thoughts? (As usual, the “comment” button, below the end of this story, is tiny. Sorry about that.)

 

Follow

Get every new post delivered to your Inbox.

Join 409 other followers