What Do the Latest NAEP Results Tell Us About Education “Reform” in Washington, DC?

The usual gang of supporters of bipartisan education “reform” never tire of telling the world how wonderful education ‘reform’ has been in Washington, DC, what with the proliferation of charter schools, Congressional support for vouchers, a seriously handicapped teachers’ union, tremendous churn of teaching and administrative staff, tons of consultants, and direct mayoral control.

I’ve been among those saying that the results are NOT so wonderful. I have documented how virtually none of the promises came true that Chancellors Michelle Rhee and Kaya Henderson made about 8 years ago. They promised that the improvements in test scores, graduation rates and much more would go through the roof, but in fact, almost none of that came to pass. The recent scandals about truancy, absenteeism, phoney grades and illegitimate graduation rates have shown that much of their supposed successes have been purely fraudulent.

In addition, I showed recently that in fact, progress for a number of DC’s subgroups (blacks, whites, and Hispanics) on the NAEP 4th and 8th grade reading and math tests are further evidence of failure, since improvement rates per year BEFORE mayoral control cemented the rule of our ‘reformista’ Chancellors wee BETTER THAN they were AFTERWARDS.

I was asked by one of the members of DC’s now-powerless board of education to analyze changes over time for ALL of DC’s students as a group (not subdivided in any way) to compare pre- and post-‘reform’.

I made my own graphs using the data on the NAEP Data Explorer page, being careful to use the same vertical scale in each case, and starting at the lowest point, or nadir, of DC’s NAEP scores back in the 1990s. I asked Excel to calculate and draw the line of best fit for the data points. In each case, that ‘trend-line’ of linear correlation fit the data extraordinarily well. In fact, the R-values of linear correlation went from a low of 94% to a high of 99%. I didn’t use the graphs that the NAEP Data Explorer page provided, because they changed the vertical scale from situation to situation – so a rise of, say, 10 points over 20 years would look just about the same as a rise of, say, 60 points over 20 years. And they aren’t! So my vertical (y-axis scale) is 200 points in each case.

I also marked on the graphs where the dividing line was between the time when we had an elected school board (abolished in 2007) and the present, when we have direct mayoral control with essentially no checks or balances on his or her power.

So here are the graphs:

4th grade math, ANSS, all dc, 1996-20174th grade reading, ANSS, all DC, 1998-20178th grade math, ANSS, all DC, 1996-20178th grade reading, ANSS, all DC, 1998-2017

So do you see any miracles?

Me neither.

So what does all of this that mean?

  1. You need a good magnifying glass to see any significant differences in progress on the NAEP test scores for ‘all students’ in Washington, DC when comparing the two eras. The slopes of the dashed lines of best fit are essentially identical on the two sides of the purple line.
  2. Since the proportion of white inhabitants of DC and of students in DC’s publicly-funded schools have both increased markedly in the past 10 years, and the proportion of black residents and black students have decreased markedly, and this has skewed the graph in a positive direction after 2007.* That means that this data, and these graphs, are actually making the overall situation look more favorable to the reformistas.
  3. Anybody pretending that there are huge increases in national test scores after the reformistas took over education in DC, is blowing smoke in your eyes.

===========================

*Why? When you remove low-scorers and add high-scorers (on anything) to a group, the overall average score will go up.

Here is a sports example: A football coach has been given a roster consisting of these players:

  • twenty big, strong, and bulky linesmen and backs and so on. Let’s pretend their average weight is 280 pounds.
  • twenty relatively small, but very fit, place-kickers (actually, they are soccer players looking for a fall sport) who weigh an average of 180 pounds each.

The team’s average weight is exactly 230 pounds (That’s (20*280 + 20*180) / 40) .

At noon,  the coach realizes there is no need for so many place-kickers, and she cuts 15 of the placekickers, leaving five of them. Their papers say that each one in fact weighs 180 pounds.

NOTHING ELSE CHANGES. In particular, none of the players gain or lose any weight during these fifteen minutes that the coach is making these changes.

At a quarter past noon, the average weight of the team has now increased markedly. It is now (20*280 + 5*180) / 25, or 260 pounds – it has gone up by 30 pounds simply by cutting 17 of its least-heavy players.

Is that coach a genius, or what, at bulking up her team?

Actually, although it’s not the direct result of what any Chancellor has done, this situation is somewhat similar to what’s happening in DC. Remember that white students in DC are the highest-scoring group of white students anywhere in the nation, because their parents overwhelmingly have graduate or professional degrees; DC’s white working class left town decades ago. So when relatively low-scoring African-American students (from working-class families) move to PG County, and white students and their relatively-highly-educated families move into DC from wherever, the averages will increase much as they did in my example with the imaginary football team.

Advertisements

Progress (or Not) for DC’s 8th Graders on the Math NAEP?

8th grade naep math, DC, w + H + B

Here we have the average scale scores on the math NAEP for 8th grade students in Washington, DC.* You will notice that we don’t have data for 8th grade white students in DC in math for the years 2003, 2007, and 2009, because there weren’t enough white students taking the test in those years for the statisticians at NCES to be confident in their data.

The vertical, dashed, red line near the middle of the graph represents the date when the old, elected, DC school board was replaced by Chancellors Rhee, Henderson, and Wilson, directly appointed by the various mayors. That year (2007) was also when Rhee and her underlings instituted brand-new teacher evaluation systems like IMPACT and VAM and new curricula and testing regimes known as Common Core, PARCC, and so on. Hundreds, if not thousands, of teachers were either fired or resigned or took early retirement. If these reforms had been as successful as Rhee promised in writing, then the lines representing scores for white, black, and hispanic students in DC would go slanting strongly up and to the right after that 2007 change.

I don’t see it.

Do you?

In fact, let’s look carefully at the slopes of the lines pre-Rheeform and post-Rheeform.

For black 8th grade students, scores went from 231 to 245 in the years 2000 to 2007, or 14 points in 7 years, which is a  rise of 2.0 points per year. After mayoral control, the scores for black students went from 245 to 257, or 12 points in 10 years. That’s rise of 1.2 points per year.

Worse, not better.

For Hispanic students, scores went from 236 in the year 2000 to 251 in 2007, a rise of 15 points in 7 years, or a rise of about 2.1 points per year. After mayoral control, their scores went from 251 to 263 in 10 years, which is a rise of 1.2 points per year.

Again: Worse, not better.

With the white students, a lot of data is missing, but I’ll compare what we have. Their scores went from 300 in year 2000 to 317 in the year 2005, which is an increase of 3.4 points per year. After mayoral control, their scores went from 319 in 2011 to 323, in 2017 or a rise of four (4!) points in 6 years, which works out to about 0.6 points per year.

Once again, Worse, not better.

Voters, you have the power to stop this nonsense, if you get organized!

—————————————————-

* Note that I’m using the numbers for Washington DC as a whole – which includes the regular DC Public School system, all the charter schools, as well as private (aka ‘independent’) and parochial schools. At one point, NAEP divided the DC scores into those for DCPS only (on the one hand) and for everybody else. In addition, they began to make it possible to separate out charter schools. However, since the regular public schools and the charter schools together educate the vast majority of students in DC, and the DCPS-only score-keeping started well after 2007, I decided to use the scores for all of DC because there was a much longer baseline of data, going back about twenty years.

=================================================

Here are my previous posts on this matter:

  1. https://gfbrandenburg.wordpress.com/2018/04/16/just-how-much-success-has-there-been-with-the-reformista-drive-to-improve-scores-over-the-past-20-years/
  2. https://gfbrandenburg.wordpress.com/2018/04/16/maybe-there-was-progress-with-hispanic-students-in-dc-and-elsewhere/
  3. https://gfbrandenburg.wordpress.com/2018/04/16/progress-perhaps-with-8th-grade-white-students-in-dc-on-naep-after-mayoral-control/
  4. https://gfbrandenburg.wordpress.com/2018/04/16/was-there-any-progress-in-8th-grade-math-on-the-naep-in-dc-or-elsewhere/
  5. https://gfbrandenburg.wordpress.com/2018/04/17/one-area-with-a-bit-of-improvement-4th-grade-math-for-black-students-on-the-naep/
  6. https://gfbrandenburg.wordpress.com/2018/04/17/more-flat-lines-4th-grade-reading-for-hispanic-and-white-students-dc-and-nationwide/
  7. https://gfbrandenburg.wordpress.com/2018/04/17/the-one-area-where-some-dc-students-improved-under-mayoral-control-of-education/
  8. https://gfbrandenburg.wordpress.com/2018/04/20/how-dcs-black-white-and-hispanic-students-compare-with-each-other-on-the-naep-over-the-past-20-years/
  9. https://gfbrandenburg.wordpress.com/2018/04/20/comparing-dcs-4th-grade-white-black-and-hispanic-students-in-the-math-naep/
  10. https://gfbrandenburg.wordpress.com/2018/04/20/dcs-black-hispanic-and-white-students-progress-on-the-naep-under-mayoral-control-and-before-8th-grade-reading/

DC’s Black, Hispanic and White Students Progress on the NAEP Under Mayoral Control and Before – 8th Grade Reading

8th grade naep reading, DC, B + W + H

We are looking at the average scale scores for 8th grade black, Hispanic, and white students in DC on the NAEP reading tests over the past two decades. Ten years ago, Washington DC made the transition from a popularly-elected school board to direct mayoral control of the school system. Michelle Rhee and Kaya Henderson, our first and second Chancellors under the new system, promised some pretty amazing gains if they were given all that power and many millions of dollars from the Walton, Arnold, and Broad foundations, and I showed that almost none of their promises worked out.

In the graph above, the vertical, dashed, green line shows when mayoral control was imposed, shortly after the end of school in 2007, so it marks a convenient end-point for school board control and a baseline for measuring the effects of mayoral control.

For 8th grade black students in reading in DC, their average scale scores went from 233 in 1998 to 238 in 2007, under the elected school board, which is a (very small) rise of 5 points in 9 years, or about 0.6 points per year. Under mayoral control, their scores went from 238 to 240, which is an even tinier increase of 2 points in 10 years, or 0.2 points per year.

Worse, not better.

For the Hispanic students, scores only increased from 246 to 249 before we had chancellors, or 3 points in 9 years, or about 0.3 points per year. After mayoral control, their scores went DOWN from 249 to 242 in 10 years, or a decrease of 0.7 points per year.

Again, worse, not better: going in the wrong direction entirely.

For white DC 8th graders, it’s not possible to make the same types of comparisons, because there were not sufficient numbers of white eighth-grade students in DC taking the test during five of the last ten test administrations for the NCES statisticians to give reliable results. However, we do know that in 2005 (pre-mayoral control) white 8th graders in DC scored 301 points. And since the mayors and the chancellors took over direct control of education in DC, not once have white students scored that high.

Again, worse, not better.

Why do we keep doing the same things that keep making things worse?

==============================================

My previous posts on this topic:

  1. https://gfbrandenburg.wordpress.com/2018/04/20/comparing-dcs-4th-grade-white-black-and-hispanic-students-in-the-math-naep/
  2. https://gfbrandenburg.wordpress.com/2018/04/17/the-one-area-where-some-dc-students-improved-under-mayoral-control-of-education/
  3. https://gfbrandenburg.wordpress.com/2018/04/17/more-flat-lines-4th-grade-reading-for-hispanic-and-white-students-dc-and-nationwide/
  4. https://gfbrandenburg.wordpress.com/2018/04/17/one-area-with-a-bit-of-improvement-4th-grade-math-for-black-students-on-the-naep/
  5. https://gfbrandenburg.wordpress.com/2018/04/16/was-there-any-progress-in-8th-grade-math-on-the-naep-in-dc-or-elsewhere/
  6. https://gfbrandenburg.wordpress.com/2018/04/16/progress-perhaps-with-8th-grade-white-students-in-dc-on-naep-after-mayoral-control/
  7. https://gfbrandenburg.wordpress.com/2018/04/16/maybe-there-was-progress-with-hispanic-students-in-dc-and-elsewhere/
  8. https://gfbrandenburg.wordpress.com/2018/04/16/just-how-much-success-has-there-been-with-the-reformista-drive-to-improve-scores-over-the-past-20-years/

 

Kaya Henderson Really Doesn’t Know How to Run a School System

DCPS Chancellor Kaya Henderson told the city two days ago, “I want to be clear. We know what we need to do, and we have what it takes to get it done.”

That is patently untrue.

Even by her own yardsticks, namely test scores, Henderson and her kind of ‘reform’ has so far been a complete failure; Continuing the churn-and-test-prep regime won’t make it any better

As I wrote in a comment on the article in the Washington Post:

All of Henderson’s boasts of continuous progress are completely bogus. 
 
If you look at the scores on the DC-CAS for every single subgroup, you can see that they have stagnated since 2009, which was the year before Rhee, Kamras and Henderson implemented their trademark reforms (IMPACT, TLF, VAM “merit pay” and eliminating seniority protections for teachers). The gaps between white students and hispanic or black students have NOT narrowed since that time. There were some increases from 2006-2009, but it’s not clear how much of that was due to adults cheating, or simply because students and teachers were adapting to a brand-new test. (You may recall that the DC-CAS was administered for the very first time in 2006, and the percentages of kids deemed ‘proficient’ dropped quite a bit in comparison to what they were under the old test, especially in math.) 
 
Also: out of the 78 measurable goals set by Rhee and four large foundations, in order to earn that $64.5 million grant in 2009, the DCPS leadership has achieved a mere one and one-half of those goals (and I’m being generous with the one-half). That is a success rate of TWO PERCENT. 
 
In other words, Rhee and Henderson have an almost perfect record of failure, none of which is publicized by the media (esp. not WaPo editorial staff) but is easy to see if you look at the official OSSE statistics and are willing to dig a little bit.  
 
I’ve done some digging and have made some pretty easy-to-understand graphs showing how much Rhee and Henderson have failed. Look at my blog, gfbrandenburg.wordpress.com , and in particular at http://bit.ly/10mna8c , http://bit.ly/10mneEY , and http://bit.ly/1ptal1K . 
 
After you read those blog posts, can you explain to me why Kaya Henderson still has a job? It is so clear that mayoral control has been a complete failure!

Latest DC Audited Enrollment Figures for all, charters, and regular public schools

The latest audited enrollment numbers have just been released, but not in a very useful format.

They show that regular DCPS enrollment is pretty close to flat, with only a small change over last year, or even over the last seven years. However, overall enrollment in all taxpayer-funded schools in the District of Columbia continues to rise, mostly because of a steady 15-year-long rise in charter school enrollment and a large increase in the overall city population.

The strangest feature I see is that the high school enrollment (grades 9-12) is down at all types of schools, with apparently many of those students moving to ‘alternative’ schools, at least on paper.

As I said, I didn’t think the graphs put out by OSSE were very informative, so I’ve re-plotted them here. For example, they put the charter school and public school enrollments on different graphs with different scales, making them hard to compare.

My first graph is of overall enrollment figures for regular public schools and for the charter schools (which several courts have decided are NOT public entities)  since the start of the millennium:

audited enrollment, dc public and charter schools, 2001-14

 

The red line is enrollment in the charter schools, and the blue line is that of the regular public schools. You can see that the blue line has been just about level since 2007-8, when Michelle Rhee was appointed chancellor of DCP.

My next graphs explores where the students are. OSSE separates students into various “bands” which are a bit hard to decipher. PreK3, PreK4, and Kindergarten totals are counted separately, and then they lump together grades 1-3 (‘primary’), then grades 4-5 (‘upper elementary’), then grades 6-8 (‘middle’), and grades 9-12 (high school). Students in alternative schools, of unspecified ages, are counted separately, as are students enrolled in Special Education schools and those in adult learning centers.

This first one is for regular DC Public Schools. You can see that preK3 though grade 3 comprises just under half of the entire DCPS population.

overall dcps - only enrollment by bands, 2013-4

 

The next graph shows the same thing but for ALL taxpayer-funded schools, both public and charter. Notice that the ‘adult’ sector is larger here.

overall dc osse enrollment by grade bands, 2013-4And the next graph shows the same thing for just the charter schools:

overall dc charter enrollment in percentages by grade bands 2013-4We see a much larger fraction of students in the adult sector. Again, Prek3 through grade 3 makes up just under half of the total.

Now let’s look a bit closer at the changes from last year to this, by grade band. My first graph shows overall changes from last year to this year, in all taxpayer-funded schools in Washington DC. Notice the large increase in the ‘alternative’ population and the ‘adult’ population, followed by a somewhat smaller rise in grades 1-3. The high school population – both public AND charter – actually dropped, as did the number of students enrolled in a special education school like Sharpe. It appears that a large fraction of that drop is students being reclassified as “alternative” instead of being in a high school.

increases, decreases by grade level, all DC OSSE schools, 2012-3 ri 2013-4Now let’s look at the corresponding graph for the regular DC public schools:

actual increases or decreases by grade level, DCPS only 2012-3 to 2013-4

 

Notice that once again, there was a big jump in the ‘alternative’ population, followed by an increase of about 250 at grades 1-3. As in overall DC stats, there was a drop in grades 9-12 and in special education. (the number for grade 6-8 is a typo: it should be 50)

Lastly, here are the changes since last year by grade band for the DC charter schools:

actual changes in enrollment, dc charter schools by grade bands, 2012-3 to 2013-4

 

I was surprised to see small drops in all of secondary charter schools (that is, grades 6 through 12). We see robust increases at all other levels, especially at the adult and alternative levels. I’m not exactly sure what’s causing this; perhaps readers closer to the trench lines than me (retired 5 years now) can comment.

My understanding  from reading US census figures is that the number of teenagers in Washington, DC – and thus, the number of students eligible to enroll in grades 6-12 continues to fall, while the number of younger kids is increasing. Obviously, those little kids generally grow older, and soon we will see a robust increase in the high school enrollment in the public and charter schools — unless they and their families all move out of town or decide to spend huge amounts for private or parochial schools. Which I doubt will happen.

In any case, claims of huge increases in enrollment in the DC public schools under chancellors Henderson and Rhee are just wishful thinking — like most of the boasts on Michelle Rhee’s famous resume.

 

Comments are most definitely welcome, even if you need a magnifying glass to see the “comments” button.

 

 

 

 

Published in: on February 28, 2014 at 3:41 pm  Comments (2)  
Tags: , , , , , ,

More on those supposedly wonderful DCPS NAEP TUDA scores…

In this post, let us look at how the District of Columbia Public Schools fared on the Trial Urban District Assessment sub-set of the National Assessment of Educational Progress. I think you will agree that there has been no significant change in trends if you compare the pre-Rhee era and the post-Rhee era, which we are in now. None of these graphs were made by me: I merely copied and pasted them from the NAEP website, and added a little color and a few labels.

The next graph shows the average scores on the NAEP for 8th-grade math for DC Public Schools and for all large-city public school systems in the US. You will have to look very hard to notice any change in slope for the lower, blue line, which represents DCPS, on either side of the orange vertical line, which separates the pre-Rhee era from the post-Rhee era.

dcps and large urban public schools math 8th grade

The next graph shows the average scores on the NAEP for 8th-grade reading in DCPS and all other large urban school systems. There has been no large change in either the national scores or the local DCPS scores since 2002, but I guess the best we can say that after two periods of small declines after mayoral control was imposed in 2007, the scores actually went up a bit in 2013 in DC. However, DCPS students on the whole are a little farther behind other urban kids now, under Chancellor Henderson, than they were at any time in the era before Rhee. But the changes are not very large or significant.

dcps and large urban public schools 8th grade reading

dcps and large urban public schools math 4th grade

The previous graph shows average Math NAEP scores for fourth-graders in DCPS and all other urban districts. Do you really see any big changes in the trends for DCPS scores? They have been going up rather steadily since 2003… It’s nice to see that DCPS kids seem to be catching up with those in other cities, but that was happening anyway.

My last graph in this post is for fourth-grade reading. It looks like I forgot to draw the vertical line separating the pre-Rhee and post-Rhee eras. Draw it in yourself. Do you see evidence of the supposed miracles that getting rid of 90% of the veteran teachers and school administrators, and hiring enormous numbers of inexperienced, highly-paid central-office administrators, has caused?

I surely don’t.

dcps and large urban public schools reading 4th grade

In a future post, I will actually dive a little deeper and ask how much of these changes (or lack thereof) are due to changing demographics….

I will also attempt to tease out how the privately-run charter schools in DC compare..

Published in: on December 19, 2013 at 1:37 pm  Comments (6)  
Tags: , , , , , ,

What I actually had time to say …

Since I had to abbreviate my remarks, here is what I actually said:

I am Guy Brandenburg, retired DCPS mathematics teacher.

To depart from my text, I want to start by proposing a solution: look hard at the collaborative assessment model being used a few miles away in Montgomery County [MD] and follow the advice of Edwards Deming.

Even though I personally retired before [the establishment of the] IMPACT [teacher evaluation system], I want to use statistics and graphs to show that the Value-Added measurements that are used to evaluate teachers are unreliable, invalid, and do not help teachers improve instruction. To the contrary: IVA measurements are driving a number of excellent, veteran teachers to resign or be fired from DCPS to go elsewhere.

Celebrated mathematician John Ewing says that VAM is “mathematical intimidation” and a “modern, mathematical version of the Emperor’s New Clothes.”

I agree.

One of my colleagues was able to pry the value-added formula [used in DC] from [DC data honcho] Jason Kamras after SIX MONTHS of back-and-forth emails. [Here it is:]

value added formula for dcps - in mathtype format

One problem with that formula is that nobody outside a small group of highly-paid consultants has any idea what are the values of any of those variables.

In not a single case has the [DCPS] Office of Data and Accountability sat down with a teacher and explained, in detail, exactly how a teacher’s score is calculated, student by student and class by class.

Nor has that office shared that data with the Washington Teachers’ Union.

I would ask you, Mr. Catania, to ask the Office of Data and Accountability to share with the WTU all IMPACT scores for every single teacher, including all the sub-scores, for every single class a teacher has.

Now let’s look at some statistics.

My first graph is completely random data points that I had Excel make up for me [and plot as x-y pairs].

pic 3 - completely random points

Notice that even though these are completely random, Excel still found a small correlation: r-squared was about 0.08 and r was about 29%.

Now let’s look at a very strong case of negative correlation in the real world: poverty rates and student achievement in Nebraska:

pic  4 - nebraska poverty vs achievement

The next graph is for the same sort of thing in Wisconsin:

pic 5 - wisconsin poverty vs achievement

Again, quite a strong correlation, just as we see here in Washington, DC:

pic 6 - poverty vs proficiency in DC

Now, how about those Value-Added scores? Do they correlate with classroom observations?

Mostly, we don’t know, because the data is kept secret. However, someone leaked to me the IVA and classroom observation scores for [DCPS in] SY 2009-10, and I plotted them [as you can see below].

pic 7 - VAM versus TLF in DC IMPACT 2009-10

I would say this looks pretty much no correlation at all. It certainly gives teachers no assistance on what to improve in order to help their students learn better.

And how stable are Value-Added measurements [in DCPS] over time? Unfortunately, since DCPS keeps all the data hidden, we don’t know how stable these scores are here. However, the New York Times leaked the value-added data for NYC teachers for several years, and we can look at those scores to [find out]. Here is one such graph [showing how the same teachers, in the same schools, scored in 2008-9 versus 2009-10]:

pic 8 - value added for 2 successive years Rubenstein NYC

That is very close to random.

How about teachers who teach the same subject to two different grade levels, say, fourth-grade math and fifth-grade math? Again, random points:

pic 9 - VAM for same subject different grades NYC rubenstein

One last point:

Mayor Gray and chancellors Henderson and Rhee all claim that education in DC only started improving after mayoral control of the schools, starting in 2007. Look for yourself [in the next two graphs].

pic 11 - naep 8th grade math avge scale scores since 1990 many states incl dc

 

pic 12 naep 4th grade reading scale scores since 1993 many states incl dc

Notice that gains began almost 20 years ago, long before mayoral control or chancellors Rhee and Henderson, long before IMPACT.

To repeat, I suggest that we throw out IMPACT and look hard at the ideas of Edwards Deming and the assessment models used in Montgomery County.

My Testimony Yesterday Before DC City Council’s Education Subcommittee ‘Roundtable’

Testimony of Guy Brandenburg, retired DCPS mathematics teacher before the DC City Council Committee on Education Roundtable, December 14, 2013 at McKinley Tech

 

Hello, Mr. Catania, audience members, and any other DC City Council members who may be present. I am a veteran DC math teacher who began teaching in Southeast DC about 35 years ago, and spent my last 15 years of teaching at Alice Deal JHS/MS. I taught everything from remedial 7th grade math through pre-calculus, as well as computer applications.

Among other things, I coached MathCounts teams at Deal and at Francis JHS, with my students often taking first place against all other public, private, and charter schools in the city and going on to compete against other state teams. As a result, I have several boxes full of trophies and some teaching awards.

Since retiring, I have been helping Math for America – DC (which is totally different from Teach for America) in training and mentoring new but highly skilled math teachers in DC public and charter schools; operating a blog that mostly concerns education; teaching astronomy and telescope making as an unpaid volunteer; and also tutoring [as a volunteer] students at the school closest to my house in Brookland, where my daughter attended kindergarten about 25 years ago.

But this testimony is not about me; as a result, I won’t read the previous paragraphs aloud.

My testimony is about how the public is being deceived with bogus statistics into thinking things are getting tremendously better under mayoral control of schools and under the chancellorships of Rhee and Henderson.

In particular, I want to show that the Value-Added measurements that are used to evaluate teachers are unreliable, invalid, and do not help teachers improve their methods of instruction. To the contrary: IVA measurements are driving a number of excellent, veteran teachers to resign or be fired from DCPS to go elsewhere.

I will try to show this mostly with graphs made by me and others, because in statistics, a good scatter plot is worth many a word or formula.

John Ewing, who is the president of Math for America and is a former executive director of the American Mathematical Society, wrote that VAM is “mathematical intimidation” and not reliable. I quote:

pic 1 john ewing

 

In case you were wondering how the formula goes, this is all that one of my colleagues was able to pry from Jason Kamras after SIX MONTHS of back-and-forth emails asking for additional information:

pic 2 dcps iva vam formula

One problem with that formula is that nobody outside a small group of highly-paid consultants has any idea what are the values of any of those variables. What’s more, many of those variables are composed of lists or matrices (“vectors”) of other variables.

In not a single case has the Office of Data and Accountability sat down with a teacher and explained, in detail, exactly how a teachers’ score is calculated, student by student, class by class, test score by test score.

Nor has that office shared that data with the Washington Teachers’ Union.

It’s the mathematics of intimidation, lack of accountability, and obfuscation.

I would ask you, Mr. Catania, to ask the Office of Data and Accountability to share with the WTU all IMPACT scores for every single teacher, including all the sub-scores, such as those for IVA and classroom observations.

To put a personal touch to my data, one of my former Deal colleagues shared with me that she resigned from DCPS specifically because her IVA scores kept bouncing around with no apparent reason. In fact, the year that she thought she did her very best job ever in her entire career – that’s when she earned her lowest value-added score. She now teaches in Montgomery County and recently earned the distinction of becoming a National Board Certified teacher – a loss for DCPS students, but a gain for those in Maryland.

Bill Turque of the Washington Post documented the case of Sarah Wysocki, an excellent teacher with outstanding classroom observation results, who was fired by DCPS for low IVA scores. She is now happily teaching in Virginia. I am positive that these two examples can be multiplied many times over.

Now let’s look at some statistics. As I mentioned, in many cases, pictures and graphs speak more clearly than words or numbers or equations.

My first graph is of completely random data points that should show absolutely no correlation with each other, meaning, they are not linked to each other in any way. I had my Excel spreadsheet to make two lists of random numbers, and I plotted those as the x- and y- variables on the following graph.

pic 3 - completely random points

I asked Excel also to draw a line of best fit and to calculate the correlation coefficient R and R-squared. It did so, as you can see, R-squared is very low, about 0.08 (eight percent). R, the square root of R-squared, is about 29 percent.

Remember, those are completely random numbers generated by Excel.

Now let’s look at a very strong correlation of real numbers: poverty rates and student achievement in a number of states. The first one is for Nebraska.

pic  4 - nebraska poverty vs achievement

R would be about 94% in this case – a very strong correlation indeed.

The next one is for Wisconsin:

pic 5 - wisconsin poverty vs achievement

Again, quite a strong correlation – a negative one: the poorer the student body, the lower the average achievement, which we see repeated in every state and every country in the world. Including DC, as you can see here:

 

pic 6 - poverty vs proficiency in DC

Now, how about those Value-Added scores? Do they correlate with classroom observations?

Mostly, we don’t know, because the data is kept secret. However, someone leaked to me the IVA and classroom observation scores for all DCPS teachers for SY 2009-10, and I plotted them. Is this a strong correlation, or not?

pic 7 - VAM versus TLF in DC IMPACT 2009-10

I would say this looks pretty much like no correlation at all. What on earth are these two things measuring? It certainly gives teachers no assistance on what to improve in order to help their students learn better.

And how stable are Value-Added measurements over time? If they are stable, that would mean that we might be able to use them to weed out the teachers who consistently score at the bottom, and reward those who consistently score at the top.

Unfortunately, since DCPS keeps all the data hidden, we don’t exactly know how stable these scores are here. However, the New York Times leaked the value-added data for NYC teachers for several years, and we can look at those scores to see.

Here is one such graph:

pic 8 - value added for 2 successive years Rubenstein NYC

That is very close to random.

How about teachers who teach the same subject to two different grade levels (say, fourth-grade math and fifth-grade math)? Again, random points:

pic 9 - VAM for same subject different grades NYC rubenstein

One thing that all veteran teachers agree on is that they stunk at their job during their first year and got a lot better their second year. This should show up on value-added graphs of year 1 versus year 2 scores for the same teachers, right?

Wrong.

Take a look:

pic 10 - VAM first yr vs second year same teacher rubenstein nyc

One last point:

Mayor Gray and chancellors Henderson and Rhee all claim that education in DC only started improving after mayoral control of the schools, starting in 2007.

Graphs and the NAEP show a different story. We won’t know until next week how DCPS and the charter schools did, separately, for 2013, but the following graphs show that reading andmath scores for DC fourth- and eighth-graders have been rising fairly steadily for nearly twenty years, or long before mayoral control or the appointments of our two chancellors (Rhee and Henderson).

 

 

pic 13 - naep reading 8th since 1998 scale scores many states incl dc

 

pic 12 naep 4th grade reading scale scores since 1993 many states incl dc

pic 11 - naep 8th grade math avge scale scores since 1990 many states incl dc

 

 

More on the 2013 NAEP

I would like to present some more results from the latest batch of released scores from the 2013 National Assessment of Educational Progress, or NAEP, so you can judge for yourselves.

As usual, the charlatans and quacks who are guiding US educational policy today claim that the results are clear proof that their ill-considered policies are working miracles, especially in the District of Columbia, my home town.

I claim that there has been no miracle. Yes, scores on the NAEP reading and math scores in the 4th grade and 8th grade are gradually but unevenly increasing — as has been the case for the past twenty years or so. But there has been no Rhee/Kamras/Henderson miracle in DC, or at least not one we can see on these graphs — no huge, enormous jump that trumps all growth prior to their mayoral takeover of the DC public schools.

Plus, we don’t yet know what weight the NAEP statisticians give to the scores of the kids in the regular public schools, those in the private or religious schools, or those in the charter schools. We do know that the proportion of white students counted in DC has increased substantially since the 1990’s, and that the proportion of black kids has shrunk, but we can only guess just what that means.

For each graph, I have drawn a thick, red, vertical line to distinguish the pre-“Rhee-form” era from the Era of Excellence and Data. See if you honestly see significant differences.

First, average NAEP math scores by states for 8th grade kids, 1990-2013. Remember, please, this is public AND private schools. I chose the states because they were the highest- or lowest-scoring ones in the nation (MA & MS) or because they were located near DC.

Fixed average 8th grade naep MATH scores by jurisdiction 1990-2013

Next, average NAEP reading scores for 4th graders:

fixed aveage 4th grade reaqding naep by states 1993-2013And lastly, average NAEP reading scores for 8th graders:

fixed average 8th grade naep reading scores by jurisdiction 1990-2013Remember: Mississippi, in an ugly shade of green on these graphs, is the lowest-performing state on both math and reading, and DC is still behind it.

 

 

 

Published in: on November 11, 2013 at 8:41 pm  Comments (9)  
Tags: , , , , , , ,

Graphs and Analysis of Trends in DC (regular and charters) from EdCORE to the DC Auditor

When the DC Mayor and City Council passed the law that got rid of the DC Board of Education and turned over the schools to the Mayor and his Chancellor, they also required there to be an audit, supposedly to see if things improved.

An audit has now been prepared by a group called EdCORE and submitted to the DC Auditor for the years 2006-7 through 2010-11, and it’s not very pretty.

So you won’t hear much about it, except, probably, from me.

Basically it shows that overall, there has been very little progress, even though the EduDeformers have been very successful in getting their way:

  •  Constant churn of teachers and administrators;
  •  Wholesale elimination of experienced personnel;
  •  Both parents and teachers have lost any ability to fight back against what they perceive as bad ideas;
  •  Near-collapse of regular public school system as parents, seeing no options, flee to charters that advertise constantly.

After six years now of the Rhee-Fenty-Henderson-Gray regime, let’s see what we get. Not much.

First, let’s look at the overall percentages of students scoring at or above the proficient level in both reading and math, as reported on the DC-CAS. The authors of the report caution repeatedly and strongly that since we know that there has been a lot of cheating by adults on the DC-CAS, these numbers may not be reliable at all.

I would also like to add that the very first year that DCPS used the DC-CAS was during SY 2006-7, at which point scores dropped drastically from previous years, when we had been using a different test altogether (The Stanford-9). It is common knowledge that when a school system starts using a new test, scores drop across the board, until teachers have learned how to prepare students for the new test; after a year or two, the scores rise a bit, then often stabilize.

That looks like what’s happened in DC, both in the regular public schools and in the charter schools.

In any case, there is no evidence of enormous progress in either one of these graphs; note that scores are essentially flat for the last three reported years. (Kind of a pity that they didn’t report on 2011-12.)

overall bb and P or A on dc-cas 2006 to 2011

The second graph shows the percentage of students scoring in the Below Basic category, which is the lowest level. Since 2008-9, there has essentially been no significant progress…

BY WARD

I’ve tried to make the next two graphs clearer than the ones given in the report. They show what percentage of students scored ‘proficient’ or better in the various wards of the city. If you know anything about Washington DC, you know that it’s mostly a very segregated town, both by income and by race. The very poorest students live in Wards 7 and 8, on the east bank of the Anacostia River, and are almost 100% African-American. The wealthiest and whitest and best-educated families live in wards 3 and 2.  (I live in Ward 5, which currently has no public middle schools at all!)

Since Rhee & Henderson and their cronies have replaced the vast majority of teachers and administrators in DC, replacing all of us with inexperienced, untrained, but “excellent” Teach for America temps and so on, then they must have CRUSHED that achievement gap between Ward Three and Ward Eight (i.e., between Spring Valley & Chevy CVhase DC on the one hand, and Barry Farms and Congress Heights on the other)?

Nope. See for yourself. I color-coded the wards. Do you see any real narrowing of those gaps? I don’t.

percentage of students at or above prof in reading by WARD

The next graph shows the percentage of students who are at the lowest reading level, “Below Basic”. If you understand real estate values, it is no surprise that the ward with the most expensive homes (Ward 3) continues to have the smallest fraction of students at the bottom reading level. And no surprise that the ward with the least expensive houses and apartments (ward 8) has the highest percentages of students at the bottom reading level.

It’s the same pattern all over the world, not just here in the US.percent below basic - reading - by ward

Next: teacher and administrator churn…

%d bloggers like this: