“Capital Gains” Program Not Effective: Experimental Schools Do Worse in both Reading and Math than the Control Group

Bribing students for good grades and conduct doesn’t seem to work.

How well did the “Capital Gains” program work? In case you have forgotten, in the “Capital Gains” program, students at participating schools could earn substantial amounts of cash by having good grades, good attendance, and good conduct.

Was it effective in raising DC-CAS scores at those schools, in comparison with the “control” group of schools that did not take part in the program?

In one word: NO.

Apparently paying students hundreds of dollars doesn’t produce academic gains as shown on the DC-CAS.

Which raises a question – why isn’t anybody else saying anything about this?

And another question: since the experiment failed, why are we paying to do it again?

The simplest way to see the lack of results is to make some bar graphs that compare the growth (or decline) in math and reading scores when we separate the two groups of schools. Below, you can see two graphs. They have exactly the same number of schools and the same scale. The first one, which is in gray and brown, shows how much change there was in reading scores on the DC-CAS from 2008 to 2009 at the 14 “experimental” schools that were in the “Capital Gains” Program. The bar graphs show the changes in percentages of students at these schools who scored ‘proficient’ or better on the DC-CAS from spring of 2008 to spring of 2009.

Note that in the first graph, which is of the experimental group, nine middle schools saw declines in their reading scores: Browne, Garnett-Patterson, Whittier, Langdon, Takoma, Emery, Brightwood, Burroughs, and Jefferson. Only five middle schools had gains in reading: Hart, Miller, Hardy, Eliot, and Stuart-Hobson. Both the median and the average changes for this experimental group were negative: -1.98% for the median, and -1./71% for the average.

Now let’s look at the fourteen schools that were in the non-participating, control group. (I do applaud the makers of this experiment that they actually did have an experimental group and a control group – something you might remember from doing science fair experiments in middle school or high school.) In this green-and-aqua graph, we see that six schools had declines in scores: Marshall, Winston, Brookland, Johnson, Francis, and Deal. There were eight schools with positive growth: Ron Brown, Truesdell, Kramer, Lasalle, West, Shaed, Walker-Jones, and Sousa. The median growth for this group was positive: +2.28%, and so was he average: +1.26% more students scoring “proficient’ or ‘advanced’ on the DC-CAS from spring of 2008 to spring of 2009.

So the winner in the category of reading scores, is definitely the control group, where the students were NOT paid to be good.

Well, what about math? Again, it’s easiest to look at the graphs. Here they are, stating with the experimental group again:

In math, in the experimental group where the students were paid for good grades, behavior, and attendance, there were six schools where the percentage of students scoring ‘proficient’ or ‘advanced’ on the DC-CAS went down: Takoma, Burroughs, Browne, Langdon, Emery, and Garnett-Patterson. There were eight schools where there were increases: Hart, Miller, Jefferson, Eliot, Hardy, Whittier, Stuart-Hobson, and Brightwood. The median change for this group was a rise of 0.32% (hardly significant), and the average growth was +0.08%, which is essentially zero.

Now, let’s look at the control group’s changes in math scores:

Here we have six schools that showed drops in math scores: Brookland, Marshall, Winston, Kramer, LaSalle, and Francis. At eight of the schools, there were increases: Deal, Johnson, Shaed, Truesdell, Ron Brown, West, Walker-Jones, and Sousa. The median change was +2.38%, and the average change was +1.26%.

So who won in math? Again, the control group, but not as decisively. Both groups had the same number of schools with declines and increases, but the average or median growth in the control group was larger than in the experimental group.

So for all of that effort, what did the students of DCPS and citizens of the District actually get? We seem to have learned that we can’t just bribe student a few hundred dollars a year to get them to raise their test scores.

The fact that the DCPS has these results and has not publicized them – and that the media have also been mum – raises some uncomfortable questions, especially since they have continued the experiment.

Suppose that a drug company wants to show the FDA that their new medicine XYZ is good for you. They set up a situation where an experimental group gets medicine XYZ, and a control group doesn’t get any medication at all. And suppose the experiment shows that the people taking XYZ get sicker, or die more often, than those taking no medicine at all. Suppose the drug company then hides that data and keeps on giving out medicine XYZ. Most of us would call that scientific fraud, and might file lawsuits.

Well, that’s essentially what we have here.

What are Rhee, Reinoso, and Fenty trying to hide, and why are they continuing this failed experiment?

Even the person who came up with the idea for the program, Roland Fryer, Jr. of Harvard University, stated publicly that it might not work:

“Fryer has launched similar programs in Chicago and New York, where about 8,500 fourth- and seventh-graders are getting cash awards based on standardized test scores. Although he has touched off a national debate about the propriety of paying students for performance, he has said that these incentive programs are not a solution for the achievement gap and that they may prove to be ineffective when the data are analyzed.”

(Washington Post, 9-30-08)

So, if it doesn’t work, try something else. Don’t keep up the same failed experiment.

PS: This reminds me of the deal with charter schools. Study after study has shown that they do no better, and often do worse, than regular public schools. But Administration after Administration keeps saying that the only cure for public schools is to close them down and convert them to charter schools! Is this magical thinking, or something much more nefarious?

Sources: Raw data on the DC-NCLB-OSSE AYP website; spreadsheet, analysis and graphs prepared by the author. See http://www.nclb.osse.dc.gov/ and be prepared to do a lot of digging.

Published in: on March 14, 2010 at 7:09 pm  Comments (9)  

The URI to TrackBack this entry is: https://gfbrandenburg.wordpress.com/2010/03/14/%e2%80%9ccapital-gains%e2%80%9d-program-not-effective-experimental-schools-do-worse-in-both-reading-and-math-than-the-control-group/trackback/

RSS feed for comments on this post.

9 CommentsLeave a comment

  1. […] Guy Brandenberg crunches DC-CAS numbers, concludes that Capital Gains pay-for-grades program doesn't […]

    Like

  2. […] Guy Brandenberg crunches DC-CAS numbers, concludes that Capital Gains pay-for-grades program doesn’t […]

    Like

  3. It appears, according DCPS website, that Hardy does not participate.

    Like

    • Hmm. Thanks. I was going by what was listed in WaPo last year. I wonder if the schools participating changed. I may have to go back, call up the principals of the schools where my original list differs from what’s on the DCPS website, and see how the data come out after re-doing my analysis. Here is the current list:
      * Brightwood EC
      * Browne EC
      * Burroughs EC
      * Eliot-Hine MS
      * Emery EC
      * Hart MS
      * Jefferson MS
      * Kelly Miller MS
      * Langdon EC
      * Lincoln MS (part of Columbia Heights Education Campus)
      * Raymond EC
      * Shaw MS @ Garnet-Patterson
      * Stuart-Hobson MS
      * Takoma EC
      * Whittier EC

      GFB

      Like

      • As I posted on the Post’s Bill Turque’s blog, the 2008 announcement about Capital Gains was held at Hardy MS.
        Someone followed up and said Hardy parents rejected it.
        Rather ironic.

        Like

  4. Hardy teachers met with the representatives from Harvard and we made it clear we had no interest in participating. The announcement was probably at Hardy b/c we had just returned to the renovated building, and it made for a pretty picture!

    Like

    • Thanks for the information; and good for you!
      GFB

      Like

  5. […] An analysis of the program by Harvard economist Roland Fryer found that Capital Gains was effective in improving reading skills compared to the control group. Results were especially pronounced among boys and  Hispanic and black students. (Note: if you do a quick Google search, you can see an argument against the program here). […]

    Like

  6. […] Fryer worked with Michelle Rhee in an abortive experiment in DCPS and elsewhere to raise student achievement by paying students for doing the right thing — an experiment which also had no positive results, according to Guy Grandenburg. […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: