Wanting to be fair and scrupulous, I decided to take another look at the scores of the 28 schools that were split into a control group and an experimental group. The reason for my second look is that some of the schools enroll other grades than just 6 through 8, and I thought that might skew the results. Also, I had no real idea whether the schools were gaining or losing population, and I didn’t know how that might affect the results, either.

For that reason, I just spent several hours tabulating two things for each school in this experiment:

* the exact number of students at each of those schools who were in grades 6, 7, and 8 and were tested for reading in 2008 and 2009; and

* the percentage of students at those schools in those grades who were deemed ‘proficient’ on the DC-CAS in reading 2008 and 2009.

I then used that data to calculate how many students in each school were proficient in reading in grades 6, 7, and 8, for the two years.

Results? Very close to no difference in percentages of students proficient over time in the two groups of schools, over time, as you can see here in this table:

As you can see, in the ‘control group’, where students were not given bribes to come to school nd do their homework, the percent of students proficient in reading went from (drumroll, please) 43% all the way to 43%!!! However, the total number of students enrolled and tested increase by over 400, for reasons I know not.

In the experimental group, where the students did receive rewards for being on time, coming to school, not getting into trouble, doing well on their report cards, and so on, the portion of students scoring “proficient” in reading went from (gasp!) 42% to 43%! Not a very big change; I don’t think it’s significant in fact, even if it is theoretically of statistical significance. Again, the number of students in these schools incresed by a lot -about 300 students. Why? I don’t know.

Sorry, I didn’t have time to retabulate all the results for math as well. I don’t know what the results would be, but I suspect that it would show just about no significant change, either. Why do I think that? Well, look at all of the churning of numbers in the following table, which gives a more detailed breakdown, by grade level (but only for reading):

Notice that there are lots of changes, but that they don’t seem to add up to any pattern. For example: the total number of 6th graders in the control group nearly doubled from 2008 to 2009, and their proficiency level went up from a very-low 32% to a still-low 39%. Why did the total numbers, and the percent of proficient students, both go up? I can only guess.

In that same grade, the experimental group increased by over 200 students, and their proficiency rate started out a lot higher than that in the control group schools, but only increased by about 3.5%. Why? I can only guess.

And so on.

Or, if we look longitudinally at more-or-less the same students as they move through the grades, we still have problems. In the sixth grade, three hundred sixteen sixth-graders somehow become 951 seventh-graders the following year, and the proficiency rate went from 32% to 42%. Where did the extra students come from? I can only guess. Why did the pass rate go up? Same answer.

A different question: is it really true that 1,048 seventh-graders in the experimental group were only joined by six new students to become 1054 8th graders? I strongly doubt it. And whether or not that is true, why did the percent of proficient 7th grade students go from 42% to 43% in the 8th grade? I don’t know. And why did the percent of proficient 6th graders in the experimental group drop by 11% when they went to the 7th grade? You know my answer by now.

Do you have a better one?

2nd to last paragraph, 2%–should be 42% I assume.

Thanks for your blog!

LikeLike

Yes, thanks, TFT! Definite slip of the missing index finger (or something). I added a bunch more text since you wrote your comment.

LikeLike

Thanks for all the detail and acknowledgement of unanswered questions. I bet Roland Fryer, the Harvard economist who is doing the Capitol Gains experiment with the complete data at his disposal, would have some answers. I also think he’d be impressed with your research. Once his comes out, it will be interesting to see his data, his methods and to compare results.

LikeLike

DCPS doesn’t really go into detail at their website:

http://dcps.dc.gov/DCPS/Beyond+the+Classroom/Special+Projects+in+Schools/Capital+Gains

I’ll agree with efavorite about Dr. Fryer.

After all, he made Esquire’s “Genius” issue.

LikeLike