What I actually had time to say …

Since I had to abbreviate my remarks, here is what I actually said:

I am Guy Brandenburg, retired DCPS mathematics teacher.

To depart from my text, I want to start by proposing a solution: look hard at the collaborative assessment model being used a few miles away in Montgomery County [MD] and follow the advice of Edwards Deming.

Even though I personally retired before [the establishment of the] IMPACT [teacher evaluation system], I want to use statistics and graphs to show that the Value-Added measurements that are used to evaluate teachers are unreliable, invalid, and do not help teachers improve instruction. To the contrary: IVA measurements are driving a number of excellent, veteran teachers to resign or be fired from DCPS to go elsewhere.

Celebrated mathematician John Ewing says that VAM is “mathematical intimidation” and a “modern, mathematical version of the Emperor’s New Clothes.”

I agree.

One of my colleagues was able to pry the value-added formula [used in DC] from [DC data honcho] Jason Kamras after SIX MONTHS of back-and-forth emails. [Here it is:]

value added formula for dcps - in mathtype format

One problem with that formula is that nobody outside a small group of highly-paid consultants has any idea what are the values of any of those variables.

In not a single case has the [DCPS] Office of Data and Accountability sat down with a teacher and explained, in detail, exactly how a teacher’s score is calculated, student by student and class by class.

Nor has that office shared that data with the Washington Teachers’ Union.

I would ask you, Mr. Catania, to ask the Office of Data and Accountability to share with the WTU all IMPACT scores for every single teacher, including all the sub-scores, for every single class a teacher has.

Now let’s look at some statistics.

My first graph is completely random data points that I had Excel make up for me [and plot as x-y pairs].

pic 3 - completely random points

Notice that even though these are completely random, Excel still found a small correlation: r-squared was about 0.08 and r was about 29%.

Now let’s look at a very strong case of negative correlation in the real world: poverty rates and student achievement in Nebraska:

pic  4 - nebraska poverty vs achievement

The next graph is for the same sort of thing in Wisconsin:

pic 5 - wisconsin poverty vs achievement

Again, quite a strong correlation, just as we see here in Washington, DC:

pic 6 - poverty vs proficiency in DC

Now, how about those Value-Added scores? Do they correlate with classroom observations?

Mostly, we don’t know, because the data is kept secret. However, someone leaked to me the IVA and classroom observation scores for [DCPS in] SY 2009-10, and I plotted them [as you can see below].

pic 7 - VAM versus TLF in DC IMPACT 2009-10

I would say this looks pretty much no correlation at all. It certainly gives teachers no assistance on what to improve in order to help their students learn better.

And how stable are Value-Added measurements [in DCPS] over time? Unfortunately, since DCPS keeps all the data hidden, we don’t know how stable these scores are here. However, the New York Times leaked the value-added data for NYC teachers for several years, and we can look at those scores to [find out]. Here is one such graph [showing how the same teachers, in the same schools, scored in 2008-9 versus 2009-10]:

pic 8 - value added for 2 successive years Rubenstein NYC

That is very close to random.

How about teachers who teach the same subject to two different grade levels, say, fourth-grade math and fifth-grade math? Again, random points:

pic 9 - VAM for same subject different grades NYC rubenstein

One last point:

Mayor Gray and chancellors Henderson and Rhee all claim that education in DC only started improving after mayoral control of the schools, starting in 2007. Look for yourself [in the next two graphs].

pic 11 - naep 8th grade math avge scale scores since 1990 many states incl dc

 

pic 12 naep 4th grade reading scale scores since 1993 many states incl dc

Notice that gains began almost 20 years ago, long before mayoral control or chancellors Rhee and Henderson, long before IMPACT.

To repeat, I suggest that we throw out IMPACT and look hard at the ideas of Edwards Deming and the assessment models used in Montgomery County.

Poverty, Segregation, and Test Scores in DC

While looking at the latest released NCLB test scores in Washington, DC, I was struck by the enormous number of students who are stuck in completely segregated schools, on the eve of the 50th anniversary of the March on Washington for civil rights and justice.

100% black student body and 100% of them on free or reduced-price lunches (i.e., poor) is the most obvious group of schools.

Followed by another very large group of schools that are 90-99% black & hispanic and 100% poor.

Very, very few schools have an actual mix of white, hispanic, black, and asian students.

This is true for both the regular public schools in DC and for the publicly-funded but privately-run charter schools.

Out of a  total of 181 DC schools for which I have data, 23 have ‘perfectly’ segregated student bodies — that is, every single kid is black and/or hispanic,  AND every single kid is eligible for free or reduced-price lunches.

Here is the list:

  1. Aiton Elementary
  2. Arts + Technology Public Charter
  3. Beers Elementary
  4. Burrville Elementary
  5. C. W.  Harris Elementary
  6. Center City (Congress Heights campus) Public  Charter
  7. Center City (Shaw campus) Public Charter
  8. Ferebee-Hope Elementary
  9. Garfield Elementary
  10. Howard  Road (MLK campus) Public Charter
  11. Howard Road (main campus) Public Charter
  12. Integrated Design Electronics Academy (IDEA) Public Charter
  13. Johnson Middle
  14. Martin Luther King Elementary
  15. Ludlow-Taylor Elementary
  16. Malcolm X Elementary
  17. Maya Angelou (Evans campus) Public Charter
  18. Meridian Public Charter
  19. Options Public Charter
  20. Randle Highlands Elementary
  21. Septima Clark Public Charter
  22. Simon Elementary
  23. Stanton Elementary

Poverty Isn’t Destiny?

Quite a few Ed Deformers say that Poverty Isn’t Destiny. They say that it doesn’t matter if a child has been subjected to lead poisoning, separation from parents, violent or otherwise cruel child abuse, inadequate nutrition, and has lacked dental or health care and the love and care of a family during the first, crucial years. All it takes is for a Bright Young Thing fresh out of college to work her butt off for two years before she goes to work for a bank — and all of those handicaps will be overcome, with no extra dollars invested, and maybe even less!

Or maybe not.

Lots of teachers have been working their butts off for many decades, doing their best, believe it or not (for the most part).

Here are two three graphs from Wisconsin that show how close the connection between the poverty rates and student achievement levels, at all of their schools for which they provide data. My data come from here and are for SY 2011-2012. In fact, you can download the entire spreadsheet for the state of Wisconsin if you click on this link:

http:reportcards.dpi.wi.gov/files/reportcards/xls/2011-12reportcarddata.xlsx

In both all three graphs, the percentage of students at the schools is along the horizontal (X) axis. In the first two, the average achievement score at the school is along the vertical (Y) axis.

In this first graph, Wisconsin uses a 100-point scale for overall student achievement.

wisconsin school overall student ach score by pct of poor kids

That is an incredibly strong correlation between poverty levels and student achievement. The fewer the proportion of poor students at a school, the better the achievement scores at that school.

I had Excel compute two correlation “trend” lines – one straight, in black, and one curved, in red following a third-degree polynomial, since it looks like we have a serious “Matthew effect” going on here. In either case, the R-squared and R values are very elevated, showing that, in fact, poverty is in fact destiny for a lot of kids.

The next graph is for reading only, but it shows essentially the same trend. School reading scores go from 0 to 50.

Wisconsin school READING scores by pct of poor kids

There are very few real-life correlations between two entities stronger than what you see in these two graphs.

This next graph is a little different, for two reasons: the y-axis is math, and it’s the percent of students deemed ‘proficient’ on whatever test Wisconsin is using. It also shows a very strong correlation.

wisconsin school poverty rate versus percent of students proficient in MATH

Widening US Educational Achievement Gap Between Rich and Poor?

A Stanford professor has analyzed data for the past 50 years, concluding that the gap in educational achievement between the wealthy and the poor has become considerably wider since about 1960; it’s now roughly twice as large as the black-white gap, when it used to be roughly the reverse!

A quote from his article,  http://cepa.stanford.edu/content/widening-academic-achievement-gap-between-rich-and-poor-new-evidence-and-possible

The widening academic achievement gap between the rich and the poor: New evidence and possible explanations

In this chapter I examine whether and how the relationship between family socioeconomic characteristics and academic achievement has changed during the last fifty years. In particular, I investigate the extent to which the rising income inequality of the last four decades has been paralleled by a similar increase in the income achievement gradient. As the income gap between high- and low-income families has widened, has the achievement gap between children in high- and low-income families also widened?

The answer, in brief, is yes. The achievement gap between children from high- and low-income families is roughly 30 to 40 percent larger among children born in 2001 than among those born twenty-five years earlier. In fact, it appears that the income achievement gap has been growing for at least fifty years, though the data are less certain for cohorts of children born before 1970. In this chapter, I describe and discuss these trends in some detail. In addition to the key finding that the income achievement gap appears to have widened substantially, there are a number of other important findings.

First, the income achievement gap (defined here as the income difference between a child from a family at the 90th percentile of the family income distribution and a child from a family at the 10th percentile) is now nearly twice as large as the black-white achievement gap. Fifty years ago, in contrast, the black-white gap was one and a half to two times as large as the income gap.

Published in: on December 23, 2012 at 11:56 am  Comments (1)  
Tags: , , , ,

In Other Countries, Kids in Wealthier Schools Do Better Than Kids in Poorer Schools. Huge Surprise, Right?

Released today, the latest international studies of students in math, science, and reading (TIMMS and PIRLS) actually make interesting reading. You can find the the math report (TIMMS) here and the reading and literacy report here. They are pretty long.

Unfortunately and as usual, the few on-line and printed reports ABOUT the studies seem to be cherry-picking data to find what they want — much like pundits pontificating and bloviating on TV the exact same ‘talking points’, whether there’s any evidence for what they believe or not. I just saw a headline: “More Bad News for US Students.”

At least I try to look at the data itself.

A brief quote from the TIMMS ‘executive summary’, which is as far as I had gotten when I first wrote this yesterday morning:

Home Resources Strongly Related to Mathematics Achievement

Research consistently shows a strong positive relationship between achievement and indicators of socioeconomic status, such as parents’ or caregivers’ level of education. At the fourth and sixth grades, TIMSS used the parents’ reports on the availability of key home resources to create the Home Resources for Learning scale, including parents’ education, parents’ occupation, books in the home, and study supports. Internationally, on average, the 17 percent of students with Many Resources had substantially higher mathematics achievement than the nine percent with Few Resources—a 119-point difference. However, 
almost three-quarters of the fourth grade students (74%) had Some Resources.
At the eighth and ninth grades, TIMSS asked the students themselves about their parents’  education, books in the home, and study supports, with similar results. Internationally, the twelve percent of eighth grade students with Many Resources had the highest average achievement, the two-thirds with Some Resources had the next highest achievement,  and the one-fifth with Few Resources had the lowest average achievement.

Note that this is in ALL of the countries: kids from poorer backgrounds do less well in school, and the inverse: wealthier kids do better in school. All over the world, not just in the USA.

Duh.

Here are the tables that accompanied that text:

POVERTY AND LEARNING

 

Now that I’ve had a bit more time to look at the data, I see that at the fourth grade, I can look at reported differences between average scores at schools where the principal says the kids come from wealthier families, and average scores at schools where the principal says the students come from more disadvantaged families. It’s one way of estimating what the famous rich-poor achievement gap is like in various nations, but we should be cautious with the data: it all has pretty large standard deviations according to TIMMS itself, meaning that the various bars in this graph are likely to be, in reality, much higher or much lower than this chart (which I made using the data in exhibit 5.3, page 514).

grade 4 rich poor schools gap TIMMS

 

If this information is accurate, the US has a somewhat larger ‘math poverty’ gap than most other countries, but it’s by no means the worst. And it also shows that lots of other countries actually have similar achievement gaps in 4th grade math between two groups of schools:

Quoting from the report:

… the More Affluent schools had more than one-fourth of their students from affluent home backgrounds and not more than one-fourth from disadvantaged home backgrounds, and the More Disadvantaged schools had the reverse situation. The other schools were “in between.” Internationally, the students were distributed relatively equally across the three types of schools. On average, across countries at the fourth grade, 36 percent of the students attended schools with relatively more affluent students than disadvantaged students, and students in these schools had the highest average achievement (508). At the other end of the range, 30 percent of the students attended schools with relatively more disadvantaged students than affluent students, and students in these schools had the lowest average achievement (470). Although this overall achievement pattern was observed in most countries and benchmarking participants, there was a wide variation among participants in the percentages of students attending the three different economic categories of schools.

MAIN POINTS:

  1. The sky isn’t falling,
  2. US schools are neither the best nor the worst in the world.
  3. This data (like most educational dat) is actually quite fuzzy.
  4. All around the world, it appears that wealthier kids these days generally tend to do better in math at the 4th grade than poorer kids.
Published in: on December 12, 2012 at 2:07 pm  Leave a Comment  
Tags: , ,
Follow

Get every new post delivered to your Inbox.

Join 378 other followers