Part Two: Cheating in DCPS

DC Education Reform Ten Years After, 

Part 2: Test Cheats

Richard P Phelps

Ten years ago, I worked as the Director of Assessments for the District of Columbia Public Schools (DCPS). For temporal context, I arrived after the first of the infamous test cheating scandals and left just before the incident that spawned a second. Indeed, I filled a new position created to both manage test security and design an expanded testing program. I departed shortly after Vincent Gray, who opposed an expanded testing program, defeated Adrian Fenty in the September 2010 DC mayoral primary. My tenure coincided with Michelle Rhee’s last nine months as Chancellor. 

The recurring test cheating scandals of the Rhee-Henderson years may seem extraordinary but, in fairness, DCPS was more likely than the average US school district to be caught because it received a much higher degree of scrutiny. Given how tests are typically administered in this country, the incidence of cheating is likely far greater than news accounts suggest, for several reasons: 

·      in most cases, those who administer tests—schoolteachers and administrators—have an interest in their results;

·      test security protocols are numerous and complicated yet, nonetheless, the responsibility of non-expert ordinary school personnel, guaranteeing their inconsistent application across schools and over time; 

·      after-the-fact statistical analyses are not legal proof—the odds of a certain amount of wrong-to-right erasures in a single classroom on a paper-and-pencil test being coincidental may be a thousand to one, but one-in-a-thousand is still legally plausible; and

·      after-the-fact investigations based on interviews are time-consuming, scattershot, and uneven. 

Still, there were measures that the Rhee-Henderson administrations could have adopted to substantially reduce the incidence of cheating, but they chose none that might have been effective. Rather, they dug in their heels, insisted that only a few schools had issues, which they thoroughly resolved, and repeatedly denied any systematic problem.  

Cheating scandals

From 2007 to 2009 rumors percolated of an extraordinary level of wrong-to-right erasures on the test answer sheets at many DCPS schools. “Erasure analysis” is one among several “red flag” indicators that testing contractors calculate to monitor cheating. The testing companies take no responsibility for investigating suspected test cheating, however; that is the customer’s, the local or state education agency. 

In her autobiographical account of her time as DCPS Chancellor, Michelle Johnson (nee Rhee), wrote (p. 197)

“For the first time in the history of DCPS, we brought in an outside expert to examine and audit our system. Caveon Test Security – the leading expert in the field at the time – assessed our tests, results, and security measures. Their investigators interviewed teachers, principals, and administrators.

“Caveon found no evidence of systematic cheating. None.”

Caveon, however, had not looked for “systematic” cheating. All they did was interview a few people at several schools where the statistical anomalies were more extraordinary than at others. As none of those individuals would admit to knowingly cheating, Caveon branded all their excuses as “plausible” explanations. That’s it; that is all that Caveon did. But, Caveon’s statement that they found no evidence of “widespread” cheating—despite not having looked for it—would be frequently invoked by DCPS leaders over the next several years.[1]

Incidentally, prior to the revelation of its infamous decades-long, systematic test cheating, the Atlanta Public Schools had similarly retained Caveon Test Security and was, likewise, granted a clean bill of health. Only later did the Georgia state attorney general swoop in and reveal the truth. 

In its defense, Caveon would note that several cheating prevention measures it had recommended to DCPS were never adopted.[2] None of the cheating prevention measures that I recommended were adopted, either.

The single most effective means for reducing in-classroom cheating would have been to rotate teachers on test days so that no teacher administered a test to his or her own students. It would not have been that difficult to randomly assign teachers to different classrooms on test days.

The single most effective means for reducing school administratorcheating would have been to rotate test administrators on test days so that none managed the test materials for their own schools. The visiting test administrators would have been responsible for keeping test materials away from the school until test day, distributing sealed test booklets to the rotated teachers on test day, and for collecting re-sealed test booklets at the end of testing and immediately removing them from the school. 

Instead of implementing these, or a number of other feasible and effective test security measures, DCPS leaders increased the number of test proctors, assigning each of a few dozen or so central office staff a school to monitor. Those proctors could not reasonably manage the volume of oversight required. A single DC test administration could encompass a hundred schools and a thousand classrooms.

Investigations

So, what effort, if any, did DCPS make to counter test cheating? They hired me, but then rejected all my suggestions for increasing security. Also, they established a telephone tip line. Anyone who suspected cheating could report it, even anonymously, and, allegedly, their tip would be investigated. 

Some forms of cheating are best investigated through interviews. Probably the most frequent forms of cheating at DCPS—teachers helping students during test administrations and school administrators looking at test forms prior to administration—leave no statistical residue. Eyewitness testimony is the only type of legal evidence available in such cases, but it is not just inconsistent, it may be socially destructive. 

I remember two investigations best: one occurred in a relatively well-to-do neighborhood with well-educated parents active in school affairs; the other in one of the city’s poorest neighborhoods. Superficially, the cases were similar—an individual teacher was accused of helping his or her own students with answers during test administrations. Making a case against either elementary school teacher required sworn testimony from eyewitnesses, that is, students—eight-to-ten-year olds. 

My investigations, then, consisted of calling children into the principal’s office one-by-one to be questioned about their teacher’s behavior. We couldn’t hide the reason we were asking the questions. And, even though each student agreed not to tell others what had occurred in their visit to the principal’s office, we knew we had only one shot at an uncorrupted jury pool. 

Though the accusations against the two teachers were similar and the cases against them equally strong, the outcomes could not have been more different. In the high-poverty neighborhood, the students seemed suspicious and said little; none would implicate the teacher, whom they all seemed to like. 

In the more prosperous neighborhood, students were more outgoing, freely divulging what they had witnessed. The students had discussed the alleged coaching with their parents who, in turn, urged them to tell investigators what they knew. During his turn in the principal’s office, the accused teacher denied any wrongdoing. I wrote up each interview, then requested that each student read and sign. 

Thankfully, that accused teacher made a deal and left the school system a few weeks later. Had he not, we would have required the presence in court of the eight-to-ten-year olds to testify under oath against their former teacher, who taught multi-grade classes. Had that prosecution not succeeded, the eyewitness students could have been routinely assigned to his classroom the following school year.

My conclusion? Only in certain schools is the successful prosecution of a cheating teacher through eyewitness testimony even possible. But, even where possible, it consumes inordinate amounts of time and, otherwise, comes at a high price, turning young innocents against authority figures they naturally trusted. 

Cheating blueprints

Arguably the most widespread and persistent testing malfeasance in DCPS received little attention from the press. Moreover, it was directly propagated by District leaders, who published test blueprints on the web. Put simply, test “blueprints” are lists of the curricular standards (e.g., “student shall correctly add two-digit numbers”) and the number of test items included in an upcoming test related to each standard. DC had been advance publishing its blueprints for years.

I argued that the way DC did it was unethical. The head of the Division of Data & Accountability, Erin McGoldrick, however, defended the practice, claimed it was common, and cited its existence in the state of California as precedent. The next time she and I met for a conference call with one of DCPS’s test providers, Discover Education, I asked their sales agent how many of their hundreds of other customers advance-published blueprints. His answer: none.

In the state of California, the location of McGoldrick’s only prior professional experience, blueprints were, indeed, published in advance of test administrations. But their tests were longer than DC’s and all standards were tested. Publication of California’s blueprints served more to remind the populace what the standards were in advance of each test administration. Occasionally, a standard considered to be of unusual importance might be assigned a greater number of test items than the average, and the California blueprints signaled that emphasis. 

In Washington, DC, the tests used in judging teacher performance were shorter, covering only some of each year’s standards. So, DC’s blueprints showed everyone well in advance of the test dates exactly which standards would be tested and which would not. For each teacher, this posed an ethical dilemma: should they “narrow the curriculum” by teaching only that content they knew would be tested? Or, should they do the right thing and teach all the standards, as they were legally and ethically bound to, even though it meant spending less time on the to-be-tested content? It’s quite a conundrum when one risks punishment for behaving ethically.

Monthly meetings convened to discuss issues with the districtwide testing program, the DC Comprehensive Assessment System (DC-CAS)—administered to comply with the federal No Child Left Behind (NCLB) Act. All public schools, both DCPS and charters, administered those tests. At one of these regular meetings, two representatives from the Office of the State Superintendent of Education (OSSE) announced plans to repair the broken blueprint process.[3]

The State Office employees argued thoughtfully and reasonably that it was professionally unethical to advance publish DC test blueprints. Moreover, they had surveyed other US jurisdictions in an effort to find others that followed DC’s practice and found none. I was the highest-ranking DCPS employee at the meeting and I expressed my support, congratulating them for doing the right thing. I assumed that their decision was final.

I mentioned the decision to McGoldrick, who expressed surprise and speculation that it might have not been made at the highest level in the organizational hierarchy. Wasting no time, she met with other DCPS senior managers and the proposed change was forthwith shelved. In that, and other ways, the DCPS tail wagged the OSSE dog. 

* * *

It may be too easy to finger ethical deficits for the recalcitrant attitude toward test security of the Rhee-Henderson era ed reformers. The columnist Peter Greene insists that knowledge deficits among self-appointed education reformers also matter: 

“… the reformistan bubble … has been built from Day One without any actual educators inside it. Instead, the bubble is populated by rich people, people who want rich people’s money, people who think they have great ideas about education, and even people who sincerely want to make education better. The bubble does not include people who can turn to an Arne Duncan or a Betsy DeVos or a Bill Gates and say, ‘Based on my years of experience in a classroom, I’d have to say that idea is ridiculous bullshit.’”

“There are a tiny handful of people within the bubble who will occasionally act as bullshit detectors, but they are not enough. The ed reform movement has gathered power and money and set up a parallel education system even as it has managed to capture leadership roles within public education, but the ed reform movement still lacks what it has always lacked–actual teachers and experienced educators who know what the hell they’re talking about.”

In my twenties, I worked for several years in the research department of a state education agency. My primary political lesson from that experience, consistently reinforced subsequently, is that most education bureaucrats tell the public that the system they manage works just fine, no matter what the reality. They can get away with this because they control most of the evidence and can suppress it or spin it to their advantage.

In this proclivity, the DCPS central office leaders of the Rhee-Henderson era proved themselves to be no different than the traditional public-school educators they so casually demonized. 

US school systems are structured to be opaque and, it seems, both educators and testing contractors like it that way. For their part, and contrary to their rhetoric, Rhee, Henderson, and McGoldrick passed on many opportunities to make their system more transparent and accountable.

Education policy will not improve until control of the evidence is ceded to genuinely independent third parties, hired neither by the public education establishment nor by the education reform club.

The author gratefully acknowledges the fact-checking assistance of Erich Martel and Mary Levy.

Access this testimonial in .pdf format

Citation:  Phelps, R. P. (2020, September). Looking Back on DC Education Reform 10 Years After, Part 2: Test Cheats. Nonpartisan Education Review / Testimonials. https://nonpartisaneducation.org/Review/Testimonials/v16n3.htm


[1] A perusal of Caveon’s website clarifies that their mission is to help their clients–state and local education departments–not get caught. Sometimes this means not cheating in the first place; other times it might mean something else. One might argue that, ironically, Caveon could be helping its clients to cheat in more sophisticated ways and cover their tracks better.

[2] Among them: test booklets should be sealed until the students open them and resealed by the students immediately after; and students should be assigned seats on test day and a seating chart submitted to test coordinators (necessary for verifying cluster patterns in student responses that would suggest answer copying).

[3] Yes, for those new to the area, the District of Columbia has an Office of the “State” Superintendent of Education (OSSE). Its domain of relationships includes not just the regular public schools (i.e., DCPS), but also other public schools (i.e., charters) and private schools. Practically, it primarily serves as a conduit for funneling money from a menagerie of federal education-related grant and aid programs

Religiosity vs Poverty and Education

This is from Quora. The USA is a real outlier, but in general the poorer a country is, the more religious its people are, and vice versa; also, the more education, the less religiosity.

Q: Have countries that have learned towards atheism failed more than countries that have acknowledged God?

A: Let’s check.

The table below has the ten most and least religious countries according to Gallup, followed by how many think religion is important, followed by GDP per capita according to IMF.


1: Estonia: religious score 16%, GDP/capita $22,990

2: Sweden: religious score 17%, GDP/capita $53,873

3: Denmark: religious score 19%, GDP/capita $60,692

4: Norway: religious score 21%, GDP/capita $81,695

5: Czech republic: religious score 21%, GDP/capita $22,850

6: Japan: religious score 24%, GDP/capita $39,306

7: Hong Kong: religious score 24%, GDP/capita $48,517

8: United Kingdom: religious score 27%, GDP/capita $42,558

9: Finland: religious score 28%, GDP/capita $42,878

10: Vietnam: religious score 30%, GDP/capita $2,551

[…]

149: Djibouti: religious score 98%, GDP/capita $2,085

150: Mauritania: religious score 98%, GDP/capita $1,143

151: Sri Lanka: religious score 99%, GDP/capita $4,068

152: Malawi: religious score 99%, GDP/capita $351

153: Indonesia: religious score 99%, GDP/capita $3,871

154: Yemen: religious score 99%, GDP/capita $872

155: Niger: religious score 100%, GDP/capita $477

156: Ethiopia: religious score 100%, GDP/capita $853

157: Somalia: religious score 100%, GDP/capita $499*

158. Bangladesh: religious score 100%, GDP/capita $1,745

*Not in IMF’s dataset; World Bank used instead.


But that data isn’t very intuitive. Sure, there’s at least a factor 10 difference between the least religious countries and the most religious countries, but how can we illustrate it more clearly? Well, how about a graph:

Although Pew chose to highlight the US and its strong outlier as a wealthy nation with high religiosity, the interesting thing is the inverse correlation between GDP/capita and religiosity. It really seems to imply that in general, success and irreligion are connected.

But how? In the same dataset, Pew also makes another important observation, namely of education.

This correlation is much stronger. And we already know that education and wealth are strongly correlated.

But it’s not quite that simple. Pew makes yet another observation, of income inequality and religion:

But what we can take away from this is that the poorer a country is, and the greater the income inequality is, and the poorer educated a country is, the more religious it is in general.

Or expressed even more bluntly: shithole country ≈ religious country.


Sources:

Importance of religion by country – Wikipedia

List of countries by GDP (nominal) per capita – Wikipedia

Religious observance by age and country

Published in: on January 10, 2020 at 9:31 am  Comments (7)  
Tags: , ,

Poverty vs Proficiency In DC Public and Charter Schools

You’ve all heard the slogan:

“A child’s course in life should be determined not by the zip code she’s born in.” Source

Reformers like Bush2, Barack Obama, Michelle Rhee, Joel Klein, Arne Duncan, Adrian Fenty, Bill Gates, the Bradleys and the Waltons, all said they were going to bust the educational effects of poverty in DC and other places around the country. Their chosen methods were gutting the teachers’ unions, establishing lots of charter schools, firing or forcing into retirement thousands of teachers, establishing a revolving door of inexperienced teachers who almost all crash and burn out after a few years, and transforming schooling into all testing and test prep, all the time, especially on-line, so as to collect lots of data.

Have they been successful at solving the zip-code-and-destiny problem?

If we look at the only publicly-available data that we have for Washington, DC, namely PARCC scores and percentages of students who are designated as ‘At Risk’, the answer is:

NO.

Look at these two graphs, which I’ve prepared by matching the percentages of students scoring ‘Proficient’ or ‘Advanced’ in Washington, DC, at every single DC public school and charter school, versus OSSE’s official list of the percentages and numbers of students officially designated as being ‘At Risk’.

Unfortunately, the correlation is extremely strong, and negative. In other words, the fewer the kids who are officially ‘At Risk’ at any given school, the higher the percentage of kids scoring ‘Proficient’ or ‘Advanced’ on the PARCC – the Big Standardized test given in April of 2017. And obversely the greater the percentage of students at risk at any school, the lower the percentage of students ‘passing’ the PARCC.

The effect is particularly strong in the English and Reading part of the test.

(Note: I didn’t make up the ‘At Risk’ category. It’s relatively new, but combines statistics regarding homelessness, receiving food, living in poverty, divorces, family members being incarcerated, and so on.)

Here is the graph I made for the English Language Arts test. That R-squared correlation, 0.7016, is one of the strongest correlations you will find anywhere in the social sciences.

2017 ELA Parcc, proficient vs at risk, public and charter

Now here is the graph for the Math section of the PARCC:

2017 math PARCC proficiency vs at risk, public and charter

This is certainly not an indication that education ‘reform’ in DC has been a success. After more than a decade.

Next time I’ll break this down into charters and public schools. I think you will find that many of the charter schools have populations near the middle of these charts, while the regular DC public schools have populations near the extremes.


Many thanks to Ruth Wattenberg, Mary Levy and Matthew Frumin for showing me where these data files were kept – here and here. Any errors are my own.

 

 

A closer look at charter and regular public school enrollments, percentages of students at risk, and percentages of students ‘proficient’

Here is another look at the brand-new data concerning four variables in the District of Columbia schools, about which I wrote a couple of days ago. The difference here is that the dots representing the schools are more-or=less proportional to the size of the student body.

1. Is this a regular public school, or a charter school (blue or red):

2. What fraction of the kids at that school are officially considered to be At Risk? (That’s the scale along the x-axis at the bottom of the page)

3. What is the average percentage of the kids at that school are ‘proficient’ in reading and math on the DC-CAS? (That’s the scale along the y-axis at the left-hand side of the page)

4. How big is the school? (That’s the size of the dot, more or less; the legend is at the bottom left-hand corner of the graph)

Time spent looking carefully at this graph will be well-spent. If you click on it, it will expand.

It will certainly show that charter schools have not revolutionized education for the better in DC: for both types of schools, there remains a very strong, negative correlation between the percentages of kids At Risk and ‘pass’ rates on the DC-CAS.

Note that most schools have between 200 and 500 students and that most of the ones that are smaller are actually charter schools. As I wrote a couple of days ago, the schools with the largest fraction of At-Risk students (say, over 2/3 of the student body) are almost all regular DC public schools.

On the second graph, which is otherwise identical to the first, I’ve labeled some of the larger schools.

fixed bicolor, size of school and at risk vs average dc cas 2014 proficiency, both regular public and charter, dc

Here is the one with names of some of the larger schools, so you can see how individual schools fall on this graph.

(Sorry, I there was not enough room to label every single one, and my non-existent HTML skills won’t allow me to make it so that any of the dots are clickable. If any of my readers know how to do that and would like to offer to make that happen, then please let me know in the comments.)

again fixed and revised names and bicolor, size of school and at risk vs average dc cas 2014 proficiency, both regular public and charter, dc

And here is the entire data table. So you can see where every single school lies on these three dimensions.

(PS: I added a few more names of schools and corrected four other small errors, two pointed out by an alert reader.. 2/22/2015)

How Well are Charter Schools in DC Educating Students Who are Officially At-Risk?

The results may surprise you.

To answer this question, I used some recent data. I just found out that the DC City Council has begun requiring that schools enumerate the number of students who are officially At-Risk. They define this as students who are

“homeless, in the District’s foster care system, qualify for Temporary Assistance for Needy Families (TANF) or the Supplemental Nutrition Assistance Program (SNAP), or high school students that are one year older, or more, than the expected age for the grade in which the students are enrolled.” (That last group is high school students who have been held back at least one time at some point in their school career.)

So, it’s a simple (but tedious) affair for me to plot the percentage of such at risk students, at each of the roughly 200 publicly-funded schools in Washington, DC, versus the average percentage of students who were proficient or advanced in math and reading on the 2014 DC-CAS.

I was rather shocked by the results. Here are my main conclusions:

1. For almost all of the schools, to get a rough idea of the percent of students passing the DC-CAS, simply subtract 90% minus the number of students ‘At-Risk’. The correlation is very, very strong.

2. There are only THREE DC charter schools with 70% or more of their students At-Risk, whereas there are THIRTY-ONE such regular public schools. So much for the idea that the charter schools would do a better job of educating the hardest-to-reach students (the homeless, those on food stamps, those who have already failed one or more grades, etc).

3. The only schools that have more than 90% of their students ‘passing’ the DC-CAS standardized tests remain, to this day, the small handful of schools in relatively-affluent upper Northwest DC with relatively high percentages of white and Asian students..(Unless you include Sharpe Health school, where students who cannot feed or dress themselves or hold a pencil are somehow deemed ‘proficient’ or ‘advanced’ by methods I can only guess at…)

4. As I’ve indicated before, it appears that for the most part, DC’s charter schools are mostly enrolling smaller percentages of At-Risk, high-poverty students but higher fractions of the students in the middle of the wealth/family-cohesion spectrum than the regular DC public schools. There are a few exceptions among the charter schools: BASIS, Yu Ying, Washington Latin and a few others are succeeding in attracting families and students at the high end of the socio-economic and academic scales.

5. It looks like we are now turning into a tripartite school system: one for affluent and well-educated familes (relatively high fractions of whites and Asians; mostly but not all in regular Ward 3 public schools); one for those in the middle (mostly blacks and hispanics, many enrolled in charter schools), and one for those at the seriously low end of the socio-economic spectrum, overwhelmingly African-American, largely At Risk, and mostly in highly-segregated regular public schools.

Very, very sad.

Here is the graph that sums it all up. Click on it to see a larger version.

bicolor, at risk vs average dc cas 2014 proficiency, both regular public and charter, dc

In blue we have the regular public schools of Washington DC for which I have DC-CAS data for 2014, from grades 3 through 8 and grade 10. In red we have the privately-run but publicly-funded charter schools. Along the horizontal axis, we have the percentage of students who are officially At Risk as defined by the DC CIty Council. Along the vertical axis, we have the average percentage of students who scored ‘proficient’ or ‘advanced’ in math and reading on the DC-CAS at those schools. The green line is the line of best fit as calculated by Excel. Notice that the data points pretty much follow that green line, slanting down and to the right.

To nobody’s surprise, at both the charter and regular public schools, on the whole, the greater the percentage of students at a school who are At Risk, the smaller the percentage of students who ‘pass’ the DC-CAS standardized tests.

The colors do help us see that at the far right-hand end of the graph, there are lots of blue dots and only a small number of red ones. This means that the vast majority of schools with high percentages of At Risk students are regular DC public schools. You could interpret that to mean that parents in more stable families in those neighborhoods are fleeing from what they see as the bad influence of potential classmates who are extremely poor, homeless, have already repeated a grade, and so on, and are flocking to charter schools who have the freedom to expel or ‘counsel out’ such students and to impose a relatively strict behavior code that the DC Council forbids the regular public schools from using. (Their latest initiative is to forbit ALL out-of-school suspensions, no matter what…)

Dots that are above the slanted green line supposedly represent schools that are doing a better job at teaching to the tests than would be predicted by the At-Risk status alone. Dots below the line are doing a worse job than would be predicted. Notice that there are dots of both colors both above and below the line.

=====

I wish to thank the indefatigable Mary Levy for collecting and passing on this data. You can find the original data source at the OSSE website, but I’ve saved the larger table (all 2008-2014 DC-CAS data) on Google Drive at this link. I took the average of the percentage of students ‘passing’ the DC-CAS in math and in reading as the proficiency rate. The note on the at-risk data table reads as follows:

Data Source: SY2013-14 student-level data from OSSE. The list includes DCPS traditional, DCPS citywide specialized, DCPS selective schools, and public charter schools, but excludes any DCPS or public charter adult education or alternative school. The definition of at risk students includes students who are homeless, in the District’s foster care system, qualify for Temporary Assistance for Needy Families (TANF) or the Supplemental Nutrition Assistance Program (SNAP), or high school students that are one year older, or more, than the expected age for the grade in which the students are enrolled.

The Real Lesson of Singapore Math!

By now you’ve probably heard that Singapore and Shanghai are the two places on earth with the smartest kids in the entire world. We can see their PISA scores (go to page 5) are right at the top.

Case closed, right? Whatever they are doing in education, we in the US need to emulate that in order to catch up! Common Core! StudentsFirst! Teach for America! Race to the Top! PARCC! Bust those teacher unions! No more recess! All test prep all the time! Charter Schools! Turn the schools over to the billionaires (Gates, Bloomberg, Koch family, Walton family, and their hirelings and shills)!

But wait a second.

Have you noticed that an ENORMOUS fraction of the low-skilled, low-paid people living in Singapore are temporary foreign workers from various parts of Asia and Africa and are not allowed to bring their kids with them? Those kids are raised back in the workers’ homelands by various relatives, far away, and only get to see their parents at long intervals (somebody has to fly somewhere); back home, jobs are even scarcer and worse-paid, so the parents go elsewhere to try support their families.

Now, everywhere in the world, family income is very, very closely linked to children’s test scores in school. It’s one of the tightest correlations there are in the social sciences, as you can see in the simple scatter-plots I have repeatedly shown in this blog over the past 4 or 5 years. (Try using terms like “poverty” “income” and “scores” together in the search box on this page and be prepared to look through a lot of posts with such graphs, from all over!)

If one-quarter to one-third of the population of a country was legally not permitted to have children in the schools, and it was the low-paying 1/4 to 1/3 of the population, then the scores of the remainder of the kids would, quite naturally, be pretty darned good, since the bottom 1/4 to 1/3 of the distribution just got cut off.

If we systematically excluded the poorest quarter or third of our American student population from taking PISA, we know that our scores would be pretty darned high as well.*

Hmm, maybe the leaning tower of PISA hype is falling.

 

=====================

*Let’s remember that this WAS official policy in many states of the USA up until 1865: a large fraction of the population (guess which one!) was forbidden to send their kids to schools at all and it was explicitly forbidden even to teach them to read privately. When Jim Crow was established from the 1870s to the early 1960s, school facilities for Blacks and Hispanics, BY DESIGN of the racist authorities, so inferior to those for whites that they were a national disgrace. Which is why the calls for going back to the good old days should be so infuriating. There WERE NO GOOD OLD DAYS.

Daily Howler on how the editors and reporters at the NYTimes get education right – or wrong

Two articles this past week by Bob Somerby on how at least one reporter (Motoko Rich) does a good job reporting on what is right and what isn’t right with education in America, and how the NYT editorial staff doesn’t get it right at all. But then again, the editors all went to cushy private schools and elite colleges and have mansions in the Hamptons, so they don’t have a clue as to how the other half lives.

First daily howler article here.

Second daily howler article here.

Published in: on March 29, 2014 at 3:06 pm  Leave a Comment  
Tags: , , ,

What I actually had time to say …

Since I had to abbreviate my remarks, here is what I actually said:

I am Guy Brandenburg, retired DCPS mathematics teacher.

To depart from my text, I want to start by proposing a solution: look hard at the collaborative assessment model being used a few miles away in Montgomery County [MD] and follow the advice of Edwards Deming.

Even though I personally retired before [the establishment of the] IMPACT [teacher evaluation system], I want to use statistics and graphs to show that the Value-Added measurements that are used to evaluate teachers are unreliable, invalid, and do not help teachers improve instruction. To the contrary: IVA measurements are driving a number of excellent, veteran teachers to resign or be fired from DCPS to go elsewhere.

Celebrated mathematician John Ewing says that VAM is “mathematical intimidation” and a “modern, mathematical version of the Emperor’s New Clothes.”

I agree.

One of my colleagues was able to pry the value-added formula [used in DC] from [DC data honcho] Jason Kamras after SIX MONTHS of back-and-forth emails. [Here it is:]

value added formula for dcps - in mathtype format

One problem with that formula is that nobody outside a small group of highly-paid consultants has any idea what are the values of any of those variables.

In not a single case has the [DCPS] Office of Data and Accountability sat down with a teacher and explained, in detail, exactly how a teacher’s score is calculated, student by student and class by class.

Nor has that office shared that data with the Washington Teachers’ Union.

I would ask you, Mr. Catania, to ask the Office of Data and Accountability to share with the WTU all IMPACT scores for every single teacher, including all the sub-scores, for every single class a teacher has.

Now let’s look at some statistics.

My first graph is completely random data points that I had Excel make up for me [and plot as x-y pairs].

pic 3 - completely random points

Notice that even though these are completely random, Excel still found a small correlation: r-squared was about 0.08 and r was about 29%.

Now let’s look at a very strong case of negative correlation in the real world: poverty rates and student achievement in Nebraska:

pic  4 - nebraska poverty vs achievement

The next graph is for the same sort of thing in Wisconsin:

pic 5 - wisconsin poverty vs achievement

Again, quite a strong correlation, just as we see here in Washington, DC:

pic 6 - poverty vs proficiency in DC

Now, how about those Value-Added scores? Do they correlate with classroom observations?

Mostly, we don’t know, because the data is kept secret. However, someone leaked to me the IVA and classroom observation scores for [DCPS in] SY 2009-10, and I plotted them [as you can see below].

pic 7 - VAM versus TLF in DC IMPACT 2009-10

I would say this looks pretty much no correlation at all. It certainly gives teachers no assistance on what to improve in order to help their students learn better.

And how stable are Value-Added measurements [in DCPS] over time? Unfortunately, since DCPS keeps all the data hidden, we don’t know how stable these scores are here. However, the New York Times leaked the value-added data for NYC teachers for several years, and we can look at those scores to [find out]. Here is one such graph [showing how the same teachers, in the same schools, scored in 2008-9 versus 2009-10]:

pic 8 - value added for 2 successive years Rubenstein NYC

That is very close to random.

How about teachers who teach the same subject to two different grade levels, say, fourth-grade math and fifth-grade math? Again, random points:

pic 9 - VAM for same subject different grades NYC rubenstein

One last point:

Mayor Gray and chancellors Henderson and Rhee all claim that education in DC only started improving after mayoral control of the schools, starting in 2007. Look for yourself [in the next two graphs].

pic 11 - naep 8th grade math avge scale scores since 1990 many states incl dc

 

pic 12 naep 4th grade reading scale scores since 1993 many states incl dc

Notice that gains began almost 20 years ago, long before mayoral control or chancellors Rhee and Henderson, long before IMPACT.

To repeat, I suggest that we throw out IMPACT and look hard at the ideas of Edwards Deming and the assessment models used in Montgomery County.

Charter Schools Have Failed to Close the ‘Achievement Gap’ their Backers Claim they would Crush.

I am reposting an article that Diane Ravitch brought to my attention, but I’m deleting the crappy and incorrect headline. I am emphasizing a few parts that I think are key, since I think the Levines “buried the Lede” as a reporter would say.

They also could have used a few graphs to illustrate what they meant.

By Adeline Levine and Murray Levine

SPECIAL TO THE News

on October 13, 2013 – 12:01 AM

Charter schools are hailed by the U.S. Department of Education, by major foundations, and by corporate and philanthropic organizations as the prime solution to the alleged failures of traditional public schools to educate children, failures underscored by the poor performance of their minority and disadvantaged students.

Four large-scale studies by two respected research institutes, CREDO and Mathematica, comparing charter schools with traditional public schools were reported in 2013. Major newspapers, apparently relying on the press releases, trumpeted that charter schools had shown astonishing results in closing the achievement gap between disadvantaged and not-disadvantaged students.

Achievement tests are the major yardstick used to assess schools. CREDO conducted three national evaluation studies comparing the achievement test performance of students in charter schools with matched students in traditional public schools. Mathematica studied middle schools in the well-regarded KIPP charter school chain. All four studies compared the amount of “gain” or “growth” in achievement test scores over a school year, not the actual levels of achievement. Even with gains, the achievement level may still be well below norms for the test.

Buried deep in its report, one CREDO study states, “Only when the annual learning gain of these student [minority/poverty] subgroups exceeds that of white or non-poverty students can progress on closing the achievement gap be made.” Charter school minority and economically disadvantaged students made some very small gains in reading and math when compared to matched controls in public schools. However, the difference in achievement growth between white non-poverty students in traditional public schools and minority/poverty students in charter schools is the most relevant comparison.

The average gain, in standard deviation units, for minority or poverty students in charter schools when compared to their counterparts in traditional public schools, was about 0.03. However, the average gain for non-minority, non-poverty traditional public school white students was 0.80. The gain was up to 27 times the gain for poverty or minority students in charter schools. The Mathematica study of KIPP middle schools showed similar large gaps in gains.

The CREDO Institute states: “For many charter school supporters, improving education outcomes for historically disadvantaged is the paramount goal.” While all of the groups in both kinds of schools show gains over the years, the achievement gap remains, as it always has when students from homes in poverty are compared to non-poor ones, in this country and internationally. The “paramount goal” to level the field is not being met by charter schools.

Charter school advocates attribute the educational difficulties of disadvantaged students in traditional public schools to ineffective, uncaring teachers, their unions and bureaucratic restrictions. They insist that having a great teacher in every classroom will overcome every limitation. They claim that low expectations for disadvantaged children are the major problem, not the complex negative effects of poverty.

Charter schools are not hindered in their selection of teachers by bureaucratic restrictions, nor are charter school teachers prevented by union restrictions from pursuing the charter school programs. Allegedly, charter schools have great teachers in every classroom. If there are “no excuses” when disadvantaged students do less well than non-disadvantaged students in traditional public schools, the same rules should apply to charter schools.

What excuse do charters have for the persistent achievement test gap between disadvantaged students in charter schools compared to non-disadvantaged students in the public schools? And why continue down a path where the numbers show that the national policy favoring charter schools will make the majority-minority gap worse?

Charter schools are protected by powerful, wealthy individuals and foundations that profess free-market choice and hold anti-union sentiments and pro-privatization beliefs; some advocates are pursuing profit motives. The advocates seem not to be influenced by data despite their insistence they are data-driven.

The reality is that problems associated with a history of discrimination and the complex negative effects of poverty are not easily solved. The solutions require an enormous, long-term societal commitment. The current reforms, however, threaten the very existence of our public schools, which have long been the envy of the entire world.

Adeline Levine, Ph.D., is professor emeritas (sociology) at the University at Buffalo. A former chairwoman of the department, she is the author of “Love Canal: Science, Politics and People,” and other books and articles on educational subjects. Murray Levine, J.D., Ph.D., is distinguished service professor (psychology) emeritus at UB. He has published extensively on educational subjects.

Published in: on October 19, 2013 at 7:04 pm  Leave a Comment  
Tags: , , , , ,

Spreadsheet for DC scores (poverty, segregation, public vs. charter)

If you want to see the spreadsheet I made and used from the District of Columbia’s Office of the State Superintendent of Education showing the links between poverty, segregation, and test scores in 2013m, you can look at it on Google Drive by employing this URL:

https://docs.google.com/file/d/0B1ZJFar_AuNBV21OazZTcEh5a0k/edit?usp=sharing

or

http://tinyurl.com/lahg74t

 

Please let me know if the link doesn’t work.

%d bloggers like this: