A must-read article in the New Yorker on exactly how teachers and administrators cheated on the NCLB tests in Atlanta, Georgia.
A must-read article in the New Yorker on exactly how teachers and administrators cheated on the NCLB tests in Atlanta, Georgia.
It’s nice when people you don’t even know back up what you’ve claimed.
This morning, a group I don’t actually recall hearing of before issued a press release saying about what I recently wrote. They agreed that there was little or no agreement between the test results on the NAEP for the public schools in Washington DC — on the one hand — and what DC’s officials claim the DC-CAS standardized tests show.
This group, Better Bolder Approach to Education (BBA), had way more facts and figures than I did, and their preliminary report is about 15 pages long. Here is the meat of their conclusions:
[We, BBA, are ] … producing a report explaining why these gains are exaggerated and, in some cases, non-existent, and how lack of data transparency, combined with cherry-picking specific numbers, has enabled DCPS and OSSE to paint a false picture of progress. Moreover, our report will show clearly that low-income and minority DCPS students (and other groups of disadvantaged students) have, in fact, lost ground to their more advantaged peers in the past few years under Chancellors Michelle Rhee and Kaya Henderson.
They go on to say,
And then they display this graph, which shows that gains in average scale scores have in fact been very, very small:
Keep in mind that these scale scores go from 0 to 99! If after 3 or 4 years of hard work, I found that average scores on my students’ final exams had only risen from a 51 to a 54, I don’t think I would be talking about ‘historic gains’!
Naturally, the DCPS spokesperson belittled the BBA report as follows:
From my own experience, I believe the BBA way more than I do anybody from DCPS bureaucracy, since they are making it harder and harder to find any information at all.
I will conclude by linking to the three blog posts that I recently wrote urging folks NOT to believe the self-serving rhetoric that is sure to emerge from the spokespeople at DC’s Office of the State Superintendent of Education (OSSE), DCPS, or the DC mayor’s office.
Olivia Chapman resigned as a DC public school teacher. Her letter explaining why was posted on Rachel Levy’s blog.
Here is tghe link:
In this installment, I’ll look at the reading scores for the District of Columbia as reported on the National Assessment of Educational Progress (NAEP) and by DC’s Office of the State Superintendent of Education (OSSE) by way of the DC-CAS (Comprehensive Assessment Program), written and published by CTB-McGraw Hill.
You may have noticed that I’ve been reporting ‘Average Scale Scores’ on both tests, rather than achievement levels — proficient, advanced, below basic and basic. It was suggested to me that this would allow us to compare the two tests (national and ‘state’) more directly, since the decision of what score falls in which category is so obviously open to negotiation and ‘fudging’ by powers-that-be.
In any case, I will continue giving Average Scale Scores, first for 4th graders in reading:
As before, the blue scale is the average scale score for DC’s fourth-graders in reading, divided by five so that it would fit on the same grid as the DC-CAS scores for the same subject, same grade level. As before, the DC-CAS scale scores for the fourth grade go from 400 to 499, which I treat as going from 0 to 100, and the NAEP scores go from 0 to 500. As before, I had to find these scores in a variety of places; I gave the URLs in the previous post. Also, for a couple of years, I found two different scores for the same year, so I plotted and reported both of them.
You will notice that since about 1996, which is almost two decades, the scores on the NAEP for successive cohorts of Washington DC fourth-graders have been more or less slowly increasing, and there does not seem to be much difference between that progress before mayoral control of DC schools (labeled “Pre-Rhee”) and after the imposition of mayoral control of the schools (which I labeled “Post-Rhee”).
That’s the blue line.
However, the red line, representing the locally-funded DC-CAS tests for the same grade level, show much more volatility and overall growth, with the jump from 2007 to 2008 being most suspicious of all, knowing what we now know about the degree to which Michelle Rhee instructed each and every principal in DCPS to magically raise test scores or get fired.
Once again, I would much prefer to rely on the federal National Center for Educational Statistics than I would rely on CTB-McGraw Hill or the very-political appointees to DC’s OSSE.
Lastly, I present an image with the same pair of graphs, but for 8th grade reading:
The same comments apply here as with every one of the other three tests. NAEP scores show a little bit of steady growth since 1998 (16 years ago), whereas the DC-CAS seems to show less even but much more impressive growth since 2006.
As usual, I would very much discount what Mayor Gray, Chancellor Henderson, OSSE or CTB-McGraw-Hill have to say. I would recommend that you put much more trust in the federal civil servants at NCES and NAEP.
What about you?
Links to my other articles on this:
Part Three (all reading) — this one right here
Sometimes, looking at the past gives you lots of clues about the future.
From looking at past evidence about the scores of Washington, DC students on both the National Assessment of Educational Progress (NAEP) and on its locally-mandated test (currently known as DC-CAS, or Comprehensive Assessment System), we can make some predictions about the importance (or lack of it) of the results for the 2014 CAS, which were administered about three months ago.
My prediction: the CAS results don’t really matter, one way or the other, because they are so volatile and do not correlate very well with much-more-reliable NAEP. Besides the fact that there have been many documented cases of cheating by adults on the DC-CAS for their own personal gain, it almost seems like the DC-CAS is designed to be manipulated for political and ideological gain, for two main reasons:
(1) In some subjects the DC-CAS has displayed enormous year-to-year score increases that are not at all reflected in the cheat-proof NAEP.
(2) This is despite the fact that between 34% and 42% of all questions are non-scored “anchor” questions designed simply to see if the test gives consistent results from year to year, according to testimony of Emily Durso last September before David Catania’s DC-Council subcommittee on education. Those question’s aren’t scored!
Think about that: something like a third to three-sevenths of all the questions a student is forced to answer isn’t even used to grade the students or teachers. It’s used to help out the testing company.
Which produces unreliable results anyway.
(And even though CTB-McGraw Hill has quite a variety of ways of using statistics to detect cheating by kids or adults, DCPS won’t pay for them to use those methods, I was told by CTB’s head econometrician.)
As promised, here is some of the evidence.
First, we have the DC-CAS (red) and NAEP (blue) scores over the past quarter-century in DC, for 8th grade math students:
Once again, the NAEP scores (in blue) for DC’s 8th graders since 1990 seem to show more-or-less steady growth, especially since the year 2000.
(Keep in mind that in order to plot the NAEP and DC-CAS scores on the same grid, and since the NAEP scores go from 0 to 500 while the DC-CAS scores go from 0 to 100, I divided the NAEP scores by 5.)
The DC-CAS was first administered in 2006, so there are no records from before that year. Note that there are two scores given for the year 2009 (46.0 and 49.8), which are pretty far apart; I can only guess as to why they contradict each other. The two sources are here and here. For the NAEP, the reason for the slightly different scores is quite straightforward: one version of the test allowed accommodations for disabilities, and the other did not.
In any case, up until 2011, the gains by successive cohorts of 8th graders on the DC-CAS math test — whom I often taught when I was a teacher in DCPS — was fairly spectacular, but not at all mirrored by the slow, steady progress shown on the NAEP. And since then, those scores have been fairly flat, especially when we remember that the CAS is scored from 0 to 100 (or, to be technical, from 800 to 899; the fourth-grade test scores go from 400 to 499, and the 10th grade tests are scored from 900 to 999 (probably because they didn’t want to add a fourth digit).
Here are the links to the other parts of this article:
Part One (fourth grade math)
Part Two (8th grade math) — this one right here
Part Three (all reading)
Sometime this month, the Mayor of DC and the Chancellor of the DC Public Schools will make some sort of announcement on how DC public and charter school students did on the DC-CAS (Comprehensive Assessment System) – the test required by Federal law to be given to every single kid in grades 3 through 8 and in grade 10.
I don’t have a crystal ball, and I haven’t developed any sources willing to risk their jobs by leaking the results to me in advance, but I can make a few predictions:
1. If the results look bad, they will be released right before a holiday or a weekend (a basic public-relations tactic that all public officials learn).
2. If the scores as a whole look good, or if there is some part of the trends that look good, that will be highlighted heavily.
3. There won’t be much of a correlation between the trends on the DC-CAS scores and the National Assessment of Ednucational Progress, which has been measuring student achievement in grades 4 and 8 in reading and math since the 1970s by giving a carefully-selected sample of students in DC and across the nation a variety of different test items in math, reading, and a number of other areas.
4. Even though the DC-CAS results won’t be released to the public for a couple more weeks, clearly DCPS officials and Mathematica staff already have them; they have been firing teachers and principals and “adjusting” – with the benefit of hindsight – the rest of their evaluations to fit the DC-CAS scores and the magic secret formula called “Value Added
You may ask, how can GFBrandenburg predict not much of a match between the DC-CAS and the NAEP?
By looking at the track record, which I will share with you.
I present the average scores of all DC students on both the DC-CAS and on the NAEP over the past quarter-century. The NAEP scores for the District of Columbia have either been pretty steady or have been rising slightly.
As far as I can tell, the statisticians at the National Center for Educational Statistics (NCES) who design, administer, and score the NAEP do a fine job of
A. making sure that there is no cheating by either students or adults,
B. making up good questions that measure important topics, and
C. gathering, collating, and reporting the data in an honest manner.
On the DC -CAS, however, we have had many documented cases of cheating (see point A), I have shown that many of the questions are ridiculous and don’t measure what we teachers were supposed to be teaching (see point B), and I hope to show you that whatever they are doing with the scores does not seem to be trustworthy.
Exhibit number one is a graph where I plot the average scale scores of the students in Washington DC on both the NAEP and on the DC-CAS for fourth grade math:
Allow me to explain.
The bottom blue curve is what DC’s fourth-graders average scale scores were on the NAEP starting in 1992 and going on through 2013. As you can see, since 1996, there has been what appears like more-or-less steady improvement.
(It is very hard, in fact, to see much of a difference in trends before mayoral control over the DC schools and after that time. I drew a vertical black line to separate the ‘Pre-Rhee” era from the “Post-Rhee” era, since Michelle Rhee was the very first Chancellor installed in the DC schools, after the annual tests were given in 2007.)
(As noted, the NAEP scale scores go from 0 to 500, but the DC-CAS scores go from 0 to 100. I decided that the easiest way to have them both fit on the same graph would simply be to divide the NAEP scores by 5. The actual reported NAEP scores are in the little table, if you want to examine them for yourself. You can double-check my numbers by looking around at the NAEP and DC OSSE websites — which are unfortunately not easy to navigate, so good luck, and be persistent! You will also find that some years have two different scores reported, which is why I put those double asterisks at a couple of places on those curves.)
But here’s what’s really suspicious: the DC-CAS scores, shown in red, seem to jump around wildly and appear to show tremendous progress overall but also utterly un-heralded drops.
Which is it?
Slow, steady progress since 1996, or an amazing jump as soon as Wonder Woman Rhee comes on the scene?
In my opinion, I’d much rather trust the feds on this. We know that there has been all sorts of hanky-panky with the DC-CAS, as repeatedly documented in many places. I know for a fact that we math teachers have been changing the ways that we teach, to be more in line with the 1989 NCTM standards and the ways that math is tested on the NAEP. It’s also the case that there has been significant gentrification in DC, with the proportion of white kids with highly educated parents rising fairly steadily.
Slow improvement in math scores, going back a quarter of a century, makes sense.
Wild jumps don’t seem reasonable to me at all.
On the contrary, besides the known mass cheating episodes, it almost seems like DC education officials get together with McGraw-Hill CTB, which manufactures the DC-CAS, and decide how they want to get the scores to come out. THEN they decide which questions to count and which ones NOT to count, and what the cut-off scores will be for ‘advanced’, ‘proficient’ and so on.
Next time: 8th grade math; and 4th and 8th grade reading.
Links to my other articles on this:
Part One (fourth grade math)— this one right here
Part Two (8th grade math)
Part Three (all reading)
EduShyster does an excellent job at stripping away the myths that are cloaking the elite-led war on public education and school teachers in particular.
Bob Braun does an excellent job at showing how completely racist and undemocratic it is that Chris Christie can re-appoint a superintendent of schools who pointedly thwarts the expressed will of all of the citizens in the city that she oversees:
National Center for Fair & Open Testing
TO: Journalists Who Cover Education
FROM: Bob Schaeffer, Public Education Director
RE: Testing Resistance & Reform News
DATE: June 24, 2014
What a week!
The impacts of “Testing Resistance & Reform Spring” protests reverberate across the nation with more states suspending testing requirements or pulling out of testing consortia. Bill Gates’ call for a moratorium on some consequences from Common Core exams, quickly implemented by some political allies, reflects another way grassroots power is forcing policy elites to backpedal. To take advantage of this opportunity, parents, educators, students, and community activists need to step up advocacy campaigns to end standardized testing misuse and press for implementation of higher quality performance assessments.
Remember that back issues of these weekly updates are archived at:http://fairtest.org/news — let me know if you want to be added to the regular distribution list
What the Gates Foundation’s Embrace of a High-Stakes Testing Moratorium Really Means
Plan for Common Core Testing “Fractures” As States Withdraw
Common Core National Field Tests Had Major, Known Data Security Flaw
Bid-Rigging Lawsuit Throws $240Million/Year Common Core Testing Contract Into Limbo
Colorado Parent’s Letter: Fixation on Testing Hurts Local Education
DC Public Schools Take “Hiatus” From Test-Based Teacher Evaluation
Supers Question Florida Writing Test Scores
Florida Teacher Who Decried “Toxic Culture of Education” Now Running for School Board
Turmoil Over Indiana School Testing Continues
Indiana Parents Protest Testing Frequency
Louisiana Governor Seeks to Block Common Core Test
Common Core Fight in Louisiana Needs Common Sense
Massachusetts Super: Adding New Tests Fits the Definition of “Insanity”
Debate Over Massachusetts Testing System Ignores Larger Educational Questions
New Jersey Assembly Votes to Put Brakes on Common Core Testing
New York Educator Union Head Seeks Overhaul of Exam-Score-Based Ratings
Common Core Testing Concerns Spur Teacher Evaluation Change
New York Parents Outraged That Testing “Pause” Does Not Include Students
Ohio Legislature Delays Common Core Testing Impact
Rhode Island Legislature Passes Grad Test Suspension Bill
Happy Rhode Island Teenager Was Prime Example for Exit Exam Repeal
South Carolina Retroactively Gives Diplomas to Adults Who Failed High School Grad Test
Tennessee Quits PARCC Testing Consortium
Legislators Seek Tennessee Ed. Head’s Resignation Over Test-Score Manipulation
Texas District Super: Test Does Not Define Your Child
Education “Reform” — A National Delusion
New Analysis Rebuts Claims About Accuracy of “Value-Added” Teacher Evaluation
Cartoon: Transforming Teachers Into Mere Test Proctors
What Real Learning Looks Like (Hint: It’s Not Test-Driven Classrooms)
How Standardized Testing Keeps Us Stuck in the 20th Century
Why Education in Finland Works
“Good Morning Mission Hill” — A Model for a U.S. School Promoting Real Learning
Why Machines Should Never Be Allowed to Grade Student Writing
College to Applicants: We Won’t Look at Your SATs (or ACTs)
Bob Schaeffer, Public Education Director
FairTest: National Center for Fair & Open Testing
office- (239) 395-6773 fax- (239) 395-6779
mobile- (239) 696-0468
With over a decade of data, now can answer quite a few questions about the billionaire-led so-called education reform that has shuttered so many schools, atomized low-income neighborhoods that used to be centered around their local schools, created so much churn in the teaching profession, and turned many urban schools into all-test-prep, all the time.
We can now tell whether students are actually learning more on national tests like the NAEP (National Assessment of Educational Progress) as a result of No Child Left Behind, Race to the Top, and the Common Core.
Along with many other researchers and commentators, I have been showing repeatedly, on this blog, that the answer is, “No.”
My latest piece of evidence comes from two sources: the NAEP Trial Urban District Assessment (TUDA) test scores for 21 urban school systems published by the National Center for Educational Statistics (NCES), and data on the percentages of students enrolled in charter schools in those cities, published by the National Alliance for Charter schools.
We have been repeatedly told that more charter schools means better education for all; supposedly the competition with the charter schools will cause the regular public school systems to improve dramatically as well.
So if you plot the “market share” of charter schools in a bunch of cities against their NAEP math or reading scores in the 4th or 8th grade, you should see a strong positive correlation, something like the data I invented in the graph below:
Well, it happens to be the other way around.
We do not see strongly focused scatterplots with linear correlations going up and to the right.
We instead see strong trendlines going DOWN and to the right.
In the graph above, the value of R-squared is 0.454, and the value of R (the coefficient of correlation ) is 0.6738. For completeness, I also plotted the average score for all US urban students (238) and he average for all US public school students (241). The topmost blue dot on the left represents Detroit, and the next highest dot, at about 43% market share, is Washington, DC. The dot near the bottom center at a NAEP score of 220 is Fresno. The system at the very bottom, with a score of 234, is Louisville KY (aka Jefferson County).
In the next graph (tan/blue), we see the exact same data, only for 8th grade:
In this one, R is almost 0.7, and R-squared is about 0.49, both quite strong correlations.
The very alert reader may notice that this is the graph that I used to make up some phony statistics that DO NOT EXIST but are predicted by many pundits who haven’t been in a public school classroom for a very long time.
My final graph for today shows the same thing, but for 8th grade reading.
All of the correlations have been rather strong, but this one is the strongest of all.
Of course, correlation isn’t necessarily causation. We don’t know from the data alone which factor causes the other, or if there is a third factor causing both changes.
But in any case, the argument that charter schools and choice — as defined by Gates, Wallton, Rhee and Duncan — would inherently lift all boats is definitely demolished.
I am quoting all of Diane Ravitch’s post on this:
Anthony Cody: Is Gates’ Moratorium Real or a Tactic to Defuse Opposition?
When the Gates Foundation issued a press release calling for a two-year moratorium on the use of test scores to evaluate teachers, its position met a mixed reception. Some saw it as a victory for the critics of high-stakes testing; others as an attempt to weaken the critics by deferring the high stakes.
Anthony Cody says, don’t be fooled. The Gates Foundation gives no indication that it understands that its path is wrong, it is simply buying time.
The question we should all be asking is how this one very rich foundation took charge of American education and is in a position to issue policy statements that should be the domain of state and local school boards. What we have lost is democratic control of public education; while no one was looking, it got outsourced to the Gates Foundation.
“As a thought experiment, what would it look like if the Gates Foundation truly was attending to the research and evidence that is showing how damaging the new Common Core tests and high stakes accountability systems are? Would they simply be calling to defer the worst effects of this system for two years?
A real appraisal of the evidence would reveal:
“VAM systems are unreliable and destructive when used for teacher evaluations, even as one of several measurements.
“School closures based on test scores result in no real gains for the students, and tremendous community disruption.
“Charter schools are not providing systemic improvements, and are expanding inequity and segregation.
“Attacks on teacher seniority and due process are destabilizing a fragile profession, increasing turnover.
“Tech-based solutions are often wildly oversold, and deliver disappointing results. Witness K12 Inc’s rapidly expanded virtual charter school chain, described here earlier this year.
“Our public education system is not broken, but is burdened with growing levels of poverty, inequity and racial isolation. Genuine reform means supporting schools, not abandoning them.
“The fundamental problem with the Gates Foundation is that it is driving education down a path towards more and more reliance on tests as the feedback mechanism for a market-driven system. This is indeed a full-blown ideology, reinforced by Gates’ own experience as a successful technocrat. The most likely hypothesis regarding the recent suggestion that high stakes be delayed by two years is that this is a tactical maneuver intended to diffuse opposition and preserve the Common Core project – rather than a recognition that these consequences do more harm than good.”
Moratorium or no, he notes, we are locked into a failed paradigm of testing and accountability. Standards and tests are not vehicles to advance equity and civil rights. If anything, they have become a way to undermine democracy and standardize education.