Judge in NY State Throws Out ‘Value-Added Model’ Ratings

I am pleased that in an important, precedent-setting case, a judge in New York State has ruled that using Value-Added measurements to judge the effectiveness of teachers is ‘arbitrary’ and ‘capricious’.

The case involved teacher Sheri Lederman, and was argued by her husband.

“New York Supreme Court Judge Roger McDonough said in his decision that he could not rule beyond the individual case of fourth-grade teacher Sheri G. Lederman because regulations around the evaluation system have been changed, but he said she had proved that the controversial method that King developed and administered in New York had provided her with an unfair evaluation. It is thought to be the first time a judge has made such a decision in a teacher evaluation case.”

In case you were unaware of it, VAM is a statistical black box used to predict how a hypothetical student is supposed to score on a Big Standardized Test one year based on the scores of every other student that year and in previous years. Any deviation (up or down) of that score is attributed to the teacher.

Gary Rubinstein and I have looked into how stable those VAM scores are in New York City, where we had actual scores to work with (leaked by the NYTimes and other newspapers). We found that they were inconsistent and unstable in the extreme! When you graph one year’s score versus next year’s score, we found that there was essentially no correlation at all, meaning that a teacher who is assigned the exact same grade level, in the same school, with very similar  students, can score high one year, low the next, and middling the third, or any combination of those. Very, very few teachers got scores that were consistent from year to year. Even teachers who taught two or more grade levels of the same subject (say, 7th and 8th grade math) had no consistency from one subject to the next. See my blog  (not all on NY City) herehere, here,  here, herehere, here, here,  herehere, and here. See Gary R’s six part series on his blog here, here, here, here, here, and here. As well as a less technical explanation here.

Mercedes Schneider has done similar research on teachers’ VAM scores in Louisiana and came up with the same sorts of results that Rubinstein and I did.

Which led all three of us to conclude that the entire VAM machinery was invalid.

And which is why the case of Ms. Lederman is so important. Similar cases have been filed in numerous states, but this is apparently the first one where a judgement has been reached.

(Also read this. and this.)

Advertisements

Having Two Separate School Systems Is Wasteful

Peter Greene keeps making the point that having a charter school system along side a public school system is wasteful. One reason is that each system would need its own set of administrators. In Washington, DC, where nearly half of the students now attend charter schools, we now have MORE school buildings than we did when I was in junior high school just over 50 years ago, but only about HALF as many students — thus, a lot of unused space.

Inventing two separate school systems has done essentially nothing to reduce the score gaps between children from white, affluent families (living mostly in upper Northwest) and children from minority, poor families (living elsewhere). The segregation is not quite as awful as it was in the 1960s, but it’s pretty close.

Here is an excellent article from Valerie Strauss’ blog where a DC parent decries the waste, segregation, and general bass-ackwardness of what passes for ‘reform’ in the nation’s capital.

An excerpt:

“Two years ago, when I moderatedthe mayoral education debate, I gave each candidate a math problem:

“–In 1965, the District had 147,000 students and 196 schools. That’s [an average of] 750 kids per school.

“–In 2014, we had 85,000 students and 213 DCPS [D.C. Public Schools] and charter school buildings. That’s [an average of] 399 kids per school.

“That means we have half the kids that we had in the 1960s, and more buildings, many of them gravely under-enrolled. Yet, we still authorize up to 20 new charters per year, and an unclear number of DCPS new schools. Enrollment is flat. At what point do we match school growth with enrollment needs, geographic balance, and transportation planning in mind? At one point do we focus on using data to invest in and manage the schools that we have?”

She also describes

“…the scene I watch from my house near North Capitol Street. It’s straight-up racial apartheid. If I see white children walking to the parks, I knew they are from Mundo Verde or Inspired Teaching schools. The lack of white faces in a group of children makes me know the kids are from Langley, Harmony, or KIPP.”

Reform the Tests! As they are, they don’t test anything important!

A brilliant article by Marion Brady, reprinted by Valerie Strauss at the Washington Post.

Brady points out that what we are actually testing with NCLB, RTTT and so in is worse than useless. What needs to happen is that the tests themselves need to be drastically changed in ways that actually teach higher-order thinking skills. I only quote a small  excerpt to try to get you to read the entire, well-reasoned article:

” If higher order thinking skills are tested, teachers will teach them. Those who don’t know how will quickly learn.

 Of course, Pearson, McGraw-Hill, Educational Testing Service, and other test manufacturers aren’t going to volunteer to test student-initiated higher order thinking skills. Neither are the politicians they help elect and re-elect going to make them even try to do so unless they think voters give them no alternative.

So voters should give them no alternative. Unless politicians and test manufacturers can make a convincing case for not teaching the young to think, they should be told what they’ve been telling teachers who say standardized tests are a waste of time and money: “No excuses!”

It’s likely that nothing short of binding agreements between states and test manufacturers will yield the new tests. To that end, in appropriate legal language, contracts should make clear that (a) every test question in every subject will evaluate a particular, named thinking skill, (b) every test will evaluate a balanced mix of all known thinking skills, and (c) a panel of experts not connected to test manufacturers or politicians will preview all test items to assure contract compliance. No excuses.

Fairtest, Parents Across America, United Opt Out National, and other state and local organizations have strategies in place to try to persuade. Petitions and referendums invite signers. Parents, grandparents — indeed, all who care about kids and country — should get on board.

 No more multimillion dollar checks for tests that no one but manufacturers are allowed to see. No more tests the pass-fail cut scores of which can be raised and lowered to make political points. No more kids labeled and discarded, every one with a brain wired to do all sorts of amazing things. If storing trivia in short-term memory doesn’t happen to be one of those things, that shouldn’t put them out of school and on the street.”

Scooped Again – By the Washington Times in 2007!

Scooped again.

I see now that my own analysis (and dismissal) of Rhee’s claims of stupendous success in the classroom, based on the UMBC study written by Lois Williams and Lawrence Leak in 1995, came about three years later than a fairly objective analysis in the Washington Times, dated June 28, 2007. Here is most of the article, with the parts highlighted that I think are important.

 

==========================

D.C. Mayor Adrian M. Fenty’s choice to reform public schools has been unable to provide proof of the remarkable student improvement she achieved during her brief teaching stint in Baltimore.

“We were told that these kids came in on this level and they were leaving on average at this level,” said D.C. schools chancellor-nominee Michelle A. Rhee, who has noted a dramatic improvement in student test scores in her resume.

“I didn’t think to ask back then for solid documentation or proof or any of those things,” she said. “As a new teacher, I didn’t think those things were particularly relevant.”

Mrs. Rhee, 37, began her three-year teaching career at Harlem Park Community School in the 1992-93 school year through the Teach for America program.

In the 1993-94 school year, when she taught second-graders at the inner-city school, those students had scored at the 13th percentile on standardized tests.

By the end of the 1994-95 year, after Mrs. Rhee had taught the same students as third-graders, 90 percent of them scored at the 90th percentile, according to her resume.

Mrs. Rhee said the test results were achieved on the Comprehensive Test of Basic Skills (CTBS).

Her biographical information on the mayor’s office Web site (http://dc.gov/mayor/news/) and on the Web site of her former nonprofit — the New Teacher Project (www.tntp.org) — both say such “outstanding success” in the classroom earned Mrs. Rhee national media acclaim.

But education experts note that most low-income schools have a high student-turnover rate and Mrs. Rhee taught her students as part of a team. Tying the percentile jump specifically to her is extremely hard to do, they said.

“Although there were some significant gains for third-grade Title 1 students in reading [during Mrs. Rhee’s tenure], there is nothing that would establish a sufficient evaluation link between that particular population of students and any particular individual staff member,” said Ben Feldman, who is in charge of testing for Baltimore schools. “You couldn’t go there.”

In addition, establishing a precise link between student achievement and Mrs. Rhee’s performance in the Baltimore school system is difficult in part because of dated information systems and antiquated storage.

Mr. Feldman said retrieving data from a decade ago is hard because his office changed its information storage systems for the year 2000.

Still, the normal curve equivalent score (which is similar to a percentile) on the CTBS for Harlem Park second-graders was 27 in reading and 43 in math in the 1993-94 school year, according to a 1995 report published by the University of Maryland-Baltimore County.

The report also shows that third-graders at the school for two years achieved a score of 45 in reading and 51 in math in 1994-95. The report does not break down scores by specific class and excludes some students from the totals, including those who received special-education services.

Those scores show significant gains at Harlem Park, but the question remains whether they support the remarkable gains highlighted by Mrs. Rhee and her backers.

“It’s nothing to sneeze at at all,” said Mary Levy, director of the public education reform project for the Washington Lawyers’ Committee for Civil Rights and Urban Affairs. “The only question is where does this 90 come from. Ninety [percent] is amazing. You get that kind of score at schools attended by advantaged children.”

Figures contained in the university study also show that Harlem Park’s elementary enrollment fell from 523 in 1992-93 to 440 in 1994-95.

Mrs. Rhee, who was in her early 20s while at the school, said she did not remember the size of her class.

Her time at Harlem Park coincided with an experiment by the Baltimore school system to let a private company — Education Alternatives Inc. (EAI) — manage nine out of 180 city schools, including Harlem Park.

The Maryland study, which focused on the EAI experiment, and a follow-up report showed that the project elicited little progress in CTBS scores among its students.

==================

Me again:

The whole thing is so bizarre – and typical of Rhee and her star-struck, reality-defying acolytes.

Rhee claimed very specific (and incredible) gains among her students. Admits that she had no actual data whatsoever – she didn’t even remember how many students she had – which means that she made the whole thing up – but pointedly refuses to admit that. She claims that there was national press coverage of her amazing success, and cites various publications (without, however, providing any dates). When I and other investigators look into those publications (such as the Wall Street Journal), we find no such acclaim at all. When I and other investigators look into the actual CTBS data for the time period, we find no evidence whatsoever of any such unprecedented, extraordinary gains.

Then, well-paid, right-wing ideologues like Rick Hess attack me (!!) for supposedly playing loose with the statistics – but exonerate Rhee for making up a pleasing fairy tale, and claim that perhaps her claims are true, but that none of her students were tested (!!!!!). One rather obsessed Rhee-lover, Chris Smyr of Eduwonk (I think), goes on to make the outrageous claim that the principal author of the UMBC study has no right to comment on my conclusions that Rhee made all of her claims for success up!!

Meanwhile, Rhee continues to make claim after claim in the national press — claims that go against all evidence, but which are accepted at face value by almost the entire establishment press, with the exception of Valerie Strauss, a paid blogger/reporter for the Washington Post. Jay Mathews, the most-printed WaPo education writer, appears to realize that Rhee made up nearly all of the claims that got her the job as DCPS chancellor, but he doesn’t quite come to the correct conclusion — which is that she is a fraud from beginning to end.

 

Welcome to the world of the Super-Rich — and to the other world, where the rest of us live

Valerie Strauss has an interesting piece today, written by a middle-class parent who had the opportunity to go to a meeting of some hyper-rich people in New York City having to do with setting up yet another extremely expensive private school in Manhattan for their own privileged children.

The writer draws a number of contrasts with the education of the vast majority of NYC schoolchildren, who attend the public schools.

Here is the URL:

http://voices.washingtonpost.com/answer-sheet/guest-bloggers/why-arent-public-school-parent.html#more

Published in: on February 4, 2011 at 1:50 am  Comments (2)  
Tags: , , , ,

The Secret Behind KIPP? Get Rid of the Low-Scoring Students

And of course, that’s what private schools do, too. You have to take entrance exams to get into St. Alban’s, Sidwell, Maret, Phillips Exeter, and so on.

At KIPP public charter school, they don’t have an entrance exam. But the parents have to sign up for some pretty heavy responsibilities (ones what we wish all parents could do, but we know, unfortunately, that some will not and can not. After all, if you are locked up or dead, it’s pretty hard to come to mandatory Saturday classes!) or the students will not be accepted.

In any case, today’s guest Answer Sheet column in the Washington Post (conducted by Valerie Strauss but written today by Richard Kahlenberg) shows conclusively that the way that KIPP achieves its “success” is by unloading its low-achievers as they progress through middle school. Enrollments drop, and those students are not replaced with new ones.

What’s more, KIPP once tried to take over an existing inner-city school, one in which they were not allowed to select the students. Guess what: they failed, and gave it up. It reminds me of Geoffrey Canada ‘firing’ an entire grade of students in his ‘Harlem miracle’.

Here is the link:

http://voices.washingtonpost.com/answer-sheet/charter-schools/myths-and-realities-about-kipp.html?referrer=emaillink

Inequality, Segregation and Education in America vs. the World

This is an excellent analysis of the PISA results, not written by me. It was published in Valerie Strauss’ column in the Washington Post.

What international test scores really tell us:

Lessons buried in PISA report

By Valerie Strauss

This was written by William J. Mathis of Goshen, Vermont. He is the managing director of the National Education Policy Center.. and a former Vermont superintendent. The views expressed are his own.

By William J. Mathis

For the 27th, government officials have yet again been surprised, shocked and dismayed over the latest international test score rankings. Secretary of Education Arne Duncan said, “We have to see this as a very serious wake-up call.” Former Reagan education official Chester E. Finn Jr. reported that he was “kind of stunned” by the results of the Program for International Student Achievement (PISA) results. In hyperbolic overdrive, he compared the results to Pearl Harbor and Sputnik.

The PISA tests were given to 15-year-old students by the Organization for Economic Cooperation and Development (OECD) in 65 nations and educational systems. Nine had higher average scores in reading, 17 in math, and 12 in science.

While ranking nations on test scores is a pretty sorry way to evaluate education systems, there is simply no reason to expect the results to have been any better than they were the last time we heard from this same chorus of surprised, shocked and dismayed pundits and politicians.

The reason is simple. Federal and state policymakers continue to embrace reforms that have little positive effect (if not downright negative effects) while ignoring reforms that make a difference. Buried within the PISA report is an analysis of educational systems that registered high test scores. Here are some of the less-reported findings:·

*The best performing school systems manage to provide high-quality education to all children.

· *Students from low socio-economic backgrounds score a year behind their more affluent classmates. However, poorer students who are integrated with their more affluent classmates score strikingly higher. The difference is worth more than a year’s education.

· *In schools where students are required to repeat grades (such as with promotion requirements), the test scores are lower and the achievement gap is larger.

· *Tracking students (“ability grouping”) results in the gap becoming wider. The earlier the practice begins, the greater the gap. Poor children are more frequently shunted into the lower tracks.

· *Systems that transfer weak or disruptive students score lower on tests and on equity. One-third of the differences in national performance can be ascribed to this one factor.

· *Schools that have autonomy over curriculum, finances and assessment score higher.

· *Schools that compete for students (vouchers, charters, etc.) show no achievement score advantage.

· *Private schools do no better once family wealth factors are considered.

· *Students that attended pre-school score higher, even after more than 10 years.

As OECD Paris-based official Michael Davidson said in National Public Radio comments, “One of the striking things is the impact of social background on (U.S.) success.”

Twenty percent of U.S. performance was attributed to social background, which is far higher than in other nations. Davidson went on to point out that the United States just does not distribute financial resources or quality teachers equally. In a related finding, students from single-parent homes score much lower in the United States than they do in other countries. The 23-point difference is almost a year’s lack of growth.

Our Educational Policies

Unfortunately, federal and state policies do little to adopt these factors that other nations have found so successful. Countless finance studies show that funding across our schools is inequitable and inadequate. Federal and state governments vaguely note this concern but actions do not match the rhetoric. Our treatment of economically deprived students is to house them in segregated schools and shunt them into tracked programs.

A number of “get tough” social promotion policies have been adopted in states even though we know they are harmful. Despite a clear research consensus, early education is still politically disputed. Tracking students still remains the national norm even as we know it increases the achievement gap.

As the federal government (under both Republican and Democratic administrations) has become even more top-down and prescriptive, local schools become less autonomous and less like our successful international counterparts. Finally, the push for privatizing public education through charters, tuition tax credits, vouchers and the like does not result in better test scores and has the effect of increasing segregation, and the inequalities that lead to low test scores.

The American Dream

The American dream is that all children have an opportunity to be successful no matter how humble their roots. Thus, the most troubling finding in the PISA results is the lack of “resilience” among our children.

OECD measured resilience by looking at the scores of the least wealthy 25% of students and seeing what proportion of these students have academic scores in the top 25% of countries with similar socio-economic levels. In the highest scoring nations, 70 percent of the students are rated resilient.

The U. S. figure is less than 30%. In a nation which sees the top 1% controlling more than 50% of the nation’s wealth and the collapse of middle class jobs, we face the specter of building a country of social, economic and educational apartheid.

Secretary Duncan calls the PISA scores a serious wake-up call for our economy and “international competitiveness.” But that is merely to misunderstand economics and global competitiveness. Due to our pursuit of ineffective and ill-focused educational and economic reforms , the rude disturbance of our slumbers is the slamming of the door on the American dream.

%d bloggers like this: