Texas Decision Slams Value Added Measurements

And it does so for many of the reasons that I have been advocating. I am going to quote the entirety of Diane Ravitch’s column on this:


Audrey Amrein-Beardsley of Arizona State University is one of the nation’s most prominent scholars of teacher evaluation. She is especially critical of VAM (value-added measurement); she has studied TVAAS, EVAAS, and other similar metrics and found them deeply flawed. She has testified frequently in court cases as an expert witness.

In this post, she analyzes the court decision that blocks the use of VAM to evaluate teachers in Houston. The misuse of VAM was especially egregious in Houston, which terminated 221 teachers in one year, based on their VAM scores.

This is a very important article. Amrein-Beardsley and Jesse Rothstein of the University of California testified on behalf of the teachers; Tom Kane (who led the Gates’ Measures of Effective Teaching (MET) Study) and John Friedman (of the notorious Chetty-Friedman-Rockoff study) testified on behalf of the district.

Amrein-Beardsley writes:

Of primary issue will be the following (as taken from Judge Smith’s Summary Judgment released yesterday): “Plaintiffs [will continue to] challenge the use of EVAAS under various aspects of the Fourteenth Amendment, including: (1) procedural due process, due to lack of sufficient information to meaningfully challenge terminations based on low EVAAS scores,” and given “due process is designed to foster government decision-making that is both fair and accurate.”

Related, and of most importance, as also taken directly from Judge Smith’s Summary, he wrote:

HISD’s value-added appraisal system poses a realistic threat to deprive plaintiffs of constitutionally protected property interests in employment.

HISD does not itself calculate the EVAAS score for any of its teachers. Instead, that task is delegated to its third party vendor, SAS. The scores are generated by complex algorithms, employing “sophisticated software and many layers of calculations.” SAS treats these algorithms and software as trade secrets, refusing to divulge them to either HISD or the teachers themselves. HISD has admitted that it does not itself verify or audit the EVAAS scores received from SAS, nor does it engage any contractor to do so. HISD further concedes that any effort by teachers to replicate their own scores, with the limited information available to them, will necessarily fail. This has been confirmed by plaintiffs’ expert, who was unable to replicate the scores despite being given far greater access to the underlying computer codes than is available to an individual teacher [emphasis added, as also related to a prior post about how SAS claimed that plaintiffs violated SAS’s protective order (protecting its trade secrets), that the court overruled, see here].

The EVAAS score might be erroneously calculated for any number of reasons, ranging from data-entry mistakes to glitches in the computer code itself. Algorithms are human creations, and subject to error like any other human endeavor. HISD has acknowledged that mistakes can occur in calculating a teacher’s EVAAS score; moreover, even when a mistake is found in a particular teacher’s score, it will not be promptly corrected. As HISD candidly explained in response to a frequently asked question, “Why can’t my value-added analysis be recalculated?”:

Once completed, any re-analysis can only occur at the system level. What this means is that if we change information for one teacher, we would have to re- run the analysis for the entire district, which has two effects: one, this would be very costly for the district, as the analysis itself would have to be paid for again; and two, this re-analysis has the potential to change all other teachers’ reports.

The remarkable thing about this passage is not simply that cost considerations trump accuracy in teacher evaluations, troubling as that might be. Of greater concern is the house-of-cards fragility of the EVAAS system, where the wrong score of a single teacher could alter the scores of every other teacher in the district. This interconnectivity means that the accuracy of one score hinges upon the accuracy of all. Thus, without access to data supporting all teacher scores, any teacher facing discharge for a low value-added score will necessarily be unable to verify that her own score is error-free.

HISD’s own discovery responses and witnesses concede that an HISD teacher is unable to verify or replicate his EVAAS score based on the limited information provided by HISD.

According to the unrebutted testimony of plaintiffs’ expert, without access to SAS’s proprietary information – the value-added equations, computer source codes, decision rules, and assumptions – EVAAS scores will remain a mysterious “black box,” impervious to challenge.

While conceding that a teacher’s EVAAS score cannot be independently verified, HISD argues that the Constitution does not require the ability to replicate EVAAS scores “down to the last decimal point.” But EVAAS scores are calculated to the second decimal place, so an error as small as one hundredth of a point could spell the difference between a positive or negative EVAAS effectiveness rating, with serious consequences for the affected teacher.

Hence, “When a public agency adopts a policy of making high stakes employment decisions based on secret algorithms incompatible with minimum due process, the proper remedy is to overturn the policy.”

Advertisements

An Immodest Proposal

If you look at the lingo used to justify all the horrendous crap being imposed by “Ed reform”, you’ll see that it’s all couched in lefty-liberal civil rights language. But its results are anything but. Very strange.

Q: Can you cite some examples?

GFB: Yes. From the TFA website:

“Everyone has a right to learn. But in our country today, the education you receive depends on where you live, what your parents earn, and the color of your skin.

“That’s a serious injustice. And in the national movement to right  our contribution is the leadership of remarkable people.

“Our people—diverse and passionate—start in low-income classrooms, where the stakes are highest. We help them become teachers who can dramatically expand students’ opportunities. But our teachers don’t just teach their students, they learn from them.

“They gain a better understanding of the problems and the opportunities in our education system and use those lessons to define their path forward. Many stay in the classroom. Others leave. Both paths matter because to set things right, we need leaders in all areas of education and social justice united in a vision that one day, all kids will have access to an excellent education.”

GFB: However, the way TFA works in practice is that the kids who need the most experienced, skillful teachers, instead get total newbies straight out of college with no teaching experience, no mentoring, and courses on how to teach whatever subject they are they are assigned to. Their five weeks of summer training are mostly rah-rah cheerleading and browbeating. Their only classroom experiences during that summer are a dozen or so hours teaching a handful of kids, **in a subject or grade level totally different from whatever they will be randomly assigned to**.

What underprivileged students do NOT need is an untrained newbie who won’t stick with them. If anything, this policy INCREASES the ‘achievement gap’.

Q: I’m sorry Guy, but none of this poses a solution. Paying the teachers more is not the answer. I know this because I would quit my engineering job in a heartbeat to teach. I honestly would. And I would do it for 1/4 the pay. But not under these  conditions. Not with “father education” telling me how to use fancy calculators to educate kids. Not when you take what I love about math and turn it into garbage. The paradigm sucks, independent of the lousy pay.

GFB: That’s yet another reason to oppose Michelle Rhee. She and her allies have figured out how to micromanage teaching down to the minute and to the very sentences teachers are required to read — from a script. Yes, she and Jason Kamras and Raj Chetty and the other billionaires friends have made it that teachers have no say whatsoever on content or methodology.

If they are not on the same page exactly, down to the minute, they can get marked down, harassed, suspended and fired.

Want to teach under those conditions for twice the pay? Me neither.

It’s not “Teach Like A Champion” as Doug Lemov puts it: it’s teach like a robot.

Q: Plus, their answer to teaching is to integrate technology. They think that if they use technology, everybody will be prepared for the “real world”. Unfortunately, the technology they use isn’t utilized in the real world. So…useless. Somebody needs to tell them this!

GFB:  That’s often true. However I think the teacher should be the one to judge how much technology to use and when. Occasionally we should show them really OLD technology like carving quill pens from turkey feathers, or making their own batteries from copper pennies and galvanized iron…

But you can’t do that with Value Added Measurements and rubrics testing whether you are on the Commin Core Crapiculum to the minute.

I wasn’t really giving THE or even A solution. I was objecting to the solution we are having imposed on us right now. If you want proposed solutions, here goes:

  1. Get people who don’t have actual, extensive teaching or research experience out of the command and control centers of education except as advisors.so, no Michelle Rhee, Andre Agassi, Arne Duncan, Billionaire Broad at the helm.
  2. For our poorer kids, make sure they have free, high quality wraparound services of every kind from the moment their mother notices she’s pregnant.
  3. So for example good well-qualified dentists, ophthalmologists, psychologists, general practitioners, and other doctors should come to each school and check eyes ears nose throat etc and give immunizations to every kid, no more than a single hour of class needs to be missed. If they get hurt on the playground or suddenly vomit in class, it’s really taken care of, right away.
  4. There should be all sorts of remedial help available for kids AS SOON AS help appears to be needed: eyesight, hearing, balance, coordination, mental math, memory improvement, spelling, reading, writing, walking, emotional difficulties, etc. (Right now, the provisions of ADA and IDEA are not funded, so school districts have an incentive to NOT diagnose those with deficiences or learning disabilities, because then they would have to take care of them. Charter schools for the most part just pretend that there are no IEPs.)
  5. Every kid gets a lot of ‘gross motor’ outdoor activities – not just team sports but also things like wilderness hikes, camping, horse care and riding, farming, boating. And music and drama and arts of all sorts – not just for the talented few, but everybody. Lots of after-school activities of these sorts.
  6. Teachers (and parents) should select their principals from among the ranks of the teachers. The principal should also teach, part-time.
  7. Teachers should have at least two years of education theory (and human psychology) and a full year and a half of student teaching, and at least a college major in their area, under experienced mentors. Teachers should be given help o0n how to defuse tense situations and child psychology, and should be chosen from the ranks of those showing
    1. academic promise and
    2. the ability to empathize and
    3. the ability to explain patiently and clearly.
  8. Classes should be much, much smaller. If 12:1 is good enough at Phillips Exeter Academy with their Harkness Tables, why not at Malcolm X ES in far Southeast Washington DC? And if it’s a hands on activity like a chemistry lab or using compasses & straightedges or making birdhouses, get an assistant or two so that it’s more like 3:1.
  9. Let the teachers wrangle over curriculum. State level is fine. County level is fine. School level is fine. To hell with these state-wide standardized tests and curricula, be they bubble type or click and drag.
  10. Actual hands-on vocational training that leads to actual jobs should be available to all who want it, and corporations must engage to hire those grads at decent rates of pay and with promises of additional training.
  11. State-college or  state-university higher education needs to be much, much cheaper. Student debt, like all other debt, should be dischargeable upon bankruptcy, and should be payoff able by many kinds of national service. (Exact provisions TBD, but teaching should definitely be one of those forms of national service. Payments and interest in limbo for the first X years, paid off at Y percent per year, fully paid off after Z years. Exact values of X, Y, Z are TBD.)
  12. Teachers should be paid well enough that they don’t need to get second jobs. Pay in DCPS is not the problem. Working conditions are the problem.
  13. I think that 3-4 hours of personal contact time with kids per day is enough. Planning for each class and heading papers can easily take 2x the amount of class time. So each paper turned in by a student should be returned the next day, marked intelligently.
  14. Since the bosses have their own organizations (NAM, Chamber of Commerce, ALEC, the Koch Brothers network, Council on Foreign Relations, the Cosmos Club, etc) so should the employees. Teachers’ unions should continue to exist but should be more democratic.
  15. Students should, in fact, be held responsible for their success or failure. It’s not all on the teacher, as it is now. Social promotion for a number of years is OK, many countries do it without bad effects, but there should be some sort of a test, I think, of all sorts: practical (eg drawing something, playing a musical piece, climbing a wall, drilling a hole, writing an essay, doing a proof, viewing something under the microscope, etc) as well as a pencil-and-paper or mouse-and-screen test of some sort. Not just arcane reading and math.
  16.  Those who don’t meet the mark should obviously be advised as to what their options are, and those options should be available and well-funded, whatever they might be.
  17. We should strike a balance between having kids go to their walk-to neighborhood schools and having truly integrated schools where each school has a mix of kids of all ethnic groups and incomes. How to do that, exactly , under our current mega-segregated urban patterns, is beyond me. The superhighways and redlinings of the last 80 years are not going to be overcome overnight, but having kids ride for hours to charter schools where there is no neighborhood connection – that’s not the answer.
  18. Anything I left out?

Consumer Math

Since my days as in HS and undergrad, computers have gone from something that was extremely rare but not impossible, all the way up to complete ubiquity in the US and much of the rest of the world (I don’t know exactly how far, but they have now reached tiny villages with no electricity or running water in Benin, west Africa, according to my niece who did a stint in the Peace Corps there: she tells me that lots of village and town folks had cell phones, which are, essentially, small, special-purpose computers. The relative cheapness of computers means that computational tasks that were only done by experts with access to rare computers, some years ago, can be done instantly and easily with Excel or an ordinary scientific or graphing calculator, which is on your smartphone anyway. (A safe bet that you, my reader, have one.)

I was one of the early folks with access to some sort of computer. In 1965, being a scholarship kid at a good New England boarding school, we had access to some early mainframe computer located at Dartmouth College (in NH like my school). We accessed the main computer via WATS lines (a fore-runner of the 800 phone system) and an honest-to-goodness modem that held the handset of the telephone in a cradle. We used some operating system referred to as “time-sharing” which meant that nobody at any of the 10 or 50 or 100 remote or local consoles noticed the presence on the computer of anybody else, which meant that we didn’t have to punch IBM cards and carefully arranged in trays and submit them to a computer operator — and then to await our turn in the queue in “batch mode” – the exact opposite of time sharing.

At that HS, we had a teletype printer that printed everything on a long roll of paper. We loved the thing and even imitated its type font. I was scornful of those kids who wrote simulations of baseball or football games, thinking that wasn’t serious. Boy, was I wrong! The games my schoolmates were writing in our spare time at the computer lab, after sports and classes and before bedtime, fit in our schedule around the time needed for completing our other homework assignments, were incredibly crude by our standards today, of course (the computer would describe the flow of the game in a few words and then ask you if you wanted to throw a fastball, a curve, or a sinker or a knuckleball, and then the computer would work out a pseudo-random number and make a decision as to what happens next (grounder to first, double play, pop fly, safe hit, double, home run, etc). But the genre of computer games has obviously earned its makers and promoters a TON of money over the years and have attracted the very, very best programmers and computer artists of all types. So don’t take everything I say as the gospel truth. Nor the words of anybody else, even your very own self.

(/brag & reminiscence ON/ I won a third place contest when I wrote a program that would do tons and tons of calculations and then print out, on the same roll of teletype paper, a very crude graph of absolutely any equation the user wanted, at any scale that might be desired, using an algorithm that I devised as a modifrication of some spaghetti code I had seen somebody else write. I mean even equations that were as complicated as

Sin(x^4.7*sqrt(tan(y))-log(x^2-y^9)+(e^cos(5x-2y^3))=0.

/brag & reminiscence OFF/)

I remember my brother and lots of sociology or psychology students having to make computer runs in various semi=specialized statistics packages that calculated all sorts of stuff about all sorts of data. (Usually the computer’s output was that you had made some sort of syntax or usage error and you needed to fix the error and try again later, whenever that might be.) I remember collecting IBM punch cards and computer printouts from the recycling bins at the University of Maryland so I could use them for scratch paper and flash cards in my classroom, back in the late 1970s and early 1980s. I also remember having to go to UM or UMUC or GWU or GtnU or CUA or AU or UDC in order to do various programming assignments in various languages . I, personally, only punched a handful of  Fortran cards, treating it as more of a historical curiosity that i was glad I never had to use, similar to the way that folks look at slide rules today.

But there is a difference: I understood exactly how every part of the punch card mechanisms and its electronic alter-egos worked together. I could write the Fortran code (not well, but I took part of a course and taught myself from books, before going on to learn many other computer programming languages. However, I bet that most folks, even those who have just finished a year or semester of working with logarithms, could not explain how or why a slide rule worked.

Let me name a few of the computer languages I have learned, more or less in order, and in no particular order as to how well I learned them:

BASIC (in lots and lots of different variations), COBOL, FORTRAN, Logo (lots of varieties), Pascal, C-64 and Apple 2 assembly and machine language, IDL, Python. Then I said that was enough.

I also learned how to use probably a dozen different types of typewriters and later word processors, and also was quite serious about calligraphy for several decades. Old Underwood standard manual typewriters were quite different from portable or electric typewriters.

Naturally, I also used lots of different types of computational aids, Let’s try listing them:

  • fingers
  • abaci
  • pencil-and-paper (learning and modifying various shortcuts along the way),
  • mental arithmetic (shortcuts, estimation)
  • and an entire panoply of calculators, ranging from the $400 calculator in 1977 dollars that could only do X, /, +, – and not even square root, on up to the ones that we can now program to do all sorts of things, including somewhat basic graphics. And lots of different spreadsheets (I just found some wonderful spiffy videos on how to do all sorts of different tasks using the latest version of Excel. Some of them really do make things much, much easier. But getting accustomed to this new version is, again, a learning process, once again.

I am sure that I am not the only person who thinks that it is not necessary to upgrade all electronic stuff all he time. Sure, the newest versions generally crash less often than the old ones, run faster, and have lots of new features, hut it’s rather expensive to keep having to buy new stuff. It especially doesn’t pay to be an early adopter, especially if you choose wrongly and you end up owning devices that are abandoned by the market and all of your hard-won detailed operating knowledge becomes useless.

Wait. That happens with everything these days!

Including cars. Here’s a little story:

The Subaru Forester my wife and I own was starting to cost a serious amount of money to repair after 10 years and 100,000 miles. We got an estimate from our most dependable mechanic that needed repairs to the window, trim, and transmission would be close to three grand on top of several grand in other repairs over the past year. Plus it would still remain noisy and not have good gas mileage, and the head gasket might be leaking, too.

When a car’s blue book value is not very much greater than the estimated repair bill, my solution is to start looking around for another car.

Not to fix it myself any more. These big jobs were never in my capacity to fix, even back when working on a car was pretty easy. In the old days (1920s through the mid-1980s) it was kind of essential for a guy to understand how to fix stuff on the car, and it often wasn’t too complicated — just really dirty and greasy. (The vast majority of women wouldn’t bother, as you probably remember, which gave rise to lots of jokes.) But today, the situation is different: almost nobody has the equipment, time, and necessary training to work on their own car. If you look under the hood, there isn’t any room left, and it’s extremely complicated to boot. Just to diagnose many problems you need a special-purpose computer.

I remember around 1980 that sometimes you could just disassemble an inoperable part such as a starter motor, clean it out really well, put it back together again, perhaps replace a washer or a nut, then reinstall it, and it would work quite well for some years. Plus, in some older Big 3 cars, one or two smallish people could easily bend over and fit in the engine compartment with the hood closed. So almost everything was easy to reach and take off and re-install.

But today, with modern cars, and going back to at least electronic ignition, a/c, and serious emission controls, all the way up to futuristic cars like the one we just bought (a prius v, level 3), you can’t do any of that.

Yeah, we bought a Prius V, which looks like it’s almost in a Matrix body, and we came to this conclusion based on some simple but useful consumer math. Our old car, the one that just reached 100 thousand miles, was great for many purposes. Perfect when it snows (except we had NONE this winter), fits my very large home-made telescopes and camping gear for remote dark-sky locations, and can hold many other things as well. It worked well up to 100 thosand miles, but then it started falling apart as I described. Our daughter had an old Subaru Forester like we did, and it also fell apart, but after more miles. (anything involving headgaskets is quite expensive!) BUT it had a big downside or two.

The overall gas mileage was around 20 mpg, which I think sucks. (Nearly every time we filled the tank over the years we would estimate what the mileagte (i.e. mpg rating) was. Some times it was a little bit below 20 mpg, sometimes a bit more. 20mpgt as an overall average is close enough and easy to work with. es, there are plenty of bigger cars that guzzle more fuel, but a Forester isn’t all that big. AWD is nice, but it really uses a lot of gas. I was envious of those who drove hybrids, but wasn’t sure they made sense for me, financially and so on.

Plus, Subarus are really noisy. You can’t stop the air from making wooshing sounds hecause of the way they made the windows, at least afrter the fist year or so. Plus, there ae lots o little rattly things all over the underside of the car – my mechanic’s approach was to take them off, one by one.

Now, Feb/March 2012, it began to make sense to buy a Prius, and let me explain why.

At 20 miles per gallon, 100,000 miles consumed roughly 5,000 gallons of fuel. It looks like gasoline is inching up to a long-term average of about $4 per gallon, which is what it used to cost in Europe about 10 years ago. So, using $4/gallon as a rough guide, that five thousand gallons costs $20,000. If I bought another forester, it would probably cost that much to fuel it in the next 10 years, all other things being equal.

However, the larger Prius V is supposed to get 41 to 45 mpg according to the famous EPA estimate. I used 40 mpg to be conservative on this for various reasons. If we drove a Prius 100,000 miles in 10 years and it gets 40 mpg, then it will use 2,500 gallons of gas. And at the same price of $4/gallon, that gas would only clost $10,000, which is half as much.

I don’t know about you, but to me, ten thousand dollars is a lot of money. It made me want to take a look. If I can save ten grand on a new type of car that can do the same thing but better than my old type of car, then that calculated $10K savings on fuel makes a big difference. It meant that the roughly $10K difference in price between a Forester and a Prius V could be essentially discounted.

Why not buy a used car? When we were young and broke (beginning teachers back in the 1970s and 1980s didn’t earn squat) my wife and I had a series of used cars, sometimes gifts from relatives (as we did for our own kids later on) and sometimes from private sellers or dealers. Some of the cars were great, some were absolutely horrible. So horrible that we laugh out loud because retelling the stories is so deliciously funny in a weird sort of way. (You could see the road through the floor! There were wasps in the upholstery! No dipstick, jack, no bald tires, trunk full of leaves!) But it’s really a huge hassle to sell your old car through classifieds or auto magazines or craigslist or whatever, and then to get a new one.

So a new car, traded in, a hybrid Prius V it is.

It’s like driving a computer. (Perhaps you’ve seen this before; I’ve driven in one a few times, but I find this a revelation.) The gas gauge almost never seems to move (exaggeration!). If I want it to, I can have the car tell me at every instant which way the power is flowing, including from the wheels through the electric motor back to the battery when I’m braking. The car’s display often claims that I’m getting over 100 mpg for a 5 minute stretch, especially when I’m coasting downhill or coming to a stop at a light or intersection. I don’t have to use the key in either the ignition nor the door, and I can tell my iphone to call my wife or anybody else. It claims its calculated the mileage at 40.3 mpg overall, but I can’t yet tell if that’s accurate, because I’m still at a half a tank of gas, so I can’t yet do the little elapsed miles divided by gallons purchased estimate to see what my real mpg is since when we bought it. We will see in a week or so.

Math is, truly, everywhere, and is part of our lives.

We all use it; better to use it well, without too much effort, than to use it poorly or only with great effort. I think we could do a much better job in school of showing how useful it really is. By that, I don’t necessarily mean we should teach formal algebra courses in 5th grade. I mean what I wrote, show kids lots of ways that they really do need to use math in many ways that are not in the old-fashioned curriculum more or less put in place in 1893 or thereabouts.

And since computers are, literally, all around us, let us use them in wise and intelligent ways that save us a huge amount of effort, to answer questions that mean a lot to us. Yes, you show kids how to do peprform standard algorithms or certain variations thereof. But you also show them how to use calculators and spreadsheets and other graphic or mathematical electronic tools as well.

Published in: on March 5, 2012 at 12:10 pm  Comments (3)  
Tags: , , ,
%d bloggers like this: