A must-read article in the New Yorker on exactly how teachers and administrators cheated on the NCLB tests in Atlanta, Georgia.
Yong Zhao and some other commentators have been criticising PISA for a number of reasons, one being that its sample populations are at times ‘gamed’ by two cities (Shanghai and Singapore – and that’s all they are, two cities that import their labor force from elsewhere and neither place educates or tests the children of that labor force) while ignoring the outstanding performance of certain individual US states on the exact same test. In her recent book “The Smartest Kids in the World” Amanda Ripley follows a handful of exchange students to and from the US and thinks that the PISA is a pretty good test and that it predicts real things about how societies are going; she appears to be a great fan of Poland these days.
Looking at some of the questions, I am beginning to have a lot less faith in PISA as a test itself and in those folks who claim that the sky is falling on American education based on our scores.
Some of the questions seem OK, some not. I have no idea whether these released items are of equal difficulty if written in French, Polish, Chinese, English, Arabic, Urdu or Swahili, but let’s pretend they are equivalent.
More importantly I read recently an argument that PISA is *not* in fact a test of creativity and original applications of things learned in school; instead, it IS things learned at school or else IQ-type logic puzzles, Even Rick Hess, a big friend of Michelle Rhee, apparently agrees, to my surprise.
Apparently there ARE tests of creativity that are, supposedly, quite reliable. I haven’t read scholarly critiques of THAT creativity test, but I’ve heard of the concept. I will need to reserve judgment on the real records of the creativity test, but I did indeed recall that one PISA question I saw really was basically a little math/logic puzzle of a sort that I had seen in various puzzle books. Let’s see if I can find it.
In any case, now that I’ve seen the sample questions, I have even less sympathy
Just now I went to look for some sample PISA items that have been declassified — i.e. it is legal to discuss and show them to people; nobody will lose their jobs for leaking their contents — as teachers and other school staff are threatened with, no matter how stupid a question might be or how many students complained that the problem didn’t make any sense at all and you saw that they weren’t kidding, yes, the problem makes no sense at all.
Let me show you one PISA test item that I think has a fatal flaw – it doesn’t make sense, because ALL of the answers are possible. Some have a higher probability of being correct, but that’s all.
Here is the question:
A seal has to breathe even if it is asleep in the water. Martin observed a seal for one hour. At the start of his observation, the seal was at the surface and took a breath. It then dove to the bottom of the sea and started to sleep. From the bottom it slowly floated to the surface in 8 minutes and took a breath again. In three minutes it was back at the bottom of the sea again. Martin noticed that this whole process was a very regular one.
After one hour the seal was
In my opinion, the phrase “Martin noticed that this whole process was a very regular one” does NOT mean the same as “Martin took very careful notes and timed a seal that he had learned to recognize for precisely one hour. What’s more, the water was so transparent that Martin could see everything the seal was doing. At exactly 9:00 AM, the seal was at the surface and took a breath that lasted ____ seconds and then dove … and so on, and then it floated to the top where it surfaced at exactly 9:08 AM, and so on”
Notice the details I added. If you are out in cold coastal waters where I myself have seen some seals during my lifetime, you often can’t see down to the bottom if you are on top; even if you are underwater in scuba gear, you generally can’t see a long way; and if it took this seal THREE WHOLE MINUTES to swim to the bottom going, I suppose, straight down at speed that no human swimmer could possibly achieve without mechanical help of some kind, then it’s gotta be pretty deep water, right? You are going to have an impossible time seeing that seal.
And then how does mythical Martin actually know that it’s the exact same seal?
Come on, now. This is a bullshit question, made up by someone who hasn’t actually watched seals at all. I’ve only watched a few dozen myself, but it’s BS.
And plus: animals do NOT act like clocks. Their behavior is not metronomic: it is influenced by what goes on around them. Even though the problem says “Martin noticed that this whole process was a very regular one”, and even if we allow that that is true, nowhere in the problem does the wording imply the kind of clock-like precise repetition that is required to be able to answer the question.
Plus: it doesn’t really say how loong the seal is breathing, nor does it say how long the seal is at the bottom. All the numbers are very vague. It is impossible to answer the question with the information that is ghiven — we are being asked to guess what the problem-writer really meant.
In my opinion, repeating that pattern as being precisely 3 minutes and zero seconds plus 8 minutes and zero seconds, for exactly 60 minutes, is absurd and unbelievable. Animals are NEVER that regular, as I complained earlier. The cycle will shift, somewhat, and those odd seconds do add up. And, as I said, the writer never told us the elapsed time on the sleeping or on the breathing.
So the question is utterly bogus.
We could talk about the PROBABILITY that the seal was in one of those four categories, but only if we knew a whole lot of information. Any child who has ever observed animals knows that they will not keep up the exact same pattern for a full hour measurable to the exact second, no matter what. Not even if they are imprisoned in a cage or a zoo and go all insane and repetitive will they repeat to the exact second.
The math problem listed here has been making the rounds. It’s supposedly from the common core. If you haven’t seen it, it supposedly shows Jack using some number line to subtract 427 minus 316.
A lot of writers have been dumping on it.
I think they’re missing something — there are at least TWO errors in the work of this imaginary Jack.
The idea of trying to figure out where someone else got something wrong isn’t the worst idea in the world. However. what Jack was allegedly doing would need to be done in the head, because this method is so unwieldy if written out — as many people have pointed out. Also, if that was a carefully printed out number line, then I hope the problem is entirely imaginary, because unless we are teaching about logarithmic plots, then mathematicians take care to make sure that scales are linear (meaning that the distance from 100 to 200 equals the distance from 0 to 100, which are each exactly ten times the distance from 90 to 80 or from 57 to 67.)
As a mental exercise, number lines like this are not an entirely useless method.
Nobody seems to have pointed out that Jack made two math errors, not just one.
Since this problem asks 427-316, if you are doing this in your head, you could either count backwards from 4 by 3 units, or ask yourself how far it is from 3 to 4 — obviously 1. Writing the number line out is a lot of work, but saying silently to yourself, “427, 327, 127″ isn’t much work. So far so good.
But it’s not only in the mathematics that the imaginary Jack made an error:
I’m not sure, but it seems like the problem writer wanted Jack to confuse 16 and 60. This is not hard to do when HEARING the number, but more difficult if you are SEEING it written out. So this makes the problem a bit more, well, problematic, because nowhere in the problem is there any hint that Jack tends to hear poorly.
So this imaginary Jack mistakenly counts backwards in the tens place by tens by going 127, 107, 97, 87, 77, 67, 57 — which appears to be his final answer. Maybe. I can’t quite tell on this sheet.
So that means that “Jack” made another error in leaving out the decade 117.
Since it looks like this problem was written by some low-paid contract worker (think of “call centers” in Malaysia or India) with little scrutiny afterwards, we don’t know if the intention of the problem writer was for the student to realize that Jack is both hard of hearing and mis-counted by skipping the 117? If so, you are really asking a lot of a kid looking at the problem — and notice, if I’m right, then a whole lot of adults missed that point as well.
Did they really intend for the problem to be that difficult?
Sounds like an error on the part of the error-writer, but I could be wrong.
ADDED LATER: If the writer were aware that there were two mistakes in the problem, shouldn’t they have written “Find his error(s)” rather than “find his error”?
So, lots of errors all around.