Yong Zhao and some other commentators have been criticising PISA for a number of reasons, one being that its sample populations are at times ‘gamed’ by two cities (Shanghai and Singapore – and that’s all they are, two cities that import their labor force from elsewhere and neither place educates or tests the children of that labor force) while ignoring the outstanding performance of certain individual US states on the exact same test. In her recent book “The Smartest Kids in the World” Amanda Ripley follows a handful of exchange students to and from the US and thinks that the PISA is a pretty good test and that it predicts real things about how societies are going; she appears to be a great fan of Poland these days.
Looking at some of the questions, I am beginning to have a lot less faith in PISA as a test itself and in those folks who claim that the sky is falling on American education based on our scores.
Some of the questions seem OK, some not. I have no idea whether these released items are of equal difficulty if written in French, Polish, Chinese, English, Arabic, Urdu or Swahili, but let’s pretend they are equivalent.
More importantly I read recently an argument that PISA is *not* in fact a test of creativity and original applications of things learned in school; instead, it IS things learned at school or else IQ-type logic puzzles, Even Rick Hess, a big friend of Michelle Rhee, apparently agrees, to my surprise.
Apparently there ARE tests of creativity that are, supposedly, quite reliable. I haven’t read scholarly critiques of THAT creativity test, but I’ve heard of the concept. I will need to reserve judgment on the real records of the creativity test, but I did indeed recall that one PISA question I saw really was basically a little math/logic puzzle of a sort that I had seen in various puzzle books. Let’s see if I can find it.
In any case, now that I’ve seen the sample questions, I have even less sympathy
Just now I went to look for some sample PISA items that have been declassified — i.e. it is legal to discuss and show them to people; nobody will lose their jobs for leaking their contents — as teachers and other school staff are threatened with, no matter how stupid a question might be or how many students complained that the problem didn’t make any sense at all and you saw that they weren’t kidding, yes, the problem makes no sense at all.
Let me show you one PISA test item that I think has a fatal flaw – it doesn’t make sense, because ALL of the answers are possible. Some have a higher probability of being correct, but that’s all.
Here is the question:
A seal has to breathe even if it is asleep in the water. Martin observed a seal for one hour. At the start of his observation, the seal was at the surface and took a breath. It then dove to the bottom of the sea and started to sleep. From the bottom it slowly floated to the surface in 8 minutes and took a breath again. In three minutes it was back at the bottom of the sea again. Martin noticed that this whole process was a very regular one.
After one hour the seal was
- At the Bottom
- On its way up
- On its way down
In my opinion, the phrase “Martin noticed that this whole process was a very regular one” does NOT mean the same as “Martin took very careful notes and timed a seal that he had learned to recognize for precisely one hour. What’s more, the water was so transparent that Martin could see everything the seal was doing. At exactly 9:00 AM, the seal was at the surface and took a breath that lasted ____ seconds and then dove … and so on, and then it floated to the top where it surfaced at exactly 9:08 AM, and so on”
Notice the details I added. If you are out in cold coastal waters where I myself have seen some seals during my lifetime, you often can’t see down to the bottom if you are on top; even if you are underwater in scuba gear, you generally can’t see a long way; and if it took this seal THREE WHOLE MINUTES to swim to the bottom going, I suppose, straight down at speed that no human swimmer could possibly achieve without mechanical help of some kind, then it’s gotta be pretty deep water, right? You are going to have an impossible time seeing that seal.
And then how does mythical Martin actually know that it’s the exact same seal?
Come on, now. This is a bullshit question, made up by someone who hasn’t actually watched seals at all. I’ve only watched a few dozen myself, but it’s BS.
And plus: animals do NOT act like clocks. Their behavior is not metronomic: it is influenced by what goes on around them. Even though the problem says “Martin noticed that this whole process was a very regular one”, and even if we allow that that is true, nowhere in the problem does the wording imply the kind of clock-like precise repetition that is required to be able to answer the question.
Plus: it doesn’t really say how loong the seal is breathing, nor does it say how long the seal is at the bottom. All the numbers are very vague. It is impossible to answer the question with the information that is ghiven — we are being asked to guess what the problem-writer really meant.
In my opinion, repeating that pattern as being precisely 3 minutes and zero seconds plus 8 minutes and zero seconds, for exactly 60 minutes, is absurd and unbelievable. Animals are NEVER that regular, as I complained earlier. The cycle will shift, somewhat, and those odd seconds do add up. And, as I said, the writer never told us the elapsed time on the sleeping or on the breathing.
So the question is utterly bogus.
We could talk about the PROBABILITY that the seal was in one of those four categories, but only if we knew a whole lot of information. Any child who has ever observed animals knows that they will not keep up the exact same pattern for a full hour measurable to the exact second, no matter what. Not even if they are imprisoned in a cage or a zoo and go all insane and repetitive will they repeat to the exact second.