PISA International Test Results Are Rigged

If you read the article, you see how the international student tests known as PISA are rigged. It’s rather simple: the high-scoring countries choose their wealthiest cities; in those cities, they choose the highest-performing schools; and at those schools they don’t let the low-performing students take the test.

In this way, Washington, DC could be the highest scoring “state” in the USA if it only allowed the highest-scoring kids from, say, Janney, Murch, Deal, Walls, BASIS, St Albans and Sidwell participate. Easy-peasy!

The Chinese government could give lessons to Cheeto45 on how to obfuscate and lie.

Advertisements

Compare ‘Education Reform’ to Ineffective but Profitable Quick-Weight-Loss Schemes

John Viall compares the past 15 years of education ‘reform’ to the past 30 or 40 years of completely counterproductive weight-loss schemes — in both cases, the results are exactly contrary to what they were promised to be. In one case, we can see that America’s obesity rates are some of the worst in the world. In the other, we have certainly not ‘raced to the top’ on TIMMS, PISA, or any other international test, despite all of promises by both the Bush and Obama administrations.

He concludes (I added some color):

“For a sixth time the PISA test was administered in 2015.

Now, 15-year-olds from seventy countries and educational systems took the test. How did U. S. students fare?
The envelope please.
In reading U. S. students scored 497. In other words, after fifteen years of school reform and tens of billions wasted, reading scores were still down seven points.
Fifteen years of listening to blowhard politicians—and U. S. students averaged 470 in math, a depressing 23-point skid.
Surely, all that meddling must have done some good? No. Science scores averaged 496, still down three points.
Fifteen years of diet plans that couldn’t possibly fail and, metaphorically, we were all just a little more fat.
PISA scores had been the foundation on which all school reform was built; and after all these years, America’s 15-year-olds were scoring 33 points worse.

What Lessons Has DC Drawn From PISA?

Basically, the lessons drawn by those in charge of education in Washington, DC, is to do exactly the opposite of everything being done by nations with high test scores. Valerie Jablow at EducationDC explains the details.

The Real Lesson of Singapore Math!

By now you’ve probably heard that Singapore and Shanghai are the two places on earth with the smartest kids in the entire world. We can see their PISA scores (go to page 5) are right at the top.

Case closed, right? Whatever they are doing in education, we in the US need to emulate that in order to catch up! Common Core! StudentsFirst! Teach for America! Race to the Top! PARCC! Bust those teacher unions! No more recess! All test prep all the time! Charter Schools! Turn the schools over to the billionaires (Gates, Bloomberg, Koch family, Walton family, and their hirelings and shills)!

But wait a second.

Have you noticed that an ENORMOUS fraction of the low-skilled, low-paid people living in Singapore are temporary foreign workers from various parts of Asia and Africa and are not allowed to bring their kids with them? Those kids are raised back in the workers’ homelands by various relatives, far away, and only get to see their parents at long intervals (somebody has to fly somewhere); back home, jobs are even scarcer and worse-paid, so the parents go elsewhere to try support their families.

Now, everywhere in the world, family income is very, very closely linked to children’s test scores in school. It’s one of the tightest correlations there are in the social sciences, as you can see in the simple scatter-plots I have repeatedly shown in this blog over the past 4 or 5 years. (Try using terms like “poverty” “income” and “scores” together in the search box on this page and be prepared to look through a lot of posts with such graphs, from all over!)

If one-quarter to one-third of the population of a country was legally not permitted to have children in the schools, and it was the low-paying 1/4 to 1/3 of the population, then the scores of the remainder of the kids would, quite naturally, be pretty darned good, since the bottom 1/4 to 1/3 of the distribution just got cut off.

If we systematically excluded the poorest quarter or third of our American student population from taking PISA, we know that our scores would be pretty darned high as well.*

Hmm, maybe the leaning tower of PISA hype is falling.

 

=====================

*Let’s remember that this WAS official policy in many states of the USA up until 1865: a large fraction of the population (guess which one!) was forbidden to send their kids to schools at all and it was explicitly forbidden even to teach them to read privately. When Jim Crow was established from the 1870s to the early 1960s, school facilities for Blacks and Hispanics, BY DESIGN of the racist authorities, so inferior to those for whites that they were a national disgrace. Which is why the calls for going back to the good old days should be so infuriating. There WERE NO GOOD OLD DAYS.

Some Released PISA Questions

Yong Zhao and some other commentators have been criticising PISA for a number of reasons, one being that its sample populations are at times ‘gamed’ by two cities (Shanghai and Singapore – and that’s all they are, two cities that import their labor force from elsewhere and neither place educates or tests the children of that labor force) while ignoring the outstanding performance of certain individual US states on the exact same test. In her recent book “The Smartest Kids in the World” Amanda Ripley follows a handful of exchange students to and from the US and thinks that the PISA is a pretty good test and that it predicts real things about how societies are going; she appears to be a great fan of Poland these days.

Looking at some of the questions, I am beginning to have a lot less faith in PISA as a test itself and in those folks who claim that the sky is falling on American education based on our scores.

Some of the questions seem OK, some not. I have no idea whether these released items are of equal difficulty if written in French, Polish, Chinese, English, Arabic, Urdu or Swahili, but let’s pretend they are equivalent.

More importantly I read recently an argument that PISA is *not* in fact a test of creativity and original applications of things learned in school; instead, it IS things learned at school or else IQ-type logic puzzles, Even Rick Hess, a big friend of Michelle Rhee, apparently agrees, to my surprise.

Apparently there ARE tests of creativity that are, supposedly, quite reliable. I haven’t read scholarly critiques of THAT creativity test, but I’ve heard of the concept. I will need to  reserve judgment on the real records of the creativity test, but I did indeed recall that one PISA question I saw really was basically a little math/logic puzzle of a sort that I had seen in various puzzle books.  Let’s see if I can find it.

In any case, now that I’ve seen the sample questions, I have even less sympathy

Just now I went to look for some sample PISA items that have been declassified — i.e. it is legal to discuss and show them to people; nobody will lose their jobs for leaking their contents — as teachers and other school staff are threatened with, no matter how stupid a question might be or how many students complained that the problem didn’t make any sense at all and you saw that they weren’t kidding, yes, the problem makes no sense at all.

Let me show you one PISA test item that I think has a fatal flaw – it doesn’t make sense, because ALL of the answers are possible. Some have a higher probability of being correct, but that’s all.

Here is the question:

SEAL’S SLEEP

A seal has to breathe even if it is asleep in the water. Martin observed a seal for one hour. At the start of his observation, the seal was at the surface and took a breath. It then dove to the bottom of the sea and started to sleep. From the bottom it slowly floated to the surface in 8 minutes and took a breath again. In three minutes it was back at the bottom of the sea again. Martin noticed that this whole process was a very regular one.

After one hour the seal was

  1. At the Bottom
  2. On its way up
  3. Breathing
  4. On its way down

In my opinion, the phrase “Martin noticed that this whole process was a very regular one” does NOT mean the same as “Martin took very careful notes and timed a seal that he had learned to recognize for precisely one hour. What’s more, the water was so transparent that Martin could see everything the seal was doing. At exactly 9:00 AM, the seal was at the surface and took a breath that lasted ____ seconds and then dove … and so on, and then it floated to the top where it surfaced at exactly 9:08 AM, and so on”

Notice the details I added. If you are out in cold coastal waters where I myself have seen some seals during my lifetime, you often can’t see down to the bottom if you are on top; even if you are underwater in scuba gear, you generally can’t see a long way; and if it took this seal THREE WHOLE MINUTES to swim to the bottom going, I suppose, straight down at speed that no human swimmer could possibly achieve without mechanical help of some kind, then it’s gotta be pretty deep water, right? You are going to have an impossible time seeing that seal.

And then how does mythical Martin actually know that it’s the exact same seal?

Come on, now. This is a bullshit question, made up by someone who hasn’t actually watched seals at all. I’ve only watched a few dozen myself, but it’s BS.

And plus: animals do NOT act like clocks. Their behavior is not metronomic: it is influenced by what goes on around them. Even though the problem says “Martin noticed that this whole process was a very regular one”, and even if we allow that that is true, nowhere in the problem does the wording imply the kind of clock-like precise repetition that is required to be able to answer the question.

Plus: it doesn’t really say how loong the seal is breathing, nor does it say how long the seal is at the bottom. All the numbers are very vague. It is impossible to answer the question with the information that is ghiven — we are being asked to guess what the problem-writer really meant.

In my opinion, repeating that pattern as being precisely 3 minutes and zero seconds plus 8 minutes and zero seconds, for exactly 60 minutes, is absurd and unbelievable. Animals are NEVER that regular, as I complained earlier. The cycle will shift, somewhat, and those odd seconds do add up. And, as I said, the writer never told us the elapsed time on the sleeping or on the breathing.

So the question is utterly bogus.

We could talk about the PROBABILITY that the seal was in one of those four categories, but only if we knew a whole lot of information. Any child who has ever observed animals knows that they will not keep up the exact same pattern for a full hour measurable to the exact second, no matter what. Not even if they are imprisoned in a cage or a zoo and go all insane and repetitive will they repeat to the exact second.

Bogus.

Published in: on April 9, 2014 at 1:09 am  Leave a Comment  
Tags:

The Chinese Way to Get High International Test Scores: Exclude Low-Scorers

Here is the secret for getting high scores on tests like PISA, TIMMS and so on: systematically exclude any student likely to produce low scores.

In Singapore, the children of the local indentured servant class and the children of migrant workers who cross from Malaysia every day simply are not counted because they are not permitted to attend schools in Singapore at all.

In China, even though students in a number of provinces are tested and measured on PISA by the OECD, the Chinese government only permits scores to be published from the city of Shanghai — where half of the school-age children simply are not allowed to attend school or receive any services at all, since they theoretically and legally belong to their home town out in the rural provinces somewhere.

I strongly recommend reading this article, on Diane Ravitch’s blog, as well as my wonderfully edifying comment.

http://dianeravitch.net/2013/12/12/tom-loveless-on-shanghai-the-scores-are-rigged-and-oecd-doesnt-care/

and here is the original article by Tom Loveless of the Brookings Institution, and here are a few paragraphs from it:

The only reasonable conclusion is this: officials in Shanghai are only counting children with Shanghai hukous as its population of 15 year-olds, about 108,000.  And the OECD is accepting those numbers.  It is as if the other children, numbering 120,000 or more, do not exist.  This is not a sampling problem.  PISA can sample all it wants from the official population.  Migrant children have been filtered out.  Professor Chan of Washington agrees with this hypothesis, saying in an email to me: “By the time PISA is given at age 15, almost all migrant children have been purged from the public schools.  The data are clear.”

What Now?

As a researcher who studies student achievement, I use PISA data.  That requires trust and confidence in the integrity of the assessment.  I can be confident, for example, that the scores from Portugal are from a representative sample of all 15 year-olds in Portuguese schools.  I have no such faith in PISA scores from China.  PISA-OECD has been silent about its special arrangement with China.  All of the data from 2009 still have not been released.  The data from Shanghai apparently only represent the privileged subset of 15 year-olds who hold Shanghai hukous.  I don’t know for sure. In the four volumes of data on PISA 2012, neither hukous nor the migrant children of China are discussed. Not a word.  Not a peep.

PISA officials are not shy about offering policy advice to countries, especially policies that the OECD believes will promote equity.  Delaying tracking and ability grouping, reforming policies governing immigration, distributing resources so that schools with less get more, and expanding early childhood education—all have been promoted as equity-based policies.  But not a word about reforming hukou.  Not a word on a discriminatory policy affecting the education of millions of Chinese children.  Not a word on the human rights story of migrant families in China and the human suffering that they must endure. 

Published in: on December 12, 2013 at 9:34 am  Leave a Comment  
Tags: , , ,

The PISA results

Most of what I want to write has already been put into print. A usual, Diane Ravitch has done a masterful job of dissecting what they mean or do not mean.

Let me just emphasize a couple of points that I think are particularly important:

1. There is a lot of evidence that being a good test-taker does not necessarily overlap with other desirable properties, either on the individual level or on the local or national or international level.

2. A lot of silly things are read into comparing how many questions they get right in one country versus another.

3. The United States has now TEN FULL YEARS in which it has based essentially ALL educational decisions on test scores, with a small but well-funded and powerful group claiming that it would produce miracles in raising American students’ test scores on every level that they can be measured.

4. Those results have been a complete and utter failure EVEN ON THE TERMS CLAIMED BY this bipartisan group of Educational DEformers (or GERM group): the US scores on the PISA are essentially the same as when PISA started, back in the 1990s. (I’ll locate the graph later and post it.)

5. Every single nation that I’ve looked at has a very large gap in test scores between poor kids and rich kids. Some are even wider than ours.

6. As usual on the PISA, the US is in the middle of the pack. We are kind of an orange, below the UK which is in black.

7. Arne Duncan and his ilk say that the fact that the same approach has failed for 10 straight years, means we need to keep doing it harder. Sensible people would say no, let’s forget about measuring with stupid standardized tests. Let the kids learn, remember that humans LOVE to learn stuff — it’s what we do as a species. And precisely nobody knows what knowledge of today is going to be the most useful or fun tomorrow. So let’s get rid of the idiotic focus on standardized tests and Big Data, and stop wasting so much money and time and energy on them. We’ve got all sorts of art and sports and drama and dance and music and technology and building stuff and real science and history and psychology to learn and to perform.

Pisa 2012 results graphic

Published in: on December 4, 2013 at 11:25 pm  Comments (3)  
Tags:

Shanghai PISA scores not representative of China

Interesting article by a Chinese-American scholar who studies Chinese migratory labor. He points out that the students in Shanghai who took the PISA tests, and who scored at the top of the various nations who engaged in the test, aren’t representative of Chinese students generally. An excerpt:

“… the contrast of the U.S. scores with Shanghai’s is not totally appropriate: It is comparing the entire U.S. population — including many who are on free or reduced-price lunches — with China’s cream of the crop, the Shanghai kids.

Even more important, but far less-known, is that in Shanghai, as in most other Chinese cities, the rural migrant workers that are the true urban working poor (totaling about 150 million in the country), are not allowed to send their kids to public high schools in the city. This is engineered by the discriminatory hukou or household registration system, which classifies them as “outsiders.” Those teenagers will have to go back home to continue education, or drop out of school altogether.

In other words, the city has 3 to 4 million working poor, but its high-school system conveniently does not need to provide for the kids of that segment. In essence, the poor kids are purged from Shanghai’s sample of 5,100 students taking the tests. The Shanghai sample is the extract of China’s extract. A fairer play would be to ask kids at [Alice Deal, Lafayette, Sidwell Friends, NCS, or St. Alban’s*] to race against Shanghai’s kids.”

Here is the link to the article:

http://seattletimes.nwsource.com/html/opinion/2013808513_guest03chan.html

* The author used some fancy private school in Seattle that I’ve never heard of, since I live in the East coast Washington. So I inserted names of some schools I do know something about.

Inequality, Segregation and Education in America vs. the World

This is an excellent analysis of the PISA results, not written by me. It was published in Valerie Strauss’ column in the Washington Post.

What international test scores really tell us:

Lessons buried in PISA report

By Valerie Strauss

This was written by William J. Mathis of Goshen, Vermont. He is the managing director of the National Education Policy Center.. and a former Vermont superintendent. The views expressed are his own.

By William J. Mathis

For the 27th, government officials have yet again been surprised, shocked and dismayed over the latest international test score rankings. Secretary of Education Arne Duncan said, “We have to see this as a very serious wake-up call.” Former Reagan education official Chester E. Finn Jr. reported that he was “kind of stunned” by the results of the Program for International Student Achievement (PISA) results. In hyperbolic overdrive, he compared the results to Pearl Harbor and Sputnik.

The PISA tests were given to 15-year-old students by the Organization for Economic Cooperation and Development (OECD) in 65 nations and educational systems. Nine had higher average scores in reading, 17 in math, and 12 in science.

While ranking nations on test scores is a pretty sorry way to evaluate education systems, there is simply no reason to expect the results to have been any better than they were the last time we heard from this same chorus of surprised, shocked and dismayed pundits and politicians.

The reason is simple. Federal and state policymakers continue to embrace reforms that have little positive effect (if not downright negative effects) while ignoring reforms that make a difference. Buried within the PISA report is an analysis of educational systems that registered high test scores. Here are some of the less-reported findings:·

*The best performing school systems manage to provide high-quality education to all children.

· *Students from low socio-economic backgrounds score a year behind their more affluent classmates. However, poorer students who are integrated with their more affluent classmates score strikingly higher. The difference is worth more than a year’s education.

· *In schools where students are required to repeat grades (such as with promotion requirements), the test scores are lower and the achievement gap is larger.

· *Tracking students (“ability grouping”) results in the gap becoming wider. The earlier the practice begins, the greater the gap. Poor children are more frequently shunted into the lower tracks.

· *Systems that transfer weak or disruptive students score lower on tests and on equity. One-third of the differences in national performance can be ascribed to this one factor.

· *Schools that have autonomy over curriculum, finances and assessment score higher.

· *Schools that compete for students (vouchers, charters, etc.) show no achievement score advantage.

· *Private schools do no better once family wealth factors are considered.

· *Students that attended pre-school score higher, even after more than 10 years.

As OECD Paris-based official Michael Davidson said in National Public Radio comments, “One of the striking things is the impact of social background on (U.S.) success.”

Twenty percent of U.S. performance was attributed to social background, which is far higher than in other nations. Davidson went on to point out that the United States just does not distribute financial resources or quality teachers equally. In a related finding, students from single-parent homes score much lower in the United States than they do in other countries. The 23-point difference is almost a year’s lack of growth.

Our Educational Policies

Unfortunately, federal and state policies do little to adopt these factors that other nations have found so successful. Countless finance studies show that funding across our schools is inequitable and inadequate. Federal and state governments vaguely note this concern but actions do not match the rhetoric. Our treatment of economically deprived students is to house them in segregated schools and shunt them into tracked programs.

A number of “get tough” social promotion policies have been adopted in states even though we know they are harmful. Despite a clear research consensus, early education is still politically disputed. Tracking students still remains the national norm even as we know it increases the achievement gap.

As the federal government (under both Republican and Democratic administrations) has become even more top-down and prescriptive, local schools become less autonomous and less like our successful international counterparts. Finally, the push for privatizing public education through charters, tuition tax credits, vouchers and the like does not result in better test scores and has the effect of increasing segregation, and the inequalities that lead to low test scores.

The American Dream

The American dream is that all children have an opportunity to be successful no matter how humble their roots. Thus, the most troubling finding in the PISA results is the lack of “resilience” among our children.

OECD measured resilience by looking at the scores of the least wealthy 25% of students and seeing what proportion of these students have academic scores in the top 25% of countries with similar socio-economic levels. In the highest scoring nations, 70 percent of the students are rated resilient.

The U. S. figure is less than 30%. In a nation which sees the top 1% controlling more than 50% of the nation’s wealth and the collapse of middle class jobs, we face the specter of building a country of social, economic and educational apartheid.

Secretary Duncan calls the PISA scores a serious wake-up call for our economy and “international competitiveness.” But that is merely to misunderstand economics and global competitiveness. Due to our pursuit of ineffective and ill-focused educational and economic reforms , the rude disturbance of our slumbers is the slamming of the door on the American dream.

Why does no-one mention Finland?

Here is a table that gives the latest PISA results in science, reading, and math.

The NYT highlighted Shanghai, Macao, and Hong Kong, individual cities within China. A number of people have pointed out that Shanghai, in particular, probably attracts some of the brightest students within China.  I don’t think there is any way that American students would tolerate the extremely long hours and hard work that Chinese students put in.

Notice that I highlighted Finland.  Finland used to score at extremely low levels on international tests like this, but has risen dramatically in the past few decades. But NOT by using any of the methods currently popular among the US Educational Deform movement. Instead, they do just about the opposite. See my previous blogs on that.

%d bloggers like this: