Can selection bias shoot down an argument?

Manhattan in Motion

Urban policy is rich in opportunities for fallacious thinking – for example, surveys that purport to show huge latent demand for a particular mode of transport, but sample only users of that mode. So I’m always interested in new examples of where we can so easily go wrong.

Alex Tabarrok at Marginal Revolution recently provided a pointer to this great historical example by John D. Cook of the importance of selection bias. It isn’t specifically related to urban policy, but nevertheless provides a valuable lesson. Cook says:

During WWII, statistician Abraham Wald was asked to help the British decide where to add armour to their bombers. After analysing the records, he recommended adding more armour to the places where there was no damage!

According to Cook, while it seems backward at first, Wald realised his data came from bombers that hadn’t been shot down:

That is, the British were only able to analyse the bombers that returned to England; those that were shot down over enemy territory were not part of their sample. These (surviving) bombers’ wounds showed where they could afford to be hit. Said another way, the undamaged areas on the survivors showed where the lost planes must have been hit because the planes hit in those areas did not return from their missions.

Wald assumed that the bullets were fired randomly, that no one could accurately aim for a particular part of the bomber. Instead they aimed in the general direction of the plane and sometimes got lucky. So, for example, if Wald saw that more bombers in his sample had bullet holes in the middle of the wings, he did not conclude that Nazis liked to aim for the middle of wings. He assumed that there must have been about as many bombers with bullet holes in every other part of the plane but that those with holes elsewhere were not part of his sample because they had been shot down.

Here’s another example via Marc Gawley, who noted a BBC story implying that three quarters of those committing crimes during the London riots had previous convictions or cautions. Is that really true, he wonders, or:

is it that those with previous convictions have their details on a police database and it was therefore possible to identify (and find) those people based on images from photographs and recordings made last month? Read the rest of this entry »


Do school league tables add any value?

Click to generate Malcolm's next bestseller!

Some new research suggests they do!

One of the arguments against My School is that information on the performance of different schools will be interpreted incorrectly by the press, the public and parents.

The concerns usually relate to fears that schools will ‘teach to the test’; high socioeconomic status parents will abandon under-performing schools; or due regard will not be given to the special conditions that apply to some schools.

While I acknowledge there are risks with performance data, I’m suspicious of anyone who tells me that parents can’t be trusted – supposedly for both their own good and the good of their children – with full information about their child’s school.

I’ve written about My School before in the context of the ICSEA Index (here and here). With the revised My School web site going live on Friday, it’s timely to look at some recently published research on school league tables in Wales and The Netherlands (H/T Alex Tabarrok).

The research on Wales takes advantage of a ‘natural experiment’ – secondary school performance tables that have been published in both England and Wales since 1992 were abolished in Wales in 2001 but maintained in England. The research examines the effects over the subsequent years in both jurisdictions.

The University of Bristol researchers found there is “systematic, significant and robust evidence that abolishing school league tables markedly reduced school effectiveness in Wales”. They estimate the negative impact on schools is equivalent to increasing class size from 30 pupils to 38.

They also find that the negative effect is “concentrated in the schools in the lower 75% of the distribution of ability and poverty” and argue that the “results show that the policy reform in Wales reduced average performance and raised educational inequality”.  Read the rest of this entry »