Limitations of My SchoolPosted: March 5, 2010
The My School web site launched by the Deputy Prime Minister earlier this year has some important lessons for any disciplines that rely heavily on spatial measures to capture social and economic information.
With two school age children, I’m sympathetic to Julia Gillard’s push for more information to be made available on schools’ performance. I think providing the NAPLAN information is a good start and I’m looking forward to seeing time-based data next year.
However I’m much less happy with the way that showing the socioeconomic profile of each school is handled. The Index of Community Socio-educational Advantage (ICSEA) was developed to provide a fairer basis for comparing the NAPLAN performance of different schools. In simple terms, each school’s ICSEA rating is calculated from the social and economic characteristics of the Census Collection Districts in which its students live.
Not surprisingly, 89% of the 800 students at a prestigious private school like Melbourne Grammar, where Year 12 fees are around $25,000 p.a., live in Collection Districts whose average score on the ICSEA index is in the top quartile. What is surprising however is that 95% of the 1,200 students at Camberwell High School, which is also located in the inner eastern suburbs, are in the top quartile.
I think it doubtful that this is explained by Camberwell and adjacent suburbs having a very high proportion of wealthy but ideologically driven residents who choose to send their children to a free State high school. The most plausible explanation is that although the average socioeconomic status of Camberwell and surrounds is very high, there are nevertheless significant numbers of residents whose incomes are too modest to afford a private school for their children. Thus Camberwell High ranks highly on ICSEA because its students live in a ritzy area, not because the students themselves come from rich families.
This highlights a fundamental risk with areal data – it is highly variable and simply using the average value can be extremely misleading. The risk is shown in the statistical calculations used to derive the ICSEA index – only 59% of the variation in NAPLAN scores across secondary schools is explained by the index. That’s pretty good as far as these sorts of statistical exercises go but it still leaves a rather large 41% unexplained.
So when you see statements like “Melbourne’s inner city is characterised by young, high income, professional singles and couples without children” we need to be very conscious that these are average characteristics and may conceal large numbers of residents in other groups.
It’s all self-evident of course, but not so obvious that the managers of My School, the Australian Curriculum Assessment and Reporting Authority, appear to appreciate the limitations of areal-based methods. EDIT: see subsequent post here.