Limitations of My School

The My School web site launched by the Deputy Prime Minister earlier this year has some important lessons for any disciplines that rely heavily on spatial measures to capture social and economic information.

With two school age children, I’m sympathetic to Julia Gillard’s push for more information to be made available on schools’ performance.  I think providing the NAPLAN information is a good start and I’m looking forward to seeing time-based data next year.

However I’m much less happy with the way that showing the socioeconomic profile of each school is handled.  The Index of Community Socio-educational Advantage (ICSEA) was developed to provide a fairer basis for comparing the NAPLAN performance of different schools.  In simple terms, each school’s ICSEA rating is calculated from the social and economic characteristics of the Census Collection Districts in which its students live.

Not surprisingly, 89% of the 800 students at a prestigious private school like Melbourne Grammar, where Year 12 fees are around $25,000 p.a., live in Collection Districts whose average score on the ICSEA index is in the top quartile.  What is surprising however is that 95% of the 1,200 students at Camberwell High School, which is also located in the inner eastern suburbs, are in the top quartile.

I think it doubtful that this is explained by Camberwell and adjacent suburbs having a very high proportion of wealthy but ideologically driven residents who choose to send their children to a free State high school.  The most plausible explanation is that although the average socioeconomic status of Camberwell and surrounds is very high, there are nevertheless significant numbers of residents whose incomes are too modest to afford a private school for their children.  Thus Camberwell High ranks highly on ICSEA because its students live in a ritzy area, not because the students themselves come from rich families.

This highlights a fundamental risk with areal data – it is highly variable and simply using the average value can be extremely misleading.  The risk is shown in the statistical calculations used to derive the ICSEA index – only 59% of the variation in NAPLAN scores across secondary schools is explained by the index.  That’s pretty good as far as these sorts of statistical exercises go but it still leaves a rather large 41% unexplained.

So when you see statements like “Melbourne’s inner city is characterised by young, high income, professional singles and couples without children” we need to be very conscious that these are average characteristics and may conceal large numbers of residents in other groups.

It’s all self-evident of course, but not so obvious that the managers of My School, the Australian Curriculum Assessment and Reporting Authority, appear to appreciate the limitations of areal-based methods. EDIT: see subsequent post here.

7 Comments on “Limitations of My School”

  1. Damien says:

    I agree that making schools more accountable for their performance is a good thing but NAPLAN is a bad start. By your own analysis its basic ‘anchor’ (ICSEA) is fundamentally flawed.

    I think its back to the drawing board for Julia not that she would ever admit that she is capable of a mistake. That’s Kevin job!


  2. I would say NAPLAN is the anchor of My School, but even so, ICSEA is not up to the job and its hard to see how it could be made adequate.

    I doubt that getting accurate socioeconomic data direct from parents is feasible either, so it might be best if My School just confined this sort of data to a handfull of ‘objective’ measures like remoteness, % indigenous enrolments, income from fees, single sex or not.

    My suspicion however is most parents don’t care a lot about relative advantage rankings – they’re interested in how well their school does on NAPLAN, not why.

    • mc says:

      “My suspicion however is most parents don’t care a lot about relative advantage rankings – they’re interested in how well their school does on NAPLAN, not why.”

      You are probably right, but when the school a parent’s children goes to does badly, there needs to something to assist them to justify keeping their children there…

  3. […] a previous post, Limitations of My School, I pointed to some problems with the Index of Community Socio-educational Advantage (ICSEA) used on […]

  4. […] written about My School before in the context of the ICSA Index (here and here). With the revised My School web site going live on Friday, it’s timely to look at some […]

  5. Johnyboy says:

    I have to agree i think that the schools website is more bang then buck. I would like simple data like. How many students passed VCE and How many students failed at each school. How many students are meeting the requirements of the literacy and numeracy tests at each school.

    I think that simple imformation is important and the rest is not so important. I think the rest of the imformation is mired in politics and debate and subject to ideology.

  6. […] inhabitants to give it a cool profile, they don’t seem to be in the majority. As the My School debacle showed, look beyond the average and suburbs are revealed as diverse places with residents from a […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s