Is the Transport Department coming out on outcomes?

xkcd on the value of time (the corresponding figure in Australia is circa 4 minutes)

I don’t ordinarily read annual reports but this story in Melbourne’s Herald-Sun acknowledged its source was the Victorian Department of Transport’s 2010-11 annual report. The newspaper noted a 3.4% increase in reported crime on Victoria’s public transport system in 2009-10. Much to my surprise, the paper acknowledged up front that patronage had also increased over the period, by 1.6%. That’s a pertinent qualification but it surprised me that a News-owned tabloid bothered to make it. Well done to the Hun and reporter Peter Rolfe – a big tick!

Looking at the performance outcomes section of the annual report, I see the Department’s been clever – its performance indicator for reported crime against the person is effectively the change in offences minus the change in patronage growth. Bit hard to ignore that and it certainly helps to put the crime figure in perspective.

This got me wondering what progress the Department is making in dealing with crime on public transport. What, I wondered, is the trend? How does the latest figure compare with the previous year? This was when I discovered some serious deficiencies in the way the Department reports on outcomes.

A key shortcoming is the annual report fails to give the previous year’s outcome figure. So, for example, the report tells us that the total volume of freight carried by rail in Victoria increased by 8.31 billion tonne-kilometres in the most recent year for which data is available, but provides no indication of whether that is an improvement on the trend or a deterioration. What is the reader to make of this number?

Similarly, bicycle path use increased 2.8% and rail capacity utilisation in the morning peak grew by 9.7%, but in neither case is there a figure on the previous year’s outcome. This is in stark contrast with the financials, which use the convention of showing 2011’s numbers against the previous years. The examples I’ve cited aren’t isolated cases – none of the performance outcome results show the previous year.

The importance of context is shown by the Herald-Sun’s report. Peter Rolfe did some legwork and found reported crime dropped 10% in the previous year, while patronage grew 6%. That’s much better than the latest year’s 3.4% vs 1.6%, suggesting a marked deterioration in performance. Although he failed to acknowledge the difference between the two years might in part be due to increased policing (a cross!), he nevertheless illustrates the value of having something against which to compare performance measures.

Unfortunately, the shortcomings of the Department’s annual report get worse. While each indicator has a target “increase” or “reduction”, most of the “results” actually provide a snapshot or balance statistic – they don’t show the change. For example, one outcome indicator purports to show the increase in the proportion of trips starting or ending in Central Activities Areas (CAA). But what we’re told is that 3% of trips started or ended in CAAs in 2009-10. That’s valuable information in its own right, but it’s not telling us if the Department is succeeding on this indicator. It’s not telling us if there was an increase. It’s not telling us what the indicator says it tells us!

Another example is the target to reduce fuel consumption of petrol vehicles – the result is given as 11.4 litres per vehicle kilometre, but again there’s no indication whether things are getting better or getting worse. Likewise, the result for the car occupancy rate indicator is 1.2 persons (it’s meant to reduce); the greenhouse emissions of the vehicle fleet is 384 g/km (it’s meant to reduce); the average weight carried per freight vehicles is 4.19 tonnes (it’s meant to increase); and so on – but do these represent an improvement or a deterioration on the trend?

This is not a minor issue. Of the 34 indicators that claim to show an “increase” in good things or a “reduction” in bad things, 20 do not even attempt to show how they’ve changed. They’re all snapshots. Experts will in many cases know the context but annual reports are prepared for parliamentarians and for the people. Read the rest of this entry »