Is the Transport Department coming out on outcomes?

xkcd on the value of time (the corresponding figure in Australia is circa 4 minutes)

I don’t ordinarily read annual reports but this story in Melbourne’s Herald-Sun acknowledged its source was the Victorian Department of Transport’s 2010-11 annual report. The newspaper noted a 3.4% increase in reported crime on Victoria’s public transport system in 2009-10. Much to my surprise, the paper acknowledged up front that patronage had also increased over the period, by 1.6%. That’s a pertinent qualification but it surprised me that a News-owned tabloid bothered to make it. Well done to the Hun and reporter Peter Rolfe – a big tick!

Looking at the performance outcomes section of the annual report, I see the Department’s been clever – its performance indicator for reported crime against the person is effectively the change in offences minus the change in patronage growth. Bit hard to ignore that and it certainly helps to put the crime figure in perspective.

This got me wondering what progress the Department is making in dealing with crime on public transport. What, I wondered, is the trend? How does the latest figure compare with the previous year? This was when I discovered some serious deficiencies in the way the Department reports on outcomes.

A key shortcoming is the annual report fails to give the previous year’s outcome figure. So, for example, the report tells us that the total volume of freight carried by rail in Victoria increased by 8.31 billion tonne-kilometres in the most recent year for which data is available, but provides no indication of whether that is an improvement on the trend or a deterioration. What is the reader to make of this number?

Similarly, bicycle path use increased 2.8% and rail capacity utilisation in the morning peak grew by 9.7%, but in neither case is there a figure on the previous year’s outcome. This is in stark contrast with the financials, which use the convention of showing 2011’s numbers against the previous years. The examples I’ve cited aren’t isolated cases – none of the performance outcome results show the previous year.

The importance of context is shown by the Herald-Sun’s report. Peter Rolfe did some legwork and found reported crime dropped 10% in the previous year, while patronage grew 6%. That’s much better than the latest year’s 3.4% vs 1.6%, suggesting a marked deterioration in performance. Although he failed to acknowledge the difference between the two years might in part be due to increased policing (a cross!), he nevertheless illustrates the value of having something against which to compare performance measures.

Unfortunately, the shortcomings of the Department’s annual report get worse. While each indicator has a target “increase” or “reduction”, most of the “results” actually provide a snapshot or balance statistic – they don’t show the change. For example, one outcome indicator purports to show the increase in the proportion of trips starting or ending in Central Activities Areas (CAA). But what we’re told is that 3% of trips started or ended in CAAs in 2009-10. That’s valuable information in its own right, but it’s not telling us if the Department is succeeding on this indicator. It’s not telling us if there was an increase. It’s not telling us what the indicator says it tells us!

Another example is the target to reduce fuel consumption of petrol vehicles – the result is given as 11.4 litres per vehicle kilometre, but again there’s no indication whether things are getting better or getting worse. Likewise, the result for the car occupancy rate indicator is 1.2 persons (it’s meant to reduce); the greenhouse emissions of the vehicle fleet is 384 g/km (it’s meant to reduce); the average weight carried per freight vehicles is 4.19 tonnes (it’s meant to increase); and so on – but do these represent an improvement or a deterioration on the trend?

This is not a minor issue. Of the 34 indicators that claim to show an “increase” in good things or a “reduction” in bad things, 20 do not even attempt to show how they’ve changed. They’re all snapshots. Experts will in many cases know the context but annual reports are prepared for parliamentarians and for the people.

There are some oddities too. The indicators for fatalities and serious injuries suffered by cyclists and pedestrians don’t have targets – perhaps that’s just a “typo”, embarrassing at worst? If not, some explanation is required. A number of the indicators are footnoted with a reference to a report, Victorian Transport Facts 2011, which I can’t find anywhere on DoT’s site. Nor has Google heard of it.

The indicators themselves aren’t always well explained. For example, 12% of motorised trips within Melbourne were made by public transport in 2009-10 (the target is 20% by 2020), but is this metric based on numbers of trips or on kilometres of travel? There’s an important difference – the former is relevant to issues like the capacity of infrastructure, whereas the latter is relevant to issues like the sustainability of travel. The report references the Department’s earlier DOT Plan: 2010 Update but there’s no elaboration of the indicators there.

Kudos to the Department, though, for publishing within four months of the end of the reporting year. And more kudos for abandoning the traditional self-serving narrative of many such reports and instead just giving us the basic facts (well most of them, anyway).

The report is well worth looking at for some of the numbers it provides. For example, while it doesn’t tell us if things are getting better or worse, it does inform us that the State-wide median trip length for train passengers is 18.3 km, whereas for drivers it’s 5.4 km. We can see that total vehicle kilometres of travel in Victoria increased by 2% and that 383 cyclists and 593 pedestrians were seriously injured in road incidents. There’re many more interesting snippets of information if you know how to interpret them.

Overall though, the report flunks its treatment of performance outcomes. This is an important area and demands improvement. I appreciate I haven’t commented on whether the indicators are in any event the “right” ones, but I’ll leave that question to another day.

2 Comments on “Is the Transport Department coming out on outcomes?”

  1. I can provide a small snippet of clarity on the car fuel efficiency. Summed up, cars are no more fuel efficient than they were in the early 60s, and only marginally more fuel efficient than at their worst in the mid 70s.

    Year – Litres/100km
    1963 – 11.4
    1971 – 12.3
    1976 – 12.6
    1979 – 12.5
    1982 – 12.5
    1985 – 12.1
    1988 – 11.9
    1991 – 12.3
    1995 – 11.5
    1998 – 11.7
    1999 – 11.6
    2002 – 11.3
    2004 – 11.5
    2006 – 11.4
    source ABS (1963-2007) Survey of Motor Vehicle Usage via Transport for Suburbia – Paul Mees

    2004 – 11.5
    2005 – 11.7
    2006 – 11.4
    2007 – 11.5
    2010 – 11.3
    Source Survey of Motor Vehicle Use 2011 taken from the ABS website

    Pretty much, every gain in more fuel efficient engines has been offset by added weight to cars.

    Hopefully this trend will start reversing, but the only real driver I see for that happening is some serious increases in the price of petrol. At the moment it seems that for every hybrid or small and efficient car on the road there is also a Hummer or equivalent.

  2. john says:

    For a transit authority, ‘how many people did we carry’ (per cost; per seat; per vehicle trip; per vehicle km of service etc) would appear to be a fair basic performance measure. In many annual reports it’s surprisingly hard to find.

    Higher billing goes to ‘our new equal opportunity employment plan’ or ‘our new system for recycling water in our head office toilets’. Plus the ragbag of frustrating, context-free titbits that you describe.

    It reflects the political view of the annual report as glossy brochure, rather than as a source of userful information.

    There should be a clear set of performance measures which the report reports on in a consistent way from year to year, so a longitudinal picture can be built up. Just like the financials.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s