– How should transit proposals be evaluated?

How your parents raised you had almost nil effect on you, according to Bryan Caplan, Professor of Economics at George Mason University. A fascinating presentation!

There’s an interesting debate in the latest issue of one of the academic journals about the application of benefit-cost analysis to urban rail transit projects in the US. The principal players are all from various campuses of the University of Southern California – Peter Gordon, Robert Cervero and Lisa Schweitzer. The journal is Public Works Management and Policy (PWMP), a “peer-reviewed journal for academics and practitioners in public works and the public and private infrastructure industries”.  The editor summarises the positions of the protagonists:

Peter Gordon and Paige Kolesar have written A Note on Rail Transit Cost-Benefit Analysis: Do Nonuser Benefits Make a Difference? which examines the rather poor performance of modern rail transit projects when measured in traditional benefit-cost terms. In counterpoint, Robert Cervero and Erick Guerra offer To t or not to t: A Ballpark Assessment of the Costs and Benefits of Urban Rail Transportation where they argue that the benefits of urban rail accumulate over long time horizons and to many who will never use the system. This is a new version of an old conversation and Lisa Schweitzer serves as a referee of sorts with Benefit-Cost Analysis of Rail Projects: A Commentary.

The full articles are gated but the links above provide the abstracts for the Gordon and the Cervero articles. There’s no abstract for Schweitzer’s commentary, however because it’s quite short (2½ pages in the journal), can be read independently and is very good, I’ve posted her piece here complete. She’s a smart, independent and sensible observer of transport issues and I agree with most of her comments in this paper. Like us, the US has some good transit projects, but there are also some that are highly questionable.

The debate in this issue of PWMP reflects a hardy perennial in the transportation community. With some consistency, rail transit fails to meet benefit–cost criteria or ridership forecasts. Planners and transit advocates—often these are the same—respond that benefit-cost analysis is only a partial measure of a project’s worthiness. How, they ask, do you monetize the benefits of something like a trolley that reinvigorates a slumping downtown? Some of the things we could never imagine living without—like the Brooklyn Bridge—would probably have failed a benefit–cost test back in 1866when the New York legislature authorized it. And now the bridge is an architectural icon in a region whose economic health has come to rely at least in part on the aesthetics of investments made more than a century ago.

As vocal as transit advocates have become in dismissing those who question rail investment based on benefit–cost evaluations, rail advocates have more than earned the suspicion that surrounds them. Promise after promise accompanies the push to get federal dollars for local rail transit projects: for example, transit cleans up the air (not so much); it clears up congestion (not even close); it makes us thin (even though study after study demonstrates transit’s minuscule effects on obesity). Whenever anyone points out that projects have not delivered on their promises, then comes the next flood of promises: Jobs, jobs, jobs! Climate change! Building social capital! Economic development! Retail revitalization! At some point, investments have to be accountable for the promises made on their behalf. Guerra and Cervero contend that the mobility benefits accrue to future generations—future riders. If so, that is an empirical claim we can and should test. Some systems over time will jump over the bar their advocates set for them while others are unlikely to do so.

U.S. Secretary of Transportation, Ray LaHood, formally announced a move away from benefit–cost analysis as a criterion for transit investment early in 2010, arguing, “We want to base our decisions on how much transit helps the environment, how much it improves development opportunities and how it makes our communities better places to live” (LaHood, 2010, Press release). Transit advocates met this announcement with applause—applause that seems terribly misguided, for transit delivers little of these joyful outcomes unless it first serves passengers. While we would all love to say that whatever we want governments to provide us transcends mere benefit–cost criteria, ignoring benefit–cost criteria does not help us grapple with the fact that there are rail projects proposed in many disparate locations with many people saying that every single one of them will help the environment/improve development/make our communities better—and an infuriatingly finite, dwindling amount of money with which to pay for those projects. It is a no-win situation. Overemphasize benefit–cost and you risk forgoing a project that, if it works, can really transform a place in wonderful ways. Ignore it and you risk wasting money on projects that do not deliver much of anything besides warm, fuzzy feelings for loudmouth boosters. Among all the issues raised in these notes, perhaps the most nettling concerns Guerra and Cervero’s question: What is the counterfactual? They unfortunately settle for the usual bogeymen—clogged highways and the looming threat of expensive energy. These are easy tropes to fall back on because everyone hates everyone else’s car and too many freeways, and everyone fears high energy costs (but few have even the slightest idea of what the nation’s energy future is going to look like). High energy prices will, indeed, bring on the pain, but they will hurt transit agencies right along with individual drivers. The energy used to power cars and transit may be different, but there is little reason to believe that transit agencies can handle energy cost shocks easily, even with rail in their inventory.

More to the point, transit counterfactuals seem far more relevant for benefit-cost analyses for rail than do highway counterfactuals. As much as rail figures as a substitute for freeways in the popular imagination, there is a simple enough answer to the desire to avoid building more freeways: don’t build more freeways. If the U.S. auto infrastructure is overbuilt, underpriced, and oversubsidized, as many contend, it hardly makes sense to build alternatives to it no matter their benefit–cost ratio, let alone to overbuild, underprice, and oversubsidize those alternatives to balance somehow our grievances about the unfair advantages allegedly enjoyed by the auto infrastructure.

Rather, rail’s counterfactuals really come down to what else we might have done with the money spent on the rail projects. Gordon and his coauthor examine some options including the following:

  • Concentrated investments in rail systems with more established ridership than all these new rail starts in smaller regions; that is, more money for workhorse systems and less for speculative systems; or
  • Geographically more compact networks serviced at greater frequencies.

In some regions of the United States, the latter portends potentially large improvements in transit service quality that could yield far more passenger benefit than more capital intensive investments serving larger geographies but carrying fewer passengers. It could also offer better transit support for infill development. This counterfactual may be particularly germane for the Los Angeles system disputed by these writers.

At the same time that regions propose one new project after another, major U.S. transit providers are in extreme budgetary crisis and cutting back operations. Many transit agencies currently cannot afford to maintain frequencies on the routes they have, and this problem will only become worse if investment decisions are made according to criteria other than ridership. Unless the United States reforms finance for transit operations, those who advocate for costly capital investments everywhere will become like the proverbial bald men fighting over a comb. Guerra and Cervero may be right in that capital investments serve the next generation of urban residents, but transit agencies must budget in the present. Their revenues suffer right along with the reputations of their forecasters the longer it takes riders to get on board. Without new sources of operating funds to support service on new rail extensions, transit agencies are not in a position to serve these routes regardless of their potential contributions to the future.

In the end, analysis helps us clarify the priorities for investment, no matter how painful that process becomes. As true as it is that benefit–cost criteria are only part of decision making, they do help us try to stay accountable for the goals set for investments. In the noisy world surrounding rail—from the noise of those who argue that no rail investment is ever worth what we pay for it to the noise of those who argue every rail investment is always worth whatever we pay for it—the numbers help the rest of us deliberate the hard choices about which projects might really benefit an entire regional system and which do not.

6 Comments on “– How should transit proposals be evaluated?”

  1. brisurban says:

    I think this is great, but I would just add a single comment: It’s not just rail.
    It seems that all of Brisbane’s busways were economically unviable when measured according to traditional CBA methods.

    “The evaluation subsequently undertaken showed a Nett Present Value for the project of -$309M and a Benefit-Cost Ratio of 0.35,
    which lead to the conclusion that the analysis results “show the project to be unviable in economic terms”, based on the evaluation tools commonly used to assess transport projects.
    Similar preliminary evaluations undertaken for the proposed Eastern and Northern Busways also reported Benefit Cost Ratios less than 1.0, generating the need to consider alternative methods of valuing the benefits of busways in Brisbane.”

    The problem is that this doesn’t seem to make sense. The busway carries a huge volume of passengers in the am peak hour and has 150 000 trips done on it per day, which is heaps.

    “Any road project of a similar capital cost to the busway project would be expected to receive a very healthy Benefit Cost Ratio with these levels of patronage.”

    “It is clear that what we observe on the South East Busway is different from what the Benefit Cost Analyses are suggesting.”

    Click to access 1834_paper152-Bitzios.pdf

    • Alan Davies says:

      150,000 per day is phenomenal! However I don’t think that paper actually shows anything inherently inadequate or limiting about BCA, rather it seems to point the finger at the assumptions used in modelling. Those sorts of errors could be made with any evaluation process, including those done on the backs of envelopes. Maybe in this instance there was too much reliance on historical data about buses, forgetting that buses are usually run on marginal routes whereas this is a full-blown CBD commuter corridor?

      • brisurban says:

        We’ll you’re correct if you take the “CBA is a calculator” view of it,
        and the inputs into the calculator are off (though this raises questions about how the patronage was calculated and which could imply questions about density here).

        I’m not sure about historical data; Though the people who designed the Ottawa busway in Ontario Canada had quite a bit of input into the design of the Brisbane busway.
        The Ottawa busway carries 240,000 trips per day (although the peak flow is somewhere around 11 000 pphd, Source: Alain, Mercier, OC Transpo.)

        It seems that perhaps the effects of speed, frequency and network connectivity were perhaps underestimated?
        Certainly the busway is doing much better than anyone imagined. Pity the Brisbane rail network which is a number of
        times larger than the busway but has very poor frequency and hard to get a bus to.

        So the question becomes– how do we know and test that the inputs and assumptions into the CBA calculator are valid?

        • Alan Davies says:

          “So the question becomes– how do we know and test that the inputs and assumptions into the CBA calculator are valid?”

          That’s always going to be somewhat uncertain. But going through the process of attempting to measure and evaluate on some rational basis has got to be better than simply leaving it to whoever shouts loudest. The trick is not to take the ‘black box’ as gospel (which frankly I thought we’d got over decades ago – I’m a little surprised at the way some observers still carry on about the perils of BCA)

          • brisurban says:

            I agree with you there. Though I think a lot of the ‘perils of BCA’ are really perils of fallible human being feeding in off or perhaps even wrong assumptions into models. (Another example springs to mind- Brisbane’s Clem 7 toll road tunnels which has truly under-whelming patronage on that , well below forecasts).

            The measure of the value of something should be it’s utility, except for perhaps emergency infrastructure
            where you hope the day where you have to use it doesn’t come around.

            One thing I find unusual is the apparent lack of alternatives analysis between different modes and car infrastructure like freeways. For example, we have a number of road tunnel projects (pick a random example, Brisbane’s Northern Link Tunnel/Legacy Way), and it would be curious to see the BCA and NPV for that compared against the alternatives (like spending money on improved existing rail and bus services in that area).

  2. Russ says:

    Familiar opponents, but not sure they are adding anything of value that hasn’t been said before. A bit over 10 years ago the BTE put out a very readable, very interesting paper on Facts and Furphies in Transport Economics (pdf). Well worth reading if you haven’t seen it. The impression I got was not that BCA wasn’t valuable, but that it ought to have more focus on the margin of error of its predictions. But maybe the variability of those numbers is itself a problem. Bent Flyvbjerg seems to indicate as much, but he also blames that on the need to show a good BCA result initially – a problem I suspect is significant for federally funded transport projects in the US (and likely to be so for Australian ones done similarly).

    The worry I have with BCA is that it doesn’t deal well with network effects, or rather, future planning. It implicitly (and for good reason) puts a fence around the project to measure its impact, but in transport, future projects (and future growth of the municipality) will have huge impacts on its final value. Cervero seems to argue the same thing, but it is more interesting to know how we might measure it ex ante, rather than ex post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s