Tag Archives: accuracy

The cone of uncertainty

One of the elements that makes prediction difficult is uncertainty. In one of the chapters of Donald Shoup’s High Cost of Free Parking (adapted for Access here), Professor Shoup poses the question:

HOW FAR IS IT from San Diego to San Francisco? An estimate of 632.125 miles is precise—but not accurate. An estimate of somewhere between 400 and 500 miles is less precise but more accurate because the correct answer is 460 miles. Nevertheless, if you had no idea how far it is from San Diego to San Francisco, whom would you believe: someone who confidently says 632.125 miles, or someone who tentatively says somewhere between 400 and 500 miles? Probably the first, because precision implies certainty.

Shoup uses this example to illustrate the illusion of certainty present in the parking and trip generation estimates from the Institute of Transportation Engineers. Many of the rates are based on small samples of potentially unrepresentative cases – often with a very wide range of observed parking/trip generation. Shoup’s concluding paragraph states:

Placing unwarranted trust in the accuracy of these precise but uncertain data leads to bad policy choices. Being roughly right is better than being precisely wrong. We need less precision—and more truth—in transportation planning

Part of the challenge is not just knowing the limitations of the data, but also understanding the ultimate goals for policy. David Levinson notes that most municipalities simply adopt these rates as requirements for off-street parking. This translation of parking estimates to hard-and-fast regulation is “odd” in and of itself. What is the purpose of a parking requirement? To meet the demand generated by new development?

Parking demand for a given building will be a range throughout the course of a day and a year, and demand for any given building category will itself fall within a large range. That range is reality, but that unfortunately doesn’t translate into simply codified regulations.

In the previous post, I discussed the challenges of accurate prediction and specifically referenced Nate Silver’s work on documenting the many failures and few successes in accurate forecasting. One area where forecasting improved tremendously is in meteorology – weather forecasts have been steadily improving – and a large part of that is disclosing the uncertainty involved in the forecasts. One example is in hurricane forecasts, where instead of publicizing just the predicted hurricane track, they also show the ‘cone of uncertainty‘ where the hurricane might end up:

Example of a hurricane forecast with the cone of uncertainty - image from NOAA.

Example of a hurricane forecast with the cone of uncertainty – image from NOAA.

So, why not apply these methods to city planning? A few ideas: as hypothesized before, the primary goal for parking regulations isn’t to develop the most accurate forecasts. The incentives for weather forecasting are different. The shifts to embrace uncertainty stems from a desire finding the most effective way to communicate the forecast to the population. There are a whole host of forecast models that can predict a hurricane track, but their individual results can be a bit messy – producing a ‘spaghetti plot,’ often with divergent results. The cone of uncertainty both embraces the lack of precision in the forecast, but also simplifies communication.

For zoning, a hard and fast requirement doesn’t lend itself to any cone of uncertainty. Expressing demand in terms of a plausible range means that the actual requirement would need to be set at the low end of that range – and in urban examples, the low end of potential parking demand for any given project could be zero. Of course, unlike weather forecasts, these regulations and policies are political creations, not scientific predictions.

Meteorologists also have the benefit of immediate feedback. We will know how well hurricane forecasters did within a matter of days, and even then we will have the benefit of several days of iterations to better hone that forecast. Comparatively, many cities added on-site parking requirements to their zoning codes in the 1960s; regulations that often persist today. Donald Shoup didn’t publish his parking opus until 2005.

There’s also the matter of influencing one’s environment. Another key difference between a hurricane forecast and zoning codes is that the weather forecasters are looking to predict natural phenomena; ITE is trying to predict human behavior – and the very requirements cities impose based on those predictions will themselves influence human behavior. Build unnecessary parking spaces, and eventually those spaces will find a use – inducing the very demand they were built to satisfy. There, the impacts of ignoring uncertainty can be long-lasting.

Here’s to embracing the cone of uncertainty!

Precisely.

Minneapolis LRV, a project built with New Starts before the total focus on the CEI. CC image from joelplutchak on flickr.

Minneapolis LRV, a project built with New Starts before the total focus on the CEI. CC image from joelplutchak on flickr.

Following up from previous discussions of precision and accuracy, Elana Schor at Streetsblog delves deeper into the subject.

While addressing the U.S. Conference of Mayors, assistant transport secretary for policy Polly Trottenberg was asked by the mayor of Clearwater, Florida, to outline how the agency might “quantify livability” in its upcoming rulemaking.

“Not everything can be measured,” Trottenberg said, adding that her colleagues wanted to avoid making the “mistake of false precision.”

She also addressed the pitfalls of relying on in-house economic predictions to assess transit projects. Several local rail lines have quickly exceeded initial federal ridership projections, casting doubt on the models used for the so-called New Starts program.

“Sometimes we’ve gotten so tangled up in the perfect mathematical science — we did it in New Starts,” Trottenberg said.

Data is good.  But we cannot limit our information inputs to just quantitative measures.  And when we do use them, we need to understand their limits.  I’m eager to see how the FTA decides to evaluate the livability criteria, but the acknowledgment that the numbers have limits is a big step forward.

Cost-effectiveness

Streetcar tracks, H St NE - CC image from flickr

Streetcar tracks, H St NE - CC image from flickr

Over the past couple of days, there have been lots of reactions to the DOT’s decision to lessen the importance of their cost-effectiveness measures in decisions on new transit starts funding (TTP, Yglesias, TNR, TOW, Streetsblog), almost all of them positive. There are, however, some key points to consider.  With the emphasis on livability as opposed to cost-effectiveness, the question will now be about measuring that livability.  Jarrett Walker notes:

Great news, perhaps, but I look forward to seeing how FTA is going to turn something as subjective as livability into a quantifiable measure that can be used to score projects, particularly since the payoffs lie in development that a proposed transit line might be expected to trigger, but that usually isn’t a sure thing at the point when you’re deciding to fund the line.  And of course, travel time does still matter.

Measurement is indeed the key.  Part of the problem of the Bush Administration’s emphasis on the CEI was an expansive definition of costs and a rather narrow definition of ‘effectiveness.’

The other problem is one that Donald Shoup talks about extensively in his book, The High Cost of Free Parking.  Namely, often imprecise data points are given undue precision in a bias towards quantifiable results and numbers – precision and accuracy are two different things, and it is important not to conflate them:

HOW FAR IS IT from San Diego to San Francisco? An estimate of 632.125 miles is precise—but not accurate. An estimate of somewhere between 400 and 500 miles is less precise but more accurate because the correct answer is 460 miles. Nevertheless, if you had no idea how far it is from San Diego to San Francisco, whom would you believe: someone who confidently says 632.125 miles, or someone who tentatively says somewhere between 400 and 500 miles? Probably the first, because precision implies certainty.

This doesn’t disprove Jarrett’s point – there are still metrics that can be used for more qualitative factors – but the larger issue here is a move away from false precision and towards outcomes that are more accurate – outcomes that better reflect the true (qualitative and quantitative) nature of cities.

With that in mind, it’s interesting to read some reactions published in the National Journal (h/t Planetizen).

Anthony Shorris: The new approach laid out by Secretary Lahood should force a re-thinking of all of our evaluative tools — cost-benefit analysis, alternatives analysis, environmental impact statements — with an eye toward re-balancing them away from an excessive reliance on only those measures that can be readily quantified.  This re-thinking should be inter-departmental (including other agencies and OMB) and inter-disciplinary (including the perspectives of urban planners and designers as well as economists).  One thing the financial crash should have taught us is that there are limitations to even the most seemingly sophisticated financial models, and that apparently crisp spreadsheets are no substitute for the prudent exercise of judgment that the American people have a right to expect of their leaders.

William Millar, APTA: With the action taken by DOT to consider all the factors required by law, transit projects can now be looked at from a holistic perspective. By judging a project on the multiple benefits it offers (i.e. mobility, economic development, environmental impact, land use improvements etc.), a well-rounded and more informed decision can be made. By removing the barrier that the Bush Administration implemented, the process is now in alignment with how it was originally intended to be.

Projects must still be cost effective and meet at least an overall medium rating in project justification and local financing. However, now, instead of a narrow prism through which to judge a project, a wider lens will offer a larger perspective. It should encourage innovative projects to be proposed and funded.