Please, no more projections. What we need are predictions, and they're harder

It is difficult to make predictions, especially about the future. (attributed to Danish physicist Niels Bohr)

The difficulty of making predictions has long been known, but the rise of the internet means that it is getting harder to hide the gaps between predictions and reality.

Perhaps the most significant example of repeated failed projections has been the International Energy Agency’s projections of the importance of renewable power – the proportion of electricity generated by solar and wind and other renewable sources other than hydro.

Beginning in 1994, when generation from these sources was negligible, the Agency has produced estimates of the future share of renewables every two years.

Read more: Getting projections right: predicting future climate

Every two years, with striking regularity, the previous estimates have been exceeded (hugely) and the new set has been revised upwards. But never by enough. Each time the upward revision has been inadequate.

This graph, prepared by Paul Mainwood from IEA reports, illustrates the point:

Projections versus reality

The dashed lines represent successive projections from 1994 to 2016.

The solid red line represents the actual growth of non-hydro renewables as a share of total electricity generation.

What can be seen is that eleven times in a row the projected growth in renewables was far too low and had to be upgraded. Eleven times in a row the upgrades weren’t big enough.

As a joke (indicated by the green line) Mainwood “corrected” the 2016 projection for the typical under-projection:

Projections plus “corrected” 2016 projection

Projections often understate change

Another striking example relates to religious belief. In 2015 the Pew Research Institute released a report suggesting a gradual decline in Christianity in the United States.

The headline finding for the US:

Christians will decline from more than three-quarters (75%) of the population in 2010 to two-thirds (66%) in 2050.

Yet only four years later in 2019, Pew released another report with this headline finding about the US:

In 2018 and 2019, 65% of American adults describe themselves as Christians when asked about their religion, down 12 percentage points over the past decade.

Thus, a decline expected to take forty years had taken place inside ten.

Moreover, Christian affiliation is highest in the older age cohorts, very few members of which will be around in 2050. Looking at the younger cohort born after 1981, only 49% claim a Christian affiliation, a proportion that has declined by 16 percentage points in a decade. Unless that trend is rapidly reversed, or later cohorts experience a religious revival, Christians will be a minority well before 2050.

They enable modellers to escape responsiblity

Faced with outcomes of this kind, modellers often defend themselves by drawing a distinction between “predictions” and “projections”.

A prediction (Latin for “foretelling”) is a claim about what is most likely to happen in the future. By contrast, a projection (“casting forward”) is a model output derived by extrapolating past trends, holding some parameters constant and allowing others to vary.

The International Energy Agency projections were characterised by the assumption that the costs of renewable energy would change only slowly, rather than continuing the extraordinary decline that began in the 1970s.

Read more: A radical idea to get a high-renewable electric grid: Build way more solar and wind than needed

In the case of the Pew projections, the crucial parameter is conversion rates – the frequency with which people brought up with one religion (including “no religion”) change to another. The Pew projections were based on the assumption conversion rates will remain unchanged.

It is the opposite of the widely-held but controversial “secularisation hypothesis” which states that modernisation leads to increased conversions to “no religion”.

Predictions force them to take responsiblity

Projections are useful in the development of models. All models are based on past experience and have to assume that in some respects the future will be like the past. By examining the projections generated under particular assumptions about which variables and parameters will remain constant, it is possible to understand how models works and make modelling choices.

But projections of this kind are of no use in informing people not familiar with the material. They mislead casual readers of the Australia’s budget.

The budget uses both predictions and projections.

For the two years following each budget the Commonwealth uses “forecasts” which it is prepared to stand by, then for the next two it presents mechanical “projections”:

When asked to defend (for example) its wage growth number of 3½%, the treasury points out that it is a mechanical projection rather than a forecast, a distinction lost on all but the most enthusiastic of budget devotees.

In practice it is impossible to separate the two.

Even though both the International Energy Association and Pew describe their estimates as “projections” they are almost invariably treated as predictions.

Read more: Vital Signs: dismal wages growth makes a joke of budget forecasts

Since any statement about the future will be treated as a prediction, the only serious option for modellers prepared to be truthful with the public is to bite the bullet and make predictions.

It means working hard to make the best possible estimate of the future paths of all model parameters, and using judgement to take account of everything else.

It’s what those of us looking to experts for guidance deserve.

John Quiggin does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.