Viw Magazine

  • Written by Eva Vivalt, Research Fellow and Lecturer, Australian National University
Putting scientific results under the microscope before they are even collected could help improve science as a whole. Konstantin Kolosov/Shutterstock

We ask experts for advice all the time. A company might ask an economist for advice on how to motivate its employees. A government might ask what the effect of a policy reform will be.

To give the advice, experts often would like to draw on the results of an experiment. But they don’t always have relevant experimental evidence.

Collecting expert predictions about research results could be a powerful new tool to help improve science - and the advice scientists give.

Better science

In the past few decades, academic rigour and transparency, particularly in the social sciences, have greatly improved.

Yet, as Australia’s Chief Scientist Alan Finkel recently argued, there is still much to be done to minimise “bad science”.

Read more: 'There is a problem': Australia's top scientist Alan Finkel pushes to eradicate bad science

He recommends changes to the way research is measured and funded. Another increasingly common approach is to conduct randomised controlled trials and pre-register studies to avoid bias in which results are reported.

Expert predictions can be yet another tool for making research stronger, as my co-authors Stefano DellaVigna, Devin Pope and I argue in a new article published in Science.

Why predictions?

The way we interpret research results depends on what we already believe. For example, if we saw a study claiming to show that smoking was healthy, we would probably be pretty sceptical.

If a result surprises experts, that fact itself is informative. It could suggest that something may have been wrong with the study design.

Or, if the study was well-designed and the finding replicated, we might think that result fundamentally changed our understanding of how the world works.

Yet currently researchers rarely collect information that would allow them to compare their results with what the research community believed beforehand. This makes it hard to interpret the novelty and importance of a result.

The academic publication process is also plagued by bias against publishing insignificant, or “null”, results.

The collection of advance forecasts of research results could combat this bias by making null results more interesting, as they may indicate a departure from accepted wisdom.

Changing minds

As well as directly improving the interpretation of research results, collecting advance forecasts can help us understand how people change their minds.

For example, my colleague Aidan Coville and I collected advance forecasts from policymakers to study what effect academic research results had on their beliefs. We found in general they were more receptive to “good news” than “bad news” and ignored uncertainties in results.

Forecasts can also inform us as to which potential studies could most improve policy decisions.

For example, suppose a research team has to pick one of ten interventions to study. For some of the interventions, we are pretty sure what a study would find, and a new study would be unlikely to change our minds. For others, we are less sure, but they are unlikely to be the best intervention.

If predictions were collected in advance, they could tell us which intervention to study to have the biggest policy impact.

Testing forecasts

In the long run, if expert forecasts can be shown to be fairly accurate, they could provide some support for policy decisions where rigorous studies can’t be conducted.

For example, Stefano DellaVigna and Devin Pope collected forecasts about how different incentives change the amount of effort people put into completing a task.

As you can see in the graph below, the forecasts were not perfect (a dot on the dashed diagonal line would represent a perfect match of forecast and result). But there does appear to be some correlation between the aggregated forecasts and the results.

Reproduced with permission from DellaVigna and Pope.

A central place for forecasts

To make the most of forecasts of research results, they should be collected systematically.

Over time, this would help us assess how accurate individual forecasters are, teach us how best to aggregate forecasts, and tell us which types of results tend to be well predicted.

Read more: Science is best when the data is an open book

We built a platform that researchers can use to collect forecasts about their experiments from researchers, policymakers, practitioners, and other important audiences. The beta website can be viewed here.

While we are focusing first on our own discipline – economics – we think such a tool should be broadly useful. We would encourage researchers in any academic field to consider collecting predictions of research results.

The Social Science Prediction Platform,

There are many potential uses for predictions of research results beyond those described here. Many other academics are also exploring this area, such as the Replication Markets and repliCATS projects that are part of a large research initiative on replication.

The multiple possible uses of research forecasts gives us confidence that a more rigorous and systematic treatment of prior beliefs can greatly improve the interpretation of research results and ultimately improve the way we do science.

Eva Vivalt receives funding from the Alfred P. Sloan Foundation and the John Mitchell Economics of Poverty Lab.

Authors: Eva Vivalt, Research Fellow and Lecturer, Australian National University

Read more

Why Sydney’s top-end property is pandemic proof

The pandemic has repeatedly delivered the unexpected for property markets. Sydney Buyers Agent - Nick Viner discusses why Sydney is pandemic proof. There were some early predictions during the ...

Nick Viner - avatar Nick Viner

Phytonutrients can boost your health. Here are 4 and where to find them (including in your next cup of coffee)

ShutterstockCoffee is regularly in the news for its potential health benefits and drawbacks. A review of the research found drinking a few cups of coffee a day was associated with a lower risk of dyin...

Clare Collins, Professor in Nutrition and Dietetics, University of Newcastle - avatar Clare Collins, Professor in Nutrition and Dietetics, University of Newcastle

New Zealand relaxes COVID-19 restrictions, except for Auckland. How much longer will the city have to wait?

Fiona Goodall/Getty ImagesNew Zealand’s government today announced that after seven days of no new community cases linked to the Auckland cluster, most of the country will return to almost norma...

Alex James, Associate professor, University of Canterbury - avatar Alex James, Associate professor, University of Canterbury

The Role Of A Construction Supervisor On A Construction Site

Those looking into building a career in the building and construction industry are afforded a lot of options they can pursue, with responsibilities often being as diverse as the projects themsel...

News Company - avatar News Company

4 Writing Hacks You Need to Become a Great Writer

What do you need to mount up your writing skills and become a cooler writer? This venture is highly individual and relies on each particular writer’s personal abilities and expertise. However...

News Company - avatar News Company

Factors to Consider When Looking For Kid’s Birthday Party Venues

Organising huge children's birthday parties begin with finding the right kids birthday party venues. Like every other major activity that requires a bunch of guests, the final venue will dictate...

Ester Adams - avatar Ester Adams

How a Share Registry Works

Australia, the Land Down Under, is a melting pot of cultures, races, and ethnicities. This country embraces diversity and has no tolerance for discrimination of any kind, especially if it is abou...

Ester Adams - avatar Ester Adams

This video shows just how easily COVID-19 could spread when people sing together

ShutterstockProduction of the reality TV show The Masked Singer was shut down last month after several crew members were infected with COVID-19.It’s one of several examples of COVID-19 transmiss...

C Raina MacIntyre, Professor of Global Biosecurity, NHMRC Principal Research Fellow, Head, Biosecurity Program, Kirby Institute, UNSW - avatar C Raina MacIntyre, Professor of Global Biosecurity, NHMRC Principal Research Fellow, Head, Biosecurity Program, Kirby Institute, UNSW

Has Australia really had 60,000 undiagnosed COVID-19 cases?

A preliminary study, posted online this week by researchers at the Australian National University and elsewhere, estimates 71,000 Australians had COVID-19 by mid-July — 60,000 more than official...

Andrew Hayen, Professor of Biostatistics, University of Technology Sydney - avatar Andrew Hayen, Professor of Biostatistics, University of Technology Sydney

Writers Wanted

News Company Media Core

Content & Technology Connecting Global Audiences

More Information - Less Opinion