• Written by Eva Vivalt, Research Fellow and Lecturer, Australian National University
Putting scientific results under the microscope before they are even collected could help improve science as a whole. Konstantin Kolosov/Shutterstock

We ask experts for advice all the time. A company might ask an economist for advice on how to motivate its employees. A government might ask what the effect of a policy reform will be.

To give the advice, experts often would like to draw on the results of an experiment. But they don’t always have relevant experimental evidence.

Collecting expert predictions about research results could be a powerful new tool to help improve science - and the advice scientists give.

Better science

In the past few decades, academic rigour and transparency, particularly in the social sciences, have greatly improved.

Yet, as Australia’s Chief Scientist Alan Finkel recently argued, there is still much to be done to minimise “bad science”.


Read more: 'There is a problem': Australia's top scientist Alan Finkel pushes to eradicate bad science


He recommends changes to the way research is measured and funded. Another increasingly common approach is to conduct randomised controlled trials and pre-register studies to avoid bias in which results are reported.

Expert predictions can be yet another tool for making research stronger, as my co-authors Stefano DellaVigna, Devin Pope and I argue in a new article published in Science.

Why predictions?

The way we interpret research results depends on what we already believe. For example, if we saw a study claiming to show that smoking was healthy, we would probably be pretty sceptical.

If a result surprises experts, that fact itself is informative. It could suggest that something may have been wrong with the study design.

Or, if the study was well-designed and the finding replicated, we might think that result fundamentally changed our understanding of how the world works.

Yet currently researchers rarely collect information that would allow them to compare their results with what the research community believed beforehand. This makes it hard to interpret the novelty and importance of a result.

The academic publication process is also plagued by bias against publishing insignificant, or “null”, results.

The collection of advance forecasts of research results could combat this bias by making null results more interesting, as they may indicate a departure from accepted wisdom.

Changing minds

As well as directly improving the interpretation of research results, collecting advance forecasts can help us understand how people change their minds.

For example, my colleague Aidan Coville and I collected advance forecasts from policymakers to study what effect academic research results had on their beliefs. We found in general they were more receptive to “good news” than “bad news” and ignored uncertainties in results.

Forecasts can also inform us as to which potential studies could most improve policy decisions.

For example, suppose a research team has to pick one of ten interventions to study. For some of the interventions, we are pretty sure what a study would find, and a new study would be unlikely to change our minds. For others, we are less sure, but they are unlikely to be the best intervention.

If predictions were collected in advance, they could tell us which intervention to study to have the biggest policy impact.

Testing forecasts

In the long run, if expert forecasts can be shown to be fairly accurate, they could provide some support for policy decisions where rigorous studies can’t be conducted.

For example, Stefano DellaVigna and Devin Pope collected forecasts about how different incentives change the amount of effort people put into completing a task.

As you can see in the graph below, the forecasts were not perfect (a dot on the dashed diagonal line would represent a perfect match of forecast and result). But there does appear to be some correlation between the aggregated forecasts and the results.

Reproduced with permission from DellaVigna and Pope.

A central place for forecasts

To make the most of forecasts of research results, they should be collected systematically.

Over time, this would help us assess how accurate individual forecasters are, teach us how best to aggregate forecasts, and tell us which types of results tend to be well predicted.


Read more: Science is best when the data is an open book


We built a platform that researchers can use to collect forecasts about their experiments from researchers, policymakers, practitioners, and other important audiences. The beta website can be viewed here.

While we are focusing first on our own discipline – economics – we think such a tool should be broadly useful. We would encourage researchers in any academic field to consider collecting predictions of research results.

The Social Science Prediction Platform, https://socialscienceprediction.org/.

There are many potential uses for predictions of research results beyond those described here. Many other academics are also exploring this area, such as the Replication Markets and repliCATS projects that are part of a large research initiative on replication.

The multiple possible uses of research forecasts gives us confidence that a more rigorous and systematic treatment of prior beliefs can greatly improve the interpretation of research results and ultimately improve the way we do science.

Eva Vivalt receives funding from the Alfred P. Sloan Foundation and the John Mitchell Economics of Poverty Lab.

Authors: Eva Vivalt, Research Fellow and Lecturer, Australian National University

Read more http://theconversation.com/predicting-research-results-can-mean-better-science-and-better-advice-125568

How Much Does Epoxy Flooring Cost?

Epoxy can be used on wood, cement, concrete, or metal surfaces or as an undercoat beneath other floorings. Discover how much does epoxy flooring cost here. Are you ready to apply epoxy flooring t...

News Company - avatar News Company

An Android Treasure Trove: 10 Youtube Tips and Tricks For Watching Videos On Android Tablets

Your app is the ideal device for you to watch videos on. But did you know about these hidden Youtube tips and tricks to make your Youtube experience better? YouTube remains the king of the hill s...

News Company - avatar News Company

What Are the Highest-Paying Online Casino Games?

If you try to picture instances of big-money wins at a casino, you will likely conjure up images of some James Bond-like figure playing high-stakes poker or roulette in a swanky setting. That’s un...

News Company - avatar News Company

The burn legacy: why the science on hazard reduction is contested

When it comes to reducing the extent of bushfires, scientists disagree on the best way to do it. Hazard-reduction burning (also known as “prescribed burning” or “controlled burning&r...

Kevin Tolhurst, Hon. Assoc. Prof., Fire Ecology and Management, University of Melbourne - avatar Kevin Tolhurst, Hon. Assoc. Prof., Fire Ecology and Management, University of Melbourne

'I will euthanise myself before I go into aged care': how aged care is failing LGBTI+ people

ShutterstockOlder lesbian, gay, bisexual, transgender and intersex (LGBTI+) people fear discrimination, exclusion and isolation in Australia’s aged care services, we found in our research. Wi...

Andrea Waling, Research fellow, La Trobe University - avatar Andrea Waling, Research fellow, La Trobe University

How vulnerable is Xi Jinping over coronavirus? In today's China, there are few to hold him to account

NOEL CELIS / POOL/ EPABrand “People’s Republic of China” is wobbling, as if the massive picture of Mao Zedong in Tiananmen Square was swaying with an earthquake tremor. But it can o...

Rowan Callick, Industry Fellow, Griffith University - avatar Rowan Callick, Industry Fellow, Griffith University

People love the idea of 20-minute neighbourhoods. So why isn't it top of the agenda?

Nils Versemann/ShutterstockWe were heavily involved in the consultation program for Melbourne’s long-term land-use plan, Plan Melbourne. The idea that resonated most with many participants was ...

John Stanley, Adjunct Professor, Institute of Transport and Logistics Studies, University of Sydney Business School, University of Sydney - avatar John Stanley, Adjunct Professor, Institute of Transport and Logistics Studies, University of Sydney Business School, University of Sydney

Life sentences – what creative writing by prisoners tells us about the inside

from www.shutterstock.comA recent project to encourage South Australian prisoners to write provides insights into how prisoners may benefit from written expression. The project, Life Sentences, gav...

Dr Michael X. Savvas, Senior Lecturer in the Transition Office (PhD in Creative Writing), Flinders University - avatar Dr Michael X. Savvas, Senior Lecturer in the Transition Office (PhD in Creative Writing), Flinders University

I made bushfire maps from satellite data, and found a glaring gap in Australia's preparedness

Image courtesy of Greg Harvie, Author providedOn the night of January 9 2020, my wife and I secured our Kangaroo Island home and anxiously monitored the South Australian Country Fire Service (CFS) we...

Wallace Boone Law, PhD Candidate, University of Adelaide - avatar Wallace Boone Law, PhD Candidate, University of Adelaide

Sick and Tired of Your Dead End Job? Try Teaching!

Tired of the same old grind at the office? Want an opportunity to impact lives both in your community and around the world? Do you love to travel and have new experiences? Teaching English is the perfect job for you! All you need is a willingness to ...

News Company - avatar News Company

The Impact of an Aging Population in Australia

There’s an issue on the horizon that Australia needs to prepare for. The portion of elderly citizens that make up the country’s overall population is increasing, and we might not have the infrastructure in place to support this. Australians h...

News Company - avatar News Company