Why brain stimulation isn't what it's cracked up to be

Interest in electrical brain stimulation has skyrocketed in recent years, both in the popularmedia and scientific literature.

Scientists and clinicians are using the non-invasive and cheap technique to treat various neurological and psychiatric disorders, including depression, epilepsy and addiction. The US military is researching whether it improves learning and attention. And those who train elite athletes can see its potential to enhance performance.

But our research shows the evidence to back electrical brain stimulation varies in quality, and the results are commonly not reproduced in other studies. Our survey also unearthed the lengths to which some researchers go to to present their findings in the best light.

What is electrical brain stimulation?

The type of electrical brain stimulation we studied is transcranial direct-current stimulation. This is when a small electric current is applied to the brain for 20 to 30 minutes. Electrodes are placed on the patient’s head, and some of the current passes through the skull to the brain.

It is thought this alters brain function mainly by inducing persistent changes in the excitability of neurones.

This isn’t to be confused with electroconvulsive therapy, which uses currents hundreds of times larger. This induces a seizure.

What we did

We used an online survey to ask researchers if they could reproduce published findings related to electrical brain stimulation. We invited all researchers who served as corresponding authors on a published scientific paper on electrical brain stimulation in humans to do so.

In all, 976 researchers from all over the world were invited to answer the question of whether they could reproduce published electrical brain stimulation effects.

We also asked whether researchers used, but didn’t report, questionable research practices in their own research – such as fiddling with statistics to make them look more favourable and selectively reporting results. And we asked if they thought other researchers used these questionable techniques, and whether they should be reported in publications.

To check what researchers actually do, we audited a random selection of 100 publications featuring research on electrical brain stimulation. We looked to see if they admitted to the dodgy practices in their publications.

What we found

For the two most popular types of electrical brain stimulation (anodal and cathodal stimulation), only 45 to 50% of researchers routinely reproduced published findings.

Some researchers were aware of others who handpicked which experimental conditions (36%) and which results (41%) to publish. They also knew researchers who manipulated results by excluding data based on a gut feeling (20%) and fiddling with the statistics (43%).

As expected, fewer researchers admitted to personally using these types of shady research practices. Still, 25% admitted to adjusting statistical analysis to optimise results – namely p-hacking, when researchers manipulate the statistics to make results appear more statistically significant than they might otherwise be.

Our research also revealed the difference between whether these questionable types of practices should be reported in research papers, and whether they are. Although 92% of respondents said all researchers should admit to the questionable practices in their publications, we found only two such admissions (2%) in our audit of published studies.

So, what do we make of this?

Meta-analyses, which are studies that pool results from several other studies, indicate electrical brain stimulation is effective in major depression. But it isn’t in fibromyalgia (where people experience widespread pain without a known cause), food craving and overeating, Parkinson’s disease, and speech problems after a stroke.

Unfortunately, a general finding is that electrical brain stimulation studies are often of low quality and that, when present, therapeutic effects are often small. So, before you decide to strap electrodes to your head, speak to an informed health professional.

Poor reproducibility and bad science are not unique to electrical brain stimulation research. Nor are these problems new. But public funds are being wasted on poorly conducted research that cannot be reproduced, which means the results are questionable. Such poor research is tarnishing the genuine efforts of researchers to improve human brain function.

The main reason researchers engage in questionable researcher practices is the continual pressure to publish scientific papers to gain funding or to progress scientific careers. If results are statistically significant, researchers are more likely to be published. So, researchers may consciously, or unconsciously, resort to questionable or fraudulent research practices.

What can we do about it?

Awareness of bad science is on the rise – and recommendations and guidelines are emerging to deal with this. But there needs to be more education and true incentives for scientists to conduct better, reproducible science.

If not, some scientists will continue to do as they have always done. Incentives to improve the culture of research include promoting researchers who do more open science, and funding projects that adhere to open science practices as well as those that attempt to replicate studies.

The responsibility to improve the quality of our science lies with research institutions and universities, funding agencies, scientific publishers and individual researchers.

Our goal of clinically useful brain stimulation techniques is a worthy one. But our progress is limited by findings of often variable and small effects currently reported, as well as the poor quality of some of the studies that claim any effects at all.

Simon Gandevia receives funding from the National Health and Medical Research Council.

Colleen Loo and Martin Héroux do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond the academic appointment above.