Many scientists are struggling to reproduce previously established findings, raising questions surrounding the funding and foundations of science, according to Nicole Nelson, an associate professor in the Department of Medical History and Bioethics at the University of Wisconsin.
Nelson was first interested in the struggle to reproduce data in the 1990s when a group of researchers completed the same set of experiments in different labs. But when they compared their results, some findings were the same and others were different. More questions surrounding the phenomenon first arose in the public sphere in the early 2010s when it was deemed the reproducibility crisis, Nelson said.
This problem impacts almost every scientist because almost all scientists have experienced a failure to replicate findings, Nelson said. But the reproducibility crisis impacts everyone outside of the scientific community, too.
“Everybody is invested in the pace of scientific progress, especially when it comes to really super relevant things like drug development or new technologies,” Nelson said. “When they fail, they take a lot of money and they also take everybody’s time, you know, for people who really need and want those results.”
The Lab Report: Alternate solutions for natural products using plant tissue
Nelson’s research focuses on how scientists first became aware of the reproducibility crisis in their work. She said there are many routes to awareness, but one common route is when graduate students experience irreproducibility in their experiments.
One of Nelson’s studies interviewed 40 UW graduate students who had experience a failure to replicate previous studies. While they eventually found other reasons for the failure, many initially blamed themselves.
“This tendency for grad students to blame themselves first I think helps us understand how it is that a lot of irreproducibility can be basically invisible because grad students aren’t talking about it,” Nelson said. “They don’t even recognize it as irreproducibility, they just think it’s them messing up.”
In Nelson’s study on UW graduate students, an initial survey found that 79 out of 80 respondents had experienced failures to replicate, and the single respondent who hadn’t was a first-year graduate student.
Some graduate students, Nelson said, are greatly impacted by failures to reproduce — some students found it difficult to eat or sleep well and others sought out therapy. A large consequence of the reproducibility crisis is the burden on graduate students, Nelson said.
Third-year PhD student in the Chemical and Biological Engineering Program Jack McAlpine has, like many researchers, experienced failures to reproduce data. McAlpine works for the Gebbie Lab and his focus is understanding the electron and ion interactions behind battery electrolytes.
McAlpine worked to accurately measure the capacitance of ionic liquid salts for two years, but the lab’s data wasn’t matching up to the data from previous findings. One of his lab’s main struggles was figuring out how previous researchers completed their experiments.
“I think a big portion of it is that you don’t really share everything you do … and as the capabilities that people have and the questions that they want to explore become increasingly diverse, people are able to build their own stuff a lot more,” McAlpine said. “And if you’re not sharing really in-depth of what you’ve made and how you’re doing it, it’s really hard to keep up with that.”
One reason the lab struggled to reproduce the data may have been the quality of wire used in the experiment, McAlpine said. The quality of raw materials that go into experiments are commonly viewed as a cause of irreproducibility, Nelson said.
UW team maintains 20-year streak in international programming competition
For example, when biomedical researchers buy antibodies, they don’t test them and use them in their experiments, generating false positives, Nelson said. Some antibodies are sold commercially and are advertised as able to find specific proteins. While the antibodies do find these proteins, Nelson said they can bind to other things as well — resulting in a false positive.
Another commonly-held cause of irreproducibility is called p-hacking, Nelson said. P-hacking is when researchers try different statistical tests until one provides a significant result. Then, Nelson said, researchers publish the significant results.
“The kind of key problem is that if you keep trying over and over and over again, you will eventually cough up a result that looks significant just by chance,” Nelson said. “And if you only go and publish that result, the one that you happened to get that looks significant, then you can end up with a whole literature of stuff that looked like it was good but actually isn’t.”
While raw material quality and p-hacking are widely known as potential causes of irreproducibility, Nelson said it’s unknown whether they’re the main culprits — there are many factors that can cause a failure to replicate.
Understanding implications of augmented reality in manufacturing, privacy
There are also many potential solutions to the reproducibility crisis, Nelson said. One widely-held solution is changing the incentive structure of science. The incentive structure is the incentives that get scientists paid and promoted and the current incentive is a good publication record, Nelson said.
According to Nelson, good publication records mean publishing findings often, but there’s no way to measure the quality of scientists’ publications.
“There is no way right now of trying to discern who’s publishing a lot because they’re publishing stuff that they haven’t really verified, versus who’s publishing stuff at the same rate but is actually publishing good stuff,” Nelson said. “And if you had better measures for doing that, then you would have better measures for figuring out who you wanted to hire, who you wanted to promote, who you wanted to give money to.”
