Early data from preprint studies at the start of the COVID-19 pandemic remained unchanged through the peer review process, according to a study by University of Wisconsin Information School assistant professor Ian Hutchins.
Preprints, or research papers shared before a peer review, are a valuable tool for research. They were especially helpful in the early days of the pandemic. Professor of population health sciences Ajay Sethi did his first COVID-19 research early in the pandemic, when he helped UW Health model metrics like future hospitalizations, ventilator use and death. He would sometimes cite preprints in his own research.
Preprint servers offered researchers the opportunity to communicate their findings quickly during the pandemic, Sethi said. While researchers could communicate their findings in places like online seminars, preprint servers were the most convenient and had a larger reach.
“I do think, especially after looking at this data, [preprint servers] probably accelerated the communication of discovery,” Hutchins said.
Traditional scientific publications were inundated with COVID-19 papers early in the pandemic. Sethi said these publications simply couldn’t review and publish all the papers worthy of publication. Preprint servers represented a pool of research that may have been worthy of publication but hadn’t been published yet.
While there is no peer review to verify the validity of a preprint’s findings, Sethi said he relied on his own judgment and his colleagues to decide whether to cite a preprint. The number of COVID-19 papers on preprint servers could be overwhelming, so he often relied on other researchers who promoted their own studies on Twitter or in seminars.
“There’s a lot of skepticism among the scientific community about the quality of preprints and whether the community is shooting themselves in the foot by going down this avenue because peer reviews take a long time,” Hutchins said. “It can add … up to a year to publication in this area. And nobody knows what happens when you get rid of that quality control.”
In order to measure how much the data in COVID-19 papers changed, Hutchins and his team looked at 100 articles in the NIH COVID-19 portfolio that were initially published as a preprint, but later went through peer review, Hutchins said. Then, undergraduate researchers went through both versions of the paper and looked for common epidemiological measurements such as viral reproduction and infection mortality rate.
The study found point estimate values change 6% of the time during peer review. Confidence intervals, or the margin of error, were reduced by 7%, according to the study. Around 10% of data points disappeared during peer review as well. In short, these findings indicate that the quality of the data between COVID-19 preprints and the peer-reviewed article is not remarkably different.
The data in COVID-19 papers can change for a variety of reasons, Hutchins said. The scientist reviewing the paper could suggest authors take out data points they don’t think are significant enough. Additionally, the change in 6% of data points could have come from authors adding or removing data.
Since the pandemic was occurring when researchers were performing these studies, they also could have added data that was unavailable while they were writing the preprint. Reviewers could have also suggested changes to how authors performed calculations on their data, and these calculation changes could affect the final result.
“And, in a way, isn’t it getting peer-reviewed as we speak, because it’s publicly available, and there’s an opportunity to engage with authors even to provide comments at the bottom of those pages that post the papers?” Sethi said.
DNR, DATCP kick off program to collect, dispose of PFAS-containing firefighting foam
The main preprint server used in biology is Bio Archive. While it started small, the server has grown exponentially since its inception, especially during COVID-19, Hutchins said.
Peer review is the process where experts in a given field provide recommendations for papers submitted to scientific journals, according to scientific publisher Biomed Central. The peer review process is an important part of scientific publishing since it aims to make papers stronger and easier to interpret.
Hutchins said it’s important for policymakers and reporters to understand how data may change in peer review so they can decide how comfortable they would be using preprint data to make decisions.
Still, while the data doesn’t change that much during peer review, it remains an important part of the scientific process. Hutchins said that peer review could increase the accuracy of the data scientists report as well as ensure research meets a proper ethical standard.
Over the years, the government has implemented different policies that encourage the use of preprints, Hutchins said. Should those policies continue, Hutchins expects to see the continued growth of preprint servers.
“The genie’s out of the bottle, so to speak,” Sethi said. “I mean, the reason I say there’s no going back is because I think everybody has seen the value of putting papers on preprint servers.”
