Researchers delve into one of biology's scientific frontiers—by studying studies

Turning the pandemic into potential, hundreds of biologists worldwide—including two researchers from Virginia Commonwealth University—joined forces to answer a troubling question: If two biologists analyze the same data, will they get the same results? (Turns out, they won't.)

In April 2020, Brian Verrelli, Ph.D., was setting up a home office in his dining room. Like many people, he was unexpectedly working away from the office for the first time during the COVID-19 pandemic.

That's when Verrelli, an evolutionary geneticist and professor in VCU Life Sciences, saw the call go out through EvolDir, an international listserv for evolutionary biologists: Would he like to join a study investigating replication issues in ecology and evolutionary biology?

In recent years, scientists across many fields have grown concerned about the reproducibility of their studies—basically, the ability to rerun their own or another scientist's study and generate the original results. And while replication issues are well-known in other fields, like psychology, they are largely unstudied in the biological sciences.

Verrelli viewed the global project as a way to stay involved in his field without leaving the house. He reached out to Michael Rosenberg, Ph.D., a professor and director of the Center for Biological Data Science, to see if he'd like to join the fun—at least, as much fun as they could have over Zoom.

"We likely would have participated regardless," Rosenberg said. "But this concept was especially appealing in the spring of 2020, during the early stages of the pandemic."

The research, recently published in BMC Biology, is the first large-scale study to test how the "replication crisis" affects ecology and evolutionary biology. Headed by researchers from Whitman College, the University of Melbourne and the University of Alberta, the study tested how small teams of scientists might analyze the same data in different ways. It quickly ballooned to include over 170 analyst teams, possibly because its timing coincided with the beginning of the pandemic.

"The main authors were blown away by having hundreds of people seeking to be involved," Verrelli said. "Maybe many of us were in the same boat, needing a different outlet or distraction to work on something completely different during COVID-19."

Verrelli and Rosenberg make up one of those 174 teams. Together, they and around half of the teams pored through data on the effect of grass cover on the spread of eucalyptus trees in Australia, while other teams examined data on juvenile blue tits, a kind of bird. Verrelli even learned an entirely new programming language to help analyze the data.

After the teams submitted their analyses, which amounted to almost an entire research paper per team, the lead researchers asked independent reviewers to look over and rate the results. Although most researchers' results clustered around a central conclusion, some found wildly different results through statistically correct analyses.

"It's as if I asked 100 people what 2 plus 2 equals," Verrelli said. "And in this study, you got some people back that said it was 3, you got some people back that said it was 5, and you got some weird outliers that said it was 10."

And while 2 plus 2 is always 4, biological data isn't that simple. Every analytical team included in the study used statistically sound methods to reach their conclusions, but the results varied widely. That could be due to tiny decisions made by the researchers that built up over the course of the analysis and changed its trajectory.

The results could have startling implications for science, as researchers grapple with the idea that they can analyze the same data in a reasonable way but reach conflicting conclusions.

"On the surface, this was a simple question, so you'd think the statistics would be more straightforward," Rosenberg said. "The fact that they're not—and that you can get this wide array of very reasonable, yet very varying results—should be a wake-up call."

That doesn't mean the scientific method doesn't work—just that scientists should be aware that different analyses could yield different results. To solve the problem, some have advocated for more "openness" in science, calling for researchers to publish their hypotheses, methods and statistical plans before conducting a study, which could possibly reduce researcher bias.

For their part, Verrelli and Rosenberg are happy to have participated in such a groundbreaking study.

"This study, to me, is sort of a one-of-a-kind thing," Verrelli said. "If someone put out a call this summer, I'd probably do it again."