Time to Overhaul Science
A noted advocate of science integrity is leading a team of experts on a program to improve scientific practice and clean up serious failings.
The Stanford Medicine News Service has big news: “Researchers announce master plan for better science.” The initiative will be led by John Ioannidis, MD, DSc, professor of medicine and of health research and policy at the Stanford University School of Medicine.
We’ve heard from Ioannidis before. He shocked the research establishment a dozen years ago, claiming “Most published research findings are false.” More recently, he complained that “the scientific reward system needs to change” (10/29/14). Last year, his influence led to widespread recognition that “Widespread failure to reproduce research results has triggered a crisis of confidence in research findings, eroding public trust in scientific methodology.”
Now, Ioannidis and colleagues at Stanford Medicine have posted a manifesto in the inaugural issue of a new journal, Nature Human Behavior, that addresses four areas of concern in scientific practice: methods, reporting, reproducibility, and evaluation. “There is a way to perform good, reliable, credible, reproducible, trustworthy, useful science,” he thinks, but it’s going to require some big commitments. The impact of bad practice on science’s customers is huge.
The manifesto suggests it’s not just scientists themselves who are responsible for improving the quality of science, but also other stakeholders, including research institutions, scientific journals, funders and regulatory agencies. All, said Ioannidis, have important roles to play.
Scientists need to understand, he explains, that busy work represents wasted potential. Science is not done for its own sake. Science is for something, and for somebody: the stakeholders, like taxpayers, who throw billions of dollars at it. They have a stake at seeing some motion, some benefit accruing to them, some return on investment.
Each year, the U.S. government spends nearly $70 billion on nondefense research and development, including a budget of more than $30 billion for the National Institutes of Health. Yet research on how science is conducted — so-called meta-research — has made clear that a substantial number of published scientific papers fail to move science forward. One analysis, wrote the authors, estimated that as much as 85 percent of the biomedical research effort is wasted.
The press release points to a human flaw that plagues science: the tendency to jump the gun and report exciting, serendipitous results, rather than to run a research program to completion according to pre-established practices. Stakeholders need to know about all results, even negative or inconsequential findings. Unexciting results may be even more valuable in the long run, providing a record of what doesn’t work. Another common flaw is massaging data to the point where chance results briefly pop out as statistically significant when they are not.
The press release mentions these specific recommendations:
- Methods: Design your study to minimize bias. Register your flight plan before taking off: what’s your plan? How will you measure success or failure? How will you analyze your results?
- Reporting: Establish policies to report all findings, even unexciting ones, instead of stuffing them in a file drawer.
- Reproducibility: Funding agencies and journals can set up standing committees to set standards.
- Evaluation: The committees can advise, monitor compliance, and incentivize best practices.
Given their prominence in society, scientists need to maintain focus on the goal of their work.
The ultimate goal is to get to the truth, Ioannidis said. “When we are doing science, we are trying to arrive at the truth. In many disciplines, we want that truth to translate into something that works. But if it’s not true, it’s not going to speed up computer software, it’s not going to save lives and it’s not going to improve quality of life.
In an age of post-truth and fake news, Ioannidis’s perspective is refreshing. Dr. Everett Piper, president of Oklahoma Wesleyan University, took some glee last Friday on FRC’s Washington Watch radio program at the secular media’s recent concern over “fake news.” It presupposes, he says, that truth is not relative. His opening commentary is worth hearing in that regard, because it touches on what Ioannidis also said about science’s goal: the truth, and a truth that works.
Update 1/12/17: The paper, “A manifesto for reproducible science,” appeared in Nature Human Behavior and is open-access.
Biblical truth is always action-oriented. It’s not just out there in the clouds, unrelated to the way we live our lives. Truth should change a person’s behavior. It should guide the individual in right living. We see a parallel here in John Ioannidis’s statement that “we want that truth to translate into something that works.” We don’t want our scientists just cranking out knowledge on all kinds of irrelevant details. An example we have used before is recording the vocalizations of each animal when you step on its foot. What good is that? Most biological research is done on a few model organisms: among them, E. coli, Arabidopsis, and C. elegans. Taking this to the extreme, suppose scientists ran the same research on 8 million other organisms. At some point busy work becomes ridiculous. How many genomes is enough?
Pure research has its place; one doesn’t always know the practical value of an experiment. Michael Faraday wisely responded to a government official who asked what was the use of electricity. Faraday answered, “Some day you will be able to tax it.” But no scientist approaches subject matter without some inkling about its potential usefulness. The practical outcome of science is especially important when others are footing the bill. The stakeholders expect truth, and they expect value. If an independently wealthy researcher wants to go off on his own to record the sounds made by animals when he steps on their feet, fine (just don’t tell PETA).
We see the sensationalism of science in many ways, often measurable by the perhapsimaybecouldness index (PCI). This molecule “might” lead to a cure for cancer. This fossil “might” shed light on evolution. This particle “might” help us understand the big bang. The scientist or reporter makes these statements and then runs off to something else, never delivering. I can’t tell you how many news reports I read each week about some new promising finding on cancer. Very few of them ever lead to a clinical trial or proof. Meanwhile, people are dying of cancer every day. It would be better not to make those kinds of statements at all than to tease, if there are no plans to follow through.
Here’s the take-home message from this manifesto: Big Science is currently getting a failing grade. The manifesto wouldn’t have been needed if, for the past 12 years at least, scientists had a good report card. Some people reading our pages may get a false impression we are anti-science. No, we are anti-bad science. And we say that scientists need to get off their pedestal and realize that everyone needs to be involved in finding truth that translates into good works. We tend to reify science as a subject apart from other forms of knowledge acquisition. But whether you are a historian, an auto mechanic or a pastor, you also need to pursue truth that translates into something that works for people. Scientists might do it studying petunias or Wolf-Rayet stars so that people can appreciate the beauty and variety of God’s creation or so that engineers can use what they learn for human benefit; historians may seek to understand Hitler’s motivations so that his errors are not repeated; auto mechanics by testing replacement parts so that people can drive safely; pastors by seeking to understand the principles of theology so that people will live righteously and harmoniously together under God. We’re all in this together, folks! We’re all scientists trying to make sense of our world (see Doug Axe’s book Undeniable for elaboration). If you are a “professional” scientist, it’s time to pull your share of the load. Start moving that F grade to an A, for the sake of your stakeholders. They don’t owe you a salary.