Corrupt Peer Review Needs a Reformation

The ostensible gold standard of scientific reliability, peer review, looks more like fool’s gold in many cases. Reforming it will require an overhaul, not just corrections.

Evolutionists sometimes hammer creationists with peer review. They sneer, ‘Point me to some of your peer-reviewed work and I might begin to take it seriously.” This attitude overlooks a number of flawed assumptions, among them: (1) that peer review elevates a paper to a higher plane of scientific reliability, (2) that creationists do not have peer review (they do, but most often in their own journals), (3) that evolutionary journals would treat creationist submissions fairly (which they do not; they are excluded a priori), and (4) that reviewers are unbiased saints without ulterior motives or flaws. Another faulty assumption is that peer review has always been a criterion of science, when in fact, many honored works, including Newton’s Principia and (ironically) Darwin’s Origin of Species (touted by atheists as the greatest scientific work ever penned), were not peer reviewed. Peer review has had a spotty history, only in recent decades following any kind of regular protocol.

Solomon recommended seeking a multitude of counselors (Proverbs 11:14) almost two millennia ago. That’s not exclusive wisdom for scientists. Most importantly, any form of peer review or counsel clearly depends on the honesty of the reviewers.

This month marks the 500th Anniversary of the Protestant Reformation. Luther (and just about everyone else at the time) saw the corruption of a religious system that arrogated to itself the sole right of review, as it were, of what was orthodox and what was heretical. Luther’s original hope for debate and rational discussion quickly degenerated into a standoff, in which the entrenched powers of tradition ordered him to recant his views. In good conscience, he could not. The rest is history; it took a Reformation, not a discussion, to achieve lasting change.

A similar situation is happening right now with peer review. There’s a revolt going on among scientists who feel that mere revisions to tradition will probably not be able to cure the corruption. Here is some recent news on the coming reformation in peer review:

Can editors save peer review from peer reviewers? (PLoS One). The authors find that reform is unlikely to help, and seem to agree that a reformation is necessary. Luther could probably relate to the human foibles corrupting today’s scientific peer review:

Peer review is the gold standard for scientific communication, but its ability to guarantee the quality of published research remains difficult to verify. Recent modeling studies suggest that peer review is sensitive to reviewer misbehavior, and it has been claimed that referees who sabotage work they perceive as competition may severely undermine the quality of publications. Here we examine which aspects of suboptimal reviewing practices most strongly impact quality, and test different mitigating strategies that editors may employ to counter them. We find that the biggest hazard to the quality of published literature is not selfish rejection of high-quality manuscripts but indifferent acceptance of low-quality ones. Bypassing or blacklisting bad reviewers and consulting additional reviewers to settle disagreements can reduce but not eliminate the impact. The other editorial strategies we tested do not significantly improve quality, but pairing manuscripts to reviewers unlikely to selfishly reject them and allowing revision of rejected manuscripts minimize rejection of above-average manuscripts. In its current form, peer review offers few incentives for impartial reviewing efforts. Editors can help, but structural changes are more likely to have a stronger impact.

Preprint ecosystems (Jeremy Berg in Science Magazine). Preprint servers are exploding on the web as alternatives to peer review. Cornell’s arXiv service for physicists has had legs for a number of years; now biologists are embracing the trend as a bioRxiv service spreads its wings. Although preprint authors may subject their work to peer review before final publication, the internet has enabled scientists fed up with tradition to do an end run around peer review and get feedback on their ideas instantly. This leaves traditional journals like Science in a quandary. They don’t want to stop the reformation, but they still want to maintain some of their power to keep ‘official’ science under their wing:

The Science family of journals accepts the submission of a research paper for which a preprint of the submitted version is posted on not-for-profit servers such as arXiv and bioRxiv, as we support mechanisms that relate to the rapid communication of findings within the scientific community. We encourage authors to discuss with our editors any postings to other servers, and we encourage our editors to become involved in discussions about preprint-related issues as they evolve. Interactions with the press related to preprints has become one challenging area. When preprints of papers that are of potential interest to the press and the public are available, reporters on occasion approach authors prior to peer-reviewed publication. If a paper is under consideration in a Science-family journal, we leave it up to authors as to how they respond to media inquiries, but do note that media coverage could be taken into account by editors when considering novelty and make it difficult to embargo the paper if accepted for publication. We believe that giving reporters access to papers that will soon appear in our journals, on an embargoed basis, leads to more complete and accurate reporting of important science stories.

The preprint dilemma (Jocelyn Kaiser in Science). Speaking for the establishment, Kaiser worries about new problems that the rush toward preprints might cause. And yet it’s a dilemma for the journals because they don’t want to position themselves as antagonists toward a popular reformation. She sounds like a medieval apologist for the church trying to give calm, rational objections to what Luther’s actions might lead to, while keeping some semblance of order for traditional power structures. In the subheading, “Will preprints replace journals?” comes this statement: “Some proponents predict that preprint servers will become the favored venue for publishing and critiquing findings, and will eventually replace peer-reviewed journals altogether,” she says, giving this diplomatically-worded talking point in favor of Holy Mother Science: “For the moment, that appears to be a minority view.”

Publishers threaten to remove millions of papers from ResearchGate (Nature). Integrity applies to both authors and reviewers. A number of scientists, fed up with the paywalls of the standard journals, have taken advantage of the internet’s easy flow of information to post copyrighted papers on “the world’s largest scholarly social network.” It’s not quite as evil as BitTorrent has been for musicians denied their fair compensation, because participants in ResearchGate postings can rationalize that publicly funded research should be publicly available. Who owns the rights to this ‘intellectual property’? The current controversy sounds historically familiar. In Luther’s day, outraged peasants felt cheated out of the Holy Scriptures intended for all mankind that a powerful religious order was keeping from them. In a kind of counter-reformation, the journals are fighting back against the undercutting of their bread and butter (i.e., journal sales) by suing and ordering take-down of illegal postings. Much as they want to stop the leaks, their suggested judicial fingers in the internet dike are unlikely to prevent a flood, since there are many ways around the fingers. Many scientists, for instance, commonly share papers via email, using their institution’s group subscriptions, or post their work on their personal web pages.

Influence, integrity, and the FDA: An ethical framework (Science Magazine). While not about peer review per se, this Policy Forum statement from the AAAS underscores the principle that review or regulation are only as good as the integrity of the reviewers or regulators.

Among the core missions of the U.S. Food and Drug Administration (FDA) are protecting public health by assuring the safety and efficacy of drugs, biologics, and medical devices and advancing public health by promoting scientific research and medical innovation. According to its mandate, the decisions made by the FDA in fulfilling these missions should be guided by scientific considerations, not economic or political ones. However, several recent, high-profile episodes have highlighted the fact that the FDA is buffeted by many external influences. Such controversies require us to distinguish between legitimate influences that would improve the FDA or enhance its regulatory mission, illegitimate influences that seek to corrupt or undermine the agency, and influences that may be legitimate but nevertheless harm public health or patient outcomes. We present a decision framework to assist regulators, policy-makers, judges, physicians, and the public in evaluating the legitimacy and value of external influences on the FDA.

This may sound all fine and good, but in reality, one group of fallible humans seeks to arrogate to itself the right to tell other fallible humans what they should do. What influences are acting on the authors of this policy statement? On what basis do they determine legitimacy or corruption in the FDA’s dealings, if not by appealing to an external standard of morality? If the government responded by issuing a policy statement requiring analysis of the legitimacy of the AAAS, what would the AAAS say? Who’s fixing the fixers?

The Bible has an absolute standard of morality, but evolutionary theory does not. In evolution, might makes right. Like in a cage match, it’s about survival in the struggle for existence. Put the AAAS and the FDA in the iron cage and let them fight to the finish. The winner gets the belt of legitimacy. If scientists and journals find this picture deplorable, let them acknowledge the Ten Commandments, or point to some other timeless, universal canon of integrity that is not composed of matter in motion. Good luck finding one in Darwin’s views.