Peer Review Evolves Without Intelligent Design

Humans have brains, but they can do things that
look downright Darwinian, as in “Darwin Awards”

We hate to upset the warm fuzzy feelings many have about peer review as a guarantee of scientific integrity, but it just isn’t. This becomes clear in a study from the University of Kansas on the “evolution of peer review” that we will look at shortly. First, to set the stage, look at another article about how good intentions can sometimes end up doing the exact opposite of whatever worthy goal was initially planned.

Burned trees and billions in cash: How a California climate program lets companies keep polluting (Evan Halper for the Los Angeles Times, reposted by Phys.org Sept 17). This sad tale will leave readers’ heads shaking in frustration. You’ve heard of “carbon credits”? Those were intended to help reduce global warming. The idea was that polluting companies unable to make draconian changes to their operations to achieve climate goals in time could “offset” their carbon footprint by doing good elsewhere. They could pay other companies or NGOs that were working to mitigate climate change. It sounds noble in theory (and many states beside California, and even other countries like China) have bought into the plan, Halper shows. Unfortunately, it ends up giving polluters license to keep polluting, while making the “good guys” dependent on the carbon credit money whether or not any climate mitigation happens.

“One of the insidious aspects of the program is it hooks conservation and Indigenous groups on a source of funds,” said Neil Tangri, who in February resigned in protest from his post as the environmental justice representative on the state’s offset task force. “So you have a constituency that will fight for this program whether or not it is doing anything meaningful in a larger sense.

“They may be using the money for good things,” Tangri said. “But in the end, it is a Ponzi scheme.“

It’s enough to make one scream, but such mismatches of goals and effects seem all too common in government programs. What about peer review? That’s not a government program. Can this traditional practice overcome the perverse incentives that legislators often create, and actually achieve its intended purpose to purify scientific research? Is peer review a guardian against fraud, ineptness and ignorance? Not if it “evolves” without intelligent design.

The Evolution of Peer Review

Study points out evolution of peer review in academic publishing, problems and alternatives (University of Kansas). This article offers a needed tonic for flower-and-bunny hallucinations about the very practice that most people think makes science self-correcting. Reporter Mike Krings discusses a study about peer review done by Ben Merriman, a man well-placed to peer-review “peer review” itself, since he “has experienced all sides of academic publishing as an author, journal editor and reviewer.”

Few laymen who have grown to trust in peer review realize just how recent the tradition is. Didn’t it come into practice at the dawn of the scientific revolution in the 17th century? No; it “evolved” over time, Merriman says:

In his study, Merriman analyzed written annual reports of the editors of American Sociological Association journals, which started in 1952. Those reports spell out efforts taken to manage the publication process and how they led to much of today’s peer-review model being implemented by the early ’80s. The process of blind review, external review, exclusive submission to one journal, revise and resubmit all were adopted over a period of 30 years of gradual, mostly unplanned change documented in the annual editorial reports.

Although he is speaking of peer review in the social sciences, its evolution mirrors the development of peer review in other fields. Scientists have always criticized one another’s work, but did not have a formal review process until relatively recently. Many scientists, including Darwin, published their ideas in books. Peer review’s modern evolution began when science became recognized as a matter of national security after World War II (see 4 Mar 2017 footnote).

Merriman, University of Kansas assistant professor public affairs and administration, is not opposed to peer review. He thinks it does some good:

“There is strong evidence that blind review protects, at least partly, against some of the biggest problems, like gender bias, for example,” Merriman said. “But you can have blind review without the extensive, lengthy process we have now. This basic model does many things quite well. It has definitely improved my work. There is something to it morally that I agree with as well. But it would be healthy if there were more of a variety of review methods.”

His halfway merry assessment at the end of the article, however, is not easy to square with Merriman’s list of problems with peer review he delineates in the article. Like carbon offsets, he shows, peer review evolved, and ended up exacerbating the very problems it was supposed to correct:

• It changed the role of journal editors from overseeing content to managing reviewers.
• It created an incentive to restrict the amount of content needing to be reviewed, which disadvantages some otherwise worthy research.
• It increased the time between authorship and publication, disadvantaging grad students trying to gain recognition for their ideas.
• The time delay also disadvantaged young researchers needing a publication record to advance their careers.
• Attempts to speed up the process, though, have increased “desk rejections” in which a submitted paper is not given adequate attention because peer review is too tedious.
• Editors can reject papers for any reason without submitting them to peer review, which they know is time consuming.
• Desk rejections tend to stifle dissent, because editors often are reluctant to publish novel ideas, or “ideas that don’t align with prevailing thoughts.”
• Permitted papers, therefore, may fossilize “prevailing thoughts” even though they go through peer review.
• “I think disagreement can be very healthy for a field,” Merriman says, but peer review works against debate.
• “Peer” review is a misconception. The reviewers are usually not peers. They are often seasoned researchers who have been in the field much longer than the authors.
• Seasoned researchers are often the very ones unlikely to give maverick ideas a hearing, because they are steeped in consensus paradigms.

For all these flaws—and others—Merriman thinks there are some gems in the junk:

While Merriman said the process has several other problems, including labor inefficiency, long delays in getting findings published, difficulty detecting fraud and predicting impact of work, Merriman points out that it does have several merits as well, such as ensuring the basic soundness of the research. And unlike other models, modern peer review allows for nearly anyone to have their work considered without regard to status or renown.

Does that square with what he said before? And in what specific way does scientific peer review differ from quality control in other fields of academic inquiry? Peer review seems synonymous with scientific publishing, but actually, all scholarship needs review and criticism. It doesn’t always have to occur prior to publication. There are other ways for Big Science to ensure the “basic soundness of the research” by applying intelligent design instead of by letting traditional peer review evolve. He invites Pollyana on stage for his final comments:

“I think it’s easy to become accustomed to the notion of being frequently evaluated without reflection on if it’s truly the best method, or why it came to be,” Merriman said. “Knowing about the historical accidents could make it clear that this method is not inevitable. There are parts worth defending, and I think at least having more editorial approaches could help. Academia is full of smart people who I’m sure have excellent ideas on how to share knowledge with the world.”

Positive thinkers can wait for this futureware to evolve. If by mutation and natural selection, it could take millions of years.

The “Darwin Award” is iconic folklore for what is karmic for nincompoops who, without adequate intelligent design expected of Homo sapiens, end up eliminating themselves from the gene pool. The classic example is sawing off the limb one is sitting on. As such, it is an award for “negative selection” – it contains no mechanism for increasing the intelligence of the remaining population.

If peer review and carbon offsets are ever to make progress toward their intended goals, success will have to be achieved by intelligent design, not by evolution.