Scholars Criticize Scientific Models
Modeling is common in the sciences, but models and reality
are different things. Two recent articles warn that
models can generate misleading conclusions.
For your weekend reading, here are two recent articles that warn about trusting too heavily in scientific models.
The model, the modeler, and the truth (Science, 15 Dec 2022). James Nguyen reviews the new book Escape from Model Land by Erica Thompson (Basic Books, 2022). He is impressed by this “tremendous book” and has little to criticize. Here’s the opening of his book review:
Scientific models are ubiquitous. They structure much of our thought, and indeed our lives. They have acted, and continue to act, as protagonists in three of the most globally important events of my lifetime: the 2008 financial crisis, the COVID-19 pandemic, and humanity’s ongoing attempts to grapple with anthropogenic climate change. It is not an overexaggeration to say that they have underpinned decisions that have affected billions of lives (not to mention the future generations whose existence may depend on these decisions). The consequences of reasoning inappropriately with models are anything but trivial.
In her tremendous book, Escape from Model Land, Erica Thompson mounts a broadside against the idea that models can be naïvely interpreted as literally true descriptions of reality. … If we want to use them as guides to action, Thompson argues, we must consciously reflect on how to escape from this domain into the messy world in which we actually live. Given the limitations of models, particularly with respect to their failure to represent unquantifiable unknowns, this exit is never straightforward.
As an example, our many reports on climate change point out serious contradictions of observable evidence with climate models that are affecting “billions of lives” around the world. We quote many papers that point out “unquantifiable unknowns” that have failed to be represented in the models. (Search on “climate change” at this site, e.g. 30 July 2022.)
How a quest for mathematical truth and complex models can lead to useless scientific predictions – new research (The Conversation, 3 November 2022). Would you waste your time on statements that might be true but useless for making decisions? Arnald Puy, an associate Professor in Social and Environmental Uncertainties at the University of Birmingham, claims that complex mathematical models common in scientific work crank out uncertain results that nobody cares about. Scientists’ obsession with more and bigger data in their models can have a downside: the amount of uncertainty grows with the inputs, giving “fuzzy results” that are useless.
Once these new additions and their associated uncertainties are integrated into the model, they pile on top of the uncertainties already there. And uncertainties keep on expanding with every model upgrade, making the model output fuzzier at every step of the way – even if the model itself becomes more faithful to reality.
Sometimes simpler is better, Puy explains. If one factor can make accurate predictions, why crunch more numbers that do nothing to the model but increase the error bars? He gives examples, then explains how the bad habit actually reduces credibility.
Why has the fact that more detail can make a model worse been overlooked until now? Many modellers do not submit their models to uncertainty and sensitivity analysis, methods that tell researchers how uncertainties in the model affect the final estimation. Many keep on adding detail without working out which elements in their model are most responsible for the uncertainty in the output.
It is concerning as modellers are interested in developing ever larger models – in fact, entire careers are built on complex models. That’s because they are harder to falsify: their complexity intimidates outsiders and complicates understanding what is going on inside the model.
These problems call to mind the human element in science. It’s more work to do sensitivity analysis and keep track of accumulating error, but more attractive to make the model bigger as a public-relations ploy. A big, complex model intimidates outsiders, who may include government officials setting policy. Puy and colleagues explained their concept of uselessness and error as functions of data in Science Advances on 19 Oct 2022.
Modeling has a long history in science. One of our scientific champions, Lord Kelvin, advocated creating mechanical models as a way to understand a phenomenon. James Clerk Maxwell also made mechanical models; once he built a contraption to understand Saturn’s rings. These, however, were mechanical models, not mathematical models; you could watch them as heuristic devices that could lead to testable hypotheses. Watch Illustra Media’s animated model of an orrery (a solar system model) that Isaac Newton used to good effect to convince an unbelieving colleague about intelligent design.
One should never confuse a model with reality. Reality is too “messy” to be reduced to human devices. On the flip side, models can oversimplify reality, too. Climate models share both extremes. They are highly convoluted and immense, taking long times for computers to crunch the numbers. But they can leave out simple factors like the effect of clouds.
Models are only as good as their inputs. Remember GIGO? (garbage in, garbage out). If you only input Darwinism into a model, you might get three kinds of error: DIDO, DIGO, or GIDO.