Simulations Can Misrepresent Reality

A new paper warns that commonly-used simulation methods in science can misrepresent the real world.

In the European Physical Journal Plus, Daan Frinkel warned about the “dark side” of simulations – those applied out of context or extrapolated beyond their capabilities (by “dark” Frenkel means under-exposed, not evil).  Simulations are very common in science; for instance, Monte Carlo simulations are frequently used to study everything from thermodynamic processes to population behavior.

The paper was summarized on Science Daily as “Simulations’ Achille’s [sic] Heel,” and on PhysOrg with the headline, “What can go wrong when computer simulations applied outside their original context.” The summary says,

Frenkel also focuses on methods that, at first blush, appear reasonable, but are flawed and are akin to attempting to compare apples and oranges. For example, computing a mechanical property of a system—say the potential energy—using a Monte Carlo simulation, which can be based on thermal averages, does not allow us to compute the thermal properties of such a system—such as entropy—in terms of thermal averages. Finally, the article also takes great care to debunk common myths and misconceptions pertaining to simulations, for instance, newer simulation methods are not necessarily better than older ones.

Looking inside the paper, Frenkel explained that Monte Carlo and Molecular Dynamics methods, though simple, are not risk-free.  They require great care, and sometimes are unreasonable even if they seem appropriate.  When used carelessly, they can pretend to provide answers – sometimes when the original question has been forgotten.

In Kuhn’s view of science, “guilds” of scientists amuse themselves with busy work on paradigms that may or may not reflect reality.  Guild members can seem mutually self-satisfied that progress is being made on the paradigm, when in fact, the work may not correspond to reality.  Hopefully many scientists really do care about correspondence with reality, but even though they produce papers with whiz-bang simulations that seem to work, their readers should keep a healthy caution in mind that internal self-consistency does not presuppose getting “the world” right.  This is especially true of evolutionary scenarios (see 5/08/2008 commentary, “How Not To Work a Puzzle”).  We reproduce that commentary here for convenience.

HOW NOT TO WORK A PUZZLE

1. Visualize your favorite picture.
2. Invite your friends.  Check their credentials.  If anyone from a “different picture club” tries to sneak in, expel them.
3. Make everyone feel good about the picture.  Reinforcement can be achieved with a few choice sermonettes from respected individuals.
4. Draw a big outline of the picture on the floor.  A gymnasium is good for this step.
5. Open the box and spill out the pieces.
6. Throw away the box top and the instructions.  After all, this is science.  We do it our way.
7. Form teams and pass out the puzzle pieces.  These can be distributed by color, shape, or any other useful scheme.
8. Allow each team to work on their part of the picture.  Rearrangement of small parts is permitted, but not changes to the outline.
9. The rules allow for complaints about how hard the work is.  If anyone complains about the big picture, though, they must be expelled.
10. Every two years, throw a party with booze and croissants and let each team share their experiences.  Throw in a few more choice sermonettes to keep spirits high.
11. Report to the media on the progress being made.
12. Draw up a curriculum and teach the next generation how to work the puzzle.

INTERPRETATION
Note: Mature, well-trained, experienced readers can skip this section.

1. Philosophy of discovery:  Theories do not emerge from raw data.  More often, scientists begin with a picture in mind.  Even deciding what to call “the data” requires a human choice, because not all inputs are relevant to the picture.  Like Benner said, “science is often what we choose to believe.”
2. Sociology of science:  Scientists tend to hang out with people they know and like.
3. Sociology, cont.:  Science is a human activity, not something that could be done by robots.  It is not purely rational but involves emotions, rhetoric, herd mentality and other non-rational considerations.
4. Kuhnian normal science:  The paradigm determines the research project.  The participants were not assembling to question the picture.  They were assembling to affirm it.
5. Underdetermination of theories by data:  There are inevitably many possible explanations for one set of data.  The same puzzle pieces could be fit to a different picture.
6. Naturalism:  Modern science has chosen to restrict itself to “natural causes” (whatever that means; see 05/11/2006).  Today’s scientists have been trained to deplore revelation (natural and/or special), no matter how well validated by empirical evidence, reason or history.
7. Pragmatism:  Nature does not determine the choice of classification scheme; people do.
8. Limits of science:  No one person can master the whole picture, especially one as broad as evolution.
9. Sociology/psychology of science:  Each researcher works under the assumption that his or her little piece will reinforce the paradigm.  Cooperation is ensured by the fear of being expelled as a maverick, or worse, a pseudoscientist.  Since no philosopher of science has successfully defended demarcation criteria for science vs pseudoscience, and since no universal scientific method has been defined, emotional and sociological judgments again come into play to determine who is “in” and who is “out.”
10. Sociology, cont.:  Social activities, though they have nothing to do with the validity of the proposition under study, serve to reinforce the paradigm and draw in more party members.
11. Positivism:  The party celebration attracts reporters and gives them some fun work to do at a nice hotel.  The atmosphere promotes a spirit of progress.  All this activity, all these smart people, and all the erudite PowerPoint slides must mean that productive science was being done, right?  It must be the case since the government is funding the work.
    The system feeds on itself.  Reporters get a share of the booze and croissants, paid for by their bosses, who get better advertising ratings for maintaining a lively science page.  The reporters make friends with some of the scientists and learn from the herd who is hot and who is not.  It is unlikely any reporter will go back to the office and write up a scathing rebuke of the entire philosophical premise underlying the event.  Party organizers will be sure to send the cheerful press releases to Senator Earmark.
12. Education:  The paradigm might lose popularity without new blood.  Captive students must be trained and inculcated into the craft before other paradigms capture their attention.  This can be accomplished by making all other paradigms illegal.  Skilled facilitators can create visuals and curricula from the Policies and Procedures Manual of the Paradigm, to inculcate the novitiates into the craft and inoculate the young herd against critical thinkers (12/21/2005).  Successful novitiates are graded on their ability to regurgitate the talking points, meditate on the non-negotiable assumptions and doctrines, sense how to tell the good guys from the bad guys (the creationists), and honor the idol of Our Leader.