Steve
2006-03-31 14:21:28 UTC
From
http://chronicle.com/free/v49/i21/21b02001.htm
POINT OF VIEW
The Seven Warning Signs of Bogus Science
By ROBERT L. PARK
The National Aeronautics and Space Administration is investing close to
a million dollars in an obscure Russian scientist's antigravity
machine, although it has failed every test and would violate the most
fundamental laws of nature. The Patent and Trademark Office recently
issued Patent 6,362,718 for a physically impossible motionless
electromagnetic generator, which is supposed to snatch free energy from
a vacuum. And major power companies have sunk tens of millions of
dollars into a scheme to produce energy by putting hydrogen atoms into
a state below their ground state, a feat equivalent to mounting an
expedition to explore the region south of the South Pole.
There is, alas, no scientific claim so preposterous that a scientist
cannot be found to vouch for it. And many such claims end up in a court
of law after they have cost some gullible person or corporation a lot
of money. How are juries to evaluate them?
Before 1993, court cases that hinged on the validity of scientific
claims were usually decided simply by which expert witness the jury
found more credible. Expert testimony often consisted of tortured
theoretical speculation with little or no supporting evidence. Jurors
were bamboozled by technical gibberish they could not hope to follow,
delivered by experts whose credentials they could not evaluate.
In 1993, however, with the Supreme Court's landmark decision in Daubert
v. Merrell Dow Pharmaceuticals, Inc. the situation began to change. The
case involved Bendectin, the only morning-sickness medication ever
approved by the Food and Drug Administration. It had been used by
millions of women, and more than 30 published studies had found no
evidence that it caused birth defects. Yet eight so-called experts were
willing to testify, in exchange for a fee from the Daubert family, that
Bendectin might indeed cause birth defects.
In ruling that such testimony was not credible because of lack of
supporting evidence, the court instructed federal judges to serve as
"gatekeepers," screening juries from testimony based on scientific
nonsense. Recognizing that judges are not scientists, the court invited
judges to experiment with ways to fulfill their gatekeeper
responsibility.
Justice Stephen G. Breyer encouraged trial judges to appoint
independent experts to help them. He noted that courts can turn to
scientific organizations, like the National Academy of Sciences and the
American Association for the Advancement of Science, to identify
neutral experts who could preview questionable scientific testimony and
advise a judge on whether a jury should be exposed to it. Judges are
still concerned about meeting their responsibilities under the Daubert
decision, and a group of them asked me how to recognize questionable
scientific claims. What are the warning signs?
I have identified seven indicators that a scientific claim lies well
outside the bounds of rational scientific discourse. Of course, they
are only warning signs -- even a claim with several of the signs could
be legitimate.
1. The discoverer pitches the claim directly to the media. The
integrity of science rests on the willingness of scientists to expose
new ideas and findings to the scrutiny of other scientists. Thus,
scientists expect their colleagues to reveal new findings to them
initially. An attempt to bypass peer review by taking a new result
directly to the media, and thence to the public, suggests that the work
is unlikely to stand up to close examination by other scientists.
One notorious example is the claim made in 1989 by two chemists from
the University of Utah, B. Stanley Pons and Martin Fleischmann, that
they had discovered cold fusion -- a way to produce nuclear fusion
without expensive equipment. Scientists did not learn of the claim
until they read reports of a news conference. Moreover, the
announcement dealt largely with the economic potential of the discovery
and was devoid of the sort of details that might have enabled other
scientists to judge the strength of the claim or to repeat the
experiment. (Ian Wilmut's announcement that he had successfully cloned
a sheep was just as public as Pons and Fleischmann's claim, but in the
case of cloning, abundant scientific details allowed scientists to
judge the work's validity.)
Some scientific claims avoid even the scrutiny of reporters by
appearing in paid commercial advertisements. A health-food company
marketed a dietary supplement called Vitamin O in full-page newspaper
ads. Vitamin O turned out to be ordinary saltwater.
2. The discoverer says that a powerful establishment is trying to
suppress his or her work. The idea is that the establishment will
presumably stop at nothing to suppress discoveries that might shift the
balance of wealth and power in society. Often, the discoverer describes
mainstream science as part of a larger conspiracy that includes
industry and government. Claims that the oil companies are frustrating
the invention of an automobile that runs on water, for instance, are a
sure sign that the idea of such a car is baloney. In the case of cold
fusion, Pons and Fleischmann blamed their cold reception on physicists
who were protecting their own research in hot fusion.
3. The scientific effect involved is always at the very limit of
detection. Alas, there is never a clear photograph of a flying saucer,
or the Loch Ness monster. All scientific measurements must contend with
some level of background noise or statistical fluctuation. But if the
signal-to-noise ratio cannot be improved, even in principle, the effect
is probably not real and the work is not science.
Thousands of published papers in para-psychology, for example, claim to
report verified instances of telepathy, psychokinesis, or precognition.
But those effects show up only in tortured analyses of statistics. The
researchers can find no way to boost the signal, which suggests that it
isn't really there.
4. Evidence for a discovery is anecdotal. If modern science has learned
anything in the past century, it is to distrust anecdotal evidence.
Because anecdotes have a very strong emotional impact, they serve to
keep superstitious beliefs alive in an age of science. The most
important discovery of modern medicine is not vaccines or antibiotics,
it is the randomized double-blind test, by means of which we know what
works and what doesn't. Contrary to the saying, "data" is not the
plural of "anecdote."
5. The discoverer says a belief is credible because it has endured for
centuries. There is a persistent myth that hundreds or even thousands
of years ago, long before anyone knew that blood circulates throughout
the body, or that germs cause disease, our ancestors possessed
miraculous remedies that modern science cannot understand. Much of what
is termed "alternative medicine" is part of that myth.
Ancient folk wisdom, rediscovered or repackaged, is unlikely to match
the output of modern scientific laboratories.
6. The discoverer has worked in isolation. The image of a lone genius
who struggles in secrecy in an attic laboratory and ends up making a
revolutionary breakthrough is a staple of Hollywood's science-fiction
films, but it is hard to find examples in real life. Scientific
breakthroughs nowadays are almost always syntheses of the work of many
scientists.
7. The discoverer must propose new laws of nature to explain an
observation. A new law of nature, invoked to explain some extraordinary
result, must not conflict with what is already known. If we must change
existing laws of nature or propose new laws to account for an
observation, it is almost certainly wrong.
I began this list of warning signs to help federal judges detect
scientific nonsense. But as I finished the list, I realized that in our
increasingly technological society, spotting voodoo science is a skill
that every citizen should develop.
Robert L. Park is a professor of physics at the University of Maryland
at College Park and the director of public information for the American
Physical Society. He is the author of Voodoo Science: The Road From
Foolishness to Fraud (Oxford University Press, 2002).
http://chronicle.com
Section: The Chronicle Review
Volume 49, Issue 21, Page B20
http://chronicle.com/free/v49/i21/21b02001.htm
POINT OF VIEW
The Seven Warning Signs of Bogus Science
By ROBERT L. PARK
The National Aeronautics and Space Administration is investing close to
a million dollars in an obscure Russian scientist's antigravity
machine, although it has failed every test and would violate the most
fundamental laws of nature. The Patent and Trademark Office recently
issued Patent 6,362,718 for a physically impossible motionless
electromagnetic generator, which is supposed to snatch free energy from
a vacuum. And major power companies have sunk tens of millions of
dollars into a scheme to produce energy by putting hydrogen atoms into
a state below their ground state, a feat equivalent to mounting an
expedition to explore the region south of the South Pole.
There is, alas, no scientific claim so preposterous that a scientist
cannot be found to vouch for it. And many such claims end up in a court
of law after they have cost some gullible person or corporation a lot
of money. How are juries to evaluate them?
Before 1993, court cases that hinged on the validity of scientific
claims were usually decided simply by which expert witness the jury
found more credible. Expert testimony often consisted of tortured
theoretical speculation with little or no supporting evidence. Jurors
were bamboozled by technical gibberish they could not hope to follow,
delivered by experts whose credentials they could not evaluate.
In 1993, however, with the Supreme Court's landmark decision in Daubert
v. Merrell Dow Pharmaceuticals, Inc. the situation began to change. The
case involved Bendectin, the only morning-sickness medication ever
approved by the Food and Drug Administration. It had been used by
millions of women, and more than 30 published studies had found no
evidence that it caused birth defects. Yet eight so-called experts were
willing to testify, in exchange for a fee from the Daubert family, that
Bendectin might indeed cause birth defects.
In ruling that such testimony was not credible because of lack of
supporting evidence, the court instructed federal judges to serve as
"gatekeepers," screening juries from testimony based on scientific
nonsense. Recognizing that judges are not scientists, the court invited
judges to experiment with ways to fulfill their gatekeeper
responsibility.
Justice Stephen G. Breyer encouraged trial judges to appoint
independent experts to help them. He noted that courts can turn to
scientific organizations, like the National Academy of Sciences and the
American Association for the Advancement of Science, to identify
neutral experts who could preview questionable scientific testimony and
advise a judge on whether a jury should be exposed to it. Judges are
still concerned about meeting their responsibilities under the Daubert
decision, and a group of them asked me how to recognize questionable
scientific claims. What are the warning signs?
I have identified seven indicators that a scientific claim lies well
outside the bounds of rational scientific discourse. Of course, they
are only warning signs -- even a claim with several of the signs could
be legitimate.
1. The discoverer pitches the claim directly to the media. The
integrity of science rests on the willingness of scientists to expose
new ideas and findings to the scrutiny of other scientists. Thus,
scientists expect their colleagues to reveal new findings to them
initially. An attempt to bypass peer review by taking a new result
directly to the media, and thence to the public, suggests that the work
is unlikely to stand up to close examination by other scientists.
One notorious example is the claim made in 1989 by two chemists from
the University of Utah, B. Stanley Pons and Martin Fleischmann, that
they had discovered cold fusion -- a way to produce nuclear fusion
without expensive equipment. Scientists did not learn of the claim
until they read reports of a news conference. Moreover, the
announcement dealt largely with the economic potential of the discovery
and was devoid of the sort of details that might have enabled other
scientists to judge the strength of the claim or to repeat the
experiment. (Ian Wilmut's announcement that he had successfully cloned
a sheep was just as public as Pons and Fleischmann's claim, but in the
case of cloning, abundant scientific details allowed scientists to
judge the work's validity.)
Some scientific claims avoid even the scrutiny of reporters by
appearing in paid commercial advertisements. A health-food company
marketed a dietary supplement called Vitamin O in full-page newspaper
ads. Vitamin O turned out to be ordinary saltwater.
2. The discoverer says that a powerful establishment is trying to
suppress his or her work. The idea is that the establishment will
presumably stop at nothing to suppress discoveries that might shift the
balance of wealth and power in society. Often, the discoverer describes
mainstream science as part of a larger conspiracy that includes
industry and government. Claims that the oil companies are frustrating
the invention of an automobile that runs on water, for instance, are a
sure sign that the idea of such a car is baloney. In the case of cold
fusion, Pons and Fleischmann blamed their cold reception on physicists
who were protecting their own research in hot fusion.
3. The scientific effect involved is always at the very limit of
detection. Alas, there is never a clear photograph of a flying saucer,
or the Loch Ness monster. All scientific measurements must contend with
some level of background noise or statistical fluctuation. But if the
signal-to-noise ratio cannot be improved, even in principle, the effect
is probably not real and the work is not science.
Thousands of published papers in para-psychology, for example, claim to
report verified instances of telepathy, psychokinesis, or precognition.
But those effects show up only in tortured analyses of statistics. The
researchers can find no way to boost the signal, which suggests that it
isn't really there.
4. Evidence for a discovery is anecdotal. If modern science has learned
anything in the past century, it is to distrust anecdotal evidence.
Because anecdotes have a very strong emotional impact, they serve to
keep superstitious beliefs alive in an age of science. The most
important discovery of modern medicine is not vaccines or antibiotics,
it is the randomized double-blind test, by means of which we know what
works and what doesn't. Contrary to the saying, "data" is not the
plural of "anecdote."
5. The discoverer says a belief is credible because it has endured for
centuries. There is a persistent myth that hundreds or even thousands
of years ago, long before anyone knew that blood circulates throughout
the body, or that germs cause disease, our ancestors possessed
miraculous remedies that modern science cannot understand. Much of what
is termed "alternative medicine" is part of that myth.
Ancient folk wisdom, rediscovered or repackaged, is unlikely to match
the output of modern scientific laboratories.
6. The discoverer has worked in isolation. The image of a lone genius
who struggles in secrecy in an attic laboratory and ends up making a
revolutionary breakthrough is a staple of Hollywood's science-fiction
films, but it is hard to find examples in real life. Scientific
breakthroughs nowadays are almost always syntheses of the work of many
scientists.
7. The discoverer must propose new laws of nature to explain an
observation. A new law of nature, invoked to explain some extraordinary
result, must not conflict with what is already known. If we must change
existing laws of nature or propose new laws to account for an
observation, it is almost certainly wrong.
I began this list of warning signs to help federal judges detect
scientific nonsense. But as I finished the list, I realized that in our
increasingly technological society, spotting voodoo science is a skill
that every citizen should develop.
Robert L. Park is a professor of physics at the University of Maryland
at College Park and the director of public information for the American
Physical Society. He is the author of Voodoo Science: The Road From
Foolishness to Fraud (Oxford University Press, 2002).
http://chronicle.com
Section: The Chronicle Review
Volume 49, Issue 21, Page B20