In this reprint,
Curmudge and Jaded Julie discuss the traditional practice of conceiving,
writing, and publishing a peer-reviewed scientific paper. It was revealed in
November of 2009 that some climate science researchers had violated aspects of
this long-accepted practice. These
events were given the name “climategate.” To a published scientist, climategate
was abhorrent. In the view of Kaizen
Curmudgeon, the perpetrators of climategate should no longer participate in the
global warming debate.
Reprinted from Curmudgeon’s
Wastebasket Originally Posted February 10, 2010
“Curmudge, the term ‘peer-reviewed’
has been in the news a lot during the past few months. This might be one of
those extremely rare blog topics about which you know something. Would you be
so kind as to enlighten me, or is this a subject—like differential
equations—that you have totally forgotten?”
“I would be delighted to share my
recollections, Jaded Julie. Because I had manuscripts peer reviewed between
1961 and 2002 and did a bit of reviewing myself, much of it has become
hardwired.”
“Please give it a try, Old Guy.
Let’s start by looking at the process from the author’s standpoint and how he
or she decides what is publishable.”
“The critical guideline in this
whole process, Julie, is integrity. If data from an experiment or series
of experiments provides the basis for a finding or story heretofore unknown to
the world or to others in your discipline, it is a candidate for publication.
But one can’t cherry-pick his data. If some of the data don’t fit your story,
they can’t simply be ignored; they can only be rejected if they are statistical
outliers.”
“I can appreciate the importance of
integrity. Doesn’t everyone have some element of vested interest in his
research?”
“Vested interest comes in many
flavors, Julie. Late in one’s career, the author’s delight in contributing to
knowledge would represent the low end of the vested interest scale. At the far
end of the scale is current work in atmospheric science. It seems that in that
discipline one’s findings impact multimillion-dollar technological ventures,
politics, and the fate of the industrialized world. The higher one goes on the
vested interest scale, the greater the temptation to compromise one’s integrity.”
“So back to the guy in the lab. The
author writes his paper, including data to support his hypothesis (remember, no
fudging or cherry-picking) and enough experimental detail to permit someone to
duplicate the work, and sends it off to a journal for publication, right?”
“Exactly. Often he must wait many
months—up to a year—for the peer review process to transpire.”
“Am I correct, Curmudge, that
publication in a peer-reviewed journal has traditionally been the ‘gold
standard’ for scientific knowledge?”
“You are essentially correct, Jaded
Julie. Remember, however, that the ultimate tests are (1) one’s work must be
independently reproducible by someone else, and (2) any forecasts must turn out
to be true. Nevertheless, despite all of its warts and blemishes and potential
for hanky-panky, the peer review process has worked pretty well. Because my
area was not highly controversial and was populated by a limited number of
scientists, peer review was not very onerous for either authors or reviewers.”
“Curmudge, what’s that ‘potential
for hanky-panky’ that you mentioned earlier?”
“Here’s an example. Consider a topic
about which there are two schools of thought and in which editors and reviewers
have a vested interest. Remember that peer review is a single-blind process in
which the reviewers know the author but not vice-versa.”
“I think I can see what’s coming,
Curmudge. If the editor’s position on the topic is ’A’ and the author is
advocating position ‘B’, the editor can reject the paper out of hand or send it
to reviewers that share position ‘A’ who will recommend rejection. I feel sorry
for the poor author. He’s been shot down, but he doesn’t know by whom. And
potential reviewers who are interested in position ‘B’ may wonder why they
never receive any manuscripts to review. ”
“That’s it, Julie. If the author
can’t find another journal whose reviewers will at least consider
evidence of position ‘B’, he and position ‘B’ are effectively suppressed. This
would appear to be ‘political’ science, i.e., science for the purpose of
politics or vested interests. My speculation is that this might represent the
untimely and unjustified demise of traditional peer review.”
“So how do we summarize the lessons
of these last four postings?”
“It’s pretty simple, Julie. Be wary
of politicians talking about science, of people confusing effect with cause, of
scientists with a vested interest using complex mathematical models that can’t
be experimentally verified, and of researchers who suppress publication of
findings that don’t agree with their own.”
Kaizen Curmudgeon
Link to posting from blog archives: Reliability and Patient Safety: 11/06/08
http://kaizencurmudgeon.blogspot.com/2008/11/reliability-and-patient-safety.html
No comments:
Post a Comment