Reviewers generally work without extrinsic rewards. Their names are not revealed, so their
reputations do not depend on their doing high quality reviews. Perhaps this leads them to spend little
5
time on their reviews. In any event, on average, reviewers spend between two and six hours in
reviewing a paper (Jauch & Wall 1989; King, McDonald & Roderer 1981; Lock & Smith 1990;
Yankauer 1990), although they often wait for months before doing their reviews. They seldom use
structured procedures. Rarely do they contribute new data or conduct analyses. Typically, they are
not held accountable for following proper scientific procedures. They match their opinions against
the scientific work by the authors.

Bradley (1981) asked
authors about their experience on the ?last compulsorily revised article published in a refereed
journal.@ When asked whether the changes advocated by the referees were based on ?whim, bias, or
personal preference,@ only 23% said none were, while 31% said that this applied to some important
changes. Forty percent of the respondents said that some of the referees had not read the paper
carefully.

Reviewers appear to base their judgments on cues that have only a weak relation to quality.
Such cues include (1) statistical significance, (2) large sample sizes, (3) complex procedures, and
(4) obscure writing. Researchers might use these cues to gain acceptance of marginal papers
(Armstrong 1982).
Although it typically has little relationship to whether the findings are important, correct, or
useful, statistical significance plays a strong role in publication decisions as shown by studies in
management, psychology, and medicine (Begg & Berlin 1988; Greenwald 1975; Hubbard &
Armstrong 1992; Salsburg 1985; Sterling, Rosenbaum, & Weinkam 1995). The case against
statistical significance is summarized for psychologists by Cohen (1994) and for economists by
McCloskey & Ziliak (1996).

...

Complex procedures serve as a favorable cue for reviewers. One wonders whether simpler
procedures would suffice. For example, in the field of forecasting, where it is possible to assess the
effectiveness of alternate methods, complex procedures seldom help and they sometimes harm
accuracy (Armstrong 1985). Nevertheless, papers with complex procedures dominate the forecasting
literature.

...

Obscure writing impresses academics. I asked professors to evaluate selections from
conclusions from four published papers (Armstrong 1980). For each paper, they were randomly
assigned either a complex version (using big words and long sentences, but holding content
constant), the original text, or a simpler version. The professors gave higher ratings to authors of the
most obscure passages. Apparently, such writing, being difficult to understand, leads the reader to
conclude that the writer must be very intelligent.

Although journal peer review improves quality, it does not assure it. It seems likely that
errors can be found in any published paper. Consider such a simple measure of quality as the
accuracy of references. Evans, Nadjari & Burchell (1990) studied the references cited in three
medical journals and found a 48% error rate. They said that ?a detailed analysis of quotation errors
raises doubt in many cases that the original reference was read by the authors.@ Eichorn and
Yankauer (1987) found that 31% of the references in public health journals contained errors, and 3%
of these were such that they could not locate the references. In addition and more important, Eichorn
and Yankauer found that the authors= descriptions of previous studies differed from the original authors= interpretations for 30% of the citations, with half of these descriptions bring unrelated to the
authors= contentions.

Horrobin=s (1982) opinion, as editor of two biomedical journals, was that only one-third of
reviews are accurate. In an analysis of 216 reviews for 108 manuscripts on reading and language,
Fagan (1990) concluded that reviewers made mistakes.


Murray (1988) evaluated the statistical procedures described in 28 papers that had been
published in the British Journal of Surgery and concluded that four papers should have been
rejected, seven needed major revisions, and 11 needed minor changes. Thus, only 21% of the papers
passed this post-publication peer review without the need for changes.

One belief is that authors should receive the same treatment as one another. This does not
always happen. Peters & Ceci (1982) conducted a quasi-experiment in which they concluded that
reviewers were biased against authors from unknown or less-prestigious institutions. The results of
this experiment were surprising. When I showed this design to a group of 22 academic researchers
as a proposed study, they did poorly at predicting the results (Armstrong 1982). For example, they
expected that 46% of the studies would be rejected as ?adding nothing new.@ In fact, none were.

...

Lloyd (1990) conducted an experiment in which she sent for reviews on the identical
manuscript on psychology with names that were obviously male or female (all fictitious) . She found
that female-authored manuscripts were accepted significantly more often by female reviewers (62%)
than by male reviewers (21%).

Mahoney (1977) showed that reviewers rejected papers with controversial findings because, they said, the methodology was poor, while they accepted papers with identical methodology that supported conventional beliefs.

For authors and funding organizations, peer review provides an important veneer of respectability. Findings published in a famous journal take on the aura of the journal's reputation. This may be well founded � some journals invest a lot of effort in selecting, checking and publishing articles. Yet, even the most-respected journals have been caught out and, despite extensive peer review, have ended up publishing fraudulent or seriously flawed material.

Confirmatory bias is the tendency to emphasize and believe experiences that support one's views and to ignore or discredit those that do not. The effects of this tendency have been repeatedly documented in clinical research. However, its ramifications for the behavior of scientists have yet to be adequately explored. For example, although publication is a critical element in determining the contribution and impact of scientific findings, little research attention has been devoted to the variables operative in journal review policies. In the present study, 7S journal reviewers were asked to referee manuscripts that described identical experimental procedures but which reported positive, negative, mixed, or no results. In addition to showing poor interrater agreement, reviewers were strongly biased against manuscripts that reported results contrary to their theoretical perspective. The implications of these findings for epistemology and the peer review system are briefly addressed.

One study found that the vast majority of scientists drawn from a national sample showed a strong preference for "confirmatory" experiments (Mahoney & Kimper, 1976). Over half of these scientists did not even recognize disconfirmation (modus tollens) as a valid reasoning form! In another study the logical reasoning skills of 30 scientists were compared to those of 15 relatively uneducated Protestant ministers (Mahoney & DeMonbreun, 1977). Where there were performance differences, they tended to favor the ministers. Confirmatory bias was prevalent in both groups, but the ministers used disconfirmatory logic almost twice as often as the scientists did.

What is Peer Review?

You can think of peer review as a "stamp of approval" from academic experts. When an article is published in a peer-reviewed journal, readers can be certain that experts in the relevant field have read it and, independent of their own particular opinions, verified it to meet a high standard of scholarship. Scholars rely on peer review to ensure that the conversation they have with each other is always based in good research and the established standards of their discipline.

If an article has been published in a peer-reviewed journal, you can assume that:

* it doesn't duplicate work that has already been done by other scholars
* it presents a worthwhile approach to the topic
* it exists as part of a standing conversation on the topic, as represented by its citations to other scholarship
* it meets the above and other standards of the discipline, as generally agreed upon by its scholars

When the Biological Society of Washington issued its embarrassing apology for publishing the paper, the organization pledged that arguments for Intelligent Design "will not be addressed in future issues of the Proceedings," regardless of whether the paper passes peer review.

But beyond the headlines, Ioannidis was shocked at the range and reach of the reversals he was seeing in everyday medical research. “Randomized controlled trials,” which compare how one group responds to a treatment against how an identical group fares without the treatment, had long been considered nearly unshakable evidence, but they, too, ended up being wrong some of the time. “I realized even our gold-standard research had a lot of problems,” he says. Baffled, he started looking for the specific ways in which studies were going wrong. And before long he discovered that the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals.

This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

Though scientists and science journalists are constantly talking up the value of the peer-review process, researchers admit among themselves that biased, erroneous, and even blatantly fraudulent studies easily slip through it. Nature, the grande dame of science journals, stated in a 2006 editorial, “Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.”