Reviewers generally work without extrinsic rewards. Their names are not revealed, so their
reputations do not depend on their doing high quality reviews. Perhaps this leads them to spend little
5
time on their reviews. In any event, on average, reviewers spend between two and six hours in
reviewing a paper (Jauch & Wall 1989; King, McDonald & Roderer 1981; Lock & Smith 1990;
Yankauer 1990), although they often wait for months before doing their reviews. They seldom use
structured procedures. Rarely do they contribute new data or conduct analyses. Typically, they are
not held accountable for following proper scientific procedures. They match their opinions against
the scientific work by the authors.

Bradley (1981) asked
authors about their experience on the ?last compulsorily revised article published in a refereed
journal.@ When asked whether the changes advocated by the referees were based on ?whim, bias, or
personal preference,@ only 23% said none were, while 31% said that this applied to some important
changes. Forty percent of the respondents said that some of the referees had not read the paper
carefully.

Reviewers appear to base their judgments on cues that have only a weak relation to quality.
Such cues include (1) statistical significance, (2) large sample sizes, (3) complex procedures, and
(4) obscure writing. Researchers might use these cues to gain acceptance of marginal papers
(Armstrong 1982).
Although it typically has little relationship to whether the findings are important, correct, or
useful, statistical significance plays a strong role in publication decisions as shown by studies in
management, psychology, and medicine (Begg & Berlin 1988; Greenwald 1975; Hubbard &
Armstrong 1992; Salsburg 1985; Sterling, Rosenbaum, & Weinkam 1995). The case against
statistical significance is summarized for psychologists by Cohen (1994) and for economists by
McCloskey & Ziliak (1996).

...

Complex procedures serve as a favorable cue for reviewers. One wonders whether simpler
procedures would suffice. For example, in the field of forecasting, where it is possible to assess the
effectiveness of alternate methods, complex procedures seldom help and they sometimes harm
accuracy (Armstrong 1985). Nevertheless, papers with complex procedures dominate the forecasting
literature.

...

Obscure writing impresses academics. I asked professors to evaluate selections from
conclusions from four published papers (Armstrong 1980). For each paper, they were randomly
assigned either a complex version (using big words and long sentences, but holding content
constant), the original text, or a simpler version. The professors gave higher ratings to authors of the
most obscure passages. Apparently, such writing, being difficult to understand, leads the reader to
conclude that the writer must be very intelligent.

Although journal peer review improves quality, it does not assure it. It seems likely that
errors can be found in any published paper. Consider such a simple measure of quality as the
accuracy of references. Evans, Nadjari & Burchell (1990) studied the references cited in three
medical journals and found a 48% error rate. They said that ?a detailed analysis of quotation errors
raises doubt in many cases that the original reference was read by the authors.@ Eichorn and
Yankauer (1987) found that 31% of the references in public health journals contained errors, and 3%
of these were such that they could not locate the references. In addition and more important, Eichorn
and Yankauer found that the authors= descriptions of previous studies differed from the original authors= interpretations for 30% of the citations, with half of these descriptions bring unrelated to the
authors= contentions.

Horrobin=s (1982) opinion, as editor of two biomedical journals, was that only one-third of
reviews are accurate. In an analysis of 216 reviews for 108 manuscripts on reading and language,
Fagan (1990) concluded that reviewers made mistakes.


Murray (1988) evaluated the statistical procedures described in 28 papers that had been
published in the British Journal of Surgery and concluded that four papers should have been
rejected, seven needed major revisions, and 11 needed minor changes. Thus, only 21% of the papers
passed this post-publication peer review without the need for changes.

One belief is that authors should receive the same treatment as one another. This does not
always happen. Peters & Ceci (1982) conducted a quasi-experiment in which they concluded that
reviewers were biased against authors from unknown or less-prestigious institutions. The results of
this experiment were surprising. When I showed this design to a group of 22 academic researchers
as a proposed study, they did poorly at predicting the results (Armstrong 1982). For example, they
expected that 46% of the studies would be rejected as ?adding nothing new.@ In fact, none were.

...

Lloyd (1990) conducted an experiment in which she sent for reviews on the identical
manuscript on psychology with names that were obviously male or female (all fictitious) . She found
that female-authored manuscripts were accepted significantly more often by female reviewers (62%)
than by male reviewers (21%).

Mahoney (1977) showed that reviewers rejected papers with controversial findings because, they said, the methodology was poor, while they accepted papers with identical methodology that supported conventional beliefs.