PERSPECTIVES IN

Publishing
No.2 August 2004

Is peer review
Perspectives in Publishing
is an occasional newsletter for
journal editors. It aims to
address generic issues in
scholarly communication.

For general information about

Elsevier, or any publishing
in crisis?
related matter contact
your Publishing Editor or visit A. Mulligan
the Elsevier website Elsevier
at:http://www.Elsevier.com/
Peer review is an essential component of scholarly
publishing. In recent years it has attracted criticism and
For further information about
this article or if you require its role has been challenged. Based upon the findings of
further copies, please contact: focus groups commissioned by Elsevier, this paper
Adrian Mulligan examines the role of peer review and discusses if, and
Elsevier
The Boulevard
how, it could be improved.
Langford Lane
Kidlington Oxford
OX5 IGB
[email protected]

Copyright © Elsevier Ltd.

Perspectives in Publishing No. 2 Is peer review in crisis?

Review by peers has been a method of evaluation since Greek editor has requested that the research upon which the paper
times (Barnes, 1981) and has been a formal part of scientific is based be repeated. This was done most famously when
communication since the first scientific journals appeared Nature published the work of Jacques Benveniste1 (as
over 300 years ago. The Philosophical Transactions of the quoted by Bhattacharya, Shaoni, 2003). Repeating the
Royal Society is widely accredited as being the first journal to experiment is exceptional, and an expensive and impractical
formalise the process (Zuckerman & Merton, 1971 pp 68-69). suggestion when considering the whole of scholarly
The referee is now at the heart of scientific publishing and publishing. It is also worth noting Nicholas Wade’s
according to Ziman (1968, p111) is ‘…the lynchpin about comments, “There are plenty of honest reasons why two
which the whole business of Science is pivoted.’ researchers may get different results from the same
experiment.” (Wade, 2003)
It is testament to the power of peer review that a scientific
hypothesis or statement, presented to the world is largely Editors who attended the focus groups agreed that recent
ignored by the scholarly community unless it is first developments, if adopted more widely, would make the
published in a peer-reviewed journal. It is precisely because process of publishing fraudulent work more difficult. Such
of this pivotal role as well as some notable incidents of fraud mechanisms include vouchsafing the article, where the
and incompetence that the process has been subjected to a author is asked to sign a declaration stating the work
variety of criticisms in recent years. submitted has not been falsified and is their own work.
Further, a journal editor may insist that each of the
Possibly the most damaging incidence was in September
corresponding authors or significant contributors sign a
2002, when Jan Hendrik Schon, tipped to be a Nobel Prize
document stating that they agree to publication, including
winner, was discovered to have published a series of
the head of the respective department (if their name is not
fraudulent papers. Subsequently, 16 papers he had published
already present). While unlikely to prevent the truly
were withdrawn from Nature, Science, Physical Review and
determined falsifier, formalisation of the institutional review
Applied Physics Letters (Lerner, 2003). Similarly fabrication
will certainly make it more difficult.
has been found in the life sciences: the German molecular
biologists, Fridhelm Herrmann and Marion Brach, were There’s a strong case for viewing the prevention of
accused of inventing data in forty-seven papers published in fraud as the direct responsibility of the lab chief. The
a number of prestigious periodicals (Hauptman, 2002). lab chief is in the best position to detect fraud. Only
he can demand to see the lab notebooks, evidence
Unfortunately fraud is not the only issue, a study published
that is beyond the reach of outsiders. (Wade, 2003).
on the effects of ‘ecstasy’ had to be retracted by Dr. George
Ricaurte of Johns Hopkins University when it was realised Arguably, the ultimate recourse would be some form of legal
that a more potent drug had been tested by mistake action; however, editors who attended the focus groups agreed
(Ricaurte, 2003). that the international nature of scholarly publication makes
enforcement difficult and thus unlikely. In the absence of a
These events have acted as a catalyst within the scholarly
legally binding framework some medical editors have
community, with many questioning the role of peer review.
collaborated to establish the Committee of Public Ethics
Recently, the Royal Society, Britain’s most eminent academy,
(COPE)2. The group acts as a sounding board for Editors
established a working group specifically to examine peer review.
struggling to deal with possible breaches in research and
So, after 300 years of scholarly publishing is it time to publication ethics. COPE is attempting to put research
radically overhaul the peer review process? Qualitative misconduct firmly on the academic and educational agenda.
research into peer review conducted by Elsevier, across a Similarly, the World Association of Medical Editors (WAME)3 seeks
range of disciplines, consulting separate groups of authors, to provide support. Unfortunately, such groups have limited
referees and editors, suggests not. Altogether 59 respondents, jurisdiction and ability to enforce any decisions they make.
drawn from the USA, UK, and Germany, attended 6 focus
The simplest course of action for the editor or reviewer who
groups with each group lasting 2-3 hours. Attendees had a
suspects fraud is to reject or recommend rejection of the
range of experience, some were new to refereeing, editing or
paper. Unfortunately, the history of science is littered with
authoring, while others had years of experience. Those
incidences of original research papers, later proven to be
participating recognised that the process of review could be
correct, being rejected because they included unbelievable
improved, but none believed that the fundamental precept of
results or equivocal conclusions. The premise of scholarly
peer review should change. Sir Peter Lachmann, President of
communication is that research published is new and
the Academy of Medical Sciences, and responsible for
original. Often, only the author is in a position to truly say
formulating proposals to deal with fraud agrees:
a piece of research is false.
Peer review is to science what democracy is to politics.
It’s not the most efficient mechanism, but it’s the 1 Beneviste claimed that homeopathic water retained ‘memory’ of
least corruptible. (Lachmann, Peter, 2002) substances once dissolved in it. The peer review process agreed
involved the experiments being repeated and observed by independent
Fraud experts – the results could not be duplicated. Benveniste’s reputation
was left in ruins; he lost his funding and subsequently his position.
What can the reviewer or editor do if they suspect fraud? In 2 http://www.publicationethics.org.uk/
the past when suspicions have been strong enough the 3 http://www.wame.org/

2
Perspectives in Publishing No. 2 Is peer review in crisis?

Editors agreed that by its very nature science is self- training good referees.
correcting. Fabricated or misinterpreted data will eventually
How best to do this depends upon fathoming the
be detected. Subsequent papers will attempt to build on the
motivations and influences of referees. A number of key
reported findings and discover the errors. This is the nature
motivations emerged during the referee focus groups. All
of scholarly publishing. It could be argued that what has
reviewers felt it was their academic duty to review,
changed is not the process of science or an increase in the
recognising that they expected someone to do the same for
number of cases of fraud, but the pressures on science,
them. Many stated they were driven by a general interest in
notably the public desire for instant and irrefutable
the area and a desire to be up-to-date with latest
solutions to complex problems.
developments. Some thought it would help their own
The purpose of peer review is often misunderstood, not only research or stimulate new ideas. For younger scientists being
by some scientists, but the media and public generally. Peer asked to review was perceived as an honour and
review cannot guarantee the correctness of the results. As confirmation of their standing in the community. Others
Marc H. Brodsky, executive director of the American hoped that reviewing would encourage the editor to view
Institute of Physics, states “Referees cannot determine if them more favourably.
data is falsified, nor are they expected to”. Nor is it likely a Marginal motivations that emerged during the groups
referee will be in a position to spot an error as a included a desire to build a relationship with the editor or
consequence of a mislabelled drug, as in the case of the being associated with a prestigious journal. A number
Ecstasy research retraction (Ricaurte, 2003). Moreover, enjoyed reading research prior to their colleagues. Finally,
scientists announcing their findings in the press prior to for some, it served as career development; participation
peer review have exacerbated the situation. An unsuspecting demonstrated a wider interest, indicating initiative and
public is not always aware of the distinction between peer commitment.
review and scientific proclamations in the media. In terms of attracting referees, many made it very clear they
Peer review: purpose and practice only wished to review papers relevant to their area of
expertise; moreover, a paper perceived as groundbreaking
If peer review is not meant to detect fraud, what is its would guarantee review. All agreed that reviewing was time
purpose? Unfortunately, there is no one text to which consuming especially if they were unfamiliar with the area;
authors, reviewers or editors alike can refer. Peer review is therefore they rejected papers outside their immediate
an imprecise term and varies across disciplines: at its widest sphere of knowledge.
it is, ‘the assessment by an expert of material submitted for
publication.’ (Carin Olson, 1990, p356-58) While the reputation of a journal was a significant factor
influencing a referee’s decision to review, often his or her
The attendees of the focus groups broadly believe that peer own experience on the journal was decisive. Referees are
review should prevent an author making egregious claims on also authors; a poor experience as an author affects a
minimal results, and where feasible, identify incidences of referee’s decision to review for a particular journal. The
plagiarism. Furthermore, it should ensure that a consistent influence of the editor is also important; a referee is more
and appropriate methodology is used, and that recent likely to review if they personally know the person
reputable work in the area, upon which the article may be contacting them. The mode of communication is also
based, is correctly referenced and acknowledged. significant, the majority accept e-mail as standard, but a
While the focus groups attendees recognised fraud was likely number stated they would reject articles if they did not
to continue to capture media attention, they believed other come as hard copy.
more damaging issues were not. Referees felt that the Obstacles identified by referees included difficulties in
overall burden of review is increasing; as more papers are understanding papers that were badly written or from non-
being published, fewer reviewers are willing to referee. English speakers. Such papers invariably took longer to
Authors were recognised by editors and referees alike as review and referees felt they should be rejected or improved
becoming increasingly demanding, requiring faster and prior to being sent to them. Editors participating in separate
faster review responses. Authors thought that bias and focus groups saw peremptory rejection as an issue. Editors
professional conflict was an issue in reviewing, that stated they often see their role as mentor to the author,
unscrupulous reviewers delayed publication or appropriated developing and improving the standard of a paper for final
ideas. Editors felt that it was becoming increasingly difficult publication, a process some believed required a substantial
to attract and retain good referees; that authors were guilty amount of effort on behalf of both the editor and referee.
of multiple submissions, fragmenting studies into ‘minimum
publishable units’, and that plagiarism was increasing. The role of the editor in evaluating a paper upon receipt is
important. A study (Lavelle, 1966, p3-12) of 166 humanities
Improving Peer Review journals revealed that only 9.6% of editors made an
acceptance or rejection decision themselves. In contrast
So how can the process of peer review be improved? The
editorial staff on the Lancet evaluate all manuscripts.
solution in part may lie in changing some review practices,
Richard Horton, Editor of The Lancet, said: “All submissions
but all attending the focus groups agreed that timely, high
are carefully read and reviewed by a team of physicians and
quality reviews depended upon attracting, retaining and

3
Perspectives in Publishing No. 2 Is peer review in crisis?

scientists in-house before a decision is made about external value. It was suggested that feedback would aid both
peer review. At that point about 25% of submissions are sent personal and professional development. Many wanted to see
out for further consideration, a vital step in our quality feedback on their own performance, perhaps an end of year
assurance process. No research paper is published in The review. Individuals who had received an acknowledgement or
Lancet which has not gone through expert and statistical some form of feedback in the past implied they were more
review”. pre-disposed to review for that journal.
By way of a compromise, and in the absence of extensive Referees did not consider incentives essential, but were
editoral resources, it was suggested by some editors that nonetheless welcome; suggestions included book discounts,
poorly constructed papers should be rejected prior to print cartridges and bottles of wine. Recognition emerged as
external peer review and the author provided guidelines on a key covert factor, for many the review process is perceived
how to write a research paper. as a ‘chore and not a pleasure’. Reviewers feel this way
because they are not rewarded or recognised for their work.
Furthermore, the author should be encouraged to resubmit
the paper after revision. For papers where language is a It was felt that many of the perceived problems afflicting the
substantial issue, it was agreed that the author should be refereeing process could be remedied if the review was
referred to a specialised translation service or encouraged to formally recognised. Such a system could take the form of
have the paper moderated by a native English speaker. accreditation to a journal, society or publisher. It might
identify the number of times an individual reviews in
The views of referees different prestigious journals; such an indicator could in part
be a measure of that individual’s contribution to science. In
According to the referees, encouraging participation in the
the absence of such a system, limited recognition could be
review process rests upon fundamental criteria: sending only
provided by a framed certificate for younger referees, an end
relevant papers of at least a minimum standard in quality,
of year acknowledgement, a listing of those who had
both in terms of content and language. Furthermore,
reviewed published in the journal, or a formal letter to the
referees indicated that communication is more effective if it
referee or the head of the referee’s department.
is personalised, polite and efficient. In order to ensure such
communication, one or two editors mentioned that they Referees raised a number of issues with the concept of
keep notes on any pertinent personal details, such as referee payment, believing strongly that any financial reward
birthdays or recent holidays, which they then use in their would undermine objectivity. Further, even with a financial
communications. incentive some were unlikely to review more thoroughly, as
both resources and time were limited.
In order to save time, a number of referees stated that they
preferred to receive prior notice before receiving a full paper. Increasingly, publishers are becoming more active in the
In fact some stated that they would not review a paper if it refereeing process through the facilities they offer. The
arrived unsolicited. Referees indicated that notification, development of online systems is perhaps the most
particularly in the form of an abstract would allow them to significant contribution, potentially speeding up the process
make a quick and informed decision. It negated the need to of refereeing. Online reviewing was perceived as
read the whole paper and also had the considerable advantageous for the author and editor rather than the
advantage of saving the referee the cost of printing. reviewer. Nearly all referees accepted the inevitability of it,
but felt they were shouldering an extra burden. They now
Editors recognised that sending advance notice to just two
had to print the manuscript, which is both time-consuming
referees allows for a quick response, and importantly, the
and expensive. Moreover, returning comments or
option for the editor to move onto another referee within a
recommendations electronically was seen as restrictive,
short period of time if an invitation is declined or ignored.
some expressed a desire for flexibility of format.
Even more importantly, it reduces the overall number of
reports produced by up to a third, diminishing the sense of Referees indicated it was more likely, that a specific and
burden among referees. reasonable timeline (3 to 4 weeks was appropriate for most),
in combination with some of the points already mentioned
Retaining the services of the best referees also involves
(e.g. limits on the number of reviews) would help speed up
setting limits on how many times a referee will be requested
review. In addition, they indicated that they did not mind a
to review, and communicating that limit to them. Many
polite reminder from the editor. In some incidences a letter
referees felt overburdened with requests and indicated that
from the editor was seen as a useful tool to help focus the
they invariably take longer to review if they believe there is
referee on their task. Some referees stated that they chose
a risk they will receive more requests.
not to commence their review until a reminder arrived.
A number of referees stated that they wished to see other
Authors and referees alike believed that professional
referees’ comments and receive feedback on the paper, not
conflicts are increasingly common in peer review,
only in terms of the overall editorial decision, but also the
particularly in an arena in which the progress of science is
criteria used. Some reviewers were upset to discover a paper
becoming more heavily dependent upon funding for its
was published when they had advised that it be rejected. In
success. It was recognised that this most affected niche
contrast others felt that there was a lack of editorial
areas, where experts reviewing a paper may be competing for
authority, too many referees’ comments were taken at face

4
Perspectives in Publishing No. 2 Is peer review in crisis?

the same funding as the author. A number of attendees scientific value. Some referees questioned whether it would
recounted incidences of sabotaged papers; where the ever work, indicating that as a researcher, it was unlikely
publication of research had been delayed in order to allow that they would look at work on the web that was not peer
the referee’s own research to be published first. reviewed, unless it was extremely important to the field,
much less comment upon the work themselves.
New referee practices
Editors suggested a simple and common sense solution to
Some referees believed that the different peer review the problem of ‘political’ refereeing: always seek a third
practices, such as double blind refereeing or open opinion when two reports contradict one another or when a
refereeing, if adopted globally, might alleviate some of the report is unnecessarily delayed. Further, it was suggested
problems with unscrupulous referees. However, they also that circulating referee reports to the other referees would
recognized that adopting these practices might create more be beneficial: not only would it satisfy the needs of referees
problems than they resolve. who wish to develop professionally: it also has the
advantage of eliminating those reviews with an agenda.
Double blind refereeing, whereby both referee and author
Moreover, it was acknowledged that referee reports were
remain anonymous, was considered ‘ideal’ as it served to
likely to be tempered, once it was known they were
avoid all potential bias, particularly for those authors who
circulated to other reviewers as standard practice.
might be working in a developing country or be associated
with an institute with an ‘ordinary’ reputation. It was also However, for some editors other concerns were more
seen as relevant in incidences where ‘prestigious’ authors’ pressing. They believe that plagiarised, re-published or
papers are reviewed with more regard to the reputation of multiple-submitted works, as well as research fragmented
the author than the content of the research paper. However, into ‘minimum publishable units’ are increasingly common.
it was questioned whether a paper could ever be truly blind, Detecting such practices in part lies with publishers. One
especially in ‘niche areas’. Many claimed they could identify approach might be to provide a utility that matches
the author, through the style or subject matter of the paper, sections of submitted text with the full text in other
or more often through self-citation. A study by Justice et al articles. Another approach might be to provide a database
(1998) supports the assertions of the referees. The study to reviewers of all abstracts to articles in-print and
shows that blinding had little effect on the quality of published. However, it is worth noting that Mabe and Amin
review, was not successful in 32% of cases, and that well- (2002) believe there is little evidence to support the theory
known authors had been far more difficult to blind. of ‘salami publishing’. Contrary, they show that the number
of papers per author is dropping.
Open refereeing, where the referee and author are known to
one another, was preferred by some respondents. They Improving peer review reports
believed it prevented ‘malicious’ comments, stopped
plagiarism, prevented referees drawing upon their own Editors indicated there was a large degree of variability in
‘agenda’, and encouraged honest open responses. However, the standards of referee reports. Some reports could be as
the majority of referees believed it would achieve the short as one sentence; others ran to several pages, while in
opposite effect, and promote less open and less honest a few incidences a referee almost completely reworked the
reports. Many felt that anonymity was key to reviewing in manuscript. Editors recognised that referees were often
order to avoid ‘politics’. A number said they would be less young, or up for tenure and may only recently have started
likely to be as forthright in their opinion if their comments reviewing. They need advice and like authors themselves,
were attributed. It was also believed that the problem of mentoring. It was generally accepted that the mechanism of
referees with an agenda would re-emerge, as a junior peer review is largely unknown and hidden. Increasing
researcher reviewing an eminent scientist’s work would be confidence and understanding in the system, it was
less likely to be honest for fear of affecting their own career suggested, is based upon establishing a transparent
or funding opportunities. Independent studies tend to framework.
support these comments. Open reviewing is practised on
Establishing such a clear framework involves clearly
the ‘British Medical Journal’ and in a study conducted by
identifying to all the mechanics and expectations of peer
the journal itself it recognised there was no discernable
review. A number of journals to varying degrees already
improvement in the quality of reviewing, and importantly it
provide guidelines, in a survey of 139 journals Weller (2002,
significantly increased the likelihood of reviewers declining
p25) estimates that about half (51.8%) indicate their
to review (Susan Van Rooyen et al, 1999).
journal’s position on anonymity of their authors and
A slightly different version of open refereeing exists on the reviewers. In the same survey just over a third of editors
worldwide web; papers are made available for comments by informed the wider community of their rejection-rate, while
readers, the article is subsequently revised and resubmitted. only 17.3% gave detailed guidelines for reviewers.
On balance, the majority of respondents attending the focus Transparency benefits readers, authors, reviewers and
groups rejected it, not only for the reasons mentioned editors alike, “There appears to be a trend in this direction,
previously, but because respondents believed that research with an obvious benefit to both journal readers and
had to pass through a formal review process in order to scholarly community as a whole” (Weller 2002, p27).
prevent poor quality papers overshadowing those of real Readers know the rigour of the review process, and thus are

5
Perspectives in Publishing No. 2 Is peer review in crisis?

able to make a judgement upon the standard of articles Acknowledgements

within the journal. Authors are able to make a more
informed decision as to where to submit a paper. Reviewers DVL Smith – Independent Research Agency who conducted four
of the focus groups.
will have a clearer idea of what is expected of them. The
editor hopefully sees a greater number of submissions, finds Bibliography
referees more willing and will receive fewer queries
regarding the refereeing process. Barnes, Jonathan. (1981) “Proof and the Syllogism”. p17-59 in
Berti. Discusses the principles of Aristotle’s endoxos
Ideally such transparency would include the following:
Bhattacharya, Shaoni (22nd October 2003). “Homeopathy
• A policy on anonymity. reduces arsenic poisoning in mice”, New Scientist
• A policy statement on whether all articles are sent out Brodsky, Marc H. (2003) as quoted by Lerner in “Fraud Shows
for review.
Peer-Review Flaw” Vol 8, Iss 6, p12-17. The Industrialist
• Information on how reviewers are selected. Physicist.
• The number of reviewers typically involved. Hauptman, Richard. (2002) “Dishonesty in the Academy”,
• The expected length of the review process. Iss. 2 Academie,.

• The protocol a referee is expected to follow. (Criteria Justice AC, Cho MK,Winker MA. (1998) ‘Does masking author
for publication evaluation) identity improve peer review quality?’ JAMA; 280 pp 240-
242
• How the final decision is made.
Lachmann, Peter. (2002) “The research integrity initiative:
• How the referee’s remarks are communicated to the progress report”, Cope Report, p11,
author.
• A statement on conflicts of interest. (e.g. referees who Lavelle. (1966). Facts of Journal Publishing, IV. Publications of
have collaborated with the author recently) the Modern Language Association of America, 81 (6).

• Rejection rates. Lerner, Eric. (2003) “Fraud Shows Peer-Review Flaw” Vol 8, Iss 6,
p12-17. The Industrialist Physicist.
• Feedback to referees.
Mabe, Michael and Amin, Mayur (2002) “Dr Jekyll and Dr Hyde:
So, is peer review in crisis? The simple answer is no, but author-reader asymmetries in scholarly publishing” Aslib
that is not to say it cannot be improved. Enhancing peer Proceedings, Vol 54, iss 33 pp149-157
review does not necessarily involve supplanting current
processes, rather developing and supplementing existing Olson, Carin M. (1990, July) “Peer Review of the Biomedical
Literature,” American Journal of Emergency Medicine 8
practices. The attraction and retention of the best referees
no.4: 356-358.
who will improve peer review by providing consistent,
timely and quality reviews is key. Facilitating such quality Smith, Richard. (1999, January 2) “Opening Up BMJ Peer
reviews is based upon practices such as asking referees to Review,” British Medical Journal 318 (7175) 4-5.
review only relevant papers, setting limits on the number of Ricaurte, George. (2003, Sept 8th) Letter to Science.
times they will be asked to review. Sharing referee reports
Van Rooyen, Susan et al. (1999) “Effect of open peer review on
between referees will encourage constructive criticism.
quality of reviews and on reviewers’ recommendations; a
Informing the referee of the final decision, providing a randomised trial”, BMJ, 318, 23-27 p12.
personalised service, one that allows flexibility of response,
giving the referee recognition will help develop the referee Wade, Nicholas. (2003, Jan 27)“Fraud Happens: What to Do
and make them feel valued. About It”, The Scientist, Volume 17, Issue 2, 56,
Weller, A. (2001) Editorial Peer Review: It’s strengths and
For researchers peer review is the bedrock upon which
Weaknesses, ASIS&T: Information Today Inc Medford, New
scholarly publishing is based. However, for much of the Jersey.
wider public, its function is misunderstood. This is an issue,
and the public need to be assured that scientific publishing Ziman, J.M. (1968), Public Knowledge: an essay concerning the
social development of science. London: Cambridge
is self-correcting and ultimately will detect fraud. While
University Press
steps can be taken to avoid deception, peer review should
not be a tool whose primary function is the categorical Zuckerman, H. & Merton, R.K. (1971, January). Patterns of
detection of fraud. Editors, referees and authors alike agree evaluation in science: institutionalisation, structure and
peer review aspires to improve the quality of scientific functions of the referee system. Minerva, 9(1), 66-100
research published, and where feasible assure the
correctness of the findings published. As Weller states:
Both accidental and deliberate mistakes do happen,
but that is not reason to scrap editorial peer review
or to underestimate the tremendous importance of
editorial peer review to the communication of
scholarly and scientific information. (Weller, 2001,
p322)