From A Storehouse of Knowledge

Peer review is the process of editors of scientific journals sending papers submitted by scientists to other scientists to review before being published. This helps ensure that the scientific research has been done properly and the conclusions drawn are legitimate. It also helps the editor decide if he should publish the paper.

Contents 1 The purpose of peer review 2 The status of peer review 3 The Peer Review Process 3.1 Shortcomings 4 Responsibilities of Reviewers 5 References

The purpose of peer review

The peer review process serves a number of overlapping purposes. The primary purpose is to ensure that research to be reported is of sufficient quality to be published. Expert reviewers are chosen so that they can evaluate the materials and methods used in a study as well as the conclusions. Since most papers are revised after being sent out for review, another important purpose of the process is to improve the paper under consideration. As “fresh eyes” on a paper, the reviewers can often catch flaws in an author’s arguments that he or she is too close to the work to see. By the time an author has submitted a paper for publication, he or she may have gone through many drafts over the course of many months (not to mention carrying out the experiments and working through the analysis). An argument that makes perfect sense to the author may not come across as clearly to someone not as familiar with the work. Peer review also serves as a first line of defense against issues of plagiarism or research fabrication. Obviously, reviewers cannot be expected to catch every potential problem, but good reviewers can raise a red flag with the editor if they see something suspicious.

The status of peer review

Economist Robert Higgs said that "Reference to “peer-reviewed research” and to an alleged “scientific consensus” are regarded as veritable knock-out blows by many commentators.".^[1] Editor of The Lancet Richard Horton said that "Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller."^[2] Wikipedia's policies for verifying claims in its articles says that "academic and peer-reviewed publications are usually the most reliable sources".^[3] Global warming proponents frequently invoked peer review as a standard by which to promote their views and reject opposing views.

The Peer Review Process

The peer review process is considered central to writing and publishing in science. In general, a scientific conclusion is not given much weight unless it has gone through the process of peer review and been published. Scientific journals generally do not invite particular authors to submit papers (except for feature articles or similar papers). Most papers are submitted to a journal based on the author’s estimation of the fit between the topic of the paper and the journal. Once the editor of a journal receives a paper, he or she needs to determine whether the paper fits the journal’s focus and whether it is of sufficient quality to be published. An editor may reject a paper out of hand if he or she does not believe the paper is a good fit or if the author did not follow the journal’s requirements. This can be quite common. For example, the British Medical Journal rejects about two thirds of submissions without going to external peer review.^[4] If not rejected internally, the editor will seek advice as to whether or not the paper should be published. This is done by sending out the paper to between two and four experts in the field covered by the paper. These experts may be people that the editor knows, people who have submitted papers to the journal, or people suggested by the author of the paper (an author can also request that certain people NOT be included among the reviewers). Since reviewing papers is voluntary (although all scientists are expected to do their fair share of reviewing), the editor will contact the potential reviewers and ask them to review the paper. If they say yes, the editor will usually give them a few weeks to read the paper and submit a written review. These reviews can cover anything relating to the paper, from broad comments about the field being studied to specific questions about individual data points. Reviewers may also comment on the quality of the writing (especially if the paper is lacking in this area), as the writing will greatly impact how understandable a paper is. Reviewers will usually classify a paper in one of three categories:

1. Accept as-is (rare)
2. Accept with revisions
3. Reject

Once the reviews have been submitted, the editor will read through them and make his or her final decision on accepting the paper. He or she is not bound by the reviewers’ suggestions, but most editors take reviewer suggestions quite seriously. If there are a mix of positive and negative reviews, the editor may choose to send the paper out to additional reviewers for more comments. Reviews are usually done anonymously, in the sense that the author of the paper usually does not know who the reviewers were, although there are cases where a reviewer will intentionally break that anonymity. Some journals use a double-blind review process, where the reviewers do not know the identity of the author, but this is less common (although there are distinct advantages to such a system). Once the final decision has been made, the author will be informed of that decision. In most cases, the text of the reviews (with any identifying information removed) is passed along to the author. This allows the author to see what revisions need to be made, but it also can provide helpful feedback on both the writing and the science behind the paper. As most papers that are accepted are accepted pending some revisions, the reviewer comments serve as a guide during the revisions. Authors should be careful to address each of the reviewer’s comments. Since reviewers are human, they do sometimes make mistakes. If an author truly disagrees with a reviewer about a point, he or she can choose not to make the suggested change, but the author should outline why he or she chose not to follow the reviewer’s advice.

Shortcomings

The peer review process, although considered a vital part of science, has a number of shortcomings and has failed spectacularly at times.

Although anonymity of authors and reviewers is often claimed to be vital, it is often difficult to achieve, due to the reviewers, by necessity, being experts in the same field as the author, and therefore likely known to each other, even if not personally. A reviewer might recognise the style or line of reasoning of an author (or an author the style of a reviewer), and this might influence the reviewer's comments. Some journals do not even try and keep the author's identity a secret.

Another problem is a potential conflict of interest, as the reviewers may be competing with the authors for research grants or publishing kudos. Many journals have Conflict of Interest policies in place to minimise this, although ultimately it often relies on the integrity of the reviewers to declare that that have a conflict of interest.

Reviewers are not expected to independently check the data or conclusions of the author, so deliberately falsified data could easily be published.

Peer review also serves to stifle ideas that are unconventional or unpopular.

Reviewers are also not necessarily neutral and may not be truly critical in the proper sense of that word. ...
Reviewers can block the publication of work antagonistic to their own views and the conventional wisdom. By blocking publication the reviewer prevents argument starting once the paper is published.
It is worth remembering that scientists are only human. It is hard for a reviewer to be objective about a paper that attacks some line that is dear to the heart (and mind) of the reviewer.^[5]

Participants in the publications process may have strong beliefs (“intellectual passion”) that commit them to a particular explanation, method, or idea. They may, as a result, be biased in conducting research that tests the commitment or in reviewing the work of others that is in favor or at odds with their beliefs. For example, if research challenging conventional wisdom is reviewed by someone who has made his or her reputation by establishing the existing paradigm, that person might judge the new research results harshly.^[6]

This is particularly the case with challenges to evolutionary ideas, and has led to creationists starting their own peer-reviewed journals.^[7]

The shortcomings of peer review are widely recognised. Economist Robert Higgs wrote:

Peer review, on which lay people place great weight, varies from being an important control, where the editors and the referees are competent and responsible, to being a complete farce, where they are not. As a rule, not surprisingly, the process operates somewhere in the middle, being more than a joke but less than the nearly flawless system of Olympian scrutiny that outsiders imagine it to be. Any journal editor who desires, for whatever reason, to reject a submission can easily do so by choosing referees he knows full well will knock it down; likewise, he can easily obtain favorable referee reports. ... Personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion, and a great deal of plain incompetence and irresponsibility are no strangers to the scientific world; ... In no event can peer review ensure that research is correct in its procedures or its conclusions.^[1]

Peter Horton, editor of The Lancet, put it even more strongly:

The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability -- not the validity -- of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.^[2]

Some scientists will go as far as trying to subvert the peer review process. Global warming proponents were caught out admitting to trying this to support their case, as revealed in their e-mails:

I can't see either of these papers being in the next IPCC report. Kevin and I will keep them out somehow - even if we have to redefine what the peer-review literature is !^[8]

This was the danger of always criticising the skeptics for not publishing in the "peer-reviewed literature". Obviously, they found a solution to that--take over a journal! So what do we do about this? I think we have to stop considering "Climate Research" as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal. We would also need to consider what we tell or request of our more reasonable colleagues who currently sit on the editorial board...^[9]

Similarly, critics of creationism charge that the peer-reviewed journals started by creationists don't count as legitimate peer-reviewed journals.

Responsibilities of Reviewers

Since reviewers play such an important part in the publication process, their role comes with a number of responsibilities. Reviewers are first and foremost responsible for carefully reviewing papers both objectively and accurately. Since reviewers are experts in the same field as the author, there are sometimes conflicts of interest, especially if the author’s work is in direct competition with the reviewers. In such cases, reviewers are asked not to read or review the work without first making the journal editor aware of the potential conflict. Another responsibility (probably the most important, as far as authors are concerned) is to complete the review quickly. Editors usually give two to three weeks for a reviewer to complete their work. If a reviewer is busy and will not be able to complete the review in that time frame, he or she should inform the editor and ask that someone else review the article. Another important responsibility is that all papers under review should be treated as confidential documents. They should not be shown to or discussed with ANYONE without prior approval of the editor. Because the work in the paper has not yet been published, releasing it early could allow someone else to “scoop” the author of the paper. A reviewer also has the responsibility to not use any information from the paper in his or her own research (unless and until it is actually published). There are two exceptions to this. The first is if the reviewer gets specific permission from the editor to contact the author and get the authors permission to use the information. The second exception is that, if a paper shows that an aspect of the reviewer’s research is unlikely to be successful, the reviewer could halt that research.

References