Peer review, schmeer review

Aug 13 2010 Published by under Uncategorized

Originally posted Friday August 13, 2010 at LabSpaces (here).

There is a recent piece in the The Scientist titled "I Hate Your Paper" that outlines the problems with the current peer review system and describes some of the alternatives that have been suggested. This has been discussed byOrac, which was followed up on by DrugMonkey.

There is no doubt that the current peer review system has some major problems. I won't defend that. However, some of the so-called "alternatives" seem to me to be prone to the same, if not worse, issues. Let us not forget,

Peer review in any shape or form is done by people.

Yes, reviewers can be snarky, unreasonable, stupid, vindictive and/or lazy. And editors can be snarky, unreasonable, stupid, vindictive and/or lazy.

So can authors.

I've certainly been on the receiving end of what I consider to be completely unreasonable reviews.

As a reviewer I've probably written reviews that the authors have considered completely unreasonable.*

And as an editor I've probably made decisions that authors have considered completely unreasonable.*

Peer review in any shape or form is done by people.

So let's consider some of the suggested alternatives (not all, just some).

Anonymous Authors

This system would work just the way current peer review does, except that the names and affiliations of the authors are not revealed to the reviewers. The idea is that the biases of the reviewers against competitors, old enemies, humanity in general, will be minimized. Nice idea, but it won't work. Most publications are chapters in an ongoing story being told by a lab (or labs). The authors cite the work done leading up to the current chapter, usually with emphasis on the preceding work done in the lab(s) of the authors. It is just too easy to figure out who the senior authors are.

Non-Anonymous Reviewers

In this model, apparently already adopted by some publications, the names of the reviewers are revealed at the time of publication. The thought here is that reviewers will take their jobs much more seriously if they know their name will be associated with the final product.

Riiiiggggghhhhhtttt.

Do proponents of this system have any idea just how hard it is to find reviewers under the current system? Any reviewers, not just good ones? Working in an editorial capacity I routinely have to ask 6-8 people before I can get two to agree to review (and here we're assuming those two will actually complete their reviews and in a timely manner).** And those lists of potential reviewers authors have to provide when submitting a manuscript? Often useless. Once I cull out cronies, former (and sometimes present!) collaborators and people with absolutely no expertise in the field, there's often no one left.

If I now have to tell potential reviewers their names will be revealed to the authors, what do you think is going to happen? If I ask someone to review a manuscript by Professor Standing-Member-Of-The-Study-Section-Your-Grant-Goes-To, what do you think they're going to do? What about a manuscript by Professor Senior-Dood-Who-Has-The-Most-Influence-In-Your-Field? Or Professor I-Have-A-Nobel-Prize? Most will find some excuse not to accept the invitation to review (especially young faculty if they have any sense). Those that do may not be as unbiased as one would hope. Are they really going to trash a manuscript (that deserves it) by someone who has some direct control over their future, given that, should the manuscript actually be published the names of the reviewers will be revealed? Maybe, maybe not. At least reviewer anonymity provides some semblance of protection against retribution.***

Under this system people will be very, very circumspect about which manuscripts they agree to review. Making it harder to find reviewers and making the whole process even longer than the current one.

Post-Publication Review

The idea here is that authors can go ahead and publish without review, likely on a web site somewhere, and the scientific community can decide whether or not the work holds up. There would be some sort of ranking system, perhaps comments associated with the manuscript in the same manner comments are linked to blog posts, or enumeration of how many times a manuscript is incorporated into someone's personal on-line library. This is kind of like a mega-peer review system.

I like this idea in principle. But...

Reviewing via comments? Despite valiant efforts by organizations such as PLoS, this just hasn't worked. I'm not sure why.

Ranking by counting how often an unreviewed manuscript is included in someone's personal on-line library? Seriously? Who has time to read all the relevant papers in their field that have been through the current peer-review system? Not me. And at least 50% (I suspect more like 70%) of all manuscripts never make it into print. So we're going to critically read twice as many, if not three times as many, manuscripts in order to figure out which are reasonable? No. What I suspect will happen is the known authors in a given field will have their manuscripts included in libraries, and the rest will be largely ignored...

I don't know what the answer is. As stated above, there is no doubt that the current peer review system has some major problems. But when trying to come up with an alternative, let's all keep in mind that

Peer review in any shape or form is done by people.

* The authors may or may not be right.

** I'm not talking just about over the summer when it's notoriously difficult to find people to review. I'm talking about year round, over at least the last 4-5 years.

*** And before you say "But I can always figure out who reviewed my manuscript!", let me tell you no, you can't. A very senior editor once told me that he often gets people coming up to him to tell him they know exactly who reviewed their manuscript. According to him, in 99% of cases they're wrong.

Comments are off for this post