A reply to Michael Eisen

Feb 19 2016 Published by under Careers, Life, NIH

Michael Eisen, strong proponent of open access and preprints, stopped by and left a comment on my previous post. You can read it in situ here. This is my response.

"There's a simple solution ... don't do it that way. Don't think anyone thinks it's a good idea to have journals run pre-print servers. This is something to be done by the National Library of Medicine and NSF in US and by comparable organizations internationally who share preprints and create a resource where anyone - researchers, healthcare works, teachers, students, interested members of the public - can get access to any paper published by anyone, anywhere. You really think that this is a bad thing?"

I absolutely agree that it's a bad idea to have journals run preprint servers. But who or what is going to stop them? Federal mandate? That hasn't worked so well for PubMed Central. Community pressure? It's certainly growing with OA, but preprints are a significantly greater shift from the cultural norm, so it will likely take much longer.

As we've seen with predatory OA journals, any git with a PC and internet connection could, in principle, set up a preprint server. They likely won't because, unlike OA, it would be hard to make a profit. But the point is it's ridiculously easy, and if journals see some advantage to doing so, they will. Glammagz will respond to anything they perceive as a threat to the glamhumping culture their empires are built upon. Will that response take the form of preprint servers? Who knows. Just don't underestimate them.

Do I think preprint servers are a bad thing? I noted in my post I was considering using them, so no, I don't think they're a bad thing. I do have concerns though and I'll discuss those below.

"Yes, it's possible to corrupt such a system if we don't make sure it's structured and run fairly. But the solution is to do it well. And you really think complaints about science publishing are a fad? You really think a system that costs billions of dollars, is deeply biased and is structured to all but ensure that the rich get richer, takes longer to publish papers than it does to send rockets to Mars, and which denies 99.99% of the people on the planet, including over 99% of teachers and students, and well over 50% of practicing scientists access isn't borked?"

I would love to see a system that's run fairly. And done well. But who's going to enforce that?

My fad comment was not to do with complaints about science publishing. It was aimed at preprints. They can be called a fad in the same way many things can. Some fads disappear, some go on to become mainstream (rock and roll was once considered a fad). At this point I don't believe preprints have gained sufficient traction in the biosciences to know which way they'll go.

I'm a supporter of making the results of research available to all, and I do believe there are issues with publishing. Not necessarily all the same ones you do. For example, the journals I submit to take way less time to review and publish manuscripts than it takes to get to Mars.* Perhaps people should try submitting to places with more reasonable turnaround times. And I believe in peer review, warts and all.

But let me ask you, will widespread adoption of preprint servers cause a new set of problems? I like the idea of making research results available to all, but without prior review? Not so much. What's to stop deposition of pseudoscience in preprint servers?** Sure practicing scientists are capable of judging the science or lack thereof, and should be doing so. But the public? Don't give me some bull about post-deposition, pre-publication review. That relies way too much on people bothering to record their thoughts (PubMed Commons hasn't exactly been a runaway success). And how are the public to judge any such comments that are recorded? >98% of all climate scientists agree climate change is an urgent issue bordering on disaster, yet large swathes of the public remain skeptical. Paid much attention to what's going on in Britain? Homeopathy is a part of their national healthcare system. Put there by college-educated people, some of whom have MDs. All of this will be made worse if preprint servers are run by the NLM and/or NSF (as I've agreed they probably should be) and have the associated implied stamp of government approval. The public isn't stupid. They just don't necessarily have the tools required to judge the science (and pseudoscience) that would become available. Shouldn't that be a concern?



* As I was typing this I got reviews back for a submitted manuscript. Three weeks from submission to reviews.

** Yes, I am aware that some pseudoscience slips through the current peer-review process. Addressing a slow drip by shearing the faucet off the wall doesn't seem terribly sensible to me.





5 responses so far

  • drugmonkey says:

    And not only should we be selecting the journals that have reasonable turnaround-- we should be *rewarding* them. Spark a positive forward feed and other journals will follow suit.

    I have a current fave sweet spot journal- good turnaround time, fast to print after accept and a JIF that works for my scenario. That is what needs to be stamped with a rainbow unicorn IMO.

  • dr24hours says:

    There's no plumbing problem that isn't fixed by burning the house down.

  • I understand all of the things you are concerned about and think there are legitimate reasons to be worried about all of them. My thoughts on these issue:

    1) On journals running pre-print servers

    Obviously there's nothing that can or should stop people from setting up pre-print servers, just like there's nothing that can or should stop people from setting up new journals. But just because people set something up, doesn't mean we have to use them, or sanction them. I think the path forward is for the NIH/NLM/etc... to set up criteria for what constitutes a legitimate source of manuscripts, just like they have criteria today for what constitutes a legitimate journal. It's not perfect, but they do a reasonable job of balancing the desire to include all legitimate journals in PubMed against the need to filter out bogus journals (none of the predatory open access journals are included in PubMed). If all pre-prints can be accessed in one place, people will go to this place to get papers, and I don't think there will be much incentive for journals to create bespoke pre-print servers. I'm not trying to trivialize the problem, but I think it's a solvable one.

    2) Regarding the issue of how the public would use preprints

    First of all, I don't buy the argument that pseudoscience gets a significant boost from the ability of its practitioners to get their work published in peer-reviewed journals. Do anti-vaxxers, GMO alarmists, homeopathists, creationists, climate denialists, etc... make use peer-reviewed publications aligned with their interests? Of course they do. Does this explain their success, or even significantly contribute to it? I don't think so. Most of what they cite is stuff from outside the peer-reviewed literature, and the people who subscribe to what they are selling are clearly not basing their views on the scientific legitimacy of their arguments. Yes, they are helped by the imprimatur of legitimacy they get. But is this so significant a problem that we should let it dictate how scientists communicate? I can't see how that's the case.

    But let's stipulate that it is. What is the solution? We already know that the current peer review system is porous enough that this kind of stuff gets published, and I don't see how this is going to get any better as the number of papers and journals proliferate. To me, problem number 1 is that we have overlegitimized peer review. People falsely believe that peer-reviewed = true, in no small part because we have told that that it is.

    So I think step 1 is to make it abundantly clear (by writing it on the manuscripts) that pre-prints have not been peer-reviewed, and should be treated as what they are, unvetted claims made by the authors. Then, step 2, as these manuscripts get reviewed, either by existing journals or by new types of post-publication review, the results of this review process can be attached to the paper in a highly visible way.

    I think this would have two positive effects. First, the boost crackpot theories get from having the imprimatur of peer-review would be blunted by a preponderance of unambiguously negative reviews that are attached to the paper. And second, by opening up the process of peer review, one can at least hope that some people will get a better appreciation for how we actually go about sciencing and deciding what is and isn't true.

    Now I'm not naive enough to think this is going to stop people from abusing science for their ends. I mean it's not like any anti-vaxxers actually read Wakefield's paper, let alone all the critical things that have been written about it. I just think such people will always find ways to attach legitimacy to their illegitimate claims, and while we can take reasonable steps to try to make it difficult for them, it's foolish to let them hold us hostage and force us to make it more difficult for us to do science.

    • odyssey says:

      Thanks for the reply. Note it's not those with a pseudoscience agenda that I'm concerned about. It's those who can't tell/don't know the difference.