Over the last couple of weeks, I have really started to agree with the proponents of the “Peer-Review is Broken” notion. Well, perhaps not broken, but sickly and in need of an overhaul.
A poster here directed me to an article in the Daily Mail (yeah, I know). So I looked at it. Then I spent some time tracking down the article. The article was interesting (and, I admit, problematic to my position on something). The particular journal has a hyperlink system for comments, citations, and citing articles (in the same journal presumably).
Here’s where I run into the problem. The comments are also paywalled. With some moderate effort, the original articles can often be found on the internet. If not, then the authors can be contacted as well. But the comments… those are nearly impossible to find.
In the case of this article, the comments were critical to the reception of the paper. Until certain issues raised in the comments are addressed by the authors, then the study is useless. It may be right, it may be wrong, but until the issues are resolved, we won’t know.
But how will we know when the issues are resolved? Indeed, if it wasn’t for the journal itself (charging $35 even for the comments), then I wouldn’t know that there were any issues.
Peer-review, just like anything else, should be taken skeptically. Recent incidents (Seralini, for example) show that just because something is peer-reviewed, it’s not automatically correct.
The problem is that our current system promotes the idea (even if unconsciously) that peer-review is the end. I’ve not been critical of this process until recently. Between Seralini, other anti-GMO scientists, creationists, and global warming deniers, it’s exceedingly difficult to parse out who is trustworthy.
What’s worse, it’s becoming increasingly easy for deniers to trot out peer-reviewed research to support their position. And non-experts can’t know if those papers are any good or not. The Seralini paper was easy to deconstruct. Almost anyone with any kind of rational thinking ability could see the gaping flaws in that paper. This paper, not so much. Unless one is a soil toxicology expert, then the issue would have gone unnoticed. I sure would have missed it.
Open access is a fine idea, but the denier groups are becoming more sophisticated. This is especially true when there is financial incentive for a scientist to promote or refute a certain position (I’m looking at you Wakefield).
A start of a solution would be for journals to allow open access to all the comments and responses to a paper, with the paper itself (not separately). I’d say that even if one downloads a PDF of a paper, it should include the comments and responses (all of them). There’s been some movement toward a clearinghouse of papers and data. I’d like to see the same for responses and comments to papers.
I fear that we’re beginning to see the backlash of simple information overload. In 2009, there were over 200 papers published on Origins of Life research. That’s a tiny section of a large field (biochemistry) of a large content area (biology). I still haven’t read them all… and they are now almost 5 years old.
It’s exceedingly difficult for non-experts to stay even partially aware. This is only made worse with all the pop-science and denial websites, journals (yes, they have journals… sometimes), and blogs.
If I wanted to, I could easily get into a massive argument on some minutiae of just about anything. While I don’t have the background for it (and I accept that), the deniers also don’t have the background, but don’t accept it. The book review of Stephen Meyer has shown us that anyone can talk confidently, even when they have no idea what’s going on (or are purposefully misrepresenting science).
And confident talk, is more important than facts and evidence in our perception-based culture.
I would very much like to hear other’s thoughts on this subject. It’s been mentioned several times here at SIN, so please leave a comment with your thoughts.
