Science 4 October 2013:
Vol. 342 no. 6154 pp. 60-65
Special Issue News

Who's Afraid of Peer Review?

John Bohannon | 234 Comments

A spoof paper concocted by Science reveals little or no scrutiny at many open-access journals.

Add a new comment

These postings do not necessarily represent the views/opinions of Science.

I agree that most open access journals are very bad, and as a researcher I have been very careful to submit papers to this type of journals. However, it would have been very good if this interesting study would have covered traditional journals, because worrying surprises would probably have emerged. I also notice the current practice of several mainframe journals, that have produced a second-grade sister journals, to which they encourage rejected papers to be submitted (and pay the price if they are accepted). I would like to see how these open access journals compares in relation to less reputed ones!!?

Submitted on Sat, 04/12/2014 - 11:11

Fascinating conversation, everyone, thanks!

Quality is, from what we’ve observed, an issue with OA and paywalled journals alike. And we think it’s a brilliant opportunity to show the huge value of peer review when properly done.

We understand that it’s very difficult to catch every potential error or problem pre-publication (although one can always try), which is where post-publication peer review becomes of vital importance: errors and inconsistencies can be picked up and discussed.

In this way, the general scientific/academic canon can be kept as free as possible of poor quality research. And it’s for these reasons that we offer post pre-and post-publication peer review capabilities on our platform: we want as much high-quality peer review information on papers as possible to be available.

And, of course, for peer reviewers to be rewarded for their contribution to the system :)

aimee whitcroft Community wrangler Publons

Submitted on Sun, 03/30/2014 - 21:39

You did not need to bother. The experiment has already been done. I hope Science will now report that the issue is not exclusive to open access journals. See

Submitted on Fri, 02/28/2014 - 04:51

One of the most annoying issues with these predatory journals are the amount of submission requests and review requests they send out. They are randomly assigned and I receive mails for anything from dentistry to chemistry, but never in my field of expertise.

Submitted on Thu, 02/20/2014 - 15:29

I sympathize with Roos's view, in that the author should have included subscription-based journals in this "sting" as a control group.

As to the defensive statements about "trust": there is a certain element of trust involved, at least regarding accurate reporting of data. This does *not* excuse an editor from the responsibility of ensuring that the conclusions formed are actually *supported* by the data that is reported, and that the data is obtained by sound methodology.

Submitted on Wed, 01/29/2014 - 22:03

I came to this article from another link... I just would like to add a word: Bohannon seems to be 'excited and happy' about his deficient 'scoop'. It would be interesting that Bohannon, with his 'high skills and intelligence' writes another good bogus paper with his 'excellent writing style' and submit it to Science itself. I am sure that his paper will be accepted immediately. You can also put your name as the fake supervisor and put any other name as a first author, your fake paper will still be acceptable by Science and its sisters... This is a paper full of conflict of interests against open access journals.. One day, not so far undoubtedly, Science will also become an open access journal... What will you say, John? We will see...

Submitted on Sat, 01/25/2014 - 18:00

Bohannon has identified the problems of scientific publishing, which are well known facts.Scientists and researchers are aware of many of these journals and their these shoddy practices publishing. The only thing Bohannon has done is to put pieces of jigsaw puzzle put together through this "sting" and through a systematic approach as is done for scientific experiments. Though Bohannon has identified the problems, i wonder why he has not attempted to provide concrete solutions. It is easy to find fault and problems, but without solid, reliable solutions problems do not wean away on their own. I would have appreciated Bohannon's efforts even more, had he attempted to provide solutions. It is a common sense to refrain from publishing in those journals that seek exorbitant money for publishing an article, in the name of open access. What is required is solutions beyond that. The other issues is of "ethical standards" which are always preached and talked about in scientific research. Communicating a manuscript with a fake name, i feel, is an "unethical way" of conducting research. This could open way for many more such enthusiast like Bohannon to communicate "similar fake articles" to various journals only to withdraw manuscripts later, thereby wasting time of reviewers and others involved in the publication process. To conduct such research, Bohannon should have communicated manuscript in his own name and not a fake "African name".

Submitted on Fri, 01/03/2014 - 14:55

This spoof experiment is valuable in revealing serious problems with open access. But it is an unethical experiment on human subjects with no informed consent. I can accept such experimentation on humans in extremely limited circumstances when there is serious danger to life. But this is different. Best wishes to all, Frank Leavitt (Yeruham)

Submitted on Fri, 12/27/2013 - 22:29

The spoof paper of John Bohannon is topped by ´Near-Death-Experience´(NDE)-research Since 1975, when ´Life after Life´was published, a lot of research on NDEs and dying experiences was done: But the term Near-Death makes no sense and should therefore never be used in a scientific paper - how close (distance / time) can the death be to a person: is the death an object?. And a person who had a dying experience has to be a corpse after this experience. Important studies to NDEs are Prof. Fischer´s ´Immortality Project´(worth $5M) and Sam Parnia´s ´The AWARE Study´. NDE-research can be found on countless universities all over the world

Submitted on Sun, 12/22/2013 - 11:14

To defend not really top science is not an easy matter. Yet, let us remember an important principle. The “Principle of charity” in published texts is as follows: When criticizing the text we should criticize only what is explicitly said or written and NOT what may be implied or not explicitly affirmed. Bite (fake articles) was very short paper and (this is my opinion) may be acceptable in principle as a «brief communication». The main drawback of the bait articles was that the introduction and discussion were very short and extremely modest. Some possible answers to the alleged “flows” could be as follows. The figure 1 does not really show dose – response? This is true. However this could be explained by the fact that the agent was probably very specific and very powerful so that it already had its maximal effect by the first concentration tested and then the effect remained constant. Is there an artefact of ethanol? No, the concentrations of ethanol were too small to have an effect (0.66% ethanol, just marginally too high). Figure 2 claims that there is an increase of sensitivity to radiation, although the controls were not irradiated? The response does not start from 100% but much lower and at that portion there is obviously an increased sensitivity to the agent. The control bars seem to be superfluous in both graphs.

Other alleged “flows” I rejected in my earlier comments. Therefore the claim of Bohannon that some OA journals may be publishing bad science is not warranted. I doubt that the author of the bait articles, not being pharmacologist, was aware of the fact that his articles were not so seriously flown. “Science” accepted his report, which is far less scientific (I this explained earlier 11.9. and 11.11. 2013). The whole enterprise of “Science” to expose some predatory journals was therefore in vain. And the method used, unfortunately, immoral.

Submitted on Wed, 12/04/2013 - 08:16

I am afraid I did not say what dr. Romario Melo seems to agree with. I have expressed my own opinion in my previous comments. Anyone is interested can read them.

Submitted on Tue, 11/26/2013 - 03:44

The academia can withstand only so much pressure before it becomes a faithful reflection of the society at large. The prevailing narrative is money. Schools continue to build new facilities like nothing happened. The administration is bloated as never before and has become grotesque. It is also grossly overpaid. Meanwhile the NIH policies promote degradation of all the morals in the scientific community. Their funding policies promote superficial research where PIs keep writing modular ($250000 grants) that cannot support a research projects forcing PIs to write myriad of those and to conduct research in as many areas as possible to support these applications (no overlap, mind you). The grant reviewers are corrupted with little favors that NIH bestows on them for participating in the dysfunctional review process. Publishing a small number of in depth research papers is not profitable any longer. The number of papers its all that matters with only lip service praising the quality these days.

Submitted on Mon, 11/25/2013 - 12:49

John Bohannon’s Science article – “Who’s Afraid of Peer Review?” – explores an important issue. The response has been impressive. Even so, some miss the key point.

Bohannon does not challenge open access, and he did not criticize all open access publishing. He looked at “Open Access Gold,” where authors or their institutions pay publication fees. The possibility of a gold rush is the cause of a Wild West mentality.

Determining the degree of similar problems in paper journals would require a similar study. This was not Bohannon’s task. We have all seen stupidity among reviewers and authors at paper journals. What we do not see in subscription journals is the cash incentive to publish utter rubbish. Paper, printing, and delivery are costly. In the absence of a publication fee, there is no incentive to pan for gold. The incentives go to building impact and attracting subscribers.

The study is entirely useful in an environment with an increasing number of unethical publishers. Those caught out may not like it, but there is no harm to human subjects in catching them.

This is an important contribution to the contemporary debate on scholarly communication. This sting focused on what I describe as “the OA Gold Rush.” The debate is timely and welcome.

Ken Friedman

University Distinguished Professor, Swinburne University of Technology, Melbourne, Australia

Guest Professor, College of Design and Innovation, Tongji University, Shanghai, China

Submitted on Tue, 11/19/2013 - 09:53

Dear Bohannon, I guess you got my comment wrongly. I know GMR and how "sh.." they are, as do so everyone who submit their papers to it, however with the "publish or perish"+pressure scientists find better to have something published (even better when it's easy and without many restrictions like in GMR) than nothing. This was my point when talking about the comunity's support for the scheme. On the other hand, my criticism was more on very basic statements from any small and simple scientific journal: "the data should present new data/insights..." yours does not; "figures should be self-explanatory..." yours is not, unless you intend the readers "to study" the figure and data for several minutes in an attempt to find a country-based pattern. I think I agree with Mrs. Raimo's post in here, who should care about geography and statistics when talking about geographical locations, numbers and percentages, right? After all we are in a new era of science in which Science AAAS is accepting papers without "control treatment" and figures not self-explanatory... not even talking about the many retractions we have seen from authors in Science AAAS in the past years!

Submitted on Mon, 11/18/2013 - 03:48

Meta-analysis is a powerful tool used in diverse disciplines to quantitatively synthesize findings across studies that address a similar question. Data from the peer-reviewed literature dominate most meta-analyses. Thus, open-access publishing has the potential to transform the field of meta-analysis through the massive influx of readily-accessible articles. However, Bohannon (2013) documents a major flaw with the rise in new, open-access journals – namely, the lack of credible, unbiased manuscript peer-review. Study quality is rarely considered in meta-analysis, except when experimental error is considered (i.e., weighted meta-analysis). Instead, meta-analysts rely on an objective, peer-review process to adequately vet most of their data sources. If Bohannan’s findings are indicative of open-access publishing, in general, I expect the utility of meta-analysis to wane in the future as meta-analysts are forced to choose which studies to consider for their analyses thus further biasing their syntheses. And, all-inclusive meta-analyses that include data from journals with suspicious review processes will not be deemed reliable. Given that meta-analysis is, in part, used to develop protocols and treatments that protect and save human lives, there are obvious serious repercussions to a flawed peer-reviewed system that includes open-access and traditional publications. The scientific community needs to develop a plan for dealing with dubious scientific publishers before meta-analysis is lost.

Submitted on Sat, 11/16/2013 - 08:28

Thank you, Bohannon, for your article showing facts. From them one can learn much more than from lessons of Statistics or Geography.

Submitted on Thu, 11/14/2013 - 04:45

Bohannon’s article is the perfect example of its own criticism. To have an article making those claims without statistics, proper controls, no references, being publish in Science (even if it is on the news section) just illustrates how badly we need proper scientific training not only among reviewers, bad desperately among our peers. Dear John, if you wanna talk the talk, you gotta walk the walk.

Submitted on Tue, 11/12/2013 - 10:26

I found the idea and outcome very interesting, however unfortunately that much work does not show anything very new to anyone, with exception of those who play blind in the nowadays scientific world. After the coining of "publish or perish", obviously in a capitalist world lots of "smart" people would take advantage on that and the issue in evidence here has happened more often ever since. We should not forget that this has been supported by many scientists, who very often heavily criticize competitors within the same field of research for approaching a specific method and even claim they are "faking data", but on the other hand do "fancy" analysis and "pull off" extreme data from their experiment to publish cleaner data without regret. What I find even more interesting is someone who conducts this kind of assay publishes a map with an absolute wrong geographical location of a City. AND THIS HERE IS PUBLISHED IN SCIENCE, isn't it??? Did Science Editors have checked the "fancy" submitted Map? I doubt it! Please check the location for Genetics and Molecular Research journal, and then type the informed location on google maps.

Submitted on Sat, 11/09/2013 - 17:28

I encourage you to read the emails that led to the acceptance of the fake paper by Genetics and Molecular Research:

Judge for yourself if you think that was a provisional acceptance. All of the emails from every submission are public record:

As for the map, the points show country location, not city location.

Submitted on Mon, 11/11/2013 - 10:43

In addition, and this is the greatest problem, the act of producing and sending over 300 fraudulent articles to the scientific journals is a moral disaster. It would be certainly impossible not to condemn, on deontological grounds, this act or to try to defend it by employing utilitarian arguments, by using the principle of double effect or some other similar justification. Simply because the effects of this fable observational report will certainly be null. And, nevertheless, it was long known that number of journals had problematic reviewing procedure and other morally acceptable methods could have been employed to warn such journals.

Submitted on Sat, 11/09/2013 - 07:37

Bohannon’s article is itself a perfect example of “bad science” as he himself defined it. Therefore NOTHING can be concluded from his study (declared as “news” by Science (!?). The serious flaws of Bohannon’s bite*, were: bad controls for the experimental groups, wrong legend for one graph, non idiomatic English, and the conclusion contained unjustified, and therefore “fallacious” recommendations. Astonishingly, if we would examine the very Bihannon’s article published in Science, what we find is that it has itself similar weaknesses i.e.: there is no randomization of his “experimental group”, there are no controls at all (he could have compared the OA journals to the ordinary big journals), there was elimination of non –responders and there was no application of the intention to treat principle in the analysis. Then, even worse: there was no inferential statistics and no references. Finally, Bohannan maintains, about the conclusion in his “bite” articles, “to be bad”. This is simply not true: it states in reality that the studies that had to be conducted, should be done “in animal and human”, which suggests only implicitly though clinical studies! The bite article of Bohannan was withdrawn from OA journals (victim), so unfortunately we do not know what type of the article it would have been if published (pilot study, short report, special report, news or else). In discussions that followed its publication in Science - it IS interpreted not as a simple interesting “news” but as a scientific study. This is indirectly suggested also in the subtitle. What follows from Bohannon’s text is that the journal where Bohannon’s text itself was published (Science) belongs also to the “victim” group of journals characterized by Bohannon as publishing weak science. Science was sufficiently clever to have published Bohannons article as “news”, which apparently liberated it from the obligation to apply scientific method. The scientific method exists to try to establish what is an what is not, and even when it is used correctly, the conclusions are uncertain. This text contradicts the very principle of science. This is how so famous journal as Science is, in fact failed badly and shot itself in the foot!

Submitted on Sat, 11/09/2013 - 02:40

There is certainly a lot of things that are not pleasant about open access. But one can write very bad stories about the traditional-journal peer review as well. Perhaps this even the reason why the open access started in the first place? Because people are fed up with the outright dishonesty and violence that reigns in these journals. Peer review in traditional journals is extremely biased by people who have managed to get influence. They would eliminate a competitor with dishonest reviews. Instead of improving the quality of a paper, they eliminate good papers. I sometimes spotted errors in my own papers after submission, but these were never discovered in the review. However, I have received many times reviews that made a lot of indignated fuss phantom errors that were just not there. Just because people were jealous and wanted to push their personal agenda by taking advantage of the secrecy of the system. Being a referee is a godsend to get rid of the guy who is better than you. And the fraud is often set up with highly sophisticated psychological tricks. These dishonest referees have the warrant of strict impunity because their name remains secret. As long as there are not dishonest referees convicted in Court, with very incisive sentences, traditional journals should just shut up complaining about what happens elsewhere. Because the Schoen affaire is only the tip of an iceberg. Just google "peer review in medicine in the US". There also a system of anonymous peer review exists. And it is abused by unscrupulous people to eliminate doctors who are more competent than they are, such that they fear to loose patients. Just having an argument about a place in a parking lot can be enough to tick somebody off. As a consequence of this mobbing an estimated 300 physicians commit suicide per year in the US. Great is it not? Also there, nobody wants to listen. If the traditional journals could start sweeping in front of their own doors and install the possibility to legally inquire into the ethics of the reviewers? And what about editorial boards who take decisions about papers without sending them out for review? Has there been a declaration of financial interests in the publication of the study?

Submitted on Fri, 11/08/2013 - 07:45

The story of Ocorrafoo Cobange, a supposed biologist at the mysterious Wassee Institute of Medicine in Asmara (‘Who’s afraid of Peer Review?’, Science, 4 October 2013, vol. 342, page 60), is one of numerous examples of scandals in peer review. The case raises two problems that publishers, journals, and the scientific community continue to struggle with. First, the internal evaluation processes of academic journals remain hidden from scrutiny. It is hard to improve something that is not observable. Allowing more systematic analyses of peer review is an essential first step to improving journals’ processes. Funding agencies, publishers, and influential journals should be establishing new initiatives to promote data sharing and analysis.

Secondly, peer review relies on voluntary work by referees. While financial rewards may be counter-productive (merely yielding a ‘crowding out’ of intrinsic motivations), exploiting reputational motives, which are the main driving force of scientists’ behavior, could be really helpful. We should be exploring incentives such as appointing excellent referees to membership of journal boards, ensuring that reviewing counts for promotion, and publishing quality assessments of referees’ reviews. Implementing changes such as these would make the existence of the next Ocorrafoo Cobange at least more problematic.

Flaminio Squazzoni Department of Economics and Management, University of Brescia, Via San Faustino 74/B, 25122 Brescia, Italy. Email:

Nigel Gilbert Centre for Research in Social Simulation, University of Surrey, Guilford, Surrey, GU2 7XH, United Kingdom. Email:

Submitted on Tue, 11/05/2013 - 17:35

John Bohannon has tackled a contentious problem in modern science; the quality of peer-review. Bohannon attempted to clarify the murky underworld of open-access journals, revealing little signs of credible peer-revision. His efforts in alerting researchers to unscrupulous publishers should be applauded. However, there is a major “red flag” in Bohannon’s article itself. “In fact, it should have been promptly rejected” as well. This fatal flaw is the lack of a control. Anyone, indeed, should be well aware of the importance of a control as the foundation of a scientific method (1). Hyperbolically, consider an experiment where 1000 women were tested on their driving ability. Can we conclude that women are poor drivers without assessing men too? Yet, by not applying a control to his study, Bohannon’s finding of “little or no scrutiny at many open-access journals” may be considered misleading. Though not openly mentioned by Bohannon, such a statement implies that the situation would be different in subscription-based journals; something that cannot be substantiated. What is even more surprising is that this was not just an unfortunate oversight by the author. In his reasoning, Bohannon honestly admitted that if he “had targeted traditional, subscription-based journals”, he would have obtained the same result, as “strongly suspected” and suggested by David Roos. Additionally, the peer-review process relies on the quality of the reviewers (2), and not solely the editorial board and the publisher, and even less importantly where their offices and banks are located. Simply passing blame onto the new comers, the open-access publications with offices based in non-Anglophone countries, does not provide a solution. The question is who is afraid of science being global, being a business, and not being immune to changes? Adapting, not neglecting, is a key to evolution and survival (4,5). References 1) Quinn, G.P. & Keough, M.J. (2002) Experimental Design and Data Analysis for Biologists. Cambridge University Press. 2) Bornmann, L. & Daniel, H.D. (2010) The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal. PLoS ONE, 5, e11344. 3) Williams, G.C. (1966) Adaptation and Natural Selection: A Critique of Some Current Evolutionary Thought. Princeton University Press. 4) Rappaport, R.A. (1971) The Sacred in Human Evolution. Annual Review in Ecology and Systematics, 4, 23-44

Submitted on Mon, 11/04/2013 - 18:50

Great article, very important, it puts the cats among the pigeons. I have been confronted myself, repeatedly, with such untrustworthy magazines, inviting an open-access paper, after I had published an article in a respectable subscription journal. The invitation sounded:

Dear Dr. Peter Prudon, We would like to invite you to submit original research manuscripts or recommend papers to be published in Psychology (PSYCH, ISSN: 2152-7199), which is a peer-reviewed open access journal and has been indexed by 57 databases so far. PSYCH has published 474 papers in 35 issues since its initial issue in November 2009. Currently the downloads of the PSYCH papers have exceeded 335,000, which means over 707 downloads/paper. More relevant areas of this journal can be found at the following website: We cordially invite you to submit your manuscript to this journal via the Online Submission System. About Our Journal ■ Full peer review: All manuscripts submitted to our journals undergo peer review. ■ Fast publication: Fast peer review process of papers within approximately one month of submission. ■ Open Access: All Published Articles Are freely Accessible from the Journal Website. ... Best regards, Editor-in-Chief Seth Kunen, Magellan Health Services & Louisiana State University, USA PSYCH Editorial Office

I did a bit of investigation. The Journal appears not to exist. One of the editors is said to be Dr. Rajendra D. Bagaiyan of Harvard University. He does exist, but is from the University at Buffalo. However, they pretend he has an email-address at Harvard: This refers to a website with a picture of the William James Hall. If you click on Psychology (my field) you get the following website:

All this is humbug, meant to tempt low-ranked scientist to spend money on non-existent journals, of which the previous volumes have been filled with stolen articles of many years back. The latter I learned from someone on, posted already in 2009. For this purpose, the good name of Harvard is used.

Submitted on Sun, 11/03/2013 - 06:56

The flaws of the open access publishers might all be true. It is also difficult and time consuming for an author to figure out who complies with high standards and who does not. However, the flaws of traditional journals are at least as troubling and should be considered as well. It is very hard not to think that established journals strongly support established views, in a self-sustaining cycle of conventional wisdom. I have a number of papers that were rejected more than once on the basis that they are "wrong". Eventually they were all published (not without the considerable delays hurting my academic career) and all the criticism that I got under the condition of anonymity never made into the literature and I wonder why. Editors never went to the bottom of the arguments and took the reviewers' judgement as sound, even in the absence of specific arguments. If the conclusions or the methodology of my papers were wrong a fair scientific debate should happen in the scientific journals after the publication (and it did not) and not in the reviewing process.

Carlo Canepa

Submitted on Sat, 11/02/2013 - 09:59

When I first read the article I thought it was a good one confirming what I think many of "us" already knew. What has been less positive is the way in which the media has responded to and reported these findings. I think the scientific publishing process has been dealt a blow and while it has some problems it is very important to getting good science done. As many have stated the "control" in terms of the better journals is not there and while most scientists understand this the media does not. In addition what is left out of the equation is the fact that for scientific results to be really accepted into science they must be repeated and cited. This is part of the process and unless this happens the publication is largely ignored and not incorporated into what we accept to be true. This article thus reports on only part of the scientific publishing process BUT is being used to sink the whole system. I think the baby is being thrown out with the bath water. The real problem is that with electronic media there are too many “fly by night” operations. We do not need to fix the scientific publishing process as such but journals do need to have some accreditation system.

Submitted on Fri, 11/01/2013 - 06:58

WIKIPEDIA QUOTE " In September 2013, he [Dr. Bohannon] submitted a fake scientific article to every fee-charging open-access publisher, revealing that less than 40% were living up to their promise of rigorously peer-reviewing what is published.This approach was criticized by many commentators, who complained that his sting only targeted one type of open-access journal and no subscription-based journals, damaging the reputation of the open access movement" UNQUOTE. The above information put everyone in dilemma is it (this article of Dr J Bohannon) damaging open access or open access has damaged the print publication. In recent past one of the fantastic publishers which published on high quality glossy sheets freely, with par excellent review processes, ceased (The Global Science Books Based initially in UK and Japan and fully shifted to Japan in later stages). It was open access journal which might be reason of its stopage in mid 2013 and they are trying to keep all journal to run in open access format. One of the reason of flourishing fraud open access journals might be impact factor which Thomson Reuters (a PVT company) are giving to journals rather than papers. It is evident that journal does not matter so much as it is paper which makes journals qulity. However it is reverse all over as the IMPACT FACTOR of journaals are evaluating papers quality?IMPACT FACTOR has put whole world in competitive zone un-necessarily and new IF companies are evolving. The time may come that every country offially access the publication factor if bloom of fraud in publication will arise as cited by Dr. Bohannon

Submitted on Mon, 10/28/2013 - 10:05

I wonder... Did any of them publish your paper anyway, after your retraction?

Submitted on Sun, 10/27/2013 - 14:42

Nowadays submitting a manuscript is like gambling. Even high impact factor journals do not really take the peer review process really seriously. the reviewers might just take a look at the manuscript for 2 or 3 minutes and then make a decision. the thing is that now matter how a bad job they do, there are almost no ways to complain. If reviewing a manuscript can have some academic credit, that would encourage scientist to spent more times on reviewing process. Also I do not know how a journal would rate the work performance of the editor. It seems to me there might not be a good way to evaluate their performance.

Submitted on Sun, 10/27/2013 - 11:15