Science 4 October 2013:
Vol. 342 no. 6154 pp. 60-65
DOI:10.1126/science.342.6154.60
Special Issue News

Who's Afraid of Peer Review?

John Bohannon | 248 Comments

A spoof paper concocted by Science reveals little or no scrutiny at many open-access journals.

Add a new comment

These postings do not necessarily represent the views/opinions of Science.

I believe rushing and cheapening a product will result in a loss of quality. This article is an example of my belief. I feel open-access journals do not have the capability to produce articles of equal value as paid scholarly journals. Why use your valuable time on a lesser product?

Submitted on Sun, 06/21/2015 - 13:48

Study indicates spoof paper is a Reality. Study is interesting and useful.Universities should have the facility to identify and stop the author/ scholar increasing their publication record through such papers.

Submitted on Mon, 09/22/2014 - 03:05

This work is missing the essential control of submission to journals that are NOT open access. Have we not all observed the publication of papers with huge flaws even in the most esteemed non-open-access journals?. Ultimately it is always up to the reader to ensure that the information they really on is reliable. Maybe actually read the paper before you start citing it for having shown this or that?

Submitted on Thu, 08/21/2014 - 17:45

Thank you. Anything that gives credibility to fake science places us all in the dark ages. How much time does the typical university science course spend on ethics? What if ethical practice would be a part of every science final exam beginning in elementary school? What about a scientific audit of ethical theory in university science class syllabi? Since ethics is cultural, could this be an international study?

Are better laws needed? Given that financial fraud can ultimately result in jail time, does science fraud have equivalent legal consequences for perpetrators? Are publishers subject to due diligence liability for damages?

Please continue your work and the discussion of ethics and truth auditing in science.

Submitted on Sat, 05/03/2014 - 08:32

I agree that most open access journals are very bad, and as a researcher I have been very careful to submit papers to this type of journals. However, it would have been very good if this interesting study would have covered traditional journals, because worrying surprises would probably have emerged. I also notice the current practice of several mainframe journals, that have produced a second-grade sister journals, to which they encourage rejected papers to be submitted (and pay the price if they are accepted). I would like to see how these open access journals compares in relation to less reputed ones!!?

Submitted on Sat, 04/12/2014 - 11:11

Fascinating conversation, everyone, thanks!

Quality is, from what we’ve observed, an issue with OA and paywalled journals alike. And we think it’s a brilliant opportunity to show the huge value of peer review when properly done.

We understand that it’s very difficult to catch every potential error or problem pre-publication (although one can always try), which is where post-publication peer review becomes of vital importance: errors and inconsistencies can be picked up and discussed.

In this way, the general scientific/academic canon can be kept as free as possible of poor quality research. And it’s for these reasons that we offer post pre-and post-publication peer review capabilities on our platform: we want as much high-quality peer review information on papers as possible to be available.

And, of course, for peer reviewers to be rewarded for their contribution to the system :)

aimee whitcroft Community wrangler Publons publons.com

Submitted on Sun, 03/30/2014 - 21:39

You did not need to bother. The experiment has already been done. I hope Science will now report that the issue is not exclusive to open access journals. See http://www.slate.com/blogs/future_tense/2014/02/27/how_nonsense_papers_e...

Submitted on Fri, 02/28/2014 - 04:51

One of the most annoying issues with these predatory journals are the amount of submission requests and review requests they send out. They are randomly assigned and I receive mails for anything from dentistry to chemistry, but never in my field of expertise.

Submitted on Thu, 02/20/2014 - 15:29

I sympathize with Roos's view, in that the author should have included subscription-based journals in this "sting" as a control group.

As to the defensive statements about "trust": there is a certain element of trust involved, at least regarding accurate reporting of data. This does *not* excuse an editor from the responsibility of ensuring that the conclusions formed are actually *supported* by the data that is reported, and that the data is obtained by sound methodology.

Submitted on Wed, 01/29/2014 - 22:03

I came to this article from another link... I just would like to add a word: Bohannon seems to be 'excited and happy' about his deficient 'scoop'. It would be interesting that Bohannon, with his 'high skills and intelligence' writes another good bogus paper with his 'excellent writing style' and submit it to Science itself. I am sure that his paper will be accepted immediately. You can also put your name as the fake supervisor and put any other name as a first author, your fake paper will still be acceptable by Science and its sisters... This is a paper full of conflict of interests against open access journals.. One day, not so far undoubtedly, Science will also become an open access journal... What will you say, John? We will see...

Submitted on Sat, 01/25/2014 - 18:00

Bohannon has identified the problems of scientific publishing, which are well known facts.Scientists and researchers are aware of many of these journals and their these shoddy practices publishing. The only thing Bohannon has done is to put pieces of jigsaw puzzle put together through this "sting" and through a systematic approach as is done for scientific experiments. Though Bohannon has identified the problems, i wonder why he has not attempted to provide concrete solutions. It is easy to find fault and problems, but without solid, reliable solutions problems do not wean away on their own. I would have appreciated Bohannon's efforts even more, had he attempted to provide solutions. It is a common sense to refrain from publishing in those journals that seek exorbitant money for publishing an article, in the name of open access. What is required is solutions beyond that. The other issues is of "ethical standards" which are always preached and talked about in scientific research. Communicating a manuscript with a fake name, i feel, is an "unethical way" of conducting research. This could open way for many more such enthusiast like Bohannon to communicate "similar fake articles" to various journals only to withdraw manuscripts later, thereby wasting time of reviewers and others involved in the publication process. To conduct such research, Bohannon should have communicated manuscript in his own name and not a fake "African name".

Submitted on Fri, 01/03/2014 - 14:55

This spoof experiment is valuable in revealing serious problems with open access. But it is an unethical experiment on human subjects with no informed consent. I can accept such experimentation on humans in extremely limited circumstances when there is serious danger to life. But this is different. Best wishes to all, Frank Leavitt (Yeruham)

Submitted on Fri, 12/27/2013 - 22:29

The spoof paper of John Bohannon is topped by ´Near-Death-Experience´(NDE)-research Since 1975, when ´Life after Life´was published, a lot of research on NDEs and dying experiences was done: But the term Near-Death makes no sense and should therefore never be used in a scientific paper - how close (distance / time) can the death be to a person: is the death an object?. And a person who had a dying experience has to be a corpse after this experience. Important studies to NDEs are Prof. Fischer´s ´Immortality Project´(worth $5M) and Sam Parnia´s ´The AWARE Study´. NDE-research can be found on countless universities all over the world

Submitted on Sun, 12/22/2013 - 11:14

To defend not really top science is not an easy matter. Yet, let us remember an important principle. The “Principle of charity” in published texts is as follows: When criticizing the text we should criticize only what is explicitly said or written and NOT what may be implied or not explicitly affirmed. Bite (fake articles) was very short paper and (this is my opinion) may be acceptable in principle as a «brief communication». The main drawback of the bait articles was that the introduction and discussion were very short and extremely modest. Some possible answers to the alleged “flows” could be as follows. The figure 1 does not really show dose – response? This is true. However this could be explained by the fact that the agent was probably very specific and very powerful so that it already had its maximal effect by the first concentration tested and then the effect remained constant. Is there an artefact of ethanol? No, the concentrations of ethanol were too small to have an effect (0.66% ethanol, just marginally too high). Figure 2 claims that there is an increase of sensitivity to radiation, although the controls were not irradiated? The response does not start from 100% but much lower and at that portion there is obviously an increased sensitivity to the agent. The control bars seem to be superfluous in both graphs.

Other alleged “flows” I rejected in my earlier comments. Therefore the claim of Bohannon that some OA journals may be publishing bad science is not warranted. I doubt that the author of the bait articles, not being pharmacologist, was aware of the fact that his articles were not so seriously flown. “Science” accepted his report, which is far less scientific (I this explained earlier 11.9. and 11.11. 2013). The whole enterprise of “Science” to expose some predatory journals was therefore in vain. And the method used, unfortunately, immoral.

Submitted on Wed, 12/04/2013 - 08:16

I am afraid I did not say what dr. Romario Melo seems to agree with. I have expressed my own opinion in my previous comments. Anyone is interested can read them.

Submitted on Tue, 11/26/2013 - 03:44

Do you have a spam problem on this blog; I also am a blogger, and I was wanting to know your situation; many of us have developed some nice methods and we are looking to exchange methods with other folks, why not shoot me an e-mail if interested.

Submitted on Mon, 01/05/2015 - 04:10

The academia can withstand only so much pressure before it becomes a faithful reflection of the society at large. The prevailing narrative is money. Schools continue to build new facilities like nothing happened. The administration is bloated as never before and has become grotesque. It is also grossly overpaid. Meanwhile the NIH policies promote degradation of all the morals in the scientific community. Their funding policies promote superficial research where PIs keep writing modular ($250000 grants) that cannot support a research projects forcing PIs to write myriad of those and to conduct research in as many areas as possible to support these applications (no overlap, mind you). The grant reviewers are corrupted with little favors that NIH bestows on them for participating in the dysfunctional review process. Publishing a small number of in depth research papers is not profitable any longer. The number of papers its all that matters with only lip service praising the quality these days.

Submitted on Mon, 11/25/2013 - 12:49

John Bohannon’s Science article – “Who’s Afraid of Peer Review?” – explores an important issue. The response has been impressive. Even so, some miss the key point.

Bohannon does not challenge open access, and he did not criticize all open access publishing. He looked at “Open Access Gold,” where authors or their institutions pay publication fees. The possibility of a gold rush is the cause of a Wild West mentality.

Determining the degree of similar problems in paper journals would require a similar study. This was not Bohannon’s task. We have all seen stupidity among reviewers and authors at paper journals. What we do not see in subscription journals is the cash incentive to publish utter rubbish. Paper, printing, and delivery are costly. In the absence of a publication fee, there is no incentive to pan for gold. The incentives go to building impact and attracting subscribers.

The study is entirely useful in an environment with an increasing number of unethical publishers. Those caught out may not like it, but there is no harm to human subjects in catching them.

This is an important contribution to the contemporary debate on scholarly communication. This sting focused on what I describe as “the OA Gold Rush.” The debate is timely and welcome.

Ken Friedman

University Distinguished Professor, Swinburne University of Technology, Melbourne, Australia

Guest Professor, College of Design and Innovation, Tongji University, Shanghai, China

Submitted on Tue, 11/19/2013 - 09:53

Dear Bohannon, I guess you got my comment wrongly. I know GMR and how "sh.." they are, as do so everyone who submit their papers to it, however with the "publish or perish"+pressure scientists find better to have something published (even better when it's easy and without many restrictions like in GMR) than nothing. This was my point when talking about the comunity's support for the scheme. On the other hand, my criticism was more on very basic statements from any small and simple scientific journal: "the data should present new data/insights..." yours does not; "figures should be self-explanatory..." yours is not, unless you intend the readers "to study" the figure and data for several minutes in an attempt to find a country-based pattern. I think I agree with Mrs. Raimo's post in here, who should care about geography and statistics when talking about geographical locations, numbers and percentages, right? After all we are in a new era of science in which Science AAAS is accepting papers without "control treatment" and figures not self-explanatory... not even talking about the many retractions we have seen from authors in Science AAAS in the past years!

Submitted on Mon, 11/18/2013 - 03:48

Meta-analysis is a powerful tool used in diverse disciplines to quantitatively synthesize findings across studies that address a similar question. Data from the peer-reviewed literature dominate most meta-analyses. Thus, open-access publishing has the potential to transform the field of meta-analysis through the massive influx of readily-accessible articles. However, Bohannon (2013) documents a major flaw with the rise in new, open-access journals – namely, the lack of credible, unbiased manuscript peer-review. Study quality is rarely considered in meta-analysis, except when experimental error is considered (i.e., weighted meta-analysis). Instead, meta-analysts rely on an objective, peer-review process to adequately vet most of their data sources. If Bohannan’s findings are indicative of open-access publishing, in general, I expect the utility of meta-analysis to wane in the future as meta-analysts are forced to choose which studies to consider for their analyses thus further biasing their syntheses. And, all-inclusive meta-analyses that include data from journals with suspicious review processes will not be deemed reliable. Given that meta-analysis is, in part, used to develop protocols and treatments that protect and save human lives, there are obvious serious repercussions to a flawed peer-reviewed system that includes open-access and traditional publications. The scientific community needs to develop a plan for dealing with dubious scientific publishers before meta-analysis is lost.

Submitted on Sat, 11/16/2013 - 08:28

Thank you, Bohannon, for your article showing facts. From them one can learn much more than from lessons of Statistics or Geography.

Submitted on Thu, 11/14/2013 - 04:45

I every time spent my half an hour to read this website's articles daily along with a mug of coffee.

Submitted on Sun, 01/04/2015 - 11:52

Bohannon’s article is the perfect example of its own criticism. To have an article making those claims without statistics, proper controls, no references, being publish in Science (even if it is on the news section) just illustrates how badly we need proper scientific training not only among reviewers, bad desperately among our peers. Dear John, if you wanna talk the talk, you gotta walk the walk.

Submitted on Tue, 11/12/2013 - 10:26

I found the idea and outcome very interesting, however unfortunately that much work does not show anything very new to anyone, with exception of those who play blind in the nowadays scientific world. After the coining of "publish or perish", obviously in a capitalist world lots of "smart" people would take advantage on that and the issue in evidence here has happened more often ever since. We should not forget that this has been supported by many scientists, who very often heavily criticize competitors within the same field of research for approaching a specific method and even claim they are "faking data", but on the other hand do "fancy" analysis and "pull off" extreme data from their experiment to publish cleaner data without regret. What I find even more interesting is someone who conducts this kind of assay publishes a map with an absolute wrong geographical location of a City. AND THIS HERE IS PUBLISHED IN SCIENCE, isn't it??? Did Science Editors have checked the "fancy" submitted Map? I doubt it! Please check the location for Genetics and Molecular Research journal, and then type the informed location on google maps.

Submitted on Sat, 11/09/2013 - 17:28

I encourage you to read the emails that led to the acceptance of the fake paper by Genetics and Molecular Research:

http://scicomm.scimagdev.org/data/journals/329/

Judge for yourself if you think that was a provisional acceptance. All of the emails from every submission are public record:

http://www.sciencemag.org/content/342/6154/60/suppl/DC1

As for the map, the points show country location, not city location.

Submitted on Mon, 11/11/2013 - 10:43

In addition, and this is the greatest problem, the act of producing and sending over 300 fraudulent articles to the scientific journals is a moral disaster. It would be certainly impossible not to condemn, on deontological grounds, this act or to try to defend it by employing utilitarian arguments, by using the principle of double effect or some other similar justification. Simply because the effects of this fable observational report will certainly be null. And, nevertheless, it was long known that number of journals had problematic reviewing procedure and other morally acceptable methods could have been employed to warn such journals.

Submitted on Sat, 11/09/2013 - 07:37

Bohannon’s article is itself a perfect example of “bad science” as he himself defined it. Therefore NOTHING can be concluded from his study (declared as “news” by Science (!?). The serious flaws of Bohannon’s bite*, were: bad controls for the experimental groups, wrong legend for one graph, non idiomatic English, and the conclusion contained unjustified, and therefore “fallacious” recommendations. Astonishingly, if we would examine the very Bihannon’s article published in Science, what we find is that it has itself similar weaknesses i.e.: there is no randomization of his “experimental group”, there are no controls at all (he could have compared the OA journals to the ordinary big journals), there was elimination of non –responders and there was no application of the intention to treat principle in the analysis. Then, even worse: there was no inferential statistics and no references. Finally, Bohannan maintains, about the conclusion in his “bite” articles, “to be bad”. This is simply not true: it states in reality that the studies that had to be conducted, should be done “in animal and human”, which suggests only implicitly though clinical studies! The bite article of Bohannan was withdrawn from OA journals (victim), so unfortunately we do not know what type of the article it would have been if published (pilot study, short report, special report, news or else). In discussions that followed its publication in Science - it IS interpreted not as a simple interesting “news” but as a scientific study. This is indirectly suggested also in the subtitle. What follows from Bohannon’s text is that the journal where Bohannon’s text itself was published (Science) belongs also to the “victim” group of journals characterized by Bohannon as publishing weak science. Science was sufficiently clever to have published Bohannons article as “news”, which apparently liberated it from the obligation to apply scientific method. The scientific method exists to try to establish what is an what is not, and even when it is used correctly, the conclusions are uncertain. This text contradicts the very principle of science. This is how so famous journal as Science is, in fact failed badly and shot itself in the foot!

Submitted on Sat, 11/09/2013 - 02:40

There is certainly a lot of things that are not pleasant about open access. But one can write very bad stories about the traditional-journal peer review as well. Perhaps this even the reason why the open access started in the first place? Because people are fed up with the outright dishonesty and violence that reigns in these journals. Peer review in traditional journals is extremely biased by people who have managed to get influence. They would eliminate a competitor with dishonest reviews. Instead of improving the quality of a paper, they eliminate good papers. I sometimes spotted errors in my own papers after submission, but these were never discovered in the review. However, I have received many times reviews that made a lot of indignated fuss phantom errors that were just not there. Just because people were jealous and wanted to push their personal agenda by taking advantage of the secrecy of the system. Being a referee is a godsend to get rid of the guy who is better than you. And the fraud is often set up with highly sophisticated psychological tricks. These dishonest referees have the warrant of strict impunity because their name remains secret. As long as there are not dishonest referees convicted in Court, with very incisive sentences, traditional journals should just shut up complaining about what happens elsewhere. Because the Schoen affaire is only the tip of an iceberg. Just google "peer review in medicine in the US". There also a system of anonymous peer review exists. And it is abused by unscrupulous people to eliminate doctors who are more competent than they are, such that they fear to loose patients. Just having an argument about a place in a parking lot can be enough to tick somebody off. As a consequence of this mobbing an estimated 300 physicians commit suicide per year in the US. Great is it not? Also there, nobody wants to listen. If the traditional journals could start sweeping in front of their own doors and install the possibility to legally inquire into the ethics of the reviewers? And what about editorial boards who take decisions about papers without sending them out for review? Has there been a declaration of financial interests in the publication of the study?

Submitted on Fri, 11/08/2013 - 07:45

The story of Ocorrafoo Cobange, a supposed biologist at the mysterious Wassee Institute of Medicine in Asmara (‘Who’s afraid of Peer Review?’, Science, 4 October 2013, vol. 342, page 60), is one of numerous examples of scandals in peer review. The case raises two problems that publishers, journals, and the scientific community continue to struggle with. First, the internal evaluation processes of academic journals remain hidden from scrutiny. It is hard to improve something that is not observable. Allowing more systematic analyses of peer review is an essential first step to improving journals’ processes. Funding agencies, publishers, and influential journals should be establishing new initiatives to promote data sharing and analysis.

Secondly, peer review relies on voluntary work by referees. While financial rewards may be counter-productive (merely yielding a ‘crowding out’ of intrinsic motivations), exploiting reputational motives, which are the main driving force of scientists’ behavior, could be really helpful. We should be exploring incentives such as appointing excellent referees to membership of journal boards, ensuring that reviewing counts for promotion, and publishing quality assessments of referees’ reviews. Implementing changes such as these would make the existence of the next Ocorrafoo Cobange at least more problematic.

Flaminio Squazzoni Department of Economics and Management, University of Brescia, Via San Faustino 74/B, 25122 Brescia, Italy. Email: squazzon@eco.unibs.it

Nigel Gilbert Centre for Research in Social Simulation, University of Surrey, Guilford, Surrey, GU2 7XH, United Kingdom. Email: n.gilbert@surrey.ac.uk

Submitted on Tue, 11/05/2013 - 17:35

Pages