Recently, I heard an interesting public radio program on Science Friday called “If Science Takes a Wrong Turn, Who Rights It?”. Host Ira Flatow interviewed Ivan Oransky, an executive editor for Reuters Health who contributes to the blog Retraction Watch, and R. Grant Steen, president of Medical Communications Consultants and a former professor of psychiatry at The University of North Carolina at Chapel Hill.
The question addressed was whether science is as good in correcting itself as we would like it to be. The first criterion for reliability is that the result is published in a journal; the vast majority of published articles are reliable. But, science is evolving and should be self-correcting.
There are more journals today than ever before. Thus, there is a larger body of scientific literature, but the same number of reviewers, so many errors are not caught during the review process. Science and Nature, the most prestigious journals in the basic sciences, have the greatest number of retractions. (They also publish more papers than other journals.)
A recent related article in Nature, “Science Publishing: The Trouble with Retractions”, noted a 10-fold increase in the number of retractions over the past decade, while the number of published articles had risen by only 44%.
Retraction notices for published papers covered by the web of science have increased from 0.001% of the total to 0.02%, so one could say the problem is very small indeed. The fact that ‘retraction’ is often associated with fraud may make authors less likely to admit errors, so the number of retractions reported might underestimate the number of erroneous papers.
As best as can be determined, just over half of retractions in the past 10 years are for fraud (and most of these involve plagiarism). Most of the remainder are for honest errors or irreproducible results, although it is difficult to tell because retraction notices are often uninformative. This may be due to fear of suit for libel by publishers or the work it would take to determine the cause of the errors.
One participant in the Science Friday program said he thought no paper was entirely error free, but that most errors were trivial.
Assuming that most statisticians are honest, I wondered how we statisticians fared with regard to corrections. I took a brief (and clearly nonrandom) look at two years of JASA (September 2010 to June 2011 and the same issues for 2001 to 2002). I found four corrections and two letters to the editor in the recent year and four corrections and three letters in the earlier year. (Letters also may point out errors.) The corrections were mainly small errors (several typographical), but in the earlier year, one corrected a wrong theorem and authors replaced a section due to notational problems in another. That appears to be a tiny number of errors, none making newspaper headlines. The statistical literature might be well refereed and self correcting, but a more formal study would be needed to demonstrate that.
Yet we should not be so sanguine as to believe bad errors are not going to happen to us. Many of us are involved in collaborative research in which each author contributes but a small part to the project. Sometimes, results seem fabulous, and it is easy to believe a fabulous result if you are part of the collaboration, but this is a time you should be especially skeptical.
The retractions that make headlines are those involving reports of fabulous discoveries, sometimes with consequences that are far reaching. One example is the microarray-based signatures of drug sensitivity derived from cell lines that were used to predict patient response to cancer drugs. The scientific community found the results impressive: Many years of genetic research had yielded something that appeared to be useful in the clinic. But, the analysis methods were flawed and the results could not be reproduced. Clinical trials were already under way when the errors were detected.
Another example is a paper that reported genetic factors for longevity, but had to be retracted because the centenarian and control subjects were analyzed using different genetic platforms without proper adjustment in the statistical analysis.
The combination of team science and competition—whether for research funding, fame, or an adrenaline surge—may lead us down paths we later regret. One should beware of results that are fabulous, especially if they are one’s own.
Leave a Reply