Michelle Schwalbe, Director of the Committee on Applied and Theoretical Statistics of the National Academies of Sciences, Engineering, and Medicine, and Claire Ji Former Mirzayan, Fellow at the National Academies of Sciences, Engineering, and Medicine

The reproducibility of scientific research is a growing concern among the scientific community. A number of high-profile articles in scientific journals and the popular press have called attention to the issue, noting that difficulties in reproducing scientific results waste resources, slow scientific progress, and could erode the public’s trust of science.

Among the many factors that contribute to poor rates of reproducibility, several are of particular interest to the statistical community, including insufficient training in experimental design, poor data management and analysis, and inappropriate understanding and interpretation of statistical concepts. The impact of these factors varies among scientific disciplines, because each field has a different research culture and a unique set of needs.

A February 2015 workshop—organized by the Committee on Applied and Theoretical Statistics (CATS) of the National Academies of Sciences, Engineering, and Medicine and sponsored by the National Science Foundation (NSF)—focused on three reproducibility issues from a statistical perspective: the extent of reproducibility; the causes of reproducibility failures; and the potential remedies for these failures.

To address these issues, workshop discussions considered, among other items, the definitions of reproducibility and associated terms, how to improve our understanding of scientific discovery through pursuing reproducibility, re-evaluating the threshold for statistical significance, enhancing and clarifying protocols, facilitating the sharing of statistical tools, and enhancing education and training.

In addition to probing the statistical aspects of reproducibility, the workshop also brought to light a broad array of efforts to address the overall problem. In particular, the computing research community, data-sharing clouds, and open-source software packages have been developed in an attempt to improve reproducibility and sharing. Companies and organizations such as the Science Exchange for biomedical sciences and the Center for Open Sciences for social sciences are establishing mechanisms to improve reproducibility by starting peer review of study designs and methodologies as early as possible in the research process, even before data are collected. Funding agencies such as the National Institutes of Health and NSF are piloting new ways of doing grant review to promote transparency, and they are considering funding opportunities for replication studies.

A summary report from the 2015 workshop, Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results, is available for free download. Visit the workshop webpage for videos and other materials.

For more information about this workshop and other CATS activities, contact Michelle Schwalbe at mschwalbe@nas.edu.