A Practicing Statistician’s Plea

Gang (John) Xie, Office of Research Services and Graduate Studies, Charles Sturt University, Australia

I am writing to express my deep concerns about the current practice of statistical analysis in scientific research. As a practicing statistician, I have observed a troubling disconnect between the principles of mathematical statistics theory and the application of these principles in real-world data analysis. In essence, many statisticians are not practicing what they preach.

Random sampling from a well-defined target population and randomization in experimental studies are two of the most fundamental principles of statistics theory for any statistical inferential analysis. Random sampling is crucial for ensuring the external validity of statistical results, while randomization in experimental design safeguards internal validity by minimizing potential confounding effects, whether known or unknown. The process begins with a necessary random sample and continues with many more random samples until a reasonable sampling distribution is constructed, which is sufficient for achieving external validity. It is through the sampling distribution of sample statistics that we approach the core objective of scientific inference: distinguishing between scientific truth and falsity.

However, technical limitations—such as research populations rarely being finite and unchanging, ethnical constraint preventing randomization, nonrandom samples, missing values, violations of model assumptions (including but not limited to independence, equal variance, and normality)—any of these factors compromise the reliability and validity of statistical inference. These issues are rarely acknowledged or adequately addressed in practice, often deliberately ignored.

In the ASA President’s Task Force Statement on Statistical Significance and Replicability the authors write, “P-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data.” However, they do not clarify what exactly constitutes the proper application and interpretation of these tools. I propose that, in addition to meeting any assumption conditions for a specific statistical model, statistical inference should only be considered properly applied when the following three conditions are met:

1. Random sampling is used to obtain the sample for inference. 

2. Experimental units are randomized with respect to the treatment conditions of interest. 

3. The study is repeated multiple times to adequately establish the sampling distribution. 

In reality, it is rare to find cases where these criteria are fully satisfied, and the level of deviations is often unknown. This leads to the conclusion that most statistical inferences in real-world research are invalid either internally, externally, or—frequently—both.

On the other hand, a 2019 editorial in The American Statistician, “Moving to a World Beyond ‘p < 0.05,’” makes a specific and necessary call to end the use of the pseudo-scientific concept of ‘statistical significance’ in statistical analysis practice. This call represents the bare minimum required to ensure good practice in statistical analysis. Therefore, I sincerely urge professional institutions such as the American Statistical Association and Royal Statistical Society to provide clear, operational, and theoretically defensible guidelines for statistical practice.

The true role of statistical analysis lies in statistically describing or characterizing quantitative evidence regarding scientific research findings and offering what-if scenarios through logically consistent statistical modeling. The validation or justification of scientific research findings, however, is a task for the scientific method itself, not achievable through statistical inference. For example, no matter how many survey studies are conducted or how statistically rigorous (or flawed) these studies are, statistical inference alone cannot prove that certain chemicals in tobacco cause lung cell damage, thereby establishing causation between smoking and lung cancer. I therefore strongly resonate with the perspectives of R. Hubbard et al. and C. Tong, who argue in The American Statistician [articles “The Limited Role of Formal Statistical Inference in Scientific Inference” and “Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science,” respectively,] that statistical inference has a limited role in the broader process of scientific inference. As Tong aptly put it, “Statistical inference enables bad science; statistical thinking enables good science.” This distinction highlights the need for a deeper understanding of the role and limitations of statistical tools and the importance of integrating them into a rigorous scientific framework.

As statisticians, we must collectively acknowledge that most current statistical practices do not align with the principles we teach in mathematical statistics. A hard truth that we must accept is statistical inference is built upon layers of assumptions. Yet [according to MD Higgs in her blog post “Assumptions Are Not Met. Period.], the reality is that assumptions are not met. It is time to realign our practice with our theory, ensuring that our analyses are both scientifically sound and practically applicable.