David Morganstein

Two topics this month related to elements of the ASA’s mission: promote the proper application of statistics and anticipate and meet member needs.

First, new editors to a social psychology journal established a policy that the journal will not accept articles that include p-values or hypothesis tests. While we may have different viewpoints about various statistical tools, do the editors’ actions address underlying issues about misuse of methods? There is an altogether different approach that ASA is taking to air those differences.

Second, several state legislatures recently passed laws that included the phrase “religious freedom” in their title. The legislation raised concerns regarding unlawful discrimination. These legislative actions led the ASA to issue a press release opposing legislation that would allow an organization to unlawfully discriminate and reaffirming its meeting conduct policy.

P-values

The widely used p-value has come under fire recently in several publications. The new editors of the Journal of Basic and Applied Social Psychology imposed the following policy: Authors will have to remove all vestiges of the null hypothesis significance testing procedure (p-values, t-values, F-values, statements about “significant” differences or lack thereof, and so on). Shortly thereafter, in Science News, this statement appeared: “P-values are at the root of all (well, most) scientific evil.”

Our statistical tent is big enough to include differing viewpoints on the proper analysis and interpretation of data to support making better decisions. An array of perspectives about the use of the p-value and the editors’ decision are discussed by colleagues in the Royal Statistical Society and the International Society for Bayesian Analysis.

Likely there is general agreement that practical significance and economic and other important consequences should accompany any decision based on sample data, regardless of a p-value. In the era of Big Data, there is wide understanding that very large samples with consequent small standard errors result in small p-values. Less widely appreciated may be the role of nonsampling errors that often dominate when large samples are taken. Banning improper usage of statistical methods from publication is sound; however, it is not sound to exclude papers that reference a widely held approach simply because an editor does not value it.

Hopefully, there is consensus that any tool can be misapplied. However, it is unwise to make the leap from “a valuable tool can be misused” to “never use it at all.” Many of us, on reading these critical comments about the p-value and hypothesis testing, concluded that the action of banning a paper that employs them fits the description of “throwing the baby out with the bath water.” Nor is a p-value in and of itself the root of any evil, rhetorical hyperbole aside. If we agree these statements are an overreaction, what is a measured, appropriate, and thoughtful reaction?

Before addressing that question, let me mention a few other common situations many of us encounter in which our statistical tools can be misused. Powerful statistical software has become readily available. Knowledge of how to use its vast capabilities correctly may be less common. It’s easy to analyze data in inappropriate ways and, of course, reach indefensible conclusions. That does not speak to the tool; rather, it is commentary on the knowledge and experience of the user.

Differences in means in “before” and “after” measures on a set of patients could be analyzed (inappropriately) as independent samples where truly significant differences might be overlooked due to a failure to incorporate the power of correlated data. Estimating differences obtained by combining data from two surveys conducted in the same places (counties, schools, homes, etc.) while failing to reflect the likely correlations in the two sets of estimates will lead to improper inferences in the same way the first example does. Estimating means and totals from data collected via a stratified, clustered, unequal probability design, using software built on the assumption of simple random samples, produces incorrect standard errors and improper inferences. Using a normal assumption on proportions near zero or one with small enough samples can yield confidence intervals that include values below zero or greater than one. As a last example, analyzing a case-control study as if it were a cohort study leads to improper conclusions (as was pointed out regarding Brian Hooker’s paper on vaccinations and autism, a paper later retracted by Translational Neurodegeneration, the journal that imprudently published it).

Let us not conclude from these examples of misusing statistical methods that the tools themselves are flawed and, because they can be misused, they should be excluded from publication. Instead, we would suggest that better education of practitioners is a much-preferred solution. Education is a significant component of ASA programs: short course offerings, support for teachers in public schools, and guidance for undergraduate and graduate programs.

(As an aside, these examples offer an important rationale behind this year’s presidential initiative to create a modern, interesting, and sound Stat 101 course for nonstatisticians. We’ll be hearing from Dick DeVeaux and his committee later this year. This speaks to the next point.)

What, if anything, is the responsibility of the ASA when statements like the above appear online and in publications? On the ASA home page, we express the concern that this psychology journal’s policy may have its own negative consequences while recognizing there needs to be open discussion in the larger research community. We also note, “A group of more than two-dozen distinguished statistical professionals is developing an ASA statement on p-values and inference that highlights the issues and competing viewpoints. The ASA encourages the editors of this journal and others who might share their concerns to consider what is offered in the ASA statement to appear later this year and not discard the proper and appropriate use of statistical inference.” Our executive director, Ron Wasserstein, organized this group to initiate just this discussion. We look forward to the results of their efforts, perhaps before this summer’s Joint Statistical Meetings.

As statisticians, do we have a role to play here? We do have an ethical obligation to “avoid condoning or appearing to condone careless, incompetent, or unethical practices in statistical studies conducted in your working environment or elsewhere” (from ASA’s Ethical Guidelines for Statistical Practice). The International Statistical Institute’s Declaration on Professional Ethics has pursuing objectivity as its first principle: “The statistician should guard against predictable misinterpretation or misuse. If such misinterpretation or misuse occurs, steps should be taken to inform potential users.”

Yes, we do have a role: to avoid condoning careless practices and inform potential users.

Rejecting a paper whose authors use inference improperly, blindly considering .05 sacrosanct, improves the quality of published research. Excluding papers solely because they contain hypotheses tests or p-values will not.

Meeting Conduct Policy

On March 30, the Indiana legislature passed a law titled the Religious Freedom Restoration Act. While the full impact of that law remained to be defined in subsequent court decisions, one interpretation of the act suggested it could be used by businesses to refuse service to a potential customer based on that customer’s gender identity or sexual orientation. Immediately after Indiana passed this legislation, Arkansas lawmakers passed a similar “religious freedom” bill.

To reaffirm the ASA’s Meeting Conduct Policy—that all participants in ASA activities will enjoy a welcoming environment free from unlawful discrimination, harassment, and retaliation—the following statement was issued:

Subsequent to Indiana’s governor signing the legislation, the legislature revised the wording to clarify that the law could not be used for unlawful discrimination. Similar clarifications were included in the bill that finally passed in Arkansas.

In light of these legislative actions, it is appropriate that the ASA state clearly it remains committed to its Meeting Conduct Policy.