Panelists Discuss How Statisticians, Evaluators Turn Evidence into Impact

During the 2024 Joint Statistical Meetings, Elizabeth Eisenhauer—lead statistical associate at Westat—organized the following panel of statisticians and evaluators, who convened to share their insights on using evidence to inform programs and policies:

• Ruth B. Etzioni, Professor and Rosalie and Harold Rea Brown Chair, Biostatistics Program, Fred Hutch Cancer Center
• Michael Baiocchi, Associate Professor, Department of Epidemiology and Population Health, Stanford University
• Kristin Linn, Assistant Professor of Biostatistics, University of Pennsylvania
• Nandita Mitra, Professor of Biostatistics and Co-Director of the Center for Causal Inference, University of Pennsylvania
• Debra Rog, Vice President for Social Policy and Economics Research, Westat

Members of the panel reflected on past successes, challenges, and the evolving partnership between statisticians and evaluators, highlighting future opportunities.

Program Evaluation in Action

Linn provided an example of how thoughtful program evaluation can lead to meaningful behavioral change. She described her evaluation of an intervention designed to promote secure firearm storage by improving conversations between clinicians and parents during child wellness visits and offering free gun locks.

The evaluation team compared two groups: the control group, in which clinicians received simple reminders to discuss secure firearm storage with parents, and the intervention group, in which clinicians also received facilitation on how to have these conversations.

The facilitation proved highly effective, significantly increasing the number of safe storage conversations and free gun lock offers.

Shaping Policy Through Evaluation

Etzioni shared her work with the National Cancer Society on breast cancer screening recommendations.

For decades, the minimum recommended age for breast cancer screenings in the US was 50. Their evaluation considered how the population and benefits of screening had shifted over time, leading policymakers to lower the recommended age to 40.

The policy update, Etzioni noted, reflected good practice in policy evaluation: adapting to evolving evidence and context.

The Secret to Collaboration

“A smattering of everything, knowledge of nothing,” a fortune cookie phrase that resonated with Etzioni during her student years, reflected the panelists’ philosophy on collaboration.

Applied statisticians, the panelists emphasized, should not confine themselves to rigid definitions of their work. They were united by a problem-solving ethos, where innovative solutions often emerged from unexpected disciplines.

Baiocchi, for example, described developing algorithms to assist in raids on human trafficking operations in Brazil. However, after discovering a high number of victims returned to the same camps within a year, Baiocchi pivoted to working with social workers to evaluate interventions rooted in cognitive behavioral therapy.

When Collaborators Disagree

In response to a story about disagreement over the ‘correct’ way to analyze data, Etzioni acknowledged that there is rarely a single right answer; every method has its limitations. Sometimes, the most technically rigorous method is not the most practical or trusted by collaborators.

Baiocchi suggested being a good collaborator sometimes means helping your team members understand solutions enough to criticize them. He compared statisticians to therapists—drawing out collaborators’ anxieties and uncovering unobserved variables that are better accounted for sooner than later.

The Role of Context

The panelists shared many examples of how context influences evaluations in unexpected ways. Rog, who has studied homelessness for 40 years, recounted a time she was asked, “Why are you collecting more data on homelessness in this study? Don’t you already know enough about homelessness?” Rog explained that homelessness is different every time she studies it; the context changes from year to year and place to place.

Rog advocated for mixed methods, combining qualitative and quantitative analyses, as essential tools for navigating complexities. Mixed methods studies can be done poorly, but the best ones start out with a clear plan to intentionally integrate the data.

Finding Value in a Negative Result

“Disappointment and opportunity coexist in a negative result,” Etzioni observed, as the panelists reflected on evaluations that didn’t produce the desired outcomes.

Mitra described a study on Philadelphia’s beverage tax, which did not significantly reduce obesity rates. However, it provided valuable insights into the system being studied, underscoring how evaluation can foster learning, even when primary goals aren’t achieved.

Etzioni shared another example: a screening test that met its primary aim but didn’t lead to timely follow-up MRIs or diagnoses due to insurance barriers. She noted implementation rarely happens as planned, underscoring the importance of candid communication between statisticians and on-the-ground experts.

Rog reflected on how the evaluation field has changed over the past 50 years, from a focus on accountability to a focus on learning. She thinks this shift presents broad challenges and rich opportunities for interdisciplinary collaboration.

Next Steps

The panelists outlined ways to bridge the statistics and evaluation fields, including developing shared resources to support mixed-methods evaluations and encouraging training opportunities to build evaluation capacity within the statistical community.

A proposed American Statistical Association/American Evaluation Association working group seeks to connect members of the ASA and AEA to tackle real-world problems more effectively. For inquiries about the working group, contact ASA Associate Executive Director Donna LaLonde.