Regina Nuzzo, ASA Senior Advisor for Statistics Communication and Media Innovation
It is fall in a presidential election year, which means journalists across the nation are once again wishing they had paid more attention in their statistics courses. What is margin of error again? What does 95 percent confidence mean? Why, exactly, do we need a random sample?
Many reporters enjoy “geeking out” on the technicalities of polling, of course, and some can debate the drawbacks of nonprobability sampling with the best of them. But there is a healthy portion of journalists who only need to report on public opinion research once every four years—and, boy, do they have a lot of questions.
That is why the ASA recently teamed up with SciLine, a philanthropically-funded free service for journalists that is housed at the American Association for the Advancement of Science (AAAS). SciLine helps put reporters on deadline in touch with relevant science experts, and it produces media briefings, fact sheets, and multi-day boot camps on various scientific topics. Until recently, however, it didn’t have resources on statistical survey methods and public opinion research.
ASA staff members worked with SciLine’s director, Rick Weiss—a former Washington Post science journalist—and his staff to create a fact sheet on polling that would be accessible to a wide audience. They also created an online media briefing on polling, co-sponsored by the ASA, which was held live on June 17 for a large number of registered journalists and is now archived for general access on SciLine’s website.
The stars of the briefing were the panelists, three ASA members who gave brief presentations and then answered a barrage of questions from reporters in the audience: Trent Buskirk, Novak Family Distinguished Professor of Data Science and chair of the applied statistics and operations research department at Bowling Green State University; Courtney Kennedy, director of survey research at Pew Research Center; and Gary Langer, president and founder of Langer Research Associates in New York.
Kennedy kicked off the briefing with a bird’s-eye view of the general polling landscape. She explained the differences between live telephone and recorded telephone polling, the differences between online probability-based panels and online opt-in polls, and why weighting poll results is so important.
“I thought it was very successful. The journalist turnout was very high for this type of event. The questions were great,” Kennedy said. “In my role at Pew Research Center, I do a good deal of media/journalist education-type events,” she continued. “I find that Rick’s SciLine events are the best organized and most effective events of that type. I’m always impressed at how many attendees they get from major national and state/local outlets. Polling is a very complicated topic these days. Journalists are not trained in today’s polling practices, and so there is a knowledge gap that SciLine is filling.”
Next up was Langer, who explained in detail why using an opt-in sample rather that a probability-based sample of respondents can be so problematic. He also illustrated principles of good question wording by showing a misleading, “triple-barreled” poll question.
“After more than 30 years as a news reporter and pollster alike, I know the pressures and demands of both fields and their importance in the public discourse,” Langer said. “Rigorous, professional public opinion surveys add unique and invaluable insight into our social and political condition. Poorly done polls take us in another direction—to misinformation, even outright disinformation,” he continued. “It’s essential for journalists to recognize the difference. I’m always grateful for the opportunity to talk with journalists and other data consumers about these issues; the stakes are too high to let it slide. We owe it to ourselves, our professions and—above all—our audiences to bear down and get this right.”
Buskirk wrapped up with advice for reporters who have to ask hard questions. He discussed potential pitfalls in question wording, target audience, survey methods, weighting and models, uncertainty quantification, and poll aggregation.
“The panel with SciLine was a great opportunity for survey researchers and statisticians to be in the same place with members of the media,” Buskirk said. “Not all numbers are equal, and not all polls are the same. It takes effort to look deeper into the methods and math behind the generation of these statistics. It was so cool that members of the media wanted to learn how to delve deeper.”
The media are already seeking out Buskirk as a source for expert information. “Since that briefing, I’ve had the privilege of serving on a panel of survey experts, including Rob Santos [vice president of the Urban Institute and ASA president-elect] and Scott Keeter [of Pew Research Center] on the show The Source from Texas Public Radio,” said Buskirk.
Reporters followed up with questions for more than a half hour, covering topics such as what went wrong with the 2016 polling, the quality of text-based survey methodology, oversampling specific populations, and judging the value of single polls versus survey aggregates.
The ASA is looking to team up with SciLine on more topics. If you have an idea you think would make a terrific fact sheet or media briefing, send an email to Regina Nuzzo.
Leave a Reply