The Washington Statistical Society and George Mason University held a joint seminar on election polls and forecasting on October 11. The seminar featured Andrew Gelman, professor of statistics and political science at Columbia University, who was interviewed by CNN Data, Polling, and Election Analytics Senior Producer Edward Wu. The seminar was organized by Jonathan Auerbach, Katherine Irimata, and Ben Lee.
Here is the interview followed by a summary of Gelman’s talk.
Ed Wu: Thanks for taking the time to chat with us. There are many data sources available to predict the outcome of the 2024 presidential election. For example, there is polling, fundraising, early voting, economic indicators, and past election results. Which data sources are most important?
Andrew Gelman: We primarily use state and national polls, as well as so-called fundamentals like economic indicators. We don’t use information from fundraising, early voting, media coverage, or the position taken by candidates. We also don’t use “derived products” such as the results of other forecasts or betting markets.
Ed Wu: So, you don’t use information like voter enthusiasm directly in your model. Do you consider this information useful for interpreting the results of your model?
Andrew Gelman: There is a lot of information that could be included in a model. For example, in previous models, we have considered including polling errors from previous years. That is, we considered accounting for nonsampling—systematic errors in the polls that didn’t go away as the election approached.
Voter enthusiasm is tricky. Enthusiastic voters are more likely to participate in polls, which can lead to over-adjustment.
Intermediate information such as early voting and campaign spending will surely be used to tell stories after the election, especially if the result is unexpected. But we don’t always have great measures of these things while they’re happening.
Ed Wu: We don’t have very many elections. Federal races happen once every two years, and presidential races occur every four years. In addition, the circumstances surrounding each election are unique. How should the accuracy of polls and forecasts be assessed?
Andrew Gelman: The polls are pretty good. In the last election, they were off by maybe two or two and a half percentage points. For most purposes, that’s good enough. It is not so great if the election is to be decided within two percentage points.
You do not want to evaluate forecasts on whether they correctly predict the winner or loser—particularly in close races. If a forecast calls a 50/50 race and gets the winner right, that’s just calling a coin flip. You should get credit for predicting that the election is close, but you shouldn’t get credit for correctly predicting an election that’s basically a tie.
Models can be evaluated based on how well the forecast captures uncertainty. For example, at some point before the 2020 presidential election, FiveThirtyEight gave Biden a 6% chance of winning South Dakota, which seemed too high. For another example, there was also a conditional forecast where if Trump won New Jersey, his probability of winning Pennsylvania wasn’t that high. These predictions suggest flaws in the underlying model.
It’s not easy to correct errors or to bet against them. People are always saying after the election that a campaign spent their money in the wrong place or that they did the wrong thing, even though they had all the information in front of them. But campaigns are making decisions in real time, and they have a lot of other constraints.
Ed Wu: You talked about what makes for a good forecast. But what makes a good poll?
Andrew Gelman: You want the respondents of a poll to be a representative sample of the population. There is also an issue of question wording, but that’s less of an issue with intention polls since the questions are pretty well established.
In previous work, we had looked at polls taken three weeks before various elections for governor, senator, and president. We compared the standard error reported by the poll with how far the poll was from the actual outcome of the election three weeks later. We found that the actual error was about twice that reported by the polls. I think most of that difference was due to nonsampling error stemming from the fact that the people who respond to the poll do not reflect the people who come out to vote.
Ed Wu: We have observed increasing partisan gaps in how Americans view the economy. For example, the gap in consumer sentiment reported by Democrats and Republicans has been considerably higher during the Trump and Biden presidencies than it was previously. How does increasing partisan polarization—where Democrats and Republicans view the economy differently—affect the usefulness of economic indicators?
Andrew Gelman: The economy has become a weaker predictor of election outcomes in recent years. There were landslide elections in 1964, 1972, and 1984, and these are traditionally attributed to the economy. We haven’t had that kind of economic boom in an election year recently, but if we did, I doubt that it would lead to the incumbent getting 60% of the two-party vote.
Ed Wu: Are you optimistic about the future of polling? Are there any new types of data you find promising for election forecasting?
Andrew Gelman: Polling isn’t going anywhere. I agree with George Gallup, the founder of the Gallup Poll, who considered polls to be a public good. But I do think we have way too many polls. For example, some people say we need a high-quality poll of Pennsylvania. But why? We already know it’s going to be close.
Regarding new types of data, we have, for example, hopes for survey questions that ask not just who someone plans to vote for but who the people they know plan to vote for.
Ed Wu: Thank you very much. We appreciate your time.
Tough Choices in Election Forecasting: All the Things That Can Go Wrong
Andrew Gelman began by reflecting on his first experience with election forecasting in 1988, working with a political scientist. At the time, Mike Dukakis was leading in the polls, but his colleague predicted George H.W. Bush would win, citing the economy and incumbency as the deciding factors. Gelman described this as a formative experience, that political scientists can know the outcome of an election before the public.
He then discussed research he and his colleagues conducted for the 1992 election. They forecasted the outcome without using any polls, relying only on information available before the campaigns. While the prediction was largely accurate, it overestimated Bill Clinton’s performance in southern states by giving him too much of a home-region benefit.
Gelman explained that the current political landscape is different. Since 2000, voter opinions have remained relatively stable throughout campaigns, with little movement in the polls. Exceptions include convention bumps and the impact of political ads, which can cause temporary shifts.
He also discussed the phenomenon of differential nonresponse, where more Democrats respond to polls when their candidate is doing well and more Republicans respond when their candidate is ahead, with Republicans generally more likely to respond to polls overall. This, he believes, accounts for most of the fluctuations seen in polling data.
Gelman described the current forecast model he uses with his colleagues, which incorporates both state and national polls, economic indicators, and historical election results. The model accounts for sampling and nonsampling error, with the latter being correlated across states. It allows for opinion drift modeled through a random walk, meaning the polls may fluctuate but will eventually converge on the actual election outcome.
As of October 11, the forecast predicts Democrats will win approximately 52% of the popular vote, but the electoral college outcome remains uncertain due to the state-by-state vote distribution.
Gelman concluded by explaining the paradox of poll aggregation: While combining polls yields more reliable predictions, it removes the daily fluctuations the media often finds newsworthy. He offered a rule of thumb for interpreting poll shifts: A 10-point swing in the polls corresponds to only about a 0.4 percentage point change in the final two-party vote share. Polls and forecasts are valuable, Gelman said, but they must be interpreted with care, particularly in today’s environment, where voters seem less responsive to news, even positive economic indicators.
This summary has been edited for brevity and is not a transcript. The full recording can be found on the Washington Statistical Society’s YouTube channel. Following CNN protocol, the questions were not shared with Gelman in advance.
Leave a Reply