Winning Isn’t the Only Thing: Why Statisticians Must Separate Process from Outcome 

Annie Duke is short-stacked and all-in-or-fold with a pair of tens. (She tells her story onThe Moth podcast) She picks up Greg Raymer’s tell. The rest of us on the couch in front of the TV, watching every player’s hole card, are sure we could have played it better. Really? The deeper question isn’t whether we can read the table—it’s whether we judge the hand by who claims the pot. Statisticians should know better.

And yet we don’t. Not consistently. In poker circles, assessing a decision by how it turned out is called resulting. Resulting is a cognitive bias we need to understand and avoid to make good decisions.

As statisticians, we are supposed to separate decision quality from outcomes. Yet, in practice, even in scientific and clinical work, we often judge decisions by how they turned out rather than what was known at the time. This shows up all the time in how we interpret evidence, evaluate studies, and make policy decisions. 

Forecasts, Smoothed Estimates, and the View from the Couch

Resulting is the difference between forecasting and smoothing. The blue line is the one-step-ahead forecast, jumping around with a wide confidence band. The green line is estimated using all future data and smoothed with a narrow confidence band. At t=50, the series jumps—a level shift. The blue forecasts continue to track the old level for a few periods before catching up. The green smooth glides up through the shift as if it had been expecting it all along.

Now the catch: In real life, we can make decisions like the blue one-step forecasts. We only have data up to the present. Our process can be sound, our model well-specified, and we will still be late to a level shift when the world changes. Watching Annie on TV, we are looking at the green line. We can see where the hand went. Judging her decision from that vantage point is the same as scoring last week’s forecast against this week’s smoothed value.

This is evident in clinical research. A well-designed trial can appear to fail because of variability or a small sample size, while a weaker analysis might appear successful by chance or bias. If we judge these decisions solely by their outcomes, we end up rewarding the wrong things and discouraging the right ones.

Decisions are process and luck. In our models, our process is the prediction, and luck is the residual; we need to be careful not to confuse them. In life, it is much harder to tell which is which, but it is worth the work—studying our decisions using the information we had at the time, no matter how they turned out.

In other words we choose one outcome from many possible ones but tend to forget the others we didn’t choose.  Each of those possible outcomes has a different likelihood of succeeding or failing in ways we cannot fully control.  That’s luck.  If we mix these up, we reward luck and penalize sound decisions.

The 2×2 That Earns Its Keep

Cross decision quality (good or bad process) with outcome (good or bad result):

A good decision should be judged by what it was likely to lead to, not by how things turned out this one time. The 2×2 framework shows how easily those two can diverge.

Our instinct is to evaluate column-wise, by outcome. The decision analyst evaluates row-wise, by process. The cell that quietly does the most damage is the bottom-left—dumb luck—because the world rewards it before we notice the process was flawed.

A Liberal Season in a Dry Year

In the early years of adaptive harvest management, waterfowl regulations were liberal year after year. Then came a dry year—low pond habitat and low duck counts—and the model surprisingly prescribed another liberal season. The model had a built-in learning component, and much could be learned from a heavy harvest in a poor year. But there was also a duty to the resource. We had a real conflict: protect ducks or protect adaptive harvest management?

In the end, the technical team adopted the model’s prescription to keep regulators and the hunting community committed to adaptive harvest management, even when the answer was uncomfortable. What they learned afterward was that none of the regulatory teams took their full liberal seasons; the duty to the resource showed up downstream as restraint.

Was it a good decision? Should the team have understood the model’s prediction more deeply before deferring to it? Did we end up in the top-left or bottom-left cell? Good outcomes deserve the same audit as bad ones because a good outcome from a shaky process is the surest way to repeat that shaky process next year.

We Were Not Trained for This

In statistics, we design the survey or run the experiment, then collect the data to answer the question. The luxury is that the data is built to fit the question. In decision analysis, we frame the decision as best we can, scrape together whatever data exists (never enough), and decide in the moment. We may know the odds, as Annie knew the odds with her tens, but we cannot know what will happen around the table. We decide on a single outcome, not the long-run distribution.

Annie fell in with a group of the world’s best players who debriefed every difficult hand in extraordinary detail—cards, position, table image, opponents’ tendencies, and other things I cannot imagine remembering. The rule was to describe the situation without revealing what you did or how it turned out. Eric Seidel cut Annie off the first time she opened with a bad-beat story. If it’s bad luck, why are we talking about something we can’t learn from? We can explore our decisions the same way.

A good decision should be judged by what it was likely to lead to, not by how things turned out this one time.

The Hard Part Is Doing It on Purpose

The hardest part is making decisions consciously. If you don’t examine your decisions, you can’t learn from them, and it’s harder to separate process from luck. You must track what you knew and when you knew it. A decision journal works for personal choices; a shared log works for projects. It feels awkward at first, but it gets easier over time.

In scientific work, we try to guard against this by setting things up in advance, being transparent about our data analysis, and checking results through replication. At the end of the day, these practices focus on the decision process rather than the outcome. Bringing that same mindset into everyday decisions is not easy, but it is important.

The payoff is real. As you sweat the outcome less and grow curious about the process, the “I should have known” of hindsight bias loses its sting. You can look back at what you actually knew and forgive yourself for not knowing more. You can lose a hand and still respect how you played it, taking things as they come, even at the poker table.

Decisions are predictions. Next time: Why are we so bad at making them? Remember, our 90% intervals were correct only about 25% of the time. Let’s see what we can do about that.

Mark Otto

Mark C. Otto is a retired, but not retiring, US Census Bureau and US Fish and Wildlife Service statistical scientist. He is chair of the American Statistical Association Committee on Applied Statisticians and has served as the president of the Washington Statistical Society and as the Membership Council vice chair. (Consider volunteering.) He is an ASA Fellow. In these interesting times, he works toward long-term cultural change with the Alliance for Decision Education and Braver Angels.

MinJae Lee

MinJae Lee is a biostatistician and professor at McGovern Medical School, UTHealth-Houston, where she leads the division of clinical and translational sciences. Her research focuses on developing and applying innovative statistical methods to address real-world research challenges. She is a member of the ASA Committee on Applied Statisticians and serves on the Nature Medicine Statistical Advisory Panel.

Deciding Factors, a column by Mark Otto, explores decision analysis and how statistical scientists can apply their expertise to make better choices in their careers, communities, and personal lives.