Decision-Making for Statisticians

Mark C. Otto

We work to model the messy reality of the world, dealing with variability and applying sound logic to heaps of data. But how often do we pause to consider the system we use to make day-to-day decisions, big or small? Do we trust our gut, seek consensus, or weigh the usual pros and cons?

Decision-making is an essential skill. It involves your decision process, the information available, and luck. On one hand, you can research everything about a new house—checking schools, commutes, and neighborhood data—and still end up with a neighbor from hell. On the other hand, a bad decision can sometimes lead to a great outcome, which, frustratingly, can bias your future judgment. Our goal isn’t to eliminate uncertainty or guarantee a perfect result; it’s to follow a solid decision process. What’s difficult, in hindsight, is to see we made the “best” decision possible based on what we knew at the time, not the result.

The greatest value comes from questioning common sense and leadership beliefs, such as the idea that leaders should persevere. Additionally, few navigate trade-offs effectively. Decision analysis helps us go beyond data analysis by framing projects from the outset and guiding clients and leaders to make sound decisions informed by data and our analysis. Here are four key areas of decision analysis that correspond with our statistical expertise. For more information, see the Alliance for Decision Education.

1. Valuing and Applying Rationality

This is where statistics meet emotion. Valuing and applying rationality involve understanding what outcomes truly matter to you, including values and objectives beyond just money. Decision analysis offers a logical way to identify the “best” option based on your preferences.

Suppose you want to buy a car. You want it to be reliable, have good gas mileage, cost less, and maybe look stylish enough to be fun to drive. You need to decide how much you value each competing objective. Here, values are more practical than lofty ideals. Each car you consider meets these criteria to varying degrees. But what if your wife comes with you to the dealership? Then, safety and space to fit the soccer team might become more important. Maybe you’re thinking too narrowly—if you live in Washington, DC, with good public transportation, you might only rent a car or truck when necessary. Decision analysis helps you align your objectives and values with your actions, even when buying a car.

2. Thinking Probabilistically

We excel at understanding probability, but when predicting real-world decisions, we often underestimate how uncertain things truly are. Thinking probabilistically means estimating probabilities, usually for single outcomes. We must recognize that our predictions are fragile. For instance, we often ignore unexpected uncertainty; research shows that when we say we’re 90% confident, we’re only correct about 25% of the time, revealing how overconfident we tend to be—the unknown unknowns.

Here, probability is a subjective judgment based on our state of mind and the available information, not on the formal or repeated measurements we usually see in statistics. It involves using our best judgment. Will it be faster to take my usual I-95 that has traffic or to try side streets? I guess 60% of the side streets will be faster. My guess helps me avoid all-or-nothing thinking and thinking of risk rather than being right. 

3. Structuring Decisions

A structured decision-making approach helps us avoid poor choices. While many problems are too complex to solve on a napkin, even small decisions benefit from starting the PrOACT process: framing the problem, establishing our objectives, generating alternatives, predicting consequences, and working through trade-offs.

We must begin by framing the problem. If we leave out certain aspects from the problem description, they won’t appear in the solutions. Also, if we take proactive steps, we create opportunities that go beyond just returning to our starting level. Next, we identify our values and objectives—these are the metrics that matter most to us. Often, conflicting groups can discuss their values even if they disagree, but they may strongly contest different actions or policies. After generating alternatives that meet these objectives, we can assess the consequences of each alternative for each objective and work through the trade-offs.

The Fish and Wildlife regulations committee used to conduct the waterfowl population survey in a smoke-filled room and emerge with that year’s hunting regulations. In 1997, wildlife biologists and statisticians developed a structured decision-making repeated-decision process to maximize hunting opportunities while keeping duck populations at safe levels. It was built on competing statistical population models that forecasted the impact of the harvest. This made the regulation process more scientific and transparent. Both state agencies and hunters have been supportive of this approach. Because the decisions are repeated, we can learn from the models and predictions. 

4. Recognizing and Resisting Cognitive Biases

Statisticians excel at identifying statistical bias and noise (variability). However, cognitive biases are psychological rather than mathematical. They can still bias or sabotage human decision-making. Many of these biases stem from confirmation bias. It’s easy to confirm what we already believe, but difficult to listen to and understand different viewpoints. 

A key way to counter bias is by examining what we overlook. Consider the tragic Challenger disaster. Engineers only examined O-ring failures on flights without incidents, ignoring those where failures occurred. When all relevant data was included and incidents were sorted by temperature, the risk became evident.

We also need the courage to quit. Don’t endure a lousy movie just because you bought a ticket or proceed with a risky space launch under public pressure (the sunk-cost fallacy).

Finally, to avoid public shame from waffling, leaders might need to set “kill criteria” to change policies that aren’t working without seeming indecisive.

Statisticians are problem solvers and can be effective decision-makers. The concepts are familiar to us, but they are applied in different contexts, and good decision-making makes us more valuable statisticians.

Mark Otto

Mark C. Otto is a retired, but not retiring, US Census Bureau and US Fish and Wildlife Service statistical scientist. He is chair of the American Statistical Association Committee on Applied Statisticians and has served as the president of the Washington Statistical Society and as the Membership Council vice chair. (Consider volunteering.) He is an ASA Fellow. In these interesting times, he works toward long-term cultural change with the Alliance for Decision Education and Braver Angels.

Deciding Factors, a column by Mark Otto, explores decision analysis and how statistical scientists can apply their expertise to make better choices in their careers, communities, and personal lives.