David Morganstein

The statistician as skeptic is well typified by Deming’s frequent use of the question “How do you know?” I offer a story and two real case studies that represent the value of asking this seemingly basic question. Whether you’re working as a statistical consultant or simply trying to improve quality within your own organization, trying to uncover the origins of common practices often reveals opportunities for improvement! You may discover no one has any real evidence, the current practice works, or the rationale for it no longer applies.

I was out in the kitchen one evening helping my wife prepare dinner for the evening’s meal with her parents. I observed her unwrap a ham and, with a knife, slice a quarter of an inch off the end and throw the end piece in the trash. Quite surprised, I asked her why she did that, “Did you read about that in a cookbook or see it on a TV show?”

She replied, “I learned it from my mother. She used to do that.”

I asked, “Why did she do that?”

After another few moments in thought, she answered, “I think it’s something about the end of the meat getting dried out, and you don’t want to serve it.”

Being curious, I cornered my mother-in-law after dinner and told her I’d been out in the kitchen helping her daughter fix the meal. (Of course, I was hoping to earn a few points from my mother-in-law for sharing the kitchen duties.) I brought up the ham-trimming incident and asked her if my wife had learned that curious culinary technique from her.

“Yes,” she replied.

“So, why do you do that?” I asked.

Her answer was that the meat was exposed to the air and that damaged the flavor, a somewhat different explanation than my wife’s. “Did you learn that in a cookbook or from someone?” I asked.

She replied, “My mother did that.”

Since I’d heard two rather different explanations for this seemingly wasteful tradition, my curiosity got the best of me. The next day, I called my grandmother-in-law and asked her, “Did your daughter see you cut a quarter of an inch off the end of the ham and throw it away?”

After a few seconds of laughter, she replied, “Yes. After the war, the only pan we had was 6 inches long!”

How do you know?

One of my statistical consulting projects took me into a steel operation’s rolling mill. We had worked with various units in the steel operation, but the rolling mill was new to us. During a tour of the unit, we saw how the mill took huge multi-ton slabs of steel produced much earlier in a steel-making operation by a continuous caster, heated the slabs red hot in a furnace, and rolled them like Play-Doh into long thin strips that were coiled up at the end of the mill. The width and thickness of each strip defined a specific product for a client.

The first thing the operator did was trim off the leading edge of the heated slab. The operator had a card pasted above him. The card was clearly old and dog-eared and listed various coil dimensions with a number of feet next to it. We asked him how he used the card. He said, “These ingots are created by pouring steel into a mold. When the steel cools, the top of the material oxidizes and converts into some nonuniform material called ‘pipe.’ We saw off the end with the pipe and throw it back into the steel-making furnaces so it can be melted down and reused.”

We commented that we were surprised by this step because we understood the casting operation had been replaced two years ago with a $250 million continuous caster, a process that no longer created the pipe and was one reason for the huge expenditure. The operator looked puzzled at first, then disturbed. He took us to a conference room and asked us to wait for him to return. The operation manager then met with us, saying we had more than earned our consulting fee! Indeed, the continuous caster was intended to save millions of dollars of wasted material that no longer needed to be cut off from the end and recycled. Here, two years on, and the mill had still not changed its practices.

How do you know?

Our second real-world quality improvement example took place some time ago when telephone surveys of households used the Waksberg method of sampling. In this process, using a 100 bank of numbers and the first eight digits of the phone number, a random choice of the final two digits was made and contacted. If this choice was a residential number, the cluster was retained and a fixed number of additional numbers were dialed within it. Waksberg demonstrated that, although the number of residential numbers in the cluster was unknown and related to the selection probability of the cluster, the two-stage process removed the unknown measure of size from the selection probability, creating an equal probability design.

During downtime in the telephone center, interviewers would make that initial call to phone clusters and then “put on the shelf” for future use those clusters that had been identified for retention. However, they chose not to call the initial phone number for a subsequent interview because either time had passed since those numbers were located and they might have converted to nonresidential or, having contacted the household one time, coming back later for an interview would be seen as a burden and adversely affect the response rate.

A sampling statistician on a project asked the question, “Why do you do that (throw away that initial known to be residential number)?” When told the rationale, he asked, “How do you know?” An experiment was then conducted that demonstrated that those initial numbers were virtually always residential numbers and that the households, in fact, were “cooperating” households since they had already participated during the screening so they were more likely to cooperate again. Until this situation had been discovered, roughly one-sixth of all the work was being thrown away, just like the slice of ham and the ends of the steel slabs.

If you are a statistical consultant or working on quality improvement in your own organization, be sure to keep the question, “How do you know?” in your hip pocket!