Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies
Two matters recently receiving considerable media attention highlight a statistical pattern that is almost universally misunderstood. Its misunderstanding undermines most efforts to appraise demographic differences in outcome rates in the law and social and medical sciences. Both matters involve federal law enforcement policies based on statistical perceptions that are the exact opposite of reality.
Over the last year, the Department of Justice (DOJ) announced settlements of claims of racial and ethnic discrimination in lending with recoveries totaling more than half a billion dollars. The largest involved Countrywide Financial Corporation ($335 million) and Wells Fargo Bank ($175 million). The complaints in both cases fault lenders for failing to implement less discriminatory alternatives to practices believed to cause minorities to receive subprime, rather than prime, loans at higher rates than whites do, as well as for various practices that led generally to greater frequency of subprime loans.
To put things in context, one must look back to the 1990s, when there existed great concern about several-fold racial and ethnic differences in mortgage rejection rates. In 1994, belief that a substantial part of the differences resulted from the greater difficulty minorities had in meeting standard lending criteria prompted federal agencies monitoring fair lending laws to announce that lenders could be held liable for unnecessarily stringent criteria that disqualified minorities at higher rates than whites.
The approach accorded with federal policy in the employment testing context, where lowering test cutoffs was universally regarded as reducing tests’ disparate impact because lowering cutoffs reduced relative differences in pass rates. For example, suppose that at a particular cutoff, pass rates are 80% for an advantaged group (AG) and 63% for a disadvantaged group (DG). At this cutoff, AG’s pass rate is 27% higher than DG’s pass rate. If the cutoff were lowered to the point where 95% of AG passes the test, assuming normal test score distributions, DG’s pass rate would be about 87%. With the lower cutoff, AG’s pass rate would be only 9.2% higher than DG’s pass rate.
Lending criteria operate just like test cutoffs, and, as with lowering cutoffs, relaxing lending criteria tends to reduce relative differences in meeting them. Lenders that responded to encouragement to relax lending criteria would tend to have lower relative differences in mortgage approval rates.
But, while lowering cutoffs tends to reduce relative differences in pass rates, it tends to increase relative differences in failure rates. In the situation above, DG’s failure rate was initially 1.85 times AG’s failure rate (37%/20%). With the lower cutoff, DG’s failure rate would be 2.6 times AG’s failure rate (13%/5%).
This pattern is not peculiar to test score data or the numbers with which I chose to illustrate it. Inherent in normal distributions of factors associated with experiencing an outcome is a pattern whereby the rarer an outcome, the greater tends to be the relative difference in experiencing it and the smaller tends to be relative difference in avoiding it. The pattern can be illustrated with almost any data showing points on a continuum of factors associated with experiencing an outcome. Income data show how lowering poverty will tend to increase relative differences in poverty rates while reducing relative differences in rates of avoiding poverty. Credit score data show how lowering a credit score requirement will tend to increase relative differences in failing to meet it while reducing relative differences in meeting it.
As reflected by the credit score example, relaxing lending standards, while tending to reduce relative differences in rates of securing mortgages, tended to increase the relative differences in mortgage rejection rates that had prompted concerns about lending disparities in the first place. Nevertheless, regulators and others concerned about lending disparities continued to measure those disparities in terms of relative differences in mortgage rejection rates. Thus, lenders that were most responsive to the encouragement to relax criteria—hence, more so than other lenders, tending to reduce relative differences in approval rates while increasing relative differences in rejection rates—were regarded as the most discriminatory lenders.
The same patterns exist when the adverse lending outcome is assigned to subprime loan status. And the more lenders follow the suggestions in the Countrywide and Wells Fargo complaints to reduce rates of such assignment, the larger the relative differences that federal regulators monitor will tend to be.
The second matter lately in the news implicating this issue involves racial disparities in public school discipline rates. In March, the Department of Education (DOE) released data showing large relative differences in rates at which minority and white students are suspended or otherwise disciplined. As both DOE and DOJ had previously done, observers attributed the disparities to “zero tolerance” discipline policies in effect in recent decades and called for relaxing those policies. Colorado has already enacted legislation modifying discipline standards at least partly out of concern for racial disparities in discipline rates, and other jurisdictions are considering similar legislative or administrative action.
As in the testing and lending contexts, however, relaxing standards will not tend to decrease relative difference in discipline rates. It will tend to increase them. And, as in the lending context, federal investigations will focus on the school systems with the largest relative differences in discipline rates, which frequently will be the systems most responsive to encouragement to relax standards. At least that will be the case until the statistical expertise of the federal enforcement agencies is substantially improved.
But the failure to understand the way relative differences tend to be affected by the prevalence of an outcome is hardly limited to federal law enforcement. Vast resources are devoted to the study of group differences in the social and medical sciences, but rarely with recognition of the way particular measures are affected by the prevalence of an outcome. Despite the utility of income data to illustrate the described patterns, changing demographic differences in poverty rates are invariably studied without recognition that, given the shapes of the income distributions, certain patterns will almost always be found when substantial increases or decreases in poverty occur.
Thousands of researchers study health and health care disparities around the world, but few recognize that features of the underlying distributions will tend to cause general declines in mortality to increase relative differences in mortality while decreasing relative differences in survival and cause general improvements in health care to reduce relative differences in appropriate care rates while increasing relative differences in rates of failing to receive appropriate care.
Should researchers therefore rely on absolute differences between rates or odds ratios, measures unaffected by which outcome is examined? By no means. Those measures also tend to be systematically affected by the overall prevalence of an outcome. The patterns are just more complicated than in the case of the two relative differences.
For fuller treatments of the patterns by which standard measures of differences between outcome rates tend to be affected by the prevalence of an outcome and the implications of failure to understand those patterns, see my guest editorial “Can We Actually Measure Health Disparities?” in the Spring 2006 issue of CHANCE and my “Race and Mortality” in the January/February 2000 issue of Society. Both are available on jpscanlan.com.
James P. Scanlan is a lawyer in Washington, DC, who specializes in the use of statistics in litigation. He has written extensively about the use of statistics in the law and social and medical sciences.