Emily Hadley is a research data scientist at RTI International. She is experienced in machine learning, predictive modeling, and natural language processing and is passionate about ethics, bias, and antiracism in statistics and data science.

With the growing use of algorithms in many domains, considerations of algorithmic bias and equity have far-reaching implications for society. Algorithmic bias emerges from the concern that algorithms are not simply neutral transformers of data but compounders of existing societal inequities, particularly when performance is substantially better or worse across mutually exclusive groups.

Algorithmic bias can occur as a result of decisions made throughout the algorithm development and deployment process. Left unaddressed, it can deeply affect equity. While the meaning of equity has been contested for millennia, it is generally considered to be a property of fairness of a decision-making agent or institution.

We already live in a world in which biased algorithms are affecting individual livelihoods. These effects have been described in books like Weapons of Math Destruction, in movies like Coded Bias, and in the growing Artificial Intelligence Incident Database (AAID). The following have occurred as a result of biased algorithms and algorithmic decision-making:

• Millions of black hospital patients received lower risk scores than similar white hospital patients, potentially excluding them from extra care
• Older workers never saw some age-targeted job ads on Facebook, denying them potential work opportunities in a direct violation of US employment law
• Faulty facial recognition led to the arrest of a man for a crime he did not commit

Few and narrow laws exist in the US to govern algorithm development and deployment or address algorithmic bias. Anti-discrimination laws cover the use of specific protected classes like race and sex in algorithms in domains such as lending and employment. The Federal Trade Commission recently took a role in holding companies accountable for deceptive and discriminatory algorithms that affect consumers, and individual states like California and New York have developed legislation related to algorithmic data inputs and the use of algorithms. Yet, there remains no comprehensive legal guidance in the US for algorithm development and usage.

Thus, it falls to the practitioners building these algorithms—including statisticians and data scientists—to interrogate decision-making throughout the algorithm development process and identify opportunities to enhance equity.

Techniques for Practitioners

There is no single tool or approach that makes an algorithm unbiased. Rather, practitioners should adopt a commitment to recognizing opportunities for bias in decisions throughout the development process and act to address these challenges when possible. The following are four techniques related to development:

Technique 1: Advocate for Representativeness of Data

Representativeness of data is a topic often covered in an introductory statistics course. Students learn it is often inappropriate to make sweeping generalizations of results from a limited data set and apply them to populations for which that data is not representative. Yet, in practice, representativeness is often not prioritized and leads directly to biased algorithms.

Facial recognition data sets used to build tools for social media companies and law enforcement have historically skewed white and male, leading to less accurate predictions for non-white and non-male individuals. Health care AI systems are overwhelmingly built using data from just three states (CA, NY, and MA), and this lack of data diversity is likely contributing to biased health algorithms.

When developing an algorithm, practitioners should analyze the representativeness of available data in comparison to the population of interest, identify disparities, and advocate for greater data diversity.

Technique 2: Interrogate Use of Sensitive Attributes in Algorithms

Sensitive attributes are protected characteristics like race, sex, or age for which bias in the algorithm could lead to inequitable decisions. A common argument is that an algorithm must be fair and unbiased if it doesn’t include these sensitive attributes, known as “fairness through unawareness.” Yet, literature has shown this argument does not hold due to the numerous and often opaque relationships of sensitive attributes with seemingly “neutral” predictors.

The algorithm where black patients received lower risk scores was one that was purported to be fair because race was not included as a predictor; however, it emerged that projected cost—another predictor—was correlated with race, which led to the discriminatory outcome.

Practitioners should recognize it is a myth that simply withholding a sensitive attribute from an algorithm will make it fair. The appropriateness of using a sensitive attribute in an algorithm should depend on the context in which it is used, and, regardless of use, practitioners should evaluate fairness (Technique 3) when possible.

Practitioners can further interrogate their use of sensitive attributes with the following questions:

• How was missingness addressed for the sensitive attributes? Was the approach ethical and thoughtful? How might the approach affect the outcome?
• Was a grouping technique such as combining groups with small numbers used for the sensitive attributes? What assumptions were made in this grouping? Who is prioritized by the grouping?
• Were groups combined to create an “Other” group? This group may not be meaningful for analysis; whose insights will be lost by inclusion in the “Other” group?

 

Is a reference level used in the analysis? Why was this reference level selected, and who is prioritized by selecting this reference level?

Technique 3: Evaluate Fairness in Algorithms

Evaluating algorithms for fairness is an evolving area of research. One reason is the competing definitions for fairness. Some metrics align with individual fairness such that individuals with similar attributes should have similar outcomes. Other metrics align with demographic parity such that the distribution of positive or desirable outcomes should mirror that of the general population class distribution. Still others align with equal opportunity such that there should be equal true positive rates across classes.

Selection of fairness metrics should include conversations about the tradeoffs between these definitions with subject-matter experts and those most likely to be affected by the algorithm. Practitioners should prioritize calculation of fairness metrics in their algorithm development workflow.

Technique 4: Consider an Algorithmic Review Board

Academic researchers may be familiar with the institutional review board, an administrative body established to protect the rights and welfare of human subjects. Given the effect of algorithms on individuals, there is increasingly a call for algorithmic review boards at companies developing algorithms with human impact.

Large tech companies and financial institutions have already begun exploring how an ARB or similar committee would work in practice. These company watchdogs can serve as an internal mechanism to evaluate the practices used to collect data and build and deploy the algorithm.

Practitioners should consider if an ARB or similar committee may be appropriate for their own organization.

Statisticians and data scientists are actively involved in developing algorithms being used to make decisions that affect individuals, often with little or no legal oversight. By incorporating individual techniques into their own work, these practitioners can contribute to key decisions that reduce algorithmic bias and improve equity.