Bias, Fairness & Discrimination in AI Systems
How AI systems encode, amplify, and perpetuate societal biases — and the technical and institutional frameworks for measuring, mitigating, and governing fairness in machine learning.
Defining Algorithmic Bias and Its Sources
Algorithmic bias refers to systematic and unfair discrimination produced by an AI system — where outcomes differ across demographic groups in ways not justified by relevant factors. Critically, bias in AI systems doesn't require malicious intent. It can emerge from data, model architecture, deployment context, or the feedback loops between a system and society.
The most important source is training data bias. Data reflects the world as it is — including historical discrimination. A hiring algorithm trained on a company's past hiring decisions will encode that company's historical preferences. A facial recognition system trained predominantly on lighter-skinned faces will perform worse on darker-skinned faces. The system is doing exactly what it was trained to do; the problem is what it was trained on.
The COMPAS recidivism prediction tool, used by US courts to assess reoffending risk, was analysed by ProPublica (2016) and found to be nearly twice as likely to falsely flag Black defendants as higher risk compared to white defendants, while white defendants were more likely to be mislabelled lower risk. The company countered with a different fairness metric. Both sides were mathematically correct — but the metrics are mutually incompatible. This is the core of the fairness impossibility problem.
A Taxonomy of Bias Sources
- Historical bias: The world being modelled contains historical discrimination. Even a perfectly representative dataset can encode unjust historical patterns that the model perpetuates.
- Representation bias: Training data over- or under-represents certain groups. Facial recognition trained on datasets that are 78% male and 83% lighter-skinned (Buolamwini & Gebru, 2018) will systematically perform worse on darker-skinned women.
- Measurement bias: The proxy variable used to train the model is itself biased. Using arrest records as a proxy for crime encodes policing patterns, not actual criminal behaviour.
- Aggregation bias: Building a one-size-fits-all model when subgroups have meaningfully different distributions. A diabetes prediction model trained on population averages may perform poorly for specific ethnic groups.
- Evaluation bias: Benchmarks and test sets that don't adequately represent the deployment population. A model that "passes" evaluation may still fail systematically in real deployment.
- Deployment bias: The model is deployed in a context different from its training context, changing who is affected and how.
The Mathematics of Fairness — and Why They Conflict
Computer scientists have formalised multiple definitions of fairness, each capturing a different intuition about what equal treatment means. The central result — the fairness impossibility theorem — shows that when base rates differ between groups, it is mathematically impossible to satisfy all major fairness criteria simultaneously.
When the base rate of a binary outcome differs between groups, it is mathematically impossible to simultaneously achieve: (1) equal false positive rates, (2) equal false negative rates, and (3) calibration. COMPAS achieved calibration. ProPublica measured equal false positive rates. Both were correct — they were measuring different things. Every fairness intervention is a values choice, not a technical solution.