COMPAS and Recidivism Scoring: The Fight Over Bias, Calibration, and Automated Justice
The COMPAS controversy showed that an algorithm can satisfy one statistical definition of fairness while violating another, making governance choices about error costs as important as the model's predictive accuracy.
COMPAS turned many personal and criminal-history variables into risk scores
COMPAS is a proprietary risk-and-needs assessment system used in parts of the U.S. criminal justice system. It combines information from records and questionnaires into scores intended to help estimate recidivism risk and identify supervision needs. The attraction of such tools is consistency: instead of relying only on intuition, agencies can use a standardized statistical assessment. But the score compresses many choices into one number—what outcome counts as recidivism, which population defines risk, how variables are weighted, and which errors matter most. Once the score enters sentencing or supervision, those modeling choices become policy choices with consequences for liberty.[1]
A score can hide the assumptions that produced it
Decision-makers see a risk category, not the full distribution of uncertainty or the normative tradeoffs embedded in its construction. High-stakes tools therefore need documentation explaining what the score can and cannot mean.
ProPublica found large racial differences in false-positive and false-negative errors
In 2016 ProPublica analyzed COMPAS scores from Broward County, Florida and compared them with subsequent arrests over roughly two years. Its investigation reported that Black defendants who did not reoffend were more likely than white defendants to be labeled higher risk, while white defendants who did reoffend were more likely to have been labeled lower risk. ProPublica argued that these asymmetric error rates showed racial bias. The company then known as Northpointe disputed the interpretation and methodology. The disagreement became important because both sides could point to valid statistics while emphasizing different properties of the same score.[2]
Northpointe argued that the system was comparably calibrated across groups
Northpointe’s response emphasized measures such as predictive parity and calibration: among people assigned similar risk levels, observed recidivism rates were broadly similar across racial groups. ProPublica replied that calibration did not erase the disparity in who was falsely classified as high or low risk. This is not merely a rhetorical dispute. When base rates differ across groups, common fairness criteria can become mathematically incompatible. A model may be calibrated and still produce unequal false-positive rates. Engineers therefore cannot solve fairness by choosing a metric after the model is built; institutions must decide which harms and rights the decision process should prioritize.[3]
Fairness metrics encode competing values
False positives matter to people wrongly labeled risky; calibration matters to decision-makers interpreting a score. Neither metric automatically dominates in every context, so the choice needs legal and policy justification rather than hidden optimization.
The Wisconsin Supreme Court allowed limited use while requiring warnings
In State v. Loomis, the Wisconsin Supreme Court held that a sentencing court could consider COMPAS if the score was used with specified cautions and was not determinative. The opinion highlighted the system’s proprietary nature, the lack of a Wisconsin-specific cross-validation study at the time, concerns about possible racial disparities, and the need to monitor and renorm assessment tools. It also stressed that the scores were not designed to decide sentence severity by themselves. This legal treatment is a useful systems pattern: when a model cannot support a decision independently, the interface and procedure must prevent users from treating it as if it can.[4]
Proprietary models complicate due process and technical auditing
A defendant may know the questions asked and the resulting score without knowing the exact weighting or transformations that produced it. Proprietary protection can be commercially legitimate, but opacity creates a special problem when the score affects public decisions. Independent auditors need enough information to test reliability, subgroup performance, drift, and sensitivity to inputs. Users also need to understand whether the model predicts arrest, conviction, or another outcome, because each target includes different institutional behavior. Accountability cannot stop at ‘the algorithm said so.’ The agency using a vendor model remains responsible for whether the tool is appropriate for its legal purpose.
The controversy changed how researchers explained fairness tradeoffs
The COMPAS debate helped popularize a result that later fairness research formalized: with unequal base rates and imperfect prediction, it is generally impossible to satisfy several intuitive group-fairness criteria simultaneously. Reporting at the time described researchers showing why calibration and equalized error rates can conflict. This does not mean bias is inevitable in every moral sense, nor that nothing can be improved. It means designers must make tradeoffs explicit. The correct question is not ‘is the model fair?’ in the abstract, but ‘which fairness property is being optimized, for which decision, for which population, and with what consequences when the model is wrong?’[5]
Mathematical impossibility does not eliminate governance responsibility
When all desirable metrics cannot be achieved together, the institution must choose openly. Hiding that choice inside a scoring system makes a policy decision look like a technical inevitability.
Risk scores can become anchors even when humans retain final authority
Human-in-the-loop designs are often presented as a safeguard, but a numerical risk score can anchor judgment. A judge, probation officer, or case manager may give a standardized score more weight than intended, especially if the model appears scientific and the alternative is subjective. Interface design should therefore show uncertainty, permitted uses, limitations, and relevant contextual evidence instead of presenting a simple red-yellow-green conclusion. Training and audit logs can help determine whether users are treating the model as one factor or effectively outsourcing judgment to it.
COMPAS remains a lesson in separating prediction from legitimate decision policy
The historical importance of COMPAS is not that one analysis definitively proved or disproved ‘algorithmic bias.’ It is that the dispute exposed how many meanings of fairness can coexist around the same predictive system. Models estimate patterns from data; institutions decide what to do with those estimates. High-stakes deployment therefore requires transparent objectives, population-specific validation, error analysis across groups, legal constraints on use, meaningful human review, and recurring revalidation as populations change. A statistically competent model can still be used unfairly, and a fairness metric can still be inappropriate for the decision it governs. Automated justice fails when those distinctions disappear behind a single score.
Human review needs rules about permissible use
A human in the loop is not automatically a safeguard. Reviewers need explicit instructions about when a score may inform judgment, when it must not be used, and how disagreement should be documented.
Works Cited
- 01ProPublica — Machine Bias propublica.org
- 02ProPublica — How the COMPAS analysis was performed propublica.org
- 03ProPublica — Response to Northpointe critique propublica.org
- 04Wisconsin Supreme Court — State v. Loomis law.justia.com
- 05ProPublica — Mathematical conflict among fairness criteria propublica.org
CodeHistory is a living archive. Citations document the evidence used for this edition; later evidence may refine the account.