The role of risk assessment instruments and algorithms in criminal justice decision making.

Jennifer Skeem, University of California, Berkeley
Nicholas Scurich, University of California, Irvine
John Monahan, University of Virginia

Objective: Use of risk assessment instruments in the criminal justice system is controversial. Advocates emphasize that risk assessments are more transparent, consistent, and accurate in predicting re-offending than judicial intuition. Skeptics worry that risk assessments will increase socioeconomic disparities in incarceration. Ultimately, judges make decisions—not risk assessments. This study tests whether pro- viding risk assessment information interacts with a defendant’s socioeconomic class to influence judges’ sentencing decisions. Hypotheses: Tentatively, socioeconomic status was expected to have a main effect; without an interaction with risk assessment information. Method: Judges (N 340) with sentencing experience were randomly assigned to review 1 of 4 case vignettes and sentence the defendant to probation, jail, or prison. Information in the vignettes was held constant, except the defendant’s socioeconomic status and whether risk assessment information was provided. Results: Risk assessment information reduced the likelihood of incarceration for relatively affluent defendants, but the same information increased the likelihood of incarceration for relatively poor defendants. This finding held after controlling for the sex, race, political orientation, and jurisdiction of the judge. Conclusions: Cuing judges to focus on risk may re-frame how they process socioeconomic status—a variable with opposite effects on perceptions of blameworthiness for past crime versus perceptions of risk for future crime. Providing risk assessment information may have transformed low socioeconomic status from a circum- stance that reduced the likelihood of incarceration (by mitigating perceived blameworthiness) to a factor that increased the likelihood of incarceration (by increasing perceived risk). Under some circumstances, risk assessment information may increase sentencing disparities.

risk assessment, algorithms, decision making, judges, bias

“Today, policymakers are keenly interested in using risk assessment as a tool for criminal justice reform. In fact, risk assessment is ‘the engine that drives’ a federal prison reform bill that was just signed into law. Across the United States, jurisdictions have been undertaking a variety of efforts to reduce unprecedented rates of incarceration without compromising public safety. Risk assessment can be helpful in this regard. One way to safely reduce the human and fiscal cost of mass incarceration is to identify the people who are least likely to reoffend and release them, supervise them in the community on probation or parole, or abbreviate their period of incarceration. Advocates argue that—when risk is a legally relevant consideration—judges should consider risk assessment instruments (RAIs) to improve the consistency, transparency, and accuracy of their decisions” (p. 52).

“Judges routinely make momentous decisions in a person’s life that include consideration of the likelihood that the person will reoffend—and must make their own intuitive judgments, without RAIs. At the pretrial stage, each of the 30,000 daily arrests in the United States requires a judge to decide whether to release a defendant until their court date or keep them in jail to prevent them from absconding or reoffending before their case disposition. At the sentencing stage, each conviction requires a judge to determine an appropriate sentence. Although sentencing traditionally focuses more on backward-looking concerns about the defendant’s blameworthiness for a past crime, the Model Penal Code also provides a limited role for forward-looking concerns about preventing future crimes. In a recent survey, eight of 10 judges believed that both blameworthiness and risk of reoffending should be considered at sentencing” (p. 52).

“Despite the clear promise of risk assessment, such suggestions have been met with intense criticism. The principal concern is that using risk assessment to inform judicial decisions will increase racial and socioeconomic disparities in incarceration. In an era of general skepticism about the fairness of algorithms critics assert that risk factors included in some RAIs (e.g., education level, marital status, neighborhood disadvantage) are ‘proxies’ for minority race and poverty. In the view of former Attorney General Eric Holder, the broad use of risk assessment ‘may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society’” (p. 52).

“In the present study, we address this essential question. Real judges with criminal sentencing experience participated in a controlled experiment to test whether the provision of risk assessment interacts with a defendant’s socioeconomic class to change sentencing decisions. Because sentences can be influenced by a host of case characteristics and judicial tendencies, we used an experimental design to permit causal inference about the variables of interest. Judges were randomly assigned to review one of four written case vignettes that described a defendant who had been convicted of a drug offense—a type of offense that is common and associated with both sentencing discretion and sentencing disparities. The case vignettes varied in only two independent factors: whether the defendant was relatively poor or affluent, and whether a set of risk assessment information was provided or omitted. After reading the case vignette, the judges then issued a sentence. If risk assessment exacerbates disparities, as Holder (2014) predicted, then providing judges with risk assessment information will increase sentencing severity significantly more for relatively poor defendants than their more affluent counterparts” (p. 52-53).

“In a case designed to maximize judicial discretion, we found that adding risk assessment information reversed the direction of judges’ disparities in sentencing relatively poor versus affluent defendants. This reversal held after controlling for judges’ jurisdiction and personal characteristics. We believe this reversal occurred because (a) many judges—and the Model Penal Code—attempt to balance competing sentencing considerations that include the defendant’s blameworthiness and risk, (b) socioeconomic status has opposite effects on perceptions of blameworthiness for committing a past crime versus perceptions of risk for committing a future crime, and (c) cuing judges to focus on risk reframes how they process socioeconomic status. Providing judges with risk assessment information transformed low socioeconomic status from a circumstance that reduced the likelihood of incarceration (perhaps by mitigating perceived blameworthiness) to a factor that increased the likelihood of incarceration (perhaps by increasing perceived risk)” (p. 56).

“Specifically, without risk assessment information, judges were less likely to sentence the relatively poor defendant to incarceration than his more affluent counterpart (45.8% vs. 59.5% probabilities, respectively). In this context, judges may have implicitly processed poverty as an unfortunate circumstance that helped explain the offense and should mitigate the sentence. Arguably, a casual laborer in construction who dropped out of high school is no less blameworthy than a degree-holding computer technician when he decides to commit a drug offense. Nevertheless, environmental deprivation has occasionally been discussed as a mitigating factor at sentencing. Perhaps in this context, judges processed the relatively poor defendants’ crime as the partial product of disadvantages in life, which mitigated his culpability. In contrast, the relatively affluent defendant had little excuse” (p. 56-57).

“When risk assessment information was added to these cases, judges were more likely to sentence the relatively poor defendant to incarceration than his more affluent counterpart (61.2% vs. 44.4%). Adding formal risk assessment information may have cued judges to process poverty as a factor that increased the likelihood that the defendant would continue committing offenses and to process relative affluence as a factor that reduced the likelihood that the defendant would continue committing offenses. This context may have activated stereotypes of poverty and affluence that led judges to interpret identical risk scores as signaling a much higher risk of re-arrest for the relatively poor defendant than his more affluent counterpart” (p. 57).

“Fundamentally, this study demonstrates that biases can shift, as a risk assessment algorithm filters through a judge into a sentencing decision. It is worth reiterating that there were sentencing disparities in this study, even in the absence of risk assessment information. Given a medium-risk defendant convicted of a drug offense who falls in “gray” sentencing territory, providing judges with risk assessment information trans- formed poverty from a mitigating circumstance that reduced the likelihood of incarceration to a risk factor that increased the likelihood of incarceration” (p. 58).
“In many jurisdictions, formal risk assessment information is routinely included in presentence investigation reports. Even when judges explicitly discredit or reject risk assessment, exposure to risk scores could influence how they intuitively process information about the defendant to reach a sentence. We believe that risk assessment has an important role to play in reducing mass incarceration in the United States, as the Model Penal Code has recently affirmed. Providing guidelines or training to raise judges’ awareness about their own intuitive biases and how they can interact with algorithms may help. Determining how to present risk algorithms so that judges can most effectively and fairly incorporate them into their decision making about defendants is essential” (p. 58).

“The present study was designed to permit valid inferences about the cause– effect relationship between risk assessment and socioeconomic status on judges’ sentencing decisions. This experiment may overestimate the causal effect, and findings may not generalize beyond the specific conditions tested. First, we deliberately created cases that fell in a gray sentencing zone (eligible for probation or incarceration), in which inappropriate considerations like socioeconomic status or race may be most likely to influence judges’ decisions. Results may not generalize to cases that involve less judicial discretion. Second, it is unclear whether the present results would generalize from a drug case to other types of offenses that may be less associated with stereotypes of poverty; and from “moderate to high” risk cases to those at lower risk of recidivism. Finally, although we developed relatively detailed presentence vignettes tailored to local jurisdictions to maximize ecological validity, the independent variables probably have a weaker effect in real courtroom settings in which judges are exposed to a richer set of case materials and interact with the parties involved. In future research, it will be important to test the extent to which the present results generalize to contexts in which judges have more limited discretion in sentencing, defendants vary in their offenses and estimated risk levels, and sentencing materials are more complete. Whether providing judges with risk assessment information increases, decreases, or has no effect on sentencing disparities probably depends on several conditions that are just beginning to be understood” (p. 57-58).

As always, please join the discussion below if you have thoughts or comments to add!