Validation of Pretrial Risk Assessments Used in Practice

Evan M. Lowder, George Mason University
David B. Wilson, George Mason University

Objectives: A growing number of studies have examined the predictive validity of pretrial risk assessments. Overwhelmingly, these validation studies are conducted in the context of routine practice, where not all individuals who are assessed receive pretrial release. Despite evidence of range restriction in pre- trial validation research, no study to date has corrected for range restriction in predictive validity estimates. To address this limitation, we examined the effects of range restriction on predictive validity estimates under varying conditions. Method: We ran simulations based on data from a local validation of 1,030 pretrial defendants to demonstrate the effects of range restriction on predictive validity estimates under different degrees of range restriction, various population correlations, and with and without the influence of a third variable, z, representing discretionary decision making. We examined the effect of range restriction on correlation coefficient (r) and nonparametric Area Under the Curve (AUC) statis- tics. Results: Under a realistic population correlation (q = .40), attenuation of r ranged from 1–29% for total scores and 8–39% for risk levels across conditions. Under similar conditions, attenuation for AUC estimates ranged from 1–13% for total scores and 1–20% for risk levels. Attenuation was greater with the influence of a secondary selection variable modeling discretionary decision making above and beyond the risk assessment tool. Conclusion: Range restriction may meaningfully reduce predictive validity estimates when greater than 40% of moderate risk and 60% of high risk defendants are detained.

pretrial, risk assessment, predictive validity, range restriction, pretrial misconduct

“Pretrial risk assessments are growing in popularity across the United States, prompting research on their predictive validity in practice. These validation studies frequently involve the loss of observations due to individuals who are detained pretrial with no at risk period in the community. Yet, there has been limited investigation looking at how range restriction due to pretrial decision-making may attenuate predictive validity estimates. We addressed this limitation by simulating data to show the effects of range restriction on predictive validity estimates under various conditions. Our findings showed that at moderate rates of detention for moderate (20% or less) or high (40% or less) risk defendants, range restriction did not change the overall magnitude of predictive validity estimates. However, there was more evidence of attenuated (i.e., lowered) predictive validity estimates when more than 40% of moderate risk or 60% of high risk defendants were detained. There was also greater attenuation with the presence of a secondary selection variable modeling discretionary decision making, particularly when the population correlation was higher and a greater number of moderate and high-risk defendants were detained” (p. 330).

“Importantly, we saw minimal evidence that predictive validity estimates were attenuated at rates of range restriction seen in practice, based on the originating dataset. At most, predictive validity estimates were lowered by 16% as a result of range restriction. However, we only saw this level of attenuation for the predictive accuracy of risk levels when evaluated with correlation coefficient r. However, there is a shortage of validation studies that have reported full information on detained cases. Thus, it is difficult to appraise generalizability of these release rates across pretrial settings. In pretrial settings more broadly, anywhere between 50 to near 90% of defendants are released pretrial. This variability in release rates has meaningful implications for predictive validity estimates in jurisdictions where release rates are close to 50% or below for moderate- and high-risk defendants. Specifically, validation estimates may show that the tool is underpredicting, which may cause jurisdictions to conclude (erroneously) that the tool is not accurate in predicting misconduct and should not be used to inform release and supervision decisions” (p. 330).

“As anticipated, there was less evidence of attenuation due to range restriction with higher population correlations between pre-trial risk assessments and misconduct outcomes, relative to lower population correlations. Specifically, although the absolute reduction in coefficient values was greater when population correlations were higher, the proportion of attenuation in population coefficients was lower. In prior meta-analytic research, pretrial risk assessments have shown small correlations with any pretrial misconduct, though estimates are likely biased downward due to low base rates and range restriction…Practically, our findings suggest—regardless of the tool under investigation or the setting—all pretrial risk assessment validations conducted in routine practice should be concerned about the potential influence of range restriction on predictive validity estimates” (p 331).

“Overall, our findings highlight the extent to which local conditions may change the magnitude of predictive validity findings. Relative to validations conducted in other criminal justice settings, the pretrial setting is a unique context. Assessments are used not only to inform release, but to inform pretrial supervision monitoring and release conditions as well. The exposure period of interest is also inherently variable. Some individuals may have formal charges filed months before they are apprehended and brought into the jail for assessment, limiting the length of their follow-up period. The entire case processing period can also elapse during an initial period of detention if a defendant’s case is disposed prior to release. Finally, there are a multitude of discretionary influences on pretrial release decision making, above and beyond risk assessments. Together, these factors have the potential to limit sample generalizability and further downwardly bias the assessment of predictive validity beyond what might occur due to range restriction Our findings underscore the need for transparent reporting of sample creation processes in pretrial validation studies to appraise the generalizability of findings and potential magnitude of attenuation. Such information would also allow both the primary authors and meta-analyst the ability to calculate study-specific attenuation- adjusted validity coefficients. In practice, complete reporting may be complicated by whether researchers play a primary role in data cleaning, the accuracy of secondary data records, and the researcher’s ability to create a sample from a population of cases. In larger validation studies, these objectives may be difficult to achieve. The limited availability and quality of data has been noted as a limitation in pretrial research more broadly” (p. 331-332).

“Furthermore, there is likely to be more substantial attenuation of predictive validity estimates with the presence of unstructured influences on decision making. These influences, modeled as a third variable z in the present investigation, may include other legal factors (e.g., charge type and severity), extralegal factors (e.g., age and sex;), judicial attitudes toward pretrial risk assessment, or judicial discretion more broadly. Our assumption was that non-risk-assessment influences on release decisions would still be related to out- comes, but less so than the actual risk assessment. We modeled this correlation at two thirds of the original risk assessment and outcome correlation. However, predictive validity estimates would be more attenuated in situations where these influences on release decisions reflect inaccurate risk management decisions and where higher proportions of defendants are detained” (p. 332).

“Overall, our findings suggest range restriction may not practically attenuate predictive validity estimates when overall release rates are high. However, severe range restriction (i.e., detention of nearly half of all moderate- and high-risk cases) may attenuate the strength of predictive validity estimates, particularly when high rates of pre- trial detention are driven by unstructured decision-making processes in addition to risk assessments. These findings have important implications for pretrial validation research conducted in the con- text of routine practice. In jurisdictions where a high proportion of pretrial defendants are detained, predictive validity estimates may be meaningfully lowered by range restriction. Researchers must be aware of these potential influences on the strength of predictive validity estimates and contextualize their findings appropriately to local stakeholders” (p. 333).

“The numerous influences on pretrial decision making support the need for structured guidelines to ensure consistent use of risk assessment information, alone or in combination with other legal factors. Our findings suggest unstructured decision making has the potential to lower predictive validity of risk assessments when validated in practice, particularly when higher proportions of defendants are detained. Importantly, only half of all jurisdictions that have implemented pretrial risk assessments report using structured guidelines to guide consideration of risk scores in decision making. Importantly, there is growing interest in departures from structured guidelines in pre- trial settings, including how judges view their discretion in adhering to risk assessment-guided recommendations. Our approach modeled discretionary decision making overall, which may have included upward or downward departures from structured guidelines. Because structured guide- lines incorporate information on risk scores as well as criminal charges, it was not possible for us to tease out the role of risk score versus charge in departing from structured guidelines. To illustrate, a downward departure from a guideline could still be risk-adherent if the defendant was given a decision corresponding to their risk level but not their charge severity. This was a primary reason why we modeled discretionary influences overall as a third variable, z” (p. 332-333).

“Our findings also showed evidence of attenuation for AUCs as well as correlation coefficients, particularly when the population correlation was near .50. This is consistent with the work of Walter (2005) who showed that AUCs are affected by range restriction. AUC values also have been criticized as an incomplete measure of predictive accuracy, focusing on discrimination rather than calibration. Notably, however, AUC values for total scores were less susceptible to attenuation, even under the most restrictive models. Use of the AUC for total scores may be more appropriate than for risk levels when severe range restriction occurs, and researchers are interested in overall discrimination. However, practitioners and other legal scholars are increasingly interested in other metrics such as the positive predictive value (PPV) and negative predictive value (NPV). PPVs in particular may be distorted by the detention of high-risk individuals, which may affect observed rates of misconduct in this group, whereas AUC values may be less affected. However, PPV and NPV assume a single risk threshold at which an individual should be detained or released. Pretrial decision making in practice is multifaceted and includes consideration of risk assessment information with offense type and severity. As a result, these metrics may be less suitable for measuring and accounting for the effects of range restriction on predictive accuracy” (p. 330).

As always, please join the discussion below if you have thoughts or comments to add!