Ignorance is Not Always Bliss: The Use of Structured Risk Assessment Instruments and Concern About Bias in Evaluations of Risk

Jennifer Kamorowski; Department of Clinical Psychological Science, Maastricht University, Maastricht, Netherlands; Department of Psychology, University of Gothenburg, Gothenburg, Sweden

Corine de Ruiter; Department of Clinical Psychological Science, Maastricht University, Maastricht, Netherlands

Maartje Schreuder; Department of Clinical Psychological Science, Maastricht University, Maastricht, Netherlands

Karl Ask; Department of Psychology, university of Gothenburg, Gothenburg, Sweden

Marko Jelicic; Department of Clinical Psychological Science, Maastricht University, Maastricht, Netherlands

The use of structured risk assessment instruments (SRAIs) has increased significantly over the past decades, with research documenting variation between countries. The use of SRAIs, their perceived utility and potential for mitigating bias in forensic risk evaluations (FREs) was investigated in a survey of Dutch forensic mental health practitioners (N 1⁄4 110) We found generally positive views regarding SRAI utility. Bias in FREs was of concern to respondents. We found no evidence of a bias blind spot (the belief that oneself is less prone to bias than peers/colleagues). SRAIs were rated as the most effective debiasing strategy, but respondents also endorsed introspection. There were few differences in beliefs about sources of bias or debiasing strategies between respondents who had bias training and those who had not, suggesting the need for development of effective strategies to mitigate bias and training related to bias in FREs.

Cognitive bias; debiasing strategies; forensic mental health; forensic risk evaluations; risk assessment

“Forensic mental health practitioners play an important role in advising courts about the treatment and management of people who suffer from a mental illness and have been accused or convicted of a crime…Specifically, an offender’s estimated risk of violent or sexual reoffending is a key consideration in determining what conditions are necessary – and legally justified – to minimize the risk of harm to potential victims and to make the best use of limited resources…Research conducted over the past decades indicates that using structured approaches in forensic risk evaluations (FREs) – particularly structured risk assessment instruments (SRAIs) – increases the accuracy of predictions about the likelihood of future violent or sexual offending, as compared to unstructured clinical judgment…Findings from more recent surveys also indicate that SRAI use and FRE practices can vary widely across countries…Yet there is relatively little country-specific research about which SRAIs forensic evaluators are required to use in practice, which SRAIs they choose to use, and their perceptions about the usefulness of those tools…” (p. 1-2).

“Cognitive bias refers to systematic errors in logic or reasoning that occur outside of conscious awareness…and are the result of the mind’s automatic processing of information based on experience and prior expectations…Therefore, cognitive bias represents a threat to the objectivity and validity of a forensic mental health evaluations…A number of recently published surveys suggest that forensic mental health practitioners are aware of, and concerned about, the potential for bias in forensic mental health evaluations…However, many remain skeptical about bias affecting their own work as evidenced by a bias blind spot…that is, the belief that they are less prone to bias than their colleagues…In this study, we surveyed Dutch forensic mental health evaluators regarding their use of SRAIs for evaluating violent, sexual, criminal, and intimate partner violence risk. Based on previous research conducted by Zapf et al. (2018), we also measured Dutch evaluators’ awareness about the potential for bias in forensic risk evaluations” (p. 2-3).

“Experience in mental health settings among respondents ranged from 0 to 46 years…Respondents worked in a variety of settings, including forensic psychiatric hospitals (5.5%), private practice (14.5%), mental health clinics (9.1%)…respondents in the current study were quite experienced in their respective fields and averaged approximately 18 years of experience in mental health settings…Our respondents reported a very high rate of SRAI use, as nearly all of them indicated that they use an SRAI always or most of the time. Further, the vast majority of respondents were required to use a specific SRAI…SPJ [structured professional judgment] tools – specifically, the HCR-20 [Historical Clinical Risk Management – 20] and the HKT-30/R [Historische, Klinische, Toekomstige – 30] – are the SRAIs most commonly used among Dutch forensic evaluators…Regardless of whether they were required or optional users, evaluators rated the HCR-20 and the HKT-30/R as moderately to highly useful…Nearly half of our respondents reported that they were required to use a psychopathy measure (PCL-R [Psychopathy Checklist – Revised] or PCL:SV [Psychopathy Checklist: Screening Version]) and a substantial majority of these required users rated the measure as very or extremely useful…Interestingly, the majority of our respondents indicated they were required or chose to use an actuarial (rather than SPJ) SRAI – the Static-99/99R – for evaluating the risk of sexual reoffending…” (p. 3-11).

“Respondents to our survey were significantly more concerned about the possibility of a false negative than a false positive outcome in their risk evaluations. This finding suggests that Dutch forensic evaluators, like evaluators elsewhere…may tend to err on the side of caution (i.e., being more averse to the potential outcome of improperly classifying someone as low risk who then reoffends than improperly classifying someone was high risk who would not have reoffended)…We were also interested in how often and with whom our respondents consulted regarding FREs…Our findings confirm that it is common practice for Dutch evaluators to seek advice from others about an FRE always or most of the time. In addition, over 40% of our respondents indicated that they consult with other treatment providers of the evaluee who is the subject of the FRE” (p. 11).

“…our respondents were inclined to agree that an evaluator’s prior beliefs and expectations can affect how they analyze a forensic case and formulate their ultimate opinion, suggesting some awareness of confirmation bias among our respondents…Nevertheless, less than half of our respondents viewed bias as a problem in FREs…Furthermore, a substantial majority of our respondents agreed that a conscious effort to set aside prior beliefs or expectations makes it less likely an evaluator will be influenced by them…Our respondents gave significantly higher effectiveness ratings to the six debiasing strategies with some empirical support than they did to the remaining 19 strategies…” (p. 12).

“…we note that very few strategies to mitigate bias have been empirically tested. Therefore, field and task-specific research regarding the effectiveness of potential strategies to mitigate bias in FREs would be of significant benefit for the discipline, forensic examinees, and legal decision-makers…Given the potential for irrelevant contextual information to introduce bias into FREs…surveying forensic mental health professionals about their views of what information is irrelevant to the FRE task seems to be an important avenue for future research…” (p. 13).

“…Only within the past decade or so have researchers begun to investigate the role that cognitive bias may play in FREs…Furthermore, many potentially effective debiasing techniques have not been empirically tested with respect to forensic evaluations, or specifically FREs. This point is worth noting because not all debiasing strategies are appropriate and/or effective across domains or tasks…Therefore, direct investigations on the effect structured evaluation methods on countering bias in FREs may shed light on the debiasing potential of these methods” (p. 13-14).

“Neal and Brodsky (2016) identified six debiasing strategies that have previously been suggested as potentially effective debiasing techniques for use in forensic evaluations. These strategies are: using structured methods to gather and analyze data…consulting with colleagues…critically examining conclusions (e.g., considering alternative hypotheses…), receiving explicit didactic training about objectivity…taking time to think about the evaluation before writing the report…and reading professional literature about the importance of objectivity…In addition to the six strategies identified by Neal and Brodsky (2016) as having been suggested in scientific literature, we think there are at least four others…that have been suggested as potentially effective in countering bias specific to forensic evaluations. These four strategies are: investigating all relevant data before forming an opinion…basing conclusions and opinions on sound data…taking careful notes during an evaluation…and examining patterns of personal decision-making…” (p.12).

As always, please join the discussion below if you have thoughts or comments to add!