The link between risk assessment and management: Not as straightforward as it seems

Jodi L. Viljoen, Simon Fraser University
Dana M. Cochrane, Simon Fraser University
Melissa R. Johnson, Simon Fraser University

Although it is widely believed that risk assessment tools can help manage risk of violence and offending, it is unclear what evidence exists to support this view. As such, we conducted a systematic review and narrative synthesis. To identify studies, we searched 13 databases, reviewed reference lists, and contacted experts. Through this review, we identified 73 published and unpublished studies (N = 31,551 psychiatric patients and offenders, N = 10,002 professionals) that examined either professionals’ risk management efforts following the use of a tool, or rates of violence or offending following the implementation of a tool. These studies included a variety of populations (e.g., adults, adolescents), tools, and study designs. The primary findings were as follows: (a) despite some promising findings, professionals do not consistently adhere to tools or apply them to guide their risk management efforts; (b) following the use of a tool, match to the risk principle is moderate and match to the needs principle is limited, as many needs remained unaddressed; (c) there is insufficient evidence to conclude that tools directly reduce violence or reoffending, as findings are mixed; and (d) tools appear to have a more beneficial impact on risk management when agencies use careful implementation procedures and provide staff with training and guidelines related to risk management. In sum, although risk assessment tools may be an important starting point, they do not guarantee effective treatment or risk management. However, certain strategies may bolster their utility.

systematic review, risk assessment, violence, offending, risk management

“In the past several decades, researchers have developed over 400 different tools designed to assess risk of violence and offending. Professionals, such as psychologists, probation officers, nurses, psychiatrists, and police, have widely adopted these tools in as many as 44 countries. In addition, administrators and policymakers have created policies and, in some cases, laws mandating the use of tools. […] Risk assessment tools are also commonly used in forensic psychiatric facilities, general psychiatric hospitals, treatment programs, correctional centers, and in a variety of court evaluations, such as those involving the transfer of adolescents to adult court and the civil commitment of individuals who have sexually offended.” (p. 181)

“Broadly speaking, risk management refers to the process of planning and implementing strategies to help prevent violence and other forms of offending. It is carried out by a variety of professionals (e.g., psychologists, probation officers, nurses, police) and encompasses not only treatment (e.g., therapy), but also strategies such as supervision, case management, and placement decisions. […] There is widespread agreement, among risk assessment researchers, that risk management is a key goal of risk assessment. […] As a result, the field of risk assessment has evolved to focus increasingly on the management and reduction of risk, rather than solely its prediction. […] However, even though many risk assessment tools aim to help manage and reduce risk, it is unclear whether tools do, in fact, achieve this goal.” (p. 182)

“Although the mechanism between risk assessment and risk management is not especially well-articulated, the primary hypothesized mechanism appears to be twofold. First, it is thought that tools will increase professionals’ level of adherence to the risk-need-responsivity (RNR) model of offender treatment. Second, it is believed that this will, in turn, decrease offending. […] [Risk assessment] tools are viewed as a means by which to deploy interventions that are evidence-based and individually tailored, thereby avoiding a “one size fits all” approach.” (p. 182–182)

“Although many researchers believe that risk assessment tools can help manage and reduce risk, some researchers have noted that the impact of tools may be contingent upon other factors, such as the type of tool and whether the tool is followed through on to match individuals to appropriate interventions. Whereas some tools focus primarily on historical or static factors, such as history of offending, other tools include dynamic or modifiable factors, such as anger management difficulties (i.e., criminogenic needs). In addition, on some tools (i.e., actuarial tools), evaluators add up risk factors to generate a total score. On other tools (i.e., structured professional judgment tools), evaluators use their discretion to make a separate summary risk rating after considering the items and additional case-specific considerations. As a result of these variations, tools may differ in their utility for various risk management decisions.” (p. 183)

“Besides variation in item content, risk assessment tools also vary in terms of their validity or their ability to predict reoffending. Some tools have been found to significantly predict reoffending in multiple studies, with effect sizes falling in the moderate range. However, other risk assessment tools have poor predictive validity, or have not yet been tested at all. Presumably, unless a tool has adequate validity in predicting reoffending, it will have limited value for risk management.” (p. 183)

“Finally, tools vary in the extent to which they include a focus on risk management. Whereas most risk assessment tools do not include explicit instructions or guidance regarding how to manage risk, some tools aim to bridge risk assessment to risk management by providing greater structure and support for risk management, such as by including case management planning forms.” (p. 183)

“Not only does the utility of risk assessment depend on the nature of the tool, it could also depend on how professionals use and apply tools. In particular, as is true of any type of assessment, the value of risk assessment likely lies primarily in what happens after the assessment. Although risk assessment tools may be a starting point for treatment and risk management, they are not a treatment in and of themselves. As such, it may be unrealistic to expect that using a tool will help manage risk or reduce reoffending unless (a) appropriate treatments are, in fact, available, and (b) professionals meaningfully apply tools to match individuals to these treatments.” (p. 183)

“Despite the debate about whether some tools may be better suited for risk management than others, most risk assessment researchers appear to believe that tools can, in principle, aid in risk management. However, this viewpoint is not embraced by everyone. Some critics have expressed concern that risk assessment tools, in general, are not only ineffective in managing risk, they might even cause harm to patients and offenders. […] Furthermore, some critics have questioned the motives that underlie the adoption of risk assessment tools. […] Finally, critics have pointed out that despite claims that risk assessment tools help to manage risk, there is little evidence to support such assertions.” (p. 183–184)

“In sum, opinions about the value of risk assessment tools for risk management efforts range considerably. […] It is concerning that many assertions about the utility of tools (or lack of utility) have been offered without reference to research findings. One possible explanation for the lack of empirical grounding for these assertions is that relevant research simply does not exist, as of yet. However, another possibility is that existing research has not yet been adequately integrated into the literature due to a lack of comprehensive reviews.” (p. 184)

“To our knowledge, only one systematic review has examined the utility of risk assessment tools for risk management, and this review was not designed to focus on risk assessment per se. […] The goal of the present systematic review was to expand on [that] review. First, rather than focusing exclusively on psychiatric patients in acute care settings, we included a range of populations (e.g., patients, offenders) and settings (e.g., jails, forensic hospitals). Second, rather than restricting our review to RCTs, our review encompassed a variety of designs (e.g., RCTs, prepost studies). […] Third, instead of solely examining whether the use of risk assessment tools reduce violence, we also investigated their impact on professional practices. […] As such, we reviewed research on whether tools facilitate professionals’ adherence to the risk and need principles, as these are the hypothesized mechanism by which tools might help manage risk. We also examined whether professionals perceive tools as useful for risk management and whether they use tools to guide their risk management effort; tools are unlikely to be effective if professionals do not apply them or view them as useful. […] Our goal was to understand, more thoroughly, the pathway between risk assessment and risk management. Furthermore, to develop an agenda for future research, we reviewed studies on strategies to enhance the utility of risk assessment tools for risk management, such as staff training.” (p. 184)

“To examine our research questions, we chose to conduct a systematic review rather than a traditional literature review because systematic reviews are more transparent, comprehensive, and objective. […] To synthesize findings, we used a narrative approach. An empirical synthesis (i.e., meta-analysis) was neither feasible nor appropriate because our review included a wide range of designs (e.g., RCTs, surveys), populations (e.g., offenders, patients), tools (i.e., 34 different risk assessment tools), and outcomes.” (p. 184–185)

“In total, 73 studies met inclusion criteria. Sixteen of these studies were unpublished; nine of the unpublished studies were dissertations or theses and the remaining seven were reports by researchers, government, or other organizations. These studies included 31,551 offenders or patients, 10,002 professionals, and 34 risk assessment tools. Most tools included dynamic or modifiable factors (i.e., criminogenic needs; 76.5%, k = 26), and were validated (i.e., have been found to significantly predict violence or reoffending; 82.4%, k = 28).” (p. 187)

“If professionals do not “buy-in” to tools or perceive them as useful, they may not adequately utilize them. Thus, as an initial step, we examined professionals’ attitudes toward risk assessment tools. We found that, although some professionals held positive views about tools, in many studies, professionals had mixed views about the utility of tools for risk management (e.g., treatment planning, placement decisions). This is not particularly surprising; professionals often feel reluctant to adopt new assessment and intervention approaches even when these approaches have strong research support. Furthermore, manuals and training for risk assessment tools often focus on how to complete item ratings rather than how to apply the tool to risk management efforts. As such, professionals’ questions about the utility of tools may be understandable.” (p. 203)

“Not only did professionals have mixed views about the utility of tools for risk management, in most of the identified studies, the use of risk assessment tools for risk management was mixed. Specifically, although some professionals reported that they relied on tools to guide their risk management decisions (e.g., decisions about services or placements), others reported that they did not use tools, even when employers mandated their use. As such, these findings illustrate that risk assessments do not necessarily flow through to risk management efforts. Slippage might be more likely to occur when risk assessors do not have direct control over risk management decisions, but instead act as intermediaries to decision-makers (e.g., judges). In such cases, the application of tools to risk management may depend not only on evaluators’ use of tools, but also on whether subsequent decision-makers perceive tools as useful and relevant.” (p. 203)

“Despite the mixed application of tools to risk management efforts overall, match to the risk principle was moderate following the use of risk assessments tools. In a number of studies, high-risk individuals were referred to more services than low-risk individuals. They were also more likely to receive secure placements. However, most studies did not have a comparison group of individuals who did not receive risk assessments. As such, it is difficult to determine if such findings are attributable to the use of the tool; some research suggests that high-risk individuals receive more intensive risk management strategies than low-risk individuals even when a risk assessment tool is not used.” (p.203)

“Contrary to the positive findings relating to the risk principle, match to the need principle appeared limited following the use of risk assessment tools (match was rated as mixed or low in all but one study). This means that many of offenders’ and patients’ needs remained unaddressed even when risk assessment tools were used. This could indicate that professionals are not paying adequate attention to risk assessments when they are making decisions about services. Alternatively, these low rates of overall match could occur because professionals opt to focus on only a couple “high impact” needs at a given time, as it may not be feasible to simultaneously target all needs. Another possibility is that low rates of match occur because services to address needs are simply not available. Clearly, identifying needs has limited value if there are no viable means by which to address these needs. Finally, low match to the needs principle may, in part, arise from limited compliance; offenders and patients may not necessarily attend or engage in the services to which they are referred.” (p. 203–204)

“In light of the preceding findings, it is perhaps not surprising that evidence on whether tools reduce violence and offending was inconsistent. Although two RCTs found that the use of the BVC resulted in decreases in violence, another RCT did not find significant changes in violence or other criminal incidents when another tool, the START, was implemented. In addition, although two pre-post studies found that the implementation of risk assessment tools was associated with decreases in violence or offending, the bulk of pre-post studies (k _ 7) did not. Thus, at present, there is insufficient evidence to conclude that tools reduce violence or offending. One possible explanation for these findings is that it may be unrealistic to expect risk assessment tools to directly reduce violence or offending. […] Another possible explanation for the modest findings is that the effectiveness of tools might vary by factors such as the setting, population, or tool. […] It is also possible that some tools may be more effective than other tools.” (p. 204)

“Overall, our results suggest that there is a disconnect between the theory of risk assessment and what actually happens in real-world practice. However, although research is limited, preliminary evidence suggests that it may be possible to enhance professionals’ risk management practices by combining the use of risk assessment tools with approaches such as risk management training and structured risk management guidelines. Though such approaches may not directly reduce violence and offending, they have been found to improve match to the risk and need principles, thus providing potential avenues by which to enhance the utility of tools. Sound implementation practices, such as policies and protocols to guide the use of tools, are also critical” (p. 204)

“Overall, this review suggests that even though risk assessment tools may be a starting point for risk management, they are not sufficient in and of themselves. Although some studies found positive results, indicating that tools might help achieve better match to the risk principle or even reductions in violence in some circumstances, the findings also revealed that “there is no guarantee that the results of these protocols flow through to front line service provision”. […] To ensure that risk assessment instruments are optimally used and do not degenerate into merely a bureaucratic exercise further efforts are needed. In particular, rather than focusing exclusively on predictive validity studies and the development of new tools, researchers need to pay greater attention to how tools are applied to guide real-world decisions, such as by testing the pathways between risk assessment and risk management, identifying areas of slippage, and developing strategies to facilitate the ability of risk assessments to translate into better risk management efforts.” (p. 205)

“Our findings suggest that risk assessment tools are not sufficient to guarantee sound risk management practices or reductions in violence. Thus, researchers and tool developers should be careful to not overstate the potential value of risk assessment for risk management. Likewise, professionals, agencies, and policymakers should not rely on risk assessment tools as their sole or primary risk management strategy. Instead, agencies who use risk assessment tools should work to build staff buy-in, regularly monitor adherence, and ensure that they are providing effective treatment, rather than funneling all their resources into assessment.” (p. 204)

“That said, even though risk assessment tools have limitations, they remain a best available practice. Although tools may not reduce violence or offending in and of themselves, there is no evidence that alternative approaches, such as assessing risk via unstructured clinical judgment or not assessing risk at all, do so either.” (p. 204)

“To ensure that our systematic review met best practice standards and followed relevant reporting guidelines, we followed the criteria set forth in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and the Assessment of Multiple Systematic Reviews (AMSTAR) tool.” (p. 185)

“Studies were required to meet the following inclusion criteria: (a) empirical study that was published or disseminated in English; (b) included a sample of individuals who were assessed with a structured risk assessment tool in real-world practice and/or a sample of professionals who used risk assessment tools in practice; and (c) included an outcome relevant to at least one of our research questions (e.g., perceived utility, adherence to the risk principle). We defined structured risk assessment tools as tools that included a designated list of risk factors and an overall rating of risk level for violence or offending. Thus, we did not include measures of psychopathy. Also, we did not include qualitative research, as systematic reviews of qualitative studies use different methodologies, nor did we include research on the responsivity principle, as this principle encompasses a wide range of constructs (i.e., culture, trauma, mental health) and is not as well-researched as the risk and need principles. When disseminations were based on the same sample, we selected the study that was the most comprehensive and rigorous (e.g., largest sample).” (p. 185)

“In interpreting our findings, several limitations of this review are important to note. Although we systematically searched 13 databases, reviewed reference lists, and contacted experts, our review likely missed some studies, such as studies written in languages other than English. In addition, although we attempted to summarize our findings with terms such as low, mixed, or high, definitions of such terms are somewhat subjective by nature. As such, to increase transparency and objectivity, we provided operational definitions of our summary terms, and summarized study findings in more detail using evidence tables. In addition, two independent raters coded each study and we conducted consensus ratings. Another limitation of this systematic review is that there is a lack of appropriate tools for appraising risk of bias in risk assessment studies. As such, we drew items from other tools, and adapted the wording for this context. However, the approach that we used to appraise observational and survey studies was brief and, as such, our review likely failed to capture some relevant study limitations. Finally, although we examined differences in general patterns of results across published and unpublished studies, it is difficult to evaluate publication bias in narrative reviews.” (p. 204)

“On the basis of this review, there are number of important areas for future research. First, many studies have lacked appropriate comparison groups, making it impossible to determine if tools improve practices per se. As such, there is a strong need for further research with comparison groups of individuals who were not assessed with a risk assessment tool, including studies with both mental health and justice populations. Second, to determine if certain tools may have a more beneficial impact on risk management than other tools, head-to-head comparisons of tools are needed. Third, adherence to risk assessment tools appears to be poor in some cases, making it difficult to evaluate the impact of tools. As such, research should routinely measure and report adherence. Fourth, given that the utility of tools for risk management likely depends heavily on what happens after the risk assessment, researchers examine the pathway between risk assessment and risk management, such as by developing and testing conceptual models. Finally, researchers should create and evaluate approaches to improve the utility of tools for risk management, such as training initiatives, structured risk management guidelines, and quality improvement or audit systems.” (p. 204–205)

As always, please join the discussion below if you have thoughts or comments to add!