Mixed Findings on the SAVRY’s and YLS/CMI’s Sensitivity to Short-term Changes in Risk - CONCEPT Professional Training

Jodi L. Viljoen, Simon Fraser University
Catherine S. Shaffer, Simon Fraser University
Andrew L. Gray, Simon Fraser University
Kevin S. Douglas, Simon Fraser University

Although many adolescent risk assessment tools include an emphasis on dynamic factors, little research has examined the extent to which these tools are capable of measuring change. In this article, we outline a framework to evaluate a tool’s capacity to measure change. This framework includes the following: (a) measurement error and reliable change, and (b) sensitivity (i.e., internal, external, and relative sensitivity). We then used this framework to evaluate the Structured Assessment of Violence Risk in Youth (SAVRY) and Youth Level of Service/Case Management Inventory (YLS/CMI). Research assistants conducted 509 risk assessments with 146 adolescents on probation (101 male, 45 female), who were assessed every 3 months over a 1-year period. Internal sensitivity (i.e., change over time) was partially supported in that a modest proportion of youth showed reliable changes over the 3-, 6-, and 12-month follow-ups. External sensitivity (i.e., the association between change scores and reoffending) was also partially supported. In particular, 22% of the associations between change scores and any and violent reoffending were significant at a 6-month follow-up. However, only 1 change score (i.e., peer associations) remained significant after the Bonferroni correction was applied. Finally, relative sensitivity was not supported, as the SAVRY and YLS/CMI was not more dynamic than the Psychopathy Checklist: Youth Version (PCL:YV). Specifically, the 1-year rank-order stability coefficients for the SAVRY, YLS/CMI, and PCL:YV Total Scores were .78, .75, and .76, respectively. Although the SAVRY and YLS/CMI hold promise, further efforts may help to enhance sensitivity to short-term changes in risk.

adolescence, dynamic risk factors, offending, risk assessment, violence

“Risk assessment tools for violence and offending have gained widespread use (Singh, Desmarais, Hurducas, et al., 2014). Considerable research has shown that these tools can predict subsequent convictions with moderate effect sizes (Skeem & Monahan, 2011; Yang, Wong, & Coid, 2010). However, little research has examined the extent to which risk assessment tools are able to assess changes in risk (Douglas & Skeem, 2005). Measuring change is important, because it might help professionals to better predict and prevent reoffending” (p. 244).

“A primary challenge in measuring change is disentangling real change from measurement error (Jacobson & Truax, 1991; Table 1). For instance, if a youth changes a few points on a risk assessment tool, this may not be reflective of true change. Instead, it might stem from imperfect interrater reliability. As such, in evaluating the ability of a tool to measure changes in risk, an important starting point is to consider measurement error and the extent to which change is reliable (Riddle & Stratford, 2013)… For instance, no published research has investigated reliable change on the YLS/CMI, and only one study, to our knowledge, has examined reliable change on the SAVRY (Viljoen et al., 2015). The authors of that study reported that, after taking into account measurement error, a youth had to show a change of 7 to 8 points on the SAVRY Risk Total Score to be able to confidently classify this change as reliable” (p. 245).

“In evaluating a tool’s ability to measure change, another important criterion is sensitivity to change (Table 1)… According to one model, there are two forms of sensitivity to change, namely internal and external (Husted et al., 2000). Internal sensitivity refers to the “ability of a measure to change over a particular pre-specified time frame” (Husted et al., 2000, p. 459). Oftentimes, internal sensitivity is examined by evaluating the extent to which a tool differentiates individuals who have received various levels of treatment. For instance, if youth who received
services showed greater reductions in SAVRY and YLS/CMI scores than youth who did not receive services, it would suggest that these tools show internal sensitivity to change…Whereas internal sensitivity to change focuses on within individual or group-level change over time, external sensitivity to change refers to “the extent to which changes in a measure over a specified time frame relate to corresponding changes in a reference measure” (Husted et al., 2000, p. 459)” (p. 245).

“Despite growing research on adult tools’ sensitivity to change, only a small number of studies have examined sensitivity to change in widely used adolescent risk assessment tools, such as the SAVRY and YLS/CMI… In the current study, we used the above-described framework to extend research on the ability of the SAVRY and YLS/CMI to measure change. First, we investigated measurement error and reliability of change. Second, we tested sensitivity to change, including the following: (a) internal sensitivity (i.e., the extent to which SAVRY and YLS/CMI scores changed over time); (b) external sensitivity (i.e., whether adolescents who showed decreases in risk scores were less likely to reoffend); and (c) relative sensitivity (i.e., whether the SAVRY and YLS/CMI detected more change than a measure of psychopathic features)” (p. 246).

“Participants included 146 youth on community probation in a large city in Western Canada. All participants had been assessed by research assistants (RAs) on at least two occasions…This study was conducted as part of a larger study on risk assessment (Viljoen et al., 2016). Although that study also includes a 9-month reassessment, we focused on the baseline, 3-, 6-, and 12-month follow-ups, as researchers have recommended reassessments at these time points (Viljoen, Cruise, et al., 2012; Vincent et al., 2012)… For each assessment, RAs conducted a standardized interview with the youth at a probation office or a quiet public place (e.g., coffee shop) and then examined youths’ justice records prior to rating the SAVRY, YLS/CMI, and PCL:YV” (p. 247).

Results

“Overall, level of measurement error appeared acceptable. Specifically, youth’s score had to have increased or decreased by 7 points on the SAVRY Risk Total Score (14% of the maximum possible score), and 9 points on the YLS/CMI Risk Total Score (21% of the maximum possible score) in order to conclude that the change was reliable. This is similar to previous research on the SAVRY (Viljoen et al., 2015) and on adult risk assessment tools (Draycott et al., 2012). Thus, as is expected, small changes in SAVRY and YLS/CMI scores may reflect measurement error rather than true change” (p. 253).

“Internal sensitivity to change (i.e., the ability to detect change over time) was partially supported. Some youth showed reliable increases or decreases in SAVRY and YLS/CMI scores across the follow-up periods. Specifically, at the 12-month follow-up, 8% to 22% of youth showed reliable change on SAVRY and YLS/CMI Risk Total Scores. However, rates of short-term change were more modest than expected” (p. 253). One possibility is that the youth in our sample truly were not demonstrating very much change…Another possibility is that youth were, in fact, changing but the tools did not fully detect these changes” (p. 253).

“External sensitivity to change (i.e., associations with an external criterion, namely reoffending) was, again, partially supported. We found some significant associations between change scores and reoffending. For instance, youth who showed decreased risk in Peer Associations were less likely to engage in any reoffending. However, in most cases (i.e., 78% of the analyses), the associations between change scores and reoffending were not significant. For example, changes in summary risk ratings did not significantly predict reoffending. Also, only 2% of the analyses remained significant after a Bonferroni correction was made for the large number of comparisons. Given that prior research is mixed (i.e., Clarke et al., 2016; Viljoen et al., 2015), the SAVRY’s and YLS/CMI’s external sensitivity to change may vary depending on the context in which tools are used (e.g., probation vs. treatment settings)…. Finally, with respect to relative sensitivity, the SAVRY showed fairly similar sensitivity to change as the YLS/CMI. However, contrary to expectations, the putatively ‘dynamic’ scales on the SAVRY and YLS/CMI generally did not appear to be more
dynamic than the historical factors scales. Also, contrary to expectations, neither tool was any more dynamic than the PCL:YV” (p. 254).

“Researchers should continue to investigate sensitivity to change for the SAVRY, YLS/CMI, and other tools (e.g., Violence Risk Scale: Youth Version [Wong, Lewis, Stockdale, & Gordon, 2011], the Risk-Sophistication-Treatment Inventory [Salekin, 2004], Short-Term Assessment of Risk and Treatability: Adolescent Version [Viljoen, Nicholls, Cruise, Desmarais, & Webster, 2014]). However, given that measuring change may be difficult, researchers should also identify approaches by which to further improve tools’ sensitivity to change (see Table 9 for a list of potential strategies). Finally, it will be important to investigate if assessing changes in risk holds clinical utility, such as whether it enhances professionals’ ability to plan treatment. Ultimately, such efforts may help move the field of risk assessment beyond prediction, and closer to effective risk management and prevention” (p. 254).

“Given that most participants were male, it was difficult to draw conclusions about gender differences. Moreover, although we compared sensitivity to change for youth from ethnic minority and nonminority groups, we were unable to conduct more refined analyses on any one particular ethnic group (e.g., Indigenous, Southeast Asian), given the small sample sizes. However, this is an important area for future research, especially as tools may not necessarily function equally across groups (Gutierrez, Wilson, Rugge, & Bonta, 2013; Shepherd, Adams, McEntyre, & Walker, 2014)” (p. 254).

As always, please join the discussion below if you have thoughts or comments to add!