community corrections risk assessment validation study ... · 2, 2015. as part of the final report...
TRANSCRIPT
Community Corrections Risk Assessment Validation Study
Examining the Validity and Reliability of the Field Reassessment Offender Screening Tool (FROST)
Supplemental Report
Prepared for the Arizona Department of Corrections – Community Corrections Bureau by:
Edward Latessa, Ph.D. Principal Investigator
&
Jennifer Lux, M.S. Project Director
University of Cincinnati Corrections Institute Center for Criminal Justice Research
Submitted on: November 20, 2015
2
TABLE OF CONTENTS
LIST OF TABLES & FIGURES .................................................................................................... 3 INTRODUCTION .......................................................................................................................... 4 IMPROVING THE VALIDITY & RELIABILITY OF THE FROST ........................................... 4
Recommendation #1 .................................................................................................................... 5
Recommendation #2 .................................................................................................................... 5
Recommendation #3 .................................................................................................................. 16
Recommendation #4 .................................................................................................................. 20
Recommendation #5 .................................................................................................................. 21
CONCLUSION ............................................................................................................................. 21
3
LIST OF TABLES & FIGURES Table 1. Items Removed from the Original FROST Instrument .................................................... 6 Table 2. Risk Level Distribution for the Overall Sample & Male & Female Samples — Original versus New Instruments .................................................................................................................. 7 Table 3. Original versus New Cut-Off Scores ................................................................................ 9 Figure 1. Rates of Recidivism based on Original & New Cut-Off Scores for Overall Sample by Risk Level ..................................................................................................................................... 10 Figure 2. Rates of Recidivism based on Original & New Cut-Off Scores for Males by Risk Level ....................................................................................................................................................... 11 Figure 3. Rates of Recidivism based on Original & New Cut-Off Scores for Females by Risk Level ............................................................................................................................................. 12 Table 4. Internal Reliability Coefficients for the Overall Sample & Male & Female Samples – Original versus New Instruments .................................................................................................. 13 Table 5. Correlations for FROST Total Score & Recidivism for the Overall Sample – Original vs. New Instruments ...................................................................................................................... 14 Figure 4. ROC Curve for Total Risk Score & Reentry to System for Overall Sample – New Tool ....................................................................................................................................................... 15 Figure 5. ROC Curve for Total Risk Score & Reentry to System for Males – New Tool ........... 15 Figure 6. ROC Curve for Total Risk Score & Reentry to System for Males – New Tool ........... 16 Table 6. Risk Level Distribution for the Sex Offender Sample — Original versus New Instruments .................................................................................................................................... 17 Figure 7. Rates of Recidivism based on Original & New Cut-Off Scores for Sex Offenders by Risk Level ..................................................................................................................................... 18 Table 7. Internal Reliability Coefficients for the Sex Offender Sample – Original versus New Instruments .................................................................................................................................... 19 Figure 8. ROC Curve for Total Risk Score & Reentry to System for Males – New Tool ........... 20
4
INTRODUCTION
The University of Cincinnati Corrections Institute (UCCI) contracted with the Arizona
Department of Corrections (ADC) in January 2014 to evaluate the validity, reliability, and
efficacy of the Field Reassessment Offender Screening Tool (FROST) for its Community
Corrections Bureau. Specifically, the study was designed to meet the following goals:
(1) Assess the inter-rater agreement of the FROST;
(2) Assess the internal reliability of the FROST;
(3) Assess the predictive validity of the FROST; and
(4) Evaluate the internal reliability and the predictive validity of the FROST specifically for sex offenders.
When appropriate, the validity, reliability, and efficacy of the FROST were explored by different
subgroups, including sex and ethnicity.
A final report describing the methods used to complete the study, as well as the findings
associated with each of the aforementioned project goals, was submitted to the ADC on October
2, 2015. As part of the final report and final presentation to the ADC, UCCI made several
recommendations for improving the reliability and predictive accuracy of the FROST. The
current supplemental report outlines the changes the ADC is making in response to UCCI’s
recommendations.
IMPROVING THE VALIDITY & RELIABILITY OF THE FROST
Based on the results of the study, UCCI made five recommendations to the ADC for
improving the validity and reliability of the FROST. The following subsection outlines each of
these recommendations and reviews the specific changes the ADC is making in response to these
recommendations.
5
Recommendation #1
Related to inter-rater reliability, UCCI recommended that the ADC develop a more
structured interview guide to include more guidance and scoring criteria for assessors. These
changes will ensure that all staff completing FROST assessments understand the purpose of each
item and are scoring out the tool in a consistent manner. The ADC accepted this
recommendation and has already begun adapting and making changes to the FROST materials.
Recommendation #2
Related to improving internal reliability and validity, UCCI recommended that the ADC
reconfigure the FROST by:
(1) Combining some domains;
(2) Eliminating other domains;
(3) Removing individual items;
(4) Adjusting cut-off scores so that more cases (i.e., offenders) fall into the low-risk and high-risk categories; and/or
(5) Creating separate risk-level categories for males and females.
UCCI provided three different options to the ADC for making these improvements.1 Ultimately,
the ADC chose Option #2, which is described further below.
Combining Some Domains, Eliminating Other Domains, & Removing Individual
Items. Three steps were taken to adjust domains and individual items in the FROST instrument.
First, the new FROST instrument consists of 23 items across three domains, including: (1)
General Needs, (2) Attitude, and (3) Criminal Behavior. The following domains were combined
to create the General Needs section: (1) Vocational/Financial, (2) Education, (3) Family/Social
Relationships, (4) Drug Abuse, and (5) Mental Health. Second, all items from three different
1 Please see the final Power Point presentation made to the ADC on October 1, 2015 for a review of all three options.
6
domains were eliminated completely: (1) Physical Health/Medical (2 items); (2) Residence and
Neighborhood (2 items); and (3) Alcohol (3 items). Third, items that were significantly related
to recidivism or consistently found to be non-significantly related to recidivism across all
outcome measures were also eliminated. More specifically, items were removed if they were
found to decrease reliability (i.e., if the item was removed, reliability actually increased) or had
correlations below 0.03. Table 1 provides a summary of the 14 items that were removed from the
original FROST instrument. Altogether with the seven items removed across the three full
domains and the 14 items removed across other domain areas, a total of 21 items were removed
from the original instrument.
Table 1 Items Removed from the Original FROST Instrument
Domain Item #/Item Name
II. Vocational/Financial 3. Fired or asked to resign from a job within the last 6 months 5. Receives regular support through financial assistance
III. Education 1. Level of formal education 3. Reading ability
IV. Family/Social Relationships
4. What is the status of the current family relationship? 6. What best describes client’s current most significant companion/romantic relationship? 7. What best describes the client’s current peer and/or associate relationships? 8. Does the client associate with at least one pro-social person on a regular basis?
VII. Drug Abuse 3. Client’s self-perception VIII. Mental Health 2. Current mental health functioning
IX. Attitude 3. Does client have a poor attitude about his/her current conviction? 4. Does client have a poor attitude about community supervision?
X. Criminal Behavior 5. Present offense designation 7. Is the present offense violent?
7
Adjusting Cut-Off Scores. While the original FROST instrument placed offenders into
low-, moderate-, and high-risk categories effectively, results indicated that the majority of
offenders were deemed moderate-risk to reoffend. As such, it was suggested that cut-off scores
be adjusted and a new category (moderate/high) be created for males, so that more offenders
would fall into the lower and higher ends of the distribution. Data presenting the new
distribution, compared to the original distribution for the overall sample and male and female
samples are presented in Table 2.
Table 2 Risk Level Distribution for the Overall Sample & Male & Female Samples — Original versus New Instruments
Overall Sample (N = 18,020)
Original New
N % N %
Low-Risk 2,573 14.3 6,449 35.8
Moderate-Risk 14,738 81.8 7,306 40.5
Moderate/High-Risk -- -- 3,094 17.2
High-Risk 709 3.9 1,171 6.5
Male Sample (N = 15,173)
Original New
N % N %
Low-Risk 2,062 13.6 5,193 34.2
Moderate-Risk 12,443 82.0 6,177 40.7
Moderate/High-Risk -- -- 2,632 17.3
High-Risk 668 4.4 1,171 7.7
8
Table 2 Risk Level Distribution for the Overall Sample & Male & Female Samples — Original versus New Instruments
Female Sample (N = 2,847)
Original New
N % N %
Low-Risk 511 18.0 1,256 44.1
Moderate-Risk 2,295 80.6 1,129 39.7
High-Risk 41 1.4 462 16.2 As can be seen, the percentage of offenders falling into the moderate-risk category is
considerably lower for the new FROST instrument, compared to the original instrument for the
overall sample (approximately 41% and 82%, respectively). In this way, more offenders now
fall into the low-risk (approximately 36% for the new tool versus 14% for the original tool) and
high-risk (approximately 7% for the new tool versus 4% for the original tool) categories.
Distributions for both the male and female samples follow a similar pattern.
Creating Separate Risk-Level Categories for Males & Females. Different risk-level
categories were also created for males and females, as study findings showed that males’ levels
of risk were inflating females’ levels of risk. More specifically, the study showed that no matter
how risky female offenders were, their scores were never as risky as their male counterparts. In
this way, women may have been over-supervised and/or over-treated in the past.
Table 3 below presents the new cut-off scores for males and females, compared to the
original cut-off scores for each subgroup. As can be seen, both males and female offenders who
score between a 0 and 6 on the new FROST instrument are low-risk, while offenders who score
between 7 and 10 are moderate-risk for both subgroups. Males who score between an 11 and a
13 are now considered moderate/high risk, while males who score a 14 and higher are considered
9
high-risk. Females on the other hand, do not have a moderate/high-risk category, scoring high-
risk if they receive a score of 11 or higher.
Table 3 Original versus New Cut-Off Scores
Risk-Level Original Cut-Off Scores
New Cut-Off Scores – Males
New Cut-Off Scores – Females
Low 0 – 6 0 – 6 0 – 6 Moderate 7 – 20 7 – 10 7 – 10 Moderate/High -- 11 – 13 N/A High 21 – 44 14+ 11+
Figure 1 below presents offenders’ rates of recidivism based on the original and new cut-
off scores for the overall sample, while Figures 2 and 3 present rates of recidivism based on the
original and new cut-off scores for males and females, respectively. As can be seen in Figure 2,
rates of recidivism continue to steadily increase as each level of risk increases. With the new
cut-off scores, however, a larger number of offenders fall into the lower and higher ends of the
distribution (as discussed in the subsection above), with rates of recidivism remaining relatively
the same at each level of risk. For example, based on the original cut-off scores, approximately
17 percent of offenders who fell into the low-risk category recidivated, while 19 percent of the
offenders who now fall into the low-risk category based on the new cut-off scores, recidivated
(only about a 2% difference). Similarly, more offenders now fall into the high-risk category
based on the new cut-off scores, but about 2 percent less than the offenders falling into the
original high-risk category actually recidivated. This is likely due to the newly created
moderate/high-risk category.
10
Figure 1 Rates of Recidivism based on Original & New Cut-Off Scores for Overall Sample by Risk Level
Rates of recidivism increase as each level of risk increases for the new male cut-off
scores as well. As can be seen from Figure 2, rates of recidivism for low-risk males increased
slightly (about 3%) between the original low-risk male group and the new low-risk male group;
however, as shown in the subsection above, an estimated 3,000 more offenders now fall into the
low-risk category. Interestingly, rates of recidivism stayed the same between the original and
new moderate-risk male group (approximately 40%), even though this group now includes
almost 6,300 less offenders than the original moderate-risk group. Finally, with the creation of
the new moderate/high-risk category, rates of recidivism for the high-risk male group decreased
slightly between the original and new cut-off scores (approximately 53% and 51%, respectively).
16.6 19.0
31.0 31.1
0
41.1
52.8 51.4
0
10
20
30
40
50
60
Original New
% R
etur
n to
Sys
tem
Low
Moderate
Moderate/High
High
11
Figure 2 Rates of Recidivism based on Original & New Cut-Off Scores for Males by Risk Level
Consistent with the discussion above, rates of recidivism increase as each level of risk
increases for the new cut-off scores for females as well. As can be seen from Figure 3, rates of
recidivism for low-risk females increased by approximately 1 percent between the original low-
risk female group and the new low-risk female group, even after placing almost 750 more female
offenders into the low-risk category. Rates of recidivism stayed the same between the original
and new tool for moderate-risk female offenders (21%) after placing almost 1,200 female
offenders into the lower and higher ends of the distribution. Finally, female offenders who are
now considered high-risk based on the new cut-off scores, recidivated approximately 10 percent
less than their original high-risk female counterparts, even after adding almost 430 females to the
high-risk group.
17.5 20.1
32.9 32.9
0
43.0
53.4 51.4
0
10
20
30
40
50
60
Original New
% R
etur
n to
Sys
tem
Low
Moderate
Moderate/High
High
12
Figure 3 Rates of Recidivism based on Original & New Cut-Off Scores for Females by Risk Level
Table 4 presents the alpha reliability coefficients for the overall sample and also for the
male and female samples for the original FROST instrument compared to the new FROST
instrument. Originally, internal consistency across the Vocational/Financial, Education,
Family/Social Relationships, Drug Abuse, and Mental Health domains ranged from 0.20 to 0.50
for the overall sample and 0.19 to 0.50 and 0.29 and 0.48 for the male and female samples,
respectively. After removing several items and combining the remaining factors into the General
Needs section, however, it can be seen that reliability is improved slightly, yielding values of
0.52 for the overall sample, 0.53 for the male sample, and 0.49 for the female sample. It is
important to note that reliability coefficients for the Attitude and Criminal Behavior domains
remained relatively similar for the overall sample and the male and female samples when
comparing the original FROST instrument to the new FROST instrument. Overall, internal
consistency for the new FROST instrument (with almost half as many items compared to the
12.9 14.3
21.0 21.0
41.5
31.8
0
5
10
15
20
25
30
35
40
45
Original New
% R
etur
n to
Sys
tem
Low
Moderate
High
13
original tool) is relatively high, yielding alpha coefficients of 0.72 for the overall sample, as well
as the male sample, and 0.71 for the female sample.
Table 4 Internal Reliability Coefficients for the Overall Sample & Male & Female Samples – Original versus New Instruments
Original Instrument
Domain Name Overall Males Females I. Physical Health/Medical
0.48 0.49 0.43
II. Vocational/Financial 0.32 0.33 0.29 III. Education 0.20 0.19 0.30 IV. Family & Social Relationships 0.45 0.45 0.44 V. Residence & Neighborhood 0.24 0.24 0.23 VI. Alcohol 0.47 0.47 0.47 VII. Drug Abuse 0.50 0.50 0.48 VIII. Mental Health 0.42 0.44 0.33 IX. Attitude 0.76 0.77 0.70 X. Criminal Behavior 0.65 0.65 0.58 Total 0.75 0.76 0.71
New Instrument
Domain Name Overall Males Females I. General Needs
0.52 0.53 0.49
II. Attitude 0.73 0.74 0.68 III. Criminal Behavior 0.65 0.65 0.59 Total 0.72 0.72 0.66
Table 5 presents bivariate correlations for the FROST total score and the reentry to
system measure for the original FROST instrument and the new FROST instrument. As can be
seen, values improved slightly between the original and new FROST instruments, while
14
remaining significantly correlated with recidivism (0.22 for the overall sample and male sample
and 0.17 for the female sample; p < 0.001).
Table 5 Correlations for FROST Total Score & Recidivism for the Overall Sample – Original vs. New Instruments Original
Instrument New
Instrument Overall (N = 18,020) 0.21*** 0.22*** Males (N = 15,173) 0.20*** 0.22*** Females (N = 2,874) 0.16*** 0.17***
Finally, Figures 4, 5, and 6 present the results from the receiver operating characteristic
(ROC) area under the curve (AUC) analyses for the total risk score and the return to system
outcome measure for the new FROST instrument for the overall sample, males, and females,
respectively. Recall that when describing the strength of AUC values, the following minimum
thresholds should be taken into consideration: weak = 0.56, moderate = 0.64, and strong = 0.71
(Rice and Harris, 2005).
Based on these thresholds discussed above, it can be seen that the new FROST
instrument predicts offenders’ recidivism about 12 percent better than chance for the overall
sample and the male sample (moderate; AUC = approximately 0.64) and approximately 11
percent better than chance for the females sample (slightly below moderate prediction; AUC =
approximately 0.62).
15
Figure 4 ROC Curve for Total Risk Score & Reentry to System for Overall Sample – New Tool
Figure 5 ROC Curve for Total Risk Score & Reentry to System for Males – New Tool
0.00
0.25
0.50
0.75
1.00
Sens
itivity
0.00 0.25 0.50 0.75 1.001 - Specificity
Area under ROC curve = 0.6392
0.00
0.25
0.50
0.75
1.00
Sens
itivity
0.00 0.25 0.50 0.75 1.001 - Specificity
Area under ROC curve = 0.6372
16
Figure 6 ROC Curve for Total Risk Score & Reentry to System for Males – New Tool
Recommendation #3
Regarding the sex offender-specific sample, UCCI recommended that the ADC remove,
combine, and/or create new cut-off scores and risk-level categories for this population as well.
The following subsection outlines the new FROST instrument for sex offenders specifically. It
is important to note that same items and domains that were eliminated for the overall sample, as
well as the same domains that were combined for the overall sample, were eliminated or
combined for sex offenders as well (i.e., the ADC adopted Option #2 for the sex offender-
specific sample).
Like the original FROST instrument for the overall sample, the original tool effectively
placed sex offenders into low-, moderate-, and high-risk categories; however, the majority of sex
offenders were deemed moderate-risk to reoffend. As such, UCCI recommended to the ADC
that cut-off scores be adjusted so that the distribution of sex offenders was more spread out
across the lower and higher ends of the risk-level continuum.
0.00
0.25
0.50
0.75
1.00
Sens
itivity
0.00 0.25 0.50 0.75 1.001 - Specificity
Area under ROC curve = 0.6195
17
Table 6 presents the new distribution, compared to the original distribution, for the sex
offender-specific sample. As can be seen, approximately 30 percent of the sample is now
considered low-risk, while almost half as many offenders are considered moderate-risk,
compared to the original distribution (approximately 41%). Finally, about 20 percent and 9
percent of the sample are considered high/moderate- and high-risk, respectively based on the
new cut-off scores.
Table 6 Risk Level Distribution for the Sex Offender Sample — Original versus New Instruments
Original New
N % N %
Low-Risk 161 9.2 521 29.9
Moderate-Risk 1,444 82.8 710 40.7
Moderate/High-Risk -- -- 355 20.4
High-Risk 139 8.0 158 9.1 Figure 7 presents the recidivism rates for the sex offender-specific sample based on the
original cut-off scores and the new cut-off scores presented during Recommendation #2. 2 As can
be seen, slightly more (about 6%) of the low-risk sex offenders from the new FROST instrument
reoffended; however, there were 360 more offenders who were considered low-risk based on the
new cut-off scores than there were in the original FROST instrument. Related to moderate-risk
offenders, almost half as many sex offenders are now considered moderate-risk, compared to the
original tool (approximately 41% versus 83%); but, recidivism rates remained the same when
looking at the original and new instruments side-by-side (approximately 39%). Finally,
2 Note, cut-off scores are the same for the sex offender-specific sample as they are for the new FROST instrument described for the overall sample: low-risk = 0 to 6; moderate-risk = 7 to 10; high/moderate-risk = 11 to 13; and high-risk = 14+.
18
approximately 50 percent and 58 percent of the moderate/high- and high-risk sex offenders
reoffended, respectively.
Figure 7 Rates of Recidivism based on Original & New Cut-Off Scores for Sex Offenders by Risk Level
Table 7 presents the alpha reliability coefficients for the sex offender-specific sample for
the original FROST instrument compared to the new FROST instrument. Originally, internal
consistency across the Vocational/Financial, Education, Family/Social Relationships, Drug
Abuse, and Mental Health domains ranged from 0.19 to 0.53 for the sex offender sample. After
removing several items and combining the remaining factors into the General Needs section,
however, it can be seen that reliability is improved slightly, yielding a value of 0.54 for the sex
offender sample. Reliability estimates for the Attitude domain decreased very slightly between
the original and new FROST tool (0.78 versus 0.74, respectively), while internal consistency
increased slightly for the Criminal Behavior domain between the original and new FROST
instrument (0.62 versus 0.68, respectively). Overall, internal consistency for the new FROST
19.3
25.5
39.3 38.6
0.0
50.1 56.1 57.6
0
10
20
30
40
50
60
70
Original New
% R
etur
n to
Sys
tem
Low
Moderate
Moderate/High
High
19
instrument (with almost half as many items compared to the original tool) is relatively high,
yielding alpha coefficients of 0.74 for the sex offender sample.
Table 7 Internal Reliability Coefficients for the Sex Offender Sample – Original versus New Instruments
Domain Name Original Instrument I. Physical Health/Medical
0.50
II. Vocational/Financial 0.36 III. Education 0.19 IV. Family & Social Relationships 0.53 V. Residence & Neighborhood 0.35 VI. Alcohol 0.33 VII. Drug Abuse 0.37 VIII. Mental Health 0.45 IX. Attitude 0.78 X. Criminal Behavior 0.62 Total 0.76
Domain Name New Instrument I. General Needs
0.54
II. Attitude 0.74 III. Criminal Behavior 0.69 Total 0.74
It is also important to note that the correlation for the FROST total score and return to
failure measure for the sex offender-specific sample remained the same between the original and
the new FROST instrument (0.22). Finally, as can be seen in Figure 8, the ROC AUC analysis
produced a value of approximately 0.62, indicating that the FROST instrument predicts
recidivism about 12 percent better than chance alone (moderate strength).
20
Figure 8 ROC Curve for Total Risk Score & Reentry to System for Males – New Tool
Recommendation #4
The fourth recommendation suggested that the ADC consider assessing offenders prior to
release from prison using another instrument that has been designed specifically for offenders
coming out of prison. It was also recommended that the FROST be completed after offenders
are released from prison and out in the community for several months.
While assessing offenders prior to release from prison is a difficult process, the ADC will
begin using the AIMS DPO4 screen that was once used by the Community Corrections Bureau,
but has not been used since 2007. The ADC believes that the AIMS will provide officers with a
more appropriate assessment score, and subsequently, supervision level, for offenders coming
out of prison initially. The ADC will then use the FROST assessment after offenders have been
out in the community for at least 30 days.
0.00
0.25
0.50
0.75
1.00
Sens
itivity
0.00 0.25 0.50 0.75 1.001 - Specificity
Area under ROC curve = 0.6289
21
Recommendation #5
At the time of UCCI’s presentation, the ADC was conducting FROST assessments at
three different time points: (1) two weeks post-release; (2) 90 days post-release, and (3) 180 days
post-release. It is likely that too many assessments were being conducted in too short of a time
period to really see any change in offenders’ behavior. In this way, UCCI’s final
recommendation suggested that the ADC consider reassessing offenders every 6 to 12 months.
The ADC agreed with this recommendation and will begin assessing offenders prior to
release (with the AIMS DPO4), at 30 days post-release (with the FROST), and then again every
six months thereafter. Assessment results will be used for supervision and case management
purposes.
CONCLUSION
The current report summarizes five recommendations made to the ADC upon completion
of the FROST Validity and Reliability Study. The report also outlines the decisions made by the
ADC based upon these recommendations. With the adjustments made to the scoring guide and
supplemental FROST materials, inter-rater agreement across assessors is likely to improve and
internal reliability and validity are likely to increase with the adjustments made to the items,
domains, and cut-off scores on the instrument. These changes will also likely lead to more
offenders being supervised at a minimum level, thereby decreasing officers’ workloads.