Fourth Generation Risk Assessment and Prisoner Reentry

Brian D. Martin

Brian R. Kowalski

Ohio Department of Rehabilitation and Correction

Ohio Risk Assessment System (ORAS)

Phase

Initial Contact with

Criminal Justice System

Probation/Community Control

and

Parole/Post-Release Supervision In Prison

Tool

Pretrial Tool

(ORAS – PAT)

Community Supervision

Risk Assessment Tool

(ORAS – CST)

Prison Intake Tool

(ORAS – PIT)

Reentry Tool

(ORAS – RT)

Supplemental Reentry Tool

(ORAS – SRT)

Domains • Criminal History

(3 items)

• Employment (1 item)

• Substance Abuse

(2 items)

• Residential Stability

(1 item)

•Criminal History (6 items)

•Education, Employment, &

Financial Situations (6 items)

•Family & Social Support

(5 items)

•Neighborhood Problems

(2 items)

•Substance Use (5 items)

•Peer Associations (4 items)

•Criminal Attitudes &

Behavioral Patterns (7 items)

•Criminal History

(7 items)

•Education,

Employment, &

Financial Situations

(6 items)

•Family & Social

Support (5 items)

•Substance Abuse &

Mental Health

(5 items)

•Criminal Attitudes &

Behavioral Patterns

(7 items)

•Criminal History

(8 items)

•Education,

Employment, &

Financial Situations

(4 items)

•Criminal Attitudes &

Behavioral Patterns

(7 items)

•Criminal History (8 items)

•Education, Employment, &

Social Support (7 items)

•Substance Abuse & Mental

Health (4 items)

•Criminal Attitudes & Behavioral

Problems (12 items)

Number

of

Variables

N= 7 N= 35 N= 31 N= 20(Administered only to

those subject to Parole or

Post-Release Supervision).

N= 32(Administered only to those subject to

Parole or Post-Release Supervision).

Outcomes

Predicted

•Failure to Appear

•Reoffend

• Technical Violations

• Reoffend

• Reoffend • Reoffend • Reoffend

Pros and Cons of 4th Generation Assessments

Pros

*Multi-stage assessment system

*Emphasizes responsivity and case planning

*Highly predictive global risk scores

Cons

*Inter-rater reliability problems

*Requires lengthy structured interview

*Questions surrounding incremental validity

*Fails to consider behavioral experiences in prison

ORAS Quality Assurance and Reliability Study

A large-scale quality assurance study examining the interview and

assessment process for a host of ORAS instruments was undertaken

across four months in early 2013.

The primary goals of the broader study are to (1) assess the

interview skills of staff members conducting the assessment

process, and (2) to establish the reliability of each individual item

and final risk level on each of the ORAS tools.

ORAS Quality Assurance and Reliability Study

Some examples of the interview skill items on the observational

tool are listed below (5 of 12 items):

Reliability Analysis

When we speak of reliability in the context of this analysis, we are

specifically talking about inter-rater reliability which measures the

consistency among staff members in scoring particular assessment

items.

Reliability Analysis

Reliability is operationalized by the percent agreement for each

scoreable item and the final risk level.

Prior literature on objective prison classification generally

considers items to be reliable if at least 80 percent of the cases are

in agreement (see Austin and Hardyman, 2004). We use this

threshold to establish a minimum level of reliability.

Reliability Analysis

Most of the items on the RT are found to be reliable by conventional

Standards (which is 80% or higher staff agreement).

However, the Criminal Attitudes and Behavioral Patterns domain is

very problematic (with 5 out of 7 items found to be unreliable).

As a consequence, these unreliable items are negatively impacting

the reliability of the final risk level of the reentry tool.

Reliability Analysis

The tool overall performs just below minimally acceptable levels of

reliability (78.6% before potential override decisions).

However, according to the interview assessment results (not

presented), staff members overwhelmingly meet or exceeded

requirements across all interviewer skill areas during the study.

These results have obvious training implications for ORAS trainers

and line staff, however, the worsening levels of agreement in the

Criminal Attitudes and Behavioral Patterns domain suggest that

these items may have some inherent limitations.

Research Questions

What is the predictive validity of each of domain subscale score used

on the RT and SRT tools?

What is the relative strength of the ORAS subscales compared

against a static risk assessment administered at intake?

What is the role of prison misconduct and prison programming in

predicting reoffending, controlling for assessed ORAS risk and an

inmate’s static risk score?

Does the predictive importance of prison incarceration experiences

vary across reentry tools?

Data and Sample

Data consists of a Post-Release Control (PRC) release sample in 2012

that served at least one year in prison.

Recidivism looked at a 2-year follow-up (both overall reincarceration

and reincarceration for a new crime).

The 2012 release cohort was chosen because these offenders would

likely have a previous RAP score at the time of RT or SRT

implementation.

Threat-Related Offenses of Conviction

Threat-Related Rules of Conduct

Additional Analysis

AUC

Overall Risk Score (ORAS) .69 (RT sample)

RAP Static (Admission) .61 (RT sample)

AUC

Overall Risk Score (ORAS) .66 (SRT sample)

RAP Static (Admission) .61 (SRT sample)

Summary and Implications

Parsimonious use of dynamic domains in a reentry context is

generally supported, compared against broader assessments like

the LSI-R.

Results raise questions about how longer periods of incarceration

may erode the predictive validity of some constructs.

Results are consistent with earlier work demonstrating that dynamic

domains, while generally predictive across most MV models, don’t

add much incrementally over standard static assessments.

Findings are helpful for informing revision to the PRC assessment

process, which could potentially be modified to consider recidivism

risk posed by prison misconduct.