Annual Site Reports 2006-2007 FY

A Training and Overview

Los Angeles County Evaluation System:An Outcomes Reporting Program

2008

Purpose of this Presentation

• Overview of the Los Angeles County Evaluation System (LACES).

• Review the content and purpose of the Site Reports.

• Discuss those content areas that are the focus of the evaluation.

• Describe future evaluation activities– Discharge Outcome Report (DOR)

Overview of LACES

• LACES is an eight year collaborative partnership between ADPA, contracted providers, UCLA/ISAP and other county and state representatives.

• The purpose of LACES is to design, implement, and conduct an evaluation of the adult alcohol and other drug treatment/recovery system in Los Angeles County.

• On-going evaluation, not a temporary study.

Accomplishments of LACES

To date LACES has:1. Fundamentals of the ongoing evaluation

system.2. Designed and implemented evidence-based,

standardized evaluation procedures.3. Determined that alcohol and other drug

treatment/recovery is cost effective.4. Review & improve existing treatment forms

and instrument.5. Designed reports to benefit treatment.

First… Some Questions…

• Have you heard of a site report?

• Have you seen a site report?

• Have you had the opportunity to review the contents and findings of the site report with the director or program staff?

A Site Report Is…

• Brief: six pages, double-sided.• Info from admission & discharge

LACPRS.• Demographics, outcomes &

performance measures.• Information is included on your site and

for all sites of the same program type.• Annual and some quarterly reports

What is the Purpose of the Site Report?

• How participants are benefiting from treatment.

• Provide information on program functioning and short-term treatment outcomes.

• Identify provider performance strengths and areas needing improvement.

What the Site Report Does Not Do?

• Provides “site specific” information – a single program type (e.g., outpatient) for a single address. – The site report does not consider funding

source.

• Provides feedback concerning performance in the focus areas of the evaluation.

Contents of the Site Report

• Executive Summary– Overall participant admission and discharge numbers.– Information on key outcomes and performance.

• Full Report– Demographic information.– More details on treatment outcomes and performance.– Charts illustrating admission to discharge changes.

Executive Summary

• Single page, stand alone document.• Admission to discharge changes.• Key Areas include

– “Participant Flow” chart.– Treatment Outcomes– Program Functioning

Executive Summary Tables

• Table A: Number of admissions and discharges for the time period covered by the report.

• Table B: Information on those discharged.– Determine areas of improvement

• Table C: Admission to discharge changes outcome areas.– Includes participants who were in treatment 30+ days

and completed an exit interview.

Program Functioning and Treatment Outcomes

Areas included in the Site Reports include:• Program Functioning

– Number of admissions and discharges.– Percent of exit interviews.– Participants with “positive compliance.”

• Treatment Outcomes– Retention in treatment (average days).– Changes in alcohol and other drug use.– Changes in other areas including health problems,

employment, and social support.

Program Functioning (Performance)

• Admission and discharge numbers– Review to ensure ADPA received all submitted

records.

• Positive Compliance– Discharge status = completed treatment or left

with satisfactory progress.– Important to determine the effectiveness of the

program.

Performance cont’d.

• Exit Interviews– Responses to all discharge LACPRS

questions constitutes an “Exit Interview.”– Information from both admission and

discharge is required to measure treatment effectiveness.

– No exit interviews = No measurement of the participant changes that occurred during AOD treatment.

Table B: Participant Flow by Fiscal Quarter and Year to Date (YTD)

Type of Transaction Total by Fiscal Quarter YTDTotalFirst Second Third Fourth

Participants Discharged 50 25 30 20 125

Participants with Positive Compliance 25 30 20 15 90

Percent with Positive Compliance 50% 60% 67% 75% 72%

Participants in Treatment at least 30 days

18 22 10 12 62

Percent in Treatment at least 30 days 36% 44% 50% 80% 50%

Participants in Treatment at least 90 days

12 15 12 11 50

Percent in Treatment at least 90 days 24% 30% 40% 55% 40%

Participants with Exit Interviews 20 25 15 15 45

Percent with Exit Interviews 40% 50% 50% 75% 36%

25 125

50%

40%

Treatment Outcomes

• Measures are similar to state and federal government.

• Retention in treatment for 30/90 days.– In LA County, some participants leave

treatment during the first two weeks. – Research shows that without an adequate

amount of time in treatment, few improvements are found.

Treatment Outcomes, cont’d.

• Performance Measures cover changes in:– Substance use (alcohol and other drugs).– Criminal justice involvement (days in

jail/prison).– Homeless at admission versus discharge.– Positive employment activities (full- & part-time

work, job training, and enrollment in school).– Social Support Activities (AA, NA, etc.).

Table C: Quarterly Performance Measures

Performance Measure Average Change from Admission to Discharge

This Facility Site All Residential Service Sites

Admission Discharge Change Admission Discharge Change

Average Days of Primary Drug/Alcohol Use*

12.3 0 ‑12.3 15.9 0.3 ‑15.5

Average Days of Criminal Justice Involvement

2.4 0 ‑2.4 4.9 0.2 ‑4.7

Average Days in Treatment 201 117.4

Percent Reported Homeless 49.6% 4.6% ‑45% 47.2% 23.9% -23.3%

Percent Reported Positive Employment Activities

3.5% 8.0% 4.5% 6.9% 22.4% 15.4%

Percent Reported Social Support Activities

57.0% 93.0% 36.0% 42.6% 92.9% 50.2%

This Facility SiteResidential Services

All SitesAdmission DischargeChange

Days of Drug/Alcohol Use

12.3 0 -12.3

Reported Homelessness 49.6% 4.6% -45%

Full Report

• Provides an overview of LACPRS admission and discharge information.

• Section includes:– General demographics.– Primary and secondary alcohol and other drug

use information.– Program performance.

Demographics

• Table provides demographic information on participants discharged .

• It includes demographic information on other sites for the benefit of the reader.

• Provides information on populations of special interest such as General Relief and homeless.

Primary & Secondary Substances

• Percentage of treatment participants by primary or secondary drug problem.

• Primary or secondary drug information from other sites is also included.

• These tables may be used to track changes in drug use patterns for the site.

Program Outcomes Section

Refers to the four major sections. 1. Discharge status.

2. Changes in alcohol and other drug use.

3. Changes in employment and social support.

4. Changes in health problems and other areas of social functioning.

Discharge Status

• Positive compliance examined separately from negative compliance – Left with unsatisfactory progress

• Average length of stay demonstrates differences in stay by discharge status – Useful to providers attempting to increase

overall retention.

Alcohol and Other Drug Use

• Bar chart shows change in average of days of primary and secondary drug use.

• Complete abstinence is not expected.

• Changes may also be found for injected drug use – especially for those sites that have a high percentage of needle users.

Chart 1: Admission to Discharge Changes

in Drug and Needle Use

40 1.2 0

7

18

0

10

20

30

Days Injected DrugUse

Primary Drug Secondary Drug

Avg

Day

s in

Las

t 30

Admission Discharge

Health Problems & Social Functioning

• Changes in jail days, family conflict, & physical health problems.

• In some instances the number of jail days will be very low.

• May show an increase in physical health problems due to increased access to health care.

Chart 2: Admission to Discharge Changes

in Health Problems and Social Functioning

24

02 1

7

0

10

20

30

Days in Jail Family Conflict Physical HealthProblems

Avg

Day

s in

Las

t 30

Admission Discharge

Employment and Social Support

• Changes in days spent employed and engaged in social support activities.

• Not all sites focus on employment or provide social support activities.

• Useful to examine the utilization of these activities by the participant.

Chart 3: Admission to Discharge Changes in Employment & Social Support

0

22

7.2

3.7

0

10

20

30

Paid for Working Social Support Activities

Avg

Day

s in

Las

t 30

Admission Discharge

Important Performance and Treatment Outcome Measures

• Completion of treatment/Positive Compliance.• Abstinence from/Reduction in drug/alcohol use.• Exit Interviews• Length of Stay/Enrollment in Treatment

(2 benchmarks)– 30 Days = Engagement.– 90 Days = Retention.

• All important aspects of the treatment plan.

Why Completion of Treatment?

• Most outside of this field are interested in completion figures.

• Positive Compliance– Completed treatment.– Left treatment prior to completion with

satisfactory progress.

• Others have adopted this mindset, but many have not.

Why Reduction/Abstinence?

• Silly picture, for a silly question.

• That is our business.• If we cannot demonstrate abstinence or

reductions in use, funders will move the money to those who can.

Interviews

• Responses to all discharge LACPRS questions.

• One of the most important of the evaluation foci.

• Information from both admission and discharge is required.

• No exit interviews = No measurement of changes.

Why Engagement & Retention?

• First 30 days are critical to the outcome of treatment.

• Without adequate treatment, few improvements are found.

• Funding organizations are beginning to take note of engagement and retention rates.

Note on Documentation

• Accuracy is extremely important.– Length of stay/enrollment very important.– Example – length of stay/enrollment.

• In 2004-2005, average length of stay was 105 days.• But only 40% were in treatment for 30+ days.

• If this information is inaccurate, others may become suspicious of all information reported.

Max LOS/LOE by Funding Source

• All funding sources support 30- and 90-day length of stay/enrollment benchmarks.

FUNDING Outpatient LOE Residential LOS Day Care LOE

CalWORKS 365 Days 365 Days 365 Days

Drug Court 730 Days 90 Days N/A

Female Offender

N/A 180 Days N/A

General Relief 180 Days 180 Days 180 Days

PPN 180 Days 180 Days N/A

Prop 36 300 Days 90 Days 204 Days

Drug Medi-Cal No max length - approval is need for 180+ days.

Average Provider Performance

• During the last 3 fiscal years, average performance in these areas was…

PERFORMANCE AREA Outpatient Residential Day Care

Reduction in Primary Drug Use 75.7% 97.7% 77.9%

30-Day Retention 71.8% 64.1% 68.9%

90-Day Retention 48% 38.1% 40.6%

Exit Interviews 48.6% 73.7% 47%

Strategies & Suggestions for Improving Outcomes

Improving Exit Interview Rates

• Some ways to increase rates of exit interviews include– Informing the participant of the exit

interview when admitted.– Include completion of the exit interview as

part of the individualized treatment plan.– Remind the participant when they are

nearing completion of their plan of the exit interview.

Improving Engagement & Retention?

• Engage them from first contact• Make same day appointments if possible.

• Appointment cards• To encourage forgetful clients.

• Phone calls• To remind clients of appointments.• To follow-up with clients who may have

dropped out.

Other Suggestions?

Challenges

• LACPRS– Provides a lot of good information.– Revisions and monitoring of information will be

needed to ensure quality & validity.

• Site Reports– Also requires revisions to format & content.– Not getting past Executive Directors’ eyes.

Your Views on the Site Reports

• What do you think about the site reports?

• What have you heard from your directors?

• Do you have suggestions or recommended changed to the reports?

Future of the Site Reports

• Assess individual sites in the key areas described in this presentation.

• Develop peer performance standards.

• Programs that fall short will be offered– Information, – Training, – and other forms of assistance.

LACES Phase III

• Focus on engagement and retention.• Continue with community contact.• Continue to expand and refine the site

reports.– Include Narcotic Treatment Programs.– Develop web-based training and educational

modules.

• Continue the expansion of LACES activities– Discharge Outcome Report

Discharge Outcome Reports (DOR)

• Provides admission and discharge information on a single individual.– A site report for one person.

• Useful when you need to report treatment outcomes to others.

• May also be useful when transferring someone to a new level of care outside your agency.

The End