kevin baldwin, ph.d. sharon johnson m.s. applied research services, inc. atlanta, ga kevin baldwin,...

Kevin Baldwin, Ph.D.Sharon Johnson M.S.

Applied Research Services, Inc.Atlanta, GA

Kevin Baldwin, Ph.D.Sharon Johnson M.S.

Applied Research Services, Inc.Atlanta, GA

Sharing Your Court’s Successes: Practical

Evaluation of Accountability Courts

What is What is Evaluation?Evaluation?

• The systematic collection & analysis The systematic collection & analysis of information (data), often for the of information (data), often for the purpose of making decisions.purpose of making decisions.

• eeVALUEVALUEation – “value is our middle ation – “value is our middle name” – this implies that we are name” – this implies that we are assigning worth to somethingassigning worth to something

Why Evaluate?Why Evaluate?• To answer critical questions about To answer critical questions about

the courtthe court• To document the court’s processes To document the court’s processes

and demonstrate outcomesand demonstrate outcomes• To assess Fidelity of ImplementationTo assess Fidelity of Implementation• To comply with funder’s mandatesTo comply with funder’s mandates• To provide information and feedback To provide information and feedback

for continuous quality improvementfor continuous quality improvement

Presentation Presentation OutlineOutline• Process Evaluation Process Evaluation - the “who, what, where, - the “who, what, where,

when, how, and how much” associated with when, how, and how much” associated with delivery of a program or initiativedelivery of a program or initiative

• Fidelity of ImplementationFidelity of Implementation - the degree to - the degree to which a program or initiative is delivered as which a program or initiative is delivered as designeddesigned

• Outcome Evaluation Outcome Evaluation - the degree to which a - the degree to which a program or initiative achieves its stated program or initiative achieves its stated objectives – the “so what?” aspect objectives – the “so what?” aspect

• Cost Benefit Analysis Cost Benefit Analysis - the costs of your - the costs of your program and compares the costs (typically per program and compares the costs (typically per participant) to the cost of not having the programparticipant) to the cost of not having the program

Process Process EvaluationEvaluation

• Helps us understand why a program Helps us understand why a program was or was not successfulwas or was not successful

• Disappointing outcomes could be Disappointing outcomes could be attributed to an array of issues such as attributed to an array of issues such as poor program design, poor poor program design, poor implementation, failure to reach your implementation, failure to reach your target audience, etc.target audience, etc.

• Provides information for program Provides information for program replicationreplication

Logic ModelLogic Model• A logical model is a series of A logical model is a series of

statements that link the problems statements that link the problems your court is attempting to your court is attempting to address (conditions), how it will address (conditions), how it will address them (activities), and the address them (activities), and the expected results (immediate and expected results (immediate and intermediate outcomes, long-term intermediate outcomes, long-term goals). goals).

Copyright 2009, Applied Research Services, Inc.

Conditionsand Assumptions

Long Term GoalsIntermediate Outcomes Immediate OutcomesInterventions

Non-custodial parents(NCP) not meeting child

support obligations

NCPs going throughrepeated cycles of

non-payment,incarceration,

release from jail, andsubsequent non-payment

of child support

Current court treatmentfor child support

is often not effective

Children of delinquentNCPs suffer emotionally

and financially

CPs and NCPs sufferemotionally and

financially

Appropriate participantsare identified,

screened and enrolled

Relevant communityand agency stakeholders

unified as partners

NCPs undergocomprehensive

assessment of issues

Provide research-basedcomprehensive

substance abuse,mental health,

literacy, and employmentservices based onidentified needs

NCPs take parentingclasses, visitation, and monitoring to increaseparent-child bonding

Fully functioningChild Support Problem

Solving Courtin Carroll County

Increased collaborationamong agencies dealing

with NCPs

Increased ability toaccurately identify

NCP issues

Increased linkagesbetween NCPs andservice providers

Increased degree to whichparticipants attend

treatment, training andrelated services

Decreased time in jail dueto non-payment of

child support

Increased payment ofcurrent, past child support

CCCSC participantshave increased rates ofemployment, literacy,

increased earnings, andcompletion of GED

CCCSC participants havedecreased jail time and

associated court, jail costs

CCCSC participantshave increased rates and

frequency of childsupport payments

CCCSC participants havedecreased rates of positive

drug screens, self-reported substance abuse,increased levels of mental

health functioning

CCCSC participants haveincreased frequency

of positive interactionswith their children

Medical insurancecoverage for childrenof NCPs increases

CCCSC participantsreport increased

attachment to children

Carroll County hasan effective collaborative

means to addresschronic non-paymentof child support and

collection of arrearages

Children and parents have permanent and stable incomes and

living situations

Families are able tomeet the financial,

material, and emotionalneeds of their children

Children’s educationaland health needs are met

Parents report feelingthat their emotional

well-beinghas improved

Substance abuse andfunctional illiteracy

are drastically reduced

The intergenerationalcycle of addiction and

poverty is broken

AOC-CCCSC Logic Model – Revised 03/27/09

Provide intensivecase management

Judge-led court teamof key agency staff

Increased frequency,quality of parent-child

visitation

NCPs have a number ofunderlying issues that

contribute to non-paymentthat are not being

adequately addressed

Graduated system of incentives and sanctions

Legal services forlegitimation, mediation

Process EvaluationProcess Evaluation• Process evaluation provides a Process evaluation provides a

descriptive study of how your descriptive study of how your program was implemented and how program was implemented and how it operates nowit operates now

• It’s concerned with history, current It’s concerned with history, current operations, participant progress, operations, participant progress, obstacles and overcoming obstacles and overcoming impedimentsimpediments

Roehl & Guertin, 2000Roehl & Guertin, 2000

11 Key Questions A 11 Key Questions A Process Evaluation Should Process Evaluation Should

AnswerAnswer1.1. How was the program developed? (i.e. How was the program developed? (i.e.

aims, why initial policy/process aims, why initial policy/process decisions were made)decisions were made)

2.2. What are the policies & procedures of What are the policies & procedures of the court? How have they changed over the court? How have they changed over time & why? Include: selection criteria, time & why? Include: selection criteria, point of referral in the CJ system, point of referral in the CJ system, program requirements, sanctionsprogram requirements, sanctions



AnswerAnswer3.3. Total eligible population for drug court? Total eligible population for drug court?

How are referrals & screenings How are referrals & screenings conducted? How many referrals are conducted? How many referrals are rejected & why?rejected & why?

4.4. Participant characteristics: demographics, Participant characteristics: demographics, criminal histories, SA/MH problems?criminal histories, SA/MH problems?

5.5. Available treatment & type of Available treatment & type of treatment/services received?treatment/services received?



AnswerAnswer6.6. What happens to participants in drug court What happens to participants in drug court

(i.e. treatment, drug testing, sanctions)?(i.e. treatment, drug testing, sanctions)?

7.7. Who are the staff & what are their Who are the staff & what are their responsibilities? Annual budget & funding responsibilities? Annual budget & funding sources?sources?

8.8. What are the roles of the team – judge, What are the roles of the team – judge, prosecutor, attorneys, advisory board, etc.?prosecutor, attorneys, advisory board, etc.?



AnswerAnswer9.9. What is the extent of collaboration What is the extent of collaboration

with other agencies such as with other agencies such as probation, parole, social services? probation, parole, social services? What information is shared between What information is shared between agencies?agencies?

10.10.What local court conditions affect What local court conditions affect your court (caseloads, community your court (caseloads, community attitudes)?attitudes)?



AnswerAnswer

11.11.How long do participants stay in How long do participants stay in drug court? Who drops out, at drug court? Who drops out, at what point, and why? How many what point, and why? How many participants, with what participants, with what characteristics, graduate from characteristics, graduate from drug court?drug court?


Sources of DataSources of Data• Meeting minutesMeeting minutes• Written mission statements, goals & Written mission statements, goals &

objectivesobjectives• Funding proposals, grant applicationsFunding proposals, grant applications• Annual reportsAnnual reports• Media (i.e. newspaper articles)Media (i.e. newspaper articles)• Caseload summariesCaseload summaries


Sources of DataSources of Data• Program documentation (i.e. eligibility Program documentation (i.e. eligibility

criteria, program rules, roles/ criteria, program rules, roles/ responsibilities of key agencies, graduated responsibilities of key agencies, graduated sanctions, phase requirements, participant sanctions, phase requirements, participant contracts, promotional materials, budget contracts, promotional materials, budget documentsdocuments

• Interviews/focus groups with stakeholders, Interviews/focus groups with stakeholders, participants, etc.participants, etc.

• SurveysSurveys


Sources of DataSources of Data• Automated data systemsAutomated data systems

– Demographic data (DOB, sex, race, ethnicity)Demographic data (DOB, sex, race, ethnicity)– Education (highest grade completed, degree earned)Education (highest grade completed, degree earned)– Employment status (FT, PT, student, unemployed)Employment status (FT, PT, student, unemployed)– Housing status (stable, unstable, homeless)Housing status (stable, unstable, homeless)– Community support (family, faith community, tx Community support (family, faith community, tx

community, work support)community, work support)– Mental health history (diagnosis, treatment history - # Mental health history (diagnosis, treatment history - #

inpatient & outpatient episodes, medication inpatient & outpatient episodes, medication compliance) compliance)

Sources of DataSources of Data– Substance abuse history (drug of choice, age Substance abuse history (drug of choice, age

began abuse, years of active substance abuse, began abuse, years of active substance abuse, # substance abuse treatment episodes) # substance abuse treatment episodes)

– Offense History (age at first arrest, nature & Offense History (age at first arrest, nature & type of charges, prior convictions, violent type of charges, prior convictions, violent convictions, previous jail episodes, previous convictions, previous jail episodes, previous prison episodes, previous probation/parole prison episodes, previous probation/parole episodes, current arrest/conviction – charges episodes, current arrest/conviction – charges & sentence)& sentence)

ResultsResults• Have all intended services been Have all intended services been

provided?provided?• Have the services been provided as Have the services been provided as

intended?intended?• What services not currently provided What services not currently provided

should be added to the program?should be added to the program?• Did the program reach the intended Did the program reach the intended

“target” population?“target” population?


ResultsResults• Did the program widen the “net” of Did the program widen the “net” of

defendants who were supervised by defendants who were supervised by the court or who received CJ the court or who received CJ sanctions?sanctions?

• What problems were encountered in What problems were encountered in program implementation, operation program implementation, operation and performance?and performance?

• How were these problems resolved?How were these problems resolved?


The Importance of The Importance of FidelityFidelity

• You could have the most powerful You could have the most powerful intervention ever devised, but it is intervention ever devised, but it is worthless if it is not delivered as worthless if it is not delivered as its developers intended. For its developers intended. For example, Excedrinexample, Excedrin®® works works wonders for headaches, but not wonders for headaches, but not when applied directly to the when applied directly to the forehead.forehead.

The Importance of The Importance of FidelityFidelity

• Your evaluation should explore Your evaluation should explore whether your program has been whether your program has been implemented as designedimplemented as designed

• Also, see how your implementation Also, see how your implementation compares to the Ten Key compares to the Ten Key Components for Drug Courts and Components for Drug Courts and our Georgia Standards for our Georgia Standards for Accountability CourtsAccountability Courts

https://www.ncjrs.gov/pdffiles1/bja/205621.pdf

https://www.ncjrs.gov/pdffiles1/bja/205621.pdf

Outcome EvaluationOutcome Evaluation• Measures the program’s influence on Measures the program’s influence on

factors such as graduation, recidivism, factors such as graduation, recidivism, abstinence, employment status, etc.abstinence, employment status, etc.

• Most useful when compared to a similar Most useful when compared to a similar group of persons that did not receive group of persons that did not receive programming (control or comparison programming (control or comparison group) – what outcomes would be group) – what outcomes would be expected without this program?expected without this program?

Evaluation Data, Evaluation Data, cont.cont.

• Evaluation has its own Evaluation has its own language, using words like language, using words like benchmarks, indicators, and benchmarks, indicators, and metrics to refer to datametrics to refer to data

• We also use phrases like “move We also use phrases like “move the needle”, “benchmarking”, the needle”, “benchmarking”, and “significance testing”and “significance testing”


• The data to be collected derive The data to be collected derive directly from the questions we directly from the questions we are asking. It helps therefore to are asking. It helps therefore to craft questions of a directional craft questions of a directional nature:nature:• Do graduates have reduced (Do graduates have reduced () )

rates of recidivism?rates of recidivism?• Do participants have increased Do participants have increased

(() employment skills? ) employment skills?


• We often cannot just report We often cannot just report however that something however that something increased or decreased. We increased or decreased. We need to say how much, and need to say how much, and compared to whom. Therefore compared to whom. Therefore we need benchmarks and/or we need benchmarks and/or control and/or comparison control and/or comparison groups.groups.


• Comparing our results to a Comparing our results to a benchmark would be to say benchmark would be to say that our participants had a 5% that our participants had a 5% positive urinalysis rate, positive urinalysis rate, compared to the 25% rate compared to the 25% rate observed in non-participants.observed in non-participants.


• A comparison group is a group of A comparison group is a group of people similar to our people similar to our intervention sample, but intervention sample, but receiving some other type of receiving some other type of intervention. A control group is a intervention. A control group is a group of people similar to our group of people similar to our intervention sample, but intervention sample, but receiving no intervention at all. receiving no intervention at all.

Outcome MeasuresOutcome Measures• Your logic model will provide you Your logic model will provide you

with some obvious areas for with some obvious areas for outcome measurementoutcome measurement

• Measure how well your program Measure how well your program met its goalsmet its goals

• Your outcome measures come Your outcome measures come from your objectives – they from your objectives – they indicate what you are trying to doindicate what you are trying to do

• They need to be measurableThey need to be measurable

Sample Data Sample Data SourcesSources• Official agency databases (i.e. GCIC, DFCS, Official agency databases (i.e. GCIC, DFCS,

hospital records) hospital records) • Drug test resultsDrug test results• Assessment resultsAssessment results• Treatment provider notes/reportsTreatment provider notes/reports• Pre/post testsPre/post tests• Interviews/focus groups with stakeholders, Interviews/focus groups with stakeholders,

participants, etc.participants, etc.• Surveys & other self-report dataSurveys & other self-report data

Sources of DataSources of Data• Automated data systems – combine process fields Automated data systems – combine process fields

with:with:– Program participation data: date referred, date entered Program participation data: date referred, date entered

program, # court hearings attended, types of treatment program, # court hearings attended, types of treatment completed (i.e. trauma group), # and nature of referrals completed (i.e. trauma group), # and nature of referrals made to other agencies/resources, Level of adaptive made to other agencies/resources, Level of adaptive functioning (i.e. can live independently, requires functioning (i.e. can live independently, requires supportive housing), degree of compliance with supportive housing), degree of compliance with medication, date left program, program completion status medication, date left program, program completion status (completed, did not complete, transferred, terminated) (completed, did not complete, transferred, terminated) and reason for any other status but completed.and reason for any other status but completed.

Sources of DataSources of Data

Outcome & performance measures: Outcome & performance measures: inpatient hospitalizations, crisis inpatient hospitalizations, crisis intervention episodes, ER visits, intervention episodes, ER visits, new arrests, new convictions, new new arrests, new convictions, new violations of probation/parole, new violations of probation/parole, new jail admissions, new prison jail admissions, new prison admissions, # failed drug testsadmissions, # failed drug tests

Comparison GroupComparison Group

• You will need data on your You will need data on your comparison group to compare comparison group to compare measures – take this into account measures – take this into account when determining the outcomes you when determining the outcomes you want to examine. How will you want to examine. How will you acquire the data you need from the acquire the data you need from the comparison group?comparison group?

Evaluation Data ExamplesEvaluation Data Examples• Outcome/performance data Outcome/performance data

examplesexamples• Number of graduatesNumber of graduates• Number of re-arrestsNumber of re-arrests• Percentage who relapsePercentage who relapse• Number of subsequent DFCS casesNumber of subsequent DFCS cases• Number of subsequent hospitalizationsNumber of subsequent hospitalizations• Number of subsequent ER admissionsNumber of subsequent ER admissions• Days cleanDays clean

Other Issues to ConsiderOther Issues to Consider• Timing & frequency of follow-up Timing & frequency of follow-up

• If your process includes post-participation If your process includes post-participation interviews, focus groups or surveys, be sure interviews, focus groups or surveys, be sure you have personal information to improve you have personal information to improve odds of keeping up with participants after odds of keeping up with participants after they have left the programthey have left the program

• Written consent for post-participation Written consent for post-participation follow-upfollow-up

• Respect the privacy & confidentiality of Respect the privacy & confidentiality of those with whom you are workingthose with whom you are working

• Federal laws (e.g., CFR 42, HIPPA) govern Federal laws (e.g., CFR 42, HIPPA) govern the use of substance abuse & health the use of substance abuse & health information – consider an IRB information – consider an IRB

• Respect and take into account the cultural, racial, ethnic and gender differences of your clients & their families

• Use results responsibly and ethically – don’t go beyond the intended use of the measures

• Federal laws (e.g., CFR 42, HIPPA) govern Federal laws (e.g., CFR 42, HIPPA) govern the use of substance abuse & health the use of substance abuse & health information – consider an IRB information – consider an IRB

• Respect and take into account the cultural, racial, ethnic and gender differences of your clients & their families

• Use results responsibly and ethically – don’t go beyond the intended use of the measures

Other Issues to ConsiderOther Issues to Consider

Cost-Benefit AnalysisCost-Benefit Analysis• Calculates the cost of a program vs. the Calculates the cost of a program vs. the

cost of the outcome to come up with a cost of the outcome to come up with a cost-benefit ratiocost-benefit ratio..

•Example: A study reveals that participants Example: A study reveals that participants of Drug Court A spend an average of 120 of Drug Court A spend an average of 120 less days in jail than similar offenders that less days in jail than similar offenders that do not go through drug court. do not go through drug court.

• Drug court participation to graduation Drug court participation to graduation = $3,800= $3,800

• 1 jail day = $45 x 120 days = $5,4001 jail day = $45 x 120 days = $5,400• Drug court savings = $1,600Drug court savings = $1,600

Cost Benefit Analysis Cost Benefit Analysis • Cost Benefit Analysis data Cost Benefit Analysis data

examplesexamples• Cost of treatment per participantCost of treatment per participant• Cost of day in jailCost of day in jail• Cost of foster care per childCost of foster care per child• Cost of adjudicating one felony Cost of adjudicating one felony

drug offensedrug offense• Cost of DFCS case investigationCost of DFCS case investigation

Evaluation Evaluation ResultsResults

• Ultimately your stakeholders and Ultimately your stakeholders and funders will ask these questions:funders will ask these questions:• Did it work? Was there an Did it work? Was there an

impact?impact?• How well did it work? How much How well did it work? How much

of an impact did you observe?of an impact did you observe?• How does this impact compare How does this impact compare

with results of alternative with results of alternative models? models?

Using Evaluation Using Evaluation FindingsFindings

• Describe your court and it’s Describe your court and it’s participantsparticipants

• Describe your court’s processes Describe your court’s processes and proceduresand procedures

• Continually improve your court’s Continually improve your court’s functioningfunctioning

• Document your impacts and Document your impacts and outcomes (sustainability) outcomes (sustainability)

Questions?Questions?

Presented at: 2013 AOC Accountability Court Conference – Atlanta, GA

Kevin Baldwin, Ph.D.Sharon Johnson, M.S.Applied Research Services, Inc.

404-881-1120 ext. [email protected]@ars-corp.com

www.ars-corp.com

Kevin Baldwin, Ph.D.Sharon Johnson, M.S.Applied Research Services, Inc.

404-881-1120 ext. [email protected]@ars-corp.com

www.ars-corp.com

mailto:[email protected]




Visit our web site at www.ars-corp.com or call (404) 881-1120

kevin baldwin, ph.d. sharon johnson m.s. applied research services, inc. atlanta, ga kevin baldwin,...

Documents

poor program design

increased earnings

jail time andassociated

designedoutcome evaluation

applied research services

rates of positivedr

poor implementation

childsupport obligationsncps