ohio alternative response project: evaluation findings presentation to the ohio alternative response...
TRANSCRIPT
Ohio Alternative Response Project:Evaluation Findings
Presentation to theOhio Alternative Response
SymposiumMay 13-14, 2010
Tony Loman, Ph.D. and Christine Filonow, MSW
Institute of Applied ResearchSt. Louis, Missouri
www.iarstl.org
CUYAHOGA
FRANK LI N
HAMI LTON
MONTGOME RY
S UMMI T
LUCAS
S TARK
BUTLE R
LORAI N
MAHONI NG
LAK E
TRUMBULL
CLE RMONT
WARRE N
PORTAGE
ME DI NA
GRE E NE
LI CK I NG
CLARK
RI CHLAND
FAI RFI E LD
WOOD
COLUMBI ANA
WAYNE
DE LAWARE
ALLE N
AS HTABULA
MI AMI
TUS CARAWAS
GE AUGA
MUS K I NGUM
E RI E
S CI OTO
J E FFE RS ON
ROS S
HANCOCK
BE LMONT
MARI ON
WAS HI NGTON
LAWRE NCE
ATHE NS
S ANDUS K Y
HURONS E NE CA
K NOX
DARK E
PI CK AWAY
AS HLAND
S HE LBY
CRAWFORD
AUGLAI ZE
LOGAN
PRE BLE
BROWN
FULTON
OTTAWA
ME RCE R
UNI ON
HI GHLAND
GUE RNS E Y
CLI NTON
MADI S ON
DE FI ANCE
WI LLI AMS
HOLME S
CHAMPAI GN
COS HOCTON
PUTNAM
PE RRY
J ACK S ON
HARDI N
MORROW
GALLI A
VAN WE RT
HE NRY
CARROLL
FAYE TTE
HOCK I NG
PI K E
ADAMS
ME I GS
WYANDOT
PAULDI NG
HARRI S ON
MONROE
MORGAN
NOBLE
VI NTON
Pilot Counties
Time Line for Evaluation
2008 2009 2010Jn Fb Mr Ap My Jn Ju Au Sp Oc Nv Dc Jn Fb Mr Ap My Jn Ju Au Sp Oc Nv Dc Jn Fb Mr ApPreparati
on
SACWIS and manual/email methods
Monthly case-specific data from workers
Family Surveys
Data collection Analysis/Report
Cost study
Site visits
Site visits
Site visits
Reception/Analysis of pathway tools (c. 10,000)
Reception/analysis of service plans
Random assignment
Cost study site visitsCommunity &
worker surveys
Community & worker surveys
Pathway Assignment and Random Assignment
TR Traditional Investigations
E Outcome/Impact Analysis
Child Maltreatment
Reports
AR Pathway Assignment
Pool of Reports Eligible for AR
Random Assignment
Experimental Group
Cases offered AR
D
Control Group
Control Cases
Inappropriate for AR
AR-appropriate
AR Family Assessments
B
C
Initial Screening For CPS
Screened out
Accepted Report
(Screened in)
A
The Evaluation
Estimated 51.7 percent of reports determined to be AR-appropriate Final study group included 4,529
families (50.5% Exp. and 49.5% control
Alternative Response: What is Different?
Removing the negative:– No formal victims, perpetrators– No substantiation or indication of
abuse/neglect– No entry of adults into a central registry of
abusers Enhancing the Positive
– Establish child safety (create a safety plan)– Engage the family– Focus on broader family needs– Emphasize family participation in decisions
Changes under AR: Negative Emotional Response of
Families
0% 5% 10% 15% 20% 25% 30% 35%
Angry (0.0110)
Afraid (ns)
Stressed …
Irritated …
Anxious …
Dissatisfied (ns)
Worried …
Confused …
Tense (0.0310)
Negative (ns)
Pessimistic (ns)
Discouraged …
Experimental
Control
Changes under AR: Positive Emotional Response
of Families
0% 5% 10% 15% 20% 25% 30% 35%
Relieved (0.0270)
Hopeful (0.0001)
Satisf ied (ns)
Helped (0.0470)
Pleased (0.0150)
Thankful (0.0020)
Comforted (0.0490)
Reassured (0.0010)
Grateful (0.0090)
Positive (0.0090)
Encouraged (0.0001)
Optimistic (0.0760)
Experimental
Control
Changes under AR: Satisfaction with Worker
49.5%
58.4%
39.2%
33.9%
6.8%
3.7%
4.5%
4.0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Control
Experimental
Very Satsified Generally Satisfed Generally Dissatisfied Very Dissatisfied
Changes under AR: Involvement in Decision
Making
41.1%
20.7%
9.0%
5.0%
24.2%
54.3%
15.8%
7.5%
5.1%
17.4%
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0%
A great deal
Somewhat
A little
Not at all
No decisions were made
Experimental
Control
Family Characteristics
Single Parent Families Over two-thirds of families reported incomes of
$15,000 or less compared to 8% for Ohio generally.
Compared with statewide statistics, a higher proportion of family caregivers in the study had less than a high school diploma (31 percent) than in the general population of Ohio (13 percent).
The caregiver in over six in ten families was unemployed.
Only 12 percent were employed full time.
79.0%
40.0%
39.3%
29.9%
25.5%
22.4%
23.1%
16.8%
10.0%
1.6%
80.9%
41.0%
39.1%
29.5%
27.9%
26.3%
23.7%
17.6%
10.4%
1.1%
79.9%
40.4%
39.2%
29.7%
26.6%
24.3%
23.4%
17.2%
10.2%
1.4%
0.0% 10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%
Food stamps
School breakfast/lunch
WIC
Child support
Utilities assistance
Social security disability
TANF
Housing assistance
Unemployment benefits
Retirement check
Total
Experimental
Control
Characteristics: Welfare and Other Support
Workers Identified Needs
Families tended to cite basic poverty-related needs. Workers indicated these needs among the families
they encountered but also emphasized problems in family functioning, especially parent-child relationships and parenting skills. Perhaps, consonant with this:– Nearly half of caregivers indicated problems in
children’s behavior—especially uncontrollable and aggressive behavior—and childhood depression and anxiety.
– About one-third of family caregivers surveyed reported that their children had problems in school.
Factors in Service Provision
Family need for services Family engagement Time devoted to cases Funding availability Service/resource availability and
worker knowledge Agency patterns of resource use
Family Reports: Level and Types of Services Received
0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%
Help for a family member with a disabilityAlcohol or drug treatment
Counseling services***Mental health services*
Parenting classesPARENTING, COUNSELING TREATMENT=====
Meetings with other parents Respite care
Assistance in homeChild care or day care
Legal servicesLEGAL, CHILD RELATED, DIRECT …
Employment helpJob training or vocational training
Help getting into educational classesEMPLOYMENT SERVICES===========
Welfare/public assistanceMedical or dental services**
MEDICAL OR WELFARE SERVICES====Appliances or furniture*
Money to pay rent**Housing
Car repair or transportation**Other financial help***Help paying utilities***
Food or clothing for your family**POVERTY-RELATED SERVICES=======
Experimental
Control
* Statistical Trend (p < .10) **Significant (p < .05)*** Significant (p < .01)
Family Reports: Level and Type of Services
Poverty-related services increased: AR workers more often provided referrals for or
helped families to receive food and clothing, help with utilities, other financial help, car repair and transportation, money to pay rent or help in obtaining appliances and furniture.
Experimental families under AR also reported receiving more referrals to traditional counseling and mental health services.
No difference was found in the number of services or the provision of direct services between Caucasian and African-American families under AR.
Family Reports: Satisfaction with Help Offered or
Received
34.4%
36.7%
5.6%
2.6%
20.7%
47.5%
33.7%
3.2%
4.5%
11.1%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
Very Satsified
Generally Satisfed
Generally Dissatisfied
Very Dissatisfied
No Services Offered
Experimental (AR Family Assessment)
Control (Traditional Investigation)
Worker Reports of Services Provided
Workers reported providing more services, support and assistance under AR and more information about where services could be found.
Workers indicated that basic poverty-related services were provided significantly more often to experimental families, such as rent payments, housing services, help with basic household needs, emergency food, and transportation.
Worker Reports of Services Provided
Under AR, 46.7 percent of AR workers said they were responsible for directly providing or connecting families to resources and services, while only 26.3 percent of TR workers reported this. Correspondingly, AR workers indicated they provided only “information & referral” for 41.2 percent of the services compared to 59.2 percent for TR workers.
AR workers directly assisted with 83.3 percent of services in the category “help with rent or house payments” compared to 30.0 percent for TR workers.
Most Frequent Types of Services Listed in Service
Plans
County (# plans
reviewed) 1st Frequent Service 2nd Frequent Service 3rd Frequent Service
Clark (72)
Mental health/ counseling
35% Appliances,
furniture, linens 26% Housing assistance 22%
Fairfield (69)
Mental health/ counseling
(in-house worker) 55%
Benefit assistance/ budgeting
(in-house worker) 25%
Education/ employment
services 22%
Franklin (214)
Beds, other household items
26% Settlement house
referral 10% Utility assistance 7%
Greene (104)
Mental health/ counseling
30% Utility assistance 10% Housing assistance 9%
Guernsey (43)
Utility assistance 40% Household items 21% Housing and food
assistance 19%
Licking (54)
Household items 35% Housing assistance 30% Utility assistance 28%
Lucas (107)
Clothing or food voucher
35% Beds/furniture/
appliances 23%
Baby items/ household items
20%
Ross Unable to review
Trumbull (136)
Mental health/ counseling/ therapy
53% Clothing or food
voucher 32% Utility assistance 26%
Tuscarawas (17)
Case-management by PCSA
47% Housing assistance 29% Mental health/
counseling 24%
Family and Worker Reports of Services Provided
Experimental families were also somewhat more likely than control families to indicate that the services received were enough to really help.
According to workers, AR families were also more likely to participate in services than control families.
Changes under AR: Level of Contacts of Workers with
Families
2.6
3.4
0.6
1.4
0.7
3.4
4.6
0.6
2.1
1.3
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
How many face to face meetings did
you have with members of the family (p = .003)
How many telephone contacts did you have with members of the family (p = .012)
How many other contacts did you
have with a family member (ns)
How many contacts did you have with others on behalf of this family (p = .008)
How many face to face contacts did
other service providers have with a family member (p
= .02)
Control Experimental
The average number of days until case close was 53.6 for experimental families and 44.7 days for control families.
What Changed?Operational Shifts
Longer assessment period Alternative Response workers provided
post-assessment services Family Service Plans Flexible funding Service partnerships
What Changed?Practice Shifts
Removal of barriers to family engagement Approach is less “incident-driven” Increased communication and trust Allows for more preventative support,
services, and follow-up
““With alternative response, I’m not there With alternative response, I’m not there to gather information to make a decision; to gather information to make a decision; I’m there to help the family come to a I’m there to help the family come to a resolution.”resolution.”
What Changed?Service Shifts
Factors: Family need for services Family engagement Time devoted to cases Funding availability Service/resource availability and
worker knowledge Agency patterns of resource use
Worker and Supervisor Responses
Workers and supervisors that performed work related to AR reported observable adjustments in their approach and practice, indicating that AR was implemented as intended and produced positive changes within the agency.
Workers reported feeling more able to intervene effectively with AR families than with non-AR families.
Reactions of AR families to assistance were seen as more positive by workers than the reactions of other families.
Worker and Supervisor Responses
The majority of staff involved with AR stated that the pilot had affected their approach to families a great deal or in a few important ways.
AR-involved staff saw AR as leading to a more friendly approach to families, more family participation in decisions and case planning, and more cooperation from families in the assessment process.
A strong minority (38.9 percent) of county staff involved with the pilot reported that AR had increased the likelihood that they will remain in the field of child welfare.
Worker and Supervisor Comments
“Alternative response has reminded me how blame driven we could be. A lot of the time, it doesn’t matter whose fault it is, as long as the family is willing to work to function better.”
“I have always tried to address concerns without labeling people, but the traditional approach makes this difficult. Now with the alternative response approach it is much easier to focus on the problems and solutions with the family.”
“Our alternative response team is outstanding. From the start to the close of an AR case, each worker brings an attitude of success which is passed on to the family. The ability to be creative in how families are assisted has been critical.”
Community Responses
Familiarity with AR among stakeholders had increased by the end of the pilot, from 45.3 percent in 2008 to 68.3 percent in 2009.
Attitudes toward AR were highly positive among those who were familiar, although a little less than half of all survey respondents were unsure of their opinion.
Nine out of ten judges or magistrates in the pilot counties reported being at least somewhat familiar with the AR pilot.
Those nine also perceived that AR had the potential to lower the number of cases coming to court to some degree.
Child Safety
Short-term child safety from the time of the original report until final contact with families was examined. Child safety problems were identified by workers in a minority of families, 33.2 percent of control cases and 25.4 percent of experimental cases.
When a child safety problem was identified, no statistically significant difference was found between experimental and control families in the extent of improvement or decline in safety.
There was no evidence that replacement of traditional investigations by AR family assessments reduced the safety of the children.
New Reports of Child Abuse and Neglect
Among families entering the study during the first 360 days, 13.3 percent of control families had a new report compared to 11.2 percent of experimental families. This difference was statistically significant.
A proportional hazards analysis that controlled for levels of past reporting on families also confirmed that experimental families that were served through the AR family assessment pathway had fewer new reports than control families that were approached through a TR investigative assessment.
New Reports of Child Abuse and Neglect
Control
Experimental
New Reports of Child Abuse and Neglect
Racial differences in later accepted reports were also examined.
Although study families as a whole were largely in poverty, African-American families were significantly and substantially more impoverished than Caucasian families. Race was taken as a proxy measure for poverty.
Analyses demonstrated that the major positive effects of AR on new reporting of child maltreatment at this point in tracking families appears to have occurred among African-American families. This was interpreted to mean that AR has its greatest effects among the poorest families in the population.
Subsequent Removal and Placement of Children
Differences in out-of-home placement were also examined in the evaluation.
Within the control group 3.7 percent of children had been removed while 1.8 percent had been removed in the experimental group, a significant difference.
This difference also remained significant in the stronger proportional hazards analysis. AR appeared to reduce the number of child removals and out-of-home placements.
Cost Analysis: Indirect Costs
Indirect costs were calculated using cost allocation data and average time that workers spent with experimental and control families.
AR family assessments averaged $940 per family compared to $732 per family for TR investigations. Reflecting increased worker time with families, AR was more expensive in the immediate term.
For subsequent work, experimental families averaged $145 per family compared to $266 for control families. Total costs for control families averaged about $999 per family compared to $1,084 for experimental families. At this point in the follow-up, experimental families were slightly more expensive ($85 per family) overall in indirect costs than control families.
Cost Analysis: Mean Indirect Cost per
Family
Cost Analysis: Mean Direct Cost per Family
Cost Analysis: Total Direct and Indirect Costs, Initial and
Subsequent to ARControl Experimental
Initial direct costs $ 99 $194
Initial indirect costs $ 732 $ 940
Subsequent direct costs $136 $ 48
Subsequent indirect costs $ 266 $ 143
Total Initial Costs $ 831 $ 1,134
Total Subsequent Costs $402 $191
Total Costs $1,233 $1,325
Quotes from AR Families
“The social worker was fantastic. She did not come to our home with predetermined ideas, but came to conclusions based on our family and our home.”
“I was surprised by how much help was offered. I didn’t know they offered you all that extra help. It was appreciated.”
“The caseworker treated us with respect and made us feel like we mattered and that we had our own voice to speak. We enjoyed our case worker coming and explaining things to us and made us feel wonderful.”
“My caseworker was awesome. She saw I wasn’t a bad mother. I just needed a little help to get back on the right track. And I love her for that.”