Combining Universal Screening and Data Mining From Schools, DSS, Mental Health, Chemical

Dependency, and Probation in a Realist Evaluation of What Works and For Whom

Discussion Session Presented at University of South Florida’s 28th Annual Research & Policy Conference: Child, Adolescent, and Young

Adult Behavioral Health, Tampa, Fl., March 22-25, 2015.

Mansoor A. F. Kazi, PhD, University at Albany, State University of New York Patricia Brinkman, Director, Community Mental Hygiene Services, Chautauqua County Rachel M. Ludwig, Project Coordinator, Tapestry/Children’s SPOA, Chautauqua County Victoria Patti, Early Identification & Recognition Specialist, Chautauqua County, NY

Tapestry of Chautauqua

County, New York

Mansoor A. F. Kazi, Assistant professor Behavioral Health, School of Social Welfare, University at Albany

mkazi@albany.edu • American Evaluation Association (eval.org) Co-Chair

Human Services Evaluation Topical Interest Group.

Based on Kazi, M. A. F.

(2003) ‘Realist Evaluation

in Practice’, London: Sage

Realist Evaluation Partnerships

SAMHSA’s

Gold Award for

Outstanding

Local

Evaluation

2010 Training

Institute STAKES/THL

Chautauqua Tapestry

Family driven ~ Youth guided ~ Culturally sensitive

Community based ~ Evidence-based

What is Chautauqua Tapestry?

An opportunity and challenge for

Chautauqua County service and support

providers to partner with youth who have

emotional and behavioral challenges and

with their families to create an

accessible, responsive, appropriate and

effective service delivery system.

“Chautauqua Tapestry brings together providers, families

and the community to share responsibility and resources to enhance the delivery

of services and improve the lives of youth and families.”

Seeking Collaborative Partners

Education

Mental Health

Public Health

Child Welfare

Family Court

Objectives for Discussion Session

• 1) How to Work with school districts to select and regularly use universal screening tools

• 2) How to evaluate the effectiveness of changes in the services as the system of care is implemented utilizing the universal screening tools

• 3) How to utilize data dumps from the management information systems of schools, mental health and other services to continuously evaluate alongside the repeated universal screening tools and to promote sustainability

• 4) How to enhance the utility of evaluation from a family perspective that places real meanings to the quantitative findings across thousands of service users

• 5) How to use this data to promote cultural competence, family and youth partnerships, and interagency collaboration.

Specific Topics To Be Covered

• How to utilize the repeated screenings.

• Using this real data to demonstrate how research methods can be applied to investigate the patterns between demographics, interventions and outcomes, in a continuous evaluation (group discussion and analysis of data undertaken with the group)

• Youth and Family perspective—how to interpret these apparently statistical findings in a meaningful way, what do these findings mean for cultural competence, youth and family partnerships, and interagency.

Local Evaluation Strategy

• Evaluation resource and service for you and for each participating agency

• How to access and to use your own MIS data

• How to analyze this data repeatedly to inform practice

• Carried out with you and for you only—you decide who to share with and how to use the findings

Local Evaluation Strategy

• Human service agencies routinely collect data, but this data is not typically used for evaluation.

• Apply this evaluation strategy to make the best use of the available data in their own agencies

• Utilize data dumps from the management information systems of schools, mental health and other services

• Continuously evaluate alongside the repeated universal screening tools and to promote sustainability

Realist Evaluation: What Interventions work & in what circumstances

• A combination of efficacy research & epidemiology traditions • Management Information System (MIS) Data routinely collected but

typically not used for evaluation in agencies • Investigate interrelationships between outcomes, client demographics,

client circumstances, & services provided (interventions) • Methods such as binary logistic regression can predict the likelihood of

effectiveness of an intervention in given circumstances • Use findings at regular intervals to better target and develop services

School Examples of Realist Evaluation

• 13 School Districts (Chautauqua County) including the largest--Jamestown Public Schools

• 2008/09 baseline and comparisons with 2009/10, 2010/11 and each marking period in the current year

• Outcomes: average school grades, state tests, discipline, attendance, drop out rates

• Demographics: ethnicity, gender, lunch status, special educational needs, etc.

• Interventions: school based interventions, summer program, mental health and other services

• 100% school data plus 100% agency data from participating agencies

• What works and for whom in achieving school, agency and system of care outcomes

Data analysis and utilization

• Single system design with each youth and one group pretest posttest design repeated at every marking period

• Comparison of outcomes between baseline and subsequent periods

• Comparisons between those receiving and not receiving interventions

• Investigation of patterns between outcomes, demographics and interventions

• Binary logistic regression to identify predictors at every marking period

• Data analysis carried out in partnership with schools and agencies

• Utilization of evaluation findings to develop and improve services for children and families at regular intervals

Universal Screening Program

• Chautauqua County Department of Mental Health is the host of the 5 year grant through NYS Office of Mental Health

• 36 other counties in NYS have also received the grant

• to screen 2,000 children/youth ages 3-21 each year for early detection of emotional concerns or difficulties

• Provide community education about the importance of early identification and mental health

• Partnerships with organizations, schools, agencies

Emotional Wellness Screenings in Public School Systems

• Screened by teacher who knows the child well and for at least 6 months to observe the child’s behaviors

• Identify children with emotional concerns before they develop into disorders; helps identify stressors

• Increase likelihood that struggling children get the help they need in school, as well as referrals to outside agencies or supports

• Minimize the impact of possible mental health concern on the child’s life

UNIVERSAL SCREENER IN

SCHOOLS

Response To Intervention

Universal Screening

Academic Screeners--- More familiar with

(i.e.., Literacy, Math)

Social/Emotional Screener---Strengths and

Difficulties Questionnaire

JPS partnering with agencies to provide

interventions

Why Universal Screening For Social

Emotional Needs?

Looking for Universal interventions for

school-wide systems (tier 1 of PBIS)

Process of finding the right

students/matching the right interventions to

them

Screeners are one piece of data.

Data Based Decisions

Goals Of Screening

Fast, efficient, and respectful

Include all children and youth in

elementary

Identify focus for interventions

One piece of data

Strengths and Difficulties Questionnaire

(SDQ)

(Goodman, 2005)

Brief behavioral screening questionnaire

for children 3-17 year old.

asks about 25 attributes, some positive, and

others negative.

http://www.sdqinfo.org/

SDQ Items in 5 Scales

Emotional Symptoms

Conduct Problems

Hyperactivity/inattention

Peer Relationships

Prosocial Scale **

Universal Screening Tool: SDQ

• Strengths and Difficulties Questionnaire (SDQ)

• Brief measure of pro-social behaviour and psychopathology, 3-17 yr olds

• Goodman (2001)—reliability .73

• Five factors: emotional symptoms, conduct problems, hyperactivity, peer relationships and pro-social

• Grade levels K to 4 in 2012, 2013 & 2014

SDQ 2012 and 2013 Scores

SDQ 2012 and 2013 Scores

SDQ 2012 and 2013 Scores

SDQ 2012 Normal Scores/Ethnicity

SDQ 2012 Normal Scores/IEP

SDQ 2012 Predictors for

abnormal/borderline total scores

SDQ Change from 2012 to 2013: Total

Scores

Paired Samples Statistics

Mean N Std. Deviation Std. Error Mean

Pair 1 Total_Difficulties2012 9.0907 1511 7.73341 .19895

TOTAL2013 9.0232 1511 7.95077 .20454

SDQ Change from 2012 to 2013: Total

Scores

Jamestown County Mental Health

Distribution of 2012 Total SDQ

Jamestown County Mental Health

Change from 2012 to 2013 Total SDQ

Jamestown County Mental Health

Change from 2012 to 2013 Total SDQ

Striders Elementary Change from 2012 to

2013 Total SDQ

Predictors for Change from 2012 to 2013

Total SDQ

Conclusions for SDQ Screening

•Main contributors to SDQ total score status were hyperactivity and conduct subscales

•Predictors for abnormal/borderline SDQ total scores were IEP, lunch status, gender, father’s level of education

•Significant improvements with Jamestown County Mental Health intervention

•Predictors for SDQ total score improvement were Striders intervention and baseline abnormal/borderline scores

Evaluation Strategy

• We are here to help you—as an evaluation resource

• Help you to use your own data for evaluation

• Can connect to others in Chautauqua County who have done this (e.g. data-dumps)

• Data findings can inform your decision-making and help make your services more effective

• This evaluation will also help you to be in a better position to apply for grants and other funding

• Enhance your own capacity for evaluation

• Utilizing SAMHSA funding, a free evaluation service for all participating school districts, DSS and other child-serving partners

LIVE INTERACTIVE DATA EXAMPLE

• THE SDQ Universal Screening for 2014

• How the situation has changed: Merging 2014 SDQ data with the SDQ data of 2012 and 2013

• Services Provided by Child-Serving Partners: The real impact on SDQ scores