AN EVALUATION FRAMEWORK FOR

SCHOOL-BASED SUBSTANCE ABUSE AND

ADDICTION TREATMENT PROGRAMS

DELIVERED BY

RIDEAUWOOD ADDICTION AND FAMILY SERVICES

IN OTTAWA’S TWO ENGLISH LANGUAGE SCHOOL BOARDS

FINAL REPORT ON EVALUATION CAPACITY BUILDING GRANT

(ECBG-900)

FROM

THE PROVINCIAL CENTRE OF EXCELLENCE FOR CHILDREN’S MENTAL HEALTH

IN OTTAWA

R. Paul Welsh MSW RSW Executive Director

Rideauwood Addiction and Family Services July 2009

2

TABLE OF CONTENTS EXECUTIVE SUMMARY ……………………………………………………………………… 3

HISTORY …………………………………………………………………………………… 3 a) THE PROGRAM ………………………………………………………………………. 4 b) THE EVALUATION QUESTIONS ……………………...…………………………… 4 c) SELECTED METHODS ……………………………………………………………… 5 d) EXPERIENCES ……………………………………………………………………….. 6 e) COLLABORATIVE EXPERIENCES ………………………………………………... 7 f) SUMMARY OF RECOMMENDATIONS ……………………………………………. 7

CONTEXT ………………………………………………………………………………............ 9

a) PURPOSE ………………………………………………………………….………….. 9 b) PROGRAM DESCRIPTION …………………………………………………...........10 c) TARGET POPULATION ………………………………………………..…………..11 d) REVIEW OF LITERATURE …………………………………………….……………12 Summary of Literature Review’s Salient Points ……………………...……….13

APPROACH ………………………………….………………………………………….……..14

a) DESIGN ………………………………………………………………..…………..................…14

b) DATA COLLECTION ………………………………………….…………………….14 c) QUALITATIVE INFORMATION …………………………………………………….15 d) LIMITATIONS …………………………………………………………………………15

LESSONS LEARNED ………………………………………………………………………...16

a) LITERATURE REVIEW ……………………………………………………………..16 b) EXPERIENCES ………………………………………………………………............16 c) CHALLENGES ………………………………………………………………………..16 d) WORKING WITH THE CENTRE OF EXCELLENCE ……………………............17 e) IMPACT ……………………………………………………………………………….17

CONCLUSIONS, RECOMMENDATIONS AND NEXT STEPS ……………………….....19

a) SUMMARY OF LESSONS LEARNED …………………………………………….19 b) SUMMARY OF IMPACTS …………………………………………………………...19 c) IMPROVING COLLABORATION …………………………………………............19 d) NEXT STEPS/PLANS FOR CAPACITY BUILDING ……………………………..20

KNOWLEDGE EXCHANGE PLAN ………………………………………………………….22

a) KNOWLEDGE EXCHANGE PLAN ……………………………………….............22 b) SHARING OF INFORMATION/EXPERIENCE ……………………………………22 c) FURTHER KNOWLEDGE EXCHANGE …………………………………………...22

3

EXECUTIVE SUMMARY

HISTORY Rideauwood developed a school-based addiction/substance abuse treatment program for serious youth substance abuse problems in 1985 at the request of a Principal. Outcomes and the model were promising. There were requests to extend the program to other schools. By 1992 four (4) schools were funded with provincial revenues and agency fundraising. In 1998 a basic Outcome Evaluation was developed which identified substantial reductions in student substance abuse, improved school performance, drop out prevention and improved family relationships. This made a case leading to additional funding over the years. Wait lists in schools with Rideauwood service and limited staff resources prevented more than a simple evaluation effort. Erratic funding resulted in uneven levels of service for schools. Rideauwood received funding from the Ottawa Catholic School Board (OCSB) in 2003 and the Ottawa Carleton District School Board (OCDSB) in 2005. OCDSB budget cuts prevented renewal of this funding. Schools lost service as erratic and uneven funding returned. A Community Foundation grant the next year sustained some service for three years. In 2006-07 Rideauwood was serving 22 schools, and began working with the Ottawa Centre for Research and Innovation (OCRI), other agencies and 4 school boards to fund service to all high schools. Rideauwood set a goal to develop a more robust evaluation to establish program successes, increase viable and stable funding, and possibly see the model adopted across Ontario and Canada. The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO funded Rideauwood’s proposal for a $10,000 Capacity Building Grant in August of 2008. In late 2008 a major funding initiative was confirmed which would establish Rideauwood’s model for service two days weekly in English-language and French language high schools in Ottawa under the aegis of OCRI. Rideauwood and Maison Fraternite were funded to receive $1 million to serve a number of English-language Catholic and District Board schools and two French-language high schools respectively. Four equal funders: Ontario MOH/LTC/ LHIN (25%), Ottawa Public Health (25%), 4 School Boards (25%) and the United Way (25%). Rideauwood began the Capacity Building Grant project funded by the Centre of Excellence when expansions were as yet undefined. Final decisions with respect to expansion rolled out in December 2008. By January 6, 2009 Rideauwood was serving 32 schools with 2 to 4 days weekly, including 20 grade 9 –12 schools; 10 grade 7 – 12 schools; and 2 elementary schools grade 1 – 8. Further expansion in September 2009 will see Rideauwood’s addictions program delivered in 41 schools, including every Catholic Board School from Grade 6 to 12 and the Adult High School. In the Ottawa Carleton District School Board, the agency’s addiction program will be in 25 elementary and high schools. The planning to develop one consistent evaluation for the four new funders and 4 Ottawa school boards under the OCRI initiative has been assisted by Rideauwood’s work with the Centre of Excellence this year. The expansion has presented challenges to the time line of the project, while the project itself has been a complete success.

4

a) THE PROGRAM

Rideauwood's School Based Substance Abuse/Addiction Treatment Program serves youth with serious alcohol and drug problems. Referrals come primarily from school personnel. Students and their parents receive substance abuse services in the school during school hours. The service also provides counselling and education to parents consistent with best practice literature through engagement in the school and extensive services at the agency. The school is trained to understand and recognize adolescent substance abuse and how to make effective referrals. Each of the schools establish a core team consisting of Vice Principal, Guidance, Special Education, Social Work, teachers and others who meet with the Rideauwood School Based Counsellor on a weekly basis. The names of students identified as at risk are presented at this meeting. The Rideauwood counsellor coaches the school to gather essential information, approach the students and make effective referrals to the Rideauwood School Counselor. Rideauwood screens for urgent needs at the first interview. Urgent needs are addressed by the Rideauwood counsellor or referral is made to other appropriate services. A strong focus on engagement and motivational counseling exists in the first and all subsequent contacts. A subsequent assessment follows at a pace and frequency which is appropriate for the student. A treatment plan is developed with the student’s input in consultation with the school and other key informants such as parents, Police School Resource Officers (SRO), CAS and Probation. Consent forms permit release of information to Rideauwood (see Appendix I, Appendix J, and Appendix K). Counselling is provided to students once or twice weekly in the school and, when more service is required, at the agency or the school. Counseling is youth-appropriate, trauma informed, involves cognitive-behavioral counselling, multi-systemic interventions, and family system interventions using a bio-psycho-social-spiritual model which operates on a Moderation (harm reduction) to Abstinence continuum. Service is provided 12 months a year to ensure that vulnerable clients are supported over school breaks. Service for parents begins with an initial contact in the school or at the agency. When ready, parents are served in an initial 8 evening series of group based psycho-educational activities (Parent Education on Addiction and Kids - PEAK) focused on substance abuse and addiction in youth, the impact of addiction on families, supportive and parenting counselling and education, and addressing additional parent needs through referral to other required services. These include extensive, targeted Rideauwood programs (Adult Addiction Treatment; Family Member Program; Family of Origin Program; Problem Gambling Treatment). Following the 8 evening programs, ongoing weekly group counselling, accompanied by individual and couples counselling, are provided for a year or more and also on a monthly basis for a year or more. b) THE EVALUATION QUESTIONS

1. Who is served by the program with respect to student age, gender, risk for dropout, school performance, drug use patterns and school status?

5

2. Are there improvements for students with respect to:

• Reduced substances use • Improved school performance • Improvement in problems in Five Factors in Daily Living in the Basis 32:

• Relation to Self and Others • Daily Living/Role Functioning • Depression/ Anxiety • Impulsive/Addictive • Psychosis

3. Are students satisfied with the service they received?

Has the program helped them? Do the students have any recommendations?

4. Are parents satisfied with the service to themselves and their child?

Has the program helped their family? Do parents have any recommendations?

5. Are schools satisfied with the service to their school and its students?

Has the programs helped their school? Has it helped their students?

6. What proportion of youth and parents have Concurrent Disorder scored by GAIN

SS? Are there any improvements in Concurrent Disorder scores for youth/parents?

c) SELECTED METHODS The project will use a Quasi-Experimental Design. It will administer addiction specific assessment tools required by the Ministry of Health and Long Term Care, and data provided by the school with respect to school performance. Data will be collected at admission and at the end of the school year. Client satisfaction is/will be elicited from youth, parents and schools. A new Concurrent Disorder screener will be piloted. Drug use at baseline will be compared to drug use at the end of the school year. An MOH/LTC required Drug Use History Questionnaire will be used. It asks clients to identify each drug used, the amounts and frequency over a 60 day period. Due to the young clients patterns of multiple use of drugs and alcohol, their ability of recall, and rapid change (increases) in alcohol/drug use Rideauwood modified this tool to 30 days. Improvement in problems in 5 areas of functioning are measured by another MOH/LTC required tool, the BASIS 32 which will be administered at Entry (Screening) and the end of the school year. The BASIS 32 has 5 domains:

• Relation to Self and Others • Daily Living/Role Functioning • Depression/ Anxiety

6

• Impulsive/Addictive • Psychosis

School Performance: Schools provide student grade average and credit achievement for the term/semester immediately preceding the referral; designation of "High Risk for Drop Out" where valid, and student status (suspended, dropped out, etc.) at referral. All are recorded again in June. Satisfaction of Students, Parents, and the Schools will be evaluated through the administration of a Client Satisfaction Questionnaire developed for use by MOH/LTC funded agencies. Rideauwood adds additional questions specific to students, parents, and schools which elicit feedback on the program. It will be administered to students at the end of school year or completion of service and to parents at the completion of PEAK. Concurrent Disorders will be identified through the delivery of the GAIN SS screening tool as part of a pilot for the Champlain LHIN. While it is not part of this Evaluation Framework per se at this time, it has been added to this Framework in hopes of being part of a full implementation in 2010-11. The burden of data entry and scoring is prohibitive. Consequently implementation and scoring of the GAIN SS participated in the budget for the 2010-11 fiscal year. In 2009-10 it will be piloted with a small sample of 100 clients. d) EXPERIENCES The project has presented considerable conceptual struggles which are inherent in developing an evaluation plan which have focused and sharpened thinking of the programs. The process has involved approximately 25 youth treatment and parent counselling staff as well as the Program Director and the Executive Director who is the project lead. At the same time, discussions between the counsellors and the staff in 32 schools has been useful but time-consuming given the need to receive feedback from many schools and distill that feedback to one consistent plan which in 2009-10 will apply to 4 elementary schools, 12 high schools from grades 7 to 12 and 20 schools from grades 9 to 12. Timelines have been extremely challenging partly due to a one-month illness of the Project Lead, partly due to the complexity and sheer mass of Rideauwood's 25 frontline counselling staff and their key school informants, and partly from a 1/3 expansion at mid-point. To date the final financial analysis has not been completed. We anticipate the costs specific to this project have been more than the $10,000 budget since there were 33% more paid counsellors. Those costs were essential to this important project. While our literature search has been extensive, we have discovered very little literature specific to school based treatment and its evaluation. We have confirmed that this program and our model are consistent with ‘evidence based best practices’. Considerable literature on school based prevention programs will assist our own prevention activities which served over 9200 students last year, primarily in the schools receiving school based treatment. Treatment literature also affirms the benefit of addiction specific agencies providing school service which are integrated within broader

7

addiction treatment systems, compared to schools delivering their own programs with their staff. A 1/3 expansion in Rideauwood’s School Based Treatment Program in December 2008 and January 2009, and the funding of a major Substance Abuse Prevention for Somali Youth initiative funded by Health Canada in February 2009 provided several challenges to timelines. These came about unpredictably from previous initiatives prior to the approval of this Centre of Excellence Capacity Building Grant. e) COLLABORATIVE EXPERIENCES Rideauwood has collaborated extensively with our partner schools and our newly formed Data and Evaluation team to determine the capacity for the schools and the agency to provide reliable information and data entry. We have collaborated with OCRI, 4 School Boards, the United Way, and 5 other organizations and agencies toward the development of on Outcome Evaluation which will be consistent across all schools in Ottawa. Rideauwood has been able to influence the planning, which is still under way, especially with practicalities which can only be identified through experience. Rideauwood has presented our experience with this and previous evaluations to the 35 agencies in the Youth Sector of Addictions Ontario which is a network of 100 provincial agencies providing youth substance abuse treatment across the province. The final version will be shared across those agencies in the fall of 2009. An abstract has been submitted to the Canadian Centre for Substance Abuse to present the new framework, as well as results from the 2009-10 evaluation in the previous format now under way at their annual conference in November 2009. f) SUMMARY OF RECOMMENDATIONS Rideauwood has been challenged in the development of this Evaluation Framework and its future implementation by the large number of staff and the very large number of clients. In the 2009-10 school year we expect this program will serve 1,000 to 1,200 students and also 300 to 350 parents in 35 to 38 schools. The data entry challenges to the number of evaluation tools and the frequency of administration of those tools, and statistical power have placed restrictions on the scope of the Evaluation Framework. Rideauwood would recommend a scaling of funding for Capacity Building and also for Implementation Grants for larger agencies with larger numbers of clients, for more complex evaluation processes where agencies are partnered with many stakeholders - in our case 35 - schools, OCRI, police, multiple funders, and so on. In particular, addiction services frequently have multiple funders. Two years ago, (before the recent June 2008 funding announcements) Rideauwood had 36 funding sources for service in 24 schools. Collaboration and consultation with all of these funders was extremely complex. For similar reasons, which may not be exclusive to our own experience, Rideauwood would recommend the possibility of extended timelines which address the peculiarities described in the paragraph above. This might also be useful where the conditions in an agency change midstream. For example, Rideauwood was engaged in

8

hiring additional staff (a 33% expansion) in less than a month following funding confirmation and the flow of funds. This required a pause many weeks in the Evaluation Framework planning activity of key staff as they accommodated recruitment, interviewed for hiring, orientation and training of six new staff, and implementation planning with several new schools.

9

CONTEXT a) PURPOSE The project attempts to provide a more robust evaluation to identify program outcomes, to provide knowledge of results to staff which support program development, and to communicate information on the benefits of the program and the efficacy of the model to motivate other communities to adopt/adapt similar services in other communities. b) PROGRAM DESCRIPTION Rideauwood's School Based Substance Abuse/Addiction Treatment program serves youth with serious alcohol and drug problems. Referrals come primarily from the school. Students and their parents receive substance abuse services in the school during school hours. The service also provides counselling and education to parents consistent with best practice literature through engagement in the school and extensive services at the agency. The school is trained to understand and recognize adolescent substance abuse and how to make effective referrals. The school established a core team consisting of Vice Principal, Guidance, Special Education, Social Work, teachers and others who meet with their Rideauwood School Based Counsellor on a weekly basis. Students of concern are presented. The Rideauwood counsellor coaches the school to gather essential information, approach the students and make effective referrals to the Rideauwood School Counsellor. Rideauwood screens for urgent needs at the first interview. Urgent needs are addressed through Rideauwood or through referral to other appropriate services. A strong focus on engagement and motivational counselling exists in the first and all subsequent contacts. A subsequent assessment follows at a pace and frequency which is appropriate for the student. A treatment plan is developed with the student’s input and consultation with the school and other key informants such as parents, police School Resource Officers (SRO), CAS, Probation, and others. Consent forms permit disclosure of information to Rideauwood. Counselling is provided to students once or twice weekly in the school and, when more services required, at the agency or the school. Counselling is youth-appropriate, trauma informed, involves cognitive-behavioral counselling, Multi-Systemic Interventions, and family system interventions using a bio-psycho-social-spiritual model which operates on a Moderation (harm reduction) to Abstinence continuum. Service is provided 12 months a year to ensure that vulnerable clients are supported over school breaks and summertime. Service for parents begins with an initial contact in the school or at the agency. When ready, parents are served in an initial 8 evening series of group based psycho-educational activities (Parent Education on Addiction and Kids - PEAK) focused on substance abuse and addiction in youth, the impact of addiction on families, supportive and parenting counselling and education, and addressing additional parent needs through referral to other required services. These include extensive, targeted Rideauwood programs (Adult Addiction Treatment; Family Member Program; Family of

10

Origin Program; Problem Gambling Treatment). Following the 8 evening programs ongoing group counselling, accompanied by individual and couples counselling, are provided on a weekly basis for a year or more and also on a monthly basis for a year or more. c) TARGET POPULATION This program is targeted to students in grade 6 to 12 who are referred for serious substance abuse problems and problems in school performance, and also their parents. The target is also the schools themselves. Stakeholders include funders, Probation, police, CAS and other ancillary services and service partners. d) REVIEW OF LITERATURE 1. Substance Abuse and Mental Health Services Administration (SAMHSA) U.S.

Department of Health and Human Services. (2007). Help is down the hall. Retrieved July 25, 2009, from http://www.nacoa.net/pdfs/SAP%20HANDBOOK.pdf

This handbook, developed by Substance Abuse and Mental Health Services Administration (SAMHSA), U.S. Department of Health and Human Services, establishes a comprehensive outline for Student Assistance Programs (SAPs) to be used in school-based substance abuse treatment. SAPs are described in detail and examples of programs that have been operational for decades are included. Outcomes: A study by Scott, Surface, Friedli and Barlow found that schools with a SAP had lower rates of alcohol use and “significantly higher levels of academic achievement than schools with no SAP” (p.106).

Described below are a few specific treatment programs discussed in this publication:

1.1 Great Falls, Montana: The Chemical Awareness/Responsive Education

(CARE) program was established in 1981, and initially provided treatment services to school board members, administrators, staff, students, and parents. The program now provides prevention education, training for teachers, administrators, staff, aides, bus drivers, cafeteria workers and community members. Moreover, they provide support groups for children of addicted parents (COAs∗), and represent the school district and coordinate activities for students participating in the Juvenile Drug Court.

1.2 Desert Sands Unified School District: This program started in 1983. It is a

holistic and family centered program that includes individualized treatment service, staff and parent training, and referrals to appropriate services for the entire family. Some of its key activities are individual and family appointments; educational support groups for COAs∗; parenting workshops; chemical awareness network; and educational intervention groups.

∗ COA refers to Children of Addicted parents. This factor represents a very high risk for substance abuse in children, including earlier use, using to change moods, high levels of consumption, and prevalence of other risks for youth substance abuse such as trauma, family disruption, mental health, violence and more.

11

1.3 Circle Pines, MN: Since the early 1980s, this SAP has adapted itself to meet

individual school treatment needs. Referrals for the program are made through a trained team of professionals, including school administration and other staff. All school staff are involved in identifying students with concerns and intervening. Several different student support groups are provided including groups for COAs∗, and sober support.

1.4 Roanoke, VA: This program, established in 1986, now includes high schools,

middle schools and elementary schools. Support is offered to the child and/or entire family.

Outcomes: 1200 referrals are made per year, including 343 self referrals, for own use or family member use. Treatment counselling sessions occur 3500 times per year.

1.5 Langhorne, PA: This SAP, operating since 1984, also includes all grades from

K-12. There are educational groups for COAs and alcohol/drug treatment aftercare that are tailored to meet the needs of students based on grade level. The program’s success is in part attributable to the commitment of the board, superintendent, administrators, and staff.

1.6 Tarrytown, NY: This program has been operational since 1985. Trained

substance abuse counsellors, who have clinical experience working with youth, deliver treatment in the schools one on one with students, and in groups. Funding is received from diverse sources including private foundations, corporations, individuals, school districts and local, state and federal government.

2. DeWit, D.J., Ellis, K., Wild C., Rye, B.J., Heathcote J., Steep B., (1997). Executive

Summary of Evaluation Report, Stigmatization Report and Process Evaluation of “Opening Doors.” Report for Addiction Research Foundation.

This is a voluntary, in-school treatment program that includes parent education, as well as staff awareness training. It takes a holistic approach to addiction work. Participants in this program were identified through self reporting questionnaires and engaged in 17, one-hour instructional sessions as well as group discussion and activities and peer support. Outcomes: This program geared to students in grades 8-10, demonstrated a reduction in frequency of alcohol use and binge drinking; lower susceptibility to peer pressure to misbehave and exhibit violent behaviour. General satisfaction with program indicated 87% of students and 91% of parents were satisfied.

3. Alberta Alcohol and Drug Abuse commission (AADAC) (1999).Program Outline,

AADAC Youth Services, Government of Alberta.

AADAC hosts an outpatient facility for youth ages 12-17. The program includes a school component at the center as well as assisting with reintegration into school after completion of the program.

12

Outcomes: Three months after treatment 69% of youths had abstained or decreased use. 75% of parents/family members were satisfied with help they received.

4. Minneapolis Public Schools Chemical Awareness Program (1998). Program

Components and Evaluation Tools, Minneapolis Public Schools, USA.

Chemical Awareness staff are integrated into the school teams to provide treatment interventions in the schools and are directed to families as well as individuals. Treatment as well as prevention groups are run for concerned persons/AlaTeen self-help, recovery support, classroom presentations, parents/families, parent education, staff training as well as one-on-one counselling for effected students. Referrals are made by staff, self referrals, parents, friends and community agencies. Outcomes: This program describes improvements in the areas of school attendance, performance and behaviour. Moreover, a reduction in the use of alcohol and marijuana was identified. Student data recording was completed by staff in two stages – baseline completed within first month of referral and comparison completed in May. Among the outcomes: students increased their knowledge of drugs and their effects, and better understood the choices that they make. Students were willing to discuss issues with counsellors, including their behaviours and demonstrated interest in changing behaviours. COAs increased knowledge about the impact of family members’ drug/alcohol abuse and increased personal support network. Students in recovery were able to maintain sobriety.

5. National Institute on Drug Abuse (1997). Preventing Drug Use Among Children and

Adolescents: A Research Based Guide. National Clearinghouse for Alcohol and Drug Information.

Key to this program is that parents are included in school-based treatment program, demonstrating the need to work with families, rather than solely with individuals. Outcomes: It is also relevant that high-risk youth should not be placed together in groups as it can worsen problems related to drug use for some clients.

6. Burt, M., Resnick, G., Novick, E. (1998). Building Supportive Communities for At-

Risk Adolescents: It Takes More Than Services. American Psychological Association, Washington DC.

This piece outlines a school based treatment program, that has a focus on counselling, enrichment, academics, increasing parental involvement in school activities and referrals to agencies (providing services at the school if possible), including community youth services, state drug and alcohol prevention office, juvenile justice agencies, big brother/sister programs, child guidance and crisis counselling agencies. Parent programs that look at drug and alcohol issues are included. Critical to the success of the program is that school based services are accepted by school administrators and principals. The program becomes most effective when feeder schools are included.

13

Outcomes: 18,045 students received services through this program. They found a high rate of retention of students in the school (95% for middle and high school).

7. Miziker, M. (1994). Counseling the Adolescent Substance Abuser: School-Based

Intervention and Prevention. Sage Publications, Thousands Oaks, London.

Miziker addresses treatment, as well as prevention, of drug/alcohol use, noting that the integration of treatment and prevention is beneficial to the success of the program. Integral to treatment work is the inclusion of parents and other family members. Individuals trained in chemical dependency with an understanding of adolescent development deliver individual, family and group counselling as well as staff development and training for teachers.

8. Eliny M., Rush, B., (1992). Effectiveness of Prevention and Treatment Programs for

Alcohol and Other Drug Problems: A review of Evaluation Studies. A Canada’s Drug Strategy Baseline Report. Health and Welfare Canada.

This study outlines that prevention on its own in 1992 had little or no benefit.

9. *Kaplan D.W., Calonge, N., Guernsey, B., Hanrahan, M. (1998). Managed Care and

School-Based Health Centers: Use of Health Services. Archives of Paediatric and Adolescent Medicine. Vol. 152, January.

School-based treatment through health centres. Offers group counselling, aftercare groups and individual counselling. Parents are contacted.

Outcomes: If available in the school, youth are 10 times more likely to use addiction treatment services. School-based health centers are successful in improving access to and treatment for substance abuse.

Summary of Literature Review’s Salient Points Rideauwood’s school-based program integrates a number of effective strategies outlined in the literature. The multi-dimensional program integrates treatment and prevention into one framework. Trained addiction professionals work in the schools to deliver services to those students who are most in need. Having the service available in the school allows Rideauwood counsellors to retain youth that may not attend outpatient services elsewhere. Support is offered to the entire family through parent education and support programs, moreover, children of addicted parents, a particularly high risk group, are able to access services through Rideauwood counsellors. Training is available for all education staff in the school, so that they are better equipped to recognize youth in need of assistance and to make appropriate referrals.

14

APPROACH

a) DESIGN The Outcome Evaluation for the Rideauwood School Based Substance Abuse/Addiction Treatment Program utilizes a Quasi-Experimental Design. At point of commencement of the implementation of this framework, baseline data will be recorded with respect to the academic performance of the student in the school term or semester immediately preceding referral. School records will be used to identify grade average, and credit achievements from the previous term. Student status will be identified, and whether or not the student is "high risk for dropout" will also be identified. A Drug History questionnaire will identify all of the drugs used and the amount and frequency of use in the previous 30 days. Assessment of functioning using the BASIS 32 will be administered to students based on 5 important domains. These will be recorded again in the same manner at the end of the school year. While this is not the end of treatment the timing lends itself to tracking school performance. Limitations in evaluation budget have, to date, prevented recording these factors at the end of treatment. The sample size will be approximately 1,100 and 1,400 students in 41 Ottawa English language schools, from Grades 6 to 12. All clients will be included in the full implementation to provide optimal statistical power for data analysis and interpretation. Parent satisfaction and recommendations for the program will be recorded at the end of the early service phase of PEAK, and at the end of service. All clients will be included in the full implementation: the sample size will be 300 to 350 clients. School satisfaction and recommendations for program will be recorded at the end of the school year. Approximately 40 schools will be served in the 2009-10 school year. Questionnaires will be administered to 6 key staff in each school who are members of the Multi-Disciplinary/Student Assistance Program Teens. The number of participants will be approximately 150 to 180 employees in the English language school boards. b) DATA COLLECTION Two tools utilized by the Ministry of Health and Long Term Care addiction treatment system will be administered at referral: The Drug History Questionnaire and the BASIS 32 previously described. The tools would be administered to the students by the counsellor who will be available to clarify any questions the student may have, and also to follow-up on any issues which arise for the student during or after the survey collection. The school will provide information to Rideauwood counsellors on the school performance of each student prior to admission and at the end of the school year, as described above. The Client Satisfaction Questionnaire for parents, for schools and for students is a modification of the CSQ 8, with specific medications for students and for parents. They will be administered at the end of the school year for students and for school staff. For parents, the questionnaire will be administered at the end of the PEAK Program, the end of treatment or the end of the school year, whichever comes first.

15

The new design will track changes in clients who are being served over 2 or more school years and will also provide ‘end of treatment’ data in addition to ‘end of school year’. c) QUALITATIVE INFORMATION Qualitative data will be collected from students, parents, and school staff using the Client Satisfaction Questionnaires including their comments and recommendations. d) LIMITATIONS The sheer volume of the number of clients to be surveyed precludes administering and recording data more frequently than at referral and end of school year. The burden of data entry on the agency will be substantial even with the current plan. Data collection points at the end of the school year will show a limited picture of clients at the end of treatment, since the end of the school year is not the end of treatment and many students and parents continue service over two or more school years. For most students a full reduction of alcohol and drug use will not be evident until the end of treatment, so the full impact on alcohol and drug use will be underreported. Conversely, school performance cannot be measured at the end of treatment since many students simply disengage from service rather than attending a formal end of treatment interview; hence the necessity of collecting data at the end of the school year. Eventually Rideauwood hopes to sustain sufficient funding to collect data at the end of treatment and potentially at more frequent data collection points such as the end of each school term, or every five or six months to obtain data on outcomes compared to the length of service. While not an element of this outcome evaluation, a new Screening Tool for Concurrent Disorders (Serious Mental Illness SMI and Substance Dependence) will be administered at admission only for all clients in the Champlain LHIN addiction and mental health treatment systems. Given sufficient resources, the GAIN SS could also be administered at the end of school year and may demonstrate more refined changes in domains addressing Concurrent Disorders. Data on students who leave school prior to completion or enter residential addiction treatment will be incomplete. For those students, this will have an impact on data analysis and interpretation. This evaluation framework will be field-tested in 2009-10 before full implementation in 2010-11. This will allow testing the impact of new tools on clients and in each school, as well as their cumulative impacts.

16

LESSONS LEARNED a) LITERATURE REVIEW The literature review identified less than a dozen articles which discuss school based substance abuse/addiction treatment. For the most part literature affirms the utility and the need for the Rideauwood model. Coincidentally we discovered a plethora of data on school-based Prevention programs. However they were generally found to have poor outcomes. b) EXPERIENCES The Project Lead has been a proponent of outcome evaluation as an element of organizational empowerment. However the agency, and in particular the school based treatment programs, have tripled in size over the past three years. Several new staff has been hired for whom their interest and ability in outcome evaluation was not well known to the management team. Early in the project three teams of volunteers were established by the School Based Treatment team. They undertook a review of potential evaluation tools and made recommendations based on criteria provided in the beginning and then revised based on further feedback. The process required modest amounts of management input and monitoring. All staff undertook the question of feasibility and practicality of data collection methods and frequency from the point of view of their ongoing workload, the capacity of schools to respond, and the impact of time commitments required for data collection and its impact on client service time, wait lists, and ability to respond promptly to client crises. The inputs, coaching, support and guidance from the Centre of Excellence with respect to written materials and teleconferences was very useful. However the one-on-one assistance provided by Susan Kasprzak was most useful of all. Her provision of knowledge and expertise was always respectful of the organizational realities. This certainly enhanced the staff teams’ response to the tasks of developing this evaluation framework by affirming the ‘ownership’ of the project and trust in the agency’s desire to improve our evaluation capacity. c) CHALLENGES First among the challenges faced in developing a new evaluation framework was the recognition that some substantial efforts had been invested over previous years with some success. It was challenging at times to refrain from throwing the baby out with the bathwater in our attempt to re-evaluate everything from the ground up. Conversely we were challenged at times to refrain from replicating what had already worked and had served us well. Furthermore, in moving to a more elaborate logic model we were challenged to be confident that the newer, apparently more elaborate models we were using included at least the same strengths of our previous Logic Model. It took nine months to produce a final Logic Model that satisfied our concerns. Secondly, early challenges included "the jitters" at commencement of the development of the framework based on concerns that pressure might be exerted by outside ‘experts in evaluation’ who had very little understanding of the treatment challenges and the interplay of the two. The issue of "Program Understanding" as

17

described by Carol Weiss in her Program Evaluation text are common to the evaluation of addiction programs which tend to be complex and multidimensional when they address the diverse needs of complex and multi-problem clients and families. These jitters were soon resolved following a meeting with the program’s 16 counselling staff and Susan Kasprzak from the Centre of Excellence. A third factor would be the cost in staff time and money. Rideauwood generally applies realistic and achievable client service outputs for all funds received for our services. There was concern that if counsellors were required to spend even one hour per week on the Evaluation Framework, that would reduce one client per week and two to four clients per year for each counsellor. While the $10,000 grant was absolutely essential for this project, it did not cover all the costs. We resolved to move ahead in the knowledge that a substantial expansion was imminent and the opportunities to do this in the future may have more constraints. Fourth, the lack of literature on the topic created some serious concern that our literature search was flawed. This concern was only allayed after the project was completed with three staff combining their efforts in comparing the results. The discussion of our literature review explains this. Fifth was the concern that our evaluation could not require a commitment of time from school administration, teachers or others. This limited the manner of seeking feedback from the schools. However the new feedback from schools was consistent with our previous experiences and feedback: schools value an opportunity to provide feedback for program evaluation providing it is simple, clear, and low maintenance. A sixth and final challenge was our concern that program evaluation may appear to be more credible when done by an independent, third-party. We were reassured by the Centre of Excellence that there is an emerging recognition of the validity of internal evaluations, and organizational self-sufficiency in evaluation is becoming more valued. d) WORKING WITH THE CENTRE OF EXCELLENCE This has been described above. Visits by our Centre of Excellence Project Officer with the entire staff team as well as frequent face-to-face visits and telephone consultations with the Executive Director who acted as Lead, and the Program Director who directs the School Based Treatment staff were very helpful and instrumental to what we believe is a successful project. e) IMPACT The agency believes that the entire Youth Addiction and Parent Program teams’ confidence and ability in outcome evaluation have been strengthened. Our process involved full participation of the whole Youth Addiction and Parent Program teams, all of which the School Based Treatment team are a part. These programs deliver integrated services to youth and their parents in several ways and their full involvement was necessary. The project began in September 2008 with about 12 staff and increased to 16 in January 2009. The activities were excellent team building exercises, allowing new staff as well as existing staff to demonstrate their skills and knowledge to each other and to undertake the task requiring both independent work and teamwork. The final outcome, which at the time of writing is not in its final format, is one in which the whole

18

staff shares, and feels, ownership and is confident in its final product. This reflects the old adage in community development "the process determines the outcome". Full participation resulted in a working team relationship at a product that the team will support. Capacity Building was accomplished in more than one respect.

19

CONCLUSIONS, RECOMMENDATIONS, AND NEXT STEPS

a) SUMMARY OF LESSONS LEARNED We began this project with the recognition that program evaluation is part of organizational empowerment. To this we have added the following learnings. It is likely that more learning will become evident. The agency had existing skill and knowledge which were valuable. The agency had considerable knowledge and experience in its staff which

became evident during the process and build the confidence of a growing team. It is important to address the challenges to develop a strong yet simple outcome

evaluation even in a large and complex array of services. It was important to engage all staff in the process, regardless of the cost. The outcome evaluation we developed will have lasting benefits. Outcome evaluation requires discipline and rigor, but is not ‘magic’ nor ‘rocket

science’. Outcome evaluation is within the capacity of clinically trained staff and best

served by their involvement in its development. b) SUMMARY OF IMPACTS This has already been described. c) IMPROVING COLLABORATION The application process begins a relationship which has a direct impact on the project. The first interactions in a granting relationship are instrumental. It is important that applicants and grant receivers know they are working in a relationship of trust, especially if the agency does not have experience with research granting. During the final stages of the application it became apparent that the Centre of Excellent may have had questions about whether the agency could absorb the additional activity involved in the Capacity Building Project, or that we might be ‘milking’ the grant to double fund activities that staff would carry out as a day to day activity. We had not anticipated such a question and were at first unable to recognize that this was indeed an issue. There also appeared to be some confusion connected to this issue as to what the Centre defined as "Lead", and what the roles of the Executive Director, Program Director or counsellors were as described in our proposal. (Please note that this paraphrasing is based on recollection some 10 months after the fact, and may not be exactly as worded). The process of responding to written questions did not resolve the questions. Eventually a telephone conversation resolved the situation. The process led to concerns ("jitters") early in the project as to what other assumptions might be at play which could have a negative impact on our evaluation project. Eventually, through first-hand work with Centre of Excellence staff, this concern on our part proved to be clearly unfounded. These two issues, presented as written questions for clarification may have delayed approval and start up. It presented us with some confusion, concern, and uncertainty as to what answers would address the

20

questions which were confusing to us. From an applicant’s point of view, there did not appear to be any content in our proposal that would give rise to those questions. Our written responses did not appear to answer the questions. They were ultimately clarified in conversation and followed up with written explanation. It may be that the Centre of Excellence is now working with several service systems or sectors which have different vocabularies, different cultures, different terminologies, and work in different research/evaluation contexts. “Children's Mental Health” appears to include hospital-based child psychiatry, and adolescent psychiatry; community-based children's mental health, and adolescent mental health; community-based residential youth addiction treatment as well as non-residential youth addiction treatment; services for street youth and homeless youth; pregnant/parenting young women under 18; young offender services; schools and school based programs, and more. This is a large reach with many organizations, systems and services sharing the term “Children's Mental Health”. It might be useful for the Centre to approach applications in a fashion which minimizes assumptions between one application and another. Perhaps this could be done by having readers with specialized focus with specific service types, organization types, or service systems (if this is not already a practice). When questions arise it may be useful for a telephone conversation to address lack of clarity. This may unearth misunderstandings with respect to the context of an agency, their approach to granting applications, and the meanings of terminology which may differ in different sectors or approaches to evaluation. For example different schools of training or theory apply different meanings to “goals” and to “objectives”. I regret not having more useful recommendations to offer. d) NEXT STEPS/PLANS FOR CAPACITY BUILDING Rideauwood received Health Canada funding for a large Substance Abuse Prevention for Somali Youth project in Ottawa. Rideauwood is the lead agency with The Centre for Addiction and Mental Health (CAMH) and the Canadian Friends of Somalia as partners in the funded project. A mandatory and funded deliverable is the development of an Evaluation Framework by July 12, 2009. Rideauwood is applying our learnings from this Centre of Excellence project as we develop that framework with CAMH. With respect to this Centre of Excellence School Based Treatment Project, our next step will be to introduce our evaluation plan to the two school boards we serve. Rideauwood has just received confirmation that we will expand into 9 new schools in September 2009. The new Evaluation Framework will assist in the orientation of new schools as well as strengthening the shared responsibilities for program operations by demonstrating shared responsibilities in program evaluation for all schools as the 2009-10 school year begins. The Evaluation Framework will be shared with the OCRI Substance Abuse and Youth in Schools Coalition to strengthen its work to expand services. Through that venue the new Evaluation Framework will be shared with the United Way and with Maison Fraternite in its work with the Francophone schools. It may assist them in their work.

21

The Evaluation Framework will be shared with the Addictions Ontario Youth Sector, which includes most of the youth substance abuse and addiction services across Ontario. It may assist them in strengthening the evaluation capacity and program development for that service system across Ontario. The new Evaluation Framework is part of an abstract submitted to the Canadian Centre on Substance Abuse for its conference in Halifax in November 2009. To date we have no confirmation on that submission. However school-based programs, the evaluation, and outcomes were accepted for presentation at previous CCAC conferences. We will also make a submission to present at a conference on the Canadian Association for School Health. We previously submitted to them at their Vancouver conference three years ago. The results of the Outcome Evaluation when it is fully implemented will be presented at Provincial, National, and potentially International conferences on addiction treatment and youth mental health treatment in an effort to demonstrate that the Rideauwood model is effective and economically viable. Beginning with the fall of 2009 all components in the Evaluation Framework developed in this Capacity Building Grant from the Children's Mental Health Centre for Excellence will be piloted. Each component in the Framework will be implemented on a small scale except for the substance use and the school performance components. These will be maintained at full strength in 2009-10. At this time Rideauwood cannot afford such a full scale implementation of a whole new model. Furthermore, given that the Rideauwood School Based Addiction Treatment Programs were expanded into 6 new schools in January 2009 and an additional 9 new schools in September 2009, the demands of expansion will take precedence over full implementation of our new Evaluation Framework. We are of the opinion that use of the BASIS 32, the Client Satisfaction Questionnaires for parents, students and schools while at the same time piloting the new system Concurrent Disorder screening tool, the GAIN SS, may present issues with respect to client response, data collection, or data entry, or have a combined impact. With a 40% expansion over a nine-month period we believe it prudent to ‘field test’ our new framework. Rideauwood will be applying for an Implementation Grant from the Centre of Excellence in 2010. The full implementation and a full Outcome Evaluation Report on these programs could lead to publishable results.

22

KNOWLEDGE EXCHANGE PLAN

a) KNOWLEDGE EXCHANGE PLAN Rideauwood intends to present our new Evaluation Framework in Eastern Ontario at the following organizations where we are members, in 2009-10:

• Children and Youth Mental Health Networks of Ottawa (14 agencies) • OCRI Substance Abuse and Youth in Schools network (12 organizations) • Champlain Addiction Coordinating Body’s Youth Cluster (8 organizations) • Addictions Ontario’s Youth Sector (35 organizations)

We will submit abstracts to present to the following within 2 years:

• Canadian Centre for Substance Abuse • Canadian Association for School Health • Youth Justice Network for Eastern Ontario • Canadian Evaluation Society National Conference • CES National Capital Chapter

b) SHARING OF INFORMATION/EXPERIENCES We have described this previously in this Report. c) FURTHER KNOWLEDGE EXCHANGE Rideauwood hopes to produce a publishable Evaluation Report based on the results of our full implementation. Over the next year we will begin discussions with experts in the field to assist us in this endeavor. At present Rideauwood has limited experience in publishing. We are hopeful that the Centre of Excellence may be able to provide some coaching and consultation in this respect. We believe we have a viable model that addresses a real need in the area of children's mental health, which includes addiction as a distinct sector. We believe that publishing the outcomes may lead to implementation of school-based substance abuse and addiction treatment services across the country and that this book would have a major impact on the incidence and prevalence of addiction in adolescents and adults in the future. This could have a substantial benefit on reducing the $23 billion yearly costs of addiction in Canada, and in particular reducing healthcare costs related to acute addiction and the chronic illnesses which are addiction’s sequellae.