State Agency Reform: The Experience of the Massachusetts Department of Youth Services

Introductory Remarks Tarek AnandanDirector, NDTAC

3

Evaluating N or D Programs

The What Formative evaluation Summative evaluation

The Why Legal requirement Program quality improvement Program expansion/contraction

The How Execution - internal or external Funding – Federal, State, local, foundations

4

Evaluating N or D Programs

“Nationwide, there is a scarcity of literature specific to effective or ‘best practice’ juvenile justice education, or, for that matter, systemic reform of similar systems”

Evaluation of the Department of Youth Services Education InitiativeUMass Donahue Institute, 2008

5

Evaluation and Title I, Part D

SEC. 1431. Program Evaluations(a) SCOPE OF EVALUATION—Each State agency or local educational

agency that conducts a program under subpart 1 or 2 shall evaluate the program, disaggregating data on participation by gender, race, ethnicity, and age, not less than once every 3 years, to determine the program's impact…

(d) EVALUATION RESULTS—Each State agency and local educational agency shall—

(1) submit evaluation results to the State educational agency and the Secretary; and

(2) use the results of evaluations under this section to plan and improve subsequent programs for participating children and youth

6

Evaluation and Title I, Part D

Funding Evaluation is an allowable use of Part D funds

State agencies and local education agencies can use a portion of their granted funds to support evaluation activities

Funds used for evaluation purposes should be reasonable in respect to the amount of funding that goes to the provision of educational services

State education agencies are allowed to reserve the greater of 1% or $400,000 for administrative purposes, including evaluation activities of SEA administrators

7

Evaluation Resources

U.S. Department of EducationUnderstanding Evaluation: The Way to Better Prevention Programshttp://www.ed.gov/PDFDocs/handbook.pdf

Bureau of Justice AssistanceCenter for Program Evaluationhttp://www.ojp.usdoj.gov/BJA/evaluation/

Justice Research and Statistics Association Program Evaluation Briefing Serieshttp://www.jrsa.org/pubs/juv-justice/index.html “Approaches to Assessing Juvenile Justice Program Performance” “Hiring and Working With an Evaluator” “Strategies for Evaluating Small Juvenile Justice Programs”

8

Presenters

Christine KenneyDirector of Educational ServicesMassachusetts Department of Youth Services

Lonnie KaufmanResearch ManagerUMass Donahue Institute

Steven EllisResearch Manager

UMass Donahue Institute

State Agency Reform: State Agency Reform: The Experience of the Massachusetts The Experience of the Massachusetts

Department of Youth ServicesDepartment of Youth Services

Jane Tewksbury, CommissionerEdward Dolan, Deputy Commissioner

Today’s PresentersChristine Kenney, Director of Educational Services, DYS

Lonnie Kaufman, Research Manager, UMass Donahue InstituteSteve Ellis, Research Manager, UMass Donahue Institute

DYS VisionDYS Vision

The Department of Youth Services is a leader in the field of juvenile The Department of Youth Services is a leader in the field of juvenile justice that collaborates with youth, families, communities, justice that collaborates with youth, families, communities, government, provider agencies, and others to prevent juvenile crime. government, provider agencies, and others to prevent juvenile crime.

We provide the most innovative and effective treatment and skill We provide the most innovative and effective treatment and skill development services available to the youth committed to our care. development services available to the youth committed to our care. The Department exemplifies diversity in the management of all our The Department exemplifies diversity in the management of all our work and relationships. We effectively manage the resources work and relationships. We effectively manage the resources necessary to achieve the Department’s mission. necessary to achieve the Department’s mission.

DYS FunctionsDYS Functions

DYS is the Commonwealth’s Juvenile Correctional Agency

Functions– DetentionDetention– Residential TreatmentResidential Treatment– Parole/CommunityParole/Community

Each Year:– 18,000 to 20,000 arraignments18,000 to 20,000 arraignments– 5,700 bail admissions with average length of stay of 17 days5,700 bail admissions with average length of stay of 17 days– 1,200 commitments with average length of stay of 2.5 years1,200 commitments with average length of stay of 2.5 years

On any given day there are 2,000 committed youth in custody.

– 49% in Residential Placements49% in Residential Placements– 51% in Community Supervision51% in Community Supervision

Building Educational Capacity Building Educational Capacity

Educational improvement efforts were spurred by the findings of the MA DYS 2001 Legislative Report

Key Findings Related to Educational Services included concerns regarding:

– Teacher Recruitment and RetentionTeacher Recruitment and Retention– TransitionsTransitions– Special EducationSpecial Education– Learning Methods, Curriculum and Other ResourcesLearning Methods, Curriculum and Other Resources– Vocational Education and Job TrainingVocational Education and Job Training– Computers and TechnologyComputers and Technology– InfrastructureInfrastructure

DYS Reform Strategies 2001-PresentDYS Reform Strategies 2001-Present

DYS in close collaboration with the Commonwealth Corporation and the Hampshire Education Collaborative, initiated a multiple phase education reform effort for DYS educational programs

– Develop capacity to support education reform across the agencyDevelop capacity to support education reform across the agency– Align all education programming to MA Curriculum FrameworksAlign all education programming to MA Curriculum Frameworks– Develop an effective community placement and transition processDevelop an effective community placement and transition process

This initiative reflected a collective commitment to meet state and federal standards for educational programming by developing a better coordinated, appropriately resourced educational system

In spring 2006, DYS initiated a system-wide evaluation to help measure progress and to guide further education improvement

Evaluation of DYS Education InitiativeEvaluation of DYS Education InitiativeWhy and How?Why and How?

DYS wanted to revisit the original legislative report and assess change in areas that had been targeted for improvement

Minimum education requirements had provided a preliminary, but limited, view of the Education Initiative’s progress

A comprehensive evaluation would encompass a broader and more systemic view of the Education Initiative

Talked to local universities about how to approach

DYS made a commitment to allocate funding for evaluation for FY07/08 and issued an RFP

RFP was based on a logic model which guided core education strategies and activities

University of Massachusetts Donahue Institute was awarded the evaluation contract

Evaluation: Phase OneEvaluation: Phase One

Phase One: June 1 through September 30, 2006

Literature Review

Preliminary Fieldwork and Evaluation– Interviews with key leaders of DYS, CommCorp and HECInterviews with key leaders of DYS, CommCorp and HEC– Site visit interviews with teachers and coordinators at eight programsSite visit interviews with teachers and coordinators at eight programs– Attendance and participation at various system events Attendance and participation at various system events – Review and cataloging of key system documentsReview and cataloging of key system documents

Development of a Comprehensive Evaluation Plan

Presentation and Discussion of Preliminary Evaluation Findings

Evaluation PlanEvaluation Plan

Evaluation Guided by Three Core Research Questions

1. To what extent has DYS successfully implemented strategies designed to improve the educational and life outcomes for youth they serve? What factors have influenced the extent or success of implementation?

2. Have the Education Initiative's core strategies led to, or are they leading to anticipated short or mid term impacts? What other impacts or outcomes have resulted from the Education Initiative?

3. Based on key findings of the evaluation, what should DYS focus on over the coming years in order to realize the long term goals/outcomes of the Education Initiative?

Evaluation PlanEvaluation Plan

The Evaluation Also Identified:

– Emerging promising practicesEmerging promising practices

– Causes and potential solutions for mixed finding and impactsCauses and potential solutions for mixed finding and impacts

– Causes and potential solutions for gaps in evidence, including Causes and potential solutions for gaps in evidence, including

additional data which need to be collected to properly measure certain additional data which need to be collected to properly measure certain

outcomes outside the realm of the evaluationoutcomes outside the realm of the evaluation

– Apparent gaps in the current Education Initiative Apparent gaps in the current Education Initiative

– Key lessons learned through the implementation of the InitiativeKey lessons learned through the implementation of the Initiative

Evaluation: Phase TwoEvaluation: Phase Two

Phase Two: October 1, 2006 through August 31, 2007

Several major data collection and analysis activities:

– A DYS teacher survey that gathered the perspectives of over 90% of A DYS teacher survey that gathered the perspectives of over 90% of respondentsrespondents

– In-depth interviews of approximately 50 educators working at 12 DYS In-depth interviews of approximately 50 educators working at 12 DYS Education programsEducation programs

– Collection and analysis of all available institutional dataCollection and analysis of all available institutional data

– Case studies of six “featured practices” resulting from the Initiative Case studies of six “featured practices” resulting from the Initiative

– A second, targeted review of literatureA second, targeted review of literature

– On-site observations of selected programs and events, including On-site observations of selected programs and events, including statewide professional developmentstatewide professional development

Top-Level Evaluation FindingsTop-Level Evaluation FindingsFindings show remarkable progress in the development and implementation of key interventions, and positive initial impacts. Evidence shows important systemic changes to:

– Educational service deliveryEducational service delivery– Support of student transitionsSupport of student transitions– Overall system managementOverall system management

Important new programs, policies, and infrastructure are now in place, which have facilitated improvements in the stability and quality of DYS’s:

– Teacher workforceTeacher workforce– Instructional resources and methods used in DYS classroomsInstructional resources and methods used in DYS classrooms– Operative culture in which teachers and students function each dayOperative culture in which teachers and students function each day

Continued progress is likely, assuming DYS can sustain its vigorous investment of time, resources, and attention

Findings underscore not only the need to continue and expand improvement efforts in the coming years, but also the challenges associated with delivering educational services to adjudicated youth

Sample of Approach to Presenting FindingsSample of Approach to Presenting Findings

Topic: Educational Service Delivery

Key Finding: DYS has dramatically improved the competitiveness of education staff salaries in order to attract and retain well-qualified teachers

1.1. Prior to the Education InitiativePrior to the Education InitiativeA 2001 DYS Legislative Report identified “concern about the extraordinarily high A 2001 DYS Legislative Report identified “concern about the extraordinarily high turnover rate amongst DYS teachers and its effect on the quality of educational turnover rate amongst DYS teachers and its effect on the quality of educational services. It concluded that high turnover and problems with recruiting qualified services. It concluded that high turnover and problems with recruiting qualified staff were primarily attributable to salary inequities between DYS and public staff were primarily attributable to salary inequities between DYS and public school districtsschool districts

2.2. In responseIn responseDYS implemented a more competitive salary scale that increased the minimum DYS implemented a more competitive salary scale that increased the minimum salary for bachelor’s-level teachers, bringing starting salaries in line with state salary for bachelor’s-level teachers, bringing starting salaries in line with state averages. The new salary structure also offers compelling incentives for averages. The new salary structure also offers compelling incentives for teachers to pursue full professional licensureteachers to pursue full professional licensure

3.3. Looking aheadLooking aheadAs DYS continues to consider important reforms to its salary structure, it may As DYS continues to consider important reforms to its salary structure, it may need to offer higher salaries in key sub-regions where public school salaries are need to offer higher salaries in key sub-regions where public school salaries are highest. Further, DYS may need to equalize teacher benefits and working highest. Further, DYS may need to equalize teacher benefits and working conditions across program vendorsconditions across program vendors

Implementation FindingsImplementation Findings

1. Educational Service Delivery

DYS has dramatically improved the competitiveness of education staff salaries in order to attract and retain well-qualified teachers

DYS has made substantial progress in the development and implementation of a system-wide curriculum approach supported with instructional guides, templates, and material resources.

DYS and its partners have implemented an extensive professional development system and instructional coaching model.

2. Transition Services

DYS has identified a need for more robust employment pathways for appropriate DYS youth and is actively piloting and assessing new programs, and taking other steps, to support this need.

DYS has improved the infrastructure to support student transitions within DYS and into academic settings in the community. New models are being pursued.

Implementation FindingsImplementation Findings

3. Education System Management

DYS has expanded its management capacity and educational expertise through use of external vendors – most notably CommCorp and HEC.

DYS has improved its education system coordination and has taken important steps toward establishing more effective system-level accountability.

Short-Term Impacts of the Education InitiativeShort-Term Impacts of the Education Initiative

Outcomes remain preliminary; however, it appears to have resulted in several positive short-term impacts within DYS Education

Perceptions of Education Quality– DYS teachers’ perceptions of the quality of education provided to DYS youth have DYS teachers’ perceptions of the quality of education provided to DYS youth have

markedly improved, and future expectations are for continued improvements.markedly improved, and future expectations are for continued improvements.

Trends in DYS Teacher Perceptions of Educational QualityProportion Indicating Good or Excellent

86%

80%

52%

40%

55%

70%

85%

100%

January 2005 January 2007 January 2008

Trends in DYS Teacher Perceptions of Educational QualityProportion Indicating Excellent

37%

19%

8%

0%

10%

20%

30%

40%

January 2005 January 2007 January 2008

Short-Term Impacts of the Education InitiativeShort-Term Impacts of the Education Initiative

Workforce Stability

Data show positive impacts on the stability of the DYS teacher workforce over the past three years, including a decrease in teacher turnover and in August vacancy rates

Data also showed a substantial improvement in teacher qualifications

DYS Teacher Vacancy Data

12 Month Average  Aug Sept

FY 04 20.2% 11.2% 6.6%

FY 06 3.8% 6.0% 4.2%

Additional Short-Term ImpactsAdditional Short-Term Impacts

Changes in Instructional Practices

According to teachers, new curriculum tools and templates have made a positive impact on instructional practice, and instructional coaches’ reports indicate that teachers are now implementing recommended practices with increasing frequency and quality.

Teachers report that statewide PD has had a moderate to high impact on their instructional practice and student learning in the past three years. Data also suggest that instructional coaching helps many teachers reflect on their teaching style and encourages them to adopt new teaching strategies.

Impact on Educational Culture

Data suggest a positive cultural shift has resulted from the adoption of education reform within DYS, and that a stronger professional culture has emerged within DYS Education. These trends are strongest among treatment programs, but evident system-wide.

Other Features of the Final Evaluation ReportOther Features of the Final Evaluation Report

Strategic ConsiderationsA set of strategic considerations intended to provoke discussion and inform debate among DYS and its partners as they move into the next phase of education reform

Featured Practice Case StudiesA section featuring six case studies that focus on new programs, tools, and processes developed through the Education Initiative

– A statewide professional development systemA statewide professional development system

– A system-wide instructional coaching model A system-wide instructional coaching model

– A process for effective piloting of youth employability programsA process for effective piloting of youth employability programs

– A work-based training program in partnership with Putnam Vocational Technical High SchoolA work-based training program in partnership with Putnam Vocational Technical High School

– A universal student transcript to support student credit recovery A universal student transcript to support student credit recovery

– A “Community Transition School” to support student re-entry into public schoolsA “Community Transition School” to support student re-entry into public schools

Other Reports

Additional evaluation reports available include two literature review reports, and results from the DYS teacher survey administered in January 2007

Lessons LearnedLessons Learned

Money, time, experience– More money would have allowed development of additional data and more in-depth More money would have allowed development of additional data and more in-depth

researchresearch

Barriers– Data still inconsistent across the system; not always available in a useful format Data still inconsistent across the system; not always available in a useful format

– Data often incomplete or not up to dateData often incomplete or not up to date

What DYS would do in the future– Look at what baseline data would be needed for future evaluations to ensureLook at what baseline data would be needed for future evaluations to ensure more more

quantitative resultsquantitative results

– Look at specific initiatives that would be evaluated and tied to evidence based Look at specific initiatives that would be evaluated and tied to evidence based research practicesresearch practices

For Further InformationFor Further Information

Christine Kenney, Director of Educational ServicesMassachusetts Department of Youth Services

Christine.Kenney@state.ma.us

617-960-3324

Lonnie Kaufman, Project ManagerUniversity of Massachusetts Donahue Institute

lkaufman@donahue.umassp.edu

413-587-2419

http://www.donahue.umassp.edu