lessons learned from a synergistic program evaluation model partnership between a state agency and...

Lessons Learned from a Synergistic Lessons Learned from a Synergistic Program Evaluation Model Partnership Program Evaluation Model Partnership Between a State Agency and University Between a State Agency and University

ProgramProgram

Michael J. Leahy

Michigan State University

Program Evaluation Summit

Salt Lake City, Utah

September 25, 2008

AgendaAgenda

Brief overview of MSU and MRS. Typical collaborations between university programs and

State Agencies. Problems, issues and needs that led to the development of

Project Excellence. Benefits of funded partnership to both parties. Project Excellence - scope of work and how it works. Lessons learned and success factors for PE in the public

rehabilitation program.

Project Excellence


Michigan Department of Labor & Economic Growth-Rehabilitation Services

Project Excellence:Project Excellence:A Program Evaluation PartnershipA Program Evaluation Partnership


Office of Rehabilitation and Disability Studies

Program began in 1956

Masters degree program in Rehabilitation Counseling

CORE Accredited

Doctoral degree program in Rehabilitation Counselor Education

REACH Program for professional development

Ranked the #1 program in the country (tied with University of Wisconsin-Madison) by US News and World Report Rankings of graduate programs in rehabilitation counseling

Long-standing, mutually beneficial relationship with the public rehabilitation program in Michigan

Department of Labor and Economic Growth - Department of Labor and Economic Growth - Rehabilitation Services (MRS)Rehabilitation Services (MRS)

1.3 million people with disabilities in the Michigan - 27,000 citizens with disabilities served at any given time by MRS counselors.

Over 7,600 customers placed in jobs in 2007. Core principles - customer-centered, equity, advocacy,

collaboration, accountability, and continuous improvement 275 Rehabilitation Counselors. Heavy urban and isolated rural areas. History of innovation in service delivery Highest unemployment rate in the nation at 8.9% High turnover -- early retirements of leadership

My Initial Background in PE and Programmatic Research

•Crossroads Rehabilitation Center in Indianapolis Indiana

•CARF Section 9 Standards on Program Evaluation (mid 1970’s)

•Trained in Program Evaluation through national CARF initiative

•Walker Model (Dr. Robert Walker)

•Became a trainer for Region V RRCEPS in program evaluation (late 1970’s)

•Worked in this capacity for a number of years with community-based organizations throughout the region

Typical Collaborations between a State Agency and a University Program

•Membership on Advisory Councils

•Input into graduate training programs

•Serving as adjunct instructors and guest lecturers

•Practicum and Internship sites for students

•Professional development for agency staff

One substantive area that is not typically an integral aspect of the partnership is research and program evaluation

Given the level of accountability for outcomes, the complexity of the process,and the limited resources of most state agencies to effectively devote time,energy and and expertise to program evaluation, this appears to be an idealplace for a university program to further contribute to the partnership.

Problems Viewed as Opportunities

MRS was experiencing a lack of stability in the program evaluation area - frequent turnover and difficulty finding and retaining qualified personnel were persistent problems.

Agency very active in implementing a series of new strategic directions and programs and were unable to adequately measure the impact of these initiatives and use data for continuous improvement in performance.

To have quality programmatic research with timely information and consistently evaluate programs over multiple years became a greater need and objective of the agency.Key factor - - State Director and Executive staff value of PE.

Benefits of the Funded Partnership

MSU

•Opportunity to provide a rich, real world, field based environment to enhance the training of doctoral students in research and program evaluation.

•Provided a unique revenue stream to fund doctoral students as Research Assistants.

MRS

•Stabilize and greatly expand its programmatic research and program evaluation capacity to meet routine regulatory reporting requirements.

•Use program evaluation for the first time, as a major component of its continuous improvement strategy for the agency.


Began in 2001as a three-year project designed to enhance MRS program evaluation, outcome measurement, and reporting processes. After seven years in operation, will enter into a new grant in October for another five years (2008-2013).

Administered by MSU Rehabilitation Program faculty, with two full-time doctoral level research associates, and six research assistants (doctoral students).

Collaborative and consultative approach to working with MRS administrators; counselors; technical, training, and support staff; and customers.


Project Goal

Project Excellence will further develop and improve the application of qualitative and quantitative data analysis with respect to Michigan Rehabilitation Services culture and programs while prioritizing regulatory responsibilities and program excellence values.

Regulatory NeedsRSA 911 Database

RSA Performance IndicatorsCustomer Satisfaction

Needs AssessmentDirector’s Requests

MCTI Annual Performance

Project Identified Needs

MSU Student and FacultyResearch Protocols

Agency Priorities

Business Services Michigan Works!

Youth ServicesSoc. Sec. Recipients

Minority Populations

Special Requestsfrom

MRS Administrators

andSenior Managers

MRS

Project Excellence

MSU

Project Excellence Scope of Work

How the Individual Projects are Selected Each Year

Each year the State Agency calls for ideas and concerns that PE could address from all levels of the organization.

These ideas are then reviewed with the Execute staff to identify priority areas.

Once these potential projects are prioritized the Agency and the PE staff meetto review and make decisions about the scope of work for the upcoming year.

Each year, over the past seven years, PE has had between 10-14 individualprojects specified within the scope of work.

Examples of ProjectsExamples of Projects

Annual review the RSA 911 Data Data Integrity Study Customer Satisfaction Services to Minority Populations Business Services Re-organization Impact Study MI Transition in the 21st Century Effectiveness of MRS sponsored CRO Grants Michigan Career and Technical Institute Orientation Process Evaluation Vendors Qualifications and Standards MI Comprehensive Needs Assessment Best Practices related to Successful Outcome

Work GroupsSMP

Transition YouthAdjudicated YouthMI HI Connections

Customer SatisfactionBusiness Services

MCTIGrants

Implementation TeamSenior Manager InputDistrict Manager Input

Implementation Team ReviewTraining

DisseminationEvaluation

Project Excellence Flow of Operations

Ongoing Projects

Special

Projects

Lessons Learned - Success FactorsLessons Learned - Success Factors Trust Commitment at the top of the organization a must Long history of collaborative efforts Learning organization with common values Creative leadership and management styles to address

implementation of findings from PE - continuous improvement.

Supportive Michigan Rehabilitation Council Enjoyment of projects, collaboration, and partnership Common goal to improve service delivery and enhance

employment outcomes Common commitment to provide a variety of research

opportunities to doctoral students

Doctoral StudentDoctoral Student ExperiencesExperiences

• What was gained

• Positive influences on development as researchers and leaders

• Approaches to team work on research—from concept to product

• Methods for increasing research skills via access to relevant data—beyond 911 data.

Final ThoughtsFinal Thoughts

Integration of PE into the fabric of the organization. Inclusive style of participation on design and

implementation teams. People need to feel involved -- not an audit function, but part of a continuous improvement strategy for the agency.

Clear communication of findings and implications for all levels of the organization.

Go beyond the analysis of passive data to design and carry out experimental and quasi-experimental research on specific interventions -- evidence-based practices.

Final ThoughtsFinal Thoughts

Seek out other program Evaluation Specialists in State Agencies and other human service institutions to share ideas, problems and challenges.

The PE Summit here in Salt Lake City is a wonderful start to develop a learning community among professionals with similar roles and responsibilities throughout the nation.

It is my hope that this group will continue meeting and communicating in the future.

Best of luck to all of you!!

lessons learned from a synergistic program evaluation model partnership between a state agency and...

Documents