evaluating transition-to-work programs: utilizing a...

Evaluating Transition-to-Work Programs: Utilizing a Cross-Systems Approach

Julie J. Christensen, PhD, LMSWSusan Hetherington, PhD

Strong Center for Developmental Disabilities

Education

Employment

Health

Recreation

TransitionAge

Transition

AdultsandAging

UCEDDStrong Center

forDevelopmentalDisabilities

Note: Children’s Services, Transition, and Aging/Adults cross over all four focus areas. Solid lines show direct relationships, while dashed lines denote emerging or planned opportunities for collaboration.

Education

Employment

Health

Recreation

ChildandYouth

Transition

AdultsandAging

UCEDDStrong Center

forDevelopmentalDisabilities

Note: Children’s Services, Transition, and Aging/Adults cross over all four focus areas. Solid lines show direct relationships, while dashed lines denote emerging or planned opportunities for collaboration.

NYSProject SEARCHTM

NYSPartnerships inEmployment Systems

Change

Employment First!

Employment FirstStateLeadership MentorProgram (ODEP)

OPWDDPathways toEmployment Evaluation

Technical Assistance,Advocacy &Outreach

ShelteredWorkshopConversion

Addressing the employment gap

4

5

Challenges and barriers

• Disability related factors• Lack of education and training

• Inadequate transportation options• Need for accommodations on the worksite

• Attitudinal factors and stigma• Low expectations

6

Systems issues

• Differing priorities• Conflicting goals and regulatory guidance

• “Language” barriers• Inadequate funding & payer of last resort

approach

7

Promoting Employment First!

A national movement that promotes employment for all people with disabilities

Objectives of Employment First!:• Person-centered planning• Building infrastructure capacity• Collaborations that increase employment• Developing strategic partnerships with business

8

E1 and raising expectations

Adopting an Employment First philosophy in NYS

Belief that employment is attainable for all

Need to assess for and train towards least restrictive employment options

Integrated, competitive employment as Gold Standard:

• 20+ hours per week at prevailing wage• Fully integrated within work environment,

utilizing natural support

9

Background: Employment options

Fully supported work environment

Independent work environment

Sheltered workshopsWork activity centers

Individual or groupsupported employment

Competitive employment

The “Gold” Standard

10






Defining competitive employment:• 20+ hours per week• Prevailing wages with commensurate benefits• Fully integrated environment

Lots of Work to Do






Source: National Core Indicators (2013-2014)

21%12%

79%88%

0%

20%

40%

60%

80%

100%

18-34(N=150)

All Ages(N=432)

Employed in the community (paid)

Not Employed

WIOA (2014)l Enhanced collaboration between core

partnersl Expansion of education and training optionsl Increased emphasis on youth workforce

trainingl Consistent accountability measures

13

Project SEARCHTM: A collaborative model

© CCHMC 1/3/06

Vocational Rehabilitation

School District

Business

Supported Employment Agency

Developmental Disabilities Agency

14

Project SEARCHTM: Key concepts

• Business led collaboration with education, rehabilitation and long term support

• Braided funding

• Immersion and impact • Training in real work settings

• Low risk, low cost for business • Hiring students who are “good fit”

Annual school schedule

15

Current programs in Monroe County, NYl 4 programs

l University of Rochester Medical Center l City of Rochesterl Wegmansl Delmonte Hotel Group (adult program)

Project outcomes: The good newsl Promising results in first six years

l 144 participantsl Completion rate: 85%

l 79% of program graduates transitioned into competitive, integrated employmentl 57% hired by partner host business

l Graduates work an average of 23 hours per week at $8.51/hour at hire (range: $7.25 = $12.43)

Monroe County, NY outcomesl Graduates who maintain employment are

growing in their positions

18

Monroe County, NY outcomesl Maintaining employment over time

(job retention)

19

0%10%20%30%40%50%60%70%80%90%

12 months 24 months 36 months 48 months

% PS Grads Remaining Employed

OPWDD Job Retention Data

Avg. Rate of Employment for People with Disabilities

Monroe County, NY outcomesl Impact on income for students receiving SSI

20

SSI = $808/month

23 hours @ $8.51 = $196 x 4 = $784/mo. (wages)

$784 – ($65 + $20) = $699 (adjusted income)$699/2 = $350 (subtracted from SSI)$808 – $350 = $458 (new SSI amount)

$784 + $458 = $1,242/month

Debunking myths:• Reduction in SSI has NO IMPACT on Medicaid health benefits

(up to earning $46,000 annually)• Loss of job or hospitalization allows for AUTOMATIC REINSTATEMENT of SSI

NYS Project SEARCH

21

Programs in operation (n=14)

Programs in development (n=2)

22

Goal of program evaluation

Strengthen program • Improved outcomes

• Enhanced cross-systems collaboration and communication

• Increased fiscal efficiency

23

Program evaluation approach

Process and outcome evaluation

Continuous and ongoing feedback and assessment

Data from multiple stakeholders (e.g., student teacher, job coach, business supervisor, family)

24

Objectives of program evaluation

Identify factors that contribute to success

Identify potential barriers to success

• Why participants don’t complete• Why participants who complete don’t pursue

competitive employment

• Why participants who obtain competitive employment are unable to maintain employment over time

25

Program evaluation tools

Weekly internship assessment (completed by teacher and job coach)

Self-assessment (completed by student)

JOBS scale (completed by internship site supervisor)

Internship rotation progress reviews (3x/year)

26

Weekly internship assessment

Completed by teacher and job coach

Reviewed at the end of each month• Opportunity to learn from different perspectives

Action plan developed to address areas of needed improvement• Includes both instructional and job coaching

supports

Weekly internship assessment

27

Student internship assessment

Completed at the end of each month

Opportunity to compare self-rating to assessments completed by teacher and job coach• Encourage self-awareness and honest self-

reflection• Practice receiving feedback• Preparation for performance appraisal

28

JOBS scale (Brady & Rosenberg, 2002)

Completed by internship supervisor at the end of each internship rotation

Standardized measure of readiness for competitive employment

JOBS scale

29

JOBS scale (Brady & Rosenberg, 2002)

Reviewed by team (including VR counselor)• Opportunity to learn from business perspective• Compared against teacher/job coach weekly

assessments

Measure of progress towards goal of competitive employment• Identify potential barriers to employment and

develop action plan

JOBS Scale Score Sheet

30

Internship progress review (3x/year)

Includes student, family, education team, job coach/adult agency, VR, DD, MSC, other appropriate supports

Goal:• Discuss potential for readiness for competitive

employment by end of program year• Develop appropriate Employment First! exit plan • Address logistics and systemic barriers to

employment (e.g., referral to outside agencies for supports, regulatory issues with independent living)

31

It’s time for some group work!

32

Let’s practice with the tools!

RED = Student

ORANGE = Instructor

YELLOW = Job coach

GREEN = Business (internship supervisor)

BLUE = VR and DD

PURPLE = Family

33


In your groups, conduct a mid-year review:

• Student, instructor, job coach, site supervisor to each share strengths and challenge areas

• Family to share desires and concerns for student

• VR & DD to listen to all input

34


As a group, discuss:

• Is this student ready to begin job development?

• What challenge areas does the student need to address during the upcoming internship?

• What barriers remain that might inhibit the ability to transition into competitive employment?

• What resources can assist with these barriers?

35


36

Case study: Gary

20yo w/ CP

Utilizes motorized wheelchair independently; has some ability to walk with supports

Interest in food services or office work

37

Case study: Gary

Weekly teacher and job coach assessments initially focused on mobility concerns and quantity/speed of work

Supervisor JOBS assessment identified significant behavioral concerns

Student acknowledged behavior concerns on self-assessment

38

Case study: Gary

Mid-year review• Agreement that student had work skills to be

competitively employed• Behavioral concerns as significant barrier to

employment

39

Case study: Gary

Mid-year review•Agreement that student does not yet have necessary skills to independent living•Group home regulations as barrier to pursuit of competitive employment

• Pushing workshop placement (applications due in March)

•VR and DD not in support of workshop placement•Adult agency acknowledged that vocational goal might not be met within necessary timeframe

40

Case study: Gary

Mid-year review outcomes:• Student and family willing to pursue outside

counseling to address behavior concerns• Group home willing to work with partners

explore other paths to employment beyond workshop placement

• VR, DD and adult agency to develop back-up plan for options if competitive employment not obtained by housing deadline

41

Let’s compare!How did your group do?

(Understanding that you had a brief period of time and limited information…)

42

Summary: Benefit of evaluation

Promotes opportunities for increased conversation and group learning across systems

43

Additional thoughts

What happens when there is disagreement between teacher and job coach?• Most frequently teacher gives higher marks

(awarded for effort vs. evaluated against employer expectations)

• Opportunity to cross-train

44

Additional thoughts

What happens with the business’ assessment does not agree with the team’s assessment?• Opportunity for discussion and adjustments:

• Is there a breakdown in communication?• Is the business being too critical (holding to

different standard)?• Is the business not being critical enough (not

holding student accountable)?

45

Additional thoughts

What if the student does not agree with the assessment (from teacher, job coach, or business supervisor)?• Teachable moment related to realistic self-

assessment• Helps student accept constructive criticism and

buy-in to improvement plan• No surprises on exit interview or if not provided

a positive reference

46

Questions & discussionStrong Center for Developmental Disabilities (UCEDD)

University of Rochester Medical Center 601 Elmwood Ave., Box 671

Rochester, NY 14642

Julie Christensen, Director of Employment Programs585-273-3713

[email protected]@uiowa.edu

Susan Hetherington, Co-Director (SCDD)585-275-6608

[email protected]

48

Appendices

Evaluation Tools

evaluating transition-to-work programs: utilizing a...

Documents