1

Overview of the Individual Self-Sufficiency Planning (ISSP) Project Evaluation

2

Type of Evaluation

3

Outcome (or impact) Evaluation

• Outcomes of interest to leaders of California’s State Partnership Initiative (SPI) project have been (1) increased employment (number working; hours); (2) increased earnings; (3) job stability and attachment to the workforce; (4) leaving the SSA benefit roles; (5) reducing reliance on such benefits; (6) reducing use of selected, publicly-funded mental health services; and (7) contributing findings to public policy.

4

Process Evaluation

• Q1. Have there been any unintended or unanticipated outcomes experienced, positive or negative?

• Q2. What barriers have been encountered in delivery of the intervention strategies or achieving intended outcomes and, if so, how were those barriers addressed?

• Q3. How have State and local structures and procedures changed as a result of the demonstrations?

5

Process Evaluation• Q4. What State and Federal policy changes

would have made systems change efforts more effective and/or permanent?

• Q5. What was the role of the project in expanding access to health care coverage for individuals with disabilities?

• Q6. How did the activities of the project assist in the design and implementation of benefits planning and assistance activities?

6

Process Evaluation

• Q7. How was the project involved in efforts to expand access to generic employment services on the part of individuals with disabilities? Was the project involved in attempts to improve the ability of One-Stop Centers to meet the needs of individuals with disabilities?

7

Process Evaluation

• Q8. Did the project design and implement new and innovative service strategies that led to increased support for individuals attempting to reduce dependence on Federal benefits and obtain or expand employment opportunities?

• Q9. How did the project collaborate with other reform initiatives underway in the State at the same time as the SPI project, such as the WIG, MIG, Medicaid Buy-In, BPAO, Ticket, etc.?

8

Research Design and Data Collection

9

Quasi-experimental Design

• Some SPI projects (e.g., New York) tested the impact of work-related information, and used experimental designs with random assignment to treatment (e.g., basic plus more advanced information and assistance) and control groups (e.g., basic information only).

10

California: A Matched Comparison Group

Design

11

Target Group

• Those signing up for enhanced services (i.e., enrolling in the project) had to have – (1) a severe psychiatric disability; – (2) be receiving SSI, SSDI, or both;

and – (3) be receiving vocational

rehabilitation and/or employment services.

12

Service Settings

• 17 of about 29 DMH/DOR Cooperative Projects expressed interest and submitted proposals.

• Two sites (Vocational Rehabilitation Services, in San Mateo County; and Kern County Vocational Services in Bakersfield) were selected.

• One criterion was engagement with One-Stop Career Service Centers, because of interest at both State and federal levels.

13

Enhanced Services

• Over and above existing DMH/DOR Cooperative program services (assessment, planning, pre-job search, job search, job coaching, etc.), “enhanced services” involve 1 Benefits Coordinator and 1 Service Coordinator for 50 individuals (50:2 ratio).

• San Mateo has had 100 slots, with 2 BCs and 2 SCs. Kern’s program is much smaller, and has had 50 slots with 1 BC and 1 SC.

14

Matched Comparison Groups

15

Three “Matched Comparison Groups”

• All selected from the DOR database. • 1- DOR service recipients in Sacramento and

Fresno counties. • 2- DOR service recipients across all of the

other DMH/DOR Cooperative Project sites. • 3- DOR service recipients in selected

counties without a DMH/DOR Cooperative Project, where the recipient received services through a “Generalist VR Counselor.”

16

Stratified Systematic Sampling

• Random start used across two strata: (1) Major Disability (Psychosis; Mood Disorder; Other Psychiatric); and (2) SSA Benefit Status (SSI; SSDI; Both).

• If information had been available, a “match” on work history would have been useful

• SSA Benefit Status was chosen as a proxy for work history

17

Measuring Outcomes

• VCU and Mathematica Policy Research (MPR), oversee the 17 SPI projects for the SSA.

• Designed a data collection system and are carrying out a multi-state evaluation.

• ISSP project staff use about 30% of their time collecting initial demographic and quarterly update information and transmitting it to VCU.

18

Measuring Outcomes

• Much of the outcome information through this system is “self-reported.”

• DOR obtains quarterly UI covered earnings data through EDD.

• MPR working with SSA to obtain benefit and related information from SSA files.

• DMH collaborated with DOR to retrieve selected Medi-Cal information on mental health services, without jeopardizing privacy and confidentiality.

19

Measuring Services or Interventions

• Some use has been made of DOR status codes, including types of case closures (e.g., 26s, 28s).

• Most project-related service intervention data comes from site reports to VCU and MPR, as part of their data collection system.

20

Measuring Services or Interventions

• Individuals participate in the project for varying amounts of time. Some individuals will leave the project; sites may “graduate” others; and some participants will become unreachable.

• Initially, it was thought that the ISSP project would touch about 500 individuals over the five years. This target cumulative enrollment was subsequently reduced to 250, which has been achieved.

21

Site Visits and Project Meetings

• Decided to make five annual visits to the two demonstration sites

• Every other year visit the two chief comparison sites.

• Subsequent visits asked about changes over the preceding 12 months, in the kinds of people entering the project, services and supports, immediate outcomes, significant community developments and the like.

22

ISSP Statewide Coordinating Committee

• Initial meetings were monthly, then decided on day-and-one-half quarterly meetings.

• First half-day devoted to site reports and data elements (Data Elements Workgroup).

• Second half-day devoted to the work of a Waiver Workgroup.

• Morning of second day devoted to summary of preceding day’s work and “big picture” matters. This group is known as the ISSP Statewide Coordinating Committee.

23

ISSP Statewide Coordinating Committee

Expressed a desire for additional information:• relationships between project personnel and local

SSA office staff; • the role and importance of natural supports; • ramifications of having someone specializing in

benefit planning/assistance for the rest of the team providing voc rehab and employment services;

• job descriptions; • the paradox of service systems helping some

individuals get benefits, while helping others reduce reliance on public benefits; and so forth.

24

Lessons Learned, Things to Think About

25

National Data Collection

• Can California influence the national data collection scheme, if there is one, to make it fit better with Bridges and generate the kind of information Bridges project leaders envision?

26

National Data Collection

• In ISSP project, data collection forms were announced, but changes not possible.

• Sites ended up developing their own basic information system to guide their work and to report quarterly.

• Some aspects of the national evaluation data collection scheme seem “excessive.”

• National evaluation uses a much different comparison group than the California evaluation.

27

National Evaluation

Should the California Evaluator serve as a facilitator for the National Evaluation, or should there be parallel evaluations (one California; one across the set of States participating in the Youth Initiative)?

28

Gathering Information

What kind of support will be needed at DOR, CDE, and SSA to get, store, and analyze information of value?

29

Resource and Information Needs

• It will make sense to follow an interactive process in reaching agreement on what exactly the California Evaluation team should do.

• Every aspect of the evaluation has resource, cost, and other considerations (e.g., whether data is shared on paper, via pen or pencil, or electronically).

30

Outreach and Intake

Very important for several reasons: (1)choice of comparison group; (2)advertising potential waivers may

encourage more “at-risk” youth and their families and more SSA benefit recipients to be referred and to apply to the project; and

(3) it may prove difficult to “stay on message.”