Post School Outcomes: What Can We Learn from Trend Data?

Pattie Johnson, WOUCharlotte Y. Alverson, UO Building Capacity Institute, 2013

Session Description•Oregon has three years of PSO data with

consistent definitions for educational and employment outcomes.

•What can we learn from examining the trends?

•Where are the key areas that districts can use to evaluate progress toward increased engagement for their students?

•Where will program changes have impact on improving performance?

Consistency•Since FFY 2008, states have had the same measure and definitions for Indicator 14, post-school outcomes.

•With FFY 2011 data collection, we have 3 to 4 years of PSO data.

•We can now start to examine trends in outcomes across years.

Data Collected Yearly by StatesStates are measured on their implementation of IDEA through 20 Part B Indicators.

#14: Percent of youth who are no longer in secondary school, had IEPs in effect at the time they left school, and were:1. Enrolled in “higher education”2. In “competitive employment”3. Enrolled in “other postsecondary education

or training”4. In “some other employment”

5Outcomes for Student with Disabilities

as Measured by Indicator 14

0.0

20.0

40.0

60.0

80.0

100.0

26.8

56.3

72.5

29.0

57.2

72.5

Median Percentage for Each Measure National FFY 2009 National FFY 2010

Perc

ent

of R

espo

nden

ts

United States Department of Education, Office of Special Education Programs (2011, 2012). Part B State Performance Plan/ Annual Performance Reports 2011 & 2012 Indicator Analyses .

Indicator 14 for Federal Reporting

1 HE 2 CE 3 OEd 4 OW 5 NE

We will look at the five outcome categories because these are more meaningful for

understanding our data and being able to use our data for program improvements

Questions Guiding the Analysis •How representative are these data? •What direction are our outcomes going?•Are there differences in outcomes by

subgroups? ▫Gender: Male, Female▫Disability: ID, ED, SLD, all other▫Method of Exit: Regular diploma, Completed, Dropout▫Ethnicity: Minority, Caucasian

•What is contributing to our outcomes?•How can we use the information?

Looking at Data • How representative are these data?

▫Aggregate of response representativeness• What direction are our outcomes going?• Graphs of:

▫Overall A, B, & C Measure x 3 years ▫Overall 1, 2, 3, 4, & 5 x 3 years

• Are there differences in outcomes by subgroups? • Gender Disability Method of Exit and Ethnicity categories x 3

years • What is contributing to our outcomes?

• What supplemental survey questions will help answer this question?

PSO in Oregon• 1-year prior to conducting the survey, districts

can collect accurate contact information on exiting students

• All Local Education Agencies (LEAs) collect follow up data, larger LEAs are provided with prioritized list of leavers selected to achieve a representative sample of leavers based on race, disability, gender, and method of exit

• LEA personnel conduct phone interviews

• Responses are recorded in online secure website

9

How representative are these data?

Representativeness: Basic Numbers from Three Years

School year 2008-09 2009-10 2010-11Interview year 2010 2011 2012Total Leavers 4295 4425 4244Selected for interview 2770 2779 2714Completed interviews 1911 1989 1748Response rate 68.9% 71.6% 64.4%

Oregon uses a stratified sample: • All districts conduct interviews with students each year• Small districts (15 leavers or less) interview all leavers• Larger districts are provided with a sample of required students to

interview.

NPSO Calculator Representativeness: Combining three years of data

  Overall LD ED MR AO Female Minority Dropout

Target Leaver Totals 12974 6630 1246 1053 4045 4410 3198 3015

Response Totals 5648 2840 497 461 1850 1907 1304 961

                 Target Leaver Representation   51.1% 9.6% 8.1% 31.2% 34.0% 24.7% 23.2%Respondent Representation   50.3% 8.8% 8.2% 32.8% 33.8% 23.1% 17.0%

Difference   -0.8% -0.8% 0.1% 1.6% -0.2% -1. 6% -6.2%

Dropouts are under underrepresented - a finding consistent with the each years’ separate response analysis. Importance: to ensure sampled group represents state population, the difference should be 3% or less. Caution should be used in interpreting any results using the dropout category.

What direction are our outcomes going?

▫Overall A, B, & C Measures▫Overall 1, 2, 3, 4, & 5 categories

National and OR State PSO Data

Measure A Measure B Measure C 0

20

40

60

80

100

27

56

72

24

51

66

25

54

68

25

55

72

National FFY 2009 State FFY 2009 State FFY 2010 State FFY 2011

Indicator 14 Measure

Perc

ent

of Y

outh

Data Source: National aggregate of FFY 2009 SPP Submitted February 1, 2011; State data reported in the SPP FFY 2009 & APR FFY 2010, 2011

14

What direction are our outcomes going: Outcomes by Three Years

Hi Ed

Comp E

mpl

Other

Scho

ol

Other

Emp

Not Eng

aged

0

20

40

60

80

100

2009 2010 2011

Outcome Category

Perc

ent

of R

espo

nden

ts

What do we see in the trends?• Higher Education initial increase, then static• Competitive Employment Increasing

▫More Oregon leavers employed than in education• Other School and Other Work relatively

unchanged▫Dip in middle year- 2010▫Slightly more leavers in Other Work than Other

Education• Not Engaged rate decreasing – right direction• Why look further?

Are there differences in outcomes by subgroups?

Gender x 3 years Disability categories x 3 years Ethnicity categories x 3 years Method of Exit x 3 years

Hi Ed CompEmp OthSch OthWork NE0

20

40

60

80

100

Engagement Males

2010 M 2011 M 2012 M

Differences in Outcomes by Gender

Hi Ed CompEmp OthSch OthWork NE0

20

40

60

80

100

Engagement - Females

2010 F 2011 F 2012 F

Perc

ent R

epor

ted

Observations for Outcomes by Gender

• More Females than Males in Hi Ed and both groups have fairly static trend

• More Males than Females in Competitive Employment with increase in trend for Males

• Other School engagement about the same rate for Females and Males

• Other Employment similar rates, but Females have increasing trend over time

• Not Engage decreasing trend for both groups

Differences in Outcomes by Disability Categories: SLD and ED

2009 2010 20110

20406080

100

Specific Learning Disability

1 HE 2 CE 3 Oed 4 OW 5 NE

2009 2010 20110

20

40

60

80

100Emotional Disturbance

Differences in Outcomes by Disability Categories: All Other (Low Incidence) and

ID

2009 2010 20110

20406080

100Intellectual Disability

2009 2010 20110

20406080

100

Low Incidence

1 HE 2 CE 3 Oed 4 OW 5 NE

Observations for Outcomes by Disability• SLD: slight positive trend for HE and CE and slight

negative trend for NE – trends going in desired direction• ED: negative trend in HE, but positive trend in CE;

negative trend in NE- need to explore HE • AO/Low Incidence: Slight increase in CE, other

engagement categories unchanged• ID: negative trend in HE, positive trend in all other

engagement categories; highest group NE, but decreasing

• Regardless of disability, about 1/3 of respondents are NE, HOWEVER, the trend is headed in the desired direction – seeing a negative trend in all disability categories - need to explore NE

Differences in Outcomes by Race/Ethnicity Categories

2009 2010 20110.0

20.0

40.0

60.0

80.0

100.0 Minority

2009 2010 20110.0

20.040.060.080.0

100.0

White

1 HE 2 CE 3 Oed 4 OW 5 NE

Combined Years for Sufficient Size of Race/Ethnicity Subgroups for Comparison

Asian n=128

Black n=185

Hispanic n=725

Nat Amer n=210

White n

=4348

0

20

40

60

80

100

Combined Three Years

1 HE 2 CE 3 Oed 4 OW 5 NE

Race/Ethnicity Categories

Perc

ent

of R

espo

nden

ts

Observations for Outcomes by Race/Ethnicity

• There are very minor outcome differences when all minorities are in one subgroup

• There are insufficient interviews to look at trends over the 3 years by each minority group

• Combining the data from the three years, as a representative sample was interviewed, allows further exploration

• Differences are seen in the students combined into the minority subgroup that need to be explored further by the state and districts

Outcomes by Method of Exit category Regular Diploma

2009 2010 20110

20

40

60

80

100

Regular Diploma

1 HE 2 CE 3 Oed 4 OW 5 NE

Outcomes by Method of Exit category Dropout

2009 2010 20110

20

40

60

80

100

Dropout- Not representative

1 HE 2 CE 3 Oed 4 OW 5 NE

Observations for Outcomes by Method of Exit Groups: Regular Diploma and Dropout

•Regular Diplomas: slight increases in HE and CE, and decrease in NE; trends are going in the desired directions

•Dropouts – not representative of the state▫Decrease in HE, slight increase in CE with a

dip in 2010▫Slight increase in Other Education; static in

Other Work▫Decrease in NE

Summary Observations•Not Engaged – negative trend

▫Rate is slow▫High number of youth in some

subgroups ID Dropout

•Higher Ed and Competitive Employment shifting to Other categories▫ED – negative trend on HE with

increase in Other Education

What is contributing to our outcomes?

Supplemental survey questions can help answer this question.

•Additional questions included on the follow- up interview in Oregon:▫Do you have a drivers license?▫What is your living situation?▫What one thing would you tell your school?▫Which independent activities can you do? ▫Do you receive benefits like co-workers?▫What do you do for recreation?▫If you haven’t worked, why not?▫Have you received support from adult

Agencies?

What Agency Services have you accessed since leaving school?The list of agencies on the follow-up

interview includes:▫Social Security Disability Insurance or

Supplemental Security Income ▫Developmental Disability services▫Office of Vocational Rehabilitation▫Temporary Assistance for Needy Families▫Supplemental Nutrition Assistance Program▫College Disability Services▫Loans, Financial Aid

Change in Outcome Classification

•For the next series of charts, the outcome groups were modified to allow a closer look at students who tried school or work, but were not successful.▫1 HE 2 CE 3/4 Other ed/work 5-Attempted 5-

None▫5-Attempted: students answered Yes to either

school/ training or employment, but did not continue long enough to qualify as ‘engaged’

▫5-None: students reported NO school/training or employment experience

Differences in Outcomes by Agency Services Received

2009 n=1695 2010 n=1792 2011 n=15560

20

40

60

80

100

21 27 2626 20 221911 912 17 17

No VR Services

2009 n = 216 2010 n = 198 2011 n = 1920

20406080

100

25 25 2526 30 3117 11 1117 21 17

Vocational Rehabilitation Services1 HE 2 CE 3/4 5 -Attempted 5 None

Differences in Outcomes by Agency Services Received

2009 n=205 2010 n=180 2011 n=1860

20

40

60

80

100 Developmental Disability Services1 HE 2 CE 3 or 4 5 Attempted

2009 2010 20110

20

40

60

80

100No DD Services

How Can We Use These Data: Next Steps

•Finalize the analysis ▫Other supplemental questions

•Share and discuss trend data▫ODE Transition Specialist▫Transition Advisory Council Stakeholders▫Agency Partners: ODDS and VR

•Determine what data to share and how to share ▫District and School Stakeholders

Looking at Data: Process summary • How representative are these data?

▫We explored the response size and how the subgroups matched the population

• What direction are our outcomes going?▫ Looked at graphs showing performance, trends, and

comparisons • Are there differences in outcomes by subgroups?

• Worked from general overview to more specific components• What is contributing to our outcomes?

• Looked at a combination of components, modified the question if necessary, and summarized what we learned at each step

•For more information:

▫Pattie Johnson Teaching Research Institute, Western Oregon University johnsop@wou.edu 503-838-8779

▫Charlotte Y. Alverson National Post School Outcomes Center, University of Oregon calverso@uoregon.edu 541-346-1390