Download - Measuring Our Success: Providing Evidence for CTE Outcomes

KC Greaney, Santa Rosa Junior College

Mallory Newell, De Anza College

Terrence Willett, RP Group

Measuring Our Success: Providing Evidence for CTE Outcomes

RP Conference, Kellogg West, CAApril 2013

2Measuring Our Success, RP Conference, April 2013

A Brief Overview

- CTE SURVEY OUTCOMES

- CTE LAUNCHBOARD PROJECT

- CTE USERS FEEDBACK SURVEY

- CTE OS DATA VISUALIZATION TOOL

3

2012 Statewide Effort:CTE Employment Outcomes Survey

• Pioneered at Cabrillo College over past 12+ years• Replicated at other individual colleges including

Chaffey, Los Rios, Riverside, Mt. SAC, Santa Monica

• Statewide effort instigated by Bay Area Vocational Deans

Measuring Our Success, RP Conference, April 2013

4

CTE Employment Outcomes Survey

• Collaborative effort among community college consortia, the Chancellor’s Office, and the RP Group

• Dual purposes• Provide insightful information for local program improvement

• Collect statewide data for documentation of CTE outcomes

• Practitioner-Driven Effort• Statewide meetings, conference calls, collaboration

• All methodology, implementation and survey questions vetted

• Report format developed in response to practitioner needs


5


Methodology• 2 Surveys: Completers, Leavers• 3 Modalities: e-mail, then US mail, then phone calls• Survey conducted approximately 1.5 years after

completing/leaving

Sample• 15 Colleges• All students included who met the criteria

• By 2009/10, had either earned a vocational award of 6+ units OR met VTEA “concentrator” definition AND did not enroll in 2010/11

• Total = 15,105 (7,681 Completers and 7,424 Leavers)Measuring Our Success, RP Conference, April 2013

6


Data• Chancellor’s Office MIS data wherever possible• Local data supplement (student contact information)• Survey data• Complete data set provided to colleges for further analysis


7


Response Rates• Overall, 30% (Completers: 34%, Leavers: 26%)• By College, overall response rate ranged from 19% to

37%

Reports• Statewide reports for Completers and Leavers• College-level reports for Completers and Leavers


8

Statewide Findings: CompletersThe big questions…

• Are our CTE students employable?• Do they find jobs in the field in which we trained them?

• Do they see an increase in income?

…in other words, was the program worth it?


9

Statewide Findings: Completers

• 34% Response rate• 94% Satisfaction rate with education and training• 30% transferred (25% pursuing related field)• 77% working in their field of study (or “close”)• 73% are interested in taking more classes or training in their field in the near future

• 45% successfully engaged in a job search, 14% did not (the remainder were not looking or were already employed)


10

Findings: Completers

79% of respondents are employed for pay


11

Findings: Completers

More students worked full-time after completing studies and training

Full time (40+ hours per week)

Part time (20-39 hours per week)

Part-time (less than 20 hours per week)

0% 10% 20% 30% 40% 50% 60% 70%

After Before


12

Findings: CompletersThe hourly wage of degree/certificate earners increased 39% from their hourly wage before their studies ($17.78) to their hourly wage after completing a degree or certificate ($24.79)


13

Statewide Findings: LeaversThe big questions…

• Are our CTE leavers “losers”? Or have we failed them?• Are leavers employable?• Do they find jobs in the field in which we trained them?• Do they see an increase in income?

…in other words, were the courses and training worth it, even if the student did not complete a program of study?


14

Statewide Findings: LeaversSimilar to Completers, except:

• Lower Response Rate (26% versus 34% for Completers)• Fewer with the goal of earning a certificate or degree (60%

versus 80% for completers)• Twice as many with the goal of upgrading job skills and/or

renewing a license/permit (19% versus 10% for Completers)• Slightly higher transfer rate (34% versus 30% for Completers)• Slightly fewer report working in their field of study (or “close” –

70% versus 77% for Completers)• Earnings increase is lower (26% increase versus 39% for

Completers)


15

Basic DemographicsCompleters Leavers All CCCs

Gender: Female 60% 51% 54%

Asian 18% 17% 11%

Black 6% 5% 7%

Latino 19% 20% 30%

White 45% 47% 32%

DSPS 8% 4% ~

ESL 7% 5% ~

Acad. Disadvantaged 47% 40% ~

Migrant Worker 8% 6% ~

n=7,681 n=7,424 n=2.8 million


16

“Swirl”

Completers Leavers

Attended multiple colleges within the same district

13.5% 14.4%

Attended multiple districts 40.6% 39.7%

Attended only one college 44.6% 44.8%


17

CTE Survey: Changes in 2013• Implement all the lessons we learned from the pilot year

• 35 colleges participating• Move away from “Completers” versus “Leavers”• Move towards “CTE Skill Builders”• Broaden cohort to students who completed 9+ units of SAM-coded vocational courses (to include those who completed a certificate or vocational degree)

• Tighter timeline (reports out by the end of June)• Add questions re: industry certification, impact of coursework, more responses taking transfer into account


18

CTE Survey: Changes in 2014

• Santa Rosa Junior College will continue to administer survey, working directly with interested colleges

• Chancellor’s Office endorses survey and will continue to define cohort via MIS and provide basic data

• RP Group is supportive, but no longer directly involved• CCCAOE continues to support survey

• At Spring 2013 annual meeting, the CCCAOE “Excellence in Collaboration” award given to RP Group (Kathy Booth), Bay Area Vocational Deans Group (Rock Pfoetenhauer and Kit O’Dougherty) and Santa Rosa Junior College (KC Greaney)

• CalPASS+ will host CTEOS survey data in future



CTE User Feedback Survey

20

CTE User Feedback Survey• A link to the survey was emailed to all college contacts

who participated in the survey in the 2011-12 academic year

• Received 7 valid responses• Respondents suggested that the results had largely

been used to inform decision making in the areas of:

• Enrollment management

• Accreditation

• Perkins or other reporting

• Program improvements

• Changes to program degree offerings


21

CTE User Feedback SurveyAdditional Highlights

• Respondents found the following to be most valuable about the survey: • Ease of administration by the RP Group

• The multiple ways students were contacted to increase response rates

• The overall findings from the survey from my college

• The model report that was provided

• Respondents stated the following technical support would have been beneficial: • Assistance with the initial data pull of students

• Assistance with analyzing the results

• Examples of how other colleges shared the results and with whom



Launchboard Project

23

CTE Launchboard• 4 Community Colleges participating in the pilot• Includes data elements from “original” Launchboard

Metrics (OLB) and metrics from the “Common Metrics” (CMLB) associated with the Doing What Matters Framework: http://doingwhatmatters.cccco.edu/.

• Currently includes 67 data elements• All metrics are presented by program TOPS code • Some metrics presented by region as defined in the

Doing What Matters Framework• Includes data from: COMIS, CalPASS Plus, EDD,

Centers for Excellence, and data files submitted by colleges


http://doingwhatmatters.cccco.edu/

24

CTE LaunchboardIncludes measures in the following areas:• Alignment – middle school to high school and high

school to CCC• Enrollments – course completion in high school and

CCC in CTE pathway• Capacity – FTEF, students served in CTE programs by

CCC, students served in similar programs in non CCC colleges

• Support – completed work readiness program, internship, took basic skills course, received financial aid, DSPS or EOPS

• Return on Investment – number of years of income to equal cost per students


25

CTE LaunchboardMilestones• Completed major requirements in a CTE area

(Primary Goal) (CMLB)• Earned more than 8 units in the same TOP code

within 3 years of first enrollment (OLB)• Term-to-term retention (OLB)• Completed college-level English or math within 3

years of first enrollment (OLB)• Course completion and success (OLB)


26

CTE LaunchboardCredentials• Industry-recognized certificates (OLB)• Non-credit certificates (OLB)• Chancellor’s Office approved credit certificates in

and not in CTE pathway (OLB)• Associate’s degree in and not in CTE pathway

(OLB)• Transfer in and not in CTE pathway (CMLB)


27

CTE LaunchboardEmployment:• Students are employed in same or similar field

(OLB)• Wage gain in field (OLB)• Wage gain for students earning 9+ units (OLB)• Average wage of students who are employed in

same or similar field (OLB) • Job retention (OLB)


28

CTE LaunchboardDemand:• Recent demand in Doing What Matters

Economic Region (OLB)• Projected demand in Doing What Matters

Economic Region (OLB)• Present demand in Doing What Matters

Economic Region (OLB)• Median (50th percentile) Hourly Earnings (OLB)


29

CTE Launchboard

Live look at the Launchboard:

http://www.calpass.org/


30

• Proposed• Help users access CTE Outcomes

Survey data with web based reporting tool


CTE OS Data Visualization Tool

Questions?


32

Find Out More

KC Greaney, Santa Rosa Junior [email protected]

Mallory Newell, De Anza [email protected]

Terrence Willett, RP [email protected](831) 461-4586

http://www.rpgroup.org/projects/CTE-Employment-Outcomes-Survey

mailto:[email protected]






Download - Measuring Our Success: Providing Evidence for CTE Outcomes

Top Related