1 accountability reporting for california community colleges patrick perry vice chancellor of...

1

Accountability Reporting for California Community

CollegesPatrick Perry

Vice Chancellor of Technology, Research, &

Info. SystemsCCC Chancellors Office

2

Data Preamble

“Information is the currency of democracy.” -Thomas Jefferson

“Get your facts first, then you can distort them as you please.” -Mark Twain

“In the twenty-first century, whoever controls the screen controls consciousness, information and thought.” -Timothy Leary

3

The CCC System

109 campuses, 72 districts, all locally governed

2.6 million students (annual unduplicated) 1.1 million FTES (annual) 35% white; half over age 25; 70% part-time No admissions requirements $20/unit; 40% get fees waived Highest participation rate of any CC system

in US; 25% of all CC students are CCC

4

Participation (and Fees)

5

CCC Chancellor’s Office

Weak authority; powers vested locally

Unitary MIS data collection (1992) Student, faculty, course, section,

session, grade level detail Data collected end of term, 3x/yr Used for IPEDS, apportionment,

accountability, research, online data mart

6

History of CCC Accountability

Simple reporting, fact books until 1998 1998: State provides $300m ongoing in

exchange for accountability reporting “Partnership for Excellence” was born

CCC developed report in isolation CCC allowed to determine “adequate progress” “Contingent funding” never triggered

Used 5 metrics to measure system and college-level performance

7

PFE Metrics

Annual volume of transfers to CSU/UC Annual volume of awards/certificates Rate of successful course completions Annual volume of Voc. Ed. Course

completions Annual volume of basic skills

improvements (lower to higher level) 4 of 5 are volume metrics, only 1 rate

8

The State Said:

Your metrics allow for no adequate college comparisons

Your method of determining “adequate progress” is suspicious

You only look good because you are growing

Partnership over (2001), but keep reporting, (until 2004) we have to spend your money buying energy

from Enron

9

What Happened Next

Gov. Gray Davis: recalled for spending money buying energy from Enron

Replaced by “The Governator”

10

The Governator

Likes Community Colleges Comes from a country that has

European “academic bifurcation” (Austria)-university vs trade paths

Attended Santa Monica Community College

Took ESL, PE, bookkeeping, micro/macroeconomics

Transferred to U. Wisconsin-Superior

11

And Arnold Said:

We shall haves deez accountabeelity seeztem for de community collegez.

A bill was passed to create the framework, and eventually the framework was enacted.

Named: Accountability Reporting for Community Colleges (ARCC).

12

Arnold Said:

There shall be no pay for performance, but there will be the ability to compare performance.

13

We Said:

Some metrics will be system only; others will be at college-level

College metrics will be rates (to mitigate size for comparison)

No rankings—we will compare colleges against their “peers”

No $$$=ARCC is a “dashboard” accountability report.

14

Arnold Said:

Colleges need to address their performance annually to the State.

15

We Said:

Colleges are more responsive to their local district Board; annual requirement to take local ARCC results to local Board and submit minutes to State

Colleges must submit 500 word response, which becomes a part of the final report.

16

Arnold Said:

The report shall be done in collaboration with the State, not in isolation.

17

We Said:

The Dep’t of Finance, Leg Analyst, and Secretary of Education shall be a part of the technical advisory committee (along with CCC researchers and stakeholders).

We will either succeed or fail together. This was a really smart move.

18

ARCC

The Model: Measures 4 areas with 13 metrics:

Student Progress & Achievement-Degree/Certificate/Transfer

Student Progress & Achievement-Vocational/Occupational/Workforce Dev.

Pre-collegiate improvement/basic skills/ESL Participation

“Process” is not measured

19

Student Prog. & Achievement: Degree/Cert/Xfer

College: Student Progress & Achievement Rate(s)

(SPAR) “30 units” Rate for SPAR cohort 1st year to 2nd year persistence rate

System: Annual volume of transfers Transfer Rate for 6-year cohort of FTF’s Annual % of BA/BS grads at CSU/UC who

attended a CCC

20

Student Prog. & Achievement: Voc/Occ/Wkforce Dev

College: Successful Course Completion rate:

vocational courses System:

Annual volume of degrees/certificates by program

Increase in total personal income as a result of receiving degree/certificate

21

Precollegiate Improvement/Basic Skills/ESL College:

Successful Course Completion rate: basic skills courses

ESL Improvement Rate Basic Skills Improvement Rate

System: Annual volume of basic skills

improvements

22

Participation

College: None yet…but coming.

System: Statewide Participation Rate (by

demographic)

23

Major Advancements of ARCC

Creating a viable alternative to the GRS Rate for grad/transfer rate.

Finding transfers to private/out of state institutions.

Doing a wage study. Geo-mapping district boundaries. Creating peer groups.

24

Defining Grad/Transfer Rate

Student Progress & Achievement Rate (SPAR Rate) IPEDS-GRS for 2-yr colleges stinks:

No part-timers How do you define degree-seeking? Tracking period too short Outcomes counting methodology terrible

AA/AS/Cert counted before transfer Transfer to 2-yr college is counted

25

SPAR Rate

Defining the cohort: Scrub “first-time” by checking against

past records (CCC, UC, CSU, NSC)

26

SPAR Rate

Define “degree-seeking” behaviorally for CC populations Not by self-stated intent; this is a poor

indicator Behavior: did student ever attempt

transfer/deg-applicable level math OR English (at any point in academic history) Students don’t take this for “fun”

27

Defining Degree-Seeking Behaviorally

Separates out remedial students not yet at collegiate aptitude Measure remedial progression to this

threshold elsewhere Creates common measurement

“bar” of student aptitude between colleges Same students measured=viable

comparison

28

SPAR Rate-Unit Threshold

CCC provides a lot of CSU/UC remediation Lots of students take transfer math/Eng

and leave/take in summer Should not count these as success or “our”

student Set minimum unit completed threshold

(12) for cohort entrance Any 12 units in 6 years anywhere in system

29

SPAR Denominator:

First-Time (scrubbed) Degree-seeking (at any point in 6

years, attempt transfer/degree applicable math or English)

12 units (in 6 years)

This represents about 40% of students in our system

30

SPAR Numerator

Outcomes the State wants: Earned an AA/AS/certificate; OR Transfer: to a 4-yr institution; OR Become “transfer-prepared”;OR

Completed 60 xferable units Became “transfer-directed”:

Completed both xfer level math AND English

No double-counting, but any outcome counts SPAR Rate=51%

31

Tracking Transfers

SSN-level matches with CSU, UC Nat’l Student Clearinghouse for

private, proprietary, for-profit, out of state Match 2x/yr, send all records since 1992 Update internal “xfer bucket”

Works great for cohort tracking Needed method for “annual volume”

32

Tracking Transfers

Annual Volume of Transfers CSU/UC: they provide these figures

based on their criteria We didn’t want to redefine this

Private/Out of State: NSC “cross-section” cut method

Validated against CSU/UC xfers from NSC source

Added another 30% to annual volumes

33

97-98 98-99 99-00 00-01 01-02 02-03 03-04 04-05 05-06 06-07

FTF → → → → → → → → T

FTF → → → → → → → R

FTF → → → → → → A

FTF → → → → → N

FTF → → → → S

FTF → → → F

MIN FTF → → E

12 FTF → R

UNITS FTF 06-07

34

Sector 01-02 02-03 03-04 04-05 05-06 06-07

CSU 50,473 50,746 48,321 53,695 52,642 54,391

UC 12,291 12,780 12,580 13,211 13,462 13,874

ISP 17,070 15,541 18,100 18,365 17,840 18,752

OOS 10,762 10,540 11,150 11,709 11,726 11,825

Total 90,596 89,607 90,151 96,980 95,670 98,842

35

Transfer: Sector of Choice

  % to UC% to CSU

% to Instate Private

% to Out of State

White 17.9% 60.7% 11.0% 10.4%

AfrAm 11.5% 51.2% 18.1% 19.2%

Hisp/Lat 15.1% 67.7% 12.1% 5.1%

Asian 37.0% 49.9% 9.2% 3.9%

36

Demography of Transfer

Demog (06-07)

FTF Stdents

All Stdents

XFER-CSU

XFER-UC

XFER-ISP

XFER-OOS

AfrAm 9% 8% 5% 3% 11% 13%

Asian 11% 12% 12% 26% 8% 7%

Hisp/Latino 35% 29% 23% 16% 23% 13%

White 29% 35% 37% 40% 44% 55%

37

The Rise of The Phoenix96-97 2,166

97-98 2,829

98-99 3,374

99-00 4,194

00-01 5,055

01-02 5,586

02-03 6,515

03-04 8,222

04-05 8,585

05-06 8,134

06-07 9,216

38

Who Transfers to Phoenix?

Ethnicity UC CSU Phoenix

Asian 29.3% 14.2% 4.6%

African American 2.4% 5.2% 16.8%

Hispanic/Latino 13.6% 23.8% 28.6%

White 39.1% 43.6% 37.5%

39

Wage Study

What was the economic value of the degrees (AA/AS/certificate) we were conferring?

Required data match with EDD Had to pass a bill changing EDD code

to allow match

40

Wage Study

Take all degree recipients in a given year Subtract out those still enrolled in a

CCC Subtract out those who transferred to

a 4-yr institution Match wage data 5 years

before/after degree

41

Wage Study

Separate out two groups: Those with wages of basically zero

before degree Those with >$0 pre wage

The result: The Smoking Gun of Success

42

43

Mapping Districts

CC Districts in CA are legally defined, have own elections, pass own bonds

We did not have a district mapping for all 72 districts So we couldn’t do district

participation rates

44

Mapping Project

Get a cheap copy of ESRI Suite Collect all legal district boundary

documents Find cheap labor—no budget for

this

45

46

Peer Grouping

“Peers” historically have been locally defined: My neighbor college Other colleges with similar demography Other colleges with similar size

47

Peer Grouping

Taking peering to another level: Peer on exogenous factors that predict the

accountability metric’s outcome Thus leaving the “endogenous” activity as the

remaining variance Cluster to create groups

We picked 6 clusters, with a min of 3 in a cluster

Each metric produces different factors, peers, clusters

48

Peer Grouping: Example

Peering the SPAR Rate: 109 rates as outcomes Find data for all 109 that might

predict outcomes/explain variance Perform regression and other magical

SPSS things See how high you can get your R2

49

Finding Data

What might affect a grad/transfer rate on an institutional level? Student academic preparedness levels Socioeconomic status of students First-gen status of students Distance to nearest transfer institution Student age/avg unit load

50

Finding Data

We had to create proxy indices for much of these (142 tried) GIS system: geocode student

zipcode/ZCTA Census: lots of data to be crossed by

zip/ZCTA Create college “service areas” based

on weighted zip/ZCTA values Different than district legal boundaries

51

52

Finding Data

The Killer Predictor “Bachelor Plus Index”, or what % of

service area population of college has a bachelor’s degree or higher

“Bachelor Plus Index” a proxy for: First gen Academic preparedness Socioeconomic status Distance to nearest transfer institution

53

Peering SPAR Rate

Exogenous factors that predict SPAR Rate: Bachelor Plus Index % older students % students in basic skills

R2 = .67 What’s left is implied institutional variance

Demo

54

Peering: What’s Bad

Its complex and somewhat confusing and labor intensive.

Colleges traditional notion of “peer” is shaken

Multiple peers for multiple metrics; can change every year

You could do well vs. State average, increasing over time, but last in your peer group

55

Peering: What’s Good

Its complex and somewhat confusing

You will likely look good in some areas, OK in others, and low in others

Its not very likely anyone will be high or low in all 6 metrics

It eliminated rankings.

56

The ARCC Report

Is almost 800 pages. Comes out every March. Takes 4 PY’s to complete (about 6

months/yr) Is generally regarded highly in CA

academic and Legislative circles. DOF and LAO and Sec. of Ed love it. Local Trustees/Boards love it.

57

The ARCC Collaboration

Has brought the system more money: $33 mil in basic skills Increased noncredit reimbursement

rates by $300/FTE Has brought about trust between

system and State stakeholders. Has educated both sides

tremendously.

58

No More “Girlie-Man” Accountability!