An Analysis of the 2007 Examination Results

Presented at the Extended Management Committee

14 October 2008Associate Professor George Subotzky

Executive Director: Information and Strategic Analysis

Overview• This presentation focuses on the final 2007 examination results• It updates the preliminary analyses presented to the EMC in February

and subsequently to Council, the Audit Committee and Senex• New elements in the 2007 analysis:

– Exam results analysed in a way that is more consistent with HEMIS (to avoid multiple versions)

– Nonetheless, for the purposes of disaggregation, these figures based on course counts, whereas HEMIS success rate is based on FTEs, which are always slightly lower (both figures are on the Portal)

– Exam results analysed according to a more sophisticated and comprehensive exam modelling process and metrics – to highlight multiple points of risk

– Analysis of exam success placed in relation to the larger success & throughput framework & initiative. The purpose of this is not only to describe trends, but to understand and explain them and ultimately to use actionable analytic/business intelligence to identify interventions to effect meaningful change and improve practice

Acknowledgements

The following staff members of DISA provided invaluable help and support in preparing the examination results information and this presentation:

• Esmé Wiid• Herbert Zemann• Herman Visser• Suzette van Zyl

Modelling and Improving Success &

ThroughputCONCEPTUAL MODELLING:

• Explaining success and throughput by systematically identifying all factors and their interrelationships

MEASUREMENT: • Profiling/tracking by means of available/obtainable data &

informationSTATISTICAL MODELLING:

• Reliably establishing cause & effect• Identifying & predicting risk• Confirming/refining conceptual model

IMPROVING THROUGHPUT & SUCCESS:• Identification & implementation of interventions • Monitoring & evaluation of impact• Further improvements

Employ-ment/

Citizenship

Through-put/

Graduation

PersistenceRetentionSuccess

Learning activities/

interactions

Entry:Choice/Inquiry/

Enrolment

PROFILING/MODELLING/PREDICTION: - Snapshot/trends - Views: Course/Qualification; Dept/School/College/Institution

1. Socio-Economic Circumstances:

• Background/ Schooling

• Current

2. Individual Factors/Attributes:

• Ability• Skills• Attitudes & Interests

3. Institutional Factors:

• Quality• Relevance• Effectiveness

TRACKING: Individual-level progress through HE process aggregated by profile elements

Tinto’s model: point of departure

BACKGROUND

• Socio-Economic

• Schooling• Culture

Individual Attributes

PREPAREDNESS

• Academic• Socio-

Economic

INDIVIDUAL FACTORS

• Epistemological & Ontological Development

• Goal Commitment• Institutional

CommitmentPERSISTENC

E

• Academic integration

• Social integration

• Functional integration

INSTITUTIONAL FACTORS

• Learner Support• Academic & Operational

Systems

SOCIO-ECONOMIC FACTORS

• Conducive Conditions & Support: Finance, time, health & wellness

SUCCESS THROUGHPU

T&

GRADUATE-NESS

ROAP: Pre-registration

engagement: academic & socio-economic readiness, qualification and course choice & load Bridging interactions across distances (ODL)

Profiling, tracking & predicting individual, academic & socio-economic riskDesired outcomes:

Quality, relevance, service excellence, efficiency, effectiveness

Towards a model

Acknowledgements to Professor Chris Swanepoel, Dr Paul Prinsloo (members of modelling task team) and Dr At van Schoor

Key premises• Indications from considerable number of

research findings, as well as throughput cohort studies: non-academic factors play a significant role in impeding success & throughput – ontological/sociological construct at the heart of the throughput model

• Graduateness: ontological versus epistemological foundation: not so much what to students need to know, but who do they need to be in order to be effective in society as critical citizens and the market place as flexible innovators

Success

Gross Enrolments

Exam Admission

Writing

Results

Course Attrition Rate= (CA + NA + Abs)/Gross Enrolments

Course Success Rate= Passed/Nett

Enrolments

Exam Pass Rate= Passed/Wrote

Exam Results Metrics

Exam Process Model

3 Key Indicators

Key Counts

Drop Out/Stop Out

Re-registration(Repeaters)

Qualified Readmission

Stuck

7 Risk Areas

Sup.Admission

Sup.Writing

Sup.Results

AdmittedAbsent

Fail Write

Pass

Fail

Absent

Not Admitted

Not Admitted

Admitted

Write

Pass

Canc.+ Non-Active

NettEnrolments

Wrote Exam

Active

Gross enrolments

Absent & Voluntary Cancel

NonActive

Active Cancel & Not Admitted

Failed

Did not QualifyQualified for Suppl Exam

Course Success Rate (Combined Pass/Active)

7,6%

20,5%

100%

92,4%

Admitted to Exam

Passed (Exam Pass Rate)

1,4%

65,4%

79,5%

98,6%

34,4%Results

Outstanding

0,2%

Absent & CancelWrote Suppl Exam

FailedPassed Suppl

49,0% 51,0%

32,1%67,9%

49,8%

55,8%

50,2%

Course Attrition Rate (Nett)

44,2%

Overall Course Success Rates, 2004-7

2004 2005 2006 2007 Target0%10%20%30%40%50%60%70%80%90%

100%

Course Success Rate by College, 2005-7

CAES CEMS CHS CLAW CSET0%10%20%30%40%50%60%70%80%90%

100%

Course Success Rate by Previous Activity, 2005-7

Technikon student University student Other activity0%10%20%30%40%50%60%70%80%90%

100%

Course Success Rate by CESM, 2005-7

BUS/MAN EDUCATION OTHER HUM SET0%10%20%30%40%50%60%70%80%90%

100%

Course Success Rate by Age, 2005-7

<20 21-30 31-40 41-50 51-60 61-70 >700%10%20%30%40%50%60%70%80%90%

100%

Course Success Rate by Race, 2004-7

African Coloured Indian White0%10%20%30%40%50%60%70%80%90%

100%

Course Success Rate by Gender, 2004-7

Female Male0%10%20%30%40%50%60%70%80%90%

100%

Course Success Rate by Nationality, 2004-7

Other Nationality South Africa0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Key Indicators by CESM, 2007

Exam Admission Rate Exam Writing

Rate Exam Pass Rate Suppl. Exam

Pass Rate Combined Pass Rate Course Success

Rate

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

SET

BUS/MAN

OTHER HUM

EDUCATION

UNISA

Key Indicators by College, 2007

Exam Admission Rate Exam Writing

Rate Exam Pass Rate Suppl. Exam

Pass Rate Combined Pass Rate Course Success

Rate

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CSET

CEMS

CAES

CLAW

CHS

UNISA

Key Indicators by Gender, 2007

Exam Ad-mission Rate Exam Writing

Rate Exam Pass Rate Suppl. Exam

Pass Rate Combined Pass Rate Course

Success Rate

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Female

Male

UNISA

Key Indicators by Race, 2007

Exam Ad-mission Rate Exam Writing

Rate Exam Pass Rate Suppl. Exam

Pass Rate Combined Pass Rate Course

Success Rate

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

African

Coloured

Indian

White

UNISA

Key Indicators by UG/PG, 2007

Exam Admiss

ion Rate

Exam W

riting R

ate

Exam Pass

Rate

Suppl. E

xam Pass

Rate

Combined Pass

Rate

Course Succe

ss Rate

0%10%20%30%40%50%60%70%80%90%

100%

Undergraduate

Postgraduate

UNISA