vvv evaluating student services 3 - conference design · kra 1 enhanced student experience key...

Making our contribution visible:Making our contribution visible:Making our contribution visible:Making our contribution visible:

Evaluating the value and impact

of student services

Karen DavisAssociate Director

Student Academic ServicesVictoria University of Wellington

Alison DowDirector

Pou Aroha Student SupportUnitec, Auckland

Mā pango, mā whero, ka ōti te mahi

>>UNITEC INSTITUTE OF TECHNOLOGY

By the end of the session we will have

• Reviewed the ways in which two NZ tertiary providers are approaching the evaluation for service delivery.

• Considered how evaluation focuses on visibility, viability and value.

• Identified some ways we might develop and share our evaluation processes and outcomes

Session Outcomes

>>DEPARTMENT TITLE EDIT IN HEADER & FOOTER

1

3

2

2

4


• New Zealand’s largest Polytechnic

• Certificate to PhD programmes

• 20,000+/- students, diverse student population

• “A great job for every graduate”

• Undergoing transformation

UnitecEvaluating Student Services

Victoria University of Victoria University of Victoria University of Victoria University of WellingtonWellingtonWellingtonWellington

• Capital city university

• One of 8 New Zealand universities

• Ranked 1st in PBRF (research ranking) in 2014

• 22,000 students

• Many from out of town, 60% of first year students living in Halls of residence

• 10% Maori students, 5% Pasifika students, 12% international

Development of an Outcomes Framework Development of an Outcomes Framework Development of an Outcomes Framework Development of an Outcomes Framework

for Student Services at Victoria Universityfor Student Services at Victoria Universityfor Student Services at Victoria Universityfor Student Services at Victoria University

Why develop a framework?

Initial focus levy funded services

Aims to:

- show how services and interventions contribute to outcomes for students and the university

- articulate direction for improving quality of services and student satisfaction

Student levy funded

services

Advocacy and legal

advice

Careers information,

advice and guidance

Counselling and

pastoral care

Employment

information

Financial support and

advice

Health services

Media

Childcare services

Clubs and societies

• Government Outcomes

- Tertiary education outcomes

• Victoria University of Wellington Strategic Plan

- Vision / Primary strategies / Enabling strategies

• Student Services Vision

- Key result measurements / Supporting strategies

• Individual Student Services

- Objectives / Outputs / Operational key performance

indicators

Student Services for Success FrameworkStudent Services for Success FrameworkStudent Services for Success FrameworkStudent Services for Success Framework

Measurement of contribution to university outcomesMeasurement of contribution to university outcomesMeasurement of contribution to university outcomesMeasurement of contribution to university outcomes

• High level Key Result Measurements

- external and internal performance indicators

• Complemented with information from surveys

- Student Experience Improvement Survey

- Graduate Destination Survey

- i-Barometer

• Engagement and outcomes

- compares students who used student services with those who did not engage

• Lower level operational KPIs for individual services

- Service specific data and staff survey data

to demonstrate we are …

How will the outcomes framework helpHow will the outcomes framework helpHow will the outcomes framework helpHow will the outcomes framework help

VISIBLE

VALUABLE

VIABLE

Next steps …Next steps …Next steps …Next steps …Data – what is the Victoria story ?

Working on a reporting template and dashboard – see draft below

KRA 1

ENHANCED

STUDENT

EXPERIENCE

Key Result MeasurementsStudents

engaged

with

student

services

All

students

*EPI, ^Internal PI

Percentage of students who engaged with at least 1 levy-funded

student service

Satisfaction rating for the overall performance of services and

faculties

Graduates that are in employment, further study and not seeking

work* who engaged with Careers & Employment

Percentage of students who have a clear sense of academic purpose

and direction

Percentage of students who have engaged with student services who

feel their daily life has been filled with things that interesting to them

Number of students contributing to extra-curricular programs run by

student services

Number of students that achieved the extra-curricular VicPlus

Award*

Number of students contributing to peer support paid & unpaid

through a student service

Number of students providing mentoring through a student service

i-Barometer International students propensity to recommend

Will also include information about external and internal / opportunities / risks


Work with the tools we have…..

• Student ID Numbers provide demographic data

• Report run annually on success and retention of students provided (currently 1:1 support, working on group/workshop)

• WuFoo surveys by specific service teams

• Planning and Insights team run annual Net Promoter Score Survey including questions about

knowledge and use of services

student willingness to recommend Unitec to their peers; and

open feedback.

Evaluating Student Services: Pou Aroha Student Support


7


How Net Promoter Score is Calculated



From this we found we can generate reports to show:

success and retention rates for students using services

awareness and use of services

And from these reports we can make some tentative conclusions

about the correlations between variables:

Success

Retention

Awareness of services

NPS scores

Evaluating Student Services



VISIBLE


6


VIABLE


8

8

7


VALUABLE


VISIBLE


Find your buddies

Collect a set of reports

For your V Theme:

• Which data is most useful and what does it tell you?

• What are the limitations of the data?

• What additional data would help?

• What questions or suggestions can you add?

Your task



• Successful students use services

• Students who use services are successful

• Students who use services have higher NPS scores

Follow up:

Your suggestions about how ANZSSA might facilitate the development and sharing of evaluation processes and outcomes

Tentative Conclusions Overall


Visibility of services – measured in NPS student survey

“The Value of Visibility”

Value: Retention

82%

85%

76%

74%

76%

81%

68%

70%

72%

74%

76%

78%

80%

82%

84%

86%

88% Retention ratesUnder 25-did

not use a

service

Under 25-used a

service

Maori-Used a

service

Maori-did not

use a service

Pacific-did not

use a service

Pacific-Used a

service

Value: Success

73%

79%

71%

68% 68%

74%

60%

62%

64%

66%

68%

70%

72%

74%

76%

78%

80%

82%

Success rates

Under 25-did not use a

service

Under 25-used a service

Maori-Used a service

Maori-did not use a

service

Pacific-did not use a

service

Pacific-Used a service


Maori and Pacific Success rates


50%

60%

70%

80%

90%

2009 2010 2011 2012 2013 2014

CIB Course Success Rate SHS Course Success Rate

TBE Course Success Rate Unitec Course Success Rate

50%

60%

70%

80%

2009 2010 2011 2012 2013 2014

Maori Students

PacificStudents

>>UNITEC INSTITUTE OF TECHNOLOGY >>DEPARTMENT TITLE EDIT IN HEADER & FOOTER

They treated me They treated me

with respect and

dignity, I would

recommend this

service to all

students

I accessed two services – [one]

was very helpful. [The other]

resulted in me wasting time

chasing assistance I could not

get….

The service I receive enables me to continue The service I receive enables me to continue The service I receive enables me to continue The service I receive enables me to continue studies. If it were not for the counselling studies. If it were not for the counselling studies. If it were not for the counselling studies. If it were not for the counselling service I would not be continuing studies service I would not be continuing studies service I would not be continuing studies service I would not be continuing studies and not be pursuing my dreamsand not be pursuing my dreamsand not be pursuing my dreamsand not be pursuing my dreams…

Really appreciateReally appreciatethe student services.. Especially the disability support and the workshops around managing anxiety and study and free counselling

I felt that may be I felt that may be

there was not

enough information

about available

services

Its important to put a face to the

service. I have also been made

aware of the level of services not

coming across as helpful or

friendly as they could of…. The student wellbeing service was very good. I was under financial pressure

and finishing my thesis. I was granted petrol vouchers that helped with my financial overheads. No improvement required in my experience, very helpful

Nothing to comment on

because all is well.

thank you

late comer for appointment

should go last, not jump que and

delay for those go on time.

the doctor is amazing and

so personable

>>UNITEC INSTITUTE OF TECHNOLOGY >>DEPARTMENT TITLE EDIT IN HEADER & FOOTER

1,169

322 340

68

151

291

1246

469

548

51

185

328

0

200

400

600

800

1,000

1,200

1,400

Health Counselling Careers service Chaplain DLC Student Advisory

Number of Students using Services 2013-2014 comparison

2013 2014

How Many…?

VIABLEHHow does service use affect NPS?


May 2014 Support v Net Promoter Score (+ve correlation)



1. http://pad2.wh11static.com/images/thumb/a/a8/Find-Waldo-Step-1.jpg/670px-Find-Waldo-Step-1.jpg

2. http://www.bing.com/images/search?q=Success+Clip+Art&view=detailv2&id=04F9029AD825BF4178DA22E6A90E9CE1D977CED6&selectedindex=91&ccid=tN99Z%2BKz&simid=608002687973786472&thid=OIP.Mb4df7d67e2b3cf4b29f42bb7ce586f75o0&mode=overlay&first=1

3. http://www.bizbuilder.com/wp-content/uploads/2013/06/valuations.jpg

4. http://www.bing.com/images/search?q=feasible&view=detailv2&id=0B923885EBC00F7663F546DA6016C3539CA8D375&selectedindex=47&ccid=vvHm9sGR&simid=608049833826320389&thid=OIP.Mbef1e6f6c1912d83dad5307be1a44eb0o0&mode=overlay&first=1

5. http://www.bing.com/images/search?q=business+viability&view=detailv2&id=034655A1645CE70AE65E556E6A9980A96E521042&selectedindex=48&ccid=ZRh59syt&simid=608047024915353167&thid=OIP.M651879f6ccad020ff6095d49fac57577o0&mode=overlay&first=1

6. http://www.bing.com/images/search?q=Great+Outcomes+Clip+Art&view=detailv2&id=1F72728284979AC16A0DE48058F290EBA2AFB6E3&selectedindex=44&ccid=aqivgR%2FP&simid=608024592313091494&thid=OIP.M6aa8af811fcf49de77a0e899c30f2040H0&mode=overlay&first=1

7. http://procto.biz/wp-content/uploads/2014/01/minimum-viable-product-demystified-part-2.jpg

8. http://www.bing.com/images/search?q=viable+clip+art&view=detailv2&id=18D835E0CAE488B273B1A97CEFC1EEB8B553D8FF&selectedindex=1&ccid=HJRfswNu&simid=608009259277484057&thid=OIP.M1c945fb3036e1efd7ff936b23b136703o0&mode=overlay&first=1

Images: Downloaded 22 November2015


vvv evaluating student services 3 - conference design · kra 1 enhanced student experience key...

Documents