www.monash.edu.au platform influences on survey delivery and response rate presenter phillip adams...

www.monash.edu.au

Platform Influences on Survey Delivery and Response Rate

Presenter

Phillip AdamsEvaluations Manager

Australasian Higher Education Evaluation Forum 2008Friday 2 October, 2008

www.monash.edu.au

Presentation Outline

• CHEQ (Centre for Higher Education Quality)• The Way We Were• The Response• The Outcomes

www.monash.edu.au

CHEQ

• Established in September 2000

• Lead and support quality assurance and improvement

• various areas of teaching, research, research training and support services

www.monash.edu.au

The Evolution

Two drivers for the change –

• Quality was an integral part of the higher education sector

• Need for institutional measures of academic activity

www.monash.edu.au

Unit EvaluationsThe way we were

• 1998 – 2002

• item bank

• academics create own survey

• 2002 – 2004

• faculty takes responsibility

• faculty-wide questionnaire

• conduct at least every five years

www.monash.edu.au

The issues

• 1998 – 2002

• singular, aggregated report per unit/subject

• no university benchmarking capacity

www.monash.edu.au

The issues

• 2002 – 2004

• patchy take up

• saw mix of academic & faculty reports

• still no university benchmarking capacity

• few opportunities for monitoring improvement

www.monash.edu.au

Major change in 2005

• 10 common items across university

• Up to 10 faculty items

• To be taken of each unit each year it is offered

• Results posted on a common website

• Results to be systematically considered by faculty

www.monash.edu.au

Re-engineering Technology And Processes

• both paper-based and online surveys for all units

• large volume of survey responses in short turnaround time

• system capable of storing data in any survey, online or paper, in single location

• improved access to results for staff and key stakeholders

• reduced cost of survey development and processing

• allowed automated reporting on WWW

2005, CHEQ introduced Survey Management System (SMS). Significant benefits from change in technology and processes:

www.monash.edu.au

PDF survey

• Benefits

• able to fill in without downloading

• presentation appealing

• Drawbacks

• practically clumsy

• poor accessibility for visually impaired

www.monash.edu.au

PDF survey limitations

• Three main types of student queries/complaints

• students unfamiliar with a fillable PDF

• browser version incompatibilities

• web browser setup

• Plus

• lack of accessibility for visually impaired

• disenfranchised students

www.monash.edu.au

Our response

• prepared “how-to” information sheets

• initially assisted VI over phone and with customised survey

• shifted from PDF to HTML form

www.monash.edu.au

Outcomes

• student familiarity

• accessible for visually impaired

• assisted over phone

• special form design considerations

• No browser compatibility issues

www.monash.edu.au

Outcomes

• Zero complaints about accessAccess Problems

157

109

0 00

20406080

100120140160180

2005 2006 2007 2008

Years

Nu

mb

er o

f co

mp

lain

ts

www.monash.edu.au

Outcomes

• Increase in web based response rate Annual Online Response Rate

26.330.8

33.8

42.8

0

10

20

30

40

50

2005 2006 2007 2008 (s1)

Evaluation period

Res

po

nse

rat

e (%

)

www.monash.edu.au

Conclusion

• Quality cycle relies on the effectiveness of the evaluation system

• Better meet the needs of stakeholders

• Survey more user friendly

•Positive influence on response rate