31st Annual EAIR ForumVilnius, Lithuania23 - 26 August 2009 Y R Mc NicollA R Luff

Is Quality Improvement fit for purpose? Comparing quality systems within a major Australian University

Background

• No universal accreditation of degree programs• In 2002 Australian Universities’ Quality Agency

adopted ‘fitness for purpose’, QI approach:

– “does not impose an externally prescribed set of standards upon auditees [our emphasis]. AUQA considers the extent to which institutions are meeting these objectives, and how institutions monitor and improve their performance.

• Bradley review (2008) advocates Standards

2

More Background

• Who are we? • Society and HE student body are

significantly multicultural – in 2007, 455,000 international students

predominantly from South East Asia• Monash University• Faculty of Medicine, Nursing & Health

Sciences• Quality process trial

> 3 participating faculties3

The problem

• This paper presents observations about the effects of a quality improvement based system,

– attempts to identify the value added, and to– determine whether the time, effort and money invested in

the process offer a good return. • How effective are quality reviews at identifying the

real problems?• What problems can they create? • What value do they add?• Do they necessitate management reviews to

address specific problems? • Are the two kinds of reviews complementary?

4

For whose benefit do we “do quality”?

• Universal suffrage: interests of society are synonymous with those of government– Education debate is limited

• University interests:– Students

– Academics

– Leadership

5

Methodology

• Significance– Monash places a high priority on excellence

in teaching and learning and of the overall “student experience”

– clear implications for Australia’s education export market

• Data– 3 quality reviews: 2 courses and 1 faculty– 1 ad hoc management review

6

Principles for Quality: What sort of Quality?• fitness for purpose;• quality is the professional responsibility of each individual and work group;• the best way to effect quality assurance and accountability is through

continuous quality improvement via collaboration and organisational learning;

• policy should assure comparable treatment in all areas of the university, while leaving room for different areas to develop implementation suitable to their contexts;

• an open, thoughtful and complementary approach to quality informed by international research and scholarship;

• a planned and systematic approach to quality including ensuring that the results of monitoring and evaluation are fed back in order to effect improvement;

• external points of reference provide valuable perspectives for further reflection and action.

7

Process 1: Quality review

• Broad-sweep, ‘helicopter view’• Improvement-oriented process

1. Self-review followed by 2. External validation by a panel,

> Interviews stakeholders, site visits, and document review

• Variations to process permitted– Refinements attempt to

> Balance student interests with FMNHS needs> Align with university direction

8

Process 2: Management review

• Conventional, but unformalised • At the Dean’s discretion• Single- step review by a largely internal panel

with a brief to address specific problems. • In the FMNHS such ad hoc reviews are not

uncommon.• This paper refers to the FMNHS process,

drawing on the Biosciences teaching review report

– Access to equivalent reports of other organisational units is limited due to their sensitivity

9

Purpose & Scope: Quality review

• Terms of Reference (ToRs)– Course: Structure, Management (including planning, QA,

development), Units and Major & Minor Sequences; Student Profiles, plus

– in common with review of Faulty organisation:> Teaching, Learning & Assessment; Human, Physical &

IT Resources, and Health & Safety issues; Professional and Community Engagement.

• Concatenation with professional accreditation requirements has been rare

• FMNHS guidance– Elaboration of ToRs: for convenors– External ToRs: for panels

10

Subtle differences of purpose

• BSc program review report– “to test and validate (or otherwise) the findings of the Self Review …;– ... “to assess and test the quality and adequacy of the

course in meeting its objectives, and follows the guiding principles outlined in the Course Review Policy”.  

• BCom review does not identify a purpose, • FMNHS identifies a two-fold purpose...

– reflection within the organisation, – identifying strengths, weaknesses threats and opportunities

for improvement,– [seeking] the perspective of outside experts (Williams and et

al 2007, p.12).

11

What value do quality reviews add?

• Accommodate discipline differences– Speak the language of the discipline

• See further or more clearly than self-review teams

• Reinforce each other’s findings with common recommendations

• Serve both professional and generalist degrees

12

How effective are these reviews at identifying the real problems?

• Do the skeletons tumble out of the closet?– Absence of (effective) course management– Understaffing– Inadequate accountability for improvement– Poor rationale for course structure & content– Disadvantageous and draconian assessment – Confused marketing

• Bold, wide-ranging recommendations emerge– Regardless of panel composition, – Panel experience or guidance is key

13

What problems does quality review create?• Recommendations

– That the evaluand has no power to address– Political recommendations to pressure the university– Conflicting recommendations– Multiple recommendations

• Compliance– With recommendations– Monitoring of quality review process

> Stricter enforcement by a central unit is required to avoid escapees

14

Lessons and implications

• Quality improvement– Pros

> External panels overcome groupthink> Comparable treatment, > Scope for creative improvement, > Collegial approach

– Cons> QI is ineffective in face of

– Unprofessional behaviour– Poor accountability

> Does not resolve – contradictions of applying managerial methods in the academic

environment

15

Lessons and implications

• Ad hoc management reviews– Pros

> Can be generated at a Dean’s discretion> Delve into serious problems

– Including those revealed by quality processes. > Existence of two review processes has sometimes proven

necessary to overcome the abuse of trust to which universities can be vulnerable.

– Cons> Does a need for two processes suggest inefficiency?; or

ineffectiveness?> Have we created an industry? > What is the cost/benefit of conducting (quality) reviews in this

manner?

16

Conclusions

• Quality activities are obviously an industry– Only a problem if the process drives down the

performance of Higher Education. > The impact of quality activities is difficult to identify,

although INQAAHE leaders report many positive signs (Harvey 2006).

• Debate regarding standards versus improvement may be ill-posed.

– Quality reviews are quite effective at identifying real problems, but they have limitations.

– They do not solve the long-standing problems of accountability that form the obverse to academic freedom, but we are unaware that any other process has resolved this dilemma.

17

Conclusions

• Take care that the value added by QI is not lost in the urgency to account for inputs by measuring outputs

• HEIs are complex, as is academic endeavour:– a sophisticated approach to quality seems indicated.

• Should we enhance the complementary nature of the two review processes to achieve a solution:

– Formalise the relationship as collegial, with option to police– Rationalise their demands– Continue to Invoke management review only as needed

• It is imperative to improve the student experience, and to protect education exports:

– the effort seems warranted

18

Contact

Professor Tony Luff

Associate Dean (International)

Faculty of Medicine, Nursing & Health Science

Monash University

AUSTRALIA

Tony.Luff@med.monash.edu.au

19