U N I V E R S I T Y O F C A N B E R R A

U N I V E R S I T Y O F C A N B E R R A

The Role of Assessment in Postgraduate Research

Education

Dr Margaret KileyUniversity of Canberra

U N I V E R S I T Y O F C A N B E R R A

Research Aimed to contribute to a theoretical dimension

of postgraduate ‘pedagogy’

Two main stages of the :

30+ interviews with very experienced examiners

26 interviews with inexperienced examiners (< three theses examined)

(Illustrative examples from the analysis of examiners’ reports)

U N I V E R S I T Y O F C A N B E R R A

Experienced Examiners report that they…

Expect the student to pass as they open the thesis Are very reluctant to fail a student with most

experiencing considerable distress if they do so Come to a decision about the quality of a PhD by about

the end of Chapter 2 Have a formative rather than summative view of thesis

examination Believe that there is risk attached to sending theses to

inexperienced examiners Are reluctant to take much notice of institutional criteria

when examining

U N I V E R S I T Y O F C A N B E R R A

Is there a difference?

Work of Trafford (2003) from 130 vivas it was possible to determine that:

Experienced Examiners tended to ask questions that can be defined as: “Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept”

Inexperienced examiners tended to ask more ‘technical’ questions

U N I V E R S I T Y O F C A N B E R R A

Trafford’s Categorisation

Innovation & Development

HIGH

Scholarship & Interpretation

C. Questions generally relate to issues such as research question, choose of topics, location of study

D. Defending doctorateness, contributing to knowledge,, critique of research, synthesizing concept

A. Types of questions include resolving research problems, content, structure

B. Implications, awareness of, and familiarity with wider literature

LOW

U N I V E R S I T Y O F C A N B E R R A

Purpose of Stage 2 of Research

To explore the contention put forward by many experienced examiners that it was risky to send theses to inexperienced examiners.

Why might this be? If it is not the case, why does this myth persist? What are the main similarities between

inexperienced and experienced examiners? What are the main differences between the two

groups?

U N I V E R S I T Y O F C A N B E R R A

Methodology

Interviewed 26 inexperienced examiners recognising two groups:

Those who were inexperienced examiners <3 theses examined and inexperienced supervisors <3 students supervised

Those who were inexperienced examiners <3 theses examined but more experienced supervisors <3 students supervised

U N I V E R S I T Y O F C A N B E R R A

Methodology (cont)

Interviewed staff from Canberra Adelaide James Cook QUT Queensland

U N I V E R S I T Y O F C A N B E R R A

Interviewees ( )= Previous Study

Sci Maths/Eng

Soc Sci H’ties Total

Male 5 (11) 5 (1) 3 (4) 3 (2) 16 (18)

Female 4 (3) 1 (2) 3 (5) 2 (2) 10 (12)

Total 9 (14) 6 (3) 6 (9) 5 (4) 26 (30)

U N I V E R S I T Y O F C A N B E R R A

Methodology (cont)

Each interview: Taped, Transcribed and Sent back to

interviewee for checking Analysed independently by two

researchers Searched for themes, although similarities

and differences from experienced examiners was not at the fore front of thinking

U N I V E R S I T Y O F C A N B E R R A

Inexperienced examiners

What is experience/inexperience? Is it related to:

Experience as a supervisor? Experience as an examiner? Experience as a researcher?

U N I V E R S I T Y O F C A N B E R R A

Inexperienced Examiners report…

Have a high level of confidence in ability to examine

Although inexperienced at the doctoral level many had had considerable experience at the Honours level

Adopt a similar approach to the actual process of examining as do their more experienced colleagues

Use criteria similar to experienced examiners Little in the way of discipline differences Most very concerned by typographical errors

U N I V E R S I T Y O F C A N B E R R A

Inexperienced Examiners report…

Surprisingly high percentage wanted to fail their first thesis or said it was ‘awful’

Almost immediate reference to own doctoral experience and advice from their supervisor & seem to have very high expectations of supervisors

More prone, than experienced colleagues, to follow institutional criteria

Tend to focus on the ‘steps’ or components of a PhD rather than the whole

Some felt that they were being examined too Appreciated feedback from institutions (UQ sends

other reports - definite support for that) Development as an examiner from 1st to 2nd etc

U N I V E R S I T Y O F C A N B E R R A

Discussion Suggest that one of their main difficulties is the

inability to benchmark What they report about levels of confidence is

not reflected in what they say in response to other questions

See their role as maintaining standards and performing their summative assessment role correctly

Appears that criteria for an acceptable doctorate are even more discipline-related than originally thought (rather than institutionally based)

Is there any truth in the assertion that “Inexperienced supervisors get the ‘dodgey’ students”?

U N I V E R S I T Y O F C A N B E R R A

From the reports and from comments: A ‘good’ thesis

demonstrates…

Critical analysis and argument Self-critical, rigorous and confident Contribution to knowledge Originality, creativity and degree of risk

taking Comprehensiveness & scholarly Presentation and structure Sound methodology

U N I V E R S I T Y O F C A N B E R R A

From the reports and from comments: A less than ideal

thesis demonstrates…

Too much detail with lack of analysis Lack of argument and rigour Shoddy presentation (typos etc) Lack of confidence, energy and engagement Lack of critique of own analysis/sweeping

generalisations Inadequate or poorly expressed methodology

and scope

U N I V E R S I T Y O F C A N B E R R A

Implications for Institutions

Confusion over gradings/ratings and recommendations

How to encourage experienced supervisors to send theses to inexperienced examiners

Providing feedback and support for inexperienced examiners

Greater use of Viva? Possible use of ‘internal’ examiner in the

formative sense

U N I V E R S I T Y O F C A N B E R R A

Mullins, G. & Kiley, M, (2002).‘It’s a PhD, not a Nobel Prize’: How experienced examiners assess research theses. Studies in Higher Education 27(4) pp. 370-386

Trafford, V. (2003). "Questions in doctoral vivas: View from the inside." Quality Assurance in Education 11(2): 114-122.