researchdegree examining–common principlesand...
TRANSCRIPT
Research degreeexamining – commonprinciples anddivergent practices
Stuart Powell andClaire McCauley
Overview
Over the past 18 months the authors have
co-ordinated a number of national seminars
and workshops on research degree examining
involving active researchers and research
supervisors/examiners from across the UK
and from a range of institutions and academic
disciplines. Focussed discussions at these
national seminars have indicated that
institutional practices concerning research
degree examination vary significantly across
the sector. It was also noted that some issues
relating to this variation may be largely
outside the remit of the current QAA (1999)
Code of Practice for the Assurance of Academic
Quality and Standards in Higher Education:
Postgraduate Research Programmes. It also
became clear that the issue of thresholds may
be contentious when applied to research
degree examining, and indeed, that current
notions of assessment may be inadequate
when applied to practice with regard to the
examination of theses. The notion of the vivawas particularly problematic in discussions. It
was noted that in many European countries
the viva was a matter of public defence and
part of a ritual ’’welcoming into the Guild’’ –
in the UK it tended to have a range of
functions, some overt but others remaining
hidden. One area of consensus was that there
is a need to focus on the function of research
degree examination as a form of assessment;
for example, it was suggested that other
aspects of the process such as ’’ rite of
passage’’ sometimes obscure this function.For the purposes of this paper, issues
discussed are grouped under twoheadings:(1) research degree examination as
assessment; and(2) examiners and examining panels.
In each case the original issues as set out in
the briefing paper that preceded the seminars
and workshops are noted – these issues are
followed by the key questions that were
presented to participants, and finally the main
points of the subsequent discussion are
summarised.
The authors
Stuart Powell is Director of Research Degrees, University
of Hertfordshire, Hatfield, UK.
Claire McCauley is Administrator, UK Council for
Graduate Education, Dudley, UK
Keywords
Qualifications, Research, Assessment
Abstract
Reports on discussions that took place at a series of
specialist seminars and workshops on research degree
examining organised by the UK Council for Graduate
Education during 2000/2001 at various venues in the UK.
Debates the processes and procedures of research degree
examination in the UK in terms of variations in practice
that exist along with principles that signal a common
identity. Takes account of the effects of developments in,
for example, professional doctorates and the PhD by
published work and on perceptions of the `̀ traditional’’
examination. Issues addressed include: the composition of
PhD examining panels and the roles of individual
examiners; the training and qualification of examiners;
the purpose and nature of the oral examination; the
tension between examination of the process of training
and that of the `̀ finished product’’ (which the thesis may
be seen to represent) . Argues for the need for more
transparency about examination processes, for challenge
to common assumptions and for a refocusing on research
degree examination as a process of assessment.
Electronic access
The research register for this journal is available at
http://www.emeraldinsight.com/researchregisters
The current issue and full text archive of this journal is
available at
http://www.emeraldinsight.com/0968-4883.htm
The authors wish to acknowledge the invaluablecontributions made to this paper by Howard Greenand Malcolm Shaw.
104
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . pp. 104±115
# MCB UP Limited . ISSN 0968-4883
DOI 10.1108/09684880210423591
Research degree examination asassessment
Setting out the issuesThere are many commonly expressed notionsabout research degree examination and it isfrequently seen in a different light to otheraspects of examining in higher education. Ithas developed its own custom and practiceand mystique with the viva (or oralexamination) taking on a significance,particularly for the candidate, which hasmoved it some way from being merely anassessment of the achievement of researchoutcomes, if it ever was!
We might usefully reflect on what we meanby assessment and ask whether the PhDexamination stands scrutiny as a form ofassessment. The QAA’s (2000a, p. 3)definition of assessment will suffice tochallenge us in our approach to PhDexamination:
Assessment is a generic term for a set ofprocesses that measure the outcomes ofstudents’ learning, in terms of knowledgeacquired, understanding developed, and skillsgained. It serves many purposes. Assessmentprovides the means by which students aregraded, passed or fail. It provides the basis fordecisions on whether a student is ready toproceed, to qualify for an award or todemonstrate competence to practise. It enablesstudents to obtain feedback on their learning andhelps them improve their performance. Itenables staff to evaluate the effectiveness of theirteaching.
Additionally, we may wish to question theelements of the assessment, as most, if not all,documents and artefacts submitted forassessment will contain the words ’’ submittedin partial fulfilment of the requirements . . . ’’ Ifthe document or artefact is part of thesubmission for examination, what is the otherpart? What is the relationship between the twoor possibly more parts, in terms of weighting?Does the candidate have to pass both parts inorder to be awarded the degree? Mostimportantly, where are any of these questionsanswered in institutional regulations?
The role of oral examination/vivaAn inspection of their regulations shows thatinstitutions vary in their stance with regard tothe obligatory nature of the viva, with someinsisting on it being held regardless ofperceived qualities in the thesis and othersoffering various kinds of dispensation, e.g.
’’where no useful purpose would be served’’ .
In one sense at least, attitude to the necessityfor an oral defence relates to the underlyingperception of whether the examination is ofthe thesis or the process of research training.A product might be examined in the absenceof its author, whereas to judge the outcomesof a process of training requires somequestioning of the individual who has beentrained.
There is wide variation in practice overwhether or not a supervisor should be presentat the oral examination and their role if theyare allowed/encouraged to be present. Insome institutions where supervisors may bepresent, they are only invited at the discretionof the examiners, in others only at thediscretion of the candidate, and yet in othersthey may be required to be available onrequest but normally not be expected toattend. In some institutions attendancerequirements or restrictions relate only to theprincipal, or first, supervisor. Contribution ofsupervisors during the examination variesbetween being expected to speak and beingallowed to speak, but only at the request/discretion of the examiners. Commonly, thesupervisor’ s unspoken role is that of acting as’’candidate’ s friend’’ .
Few institutions make the viva a publicevent, though some departments claimopenness within certain categories of staff.The merits and demerits of public defence arerarely discussed within institutions. Practicein other countries differs greatly from that inthe UK. It may be argued that practice in theUK needs to be debated in the context ofpotentially increasing European convergence.Clearly, differences in current UK practicerelate largely to differences in the kind andquantity of research training and researchactivity seen as necessary for the award ofPhD, and subsequently to differences in thebases for assessment. This area is explored inmore detail in Powell (1999).
There are now possibilities for making theviva a more flexible event by means of videoconferencing. There may, however, be issuesarising from the use of technology in this waywithin the process of assessment, for examplewhere confidentiality may be threatened.
Generally there is lack of clarity about thepurpose of the viva. For example, whether theviva is intended to be an examination in thebroadest sense of the term or merely a
105
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
verification of authenticity. Indeed, in reality,many vivas seem to become effectivelyopportunities to fine-tune the written work ofthe candidate in order that it reaches anotional standard acceptable for scrutiny bythe wider intellectual community.
Assessment criteriaThere also appears to be little consistency interms of the level or kind of guidance that isgiven to examiners on criteria for assessingsubmissions. Many institutions cite the needfor criteria such as originality andpublishability, but detailed checklists ofcriteria for assessing theses are not inevidence. This area is explored in some detailin Shaw and Green (1996). This kind ofvariation reflects the underlying lack of clarityabout threshold performances that mightdefine differences between degrees at Masters(e.g. MRes), MPhil and PhD levels. Also,there is lack of clarity concerning anyweighting that might be given to differentcomponents within a thesis and to differentaspects of the examination. For example,should critical self-appraisal be allowed tocompensate for poor research design?
Assessment of process or product?Differing examiners’ views indicate a tensionin the PhD examination over the extent towhich it is the process of research training orthe finished product that should be examined.Different views in this respect will lead todifferences in notions of the way in which theviva ought to proceed, and indeed, thebreadth of the remit given to examiners. Forexample, should examiners be:
able to ’’ explore records of researchsupervision’’ (CVCP, 1993) or otherdocuments such as progression ortransfer reports;asked to comment on quality ofsupervision, training opportunities andfacilities provided by departments – seenin the BPS document, UCoSDA/BPS(1995), as ’’ an important subsidiary roleof the examination board’’ ;given access to a candidate’ s criticalself-appraisal of their own learning as wellas the research findings andinterpretations?
In order to explore these sorts of issues morecarefully with practitioners, the key questions
indicated below were compiled and used tofocus discussion and group activity duringseminars/workshops:
What is being assessed? When assessing atresearch degree level – is it the candidate(i.e. the sum total of the learningoutcomes for the individual (includingperhaps any such outcomes from generictraining)) or the substantive nature of anycontribution to knowledge (i.e. asexpressed in the thesis) that is beingassessed?Who is being assessed? Should a judgementbe made as to the quality of supervisionthat has led to the submission and if socould ’’negligent supervision’’ (CVCP,1993) influence decisions when awardingthe degree? Should judgements be madeby the examiners about overall provisionwithin the institution (e.g. of resourcesand training opportunities for students)?Are there identifiable (and usable)assessment criteria for PhD? What mightsuch criteria be? Should criteria for theaward of PhD be made explicit? Shoulddifferent criteria for different parts of thethesis and/or the examination beweighted? Should such criteria be madeavailable to all students? What mightcount as ’’publishability’’ (but see below)?How should the assessment beconducted? Is a viva necessary foreffective assessment of a PhD? Is morethan one examiner necessary? What is therole of supervisors in the examinationprocess?
Summary of discussion on researchdegree examination as assessment
The discussions were allowed to befree-ranging and the foci and responses werenot always neatly confined to the keyquestions posed. The summary is presentedhere under theme headings that include thekey questions, but that range beyond them.
What or who is being assessed?It was generally agreed that it is more than thewritten word that is being examined. It shouldbe the case that the key skills, reflected by thework, are being judged and therefore thestudent himself/herself is the real point ofscrutiny. Examiners should be asking
106
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
themselves – has this student integrated keyskills and understanding? The viva issupposed to be the most satisfactory place toinvestigate this, yet this point of focus is oftenlost as the marking of the thesis and the viva,as already noted, become contexts forchecking text and improving literary qualities.
Special mention was made that thesesshould in one sense be valued ’’ in their ownpart of the academy’’ , but this is problematicwhere the work is genuinely interdisciplinary– an area which is growing. Indeed, one keyarea of potential difficulty was identified asthe nomination of an examination team thatcould properly and fairly examine a thesis thatcrosses disciplinary boundaries and that thusemploys constructs and methodologies thatmay not be universally recognised.
Assessment criteriaIt was generally agreed that criteria might bedifficult to standardise across HEIs and thatin any case such standardisation would notnecessarily be desirable. But it was alsoargued strongly that there was a need formore transparency. At present the situationacross the sector is not only varied but it isalso opaque. This makes quality andstandards auditing a difficult exercise anddisadvantages the student who may enter anassessment scenario unaware of the hiddenagenda in operation.
The notions of ’’ publishability’’ , ’’ originaland significant contribution to knowledge’’ ,that are commonplace criteria used byinstitutions, often depend on a number offactors and are linked to discipline. It isquite common for there to be no agreementmade between examiners with regard tocriteria – by default, decisions are thereforebased on implicit criteria, which mayremain idiosyncratic to individualexaminers.
Some seminar attendees made theargument for ’’ threshold’’ levels for criteriarather than standardisation and it wasgenerally agreed that there should not besubclasses for PhDs. It was generally agreedthat the PhD is a ’’ threshold’’ qualification.
Status and nature of the viva and thethesisThe thesis was seen to be prioritised as themedium for assessment. It was felt that theviva, as currently operated in the UK, could
not function adequately as a means of
reflecting the student’ s expertise accuratelyenough for judgement to be made andtherefore the student’ s individual learning wasseen to be secondary. It is then the thesis thatis assessed, though discussants did claim thatexaminers were making judgements about thecandidate on the basis of what was presentedin the thesis.
There was general agreement that the vivashould be an area for learning for allconcerned. But there was recognition that itgenerally functioned as a means of’’ polishing’’ the thesis and tuning the clarityof the contribution to knowledge and overallliterary quality of the written work.Throughout the various discussion groupsthere was an emerging consensus that thelearning opportunities offered for allparticipants by research degree examinationwere not regularly pursued to fulladvantage. Certainly there was littleevidence of the process of examinationbeing reflected on in any formal sense, norwas evaluative feedback on the processregularly obtained.
It was felt by many that the nature of theviva depends on the nature of the thesis.Some colleagues felt that if the document wasstrong then the viva functioned as asummative process whereby the examinerwould sum up the strengths of the thesis anddiscuss post-doctoral progression. If, on theother hand, the document was felt to befailing, then the viva functioned as aformative process whereby the examinerwould inform the students of the weaknessesthat would lead to resubmission and possiblere-examination. Time would then be spent ongiving formative guidance on what is requiredto amend current failings. We should notehere that the converse view was alsoexpressed, namely that the function issummative when the thesis fails (the examineris making a judgement) and formative when itis successful – the examiner is consideringprogression beyond the PhD and into theacademic community.
The view was expressed that the vivashould function also as a way of assessingwhether students have integrated key researchskills and understanding of methodology as aresult of research training. This was seen as anew aspect of the modern PhD along withother value-added aspects such as the
107
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
importance of a PhD resulting in a newartefact, product or innovation.
Monitoring and evaluation of processThere was a suggestion from some discussiongroups that the final examination of a PhDcould be informed by various earlier stages inthe process, such as a first year or transfer/upgrading report. Some colleagues felt thatexaminers should have access to these earlierreports. Again, opinions were divided alongsimilar fault lines to those that existed in the’’ teacherly-confrontational’’ debate (see ’’Roleof the examiner’’ section later in this paper).There was general agreement that asatisfactory assessment process was likelywhen each assessment point during astudent’ s progress acted clearly as a device tokeep the student on track. In particular, theupgrading process should not been seen as anautomatic progression to PhD status. Inshort, it was generally agreed that thereshould be delineated points in the process of aresearch degree programme which indicate ifthe student is likely to complete or fail, andthat a more rigorous evaluation of the processwould affect the type of viva and ratio ofpass/fails.
As already noted, there was some strongsupport for the notion that the sameexamining panel should examine throughoutthe process of a programme of study and atthe final thesis stage. In this sense theexamination of the final thesis was seen as asynoptic assessment of a complete process.This was seen as analogous with assessmentsof taught postgraduate courses whereexaminers might have access to recordsindicating the whole of the progress of anindividual student through a programme ofstudy.
There was some agreement, though notunanimous, for the notion that a reportshould be included at the end of the process(i.e. at the final examination stage) whichdescribes ’’ special measures’’ encounteredand perhaps overcome successfully during theresearch process.
The main conclusions and areas ofconsensus emerging from the discussiongroups are presented concisely in Table I.The rows indicate function and role whilethe columns show aspects of the process.What is clear from this table is that functionand role shift as one considers the different
aspects of the whole process. In particular,there are tensions between what is valuedwithin the written (or alternativelypresented) ’’ product’’ itself and the processof the student’ s learning.
Examiners and examining panels
Setting out the issuesDifferent notions of what is being examinedand how examinations should be carried outlead to different notions of the composition ofpanels. Some institutions have anindependent chair of PhD examination panels– many do not. Sometimes a senior internalmember of staff, not necessarily possessingsubject expertise, acts with a brief to ensurethe good conduct of the examination processand the dissemination of good practice.Institutions differ in the extent to which theyallow or encourage assessors at anyprogression stage to become examiners at thefinal assessment stage. Some institutionsroutinely appoint two examiners – oneinternal and one external – other institutions,and many outside the UK, routinely appointthree examiners. Institutions vary in whetheror not they require that an examiner shouldhave a PhD himself/herself.
The question has been raised as to whetheror not the external examiner should’’comment on the remainder of theexamination board’’ (UCoSDA/BPS, 1995).Additionally, the extent and nature of anystudent input into the appointment of theexamining team differs.
The differing status of examiners within thepanel is also far from well defined. In theCVCP (1993) document Handbook forExternal Examiners in Higher Education, it isstated that:
. . . an external examiner of a PhD is theexaminer. He or she decides whether the thesispasses or fails (VCP, 1993, p. 73, theiremphasis).
This is a view held in some universities butnot in others. In some the external examinerovertly holds the key vote, whereas in othersall examiners are equal – though again somemay be more equal than others! Clearly, therole of the internal examiner is defined inrelation to that given to the external.
The procedures for nominating examiners,and the mechanisms that enable institutions
108
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
to monitor them, are generally seen to beimportant if independent and fair assessmentis to be achieved, see, for example, QAA(1999) precept 10a which stresses – ’’ themechanisms used for communicatingprocedures relating to the nomination ofexaminers’’ . Yet there seems to be littleconsistency in practice. Certainly mostinstitutions take particular care over thisphase of the administration of research degreeexamination.
Many institutions do not train examiners,yet most workshop attendees thought of theprocess as complex and important enough forsome minimal training to be desirable. Withregard to qualifications for examinership,UCoSDA/BPS (1995, p. 8) suggests that ’’anexternal examiner should possess a PhD orother evidence of a similar level of scholarshipand should normally have been principalsupervisor of at least one successful PhDstudent’’ . The wording of thisrecommendation lacks precision and allowssome considerable latitude and, whilst thismay not appear to be serious, it does leave theissue of just what is expected of an examiner
largely unresolved. The relationship betweenexamining, scholarship and supervising mightusefully be explored further. Manyinstitutions require an examining team tohave experience of examining but are lessconcerned to consider supervisory experience,presumably on the grounds that these twoactivities are discrete.
The key questions set out below weresuggested by the issues identified above andwere used to focus the discussions in thenational seminars and workshops:
What are the qualifications andexperience required of an individualexaminer?Should examiners be trained?How should external scrutiny functionwithin the assessment?How should an examining panel benominated?Should the process of examination bemonitored/evaluated?Should any one examiner have the castingvote?Should the student have any role in theprocess of determining panels?
Table I Key factors in research degree assessment in the UK
The finished product The research process The viva
Function Verification of learned research skills
Indicate contribution to research
community
Indicate contribution to knowledge
Facilitate research
apprenticeship
Polishing/fine tuning the literary work
Examination of the candidate’s
knowledge and skills
Verification of authenticity
Rite of passage
Status Prioritised as medium for
assessment
Rarely prioritised during
assessment
Varies from absolute to probable to
possible
Mode Written thesis
Alternative forms of doctoral
submission
Culmination of discrete parts of a
professional doctoral programme
Collection of published work with
exegisis
Supervision
Taught elements (including
training in methodology and in
generic skills)
Retrospectively acknowledged
(PhD by published work)
Real time
Video conferencing
Appraisal of work in situ (e.g. art
submission)
Assessmentcriteria
Publishability (does not always apply,
e.g. in practice-based doctorates)
Originality
Innovation
Significance of contribution to
knowledge
Level of scholarship
May form part of assessment in
professional and practice-based
doctorates
Records of progress may be
referred to
Determined by evaluation of finished
product
Ability to communicate ideas and
defend positions
Evidence of authorship
Progression andimprovement
Increased transparency/clarity (from
earlier submissions where made)
Threshold markers for different awards
Weighting for different components
Evaluation of final thesis may
be interpreted as a synoptic
assessment of the complete
process
May lead to amendments and further
supervisor input
109
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
Summary of discussion on examiners andexamining panelsOnce again the themes emerging in discussiondid not match exactly the set of questions in aneat one-to-one correspondence, but theheadings below do give a clear indication ofthe preoccupations of the participants.
Experience/qualifications of the examiner. Itwas generally agreed during the variousworkshops that examiners only gainexperience of examining by examining, butthat training would help to bridge the gapbetween novice and expert. It was also felt bysome participants that examiners should havea record of successful supervision. Thegrounds cited were that in order to passjudgement on the outcomes of researchdegree study it is necessary to understand theprocess of that study, and in turn it istherefore necessary to understand therelationship between supervisor, student andproject. Examiners should therefore betrained to understand the formative nature ofsupervision and the relationship between thatsupervision, individual student achievementand tangible research outcomes.
But it was also noted that there is nonationally recognised forum for exploring theissues of examiner training across the UK –although several universities now offertraining to their own staff on their role in theexamination of the PhD. It was also notedthat there are severe limits to what can beexpected of examiners given the level ofremuneration and the amount of workinvolved.
While it was agreed that the examinationshould act as a learning process for examiners,as noted elsewhere in this paper, it was alsorecognised that a context needed to bedeveloped in which examiners could reflectand develop such learning. Suchopportunities did not exist at present formany colleagues
Role of the examiner. There was a generalconsensus in the workshops that whileexamining should be seen as a distinct ’’ role’’with associated specific skills, all too oftenexaminers are chosen for their individualsubject specialism and their expertise in thisfield. If perceived as a ’’ role’’ , the notion ofexaminer training could be moremeaningfully explored and hence betterunderstood. The issue of ’’ super examiners’’who can step outside their particular ideology/specialism to judge scholarly work more
widely was raised. There was some debateabout whether or not it was practicable tomake use of an independent chairperson inthis respect and it was suggested by some thatthe burden on particular individuals mightbecome onerous if this latter scenario were todevelop.
It was generally agreed that examiningshould not be driven by an examiner’ s ownpersonal research agenda and that examinersmust be aware that other ideologies andmethodologies exist outside their ownpreferences. They should not be influenced tofail a thesis purely on the basis of their ownpersonal convictions, feelings or passion.
Some colleagues conceptualised the role ofexaminer as either ’’ teacherly’’ or’’confrontational’’ . In the first instance thenotion of ’’ role’’ is highlighted and examinersof this type act as positive gatekeepersallowing access into the research community.In the second instance the notion of theindividual is highlighted and the role assumesmore of a function of negative gatekeeping:keeping people out of the researchcommunity; disallowing those students whodo not share the same ideology/methodologyand passions. Along with this negativegatekeeping is suggested the notion that someexaminers may even use the examinationprocess, consciously or otherwise, as a chancefor ’’ settling old scores’’ .
Notions of using the individual who judgedthe progress of the work at a transfer (orprogression or upgrading) stage as anexaminer at the final examination stage variedalong with the view taken of the gatekeepingrole. Those who saw the role as a teacherlyone tended to see the role of formativeassessor as positive and as compatible withthat of final assessor. Conversely, those with amore confrontational view tended to see aneed for the examination at the final stage tobe ’’blind’’ to processes that had gone before.Judgements here were to be made on the finalproduct only. Roles tended to correspondwith ’’assessment of process’’ (teacherly) or’’examination on the day as single event’’(confrontational).
Most participants agreed that examinersneed to be informed about the work they areexamining and need to understand theapproach used, even if they do not practise itthemselves. It was clear that problems werelikely to arise where the examiner had notbeen made fully aware of the nature of the
110
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
substantive content of the thesis or theprocedures and methods employed in thestudy at the time of appointment. Theusefulness of briefing the potential examinersby giving them access to an abstract (orsimilar) of the thesis was suggested.
Views on the remit that should be given toexaminers tended to be constrained byrecognition that remuneration levels arealready either poor or derisory. Therefore toexpect the examiner to comment on morethan the work in front of him/her (e.g. tocomment on supervisory arrangements atthe university or on departmentalresourcing) was seen as unrealistic by manyparticipants. This whole issue of the statusof examining and its relationship to theworkings of the academic community as aself-perpetuating guild was noted, but littleconsensus was achieved.
Examination panels. The role of anindependent chair was felt to be useful insome situations but not others. Somecolleagues felt that in 95 per cent of casesthere was no need for a chair, as the result ofthe examination process was clear-cut andtherefore any panel members in addition tothe actual examiners would only add a burdenof unnecessary bureaucracy.
However, others felt that an independentchair could enable the sound running of theexamination process. He/she could act as athird party or witness in any appeal process –particularly if they have been involved in theexamination process since its inception. Thosein favour of an independent chairperson tendedto see the role as that of a ’’ super’’ examinerwho would judge the work on its scholarly valuerather than be biased towards any one kind ofideology or methodology.
A majority of discussants felt that theexternal examiner should not be all-powerfulnor have the casting vote because this raisedquestions regarding the purpose and role ofthe internal examiner.
Student role in the nomination of theexamination panel. There were differingopinions concerning the strength of the studentvoice: the student should have a veto; thestudent should have one veto; the studentshould ’’have a say’’ in the examination panel.But there was a fairly consistent view that thestudent should have a voice of some kind, i.e.not be totally shut out of the process. Onereason given was that the student might need atthe very least to declare that he/she has had
contact with an individual examiner in terms ofcomments on work. There was some agreementthat while the student should not be in aposition to choose their own examiner theirviews on the kind of expertise that would beneeded to ensure fair and rigorous examinationof their work should be taken into account.
A summary of some of the main issuesraised in relation to examiners and examiningpanels is set out in Table II. In general, whatis clear from the table is that there isconsiderable variation in terms of function,status, level of experience and qualificationthat permeates the constitution of and theroles of the various participants within theexamination scenario.
Recent significant developments
The QAA (2001) has published its finalversion of the framework for higher educationqualifications. Awards are contained withinthree undergraduate levels (levels C, I, H)and two postgraduate levels – Masters (levelM) and Doctorate (level D). A set ofqualification descriptors which indicate thegeneral learning outcomes that should beachieved within the range of qualification ateach of the five levels are also indicated withinan Annex to their framework paper.
By carefully cutting and pasting thedescriptors, it is possible to construct a matrixsuch as is indicated, for levels M and D, inFigure 1 (reproduced here with the kindpermission of Professor Malcolm Shaw whohas been responsible for its development anduse at Leeds Metropolitan University). Thisprovides a more convenient means to scan thedescriptors both within and across levels. Thematrix also has allowed some grouping of thedescriptors into categories (knowledge andunderstanding, problem solving,communication, etc.) which helps withattempts to trace notions of progressionthrough the levels. It should be noted that thismatrix is not in any way a part of QAApromotion of the framework but it does use,for obvious reasons, the precise wording oftheir descriptors.
These descriptors at level M and level Dclearly impinge considerably on our view ofwhat should be achieved within awards such asMRes (level M), MPhil (also level M), PhD(level D) and other doctorates (also level D). As
111
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
such, they impact on the function of ourassessment of students and the criteria that weuse to assess student achievement in theseawards. They suggest a more systematic andconsistent approach to assessment andstandards across all examiners and across allawards in the HE sector. The descriptors doprovide an explicit device with the potential tobegin to identify and explore answers to some ofthe key questions already posed. Inevitablyhowever, these descriptors raise a number ofrelated fundamental questions of their own,such as:
Do the descriptors differentiateadequately between Masters andDoctorate levels?Do the outcomes present new factors thatare not currently examined?What implications do the outcomespresent for the role of the thesis and theviva?Do the outcomes accommodate thedifferent types of masters and doctoralawards?
These are questions that require a more
extensive and rigorous examination than has
been possible in the seminars to date. Certainly
it becomes clear, from even a cursory glance at
the matrix, that QAA’s descriptors provide
more scope for a systematic approach to the
assessment of the PhD whilst differentiating it
from other awards than has typically existed to
date. At the same time, most of these
descriptors at doctoral level might well be seen
to be implicit in much PhD assessment that is
currently carried out, but whether they are
valid, adequate and sufficient is another matter.So we are not suggesting here that the
descriptors should be directly translated intoassessment criteria, but that they might forma basis for discussion leading to a morestructured approach to assessment and hencethe possibility of more transparency andequity across the sector.
Conclusions
Codes of practiceThere was some discussion during seminars
of the need for a ’’Code of practice for
assessing research awards’’ . It is worth noting
here that the QAA (2000b) in its code of
practice on external examining gives little
specific attention to the examination of the
highest award that most universities
commonly make. Similar lack of clarity and of
detail is found within precepts 10a and 10b of
the QAA (1999) code of practice for
postgraduate research awards.
Table II Constitution of and roles within examination panels
External examiner Internal examiner Chair Supervisor Student
Composition Invariably necessary Often but not always
necessary
External examiner
External to panel
Chosen on the day
Not necessary in examination
process
May attend viva at discretion of
examiners or student
Presence necessary if
viva required
Status ofindividuals
Expert in field of
research?
Experience of
examining?
Has casting vote?
Aware of university
regulations?
Candidate’s friend?
Chair independent of
panel ± aware of
university
regulations?
External?
Candidate’s friend
Advisor on process
Voice in composition
of panels?
No voice ± merely
subject of enquiry
Function ofindividuals
To gatekeep?
To be teacherly?
To endow credibility
To ensure
comparability across
institutions
To represent
institution?
To ensure fairness
for candidate?
To balance against
external?
To mediate?
To control process of
oral?
To enable fair
process?
To act as note-taker for candidate
in respect of any amendments
To interpret for the candidate for
the examiners?
To authenticate
research?
To prove research
skill as an individual?
To give evidence of
learning
Experience andqualifications
Successful
supervision?
Experience of
examining?
Holder of a PhD?
Domain expert
Awareness of
regulations
Awareness of
regulations
Research skills?
First/pg degree?
112
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
Figure 1 Matrix of QAA descriptors for qualifications at levels M and D
113
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
Figure 1
114
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115
PrinciplesThe following principles for assessment andexamination emerged strongly from thediscussions:
Institutions need to ensure balancebetween research degree examination as anassessment of the process of training andthe final product that may well be thethesis. Examiners need to recognise that itis the candidate who is being examined,albeit on the evidence of learning andachievement presented in the thesis and theviva.The composition of the examinationpanel, its remit and roles of individualsshould be clear to all participants.The relationship between the variouselements of the examination must beclearly specified along with a statement ofthe role of each element in the overallassessment.In any oral examination situation thegeneral procedure and the criteria formaking judgements should be laid downin advance by the institution.There should be some way of monitoringthe process of an oral examination toenable fairness to be judged and goodpractice to be identified anddisseminated.There are skills in examining which can betaught – assumptions to the contraryshould not be made. Training in researchdegree examining should become ascommonplace as training in the processand methods for assessment itself.There is need for transparency inconceptions of threshold performanceand in the weighting of assessmentcriteria.New forms of research degree require arethink of assessment processes.There should be comparability in thelevel, if not the kind, of assessmentapplied to traditional and to newer formsof research degree.
Finally, we must bite the bullet, accept that weare assessing a total programme of research anddevelop appropriate criteria. Why should therigour of the assessment of doctoral work beregarded any differently to that pursued in other
aspects and levels of student achievementwithin higher education?
References
CVCP (1993), Handbook for External Examiners in Higher
Education, UK Universities Staff Development Unit,
Sheffield.
Powell, S.D. (1999), `̀ A Swedish disputation: reflectionson PhD examination’’, Newsletter, UK Council for
Graduate Education.
QAA (1999), Code of Practice for the Assurance ofAcademic Quality and Standards in Higher
Education: Postgraduate Research Programmes,
QAA, Gloucester.
QAA (2000a), Code of Practice for the Assurance ofAcademic Quality and Standards in Higher
Education: Assessment of Students, QAA,
Gloucester.
QAA (2000b), Code of Practice for the Assurance ofAcademic Quality and Standards in Higher
Education: External Examining, QAA,
Gloucester.
QAA (2001), The Framework for Higher EducationQualifications in England, Wales and Northern
Ireland, QAA, Gloucester.
Shaw, M. and Green, H. (1996), `̀ Standards in researchawards: length, weight or quality? Developing an
approach for resolving the dilemma’’, Innovation &
Learning in Education: The International Journal for
the Reflective Practitioner, Vol. 2 No. 3, pp. 4-10.
UCoSDA/BPS (1995), Guidelines for the Assessment of thePhD in Psychology and related Disciplines,
Universities’ and Colleges’ Staff Development
Agency, Sheffield.
Further reading
Hartley, J. and Jory, S. (in press), `̀ Lifting the veil on the
viva: the experiences of psychology PhD candidates
in the UK’’, paper submitted for publication (copies
from the authors ± Keele University).
Jackson, C. and Tinkler, P. (2000), `̀ Examining thedoctorate: institutional policy and the PhD
examination process in Britain’’, Studies in Higher
Education, June (note: available from CarolynJackson at Department of Education, University of
York).
Johnston, S. (1997), `̀ Examining the examiners: an
analysis of examiners’ reports on doctoral theses’’,Studies in Higher Education, Vol. 22 No. 3,
pp. 333-48.
Underwood, S. (1999), What is a PhD?, Higher Education
Development Centre, Lancaster University,Lancaster.
115
Research degree examining ± common principles and divergent practices
Stuart Powell and Claire McCauley
Quality Assurance in Education
Volume 12 . Number 2 . 2002 . 104±115