portfolios as a method of student assessmentctld.kau.edu.sa/files/322/files/154767_theme 4 guide 24...

-1 -

Portfolios as a method of student assessment

Portfolios as a method of student assessmentAMEE Medical Education Guide No 24

This AMEE Education Guide was first published in Medical Teacher:

Friedman Ben David M, Davis M H, Harden R M, Howie P W, Ker J and Pippard M J (2001) AMEE MedicalEducation Guide No 24: Portfolios as a method of student assessment. Medical Teacher 23(6).

The Authors

Miriam Friedman Ben David is Professor of Medical Education, formerly Co-Director of the Clinical SkillsCertification Programme at the Educational Commission for Foreign Medical Graduates (ECFMG) andAssociate Dean at the Medical College of Pennsylvania, Philadelphia. Professor Friedman Ben David iscurrently at the Centre for Medical Education, Dundee, Scotland. Her main areas of interest are physicianand student assessment, programme evaluation and development of leadership in the medical profession.

Margery Davis is Senior Lecturer in Medical Education at the University of Dundee’s Centre for MedicalEducation. She has provided educational support for the Dundee Medical School since 1992.

Ronald M Harden is Director of the Centre for Medical Education, University of Dundee and Director of theScottish Council for Postgraduate Medical and Dental Education’s Education Development Unit.

Peter Howie is Professor of Obstetrics and Gynaecology at the University of Dundee and Chairman of theScottish Council for Postgraduate Medical and Dental Education. He has had a long-standing interest inmedical education and the development of portfolio assessment in particular.

Jean Ker is a Clinical Lecturer in Medical Education in the School of Medicine, University of Dundee. Sheis a practising General Practitioner and provides support to the undergraduate curriculum in clinical skillsand assessment.

Martin J Pippard has been Professor of Haematology in Ninewells Hospital and Medical School, Universityof Dundee, since 1989. He trained in Oxford UK and Seattle USA and was formerly Consultant Haematologistat the MRC Clinical Research Centre and Northwick Park Hospital, Harrow UK. As convenor for Phase 2(years 4 and 5) of the Dundee undergraduate medical curriculum, he has coordinated the implementation ofthis phase of a new curriculum over the last 3 years.

Guide Series Editor: Pat Lilley

Production and Desktop Publishing: Lynn Bell

© AMEE 2001

ISBN: 1-903934-04-4

Copies of this guide are available from:

AMEE, Centre for Medical Education, Tay Park House, 484 Perth Road, Dundee DD2 1LR, UKTel: +44 (0)1382 631953; Fax: +44 (0)1382 645748; Email: [email protected]

-2 -


ContentsPage

Purpose of this guide .. .. .. .. .. .. 3

Summary .. .. .. .. .. .. .. .. 4

A: Portfolios as a method of student assessment .. .. .. .. 4

1 Background to the use of portfolios for assessment purposes .. .. 4

2 What are portfolios? .. .. .. .. .. .. 5

3 Types of portfolios .. .. .. .. .. .. 5

4 What portfolios might contain .. .. .. .. .. 6

5 Why use portfolios? .. .. .. .. .. .. 6

6 The assessment of professionalism .. .. .. .. .. 7

B: The Dundee case study .. .. .. .. .. .. 9

C: Psychometric issues relating to use of portfolios for assessment .. .. 13

1 Summary of the portfolio’s main assessment features .. .. .. 13

2 Raters’ consistency .. .. .. .. .. .. 14

3 Decision consistency .. .. .. .. .. .. 15

4 Sampling .. .. .. .. .. .. .. 15

5 The consensus approach.. .. .. .. .. .. 15

6 Forms of validity .. .. .. .. .. .. 16

7 External vs. internal examiners .. .. .. .. .. 17

D: Implementation of portfolio assessment .. .. .. .. 17

1 Defining the purpose .. .. .. .. .. .. 17

2 Determining competences to be assessed .. .. .. .. 18

3 Selection of portfolio material .. .. .. .. .. 18

4 Developing a marking system .. .. .. .. .. 18

5 Selection and training of examiners .. .. .. .. .. 19

6 Planning the examination process .. .. .. .. .. 20

7 Student orientation .. .. .. .. .. .. 20

8 Developing guidelines for decisions .. .. .. .. .. 20

9 Establishing reliability and validity evidence .. .. .. .. 20

10 Designing evaluation procedures .. .. .. .. .. 21

E: Conclusions .. .. .. .. .. .. .. 21

References .. .. .. .. .. .. .. 22

-3 -


This guide informs medical teachers about the use of portfolios for the purpose of student assessment.It looks at:

• What are portfolios?

• Why use portfolios and the contributions that portfolios can make to student assessment andthe psychometric issues relating to use of portfolios in assessment.

• Guidelines for the implementation of portfolio assessment.

The educational/theoretical considerations presented in this guide along with principles of practicalapplications should enable the reader to design and implement a portfolio assessment.

Purpose of this guide

-4 -


SummaryThis guide is intended to inform medical teachersabout the use of portfolios for student assessment. Itprovides a background to the topic, reviews the rangeof assessment purposes for which portfolios havebeen used, identifies possible portfolio contents andoutlines the advantages of portfolio assessment withparticular focus on assessing professionalism. Theexperience of one medical school, the University ofDundee, is presented as a case study. The currentstate of understanding of the technical, psychometricissues relating to portfolio assessment is clarified.The final part of the paper provides a practical guide

for those wishing to design and implement portfolioassessment in their own institutions. Five steps inthe portfolio assessment process are identif ied:documentation, reflection, evaluation, defence anddecision. It is concluded that portfolio assessment isan important addition to the assessor’s toolkit.Reasons for using portfolios for assessment purposesinclude the impact that they have in driving studentlearning and their ability to measure outcomes suchas professionalism that are difficult to assess usingtraditional methods.

A: Portfolios as a method of student assessment1 Background to the use of portfolios

for assessment purposesRecent changes in medical education and trainingemphasise the development of teaching and learningprogrammes which meet the needs of the medicalprofession and society as a whole. Professionalorganisations and the public at large demanddemonstration of professional attributes which ensuredoctors’ fitness for practice while adhering to highstandards of care (GMC 1997).

The medical schools and postgraduate trainingprogrammes have responded by introducing newteaching and learning strategies which will enhancedoctors’ accountability. One important innovation incurriculum reform is the development of anoutcome-based educational framework (Harden etal 1999). Learning outcomes of undergraduate andpostgraduate training are defined broadly to allowstudents to recognise their progress along the scopeand depth of their professional competences. Theuse of such frameworks facilitates the developmentof traditional learning outcomes, such as clinicalskills, as well as learning outcomes defining thedoctor as a professional which, in the past, have beenignored. The growing interest and concern withdoctors’ professional attributes is shared worldwideas indicated by the vast number of medicaleducation papers published on the topic ofprofessionalism during the year 2000 alone(Ginsburg 2000).

Furthermore, the concept of professionalism hasfiltered into teaching and learning strategies wherestudents are encouraged to take responsibility fortheir own learning, and personalise their learning

experiences. Student-centred programmesincreasingly incorporate the learners’ needs, as wellas listen to the learners’ voice. Consequently,medical schools’ curricula are seeking to broadenstudent experiences inside and outside the hospitalsto allow better appreciation of the multi-contextsystem within which doctors are now practising. Thenew direction of medical schools’ curricula resultsin somewhat less structure and more authenticexperiences for the trainee which, in turn, increasesself-directed learning.

Concern with the lack of continuity betweenundergraduate and postgraduate education is anotherimportant factor. Transition from the undergraduateto the postgraduate phase of medical educationshould be consistent with progression from technicaldiscrete abilities to full integration of professionalcompetencies, which is the ultimate outcome ofmedical training.

Concurrent to these educational reforms, newassessment strategies are being developed to meetthe needs of recent innovations in the healthprofessions. The search for new assessment tools isa reaction against existing methods of assessment,which often have had adverse effects on the learner,the teacher and the curriculum as a whole. It isrecognised that assessment tools should enhance andsupport learning as well as measure performance.Much current interest is in authentic, performance-based assessment (Koretz et al, 1998) which alsoencourages learners to take responsibility for theirown learning and guides the learners to accumulateevidence of learning, while incorporating a criterion-referenced interpretation of their performance.

-5 -


The portfolio is an attempt to counteract thelimitations of a reductionist approach to assessment.It facilitates assessment of integrated and complexabilities and takes account of the level and contextof learning. It provides an assessment solution for acurriculum that designs learning along broadeducational and professional outcomes. Itpersonalises the assessment process whileincorporating important educational values. Itsupports the important principle of “LearningThrough Assessment” (Friedman 2000).

This guide describes the use of portfolios as a toolfor assessment and explores the contributions theycan make. It examines how they can be implementedin practice and looks at their psychometric properties.A previous AMEE guide, Portfolio-based Learningand Assessment (Snadden et al 1999), introducedthe concept of portfolios in undergraduate,postgraduate and continuing medical education andfocused on their content and construction.

2 What are portfolios?

Definitions

The idea of a portfolio is drawn from the study ofart, “where the term signifies a purposeful collectionof work”. Stecher 1998

“A portfolio is a collection of papers and other formsof evidence that learning has taken place”.

Davis et al 2001

“It is a collection of student work that exhibits thestudent’s efforts, progress and achievements in oneor more areas”… “This collection represents apersonal investment on the part of the student – aninvestment that is evident through the student’sparticipation in the selection of the contents, thecriteria for selection, the criteria for judging the meritof the collection and the student’s self reflection”.

Gisselle O, Martin-Kniep 2000

These definitions imply that the purpose of studentassessment will guide the content of the portfolio,i.e. best work, typical work, most diverse work, workon a theme, or work on one project as it evolvesover time (Stecher 1998). Definition of purpose, froman assessment perspective, will determine theportfolio content, the process of creating it and theinterpretation of the evidence. Therefore, portfolioswill vary from context to context depending on theirspecialised purpose. Most portfolios will share acommon characteristic of being cumulative, i.e.contain work completed over a period of timerelevant to students’ ongoing work or studies.

3 Types of portfoliosPortfolios are used in all stages of education:elementary, higher education and in professional andvocational programmes.

Portfolios are used in elementary school classroomsas part of ongoing assignments (Herman & Winter1994). Students prepare draft written assignmentsover time and, at a pre-determined date, choose thebest work for a “show case”. Such portfolios couldbe scored at a school level or at state level (at a statescoring site) - for high stakes examinations or forquality control procedures.

In the USA, the State of Vermont introduced in 1992a large-scale portfolio assessment for grades 4 to 8in two subjects, mathematics and writing. Studentsselect from their on-going assignments inmathematics, five to seven best pieces of work whichare scored on seven dimensions of performance,three pertaining to communication and four toproblem solving. The students submit a portfolio withone best piece of work and other specified pieces.The best piece was scored separately and theremaining pieces were scored as a set. The writingpieces were scored on five dimensions: purpose,organisation, details, voice/tone, usage/mechanism/grammar. Portfolio materials were scored by teachersto provide individual scores and by the State toprovide school profiles in those two subjects (Koretz1998).

The State of Kentucky developed a similar large-scale portfolio assessment in writing for 4th gradestudents. Over a period of one year, the studentscompiled a portfolio comprising six pieces: apersonal narrative, a poem, a play, a piece of fiction,one information or persuasive piece, one piece fromany subject area other than English and languagearts, a best piece and a letter to the reviewer aboutthe best piece and student’s growth as a writer (Koretz1998).

In the professions, portfolios are used, for example,in recording nurses’ career and professionaldevelopment through formal and experientiallearning for periodic registration with the UK CentralCouncil (Jasper 1995). In medicine, portfolios areused to study critical thinking and self-directedlearning in the daily practice of general practitioners(Al Shehri 1995).

The Royal College of Physicians and Surgeons ofCanada introduced a learning portfolio, TheMaintenance of Competence Program (MOCOMP),using computer software for physicians to create apersonal learning portfolio. The collection of

-6 -


information shows how physicians learn, whichfactors affect changes in practice and what sourcesof learning physicians use. It also creates databasesfor establishing standards of physicians’ performanceand for continuing medical education programmes(Bashook et al 1998).

The National Council for Vocational Qualificationsestablished assessment procedures for qualificationin England, Wales and Northern Ireland in whichportfolios play a central role. It is aimed for youngpeople in upper secondary schooling and youngpeople and adults in work or in government training.Wolf (1998) reports that the portfolio approach toassessment has affected the desired learning styles,but major problems have arisen relating to themanageability of the approach and reliability ofassessors’ judgements.

The General Medical Council in UK has put forwarda revalidation proposal for practising physicians(GMC 2000). The 3-stage proposal suggests aportfolio (folder) which will include information andevidence regarding doctors’ cumulative performanceto be assessed by groups of doctors.

4 What portfolios might containPortfolios contain students’ ongoing work over timewhich may provide evidence for learning andprogress towards educational and professionaloutcomes or learning objectives. These may includemaintenance of competences, keeping up to datewith professional practice, f itness for practice,adherence to professional standards and more.Portfolios may also contain student’s or candidate’sreflection on the submitted evidence.

By definition any material which provides evidencefor the above mentioned educational and work relatedcriteria can be included in portfolios. This mayinclude:

• best essays

• written reports or research projects

• samples of evaluation of performance, e.g. tutorreports from clinical attachments

• video tapes of interactions with patients or withpeers

• records of practical procedures undertaken (logbooks)

• annotated patient records

• letters of recommendation

• CVs

• written reflection on the evidence and onprofessional growth.

Evidence included in portfolios is limited only bythe degree of the designer’s creativity.

5 Why use portfolios?The use of portfolios for students’ assessment enablesstudents and teachers to engage in a process oflearning through assessment. This implies that theassessment procedure not only measures andreinforces the desired learning outcomes but ratherenhances the development of strategies, attitudes,skills and cognitive processes essential for life longlearning. Consequently, the use of portfolios not onlybroadens the scope of assessment but also introducesa number of educational benefits.

Portfolios’ contribution to assessment

• The assessment of learning outcomes:This includes outcomes not easily assessed byother methods, e.g. personal growth, self-directedlearning, reflective ability, self-assessment ofpersonal growth, professionalism and more. Anextended discussion of the use of portfolios forassessment of professionalism appears at the endof this section.

• The provision of evidence of performance:This evidence may be collected from a range ofsources, e.g. student’s on-going work over time,over settings and over subject matters.

• The representation of evidence collected over aperiod of time:Portfolios provide evidence of studentdevelopment over time and not just a snapshot atone time or a series of snapshots, as in traditionalassessment methods.

• Student progression towards the learningoutcomes:Portfolios allow assessment of progress towardsthe learning outcomes by using chronologicalwork samples collected at different points of time.

• Summative and formative assessment:The evaluation of the portfolio content generatessummative statements regarding studentperformance for promotion or pass/fail decisions.It also provides an in-depth profile of studentabilities. Summative evaluation statements mayalso contain information regarding students’strengths and weaknesses, thus integratingsummative decisions with formative profiles.

The current challenge facing portfolio assessmentis to judge the qualitative and quantitativeevidence and yet maintain reliability and validity

-7 -


of the judgement. Section C in this guide expandsthe discussion of this issue.

In addition to the contribution that the portfoliomakes to student assessment, it also reinforces otherimportant educational aspects central to effectiveteaching and learning.

Focus on personal attributes

• Provides personal and professional educationalevidence for student learning (Glen and Hight,1992).

• Makes provision for students to receive feedbackabout their own personal values, feelings, stylesof interaction and ways of handling significantexperiences (Beattie 1991)

• Provides a personalised sensitive portrait of whatstudents know and are able to do (Herman &Winter, 1994).

• Contextualises learning and links experience withpersonal interpretation.

Enhances interactions between studentsand teachers

• Allows dialogue between students and educatorsand ensures individual protected time.

• Reminds students that learning is a two-wayprocess between learner and educator.

• Mirrors students’ and teachers’ work andstimulates teachers to re-assess teaching strategies(Aschlacher 1993).

• Raises teachers’ expectations in relation tothinking ability and problem solving (Finlay et al1998).

Stimulates the use of reflective strategies

• Facilitates the use of past experiences to definelearning and to recognise progress.

• Stimulates the use of reflective skills in order toanalyse and synthesise experiences. Students canalso describe and analyse learning strategies thusengaging in metacognitive processes.

• Allows educators to separate the quality ofevidence from the student ability to reflect on theevidence.

Expands understanding of professionalcompetence

• The variations in student perception andinterpretation of their experiences increasesunderstanding of students’ professional growth.Those newly acquired insights inform medicaleducation and practice (Herman & Winter 1994).

• Encourages a holistic and integrative approachto medical practice.

6 The assessment of professionalism

Why assess professionalism?

The acquisition of the professional behaviourrequired to practise medicine is slowly emerging asa central focus in the new undergraduate medicalcurriculum (Cohen 2001) and with it increasingconsensus on professional behaviour (Swick 2000).

The introduction of new methods of assessment toexpand professional horizons (Friedman 2000)presents the profession with the opportunity torealign curricula to incorporate the development ofprofessionalism as a central outcome and find waysto assess professionalism and professional growth.

What is professionalism?

One of the unique problems for medicine in definingprofessional identity is that it develops from practicewhich in recent times has seen such enormouschanges both in biomedical advances and in shiftingrole boundaries in the delivery of care (Shapiro1999). There has been little appreciation thatprofessionalism is a dynamic concept which needsto be defined in the context of changing expectationsof society and changing times. Recently, however,clearer definitions of professionalism have startedto emerge. For example, reflective ability isincreasingly being identified as a key componentof medical professionalism (Grundy 1987, Lundberg1991, Parboosingh 1996, Wear 1997, Hafferty 2001).This builds on Schon’s work in relation to thedevelopment of the reflective practitioner forprofessional practice (1983).

Reflection requires both cognitive and humanitarianattributes. It is a process that enables practitioners todetermine their own actions, critically review theseactions and act on the outcome in the best interestof the client or patient.

Project Professionalism, which was introduced bythe American Board of Internal Medicine (ABIM)(1994), def ined medical professionalism as anideology consisting of the following elements: acommitment to:

• The highest standards of excellence in the practiceof medicine and in the generation anddissemination of knowledge (cognitive attributes)

• Sustain the interests and welfare of patients

• Be responsive to the health needs of society

-8 -


These elements were further def ined as thehumanitarian qualities of altruism, accountability,excellence, duty, honour, integrity and respect forothers. These are required of all candidates seekingcertification and recertification from the ABIM.

The Association of American Medical Colleges(AAMC) (1998), in their Medical Schools Objectivesproject, have also focused on professionaldevelopment in relation to both humanitarian andcognitive qualities. They have identified four coreattributes that all medical students should be able todemonstrate on graduation. The four attributesinclude altruism, knowledge (cognitive) skill anddutifulness. They further identify learning objectivesfor each of these attributes. In relation to altruismAAMC include characteristics such as compassion,honesty, integrity, team relations, commitment topatients interests, understanding threats to medicalprofessionalism and an awareness of one’s ownlimitations.

In the UK, the Scottish Deans Medical CurriculumGroup (2000), which consists of five medical schoolshas identif ied a Scottish medical graduate as acompetent and reflective practitioner. In their newoutcome based model, they have identif ied anaptitude for personal and professional developmentas one of the twelve curriculum outcomes of aScottish doctor at graduation. The key features ofthis outcome include self awareness, (developingreflective practice to be aware of one’s owncompetence), self learning (identifying one’s ownstrengths and weaknesses) self care (being aware ofinternal and external factors on one’s ownprofessional development, career choice andcommitment).

Why are portfolios a useful means ofassessing professionalism?

The development of multiple sampling in theassessment of clinical practice is receiving increasingattention particularly in relation to the assessmentof professional behaviour. Turnbull (2000) has useda system of clinical work sampling to assess studentsduring a rotation and found it to be a reliable, validand feasible form of assessment.

Portfolios, which sample evidence over time, alsodemonstrate a potential and reliable tool for theassessment of acquisition of knowledge, skills,attitudes, understanding and achievements and

“...reflect the current stages of development andactivity of the individual” (Brown 1995). The use ofportfolio assessment has recently been introducedat the University of Dundee School of Medicine(Davis 2001) as a key component in the f inalundergraduate degree examination, in order to assessaspects of medical practice such as professionaldevelopment.

Portfolios present an ideal method of assessingprofessional development as they enable theassessment of the doctor’s progress matched againstprofessional criteria over time.

Portfolios also enable professional development tobe assessed at different stages of a practitioner’scareer.

The sampling of work-related activity reflects thereality or authenticity of an individual’s professionalclinical practice. Having a variety of authentic eventsrecorded in the portfolio enables the examiner toidentify any cognitive or humanitarian deficienciesthat candidates may have in relation to theirprofessional development.

The portfolio assessment process allocates time forthe examiner to focus on the students’ development,during the oral component of the assessment, byasking the examinees to reflect on their professionalrole and related experiences.

The portfolio provides an ideal context to assessreflective ability as a key component of professionaldevelopment. Time is provided during the portfolioassessment for examiners to assess the ability ofstudents to reflect in relation to their portfolioevidence. In addition, reflective writing for aportfolio assessment demonstrates the examinee’sexperience separated from his/her personalinterpretation of the experience. Organising students’experiences through reflective writing modifies theirperception of the experience and enables it to beintegrated into prior learning or used to initiate newlearning (Eisner 1991).

The portfolio assessment process also provides a safeopportunity to explore errors in professional medicalpractice and strategies to avoid future errors. Thesesensitive matters can be discussed within theoutcome of an aptitude for personal and professionaldevelopment.

-9 -


B: The Dundee case studyDundee medical school moved to outcome-basededucation in the 1996-1997 academic year with 12learning outcomes identif ied. A three-circleclassif ication model was developed for thepresentation of the learning outcomes (Harden et al1999). The inner circle describes “what the doctoris able to do” and this includes seven competencies:1) clinical skills, 2) practical procedures, 3)investigating a patient, 4) patient management, 5)health promotion and disease prevention, 6)communication skills and 7) information handlingand retrieval.

The middle circle describes “how the doctorapproaches the task” and it includes 3competencies: 8) understanding of basic, clinical andsocial sciences, 9) appropriate attitudes, ethicalunderstanding and legal responsibilities, 10)appropriate decision making, clinical reasoning andjudgement.

The outer circle describes the “doctor as aprofessional” and it includes: 11) the role of thedoctor within the health service and 12) aptitude forpersonal development.

In Dundee University School of Medicine, thelearning outcomes provide the framework for studentlearning, teaching and assessment. Individualstudents’ progress towards the 12 outcomes, is thefocus of the portfolio assessment. In their portfoliosthe students provide the evidence that they haveachieved the outcomes at the appropriate levelexpected of a graduating student. The outcomedef initions are the criteria for the portfolioassessment.

The final examinations were redesigned to meet theneeds of the Dundee new curriculum (Davis et al2001). The curriculum incorporated an integrated,task-based approach that is interwoven with theoutcome model. The f inal examinations areadministered in two parts: Part 1 occurs in the endof year 4 when students have completed their coreclinical attachments. It includes a knowledge andproblem solving component measured by extendedmatching items (EMI), and constructed responsequestions (CRQ); and an OSCE to assess clinicalskills. Part 2 of the final examinations takes placetowards the end of year 5 by employing a portfolioassessment. Students and faculty were given theopportunity to evaluate, in a holistic approach,student progress towards all the outcomes includingthose dealing with the doctor as a professional andpersonal development which are not easily assessedby traditional assessment methods. The content of

the portfolio includes students’ summaries ofprogress towards each of the outcomes and a varietyof materials prepared during years 4 and 5 (Table1). The notes to examiners and students containdetailed descriptions of student work which are listedin Table 1.

The portfolio examination in Dundee is part of asequential assessment process in that students needto pass the year 4 final examinations before theyare permitted to sit for the portfolio. End of year 5students, who pass the portfolio examination, requireno further assessment and thus fulfil requirementsfor graduation. Students who present with minor ormajor deficiencies are referred either for furtherremedial work or for a year 5 f inal examinationcomprising an OSCE and review of the portfolio afterappropriate correction or additions.

Figure 1 outlines the process of portfolioexamination in Dundee from the examiners’perspective. Two examiners independently read eachstudent portfolio a few days prior to the date of theportfolio oral review with the student. The twoexaminers independently grade the portfolio contentemploying criteria and guidelines set out by themedical school. They use a marking sheet, theportfolio assessment scoring sheet (PAS) to inserttheir outcome grades based on their reading ofstudents’ portfolio work. The examiners’ gradingsof the 12 outcomes are independent but take accountof the already pre-marked student work. They focuson the overall patterns of achievement and progressover the two years rather than the grading of specificpieces of evidence. They make judgements on howthe different sets of portfolio evidence relate to thestudent progress towards the 12 outcomes. In a pre-review session on the day of the oral review, the twoexaminers meet and discuss the students’ portfolios.They agree on the students’ strengths andweaknesses and specif ic issues that should beexplored during the oral review of the portfolio withthe student. During the oral review of the portfoliowith the student, which lasts 40 minutes, theexaminers pursue the issues of concern and, at theend, they independently assign grades for eachoutcome based on students’ performance during theoral review. The examiners then reach consensus ona third set of outcome grades which constitutes thef inal grading based on which the pass/referreddecisions are made. They also makerecommendations for distinction. At the end of eachexamination day, all examiners for this day meet asan Examiners Committee to consider the referredand distinction students, to challenge the decisions

-10 -


Student work Year Number Pre-marked*

Student’s personal summary of progress towardseach outcome. 5 12 No

Patient presentations: short summaries of patientsseen by the student, selected from the 100 coreclinical problems on which teaching and learning isbased on this phase of the curriculum. 4 10 Sometimes marked

Case discussions: reports of approximately 1500words, each analysing a patient’s history and findingsin terms of one of the curriculum themes. 5 7 Yes

Year 4 assignment: a project report with a gradeawarded by the project supervisor together withfeedback for the students. 4 1 Yes

Record of achievement: a record of procedures thatthe student was expected to have completed orobserved during the phase. 4&5 1 Signed by Faculty

GP special study module assessment form: a reporton the student by their general practice supervisorwith a grade awarded. 5 1 Yes

Clinical special study module assessment form:theme special study module assessment form: areport on student performance during the modulewith a grade awarded for relevant outcomes. 5 2 Yes

PRHO apprenticeship in medicine assessment form:a learning contract between the student and his/hereducational supervisor with grades awarded for eachlearning outcome. 5 1 Yes

PRHO apprenticeship in surgery assessment form: alearning contract between the student and his/hereducational supervisor with grades awarded for eachlearning outcome. 5 1 Yes

Elective report: a report completed by the studentafter the elective period. This was read by one of thetwo members of staff responsible for elective studiesand written feedback provided. 5 1 Yes

Table 1: Description of portfolio contents

* The pre-marked student work is scored using the University grading system (A-G) and contains comments provided by Faculty.

-11 -


Broad sampling of students’ work(graded & not graded)

Agreement on strengths andweaknesses, to be explored

during the portfolio oral reviewwith the student

Oral review of portfolio byexaminers with the students

Examiner IIndependent outcome

grading

Examiner IIIndependent outcome

grading

Examiner IIndependent grading

of outcome

Examiner IIIndependent grading

of outcome

Portfolio readingprocess

Pre-portfoliooral review

Post-portfoliooral review

Portfolio material

Consensus on 12 outcomegrades by Examiners I and II

Consensus bygroup of examiners

DistinctionPass Referred

t

t

t t

t

t

t

t

t

tt

Figure 1: The portfolio examination process: the examiners’ perspective

and reach consensus for the final list of the passed,referred and distinction students.

Grades awarded for each outcome correspond to thefollowing criteria:

Grade A – excellentGrade B – very goodGrade C – satisfactoryGrade D – borderline passGrade E – marginal failGrade F – definite failGrade G – bad fail

Following the Examiners’ Committee meeting,students are allocated to one of the followingcategories:

• Exempt from further pass/fail outcome-basedassessment, (all grades are D or above).

• Exempt from further pass/fail outcome-basedassessment, but to be considered for distinction.These students will sit for a distinctionexamination which will determine who isawarded a distinction.

t

t

-12 -


• Potential exemption from further examination(only one outcome is graded E with all othersgraded D or above). Further work will be assignedto address the limited deficiency identified in theportfolio examination.

• Proceeds to further examination (about 6 weekslater) if two or more outcomes at Grade E or belowor one or more outcomes at Grade F or G. Thosestudents will sit an OSCE and a further portfolioassessment.

In summary, the portfolio examination in Dundee isone sequential component of the final examinationprocess which starts at the end of year 4. Studentsneed to pass the year 4 f inal examinationcomponents before they can sit the portfolioassessment. Students who pass the portfolioassessment, meet the final examination requirements.Students who fail the portfolio assessment proceedto either remediation or further testing via an OSCEand further review of the portfolio.

Figure 2 demonstrates the portfolio process from thestudent perspective. For each phase of the processthere is an associated educational value which

indicates how the portfolio process supports studentlearning.

In the beginning of year 4 students learn in advancewhat is expected of them. As they select the evidenceduring years 4 and 5, they decide what informationthey need in order to meet the 12 outcome criteria.When they compile the evidence in the portfolio,they demonstrate their organisational skills, timemanagement and prioritisation ability. The 12outcome summary sheet allow students to reflect ontheir progress and prepares them to “think on theirfeet” during the oral review. Students are asked tocomplete an evaluation form at the end of theportfolio assessment which allows them to furtherreflect on the whole process.

Students who pass the portfolio complete the cycleof demonstrating performance achievements.Students who were selected for distinction havefurther opportunities to demonstrate problem solvingability, information handling and retrieval andknowledge. The ones who were referred to furtherwork or testing have to act on their own deficiencies.

Educational value

Advance organiser

Meet learning needs within theoutcome framework

Organisational skills, timemanagement and prioritisation

Reflective ability

Think on one’s feet. Demonstrateprofessional behaviour

Reflection on the whole process.Have curriculum input.

Demonstrate problem solving ability, informationhandling and retrieval and knowledge

Meetrequirement

Act on owndeficiencies

Read orientation book +attend induction Start of Year 4

Selection of the evidence Start of Years 4&5

Compiling evidence in the portfolio Year 5

Write 12 outcome summary sheets Year 5

Oral review. Defending the evidence Year 5

Complete evaluation forms Year 5

Task Time

Pass ReferredDistinction

Figure 2: The portfolio process: student perspective

-13 -


C: Psychometric issues relating to the use of portfoliosfor assessment

1 Summary of the portfolio’s mainassessment features

Table 2 provides a summary of the portfolio’s mainassessment features.

1 Formative and summative

2 Qualitative and quantitative

3 Personalised

4 Standardised

5 Authentic

Table 2: The portfolio’s main assessment features

Formative and summative

The main assessment characteristics of portfoliosfocus on the FORMATIVE and SUMMATIVEaspects in which students collect evidence of theirown performance for pass/fail decisions, feedback,diagnostic purposes and intervention. The formativevalue of the portfolio, when linked to summativedecisions, present a very powerful assessment tool.In this section, a forward strategy is discussed forthe use of portfolios for summative decisions suchas promotion, graduation, certification and other highstakes applications.

Qualitative and quantitative

An important feature of portfolios is the combinedQUALITATIVE and QUANTITATIVE approach,unlike many other performance tests that rely mainlyon the quantitative aspect of performance. Theportfolio contains descriptive recorded material aswell as graded evidence. This approach to assessmentallows the integration of qualitative judgement withquantif ied information to generate a morecomprehensive interpretation of studentachievement.

Personalised

The STUDENT CENTERED aspect of portfoliosensures that the student assumes an active role indirecting the evidence while teacher involvement isessential in monitoring the process. By personalised,we imply the student individualised selection ofevidence, the individualised experience containedin the evidence, the review process by which theexaminer learns the unique student’s characteristics

and the opportunity given to students to reflect anddefend their work. This process creates a highlyPERSONALISED assessment method. When theportfolio approach is contrasted with the commonforms of an OSCE clinical station, the OSCE willfocus mainly on STANDARDISATION of theencounter while providing an individual assessmentprofile.

The portfolio seeks to balance the personalisedapproach to assessment and the standardisation ofthe process.

Standardised

It is diff icult to achieve maximum control ofstandardisation for the portfolio content and process.It is possible to determine the type of work to besubmitted (case studies, projects, essay, etc).However, human performance with regard tostructure, depth, volume, material exhibition ishighly variable (Pitts et al 2001).

In their arguments Pitts et at (2001) fear that a toostructured approach to developing guidance forusing a portfolio as an assessment tool mightdiminish its strength. The dilemma is how to maintainthe personalised aspect of portfolio but at the sametime standardize the approach to enhance theportfolio’s reliability. Some suggestions forstandardisation are provided in Table 3.

1 Same portfolio’s units of evidence are assignedto all students.

2 Tasks and criteria for assessment are definedand made clear.

3 Instructions to students provide clearguidelines.

4 The portfolio reading process and rating ofmaterial follow standardised guidelines – mainlyby written instructions and training workshopsfor examiners.

5 The probing in an oral review of the portfoliowith the student follows standardisedguidelines.

6 Pass/fail decisions follow a pre-determinedpolicy.

Table 3: Suggestions for standardisation of portfolio assessment

-14 -


In cases where portfolio assessment incorporates anoral portfolio review component (note this is not atraditional oral examination) one has to considerstrategies to avoid the many pitfalls of oralexamination (Missin 1985). Factors which accountfor problems in oral examinations include:taxonomic level of questions (Des Marchais 1995),the subjectivity of examiners, the lack ofstandardised criteria employed by examiners eitherfor probing or for grading, lack of sharedinterpretation of examinees’ behaviour and variedlevels of expectations from examinees. All maycontribute significantly to lack of standardisationand low reliability.

However, some of these problems could be rectifiedby training and by extensive reading of individualstudent’s portfolios which helps prepare theexaminers for a contextualised and personalised oralencounter while maintaining a holistic frameworkof student performance.

The standardisation process is central to the reliabilityand the validity of the portfolio. The validity ofgrading relates to the interpretation by the examinersof students’ work, the expectations, sharedunderstanding of criteria for assigning the grade andthe use of the results (Messick 1994).

Authentic

Authenticity is defined as “the extent to which theoutcomes measured represent appropriate,meaningful, signif icant and worthwhile forms ofhuman accomplishments”. (Archibald and Newmann1988). Characteristics of authentic achievements asjudged in portfolio assessment are provided inTable 4.

1 Production of knowledge rather thanreproduction or citing others’ work

2 Disciplined enquiry, dependent on:

• A prior knowledge base (to be used to helpproduce knowledge).

• In-depth understanding• Integration – the production of knowledge

requires the ability to organise, synthesiseand integrate information in new ways.

3 Value beyond assessment – aesthetic,utilitarian or personal value.

4 Higher order thinking.

Table 4: Characteristics of authentic achievements as judged inportfolio assessment (Cumming et al 1999)

The portfolio is authentic in the sense that itincorporates past academic and work-relatedexperiences which form the foundations for theabove-mentioned criteria. Messick (1994) states thata portfolio is an authentic simulation in that it doesnot simulate a particular task, but rather requires thatthe examinees by conceptualizing about the case willgeneralize to other occasions. The authentic aspectof the portfolio is central to the validity of theportfolio material and the scores’ interpretations.

It is also an essential aspect of developingprofessional attitudes while reflecting on one’s ownexperiences (Schon 1987).

For this reason Case (1994) states her concern forthe loss of portfolio authenticity in large scale, morestandardised, less personalised examinations. Thepersonalised approach of portfolio assessment is theunique component which contributes to itsauthenticity, i.e. its validity. The lack of it mayincrease its reliability but may also decrease itsvalidity.

2 Raters’ consistency

Discussions of reliability are often centered aroundterms of “generalisability”; that is, the extent to whichperformance is consistent across various sources oferror labelled “facets”. (Koretz 1998). In theportfolio case, raters are considered as one facet oferror that threatens generalizability of performance.

The raters’ effect includes issues of raters’ stabilityover time (Test - retest), stability over raters (Inter-rater) and reproducibility or decision consistency ofpass/fail marks (Livingston 1995).

Given the complexity of portfolio assessment, theamount of time involved in processing the test andthe actual knowledge learned about examinees fromthe f irst administration, present diff iculties forstudying raters’ stability over a relatively short time.As for inter rater agreement, Le Mahier et al (1993)reported .60 to .70 inter-rater correlations andgeneralizability estimates between two raters for eachunit of portfolio evidence in the range of .80.Herman et al (1995) reported average correlationsbetween pairs of raters of .82. Percentages of absoluteagreements of all pairs averaged resulted in 98%.

Koretz (1998) states that much of the total error inscores was attributed to factors other than raters inthe portfolio. High agreement depends on clearcriteria, adequate examiners’ training,communicating criteria to students, good studentorientation materials, examiners’ familiarity with thecontext and shared understanding of expected

-15 -


student performance and the assessment purpose.In the Dundee case study, as in other portfolioassessments, the stability over raters is mainly relatedto the consistency of their interpretation of patternsof performance over time and over tasks, rather thanagreement on a single unit of performance or on asingle grade. The shared understanding of studentprogress towards the curriculum learning outcomesin Dundee is the focus of examiners’ training.Authentic criteria, which incorporate progressionalbenchmarks towards the global outcomes, is theDundee current challenge for achieving high inter-rater reliability. We will further discuss developmentof criteria in the section regarding implementation.However, one needs to question the meaning of inter-rater consistency in portfolio assessment. Forexample, in a portfolio which contains one sampleof “best work”, for the purpose of assessing an abilitysuch as writing, high absolute inter-rater agreementwill be essential if decisions are based on the writingscore attained. However, for portfolios which containbroad, complex, multiple tasks, one needs to questionthe likelihood of two independent examiners arrivingat identical grading of individual sets of evidence.In this case, it might be important that examinersagree on the consequences of the grading rather thanon the actual grade obtained.

3 Decision consistency

Misclassification of students into pass/fail or referredcategories due to raters’ source of error, is a mainconcern regarding disagreement among raters. Thelack of reproducibility as a function of low inter-rater agreement, is a major threat to the reliability ofthe portfolio examination in high stakes examinationor any portfolio examination with summativeconsequences.

Inter-rater agreements for units of portfolio mayresult in high correlations. However, what isimportant is the reproducibility of pass/fail decisionsamong raters. For example, do the pass/fail standards/criteria or other policy decisions assigned to themarks produce the same classification consequencesacross raters? If correlations are high among raters,while one rater consistently rates lower, students whoobtained the low ratings may fail based on theexamination standards, while students who achievedhigher ratings may pass even though the two raters’grading correlated very highly. Consequently,agreements expressed in correlation statistics are notsuff icient if pass/fail standards are employed.Correlation statistics and percent agreements shouldaccompany the following questions: 1) What is theextent of agreement for individual units among ratersin reaching the same pass/fail results? (Livingston

1995). 2) Irrespective of the grades assigned, doraters reach the same pass/fail decision?

Data available from the f irst year of the Dundeeportfolio administration, shows 98% pass/referredagreement between two pairs of examiners, whoindependently examined the students.Generalizability coefficients will define the amountof error attributed to raters. It will also assist inestimating the number of raters needed to achievehigh reproducibility (Brennan 1983).

4 Sampling

Sampling of tasks is another important “facet” ofmeasurement error in portfolio assessment. In fact,given the complexity of behaviours measured by theportfolio assessment, and in order to achieveconsistency of performance across a number oftasks, sampling needs to be extensive.

Linn (1994) raises the issue of limited sampling inperformance assessment and urges the use ofsampling of student work over time and over tasksto reduce sampling error due to task, by employingportfolios and/or multiple sources of performanceassessment. Appropriate sampling is a commonproblem in performance assessment due to limitedexamination time. The Dundee approach allows suchsampling through the preparation of evidence overtime, over occasions, employing multiple tasks whileaddressing the 12 outcomes. This approach isdifferent from a portfolio in which one unit of “bestwork” is presented as evidence. In the “best work”case, when a limited number of competences areassessed, the support provided by more evidence ofthe same ability as in the Vermont example, willincrease the portfolio validity due to broadersampling.

5 The consensus approach

In the Dundee portfolio examination, judgementson student performance are generated byincorporating a consensus approach. Can one employa consensus approach for pass/fail decisions? Howscientif ic and accurate is the consensus processbetween two raters or among committee members?There is evidence that scientif ic theories aredetermined by a consensus approach rather than bya critical thinking process, or by consistency amongindependent research findings (Hand 1999).

It is common practice in assessment centres toincorporate a consensus approach for judgingtrainees performance (Gagler et al 1987). In practice,the selection of trainees and other higher level

-16 -


personnel is conducted by selection committeeswhich employ a consensus process rather than inter-rater agreements. Juries also operate on a consensusprinciple. In both cases evidence is presented, thepanel argues the case and decision is grantedfollowing a consensus process.

Relevant to personalizing assessment in aperformance domain, is the issue of how one guardsagainst subjectivity of raters. The portfolio approach,by its complex nature, calls for judgement ofmultiple performance aspects, cognitive, attitudinal,professional and technical skills that are bestexpressed at the personalised level. The threats toreliability among raters, in such a complex andpersonalised undertaking, raises the question: shouldconsensus procedures be recognised as a legitimateand reliable assessment strategy and can consensusprocedures replace independent ratings? Linn (1994)mentions the work of Moss (1994) on validity versusreliability and brings examples of search committeesexamining the credentials of candidates and arrivingat an integrated decision. Delandshere and Petrosky(1994) talk about confirmation of prior judgementrather than replication, analogous to a secondopinion from a physician. Linn (1994) states thatthe confirmation approach may be a “useful way oflooking at student portfolios and exhibitions?”However, in such cases examinee performance isjudged by internal examiners who are familiar withthe educational context and the outcome frameworkas it is practised, for example, in Dundee.

Due to the highly qualitative nature of portfolios, itseems reasonable to maintain independent rating andstrive for high inter-rater agreement. At the sametime, we may ask: how can we equip examiners withthe right tools and the appropriate definitions to judgeexaminees independently and further submit thesejudgements to a rigorous consensus process?

The use of examples and evidence to support raters’judgements against pre-determined criteria is oneway to achieve a meaningful ratings and consensusprocess. Justif ication of grading within definedframeworks and shared criteria may create ameaningful and reliable consensus process.

In the Dundee example, two stages of independentratings provide the basis for the final consensusprocess, which leads to an overall judgement andagreement between the two raters. It seems that thethree forms of reliability: agreement amongindependent ratings, reproducibility of pass/faildecisions and rigorous consensus procedures mayall provide an integrated solution for achieving highreliability and for the use of portfolios for summativepurposes.

When raters provide their independent grading theyown this decision. In the consensus process they mustnegotiate the grades by supplying evidence of studentperformance and explain their interpretation of theperformances. Inter-rater agreements could bestudied as well as reproducibility of pass/faildecisions to establish the overall reliability of theportfolio examination. A stronger design mayincorporate two pairs of examiners, each providingindividual ratings as well as consensus judgement.

This approach was implemented in Dundee in thefirst year of the portfolio examination. Each pair ofexaminers conducted a 20-minute oral review of theportfolio with the student. However, it was felt thatsome students needed more time with each pair and,on many occasions, students faced the samequestions by the two pairs albeit concerning differentcomponents of the portfolio. There was also an issueof manpower. Currently, Dundee employs only onepair of examiners for the 40 minute oral review ofthe portfolio with the student, but this policy is underreview. One proposal is to have two pairs ofexaminers with a student for 30 minutes each.

Work needs to be done to further study the validityand reliability of the consensus approach ascomplementary to other forms of reliability.

6 Forms of validity

Discussions of validity include the appropriatenessof score interpretation and the specific inferencesmade from the test (Messick 1994). The validity ofthe inferences depends also on the reliability of thetest. If test scores suffer from low accuracy due tolow inter-rater agreement or poor sampling,inferences cannot be made.

The authenticity aspect of the portfolio strengthensthe predictive validity of the portfolio assessment.If portfolio examinations are adequate to measurehigh level, complex abilities applicable in real lifesituations, then its predictive validity shoulddemonstrate strong relationship with laterprofessional or career performance.

Such long-term studies were not found. Finlay et al(1998) conducted a randomized controlled study andhas shown better results for the portfolio(experimental) group on Oncology factualknowledge. Those submitting portfolios had overallhigher marks than those who have selected not tosubmit. This study may also indicate differences inmotivation.

As for construct validity, Koretz et al (1993) foundmoderate correlations (.47-.58) between writing

-17 -


portfolio scores and other direct writing assignment.Where pass/fail decisions are considered, thedilemma of low correlations between differentmeasures becomes more problematic. One has toensure that students will achieve mastery on boththe portfolio and other forms of writing ability toestablish comparability of measures. Also it wasfound that different scoring approaches to theportfolio may produce different classification results.(Koretz 1993).

Currently, a central validity aspect of the portfolioassessment is its face validity. The process, theevidence and the personalised approach are attractiveto both students and educators. However, work isneeded to further establish its predictive validity andits relationship to other measures of studentperformance.

7 External vs internal examiners

It is argued that portfolio assessment is more suitablefor internal examiners (Linn 1994) where familiaritywith the educational programme and with studentprogression ensures consistency of marking. Thepersonalised approach of portfolio assessmentrequires internal knowledge of the educationalmilieu. External examiners at the state or district levelmay be recognised more as examiners for qualitycontrol of the portfolio process. This may haveimplications for judging portfolio work at a centralsite without seeing the examinee or lack of familiaritywith the educational context. The purpose of suchcentral marking of portfolio assessment should be

clarified and the criteria employed for judgementshould incorporate standardisation procedures as wellas generic criteria suitable to all examineesindependent of educational programme. Externalexaminers who sit in a personalised portfolio maydiffer in their interpretation of examinee performancedue to a different educational philosophy in theirown schools. External examiners, however, arecrucial in the assessment systems of UK medicalschools. When external examiners participate inportfolio examinations, steps should be taken toensure that they do not bias the results. Specifictraining programmes should be designed for externalexaminers to familiarise them with the curriculum,student assessment and educational philosophy ofthe school.

The contextual effect on portfolio performance aswell as the personalised approach to students areforces which attempt to localize the portfolio ratherthan use it for external testing.

The large number of studies on portfolioimplementation indicate that internal as well asexternal uses are reported. The difference in theapplication lies in the purpose of the portfolio andthe criteria employed for judgement.

External uses may employ general criteria forstudents’ work which is independent of educationalor work-related context. The external marking ofstudent work may generate group data on schoolprof iles, students’ level of achievement and thequality control of educational programmes.

D: Implementation of portfolio assessmentImplementation of portfolio assessments commonlyincorporates a sequence of steps. These steps areidentified in Table 5. For each step we will bringexamples to illustrate some practical considerations.

1 Defining the purpose2 Determining competences to be assessed3 Selection of portfolio material4 Developing a marking system5 Selection and training of examiners6 Planning the examination process7 Student orientation8 Developing guidelines for decisions9 Establishing reliability and validity evidence10 Designing evaluation procedures

Table 5: Steps in developing portfolios

1 Defining the purpose

Portfolios are used for summative and formativedecisions, for different trainees’ levels and in a varietyof settings. The variation in its content, process andapplication requires that the purpose will be clearlydefined. The non-traditional assessment approachesemployed by portfolios may introduce confusion anduncertainty regarding its purpose. The clearer thepurpose, the higher the quality of the portfolioassessment.

An example of a purpose definition

“The purpose of the portfolio assessment is todetermine, in line with the GMC guidelines, that thegraduating student is ready to proceed to the Pre-registration House Office (PRHO) year and that the

-18 -


student has achieved the required leaning outcomes.The portfolio is only one component of the finalexamination. Successful performance on theportfolio is a requirement for graduation.”

The purpose statement directs the test designers toconsider the competencies essential for entering thePRHO year, to determine which competencies arealready measured by the other f inal examinationcomponents, and how the portfolio shouldsupplement the other assessment methods. In thisstatement, the portfolio is one hurdle whichnecessitates a summative decision in order to fulfilgraduation requirements. The purpose defines theassessment system and the relationship between theportfolio and the other assessment components.

2 Determining competences to beassessed

Faculty, medical educators, assessment committeeand other medical school staff will meet to discussthe purpose of portfolio assessment. They will reviewthe existing assessment procedure and will identifygaps in assessment which should be supplementedby the portfolio.

In Dundee, where the portfolio is def ined as arequirement towards graduation, the f irst 11outcomes are assessed at the “knows” and “knowshow” level (Miller 1991) by EMIs and CRQs, and atthe “shows how” level by the OSCE. At the level of“does”, however, outcome 12 is only assessed bythe portfolio.

Table 6 shows the grid of final examination and theassociated methods for each outcome.

The portfolio, through students’ cumulative work,identifies strengths and weaknesses in all outcomeswith emphasis on outcomes 6-12. The portfoliodesigners may also emphasise the interrelationsamong the outcomes and students’ overall progresstowards all outcomes.

The process of identification of competencies to beassessed signifies a systemic and systematic approachto assessment (Friedman 1999) which ensures acomprehensive strategy to determine qualificationfor graduation. The principle that only oneassessment method is inadequate to determinefitness for practice guides this design.

3 Selection of portfolio material

The identification of competencies to be measuredis a starting point for selection of portfolio material.It is important to identify key performance

behaviours relevant to the competency measured.The identified behaviours will guide the portfoliocontent selection.

For example, in assessing trainees’ patientmanagement skills, one of the selected performancebehaviours could be patient education. The portfoliodesigners will search for a work sample of patienteducation. Evidence might include written outlineof a patient education programme in the community,or a video of individual patient education sessionwith patient discussing smoking cessation. It isimportant to remember that evidence is selected fromongoing trainee’s work and is not created in anartificial way just for the portfolio examination. It isalso essential that the compilation of evidence willserve as learning and educational experience for thetrainee. For example, in selecting a video for patienteducation, trainees will employ criteria for selectingthe best patient education encounter. In defendingthe selection, the trainee demonstrates understandingof criteria for optimal performance.

For outcome 9, appropriate attitudes, ethical and legalunderstanding and responsibility, the portfoliomaterial in Dundee contains the elective report –which provides evidence if the student shows ethicalunderstanding of issues inherent in the elective. In acase discussion on ethics, the students will provideevidence of ethical judgement and moral reasoning– and could be questioned about the case.

In the Dundee case study, where the purpose ofassessment is progress towards the 12 outcomes, theevidence should indicate progress over time. Aselection of samples of student work will facilitatediscussion of patterns of performance and overallstrengths and weaknesses.

4 Developing a marking system

In the Vermont application of portfolios, genericcriteria were developed specifically for grade 4 inmathematics. Students’ work is judged by criteriawhich will specify the level of their academicachievements and will determine their progresstowards state standards. This approach is possiblewhen one unit of portfolio content is evaluated.However, in the Dundee case study, the 12 broadlydefined outcomes do not allow the application ofspecif ic criteria, but rather general statementsregarding student performance relative to theoutcome specif ication. For example, under theoutcome Information Handling the followingdimensions are specified: “the doctor is competentin recording, retrieving and analysing informationusing a range of methods including computers”.Criteria for marking may include evaluation of

-19 -


Outcomes EMI1 CRQ2 OSCE Portfolio

1 Clinical skills xxx xxx xxx xx

2 Practical procedures xx xx xxx xx

3 Investigating a patient xx xx xxx xx

4 Patient management xxx xxx xxx xx

5 Health promotion and disease prevention xx xx xxx xx

6 Communication skills x x xxx xxx

7 Information handling xx xx x xxx

8 Understanding of basic, clinical and social sciences xxx xxx x xxx

9 Appropriate attitudes ethical and legal responsibility xx xx xx xxx

10 Decision making, clinical reasoning xxx xxx xxx xxx

11 Role of the doctor x x xx xxx

12 Personal development x x xx xxx

1 EMI (Extended Matching Items), 2 CRQ (Constructed Response Questions)x – considered for use, xx – appropriate and is currently used, xxx – most appropriate assessment method

Table 6: Final examination grid in Dundee outcome-based curriculum

patient records (including specific marks for qualityof records). Criteria for search strategies (includingmarks for formulating research questions andgenerating key words for multiple databases) andcriteria for analysing information (including marksfor critique of a research paper).

The portfolio material should direct the examiner toconsider student progress according to the outcomespecif ication and should enable the examiner toidentify strengths and weaknesses. As mentionedearlier, highly specific criteria could be employed ifone competence is assessed. For multiplecompetences, general standards should bedeveloped.

It is also possible as in the Dundee case study toassign grades to student global work. However, priorto grades’ assignment, examiners must shareunderstanding of the specific or general criteria andconsistently negotiate what constitutes an A levelgrade (excellent) versus an F level (fail) and furtherdefine and refine the points on the continuum.

Schools may be bound by university gradingsystems. The best approach will be to follow theabove procedures. Def inition of criteria forperformance and identif ication of level ofperformance which will correspond to the universitygrading system.

5 Selection and training of examiners

If one considers only an internal examiners’ systemfor portfolio assessment, selection of examiners will

follow the internal structure and philosophy of theeducational program. Each portfolio according toits purpose will dictate the appropriate examiners.These will include a wide range of staff; teachers inthe basic sciences and laboratory-based disciplines,clinicians, faculty who indicate special interest ineducation and in student development, faculty whoare motivated to spend the extra time needed andfaculty who are good oral examiners (mainly studentcentred). Another selection issue relates to theseniority of examiners. Pairing of new examinerswith more senior colleagues with experience ofportfolio assessment has been found to be helpfulin Dundee. Whatever the selection process employed,the training of faculty examiners and maintainingthem in the examiners’ pool is a key point for thesuccess of the programme. Naturally the amount oftime and effort invested in the examiners’ poolthrough training and orientation and theiraccumulated experience with portfolio examination,should be preserved and reinforced. Consequently,they should be treated accordingly (dinners, lunches)and given help to form a group with recognisedacademic respect and appreciation. The fact that afaculty member says “ I can’t join you for tomorrow’smeeting – I am a portfolio examiner”, should bringpride to the speaker and respect from the listener.

Faculty in general appreciate their participation inportfolio examinations (Davis 2001). It provides anopportunity to know the students and to shareopinions with their colleagues. They obtain betterperspectives on student accumulated work andprogress. Although the amount of reading might be

-20 -


excessive (a problem if one decided to maintain thebroad sampling principle) realistic approaches toreading plans can keep this burden at manageablelevels.

6 Planning the examination process

Fortunately, the use of OSCEs has introduced theconcept of individualised student assessment. Thisconcept, together with the time needed, facultymanpower, logistics, administration issues andavailability of support staff are all issues shared withportfolio examinations.

In Dundee, with a class of 150, each pair ofexaminers sees five students for a total examinationtime of 200 minutes (40 minutes for each oral reviewof the portfolio with the student). If we include lunch,coffee breaks, discussion time between candidates,f illing up forms and the Examiners Committeemeeting at the end of the day – five students per dayis the maximum number. If 10 pairs of examinersare recruited for each day – all 150 students couldsit the oral review examination in three days withdifferent sets of examiners. The number ofexaminers needed for such an assessment operationis 60 with each examiner being committed for justone day. There is, of course, the additional timeexaminers spend reading the portfolios before themeeting with the students, approximately one hourper portfolio in Dundee.

Faculty needs to buy into the concept of portfolio tobe willing to participate in such an importantundertaking. As with all aspects of reform, strongleadership is needed within the medical school tofacilitate the “launch” of the first administration andkeep the momentum in subsequent years in spite ofany reservations among faculty, students andadministrators.

7 Student orientation

Students must be informed at the beginning of theundergraduate course about the portfolioexamination and kept up to date with any changesin subsequent years.

A special orientation booklet for students is issuedto inform them of the purpose and content ofportfolio building, the portfolio assessment process,the examination day, the marking system and theuse of results.

The more complex the task of portfolio building,the more anxious the student becomes. Problems mayoccur if the guidelines and criteria for judgingperformance are not entirely clear as may happen

when the guidelines are too broad. However, in theDundee case study, students use their on-going workfor selection of material – and usually, if theydemonstrate good progress in their achievements,they have confidence in their ability to pass theportfolio.

The more information given to students the morepositive they become towards the portfolio. Followingthe portfolio examination, students are pleased thatsenior examiners have given detailed attention totheir work over their last two years of medical school.The students, in general, like this personalised,student centered, contextualised assessmentapproach.

8 Developing guidelines for decisions

Once a grading system is developed, the portfoliodesigners must develop policy and guidelines forstandards of performance. If the portfolio is usedfor summative pass/fail decisions, standards shouldspecify what constitutes a fail or a pass. If portfoliosare used for formative decisions, standards forstrengths and weaknesses should be defined alongwith due process for intervention and remediation.The process should also incorporate the proceduresfor evaluating the intervention and the decisionmechanisms to further promote or to hold back/expelthe student. The decision making process may beoutlined by the following flow diagram.

9 Establishing reliability and validityevidence

It is important to determine, prior to theimplementation of the portfolio, what will constitutegood reliable evidence and plan the examination

Portfolio examination

Pass Fail Referred

Pass Fail

Graduate Committee review

Sit foranother exam

t t t

t

t t

t t

t

Undergoremediation

Fail Pass

Graduate

t t

t

t

Figure 3: Flow diagram for the decision-making process

-21 -


accordingly. For example: 2 pairs of independentexaminers, 1 pair of examiners, independent rating,consensus or both, minimum desired reliability orgeneralisability co-eff icient. Def ine desiredcorrelations or absolute inter-rater agreement and setthe minimum standards for tolerance ofmisclassification error. The degree of reliability inpilot studies, prior to actual administration, shoulddirect faculty to make decisions regarding whetherthe portfolio should be used as a formative or asummative assessment.

Triangulation of portfolio results with other formsof assessment will increase the validity of thedecision and will guide faculty as to the use of theportfolio results. If portfolio assessment does notcorrelate with any of the other forms of examination– its results could be aggregated as a weightedcomponent towards graduation. Recent studiessupport the use of a battery of tests to generatecomposite reliability for test components combined(Wass 2000).

10 Designing evaluation procedures

Students’ and examiners’ opinions on the portfolio’sstrengths and weaknesses should be sought in order

that feedback can introduce changes andimprovements. Questionnaires, focus groups,individual interviews and requests for writtencomments can all be employed. The appropriatecommittees may initiate changes based on evaluationresults and will inform the subsequent years aboutthe areas of satisfaction with the portfolio in orderto market its usefulness.

Students’ performance on the portfolio with relationto performance on other assessment tools areimportant evaluation strategies in order to detect ifstudents who are generally high academic achieversdo not fail the portfolio or vice versa. Too manysurprises may not be good news for the medicalschools. Many concerns may arise: either theundergraduate course does not detect early signs ofdef iciencies, the other assessment tools are notadequate, or the portfolio examination is not validfor its purpose. With any form of new examination,it is important to conduct pilot studies and use theprocedures as formative until the necessaryevaluation data is obtained.

E: ConclusionThe added value of portfolios for students’ assessmentis increasingly recognised by the medical profession.In the UK, medical schools have reported a varietyof forms of portfolio application (McKimm 2001).The portfolio method fits the triangulation approachin assessment in which not one method but rather acombination of assessment methods can capture thecomplexity of professional competencies. With theaddition of innovative methods, like portfolios, weare increasing the “menu” from which we may chooseappropriate assessment methods. The portfolioassessment process can be summarized by a numberof steps.

Traditionally, assessment procedures mayincorporate only few steps, for example, Steps I, 111and V only. Thus, the learner takes a passive role.The portfolio process allows learners to take an activerole in the assessment process through selection ofmaterial, reflection on progress and learning and,defending the evidence, similar to a dissertation inwhich the candidate assumes responsibility for theentire process.

Documentation ofexperience by the learner

Evidence

Commentary by the learneron experiences and learning

that has resulted

Reflection

Step I

Step II

Studying the evidenceby examiners

Evaluation

A dialogue betweenlearner and examiner

Defending the evidence

Step III

Step IV

Formativeand summative

Assessment decision

Step V

-22 -


References

Use of portfolio in all stages of medical educationwill be most benef icial for the education ofprofessionals. Each phase of the medical continuum:undergraduate, postgraduate and continuing medicaleducation, should consider how to use portfolios fordetermining fitness for practice. The portfolios mayprovide the methodological aspect which focuses onprofessionalism. The Quality Assurance Agencystipulation that UK universities introduce personaldevelopment planning is another potentialopportunity for introducing portfolios both foreducation and assessment.

It is important to consider the implication ofportfolios as a new assessment method in theundergraduate curriculum. Students are normallyprone to demonstrate anxiety and stress with theintroduction of new assessment methods. Thedevelopment of explicit criteria for portfolio contentselection, with examples of portfolio evidence, andexplicit criteria for judging the material, will certainlyalleviate some of the anxiety.

Students must feel that the content selection ismeaningful and that they are not involved in a paper

chase exercise. Strong staff support is anotherimportant aspect of implementing a portfolioassessment.

As was shown in the Dundee case study (Davis,personal communication) students change theirperception of portfolios with time. Acting on studentfeedback is a very important strategy to orientstudents not only to the assessment process but alsoto the design.

However, portfolios for examination have yet to winpublic and professional acceptability. They are usedsuccessfully for formative purposes but summativeapplications of portfolios are yet to be furtherdeveloped. The portfolios’ psychometric aspects ofvalidity and reliability are currently being exploredfor high stakes examination as well as for summativepurposes. However, in seeking technical rigor weneed to be sure not to lose the appeal of the portfolioconcept. It is hoped that this Guide will stimulatefurther studies and innovative approaches that willstrengthen the psychometric aspects of portfolios aswell as its educational implications.

Al Shehri, A. (1995). Learning by reflection ingeneral practice. A study report, Education forGeneral Practice 7: 237-245.

American Board of Internal Medicine (1995). ProjectProfessionalism. Philadelphia PA.

Archibald, D.A. & Newmann, F.M. (1988). Assessingauthentic academic achievement in the secondaryschool, Reston, V.A., National Association ofsecondary school principals.

Aschlacher, P.R. (1993). Issues in innovativeassessment for classroom practice, barriers andfacilitators (Tech. Rep No 359) Los AngelesUniversity of California, CRESST: Center for theStudy of Education.

Association of American Medical Colleges (1998).Learning Objectives for Medical Student Education:Guidelines for Medical Schools: Medical SchoolsObjectives.

Bashook, P.G. & Parboosingh, J. (1998).Recertification and the maintenance of competence,British Medical Journal 316.

Beattie, A. (1991). Curriculum Co-ordination.National Board for Nursing, Midwifery and HealthVisiting for Scotland 1(5): 47-59.

Brennan, R.L. (1983). Elements of generalisabilitytheory. The American college testing program.

Brown, R. (1995). Portfolio Development andProfiling for Nurses, Journal of Clinical Nursing 4:249-255.

Case, S.H. (1994). Will mandating portfoliosundermine their value? Educational Leadership.

Cohen, R. (2001). Assessing professional behaviourand medical error, Medical Teacher 23(2): 145-151.

Cumming, J.J. & Maxwell G.S. (1999).Contextualising Authentic Assessment. Assessmentin Education 6(2): 177-194.

Davis, M.H., Friedman Ben-David, M, Harden, R.M.,Howie, P., Ker, J., McGhee, C. Pippard, M.J.,Snadden, D. (2001). Portfolio assessment in medicalstudents’ final examinations, Medical Teacher 23:357-366.

-23 -


Delandshere, G. & Petrosky, A. (1994). Capturingteachers knowledge, performance assessment a) andpost-structuralist epistemology, b) from post-structuralist perspective, c) and post-structuralism,d) none of the above, Educational Researcher 23(5):11-18.

Des Marchais, J.E. & Jean, P. (1995). Effects ofExaminer Training on Open-ended, highertaxonomic level of questioning in oral certification,Examinations Teaching and Learning in Medicine5(1): 24-28.

Eisner, E. (1991). Forms of Understanding and theFuture of Education, Educational Researcher 22: 5-11.

Finlay, I.G., Haughan, T.S., Webster, D.J.T. (1998).A randomised controlled study of portfolio learningin undergraduate cancer education, MedicalEducation 32: 172-176.

Friedman Ben-David, M. (2000). The role ofassessment in expanding professional horizons,Medical Teacher 22(5).

Friedman Ben-David, M. (1999). Assessment inoutcome-based education, Medical Teacher 21(1):23-25.

Gahler, B.B., Rosenthal, D.B., Thoronton, 111, G.C.,Benston, C. (1987). Meta analysis of assessmentcenter validity, Journal of Applied Psychology 72(3).

Ginsburg, S., Regehr, G., Hatala, R., McNaughton,N., Frohna, A., Hodges, B., Lorelei, L., Stern, D.(2000). Context, conflict and resolution: A newconceptual framework evaluating professionalism,Academic Medicine 75(10).

Gisselle, O. & Martin-Kneip. (2000). Becoming aBetter Teacher. ASCD, Alexandria, Virginia, USA.

Glen, S. & Hight, N. (1992). Portfolios: an AffectiveAssessment Strategy? Nurse Education Study 12:415-423.

GMC (2000). Revalidating Doctors: ensuringstandards, securing the future.

GMC (1997). The New Doctor.

Hand, R.J. (1999). Science Education: Consensusvs. Critique, Teaching in Higher Education 4(4): 501-510.

Grundy, S. (1987). Curriculum: Product or process.Lewis Falmer.

Harden, R.M., Crosby, J.R., Davis, M.H., FriedmanBen-David, M. (1999). Outcome-based educationfrom competency to meta competency: a model forspecification of learning outcomes, Medical Teacher21(6).

Herman, J.L., Gearhart, M, Baker, E.L. (1995).Assessing writing portfolios: issues in the validityand meaning of scores, Educational Assessment 1(3):201-224.

Herman, J.L. & Winter, L. (1994). Portfolio researcha slim collection. Education Leadership.

Jasper, A.U. (1995). The potential of the professionalportfolio for nursing, Journal of Clinical Nursing 4:249-255.

Koretz, D. (1998). Large scale portfolio assessmentsin the US: Evidence pertaining to the quality ofmeasurement, Assessment in Education 5(3): 309-334.

Koretz, D., Broadfoot, P., Wolf, A. (1998). Editorial,Assessment in Education 5(3).

Koretz, D., Stecher, B., Deibent, E. (1993). Thereliability of scores from the 1992 Vermont portfolioassessment program. (Tech. Rep. No. 355). LosAngeles University of California CRESST: Centerfor the Study of Evaluation.

Le Mahieu, P.D.H., Gitumer and J.T. Evesh (1993).Portfolios in Large Scale Assessment: Difficult butnot impossible. Unpublished manuscript, universityof Delaware (principal author).

Linn, R.L. (1994). Performance Assessment: PolicyPromises and Technical Measurement Standards,Educational Researcher December pp. 4-14.

Livingston, S.A. & Lewis, C. (1995). Estimating theconsistency and accuracy of classifications basedon test scores, Journal of Educational Measurement32(2).

Lundberg, G. (1991). Promoting Professionalismthrough Self Appraisal in this Critical Decade,Journal of the American Medical Association 265:2859.

McKimm, J. (2001). Using portfolios in medicaleducation. Imperial College School of Medicinepreparedness to practice project. Internal report.

Messick (1994). The interplay of evidence andconsequences in the validation of performanceassessment, Educational Research 2(2): 13-23.

Miller, G.E. (1991). The Assessment of ClinicalSkills/Competence/Performance, Academic Medicine65(9): S63-67.

Missin, L.J. & Hart, L. (1985). Oral examinations inassessing clinical competence, Neufeld, V.R.,Norman, G.R. (eds):.71-93. New York. SpringerPublishing Company.

-24 -


Moss, P.A. (1994). Can there be validity withoutreliability, Educational Researcher 23(2): 5-12.

Parboosingh, J. ((1996). Learning Portfolios:Potential to Assist Health Professionals with Self-Directed Learning, Journal of Continuing Educationin the Health Professionals 16: 75-81.

Pitts, J., Coles ,C., Thomas P. (2001). Enhancingreliability in portfolio assessment - shaping theportfolio. In press, Medical Teacher.

Schon, D. (1983). The Reflective Practitioner - HowProfessionals Think in Action. Basic Books. HarperCollins.

Schon, D. (1987). Educating the reflectivepractitioner, San Francisco, C.A., Jossey-Bass.

Scottish Dean’s Medical Curriculum Group (2000).Learning Outcomes for the Medical Undergraduatesin Scotland: A Foundation for Competent andReflective Practitioners.

Shapiro, J. & Talbot, Y. (1999). Applying the Conceptof the Reflective Practitioner to understanding andteaching Family Medicine, Family Medicine 32(6):451-457.

Snadden, D., Thomas, M., Challis, M. (1999).Portfolio-based learning and assessment, AMEEGuide No. 11.

Stecher, B. (1998). The local benefits and burdensof large-scale portfolio assessment, Assessment inEducation 5(3).

Swick, H.M. (2001). Towards a Normative Definitionof medical Professionalism, Academic Medicine75(6): 612-616.

Turnbull, J. (2000). Clinical Work Sampling, Journalof General Internal Medicine 15: 5556-5561.

Wass, V., McGibbon, D., Van Der Vleuten, C. (2001).Composite undergraduate clinical examinations: howshould the components be combined to maximisereliability? Medical Education 35, 4, 326-330

Wear, D. (1997). Professional Development ofMedical Students: Problems and Promises, AcademicMedicine 72:1062.

Wolf, A. (1998). Portfolio Assessment as NationalPolicy. The National Council for VocationalQualif ications and its quest for a pedagogicalrevolution, HSS in Education 5(3).

portfolios as a method of student assessmentctld.kau.edu.sa/files/322/files/154767_theme 4 guide 24...

Documents