Patient feedback formedical studentsOliver Lyons, Helen Willcock and John Rees, King’s College London School of Medicine,Guy’s Hospital, London, UKJulian Archer, Paediatrics & Medical Education, Peninsula Medical School, Plymouth, UK

INTRODUCTION

Medical students unskilledin professional interac-tion with patients are

likely to lack the meta-cognitiveability to assess themselvesaccurately, and identify ways inwhich they can improve.1 Thereis a growing feeling that profes-sionalism should be activelytaught and assessed as part ofthe explicit undergraduatecurriculum. Multi-source feedback(MSF) has been advocated as avalid way of assessing profes-sional behaviours in undergradu-ates.2,3 It has the potential toraise awareness amongststudents of expected communi-cation skills standards, help

students to identify specificmutable behaviours and fosterreflection on students’ potentialfor improvement. MSF is nowused extensively in postgraduateeducation and appraisal3–5 inthe UK, but there has beenrelatively little published workutilising MSF for undergraduatemedical students.2,6 Some com-ponents of undergraduate MSFhave been developed,2,7 but weare not aware of any patientassessment tool for specific usewith undergraduates. As receiv-ing feedback does not necessarilyimprove results, and indeed mayworsen performance,8 feedbackinterventions require carefulevaluation. We set out to devel-op and then test a patient

assessment tool (with a focus onprofessional behaviour) for useby medical students after inter-action with hospital patients.

IMPLEMENTING PATIENTFEEDBACK AT KING’SCOLLEGE LONDON SCHOOLOF MEDICINE

We developed a questionnaire(Box 1) that was directly mappedto core domains of the GeneralMedical Council’s Good MedicalPractice, Medical students:professional behaviour and fitnessto practise and Tomorrow’sDoctors.9–11 Patients rate studentsusing a six-item Likert scale (theresponses available were from 1,

Multi-sourcefeedback (MSF)

has beenadvocated as a

valid way ofassessing

professionalbehaviours in

undergraduates

Thepatientvoice

254 � Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 254–258

completely disagree to 6,completely agree) in 11 areasrelated to communication andprofessional behaviour following apatient–student interaction.Patients were asked their genderand the number of times that theyhad previously seen the student.In addition, patients could makeany comments they wished in afree text box. No patient-identi-fiable information was collected.

The instrument was initiallypiloted with 40 students. A totalof 210 undergraduate medicalstudents in their first year ofclinical attachments, during a13-week placement in generalmedicine or surgery in one of fourLondon teaching hospitals, weresubsequently assessed. Studentswere each given the new feedbackforms with pre-addressed enve-lopes. Students were asked to givea form and envelope to 10patients following a student–patient interaction, for exampleeliciting a history or performingan examination. They were in-structed to leave patients to fill inthe form on their own withoutsupervision. Patients placed thecompleted form in the envelope,and gave the sealed envelope totheir nurse who posted it to theauthors. Form data were enteredinto spreadsheets and analysedwith SPSS 14.0. The number ofstudents and patients participat-ing were calculated, and aggre-gate scores at the level of theform were used to calculate meanscores per student, and for thecohort. Cronbach’s alpha wasestimated as a measure of internalconsistency. Item performancewas explored using ‘unable tocomment’ counts, and explorationof extreme outlier scores. Imple-menting assessment in the fieldpredicates a naturalistic fullynested design, and so patientswere nested within students. Themodel allowed the estimation oftwo variance components: true(attributable to the student) andresidual using VARCOMP (mini-mum norm quadratic unbiased

estimation, MINQUE, procedure)in SPSS.12 A decision study wasthen undertaken using the calcu-lated variance components.

At the start of this exercise210 students were emailed a linkto an online questionnaire(Box 2). Students used a Likertscale (the responses availablewere from 1, completely disagree,to 6, completely agree) to ratetheir agreement to four state-ments about their expectedchange in professional behaviourafter receiving feedback from dif-ferent sources: patients, them-selves, clinicians on the wards

and other medical students. Theywere then asked whether patientscan give important feedback, andwhether this feedback is moreimportant than the feedback fromclinicians. Lastly they were askedif they would be happy for patientfeedback to be used (with otherinformation) to formally assesstheir communication skills.

The students’ patient feedbackscores were compared with theirpass rate in an Integrated ClinicalExamination (ICE) consisting ofnine objective structured clinicalexam (OSCE) stations, spreadthroughout the year.

Box 1. The statements used in the patient feedbackquestionnaire

1. The student explained what they wanted to do

2. The student asked your permission to speak to ⁄ examine you

3. The student treated you politely ⁄ considerately

4. The student was professionally dressed

5. The student understood you

6. The student attempted to answer any questions you have about yourcondition ⁄ problem

7. The student was open and honest with you

8. Your views were listened to

9. The student was interested in your point of view

10. It was clear that the person who gave you this form was a medicalstudent ⁄ trainee

11. The student listened to you carefully

Box 2. Online questionnaire assessment of student’sopinions on the feedback from different sources

1. I would alter my professional behaviour after receiving feedback frompatients

2. I would alter my professional behaviour having performed aself-assessment of my communication skills

3. I would alter my professional behaviour after receiving anonymousfeedback from clinical colleagues on the wards

4. I would alter my professional behaviour after receiving anonymousfeedback from other medical students

5. I think that patients can give important feedback to medical students

6. I think that feedback from patients is more important than feedbackfrom clinical colleagues

7. I would be happy for patient feedback (with other information) to beused to formally assess the communication skills of medical students

Students wereasked to give aform andenvelope to 10patients,following astudent–patientinteraction

� Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 254–258 255

OUTCOMES FROM PATIENTFEEDBACK

Patient feedbackquestionnairesA total of 88 students took part,35 per cent of the 250 asked toparticipate, and 315 patientfeedback questionnaires werereturned. On average, 3.6 formswere returned for each student.The overall mean score for thecohort was 5.8 on the 6-point

scale (with a range of 4.7–6.0),with a negative skew. Reliabilitywas estimated using Cronbach’salpha, and was 0.68. Exploringthe item-by-item performancerevealed that question 6 (‘Thestudent attempted to answer anyquestions you have about yourcondition ⁄ problem’) was notcompleted 18 per cent of the time(with 48 ‘unable to comment’responses), and generated 18extremely low scores, compared

with an average of five ‘unableto comment’ responses andvery few extremely low scoresfor the other items. Excludingquestion 6, Cronbach’s alphawas 0.77.

Variance component analysiswas able to calculate the varianceattributable to the student(true = 0.005) and the varianceattributable to all other factors(error = 0.58). Using theseestimates, generalisability (G)theory enables the calculation ofthe number of patients requiredper student to achieve increasinglevels of reliability. This isshown in the decision study inTable 1.

Neither the gender of thepatient (t = 0.18, df = 270,p > 0.05) nor having seen thestudent before (df = 3, F = 1.17,p > 0.05) made a significantdifference to the score awarded bythe patient to the student.

In total, 103 patients (33%)made comments in the free-textbox, which were generally verypositive, but which also offeredconstructive criticism (Box 3).

Students’ perceptions of theirlikelihood of behaviouralchange with patient feedbackIn total 87 students (41% of210 students) completed theonline questionnaire at the startof the patient feedback exercise.On average, students thoughttheir professional behaviourwould change after receiving allfour forms of feedback (Fig-ure 1), but agreed most stronglyfor patient feedback (meanscore, 5.36; 95% confidenceinterval, 5.23–5.49). This wasnot significantly different tofeedback from clinicians (meanscore, 5.29; 95% confidenceinterval, 5.12–5.46; p > 0.05).Students were less positive(mean score, 3.73; 95% confi-dence interval, 3.37–4.09) aboutpatient feedback being used aspart of the formal assessment

Table 1. Decision study: the number ofpatients required to produce a given reliability

DecisionNumber ofpatients required

0.35 6

0.45 10

0.55 14

0.65 21

0.75 34

0.85 63

Box 3. Examples of patient’s comments on students

The following are a representative sample of 10 per cent of the 103comments made by patients about students in their feedbackquestionnaire.

• ‘Totally professional. Extremely polite, showed genuine interest.’

• ‘Stay open minded as you are, patients appreciate that.’

• ‘Very pleasant and because I can’t hear did her very best to make meunderstand. Thank you.’

• ‘Student explained what she was going to do. I also like to know(in nice simple language) why she was doing it’

• ‘Needs to engage more confidently with the ‘‘patient’’ rather than the‘‘task’’. Attitude and approach much better extracting information andrecording patient’s history than during physical examination.’

• ‘Possibly struggling a little with ‘‘isolation’’. Needs to act moreconfidently.’

• ‘The student was very nervous, but kept a clear focus on the task in hand.’

• ‘The student approached me professionally. The student showed aninterest in listening and understanding me. The student was polite,considerate, honest and well dressed. Well done.’

• ‘I found this student was very professional, very understanding, verypolite.’

• ‘She treated me with respect, I felt as though she listened carefully tomy story and was genuinely concerned. She had a very good bedsidemanner.’

Students wereless positive ...

about patientfeedback beingused as part of

the formalassessment of

communicationskills

256 � Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 254–258

of communication skills(Figure 2).

Comparison with ICE (OSCE)Students that participated in thispatient feedback study were morelikely to pass all stations in theirICE exam (50%), compared withthose that did not participate(20%, p < 0.01, v2). Of 161 stu-dents in the same year not offeredpatient feedback (i.e. notincluded in this study as theywere not in chosen generalmedical and surgical placements),39 per cent passed all stations inthe ICE exam.

WHAT THIS RESEARCHADDS

Professionalism has been vari-ously defined, and so we selectedbehaviours that the GeneralMedical Council expect to beevident upon qualification,and that could reasonably beassessed by patients. Theinstrument appeared to functionwell, but large numbers ofpatients would be needed to givefeedback in order to produce

results reliable enough for use insummative assessment. Thisstudy is limited by the lownumber (35%) of students thatparticipated, and the low number(3.6) of forms returned perstudent. Poor participation maybe related to the new andformative nature of this work.Unfortunately, we do not havedata on the number of formsgiven to patients by students, soit is impossible to differentiate alack of forms given out bystudents (perhaps related to

uncertainty with this new form ofassessment) from an unwilling-ness of patients to completeforms, or from loss of forms ‘inthe post’. Informally, theinformation collected suggeststhat all three factors contributedto the poor response. We also donot have data to demonstrate thehonesty of our students in thisassessment process.

Patients tended to award highscores to most participatingstudents. Comparison with theICE results of students (Figure 3)suggests it was the betterstudents that participated.Non-participation in such volun-tary assessments linked to pro-fessionalism may help to identifystudents struggling with theirmotivation or performance.13 Itis likely that if this assessmentwere mandatory the participationof the non-participants thatperformed less well in their ICEexam may increase the distribu-tion of patient-feedback scores.Positive feedback and ‘optimism’may build confidence, and pro-duce improved performance, butawarding high scores to poorperformers could produce falseconfidence or reinforce undesir-able professional behaviours. Inthe right conditions, feedbackfrom patients could provide adriver for positive change inbehaviour, especially as studentsfeel that this feedback is impor-

0 1 2 3 4 5 6

...after receiving feedback from

patients"

...having performed a self assessment of my communication

skills"

...after receiving anonymous feedback

from clinical colleagues on the

wards"

...after receiving anonymous feedback from other medical

students"

Likert Scale ScoreMean +/-95%CI

Perceived likelihood ofbehavioural change

“I would alter my professional behaviour….

Figure 1. Online questionnaire: change instudents’ behaviour.

Patients can give important feedback to

medical students

Feedback from patients is more important than feedback from clinical

colleagues

I would be happy for patient feedback (with other information) to be used to formally

assess the communication skills of

medical students

Likert Scale ScoreMean +/-95%CI

Students’ Opinions

0 1 2 3 4 5 6

Figure 2. Online questionnaire: students’opinions.

In the rightconditions,feedback frompatients couldprovide a driverfor positivechange inbehaviour

Figure 3. Students’ performance in the Integrated Clinical Examination. Of those offered inclusion

in this study, participants performed better than non-participants (p < 0.01, v2).

� Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 254–258 257

tant and is likely to producechange, although there is noevidence that students benefitsimply from being given theinformation that patients findthem ‘nervous’ or ‘struggling withisolation’ (Box 3). The process ofseeking feedback from patientsmay encourage students to thinkmore about patients’ views, andto engage positively and profes-sionally with patients.

Feedback is best given by atrained supervisor that has long-term contact with the student,who can engage with the studentin producing a management planfor behavioural change and reviewthe outcome over time. Whereashigh-quality feedback is requiredto inform and structure discus-sion, it is likely that the ongoingrelationship between student andmentor actually facilitateschange.14 Goodstone describesthe critical elements of thisrelationship.15 These include anon-judgmental approach, asetting focused on personalgrowth, the presence of empathyand a model of change that earnsthe respect of the student. Wehave yet to demonstrate thatstudents who receive feedbackfrom patients go on to changetheir behaviour and become moreprofessional.

The students that answeredour online questionnairestrongly agreed that patient

feedback will alter theirprofessional behaviour, and find itan acceptable form of formativeassessment, comparable withfeedback from doctors. They werefar less accepting of its use insummative feedback. We did notseek to restrict students’ inter-pretation of ‘professionalbehaviour’, which may differbetween individuals.

CONCLUSION

Anonymised patient feedback onencounters with students isfeasible, being inexpensive andeasy to organise. Participatingstudents thought it both accept-able and likely to change theirbehaviour. The numbers of pa-tients needed to reach high levelsof reliability may prohibit its usein the summative assessment ofundergraduates. Feedback frompatients could support the devel-opment of a culture of profes-sionalism (teaching, learning andassessing) within the undergrad-uate curriculum.

REFERENCES

1. Kruger J, Dunning D. Unskilled and

unaware of it: how difficulties

in recognizing one’s own incompe-

tence lead to inflated self-assess-

ments. J Pers Soc Psychol 1999;77:

1121–134.

2. Arnold L, Willoughby L, Calkins V,

et al. Use of peer evaluation in the

assessment of medical students.

J Med Educ 1981;56:35–42.

3. Archer JC, Norcini J, Southgate L,

et al. mini-PAT (Peer Assessment

Tool): a valid component of a

national assessment programme in

the UK? Adv Health Sci Educ

2008;13:181–192.

4. Crossley J, Davies H, McDonnell J,

et al. A district hospital assessing its

doctors for re-licensure: can it work?

Med Educ 2008;42:359–363.

5. Chisolm A, Askham J. What do you

think of your doctor. A review of

questionnaires for gathering patients’

feedback on their doctor. Oxford, UK:

Picker Institute Europe; 2006.

6. Rees C, Shepherd M. The acceptabi-

lity of 360-degree judgements as a

method of assessing undergraduate

medical students’ personal and

professional behaviours. Med Educ

2005;39:49–57.

7. Owen C, Ramsey W. Peer review

by students. Clin Teach 2007;

4:31–35.

8. Kluger AN, DeNisi AS. The effects of

feedback interventions on

performance: A historical review, a

meta-analysis and a preliminary

feedback intervention theory.

Psychol Bull 1996;119:254–284.

9. Good Medical Practice. The General

Medical Council, London 2006.

10. Medical students: professional

behaviour and fitness to practise. The

General Medical Council, London

2007.

11. Tomorrow’s Doctors. The General

Medical Council, London 2003.

12. Giesbrecht FG. An efficient proce-

dure for computing MINQUE of vari-

ance components and generalized

least squares estimates of fixed

effects. Communications in Statistics,

Part A - Theory and Methods 1983;

12:2169–2177.

13. Wright N, Tanner MS. Medical

students’ compliance with simple

administrative tasks and success in

final examinations: retrospective

cohort study. BMJ 2002;324:

1554–1555.

14. Huggett M. 360-degree feedback -

great expectations? Industrial

and Commercial Training 1998;30:

128–130.

15. Goodstone MS, Diamante T. Organi-

zational use of therapeutic change:

Strengthening multisource feedback

systems through interdisciplinary

coaching. Consulting Psychology

Journal: Practice and Research 1998;

50:153–163.

Feedback frompatients could

support thedevelopment

of a culture ofprofessiona-

lism ... withinthe

undergraduatecurriculum

258 � Blackwell Publishing Ltd 2009. THE CLINICAL TEACHER 2009; 6: 254–258