rici

t

iAPSvppdj

dah

ORIGINAL REPORTS

Acquisition of Evidence-Based Surgery Skillsin Plastic Surgery Residency Training

Claire L. F. Temple, MD, MSc, FRCSC, and Douglas C. Ross, MD, FRCSC

Division of Plastic Surgery, University of Western Ontario, Schulich School of Medicine and Dentistry, London,

Ontario, Canada

a1P

INTRODUCTION: The teaching and learning of critical ap-praisal skills and evidence-based practices by surgical residentshas been identified as an unmet need in many surgical trainingprograms.

METHODS: Monthly journal clubs over a calendar year werethe setting for a critical appraisal curriculum. Preassignedhomework assignments and carefully selected articles with spe-cific methodologies were posted electronically and formed thecourse material. Pretests and posttests on medical statisticsand methodology were administered. Presurveys and post-surveys on attitudes toward evidence-based surgery (EBS)were administered.

RESULTS: Precourse surveys revealed a lack of confidence inesidents’ knowledge of epidemiology and biostatistics, with anncrease in confidence postcourse (2.6 vs 2.9; p � 0.4). Pre-ourse and postcourse, there was strong support for more crit-cal appraisal training in residency (5.1 vs. 4.8; p � 0.1) and an

agreement that understanding evidence-based practices is im-portant for the clinical practice (4.6 vs. 4.6; p � 0.4) as well ashe research endeavors of a plastic surgeon (5.4 vs. 5.5; p � 0.8).

Pretest scores, when compared with PGY level, showed anncrease in knowledge with increasing PGY level (p � 0.6).verage pretest scores were 6.5 of a total of 15 points, or 43%.osttest scores were improved, at 7.8 of 15, or 52% (p � 0.6).ixty-four percent of learners felt that journal club was a goodenue for teaching critical appraisal skills precurriculum. Fiftyercent of learners were still of that impression at course com-letion (p � 0.3). The modest improvement in test scores in-icates an impact on critical appraisal skills, but reliance on

ournal clubs to teach these skills is insufficient.

CONCLUSIONS: Through monthly journal clubs and self-irected assignments, critical appraisal skills were improvedcross PGY levels in an academic surgical training program;owever, other settings and methods of teaching are required to

Correspondence: Inquiries to Claire Temple, MD, MSc, FRCSC, Division of Plastic Sur-gery, University of Western, Ontario, Schulich School of Medicine and Dentistry, 268

Grosvenor Street, London, Ontario, Canada; fax: (519) 646-6049; e-mail:ctemple4@uwo.ca

Journal of Surgical Education • © 2011 Association of Program DirecPublished by Elsevier Inc. All rights res

ugment a curriculum in evidence-based surgery. (J Surg 68:67-171. © 2011 Association of Program Directors in Surgery.ublished by Elsevier Inc. All rights reserved.)

KEY WORDS: evidence-based medicine, critical appraisal,plastic surgery, Journal Club

COMPETENCIES: Patient Care, Medical Knowledge, PracticeBased Learning and Improvement

INTRODUCTION

Evidence-based surgery (EBS) involves integrating clinical ex-pertise and the best available clinical evidence from systematicresearch to make sound, informed surgical decisions.1 The “Of-fice of Evidence-Based Surgery,” established within the Amer-ican College of Surgeons, perceives that “most surgical special-ists base their practice on uncontrolled case series anduncontested expert opinion.”2

To alter this behavior, ideally students would be introducedto evidence-based practices in medical school.3 Unfortunately,even when exposed to critical appraisal principles, many medi-cal school graduates feel that their clinical practice is not basedon the best available evidence.4

Thus, often it falls on the shoulders of the academic surgeonto train residents in EBS. This is challenging as few surgeonsthemselves are trained in EBS. From a systematic review of theeffectiveness of critical appraisal skills training for clinicians,Taylor et al.5 concluded that there is a paucity of educatorswithin the field of evidence-based health.

A second challenge is to find the right venue for instructionof these skills. Surgical residents are clinically busy, and aca-demic teaching time is squeezed by a continuously increasingvolume of knowledge coupled with a mandated decrease inwork hours. Yet, critical appraisal cannot just be absorbed—these skills need to be taught explicitly and then reinforcedduring the care of specific patients. The weekly formal lectureseries is one venue to integrate acquisition of these skills; how-ever, dry lectures may not produce lasting retention. Self-di-rected learning alone may not be realistic for the busy surgical

resident. “One-off” half-day clinical courses in critical appraisal

tors in Surgery 1931-7204/$30.00erved. doi:10.1016/j.jsurg.2010.12.004

167

ni

sTf

4e0in

yfmmws

have not been particularly effective.6 Integrating EBS into jour-al clubs is an acceptable way to begin to integrate formal teach-

ng of critical appraisal skills into residency training.7

As a first step to tackling this deficiency in our program, weelected to introduce formal teaching into the University ofWestern Ontario Plastic Surgery Residency Training Programvia a combination of journal club sessions and self-directedlearning assignments. Our hypothesis was that “doing” EBS inan integrated fashion would allow it to be done efficiently andwould facilitate its routine use in daily practice.

METHODS

A formal program for teaching EBS was run over one academicyear. Materials were provided to the learners on a password-protected website, including links to the “Users Guides to theSurgical Literature” series from the Canadian Journal ofSurgery.8-17 Three homework assignments were posted, withtopics covering randomized controlled trials (RCTs), meta-analyses, and diagnostic tests. Carefully selected journal articleswith methodologies matching the homework assignments weredelivered electronically.

Formal teaching was incorporated into monthly, 3-hour eve-ning journal clubs. At the first journal club, residents were askedto list any courses they had taken in epidemiology and biosta-tistics. A short questionnaire sampling attitudes toward EBSwas administered. A 6-point Likert scale was used, with re-sponses ranging from “strongly disagree” to “strongly agree.”Domains sampled included the student confidence in their EBSskills, the need for better teaching of EBS in residency, theusefulness of EBS skills in clinical and academic surgery, andthe preferred forum for acquiring these skills. Finally, residentscompleted a pretest covering basic topics in EBS. The test wasproctored to promote independent test taking among the resi-dents. No time limit was set on this examination.

At alternating journal clubs, a theme for the journal club (ie,RCTs) was selected, and a carefully selected journal article il-lustrative of that topic was assigned. In addition, a correspond-ing homework assignment on that topic was completed by eachresident and brought to the journal club. The learners wereencouraged to access multiple resources for this assignment,including textbooks, Internet resources, and the applicable us-er’s guide. An in-depth discussion on the methodology of thearticle was undertaken, and the homework assignment was re-viewed. Additional articles were also reviewed as per the con-sultant hosting the journal club that evening.

At the final journal club in the year-long program, a posttest,identical to the pretest, was administered. Likewise, an identicalsurvey on attitudes toward EBS was given. Each pretest andposttest was then anonymized for features, including name(which was optional), PGY level, and whether the test was thepretest or posttest. Examinations were then graded using a stan-dardized answer key by a single surgeon (C.T.).

Precourse and postcourse attitudes and pretest and posttest

scores were compared with paired t-tests. Linear regression was 0

168 Journal o

used to assess PGY level and test score results. A p-value of lessthan 0.05 was considered significant. Analyses were completedusing InStat 3, version 3.0 b (In-STAT Inc, Scottsdale, Ari-zona) for Macintosh.

RESULTS

Ten plastic surgery residents in the University of Western On-tario plastic surgery training program participated. There weretwo each of PGY 1, 2, and 3 residents, three PGY 4 residents,and one PGY 5 resident. Six had had prior undergraduate ormedical school courses in epidemiology or biostatistics. Fourparticipants had had none.

Eight journal club sessions were held over a single academiccalendar year. Of the 10 residents, on average 8 were present ateach session. No resident consistently missed the sessions. Twowere invariably away either on vacation or were busy on call. Onaverage, 4 consultant surgeons attended each session.

Although no time limit was set for the examination, all learn-ers had completed both the survey on attitudes and the test by20 minutes. Of the 10 residents, 7 had complete results, includ-ing a pretest and posttest. Two had a pretest only, and 1 had aposttest only.

Regarding precourse perceptions, most participants lackedconfidence in their knowledge of epidemiology and biostatis-tics, scoring 2.6 on a 6-point Likert scale with “poor” confi-dence a 1 and “high” confidence a 6. After the course, there wasa nonsignificant increase in confidence (2.6 vs 2.9; p � 0.4).

Precourse, there was strong agreement that there should bemore critical appraisal training in residency, scoring 5.1 on a6-point scale of agreement. After the course, there was a similarsentiment (5.1 vs 4.8; p � 0.1).

Precourse and postcourse, there was strong endorsement thatunderstanding critical appraisal is important for the clinicalpractice of a plastic surgeon (4.6 vs. 4.6; p � 0.4) and for aplastic surgeon scientist (5.4 vs. 5.5; p � 0.8).

Pretest scores, when regressed on PGY level, showed no as-ociation (p � 0.6) in knowledge with increasing PGY level.his would indicate that these skills are not being learned in-

ormally within the existing training program curriculum.Average pretest scores were 6.5 of a total of 15 points, or

3%. Posttest scores were improved at 7.8 of 15, or 52%; how-ver, this difference did not reach statistical significance (p �.6). This might represent beta error, in that there may be annsufficient number of subjects to reject the null hypothesis ofo difference in scores.Qualitative impressions from informal feedback from the

ear-long course included a widespread appreciation of the ef-ort that went into the program. There was moderate resent-ent for the time required to complete the self-directed assign-ents. Although initially 64% of students felt that journal clubas a good venue for teaching critical appraisal skills, 50% were

till of that impression upon completion of the curriculum (p �

.3).

f Surgical Education • Volume 68/Number 3 • May/June 2011

ite

aktcgateo

gEgwedaloIcaaawrttcgiet

acfia

DISCUSSION

EBS is a critical part of daily clinical practice in surgery. Thesurgeon must integrate the best-available literature with patientpreferences to develop a treatment plan.1 Inspiring learners tosee the practical importance of incorporating evidence-basedprinciples in patient management is, in and of itself, an impor-tant training goal.

The clinic and operating room settings are useful to in-spire questions. Although self-directed learning with the ap-propriate tools and reading guides to assist in interpretingvarious study designs could be used, specific small groupteaching in a clearly formatted and structured curriculumdedicated to teaching quantitative methods has been shownto be most effective.18 Short-burst educational interventionswhere clinicians attend a single, half-day critical appraisalcourse have not been an effective method of enhancing EBMskills.5 Although traditional journal clubs have been criti-cized for focusing on critical appraisal skills while underval-uing other areas, such as biostatistics,19 Ebbert et al.7 con-cluded that journal clubs are a useful way to improveunderstanding of epidemiology and biostatistics, readinghabits, and the use of medical literature in practice.

Our combination of integrating journal club with prepara-tory self-directed assignments was meant to optimize teachingcritical appraisal skills. Interestingly, however, only 50% of thelearners felt that journal club was the ideal venue for EBS learn-ing at the completion of the curriculum. Although it has beenpostulated that the teaching and learning of EBM is most effec-tive when contextually based in clinical practice (vs didacticlectures or as part of traditional journal clubs), the evidence forthis is lacking.20,21 Perhaps other suitable locations for acquir-ng these skills include the weekly, didactic plastic surgeryeaching sessions or the principles of surgery lecture series forarly year surgical residents.

There is a need for improvement in critical appraisal skillsmongst surgical residents and consultants. This gap innowledge was evident in the poor performance on the pre-est in our study. Although the examination tested basiconcepts in medical statistics and research methodology, theroup scored just 43%. The pretest survey of attitudesmong the resident group demonstrated an appreciation ofhe importance of more EBS training as well as an acknowl-dgment of their own deficit in EBS knowledge. In a surveyf senior otolaryngology residents, Amin et al.22 also found a

welcoming attitude to evidence-based medicine and an ap-preciation of the need to teach these skills formally. Thus,surgical trainees understand the importance of EBS skillsand are willing to “work” to enhance those skills.

A reasonable question to ask is why, despite a year-longhorizontal program to enhance EBS knowledge, the posttestscores only increased to 52% from 43%? Similar findingswere noted in a meta-analysis on effectiveness of instructionin critical appraisal, in which residents showed a small

change in knowledge (mean gain: 1.3%; standard deviation

Journal of Surgical Education • Volume 68/Number 3 • May/June 20

[SD]: 1.7%), even less than medical students (mean gain:17.0%; SD: 4.0%).23 It is possible that substantial gains inEBS knowledge might not be detected by our posttest exam-inations because of a lack of valid assessment of critical ap-praisal skills.24

Multiple obstacles are likely to inhibit the effective teach-ing of EBS. In a qualitative study using grounded theory,Bhandari et al.25 identified multiple barriers that limited theresidents’ ability to apply EBS to their care of patients. Someof these included lack of ready access to surgical EBS re-sources, lack of education in EBS, time constraints, lack ofpriority, and fear of staff disapproval. Suggestions for im-provement included hiring staff surgeons with EBS training,offering coursework in EBS to surgical staff, improving in-terdepartmental communication, and providing greater flex-ibility for EBS training.

An intriguing study by Kitto et al.26 suggested that “sur-ical cultures” played a significant role in the learning ofBS. Using qualitative methods, the authors found that sur-eons who demonstrated both clinician and scientist rolesithin their practices were more likely to be effective teach-

rs of EBS within the traditional residency structure. Evi-ence to support an integrated teaching model of “scientist”nd “clinician” for EBS was reported by Haines and Nicho-as,27 who described an intensive program in the principlesf EBS incorporated into a neurosurgical training program.n place of the usual 2-hour biweekly professors’ rounds,ase-based sessions were led jointly by both a neurosurgeonnd an epidemiologist. Residents were assisted in developingppropriate questions, searching the literature, critically an-lyzing the papers, and summarizing the results. Six topicsere analyzed in the first 2 years of the course, with results

anging from finding insufficient evidence to formulatereatment recommendations to identifying solid literaturehat permitted the development of criteria for cervical spinelearance that was incorporated into the institution’s emer-ency department. The resources used were already in placen the institution, and the addition of specific time andpidemiologic expertise was all that was required to makehis program successful.

The 4 steps required to practice EBS include (1) formulatequestion based on a clinical situation, (2) conduct a fo-

used literature search, (3) critically appraise the literature tond the best evidence, and (4) integrate the information andct in accordance with the best available evidence.28 We have

been focusing primarily on the (3), critical appraisal. Ourprogram did include framing a question into Pico format29

but did not incorporate formal instruction on literaturesearch. Rather, trainees focused on appraisal of an assignedarticle. Via mandatory preassignments, the methodology ofcertain study designs was learned. As we expand the pro-gram, we will add focus on literature search skills. Securing

the support of an expert medical librarian will be critical.

11 169

CONCLUSIONS

Through monthly journal clubs and self-directed assignments,critical appraisal skills were improved across PGY levels in anacademic surgical training program. Knowledge deficiencieswere identified and were improved modestly by the curriculum.This pilot study suggested other techniques and areas to expandthe curriculum, including incorporating some didactic lecturesin medical statistics as well as how to do a literature search. Acommitment to critical appraisal teaching is an ongoing com-ponent of teaching core competencies to residents. The ulti-mate goal is to make our residents not only good surgeons butalso good doctors, and this goal will be enhanced by an ability tobring the most up to date and accurate information on how totreat their patients.

REFERENCES

1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Rich-ardson WS. Evidence-based medicine: what it is and whatit isn’t. BMJ. 1996;312:71-72.

2. Toedtler LJ, Thompson LL, Rohatgi C. Training sur-geons to do evidence-based surgery: a collaborative ap-proach. J Am Coll Surg. 2004;199:293-299.

3. Rubin P, Franchi-Christoper D. New edition of tomor-row’s doctors. Med Teach. 2002;24:368-369.

4. Doran T, Maudsley G, Zakhour H. Time to think? Ques-tionnaire survey of pre-registration house officers’ experi-ences of critical appraisal in the Mersey Deanery. MedEduc. 2007;41:487-494.

5. Taylor R, Reeves B, Ewings P, et al. A systematic review ofthe effectiveness of critical appraisal skills training for cli-nicians. Med Educ. 2000;34:120-125.

6. Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Criticalappraisal skills training for health care professionals: a ran-domized controlled trial. BMC Med Educ. 2004;4:30.

7. Ebbert JO, Montori VM, Schultz HJ. The journal club inpoastgraduate medical education: as systematic review.Med Teach. 2001;23:455-461.

8. Archibald S, Bhandair M, Thoma A. Users’ guides to thesurgical literature: how to use an article about a diagnostictest. Can J Surg. 2001;44:17-23.

9. Thoma A, Sprague S, Tandan V. Users’ guides to thesurgical literature: how to use an article of economic anal-ysis. Can J Surg. 2001;44:347-354.

10. Urschel JD, Goldsmith CH, Tandan VR, Miller JD. Users’guides to the surgical literature: how to use an article evalu-ating surgical interventions. Can J Surg. 2001;44:95-100.

11. Hong D, Tandan VR, Goldsmith CH, Smunovic M. Users’

guides to the surgical literature: how to use an article report-

170 Journal o

ing population-based volume-outcome relationships insurgery. Can J Surg. 2002;45:109-115.

12. Birch DW, Eady A, Robertson D, Pauw SD, Tandan V.Users’ guides to the surgical literature: how to perform aliterature search. Can J Surg. 2003;43:136-141.

13. Bhandari M, Devereaux PJ, Montori V, Cina C, TandonV, Guyatt GH. Users’ guides to the surgical literature:how to use a systematic literature review and meta-analy-sis. Can J Surg. 2004;46:60-67.

14. Thoma A, Farrokhyar F, Bhandari M, Tandan V. Users’guides to the surgical literature: how to assess a randomizedcontrolled trial in surgery. Can J Surg. 2004;47:200-208.

15. Birch DW, Goldsmith CH, Tandan V. Users’ guides tothe surgical literature: self-audit and practice appraisal forsurgeons. Can J Surg. 2005;48:57-62.

16. Mastracci TM, Thoma A, Farrokhyar F, Tandan VR,Cina CS. Users’ guides to the surgical literature: how touse a decision analysis. Can J Surg. 2007;50:403-409.

17. Thoma A, Carnacchi SD, Lovrics PJ, Goldsmith CH. Users’guides to thesurgical literature:howtoassessanarticleonhealth-related quality of life. Can J Surg. 2008;51:215-224.

18. Fingerhut A, Borie R, Chadii D. How to teach evidence-based surgery. World J Surg. 2005;29:592-595.

19. Hatala R, Keitz SA, Wilson MC, Guyatt G. Beyond journalclubs. Moving toward an integrated evidence-based medi-cine curriculum. J Gen Intern Med. 2006;21:538-541.

20. Khan KS, Coomarasamy A. A hierarchy of effective teach-ing and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006;15:59.

21. Kulier R, Coppus SF, Zamora J, et al. The effectiveness ofa clinically integrated e-learning course in evidence-basedmedicine: a cluster randomised controlled trial. BMC MedEduc. 2009;12:21.

22. Amin M, Saunders JA, Fenton JE. Pilot study of theknowledge and attitude towards evidence based medicineof otolaryngology higher surgical trainees. Clin Otolaryn-gol. 2007;32:133-135.

23. Norman GR, Shannon SI. Effectiveness of instruction incritical appraisal (evidence-based medicine) skills: a criti-cal appraisal. CMAJ. 1998;158:177-181.

24. MacRae HM, Regehr G, Brenneman F, McKenzie M,McLeod RS. Assessment of critical appraisal skills. Am JSurg. 2004;187:120-123.

25. Bhandari M, Montor V, Devereaux PJ, et al. Challengesto the practice of evidence-based medicine during resi-dents’ surgical training: a qualitative study using grounded

theory. Acad Med. 2003;78:1183-1190.

f Surgical Education • Volume 68/Number 3 • May/June 2011

26. Kitto S, Petrovic A, Gruen RL, Smith JA. Evidence-based med-icine training and implementation in surgery: the role of surgicalcultures. J Eval Clin Pract. 2010; e-pub ahead of print.

27. Haines SJ, Nicholas JS. Teaching evidence-based medi-cine to surgical subspecialty residents. J Am Coll Surg.

2003;197:285-289.

Journal of Surgical Education • Volume 68/Number 3 • May/June 20

28. Sackett DL, Straus SE, Richardson WS, et al. EvidenceBased Medicine. How to Practice and Teach EBM. 2ndedition. Edinburgh: Churchill Livingstone; 2000.

29. Richardson WS, Wilson MC, Nishikawa J, Hayward RS.The well-built clinical question: a key to evidence-based

decisions. ACP J Club. 1995;123:A12-A13.

11 171