Teaching critical appraisal and statistics in anesthesiajournal club

R.S. MOHARARI1, E. RAHIMI1, A. NAJAFI1, P. KHASHAYAR2,M.R. KHAJAVI1 and A.P. MEYSAMIE3

From the 1Department of Anesthesiology, 2Research and Development Center and 3Department of

Community Medicine, Tehran University of Medical Sciences, Sina Hospital, Tehran, Iran

Received 21 April 2008 and in revised form 9 September 2008

Summary

Background: This study was designed to assessthe effectiveness of a series of journal clubs heldfor anesthesiology residents in promoting theirawareness of research methods and statisticalanalysis, as well as their skills in critical thinkingand appraisal.Material and methods: Twenty-four journal clubsessions were held between September 2006 andAugust 2007 for 16 residents of anesthesiology.A 31 multiple-choice question (MCQ) was taken aspretest and posttest to evaluate the participants’level of awareness in research methodology andstatistical analysis. Their competence in criticalthinking and appraisal was also evaluated by

evaluating a randomized controlled trial paperusing the CONSORT checklist before and after thecourse.Results: Residents’ awareness in the application ofinformation improved (P = 0.012), as well asresearch methodology (combined study design andapplication of information, P = 0.017). Their abilityin critical appraisal did also significantly rise at theend of the course (P< 0.001).Conclusion: Journal clubs can enable residents todevelop the knowledge, expertise and enthusiasmneeded to undertake research plans and can alsoenhance their ability in critical thinking andscientific reading.

Introduction

Residency training represents an ideal time when

clinicians may learn and practice the necessary

skills for searching, evaluating and applying medical

knowledge. The journal club may play an important

role in this regard.1 The present study was designed

to assess the effectiveness of a regular journal club

in teaching anaesthetic trainees research methodol-

ogy and critical appraisal skills.

Material and methods

Twenty-four journal-club sessions (biweekly for 1 h)

were held between September 2006 and August

2007 for residents of anesthesiology (years 1–3).

Attendance was closely monitored. In each session,a single article relevant to anesthesiology and

critical care medicine was discussed with respectto statistical analyses and research methodology(type of study, biases and methods). Suitable articles

were identified by Medline and EMBase Englishsearches and were given to the participants 4 days

beforehand. At the beginning of each session, asenior resident presented a 10-min summary of thearticle including its objectives and findings. Then, a

senior clinician focused in detail on a singlemethodological issue according to a prepared

syllabus (principles of statistics and research; con-cepts of risk, exposure, odds and bias; blockrandomization; case–control studies, cohort studies,

Address correspondence to M.R. Khajavi, MD, Sina Hospital, Tehran-11367-46911, Iran.email: mohammadreza.khajavi@gmail.com

! The Author 2008. Published by Oxford University Press on behalf of the Association of Physicians.All rights reserved. For Permissions, please email: journals.permissions@oxfordjournals.org Page 1 of 3

Q J Meddoi:10.1093/qjmed/hcn131

QJM Advance Access published October 8, 2008

randomized controlled trials and use of theCONSORT checklist).

In the early sessions, theoretical issues werediscussed in a didactic style. As the knowledge ofthe residents improved, the sessions became mainlystudent-centered. To assess the overall improvementin the participants’ awareness level, an identicalpretest and posttest including 31 multiple-choice(MCQ) questions was administered (Chronbach-ascore = 0.7); their mean values were compared usingpaired samples test. Subgroup analyses were alsoperformed for research methodology (22 MCQs, 11for study design and 11 for application of informa-tion) and statistical analysis (nine MCQs), usingStudent’s t-test. In addition, to evaluate the partici-pants’ competence in critical thinking and appraisal,they were asked to appraise a single randomizedclinical trial (RCT) paper, using the CONSORTchecklist, before and after the course. Two profes-sors were also asked to do the same test to providethe reference answers; in case of any discrepancy, athird professor was asked to do the test. Participantswere instructed to give points to each of the 22individual items in the CONSORT checklistbetween 0 (not mentioned in the article at all) and1 (discussed in the article). Then, the scores ofindividual items on every answer sheet weresummed to yield a total score for its respondent.The mean values of these total scores before andafter the course were compared using PairedSamples Test.

Results

There was a significant improvement in the partici-pants overall awareness level (Table 1) and this wasclearly for ‘application of information’ but not for‘study design’ or ‘statistical analysis’. After combin-ing study design and application of data (method-ology), the study revealed a significant differencebetween the pre and posttest scores (P = 0.017).

The pretest score on critical thinking and apprai-sal was deliberately considered zero as all residents

declared a lack of knowledge in this area. However,their posttest appraisal total score (mean� SD,27.06� 2.98) showed significant improvement(Table 2). However, when we segregated thetotal 22 items of the CONSORT checklist into 14well-discussed items (19.56� 1.82) and 8 poorly-discussed items (7.50�1.58), there appeared asignificant difference between the two compart-ments (mean difference = 12.06; 95% CI10.90–13.21; P< 0.001 using unpaired t-test).

Discussion

The potential use of a journal club in improvingreading habits1 and awareness of biostatistics2 hasalready been described. Other authors have also

Table 2 The CONSORT checklist showing the mean

score in terms of each item after the course

Item Posttest

1 Title and Abstract 0.78� 0.9

2 Background 1.56� 0.6

3 Participants 0.89� 0.7

4 Intervention 1.00� 0.8

5 Objectives 1.06� 0.7

6 Outcomes 1.17� 0.7

7 Sample size 1.61� 0.5

8 Randomization—sequence generation 1.11� 0.5

9 Randomization—allocation concealment 1.17� 0.4

10 Randomization—implementation 0.67� 0.9

11 Blinding 0.67� 0.7

12 Statistic analysis 1.17� 0.8

13 Participant flow 1.06� 0.8

14 Recruitment 1.50� 0.8

15 Baseline data 0.89� 0.9

16 Numbers analyzed 1.72� 0.6

17 Outcomes and estimation 1.72� 0.6

18 Ancillary analyses 1.50� 0.9

19 Adverse events 1.11� 0.9

20 Interpretation 1.72� 0.7

22 Generalizability 1.50� 0.7

22 Overall evidence 1.50� 0.6

Table 1 Comparison of the scores of the pretest and posttest multiple choice questions to assess the improvement of the

participants’ level of awareness

Pretesta Posttesta Mean differencea 95% CI P-value

Methodology 6.8� 2.5 8.8� 3.9 2.0� 3.3 0.4 to 3.7 0.017

Study design 4.8� 2.0 5.2� 2.2 0.4� 2.0 �0.6 to 1.5 0.367

Application of information 2.0� 1.2 3.6� 2.4 1.6� 2.4 0.4 to 2.8 0.012

Statistical analysis 3.8� 2.2 3.8� 2.3 0.0� 2.0 �0.9 to 0.9 1.000

Total 10.6� 4.0 12.6� 5.5 2.0� 4.0 0.0 to 4.0 0.045

aMean� SD.

R.S. Moharari et al.

Page 2 of 3

indicated that teaching the principles of criticalappraisal may also be achieved by this means.3–5

Another review of six studies confirmed the journalclub’s significant role in promoting critical appraisalskills6 but this view is not widely held.7

Our study similarly suggested an improvement incritical appraisal competency and in epidemiologicawareness level of the residents. We concur withKuhn et al.8 that the unavailability of trained facultymembers in the very areas, lack of sufficient timeand budget shortages are the factors which can limitthe effectiveness of the journal club. We also foundthat the participants were unaware of their lack ofknowledge of epidemiology and critical appraisalskills, a situation that improved by the end of thesessions.

In summary, we conclude that the journal clubrepresents a simple but effective means of effectivelyengaging residents in developing their research andcritical appraisal skills. We recommend that furtherresearch we control groups and larger sample sizesshould be performed in order to fully exploit theteaching potential of the journal club.

Acknowledgement

We acknowledge the support of the Research andDevelopment Center of Sina Hospital.

References1. Linzer M, Brown JT, Delong ER, Siegel WC. Impact of

medical journal club on house-staff reading habits, knowl-

edge and critical appraisal skills: a randomized, control trial.

J Am Med Assoc 1988; 260:2537–41.

2. Cheatam ML. A structured curriculum for improved resident

education in statistics. Am Surg 2000; 66:585.

3. Kellum JA, Rieker JP, Power M, Powner DJ. Teaching critical

appraisal during critical care fellowship training: A founda-

tion for evidence-based critical care medicine. Crit care Med

2000; 28:3067–70.

4. Seeling CB. Affecting residents’ literature reading attitudes,

behaviors and knowledge through a journal club interven-

tion. J Gen Intern Med 1991; 6:330–4.

5. Cramer JS, Mahoney MC. Introducing evidence based

medicine to the journal club, using a structured pre a post

test: a cohort study. BMC Med Educ 2001; 1:6.

6. Ebbert JO, Montori VM, Schultz HJ. The journal club in post

graduate medical education: a systematic review. Med Teach

2001; 23:455–61.

7. Bennett KJ, Sackett DL, Haynes RB, Neufeld VR, Tugwell P,

Roberts R. A controlled trial of teaching critical appraisal of

the clinical literature to medical students. J Am Med Assoc

1987; 257:2451–4.

8. Kuhn GJ, Wyer PC, Cordell WH, Rowe BH. A Survey to

determine the prevalence and characteristics of training in

evidence – based medicine in emergency medicine. J Emer

Med 2005; 28:353–9.

Critical appraisal and statistics

Page 3 of 3