Background:Background: As students complete their clerkships throughout their M3 year they gain in clinical experience and confidence, which may translate into improved performance on various end-of-rotation assessments. Specialties such as surgery, obstetrics/gynecology, internal medicine, family medicine, and pediatrics have demonstrated a “timing bias” of improved scores on the NBME subject exams as students progress through

their third year 1-2. This is relevant since students who wish to pursue a specific specialty may purposely choose that rotation later in the academic year, thereby “gaming the system.” Conversely, students who take a clerkship early due to interest may be penalized with a lower NBME grade.

The literature for psychiatry clerkships has shown no change or slight increases in NBME scores as the year

progresses 3. One study extended the concept of timing bias to 3 areas of student evaluation: the supervisors’ evaluations, SP exam (SPE), and NBME subject exam scores. While there was a weak linear trend for the NBME, the 2 other measures showed no positive trend 4. We wondered if these findings could be replicated at our institution.

Methods:Methods:Data was collected from 2 academic years, 2005-2007. Rotations for each year were numbered from 1-7, beginning with the first rotation in July of each year. The mean scores for three score components – supervisors’ evaluations, SPE scores, and NBME scores – were calculated for each rotation. Data from rotation #4 (December) was not included as that rotation was composed of 4th-year medical students taking their clerkship out of sequence. We conducted ANOVAs for each of the three score components across time looking for linear trends.

Does Increased Clinical Experience Improve Performance in the Psychiatry ClerkshipDoes Increased Clinical Experience Improve Performance in the Psychiatry Clerkship?? Sean M. Blitzstein M.D.Sean M. Blitzstein M.D.11, Rachel Yudkowsky M.D. MHPE, Rachel Yudkowsky M.D. MHPE22, and Mihaela Popescu, and Mihaela Popescu22

11Department of Psychiatry and Department of Psychiatry and 2 2Department of Medical EducationDepartment of Medical EducationUniversity of Illinois at ChicagoUniversity of Illinois at Chicago

Discussion:Discussion:Our study demonstrated a significant upward trend for scores on the NBME over time, consistent with findings in other medical specialties. However, the increased knowledge and the clinical experience gained in prior clerkships did not lead to improved supervisors’ evaluations or SPE grades.

The lack of improvement in supervisor evaluations was surprising, given the commonly held belief that students improve in their performance over the year. While the evaluations used at our institution have some items particular to psychiatry (MSE, etc), there are many general skills which could be transferable across specialties (interviewing, data gathering, professionalism, response to feedback, etc). Our results may reflect the limited reliability and validity of end-of-rotation supervisor evaluations, well documented in the literature 5.

The lack of a trend for the SPE grades over time was not surprising. The knowledge and skills for the SPE are likely case-specific as well as specialty (psychiatry)-specific, therefore not transferable from other specialties as the year progresses.

Conclusions:Conclusions:Students who are considering a residency in psychiatry and want to ensure a good clerkship grade may feel they are at a disadvantage if selecting their psychiatry rotation earlier in the academic year. The 5-point NBME disadvantage could, in some cases, make the difference between an “outstanding” and “advanced” grade in the clerkship. Whalen1 suggested adjusting NBME scores to compensate for the advantage of later experience, but this raises additional complications. In our opinion the NBME disadvantage does not outweigh the career-planning advantages of an early clerkship. We do not recommend that students interested in psychiatry postpone their rotation to later in the year.

References:References:1. Whalen JP, Moses VK. The effect on grades of the timing and site of

third-year internal medicine clerkships. Acad Med. 1990; 65:708-7092. Baciewicz FA, et al. Influence of clerkship structure and timing on

individual student performance. Am J Surg. 1990; 159:265-2683. Manley M, Heiss G. Timing bias in the psychiatry subject examination

of the national board of medical examiners. Acad Psychiatry. 2006; 30:116-119

4. Park RS, et al. Relationship of rotation timing to pattern of clerkship performance in psychiatry. Acad Psychiatry. 2006; 30:116-119

5. Williams RG, et al. Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med. 2003; 15:270-292

Purpose:Purpose: At UIC the 6-week Psychiatry Clerkship begins at seven different times during medical students’ third (M3) year. We wanted to determine if scores on the psychiatry clerkship supervisor evaluations, standardized patient exam (SPE), and/or the National Board of Medical Examiners Psychiatry Subject Examination (NBME) increased in psychiatry rotations taken later in the M3 year.

For more information, please contact

Sean Blitzstein, MD <sblitz@uic.edu>

The Three Score Components:The Three Score Components: The final grade of the six-week psychiatry clerkship is based on three score components:

1. Supervisor Evaluations

• Supervisors rate eight competencies (e.g., interviewing, professionalism, MSE) resulting in a summary global rating on a 4-point scale: 1 = needs remediation, 2 = proficient, 3 = advanced, and 4 = outstanding

2. Standardized Patient Exam (SPE)

• Each student encounters 3 SPs portraying common psychiatric disorders for 30 minutes each and documents the MSE and differential diagnosis.

• The SPE score is composed of 4 elements: SPs complete (1) an interview checklist and (2) a communications skills scale after each encounter. Faculty score (3) the MSE and (4) differential diagnoses.

• These four components are averaged to obtain a final SPE grade of needs remediation, proficient, advanced, or outstanding (same scale as supervisor evaluations).

3. NBME psychiatry subject exam

•Passing score of 60

Results:Results:A total of 298 students participated in the psychiatry clerkship in 2005-2007.

Supervisors’ evaluations (N=297) were not equivalent across rotations. ANOVA F (5) =4.370, p=.001 Post-hoc analysis (Scheffe) showed rotation #3 differed significantly from #5 and #6, however no linear trend was observed.

SPE scores did not show significant differences or linear trends over time (N=268), ANOVA F (5) =1.138, p=.341

NBME scores demonstrated a significant upward linear trend: (N=298), ANOVA F (5) =3.909, p=.002, Linear contrast F=16.760, p<.005

Oneway of Supervisor, SPE, and NBME grades by Rotation number

0

0.5

1

1.5

2

2.5

3

3.5

4

1 2 3 5 6 7

Rotation number

Mean

Su

perv

iso

r an

d S

PE

gra

des

70

72

74

76

78

80

82

Mean

NB

ME

gra

des

Supervisor grades SPE grades NBME grades

Supervisor Grade x2

SPE Grade

NBME Grade

ClinicalGrade x2

Final ClerkshipGrade

Rotation number

Month Supervisor Grade Mean (SD)

SPE gradeMean (SD)

NBME gradeMean (SD)

1 June 3.25 (.52) 2.64 (.65) 73.70 (8.79)

2 August 3.19 (.58) 2.74 (.74) 73.28 (8.04)

3 October 3.48 (.64) 2.80 (.67) 75.15 (8.89)

5 January 3.00 (.65) 2.62 (.70) 78.12 (8.30)

6 March 3.09 (.64) 2.74 (.75) 76.96 (8.61)

7 April 3.36 (.62) 2.93 (.66) 79.79 (7.21)