Donald J. Sefcik, DO, MBASenior Associate Dean

College of Osteopathic MedicineMichigan State University

• In-Service Examination (ISE) scoresWhat do they represent?How should you interpret them?What should you do with them?

• Techniques used to raise ISE scores• Recommendations

Emergency Medicine• Hern et al (2009); Cheng (2008); Gillen (1997)Family Medicine• Sevensma SC, Navarre G, Richards RK (2008)Internal Medicine• McDonald, Zeger & Kolars (2008)• Garilbaldi et al (2002)OB/Gyne• Withiam-Leitch & Olawaiye (2008)Pediatrics• Aeder, Fogel & Schaeffer (2010)• Langenau, Fogel & Schaeffer (2009)RadiologySurgery• Shellito et al (2010); Subhas et al (2009); Kosir et al (2008)• Ferguson & Warshaw (2006); Virgilio et al (2003); Bull et al (2001)

A 72 year-old patient presents with progressive difficultybreathing, jugular venous distention, orthopnea, bilaterallower extremity edema , and bilateral rales. Which of thefollowing is most important when deciding upontreatment options?

A. minute respirationB. heart rateC. degree of lower extremity edemaD. respiratory rateE. none of the above

A 72 year-old patient presents with progressive difficultybreathing, jugular venous distention, orthopnea, bilaterallower extremity edema , and bilateral rales. Which of thefollowing is most important when deciding upontreatment options?

We don’t treat patients without understanding theetiology of their predicament and the nature of theintervention. Guiding residents’ learning should followsimilar guidelines.

XR = XT + ME

XR = XT + ME

XR = Reported ISE scoreXT = True Score ME = Measurement Errors

XR = XT + ME

XR = Reported ISE Score ScoresPercent = items correct = raw scorePercentile = resident rank compared to peers

The report of the performance of a PGY2 on his ISEreveals a percent correct of 63 and a percentile rank of 67.What should you do?

A. mandate the resident enter a remediation programB. reduce work hours by 25% to increase study timeC. do not promote the resident to the PGY3 yearD. encourage the resident to keep studying E. resign as program director

• What is an “average” percent score?

http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf

• What is an “average” percent score?2009 IM (n = 402 programs)PGY1 = 54%PGY2 = 59%PGY3 = 63%

http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf

• What is an “average” percent score?2009 IM (n = 402 programs)PGY1 = 54%PGY2 = 59%(example was PGY2 = 63%)PGY3 = 63%

http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf

Your PGY2 from the previous example is now a PGY3.His percentile rank while in your program has been PGY1 = 62, PGY2 = 67, PGY3 = 64. What should you do?

A. realize that this is a common pattern of progression B. worry that this resident will fail his certifying examC. do not promote the resident to the PGY4 yearD. terminate this resident from your program E. delegate the situation to your associate PD

Why is percentile important?Increased risk of failing the certifying exam

%F

50th30th 100th

Why is percentile important?Increased risk of failing the certifying exam

%F

50th30th 100th

The lower the percentile rank –

The greater the risk of failing the certifying examination

Percent Correct Percentile

Increases each year

PGY1 PGY2 PGY3 PGY4

Percent Correct Percentile

Potential Problem

PGY1 PGY2 PGY3 PGY4

Percent Correct Percentile

Tends to be stable

PGY1 PGY2 PGY3 PGY4

Percent Correct Percentile

Potential Problem

PGY1 PGY2 PGY3 PGY4

Tracking percentiles• Three different residents • Three years of scores

%F

50th30th

A1B1C1

100th

Tracking percentiles• Three different residents • Three years of scores

%F

50th30th

A1

A2

B1

B2

C1

C2

100th

Tracking percentiles• Three different residents • Three years of scores

%F

50th30th

A1

A3

B1

B3

C1

C3

C2 B2 A2

100th

Tracking percentiles • Three different residents • Three years of scores

%F

50th30th

A1

A3

B1

B3

C1

C3

C2 B2 A2

100th

What can you do to assess the residents between the annual ISE’s?

• Make decisions about • individual’s educational progression• program content areas that are outliers

• Make decisions about • individual’s educational progression• program content areas that are outliers

• Do not make decisions about:• promotion or termination

XR = XT + ME

XT = True Score = two components

XR = XT + ME

XT = True Score = two components

XC = Score based on contentXB = Score based on beyond content

XR = XC + XB + ME

XR = Reported ISE Score XC = Score based on contentXB = Score based on beyond contentME = Measurement Error

• Choosing your intervention options• Content • Beyond Content

XC + XB

Content

Beyond Content

PGY1 PGY2 PGY3 PGY4

• Assigned / Programmed Reading• Conference Attendance

• Didactic lectures • Grand Rounds / M&M

• Practice Questions• Board Review Courses

• Assigned / Programmed Reading• Conference Attendance

• Didactic lectures / Journal Clubs• Grand Rounds / M&M

• Practice Questions• Board Review Courses

• Self-directed Studying• Preparing Lectures for Peers• Reading Fluency and Accuracy• Test Anxiety

• Self-directed Studying• Preparing Lectures for Peers

• Questions from / Clarification by an Attending• Reading Fluency and Accuracy

• Developing theme in repeat failures• Test Anxiety

• Definite impact on memory / recall

• Intervention options • Content = more impact earlier in training

PGY<2 - Programmed Reading / Didactics PGY>3 - Programmed Reading / Didactics

• Beyond Content • Resident Heterogeneity (all years of training)

Studying SkillsTest-taking Skills

• Beyond Content • Resident Heterogeneity

Studying Skills• Memory techniques • Reading and Note-taking • Transfer Test-taking Skills• Reading- Fluency and Accuracy• Test Anxiety• Test-Wiseness

1 / 6

2 / 6

3 / 6

1. Assess the resident’s situationDevelop an individualized planIt’s not just about XC

2. Create a teamResident and AttendingAttending = Coach to Mentor

3. Establish a scheduleRegular meetings

4. Monitor and Refine

4 / 6

1. Assess the resident’s situationDevelop an individualized planIt’s not just about XC

2. Create a teamResident and AttendingAttending = Coach to Mentor

3. Establish a scheduleRegular meetings

4. Monitor and Refine

There are resourcesto help youhelp your resident

5 / 6

6 / 6

We need to transition away from our focus on teaching (faculty-centered) and instead begin to focus on learning (resident-centered).

Are there some Questions?

Complete Reference List will be posted (pdf)

on SCS website

Emergency Medicine• Hern et al (2009); Cheng (2008); Gillen (1997)Family Medicine• Sevensma SC, Navarre G, Richards RK (2008)Internal Medicine• McDonald, Zeger & Kolars (2008)• Garilbaldi et al (2002)OB/Gyne• Withiam-Leitch & Olawaiye (2008)Pediatrics• Aeder, Fogel & Schaeffer (2010)• Langenau, Fogel & Schaeffer (2009)RadiologySurgery• Shellito et al (2010); Subhas et al (2009); Kosir et al (2008)• Ferguson & Warshaw (2006); Virgilio et al (2003); Bull et al (2001)