Does resident recruitment ranking predict subsequent

performance?Jonathan Fryer, Noreen Corcoran,

Brian George, Ed Wang, Deb DaRosaNorthwestern University Feinberg School of Medicine,

Chicago Illinois

Screening of Applications

Assessment of

Candidates

Ranking of Candidates

Resident Selection Process

1) How effective is the ranking process in selecting residents who will perform well?

2) How effective is the ranking process in predicting subsequent resident performances?

3) How effective are “adjustments” made to the preliminary rank list in improving the selection of residents who will perform well?

Questions?

General surgery categorical resident recruitment between 2002-2011 inclusive (n=46).◦ 4 categorical residents 2002-2004.◦ 5 categorical residents 2005-2011.

Residents who dropped out (n=1) or who were recruited after the PGY1 year (n=2) were excluded from the analyses.

Methods

We compared how successful candidates were ranked during recruitment with their subsequent performance in our program.

Methods

1) USMLE˄ Scores alone 2) Unadjusted Ranking Score (URS˄):

based on sum of 3 assessment scores ◦ Academic Profile (Coordinators)

Medical school rank, USMLE Step 1, Class Rank, Honors in Surgery

◦ Program Director Review (PDs) Research experience, extracurricular/community

involvement, LORs, Personal Statement, Dean’s Letter

◦ Faculty Interview score (Faculty) Averaged for 2 independent faculty interview scores.

Recruitment Ranking Parameters

˄ Higher score is better

3) Final Adjusted Ranking (FAR˅)◦ Modification of the preliminary rank list generated

by the URS ◦ Based on additional insights about specific

candidates provided by the resident selection committee and/or leadership

Endorsements from trusted colleagues Negative interactions with staff Concerns raised by residents, coordinators Other?

Recruitment Ranking Parameters

˅ Lower is better

Screening Academic Profile *

Assessment

URS *Ranking

FAR

Resident Selection ProcessNUFSM

*USMLE included

1) ABSITE˄ percentile alone 2) Resident Evaluation Grade (REG˄)

◦ Semiannual evaluation scores (Letter grade: A-F)◦ Group discussion and grade assignment based on:

Clinical Evaluations (360°): faculty, peers, med students, nurses, patients, etc.

Compliance: evaluations, case log, duty hour log, conference attendance, etc.

ABSITE, Mock Oral, PAME scores.

Resident Performance Measures

˄ higher is better

3) Independent Faculty Rating/Ranking (IFRR˅) ◦ Confidential survey with faculty independently

rating all residents using a 7-point Likert scale and ranking resident within their PGY1 recruitment cohorts.

◦ * not part of standard resident evaluation at NU

Resident Performance Measures

˅ lower is better

Full IRB approval was obtained. All resident ranking and performance data

was de-identified after collection and aggregated to protect resident confidentiality.

Methods

Semiannual Resident Evaluation Grades (A-F) were converted to numerical values (5-0, respectively) and averaged for analyses.

Data from Individual Faculty Rating/Ranking surveys were averaged for individual residents.

Methods

Statistical Analyses performed using SAS 9.2 software (Cary, NC).◦ Associations between ranking and performance

parameters were analyzed using Spearman’s correlation coefficient.

◦ Comparison of ranking parameters between poor and satisfactory performance used student t-test.

◦ Differences in performance based on ranking range were compared using F-test.

Methods

Poor Resident Performance Parameter

# (%) comments

Drop outs 1 of 46 (2.2%) excellent evaluations

ABS QE/CE fail 0 of 13 (0%)

REG < C * 1 of 46 (2.2%) Probation

IFFR > 4.0 6 of 46 (13.0%) >4.0 = below average

ABSITE < 35% * 12 of 46 (26.1%)

ResultsOverall resident performance

* Occurring at any time during residency training

Recruitment Ranking

USMLE˄ URS˄ FAR˅

Resident Performance

R* p R* p R* p

ABSITE (%tile)˄

0.61 <0.0001

0.06 0.6952 0.09 0.5891

PGY1 REG˄ 0.12 <0.4087 0.40 0.0058 0.17 0.2597

Overall Grade˄

0.16 <0.2783 0.34 0.0219 0.16 0.2783

Independent Faculty Rating˅

0.22 <0.1409 0.02 0.9020 -0.12 0.4245

Recruitment Ranking vs Performance

* Spearman correlation coefficient

˄ higher is better˅ lower is better

USMLE˄ URS˄ FAR˅Poor Resident Performance Criteria

Mean SD Mean SD Mean SD

*REG ˄< C (n=1) 239.00 - 111.00 - 27.00 -

≥ C (n=46) 238.42 15.27 110.44 6.00 18.48 11.83

p value 0.8008 0.6041 <0.0001

IFRR ˅ ≥ 4.0 (n=6) 232.50 17.44 111.33 5.20 13.14 9.12

< 4.0 (n=41) 239.33 14.76 110.33 6.08 19.52 12.00

p value 0.3073 0.7023 0.2199

*ABSITE ˄ % < 35 (n=12)

228.92 17.70 110.25 5.93 19.25 13.59

≥ 35 (n=35) 243.48 13.05 110.32 6.37 17.95 11.95

p value 0.0057 0.9752 0.7625

Predicting Poor Resident Performances

Predictors of Poor Resident Performances

* Occurring at any time during the residency

USMLE˄ URS˄ FAR˅Poor Resident Performance Criteria

Mean SD Mean SD Mean SD

*REG ˄< C (n=1) 239.00 - 111.00 - 27.00 -

≥ C (n=46) 238.42 15.27 110.44 6.00 18.48 11.83

p value 0.8008 0.6041 <0.0001

IFRR ˅ ≥ 4.0 (n=6) 232.50 17.44 111.33 5.20 13.14 9.12

< 4.0 (n=41) 239.33 14.76 110.33 6.08 19.52 12.00

p value 0.3073 0.7023 0.2199

*ABSITE ˄ % < 35 (n=12)

228.92 17.70 110.25 5.93 19.25 13.59

≥ 35 (n=35) 243.48 13.05 110.32 6.37 17.95 11.95

p value 0.0057 0.9752 0.7625

Predicting Poor Resident Performances

Predictors of Poor Resident Performances

* Occurring at any time during the residency

Performance criteria

ABSITE(% tile)

PGY1 REG Overall REG IFRR

Rank range( % tile)˅

Mean SD Mean SD Mean SD Mean

SD

<15 % (n=14) 61.69 28.82 4.57 0.55 4.50 0.44 3.10 0.78

15 -30% (n=17)

65.57 21.77 4.29 0.79 4.26 0.64 3.16 0.80

>30% (n=22) 66.43 23.73 4.00 0.76 4.07 0.56 3.22 0.68

P value (F-test) 0.8831 0.1106 0.1338 0.9158

Rank range vs. performance(URS)

Performance criteria

ABSITE˄ PGY1 REG˄

Overall REG˄

IFRR˅

Rank Range(% tile)˅

Mean SD Mean SD Mean SD Mean SD

<15% (n=19) 60.84 26.40 4.63 0.62 4.42 0.50 3.13 0.83

15-30% (n=9) 66.17 22.26 3.81 0.66 3.92 0.61 3.52 0.65

>30% (n=22) 70.00 28.81 4.36 0.71 4.54 0.38 2.70 0.43

P value (F-test) 0.6475 0.0025 0.0051 0.0156

Rank range vs. performance(FAR)

Performance criteria

ABSITE˄ PGY1 REG˄

Overall REG˄

IFRR˅

Rank Range(% tile)˅

Mean SD Mean SD Mean SD Mean SD

<15% (n=19) 60.84 26.40 4.63 0.62 4.42 0.50 3.13 0.83

15-30% (n=9) 66.17 22.26 3.81 0.66 3.92 0.61 3.52 0.65

>30% (n=22) 70.00 28.81 4.36 0.71 4.54 0.38 2.70 0.43

P value (F-test) 0.6475 0.0025 0.0051 0.0156

Rank range vs. performance(FAR)

Single center study URS confounded by USMLE score REG confounded by ABSITE No formal faculty orientation for IFRR survey

Study Limitations

USMLE scores were predictive of subsequent ABSITE performance only.

Unadjusted Ranking Scores (URS) were predictive of subsequent performance based on resident evaluation grades (REGs), while Final Adjusted Rankings (FAR) were not.

Summary

Our resident selection process has generally been successful in providing us with residents who perform well.

Our unadjusted ranking score appears to be a better predictor of subsequent resident performance than our final adjusted ranking… ….therefore caution should be exercised when considering adjustments to the preliminary rank list, as they may not engender selection of better performing residents.

Effectively defining a reliable rank list “cutoff”, beyond which performance will predictably decrease, may not be possible in our system.

Conclusions

◦ Drop outs: 1 (after PGY1 despite excellent performance)

◦ ABS exams first try pass rate (n=13): 100%

◦ Ever with REG <C 100% (i.e. probation): 1◦ IFR > 4.0 (i.e. below average): 6◦ Ever with ABSITE scores < 35%tile ever: 12

ResultsOverall resident performance

Range range vs. performance

USMLE URS FAR

Poor Resident Performance Criteria

Mean SD Mean SD Mean SD

REG < C (n=1)

239.00 - 111.00 - 27.96 -

≥ C (n=46)

238.42 15.27 110.44 6.00 18.48 11.83

p value 0.8008 0.6041 <0.0001

IFR ≥ 4.0 232.50 17.44 111.33 5.20 13.14 9.12

< 4.0 239.33 14.76 110.33 6.08 19.52 12.00

p value 0.3073 0.7023 0.2199

ABSITE % < 35

228.92 17.70 110.25 5.93 19.25 13.59

≥ 35

243.48 13.05 110.32 6.37 17.95 11.95

p value 0.0057

0.9752 0.7625

Predicting Poor Resident Performances

Predictors of Poor Resident Performance

Absolute ranking correlation with resident performance:◦ Absite◦ Semiannual evaluation grade◦ Faculty survey rating

Results Absolute Ranking

Absolute ranking (AR): Ranking among entire candidate group (n= 60-80).

Relative ranking (RR): Ranking among cohort of successful PGY1 applicants (n=4 or 5).

Recruitment Ranking

Within resident cohorts FAR did not correlate significantly with subsequent: ◦ ABSITE scores (r=0.22; p=0.1760) ◦ Semi-annual evaluation scores (r=0.20; p=0.1987)◦ Faculty survey cohort rankings (r=0.23;0.1175)

Conversely, USMLE scores exhibited a significantly positive correlation with subsequent:◦ ABSITE scores (r=0.46; p=0.0022), ◦ Semi-annual evaluation scores (r=0.41; p=0.0163)◦ Faculty cohort rankings (r=0.35; p=0.163)

Results (ABSTRACT)Relative Ranking

Resident recruitment involves a formal evaluation of candidates where a variety of objective and subjective criteria are used to rank candidates from best to worst.

Preliminary rank lists are often subsequently “adjusted” based on additional insights about the candidates.

Introduction

Screening of Applications

Assessment and Preliminary Ranking of Candidates

URS

Final Ranking of Candidates

FAR

Resident Selection ProcessNUFSM