Performance on Brief Practice Exam Identifies residents at Risk for Poor

ABSITE and ABS Qualifying Exam

PerformanceMichael Corneille MD, Ross Willis PhD, Ronald Stewart

MD, Daniel Dent MD

BackgroundBackground ABSITE

First offered 1975 to gauge residents’ acquisition of fundamental surgical knowledge

Not designed as predictive tool Strong correlation between performances on ABSITE and

ABS QE Shellito et al. (2010) Am J Surg

No pass/fail cutoff ABSITE vs ABS QE pass rates

ABSITE >30th %ile - 93% 1st attempt QE passed ABSITE ≤30th %ile - 73% 1st attempt QE passed

2009 ABSITE .51 correlation between score and QE score (1st

time) .92 correlation between score and QE pass (1st

time)

Act as a study tool Identify residents at risk for poor

performance on ABSITE and QE

2 part exercise Constructing the exam Taking the exam

Objectives based on potential reasons for poor performance

Purpose of Practice Exams

Objectives How goal met by practice exam

Understanding that the exam comprised a finite list of topic areas.

Distribution of ABSITE keywords

Master the 3-5 topics resident was assigned. Writing questions

Develop insight into how a test is constructed. Writing questions

Encourage resident to focus on the “board relevant” material.

Writing questions

Understand that it is possible to prepare for the exam.

Test taking

Identify residents earlier than March who will need help.

Test taking

Serve as a wake-up call to resident that his/her study method is not effective and his/her performance needs to improve.

Test taking

Keep residents engaged in continuous studying for the ABSITE.

Offering exam various times throughout the year

Background Practice test not designed as a

predictive tool

Same guidelines as ABSITE Scores not used to determine promotion

Allow program director to objectively compare resident’s fund of knowledge to peers

HypothesisHypothesis We lacked objective data to counsel

residents as to true interpretation of practice test scores

Hypothesis A brief practice exam

May identify residents at risk of poor ABSITE performance

May identify early in the chief resident year those at risk for poor QE performance.

Methods Study period 2005-2010

RRC approved to graduate 9 chiefs/year 1-3 practice exams/year 60 mins/exam (50-55 basic science & clinical

questions) Questions based on the ABSITE keywords

authored by residents and edited by faculty Assigned 3 to 5 topics

Standardized format Question 5 response options Explanations with references

Optically graded

Methods Exam was considered mandatory

Educational time allotted within the training program

Scores reported to residents within two weeks % correct and standard deviation within PGY class

Residents >1 standard deviation below the mean for their class counseled Expectation of the program > 30th %ile reiterated Referred to a faculty mentor to develop study plan

Data Analysis

Compared resident’s practice exam most recently preceding ABSITE to that year’s ABSITE

Compared chief year ABSITE to ABS QE Pass/fail status

Spearman rank correlation coefficients used to compare dependent measures

ResultsResults 9 practice exams administered to all resident

levels 432 individual practice exams completed 309 residents have taken the ABSITE 33 have taken the ABS QE

263 residents completed at least one practice exam in the year preceding the ABSITE

26 chief residents completed at least one practice exam immediately preceding the ABS QE

ABSITE Year N r (Spearman) Significance

2010 51 .285 p < .05

2009 67 .603 p < .01

2008 51 .62 p < .01

2007 45 .422 p < .01

2006 49 .413 p < .01

Practice Test vs ABSITE

Correlation between Practice Exam and first attempt QE score was significant (r = .416, p < .05)

Correlation between chief year ABSITE and the first attempt QE score was significant (r = .743, p < .01)

ABSITE score <30th %ile 89% passed the QE on the first attempt

Data Analysis

DiscussionDiscussion

What to do with the data?

Can we intervene?

Is the exercise beneficial?

Several authors have reported that intervention can improve scores 1,2

Our data show that performance on practice exam is correlated to ABSITE and ABS QE

May suggest that our interventions are inadequate or that there is a cohort of residents destined to struggle with standardized assessments

DiscussionDiscussion

1Borman KR, Does Academic Intervention Impact ABS Qualifying Examination Results? Current Surgery, Vol 63, Issue 6, Pp 367-372, Nov 2006

2Hirvela ER, Becker DR Impact of Programmed Reading on ABSITE Performance. The American Journal Of Surgery, Vol 162 Nov 1991)

DiscussionDiscussion

Silver lining May seem overall pessimistic

Those who do struggle may be excellent surgeons

Those who struggle may pass their boards on the first attempt

89% first attempt pass with ABSITE scores less than 30th percentile

ConclusionsConclusions

Poor performance on a brief practice exam

Does identify residents at risk for poor ABSITE performance

Does identify residents at risk for poor ABS QE performance and earlier than ABSITE scores.

ConclusionsConclusions

More data is necessary on what More data is necessary on what interventions truly work to improve interventions truly work to improve ABS QE pass rates in very high risk ABS QE pass rates in very high risk residentsresidents

Thank You