Performance on Brief Practice Exam Identifies residents at Risk for Poor
ABSITE and ABS Qualifying Exam
PerformanceMichael Corneille MD, Ross Willis PhD, Ronald Stewart
MD, Daniel Dent MD
BackgroundBackground ABSITE
First offered 1975 to gauge residents’ acquisition of fundamental surgical knowledge
Not designed as predictive tool Strong correlation between performances on ABSITE and
ABS QE Shellito et al. (2010) Am J Surg
No pass/fail cutoff ABSITE vs ABS QE pass rates
ABSITE >30th %ile - 93% 1st attempt QE passed ABSITE ≤30th %ile - 73% 1st attempt QE passed
2009 ABSITE .51 correlation between score and QE score (1st
time) .92 correlation between score and QE pass (1st
time)
Act as a study tool Identify residents at risk for poor
performance on ABSITE and QE
2 part exercise Constructing the exam Taking the exam
Objectives based on potential reasons for poor performance
Purpose of Practice Exams
Objectives How goal met by practice exam
Understanding that the exam comprised a finite list of topic areas.
Distribution of ABSITE keywords
Master the 3-5 topics resident was assigned. Writing questions
Develop insight into how a test is constructed. Writing questions
Encourage resident to focus on the “board relevant” material.
Writing questions
Understand that it is possible to prepare for the exam.
Test taking
Identify residents earlier than March who will need help.
Test taking
Serve as a wake-up call to resident that his/her study method is not effective and his/her performance needs to improve.
Test taking
Keep residents engaged in continuous studying for the ABSITE.
Offering exam various times throughout the year
Background Practice test not designed as a
predictive tool
Same guidelines as ABSITE Scores not used to determine promotion
Allow program director to objectively compare resident’s fund of knowledge to peers
HypothesisHypothesis We lacked objective data to counsel
residents as to true interpretation of practice test scores
Hypothesis A brief practice exam
May identify residents at risk of poor ABSITE performance
May identify early in the chief resident year those at risk for poor QE performance.
Methods Study period 2005-2010
RRC approved to graduate 9 chiefs/year 1-3 practice exams/year 60 mins/exam (50-55 basic science & clinical
questions) Questions based on the ABSITE keywords
authored by residents and edited by faculty Assigned 3 to 5 topics
Standardized format Question 5 response options Explanations with references
Optically graded
Methods Exam was considered mandatory
Educational time allotted within the training program
Scores reported to residents within two weeks % correct and standard deviation within PGY class
Residents >1 standard deviation below the mean for their class counseled Expectation of the program > 30th %ile reiterated Referred to a faculty mentor to develop study plan
Data Analysis
Compared resident’s practice exam most recently preceding ABSITE to that year’s ABSITE
Compared chief year ABSITE to ABS QE Pass/fail status
Spearman rank correlation coefficients used to compare dependent measures
ResultsResults 9 practice exams administered to all resident
levels 432 individual practice exams completed 309 residents have taken the ABSITE 33 have taken the ABS QE
263 residents completed at least one practice exam in the year preceding the ABSITE
26 chief residents completed at least one practice exam immediately preceding the ABS QE
ABSITE Year N r (Spearman) Significance
2010 51 .285 p < .05
2009 67 .603 p < .01
2008 51 .62 p < .01
2007 45 .422 p < .01
2006 49 .413 p < .01
Practice Test vs ABSITE
Correlation between Practice Exam and first attempt QE score was significant (r = .416, p < .05)
Correlation between chief year ABSITE and the first attempt QE score was significant (r = .743, p < .01)
ABSITE score <30th %ile 89% passed the QE on the first attempt
Data Analysis
DiscussionDiscussion
What to do with the data?
Can we intervene?
Is the exercise beneficial?
Several authors have reported that intervention can improve scores 1,2
Our data show that performance on practice exam is correlated to ABSITE and ABS QE
May suggest that our interventions are inadequate or that there is a cohort of residents destined to struggle with standardized assessments
DiscussionDiscussion
1Borman KR, Does Academic Intervention Impact ABS Qualifying Examination Results? Current Surgery, Vol 63, Issue 6, Pp 367-372, Nov 2006
2Hirvela ER, Becker DR Impact of Programmed Reading on ABSITE Performance. The American Journal Of Surgery, Vol 162 Nov 1991)
DiscussionDiscussion
Silver lining May seem overall pessimistic
Those who do struggle may be excellent surgeons
Those who struggle may pass their boards on the first attempt
89% first attempt pass with ABSITE scores less than 30th percentile
ConclusionsConclusions
Poor performance on a brief practice exam
Does identify residents at risk for poor ABSITE performance
Does identify residents at risk for poor ABS QE performance and earlier than ABSITE scores.
ConclusionsConclusions
More data is necessary on what More data is necessary on what interventions truly work to improve interventions truly work to improve ABS QE pass rates in very high risk ABS QE pass rates in very high risk residentsresidents
Thank You