the new mcat exam - aamc · pdf filethe new mcat exam: where we are and where we’re...
TRANSCRIPT
The New MCAT Exam: Where We Are
and Where We’re GoingRobert Witzburg, M.D.
Dean of Admissions, Boston University School of Medicine
Judith Byrne and Cynthia A. SearcyAAMC
CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2
What will we talk about today?• What do we know about applicants to the class we just
admitted?• What do we know about this year’s test takers (many of whom
are in our current applicant pool)?• What are we learning about the impact, use, and validity of the
new exam?• Q&A
CONFIDENTIAL – DO NOT DISTRIBUTE3 CONFIDENTIAL – DO NOT DISTRIBUTE3 CONFIDENTIAL – DO NOT DISTRIBUTE3
Today’s information appears on the Admissions Hub• Using MCAT Data in 2018 Medical
Student Selection
• Interactive Score Report • Videos• Presentations and other materials
on the new exam, new scores, and validity research
• Research articles
www.aamc.org/admissions
CONFIDENTIAL – DO NOT DISTRIBUTE4 CONFIDENTIAL – DO NOT DISTRIBUTE4 CONFIDENTIAL – DO NOT DISTRIBUTE4
Just finished the third year of testing w/ the new exam
Applicants/Matriculants
Year 1Coursework
Year 2Coursework
USMLEStep 1
Clerkships
Graduate in4 Years
Graduate in5 Years
USMLE Step 2 Exams
2015 2016 2017 2019 2020 20212018
2016 2017 2018 2019 2020 2021 2022
2017 2019 2020 2021 2022 20232018
Examinees
CONFIDENTIAL – DO NOT DISTRIBUTE5 CONFIDENTIAL – DO NOT DISTRIBUTE5 CONFIDENTIAL – DO NOT DISTRIBUTE5
Today, we’ll look at data for examinees, applicants, and M1s who took the new test
Applicants/Matriculants
Year 1Coursework
Year 2Coursework
USMLEStep 1
Clerkships
Graduate in4 Years
Graduate in5 Years
USMLE Step 2-CK
USMLE Step 2 Exams
2015 2016 2017 2019 2020 20212018
2016 2017 2018 2019 2020 2021 2022
2017 2019 2020 2021 2022 20232018
Examinees
CONFIDENTIAL – DO NOT DISTRIBUTE6 CONFIDENTIAL – DO NOT DISTRIBUTE6 CONFIDENTIAL – DO NOT DISTRIBUTE6
Let’s see how it’s going!
CONFIDENTIAL – DO NOT DISTRIBUTE7 CONFIDENTIAL – DO NOT DISTRIBUTE7 CONFIDENTIAL – DO NOT DISTRIBUTE7
What do we know about applicants to the class we just admitted?
CONFIDENTIAL – DO NOT DISTRIBUTE8 CONFIDENTIAL – DO NOT DISTRIBUTE8 CONFIDENTIAL – DO NOT DISTRIBUTE8
About 90% of 2017 applicants submitted scores from the new exam
These data include applicants who applied through AMCAS AND non-AMCAS application services
12%
88%
0%
20%
40%
60%
80%
100%
Took only the old exam Took the new exam
Applicants to the 2017 entering class
Only the new exam
Both the old and newexams
CONFIDENTIAL – DO NOT DISTRIBUTE9 CONFIDENTIAL – DO NOT DISTRIBUTE9 CONFIDENTIAL – DO NOT DISTRIBUTE9
Our admissions committees put new scores in contextPercentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
CONFIDENTIAL – DO NOT DISTRIBUTE10 CONFIDENTIAL – DO NOT DISTRIBUTE10 CONFIDENTIAL – DO NOT DISTRIBUTE10
Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
Some 2017 applicants w/ high UGPAs and MCATs weren’t accepted
11% of applicants with GPAs at or above 3.8 and
MCAT scores at or above 518 were
not admitted into any medical
schools
CONFIDENTIAL – DO NOT DISTRIBUTE11 CONFIDENTIAL – DO NOT DISTRIBUTE11 CONFIDENTIAL – DO NOT DISTRIBUTE11
Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
Other 2017 applicants w/ modest credentials were accepted
14% of applicants with GPAs 3.00 - 3.19
and MCAT scores 498-501 were
admitted into at least one
medical school
CONFIDENTIAL – DO NOT DISTRIBUTE12 CONFIDENTIAL – DO NOT DISTRIBUTE12 CONFIDENTIAL – DO NOT DISTRIBUTE12
Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
42% of 2017 applicants with new scores got one or more acceptances
42% of applicants with
new scores received one or
more acceptances
CONFIDENTIAL – DO NOT DISTRIBUTE13 CONFIDENTIAL – DO NOT DISTRIBUTE13 CONFIDENTIAL – DO NOT DISTRIBUTE13
Interested in learning more about our colleagues’ use of new exam scores?
Strategies for Mission-Oriented Admissions in Light of the New MCAT Exam Monday, November 6 at 3:00 PM ET
Center 304-306
Tanisha Price-Johnson, PhD, University of Arizona-Tuscon College of MedicineTheodore Hall, MD, David Geffen School of Medicine at UCLAAaron Saguil, MD, F. Edward Herbert School of Medicine, Uniformed Services University Sunny Nakae, PhD, Loyola University Chicago Stritch School of MedicineCarol Teitz, MD, University of Washington School of Medicine
CONFIDENTIAL – DO NOT DISTRIBUTE14 CONFIDENTIAL – DO NOT DISTRIBUTE14 CONFIDENTIAL – DO NOT DISTRIBUTE14
What do we know about this year’s test takers (many of whom are in your current
applicant pool)?
CONFIDENTIAL – DO NOT DISTRIBUTE15 CONFIDENTIAL – DO NOT DISTRIBUTE15 CONFIDENTIAL – DO NOT DISTRIBUTE15
3% more examinees tested in 2017 than in 2016
78,410 80,997
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
100,000
2016 2017
Unique Examinees with Scores
CONFIDENTIAL – DO NOT DISTRIBUTE16 CONFIDENTIAL – DO NOT DISTRIBUTE16 CONFIDENTIAL – DO NOT DISTRIBUTE16
Examinees tested in the same numbers and proportions as the past
Percentage of Examinees Taking the New MCAT Exam by Gender, Race/Ethnicity (N = 186,450)
46%
54%
48%
10%
11%
27%
1%
<1%
4%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Male
Female
White
Black or African American
Hispanic
Asian
American Indian or Alaska Native
Native Hawaiian or Other Pacific Islander
Other
Gender
Race/Ethnicity
CONFIDENTIAL – DO NOT DISTRIBUTE17 CONFIDENTIAL – DO NOT DISTRIBUTE17 CONFIDENTIAL – DO NOT DISTRIBUTE17
Percentage of Examinees Taking the New MCAT Exam by Fee Assistance, Parental Education, Testing Condition,
and Repeater Status (N = 186,450)
Examinees tested in the same numbers and proportions as the past
6%
82%
18%
1%
76%
24%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Did not receive
Received
Not first-generation college
First-generation college
Standard
Nonstandard
Non-repeater
Repeater
Fee Assistance
ParentalEducation
Testing Condition
Repeater Status
94%
99%
CONFIDENTIAL – DO NOT DISTRIBUTE18 CONFIDENTIAL – DO NOT DISTRIBUTE18 CONFIDENTIAL – DO NOT DISTRIBUTE18
How did examinees prepare for the new test?
CONFIDENTIAL – DO NOT DISTRIBUTE19 CONFIDENTIAL – DO NOT DISTRIBUTE19 CONFIDENTIAL – DO NOT DISTRIBUTE19
Most examinees completed coursework in the natural, behavioral, and social sciences
87%
84%
50%
42%
83%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Biochemistry
Psychology
Sociology
Research Methods
Statistics
Biology
General Chemistry
Organic Chemistry
Physics
95%
92%
96%
94%
Coursework(N = 69,997)
CONFIDENTIAL – DO NOT DISTRIBUTE20 CONFIDENTIAL – DO NOT DISTRIBUTE20 CONFIDENTIAL – DO NOT DISTRIBUTE20
Many examinees took preparation courses, used Khan tutorials, and used practice materials
41%
6%
52%
26%
75%
52%
22%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Using commercial prep courses
University-based prep courses
Khan Academy
None of the three
Practice exams
Questions packs & section banks
None of the three
Practice Materials(N = 31,723)
Preparation Courses (N = 69,525)
CONFIDENTIAL – DO NOT DISTRIBUTE21 CONFIDENTIAL – DO NOT DISTRIBUTE21 CONFIDENTIAL – DO NOT DISTRIBUTE21
Khan Academy tutorials and some AAMC resources are free• The Khan Academy has over 1,100 free tutorials on exam content Free practice materials and resources are available on AAMC’s website: What’s on the MCAT Exam? Interactive Content Outline Roadmaps to MCAT Content in Biochemistry, Psychology, and
Sociology Textbooks Guide to Creating a Study Plan How I Prepared for the MCAT Exam Testimonials AAMC Pre-Med Navigator
Students-residents.aamc.org/mcatprep
CONFIDENTIAL – DO NOT DISTRIBUTE22 CONFIDENTIAL – DO NOT DISTRIBUTE22 CONFIDENTIAL – DO NOT DISTRIBUTE22
There are a number of low-cost materials
Students-residents.aamc.org/mcatprep
CONFIDENTIAL – DO NOT DISTRIBUTE23 CONFIDENTIAL – DO NOT DISTRIBUTE23 CONFIDENTIAL – DO NOT DISTRIBUTE23
How well did examinees score on the new test?
CONFIDENTIAL – DO NOT DISTRIBUTE24 CONFIDENTIAL – DO NOT DISTRIBUTE24 CONFIDENTIAL – DO NOT DISTRIBUTE24
Interpreting Box-and-Whisker PlotsMCAT total scores for exams administered in 2015-2017
472 476 480 484 488 492 496 500 504 508 512 516 520 524 528
Overall (mean=500.5; N=239,681)
Total Score
90th Percentile10th Percentile
Median = 50th Percentile
25th Percentile 75th Percentile
501
493 508
486 514
CONFIDENTIAL – DO NOT DISTRIBUTE25 CONFIDENTIAL – DO NOT DISTRIBUTE25 CONFIDENTIAL – DO NOT DISTRIBUTE25
There was a wide range of scores overall and w/in group
472 476 480 484 488 492 496 500 504 508 512 516 520 524 528
Total (mean = 500.5; N = 239,681)
Male (mean = 502.1; N = 108,262)
Female (mean = 499.2; N = 130,777)
White (mean = 502.3; N = 105,696)
Black or African American (mean = 493.5; N = 24,559)
Hispanic (mean = 495.7; N = 25,668)
Asian (mean = 502; N = 63,201)
American Indian or Alaska Native (mean = 469.9; N = 2,452)
Native Hawaiian or Other Pacific Islander (mean = 498.6; N = 690)
Total Score
Gender
Race/Ethnicity
Overall
CONFIDENTIAL – DO NOT DISTRIBUTE26 CONFIDENTIAL – DO NOT DISTRIBUTE26 CONFIDENTIAL – DO NOT DISTRIBUTE26
There was a wide range of scores overall and w/in group
472 476 480 484 488 492 496 500 504 508 512 516 520 524 528
Did not receive (mean = 500.8; N = 224,656)
Received (mean = 496.4; N = 15,025)
Not first-generation college (mean = 501.8; N = 124,985)
First-generation college (mean = 496.3; N = 27,493)
Standard (mean = 500.5; N = 237,577)
Nonstandard (mean = 502; N = 2,104)
Non-repeater (mean = 502.6; N = 141,592)
Repeater - 1st attempt (mean = 496.1; N = 44,858)
Repeater - 2nd attempt (mean = 499.1; N = 44,858)
Total Score
Testing Condition
Fee Assistance
Repeater Status
Parental Education
CONFIDENTIAL – DO NOT DISTRIBUTE27 CONFIDENTIAL – DO NOT DISTRIBUTE27 CONFIDENTIAL – DO NOT DISTRIBUTE27
This variation is similar to that on the old exam and other standardized tests Old MCAT exam GRE LSAT GMAT
CONFIDENTIAL – DO NOT DISTRIBUTE28 CONFIDENTIAL – DO NOT DISTRIBUTE28 CONFIDENTIAL – DO NOT DISTRIBUTE28
Academic achievement is affected by multiple factors over a long period of time
Academic Achievement
and Test Scores
Student
Family Community
Prior learning opportunity and experience• K-12 schools• Undergraduate
study
CONFIDENTIAL – DO NOT DISTRIBUTE29 CONFIDENTIAL – DO NOT DISTRIBUTE29 CONFIDENTIAL – DO NOT DISTRIBUTE29
Academic Achievement
and Test Scores
Student
Family Community
Prior learning opportunity and experience• K-12 schools• Undergraduate
study
Students may experience multiple barriers
Food insecurity
Less support for learning at home
Fewer role models and mentors
Fewer experienced
teachers
Dietrichson, J., Martin B., Filges, T., and Jorgensen, AMK (2017). Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. Review of Educational Research. 87, 243-283.
CONFIDENTIAL – DO NOT DISTRIBUTE30 CONFIDENTIAL – DO NOT DISTRIBUTE30 CONFIDENTIAL – DO NOT DISTRIBUTE30
With this in mind, what can we do about differences? Look for points of leverage to improve equity in access for all
students Monitor data on academic preparation, diversity, and
performance Expand outreach to faculty at under-resourced and minority-
serving institutions Expand outreach directly to students from sociodemographic
groups underrepresented in medicine
Example: Expanding outreach to faculty at under-resourced and minority-serving institutionsDeveloped new contacts at minority-serving and/or under-resourced colleges and universities—adding 350 contacts from:• Almost 90 HBCUs• More than 200 Hispanic-serving
institutions• More than 30 Tribal colleges and
universities
Please encourage your colleagues to add their names to our mailing list. They can contact us at [email protected]
If you’re interested in contributing to the Pre-Med Navigator, please don’t forget to sign up before you leave.31
CONFIDENTIAL – DO NOT DISTRIBUTE32 CONFIDENTIAL – DO NOT DISTRIBUTE32 CONFIDENTIAL – DO NOT DISTRIBUTE32
Example: Expanding Outreach Directly to Students from Sociodemographic Groups Underrepresented in Medicine Reaching out to students directly through emails and
newsletters Health professions career fairs focused on minority populations
(e.g., UC Davis Pre-Health Conference, Stanford University Minority Medical Alliance Conference)
Virtual events like Health Professions Week Virtual Fair and AAMC Medical School Virtual Fair
Monitoring students’ use of the MCAT collection of the Khan Academy and other free and low-cost materials to learn if better outreach about available resources is needed
CONFIDENTIAL – DO NOT DISTRIBUTE33 CONFIDENTIAL – DO NOT DISTRIBUTE33 CONFIDENTIAL – DO NOT DISTRIBUTE33
This topic is of great concern to the MCAT Validity Committee
The committee is conducting qualitative and quantitative research to understand students’ preparation strategies and barriers What is easy and difficult for examinees when they prepare for
the MCAT exam? What is easy and difficult about using the AAMC’s free and low-
cost materials to prepare for the MCAT exam? Are these barriers different or greater for those from
sociodemographic groups underrepresented in medicine? What additional resources and information do examinees and
their advisors need?
CONFIDENTIAL – DO NOT DISTRIBUTE34 CONFIDENTIAL – DO NOT DISTRIBUTE34 CONFIDENTIAL – DO NOT DISTRIBUTE34
Interview Students
andAdvisors
Revise PMQ
Administer PMQ and Compare
Results for Examinees
from Different Groups
Learn what barriers
exist
Develop new
information and
resources
Results will help us improve access to information and resources
CONFIDENTIAL – DO NOT DISTRIBUTE35 CONFIDENTIAL – DO NOT DISTRIBUTE35 CONFIDENTIAL – DO NOT DISTRIBUTE35
What are we learning about the impact, use, and predictive validity of the new exam?
CONFIDENTIAL – DO NOT DISTRIBUTE36 CONFIDENTIAL – DO NOT DISTRIBUTE36 CONFIDENTIAL – DO NOT DISTRIBUTE36
21 medical schools are working together to evaluate the new exam
CONFIDENTIAL – DO NOT DISTRIBUTE37 CONFIDENTIAL – DO NOT DISTRIBUTE37 CONFIDENTIAL – DO NOT DISTRIBUTE37
The MCAT validity research addresses multiple goals Provides evidence about the value of the new
MCAT exam in admissions decisions Answers questions about the fairness and
consequences of introducing the new MCAT exam for examinees, applicants, and medical students
Presents data to admissions officers that they can act on to improve their admissions decisions
Uses findings about the needs of aspiring physicians from underrepresented backgrounds to improve test preparation resources and outreach
CONFIDENTIAL – DO NOT DISTRIBUTE38 CONFIDENTIAL – DO NOT DISTRIBUTE38 CONFIDENTIAL – DO NOT DISTRIBUTE38
The MCAT validity agenda includes three broad research topics
Academic Preparation,
Diversity, and Fairness
Admissions Decision Making
Predicting Academic
Performance
CONFIDENTIAL – DO NOT DISTRIBUTE39 CONFIDENTIAL – DO NOT DISTRIBUTE39 CONFIDENTIAL – DO NOT DISTRIBUTE39
Let’s zoom in on the predictive validity research
Academic Preparation,
Diversity, and Fairness
Admissions Decision Making
Predicting Academic
Performance
CONFIDENTIAL – DO NOT DISTRIBUTE40 CONFIDENTIAL – DO NOT DISTRIBUTE40 CONFIDENTIAL – DO NOT DISTRIBUTE40
How well did the new scores predict students’ academic performance in the first year of
medical school?
CONFIDENTIAL – DO NOT DISTRIBUTE41 CONFIDENTIAL – DO NOT DISTRIBUTE41 CONFIDENTIAL – DO NOT DISTRIBUTE41
Nationally, 2016 entrants w/ a wide range of scores progressed to M2 w/out delay
% S
tude
nts
Prog
ress
ing
to M
2
MCAT Total Score Range
50%
80% 81%
93% 94% 96% 97% 98% 98% 99%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528
% of 2016 Entrants Who Progressed to M2 Without Delay
(N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778)
Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis.
CONFIDENTIAL – DO NOT DISTRIBUTE42 CONFIDENTIAL – DO NOT DISTRIBUTE42 CONFIDENTIAL – DO NOT DISTRIBUTE42
At the validity schools, new MCAT scores predict performance across M1 courses
r Effect Size0.1 Small0.3 Medium0.5 Large
0.43
0.69
0.55
00.
20.
40.
60.
81.
0C
orre
cted
Cor
rela
tion
Overall(NSchool = 15)
Medium Effect
Correlation of MCAT Total Scores with Performance Across M1 Courses: Median and Interquartile Range
Overall, the associations of the new MCAT total scores with performance across M1 courses are medium to large
CONFIDENTIAL – DO NOT DISTRIBUTE43 CONFIDENTIAL – DO NOT DISTRIBUTE43 CONFIDENTIAL – DO NOT DISTRIBUTE43
These validities compare well to those for other exams
Author(Year) Exam
Type of ExamScore
Type of Outcome
Median Validity
CoefficientUnit of
AnalysisAAMC (2017) New Exam Total Score First-Year Course Average .55 School
(Nschool=15)AAMC (2017) Old MCAT Total score First-Year Course Average .53 School
(Nschool=17)Kuncel, et al. (2007) GMAT Total score First-Year Graduate GPA .47 Meta-
analysis of multiple studies
Stilwell, et al. (2011) LSAT Total score First-Year Average Course Grade .58 School(Nschool=170)
CONFIDENTIAL – DO NOT DISTRIBUTE44 CONFIDENTIAL – DO NOT DISTRIBUTE44 CONFIDENTIAL – DO NOT DISTRIBUTE44
New scores predict equally well for students from different sociodemographic backgrounds Research studied these early relationships for students grouped
by: Race/ethnicity Socioeconomic status
So far, MCAT scores neither over- nor under-predict the performance of students from underrepresented backgrounds based on two types of performance outcomes: National outcome: Progression to M2 without delay Validity school outcome: Average performance across M1
coursesWe have a lot more to learn about how students do in their second year, in their clerkships
and on their USMLE exams, and their graduation from undergraduate medical school
Each year, we’ll learn more about the new exam
Applicants/Matriculants
Year 1Coursework
Year 2Coursework
USMLEStep 1
Clerkships
Graduate in4 Years
Graduate in5 Years
USMLE Step 2-CK
USMLE Step 2 Exams
2015 2016 2017 2019 2020 20212018
2016 2017 2018 2019 2020 2021 2022
2017 2019 2020 2021 2022 20232018
2018 2019 2020 2021 2022 2023 2024
Examinees
CONFIDENTIAL – DO NOT DISTRIBUTE46 CONFIDENTIAL – DO NOT DISTRIBUTE46 CONFIDENTIAL – DO NOT DISTRIBUTE46
Questions?