Catherine Lucey, MD Vice Dean for Education University of California, San Francisco School of Medicine

Evaluating the Fairness, Impact, Use and Predictive Validity of the New MCAT Exam Catherine Lucey, MD Vice Dean for Education University of California, San Francisco School of Medicine Joshua Hanson, MD, MPH Assistant Dean for Student Affairs, Assistant Professor The University of Texas School of Medicine at San Antonio Carol Terregino, MD Senior Associate Dean for Education, Associate Dean for Admissions; Professor of Medicine Rutgers Robert Wood Johnson Medical School Daniel Clinchot, MD Associate Vice President for Health Sciences Education; Chair, Department of Education and Anatomy The Ohio State University College of Medicine

Diversity is a Core Driver of High Quality Health Care Minority physicians are significantly more likely to practice primary care and in underserved areas Similar race/ethnicity identification between physicians and patients results in longer visits and increased patient satisfaction Communicating in patients native language often improves adherence to treatment Medical students uniformly agree that a diverse student body improves the experience of medical school; enhances classroom discussions Black/African-Americans comprise only 4 percent of the physician workforce The majority of younger minority physicians in the workforce are women Whitla DK et al. Acad Med. 2003; 78: 460 2 Hung R et al. Acad Med.2007; 82(2):184 AAMC Diversity in the Physician Workforce Facts and Figures 2014

70% 60% 50% Slightly More Than Half of 2016 Matriculants Self- Identified as White Percentage of U.S. Medical School Matriculants by Race/Ethnicity, 2016 51.5% 40% 30% 20% 10% 0% 0.3% American Indian or Alaska Native 21.3% Asian 7.1% 6.3% Black or African American Hispanic or Latino 0.1% Native Hawaiian/ Other Other Pacific Pacific Islander Islander White 8.8% More than One Race 1.7% 1.6% 1.3% Other Unknown Race/ Ethnicity Non-U.S. Citizen/ Non-Perm. Resident Note: The Non-U.S. Citizen and Non-Permanent Resident category may include students with unknown citizenship. Matriculants who declined to report sex are not reflected. Students who reported more than one race/ethnicity are included under More than One Race. During the 2016 application cycle, a technical malfunction in 3 the collection of race/ethnicity data necessitated a request that applicants CONFIDENTIAL review and DO re-submit NOT DISTRIBUTE responses to the race/ethnicity question in their AMCAS applications. Applicants in previous years were not asked to review responses to this question.

The Path to Becoming a Physician Starts Early Pre-College Experiences Early environment Family support K-12 education Extracurricular experiences Premedical Preparation # colleges attended Quality of instruction Coursework Extracurricular experiences Medical School Admissions Holistic review (Including MCAT) Diverse class Medical Education Family and financial support Academic and other support Culture and climate Residency Program Academic and other support Culture and climate Residency Selection Holistic review (Including USMLE) Diverse cohort 4

Group Differences in Academic Achievement Are Associated with Societal Inequalities MCAT, LSAT, GRE, GMAT, and other exams show population group differences Presence of differences does not equate to test bias (i.e., construct irrelevant content or alterations in administration) Structural racism and privilege likely contribute to the differences seen across the spectrum of exams Compared with majority examinees, minority examinees: More likely to experience adverse environmental factors (poverty, food insecurity, low quality day care) More likely to have had disrupted or low quality K-12 education Less likely to have high quality exam prep experiences or advising experiences in college Davis D et al. Acad Med. 2013;88:593 5 Dietrichson, J., Martin B., Filges, T., and Jorgensen, CONFIDENTIAL AMK (2017). Academic DO NOT interventions DISTRIBUTEfor elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. Review of Educational Research. 87, 243-283.

Fairness was Front and Center in Designing and Developing the New Exam Type of Fairness Societal Fairness Procedural Fairness Exam Fairness Definition Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam. Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions. MCAT scores have the same meaning and predict student performance equally well for examinees from different backgrounds. Influence on New MCAT Exam Blueprints, test preparation resources New score scales, score reports, and resources for admissions officers Item development/review, test form development 6

Overview of the MCAT Validity Research Program 7

21 Medical Schools and 2 Pre-Health Advisors Are Working Together to Evaluate the New Exam 8

The MCAT Validity Research Program is Complex Multiple research questions in three broad areas Qualitative and quantitative data from Examinees Applicants Medical students Medical schools Multiple methods and data collection designs (longitudinal, cross sectional) The validity study will last about nine years (2014 to 2023) 9

The MCAT Validity Research Addresses Multiple Goals Provides evidence about the value of the new MCAT exam in admissions decisions Answers questions about the fairness and consequences of introducing the new MCAT exam for examinees, applicants, and medical students Presents data to admissions officers that they can act on to improve their admissions decisions Uses findings about the needs of aspiring physicians from underrepresented backgrounds to improve test preparation resources and outreach 10

The MCAT Validity Research Agenda Includes Three Broad Areas Predicting Medical Student Performance Admissions Decision Making Academic Preparation, Diversity, and Fairness 11

We re Testing 3 Hypotheses about Predicting Medical Student Performance Evaluating the predictive validity of the new exam Examining the predictive validity of the newest test section Comparing the predictive validity of the new MCAT exam to other predictors Do scores from the new exam correlate with academic performance throughout medical school? Will scores from the Psychological, Social, and Biological Foundations of Behavior section correlate with performance in medical school courses that call on the behavioral and social sciences better than section scores from the old exam? Do scores from the new exam add value to the academic information applicants already provide about themselves through applications and transcripts? 12

We re Testing 2 Hypotheses about Admissions Decision Making Acceptance of a wide range scores Use of section scores Will medical schools increase the percentage of applicants with total scores in the middle of the MCAT score scale who are invited to interview and receive acceptance offers? Will admissions committees use information about applicants strengths and weaknesses from the MCAT score reports to identify applicants who best fit their academic missions and goals? 13

We re Testing 9 Hypotheses about Academic Preparation, Diversity, and Fairness, For Example: Change in breadth of academic preparation Will more individuals learn about psychology, sociology, and biochemistry in preparation for the MCAT exam? What resources, information, and outreach will provide equity in access for students from sociodemographic groups underrepresented in medicine? Diversity of aspiring physicians Will the diversity of examinees, applicants, and medical students who took the new exam change? Fairness in score meaning Will scores from the new exam predict academic performance equally well for medical students from different racial, ethnic, or disadvantaged backgrounds? 14

Today s Presentation Will Focus on Three Questions How well did scores from the new MCAT exam predict students performance across the first year of medical school? How did admissions officers work with new MCAT scores in 2017 admissions decisions? What can we learn about preparation resources needed by examinees from underrepresented backgrounds? 15

16 How well did scores from the new MCAT exam predict students performance across the first year of medical school?

The Predictive Validity Research Includes National and Validity School Data (2015-2017 Entrants) National Population National Population with New Scores Validity School Participants 17

MCAT Scores Will Be Correlated with Multiple Types of Student Performance Outcomes Holistic Medical Student Academic Performance Academic Readiness to Graduate Holistic Outcomes Academic Readiness for Clerkships Year-End Academic Performance Course-Based Academic Performance Granular Outcomes 18

Some Analyses Will Include Outcomes from the National Population National Outcomes Progression to M2, M3 USMLE Step exams Graduation Holistic Medical Student Academic Performance Academic Readiness to Graduate Academic Readiness for Clerkships Year-End Academic Performance Course-Based Academic Performance Holistic Outcomes Granular Outcomes 19

Other Analyses Will Include Outcomes from the Validity Schools Outcomes from Validity Schools Course grades NBME Subject exams Clerkship grades GPA/Class Rank Holistic Medical Student Academic Performance Academic Readiness to Graduate Academic Readiness for Clerkships Year-End Academic Performance Course-Based Academic Performance Holistic Outcomes Granular Outcomes 20

Today s findings are from the 2016 entrants with new scores 2016 Matriculants N = 21,030 Matriculants w/ new scores N=8,521 Participants N=1,027 w/ new scores 21

2016 Matriculants and Validity School Participants with New Scores Are Similar to the 2016 Matriculant Population 2016 Matriculants N = 21,030 Matriculants w/ new scores N=8,521 Participants N=1,027 w/ new scores Demographics Fee assistance status Parental education/occupation Race/ethnicity Academic metrics MCAT total and section scores Undergraduate GPAs 22

Today s Results Include Two M1 Year-End Outcomes Holistic Medical Student Academic Performance Holistic Outcomes National Outcome Progression to M2 without delay Outcome from Validity Schools Performance Across M1 Courses Academic Readiness to Graduate Academic Readiness for Clerkships Year-End Academic Performance Course-Based Academic Performance Granular Outcomes 23

Nationally, 2016 Entrants w/ a Wide Range of Scores Progressed to M2 Without Delay % Students Progressing to M2 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% % of 2016 Entrants Who Progressed to M2 Without Delay 93% 94% 96% 97% 98% 98% 99% 80% 81% 50% 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 (N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778) MCAT Total Score Range Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis. 24

At the Validity Schools, New Scores Predict Performance across M1 Courses Corrected Correlation 0 0.2 0.4 0.6 0.8 1.0 Correlation of MCAT Total Scores with Performance Across M1 Courses: Median and Interquartile Range 0.55 0.69 0.43 Overall (N School = 15) Medium Effect 25 Overall, the associations of the new MCAT total scores with performance across M1 courses are medium to large r Effect Size 0.1 Small 0.3 Medium 0.5 Large Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. doi:10.1037/0033-2909.112.1.155

These Validities Compare Well to Those for Other Exams Author (Year) Exam Type of Exam Score Type of Outcome Median Validity Coefficient Unit of Analysis AAMC (2017) New Exam Total Score First-Year Course Average.55 School (N school =15) AAMC (2017) Old MCAT Total score First-Year Course Average.53 School (N school =17) Kuncel, et al. (2007) GMAT Total score First-Year Graduate GPA.47 Metaanalysis of multiple studies Stilwell, et al. (2011) LSAT Total score First-Year Average Course Grade.58 School (N school =170) 26

At the Validity Schools, We Also Analyzed Outcomes from Individual M1 Courses Holistic Medical Student Academic Performance Holistic Outcomes Academic Readiness to Graduate Academic Readiness for Clerkships Year-End Academic Performance Course-Based Academic Performance Granular Outcomes 27

At the Validity Schools, New Scores Predict Performance in Individual M1 Courses Corrected Correlation 0 0.2 0.4 0.6 0.8 1.0 Correlation of MCAT Total Scores with Performance in Individual M1 Courses: Median and Interquartile Range 0.51 0.61 0.40 Medium Effect Overall, the associations of the new MCAT total scores with performance in M1 courses are medium to large r Effect Size 0.1 Small 0.3 Medium 0.5 Large (N Course = 109) 28

At the Validity Schools, New Scores Predict Performance in Courses about Foundational Sciences and Body Systems Corrected Correlation 0 0.2 0.4 0.6 0.8 1.0 Correlation of MCAT Total Scores with Performance in Individual M1 Courses by Course Content Area: Median and Interquartile Range 0.61 0.62 0.64 0.51 0.51 0.50 Overall (N Course = 109) 0.40 0.40 0.40 Medium Effect General Principles of Foundational Science (N Course = 66) Body Systems (N Course = 46) r Effect Size 0.1 Small 0.3 Medium 0.5 Large Course Content Area 29

New Scores Predict Equally Well for Students from Different Sociodemographic Backgrounds Research studied these early relationships for students grouped by: Race/ethnicity Socioeconomic status So far, MCAT scores neither over- nor under-predict the performance of students from underrepresented backgrounds based on two types of performance outcomes: National outcome: Progression to M2 without delay Validity school outcome: Performance across M1 courses 30

Predictive Validity Findings Will Be Reported from 2017 through 2022 for Medical Students Who Took the New Exam We have a lot more to learn about how students do in their second year, in their clerkships and on their USMLE exams, and their graduation from undergraduate medical school 2016 Entrants 2017 Entrants 2018 2019 2020 2021 2022 31

32 How did admissions officers work with new MCAT scores in 2017 admissions decisions?

Admissions Committees Put MCAT Scores in Context There are more qualified applicants to medical school than there are places in medical school Medical schools weigh information about attributes, experiences, and academic metrics in deciding whom to admit Applicants from sociodemographic groups underrepresented in medicine obtain lower mean scores on the MCAT exam and lower undergraduate GPAs than applicants from other groups Medical schools aim to select students who will succeed and who will help build a diverse physician workforce 33

The New Score Scales Draw Attention to the Center of the Scale Because on the old exam, students who entered medical school with scores in the center of the MCAT score scale succeeded The new scale uses a nice round number at 500 to draw attention to applicants who might otherwise be overlooked The new test also includes more questions per section, providing better information about examinees strengths and weaknesses on the exam The new score reports use confidence bands to describe measurement precision and score profiles to describe strengths and weaknesses 34

Admissions Committees Put MCAT Scores in Context in 2017 Selection Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range New MCAT Total GPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All 3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67% 2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,547 3.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48% 0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,405 3.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33% 4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,812 3.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22% 0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,646 3.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15% 0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,209 2.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10% 2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,608 2.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6% 0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/868 2.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5% 0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/398 2.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5% 0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/192 2.00-2.19 0% 0% 0% - - - - - 0% 0/41 0/13 0/10 0/86 <2.00 0% - - - - 3% 0/15 1/31 All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42% 35 8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

Some 2017 Applicants w/ High UGPAs and MCATs Weren t Accepted Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range New MCAT Total GPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All 3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67% 2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,547 3.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48% 0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,405 3.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33% 4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 11% of 694/1,260 applicants 405/660 235/329 2,868/8,812 3.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22% 0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 with 255/631 GPAs at 176/314 or 63/109 1,256/5,646 3.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15% above 3.8 and 0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,209 2.80-2.99 1% 1% 2% 5% 6% 17% 25% MCAT 30% scores at 29% or 42% 10% 2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,608 2.60-2.79 0% 1% 0% 2% 6% 18% 18% above 17% 518 were 40% - 6% 0/157 1/117 0/142 3/132 8/126 14/79 10/57 not admitted 6/36 into 6/15 53/868 2.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5% 0/118 1/57 1/48 3/59 1/40 4/30 6/24 any 4/17medical 21/398 2.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5% schools 0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/192 2.00-2.19 0% 0% 0% - - - - - 0% 0/41 0/13 0/10 0/86 <2.00 0% - - - - 3% 0/15 1/31 All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42% 36 8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

Other 2017 Applicants w/ Modest Credentials Were Accepted Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range New MCAT Total GPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All 3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67% 2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,547 3.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48% 0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,405 3.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33% 4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,812 3.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22% 0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,646 3.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15% 0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,209 2.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10% 14% of applicants 2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,608 2.60-2.79 0% with 1% GPAs 0% 3.00-2% 3.19 6% 18% 18% 17% 40% - 6% 0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/868 2.40-2.59 0% 2% and 2% 5% 3% 13% 25% 24% - - 5% 2.20-2.39 0/118 1/57 MCAT 1/48 scores 3/59 1/40 4/30 6/24 4/17 21/398 0% 0% 0% 0% 12% 33% 8% - - 5% 0/69 0/29 498-501 0/24 were 0/21 2/17 4/12 1/12 9/192 2.00-2.19 0% 0% 0% - - - - - 0% admitted into at 0/41 0/13 0/10 0/86 <2.00 0% least - one - medical - - 3% 0/15 1/31 All 1% 1% school 4% 11% 22% 36% 50% 64% 75% 84% 42% 37 8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

Progression of 2016 Entrants to M2 Without Delay, By New MCAT Total Score Range % Students Progressing to M2 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% % of 2016 Entrants Who Progressed to M2 Without Delay 93% 94% 96% 97% 98% 98% 99% 80% 81% 50% 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 (N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778) MCAT Total Score Range Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis. 38

Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range New MCAT Total GPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All 3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67% 2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,547 3.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48% 0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,405 3.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33% 4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,812 3.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22% 0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,646 3.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15% 0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,209 2.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10% 2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,608 2.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6% 0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/868 2.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5% 0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/398 2.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5% 0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/192 2.00-2.19 0% 0% 0% - - - - - 0% 0/41 0/13 0/10 0/86 <2.00 0% - - - - 3% 0/15 1/31 All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42% 8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802

Let s put these two data sources together 40

Early Data Suggest Students with Scores in the Middle of the New Scale Do Well % of 2016 Entering Medical Students Who Progressed to M2 Without Delay, and % of 2017 Applicants Accepted to Medical School 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 93% 94% 96% 97% 98% 98% 99% 80% 81% 84% 75% 64% 50% 50% 36% 22% 11% 1% 1% 4% 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 Progression to M2 (2016 Entrants) Acceptance Rates (2017 Applicants) 41

As a community, should we ask ourselves if there is more we want to do? 42

As a Community, Should We Ask Ourselves if There Is More We Want to do? Admissions committees identify the applicants with the ingredients for success, and they support these students once they re admitted Are we missing some opportunities to pay more attention to students with scores in the middle of the MCAT score scale? There is still a lot to learn about students performance in the second year and in their clerkships On the new exam, findings so far suggest that, just like they did with the old exam, students admitted with scores in the middle of the new MCAT score scale do well 43

Interested in Learning More about Your Colleagues Use of New Exam Scores? Strategies for Mission-Oriented Admissions in Light of the New MCAT Exam Monday, November 6 at 3:00 PM ET Center 304-306 Tanisha Price-Johnson, PhD, University of Arizona-Tuscon College of Medicine Theodore Hall, MD, David Geffen School of Medicine at UCLA Aaron Saguil, MD, F. Edward Herbert School of Medicine, Uniformed Services University Sunny Nakae, PhD, Loyola University Chicago Stritch School of Medicine Carol Teitz, MD, University of Washington School of Medicine 44

45 What can we learn about preparation resources needed by examinees from sociodemographic groups underrepresented in medicine?

The New MCAT Exam Was Designed to Increase Access Type of Fairness Societal Fairness Procedural Fairness Exam Fairness Definition Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam. Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions. MCAT scores have the same meaning and predict student performance equally well for examinees from different backgrounds. Influence on New MCAT Exam Blueprints, test preparation resources New score scales, score reports, and resources for admissions officers Item development/review, test form development 46

The New Test Blueprints Were Developed to Advance Societal Fairness Test concepts widely taught at undergraduate institutions, including minority-serving and under-resourced institutions Test psychology and sociology concepts like discrimination, stereotype threat, and socio-economic inequalities Pay increased attention to population health, studies of diverse cultures, and ethics Balance the percentage of questions devoted to natural sciences concepts with the percentage devoted to behavioral and social sciences concepts and information processing 47

Preparation Resources Were Developed to Advance Societal Fairness The Khan Academy has over 1,100 free tutorials on exam content Practice materials and resources are available on AAMC s website: What s on the MCAT Exam? Interactive Content Outline Roadmaps to MCAT Content in Biochemistry, Psychology, and Sociology Textbooks Guide to Creating a Study Plan How I Prepared for the MCAT Exam Testimonials Practice tests and question banks Students-residents.aamc.org/mcatprep 48

New Outreach Strategies Address Societal Fairness Expand outreach directly to students from sociodemographic groups underrepresented in medicine Expand outreach to students through their advisors, with a particular focus on faculty at under-resourced institutions Distribute a monthly newsletter Premed Navigator with important information, resources, and tips Work with prehealth advisors on the MCAT Validity Committee to share findings and promote resources, such as the Find an Advisor resource for students at schools with no access to an advisor (volunteer.advisor@naahp.org) 49

Societal Fairness: Expanding Outreach to Faculty at Under-resourced and Minority-Serving Institutions Developed new contacts at minority-serving and/or underresourced colleges and universities adding 350 contacts from: Almost 90 HBCUs More than 200 Hispanic-serving institutions More than 30 Tribal colleges and universities Add to the list multiple times/year Please encourage your advisor colleagues to add their names to our mailing list. They can contact us at advisors@aamc.org 50

Societal Fairness: Expanding Outreach Directly to Students from Sociodemographic Groups Underrepresented in Medicine Reach out to students directly through emails and newsletters Participate in health professions career fairs focused on minority populations (e.g., UC Davis Pre-Health Conference, Stanford University Minority Medical Alliance Conference) Participate in virtual events like Health Professions Week Virtual Fair and AAMC Medical School Virtual Fair Monitor students use of MCAT preparation products to determine if better outreach about available resources is needed 51

Despite This Work, Group Differences Still Exist The group differences in scores on the new exam are similar to those on the old exam They are similar to those on other standardized tests: o GRE o LSAT o GMAT 52

Now We re Studying How Students Prepare for the MCAT Exam Pre-College Experiences Early environment Family support K-12 education Extracurricular experiences Premedical Preparation # colleges attended Quality of instruction Coursework Extracurricular experiences Medical School Admissions Holistic review (Including MCAT) Diverse class Medical Education Family and financial support Academic and other support Culture and climate Residency Program Academic and other support Culture and climate Residency Selection Holistic review (Including USMLE) Diverse cohort 53

Review Courses Are Used by Proportionately Fewer Examinees from Underrepresented Backgrounds Use of Preparation Courses and Khan to Prepare for the Exam By URiM Status (Race/Ethnicity) Use of Preparation Courses and Khan to Prepare for the Exam By First-Generation College Status 0 20 40 60 80 100 43% 40% Commercial Prep. Courses 6% 9% University-Based Prep. Courses 52% 51% Khan Academy 25% 27% None of the Three 0 20 40 60 80 100 44% 33% Commercial Prep. Courses 7% 7% University-Based Prep. Courses 52% 52% Khan Academy 24% 30% None of the Three Non-URiM URiM Non-First Generation College Students First-Generation College Students 54 Note. Data came from examinees who took the CONFIDENTIAL MCAT exam and DO in NOT 2017. DISTRIBUTE

Practice Resources Are Used by Proportionately Fewer Examinees from Underrepresented Backgrounds Use of AAMC Resources to Practice for the Exam By URiM Status (Race/Ethnicity) Use of AAMC Resources to Practice for the Exam By First-Generation College Status 0 20 40 60 80 100 76% 71% Practice Exams 53% 51% Questions Packs & Section Banks 21% 24% None of the Two 0 20 40 60 80 100 75% 73% Practice Exams 53% 53% Questions Packs & Section Banks 21% 23% None of the Two Non-URiM URiM Non-First Generation College Students First-Generation College Students Note. Data came from examinees who took the MCAT CONFIDENTIAL exam and 55 responded DO NOT DISTRIBUTE to the Post-MCAT Questionnaire in 2017.

Gaps in the Use of Review Courses and Practice Resources Are Larger for Those Underrepresented on Both Race/Ethnicity and Parental Education Use of Preparation Courses and Khan to Prepare for the Exam By URiM Status (Race/Ethnicity) and First-Generation College Status Use of AAMC Resources to Practice for the Exam By URiM Status (Race/Ethnicity) and First-Generation College Status 0 20 40 60 80 100 44% 33% Commercial Prep. Courses 6% 9% University-Based Prep. Courses 52% 50% Khan Academy 24% 31% None of the Three 0 20 40 60 80 100 76% 71% Practice Exams 53% 52% Questions Packs & Section Banks 21% 24% None of the Two Neither URiM Nor First-Generation College Students URiM and First-Generation College Students Neither URiM Nor First-Generation College Students URiM and First-Generation College Students Note. Data came from examinees who took the MCAT CONFIDENTIAL exam and 56 responded DO NOT DISTRIBUTE to the Post-MCAT Questionnaire in 2017.

We are Conducting Qualitative and Quantitative Research to Understand Students Preparation Strategies and Barriers What is easy and difficult for examinees when they prepare for the MCAT exam? What is easy and difficult about using the AAMC s free and lowcost materials to prepare for the MCAT exam? Are these barriers different or greater for those from sociodemographic groups underrepresented in medicine? What additional resources and information do examinees and their advisors need? 57

Findings Will Be Used to Improve Access to Resources and Information Interview Students and Advisors Revise PMQ Administer PMQ and Compare Results for Examinees from Different Groups Learn what barriers exist Develop new information and resources 58

Questions? Stay tuned for new research findings aamc.org/validitycommittee 59

Don t forget to join us tomorrow! The New MCAT Exam: Where We Are and Where We re Going Time: 7:00 AM - 8:30 AM Location: Center Ballroom C 60

References AAMC. (March, 2017). Predicting medical student performance. Presented at the MCAT Validity Committee meeting at the Association of American Medical Colleges in Washington, D.C. Kuncel, N., Credé, M., & Thomas, L. (2007). A meta-analysis of the predictive validity of the graduate management admission test (GMAT) and undergraduate grade point average (UGPA) for graduate student academic performance. Academy of Management Learning & Education, 6(1), 51-68. Retrieved from http://www.jstor.org/stable/40214516 Stilwell, L. A., Reese, L. M., & Diamond-Dalessandro, S. P. (2011). Predictive validity of the LSAT: A national summary of the 2009 2010 LSAT correlation studies. LSAT Technical Report Series, 11(2), 1-19. Retrieved from https://www.lsac.org/docs/default-source/research-(lsac-resources)/tr-11-02.pdf 61