Identifying Predictors of Performance on USMLE® Step 1

BACKGROUND: USMLE® Step 1 is considered by residency program directors to be one of the most important factors in selecting medical students for interviews, so it is important for both students and medical schools to maximize scores. However, despite its importance, very little has been studied to determine indicators of performance on the exam. OBJECTIVE: The goals of this study are: 1) to determine if data available prior to admission can predict Step 1 performance, 2) to evaluate if success in pre-clinical courses at UT Southwestern correlates to success on Step 1, 3) to analyze survey data and determine correlations between studying resources/strategies and Step 1 scores, and 4) to develop a mathematical model to flag students at risk of scoring poorly on Step 1. METHODS: This study utilized data from the UT Southwestern Medical School Class of 2018 (n=238). First, Undergraduate GPA, MCAT® scores, and demographic information were correlated with Step 1 scores and medical school grades to determine if pre-admissions factors could predict Step 1 or medical school performance. Second, medical school exam scores from the second pre-clinical year and NBME® Comprehensive Basic Science Self-Assessment (CBSSA) scores were analyzed to determine their correlation with Step 1 scores. Third, the class was surveyed during a dedicated six-week study period before the exam as well as after the exam, and results were analyzed to determine how students prepared. Each question from the surveys was correlated with exam scores to identify which factors led to higher test scores. Finally, the factors with the highest correlations to Step 1 scores were used to develop a mathematical model to predict Step 1 scores using multiple linear regression. This model was then tested to determine its effectiveness at identifying at-risk students. RESULTS: MCAT® biological and physical sciences scores and undergraduate GPA had moderate correlation with Step 1 scores (both R2 = 0.10) and weak correlation with medical school grades (R2 = 0.060 and 0.058, respectively). Of all factors studied, the initial CBSSA scores had the highest predictive value of Step 1 scores (R2 = 0.60). Cumulative medical school grades were also highly correlated with Step 1 scores (R2 = 0.52). The weekly pre-exam surveys indicate that each successive week of study produced smaller gains in points on Step 1, and most students did not see significant point increases after 6 weeks of study. The post-exam survey shows that students who answered greater than 4000 practice questions scored an average of 254 ± 5.3 (95% CI), whereas those who answered fewer than 1700 questions scored an average of 230 ± 10.6 (95% CI). There was no significant difference between those who studied 4 versus 7 dedicated weeks, or those who studied fewer than 250 hours versus more than 600 hours during the dedicated study time. However, students who started to study 6 months prior to the exam scored 252 ± 6.9 (95% CI), whereas students who waited until the dedicated preparation time scored 237 ± 3.6 (95% CI). A model incorporating a pre-admissions and medical school performance factors was developed that accounted for 64% of the score variability, with a standard error of ±8.87. When used to identify those at risk of scoring below 220, the model had a sensitivity of 81% and specificity of 86%. CONCLUSION: The study finds that MCAT® and undergraduate GPA are mediocre determinants of medical school and USMLE performance and should be used cautiously in the admissions process. Conversely, medical school grades and CBSSA scores are very accurate predictors of Step 1 scores, and students who do well on medical school exams generally do well on Step 1. Students should plan five to six weeks for dedicated study, but start reviewing material as early as possible and focus on question-based resources. The model developed from this data can be a useful tool to identify at-risk students for early intervention. This project demonstrates that objective data analysis can be used to guide students towards optimal preparation for Step 1, as well as identify those at risk of performing poorly on the exam.