We developed a formula for predicting the pass rate in the NMLE. Using this formula, we evaluated a new cohort of students in 2018 and predicted 12 students who had a higher risk of failing the NMLE. After guidance by faculty members, 7 of the 12 students passed the NMLE and the actual pass rate improved.
Predictors for passing the NMLE
We identified four significant internal predictors for passing the NMLE: 1) total score in pre-clinical medical sciences in the third and fourth years, 2) CBT-IRT score in the fourth year, 3) performance in clinical clerkship in the fifth and sixth years, and 4) score in the graduation examination in the sixth year. We also identified two external predictors: age at admission and HS located in surrounding area.
Among them, CBT is a nationwide examination administered by the Common Achievement Tests Organization19 for medical students in all Japanese medical schools before clinical clerkship using a computer to estimate the student’s knowledge for the clinical clerkship. CBT corresponds to Step 1 of USMLE, and a number of studies on risk factors and outcome for those who failed Step 1 2-5,20,21 and studies investigating Step 1 score as one of the predictors of performance after Step 1.8-11,22-26 The latter may be correlated to our result for CBT. Most studies including the study by Koenig et al.22 have indicated that a high score in Step 1 is a predictor of success in many fields in the medical profession (i.e., internal medicine, dermatology, ophthalmology, orthopedic surgery, gynecology, and family medicine),8-11,23-25 with some opposite results 27,28. Casey et al. 26 noted that the medical college aptitude test (MCAT), Step 1 and Step 2, and subsequent clinical performance parameters correlated with NBME scores across all core clinical clerkships. They also emphasized that Step 1 scores identified students at risk of poor performance in NBME subject examinations, facilitating and supporting implementation of remediation before clinical years.26 Accordingly, it is very reasonable to assume that the score of CBT which is compatible with Step 1 is one of the predictors of passing the NMLE in Japan which is compatible with NBME.
In the present study, additional three other internal predictors for passing the NMLE were also identified: score in pre-clinical medical sciences, performance in clinical clerkship, and graduation examination scores. The NMLE was taken within 3 months after clinical clerkship and graduation examination. The logistic regression analyses in our study showed a negative correlation between the score in pre-clinical medical sciences in the third and fourth academic years and passing the NMLE (Table 2). This result is in conflict with a general thought that the students with higher academic score in preclinical medical science would likely be to pass NMLE. Furthermore, the total scores for pre-clinical medical sciences in the students who failed NMLE in 2012–2017 were actually lower than those who passed NMLE (Table 1). However, when we closely looked into the 36 students who failed, we found that they had older age at admission and better scores in pre-clinical medical sciences but worse performance in the graduation examination. Hence, we hypothesize that older medical students might have insufficient study time because some of them had family or need to work part-time, diminished ability to memorize, or burnout due to longer years of schooling and/or working since they graduated high schools. Further studies are needed to confirm this hypnosis. The four significant internal predictors of passing the NMLE shown in this study can be used to predict those who may fail the NMLE.
Moreover, significant external predictors of passing the NMLE were age at admission and HS located in Gifu and Aichi Prefecture. Using linear regression analysis, Kleshinski et al 20, identified predictors of performance on Step 1 and Step 2 as follows: science GPA, biologic science section of MCAT, college selectivity, race, and age. Furthermore, McDougle et al. 3 indicated that the relative risk of first-attempt Step 1 failure for medical school graduates was 3.6 for matriculants aged >22 years (95%CI: 2.0–6.6, p < 0.0001). Consequently, older medical students have a higher risk of failing Step 1, Step 2, and the NMLE. It is unclear why medical students who belonged to a neighborhood HS have better chance of passing the NMLE, and we found no such study on the relationship between performance in medical school and the location of HS or hometown. Students from the neighborhood city/town might be able to receive various kinds of supports from their families physically, economically, and psychologically. Further investigations are required.
Predicting NMLE with data in lower grade
Several studies have predicted the performance of medical students in Step 1 and primarily focused on first-time test takers.4,20,24,29,30 Determining the characteristics of a student who will fail Step 1 is challenging4 because it is difficult to create models that predict the failure of first-time test takers given the low number of students who fail in most schools.4,21 Keeping this in mind, Coumarbatch et al. attempted to create models to predict those who will fail among first-time test takers using logistic regression analysis in 256 students from the graduating class of 2008 at Wayne University.4 They found that the year-2 standard score and MCAT biological science score were significant predictors of failing and concluded that using internal and external predictors, identifying students at risk of failing Step 1 is possible.4 Moreover, they described at-risk groups and current educational intervention strategies. In the current study, the year-2 standard score and MCAT score might correspond to the total score in pre-clinical medical science and the NCTUA (Tables 1 and 2), however, there is a difference of the competencies required and the level of difficulty between MCAT and NCTUA, so it would be reasonable that our results using logistic regression analysis were not consistent with theirs 4. More recently, Baars et al. developed a model for the early and reliable prediction of students who fail to pass the first year in the undergraduate medical curriculum within 2 years after starting.1 However, we cannot directly compare our results and theirs. In the GUSM, the students who failed the NMLE did not have better or worse scores in liberal arts and basic science during their first year in medical school (Tables 1 and 2).
Thus, in the current study, we determined the PPR using several information which can be obtained easily during medical schools, and predicted students who have higher risk to fail NMLE using the PPR for the first time.
The pass rate in NMLE 2018 after support based on the PPR prediction
Before the current study, faculties noticed that some young students with poor performance in the mock examination (ME) may pass the actual NMLE, while the older students with good performance in the ME sometimes failed NMLE, but the reason was unclear. For a new cohort in 2018, we chose students who had lower PPRs in the NMLE (95% or less), indicating a strong likelihood to fail the NMLE, to confirm the validity of the formula (Table 3). The PPR predicted all five students who would fail. Therefore, this result showed that risk analysis from data such as the PPR can enable effective support from multiple points of view, such as the use of MEs. Further prospective studies are needed in other cultural areas, although we need to confirm the validity of the PPR.