The role of personal interview and cognitive abilities at admission to medical school in predicting performance of medical students in internal medicine sub-internship

Abstract

Admission process to medical school is a complicated task, stemming from the unresolved debates on a set of skills defining an ideal physician. The Goldman school at Ben-Gurion University in Israel is known for weighing a great emphasis on personality domain on top of academic excellence, by administering 1-hour long consecutive interviews embedded into the admission process.  

We compared the skills evaluated during the admission process with the students’ performance at the internal medicine rotation in their last (6^th) year of medical school. Evaluations were administered by medical personnel and included questions prompting to choose those with the most potential to be a good internist.

Interview evaluation at admission was associated with higher chances of being evaluated as a good internist 6 years later (Odds Ratio (OR)=9.4, p-value=0.049), independent of the students' gender (OR for male vs female=0.2, p-value=0.025) and age (OR=1.3 per each year, p-value=0.115).

Our study shows that a high personal evaluation of a candidate is indicative of high personal evaluation of a student's performance at the internal medicine department 6 years later. These findings strengthen and emphasize the importance of using a personal semi structured interview in the admission process to medical school. 

Introduction

Selecting the most suitable candidate to be a medical student who will later perform as the best physician constitutes a great challenge and is discussed in many medical schools worldwide. In the past, it was common to rely only on cognitive abilities and previous academic success. Cognitive skills and academic grades were shown to predict success in medical school and in measures of career success[1]. However, this association between academic performance and success beyond medical school is relatively week[2]. Some studies [3, 4] provide clear evidence that candidates selected on the basis of high academic performance alone are much more likely to drop out from medical school than candidates selected through a complex admission process. Moreover, relying only on cognitive tools introduces a significant socio-economic class bias [5, 6]. Therefore, in recent years there is increasing agreement that the admission process to medical school that is aimed at selection of candidates with the highest propensity of becoming the best physicians includes the evaluation of two principal domains, 1-basic intelligence and cognitive aptitude and 2- personality, including candidates’ emotional ability, conscientiousness and self-esteem[7]. Various methods have been developed to examine these personal characteristics including: personal interviews, biographical questionnaires, Situational Judgement Tests, Multiple Mini Interview and others. The process varies between medical schools between countries and within Israel, reflecting the diversity in vision and mission of each one of them. Since it is complicated to define the best physician, it is an unresolved and ongoing debates which admission process predicts which candidate will be the best physician.

Since its establishment 47 years ago, the six years Goldman medical school at Ben-Gurion University (BGU) advocates for personality traits been of highest importance, and therefore deliberately emphasizing the personality evaluation to the first domain, once the basic (although very high) intelligence criterion has been met [8–10]. The admission process is open for all high school graduates and includes a three-staged process (Fig. 1). The first stage includes a score a.k.a. "Sekhem" that represents a weighted average between the Israeli national matriculation exams and the grade of the psychometric test administered by the Israeli National Institute for testing and evaluation. This combined mark ("Sechem") is a prerequisite for eligibility for initiating an admission process in all medical schools in Israel. In the BGU school, Sekhem has been historically set lower relative to other schools following an understanding that the highest combined mark does not necessarily predict the best medical student and physician [11]. Additionally, a slightly lower cut off ensures a socioeconomic diversity of the candidates. Once passed the Sekhem cut off, the candidates are eligible for the second stage. It is comprised of a computerized questionnaire, based on the "Big Five" [12]- testing for different variables of personality including conscientiousness, extraversion, agreeableness, openness to experience and emotional stability. This computerized test enables widen the funnel so all candidates with the basic cognitive abilities are given a chance. The candidates with the highest scores on the computerized test are invited for an interview by two trained interviewers in a session that may take up to one hour. One interviewer is a physician and the other is a community representative (stage 3).

The interview is based on a pre-written structured resumes and the questions are directed to specific abilities including thinking ability, emotional ability, social skills, social awareness and social involvement, as well as the ability to stand under pressure.

The scores for the interview are standardized separately for each interviewer by calculating a t-score relative to his/her history of grading. Thus the t-score of the grade indicates the extent to which a candidate scored high or low relative to other evaluations of the same interviewer. An average t-score for both interviewers represent the final interview score. The scores are back-transformed into grades 0–10 for the final comparison. Over the last decades the admission process varied in terms of the interview structure, while often times it included 2 consecutive interviews (by overall 4 interviewers), while the second interview was administered within candidates passing a threshold score in the first. The "2-interview" structure was always preferred over the "1-interview" and depended entirely on the capacity of the admission committee on a certain year. The 120 candidates with the highest score out of approximately 1600–1800 candidates passing the first stage are the ones accepted to medical school, with an acceptance rate around 7%.

Over the years, the graduates of the Goldman medical school at BGU are known for their excellence and high human-factor quality. Friedberg & Glick from BGU[13] analyzed evaluations made by department heads in hospitals throughout the country comparing BGU graduates to their counterparts trained in other medical schools in Israel. Seventy-four (74) % thought that BGU graduates were better in physician-patient relations and 49% felt they excelled in physician-team relations.

Despite these findings, the admission process at BGU and its association with the students' performance has never been assessed.

In this study, we focused on the students’ performance at the internal medicine rotation – which is practically sub-internship in their last (6th ) year of medical school. The sixth-year internal medicine clinical rotation lasting for 6 weeks is designed to prepare students for their role as house officers. During the rotation the student should function as a first-year resident under the supervision of the senior medical staff. The student will assume primary responsibility for the patients he/she admits to the service. Students are evaluated by their skills to accurately and efficiently assess, diagnose, and treat general medical conditions requiring hospitalization and provide an immediate care. Additionally, at this point students are assessed for their problem-solving skills needed to facilitate future medical care/compliance in a patient population frequently without prior medical care. During this rotation students are expected to effectively care for patients during their hospitalization and provide for a safe transition at discharge. As well as foster effective communication with patients, families, colleagues, consultants, primary care physicians, and allied health professionals.

In our opinion, the medical student sixth year internal medicine rotation performance may potentially serve as a close proxy to their future performance as physicians.

Methods

The study was aimed to validate the admission process based on comparison of the grades at admission with the students' performance during 6th year clinical rotation of internal medicine. Specifically, we estimated if students' evaluations provided by medical personnel at a clinical round of internal medicine are associated with their grades available at admission.

Additionally, we compared evaluations made by medical personnel vs students' academic evaluations throughout the 6 years of the school curriculum and the students' personal evaluations of themselves as future physicians and their aspirations in medical career.

Results

We collected the information on 136 students overall. In all, 115 students (88.5%) were admitted in 2015 and the rest of the students started their studies in earlier years due to combining an additional degree during medical school (MD-PhD, MSc, etc). For 133 of them we had a full information on their evaluations at admission. The information for 3 students could not be retrieved from the records, as the students were admitted prior to 2012, and the researchers did not have an access to that data. Evaluations by the medical personnel were received for 110 of the students, and the rest were not available owing to the complicated logistics related to the onset of the COVID-19 pandemic half-way through the study completion. Not all medical personnel agreed to provide sociometric evaluations considering them unethical, and therefore students' ranking was available for 93/110 students. Overall, 134 students filled out the personal questionnaires.

The number of medical personnel (raters) who provided evaluation ranged between 1–7 people, with half of the students graded by 3 and more raters. There were 31 out of 107 students (29.0%), for whom all the raters in the department have admitted that they would like working with the student as a physician in future (grading this question by 6 out of 6). There were 6 out of 93 students who were ranked as first out of their group (6.5%). In all, we have 31.5% (34/108) who met one of the two conditions above and therefore defined as "good internists", others defined as "not as good" internists. Descriptive tables are stratified by the "good internist" status (Table 1).

Overall, the students in the study were on average 30.0 ± 2.0 years old, 54.1% of them married and 70.2% (87/124) of them were female. Over 70% of the students were secular and served in the army in their absolute majority (88.8% (87/98)). None of these characteristics varied between the "good" and "not as good" internists, except for the type of the army forces where the students defined as the good internists were more likely to have served in education corps, intelligence forces and navy.

The students' satisfaction with their choice of career or perception of their future plans regarding research were not different between the groups (Table 2). The specialties preferences varied in fact between the groups with more students prioritizing orthopedics, ophthalmology or dermatology in the group of "not so good internists".

Inspection of evaluations at admission as compared to the students' status as good or not as good internists, provided different conclusions for different composites of the final grade (Table 3). Specifically, the two groups varied only in their evaluation at the interviews, and not in intelligence or behavior scores. For instance, ranking of a personal interview was higher for the good internists as compared to all others (with median 9.5 vs 9.4, p-value 0.013), but not for Israeli national matriculation exams, psychometric test and their composite (Sekhem). Furthermore, the score featuring the students' character traits appeared to be statistically even between the two study groups (p-value = 0.922). A close inspection of the medical personnel evaluation showed that students ranked higher at the interview at admission were getting more credits for interaction with the patients (p-value 0.044), accomplishing tasks (p-value 0.027), for participating and being present (p-value 0.024), handling complex situations (p-value 0.020) and being reasonable (p-value 0.070) (appendix 3, supplementary materials).

Aiming to find a course that could have been indicative of the good or not as good evaluation of an internist, we analyzed all the grades the students received throughout their 6 years of school curriculum. We chose the courses featured by the coefficient of variance beyond 78 (on a scale 0-100), and therefore a higher potential to discriminate between the good and not as good internists status. A simple comparison between the two groups did not provide a statistically significant conclusion but indicated a trend of higher grades in physics and a course in "body and soul" within the good internists' group, both taught at the first year of the medical school (Table 4).

Based on a multivariable analysis in a logistic regression, interview evaluation at admission was positively associated with higher chances of being evaluated as a good internist 6 years later (OR = 9.4, p-value = 0.049), independent of the students' gender (OR for male vs female = 0.2, p-value = 0.025) and age (OR = 1.3 per each year, p-value = 0.115) (Table 5). Similar results were obtained in a sensitivity analysis by using a Poisson model, although with a borderline significance for the interview grade (p-value = 0.101 and relative risk = 4.0).

Discussion

The purpose of this study was to examine which of the components of the admission process, i.e. (1) basic cognitive abilities, a (2) computerized personality test or (3) an admission interview, predicts student success toward graduation as reflected at the Sixth year internal medicine rotation. The later is a final round in which students are asked to function as "physicians" under very close supervision. Success examined by peer evaluation in various parameters such as knowledge, performance, attitude to patients and relationship with the staff. The main finding was that the interview via admission process had the highest correlation to success in sixth year while the other parameters including the computerized test, cognitive abilities as well as academic achievements throughout medical school did not impact students performance as a physician.

The findings contradict most of the published research showing low reproducibility and consequently, low validity, of personal interviews at admission to medical schools [14–16]. Nevertheless, there is a growing recognition of the importance of non-cognitive tests. Powis et all discuss in a number of articles[17–19] the importance of selecting candidates with personal qualities appreciating the significance this has for the future quality of care these doctors will provide. They discuss the various methods for examining non-cognitive parameters and the barriers that exist in medical schools for using them. The high cost of non-cognitive evaluations is one the reasons they raise. Another - is the desire to excel academically, explaining why many medical schools prioritize this aspect over personality skill. The latter have been developed in the field of organizational psychology and mostly for admission to work rather than medical school[20]. There are additional barriers we found in assessing non cognitive tests and interviews. For instance, what defines a good doctor? And will a good student necessarily be a good doctor?

The choice of the internal medicine rotation of the sixth year seemed to us to be the closest way of examining the success of students as future doctors. The students are close to graduation, it is a 6-week rotation so there is much time and ability for the staff including senior physicians, residents and nurses to get to know the students in depth. They function as supervised physicians and both their knowledge and their approach towards patients and teamwork can be assessed. Studies have shown that it is precisely in internal medicine that doctor-patient communication and the ability to interview is highly important and has influence on patients' response to treatment, patient satisfaction and treatment success [21]. Although good communication, empathy, integrity are important qualities for all physicians but in internal medicine these abilities are manifested in a higher intensity. Our medical school has emphasized the importance of these features and therefore in admission to medical school they are tested and therefore we saw fit to test these capabilities along with the ability to integrate the theoretical material along with interpersonal communication as reflected in the sixth-year rotation. A study by Manuel et al [22] they found that there is a correlation between personality components and clinical success by examining second year medical students in two parameters, first, the Sixteen Personality Factor Questionnaire (16PF) and second, the Clinical Skills Assessment II (CSA II).Their findings reinforce our findings.

We found an association between the assessment the students received by the medical staff and the choice of internship later on. A higher percentage of students who received the least good grades were those who chose future professions such as dermatology, orthopedics and ophthalmology. Various studies have examined the relationship between different personality components and the choice of specialization below. In a study by Preece et al[23] the researchers compared personality components of surgery residents to medical students who expressed interest in surgery. The study found that the personality components were similar and so was the visual modality learning style. This reinforces our findings on students with preferences for surgical departments received lower grades.

The obvious limitation of the study is not accounting for the candidates who were not accepted to the medical school who might have qualified for being good internists and thus representing a false negative fraction of the admission process. However, approaching and studying them would have been severely biased by their compliance, expected to be low, coupled with their recent rejection from the medical school causing much anger and frustration.

Second, the group that is accepted is comprised of candidates with the highest scores in the interviews which turns our comparison of outstanding with the excellent.

Another limitation discussed is whether the internal medicine sixth year rotation predicts who will be the best surgeon, or other non-internal medicine specialties?

The study was aimed to evaluate a performance of a student towards the end of medical school and not as a physician. Studies should be conducted while working as a doctor to reassess these findings.

Conclusion

In summary, our study shows that a high personal evaluation of a candidate is indicative of high personal evaluation of a student's performance at the Internal Medicine department 6 years later. These findings strengthen and emphasize the importance of using a personal semi structured interview in the admission process to medical school.

Declarations

Ethics approval and consent to participate

The study was approved by the Ethics committee of the Faculty of Health Sciences in BGU. All subjects signed an informed consent form prior to their participation in the study.

Consent for publication

The manuscript does not contain details requiring consent for publication from the third party.
 

Availability of data and materials

The datasets analysed during the current study are not publicly available due the fact that this permission has not been granted by the ethics committee, however the deidentified information will be available from the corresponding author on reasonable request.
 

Competing interests

Authors have no competing interests to declare.
 Funding

The investigators did not receive any funding.

 Authors' contributions

IL has initiated the project, contributed to its design, assisted with the study logistics and interpreted the results. Likewise, she drafted  a substantial part of the manuscript. LN participated in study design, data collection and its analysis. She also drafted the methodology and results sections. RH conceptualized the study, participated in its design, and assisted with data collection. In addition, RH critically revised the manuscript. AK conceived the study ide, participated in its design, and assisted with data collection. He wrote a substantial portion of the manuscript (introduction) and critically revised the rest of it.

 Acknowledgments

We are grateful to Prof. Shimon Glick, the former Dean of our Faculty, for his review and valuable suggestions in preparing the manuscript.

References

1. Julian, E.R., Validity of the Medical College Admission Test for predicting medical school performance. Acad Med, 2005. 80(10): p. 910–7.
2. Ferguson, E., D. James, and L. Madeley, Factors associated with success in medical school: systematic review of the literature. BMJ, 2002. 324(7343): p. 952–7.
3. Powis DA, R.I., Selection and performance of medical students at Newcastle, New South Wales. Education for Health, 1998. 11(1): p. 15–23.
4. O'Neill, L., et al., Medical school dropout–testing at admission versus selection by highest grades as predictors. Med Educ, 2011. 45(11): p. 1111–20.
5. McManus, C., K. Woolf, and J.E. Dacre, Even one star at A level could be "too little, too late" for medical student selection. BMC Med Educ, 2008. 8: p. 16.
6. Cleland J, D.J., McLachlan J., Nicholson S., Patterson F., Identifying best practice in the selection of medical students (literature review and interview survey). 2012.
7. Doherty, E.M. and E. Nugent, Personality factors and medical training: a review of the literature. Med Educ, 2011. 45(2): p. 132–40.
8. Antonovsky, A., Student selection in the School of Medicine, Ben-Gurion University of the Negev. Med Educ, 1976. 10(3): p. 219–34.
9. Glick, S.M., Selection of medical students: the Beer-Sheva perspective. Med Educ, 1994. 28(4): p. 265–70.
10. Antonovsky, A., Medical student selection at the Ben-Gurion University of the Negev. Isr J Med Sci, 1987. 23(9–10): p. 969–75.
11. Powis, D.A., et al., The objective structured interview for medical student selection. Br Med J (Clin Res Ed), 1988. 296(6624): p. 765–8.
12. Sobowale, K., et al., Personality Traits Are Associated with Academic Achievement in Medical School: A Nationally Representative Study. Acad Psychiatry, 2018. 42(3): p. 338–345.
13. Friedberg, M. and S.M. Glick, Evaluation of innovative school's graduates by department heads at other hospitals. Medical Teacher, 1997. 19(1): p. 36–39.
14. Goho, J. and A. Blackman, The effectiveness of academic admission interviews: an exploratory meta-analysis. Med Teach, 2006. 28(4): p. 335–40.
15. Basco, W.T., et al., Medical school application interview score has limited predictive validity for performance on a fourth year clinical practice examination. Adv Health Sci Educ Theory Pract, 2008. 13(2): p. 151–62.
16. Wilkinson, D., et al., Medical school selection criteria and the prediction of academic performance. Med J Aust, 2008. 188(6): p. 349–54.
17. Powis, D., Selecting medical students: An unresolved challenge. Med Teach, 2015. 37(3): p. 252–60.
18. Powis, D., Improving the selection of medical students. BMJ, 2010. 340: p. c708.
19. Powis, D., et al., Why is it so hard to consider personal qualities when selecting medical students? Med Teach, 2020. 42(4): p. 366–371.
20. N., B., The big five of personality and work involvement. Journal of Managerial Psychology, 2004. 19(1): p. 69–81.
21. Mueller, P.S., et al., Views of new internal medicine faculty of their preparedness and competence in physician-patient communication. BMC Med Educ, 2006. 6: p. 30.
22. Manuel, R.S., N.J. Borges, and H.A. Gerzina, Personality and clinical skills: any correlation? Acad Med, 2005. 80(10 Suppl): p. S30-3.
23. Preece, R.A. and A.C. Cope, Are Surgeons Born or Made? A Comparison of Personality Traits and Learning Styles Between Surgical Trainees and Medical Students. J Surg Educ, 2016. 73(5): p. 768–73.

Tables