Study participants and setting
This study was conducted on candidates that participated in admission interviews at Dongguk University School of Medicine (DUMS), a private medical school in South Korea, for matriculation in 2019. DUMS has a four-year basic medical education program for graduate-entry students and an annual intake of 50 students. DUMS has implemented admission interviews for those that passed the initial screening stage based on considerations of prior academic achievements, including undergraduate Grade Points Average (GPA) and performance at the Korean medical school entrance exam (the Medical Education Eligibility Test). As a result, 94 candidates were selected for admission interviews, which were conducted in December, 2018.
DUMS has used MMIs for admission interviews since 2014. Interview schedules are composed of six mini interviews conducted at separate stations, and the allocated time per station is 10 minutes. There was one assessor for each station, who evaluated the candidates’ performance on a 5-point scale of 1 being “unsuitable” to 5 being “outstanding” on two to three items presented in a scoresheet. Candidates’ performances are assessed by each station score and their overall performance is determined by summing up the scores at each and every station. Previous experiences with the MMI at DUMS have shown that it is a feasible tool for student selection [27, 28].
Study design and procedures
A video-based case was developed for one MMI station to assess candidate’s empathic abilities. MMIs at the other five stations were implemented in traditional paper-based format. Three medical faculty members participated in the development of the video-based case. Two were experts in MMIs and the other was a psychiatrist, who wrote the script for the scenario. Two investigators with experience of MMIs reviewed and revised the draft scenario. The video was produced in-house and was pilot tested on a volunteer medical education graduate student. The student was asked to think aloud what she thought of the video as she watched and reported the situation was presented clearly in the video and that the dialogue was unambiguous.
The video vignette presented a fictive clinical situation where a doctor interviewed a patient who seemed to be in a depressive mood. The vignette lasted around two-minutes because of time constraints for the candidate to view and prepare for him/her to discuss it with the assessor. Candidates used a tablet and a headset to watch the video, and were asked to assess the extent to which the doctor showed empathy and elicited the feelings and views of the patient, as these are considered key elements of empathy in doctor-patient interactions [20]. During the interview, the candidate discussed with the assessor on the extent to which the doctor showed empathy in communicating with the patient and about the importance of empathic communication in patient-doctor relationship.
Data on candidate perceptions and performance in the MMI stations were obtained and analyzed to investigate its acceptability, fairness, validity, and reliability as evidence regarding feasibility of the test. Acceptability by candidates was examined by investigating their perceptions of the MMI and the video-based case using a post-MMI questionnaire. The questionnaire used in this study consisted of 41 statements with Likert-type responses ranging from Strongly Disagree (1) to Strongly Agree (5). The items in the questionnaire were classified as follows. The first section included 7 items regarding candidate demographics and backgrounds. The second section consisted of 17 items that elicited candidates’ overall perceptions of the MMI. These items were adapted from the instrument developed by Eva [6] and translated into Korean by Kim et al. [28], and have been used in other studies [27, 28]. The third section included 12 statements on respondent perceptions of the video-based case used in the empathy station, which consisted of the following four sub-scales: (a) level of difficulty in understanding the situation presented in the video (3 items), (b) authenticity of the situation portrayed in the video (3 items), (c) interest (3 items), and (d) overall satisfaction (3 items). This section also included five items regarding candidate perceptions of the patient-doctor relationship presented in the video clip. The 17 items in this section of the questionnaire were developed by the authors and were pilot tested in the previous year with a sample of medical school applicants. The last item was a single open-ended question that elicited candidates’ overall opinions of the MMI.
The questionnaire was administered during a wrap-up session conducted immediately after all interviews had ended in the morning and afternoon sessions. Participation in the study was voluntary and consent was implied by return of the questionnaire as responses were collected anonymously. An ethical review was conducted and the study was exempted from the requirement for informed consent by the institutional review board of Dongguk University, Gyeongju.
Fairness of the test was assessed by means of differences in candidate perceptions of the video-based case and of the patient-doctor relationship presented in the video clip across different demographics or backgrounds. In addition to candidate perceptions, his or her test scores in the empathy station were compared across different demographics or backgrounds. Construct validity was assessed by examining the relationship of candidate scores in the empathy station with those in other stations. Moreover, we analyzed generalizability of the test to investigate it reliability.
Data analysis
Descriptive statistics were used to analyze candidate responses to the post-MMI questionnaire and their test scores in the MMI stations. Reliability of the research instrument was assessed using Cronbach’s alpha values. The independent t-test was used to compare candidates’ responses and performance with respect to gender and age, for which candidates were dichotomized about median age (25 years), and their geographic locations (urban vs. rural areas). ANOVA (analysis of variance) was used to compare candidates’ perceptions with respect to undergraduate backgrounds, which were categorized into seven groups. The G-coefficient was analyzed to investigate the reliability of this test, which indicates the proportion of variance in MMI score attributable to differences in candidates’ non-cognitive abilities [13]. The data were analyzed using SPSS version 23 for Windows (IBM Corp., Armonk, USA), and statistical significance was accepted for p values < 0.05.