Using OSCE for Capacity Building Program Outcome Evaluation: A First SP-based OSCE in Laos

Kye-Yeung Park Hanyang University College of Medicine Hoon-Ki Park (  hoonkp@hanyang.ac.kr ) Hanyang University College of Medicine Jwa-Seop Shin Seoul National University College of Medicine Youngjoo Jung KDI School of Public Policy and Management Min Young Seo KDI School of Public Policy and Management Taejong Kim KDI School of Public Policy and Management Ketsomsouk Bouphavanh University of Health Sciences Vientiane Sourideth Sengchanh University of Health Sciences Vientiane Ketmany Inthachack University of Health Sciences Vientiane


Background
In 2011, a Korean study-abroad project called the Dr. Lee Jong Wook-Seoul Project (DLSP) was launched as a medical educational aid program for Laos. This project was to provide Laos with an opportunity to enhance its medical technology and education by sending Laotian healthcare professionals from the Lao University of Health Sciences (UHS) to Seoul National University (SNU) and Seoul National University Hospital (SNUH) in Korea. In 2011, the rst year of the program, 8 Laotian professors received training as DLSP fellows at SNU and SNUH, and since then, the DLSP has had 8 fellows in 2012, 10 fellows each in 2013 and 2014, 11 fellows in 2015, 13 fellows in 2016, and 11 fellows in 2017. All fellows receive one year of training in Korea and then return to Laos. The DLSP is intended to take social and educational responsibility for developing countries, similar to the Minnesota Project, which was carried out by the United States half a century ago to transfer its advanced medical technology and welfare to Korea [1].
A previous evaluation of the rst-year outcomes of the DLSP used a 360-degree feedback format, in which the individual DLSP fellows reported a high level of satisfaction, and their trainers and colleagues reported improvement in the fellows' medical knowledge and clinical skills [2]. Another study examining the effects of the DLSP on the academic achievement of medical students in Laos found that the DLSP could be effective in improving the educational environment and teaching capacity of Laos [3]. Now, in the eighth year since the implementation of the DLSP, determining how this capacity-building program has affected the clinical competency of medical doctors in Laos, including the DLSP fellows, is an important challenge.
Whereas the previous relevant studies have examined the educational effects of the DLSP at the cognitive level of competence, this study assesses clinical competencies not evaluated previously, such as information gathering, communication skills, building patient-physician relationships, and clinical reasoning ability. An objective structured clinical examination (OSCE) can be considered an appropriate evaluation method because OSCEs are used worldwide as a validated method for assessing clinical competencies in medical education [4][5][6][7]. However, no standardized patientbased (SP-based) OSCE had been conducted in Laos before this study, although simulation-based tests have been used to assess clinical skills in low-and high-stakes examinations for health professionals. Accordingly, the process of adopting a Korean SP-based OSCE and implementing it in Laos became another aim of this study. Introducing an OSCE into a new setting or country requires great care and faces inevitable challenges, as described in previous literature [8][9][10]. Beyond its signi cant novelty, however, the introduction of this new assessment method could also be expected to improve the quality of medical education in Laos.
Therefore, the rst objective of this paper was to investigate the educational effects of the DLSP on the clinical competence of Laotian doctors by assessing the SP-based OSCE results of medical doctors, including DLSP fellows; mentee doctors taught by DLSP fellows; and control groups of doctors for comparison. Second, this study describes the process for implementing the SP-based OSCE in Laos.

Methods
Project timeline for implementing the OSCE Figure 1 shows the project timeline for implementing the SP-based OSCE in Laos. A Korean team and a faculty team from Lao UHS rst met in August 2018 to organize the OSCE in Laos. They discussed the timeline for the implementation strategy and the schedule for a workshop and seminar on the OSCE. They decided that the overall time required to implement both the workshop and the OSCE was at least three consecutive days for the workshop and another three consecutive days for the OSCE. Accordingly, the Korean team planned to make three return trips to Laos.
After the rst meeting, the Korean team identi ed OSCE case materials that it considered suitable for evaluating the clinical competence of Laotian doctors in their clinical context and chose the following six OSCE cases from among the cases used in medical colleges in Korea: headache, fatigue, fever, baby seizure, weight loss, and cough. Those cases were then translated into the Lao language by the Laotian translator. Twelve evaluators for the OSCE were recruited because each OSCE case requires two evaluators. The 12 evaluators were all experienced physicians from different departments of medicine at Lao UHS.

Workshop for the OSCE
The workshop on the SP-based OSCE was held for the twelve evaluators and run by a Korean team in Laos for three days. The workshop was intended to help the evaluators prepare for OSCE module development, SP training, and OSCE scoring and coordination. The workshop consisted of ve sessions of core lectures (using SPs in medical education, case development for OSCE, SP training, physical simulation, and coordination of the OSCE), four sessions of small group activities (modi cation and re nement of OSCE assessment modules, ne tuning of OSCE modules, SP training, and netuning of OSCE cases), and four sessions of large group activities (mock OSCE, debrie ng of mock OSCE, orientation for the upcoming OSCE event, and nal wrap up).
After the workshop, the twelve SPs for the OSCE were recruited and trained. They were all Laotian laypersons who matched the speci c traits of age and sex needed for the individual OSCE cases. Next, 90 OSCE examinees were recruited in two case-control groups to answer two research questions: (1) What is the primary effect of the DLSP? i.e., does the clinical competence of DLSP fellows differ from that of doctors without DLSP experience? (2) What is the secondary cumulative effect of the DLSP on UHS graduates who were educated by DLSP fellows? i.e., does the clinical competence of mentee doctors taught by DLSP fellows differ from that of doctors not taught by DLSP fellows?
First, to determine the direct educational effect of the DLSP, 17 doctors with DLSP experience were selected as a case group, and 17 age-, sex-, major-matched doctors without project experience were selected as the control group. Second, to determine the indirect and cumulative effects of the project, we divided UHS graduates into two groups. The case group, referred to as the mentee doctors of the DLSP fellows, contained 28 doctors trained since 2014 who were taught by UHS professors that had been DLSP fellows. The control group contained 28 age-, sex-, major-, and job-matched doctors trained at UHS before 2011, i.e., before the rst DLSP fellows had nished the program.

Implementation of the OSCE
Each OSCE case was given 11 minutes, one minute of door sign reading and a ten-minute SP encounter. Each examinee rotated through all six OSCE stations. As the venue, the conference rooms at a hotel in Vientiane, Laos, were organized into twelve examination rooms and a couple of waiting rooms.
On the day of the OSCE, the examinees signed up at the entrance of the venue and were given an onsite orientation using a recorded video in a waiting room. The video contained a brief introduction to the OSCE, provided the examination design and rules, and inventoried the medical equipment in each examination room. After the examinees nished the six cases, they were guided to a survey room to answer written questionnaires that asked about their baseline characteristics and the interview questions used for the qualitative study.

OSCE scoring and statistical analysis
The OSCE overall and subcomponent scores were rated on a pre-determined checklist by twelve evaluators who made direct observations inside each examination room. The questions and tasks on the checklists for the six stations were designed to elicit information about the six following clinical skills: (1) patient satisfaction, (2) history taking, (3) physical examination, (4) patient education, (5) clinical courtesy, and (6) patient-physician interaction (PPI). The subcomponent scores on history taking and patient education were rated dichotomously as 'did ask or tell = 1 point' or 'did not ask or tell = 0 points.' Items on the clinical courtesy subcomponent were also dichotomous based on tasks that examinees should do or show during a physical examination, and they were scored as 'did perform = 1 point' or 'did not perform = 0 points.' Subcomponent scores on the physical examination were scored as 'did perform correctly = 2 points,' 'did try but did not perform correctly = 1 point,' and 'did not perform at all = 0 points.' An item on the patient satisfaction subcomponent was 'I feel satis ed with the treatment of the examinee' and was scored by the SP at each OSCE station using a six-point Likert scale (excellent=5, very good=4, good=3, must improve=2, poor=1, very poor=0). The PPI subcomponent contained 7 items about how well the examinee established a good patient-physician relationship, and those items were scored on the same six-point Likert scale. To compute the six subcomponent scores, the number of items performed was divided by the total number of checklist items. For example, the sums of scores performed for history taking, patient education, and clinical courtesy were divided by the total number of checklist items for each subcomponent; the divided values on patient satisfaction and PPI were further divided by 5, and those on physical examination divided by 2. This yielded the station score for each examinee and gave the percentage OSCE total score. We computed the mean and standard deviation for the OSCE overall scores and each of the six subcomponent scores.
For the statistical analyses, we used the Mann-Whitney test to compare baseline characteristics such as the age and number of post-graduate years of the rst case group, i.e., DLSP fellows and their controls, and the second case group, i.e., mentees of DLSP fellows, and their respective control groups and Fisher's exact test or the chi-square test for categorical variables such as sex, profession, workplace, residency training, and major. In the comparison of the OSCE overall and subcomponent scores between each case group and its control group, the scores are presented as the mean and standard deviation (SD), and they were compared using Mann-Whitney testing due to their non-normal distribution.
Subgroup analyses comparing the overall OSCE scores between two groups according to sex, major, and profession used the Mann-Whitney test. P-values less than 0.05 were considered statistically signi cant, and all analyses were conducted using SPSS, version 21.0 (SPSS Inc., Chicago, IL).

Results
Baseline characteristics of the study subjects Table 1 compares the baseline characteristics of the rst and second case-control groups. In the rst case and control groups, the distribution of sex did not signi cantly different between the two groups. The DLSP fellow group was slightly older than the control group (38.8 years vs. 34.5 years, p-value = 0.03). The rst case and control groups comprised residents and staff, and staff accounted for the larger proportion in both groups. Most of the participants worked in the central hospital or UHS and had completed their residency training. Among the DLSP fellows, the majors varied from pediatrics, internal medicine, surgery, obstetrics and gynecology, radiology, anesthesia, and family medicine to pathology.
On the other hand, 12 of the 17 doctors in the control group were pediatricians. In the second set of case and control groups, the distributions of age, sex, and profession did not differ signi cantly between the two groups. The second case and control groups comprised interns, residents, and staff, and residents accounted for the largest proportion in both groups. More than half of the participants in both groups worked in the central hospital or UHS. Half of the participants were being trained for residency in the mentee doctor group, similar to the proportion in the control group. In both groups, the distribution of major varied from pediatrics, internal medicine, surgery, obstetrics and gynecology, radiology, anesthesia, family medicine, and otolaryngology to medical biology.
The educational effect of the DLSP on the clinical competence of Laotian doctors Table 2 compares the OSCE overall and subcomponent scores from the rst and second case and control groups. The comparison of the rst case-control group evaluated the direct educational effects of the DLSP as learning. The overall OSCE score did not differ signi cantly between the two groups, but it was marginally higher in the control group (57.1 vs 62.9, p-value = 0.06). Among the six OSCE subcomponent scores, the scores on patient satisfaction, physical examination, clinical courtesy, patient education, and PPI did not differ between the case and control groups (56.  Subgroup analyses by sex, major, and profession Given evidence that female doctors achieve higher OSCE scores than male doctors, we analyzed the female doctors separately (Supplement Table 1) [11,12]. We found no signi cant differences in either the rst or second set of case and control groups.
Subgroup analyses of the overall OSCE scores by major are presented in Table 3. Among the participants in the rst set of case and control groups, we compared the scores of clinical physicians such as pediatricians, internal medicine doctors, family physicians, and emergency medicine doctors between the case and control groups because we considered that doctors who do not regularly see patients might perform worse on the OSCE than clinical physicians (Supplement Table 2). In that comparison, we found no signi cant differences in the OSCE overall or subcomponent scores between the clinical physicians from each group. Because many of the doctors in the second set of case and control groups were residents or general physicians just out of college, we did not assess clinical physicians separately in that analysis.  3) How has the Dr. Lee Jong Wook-Seoul Project experience affected your clinical competence?
Increased self-con dence for medical practice (e.g., high-risk pregnancy) and research No impact ( ve fellows' answers).
4) How did you apply the Dr. Lee Jong Wook-Seoul Project experience to the education of medical students and residents?
Applying Korean teaching methods in educating medical students and residents (e.g., how to do a presentation, how to collect patient information and plan patient treatments) Sharing DLSP fellow experience with medical students and residents and teaching them specialized medical knowledge from Korea (intensive care, anesthesiology) Helpful for education by keeping up with the research.

5)
How do you think the Dr. Lee Jong Wook-Seoul Project experience has affected the education of medical students and residents?
Helps medical students think and express bravely, which is a good sign of development in Laos Opportunity for medical students to indirectly experience through sharing by DLSP fellows Encouraging medical students to join the project Helpful for medical students and doctors from the district hospital by teaching newly accumulated medical knowledge.
Interview results from the mentees of DLSP fellows (N = 28) 2) How has being taught by DLSP-experienced faculty affected your clinical competence?
Through specialized medical knowledge (e.g., blood vessels of the heart system, how to do a throat examination) Encouraged me to work more actively.

Discussion
In this study, we have described the implementation process for the rst SP-based OSCE in Laos and reported the results of a program outcome evaluation for a capacity building program for Laotian medical doctors. A series of processes was successfully implemented through close cooperation between faculty teams from Lao UHS and Korea. As for the direct educational effects of the project on OSCE scores, we found that the overall OSCE scores did not differ signi cantly between the DLSP fellow group and the control group. We found the same results when we compared female doctors and clinical physicians separately. Likewise, when we compared the OSCE scores between the group of mentees taught by the DLSP fellows and that control group, the OSCE overall and subcomponent scores did not differ signi cantly, which means that the project had no apparent secondary educational impact.
From the perspective of the Kirkpatrick model, this study measured the project outcomes at the level of learning and the level of transfer using the OSCE. Evaluating the learning outcome, corresponding to Kirkpatrick level 2, usually compares improvements between pre-test and post-test scores after the completion of an educational program [13,14]. We measured the direct educational effect of the project as the difference in OSCE scores between DLSP fellows and a control group of doctors who did not experience the project. The transfer outcome of the trainees, Kirkpatrick level 3, is the indirect secondary effect of a project, which we evaluated by comparing the OSCE scores of doctors taught by DLSP fellows with those from a control group who had not been taught by DLSP fellows.
The OSCE results indicating that the DLSP had no signi cant effect on the learning or transfer levels could rst be explained by the discrepancy between the goals, objectives, and tasks of the OSCE and the DLSP's educational objectives and training environment. The main goal of the OSCE is to grade the examinee's clinical practice skill or clinical competence, such as their ability to understand patient reports about their illness and achieve a comprehensive understanding of each patient as a whole person [15][16][17]. The OSCE assessment thus re ects the examinee's communication and interpersonal skills, along with their problem-solving skills [16,18,19]. On the other hand, the primary goal of the DLSP was to strengthen the education, research, and medical practices of UHS faculty, with the ultimate goal of promoting the development of UHS by transferring that information to other medical professionals and universities [2,20]. In other words, the physician competencies evaluated by the OSCE do not match the goals that the DLSP was originally intended to meet.
Despite that discrepancy, we hypothesized that eight years of project implementation would have increased the general competence of Laotian doctors through direct learning outcomes or indirect transfer outcomes. It has been reported that the teaching capacity of DLSP fellows improved since project implementation, and written test scores for medical knowledge increased in a study examining the effect of the DLSP on Laotian medical students' academic performance [3]. In another study, which evaluated the rst-year outcomes of the project, project participants reported substantial subjective satisfaction with the project [2]. However, that study reported that although the project signi cantly improved the knowledge and skills of the participants in a eld related to their majors and in basic medical knowledge, its contribution to improving their clinical ability was poor [2]. As shown in the written interview data in our study, DLSP fellows reported that the project had no real effect on their clinical competence in seeing patients, other than improving their self-con dence, whereas their experience in the project did strengthen their clinical knowledge. Given that the DLSP fellows had to experience the project in the busy medical environment of Korea and that the clinical encounters they observed were mainly conducted in the Korean language, it would have been di cult for that experience to increase their OSCE scores. Certainly, various training activities, including lectures, exercises using simulation devices, and practicing laboratory experiments, helped to strengthen their theoretical grounding in the diagnostic and therapeutic approaches used in their respective specialties. However, the DLSP fellows had few opportunities to hone their skills in building patient-physician relationships, which can be acquired only through direct patient experience. Although they were allowed to train themselves by observing outpatient care and round visits at hospitals, participating in surgeries, having access to electrical hospital records, and presenting clinical cases, it was di cult to verify any associated improvements using the SP-based OSCE.
Another possible explanation of our results is that the DLSP implementation period might still be too short for the project effects to have accumulated in the Laotian medical environment enough to be visible in Laotian doctors' clinical performance. A previous study showed that the DLSP fellows made great efforts despite the limited medical environment in Laos, and notable changes were made in education, research, and patient care [2]. Nevertheless, a fundamental gap remains between the educational and technological capacities of Laos and Korea.
Therefore, we suggest that the DLSP project supervisors consider adjusting the goals of the project to promote patientcentered problem-solving skills among the trainees. Supervisor support for educational training and an effective system need to be established to organize an appropriate learning experience for the learners [21]. Any long-term educational project for medical professionals needs to t into the local context and needs; therefore, the gap in the medical environment between Korea and Laos would need to be considered. To reinforce the effect of the project, selecting DLSP fellows whose Korean or English language skills are above a certain level might be needed, or pre-DLSP training sessions could be conducted for selected fellows to discuss the multiple clinical competencies they will need to learn more effectively during the project and better transfer what they have learned after returning to their country. We suggest that DLSP fellows returning to Laos create their own cyber education program in Laotian to enable continuous professional development.
A novel signi cance of this study is our success in implementing the rst SP-based OSCE in Laos. As described in previous studies, the implementation of an OSCE at the school or national level is challenging and requires many practical considerations [22,23], such as recruitment, training, and retention of personnel; test development and implementation costs; and facilities to administer the test [24]. Even though they are complex, costly, and resource intensive, many studies have reported that comprehensive OSCEs have numerous educational bene ts [23,[25][26][27]. For example, one study showed that over a 9-year period of administration, OSCEs produced important improvements in faculty teaching and student performance [25]. In their paper regarding the rst experience of implementing large-scale OSCEs at the national level, Rahayu et al. reported that the OSCE drove both students and teachers to spend more time learning clinical competence and stimulated improvements in clinical teaching [23]. As demonstrated previously in other research [27], further studies to investigate feedback about the testing from examinees and evaluators would be useful to enhance the further development of OSCEs in Laos. This rst SP-based OSCE in Laos was designed for summative purposes according to our study aims, but OSCEs have historically been introduced as formative assessment tools in medicine [28]. Based on our experience, the evolution of a valid and reliable SP-based OSCE t for the Laotian medical context could be expected to complement this study. This study has provided a set of baseline results that will allow further research with different goals to be conducted to track long-term improvements in the clinical performance of Laotian doctors.
This study has some limitations. We ran six OSCE stations per examinee due to the limited availability of time and personnel and the examination cost, which could have in uenced the reliability of our testing. Also, it was beyond the scope of this study to examine the inter-rater reliability or validity, though the workshops we ran prior to OSCE implementation ensured that our evaluators had the necessary psychometric expertise. Although the face and content validity were established by examining the case scenarios and checklists for the six OSCE subcomponent scores with faculty evaluators through the workshops, the precise OSCE measurement instrument and settings appropriate to the Laotian environment need to be determined. In addition, the fact that many of the DLSP fellows were from the Faculty of Basic Science at UHS could have affected the study results measuring clinical performance. Their capacity for teaching and conducting research in their major elds might have been improved by the variety of activities in the program while producing only limited improvement in their performance of primary care. Finally, we used the same OSCE stations on two days because of the large number of examinees, which could have enabled information-sharing among examinees.

Conclusions
In this study, we used an OSCE to examine the educational effects of the DLSP on the clinical performance of Laotian doctors. At the same time, we demonstrated that an SP-based OSCE was feasible in Laos. Even though the OSCE scores did not differ signi cantly in any of the comparisons, we should take into consideration that the primary goals of the DLSP have not focused on improving physicians' clinical performance. Furthermore, given the educational and technological gap between Korea and Laos, the DLSP implementation period might still be too short for the effect of the DLSP to have accumulated in the Laotian medical environment. We believe the experience that the faculty team of Lao UHS gained in conducting the SP-based OSCE for this study will bene t Laotian medical practice, as well as improve the quality of medical education in Laos. As further research, we expect that the clinical competence of doctors experiencing the DLSP will again be evaluated to verify the long-term educational effect of this capacity-building program. The participants were explained the aim and procedures of the study, potential risk and bene ts of participating the study. Anonymity of the questionnaire and the data analysis was reassured to the participants. All participants gave their written informed consent before the study. The study was conducted in compliance with the Helsinki Declaration.

Consent for publication
Not applicable Availability of data and materials The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.