Standardized Assessment of Simulation Based ECMO Educational Courses: A Pilot Study

Ahmed S Said Washington University in St Louis School of Medicine Elaine Cooley Extracorporeal Life Support Organization Rodrigo Diaz Clínica Las Condes.: Clinica Las Condes Elizabeth A Moore UI Heart and Vascular Center: The University of Iowa Hospitals and Clinics Kiran Shekar The Prince Charles Hospital, University of Queensland Christopher Beaty Nemours A.I. duPont Hospital for Children Timothy M Maul Nemours Children's Hospital Ramanathan Kollengode (  ram_ramanathan@nuhs.edu.sg ) National University Singapore Yong Loo Lin School of Medicine https://orcid.org/0000-0003-18229455 Bishoy Zakhary Oregon Health and Science University

and post-course knowledge forms). Physician-intensivists were the largest group (53%) and the majority practiced at academic hospitals (51%) and had less than 1-year of ECMO experience (50%). Post-course, participants reported signi cant increases in con dence across all domains (cognitive, technical, and behavioral; p<0.0001 CI:1.2-1.5, p<0.0001 CI:2.2-2.6, and p=0.002 CI:1.7-2.1, respectively) as well as an increase in knowledge scores (p<0.001, CI: 1.4-2.5). These ndings were consistent for all specialties and prior ECMO experience. There were also signi cant reductions in the times to critical actions in 3 of the 4 scored simulation scenarios.
Conclusions: We successfully developed and tested a comprehensive standardized ECMO course assessment tool, demonstrating participants' self-reported bene t as well as improvement in both knowledge and psychomotor skill acquisition . Standardized course evaluation is feasible and potentially provides important information to improve ECMO courses. Future steps include national implementation, addition of questions targeting clinical decision making to further assess knowledge gain, and multilanguage translation for implementation at international courses.
With the exponential increase in ECMO use, there has been a parallel surge in ECMO educational programs worldwide re ecting recognition of the high cost, relatively low volume, intense resource utilization, and high risk nature of ECMO support, mandating that clinicians be properly trained [9,10].
Despite this growth, however, there is a lack of consensus on the curriculum and structure of these educational programs [11]. As such, ELSO created a dedicated taskforce, ELSOed, charged with de ning international educational needs and identifying opportunities for standardization and international collaboration. An ELSOed position statement outlined educational priorities [12] to include standardizing an ECMO curriculum for delivery at ECMO courses, to standardize a process for ECMO course evaluation, and to outline mechanisms to evaluate the educational bene ts of ECMO courses worldwide. The Kirkpatrick evaluation model is a well-de ned method for objectively measuring the effectiveness of training programs [13]. This model outlines the four level evaluation framework: reaction, learning, behavioral change, and organizational performance. Reaction evaluates the trainees' perceptions about the program and the usefulness of the material to their work; learning gauges the participants' developed expertise and knowledge and is ideally assessed prior to ("pre-test") and following ("post-test") training.
Behavioral change and organizational performance assess the differences in participants' behavior and impact on their work after completing the program, thereby requiring long-term follow-up.
The goal of the present study was to develop a standardized online course assessment tool, based on the Kirkpatrick evaluation model, to assess the educational bene ts of ECMO courses utilizing selfassessment, knowledge assessment, and psychomotor skill evaluation forms in a simulation setting.

Study design
We performed a prospective assessment of US-based adult ECMO courses, ELSO-run or ELSO-endorsed, using online forms. The study was reviewed by the institutional review board at Washington University in St. Louis and was granted waiver of consent.
ELSO and ELSO-endorsed ECMO courses have a standard structure for 25 hours over 3.5 days with 14.5 hours of simulation. Participants are provided with a copy of the ELSO Red Book and Speciliast Training manual on arrival. Didactics consist of structured lectures covering ECMO basics, circuit components, physiology, cannulation, complications, and literature review as well as interactive case studies provided by physicians, nurses, and perfusionists with ECMO expertise. Simulation makes up over 50% of course duration and is immersive with 2-3 instructors per 6 participants. Scenarios are high-delity incorporating an ECMO circuit connected to a mannequin via a simulated dual site cannulation [14]. In addition to realistic circuit variables, this setup incorporates patient vital signs, ventilator settings, laboratory test results, and radiologic images. Simulation scenarios are followed by structured debrie ng sessions according to the "Three-Phase Debrie ng Technique" [15].

Study Subjects
All course participants were eligible for this study and received individual links to pre-and post-course assessment forms hosted on the Google Forms platform. Completed form data was centrally collected, de-identi ed, and analyzed.

Assessment Forms
Pre-and post-course forms were developed by representatives from the ELSOed Course workgroup (AS, EC, RK, and BZ), following the rst 2 levels of the Kirkpatrick evaluation model: reaction and learning.
Course administrators were blinded to the form results. Pre-course forms included: 1) participant demographics, 2) participant self-assessment regarding ECMO management, and 3) knowledge assessment. Post-course form included: 1) participant self-assessment regarding ECMO management, 2) knowledge assessment, and 3) skills assessment during the course. The self-assessment forms covered cognitive, technical, and behavioral aspects of ECMO management. Assessment categories were de ned to cover the basic knowledge and management skills identi ed as learning objectives for the courses. All questions were scored on a 5-point Likert scale (from "least con dent" to "most con dent") (Supplementary digital content 1). Knowledge assessment forms were composed of 15 multiple choice, simple recall questions on ECMO physiology, management, and circuit setup. There were no repeat questions between pre-and post-knowledge assessments.
Four ECMO emergency simulation scenarios were repeated during the courses -initially as a standalone simulation, and then on the nal day of the course as part of a mega-simulation where several scenarios were presented back to back as part of a clinical scenario. For each of these scenarios, actions critical to successful resolution of the scenario were identi ed for timing. As the multidisciplinary teams participated in these scenarios, times to each of these critical actions were recorded. The scenarios included: breach of circuit, recirculation on veno-venous (VV) ECMO, ventricular tachycardia (VT) on VV ECMO with conversion to veno-arterial (VA) ECMO, and air entrainment into the ECMO circuit.

Statistical Analysis
Statistical analysis was conducted in R (R Core Team, 2020) and gures were produced using the package ggplot2 (H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016). Categorical variables are presented as n (%) and quantitative variables are presented as median and interquartile range, unless otherwise stated. Wilcoxon signed rank sum test was used to compare preand post-course results. Pre-and post-course intergroup analyses were performed using Kruskal-Wallis one-way ANOVA tests. These subgroup analyses were solely explorative. A p-value of <0.05 was considered signi cant. Cohen's d was used to assess effect size based on differences between means with d of 0.2 representing small, 0.5 medium, and 0.8 large effect sizes.

Results
Between March 2019 and February 2020, 211 participants in 5 ELSO and ELSO-endorsed courses participated in the study. Of these, 107 (51%) completed both pre-and post-course assessment forms with 97 participants (91%) completing the pre-and post-course knowledge assessment forms ( Figure 1).
The median duration between pre-course form completion to course start day was 2.61 days IQR [1.05, 5.7] with 1.05 days [0. 2,4.95] between the course end day and post-course form completion. Physician intensivists made up over half of participants (53%), followed by nurses (11%), with the majority working in academic hospitals (51%). The majority of respondents had more than 5 years of clinical experience (62%) though half had less than 1 year of ECMO experience (50%). Table 1 displays the participant demographics and ECMO practices provided at the participants' home institutions.

Knowledge Assesment
There was a signi cant improvement in knowledge assessment scores (10 [9, 12] vs 12 [11,13], pre-vs post-course, p<0.001, CI 0.8 to 1.9). There were no signi cant differences in either the pre-or post-course test scores among participants' specialty, age group, or clinical experience. There were signi cant improvements in the post-course scores across specialties and experience level. On Cohen's d calculation, there was a moderate effect size (³0.5) in all subgroup analyses with larger effect sizes (³0.7) in nonintensivists, non-physicians, and participants with less than 1 year of ECMO experience (Figure 3

Discussion
This study presents the rst implementation of a structured assessment tool for simulation-based ECMO courses evaluating changes in participants' self-reported con dence in managing patients on ECMO in addition to assessments of their knowledge gain and change in psychomotor skills. We present the preliminary results from ve US-based courses. Our ndings show (1) improved participants' self con dence in managing ECMO patients across all specialties and experience level; (2) improved knowledge, and (3) signi cant reductions in times to complete critical actions during simulated ECMO emergency scenarios.
To objectively assess the educational bene ts of ECMO courses, we followed the commonly accepted Kirkpatrick model of training course evaluation by prospectively evaluating course participants both before and after the courses. We selected levels one and two of the Kirkpatrick model, as they are the short-term levels. Our study population was heterogenous with a variety of disciplines and specialties as well as varied clinical and ECMO experience. This is of pragmatic value representing the typicaly multidisciplinary team involved in managing ECMO patients.
Our study showed the ability of the assessment tool in identifying signi cant improvement in participants' self-con dence with ECMO management. These ndings are consistent with previously reported improvements in self-con dence in participants of one-day simulation-based ECMO courses for intensive care physicians, nurses, thoracic surgery residents, and ECMO specialists [16] [17] [18] . Others have reported improvement in con dence levels for a group of ECMO-novice practitioners following simulation training [19]. In addition to including a signi cantly larger cohort at 5 different courses nationwide, our results complement these studies by demonstrating improvement across specialties and clinical or ECMO experience. Additionally, the comprehensive tool presented in this study assesses improvements across the 3 categories: cognitive, technical, and behavioral. Such a comprehensive assessment is of special importance in ECMO education where the multidisciplinary team's con dence in both technical and non-technical aspects is essential to effectively provide this life-saving therapy [20].
We also found improvement in knowledge assessment scores across specialties and experience levels. These ndings are consistent with other studies showing signi cant knowledge gain in ECMO-naive critical care fellows following simulation-based training [21]. Similar knowledge gain following one-day and two-day high-delity ECMO simulation courses has also been shown with a other course evaluations [22,23]. Despite the statistically signi cant knowledge gain in our participants, regardless of specialty or experience, there was a noticeably narrow range for both pre and post-course scores. We supplemented the evaluation with further effect size analysis that demonstrated at least a moderate postcourse knowledge gain in all groups. There was a more signi cant effect in participants with less ECMO clinical decision-making experience, including non-physicians, non-intensivists, and those with less clinical and ECMO experience.
For the simulation assessments, there was a signi cant decrease in the time to critical action completion in all tasks for 3 of the 4 evaluated sceanarios. These ndings are consistent with previous reports [21]. Zakahry et al. found that simulation-based training improved time to completion of critical actions for novice ECMO practitioners. More importantly, these results persisted on long-term follow up [21]. Similarly, it has also been shown that there was a decrease in the number of technical errors following simulation, even though reduction in reaction time did not reach statistical signi cance [24], while Burkhat et al. reported a signi cant improvement in time to critical action following simulation training [18].
Previous studies of ECMO courses have been limited with the majority focusing on the feasibility and bene t of high-delity simulation [18,21,[25][26][27][28]. A recent publication highlighted an abbreviated ECMO course for non-surgical clinicians [22] and reported an improvement in participants' knowledge and con dence following the 1-day ECMO course. Their study was limited to participants with no previous ECMO experience and focused on the initiation of ECMO with no training on the subsequent management of patients or on weaning of ECMO support. The nature of the knowledge tests and the con dence assessments were not clearly delineated in the study. The current study developed and utilized a tool for assessing the impact of ECMO courses with participants of multidisciplinary representation and of varying experiences in addition to providing a structured assessment of post-course knowledge, con dence, and skill gain.

Strengths and limitations
This study has several strengths. First, it provides a comprehensive structured evaluation for simulation based ECMO courses. Second, the study re ects the applicability of the assessment model across a large multidisciplinary cohort that represents the different specialties involved in caring for patients on ECMO.
Finally, we outline a comprehensive approach to assess the short-term educational bene ts of simulation-based ECMO courses that can be applied on both national and international levels.
The current study should be interpreted within the context of certain limitations. First, only 50% of participants completed all the pre-and post-course forms. This is likely due to the fact that completion was voluntary. Our results, in addition to the constructive feedback collected on these forms, provide an incentive for course organizers to include assessment form completion as a mandatory part of the course in the future. Second, the study was limited to US-based courses.We are currently translating the assessment forms to allow for implementation at international courses. Third, as mentioned above, the knowledge questions were limited to rapid recall questions, which likely limits our ability to detect signi cant differences in knowledge acquisition and critical application; the incorporation of multi-step logic questions may allow for a more in-depth evaluation of participants' knowledge gain. Furthermore, evaluation on Kirkpatrick levels 1-2 has limited ability to extrapolate ndings to real-life performance and patient outcome improvements. Additionally, it is important to acknowledge the limitation of selfcon dence as a surrogate for competence, as extensively studied in medical education. This highlights the need for long-term follow up of participants to evaluate the impact of the ECMO course on their daily work practices and while allowing for assessment of levels three and four of the Kirkpatrick model.

Conclusions
This study demonstrates the successful development of a comprehensive ECMO course assessment tool. Implementation of this new tool demonstrated improvement in participants' self-con dence with management of ECMO patients in addition to improvement in both knowledge acquisition and time to critical action in simulated scenarios. Future directions include national and international implementation and evaluation of long-term effects of ECMO courses on participants' work practice.   Pre and Post-course self-assessment results