Needs Assessment: Integrating Medical Simulation in Residencies Programs in Saudi Arabia


 ObjectivesThe use of simulation in the curricula of training programs has expanded significantly over the last decade. This study conducted a simulation needs assessment on diverse ongoing residency training programs supervised by the Saudi Commission for Health Specialties (SCFHS) in Saudi Arabia. The goal was to introduce a standardized methodological approach to integrate simulation as a teaching tool for any ongoing training programs.Methoda mixed-methods approach is used in four steps to focus on top educational needs and integrate simulation within the curriculum. In the first step, thirty-eight residency training programs underwent a selection process using the scoring criteria tool. Nine residency training programs were selected to be target programs. This step was followed by champions recruitment, where two faculty representatives from each specialty were oriented about the steps to be process facilitators. The third step is targeted audience needs assessment, consisting of four phases: curriculum review, targeted audience survey, a focus group of stakeholders' interview, and selecting the top identified educational needs. A metrics-based approach was used to rank the list of skills and behaviors by invited stakeholders. The last step was to build common themes using an integration simulation sheet to facilitate integrating simulation into the program curriculum.ResultsOut of 38 programs, nine top-ranked specialties completed the process, and their roadmaps were developed. A critical appraisal facilitated by champions and simulation experts helped the stakeholders to identify the top needed skills and behaviors in each of the nine specialties. The final score proportion for each skill and behavior of the entire combined list was calculated then ranked. The proportion of top needed list of skills and behaviors were obtained as followings: Obstetrics and Gynecology 10 skills and behaviors concluded out of 84 (10/84), Emergency Medicine (ER) 80/242, Intensive care unit (ICU) 20/139, Internal Medicine (IM) 37/102, Pediatric 82/125, Ear Nose Throat (ENT) 49/125, General Surgery (GS) 55/114, Plastic Surgery 24/165, and Family Medicine (FM) 59/168.ConclusionsFindings from this process could provide a standardized approach to supervising bodies at a country level and support decision-makers with prioritizing criteria to be used in the needs assessment to integrate simulation in any other ongoing residency training programs.


Introduction
Postgraduate medical education, particularly residency training programs, is undergoing a signi cant transition. Many countries introduce competency-based training with outcome-based approaches as well as competence-dependent certi cation rather than time-dependent. In the 1990s, the CanMEDS framework (Canadian Medical Education Directives for Specialists) [1], followed by the Outcome Project of the ACGME (Accreditation Council for Graduate Medical Education in the USA) [2][3], were created with ongoing improvement. In 2005, the introduction of the "Entrustable professional activity" (EPA) came about, which gave a trainee the ability to be self-su cient enough for unsupervised practice [4][5][6].
This transition expanded the need for simulation in these programs, whether additional supportive or essential critical needs. Besides the amount of simulation needed in every program, matching ongoing residency training programs with a variant level of mastering competencies during the training period and immediately after graduation is a challenge [7]. Simulation training helps residents' practice and perform basic skills and behaviors they need with a high level of accuracy before they step into a real-life clinical scenario.
Moreover, simulation can be integrated into residency programs to enhance complex medical management and crisis resource management. [8].
Residency training programs are usually supervised by national healthcare regulatory bodies that monitor educational standards. The Saudi Commission for Health Specialties (SCFHS) is the national body that supervises more than 38 residency training programs. Over the last decade, the SCFHS has been affected by the transition from traditional medical education to competency-based training. So much so that it is in the process of working to implement the entrustable professional activity (EPA) education program [9-10].
Multiple factors provide challenges for integrating simulation-based education into residency programs for the SCFHS. For example, introducing simulation activities to the current residency curricula that have never implemented simulation before, transforming part of ongoing training activities into simulation-based activities, aligning simulation-based activities with the Entrustable Professional Activities (EPA) system, and competency-based training all create challenges for a new transformation. Many needs assessment methodologies were described in the literature that lls the training gaps and targets speci c specialties like interviews, surveys, focuses group discussions, and workshops. On the other hand, mixing these methods is recommended to accurately represent the gap and be relevant to the learners' needs. Designing a needs assessment that ts multiple diverse specialties is a national challenge [11][12].
The SCFHS started an initiative to integrate simulation in ongoing residency training programs between 2018 and 2019. The initiative targeted training gaps in every specialty and allowed ongoing training continuity for different specialties. Therefore, this manuscript aimed to build a structured methodology to conduct a simulation needs assessment on the ongoing residency training programs belonging to the Saudi Commission for Health Specialties in Saudi Arabia, prioritizing top needed specialties. As part of gradual implementation, the focus was on nine selected specialties to identify target audience gaps in skills and behaviors in the current curriculum, whether on an individual basis within the specialty or shared gaps between multiple specialties. Any regulatory or national body may utilize the steps in this structural approach of integrating simulation in residency programs [13][14][15][16][17][18][19][20][21][22].

Method
As a national training umbrella, the Saudi Commission for Health Specialization supervises 38 Residency Training Programs all over Saudi Arabia.
Each program has a scienti c council responsible for building a curriculum and ensuring its implementation will be maintained with a high level of training. These programs have different simulation needs ranging from additional supportive needs to essential critical needs. The needs assessment methodology used in this study contains four steps, and four direct simulation needs assessment phases with the same sequence. Table 1 showed a summary of methodology steps with roles and responsibilities to facilitate conduction.
Step 1: Selecting targeted specialties: A brainstorming panel of three physicians with more than ve years of experience in simulation-based education used nominal group discussion to identify the criteria to maximize the chances of selecting the best target specialties. The panel members came up with nine criteria against a linear scale from 1-10 where ten indicate high/strongly agree while 1 indicate the reverse., The criteria were: Was simulation in this specialty applicable and simulation resources available? Was simulation internationally used in this specialty? Did psychomotor skills in this specialty are highly needed? Did the chances for skills training are limited in the form of clinical exposure? Were clinicians in specialty dealing with high patients' acuity? Were clinicians in specialty dealing with crises? Did this specialty has a signi cant impact on patient safety? Was this specialty need strong interprofessional skills? Was this specialty need strong communication skills?
Each program of the 38 specialties was reviewed based on each panel member's voting rank (1)(2)(3)(4)(5)(6)(7)(8)(9)(10). Nine specialties (24%) out of 38 SCFHS programs scored high and were included in the study ( Table 2). The panel considered ve specialties targeted in stage one ( rst year) based on a large number of residents like Obstetrics and Gynecology, IM, and Pediatrics. Also, they considered acute care patterns of practice like ER and ICU.
The remaining four specialties were considered in stage two (second year). The study project for achieving this aim was conducted between 2018-2019 and supervised by three physicians with experience in simulation-based education.
Step 2: Selecting targeted champions For better engagement, it is essential to involve specialty content experts. As a national reference body, SCFHS created a list of champions in each specialty from their data, then the specialty's scienti c council nominated three names then the panel of simulation experts selected 2 of them. The process ended up with two champions representing each specialty. The criteria for selection in all steps were based on one or more of the four following criteria: specialty content quali cation, simulation quali cation, medical education quali cation, and being an instructor in the trainer's courses. A detailed description of champions' tasks was explained in an orientation workshop then handled. Each champion received electronic folders organized in a sequence of numbers with a folder labeled as an orientation package, including a curriculum folder, an electronic survey list, focus group information, and integration simulation sheet and submission document (Table 3).
Step 3: Targeted Audience Simulation Need Assessment: The needs assessment conducted followed a chronological sequence of four phases. The rst three phases ended with a product that contained a list of skills and behaviors. These initial three phases aim to expand skills and behaviors to a larger number so as not to miss any important training gaps. The phases are: Curriculum review, targeted audience survey, and focus group of stakeholders' interviews. The last phase aims to narrow the number of skills and behaviors to a smaller number that is more relevant to be implemented.
Step3. Phase 1: Curriculum Review Simulation champions extracted the rst list of skills and behaviors by reviewing current curricula after the panel of simulation experts reviewed it. The review process aimed to produce the rst broad objective-based list of skills and behaviors.
Step3. Phase 2: Targeted Audience Survey: A simple survey consists of open-ended questions developed by the panel members, including demographic data, skills that impact the residents with limited clinical exposure, the most critical behaviors needed for residents with limited opportunity to practice, and inter-professional skills and behaviors ( Table 4). The survey questionnaire was distributed to the new graduates, program directors, and practicing consultants. Recent graduates and program directors might recall exam performance re ecting training gaps while practicing consultants might recall incidents in daily clinical practice. The Number of responses per each specialty were variant and they are listed in Table 5.
Step 3. Phase 3: Focus Group of Stakeholders' Interview: The specialty champions conducted a half-day focus group meeting for targeted consultants, program directors, and nal-year residents of each of the nine targeted specialties separately. Champions facilitated the meeting by asking structured questions to the participants to obtain more information on needed skills and behavior changes for each specialty. The meeting included three round table discussions. The rst-round table aimed to add skills and behaviors not addressed in curriculum analysis and survey results. By the end of round 1, the expanded list of skills and behaviors (Divergent) is completed.
Step 3. Phase 4: Selecting the top-ranked list of skills and behaviors: During the same interview, the second-round table discussion focused on the ranking list of skills and behavior from the curriculum, survey, and rst round table discussion. The focus group ranked each item from (1-10) using google form as an electronic scoring system. The third-round table discussion focused on re-arraigning skills and behaviors from higher needed to lower needed to be followed by agreement on an imaginary line (threshold line) that include all skills and behavior will be concluded and funded above this line. The champions facilitated the discussion and negotiation to reach an agreement for the threshold line. For focus group discussion workshop curriculum, you can look to Table 6.
Step 4: Building common themes using integration simulation sheet: The panel members established a roadmap integration template to guide specialty champions in integrating simulation to the nal ranked list integration. The sheet has ve primary columns: Themes, needs assessment objectives, type of current delivery method of these objectives, if any, suggested future simulation delivery method, where to t new suggested simulation activities in the academic calendar, and any remarks or ideas ( Figure 1) and (Table 7). The panel members also suggested a drop list to help champions plan for suggested future delivery methods and t new suggested simulation activities in the academic calendar (Table 8). A nal list of skills and behaviors was inserted into the roadmap integration simulation sheet from the top-ranked skills and behaviors obtained from the above three phases. Table 9 include a summary of the number of skills and behaviors extracted from every specialty in step 3 (Simulation Need Assessment Methodology) and Step 4 (Development Integration Simulation Sheet).

Results
At the end of the initiative, nine specialties completed the process, with nine roadmaps completed and submitted to their scienti c councils as the higher national supervising body for specialty training. In addition, roadmaps are all standardized with similar titles, and these titles preceded methodology into practical steps. Table 10 include components of submission roadmap documents.
In Table 9, a stepwise summary of each number of skills and behaviors in each specialty was conducted and correlated to the corresponding educational themes. Interestingly, we found communication issues with patients (e.g., consent taking, breaking bad news, and patient counseling, etc.) and between staff (e.g., crisis recourse management, team-based training) presented as a training gap theme in all specialties except plastic surgery, which focuses on skills mainly. The panel members describe the threshold line that produces the nal ranked list for each specialty out of the combined list of skills and behaviors required for each specialization.

Discussion
Needs assessment is an essential step for the implementation of any simulation curriculum. Suppose we select Obstetrics/Gynecology specialty as an example to explain the process. After reviewing the curriculum, their champions listed 49 skills and behaviors, then they added ve skills and behaviors not mentioned in the curriculum after survey analysis. When champions conducted focus group discussion (Phase 3), the participants listed an additional 30 skills and behavior after the rst round table discussion and came up with 84 as a combined list. After the third-round table brainstorming, participants agreed to have ten skills and behaviors above the agreed imaginary threshold line and will be concluded and worth funding. The champions used Integration Simulation Sheet and divided the nal ranked list into 3 themes as step 4. See Table 9 to compare other specialties.
Moreover, the panel developed the specially designed train the trainers (TOT) guidelines on conducting the simulation process. These guidelines include many areas related to the instructor's quali cations (see Table 11 for TOT guidelines), such as developing the course objectives, design, and facility. The TOT simulation education program aims to create a safe learning environment for residents that will help them understand essential concepts and gain the knowledge and on-the-job training experience to help them increase positive patient outcomes. Trainers who participate in this program must have a healthcare background, be registered with the SCFHS, and hold a quali cation in healthcare simulation. More importantly, they must prove they have experience in the Train the Trainers (TOT) simulation course [23][24][25][26][27].
Many works of literature described using multi-sources to collect training gaps' information to facilitate de ning more accurate training gaps. Mann and Chaytor described conducting a large-scale survey and then focus group discussion as a needs assessment methodology to assess a speci c group of physicians [28][29]. The authors used the chronological sequence of reviewing the curriculum, conducting the survey, and then the focus group discussion.
We found in the sequence of targeted audience simulation needs assessment that the curriculum might not cover some skills and behaviors that are sometimes taught individually in scattered activities. On the other hand, sending a simple targeted audience survey through emails could identify critical learning gaps, and the focus group of stakeholders' interviews added additional learning gaps not addressed by earlier methods. Interestingly, several skills and behaviors showed similar trends in most of the nine specialties in all phases: total combined list and nal ranked list (Table 9).
Salzman et al. and his group used a six-step approach to establish a standardized proposal submission and evaluation mechanism. They targeted 190 training programs and 1136 trainees. Their approach was based on recruiting simulation experts to build a structural format to review curricula. Then they recognized those curricula might bene t from simulation experts mentoring and supporting by making proper decisions regarding funds [30].
Moreover, Wehbi et al. published a needs assessment methodology for simulation-based training focused on a single-specialty (i.e., emergency medical practice) to build customized curricula targeting on-site training using mobile units in rural areas to cover training gaps [31].
The methodology sequence started with a focus group, conducted surveys, and analyzed results. As a comparison, the needs assessment gap in our work started with a curriculum review of ongoing structural training programs, followed by an open-ended question survey, then a focus on specialized group workshops. It would be worth comparing emergency medicine themes from our work with Wehbi et al. work to nd overlapping areas which is out of this paper's scope.
A previous work of Arab A et al. was found using other approaches in assessing simulation in anesthesia residency training programs in Saudi Arabia [32]. The author set an implementation plan to be staged gradually between 2014-2018. It started by implementing mandatory standardized workshops that matched specialty curriculum and ongoing training activities [33]. Because of that implementation plan, we exclude Anesthesia from our work. We described an approach that has not been previously implemented in Saudi Arabia. Nine medical programs were included in this process and described as a national standardized needs assessment that can be used for different postgraduate specialties. Therefore, the allocation of the existing training programs under the umbrella of the SCFHS was our rst step in this study to facilitate the standardized communication between simulation experts and specialty content experts (specialties scienti c councils).
One of our limitations in this study was retrieving the process retrospectively. That was related to the process pattern, which is based on re ective learning until we reach a mature process. Another limitation was listing skills and behaviors in the survey without writing structural objectives, which might not re ect the exact meaning of the gap, e.g., communication with the patient might be related to breaking bad news or dealing with di cult patients... etc. This gap was raised and managed during focused group discussion by participants and lling up the integration simulation sheet by specialty champions. Also, we found in sending the survey in Step 2 that we missed the exact number of targeted groups to calculate the response rate. It happened because we sent the survey to all specialty stakeholders without con rming receiving or counting sent surveys.
The needs assessment detailed gaps in every specialty regarding a list of objectives, themes, modules, suggested educational activities, and others that were not addressed in this paper since it is out of scope. Going deep into these details in every specialty might help content experts review these gaps and support their decisions and future publications. It will be interesting to measure secondary outcomes from applying described methodology on training programs, e.g., level of graduates' competencies or patient safety [34]. Many previous studies elsewhere illustrated the importance of assessing the secondary outcomes for any methods used in establishing simulation needs assessment, particularly for required skill and behaviors [35][36][37][38][39].
Implementation Challenges: The approach used and described in our study was a creative idea supported by previous needs assessment work. We described some challenges for better learning in the future and might support the simulation community [11], [40][41].

1-Time and space
For example, the regular meetings conducted between the simulation experts' panel and champions had enriched the feedback on the process used. There were mainly developed to obtain a concurrent re ection, and they reported immediately after each step with review and feedback.
Consequently, signi cant modi cations and reviews of ongoing processes were done, which challenged the speed of the process and prolonged the required time to nish the process. However, we can a rm that our approach would reduce the required time for practicing this needs assessment method for simulation at any program. It would take no more than 10 -12 weeks to complete each cycle and develop a specialty road map without overstretching involved champions and stakeholders.

2-Champion commitment
One of the challenges was champion commitment. Two of the champions were replaced because of no response. At the same time, another specialty needed a third additional champion to re-do part of their work because it does not follow the same methodology.

3-Variation in required skills by specialties
Other challenge we found was the variation between specialties regarding the pattern of training and method of delivering their education activities. For example, surgical specialties had more skill-based activities. Another example was emergency medicine; they had a long list of required skills and behaviors. Their training was usually based on joined rotations with other specialties.

4-Logistics and Accreditation
For successful implementation, practical steps need to be addressed within the scope of this initiative, such as readiness of simulation centers, availability of equipment, and level of educators in terms of using simulation tools. The panel of three physicians with experience in simulationbased education at SCFHS studied infrastructure preparedness to ensure proper and smooth implementation of the initiative. Consequently, summary road map reports were submitted to simulation centers and program directors, asking for feedback. The feedback focused on the availability of similar training centers, the possibility of creating new activities, and the cost required for establishing such a site.

Conclusion
We think lessons learned from this work can add to any group planning for similar work. As an example: 1) compare literature needs assessment having different chronological sequences with our work in terms of primary outcome 2) support any organizations or national level umbrella with suggested needs assessment methodology and prepared guided forms 3) learn from challenges during this process 4) awareness of cost-effective needs assessment methodology that focuses on implementing what is needed 5) provide solutions for logistics and how it affects our needs assessment implementation.
We offer medical practitioners interested in implementing the concluded approach in simulation needs assessment to be approachable and easily practiced with a standardized method following the Needs Assessment method (steps and phases).
In this work, we found that implementing this descriptive methodology might help target the training gaps and support organizations and decisionmakers to properly supervise residency training programs and enhance the required integration of simulation into ongoing residency training programs in the country.

Declarations
Step Name Phase Name Roles and responsibilities Step Stakeholders will vote to narrow the list of skills and behaviors to most needed Step 4: Building common themes using integration simulation sheet     After ranking, the average ranks will be taken for every skill/behavior. All skills and behaviors will be rearranged and prioritized based on total marks (higher above and lower below).
All attendees brainstorm and agree on one imaginary line (threshold line) after discussing different opinions, including minimum skills and behaviors that need to be involved and funded above this line.