Simulation-Based Assessments in Health Professional Education: A Scoping Review

doi:10.21203/rs.3.rs-3340867/v1

Download PDF

Research Article

Simulation-Based Assessments in Health Professional Education: A Scoping Review

https://doi.org/10.21203/rs.3.rs-3340867/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

The utilization of simulation in healthcare education has grown significantly in the last 20 years. Although typically used for training purposes, simulation is increasingly used for assessing technical skills needed in clinical practice. Despite its common use, there has been no investigation into the full potential of simulation in healthcare education. This study aims to explore the scope of simulation-based assessments in healthcare education.

Methods

Using the Joanna Briggs Institute’s scoping review methodology, we searched Ovid MEDLINE, Embase, Scopus, WOS, CINAHL, APA PsycINFO, and Embase databases and two well-known simulations in medical education, Clinical Simulation in Nursing and Simulation in Healthcare, from 2021 to 2023. We also checked the reference lists of review articles. We imported, deduplicated, and screened the search results in EndNote X20. We included 49 studies that met our criteria and analyzed them descriptively and narratively.

Results

We specified the target groups and applications for simulation-based assessment in healthcare professional education. We also identified challenges, advantages, and disadvantages, as well as the necessary conditions for conducting such assessments.

Conclusions

Simulation-based evaluation is beneficial for health education in evaluating competence and safety. Nevertheless, limitations such as high costs and low realism should be taken into account. Stakeholders and educators must study these concerns, establish standards for evaluation and sampling, and conduct planning and evaluation before using these methods.

Simulation-Based Assessments

Health Professional Education

Scoping review

In its broadest sense, assessment refers to the process of selecting relevant standards and criteria and using judgment to determine quality (1). At various points in the learner's educational journey, numerous assessment types can be used to measure performance or skill development, among other goals. However, there has been substantial discussion regarding the efficacy of certain assessment methods, such as multiple-choice question exams, and this has affected educators' desire to create evaluations that are more performance-based and realistic (2). There have been numerous reviews of the evaluation methods used in pre- and post-registration health professional education (HPE) (3, 4, 5). Determining competency in practice is a compounded challenge for assessments of health professionals and health students. This is a difficult yet important part of education and training. Since educators have become aware that these kinds of evaluations have a significant impact on how students learn and how they are taught in more recent decades, performance-based assessment approaches have gathered significant momentum (6). The use of simulation-based assessment (SBA) resulted from this demand for authentic assessment.

Since the early 1900s, the aeronautical industry and military have employed simulation as a training and evaluation method. The first flight simulator was created in 1929 (7).

Since the 1950s, simulation has become increasingly complicated and sophisticated, mostly due to the integration of computer-based systems. The usage of simulation as a teaching tool has increased practically exponentially as a result of its translation into health education. The goal of simulation is to mimic real patients, anatomical areas, clinical duties, or actual events in clinical settings (8).

The use of simulation-based learning and assessment in health education is on the rise, which can be attributed to a variety of factors, including new diagnostic and therapeutic approaches, increased competition for clinical educators' time, training opportunities to simulate difficult or unusual clinical events, and a stronger emphasis on patient safety (9, 10, 11).

As a result, simulation has been accepted by health educators as an effective teaching strategy for a variety of clinical and nonclinical abilities. Health education uses a variety of simulation modalities, including standardized patients (SPs), anatomical models, part-task trainers, computerized high-fidelity human patient simulators, and virtual reality (12, 13).

These methods have been employed mostly in preregistration health professional training because they maximize learning possibilities and patient safety by allowing students to practice before clinical placement and patient contact (14, 15).

A safe environment is provided via simulation to practice clinical skills in a staged progression of increasing difficulty that is suitable for the learner's proficiency. It can be expensive, time-consuming, difficult, perhaps dangerous, and unethical to practice skills on actual patients (11, 16).

SBA is popular among health professional educators for evaluating students and professionals. It replicates real clinical practice. Despite simulation's widespread use in health education, there is no study on its scope for health professions education. Conducting such a review could provide useful results. Thus, this study aims to investigate the reach of simulation-based assessments (SBAs) in health professions education.

This proposed scoping review was carried out methodically using the five-stage methodology developed by Arksey and O'Malley for conducting scoping reviews, namely: 1- Identifying the research question; 2- Identifying Relevant Studies; 3- Study Selection; 4- Charting the Data; 5- Collating, Summarizing and Reporting the Results; with an additional Optional Stage, Consultation (17). The review was submitted following the checklist for Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) (18) and the Joanna Briggs Institute (JBI) Reviewers' Manual (19).

Stage 1: Identifying the research question(s)

Research questions were formulated to direct the search and find literature for this scoping review.

1. What are the target groups of SBA in health professions education?

2. What are the applications of SBA in health professions education?

3. What are the challenges of SBA in health professions education?

4. What are the advantages of SBA in health professions education?

5. What are the disadvantages of SBA in health professions education?

6. What are the necessary conditions for conducting SBA in the education of health professions?

Stage 2: Identifying relevant studies

The following databases and search engines were used in the review: Ovid MEDLINE, Embase, Scopus, WOS, CINAHL, APA PsycINFO, and Embase. Articles published in two renowned simulations in medical education journals from January 2021–2023, including Clinical Simulation in Nursing and Simulation in Healthcare, and the reference lists of review articles were searched manually for a comprehensive search.

The key search terms included (physician* OR anesthesiologist* OR anesthetist? OR cardiologist* OR dermatologist* OR endocrinologist* OR gastroenterologist* OR hepatologist* OR "General Practitioner*" OR geriatrician* OR gerontologist* OR gynecologist* OR nephrologist* OR neurologist* OR obstetrician* OR oncologist* OR ophthalmologist* OR otolaryngologist* OR otologist* OR pathologist* OR pediatrician* OR neonatologist* OR pulmonologist* OR radiologist* OR rheumatologist* OR surgeon* OR urologist* OR nurse* OR paramedic* OR anatomist* OR audiologist* OR dentist* OR pharmacist*) AND (simulation OR simulator* OR "virtual reality" OR "augmented reality" OR "mixed reality" OR manikin* OR mannequin*) AND (assess). All database searches were carried out by a knowledgeable librarian, who also managed the review's records and data. Supplementary 1 outlines our search strategy.

Stage 3: Study selection

All studies were imported into EndNote X20 and duplicates were removed. Two independent reviewers screened titles, abstracts, and papers based on inclusion/exclusion criteria. Discrepancies were resolved by a third reviewer. Additional papers were found through reference list analysis. A Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flow diagram (18) (Fig. 1) will display the complete article selection procedure. Table 1 lists inclusion and exclusion criteria.

Table 1

Inclusion and exclusion criteria.
Inclusion criteria	Exclusion criteria
- From the beginning of 1994 until the January of 2023 - All SBAs in health profession - All research designs except reviews - Original research, editorials. Conference papers, reports, thesis - In all countries and ethnic groups - Include all Undergraduate and Postgraduate programs - The full text of the article is available	- Outside the mentioned time period - Review articles and books - Non-health profession education - Lack of access to the full text of the article

Stage 4: Charting the data

Data were collected by two independent researchers who were blind to each other's choices, using the JBI-recommended methodology (19) Based on the study team's agreement, a data extraction tool (17) was created to capture the essential details about the source and conclusions pertinent to the review questions. Then, to guarantee consistency in their methodology and to further agree on the data extraction tool, two of the authors independently extracted the data from the first five included studies. For each of the included papers, we collected the title, authors, year and type of publication, location of studies, study population, study design, publication type, simulation method and challenges, advantages, disadvantages, application, and useful tips for simulation (Supplementary 2). Finally, a total of 49 retrieved papers were determined to be pertinent to the review questions. In Additional File 2, a summary of these documents is provided.

Stage 5: Collating, summarizing, and reporting results

A descriptive technique was used to collate, summarize, and categorize the literature. The scoping review's six research questions are addressed in the framework's fifth and final step, which also analyzes the results and provides a report on them.

The results are presented as follows while the featured studies are first given a general overview and descriptive-narrative summary (Supplementary 3). Thereafter, six research questions were answered.

Descriptive summary of the included studies

This review includes 49 studies in total. There were published research between 2000 and 2023. The majority of studies (26, 53%) were carried out in the USA. Canada with (7, 14/3%), Australia (5, 10/2%), England (4, 8/2%), Netherlands and Denmark each (2, 4/1%), Finland are Scotland and Switzerland each (1, 2%) published articles were in other levels.

The highest percentage of studies were focused on surgical specialists (20.4%, 10%), anesthesiology groups including anesthetists and nurse anesthetists (24.6%, 12.0%), and emergency groups (24.6%, 12.24%). The target group for these studies was specialists in various medical fields (22, 9.44%), followed by nurses (9, 18.4%).

The type of simulation methods used for (SBAs) in (HPE)

Simulation methods used in SBAs for health professions education include Standardized/patient simulators (20, 21, 22), anatomical models, advanced patient simulators (APS) (23), Structured Clinical Skills Examination (OSCE) (21), high fidelity manikins and synthetic models (21, 24), virtual clinical stations and virtual structured clinical examination (21), cadavers (25), animal tissues and models (24, 26), simulated environment (i.e., theater) (27), computer and electronic simulations including NeuroTouch simulator (28), bronchoscopy simulator (29), colonoscopy simulator (30), transesophageal echocardiography simulator (31), Cisro endoscopy simulator (32), virtual reality (VR) simulators like Mimic TM Flex VR (26, 29) and EyeSi,(33), and augmented reality simulator like CHARM (34).

Target groups examined in SBAs of HPE

The study of SBAs in health professions education includes a range of target groups such as specialists (29, 30, 32, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45), pre-hospital care providers (PHPs) (46), nurses (20, 47, 48, 49, 50, 51, 52), emergency medicine personnel (21, 53), respiratory therapists (54), medical emergency technicians and paramedics (54), pharmacy personnel and pharmacists (27, 55), intensive care providers,(56) family physicians (39, 43), and midwives (57).

Applications of SBA methods in HPE

In the extracted studies, the final applications of (SBA) methods are mentioned, including the following cases:

- Assessment of competence, knowledge, performance, and performance sequence (49, 56, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72),

- Clinical judgment and diagnostic power (56, 60, 62, 65),

- Critical thinking (60),

- Technique of obtaining case history (73),

- Patient counseling and guidance abilities (73),

- Simple and complex clinical reasoning and decision-making skills (73),

- Communication and remote patient management skills (in simulated patient method) (65, 74),

- Technical and non-technical skills (42, 45, 56, 60, 61, 63, 65, 69, 72, 75, 76, 77, 78, 79, 80, 81),

- Deep reflective and contemplative thinking skills (62),

- Management of uncommon findings (62),

- Examination skills for sensitive organs (such as chest and pelvis) (62, 82),

- Teamwork and interaction skills (62, 65, 72, 83, 84),

- Leadership skills in emergencies (62),

- Risk management and preparedness in disasters and crises (69),

- Management of common and rare cases (85).

- Evaluations of both formative and summative aspects that have a significant impact on the fate of individuals, such as the issuance of board licenses and certificates, or the renewal of board licenses (21, 37, 39, 43, 86, 87, 88)

- Assessment of different levels of Miller's pyramid (knows, knows how, shows, does) (89)

- Evaluation of curriculum components that cannot be assessed using traditional methods (37)

Challenges of SBAs methods in HPE

In the extracted studies, the following challenges of using SBA methods have been mentioned:

Lack of realism in current simulators, such as not simulating mental and psychological states (62),
Difficulty in generalizing performance from simulators to real-world environments (41),
Limited assessment of a small sample of skills with simulators and the lack of a comprehensive indicator of overall physician performance (41),
Individual performance assessment instead of team-based assessment (41),
Failure to evaluate factors such as environmental impact on performance (41),
Difficulty in defining precise performance (41),
Challenges in defining and measuring non-technical behaviors (e.g., timely arrival to the workplace, individual behavior in a group, listening to recommendations, empathy towards patients, professional and courteous behavior) (41),
Discussion of the comfort level of test-takers during work in familiar real-world environments versus unfamiliar simulations (41),
Difficulty in defining competencies and capabilities (63),
Challenges in evaluating competencies through simulation due to the lack of stimulation of real-world performance conditions (65),
Discussion on setting standards for assessment (65),
Discussion on appropriate sampling (65),
Discussion on behavior change in simulated patients (65),
Challenges in creating scenarios and synthesizing cases (65),
Difficulty in simulating the human body (65),
Difficulty in reducing performance and outcome measures (65),
The presence or absence of a culture and policy of simulation use (69),
The complexity of healthcare (69),
Existence of multiple stakeholders (69),
Variability of specialized care performances (56),
Difficulty in the development and design of simulations (56),
Difficulty in creating valid assessment tools (56),
Designing appropriate scenarios (83),
Lack of studies on suitability for high-stakes assessments (83),
Religious and/or ethical challenges (e.g., in animal models) (82),
Discussion on appropriate grading methods (85),
Presence of high error sources in (SBAs), including assessors (85),
Development of suitable programs (90),
Discussion on engaging participant involvement (90),
Discussion on appropriate checklist design (91),
Limitations in submitted cases for assessing each physician in SP (91),
Lack of standard scenarios (92).

Advantages of SBAs methods in HPE

The final advantages of SBA methods include high sensitivity, High specificity (20), High validity (23, 25, 30, 35, 36, 38, 48, 53, 93), Satisfaction of learners and test takers (27, 28, 34, 54), similarity to the real environment (23, 26, 43), reduction of patient harm (26, 29, 36, 37, 39, 87, 94, 95), reduction of learner and test taker anxiety, (21, 24, 26, 47, 86) complementarity of learning and testing (21, 24), correlation between simulator and in real performance environment (33, 45), provision of feedback (26, 37, 87, 95), objectivity of assessment (24, 96), usefulness emergency conditions (24), case diversity (87), replicability of simulation, reduction of disparities in assessment (95), ethical superiority (39, 95), increased interest in simulation (39) and structured assessment high reliability (89).

Disadvantages of SBAs methods in HPE

The use of SBA methods has been criticized due to high cost, (26, 57, 86, 89, 94) anxiety among users (21), lack of realism (94), limited validity (89), and disposal concerns (26). Some methods also evaluate only simple skills rather than advanced ones and may suffer from sampling errors.

Necessary conditions for using SBAs in HPE

Test objectives

It must be clearly defined and examined by counterparts, as stated by Decker et al. (23) and Glavin et al. (23, 37) Evidence can be used to identify objectives, competencies, and skills, according to Glavin et al. (23, 37) Assessment objectives should be explicit, which has been emphasized by Rizzolo and colleagues, Boulet and colleagues (52, 94), and Ennan et al. (57) It is important to precisely determine the assessment objective and match the test design to the level of learners and staff, as highlighted by Kardong-Edgren and colleagues (51, 57).

Performance standards in assessment

Individuals such as Macario and his colleagues emphasize the need for specified performance standards and consensus on assessment checklists (41). Identification of clinical activities that can be tested in simulation is also important (48). According to major researchers such as Crosby and his colleagues, precise determination of assessment criteria and use of guidelines for developing assessment tools and criteria is necessary (36). Grand and his colleagues suggest that performance assessment items should be specific and accurate, with a defined minimum passing score (21).

Assessment of examinees and standardized patients

Decker et al. and other researchers emphasize the importance of training both examinees and standardized patients. Prior practice in a simulated testing environment can lead to increased comfort and decreased anxiety for examinees (23, 45). Assessors and facilitators must also be trained according to Rizzolo et al., Lien et al., and Boulet et al. (80, 88) Dunkley et al. notes that if simulation is used through a standardized patient method, the SP must be properly trained (24). Ennen et al. suggest that trained personnel and individuals should work as a team for SBA and education (57). Strinivasan et al. states that if a standardized patient is utilized, the individual must receive proper training (21).

Criterion fidelity (real-world similarity) testing

Fidelity is a complex phenomenon that includes physical and psychological aspects (56). Reliable equipment enhances fidelity testing (23). High fidelity is a key feature of assessment methods (54). Simulations with high fidelity are more effective for education and evaluation. High-fidelity simulation is more important for competency assessment than performance assessment (24). Scenario fidelity is a central and fundamental principle for assessment validity. Technology can be used to increase fidelity (56, 86).

Video recording of examinations

Proposed by Strinivasan et al., the use of high-powered fiber-optic cameras and microphones for video recording is essential for performance assessment (21). Podolsky et al. also recommend video recording for evaluating examinees (93). Video recording allows for repeated viewing of performance according to Rizzolo et al. (52) Shah et al. suggest video recording for assessment and re-evaluation (26).

Expert opinions on assessment

Various studies suggest the use of SBA and involving experts and knowledgeable sources in administering assessments (41). Physicians and faculty members are recommended for validating assessment tools and identifying test cases. The content of the assessment tool should be clear (37, 48). The designed tools should undergo consultation and expert opinions and accuracy should be determined precisely (30). Skilled individuals should be used in the assessment team (87). Obtaining and implementing expert opinions on scenarios is important. Simulation training should be observed before evaluation. (57).

The Application of SBAs Technique: Dunkley et al. suggest that simulations used for competence assessment should recreate all aspects of everyday reality and results should be interpreted carefully. Performance criteria can be identified using evidence (24). Rizzolo et al. suggest that behaviors indicative of competence and ability should be clearly defined when using simulation for competence measurement (52). Hall et al. believe that simulations should not be used alone for evaluating knowledge (87). Technical and non-technical skills should be defined accordingly (24). Nabzdyk et al. suggest checklists for assessing technical skills and overall scoring scales for non-technical skills (56). Ennan et al. recommend valid and standardized scenarios and assessment methods for high-stakes assessments (57). Further studies are needed to establish the appropriateness of simulation for high-stakes assessment according to Kardong-Edgren et al. (51).

Different simulation methods and techniques

Different simulation methods have specific conditions. Dunkley et al. suggest using artificial models and animal tissues for invasive procedures, while electronic simulation can evaluate knowledge and competency (24). Simulated environments can assess teamwork, communication, and crisis management skills (22). Nabzdyk et al. recommend modeling simulation scenario techniques based on real events and choosing criteria and tools based on the skill or task being assessed (56). Schuwirth et al. advise against using the PMP technique for fate-determining examinations, instead preferring open-ended questions and limiting the number of questions (89). Checklists should be designed to measure the consequences of problem-solving correctly. Strinvasan et al. suggests creating an appropriate clinical environment for comprehensive mannequin simulation (21). Shah et al. recommend an SBA simulation tool with accurate anatomy, interactivity, physical, visual, and physiological reality, and haptic reality (26).

Validity and reliability of assessment tools

Studies have discussed the validity and reliability of assessment tools. The validity of assessment tools should be measured using frameworks such as Messick's framework of validity theory (97). Enhancing the reliability of assessment can be achieved through increasing the number of cases, using blind assessors, and standardizing and training assessors. Consultation with experts and piloting simulations is important for creating valid and reliable scenarios (52). Scenarios need to be valid for content validity (23). Providing sufficient time for assessment increases its reliability. Increasing the number of scenarios improves the validity and reliability of the assessment and using expert consensus can enhance validity (94). A large number of cases and a significant amount of time are required to achieve reliability in the progress test (21). The reliability of the Angoff method increases with an increase in the number of stations, appropriate time duration for each station, and having a reliable and accessible checklist. The reliability of the Angoff method depends on the number of observations if the assessment goal is at the "does" level in Miller's pyramid. Well-trained standardized patients must be selected to increase the validity of the assessment using the standardized patient method (89).

Feedback

Feedback is necessary for evaluating the assessment method. Simulations that offer feedback are recommended by Hall et al (87) and Roussin et al. for formative assessment (88).

Curriculum change when using simulation-based assessment

Ennen et al. suggest designing an appropriate curriculum for simulation-based assessment, considering the field and level of test takers (57). They advise that teamwork scenarios should be more complex than those assessing individual performance. Rizzolo et al. recommend modifying and reviewing the curriculum for simulation use, ensuring contextual and conceptual alignment with teaching. They also suggest evaluating individuals under clinical conditions with the use of SBA and other assessment methods (52).

Timing of the assessment

Rizzolo et al. stress the importance of allowing ample time for test takers to complete assessments (52) Boulet et al. also suggest that appropriate time allocation can enhance the reliability of assessments (94).

Cost of assessment

Boulet et al. suggest that using the SBA assessment method is expensive and requires investor attention (94) Rizzolo et al. also emphasize considering the cost-effectiveness of this method (52).

This review adds to the existing literature on simulation-based assessments in health professions education by examining 49 relevant studies. It provides insight into the different dimensions of the subject, including the types of simulation methods, target groups, applications, advantages, disadvantages, and necessary conditions for using simulation-based assessments. New computer simulation techniques and standard/patient simulation are the most common methods used for evaluating competencies in various skills. These methods have high validity and are safe for patients and examinees. Immediate and rapid feedback are additional advantages. However, the high cost is a common drawback mentioned by studies.

The current study discusses important considerations for using SBA methods in health professions education. These include determining test objectives, establishing performance standards, training assessors and examinees, considering fidelity, recording video, incorporating expert opinions, and considering validity and reliability. Feedback and curriculum modification should also be considered. Finally, cost and time requirements should be examined before utilizing these assessment methods. The identification of test and evaluation goals has been extensively discussed by scholars. Boulet and his team suggest that the aim of the test and the abilities assessed should be defined, while the objectives and abilities to be evaluated should depend on the level of the test taker and the type of test administered (94). Assessing a wide range of skills, such as management, diagnosis, treatment, and teamwork, is crucial in professional health education studies. This evaluation method involves integrating knowledge, clinical judgment, and communication (21, 23, 24, 30, 32, 36, 37, 38, 39, 43, 47, 49, 53, 56, 87, 88, 89, 95, 96).

Alsulimani's study on high-fidelity SBA in emergency medicine suggests that a balance is needed between measuring competencies and specific skills and the limitations of simulators and assessment methods. The study also concludes that SBA should not be the only method for evaluating learners' performance (98). Hall et al., Macario et al., and Srinivasan et al. concur that SBA should be combined with other evaluation techniques. This is particularly important for formative and summative assessments. Another requirement for healthcare worker performance assessment through simulation methods is designing appropriate tests for different employee and specialist levels (21, 41, 87, 98).

The validity of evaluation methods and results increases with higher simulator fidelity. Many experts, including Nabzdyk and LeFlore, stress the importance of high fidelity as a requirement for selecting simulation methods in this field (54, 56). Nevertheless, certain studies suggest that low similarity and fidelity simulations can evaluate the clinical competence of specialists. However, it is important to accurately define and specify qualifications in this aspect (29, 54, 56). Glavin et al. found that high and low-fidelity manikins are both appropriate for evaluation (37). Several studies have emphasized the importance of content and simulated scenarios in terms of validity (23, 25, 31, 35, 38, 89, 93). To maintain validity, content should be reviewed by peers, and standards should be established. Expert opinions on scenarios are obligatory (21, 57). Suggestions have been made for optimal utilization of evaluation methods, including emphasis on properly training standardized patients. Invasive procedures are not suitable for this technique (89). When selecting evaluation techniques, one must also consider their cost-effectiveness, as noted by Alsulimani, Rizzolo, and colleagues who identified high expenses as a significant obstacle (52, 98).

VR simulators are used for education and health assessment. Surgeons can practice surgery safely and learn technical skills before operating on patients. VR simulators are cost-effective and efficient compared to live animal models. VR simulation technology will expand its usage. It is better to use SBA methods alongside other evaluation methods for employee and learner performance. Some studies emphasize this idea (41, 87, 89).

The evaluation methods mentioned in current literature reviews can reduce patient complications and protect their physical and mental safety during employee evaluations. Additionally, these methods provide a risk-free environment for evaluation and testing, benefiting both the patient and the examinee (26, 36, 37, 39, 87, 94, 95). The use of anxiety and worry reduction methods in simulator training requires training on the simulation environment and operations, making it more popular among participants than working with real patients (24, 26, 29, 47, 86). The goal is attainable when both parties are familiar with the simulation environment and stress does not affect performance (23, 41, 45, 52). Studies, such as Konge et al., have shown that there is a notable difference in performance between respiratory and bronchoscopy physicians of varying experience levels when working with standardized simulations of transbronchial needle aspiration under ultrasound guidance (38).

Recent studies have not used evaluation methods to assess healthcare professionals like radiologists, radiation therapists, operating room assistants, and medical physicists. Similar studies to evaluate these individuals are recommended. In Iran, no simulation-based evaluation methods have been used for healthcare professionals, particularly in surgery. The use of these methods is widely recognized as beneficial and efficient. There are limited studies in this field due to its novelty, resulting in differing perspectives and complex information. Important topics were extracted and evaluated in an organized approach. The main limitations were the exclusion of incomplete or inaccessible articles and non-English language articles. No SBA studies for healthcare professionals have been conducted in Iran, which is strongly felt, especially in surgery where their use is recommended.

Simulation-based evaluation methods have advantages for health education, including assessing competence and safety. However, drawbacks and challenges must be considered, such as low realism and high costs. These concerns need to be examined by stakeholders and educators, and standards for evaluation and sampling must be set. Planning and evaluation should be conducted before deciding on the use of these methods.

HPE

Health Professional Education

SBA

Simulation-Based Assessment

SPs

Standardized Patients

SBAs

Simulation-Based Assessments

PRISMA-ScR

Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews

JBI

Joanna Briggs Institute

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-analyses

APS

Advanced Patient Simulators

OSCE

Structured Clinical Skills Examination

Virtual Reality

PHPs

Pre-Hospital care Providers

Ethics approval and consent to participate: The Shahi Beheshti University of Medical Sciences (SBMU) review board approved the study protocols with the Ethics Code of IR.SBMU.SME.REC.1401.079.

Consent for publication: Not applicable.

Availability of data and materials: The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests: The authors declare that they have no competing interests.

Funding: Not applicable.

Authors' contributions: Study concept and design: Zohra Khoshgoftar, Somaieh Bosak, and Zahra Siavashpour; Designing the searching protocol: Zohra Khoshgoftar, Somaieh Bosak, and Zahra Siavashpour; Searching and data acquisition: Zahra Siavashpour, Abstract and full text review: Zohra Khoshgoftar, Somaieh Bosak, and Zahra Siavashpour; Revision of the results and writing the manuscript: Somaye Sohrabi, Zahra Siavashpour.

Acknowledgements: Not applicable.

Evans C. Making sense of assessment feedback in higher education. Rev Educ Res. 2013;83(1):70–120.
Ryall T, Judd BK, Gordon CJ. Simulation-based assessments in health professional education: a systematic review. J multidisciplinary Healthc. 2016:69–82.
Cant RP, Cooper SJ. Simulation-based learning in nurse education: systematic review. J Adv Nurs. 2010;66(1):3–15.
Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226–35.
Swanson DB, Norman GR, Linn RL. Performance-based assessment: Lessons from the health professions. Educational researcher. 1995;24(5):5–11.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29(9–10):855–71.
Jones F, Passos-Neto CE, Braghiroli OFM. Simulation in medical education: brief history and methodology. Principles and practice of clinical research. 2015;1(2).
Qayumi K, Pachev G, Zheng B, Ziv A, Koval V, Badiei S et al. Status of simulation in health care education: an international survey. Advances in medical education and practice. 2014:457 – 67.
Gaba DM. The future vision of simulation in health care. BMJ Qual Saf. 2004;13(suppl 1):i2–i10.
Koukourikos K, Tsaloglidou A, Kourkouta L, Papathanasiou IV, Iliadis C, Fratzana A, et al. Simulation in clinical nursing education. Acta Informatica Medica. 2021;29(1):15.
Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical education: an ethical imperative. Simul Healthc. 2006;1(4):252–6.
Ker J, Mole L, Bradley P. Early introduction to interprofessional learning: a simulated ward environment. Med Educ. 2003;37(3):248–55.
Rosen KR. The history of medical simulation. J Crit Care. 2008;23(2):157–66.
Bland AJ, Topping A, Wood B. A concept analysis of simulation as a learning strategy in the education of undergraduate nursing students. Nurse Educ Today. 2011;31(7):664–70.
Seropian MA, Brown K, Gavilanes JS, Driggers B. An approach to simulation program development. J Nurs Educ. 2004;43(4):170–4.
Tun JK, Alinier G, Tang J, Kneebone RL. Redefining simulation fidelity for healthcare education. Simul Gaming. 2015;46(2):159–74.
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:1–9.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group* P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9.
Peters MD, Godfrey CM, McInerney P, Soares CB, Khalil H, Parker D. The Joanna Briggs Institute reviewers' manual 2015: methodology for JBI scoping reviews. 2015.
Heiner JD, Proffitt AM, McArthur TJ. The ability of emergency nurses to detect simulated long bone fractures with portable ultrasound. Int Emerg Nurs. 2011;19(3):120–4.
Srinivasan M, Hwang JC, West D, Yellowlees PM. Assessment of clinical skills using simulator technologies. Acad Psychiatry. 2006;30:505–15.
Weidner AC, Gimpel JR, Boulet JR, Solomon M. Using standardized patients to assess the communication skills of graduating physicians for the comprehensive osteopathic medical licensing examination (COMLEX) level 2-performance evaluation (level 2-PE). Teach Learn Med. 2010;22(1):8–15.
Decker S, Utterback VA, Thomas MB, Mitchell M, Sportsman S. Assessing continued competency through simulation: A call for stringent action. Nurs Educ Perspect (National Leag Nursing). 2011;32(2).
Dunkley M. Competence assessment using simulation. Minim Invasive Therapy Allied Technol. 2000;9(5):341–5.
Holzmacher J, Sarani B, Puche A, Granite G, Shalin V, Pugh K, et al. Can hyper-realistic physical models of peripheral vessel exposure and fasciotomy replace cadavers for performance assessment? J Trauma Acute Care Surg. 2017;83(1):130–S5.
Shah J, Darzi A, editors. Simulation and skills assessment. Proceedings International Workshop on Medical Imaging and Augmented Reality; 2001: IEEE.
Daupin J, Atkinson S, Bédard P, Pelchat V, Lebel D, Bussières JF. Medication errors room: a simulation to assess the medical, nursing and pharmacy staffs' ability to identify errors related to the medication-use system. J Eval Clin Pract. 2016;22(6):911–20.
Alotaibi FE, AlZhrani GA, Mullah MA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, et al. Assessing bimanual performance in brain tumor resection with NeuroTouch, a virtual reality simulator. Operative Neurosurg. 2015;11(1):89–98.
Steinfort DP, Yong YH, Byrne T, Gorelik A, Colt H, Irving LB. Assessment of bronchoscopic dexterity and procedural competency in a low-fidelity simulation model. J Bronchol Interventional Pulmonol. 2018;25(3):198–203.
Grantcharov TP, Carstensen L, Schulze S. Objective assessment of gastrointestinal endoscopy skills using a virtual reality simulator. JSLS: J Soc Laparoendoscopic Surg. 2005;9(2):130.
Bick JS, DeMaria S Jr, Kennedy JD, Schwartz AD, Weiner MM, Levine AI, et al. Comparison of expert and novice performance of a simulated transesophageal echocardiography examination. Simul Healthc. 2013;8(5):329–34.
Zupanc CM, Wallis GM, Hill A, Burgess-Limerick R, Riek S, Plooy AM, et al. Assessing colonoscopic inspection skill using a virtual withdrawal simulation: a preliminary validation of performance metrics. BMC Med Educ. 2017;17:1–10.
Thomsen ASS, Smith P, Subhi Y, Cour Ml, Tang L, Saleh GM, et al. High correlation between performance on a virtual-reality simulator and real‐life cataract surgery. Acta Ophthalmol. 2017;95(3):307–11.
Qian J, Rama A, Wang E, Wang T, Hess O, Khoury M et al. Assessing Pediatric Life Support Skills Using Augmented Reality Medical Simulation With Eye Tracking: A Pilot Study. J Educ Perioperative Medicine: JEPM. 2022;24(3).
Black S, Nestel D, Kneebone R, Wolfe J. Assessment of surgical competence at carotid endarterectomy under local anaesthesia in a simulated operating theatre. J Br Surg. 2010;97(4):511–6.
Crosby E. The role of simulator-based assessments in physician competency evaluations. Can J Anesth. 2010;57(7):627.
Glavin RJ, Gaba DM. Challenges and opportunities in simulation and assessment. LWW; 2008. pp. 69–71.
Konge L, Annema J, Clementsen P, Minddal V, Vilmann P, Ringsted C. Using virtual-reality simulation to assess performance in endobronchial ultrasound. Respiration. 2013;86(1):59–65.
Levine AI, Schwartz AD, Bryson EO, DeMaria S Jr. Role of simulation in US physician licensure and certification. Mt Sinai J Medicine: J Translational Personalized Med. 2012;79(1):140–53.
Lien CA, Warner MA, Rathmell JP. Simulation for assessment of the practice of board-certified anesthesiologists. Anesthesiology. 2017;127(3):410–2.
Macario A. Can physician performance be assessed via simulation? Anesthesiology. 2014;120(1):18–21.
Noveanu J, Amsler F, Ummenhofer W, von Wyl T, Zuercher M. Assessment of simulated emergency scenarios: are trained observers necessary? Prehospital Emerg Care. 2017;21(4):511–24.
Ross BK, Metzner J. Simulation for maintenance of certification. Surg Clin. 2015;95(4):893–905.
Rovamo L, Mattila M-M, Andersson S, Rosenberg P. Assessment of newborn resuscitation skills of physicians with a simulator manikin. Archives of Disease in Childhood-Fetal and Neonatal Edition. 2011;96(5):F383–F9.
Sanford DI, Ma R, Ghoreifi A, Haque TF, Nguyen JH, Hung AJ. Association of suturing technical skill assessment scores between virtual reality simulation and live surgery. J Endourol. 2022;36(10):1388–94.
Skube ME, Witthuhn S, Mulier K, Boucher B, Lusczek E, Beilman GJ. Assessment of prehospital hemorrhage and airway care using a simulation model. J trauma acute care Surg. 2018;85(1):27.
Courtney Crannell R, editor. Chemotherapy administration: using simulation case-based scenarios to assess chemotherapy competency. Oncology Nursing Forum; 2012: Oncology Nursing Society.
Heyes ME, Schnitzen L, Starr DG, Vacchiano C, Muckler VC, Thiemann L, et al. Recertification and reentry to practice for nurse anesthetists: determining core competencies and evaluating performance via high-fidelity simulation technology. J Nurs Regul. 2018;8(4):43–55.
Hinton JE, Mays MZ, Hagler D, Randolph P, Brooks R, DeFalco N, et al. Measuring Post-Licensure Competence with Simulation: The Nursing Performance Profile. J Nurs Regul. 2012;3(2):45–53.
Hock S, Sergel M, Christian E, Carizey R, Perumalsamy P, Patwari R, et al. 293 Development and Assessment of a Simulation Model of Focused and Ongoing Professional Practice Evaluations. Ann Emerg Med. 2016;68(4):114–S5.
Kardong-Edgren S, Hanberg AD, Keenan C, Ackerman AA, Chambers K. A discussion of high-stakes testing: An extension of a 2009 INACSL conference roundtable. Clinical Simulation in Nursing. 2011;7(1):e19-e24.
Rizzolo MA, Kardong-Edgren S, Oermann MH, Jeffries PR. The National League for Nursing project to explore the use of simulation for high-stakes assessment: Process, outcomes, and recommendations. Nurs Educ Perspect. 2015;36(5):299–303.
O'reilly DJ. Computer simulation in mass emergency and disaster response: An evaluation of its effectiveness as a tool for demonstrating strategic competency in emergency department medical responders. Wayne State University; 2011.
LeFlore JL, Anderson M. Effectiveness of 2 methods to teach and evaluate new content to neonatal transport personnel using high-fidelity simulation. J Perinat Neonatal Nurs. 2008;22(4):319–28.
MacFarlane B, Bergin J, Peterson GM. Assessment and management of serotonin syndrome in a simulated patient study of Australian community pharmacies. Pharm Pract (Granada). 2016;14(2):0.
Nabzdyk CS, Bittner EA. One (Not So Small) Step for Simulation-Based Competency Assessment in Critical Care. Crit Care Med. 2018;46(6):1026–7.
Ennen CS, Satin AJ. Training and assessment in obstetrics: the role of simulation. Best Pract Res Clin Obstet Gynecol. 2010;24(6):747–58.
Zupanc CM, Wallis GM, Hill A, Burgess-Limerick R, Riek S, Plooy AM, et al. Assessing colonoscopic inspection skill using a virtual withdrawal simulation: a preliminary validation of performance metrics. BMC Med Educ. 2017;17(1):118.
Roussin C, Sawyer T, Weinstock P. Assessing competency using simulation: The SimZones approach. BMJ Simul Technol Enhanced Learn. 2020;6(5):262–7.
Decker S, Utterback VA, Thomas MB, Mitchell M, Sportsman S. Assessing continued competency through simulation: A call for stringent action. Nurs Educ Perspect. 2011;32(2):120–5.
Steinfort DP, Yong YH, Byrne T, Gorelik A, Colt H, Irving LB. Assessment of Bronchoscopic Dexterity and Procedural Competency in a Low-fidelity Simulation Model. J Bronchol Interventional Pulmonol. 2018;25(3):198–203.
Srinivasan M, Hwang JC, West D, Yellowlees PM. Assessment of clinical skills using simulator technologies. Acad Psychiatry. 2006;30(6):505–15.
Glavin RJ, Gaba DM. Challenges and opportunities in simulation and assessment. Simul Healthcare: J Soc Med Simul. 2008;3(2):69–71.
Crannell C. Chemotherapy Administration: Using Simulation Case-Based Scenarios to Assess Chemotherapy Competency. Oncol Nurs Forum. 2012;39(1):19–22.
Dunkley MP. Competence assessment using simulation. Minim Invasive Ther Allied Technol. 2000;9(5):341–5.
O'Reilly DJ. Computer simulation in mass emergency and disaster response: An evaluation of its effectiveness as a tool for demonstrating strategic competency in emergency department medical responders. ProQuest Information & Learning; 2012.
LeFlore JL, Anderson M. Effectiveness of 2 methods to teach and evaluate new content to neonatal transport personnel using high-fidelity simulation. J Perinat Neonatal Nurs. 2008;22(4):319–28.
Hall AK, Chaplin T, McColl T, Petrosoniak A, Caners K, Rocca N, et al. Harnessing the power of simulation for assessment: Consensus recommendations for the use of simulation-based assessment in emergency medicine. CJEM Can J Emerg Med Care. 2020;22(2):194–203.
Carroll JD, Messenger JC. Medical simulation - The new tool for training and skill assessment. Perspect Biol Med. 2008;51(1):47–60.
Crosby E. The role of simulator-based assessments in physician competency evaluations. Can J Anaesth. 2010;57(7):627–35.
Levine AI, Schwartz AD, Bryson EO, Demaria S Jr. Role of simulation in U.S. physician licensure and certification. Mt Sinai J Med. 2012;79(1):140–53.
Ross BK, Metzner J. Simulation for Maintenance of Certification. Surg Clin North Am. 2015;95(4):893–.
MacFarlane B, Bergin J, Peterson GM. Assessment and management of serotonin syndrome in a simulated patient study of Australian community pharmacies. Pharm Pract. 2016;14(2):1886–3655.
Weidner AC, Gimpel JR, Boulet JR, Solomon M. Using standardized patients to assess the communication skills of graduating physicians for the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2-Performance Evaluation (Level2-PE). Teach Learn Med. 2010;22(1):8–15.
Alotaibi FE, AlZhrani GA, Mullah MAS, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, et al. Assessing bimanual performance in brain tumor resection with neurotouch, a virtual reality simulator. Operative Neurosurg. 2015;11(1):89–98.
Black SA, Nestel DF, Kneebone RL, Wolfe JH. Assessment of surgical competence at carotid endarterectomy under local anaesthesia in a simulated operating theatre. Br J Surg. 2010;97(4):511–6.
Holzmacher J, Sarani B, Puche A, Granite G, Shalin V, Pugh K, et al. Can hyper-realistic physical models of peripheral vessel exposure and fasciotomy replace cadavers for performance assessment? J Trauma Acute Care Surg. 2017;83(1 Suppl 1):130–S5.
Podolsky DJ, Fisher DM, Riff KWW, Zuker RM, Drake JM, Forrest CR. Assessing Performance in Simulated Cleft Palate Repair Using a Novel Video Recording Setup… Gallie Day, University of Toronto, Toronto, Canada, on April 28, 2017. Cleft Palate Craniofac J. 2020;57(6):687–93.
Qian J, Rama A, Wang E, Wang T, Hess O, Khoury M, et al. Assessing Pediatric Life Support Skills Using Augmented Reality Medical Simulation With Eye Tracking: A Pilot Study. J Educ Perioperative Med JEPM. 2022;24(3):E691–E.
Rovamo L, Mattila MM, Andersson S, Rosenberg P. Assessment of newborn resuscitation skills of physicians with a simulator manikin. Archives of Disease in Childhood -- Fetal & Neonatal Edition. 2011;96(5):F383–9.
Skube ME, Witthuhn S, Mulier K, Boucher B, Lusczek E, Beilman GJ. Assessment of prehospital hemorrhage and airway care using a simulation model. J Trauma Acute Care Surg. 2018;85(1 S):27–S32.
Shah J, Darzi A, editors. Simulation and skills assessment2001.
Brazil V. Performance assessment and simulation fidelity for dummies. Emerg Med Australasia. 2008;20(6):456–7.
Grand JA, Pearce M, Rench TA, Chao GT, Fernandez R, Kozlowski SW. Going DEEP: guidelines for building simulation-based team assessments. BMJ Qual Saf. 2013;22(5):436–48.
Boulet JR. Summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med. 2008;15(11):1017–24.
Brazil V. Performance assessment and simulation fidelity for dummies. Wiley Online Library; 2008. pp. 456–7.
Hall AK, Chaplin T, McColl T, Petrosoniak A, Caners K, Rocca N, et al. Harnessing the power of simulation for assessment: consensus recommendations for the use of simulation-based assessment in emergency medicine. Can J Emerg Med. 2020;22(2):194–203.
Roussin C, Sawyer T, Weinstock P. Assessing competency using simulation: the SimZones approach. BMJ Simul Technol Enhanced Learn. 2020;6(5):262.
Schuwirth LW, Van der Vleuten CP. The use of clinical simulations in assessment. Med Educ. 2003;37:65–71.
Ennen CS, Satin AJ. Training and assessment in obstetrics: the role of simulation. Best Pract Research: Clin Obstet Gynecol. 2010;24(6):747–58.
Schuwirth LWT, van der Vleuten CPM. The use of clinical simulations in assessment. Med Educ. 2003;37:65–71.
Kardong-Edgren S, Hanberg AD, Keenan C, Ackerman A, Chambers K. A Discussion of High-Stakes Testing: An Extension of a 2009 INACSL Conference Roundtable. Clinical Simulation in Nursing. 2011;7(1):e19-e24.
Podolsky DJ, Fisher DM, Riff KWW, Zuker RM, Drake JM, Forrest CR. Assessing performance in simulated cleft palate repair using a novel video recording setup. The Cleft Palate-Craniofacial Journal. 2020;57(6):687–93.
Boulet JR. Summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med. 2008;15(11):1017–24.
Carroll JD, Messenger JC. Medical simulation: the new tool for training and skill assessment. Perspect Biol Med. 2008;51(1):47–60.
Lee KHK, Grantham H, Boyd R. Comparison of high-and low‐fidelity mannequins for clinical performance assessment. Emerg Med Australasia. 2008;20(6):508–14.
Kane M. Validating score interpretations and uses. Lang Test. 2012;29(1):3–17.
Alsulimani LK. The feasibility of simulation-based high-stakes assessment in emergency medicine settings: a scoping review. J Educ Health Promotion. 2021;10.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Simulation-Based Assessments in Health Professional Education: A Scoping Review

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Methods

Stage 1: Identifying the research question(s)

Stage 2: Identifying relevant studies

Stage 3: Study selection

Stage 4: Charting the data

Stage 5: Collating, summarizing, and reporting results

RESULTS

Descriptive summary of the included studies

Target groups examined in SBAs of HPE

Challenges of SBAs methods in HPE

Advantages of SBAs methods in HPE

Disadvantages of SBAs methods in HPE

Necessary conditions for using SBAs in HPE

Discussion

Conclusions

List Of Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1