The present study aimed to investigate the feasibility of using VR-based scenarios to evaluate the EMC skills of junior doctors A representative sample of junior doctors was recruited, with their age and gender distribution mirroring that of the broader population of medical novices.
In the self-assessment for EMC, participants generally rated their abilities in history taking, physical examination, and diagnostic procedures (such as laboratory tests and ECG) as above average. However, deficits were primarily noted in therapeutic areas and overarching skills (e.g. prioritization or time management), aligning with the focus of many current medical curricula and consistent with finding from previous studies [17]. Despite these plausible discrepancies in competency facets, there was no significant correlation between the self-assessment results in general and the outcomes from the EMC assessment using VR simulation or the CRA performance test using the PEF. This lack of correlation underscores that, although frequently used in clinical competency evaluations [17, 22, 23], self-assessment tends to reflect personal motivation and satisfaction with educational experiences [24], rather than being an objective indicator of clinical competence.
The assessment of EMC using VR simulation revealed an overall sufficient performance, with approximately 66% of actions performed correctly. It is important to note that the actions were not weighted by the authors (‘life-saving’ actions were equally valued in the checklist alongside ‘supplementary’ medical actions), thus requiring further interpretation: Indeed, most junior doctors were successful in stabilizing patients in terms of circulation. However, significant deficiencies were observed in specific actions related to disease management, including critical measures like initiation of non-invasive ventilation). This is an important finding, as it suggests, that such deficiencies may not be adequately captured by traditional final examinations (written or oral). In contrast, participants’ CRA was generally well developed. However, it must be noted that the created rubric was highly context-dependent and focused on applied knowledge, since it was based on specific national guidelines. Due to the lack of adequate objective data on EMC skills of graduates [4], comparing results is challenging. In a narrative interview study from the UK, 185 representatives of various levels of experience in clinical patient care agreed that graduates possess sufficient skills for diagnosing and treating typical clinical conditions. However, significant uncertainties were described when cases became more complex or those requiring emergency actions [25]. A more recent review, which primarily relies on the external assessment of supervisors, reached a similar albeit somewhat more heterogeneous conclusion regarding EMC among junior doctors [26]. Both review articles underscore the need to engage more frequently with complex clinical conditions and scenarios either during medical school or at the beginning of professional practice, an area which is currently lacking. VR-based learning and assessment environments offer optimal conditions for this purpose, as their complexity can be increased almost indefinitely. This is particularly beneficial at the transition from education to further training, without the need for additional material and personnel.
Lastly, the present study demonstrated a strong correlation of assessment of EMC using VR simulation with CRA performance test using the PEF. This suggests that VR-based simulations, in addition to allowing for the assessment of knowledge and skills, also possess a certain predictive validity for overarching skills like CRA. As a limitation, the PEF consists of items (open ended questions) that primarily focus on the diagnostic process. Further studies could explore the correlation of VR-based assessments and measurement tools that also cover the therapeutic process, such as the script concordance test [27]. Even if the results shown here are plausible and the VR simulation provides a direct operationalization of the construct EMC, no conclusions about test quality criteria of VR-based assessments can be drawn from this pilot study. However, there is already some evidence from other work demonstrating validity and discriminative power of such approaches: Assessment of emergency medical skills based on VR 360° videos has yielded valid results across different levels of prior experience [14]. Additionally, the application of VR for assessing the effectiveness of resuscitation measures was considered realistic and valid by a larger group of experienced OSCE examiners [13]. We recently demonstrated that the difficulty of a VR-based OSCE station was comparable to an analog station and the discriminative power regarding an entire curricular OSCE was even slightly superior [12]. Even though further evidence on the validity and reliability of VR-based assessments would be beneficial, these platforms show promise for evaluating preparedness for real-world situations. This can be particularly valuable in entry tests for junior doctors, ensuring that they possess the necessary skills and knowledge to effectively navigate complex clinical environments. Thus VR-based assessments can help ensure that new medical professionals possess the requisite skills and competencies to deliver high-quality patient care.
Strengths
This study tested, for the first time, the utilization of VR-based complex emergency scenarios for competency assessment of junior doctors. An objective picture of EMC across three scenarios was obtained from a representative sample of graduates at the study site. Furthermore, the scenarios used have been employed in teaching since 2020 and have been continuously refined since then. The analyses conducted were extensive and involved personal factors, subjective assessments, and a validated tool for measuring clinical reasoning.
Limitations
A relatively small number of participants was recruited at only one site, making the results obtained not readily generalizable. However, it must be emphasized that recruiting junior doctors from their clinical duties for a relatively elaborate experimental setup poses a general challenge. Despite this, the present sample size was sufficient to allow for plausible and statistically significant conclusions. The items for assessing performance in the VR scenarios used in this study were created based on guidelines and clinical experience by subject matter experts. However, they have not yet been analyzed for characteristics such as difficulty or discriminative power within a larger collective.