Team training with Mixed Reality and Virtual Reality – benefits and limitations in educational practice illustrated by the example of two paramedic classes

doi:10.21203/rs.3.rs-2374530/v1

Download PDF

Research Article

Team training with Mixed Reality and Virtual Reality – benefits and limitations in educational practice illustrated by the example of two paramedic classes

https://doi.org/10.21203/rs.3.rs-2374530/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Case simulations as an imitation of clinical experience are an accepted learning method in health professions. This is also the case in Emergency Medical Services. In the field of case simulations, a strong trend toward the use of more technology has been observed for years, such as the use of Extended Reality. This paper analyzes specific media perception factors and the motivation of paramedic trainees in Extended Reality. It also analyzes the differences in media perception and motivation resulting from the inclusion of a haptic patient manikin in the Extended Reality and its meaning for educational practice.

Methods

The participants were assigned to either the experimental group (Mixed Reality with manikin) or the comparison group (Virtual Reality without manikin) on a class-by-class basis. After an Extended Reality case simulation, a survey was performed using a questionnaire to elicit side effects, sense of presence, motivation, and sociodemographic variables. After evaluating the sociodemographic data, parametric tests were used for mean comparison and correlation analysis.

Results

20 participants were evaluated in the experimental group and 16 in the comparison group. In both groups, high motivation, an acceptable sense of presence, and advanced usability were observed. Sickness symptoms were present. In both groups, high correlation between the two motivation dimensions Identified Regulation and Intrinsic Regulation with Spatial Presence and Usability was found. Spatial Presence and Usability were also significantly correlated in both groups. A significant mean score difference was found for Amotivation between the experimental group (mean = 1.56/6, SD = 1.25) and the comparison group (mean = 2.36/6, SD = 1.14).

Discussion

VR and MR are a promising learning method if implemented in a planned manner by a competent simulation crew. This can be concluded from the participants’ high level of self-regulated motivation as well as from the identified media perception factors. We recommend that the simulation crew use setup checklists, but also a competence-oriented approach, considering fiction contract and structured debriefing. Use of wireless head-mounted-displays and data-gloves still need to be done.

Overview Extended Reality

Case simulations as an imitation of clinical experience are accepted educational resources for enhancing patient safety in prehospital patient care and for training non-technical skills such as teamwork, communication, and situation awareness [1, 2]. The necessary practical transfer can be ensured by medical case simulation. Such case simulation can take place in conventional settings, for example in a classroom or a simulation laboratory, but can also be done by using highly immersive technologies such as virtual reality.

Virtual reality (VR) has been researched for many decades, has applications in various fields, and has also gained popularity in medical training. VR plays an important role in the professional education of healthcare professionals and is comparable to traditional media in learning gains [3]. First, the terms Extended Reality (XR), Virtual Reality (VR), Augmented Reality (AR), Augmented Virtuality (AV), and Mixed Reality (MR) need to be distinguished. XR is an umbrella term for technologies that create computer-generated environments or objects. Referring to Milgram's mixed reality continuum, VR describes a completely computer-generated world as diametrically opposed to the real environment [4]. AR means that the user still perceives the physical reality, but this is additionally enriched or extended with digital information. MR combines real and virtual objects. AV consists predominantly of virtual spaces that incorporate real physical objects and people who can interact with the virtual world in real time. Figure 1 shows the Milgram continuum in relation to medical case simulation.

This paper focuses on the results of the ViTAWiN project [5], which developed a multi-user Virtual Learning Environment (VLE) to enable interprofessional training of paramedics and emergency nurses in a virtualized, highly immersive scenario, independent of location. This means that, for example, paramedics, emergency nurses, and physicians can train at the same time in the same case scenario, but from different locations.

In ViTAWiN, an iterative and interdisciplinary research process with user-centered requirements analysis and an evaluation that focused on the participants at each milestone led to the design of the prototype VLE [6]. A virtual environment was first developed focusing on a prehospital scenario in a virtual patient’s garden and a handover scenario as well as a clinical scenario in a trauma center. The storyline considers a virtual patient who has suffered severe burn injuries to the thorax as a result of a barbecue accident, must be cared for by the ambulance team, and is later handed over to the emergency department. After the handover, the emergency nurses take over the clinical care in the virtual environment. During development, a patient manikin was incorporated through machine vision to provide haptic interaction at a basal level. The integration of the patient manikin marks the transition from VR to MR.

In the course of the project, aspects of facial expressions and gestures were implemented on the virtual patient to make pain expression and fear (Fig. 2), in particular, visible to the participants [7]. The realization of location-independent training was also an important milestone in the project due to the restrictions of the COVID-19 pandemic.

Both the integration of a basic haptic interaction option and the integration of facial expressions and gestures with feedback options were developed in the course of the project together with the participants on the basis of their feedback.

Following the end of the three-year project, in this paper we report the results of the final evaluation and discuss the implications for paramedic educational practice. In discussing these results, we will focus primarily on a competence-based perspective that, through its output orientation, focuses on paramedics' ability to solve problems in the safest, most error-free, and most creative way possible.

Review of relevant scholarship

As additional resource – “Procedure of the narrative review”: For the narrative literature review, the PubMed, IEEE-Xplore, and Scopus databases were screened using the keyword combination "virtual reality" or "mixed reality" and "paramedic," "emergency medical service," "paramedicine," and "nurses," respectively. Due to rapidly evolving technology, especially regarding head-mounted displays (HMD), the search was narrowed to publications between January 2018 and November 2022. Studies that address only skills training are not reported, as the focus of this paper is on non-technical aspects. In addition, we limit ourselves to the most important publications for this work. In their "Trauma bay VR study," Colonna et al. describe learning successes of their convenience sample in clinical decision-making using critical decision points, i.e., intubation, cricothyroidotomy, chest tube, and intravenous access [8]. Chabaane et al. showed that a single VR application can increase competence perception and significantly reduce anxiety in paramedics [9]. How long this effect lasts is unknown as this was a cross-sectional study. Effect sizes were small (d = 0.33 for anxiety reduction and d = 0.35 for increased competence perception). When training triage processes during mass casualty incidents, Mills et al. showed that VR simulation was nearly identical in terms of simulation efficiency [10]. They concluded that there were no measurable differences regarding mental demand, temporal demand, performance, effort, or frustration between the VR and control groups. Equal levels of satisfaction and triage results were measurable in both groups. Mills et al. were able to reduce personnel costs with VR in their experimental setup. However, the application area of mass casualty incidents should be pointed out here, which is particularly personnel-intensive depending on the type of game. Harrington et al. used the example of participants in an Advanced Trauma Life Support (ATLS) course to show that participants enjoyed using VR, rated it as their platform of choice, and considered VR to be a cost-effective tool [11]. Regarding the intensity of cognitive-emotional stress, it can be assumed that case simulations in VR can reach high intensity [12]. Other than ViTAWiN, no other MR simulations were found that address patient handover in interprofessional teams [6].

Objective

The primary aims of the study were formulated according to the Goal-Question-Metric paradigm [13] as follows: Analyze the media perception and situative motivation of paramedic trainees in a highly immersive environment (MR and VR) for the purpose of understanding the meaning for competence acquisition in paramedic education. The secondary objective was to analyze whether any differences can be observed by incorporating a haptic tactile patient manikin into the highly immersive simulation in terms of media perception and motivation in order to extrapolate the relevance for highly immersive training in paramedic education.

RQ 1

To what extent do situational motivation, VR sickness, presence, and ease of use occur in paramedic trainees after an XR training sequence as measured by a questionnaire survey?

RQ 2

How does a group of paramedic trainees in a training sequence with a fully computer-generated VR differ in terms of usability, VR sickness, and presence from a group of paramedic trainees whose training includes the haptic integration of a palpable manikin in the MR setting?

RQ 3

What consequences do these findings imply for the implementation of VR and MR scenarios in competence-based educational practice?

Research design and organizational background

The cross-sectional quasi-experimental study was based on the examination of a VR and MR training sequence in paramedic trainees, with a subsequent survey on media effect factors, motivation, and experience of the highly immersive environment via online questionnaire.

Data collection

The data was collected from the two educational Emergency Medical Service (EMS) project partners in the ViTAWiN project during a VR or MR simulation. The two educational partners provide vocational training for EMS in the German states of Lower Saxony, Bremen, and Hesse. The study was conducted at two separate sites in Lower Saxony (11.10.22) and Hesse (22.02.22). There are eight months between the two study dates because the trainees undergo dual training and are only at the "school" learning site (for a total of 1960 hrs.) in blocks, alternating between hospital (720 hrs.) and rescue station (1960 hrs.). In addition, problems with the robustness of the tracking of the manikin had to be resolved between the two dates. This was mainly due to the extremely low tolerance between the actual position of the manikin in the room and the virtualized position. A small tracking offset (< 1 cm) already leads to significant distractions for the participants.

As additional resource – “Roles and responsibilities at the study site”: The complete situation on site was controlled and documented by a study supervisor, enabling simulation technicians and educational professionals to concentrate on their roles, and documentation and quality assurance were ensured throughout. The role of the supervisor was also to instruct the participants and subsequently obtain consent forms. The simulation technicians were primarily tasked with expertly performing the setup of the XR technology and ensuring constant operational readiness of the system. The pedagogical professionals had the task of instructing the participants in the EMS scenario ("mission report") and conducting a short debriefing. They were also the point of contact for any questions of the participants during the simulation.

Technical setup & hygiene measures

The simulation can be displayed as a two-dimensional desktop variant, but it is primarily intended as a so-called highly immersive learning environment using head-mounted displays (HMD), operating elements (controllers), and position-tracking technology for capturing and processing the movements of the participants in the VLE. Although a simulation in a sitting position would be technically possible, all simulations were performed in room movement. During the entire simulation, the participants wore hairnets to improve the hygienic conditions of the HMD, which partially has fabric straps for fixation. In addition, disposable gloves and FFP2 masks (comparable to N95 and KN95) were used due to the COVID-19 pandemic.

As additional resource – “Technical details”: A free area of 3x4 meters for the play area proved sufficient but could be adapted to local conditions. In this study, we used the VR set Valve Index® from Steam, which offers a resolution of 1.440x1.600 pixels per eye and a field of view of up to 130 degrees. The position of the HMD and the controller is detected by laser sensors through so-called lighthouses. This position information is transmitted to and processed by a computer, which mirrors the updated VLE back to the HMD. In the classroom, the VLE can be additionally displayed on a screen from different perspectives (first person, third person, or stationary angle) so that other learners can observe the virtualized patient care. To project the real manikin into the VLE, predefined optical markers were attached to the manikin, which allowed the position in the play area to be calculated automatically based on pre-simulation photographs. During the simulation, the position of the markers was captured by the built-in cameras of the "VR glasses".

In the environment, it was only necessary to move around the virtualized patient with a few steps. As a locomotion technique, participants could use teleportation and movement via joystick in single-player mode. When multiple participants shared the same playing area, this was intentionally disabled to avoid potential serious injuries. Jumping, running up and down stairs, overcoming obstacles, or complex movements of the entire body were not necessary.

Experimental manipulation

In the comparison group (VR), prehospital patient care was completed without a manikin (Fig. 3), while in the experimental group (MR), a patient manikin was integrated as a haptic element (Fig. 4). In the experimental group, the patient manikin was integrated into the virtualized environment by a simulation technician (ST). The ST was able to draw on extensive knowledge in the field of computer science and was specifically familiar with this process through their own programming and co-design of the interface. In addition, the ST assisted the educational professional in controlling the simulation technology. This was also the case for the comparison group, except for the involvement of the manikin, which was not required here. This ensured that the pedagogical professionals could fully concentrate on the simulation and the participants. The experimental design meant that the participants of the experimental group could "feel" the patient on a rudimentary level. This enables examination steps such as palpating the pulse. The comparison group did not have this possibility and could only rely on the optically virtualized patient. This meant that a haptic reference was missing, such as the resistance of the patient's body during auscultation of the lungs on which the stethoscope corpus is placed.

The participants had to assess the scene in terms of safety and had to provide initial assessment and structured care to a burn patient in the MR/VR environment using medical guidelines and regional protocols (Standard Operating Procedures) [14].

Assignment method

Assignment to an experimental condition was made in a non-randomized manner on a class-by-class basis due to the extensive technical setting. Assignment at the individual level was not feasible from a research economics perspective as it would have meant hugely higher time and personnel requirements. The increased resource requirements are mainly explained by the technical setup and the room arrangement, but especially by the still very complex integration and calibration of the tracking of the patient manikin.

Inclusion and exclusion criteria

The participants were verbally interviewed about their health status on the day of the simulation. Only participants who did not report being acutely ill, injured, or in any way acutely impaired were included in the evaluation. All participants were also alert, visually inconspicuous in terms of coordination, and sure-footed. They were given verbal and written information about the study and had the option of not participating or dropping out, as well as withdrawing consent.

Participants’ characteristics

The participants were students of the German three-year paramedic training program. Unfortunately, due to staffing difficulties during the pandemic, we were not able to recruit emergency nurses from our educational partner. Prior to MR/VR exposure, experienced instructors from the educational partners gave the participants the opportunity to familiarize themselves with the hardware and software as well as the elements of the virtual environment. The questionnaire was not filled out in the simulation room, but in a separate classroom. Age and gender of the complete sample and subgroups are shown in Table 5. The participants had little or no prior experience with VR.

Base population: According to federal health reporting [15], 78000 people were employed in the German rescue service in 2020, 52000 of them full-time. 53000 employees were male, 25000 female; no information is available on non-binary persons. The age groups are shown in Table 1.

Table 1

Age groups of the population
Under 30 years	30–40 years	40–50 years	50–60 years	60 years and older
25000	20000	15000	12000	5000

Instrumentation

Immediately after the VR/MR exposure, the participants were asked to complete an online questionnaire. This comprised the collection of sociodemographic data as well as the Situational Motivation Scale – SIMS [16], the System Usability Scale – SUS [17, 18], and the IGroup Presence Questionnaire – IPQ [19]. The Simulator Sickness questionnaire – SSQ [20] was completed immediately after the evaluation. Since stricter hygiene requirements had to be observed due to the COVID-19 pandemic, approximately three to five minutes elapsed between the end of the VR/MR exposure and the start of the post-test questionnaire. The instruments used are described below:

Based on self-determination theory [21], the SIMS measures constructs of Intrinsic Motivation, Identified Regulation, External Regulation, and Amotivation. The higher the score, the higher the level. For Intrinsic Motivation and Internal Regulation, high scores are to be evaluated positively, for External Regulation only to a limited extent, and pronounced Amotivation is to be evaluated negatively. Table 2presents a brief description of the SIMS constructs.

Table 2

Description of the SIMS constructs
SIMS Construct	Amotivation	External Regulation	Identified Regulation	Intrinsic Motivation
Motivation	Absence of motivation	Externally determined	Rather autonomous	Intrinsic
Motivation control	Not applicable	External control variables, e.g., constraint by curriculum, high reward promises	A high personal value for action outcomes and action consequences is expected	Completely voluntary; goals are integrated into the sense of self

The instrument was tested with moderate to excellent reliability (Cronbach’s alpha: intrinsic motivation = .95, identified regulation = .85, external regulation = .62, amotivation = .83) and factor validity. “Results of the confirmatory factor analysis showed that the χ2 was significant, χ2(98, n = 907) = 856.50, p < .05, and the NNFI (.89) was somewhat lower than the .90 cut-off value. However, the CFI (.90) was satisfactory. […] All hypothesized factor loadings, covariances, error residuals, and factor residuals were found to be significant (z > 1.96).” [16]

The IPQ addresses questions representing Involvement, Spatial Presence, Realism, and a General Factor with responses using seven-point rating scales (0–6), with high scores being positive for each construct. The IPQ was tested with good reliability (Cronbach’s alpha > .7 for all subscales) and factor validity. The subscale items load on their respective subscales fairly highly, mostly above 0.6; for details, refer to Schubert et al. [19].

There are objective and subjective methods for assessing the severity of simulator sickness, the most popular being the self-report using Kennedy’s SSQ [22]. The SSQ includes three constructs – Disorientation, Nausea, and Oculomotor –, which are answered using a four-point rating scale. The participants were asked about any symptoms and, if applicable, their severity. Each construct, as well as the total score, has its own weighting, which is obtained by multiplying the series means. The use of the SSQ is critically debated in the application domain of VR due to its development, which was intended for flight simulators rather than the latest generation of HMD-VR [23, 24]. Nevertheless, because SSQ has been used in numerous studies to report VR-related adverse events [22, 25], it was used here to allow comparability. Weighted scores (see Table 3) < 5 are considered "negligible", 5–10 are considered "minimal", 10–15 are considered "significant", and 15–20 are considered "of concern." A total score above 20 is considered "bad" [24]. Recent work also shows psychophysiological changes associated with specific EEG abnormalities that correlate with the SSQ [26]. To date, a variety of theories and assumptions exist that try to explain simulator sickness in VR (11). The symptoms can last from minutes to hours and increase with the duration of VR exposure, although it is not currently possible to conclude a general pattern [22]. Studies in VR settings have shown that the probability of symptoms is heterogeneous. For example, Chen et al. [27] showed a frequency of 30%, while Kim et al. [26] showed a frequency above 80%.

The System Usability Scale (SUS) is a simple and technology-independent questionnaire for evaluating the usability of a system. It comprises ten questions, which are answered on a rating scale and evaluated as a percentage on the grade scale. Rule of thumb: “[…] products that scored in the 90s were exceptional, products that scored in the 80s were good, and products that scored in the 70s were acceptable. Anything below a 70 had usability issues that were cause for concern.” [28] The SUS was tested with excellent reliability (Cronbach’s alpha = .91) and acceptable factorial validity (for eigen values and factor loadings, see at Bangor et al., 2009 [28]).

Table 3: Constructs, scale, and sample items

Measures and covariates, power and precision

All statistical operations were performed using RStudio (version 2022.02.1 + 461). First, the data was prepared, then a pattern analysis of missing values and a descriptive-statistical evaluation were performed.

The subgroups were compared using Welch’s t test (α = .05, two-sided) after testing the dependent variable for extreme deviation from the normal distribution (R-package “stats”). Correlations were tested using Pearson correlation (R-package “rstatix”). The p values were adjusted for multiple testing according to Holm. The sample size of the two subgroups (n = 16 and n = 20) results in a beta error probability of 0.43 for the t test for two independent means (effect size d = 0.5) and 0.49 for the calculation of the correlation coefficient by Pearson (effect size q = 0.6) for independent samples (“g*power”, version 3.1). Considering that the size of the two subgroups is not extremely unbalanced, it can be assumed that the calculation of the Welch’s t tests and the Pearson correlation is robust for this sample [29]. The three genders were compared using the Tukey-Kramer test (α = .05) with adjustments for multiple comparisons (“stats” package). This test uses the harmonic mean of the sample numbers and can therefore be used when the sample numbers are different [30]. To test interrelatedness, Cronbach’s Alpha and McDonald’s Omega were computed with the R-package “psych” (Table 4).

Table 4: Reliability measurements

The raw data is publicly available at the FORDATIS research data repository [31].

Participant flow

A total of 42 participants were distributed class-wise between the experimental group (n = 21) and the comparison group (n = 18). None of the participants reported any pre-existing health problems. All participants in both groups completed the experiment. In the experimental group, one person opened the online questionnaire but did not submit it. In the comparison group, there were two persons with this same behavior. Figure 5 shows the flow of participants.

Baseline data

Age was distributed without any practice-relevant differences in the two groups, but age gender was not. As expected (see population description), there were significantly more males than females and non-binary persons.

Table 5

Age and sex - Complete sample and subgroups
	Age Mean (SD)	Gender Absolute Frequency
	Age Mean (SD)	male	female	non-binary
Total	22.2 (2.1)	22	12	1
MR Group (EG)	22.4 (1.7)	13	5	1
VR Goup (CG)	22.0 (2.5)	9	7	0

Descriptive statistics

Missing data: Sociodemographic questions were not answered at all by one person; age was not answered by three persons. The remaining variables were partially not answered by one person, without any specific pattern. Three persons did not submit the questionnaire at all. Table 6 shows the means and standard deviations as well as the minimum and maximum values of the total sample (RQ1) and the differences between the experimental groups for each case (RQ 2).

For convenience in understanding the SSQ results, Fig. 6 shows the results of the SSQ dimensions for the total sample (RQ1) grouped by experimental conditions (RQ2). Extreme outliers have been removed. We removed values that were 1.5 times the interquartile range above the third quartile or 1.5 times the interquartile range below the first quartile.

In reference to Bangor et al. [17, 28], the SUS is additionally interpreted as a percentage on the grade scale. Table 7 shows the grouped results. Here, the SUS score was divided into sections (F to A) and underlaid with acceptability ranges to simplify interpretation according to Bangor. In each case, the numbers represent the frequency of participants allocated to the respective section after weighting. To avoid biasing the interpretation, we report absolute values, not percentages, with reference to the sample size.

Table 7

System Usability Scale – Grade scale (according to Bangor et al., 2009)
Scale	F (0–60)	D (60–70)	C (70–80)	B (80–90)	A (90–100)
SUS total	8	11	6	10	1
MR Group	5	3	5	4	3
VR Group	3	7	2	3	1
Acceptability ranges	not acceptable or marginal low	marginal high	acceptable
Note: Absolute frequencies are shown

Inferential statistics

Mean score differences: After testing for relevant deviation from the normal distribution assumption, the t test for unpooled standard deviation was used regardless of the presence of heteroscedasticity. Construct-level (where applicable) and factor-level t tests revealed a significant mean score difference for amotivation (p = .021 [CI95=-1.47,-0.13]) between the two groups. No other significant differences were found regarding the groups. A significant mean difference (p = .03[CI95:-26.47,-1.33) between the three genders was found only for SUS between females (mean = 60.42, SD = 18.49) and males (mean = 74.32, SD = 11.42).

Correlations: With reference to RQ2, significant correlations r > 0.3 are reported below. In both the VR group and the MR group, Identified Regulation (VR group: r = 0.67, MR group: r = 0.64) and Intrinsic Motivation (VR group: r = 0.70, MR group: r = 0.83) are highly correlated with Usability. Similarly, Spatial Presence is highly correlated with Usability (VR group: r = 0.61, MR group: r = 0.71), Identified Regulation (VR group: r = 0.55, MR group: r = 0.70), and Intrinsic Motivation (VR group: r = 0.51, MR group: r = 0.61) in both groups. Involvement was highly negatively correlated with External Regulation (r = -0.67) and Amotivation (r = -0.53) only in the VR group. The MR group was further distinguished by the high negative correlation of VR Sickness with Intrinsic Motivation (r = -0.53) and Usability (r = -0.63). Further differences of the MR group were the high negative correlation of Usability with External Regulation (r = -0.69) and Amotivation (r = -0.51). Table 8 shows a summary of the correlations with p values and their 95% confidence interval.

Table 8: Significant correlations

Interpretation

Regarding RQ 1, which includes the complete sample, it can be stated that the participants had a high level of Intrinsic Motivation and Identified Regulation. Low Amotivation and low External Regulation complete the picture. Overall, this means that the participants indicated a high level of intrinsic interest. The low External Regulation may have been caused by the integration into the curriculum itself, but, like amotivation, it must not be overestimated due to the low expression. Nevertheless, it is worth questioning how the participants were prepared for the XR simulation and the evaluation by the training centers and whether competing events, such as imminent examinations, should be considered better in terms of time. Identified Regulation and Intrinsic Regulation were highly correlated with Spatial Presence, similar to Lerner et al. [32].

Regarding VR Sickness. the complete sample shows a mean value of the total score, which must be considered “significant”. Likewise, the mean values for Disorientation and Oculomotor should be discussed in order to initiate improvement measures. Reasons for the high expression of Disorientation and Oculomotor are due to the structure of the SSQ: Both constructs contain items that address eye problems; partly, the same item is used in several constructs, like "Difficulty focusing". In both Oculomotor and Disorientation, this item was the leading problem. Therefore, it must be emphasized that individual adaptation of the HMD is essential. Not only must the HMD fit securely without slipping, but the interpupillary distance and the distance from eye to lens must also be adjusted. It should be noted here that a secure fit is difficult to achieve due to the personal protective precautions, in particular the hairnet. A secure fit is also crucial for a good visual user experience and can possibly explain the high symptom expression in our case. Also, the limitations of the system for people with visual impairments need to be explored. In case of continuous use, individual solutions, such as corrective lenses for the HMD, may have to be provided. This can be a time factor that should not be underestimated in a learning arrangement with large classes.

Presence

The participants indicated a medium to high level for all facets. In detail, this means that Spatial Presence has already reached a satisfactory level, while Realism and Involvement are not a cause for concern, although there is still room for improvement.

In this respect, reference can be made to Muckler [33], who makes effective simulation dependent on the ability of teachers to enable trainees to overcome their disbelief. The following suggestion for improvement should be considered: Effective simulation depends on the ability of teachers to enable learners to overcome their disbelief. Here, the pedagogical professional must also be held responsible. Of course, positive and intrinsically dominated learner cooperation is important in this regard. Furthermore, Muckler’s goal can be shared, namely, to enable learners to accept the otherwise unrealistic aspects of a clinical simulation. Muckler further notes: "The ability to suspend disbelief enhances the participant's level of immersion in the simulation." [33] To overcome the suspension of disbelief, a fiction contract can be used. A fiction contract is a "psychological contract" between learners and teachers, in the sense of establishing a "safe container", which includes the following components: clarifying expectations, establishing a "contract" with the participants, attending to logistic details, and declaring and enacting a commitment to respecting learners and having concern for their psychological safety [34].

The high correlation of Usability and Spatial Presence in both groups highlights the great importance of the spatial dimension of presence. Due to the connection between embodiment and presence, improvements could also be achieved through stronger embodiment [35]. This includes, for example, individualizable avatars, which are very important in embodiment [36], and professional clothing with which the respective group can identify itself. In terms of embodiment and self-identification through avatars, we would like to highlight the paper by Dalgarno and Lee for further and insightful explanations [37].

In terms of usability, the scores are close to the cut-off of 70%, which suggests that improvements are necessary, but also that a considerable intermediate level has been reached. Participants' comments indicate that the wired HMD was particularly annoying. Similarly, there was criticism that very small-scale movements, such as drawing up medications, would require a high level of attention and distract from the actual learning goal due to the unnatural imitation of movement by the controllers and occasional tracking problems with very fine movements. Unfortunately, due a lack of adequate devices at the time the study was performed, it was not possible to introduce "data gloves" or wireless HMD as originally planned. However, this idea should be pursued in future projects. Furthermore, it should be tested whether a higher number of sensors would lead to less tracking shadows and thus improve haptic movements. The high positive correlation between Usability and Identified Regulation in both groups as well es the high negative correlation between Involvement and External Regulation and Amotivation in the VR Group suggests the immense importance of targeted, planned pedagogical guidance, in the sense of the principles of Hattie [38].

The results regarding usability and presence are similar to the findings of Lerner et al. [32] in the predecessor VR project EPICSAVE and to the results of Schild et al. [6] from previous evaluations.

With respect to RQ 2, a significant mean difference between the experimental group and the comparison Group was found only for Amotivation. It is surprising that Amotivation was lower in the experimental group, but no significant differences could be found for Intrinsic Motivation and Identified Regulation and External Regulation. A possible explanation is that the extraneous cognitive load required by operation and orientation outweighed the perception of the manikin during training. In future studies, it would be important to perform cognitive load measurements in addition to the current study design [39]. These should also be longitudinal, over several training sequences on different days, in order to be able to include learning effects.

RQ 3: In summary, our findings regarding Realism, Involvement, and Usability imply that there are two main prerequisites that enable the acquisition of competencies as a basal criterion: 1. The technical setting must be mastered in a suitable classroom without ifs and buts in order to avoid presence dropouts and distractors. 2. XR can only develop its full effect in planned use by pedagogical professionals. Pre-briefing, a fiction contract, and structured debriefing can be preliminarily assumed as vital, especially with regard to the perception of realism and involvement. This hypothesis needs to be tested in future studies oriented toward learning outcome. Independent variables to be considered also include the impact of familiarization and habituation on learning outcome.

Generalizability and limitations

The results show that participants in paramedic schools are open to technology and highly motivated. These are two important foundations for the use of new educational technologies.

In this study, internal validity may be biased by the so-called novelty effect, which “refers to the human tendency for heightened engagement and/or performance when encountering the introduction of a […] a new technology” [40]. With reference to the concept of the “hedonistic treadmill”, it should be noted that (without competent pedagogical supervision) the trainee's engagement gradually wanes as the now familiar game elements and mechanisms no longer entertain, challenge, or satisfy the user [41]. Furthermore, it can be assumed that orientation in and operation of the virtual learning environment represents a moderate to high extrinsic cognitive demand [39]. This can lead to the failure to perceive central components of the simulation, such as assessment of specific aspects of the simulation, but also achievement of learning outcomes. Both of these limitations can be addressed with measurement repetition of the MR and VR application over multiple instructional sessions on different days. Otherwise, we suggest discussing the use of SSQ for immersive HMD simulations very critically, as already presented in the Instrumentation chapter. Regarding the low level of interrelatedness from the presence dimension Involvement, this has to be tested with a larger sample in the field of HMD simulation.

With regard to external validity, we can highlight an important strength of this study. This is mainly due to its practical relevance, as it was evaluated in real learning situations at the educational partners. Thus, it can be assumed that the results can be transferred very well to social reality. It should be noted, however, that the sample cannot be compared directly to an international sample. In this respect, differences in training prerequisites (e.g., curricular design), professional socialization, and understanding of roles must be considered. However, this is mainly a question of pedagogical design and less of basic technology-related impact factors. However, the findings of Schild et al. [6] and Lerner et al. [32] were reproduced in our study in the broadest sense.

With respect to the sample size, we again refer to the robustness of the methods used [29].

Implications

The results of the survey and the evaluation itself have shown that the technical setting is demanding. Aspects such as customization of the HMD must be mastered to enable appropriate usability and a positive learning experience. To realize both, a "simulation crew" is needed that can synergistically combine technical aspects and pedagogical aspects. Working with a checklist is recommended.

The results show that there is high motivation to learn in XR scenarios on the part of the paramedic trainees, which should be exploited as a potential. Where this potential should be used in comparison to conventional settings in terms of the relationship between costs, time required for the setup, and learning success needs to be weighed. Its use is conceivable above all in learning situations that are difficult to produce by conventional means, such as certain injuries or mass casualty incidents.

This leads us to the next consideration: Educational variables must be clarified and planned in advance. This is the same for conventional simulation settings as it is for simulations in XR. Even before the simulation, the learning objectives must be defined, just as in conventional case simulations. Here, we recommend attaching high importance to competence orientation. Competencies are to be understood in this work as value and norm inherent self-disposition abilities of individuals, which are beyond imparting of knowledge, based on acquisition and adoption, allowing them to perform safe, creative actions in complex, open-ended situations. These considerations lead to the need to consider ex ante the competence-oriented learning goal in terms of cognitive, affective, and psychomotor learning goals, also with regard to their achievability within the available time. In order to consider an adequate level of stress and strain, the extrinsic cognitive load caused by the learning environment must also be included in these considerations. To strengthen the trainees’ involvement, the conclusion of a fiction contract is recommended. To enable reflection as an integral part of a case simulation, a structured debriefing should take place. This can be considered analogous to conventional case simulations.

In this study, we were unable to show that an MR scenario is superior to a VR scenario. For future research, this means that further studies are recommended that are designed as repeated measures in a within-subjects design to investigate the role of confounding variables such as familiarization effects, learning effects, and cognitive load.

With regard to the motivation of the participants, XR seems to be a learning environment that allows presence experience and realism and enables psychological involvement of the participants. Participants in paramedic training were found to have high intrinsic and self-regulated motivation. For a planful and goal-oriented approach, we recommend creating a simulation checklist that addresses specific points of the XR simulation. Apparent trivialities, such as adequate fit of the HMD, can be critical factors for success. A competence-oriented approach is also recommended for XR simulations in the form of a fiction contract and a structured debriefing, just like in a conventional case simulation. Regarding technical aspects, the availability and use of wireless HMDs and "data gloves" is desirable. In the overall appraisal, we conclude that XR is not just a gimmick, but a promising learning method when used by a competent simulation crew. XR case simulations are difficult to run by teachers acting as "lone wolves". Further studies addressing competence-based learning outcomes and the role of habituation effects, learning effect, and cognitive load are needed.

Ethics approval and consent to participate:

Informed consent for voluntary participation in the study was obtained from all subjects in advance. Information for the subjects was provided verbally and in written form according to the “Ethical Principles of Psychologists and Code of Conduct” of the American Psychological Association (APA), especially Section 8. Written consent has been obtained.

We confirm that the experimental protocols have been approved by the Commission for Transparency and Ethics in Research (Chairman Prof. Dr. Dieter Weber) of the Hannover University of Applied Sciences and Arts (Hannover Germany). A registry number is not available.

All methods were carried out in accordance with "Ethical Principles of Psychologists and Code of Conduct" of the APA and current German law, especially the local COVID-19 protection regulations.

Consent for publication: Informed consent for publication was obtained from all subjects verbally and in written form. The consent also included the publication of the raw data and the production and use of images for scientific purposes, in particular for publications in scientific journals.

Availability of data: The dataset generated and analyzed during the current study is available in the FORDATIS repository, https://fordatis.fraunhofer.de/handle/fordatis/296. doi: http://dx.doi.org/10.24406/fordatis/229. The dataset (CSV file) is platform-independent publicly accessible via web browser. No license or registration is required.

Competing interests: PD declares no conflicts of interest. SD and CE were research fellows in the ViTAWiN project, which is the starting point of the paper. As team members, both were responsible for didactic evaluation, medical didactic consulting with the consortium, and curricular integration. There are no other conflicts of interest.

Funding: The ViTAWiN project, which is the paper's starting point, was funded by the German Federal Ministry of Education and Research (grant number 01PV1800). Open Access fees are covered by the Fraunhofer-Gesellschaft's Open Access funding.

Author´s contributions: CE as a paramedic, economic and organizational psychologist, neuroscientist, and adult educator prepared and analyzed the data and was mainly involved in writing the manuscript. PD, as a paramedic and expert on didactic aspects in emergency medical services, was particularly involved in the literature review and derivation of implications together with CE. DS, as a paramedic and medical informatics expert, was mainly responsible for reporting the technical setup, technical details, and compiling the figures.

Acknowledgments: We thank Prof. Dr. Vera Hagemann (University of Bremen, Germany) for the unpaid scientific review within the institute's internal review process. We also thank Priv.-Doz. Dr. Thomas Luiz (IESE) and Sonnhild Namingha (IESE) for their medical, resp. linguistic layout advice.

Boyle M, Williams B, Burgess S. Contemporary simulation education for undergraduate paramedic students. Emerg Med J 2007; 24(12):854–7. doi: 10.1136/emj.2007.046318.
Bracq M-S, Michinov E, Jannin P. Virtual Reality Simulation in Nontechnical Skills Training for Healthcare Professionals: A Systematic Review. Simul Healthc 2019; 14(3):188–94. doi: 10.1097/SIH.0000000000000347.
Ryan GV, Callaghan S, Rafferty A, Higgins MF, Mangina E, McAuliffe F. Learning Outcomes of Immersive Technologies in Health Care Student Education: Systematic Review of the Literature. J Med Internet Res 2022; 24(2):e30082. doi: 10.2196/30082.
Paul Milgram, Fumio Kishino. A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems 1994; 77:1321–9.
Hochschule Hannover. Project Homepage ViTAWiN; 2019 [accessed: 08.11.2022]. https://vitawin.info/.
Schild J, Elsenbast C, Carbonell G. ViTAWiN - Developing Multiprofessional Medical Emergency Training with Mixed Reality. In: 2021 IEEE 9^th International Conference on Serious Games and Applications for Health(SeGAH): IEEE; 2021. S. 1–9 ([).
Carbonell G, Schild J. Conceptual Design of Emotional and Pain Expressions of a Virtual Patient in a Virtual Reality Training for Paramedics. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW): IEEE; 2022. S. 222–6 ([).
Colonna AL, Robbins R, Stefanucci J, Creem-Regeh S, Patterson B, Engel BT et al. Trauma bay virtual reality-A game changer for ATLS instruction and assessment. J Trauma Acute Care Surg 2022; 93(3):353–9. doi: 10.1097/TA.0000000000003569.
Chaabane S, Etienne A-M, Schyns M, Wagener A. The Impact of Virtual Reality Exposure on Stress Level and Sense of Competence in Ambulance Workers. J Trauma Stress 2022; 35(1):120–7. doi: 10.1002/jts.22690.
Mills B, Dykstra P, Hansen S, Miles A, Rankin T, Hopper L et al. Virtual Reality Triage Training Can Provide Comparable Simulation Efficacy for Paramedicine Students Compared to Live Simulation-Based Scenarios. Prehosp Emerg Care 2020; 24(4):525–36. doi: 10.1080/10903127.2019.1676345.
Harrington CM, Kavanagh DO, Quinlan JF, Ryan D, Dicker P, O’Keeffe D et al. Development and evaluation of a trauma decision-making simulator in Oculus virtual reality. Am J Surg 2018; 215(1):42–7. doi: 10.1016/j.amjsurg.2017.02.011.
Paletta L, Schneeberger M, Reim L, Kallus W, Peer A, Schonauer C et al. Work-in-Progress—Digital Human Factors Measurements in First Responder Virtual Reality-Based Skill Training. In: 2022 8^th International Conference of the Immersive Learning Research Network (iLRN): IEEE; 2022. S. 1–3 ([).
Basili VR, Caldiera G, Rombach HD. The goal question metric approach. Encyclopedia of software engineering 1994:528–32.
Thim T, Krarup NHV, Grove EL, Rohde CV, Løfgren B. Initial assessment and treatment with the Airway, Breathing, Circulation, Disability, Exposure (ABCDE) approach. Int J Gen Med 2012; 5:117–21. doi: 10.2147/IJGM.S28478.
Statistisches Bundesamt. Gesundheitspersonal in 1.000. Gesundheitsberichterstattung des Bundes: Gliederungsmerkmale: Jahre, Deutschland, Alter, Geschlecht, Beschäftigungsart, Einrichtung; 2022 [accessed: 06.11.2022]. https://www.gbe-bund.de/gbe/!pkg_olap_tables.prc_set_orientation?p_uid=gast&p_aid=15871178&p_sprache=D&p_help=2&p_indnr=93&p_ansnr=84569984&p_version=11&D.000=3&D.002=1&D.003=3&D.767=3&D.734=2.
Guay F, Vallerand RJ, Blanchard C. On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS). Motivation and Emotion 2000; 24(3):175–213. doi: 10.1023/A:1005614228250.
Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 2008; 24(6):574–94. doi: 10.1080/10447310802205776.
Brooke J. SUS: A quick and dirty usability scale. Usability Eval. Ind. 1995; 189.
Schubert T, Friedmann F, Regenbrecht H. The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators & Virtual Environments 2001; 10(3):266–81. doi: 10.1162/105474601300343603.
Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The International Journal of Aviation Psychology 1993; 3(3):203–20. doi: 10.1207/s15327108ijap0303_3.
Deci EL, Ryan RM. The Empirical Exploration of Intrinsic Motivational Processes. In: Advances in Experimental Social Psychology Volume 13: Elsevier; 1980. S. 39–80 (Advances in Experimental Social Psychology[).
Dużmańska N, Strojny P, Strojny A. Can Simulator Sickness Be Avoided? A Review on Temporal Aspects of Simulator Sickness. Front Psychol 2018; 9:2132. doi: 10.3389/fpsyg.2018.02132.
Stone Iii WB. Psychometric evaluation of the Simulator Sickness Questionnaire as a measure of cybersickness; 2017.
Stanney KM, Kennedy RS, Drexler JM. Cybersickness is Not Simulator Sickness. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 1997; 41(2):1138–42. doi: 10.1177/107118139704100292.
Kim HK, Park J, Choi Y, Choe M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Appl Ergon 2018; 69:66–73. doi: 10.1016/j.apergo.2017.12.016.
Kim J-Y, Son J-B, Leem H-S, Lee S-H. Psychophysiological Alteration After Virtual Reality Experiences Using Smartphone-Assisted Head Mount Displays: An EEG-Based Source Localization Study. Applied Sciences 2019; 9(12):2501. doi: 10.3390/app9122501.
Chen W, Chen JZ, So R. Visually induced motion sickness: Effects of translational visual motion along different axes. Contemporary Ergonomics and Human Factors 2011 2011:281–7. doi: 10.1201/b11337-47.
Bangor A, Kortum P, Miller J. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Stud. 2009; 4:114–23.
Norman G. Likert scales, levels of measurement and the “laws” of statistics. Adv Health Sci Educ Theory Pract 2010; 15(5):625–32. doi: 10.1007/s10459-010-9222-y.
Tukey JW. Comparing Individual Means in the Analysis of Variance. Biometrics 1949; 5(2):99. doi: 10.2307/3001913.
Elsenbast C. Psychometric and sociodemographic raw data of training evaluation of paramedic trainees with extended reality in classroom setting within project ViTAWiN [dataset]: Fraunhofer-Institute for Experimental Software Engineering IESE; 2022 [accessed: 14.11.2022]. https://fordatis.fraunhofer.de/handle/fordatis/296. doi: http://dx.doi.org/10.24406/fordatis/229.
Lerner D, Mohr S, Schild J, Göring M, Luiz T. An Immersive Multi-User Virtual Reality for Emergency Simulation Training: Usability Study. JMIR Serious Games 2020; 8(3):e18822. doi: 10.2196/18822.
Muckler VC. Exploring Suspension of Disbelief During Simulation-Based Learning. Clinical Simulation in Nursing 2017; 13(1):3–9. doi: 10.1016/j.ecns.2016.09.004.
Rudolph JW, Raemer DB, Simon R. Establishing a safe container for learning in simulation: the role of the presimulation briefing. Simul Healthc 2014; 9(6):339–49. doi: 10.1097/SIH.0000000000000047.
RIVA G. From Technology to Communication: Psycho-social Issues in Developing Virtual Environments. Journal of Visual Languages & Computing 1999; 10(1):87–97. doi: 10.1006/jvlc.1998.0110.
Fowler C. Virtual reality and learning: Where is the pedagogy? Br J Educ Technol 2015; 46(2):412–22. doi: 10.1111/bjet.12135.
Dalgarno B, Lee MJW. What are the learning affordances of 3-D virtual environments? Br J Educ Technol 2010; 41(1):10–32. doi: 10.1111/j.1467-8535.2009.01038.x.
Hattie J, Yates GCR. Visible Learning and the Science of How We Learn: Routledge; 2013.
Sweller J. Element Interactivity and Intrinsic, Extraneous, and Germane Cognitive Load. Educ Psychol Rev 2010; 22(2):123–38. doi: 10.1007/s10648-010-9128-5.
Tsay CH-H, Kofinas AK, Trivedi SK, Yang Y. Overcoming the novelty effect in online gamified learning systems: An empirical evaluation of student engagement and performance. J Comput Assist Learn 2020; 36(2):128–46. doi: 10.1111/jcal.12385.
Brickman P, Campbell DT. Hedonic relativism and planning the good society. Economics 1971:287–301.

Competing interest reported. PD declares no conflicts of interest. SD and CE were research fellows in the ViTAWiN project, which is the starting point of the paper. As team members, both were responsible for didactic evaluation, medical didactic consulting with the consortium, and curricular integration. There are no other conflicts of interest.

Checklist.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Team training with Mixed Reality and Virtual Reality – benefits and limitations in educational practice illustrated by the example of two paramedic classes

Status:

Version 1

Abstract

Background

Methods

Results

Discussion

Figures

Background

Methods

Results

Discussion

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1