An Observer Tool for to Enhance Learning of Anesthesia Resident’s Non-Technical Skills During High-Fidelity Simulation: A Randomized Controlled Trial

Background: An observer tool (OT) dedicated to technical skills could improve learning of medical knowledge during simulation. However, it remains uncertain whether non-technical skills learning outcomes might be improved by using an OT. Methods: After consent, anesthesia residents enrolled to a crisis management training simulation were randomized to use an observer tool (OT +) (based on non-technical skills) or not (OT-) when not role-playing. The main outcome parameter was non-technical skills learning outcomes assessed by the global score of non-technical skills learning after the training (self-assessment using the 15 items of 4 categories of the Anaesthetists’ Non-Technical Skills (ANTS); 0 to 10 Likert scale; /40). The perceived stress level, satisfaction and the score of medical knowledge were also assessed. Results: All anesthesia residents were randomized (n = 48; OT+ group, n = 37; OT- group). At the end of the session, the global score of ANTS learning and the medical knowledge score were similar in the two groups: 31 ± 4 (OT+) and 31 ± 5 / 40 (OT-) (p = 0.55) and 12 ± 2 (OT+) and 12 ± 2 / 20 (OT-) (p=0.47). The 2 groups had a positive and similar perception of learning stress management, improvement of crisis skills management, satisfaction and changes in professional practice after the training session. Conclusions: This study has shown a positive perception of ANTS learning after crisis training without difference between using or not an OT in anesthesia residents. More studies are necessary to dene the place of this educational tool.


Background
In crisis situations, anesthesiologists must quickly and accurately mobilize technical (medical and protocol knowledge, procedural skills) and non-technical (team management, leadership...) skills. The low incidence of clinical crises does not allow traditional learning at the patient's bedside nor maintenance of competence. Simulation based education is recommended for enhance technical and non-technical skills (NTS) [1][2][3][4][5]. In France, the increase in the number of anesthesia residents and the limited number of instructors leads to organizational problems for simulation centers. For example, during high-delity simulation training scenarios, some participants are active participants and some are only observers of their colleagues. It is necessary to assess whether observers obtain the same learning outcomes than those who are role-playing in scenarios. Some studies have shown that some learnings outcomes are similar between active participants and observers [6][7][8][9][10][11] while others show a greater learning bene t when the learner is an active participant [11][12][13][14]. The bene ts of observation in simulation-based medical education is increasingly recognized [15,16]. The social learning theory proposed by Bandura [17] and adapted to simulation states that vicarious learning occurs because from the observation of others, one can get an idea of how behaviors are produced and then reproduce them [18]. This bene cial effect has recently been con rmed in a meta-analysis of 13 randomized studies [16]. To reinforce the positive effects that seem nevertheless achieved in the role of observer, it has been proposed to involve observers in the scenarios through the use of an observer tool (OT) that observers must ll out as the scenario unfolds [15]. The observer tool is usually a list of key points, in a paper format to highlight the pedagogical objectives which can be technical, non-technical or mixed. In a recent literature review, O' Regan et al. encourage the use of an OT [15] suggesting that it is associated with improved learning and satisfaction by making the observer more active. However, literature lacks randomized studies comparing learning outcomes of passive observers to a group of active observers who are using an OT. In a previous study [19], an OT dedicated to medical knowledge improved learning in anesthesia residents during simulation. However, it remains uncertain whether NTS learning outcomes might also be improved by using an OT.
The aim of the study was to evaluate the impact of an NTS-based OT on NTS learning of anesthesia residents using a randomized design.

Study description
This prospective and randomized controlled study was conducted at the LabForSIMS simulation center of Paris-Saclay Medical school. The aim was to evaluate the impact of an NTS-based OT on NTS learning of anesthesia residents. Approval had been obtained from the Ethical Committee for Research of the French Society of Anesthesia and Intensive Care Medicine (SFAR, IRB 00010254-2018-102). The trial has been retrospectively registered on researchregistry.com (August 13, 2021; registration number: researchregistry7055). All methods were performed in accordance with the relevant guideline and regulation. The study was carried out with the use of the CONSORT tool adapted for simulation studies [20] and the GREET Tool for educational studies [21].
Third and fourth year anesthesia residents of the Paris area were enrolled after having given their written consent (GC.A.). This high-delity simulation session is part of their mandatory training and apart from their refusal to participate, no exclusion criteria were used.
Each resident participated in a full day of simulation-based training composed of 6 scenarios (cardiac arrest, malignant hyperthermia, local anesthetic systemic toxicity and grade III anaphylactic shock, residual muscle relaxant, di cult intubation). Scenarios and the educational objectives were the same every day during the training week. Thus, only the role played varied during the day. For each scenario, two residents played the role of the senior anesthesiologist and of the anesthesia resident respectively. A third resident played the role of another senior anesthesiologist who could be called as a backup person. This third active participant was always involved at some time in the scenario, either being called by the participants themselves or the call was suggested by an instructor playing the role of a facilitator and this was always agreed by the participants. Participants were neither aware in advance of the role to which they would be assigned nor of the scenario in which they would be involved. Observers were seated in an adjacent room in which the scenario was broadcasted by live video transmission.
Each resident was active participant at least once and observer of the 5 other scenarios. After each scenario, all residents participated to a debrie ng with instructors. The debrie ng was carried out using the RAS method (reaction, analysis and synthesis) [22]. The actors were initially invited to give their feelings and emotions. Then the analysis phase allowed for a re ective analysis of the situation by encouraging active participants and observers to highlight the positive and negative points concerning technical and non-technical skills (de ned by the educational objectives of each scenario). In addition, the residents recalled, if necessary, the concepts of technical and non-technical skills if the latter had not been used correctly during the scenario and decontextualized the situation. Finally, the synthesis made it possible to highlight the key messages of the educational objectives of each scenario.
To facilitate the organization of the day (withdrawal, change), the randomization took place just before the simulation days based on invitations scheduled over 5 days (A.B., using the random function of the Excel© software). The randomization number corresponded to the order of presence.
Anesthesia residents ( Fig. 1) were randomized into two groups according to the strategy used when residents observed the simulation scenarios (not role-playing): OT + group: an observer tool (based on non-technical skills) was used OT-group: no observer tool was used Observer tool In the OT + group, an observer tool in paper format was distributed immediately before each scenario. The non-technical skills-based observer tool was constructed using the items of the Anaesthetists' Non-Technical Skills (ANTS) from the work of Fletcher et al [23]. Observers in the OT + group had to use this observer tool over the 5 scenarios by rating each item of the non-technical skills grid using a Likert scale (0 to 10). If the ANTS item was not relevant to the situation, then the resident wrote "not applicable"

Assessment method
The main outcome parameter was the non-technical skills learning outcomes assessed by the global score of self-assessed non-technical skills learning at the end of the training day (Kirkpatrick level 2). Assessment of learning was based on questions which used the ANTS scoring system [23] and asked the perception of the resident regarding his (her) understanding and knowledge of NTS. The ANTS scoring system uses 15 questions divided in four categories and assesses task management, teamworking, situation awareness and decision-making and uses a Likert scale (0 to 10) (global score /40). The four subcategories of the ANTS score were also compared separately as a secondary outcome. Additional questions assessing satisfaction on the formative value of the day (Kirkpatrick level 1), perception of stress, self-assessment of learning (Kirkpatrick level 2) and expected changes in future practice (Kirkpatrick level 1) were used. Moreover, assessment of medical knowledge (speci cally dedicated to the training scenarios) was carried out by a 20-item multiple-choice questionnaire (MCQ) which had been previously established by the investigators.

Statistical analysis
In a previous study [19] carried out in our simulation center, the average overall ANTS score, established by self-assessment was 30 points out of 40 among residents who had been at least once active participants during the simulation day.
Assuming that the use of an observer tool would increase the overall self-perceived ANTS score by 3 points on average, with an expected mean ANTS score of 29 out of 40 in the control group; using a standard deviation of 4 points, and considering alpha risk = 5% and beta risk = 20% (80% power) with a bilateral test, 28 residents per group had to be included in each group to observe a signi cant difference https://biostatgv.sentiweb.fr/?module=etudes/sujets).
Results are presented as mean ± standard deviation or percentage. The statistical analyses were carried out with the software JMP® software (Cary, NC 27513 − 2414, USA). Statistical comparisons used twotailed Student's t-test and analysis of variance for parametric and continuous variables, a Chi-square test for proportions, and a Wilcoxon test for non-parametric variables. A value of p < 0.05 was considered signi cant.

Inclusion
In June 2018, all 85 residents consented to participate in the training day and were randomized: n = 48 in OT+ group and n = 37 in OT-group. Individual characteristics of the residents are described in Table 1. Each resident was an active participant at least once and observer of the 5 other scenarios.

Primary outcome: ANTS learning
At the end of the session, the global score of the self-perceived ANTS learning was 31 ± 4 in the OT+ group and 31 ± 5 / 40 in the OT-group (p=0.55) ( Table 2). Furthermore, no signi cant difference was found between the groups among the different ANTS categories or among the fteen ANTS items analyzed separately in a secondary analysis (Table 3).

Discussion
This study showed a positive perception in terms of NTS learning after crisis management training by high-delity simulation with no difference when using or not a NTS-based OT among residents who observed scenarios. No signi cant difference was also found in the assessment of knowledge. Likewise, a positive and similar perception was observed in terms of learning stress management, improving skills to manage a crisis, satisfaction and the professional impact of the training.
Some studies have tried to improve the learning of observers during simulation sessions using tools [8,24,25]. Using an observer tool is believed to allow more active learning [15]. The literature shows that active learning facilitates attention during training and improves students' performance [26]. The improved learning during active observation can be explained by the attention boost effect [27], a theory which suggests that when two actions are performed simultaneously, attention is then increased, and even more when the different elements to be observed are frequent [28]. Implementation of almost all NTS was indeed necessary in each of our crisis scenarios and using the OT could have made it possible to reinforce the learning of these items. In our study, although observers had a positive feeling in terms of NTS learning, we could not demonstrate any bene cial effect on learning these skills when using a NTSbased OT. To our knowledge, this study is the only one which randomly studied the speci c impact of an observer tool on the learning of NTS. In 2012, Kaplan et al.
[8] provided observers with a "critical action checklist" observation grid including a set of technical and non-technical actions to improve patient care but all observers used the observer tool and the post-test evaluation which was carried out by a questionnaire based on non-technical skills did not display any difference in the average score.
Only the study by Stegmann et al.
[24] studied the impact of an observer tool in a comparative study among observers. In this study, 200 medical students were trained with a sham patient with rectal bleeding and abdominal pain. The observers used or not a checklist targeting technical skills (performing a rectal examination) and non-technical skills (patient information, doctor-patient relationship). For each skill thirteen items were de ned and the observer had to judge whether they had been performed correctly or not. A signi cant improvement in knowledge relating to doctor-patient communication was recorded among observers equipped with the observer tool but unfortunately the study was non-randomized.
A previous work [19] randomized anesthesia residents to use an observer tool when not role-playing. The observer tools were based on crisis cognitive aids (i.e. emphasizing technical skills and medical knowledge). This study showed an increased acquisition of medical knowledge skills when using an observer tool. The acquisition of non-technical skills was also assessed using the same self-assessment than this study (secondary endpoint). As in the current study, no signi cant difference was shown in the acquisition of non-technical skills and absolute values were in the same range. Thus, compared to a technical-medical knowledge OT, the use of a non-technical OT under similar conditions had no effect on the learning of NTS. This could be explained by the fact that a non-technical OT seems more abstract than a technical OT. Indeed, this is an area which is not much taught in initial training as the importance of NTS in professional practice has been recognized only recently. In addition, the contribution of a non-technical OT as such could be less useful for learning non-technical skills because the debrie ng, in which all residents participate, frequently emphasizes non-technical skills.
A greater satisfaction score is often obtained with simulation training but the value of this outcome is debated. In our study, satisfaction ratings were very high (> 9/10), but not signi cantly different. The use of an OT therefore had no impact on satisfaction. This result agrees with our previous study in which satisfaction was similarly high in the two groups [19]. In the study by Hober [25] observing learners reported great satisfaction but in this study satisfaction was not measured objectively.
Regarding the change in professional practices, observers had a similar and very favorable perception (> 8/10) whether or not they had used the observation grid. This result is also in agreement with our previous work [19]. This lack of signi cant difference could be explained by a measurement that may be done too early. Indeed, as the immediate self-assessment was being carried out at the end of the day, projection into the future is not easy and awareness of the change in professional practice may only occur after having been again exposed to a situation requiring the use of NTS.
Regarding the assessment of knowledge (level 2 according to Kirkpatrick), our study found no signi cant difference whether or not learners had used the observation grid. Likewise, a positive and similar perception was observed in terms of learning to manage stress and skills to manage a crisis. These results are in contradiction with those of our previous study but this can easily be explained by the fact that the technical skills were not addressed in the present study [19].
The literature regarding the use of tools to increase observer learning is limited [8, 15, 24, 25] , [19] and research must continue to de ne their pedagogical value. As shown above, the design of the studies often remains of limited quality, making interpretation still uncertain.
The strengths included the fact that our study was carried out prospectively and randomized and that we used the ANTS grid which is one of the scores which have been well validated [29]. However, it has also several limitations. One of the rst limitations is the use of self-assessment. However, it would have been di cult to set up a study design in which external evaluation could have been used. Although the ANTS scoring system is well validated and widely used [29], this scale is however complex to apply even after speci c training 26 . Another limitation is the lack of assessment of NTS before training. In order to study the impact of a measure on learning, a pre-test evaluation is the reference method. The greater the variation between the pre and post-test assessment, the more effective the action is. However, we were unable to perform a pre-test assessment and we assumed that the residents had the same level of NTS at the start as they had the same previous clinical experience. Moreover, because of monocentric study, the results weren't generalizable. Finally, our study assessed the perception of learning NTS with or without using an OT immediately after training. A remote assessment could also have been of interest to assess knowledge retention.

Conclusions
This randomized, controlled study has shown a positive perception in terms of non-technical skills learning after high-delity simulation training in crisis management but found no difference in most outcomes when a NTS-based OT was used or not in in anesthesia residents.
As this educational tool is believed to be useful to increase the involvement of observers during simulation sessions, additional work is necessary to clarify the place of this tool in improving learners' training. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. AB designed the study, performed the inclusion, analyzed and interpreted the data and was a major contributor in writing the manuscript.

Abbreviations
GCA performed the inclusion, analyzed and interpreted the data regarding and was a contributor in writing the manuscript.
PR performed the inclusion, analyzed and interpreted the data regarding DB designed the study, analyzed and interpreted the data and was a contributor in writing the manuscript.     Figure 1 Study ow chart