The first tool for assessing NTS in healthcare was developed by Gaba et al[29] in 1998. This was an adaption of an instrument called NOTECHS where performance was assessed using video recordings from simulated resuscitations[30] in the context of anaesthetic practice. They found high levels of team variability and concluded that the rating system needs refinement before effectively assessing clinical competence.[29] A number of tools have been validated in clinical context, and although Gaba et al[29] is a different context than this review, it demonstrates validity of using video review to assess NTS in simulated resuscitations.
Bergs et al[19] used the Communication tool to assess presence of audible information transfer from physcian to team members. The tool focused on a single element, communication, an important NTS and function of leadership and teamwork. They assessed 204 recordings in a single centre. There was a trend towards better communication during care of the severely injured patient (p = 0.06). Some information may not have been picked up due to background noise, a confounder which is not corrected for. Bergs et al[19] concluded communication was sub-optimal.
T-NOTECHS was adapted from NOTECHS, a tool previously used in aviation[5]. which had to be validated for clinical application using several steps.[31] Firstly, a draft tool must be developed. This was done for use in the trauma context by Steinemann et al.[23]. Then, a tool must be adapted based on findings of pilot data. Adaptions of T-NOTECHS between papers in this review are the variation in the number of points in the Likert scale used. Three papers used the original 5-point Likert scale.[20, 23, 25] The other two papers [21, 22] utilised the same headings, but reduced the respective scales to a 3-point Likert scale. No study has been identified to validate this contraction. The 5-point scale is more accepted in practice due to increased reliability and validity, alongside its ability to identify extreme attitudes.[32] One paper argued that 3-point Likert scales introduce rounding error but they are quicker to complete which increases the usability.[33] Finally, a tool becomes validated when “researcher has come to the opinion that the instrument measures what it was supposed to measure”.[20, 31] In the context of measuring NTS in a trauma setting, the application of T-NOTECHS by more studies shows that authors of further studies agree with the findings of Steinemann et al,[23] and applied the tool to their own studies.[20–23, 25] The T-NOTECHS scale is shown in Fig. 2.
The CALM tool was developed by Nadkarni et al[28] in 2018 and validated in paediatric simulations to assess team leader performance. It was applied to adult real-life resuscitations by Kava et al[24] to assess individual resident performance as team leader. The CALM tool is shown in Fig. 3. It assess 15 NTS components which is more than the 5 components assessed in T-NOTECHS, providing a greater scope of assessment. T-NOTECHS may be able to give a greater insight into smaller range of NTS assessed.
The T-NOTECHS and CALM tools both assess leadership, communication and team managment. T-NOTECHS emphasises decision making and situational awareness, whereas CALM focuses on medical management and knowledge. These are not distinct categories and demonstrate overlap in some areas, as shown in Fig. 4. T-NOTECHS recognises the response to “untoward findings”, a useful inclusion that helps to validate its use in real-life resusciations, as this is common in the ED.[34] T-NOTECHS is designed to assess team performance, whereas CALM is better suited to assessing individual performance.
Both tools demonstrate a high level of usability. T-NOTECHS provides an explanation for the lowest, highest and middle score to guide the user. CALM uses a simple scoring system which enables the user to assess the frequency at which each NTS is exhibited. T-NOTECHS is potentially easier to complete as limited number of components to rate. When paired with video review, reviewers can pause or rewind the video for a more accurate assessment of NTS.[23] The inter-rater reliability of T-NOTECHS is reported higher than the CALM score,[20, 23, 24] which suggests that T-NOTECHS is a more reliable tool.
Higham et al[1] evaluated tools used for assessment of NTS in healthcare. Due to broader inclusion criteria, this study identified 76 distinct tools, including T-NOTECHS, for assessment of NTS. They noted a large amount of variation between methodology of design of tools, extent of their validity and usability. This was also evident in the comparison of our three assessed tools. They suggest that there is a “need for rationalisation and standardisation in the way we assess non-technical skills in healthcare”. This study was published in 2019 and included Steinemann et al [23], and 4 out of 5 of the studies we reported that used T-NOTECHS were published later. The inclusion of the newer studies in our review furthers the research into the standardisation of assessment of NTS.
The aim of this review was to provide an overview of tools used to assess NTS in resucitation teams within the ED using video review and to explore the evidence for the validity and usability of the tools. This review has answered the stated aims despite having limited number of papers included. We found T-NOTECHS to be the most valid and reliable tool to assess NTS during resuscitation in the emergency deparment using video review, however, it was the only tool validated in this setting. The authors are aware of the difficulty of excluding bias and can hope that the techniques utilised minimised bias.
Due to the heterogeneity of studies, there was limited application of statistical approaches to compare tools. A similar review identifies a need to benchmark outcomes between studies, thus enabling a potential future meta-analysis.[35] The findings of our review provide more clarity on the use of T-NOTECHS as a standardised tool which would enable use of video review as a tool in education and quality improvement.[36] One study translated T-NOTECHS into Finnish to assess translatability and validity and found that it can still be used to assess efficacy of trauma team resuscitations. This study used simulated trauma resuscitations, which was an exclusion criteria for our review.[32]
Steinemann et al[23] also looking at use of T-NOTECHS in the context of simulated resuscitations using video review. Rater agreement was higher in simulated resuscitations than in real-life resuscitations (ICC = 0.71). There was a significant correlation found between the number of completed resuscitation tasks (r = 0.50, P = < 0.01) and faster time to completion of the 3 common resuscitation tasks (r=-0.38, P < 0.05).[23] Simulated resuscitations are a useful tool to assess NTS of staff as there are less ethical considerations when filming patients. However, the nature of the simulated environment does not provide assessors with a true picture of how teams would perform in a real life clinical setting, hence the exclusion from our review.
This review highlights the tools used in this setting and recommends use of T-NOTECHS to assess NTS in resuscitation teams within the ED using video review. In terms of future study, using T-NOTECHS with larger sample sizes, such is in a multi-centre study may greatly establish utility of this tool. A study that uses the same videos but uses two forms of assessment tool could assess superiority in terms of interrater reliability. Furthermore, there is scope to formally compare NTS with TS using video review within the ED.
Limitations
One of the limitations of this review is the small sample size. There are breadth of tools available that assess NTS across all domains of healthcare, however, use of video review in the ED is a growing field and excluding studies without video review reduced the number available. Due to the infrastructure and resource demands to review video creation and validation of a new tool and demonstrating generalisability will be challenging. Use of tools developed and validated in the simulation context requires demonstration of their utility in real-world clinial care.
Many institutions lack audio-visual recording access due to finanacial and ethical restraints, therefore there is limited generalisability for these findings. Researchers may be faced with a reluctance to be filmed due to privacy concerns from staff regarding patients and themselves. There should be strict measures in place to ensure recordings are only accessed by appropriate personnel to ensure privacy and security.