While intubation remains an essential skill in critical care training, there are few educational studies that have been published concerning our learner population. Most of the literature on this topic is related to trainees in anaesthesiology and emergency medicine. We successfully implemented an asynchronous module that teaches CCM and PCCM fellows how to accurately assess the difficulty of an airway and to prepare for an intubation procedure. The high response rate on our survey allows for a satisfactory representation of learner reaction and self-assessment of skills.
Participant reaction to the module was overwhelmingly positive. This is supported by feedback from the learners who pointed out that the ease and convenience of technology use, its concise nature and clear organization were strengths. While learner satisfaction is critical to the quality of learner-centered curricula, learner satisfaction remains subjective and does not necessarily represent improvement in knowledge and skill.24 We therefore supplemented this finding with data on other learning and behavior constructs in keeping with the Kirkpatrick training evaluation model.25
Our form of online instruction is independent and requires motivation and self-efficacy. Self-efficacy can impact learner behavior.26,27 For example, greater self-efficacy has been correlated to persistence in learning and other domains.26,28 The positive response on confidence as reported by our learners could explain improvement in knowledge, practice behavior and skills of fellows.
Baseline knowledge of the content matter was high (80%) but these results are likely skewed by the inclusion of senior fellows (2nd and 3rd year) in the analysis. The mean pre-module quiz score for first year was 76%. The overall mean score increased by 13% which supports the module’s effectiveness in knowledge transfer.
We note that first pass success rates by fellows were 88.6% which is somewhat higher than what is reported in the literature for trainees.10 This is likely due to the inclusion of pre-fellowship emergency medicine trained participants whose prior experience emphasized intubation skills. Additionally the heterogeneity of post-graduate year training amongst the group likely contributed to this high first pass success rate. Despite that, the improvement in first pass success post module was not statistically significant. It is likely that the intubation data underrepresented the fellows since the data collection overlapped with a change in academic year where graduating fellow data was eliminated since they would not be completing the module. Another reason for underrepresentation, is the fellows schedule. For example, it would be unlikely for a fellow assigned to a non-ICU block to be intubating and if that happened to overlap with the study period then their skills would not be captured.
Self-reporting of practice change can represent actual behaviour change in the context of measuring outcomes related to medical training programs. 29 The majority of learners indicated that they would change how they document airway risk in their procedural notes and this was in fact observed in the procedural notes we reviewed. All fellows use templates for intubation procedural documentation, though these do not require the inclusion of the components of the MACOCHA. Improving the template could have the potential of conditioning the fellows into needing to think of assessment risk in a more systematic way though this would require institutional and inter-departmental buy-in.
When evaluating the ‘Prepare to Intubate’ module, the authors sought to determine how the content was applied to daily clinical practice and what barriers exist in implementing best practices that were taught in this module. Some potential barriers included patient differences such as the variations in patient population. An intubation plan must be individualized and one cannot teach every nuanced scenario that may arise. Different disease processes may need different airway preparation including allowing airway experts to take over. This was echoed by one of the participants who pointed out that there are circumstances where airway intubation simply should be attempted by the DART team. Another barrier to implementation was simply remembering the content and a suggested solution by one of the participants for this was, “a one-page cheat sheet would be helpful as a quick guide.” Visual aids to help remember technical concepts such as ramping could also assist as a solution for this.
Limitations
Using an experimental design for this study would not have been appropriate in part because our convenience sample of participants was small. A large multicentred experimental design would have protected against threats to internal validity. However, since there was a potential to improve patient care by improving learner behavior and knowledge, we did not want to withhold the educational content from a potential control group.
There are some factors that were identified as potential confounders which may limit the ability to measure the attributable efficacy of the module and its generalizability. The factors include experience in the procedure as reflected by post-graduate year (higher year = more experience) and prior residency training (EM versus Internal Medicine) since EM residents will have more intubation experience than their internal medicine counterparts. Post graduate year (PGY) and prior training were singled out as operator-related variables that could impact first pass success at intubation in the Jung and Kim study thus justifying our need to take this into account when interpreting our results.8 Contrarily, in their retrospective study, Sakles and colleagues found that PGY was not a predictor of adverse events when intubating.6 Though, selection bias may account for this finding since junior EM residents are more likely to intubate the less complicated patients compared to their senior peers.6 Both studies were limited by their observational design but a metanalysis by Crewdson et al supports Jung and Kim’s findings that intubation success is attributed to personnel experience. 6,8, 30
Another limitation to this study was our assumption of attributable educational efficacy. This is difficult to study because while we assume that all fellows completed the online module and learned from it we cannot control for what is learned independently. A more robust way to achieve evidence for response process validity for the knowledge quiz would be to use the ‘think-aloud’ approach by having learners write out their reasoning for selecting an answer on the quiz. Increasing the item numbers would also help reduce the threat of construct-underrepresentation.31 Additionally, submitting a formal external expert content review of the test would have helped in reducing the impact of construct irrelevant variance.31
Basing the skills evaluation on procedural documentation is appropriate to assess for the first pass success construct though it lends itself to social desirability and reporting bias. It may be that the fellows only submit procedure notes and new innovations logs where they were successful in completing the procedure. One way to improve this is by corroborating this with an observer checklist grading of the fellows’ performance, though this is resource exhaustive.
Since the assessment and surveys were completed shortly after the completion of the module, this may only assess their immediate recall which might result in recall bias. This study would have been strengthened by reassessing fellows knowledge and skills at the end of the academic year. Since participants would likely be susceptible to knowledge decay, repeat assessment would be necessary to ensure continued competency.