We have reported the implementation of a Paediatric Early Warning System as a complex Implementation Science QI natural experiment using the MRC Guidance for the Development of Complex Interventions, Action Research method and Action Research theory. The problem was that patients deteriorated in the hospital without being recognised or escalated for critical care, and some of these experienced predictable cardiac arrest or potentially avoidable death. The missed opportunities were due to variation in the observation and monitoring of patients, a clinical knowledge and skills gap, and lack of feedback about optimal care. A complex of interventions was needed to address the multilayer, interconnected issues. The interventions were 1) developing observation and monitoring guidance to standardise practice and provide a template for optimal care, 2) standardised charting with an embedded PEW score, 3) clinical skills training, 4) clinical process audit and feedback and 5) outcome surveillance.
Over time, the process outcomes showed that the proportion of timely unplanned PIC admissions, clinical skills and chart completion compliance improved and was associated with a lower risk of actual mortality but a longer length of PIC stay. This is likely to reflect a change in clinician behaviour with earlier escalation and PIC admission, similar to other studies. [6,9,11] However, these patients also had a longer length of PIC stay, in contrast to Kovolos et al., which reduced capacity for 56 average elective admissions/year. [22] This contradicts our hypothesis for our population that earlier intervention will reduce PIC length of stay. The patients requiring unplanned admissions were younger (median age 12 months) and younger (8 months) over time. It is difficult to explain this trend, but it does provide some information into at-risk populations in our hospital.
A review of the clinical decision-making for patients with unplanned PIC admissions identified a significant improvement in timely PIC admission following implementation from 39% in 2008 to 92% in 2018, similar to the EPOCH trial. [11] Detailed reviews provided educational and institutional learning opportunities and fed back into the observation and monitoring guidance and the training system. These included the need for patient-specific risk factors, parent/nurse concern, frequency of observations, escalation documentation and sepsis triggers. These new parameters appear to be mostly embedded since they were introduced in 2016. This area of learning is rich, vital for quality improvement and needs further investigation.
Routine clinical observations were sometimes performed inaccurately and in a similar range to other studies.[33-40] With purposeful scenario-based training, the clinical observation skills improved from 66 to 82% and appeared to be sustained, similar to other educational interventions. [6,41-45] Despite training during and after implementation, there is inaccuracy and inconsistency in routine clinical observations that may have contributed to missed or delayed opportunities to identify and escalate clinical concerns about deterioration. Only half of the staff who participated in the sessions had read guidance or completed online training a year after implementation. Possible solutions to this could be to ensure that mandated training is completed or that multiple modalities are required to improve the standard of routine clinical observation and adjust to different learning styles. The clinicians were also not able to accurately assess the level of their own skills, which had important implications for ongoing training and assessment.
The seven clinical observations contributing to the PEW score were completed well after implementation in 2008 and had improved in 2018 (87 to 97%). The intermittent audits showed that in 2018, there were 2% of observations that had missing or inaccurate parameters with a consequent inaccurate total PEW score. However, those we identified could have had clinical consequences. The rate of documentation inaccuracy compares favourably to the reported rates of 7.5% in paediatrics versus 7.3 to 42% in adults. [34-36, 46-49].
The aggregate PEW score we use is the most widely validated, but there is little information on real-life performance [1,2,10,11,50]. Our use of the PEW score requires all seven physiological parameters to calculate, and it was documented in 95% of observations in contrast to the EPOCH trial pre-intervention 60 to 75% and post-intervention documentation of 99%. [11] However, the EPOCH trial only required five of the seven parameters for calculation, and one centre reported 93% documentation of all seven parameters. [51] Documentation in paediatric patients appears to be better than the 70 to 90% reported in adult studies. [34-37, 52-54] The score was accurate in 92% of patients (97% of the 95% that had a score documented), which is considerably higher than the 54 to 84% summation error in paediatric and adult studies. [34,46,51,52] Incorrect summation and plotting of parameters could contribute to inaccurate PEW scores, misrepresent the graphic trend and lead to omitted escalation. It is unclear what the acceptable or optimal accuracy threshold is for the score as part of a larger system of overlapping safety.
The overarching aim of the programme was to reduce harm from potentially avoidable critical illness leading to cardiac arrest and death. We identified a reduction in cardiac arrest and crude mortality two years after the implementation of a PEW system. There is a post-intervention halving of the cardiac arrest rate that, given the before and after rates, appears to be associated with PEW system implementation. The predictable cardiac arrest rate also appears to be reducing, but the events are rare. Qualitative review of these cases provides valuable insights and opportunities for improvement, similar to cases with unplanned PIC admission. After three improvement cycles, the total cardiac arrest rate decreased significantly and was associated with a trend towards reduced predictable cardiac arrest and mortality reduced significantly. No further improvement was observed after cycles five and six. The consensus on RRS outcome measurement recommends using ward bed days as a denominator, which may help with future trend analysis. An interesting observation is that the median age (36 months) and severity of illness (PIM2R) of patients suffering cardiac arrest have not changed over time. Patients who have unplanned PIC admissions are younger with a lower severity of illness. Could it be that the system is better at detecting younger patients before cardiac arrest?
Mortality is reduced over time in all populations, which makes it a challenging marker of effectiveness. [11,18-20,55] Taking time trends into account, in an attempt to reduce the bias in a before and after study, it is possible that the reduction in mortality was attributable to the PEW system implementation. The EPOCH cluster randomised control trial showed no difference in mortality across 21 sites internationally. [11] The implementation was six months, and the assessment period was 12 months after intervention. We would argue that our data support longer implementation and embedding to demonstrate effect. [56] With early warning systems, it is difficult to determine from the published literature whether improved outcomes are related to the time or implementation fidelity of a complex intervention in a receptive culture. Single-centre studies that report improved outcomes report longer follow-up from 18 to 36 months. [9,19-22] A longer-term follow-up study of the outcomes at the EPOCH intervention sites would be an interesting test of this hypothesis. In contradiction of the ‘longer time to embed’ hypothesis is a longitudinal time series with the introduction of paediatric Medical Emergency Teams in the USA. [25] There is no additional improvement in mortality or cardiac arrest over the baseline time trends. What isn’t known about these centres is how well the early identification was developed and embedded. The improved outcomes in our centre were also unchanged by the introduction of a nurse-led PART five years later and increased critical care resources in and out of PIC. This supports the benefits of improving identification and escalation without the investment in a RRT. It could be that introducing response resources does not necessarily improve detection and may even distract efforts from the afferent limb to the efferent or response limb of the system.
Implementing a complex dynamic and inter-related intervention in a specialist healthcare environment is challenging. [11, 57-59] Guidance on developing, implementing, evaluating and publishing complex interventions are available [13,59], but there are few actual case studies of implementing complex interventions in healthcare and relating to early warning systems. [60,61] The early warning or rapid response literature has focused broadly on ‘does the score/response team work’ [1,2,3,8,10] and, more recently, attempts to understand why the score/response team appears to work in single-centre studies and not large multi-centre studies [3,10,25,56,62,63]. It could be that the score/response team doesn’t work or it could be that the implementation science is key. The deep dive qualitative studies reveal that compliance with early warning systems requires effective and meaningful communication between multidisciplinary staff as well as the overarching organisational context, including culture, quality improvement, resources, training and staffing. This highlights that we cannot implement or study parts of these systems in isolation; they need to be embedded, maintained and assessed within their appropriate context. Given the iterative improvement cycles required to improve the intervention and compliance and therefore outcomes, it will be difficult to determine the efficacy of these necessarily bespoke interventions with randomised trials. Single centres show that outcomes can improve with or without teams. [19] Despite the methodological bias, it may be worth looking more closely at the interventions in successful single-centre implementation to understand the system and culture change. What appears to be important is measuring and managing the same outcomes, with purposeful quality improvement. [13].
What we have learned from audits about embedding a complex system is the importance of the link with education. Our implementation required a PM followed by a nurse educator and then continued the education process within our normal business. We have updated our training for PEW system e-learning, guidance, charts and escalation to be mandatory and require periodic updates. Predictable and potentially avoidable life-threatening events are debriefed and investigated with root cause analysis; the lessons learned are shared and contribute to revisions of guidance and education. [17,64-69]
We would recommend keeping records of staff who participate in each form of training to help target staff areas that need more or different input. Multimodal educational opportunities, in particular active learning within the real clinical context, are needed for different learning styles. The case reviews, with feedback on the monitoring and identification of each child that has deteriorated, offer a valuable reflective learning experience. Appreciation and recognition of excellent observation, assessment and documentation, enabling early identification, is quick and effective in reinforcing careful monitoring. A Nurse Educator working clinical shifts is another effective way of embedding the skills in real-time in work areas that perceive themselves as too busy to participate in training. Encouragement and reinforcement from nursing and medical leaders is required during daily ward rounds to ensure that the system is being used correctly and to establish a good example with effective role modelling.
Monitoring the rates of all and predictable cardiac arrest is recommended as a quality metric for the evaluation of RRS. [13] The value of reviewing predictable cardiac arrests extends beyond measuring the rate. The qualitative information from the investigations helps to improve the system, education and knowledge. Review against guidelines and standards identifies avoidable factors and creates opportunity for learning. Feedback on each patient helps to reinforce new behaviours and optimise shared knowledge. [66,69] Our avoidable factors remain similar to those reported in 54% of hospital deaths examined in ‘Why children die?’. [4] When our PEW system was introduced, senior clinical decision-makers took time to ‘believe’ that the system was indeed beneficial. These real cases contributed greatly to the visibility of avoidable factors and their faith in the systems detection of deterioration. Dismissive attitudes changed to embrace retrospective review for timely and appropriate management, and in time, behaviour has shifted to prospective review and anticipation. This culture change is a hard outcome to measure, but we believe it is important. [17,61,65]
The main limitation of this study, from a single specialist centre, is the before and after bias for which we have tried to compensate by comparing rates and accounting for time trends. There were also many system changes over the study period, and it is not certain that the PEW system implementation influenced the outcomes. In 2008, the implementation of PEWS coincided with a reduction in overnight doctors in training and the introduction of specialist nurses. It is possible that the improved outcomes could be due to the staffing change or a time effect. There was, however, no change in the measured outcomes with the staffing changes or time in the period from 2011 to 2018. It is disappointing that the reduction in poor outcomes returned to the previous slow rate of improvement during the long follow-up. It may be that we need another ‘innovative disruptive bump’. An example would be using new technologies to improve the ergonomics of detection and escalation, making it easier for clinicians to see deterioration and respond appropriately, as well as have more timely feedback. The process measures were infrequent during implementation and maintenance; it would have been better to have regular measures to inform us when changes occurred. However, being a single centre has had the strength of being able to track the changes within a single but evolving culture. We have gained an in-depth understanding of our system and adapted it in this environment. It is possible that the application of these processes may not be applicable in different or less specialist institutions; however, a version of these processes has been implemented in regional hospitals. By the very complex nature, PEW system implementation will need to take into account the local environment, context, culture and resources. By measuring implementation fidelity with process outcomes as well as the important outcomes hopefully, each journey will be able to minimise missed opportunities and to improve the outcomes for deteriorating patients. [13]
What this natural experiment over 10 years adds to the extensive literature is the time trend of improved detection, reduced cardiac arrest rate and mortality as well as the need for focussed theoretically based QI for these complex interventions in complex systems. The outcomes in our and other centres took 18 months to improve. This will hopefully inform future study designs and national outcome surveillance. We have made significant progress towards improving the detection of deterioration, but unfortunately, we have not quite met our goal of preventing all predictable and potentially avoidable cardiac arrest and deaths. For the future, we will continue to look for opportunities to learn about avoidable factors. We will also continue to implement changes, such as ergonomics, wireless monitoring, electronic charting, smart alarms and advanced communication systems. We will use the Framework for Theorising and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies to help with tinkering and embedding in the pursuit of zero avoidable harm. [61]