Immediate and long-term effects of an electronic medication management system on paediatric prescribing errors: A stepped-wedge cluster randomised controlled trial with one-year follow-up

doi:10.21203/rs.3.rs-2083575/v1

Download PDF

Article

Immediate and long-term effects of an electronic medication management system on paediatric prescribing errors: A stepped-wedge cluster randomised controlled trial with one-year follow-up

https://doi.org/10.21203/rs.3.rs-2083575/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 13 Dec, 2022

Read the published version in npj Digital Medicine →

You are reading this latest preprint version

Objectives: To assess the immediate (first 70 days of use) and long-term (one-year) effectiveness of an electronic medication management (eMM) system to reduce prescribing errors, and their potential and actual harm in paediatrics.

Design: A stepped-wedge cluster randomised controlled trial (SWCRCT) over 11 weeks (April–July 2016): 8 clusters randomised for eMM implementation. All medication orders during the trial and for a random sample of medication orders one-year post-eMM implementation (June–September 2017) were reviewed. Clinical prescribing errors were rated for the severity of potential harm on a 5-point scale. Errors (score ≥3 ie potential adverse drug events [ADEs]) identified in the SWCRCT were assessed for actual harm (actual ADEs).

Setting: A 310-bed paediatric referral hospital in Sydney, Australia.

Participants: Inpatients.

Control and Intervention: Paper medication charts (control); eMM (intervention).

Primary outcomes: clinical prescribing errors, potential ADEs. Secondary outcomes: procedural errors, clinical prescribing errors for high-risk medications and by error type, clinical errors associated with actual harm (ADEs) during the SWCRCT.

Results: 35,260 medication orders for 4,821 patients were reviewed. In the first 70 days of eMM use there was no significant change in overall clinical prescribing error rates (Incident Rate Ratio [IRR] 1.05[95%CI 0.92-1.21], p=0.45). However, potential ADEs significantly increased by 62% (IRR 1.62[95%CI 1.28-2.04], p<0.001).

One-year post-eMM, clinical prescribing errors declined by 36% (IRR 0.64 [95%CI 0.56-0.72], p<0.001) and errors associated with high-risk medications decreased by 33% (IRR 0.67 [95%CI 0.51-0.88], p=0.004) compared with rates pre-eMM. Dose, route, frequency and duplicate therapy errors accounted for over 90% of errors in each study period. Dose error rates were more than double that for any other error type.

Few errors in the SWCRCT were associated with actual harm (paper 10.01/1000 clinical prescribing errors[95% CI 5.97-16.73]; immediately post-eMM 8.84/1000 errors[95%CI 6.04-12.92]). 71%[95%CI 50-86] of patients with actual harm experienced a dose error.

Discussion: This is the first randomised controlled trial of the effects of eMM on paediatric prescribing errors. The first 70 days of use showed no overall improvement and an increase in some categories of errors. The results suggest immediate risks to medication safety during the early periods of system use. A year after implementation error rates significantly declined from baseline suggesting long-term benefits to safety. Optimisation of eMM should focus on features to reduce dose errors due to their high frequency and greater capacity to cause harm.

Medication errors are a recognised persistent problem in paediatrics.¹ Information technologies including electronic medication management systems (eMM, also known as e-prescribing or Computerised Provider Order Entry [CPOE]) have been advanced to reduce medication errors in this population.^2–4 Multiple studies of eMM use and its effects in adult hospitals have been conducted,⁵ with a recent systematic review finding nine of 10 studies reported a significant reduction in prescribing error rates.⁵ However, reviews on this topic persistently report marked inconsistencies in study quality, error definitions and denominators.^5–7 Few studies have investigated the effects of this technology in paediatric hospitals.¹

Prescribing in children is complex as decisions often require information about weight, age, dosing ranges and off-label use of medicines, and this increases the risk of medication errors.^{8 9} The consequences of prescribing errors can be significant given children's physiological and developmental vulnerabilities.¹⁰ Concerns have been raised that eMM systems have often not been well-adapted to paediatric medication challenges^11–14 (e.g., the increased complexity of weight-based dose calculations in children and the absence of detailed paediatric-specific medicines information in the decision support provided). Hence findings from eMM studies in adult hospitals may not be directly extrapolated to paediatric populations.

eMM systems have multiple mechanisms by which they are designed to support safe prescribing and reduce errors. These include: improved legibility of medication orders; structured order sentences for selection and editing; decision support (both passive e.g. online information resources and active e.g. alerts) available at the point of prescribing.

eMM evaluation studies undertaken in paediatrics have largely employed before and after designs, and most have focused on systems used in intensive care.^{5 9 15} The existing limited evidence suggests that eMM may be more effective in reducing only certain error types in paediatrics. For example, Walsh et al¹⁶ found paediatric eMM systems were effective in reducing incomplete/wrong order errors but not dose errors, and a review by Tolley et al¹⁷ identified eMM features which contributed to dose errors in paediatrics. Greater knowledge of these outcomes is needed to inform system design and use. No randomised controlled trials of the effectiveness of eMM to reduce medication errors on general paediatric wards have been conducted, and few studies have attempted to quantify changes in actual harm associated with eMM use.^{1 5 18}

As the adoption of clinical information systems continues worldwide, questions about how best to implement these systems in safe and effective ways remain. The strengths and weaknesses of sequential versus whole of organisation (big bang) roll-outs are advocated with limited evidence to guide decision-making.^{19 20} Feasibility and resource issues will often drive the implementation strategy. eMM evaluation studies have focused on measurement at specific points in time and have provided limited evidence of the temporal impact of systems on medication safety.⁵ Such information would be valuable in identifying both when implementation support is most required and at what point medication safety benefits may be expected.

In this study, we conducted a stepped-wedge cluster randomised controlled trial (SWCRCT) with an additional one-year follow-up pre/post study to assess the immediate and long-term effectiveness of an eMM system to reduce prescribing errors, their potential severity and associated actual harm at a major paediatric referral hospital.

Characteristics of study cohorts

In total 35,260 medication orders for 4,821 patients across all cohorts (4,509 unique patients) were retrospectively reviewed for errors. Table 1 reports the characteristics of patients and orders.

Table 1

Characteristics of patients and medication orders in the three cohorts
	Paper	Immediate post-eMM	One-year post-eMM
Number of patients	1686	2096	1039
Number (%) female	691 (40.98)	838 (39.98)	460 (44.27)
Mean (SD) age, months	73.9 (62.4)	88.4 (63.4)	86.1 (63.5)
Median (IQR) age, months	55.5 (18.0-124.8)	85.0 (27.0-142.0)	80.0 (26.0-139.0)
Number of admissions	1769	2298	1201
Number of patient-days	5252	8384	4458
Number of orders	9635	16734	8891
Number (%) orders for high-risk medications (See Box 1)	1200 (12.45)	2274 (13.59)	1334 (15.00)

Changes in prescribing error rates in the immediate and one-year post-eMM periods

Overall, clinical prescribing error rates did not significantly change (IRR 1.05; p = 0.45) in the immediate post-eMM period, but there was a significant 62% increase (IRR 1.62; p < 0.001) in potential ADEs (Table 2). Procedural prescribing errors significantly declined (IRR 0.65; p < 0.001) during the immediate post-eMM period (Table 2 and further details available in supplement 5).

One-year post-eMM, clinical prescribing error rates had significantly declined by 36% (IRR 0.64; p < 0.001) compared to when paper medication charts were used (Table 2). There was also a 33% reduction in errors associated with high-risk medications between the paper and one-year post-eMM (IRR 0.67; p = 0.004) periods (Table 2) and potential ADEs returned to a similar level to the paper period.

Table 2

Clinical and procedural prescribing crude error rates and adjusted incidence rate ratios (IRRs) for all study periods
	Paper		Immediate Post-eMM				One-year Post-eMM
	N	Errors/ 100 orders (95% CI)^	N	Errors/ 100 orders (95% CI)^	IRR: eMM vs Paper (95% CI)†	P	N	Errors/ 100 orders (95% CI)^	IRR: One-year post- eMM vs Paper (95% CI)†	P
All patients	9635 orders (1686 patients)		16734 orders (2096 patients)				8891 orders (1039 patients)
All clinical prescribing errors	1399	14.52 (13.78–15.30)	2942	17.58 (16.96–18.23)	1.05 (0.92–1.21)	0.45	1043	11.73 (11.04–12.46)	0.64 (0.56–0.72)	< 0.001
Clinical prescribing errors, with potential severity ≥ 3 (potential ADEs)	353	3.66 (3.30–4.07)	812	4.85 (4.53–5.20)	1.62 (1.28–2.04)	< 0.001	299	3.36 (3.00-3.77)	0.93 (0.72–1.19)	0.55
Patients with high-risk drug orders	1200 orders (799 patients)		2274 orders (1228 patients)				1334 orders (607 patients)
Clinical prescribing errors in high-risk drug orders²¹	232	19.33 (17.00-21.99)	390	17.15 (15.53–18.94)	0.88 (0.66–1.16)	0.35	176	13.19 (11.38–15.29)	0.67 (0.51–0.88)	0.004
Patients with procedure errors reviewed	6698 orders (1162 patients)		11441 orders (1429 patients)
All procedural errors	7652	114.24 (111.71-116.83)	7699	67.29 (65.81–68.81)	0.65 (0.61–0.70)	< 0.001
*Procedural errors were assessed from a random sample comprising 68.5% of medication orders during the entire SWCRCT period. Details of changes in procedural errors by error type are in supplement 5. ^Crude error rates and 95% CIs are from simple Poisson models. †Incidence Rate Ratio (IRR) and 95% CIs from models accounting for patient age, study time (immediate post-eMM model only), and clustering by ward (unit of randomisation) and patient. ADE is adverse drug event.

Changes in types of clinical prescribing errors in the immediate and One-year post-eMM periods

The most frequent types of clinical prescribing errors in all cohorts were dose, route, frequency and duplicate drug therapy errors, in total making up over 90% of errors in each period (Table 3). Dose errors comprised more than 40% of all clinical prescribing errors in all cohorts. Overdose error rates were consistently higher in all periods compared to underdose errors (Table 3).

During the immediate post-eMM period, rates for three error types increased: wrong dose (IRR 1.22), route (IRR 1.65) and duplicate therapy (IRR 1.82) errors (Table 3). Wrong frequency errors remained relatively constant during the immediate post-eMM period.

One-year post-eMM, wrong dose (IRR 0.79), wrong route errors (IRR 0.44) and wrong frequency errors (IRR 0.47) declined compared to when paper charts were used. However, duplicate drug therapy error rates continued to be higher than when paper medication charts were used (IRR 1.48) (Table 3).

Table 3

Clinical prescribing error rates by type and changes over study periods
Error type	Paper (n = 9635 orders)		Immediate post-eMM (n = 16734 orders)			one-year post-eMM (n = 8891 orders)
	Errors	Errors/100 orders (95% CI)^	Error	Errors/100 orders (95% CI)^	IRR, eMM vs Paper	Errors	Errors/100 orders (95% CI)^	IRR, one-year post vs Paper
Wrong dose	644	6.68 (6.19–7.22)	1288	7.70 (7.29–8.13)	1.22 (1.04–1.43)	534	6.01 (5.52–6.54)	0.79 (0.68–0.93)
Overdose	308	3.20 (2.86–3.57)	674	4.03 (3.74–4.34)	1.22 (0.96–1.55)	247	2.78 (2.45–3.15)	0.79 (0.63–1.01)
Underdose	223	2.31 (2.03–2.64)	408	2.44 (2.21–2.69)	1.25 (0.95–1.66)	183	2.06 (1.78–2.38)	0.83 (0.64–1.07)
Wrong frequency	315	3.27 (2.93–3.65)	522	3.12 (2.86–3.40)	0.92 (0.73–1.16)	157	1.77 (1.51–2.06)	0.47 (0.36–0.61)
Wrong route	199	2.07 (1.80–2.37)	396	2.37 (2.14–2.61)	1.65 (1.20–2.27)	77	0.87 (0.69–1.08)	0.44 (0.27–0.71)
Duplicated drug therapy	140	1.45 (1.23–1.71)	457	2.73 (2.49–2.99)	1.82 (1.26–2.63)	205	2.31 (2.01–2.64)	1.48 (1.05–2.09)
Wrong drug	24	0.25 (0.17–0.37)	62	0.37 (0.29–0.48)	1.82 (0.66-5.00)	15	0.17 (0.10–0.28)	0.49 (0.11–2.23)
Drug-drug interaction	20	0.21 (0.13–0.32)	71	0.42 (0.34–0.54)	0.69 (0.23–2.09)	17	0.19 (0.12–0.31)	0.39 (0.09–1.73)
Allergy	20	0.21 (0.13–0.32)	19	0.11 (0.07–0.18)	0.45 (0.12–1.75)	13	0.15 (0.08–0.25)	3.17 (0.48–20.9)
Inadequate monitoring	19	0.20 (0.13–0.31)	40	0.24 (0.18–0.33)	1.19 (0.34–4.19)	5	0.06 (0.02–0.14)	---
Wrong formulation	7	0.07 (0.03–0.15)	36	0.22 (0.16–0.30)	0.74 (0.15–3.74)	7	0.08 (0.04–0.17)	---
Wrong timing	4	0.04 (0.02–0.11)	34	0.20 (0.15–0.28)	---	1	0.01 (< 0.01–0.08)	---
Wrong strength	3	0.03 (0.01–0.10)	10	0.06 (0.03–0.11)	---	9	0.10 (0.05–0.19)	---
Wrong patient	2	0.02 (0-0.07)	2	0.01 (< 0.01–0.04)	---	0	0 (0-0.04)
Wrong rate	2	0.02 (0.01–0.08)	2	0.01 (< 0.01–0.05)	---	3	0.03 (0.01–0.10)	---
No longer indicated	0	0 (0-0.04)	3	0.02 (< 0.01–0.05)	---	0	0 (0-0.04)	---

^Crude error rates and 95% CIs are from simple Poisson models. IRRs are from models adjusting for study time (immediate post-eMM model only), patient age, and clustering by ward and patient. --- = too few errors for modelling.

Potential ADEs and evidence of error detection by staff

In total 27.2% (1464/5384) of clinical prescribing errors were rated with a potential harm score of ≥ 3. There was little change in this proportion over the study periods (Table 4).

Table 4

Severity of potential harm from clinical prescribing errors by cohort
Potential harm* severity	Paper		Immediate post-eMM		One-year post-eMM
	n	% (95% CI)	n	% (95% CI)	n	% (95% CI)
1: No/minimal harm	382	27.3% (24.5–30.1)	797	27.1% (25.2–29.1)	281	26.9% (23.7–30.3)
2: Temporary harm requiring monitoring	664	47.5% (44.7–50.3)	1333	45.3% (43.4–47.3)	463	44.4% (41.1–47.7)
3: Temporary harm requiring intervention	299	21.4% (18.6–24.2)	686	23.3% (21.4–25.3)	266	25.5% (22.2–28.8)
4: Permanent harm requiring intervention	48	3.4% (0.6–6.3)	112	3.8% (1.9–5.8)	28	2.7% (0–6.0)
5: Potential death	6	0.4% (0-3.3)	14	0.5% (0.0-2.5)	5	0.5% (0.0-3.8)
Total of prescribing errors with a score ≥ 3	353	25.2% (23.0-27.6)	812	27.6% (26.0-29.3)	299	28.7% (25.9–31.5)
*Definitions provided in supplement 2. Confidence intervals (CIs) for the potential harm categories are 95% simultaneous multinomial CIs. For the total, CIs are exact (Clopper–Pearson) binomial 95% CIs.

A small proportion of clinical prescribing errors were detected by staff but this proportion increased significantly with eMM implementation (test of equal proportions χ² = 29.6, 1 df, p < 0.001). In the paper period 9.3% (95%CI 7.8–10.9) of errors had some evidence of detection. In the immediate post-eMM period this increased to 15.4% (95%CI 14.1–16.7) and at one-year post-eMM was 17.3% (95%CI 15.0-19.7).

Actual harm associated with clinical prescribing errors in the paper and immediate post-eMM periods

In total, 368 errors involving 172 patients were reviewed by the harm assessment panels. For 21 (12.2%) patients with 40 clinical prescribing errors, there was evidence in patients’ records of associated harm. Of these 21 patients, 8 (18 errors) experienced errors resulting in minor harm, 12 (21 errors) moderate and 1 serious harm (1 error). In 9 cases, the harm panel rated the likelihood that the harm was caused by the error as possible, in 8 cases probable and in 4, certain. Dose errors were attributed to the actual harm occurring in 71% (15/21; 95%CI 50–86) of patients experiencing harm; (paper period 8 dose errors/11 patients with harm; immediate post-eMM 7/10) (supplement 3). Table 5 reports prescribing errors associated with actual harm by a range of denominators.

Table 5

Clinical prescribing errors associated with actual harm (actual ADEs) during the paper and immediate post-eMM period
Prescribing errors with actual harm	Paper (Number of errors with actual harm = 14)	Immediately post-eMM (Number of errors with actual harm = 26)
per 1000 orders	1.45 [95% CI 0.87–2.44]	1.55 [95% CI 1.06–2.28]
per 1000 errors	10.01 [95% CI 5.97–16.73]	8.84 [95% CI 6.04–12.92]
per 1000 patient-days	2.66 [95% CI 1.59–4.47]	3.10 [95% CI 2.12–4.54]
per 100 admissions	0.79 [95% CI 0.47–1.32]	1.13 [95% CI 0.77–1.65]
eMM – electronic medication management system; CI – confidence interval (binomial Wilson method) Denominator numbers are presented in Table 1.

Effectiveness of eMM in paediatrics to reduce prescribing errors

We aimed to assess the effectiveness of eMM use in paediatrics to reduce prescribing errors both in the short and long-term. We found that during the immediate post-eMM period there was no significant decrease in overall clinical prescribing error rates (IRR 1.05; p = 0.45). However, potential ADEs increased by 62% (IRR 1.62; p < 0.001).

Our pre-post one-year follow-up study showed a significant 36% reduction (IRR 0.64, p < 0.001) in overall clinical prescribing errors. Prescribing errors involving high-risk medications had decreased by 33% (IRR 0.67; p < 0.004) and potential ADEs returned to a rate similar to that pre-eMM (IRR 0.93; 95%CI 0.72–1.19).

Very limited comparative data are available, with differences in study designs and error definitions and identification processes applied.²² To date, decision makers have largely relied upon evidence of eMM effectiveness in paediatrics from two studies conducted on general paediatric wards in the United States undertaken between 2006 and 2008.^{16 23 24} One^{23 25} uncontrolled pre/post study showed a reduction in medication errors post-eMM and the second¹⁶ (an interrupted time series study) no change. A further small uncontrolled pre-post 2006 UK study²⁶ reported an increase in prescribing errors 6 months post-eMM.

The time periods over which previous studies have examined effectiveness have also varied. Studies by Walsh et al¹⁶ and Barber et al²⁶ reported no reduction in medication error rates at 6 and 9 months post-eMM, while Holdsworth et al²³ found significant reductions at 15 months post-eMM. While these studies provide a small comparative evidence-base, together with our findings they are broadly consistent in demonstrating that beneficial effects of eMM on prescribing may not be apparent for a period of around 12 months. This provides a useful time parameter to inform future studies regarding when post-implementation assessments should be considered and when benefits may be likely to commence.

To understand the experiences of clinical staff during the implementation and early stages of eMM use, in parallel with the SWCRCT, we conducted 122 short interviews with doctors and nurses on all wards 1, 3, 6 and 26 weeks post-eMM.²⁷ Inexperience with system use and issues with system usability, functionality and accessibility were frequently reported in the immediate weeks following implementation, with clinicians suggesting that these factors contributed to delayed medication delivery and reduced medication safety.²⁷ Dealing with hybrid paper and computer medication systems was also identified as problematic. These themes were generic across wards during the early weeks of system use and are factors which assist in explaining the failure of prescribing errors to decrease in the immediate post-eMM period.

Interview findings demonstrated that issues raised by staff changed over time.²⁷ Six months post-eMM, many initial problems reported by staff (e.g. inexperience with use) were no longer mentioned. At 6-months post-eMM new themes emerged with clinicians increasingly reporting that the system was making a positive contribution to medication safety, consistent with the changes in error rates observed longer term. However, new issues also became apparent, including concerns about new types of errors, e.g. drop-down menu selection errors, associated with system use.^{17 28} Some of these technology-related errors may decrease as users become more familiar with the system and identify ways to work around system limitations. Thus a reduction in these technology-related errors following increased eMM system familiarity may also have contributed to both the lack of change in prescribing error rates in the immediate post-eMM period and the later observed reduction in error rates 12 months post-eMM.

An important new finding was evidence of an increase in some categories of high-volume clinical prescribing errors during the first 70 days post-eMM. Four error types accounted for over 90% of all errors in each study period, and rates for three of these common error types increased immediately post-eMM. These results suggest the eMM system had generalised effects on prescribing, consistent with users’ initial unfamiliarity and inexperience with system use, and broad disruptions to workflows,²⁹ all of which were reported by users in the early weeks of system use.²⁷

Training was mandatory for staff in order to use the system and was provided close to the time of implementation. Yet many still reported considerable challenges in system use. This situation is likely a reflection of the differences between using a system in a training situation compared to a real-world setting where contextual factors such as time pressures, interruptions and integration with other related work practices come into operation. It is very difficult to mimic these real-world work demands and pressures within a system training environment. These findings re-emphasise the importance of high levels of support, training and close monitoring during the first several months of use.

One-year post-eMM, reductions in three of the four common paediatric prescribing errors had occurred. These reductions suggest that, in the longer-term, with users’ increased familiarity, with the system, the core functionality of eMM (e.g. the ability to select from standardised order sentences, legibility of all orders, decision support such as dose guidance) contributed to reduced prescribing errors. Duplicate drug therapy errors increased immediately post-eMM and remained higher one-year post-eMM. A doubling of duplicate therapy errors was identified in a recent before and after study of eMM in a paediatric intensive care unit in Ireland.³⁰ Feedback from our study pharmacists indicated a likely contributing factor was that clinicians seeking to correct an order in the eMM would type a new order but then fail to cease the original order, leading to duplicate orders being active. Hybrid paper-eMM situations also contributed to duplicate therapy errors, with some orders appearing both on paper and electronic charts during the implementation period. Some medications remained on separate paper charts, for example sliding scale insulin orders and some specialised pain medications, and these are likely to have contributed to duplicate drug therapy errors in the longer term. These error types should be amenable to improvement by reducing hybrid systems, alerting users via training, and system design enhancements.

Future optimisation of eMM

Wrong dose errors are the most concerning type of error, given their high frequency and potential for harm. Dose errors, including 10-fold dose errors,^{30 31} have consistently been identified as problematic in paediatrics, despite the introduction of eMM.^{9 14 16 30 31} Reasons include the complexity of dose calculations for paediatrics and the sometimes limited guidance available to clinicians regarding medication use in children with specific conditions.

These findings indicate that a key focus for optimisation of eMM should be reducing dose errors. System features which provide support and guidance for prescribers, such as indication-dose alerts, weight-based filtering of order sentences, alerts to indicate major variations in patient weight that may result in dosing errors, and dose-weight calculators should be considered.^{17 32} Such efforts to reduce dose errors are likely to bring the most benefit to reducing the risk of patient harm, in tandem with education in safe paediatric prescribing principles.^{8 33}

Harm associated with clinical prescribing errors

Overall we found a low rate of prescribing errors associated with actual harm. Harm rates both before and immediately post-eMM were similar. However, of patients with errors resulting in actual harm, the majority of cases (71%) involved dose errors (both with paper prescribing 72% and immediately post-eMM 70%).

Few studies have assessed actual harm associated with medication errors. Holdsworth et al²³ compared harm associated with multiple types of medication errors (including administration errors) occurring in paediatric wards and intensive care units. They reported a change in errors causing harm from 3.8/100 admissions to 2.2/100 post-eMM, considerably higher than found in our study (0.79/100 admissions pre and 1.13 post). A limitation of our study was that our harm identification relied upon retrospective record reviews. In contrast, Holdsworth et al²³ conducted prospective record reviews and interviews with staff close in time to when errors occurred. This enabled the capture of information which may never be recorded in patients' records. Thus, our rates of prescribing errors causing actual harm are likely to be an underestimate of the true rate.

For over 80% of all clinical prescribing errors identified in our retrospective record audits there was no evidence that errors had been detected by staff. Thus, specific actions, such as monitoring or testing for signs or symptoms indicative of adverse effects, rarely occurred. An absence of investigations and monitoring information, which might provide clues as to possible adverse effects, limited the capacity of our panels to identify harm. An absence of documented evidence was generally equated to be an absence of adverse effects by our clinical harm panels. The clinical complexity of cases reviewed was also noted to be generally very high and abstracting the potential consequences of an error for children with complicated clinical presentations who were on a range of medications was often difficult for the panel. These challenges should be considered by future researchers embarking upon this process. Due to resource constraints we did not conduct harm panels one-year post-eMM and so cannot determine whether errors associated with harm decreased in the long-term.

Strengths and limitations

The SWCRCT design with one-year follow-up allowed a rigorous evaluation of the effectiveness of the eMM system to reduce prescribing errors. The 12-month follow-up study was an uncontrolled pre-post assessment. The research team had close and ongoing consultation with the hospital during the entire study and no significant medication safety interventions were instituted during this time period. However, we cannot rule out other factors that may have contributed to the reduction in error rates seen at the one-year follow-up. The study was limited to a specialist paediatric hospital, and so generalisability to other sites is unclear. Retrospective prescribing error harm assessments were limited by the level and quality of medical record documentation.

Our findings demonstrated no initial benefit from eMM in the first 70-days of use and suggested measurable risks to patient safety in this period due to increases in potentially harmful prescribing errors. Thus, close attention to support staff in training and integration of the system use into clinical work practices, along with monitoring of errors during the first few months of system use should be a priority. One year later, the pre-post study showed prescribing error rates had significantly declined. Future optimisation of paediatric eMM systems should focus on features to reduce dose errors due to their high frequency and greater capacity to cause harm in this population.

Study design and setting

A SWCRCT with an additional one-year pre/post follow-up study was undertaken at a 310-bed major urban paediatric tertiary referral hospital, one of two such hospitals serving the 8.2 million population of New South Wales, Australia.³⁴ All inpatients during the study periods were included, except those in oncology and intensive care units, as the hospital determined that eMM implementation was not possible on these wards as part of the SWCRCT.

Nine wards were grouped into eight clusters prior to randomisation. Two wards were randomised together as there was direct physical access from one ward to the other, and they treated similar patients. Researchers, blinded to cluster/ward identity and independent of the hospital, randomly assigned (using a computer number generator) the sequence of cluster eMM implementation. The trial had 11 steps of one week duration (Fig. 1). The hospital determined the one-week step between cluster implementations. Patients could be in different steps and/or clusters due to ward transfers or readmissions.

Following the publication of the trial protocol,³⁴ additional data were collected 12 months post-eMM implementation to allow conduct of a pre/post eMM comparison at this time point. This was not part of the SWCRCT study. Medication administration error data were collected using direct observation during the SWCRCT and will be reported in a separate publication, as will the impact of eMM on patient length of stay. The second planned hospital site was not able to participate in a SWCRCT as originally planned in the protocol due to changes to the implementation plan by the hospital.

The control and intervention

In the control period, medication orders were hand-written on paper charts. During the SWCRCT, clusters sequentially implemented the eMM. The intervention was an eMM module added to the hospital’s existing clinical information system (Cerner Corporation). The eMM allows electronic prescribing, recording of drug dispensing, drug administration and medication reconciliation and monitoring. The eMM contained passive and active decision support, e.g. guidelines, policies, protocols, order-sets, order sentences, dose-range limits, therapeutic drug monitoring and related pathology parameters, drug-drug interaction alerts, and dosage calculators. All wards had access to the same decision support. Specialised pain management prescribing remained on paper, as did sliding scale insulin and insulin pump orders.

The hospital organised an extensive staff training program, in how to use the eMM system, which prescribers were required to complete. Training was provided for staff on each ward two weeks prior to implementation, adopting a “just in time” training approach. Multiple online resources were made available to assist with specific processes.³⁵ Training was classroom-based with further self-directed modules. Junior medical officers, who complete the majority of prescribing tasks, were required to demonstrate competency during the class via workbook exercises and only then allowed access to the system. Further, during the implementation, support staff were available on individual wards to assist with implementation and trouble-shooting. On the first day of implementation on each ward, paper medication charts were copied to the eMM system. All new patients entering the ward from that time onwards were entered into the eMM system. If patients were transferred to a ward without the eMM system, their medications would be transferred to a paper medication chart. Once the ward implemented the eMM, it was mandatory for staff to prescribe medications using the system. If an order was not placed in the system, the pharmacy could not dispense and a nurse could not administer the medication. All paper prescription pads and paper medication charts were removed from the wards. Downtime medication charts were available in the event of a downtime, but these were stored separately.

Prescribing error data collection

Medication orders for inpatients in all study periods (April-July 2016; June-September 2017) were reviewed retrospectively by one of three clinical pharmacists, independent from the hospital, to identify and classify clinical and procedural prescribing errors³⁶ (supplement 1). The order in which records were reviewed was randomised across all wards. All errors identified by the pharmacists were rated for their potential harm severity (to identify potential adverse drug events [ADEs]) using a five-point scale based on an adapted version of the NCC MERP Index for categorizing medication errors³⁷ (supplement 2). Any documentation which indicated that staff had detected an error (e.g. note of an order correction; a case conference with parents) was recorded. Details of all medications ordered (date of prescribing, medication name and route) for each patient admission were recorded to provide denominators for error rate calculations.

Interim analysis of procedural errors was conducted after medication charts of 2000 admissions had been audited. Collection of procedural errors ceased because of the high frequency and time taken to record these errors. In total, procedural prescribing errors were collected for a random sample of 2591 patients (18,139 medication orders), which accounted for 68.5% of admissions across the entire 11-week SWCRCT. Procedural errors were not collected during the one-year follow-up.

A random stratified sample of patient records in the eight clusters was obtained one year from the date of last cluster implementation (between 20 June 2017 and 20 September 2017). Stratification was conducted by cluster/ward. Number of patients for each ward was decided based on the distribution of patients admitted in study wards. Hence the number per ward was different. Patients were then randomly selected using a random number generation code (SAS).

Training and inter-rater reliability assessment

The prescribing error definitions and detailed guidelines for how to review records and classify errors were developed based upon an existing error classification applied in studies in adult hospitals by the research team.³⁶ This classification was modified to account for differences in paediatric prescribing informed by hospital prescribing guidelines (supplement 1).

Training in the use of the error classification involved pharmacist reviewers applying the classification to the same subsets of patient records and then meeting to discuss any differences in outcomes. Once training was complete, inter-rater reliability between the three pharmacist reviewers (the most experienced, selected as the gold standard) was conducted to ensure all reviewers had good agreement in classifying prescribing errors and potential severity of those errors. The pharmacist reviewers obtained a minimum Cohen’s kappa score of 0.77 on error type and 0.76 on severity level before they commenced independent data collection. The second round of inter-rater reliability was conducted a year later (minimum kappa 0.95 and 1.0 respectively).

Assessing actual harm associated with clinical prescribing errors

A multi-disciplinary harm assessment panel (comprising a paediatrician, paediatric nurse and pharmacist) met on 12 occasions to determine actual harm associated with errors (ie actual ADEs). The medical records of patients with errors with a potential harm severity score of ≥ 3 were reviewed by clinically trained members of the research team, prior to cases being referred to the harm panel, to exclude errors not administered to patients (e.g. order with dose error that was not administered; or duplicate orders not administered).

Detailed case reports (outlining a patient’s medical history, prescribing errors, other medications, signs and symptoms, changes to treatment/care before and after the error, and relevant test results) were prepared by clinically trained members of the research team. The harm panel members each reviewed the cases individually, and then met to gain consensus about any actual harms and the severity. Some patients experienced multiple errors (supplement 3). Where harm was identified, the severity was classified using the Harm Associated with Medication Error Classification (HAMEC)³⁸ and the probability that the error caused the harm was classified using a modified version of the WHO Uppsala Monitoring Centre categories³⁹ (Supplement 2). Overall, this was a laborious process which commenced in 2019, and completed in 2021. Reduced access to clinicians for the panels due to the pandemic slowed progress.

Sample size calculations

As outlined in our protocol, the expected reduction in overall prescribing error rate was 60%, from 4.06 errors per admission (SD = 5.27) to 1.62 (SD = 2.87) with an estimated intraclass correlation coefficient (ICC) of 0.06. Collecting data on eight clusters would allow detection of a minimum change of 20% for overall errors with 100% power and 42% change for potential ADEs with 83% power (for two-sided tests; 𝛼<5%). At each step, records for 112 patient admissions were estimated to be required, totalling 1,232 across the study.

The one-year follow-up sample size was calculated based on data from 1817 patients generated during the SWCRCT. Using these data, we calculated a clinical prescribing error rate of 0.24/patient-day (SD = 0.78) with an ICC of 0.04 for eight clusters. To detect a 35% change one-year post-eMM, we required 1230 patient records for the one-year post-eMM follow-up sample (for a two-sided test; 80% power; 𝛼<5%).

Analysis

The study involved comparison of error rates for three cohorts: 1) Paper, when prescribing occurred on paper medication charts in the control period during the SWCRCT; 2) immediately post-eMM, (up to 70 days of eMM use during the SWCRCT); and 3) one-year post-eMM, when prescribing using the eMM had been in place for one year on all wards. Outcome measures are listed in Box 1.

Box 1: Outcome Measures

Primary Outcome Measures

Clinical prescribing error rates
Potential adverse drug event rates (potential ADE, potential harm severity score ≥3)

Secondary Outcome Measures

Clinical prescribing error rates by type, and those associated with high-risk* medications
Error detection rates - Prescribing errors where there was documentation in patient records indicating that errors had been detected by staff
Actual ADEs (Prescribing errors that resulted in harm, injury or required responsive action (only for the SWCRCT)
Procedural prescribing error rates (only for the SWCRCT)

*High-risk medications were defined by the hospital as anti-infectives, potassium and other electrolytes, insulin, narcotics/opioids and sedatives, chemotherapy agents and heparin and other anticoagulants. (Sydney Children's Hospital Network. High risk medicine register, 2015.)

We applied an intention to treat analysis, with all orders analysed according to whether the ward was using eMM or paper medication charts. Crude error rates per 100 orders with 95% confidence intervals (CIs) were estimated from simple Poisson regression. Errors per 100 orders in the SWCRCT, i.e. paper and immediate post-eMM periods, were modelled using mixed-effects negative binomial models accounting for the stepped-wedge design.^{40 41} The dependent variable was the number of errors per patient-day, with an offset of the logarithm of the number of orders per patient-day. Random effects terms were included for cluster (unit of randomisation) and patient (to account for admissions over multiple days). As observations for each day were available, we modelled the secular trend as continuous using restricted cubic splines with 4 degrees of freedom.^{41 42} As the average patient age was younger in the paper cohort (because of the order of eMM rollout to wards with different patient profiles), we adjusted for age in months as a restricted cubic spline with 2 degrees of freedom (supplement 4). We used splines for the secular trend and patient age to avoid assuming linear associations with errors. Knots were placed at recommended quantiles of the variables.⁴² An indicator variable was included for eMM vs paper. This model assumes a constant effect of the intervention throughout the study period and across all clusters. From the models, we estimated incidence rate ratios (IRRs) for immediate post-eMM vs paper. We used similar models without the secular trend to estimate IRRs for one-year post-eMM vs paper. Clinical and procedural errors were modelled separately, with additional models for errors with potential severity score ≥ 3 (ie potential ADE) and for orders involving high-risk drugs. To model rates of individual error types we used the same model specifications used for the main results, but with Poisson instead of negative binomial distribution as there was no overdispersion at this level and negative binomial models failed to converge.

We used a χ² test to assess any difference in the proportion of errors detected by staff in the paper and eMM periods. Data management was performed in SAS version 9.4, and analysis in R version 4.2. Model estimation used glmmTMB⁴³, and model fit was assessed using DHARMa.⁴⁴

Data availability

This study used individual patient health data that cannot be shared without ethical approval. Analysis datasets are stored according to the ethical approval and access can only be provided to researchers who have received approval from the ethics committee.

Code availability

The code used for preparing and analysing the data is available from the corresponding author on reasonable request.

Trial registration and Ethical approval: Australian New Zealand Clinical Trials Registry (ANZCTR) 370325. Registered 17 October 2016; Sydney Children's Hospital Network Human Research Ethics Committee (HREC/15/SCHN/370); Individual patient consent to access retrospective medication and clinical records was waived.

Author contributions

JW, RD, TO, LD, LL, MB, GA, MR, VM, CM, DD designed the study and were investigators on the NHMRC grant awarded to undertake the research. MR, VM, AW, MG, PB, AG, EF, AM, KS provided specialist expertise in medication errors and participated variously in the design of the error categories, review of medication charts, and application of the harm scales. JW, LL and TB designed the analysis strategy. LL and TB undertook the analyses. VM, AW, CM, MG, LD, GA, LW, PG, RD, JW, MR developed the actual harm assessment process and participated in the harm panels. MD, CM, PB, AG, LD provided specific input in relation to the eMM system implementation, use and design features. All authors contributed to multiple project steering committee meeting discussions to interpret the findings and provide clinical and IT system expertise. JW prepared the manuscript which was then reviewed, revised and approved by all authors.

We thank N Board, P Kennedy, J Karnon, C Cowell, CU Lehmann, S Whalen, R Worthington and hospital staff who provided valuable expert input at various stages of the project.

Competing interests

Cheryl McCullagh, Michael Dickinson, Richard Day, Luciano Dalla-Pozza, Geoff Ambler, Peter Barclay, Alan Gardo, and Tracey O’Brien have no competing interests to declare.

Johanna Westbrook and Ling Li are chief investigators, and Draga Barbaric, Magdalena Raban, Melissa Baysari, Virginia Mumford, Peter Gates, Erin Fitzpatrick, Alison Merchant, Timothy badger-Parker, Karla Seaman, and Amanda Woods are associate investigators on the National Health and Medical Research Council (NHMRC) Partnership grant with the Sydney Children's Hospital Network (1094878) which funded the project.

Les White, at the time of the study was on the Sydney Children’s Hospital Network Board and received a regular stipend.

Madlen Gazarian is a steering group member (unpaid role) representing the Royal Australasian College of Physicians with the Australian Digital Health Agency (ADHA) Medicines Safety Program (2017-2020)

Gates PJ, Baysari MT, Gazarian M, et al. Prevalence of medication errors among paediatric inpatients: Systematic review and meta-analysis. Drug Saf 2019;42:13–25.
Maaskant JM, Vermeulen H, Apampa B, et al. Interventions for reducing medication errors in children in hospital. Cochr Database System Rev 2015(3)
Manias E, Kinney S, Cranswick N, et al. Interventions to reduce medication errors in pediatric intensive care. Ann Pharmacother 2014;48:1313–31.
Rinke ML, Bundy DG, Velasquez CA, et al. Interventions to Reduce Pediatric Medication Errors: A Systematic Review. Pediatri 2014;134(2):338–60.
Gates PJ, Hardie R-A, Raban MZ, et al. How effective are electronic medication systems in reducing medication error rates and associated harm among hospital inpatients? A systematic review and meta-analysis. J Am Med Inform Assoc 2020;28(1):167–76.
Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014;23(9):773–80.
Weir CR, Staggers N, Laukert T. Reviewing the impact of computerized provider order entry on clinical outcomes: The quality of systematic reviews. Int J Med Inform 2012;81(4):219–31.
Conn RL, Kearney O, Tully MP, et al. What causes prescribing errors in children? Scoping review. BMJ Open 2019;9(8):e028680.
Gates PJ, Meyerson SA, Baysari MT, et al. The prevalence of dose errors among paediatric patients in hospital wards with and without health information technology: A systematic review and meta-analysis. Drug Saf 2019;42(1):13–25.
Ivanovska V, Rademaker CMA, van Dijk L, et al. Pediatric drug formulations: a review of challenges and progress. Pediatri 2014;134(2):361–72.
Caldwell NA, Power B. The pros and cons of electronic prescribing for children. Arch Dis Child 2012;97(2):124–28.
Dufendach KR, Eichenberger JA, McPheeters ML, et al. Core functionality in pediatric electronic health records. Technical Briefs, No 20. Rockville, MD: Agency for Healthcare Research and Quality, 2015.
Johnson K, Lehmann C. Electronic prescribing in pediatrics: Toward safer and more effective medication management. Pediatri 2013;131:1350–56.
Healthcare Safety Investigation Branch. Weight-based medication errors in children. UK, 2022:107.
Prgomet M, Li L, Niazkhani Z, et al. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis. J Am Med Inform Assoc 2017;24(2):413–22.
Walsh KE, Landrigan CP, Adams WG, et al. Effect of computer order entry on prevention of serious medication errors in hospitalized children. Pediatri 2008;121(3):e421-7.
Tolley CL, Forde NE, Coffey KL, et al. Factors contributing to medication errors made when using computerized order entry in pediatrics: a systematic review. J Am Med Inform Associ 2017;25(5):575–84.
Nuckols T, Smith-Spangler C, Morton S, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. System Rev 2014;3(1):56.
Devin J, Cleary BJ, Cullinan S. The impact of health information technology on prescribing errors in hospitals: a systematic review and behaviour change technique analysis. System Rev 2020;9(1):275.
Owens K. EMR implementation: Big bang or a phased approach? J Med Prac Mngt 2008;23(5):279–81.
Sydney Children's Hospital Network. High Risk Medicine Register, 2015.
Maaskant JM, Vermeulen H, Apampa B, et al. Interventions for reducing medication errors in children in hospital. Cochrane Database Syst Rev 2015(3):CD006208.
Holdsworth MT, Fichtl RE, Raisch DW, et al. Impact of computerized prescriber order entry on the incidence of adverse drug events in pediatric inpatients. Pediatri 2007;120(5):1058–66.
Walsh K, Adams W, Bauchner H, et al. Medication errors related to computerized order entry for children. Pediatri 2006;118:1872–79.
Holdsworth MT, Fichtl RE, Behta M, et al. Incidence and impact of adverse drug events in pediatric inpatients. Arch Pediatr Adolesc Med 2003;157(1):60–65.
Barber N, Franklin BD, Cornford T, et al. Safer, Faster, Better? Evaluating Electronic Prescribing: Report to the Patient Safety Research Programme. London: Policy Research Program, Department of Health, 2006.
Baysari MT, Hardie R-A, Lake R, et al. Longitudinal study of user experiences of a CPOE system in a pediatric hospital. Int J Med Inform 2018;109:5–14.
Westbrook JI, Baysari MT, Li L, et al. The safety of electronic prescribing: manifestations, mechanisms, and rates of system-related errors associated with two commercial systems in hospitals. J Am Med Inform Assoc 2013;20(6):1159–67.
Niazkhani Z, Pirnejad H, Berg M, et al. The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review. Journal of American Medical Informatics Association 2009;16(4):539–49.
Howlett M, Butler E, Lavelle K, et al. The impact of technology on prescribing errors in pediatric intensive care: A before and after study. App Clin Inform 2020;11:323–35.
Doherty C, Mc Donnell C. Tenfold Medication Errors: 5 Years’ Experience at a University-Affiliated Pediatric Hospital. Pediatri 2012;129(5):916–24.
Hou J-Y, Cheng K-J, Bai K-J, et al. The effect of a computerized pediatric dosing decision support system on pediatric dosing errors. J Food Drug Analysis 2013;21(3):286–91.
Gazarian M, Graudins LV. Long-term Reduction in Adverse Drug Events: An Evidence-Based Improvement Model. Pediatri 2012;129(5):e1334-e42.
Westbrook JI, Li L, Raban MZ, et al. Stepped-wedge cluster randomised controlled trial to assess the effectiveness of an electronic medication management system to reduce medication errors, adverse drug events and average length of stay at two paediatric hospitals: a study protocol. BMJ Open 2016;6(10)
Sydney Children's Hospital Network. Accessing eMM training [Available from: https://learning.schn.health.nsw.gov.au/search_results?search_api_fulltext=emm accessed June 2022.
Westbrook J, Reckmann M, Li L, et al. Effects of two commercial electronic prescribing systems on prescribing error rates in hospital inpatients: a before and after study. PLoS Med 2012;9(1):e1001164. doi:10.1371/journal.pmed.64.
National Coordinating Council for Medication Error Reporting and Prevention. NCC MERP Index for categorizing medication errors 2001 [Available from: .
Gates PJ, Baysari MT, Mumford V, et al. Standardising the Classification of Harm Associated with Medication Errors: The Harm Associated with Medication Error Classification (HAMEC). Drug Saf 2019;42(8):931–39.
Uppsala Monitoring Centre. The use of the WHO-UMC system for standardised case causality assessment Sweden: Uppsala Monitoring Centre, 2018.
Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trial 2007;28(2):182–91.
Hemming K, Taljaard M, Forbes A. Analysis of cluster randomised stepped wedge trials with repeated cross-sectional samples. Trials 2017;18(1):101.
Harrell F. Regression Modelling Strategies. Switzerland: Springer 2015.
Brooks M E, Kristensen K, van Benthem K J, et al. glmmTMB Balances Speed and Flexibility Among Packages for Zero-inflated Generalized Linear Mixed Modeling. The R Journal 2017;9(2):378–400.
Residual diagnostics for hierarchical (Multi-level/mixed) regression models [program]. 0.4.3 version, 2021.

There is a conflict of interest Johanna Westbrook and Ling Li are chief investigators, and Draga Barbaric, Magdalena Raban, Melissa Baysari, Virginia Mumford, Peter Gates, Erin Fitzpatrick, Alison Merchant, Timothy badger-Parker, Karla Seaman, and Amanda Woods are associate investigators on the National Health and Medical Research Council (NHMRC) Partnership grant with the Sydney Children's Hospital Network (1094878) which funded the project.

Les White, at the time of the study was on the Sydney Children’s Hospital Network Board and received a regular stipend.

Madlen Gazarian is a steering group member (unpaid role) representing the Royal Australasian College of Physicians with the Australian Digital Health Agency (ADHA) Medicines Safety Program (2017-2020)

Download PDF

Journal Publication

published 13 Dec, 2022

Read the published version in npj Digital Medicine →

Editorial decision: revise
17 Oct, 2022
Review #2 received at journal
16 Oct, 2022
Review #1 received at journal
05 Oct, 2022
Reviewer #2 agreed at journal
05 Oct, 2022
Reviewer #1 agreed at journal
27 Sep, 2022
Reviewers invited by journal
26 Sep, 2022
Editor assigned by journal
26 Sep, 2022
Submission checks completed at journal
26 Sep, 2022
Unknown event
23 Sep, 2022
First submitted to journal
20 Sep, 2022

You are reading this latest preprint version

Immediate and long-term effects of an electronic medication management system on paediatric prescribing errors: A stepped-wedge cluster randomised controlled trial with one-year follow-up

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Results

Characteristics of study cohorts

Changes in prescribing error rates in the immediate and one-year post-eMM periods

Changes in types of clinical prescribing errors in the immediate and One-year post-eMM periods

Potential ADEs and evidence of error detection by staff

Actual harm associated with clinical prescribing errors in the paper and immediate post-eMM periods

Discussion

Effectiveness of eMM in paediatrics to reduce prescribing errors

Future optimisation of eMM

Harm associated with clinical prescribing errors

Strengths and limitations

Conclusion

Methods

Study design and setting

The control and intervention

Prescribing error data collection

Training and inter-rater reliability assessment

Assessing actual harm associated with clinical prescribing errors

Sample size calculations

Analysis

Declarations

Data availability

Code availability

Author contributions

Competing interests

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1