Use of Run Charts and Statistical Process Control Charts to Monitor Quality Indicators in Adverse Drug Reactions Reporting: An Australian Health Service Network Experience During the COVID-19 Pandemic

doi:10.21203/rs.3.rs-1943048/v1

Download PDF

Research Article

Use of Run Charts and Statistical Process Control Charts to Monitor Quality Indicators in Adverse Drug Reactions Reporting: An Australian Health Service Network Experience During the COVID-19 Pandemic

https://doi.org/10.21203/rs.3.rs-1943048/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Purpose

The utility of run charts and statistical process control (SPC) charts in monitoring adverse drug reaction (ADR) management processes has not been well-described. In this retrospective observational cohort study, we explored the use of these tools to evaluate process variations over time in the quantity and quality of ADR reporting at a quaternary health institution, amidst the national Coronavirus-19 (COVID-19) vaccination program roll-out.

Methods

All submitted ADR reports to the institutional review committee between 1st January and 31st December 2021 were evaluated with regards to the total number of reports, reporter characteristics and quality, using locally validated ADR Reports Quality Algorithm (AQUA-12).

Results

Of 271 reports submitted, 187(69%) were by hospital pharmacists and > 30% concerned severe reactions. The median AQUA-12 score was 10 (IQR: 9–12), and 26.6% of the reports were of low-quality. Using run charts, significant increases in the number of ADR reports submitted were detected during March and June 2021 periods, corresponding to the roll-out of Phases 1 and 2a of the COVID-19 vaccination program. These signals of change were driven mainly by COVID-19 vaccine-related reports. Using SPC charts, corresponding increase in the proportion of ADR reports submitted by nurses, with an inverse decrease in that submitted by pharmacists, was detected for the same periods. Despite these variations, SPC charts indicated that the proportion of ADR reports of severe nature and the quality of reports remained stable.

Conclusions

This study demonstrated that the overall state of ADR reporting processes in hospitals can be monitored using suitable quality indicators and tools.

Quality improvement

run charts

statistical process control charts

quality indicators

adverse drug reactions

COVID-19 pandemic

Adverse drug reactions (ADR) cause significant morbidity and mortality and contribute a substantial economic burden to the healthcare system [1–4]. Yet a significant proportion of ADRs are preventable through appropriate assessment and reporting [4].

It is recognised that there are large variations in management of ADRs [5]. Unintended variations in ADR management processes may compromise patient safety and diminish the quality of care. To date, quality indicators that can be used to monitor ADR management processes in hospitals remain ill-defined [5]. Two parameters that have been used as surrogate indicators: the quantity of reports, and the quality of information included in the report. While the quantity of reports is easily measured, assessing the quality of information is more difficult. Quality refers to a minimum dataset is required to determine medication reaction causality [6]. An incomplete or a low-quality report has the potential to undermine the strength of association between a medication and a reaction.

Run charts and statistical process control (SPC) charts are tools used in quality improvement initiatives [7]. Intended or unintended variations within a system are evaluated by use of run charts and SPC charts. Systematic monitoring of quantity and quality parameters for ADR management processes with individual hospitals allow identification of opportunities to further learn, understand and improve those processes. However, there has been little research describing the use of these methods in ADR management, with only a few studies demonstrating their role in a tertiary hospital setting [8, 9].

In the present study, we explored the utility of run charts and SPC charts using quality indicator data to monitor the overall state and variations over time of an ADR management system. Additionally, we evaluated if changes occurring within the broader health system during the Coronavirus-19 (COVID-19) pandemic, involving the national vaccination program, had any influence on the local ADR management processes. Secondary process measures included the proportion of low-quality ADR reports and individual data elements in reports that were incomplete or missing.

Setting

This retrospective observational study was conducted at Alfred Health in Melbourne, Australia. Alfred Health is a university-affiliated health service network inclusive of a quaternary institution that provides specialist clinical services in solid organ and bone marrow transplantation, human immunodeficiency virus (HIV) infection, cystic fibrosis, haemophilia, trauma, and burns. The Adverse Drug Reaction Review Committee (ADRRC) is a multidisciplinary group consisting of senior pharmacists in medication safety and medicines information, and specialist physicians in allergy/immunology, dermatology, clinical pharmacology, infectious diseases, and internal medicine, who meet every two weeks to discuss ADR reports and provide recommendations regarding safe medication management. Around 250–300 ADR episodes per year (an average of 20 to 25 reports per month) are reported to ADRRC by hospital staff, including doctors, pharmacists, and nurses. Reporting system is spontaneous and predominantly via an electronic form embedded in the electronic medical record (EMR).

Data Source and Variables

All reports submitted to the ADRRC between 1st January and 31st December 2021 were included in this study. Relevant information was extracted from each ADR report to determine the vocation of reporter, nature and severity of reaction and medications implicated. The nature of each reaction was classified based on our previous publication, for consistency [10]. The quality of each ADR report was assessed using Adverse Drug Reactions Reports Quality Algorithm (AQUA-12). AQUA-12 was initially developed at the study institution by ADRRC members, to assess the quality of ADR reports [11]. The performance of AQUA-12 as a quality monitoring tool was previously reported [11]. The data domains assessed in AQUA-12 included: (i) previous ADR history, (ii) diagnosis or description of actual ADR, (iii) description of key events concerning ADR (i.e., the narrative), (iv) list of suspected medications, (v) consideration of medication timeline relevant to the nature of ADR, (vi) management of ADR episode, and (vii) outcome/sequelae. The rationale behind inclusion of each data element was previously described [11].

Report Characteristics and Process Measures

Quality indicators representing the ADR management processes included the total number of ADR reports, report/reporter characteristics and quality of reports. The following characteristics of ADR reports were evaluated: reporter types, severity of reaction, type of reaction, overall quality of reports as determined by AQUA-12 scores, and individual data elements that were incomplete or missing.

Using run charts, the following processes were monitored over twelve consecutive months: number of (i) ADR reports, (ii) reports involving COVID-19 vaccines, and (iii) reports involving antimicrobials. Further, to understand the underlying reasons for observed variations over time, SPC charts were used to evaluate proportion of reports: (i) submitted by healthcare professionals of specific vocations, (ii) high severity rating, and (iii) that were low-quality, as defined by AQUA-12 score < 10.

Data analysis

Summary statistics for discrete variables are presented as counts and proportions and continuous variables as median and interquartile ranges (IQR). Trends, variations, and control limits over time for process measures of interest were analysed and interpreted using quality improvement research methodology as described by Provost and Murray [7, 12]. Specifically, run charts and statistical process control (P-control charts) were utilised in this study, and the results were presented using median values for run charts, and mean values with 3 sigma values as upper and lower control limits for SPC charts. Statistically significant signals or changes in process measures over time were identified using published run chart and SPC chart rules (Appendix) [7, 12]. Assuming the average proportion of low-quality reports is 25%, and the common guidelines requirement of n > 300/p is met, the minimum subgroup size was determined at 12. Statistical tests were performed using SPSS version 28 (IBM Corporation, Armonk, NY, USA) and QI-Charts version 2.0 (Scoville Associates, Institute for Healthcare Improvement, Boston, MA, USA).

Ethics approval

Approval to conduct this study as a low-risk research project was granted by the Alfred Health human research and ethics committee (project number: 317/22)

A total of 271 ADR reports were included in the analysis. The overall characteristics of ADR reports received by ADRRC in 2021 are summarised in Table 1. Majority of reports (187, 69%) were submitted by pharmacists and more than 30% were rated at least severe. Hypersensitivity reactions constituted approximately two-thirds of reports (18.1% immediate and 48.7% delayed hypersensitivity reactions). The median AQUA-12 score was 10 (IQR: 9–12) for all reports, with 72 (26.6%) reports scoring as low-quality. the main data elements that were frequently incomplete or missing were the description of the index reaction (42.8%) and of key events concerning the index reaction (i.e. ADR narrative) (46.1%). A run chart for the number of ADR reports received by ADRRC over the 12-month period is shown in Fig. 1. Notably there was a significant increase in the number of ADR reports in March 2021 (N = 44, the ‘astronomical point’ rule for run charts [7, 12]), which coincided with the introduction of Phase 1 of the national COVID-19 vaccination program by the Government of Australia [13]. No other signals of change in the process were observed for Phases 2a and 2b of the vaccination program.

Table 1

Characteristics of ADR reports received by ADRRC in year 2021 (n = 271)
Characteristics	Frequency, n (%)
Reported by: Pharmacists Nurses Doctors Others	187 (69.0) 43 (15.9) 34 (12.5) 7 (2.6)
Severity of reaction: Mild Moderate Severe Life-threatening Fatal	67 (24.7) 111 (41.0) 57 (21.0) 33 (12.2) 3 (1.1)
Type of reaction: Immediate hypersensitivity Delayed hypersensitivity, non-SCAR Delayed hypersensitivity, single organ Delayed hypersensitivity, SCAR Delayed hypersensitivity, other Non-immunological Others	49 (18.1) 44 (16.2) 36 (13.3) 20 (7.4) 32 (11.8) 25 (9.2) 65 (24.0)
Quality of reports (based on AQUA-12): Excellent (Score 12) Good (Score 10–11) Moderate (Score 8–9) Poor (Score ≤ 7) High-quality (Score ≥ 10) Low-quality (Score < 10)	76 (28.0) 123 (45.4) 50 (18.5) 22 (8.1) 199 (73.4) 72 (26.6)
Data elements not or only partially completed (based on AQUA-12): Previous ADR history Actual reaction Description of key events Suspected drugs Timeline relevant to ADR Management of reaction Outcome/sequelae	72 (26.6) 116 (42.8) 125 (46.1) 60 (22.1) 18 (6.6) 12 (4.4) 22 (8.1)
ADR: Adverse drug reactions, AQUA-12: Adverse drug reaction reports quality algorithm, SCAR: Severe cutaneous adverse drug reactions

To confirm that these signals observed in ADR reporting processes were indeed driven mainly by COVID-19 vaccine reports, a run chart for the total number of COVID-19 vaccine-related reports (Fig. 2A) was constructed. The run chart confirmed corresponding increase in the number of vaccine-related reports received by ADRRC during Phase 1 of the vaccination program (N = 16, March 2021). Additionally, there was also an increase in the number of vaccine-related reports when Phase 2a was introduced (N = 19, June 2021). Both occasions represented significant signals of change in reporting processes by ‘astronomical point’ rule [12]. However, the reporting processes returned to a stable state by July 2021.

Interestingly, there was also an increase in the number of antimicrobial-related reports after Phase 1 vaccination (N = 21, March 2021). However, this was followed by a significant decrease in the number of reports over the following months till August 2021 (‘trend’ rule for run chart[7, 12]) (Fig. 2B).

To explore the underlying factors possibly explaining signals of change detected on run charts, P-control charts for the proportion of ADR reports submitted according to reporter vocations were constructed (Figs. 3A and 3B). An increase in the proportion of ADR reports submitted by nurses during Phases 1 and 2a of COVID-19 vaccination program (from February to July 2021), with inverse decrease in the proportion of reports submitted by hospital pharmacists (Figs. 3A and 3B) were observed on P-control charts.

Despite these variations in reporting processes, the proportion of ADR reports that were of at least severe category, and the quality of ADR reports remained stable throughout the study period (Figs. 4 and 5), indicating that observed variations did not have an influence on the overall quality of reports.

This study demonstrated that it is possible to monitor the state of ADR reporting system in a hospital setting and detect signals of change in the processes by use of run charts and SPC charts. We found that external factors, such as roll out of the national COVID-19 vaccination program, resulted in variations in ADR reporting processes at our institution. Nevertheless, these changes were transient, and other factors intrinsic to the ADR reporting processes, such as severity and quality of ADR reports, remained uninfluenced by external factors.

Very few studies have demonstrated the utility of run charts or SPC charts for monitoring ADR management systems and quality improvement purposes [8, 9]. In quality improvement research methodology, a ‘family’ of measures are usually taken into consideration to understand the overall state of a system [7]. For this study, we selected a combination of quality indicators, including the total number of reports, the nature and characteristics of reports (i.e., severity, implicated medications), reporter types, and quality of reports, to provide a holistic understanding of the ADR management system at our institution. We chose these parameters as they have been widely studied and are known to display large variations in different healthcare settings [5]. Other quality indicators, such as incidence rates, proportion of preventable ADRs, total time taken to assess and report an episode of ADR, proportion of reports referred to specialist clinics for further assessment, or the frequency and quality of information provided to patients/carers and other healthcare providers regarding index ADR episode, would also constitute as equally valid measures of overall quality of ADR management processes [5, 8, 14]. However, these are beyond the scope of current study.

The Government of Australia embarked on the national COVID-19 vaccination program in early 2021 [13]. Phase 1a was rolled out on the 21st of February to vaccinate front-line health care workers, selected group of aged care and disability care staff and residents. This was closely followed by Phase 1b within a month to include all elderly aged ≥ 70 years, other healthcare workers and at-risk group of patients. The eligible age group was extended to adults and elderly aged 40–69 years in Phase 2a in May 2021, along with the recommendation to reserve Pfizer-BioNTech vaccine to those under 50 years of age, due to reported risk of vaccine induced thrombotic thrombocytopenia (VITT) with AstraZeneca vaccine in younger population [15]. From 30th August onwards, Pfizer-BioNTech vaccine was made available all Australian residents aged 16–39 years (Phase 2b). In response to the government's directives, our institution commenced staff vaccination program, as well as vaccination of aged care residents at affiliated care facilities in February 2021, and this continued till later in the year to include the general public as progressive stages of vaccination program were implemented. To meet the demands of the vaccination program efficiently, qualified nurses were employed at vaccination clinics, on an ad-hoc basis, within the study institution.

The impact of the national COVID-19 vaccination program on local ADR reporting processes was evident in the variations observed on run charts. Signals of change were detected on the run chart for the total number of reports received by ADRRC, along with corresponding increases in the number of COVID-19 vaccines-related reports, which correlated with the Phases 1 and 2a of the COVID-19 vaccination program. These increases in reporting rates were not surprising as prior studies showed that serious known, or any unknown reactions to any new medicinal product carried the highest probability of being reported, whereas once the clinicians became familiar with the product and side effects/adverse reactions became more well-known, or too trivial, this probability decreased [16]. This phenomenon was clearly demonstrated on the run chart where observed changes in reporting rates appeared to be transient with the rates returning to the baseline level in later period of the vaccination program (on average 20–25 reports per month are received by ADRRC from previous study [10]). Additionally, in the later phases of the vaccination program, the vaccination clinic nurses were recommended to report the adverse reactions directly to the state vaccine centre, the Victorian Specialist Immunisation Services (VicSIS), instead of via ADRRC, which may also have contributed to the decrease in numbers of reports at our institution [17]. As antimicrobials were known to be implicated in up to 50% of suspected medication reactions [10], we also analysed that trends in antimicrobial-related reports. Interestingly, we also noted concurrent increase the number of antimicrobial-related reports following Phase 1a, with subsequent decrease in the months that followed. The reasons behind this observation are less obvious, perhaps driven by possible increase in antimicrobial utilisation during that particular time period, and may warrant further exploration.

Compared to our previous study, there was a higher proportion of nurses reporting ADRs in current study (15.9% vs 5.7%), with nurse-initiated reports concerning COVID-19 vaccines mostly originating from the vaccination clinics [10]. Vaccine reactions are often immediate, thus, they were more likely to be reported by the nurses who administered them, than by the hospital pharmacists. Despite this, the proportion of reports that were of at least severe in nature and the proportion of low-quality reports remained stable throughout the study period. This implied that nurses working in the vaccination clinics, many of whom were part of the transient workforce, were able to competently assess vaccine-related reactions and that they were also familiar with the ADR reporting requirements at our institution.

Several tools exist to assess the quality and completeness of ADR reports, however, almost all are designed for research setting. For the purposes of quality improvement initiatives in hospital settings, processes and outcomes need to be measured in a frequent manner to rapidly detect changes in trends, variations, and control limits over time [5, 7, 11]. To suit this purpose, AQUA-12 was developed to enable regular monitoring of the quality of ADR reports at our institution [11]. AQUA-12 was robustly tested by independent assessors from multiple specialties and yielded an excellent inter-rater reliability score. The tool assesses the practical application of knowledge and skills of the reporting healthcare professionals through completeness of information provided in ADR reports. Data elements in AQUA-12 are based on ADR principles and closely reflect the information required in the hospital ADR report form, which adheres to the reporting requirements by the Therapeutic Goods Administration, the national pharmacovigilance authority in Australia [18]. The ease of application of AQUA-12 also provided a distinct advantage, compared to other published methods of quality assessment for ADR reports. The present study found a similar rate of low-quality reports (26.6%) compared to the initial evaluative study of AQUA-12 (25.7%) [11], and over time, this rate was unaffected by the vaccination program. The proportion of individual data elements that were incomplete or omitted were also similar. The quality of ADR reports is thus a parameter which is more likely to be intrinsically influenced by the healthcare providers’ competency in ADR management, rather than external processes. Therefore, attempts at improving this parameter may require interventions that are directed at healthcare providers’ knowledge and skills in ADR management, for instance, through comprehensive education programs or medication safety programs [9].

The main limitation of this study was that we did not evaluate processes and/or outcomes relevant to patients’ ADR experience, such as the rate and quality of information provided by the clinicians upon discharge, as part of the ‘family of measures’ to reflect the overall quality of ADR management processes. To date, studies exploring this very important aspect of ADR management have been limited and further quality improvement initiatives are urgently needed in this area. Additionally, COVID-19 has brought about unprecedented challenges and system changes to every healthcare institution around the world. Often these changes took place at a very rapid pace to battle each rising wave of COVID-19 infections during the pandemic response. As such, there might have been other institutional or external factors that could have affected the observed trends in ADR management processes, other than the national vaccination program. Nevertheless, this would not change the key message in this study that a combination of several quality indicators can be used to reliably monitor the state of ADR reporting processes in a hospital setting.

This study demonstrated that trends and variations in ADR management processes in hospitals can be closely monitored using a combination of indicators that include the quantity of reports, the characteristics/nature of reactions reported, reporter types and quality assessment, to reflect the overall state of the system. By using quality improvement research methodology, consisting of run charts and statistical process control charts, unintended variations due to intrinsic or extrinsic influences can be easily detected. Further, such tools may prove useful in evaluating improvements from certain quality initiatives, such as a multidisciplinary education program.

Funding: AKA receives salary from the Alfred Health for this sabbatical leave project

Conflict of Interest: The authors declare no competing interests

Data Availability Statement: Data sharing is not available

Authors’ Contributions:

AKA: Conception, study design, data collection, data analysis and interpretation, writing up of first draft, revision, and approval of final manuscript

LVG: Conception, study design, data collection, data analysis and interpretation, writing up of first draft, revision, and approval of final manuscript

JIL: Conception, study design, data analysis and interpretation, revision, and approval of final manuscript

Ethics Approval: Approval to conduct this study as a low-risk research project was granted by the Alfred health human research and ethics committee (project number: 317/22)

Consent to participate: Not applicable

Consent for publication: Not applicable

Baiardini I, Gaeta F, Molinengo G, Braido F, Canonica G, Romano A (2015) Quality-of‐life issues in survivors to anaphylactic reactions to drugs. Allergy 70 (7): 877–879
Lorimer S, Cox A, Langford N (2012) A patient’s perspective: the impact of adverse drug reactions on patients and their views on reporting. Journal of clinical pharmacy and therapeutics 37 (2): 148–152
Impicciatore P, Choonara I, Clarkson A, Provasi D, Pandolfini C, Bonati M (2001) Incidence of adverse drug reactions in paediatric in/out-patients: a systematic review and meta‐analysis of prospective studies. British journal of clinical pharmacology 52 (1): 77–83
Roughead EE, Semple SJ, Rosenfeld E (2016) The extent of medication errors and adverse drug reactions throughout the patient journey in acute care in Australia. International journal of evidence-based healthcare 14 (3–4): 113–122
Aung AK, Walker S, Khu YL, Tang MJ, Lee JI, Graudins LV (2022) Adverse drug reaction management in hospital settings: review on practice variations, quality indicators and education focus. European Journal of Clinical Pharmacology: 1–11
Bergvall T, Norén GN, Lindquist M (2014) vigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues. Drug safety 37 (1): 65–77
Provost LP, Murray S (2011) The health care data guide: learning from data for improvement. John Wiley & Sons.
Schumock G, Seeger J, Kong S (1995) Control charts to monitor rates of adverse drug reactions. Hospital pharmacy 30 (12): 1088, 1091 – 1082, 1095
Cohen M, Kimmel N, Benage M, Cox M, Sanders N, Spence D, Chen J (2005) Medication safety program reduces adverse drug events in a community hospital. BMJ Quality & Safety 14 (3): 169–174
Aung AK, Tang MJ, Adler NR, de Menezes SL, Goh MSY, Tee HW, Trubiano JA, Puy R, Zubrinich CM, Graudins LV (2018) Adverse drug reactions reported by healthcare professionals: reaction characteristics and time to reporting. The Journal of Clinical Pharmacology 58 (10): 1332–1339
Aung AK, Zubrinich CM, Goh MSY, Snyder B, Tang MJ, Khu YL, Lee JI, Graudins L Development and application of Adverse Drug Reactions Reports Quality Algorithm (AQUA-12) score: A single-centre quality improvement initiative. Unpublished manuscript
Perla RJ, Provost LP, Murray SK (2011) The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ quality & safety 20 (1): 46–51
COVID-19 vaccination in Australia. In: ed.
Mazzoni D, Tee HW, de Menezes SL, Graudins LV, Johnson DF, Newnham ED, Kelley PG, Zubrinich CM, Goh MSY, Trubiano JA (2020) A Survey on Knowledge Gaps in Assessment and Management of Severe Drug Hypersensitivity Reactions: Multicenter Cross-Sectional Study of Australian Health Care Providers. The Journal of Clinical Pharmacology
Franchini M, Liumbruno GM, Pezzo M (2021) COVID-19 Vaccine‐associated Immune Thrombosis and Thrombocytopenia (VITT): diagnostic and therapeutic recommendations for a new syndrome. European Journal of Haematology 107 (2): 173–180
Hasford J, Goettler M, Munter K-H, Müller-Oerlinghausen B (2002) Physicians' knowledge and attitudes regarding the spontaneous reporting system for adverse drug reactions. Journal of clinical epidemiology 55 (9): 945–950
Gordon SF, Virah Sawmy E, Duckworth E, Wolthuizen M, Clothier HJ, Chea M, Tenneti N, Blow N, Buttery JP, de Luca J (2022) Victorian Specialist Immunisation Services (VicSIS)–bolstering adult clinics for COVID-19 vaccines. Human Vaccines & Immunotherapeutics: 1–6
Department of Health TGA Reporting adverse events. In: ed.

No competing interests reported.

APPENDIX.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Use of Run Charts and Statistical Process Control Charts to Monitor Quality Indicators in Adverse Drug Reactions Reporting: An Australian Health Service Network Experience During the COVID-19 Pandemic

Status:

Version 1

Abstract

Purpose

Methods

Results

Conclusions

Figures

Introduction

Methods

Setting

Data Source and Variables

Report Characteristics and Process Measures

Data analysis

Ethics approval

Results

Discussion

Conclusions

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1