This study was approved by the Research Ethics Board of the CHU of Québec – Université Laval.
Study design, setting, and participants
We conducted a retrospective cohort study using data we had already collected in three academic centers in the province of Québec, Canada. The research program for whom data were primary collected aimed at evaluating the risks and benefits associated with the use of a single dose of activated charcoal in patients presenting to the ED in the first twelve hours after the ingestion of a potentially toxic dose of a carbo-adsorbable substance (both adults and children were involved). We applied four exclusion criteria for the initial data collection: (1) ingestion > 12 hours at arrival in the ED or unknown time of ingestion, (2) route of exposure other than ingestion, (3) the ingested substance was not carbo-adsorbable or was a corrosive agent [15], (4) a method of gastrointestinal decontamination other than oral activated charcoal was used (e.g., gastric lavage or whole bowel irrigation). For this present retrospective multicenter cohort study, we used data from the pediatric cases of this poisoned cohort. We included all children aged 0 to 17 years who presented to the ED between January 2013 and December 2016 with acute intoxication following ingestion of a potentially toxic dose of a carbo-adsorbable substance. No additional exclusion criterion was applied.
Data collection
We traced patient records in the archives of the health care institutions using ICD-10 codes related to intoxications with cross-referencing to data archived at the local poison control center. Data were extracted from electronic or paper medical records using a standardized extraction grid by trained abstractors.
Poisoning severity scores
The PSS was measured using clinical data of the poisoned cases to define severity according to five categories: 0 for asymptomatic, levels 1 to 3, respectively, for minor, moderate, and severe intoxication, and 4 for death [13]. Biological and hemodynamic parameters of the patients were used to calculate the PELODS, between 0 and 71 points. Three levels of severity were then applied for the PELODS according to literature: 1 for low severity (<10 points), 2 for moderate severity (10-19 points), and 3 for high severity (≥20 points) [9-12]. We measured both scores every 12 hours during the first 72 hours of hospitalization or until discharge, depending on the first occurring event. Data abstractors were asked to assess a minimal score of PSS and PELODS for each time according to data collected in the medical record, along with a maximal PSS and PELODS under the hypothesis of missing information in the charts. We assumed that the minimal scores were the most representative of the severity of poisoned children and used them in the main analyses under the assumption that clinicians reported significant events and symptoms in the charts. Two different measures were used for our study: the initial PSS (PSS(t0)) and the 12-hour delta PSS (∆PSS = PSS(t0) - PSS(t12)). For comparison purposes, we measured the same parameters with the PELODS. Two assessors independently reviewed all case records from one hospital facility (14% of the study population) to assess the reliability of the PSS measures.
Outcomes
Our primary outcome was hospital admission after visiting the ED. Our secondary outcomes were the need for vasopressors, need for extra renal replacement therapy, use of mechanical ventilation, admission to the intensive care unit (ICU), hospital length of stay (LOS) ≥ 12 hours (ED stay included), discharge status, and death.
Statistical analyses
Given the low rate of deaths and severely affected subjects, we aggregated levels 2 and 3 of the PSS to obtain a 3-category PSS(t0) in our analyses. We categorized the ∆PSS and the ∆PELODS as decreasing, stabilizing, or increasing scores between t0 and t12.
To address our first objective of assessing the predictive validity of the PSS, we evaluated its discrimination and internal calibration to predict our outcomes of interest. First, we performed a Receiving Operating Curve (ROC) analysis with the PSS(t0) or the ∆PSS. We used the Area Under the Receiving Operating Curve (AUC) and its 95% confidence interval (95% CI) to assess the internal discriminatory ability of each score [16,17]. We interpreted the AUCs according to the thresholds preferred in clinical epidemiology: poor discrimination if <0.70, acceptable between 0.70 and 0.79, excellent between 0.80 and 0.89, and outstanding if ≥ 0.90 [18]. Then, the plot of predicted versus observed values for each model allowed a graphical analysis of the internal calibration of each score. Spiegelhalter's goodness-of-fit test for sparse data was also performed [16,17,19,20].
To address our second objective, we compared the predictive validity of the PSS to that of the PELODS, first by repeating the ROC analysis using the PELODS(t0) and the ∆PELODS instead of the PSS, and then independently of the other potential predictors of the outcomes studied. For this second step, we added to our logistic models seven potential predictors previously identified using a directed acyclic graph constructed from the literature and validated by experts: sex, age group, presence of comorbidities, ingestion of ≥ 2 toxic substances, poisoning mechanism, the delay between the intoxication and arrival at the ED, and the ED physician's main specialty. We compared the ROC curves of the PSS and the PELODS in all models, with and without the other predictors, using the DeLong test for paired data [21], and we graphically assessed the calibration goodness-of-fit [19]. We performed the Hosmer and Lemeshow test on adjusted models [19,22].
We measured the inter-rater reliability for our third objective by calculating the Cicchetti-Allison weighted Kappa coefficients for the PSS at each measurement time [23]. The level of agreement was considered moderate between 60 and 79%, high between 80 and 90%, and almost perfect if > 90% [24].
Subgroup and sensitivity analyses
We performed a subgroup analysis for children aged 0 to 5 years to investigate the internal validity of the PSS in these children who are particularly vulnerable to accidental poisonings [2,4]. We performed sensitivity ROC analyses by excluding subjects transferred from another hospital, subjects who left the ED against medical advice, and subjects treated with antidotes or activated charcoal. We also performed a sensitivity analysis by using the mean PSS and PELODS values at ED arrival and after 12 hours in each model, instead of the minimum values. These mean values consisted of an average between the minimum and the maximum score for each time. We equally performed a complete case analysis for comparison between the two poisoning scores.
Sample size
To be able to detect a difference ≥ 1.5% between the AUCs of PSS and PELODS using the DeLong test for paired data, we calculated that we needed to include from 442 to 454 children assuming that respectively 25% to 20% of the children presented the clinical outcome of interest (type I error of 5% and power of 80%).
Treatment of missing data
Data on the PSS and PELODS scores at 12 hours were missing for 280 unadmitted participants (59.7%) discharged before the 12-hour assessment and one admitted child (0.002%). Considering that the 12-hour missing data could be directly explained by baseline information and discharge status, the mechanism of Missingness At Random (MAR) was plausible [25]. We simulated these missing data using multiple imputations [25], with the Multiple Imputations by Chained Equations (MICE) method [25,26]. We entered all independent variables and the potential baseline predictors of the missing values in the imputation models and simulated 60 imputations for each missing value [25]. We performed all statistical analyses using SAS software v9.4 (SAS Institute Inc., Cary, NC, USA). A type I error of 5% was considered.