DOI: https://doi.org/10.21203/rs.2.20595/v1
Background
Out-of-hospital cardiac arrest (OHCA) is a global medical problem. The newly-developed simplified out-of-hospital cardiac arrest (sOHCA) and cardiac arrest hospital prognosis (sCAHP) scores used for prognostication of patients admitted alive have not been validated externally. This study was, thus, conducted to externally validate sOHCA and sCAHP scores in a Japanese population.
Methods
Adult patients resuscitated and admitted to hospitals after intrinsic OHCA (n=2,428, age ≥18 years) were selected from a prospectively collected Japanese database (January 2012–March 2013). We validated sOHCA and sCAHP scores with reference to the original ones in predicting 1-month unfavourable neurological outcomes based on discrimination and calibration measures. Discrimination and calibration were assessed using area under the receiver operating characteristic curve (AUC) and the Hosmer-Lemeshow goodness-of-fit test with calibration plot, respectively.
Results
One-month unfavourable neurological outcome was observed in 82% of patients. Score availability was significantly higher in the simplified scores than in the original ones and was highest in the sCAHP score (76%). The AUCs of simplified scores were not significantly different from those of original ones, whereas the AUC of the sCAHP score was significantly higher than that of the sOHCA score (0.88 vs. 0.81, P <0.001). Goodness-of-fit was poor in the sOHCA score (ν= 8, χ 2 =19.1, Hosmer-Lemeshow test: P =0.014) but not in the sCAHP score (ν= 8, χ 2 =13.5, Hosmer-Lemeshow test: P =0.10).
Conclusion
Performance of original and simplified OHCA and CAHP scores in predicting neurological outcomes in successfully resuscitated OHCA patients were acceptable. Based on the highest availability, similar discrimination, and good calibration, the sCAHP score was the better candidate for clinical implementation. The validated predictive score can help patients’ families, healthcare providers, and researchers by accurately stratifying patients.
Out-of-hospital cardiac arrest (OHCA) is a global medical problem1,2. The overall incidence of OHCA is 80–110 cases per 100,000 population, with approximately 275,000 cases in Europe, 350,000 cases in the US, and 110,000 cases in Japan each year3 − 5. Outcomes after OHCA have improved because of improvements in cardiopulmonary resuscitation (CPR) and treatment strategies. However, even in those who have had successful return of spontaneous circulation (ROSC) and been admitted to hospital for further treatment, outcomes after OHCA remain poor because of post-cardiac arrest syndrome.
Post-cardiac arrest patients often require long-term and expensive life support, resulting in a healthcare burden on families and caregivers. Therefore, prognostication of patients admitted alive is important, as it aids in identifying patients who will benefit most from intensive care, allocating healthcare resources, and counselling patients’ families. OHCA6 and cardiac arrest hospital prognosis (CAHP) scores7 were developed to predict favourable neurological outcomes of patients with ROSC who were admitted to hospital, using variables recorded immediately after admission. The OHCA score is based on measures, including the presence of a shockable rhythm, the no-flow interval (time from collapse to initiation of CPR), the low-flow interval (time from CPR to ROSC), the serum creatinine level, and the arterial lactate level. The CAHP score is based on measures, such as the presence of a shockable rhythm, no-flow interval, low-flow interval, total dose of adrenaline (epinephrine) required during CPR, patient's age, arrest setting, and arterial pH.
Both the OHCA and CAHP scores have been validated6 − 9. However, the use of the no-flow interval limits their clinical application, because estimating the time of collapse is quite difficult in unwitnessed OHCA, which account for up to 40% of OHCA10,11; furthermore, the Utstein template no longer requires no-flow time recording. To address these limitations, Wang et al.12 developed the simplified OHCA (sOHCA) score and simplified CAHP (sCAHP) score, in which no-flow time was removed from the measures. They reported similar and excellent accuracy of these scores compared to those of the original ones in predicting neurological outcomes in successfully resuscitated OHCA patients.
External validation is an important step before widespread implementation in clinical routine. However, to our knowledge, no study to date has externally validated the sOHCA and sCAHP scores. Since the sOHCA and sCAHP scores were developed based on a single-centre observational study, validation of these predictive scores based on a distinct multi-centre dataset was warranted. We therefore aimed to externally validate these predictive scores using the Japanese OHCA registry.
This retrospective observational study evaluated data from the 2012 Survey of Survivors after Out-of-Hospital Cardiac Arrest in the Kanto Region (SOS-KANTO), which was a prospective multicentre observational study in the Kanto Region of Japan that was performed with the support of the Kanto Regional Group of the Japanese Association for Acute Medicine.13 Data from adult OHCA patients (≥ 18 years old) who were admitted to hospital after successful ROSC were analysed, although cases were excluded if they involved: (1) extrinsic aetiology (trauma, burn, hypothermia, hanging, drowning, asphyxiation, or drug overdose); or (2) missing data regarding 1-month outcomes after cardiac arrest.
The study’s retrospective protocol was approved by the institutional review board of the Tokyo Metropolitan Bokutoh Hospital (approval number: 31–041). All data were de-identified prior to the analysis, and the need for informed consent was waived.
The SOS-KANTO study included 16,452 cardiac arrest patients from 67 emergency medical centres between January 2012 and March 2013.13 The emergency medical services (EMS) personnel collected data, including age, sex, bystander-initiated CPR with or without automated external defibrillator (AED) use, cardiac arrest location, witnessed status, initial cardiac rhythm, various time points (e.g., collapse, initiation of CPR, and ROSC), intravenous adrenaline use, and 1-month neurological outcomes, using a form based on the Utstein reporting guidelines. The institutional researchers collected relevant in-hospital information, and neurological outcomes were reported using the cerebral performance category (CPC) scale14: 1 (good cerebral performance), 2 (moderate cerebral disability), 3 (severe cerebral disability), 4 (coma or vegetative state), and 5 (death). Cardiac arrest was defined as the cessation of cardiac mechanical activities, confirmed by the absence of pulse and normal breathing. Physicians caring for the patients determined the aetiology of the cardiac arrest.
The original OHCA, sOHCA, original CAHP, and sCAHP scores were calculated based on previous studies6,7,12 (see additional file 1). The no-flow interval was defined as the time from collapse to initiation of CPR, and the low-flow interval as the time from initiation of CPR to ROSC. The primary outcome was 1-month unfavourable neurological outcome, defined as CPC 3–5.
For descriptive statistics, numeric variables are presented as medians with interquartile ranges (IQRs) and categorical variables as counts and percentages. All statistical analyses were performed using EZR (Saitama Medical Center, Jichi Medical University, Saitama, Japan)15, a graphical user interface for R (The R Foundation for Statistical Computing, Vienna, Austria)16 with add-on packages of “PredictABEL” for creating calibration plots and “mice” for multiple imputations. Predictive scores were estimated in each imputed dataset and were integrated based on Rubin’s rules.17
The external validity of the predictive scores was based on measures of discrimination (ability of the model to discriminate between favourable and unfavourable outcome) and calibration (whether probabilities predicted by the model match observed probabilities). Receiver operating characteristic (ROC) curve analysis was performed, and the area under the ROC curve (AUC) and 95% confidence intervals (CI) were used to assess discrimination. The AUC values of original and simplified scores were compared using paired DeLong test. We calculated the optimal cut-off value for predictive scores to distinguish between favourable and unfavourable neurological outcomes. In calculating the cut-off value, we explored both the value that provided the highest combined sensitivity and specificity (Youden Index) and the value that provided positive predictive value (PPV) > 0.99. Calibration was assessed using the Hosmer-Lemeshow goodness-of-fit test. We considered 10 groups in the Hosmer-Lemeshow test. The extent of overestimation or underestimation relative to the observed and predicted rate was explored graphically using calibration plots.
In the main analysis, missing values were handled using the pair-wise deletion method. Additionally, we performed sensitivity analysis using imputed datasets. To obtain approximately unbiased estimates of the parameters, we performed multiple imputations to handle missing data in the analyses. After the above-mentioned inclusion and exclusion criteria were applied, multiple regression, including variables potentially related to missing data (variables listed in Table 1, score availabilities excepted), was conducted. Twenty imputation datasets were generated. Constraint for minimum imputed values was introduced to avoid errors in predictive score calculation. Based on a previous study6 and minimum values in the dataset, the lowest possible values for no-flow interval, low-flow interval, lactate, and creatinine were determined as 0.5 min, 0.5 min, 0.5 mmol/L, and 10 µmol/L, respectively.
| Variable | Value | |
|---|---|---|
| Number of patients | 2,428 | |
| Patient age, years | 70 [60–80] | |
| Male sex | 1,651 (68) | |
| Cardiac arrest at home | 1,326 (55) | |
| Witnessed arrest | 1,748 (72) | |
| Bystander CPR | 1,024 (42) | |
| AED use by a bystander | 180 (7) | |
| Initial shockable cardiac rhythm | 528 (22) | |
| Response time, min | 7 [5–9] | |
| No-flow interval, min | 10 [4–14] | |
| Low-flow interval, min | 30 [16–41] | |
| pH | 6.97 [6.84–7.13] | |
| Lactate (mmol/L) | 153 [129–187] | |
| Creatinine (µmol/L) | 98 [80–135] | |
| Total dose of adrenaline (epinephrine) during CPR, mg | 2 [1–4] | |
| Original OHCA score available | 855 (35) | |
| Simplified OHCA score available | 1,359 (56) | |
| Original CAHP score available | 1,130 (47) | |
| Simplified CAHP score available | 1,834 (76) | |
| 1-month unfavourable neurological outcome | 1,985 (82) | |
| Data are presented as number (percentage) or median [interquartile range]. | ||
| CPR: cardiopulmonary resuscitation, AED; automated external defibrillator, OHCA, out-of-hospital cardiac arrest; CAHP, Cardiac Arrest Hospital Prognosis | ||
All statistical tests were two-tailed, and statistical significance was defined by a P-value < 0.05 or assessed with 95% CI. To address missing values for variables, we excluded incomplete cases on an analysis-by-analysis basis.
The SOS-KANTO 2012 study evaluated 16,452 patients, and 3,335 patients were admitted to the hospital after ROSC. After 907 patients (784 patients with extrinsic aetiology and 123 patients with unknown outcome) were excluded, a total of 2,428 patients were deemed eligible for the analysis (Fig. 1). The percentages of missing values for the 15 variables varied between 0% and 43% (age: 0%, sex: 0%, cardiac arrest location: 2%, witnessed status: 0%, bystander CPR: 0%, AED use by a bystander: 0%, initial shockable cardiac rhythm: 1%, response time: 0%, no-flow interval: 43%, low-flow interval: 18%, pH: 7%, lactate: 24%, creatinine: 12%, total dose of adrenaline during CPR: 0%, and 1-month neurological outcome: 0%). In total, 2,606 of 36,420 items (7%) were missing, and 1,595 of the 2,484 patient records (64%) were incomplete. Table 1 shows the patients’ baseline characteristics. Median age was 70 (IQR: 60–80) years, and 68% were men. Of the 2,428 patients, 1,985 (82%) had unfavourable neurological outcome 1 month after OHCA. A summary of the performance measures for the prediction models is presented in Table 2.
| Score availability | AUC (95% CI) | Model fit | Sensitivitya | Specificitya | Proportion of PPV > 0.99 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Complete case | Imputed dataset | Complete case | Imputed dataset | ||||||||
| OHCA score | |||||||||||
| Original | 35% | 0.83 (0.79–0.86) | 0.83 (0.81–0.85) | Good | Poor | 43 | 91 | 2% | |||
| Simplified | 56% | 0.81 (0.77–0.84) | 0.82 (0.79–0.84) | Poor | Poor | 34 | 74 | 4% | |||
| CAHP score | |||||||||||
| Original | 47% | 0.90 (0.87–0.92) | 0.92 (0.91–0.94) | Poor | Poor | 205 | 313 | 5% | |||
| Simplified | 76% | 0.88 (0.85–0.90) | 0.91 (0.90–0.93) | Good | Good | 165 | 240 | 13% | |||
| aCut-off values were determined based on the Youden Index that determines the value that provides the highest combined sensitivity and specificity. The cut-off values based on Youden Index were 43 for the original OHCA score, 34 for the sOHCA score, 205 for the original CAHP score, and 165 for the sCAHP score. Goodness-of-fit was assessed using Hosmer-Lemeshow test. A score with P > 0.05 in the test was considered to have good calibration. | |||||||||||
| OHCA, out-of-hospital cardiac arrest; CAHP, cardiac arrest hospital prognosis; AUC, area under the receiver operating characteristic curve; CI, confidence interval; PPV, positive predictive value | |||||||||||
Of the 2,428 patients, the sOHCA score was obtained for 1,359 (56%). Due to one or more missing values in the parameters, the sOHCA score was not available for 1,069 patients.
Of the 1,359 patients whose sOHCA score was obtained, no-flow interval was not recorded for 504 patients, and the original OHCA score was available for 855 (35% of 2,428) patients. Since the collapse of 395 patients was witnessed, the no-flow interval was inherently unavailable for these patients. Score availability was significantly higher for the sOHCA score than the original OHCA score, with a difference of 21% (95% CI, 18–24%).
Of all 2,428 patients, sCAHP score was obtained for 1,834 (76%). Of the 1,834 patients whose sCAHP score was obtained, no-flow interval was not recorded in 704 patients, and the original CAHP score was available for 1,130 (47% of 2,428) patients. No-flow time was unavailable in 529 patients, because the collapse of these patients was not witnessed. Score availability was significantly higher for the sCAHP score than the original CAHP score, with difference of 29% (95% CI, 26–32%). When the sOHCA and sCAHP scores were compared, score availability was significantly higher in the sCAHP score, with a difference of 20% (95% CI, 17–22%).
Figure 2 shows the ROC curves of predictive scores. The AUC of the original and simplified OHCA scores were 0.83 (95% CI, 0.79–0.86) and 0.81 (95% CI, 0.77–0.84), respectively. There was no significant difference between the AUC of these two scores (P = 0.41).
The AUC of the original and simplified CAHP scores were 0.90 (95% CI, 0.87–0.92) and 0.88 (95% CI, 0.85–0.90), respectively. There was no significant difference between the AUC of these two scores (P = 0.47). Comparing the sOHCA and sCAHP scores, the AUC was significantly higher in the sCAHP score (P < 0.001). The cut-off values based on Youden Index and PPV > 0.99, respectively, were as follows: 43 and 91 for the original OHCA score, 34 and 74 for the sOHCA score, 205 and 313 for the original CAHP score, and 165 and 240 for the sCAHP score. The cut-off values with PPV > 0.99 gave a false positive rate of < 0.01 in each score, and identified 14/855 (2%) patients in the original OHCA score, 58/1359 (4%) patients in the sOHCA score, 53/1130 (5%) patients in the original CAHP score, and 241/1834 (13%) patients in the sCAHP score.
Figure 3 shows the calibration plots, illustrating how the predictive scores performed in predicting 1-month favourable neurological outcome in the study population. The Hosmer-Lemeshow goodness-of-fit test demonstrated that the sOHCA and original CAHP scores had significantly poor model fits (ν = 8, χ2 = 19.1, P = 0.014 and ν = 8, χ2 = 23.8, P = 0.002; respectively), whereas the original OHCA and sCAHP scores did not (ν = 8, χ2 = 11.1, P = 0.196 and ν = 8, χ2 = 13.5, P = 0.097; respectively). A calibration plot showed that the sOHCA and original CAHP scores overestimated the probability of the 1-month unfavourable neurological outcome in patients with relatively better prognosis.
In the sensitivity analysis, predictive scores were estimated in each of the multiple imputed datasets and were integrated. The AUC of the original and simplified OHCA scores were 0.83 (95% CI, 0.81–0.85) and 0.82 (95% CI, 0.79–0.84), respectively. The AUC of the original and simplified CAHP scores were 0.92 (95% CI, 0.91–0.94) and 0.91 (95% CI, 0.90–0.93), respectively. Comparing the sOHCA and sCAHP scores, the AUC was significantly higher in the sCAHP score (P < 0.001). The Hosmer-Lemeshow goodness-of-fit test revealed that the original OHCA, original CAHP, and sOHCA scores had significantly poor model fits (ν = 8, χ2 = 72.9, P < 0.001; ν = 8, χ2 = 64.1, P < 0.001; and ν = 8, χ2 = 33.4, P < 0.001; respectively), whereas sCAHP score did not (ν = 8, χ2 = 4.0, P = 0.86)(see Additional file 2).
Our analyses showed that both the original and simplified OHCA scores, as well as the CAHP scores, had similar, acceptable discrimination in predicting unfavourable neurological outcome 1 month after OHCA in a Japanese cohort. With the exception of the sOHCA score, these predictive scores also calibrated well. Our findings of acceptable discriminatory performance of the original OHCA and CAHP scores were in agreement with those of previous internal and external validations6 − 9. The result of the sensitivity analysis was largely consistent with these findings. To the best of our knowledge, this is the first study to externally validate the sOHCA and sCAHP scores.
It is desirable for a predictive score to have immediate and wide availability, high discrimination, and good calibration. Both OHCA and CAHP scores are immediately available after routine arterial blood gases are obtained; however, their availability was relatively low in the present study. By removing the no-flow interval from the calculation of these predictive scores, their availability increased 1.6-fold. The proportion of unwitnessed OHCA in our study, 28%, was between the reported range of 11–40%7−11. Since the no-flow interval is inherently unavailable in these unwitnessed OHCA, our results showing significantly higher availability of simplified scores are to be expected. Besides the higher availability, a previous study suggested that removing the no-flow interval from the calculation could have the advantage of improved discrimination, because it decreases the uncertainty caused by estimating the no-flow interval, especially in an area where the proportion of unwitnessed cardiac arrest is relatively high12. In the present study, discrimination did not improve, but it was maintained. These results indicate that removing the no-flow interval from the calculation is acceptable in terms of discrimination.
A validated predictive score assists clinical decisions and facilitates further clinical research by enabling researchers to compare outcomes between cohorts with different demographics and emergency care systems on a fair basis. Furthermore, validated predictive scores may be considered as a tool to help decide whether further aggressive life-sustaining therapy should be provided. Given that treatment after cardiac arrest costs a great deal and lasts for long periods, an objective and accurate indicator of no chance of favourable neurological outcome could decrease the difficulty for patients’ families, as well as healthcare providers. Although there are controversial ethical issues surrounding the futility of the treatment, we explored the cut-off value that provides PPV > 0.99, based on a traditional definition of medical futility as intervention that provides a < 1% possibility of favourable neurological outcome18. When the cut-off value that provides PPV > 0.99 was applied to the OHCA patients who had ROSC and were admitted to hospital, the proportion of patients who were considered to have unfavourable neurological outcome was highest in the sCAHP score, in which futility was indicated for 13% of the patients. These results suggest that the predictive score may enable the identification of futility in the care of OHCA patients after successful ROSC at the very beginning of their treatment in the hospital.
Among the predictive scores assessed, the sCAHP score had the highest availability, and discrimination and goodness-of-fit were comparable to or better than the other scores. Therefore, we believe that sCAHP is the best candidate for clinical implementation. We recognise that prognostication requires caution and should be multimodal whenever possible. Various clinical examinations, electrophysiological measurements, imaging studies, and evaluation of biomarkers have been performed to estimate prognosis after OHCA19. Predictive scores are expected to reinforce the accuracy of these predictive factors, and vice versa. We believe our results showing the validity of predictive scores would help families of patients, healthcare providers, and researchers involved in the care of OHCA patients who have been resuscitated and admitted to hospital.
There are several limitations to the present study. First, the retrospective study design is prone to various biases. The original OHCA score was developed before the study period; thus, it is possible that clinicians used the score during the study period, which possibly affected the results. Although the original CAHP score was described later than the period of the current study, clinicians may have considered some of the factors incorporated into the CAHP score, while making their clinical decisions. Therefore, self-fulfilling prophecy is a potential bias. Second, the reason for missing values was not clear. Although we addressed the missing values using the pair-wise method, this may have biased our results. Third, 1-month CPC grade is a standard assessment of functional outcome after cardiac arrest; however, further studies that take long-term functional outcomes into account are warranted because the functional outcome may change over a long period after cardiac arrest20. Fourth, there are no available data regarding the validity of the various measurements in the studied database. Finally, further validation is desirable before implementing a predictive score into practice in a different geographical area, because outcomes after OHCA are related to both patient demographics and EMS services.
Our analyses externally validated the original OHCA and CAHP, as well as sOHCA and sCAHP scores. By removing the no-flow interval from the calculations, simplified scores increased their availability without significant reduction of discrimination. Among these predictive scores, the sCAHP score demonstrated superior performance; thus, we recommend the use of the sCAHP score in predicting the outcome of patients who are successfully resuscitated and admitted to hospital after OHCA. The use of the sCAHP score is expected to help families of patients, healthcare providers, and researchers by providing accurate stratification of patients.
AED, automated external defibrillator; AUC, area under the receiver operating characteristic curve; CAHP, cardiac arrest hospital prognosis; CI, confidence interval; CPC, cerebral performance category; CPR, cardiopulmonary resuscitation; EMS, emergency medical service; IQR, interquartile range; OHCA, out-of-hospital cardiac arrest; PPV, positive predictive value; ROC, receiver operating characteristic; ROSC, return of spontaneous circulation; SOS-KANTO, Survey of Survivors after Out-of-Hospital Cardiac Arrest in the Kanto Region
Ethics approval and consent to participate
The study’s retrospective protocol was approved by the institutional review board of the Tokyo Metropolitan Bokutoh Hospital (approval number: 31-041). All data were de-identified prior to the analysis, and the need for informed consent was waived.
Availability of data and materials
Registry data is available with permission of the Kanto Regional Group of the Japanese Association for Acute Medicine.
Competing interests
None.
Funding
None.
Acknowledgements
None.
Authors' contributions
KeS contributed to study conception and design, interpretation of data, statistical analysis, drafting of the manuscript, critical revision of the manuscript for important intellectual content, and provided intellectual input into the research and the manuscript. AS, NK, TT and TN contributed to acquisition of data, critical revision of the manuscript for important intellectual content, and provided intellectual input into the research and the manuscript. KaS, YK, TI, and YH contributed to critical revision of the manuscript for important intellectual content, and provided intellectual input into the research and the manuscript. All authors read and approved the manuscript.
Consent for publication
Not applicable.
Additional file 1
File format: .docx
Title of data: Formulae for original and simplified Out-of-Hospital Cardiac Arrest (OHCA) and Cardiac Arrest Hospital Prognosis (CAHP) scores (with permission)
Description of data: a VF, ventricular fibrillation; VT, ventricular tachycardia.
b Natural logarithm of the no-flow interval (min); lowest possible value = 0.5.
c Natural logarithm of the low-flow interval (min), lowest possible value = 0.5.
d Serum creatinine (μmol/L).
e Natural logarithm of plasma lactate (mmol/L) on admission.
f Total dose of adrenaline during cardiopulmonary resuscitation: 0 points for 0 mg; 27 points for 1 to 2 mg; and 43 points for 3 or more mg.
Additional file 2
File format: .tif
Title of data: Calibration plots for the predictive scores in predicting 1-month unfavourable functional outcome in the sensitivity analysis
Description of data: Patients were grouped according to 10 percentile increments of predictive probability of unfavourable neurological outcome (cerebral performance category 3–5). The predicted probabilities of unfavourable outcome in each group are plotted against the actually observed proportions. The line shows the points where the predicted and observed values match.