The SOS-KANTO 2012 study evaluated 16,452 patients, and 3,335 patients were admitted to the hospital after ROSC. After 907 patients (784 patients with extrinsic aetiology and 123 patients with unknown outcome) were excluded, a total of 2,428 patients were deemed eligible for the analysis (Fig. 1). The percentages of missing values for the 15 variables varied between 0% and 43% (age: 0%, sex: 0%, cardiac arrest location: 2%, witnessed status: 0%, bystander CPR: 0%, AED use by a bystander: 0%, initial shockable cardiac rhythm: 1%, response time: 0%, no-flow interval: 43%, low-flow interval: 18%, pH: 7%, lactate: 24%, creatinine: 12%, total dose of adrenaline during CPR: 0%, and 1-month neurological outcome: 0%). In total, 2,606 of 36,420 items (7%) were missing, and 1,595 of the 2,484 patient records (64%) were incomplete. Table 1 shows the patients’ baseline characteristics. Median age was 70 (IQR: 60–80) years, and 68% were men. Of the 2,428 patients, 1,985 (82%) had unfavourable neurological outcome 1 month after OHCA. A summary of the performance measures for the prediction models is presented in Table 2.
Table 2
Performance measures of the prediction models
| Score availability | AUC (95% CI) | | Model fit | | Sensitivitya | Specificitya | Proportion of PPV > 0.99 |
Complete case | Imputed dataset | | Complete case | Imputed dataset |
OHCA score | | | | | | | | | | |
| Original | 35% | 0.83 (0.79–0.86) | 0.83 (0.81–0.85) | | Good | Poor | | 43 | 91 | 2% |
| Simplified | 56% | 0.81 (0.77–0.84) | 0.82 (0.79–0.84) | | Poor | Poor | | 34 | 74 | 4% |
CAHP score | | | | | | | | | | |
| Original | 47% | 0.90 (0.87–0.92) | 0.92 (0.91–0.94) | | Poor | Poor | | 205 | 313 | 5% |
| Simplified | 76% | 0.88 (0.85–0.90) | 0.91 (0.90–0.93) | | Good | Good | | 165 | 240 | 13% |
aCut-off values were determined based on the Youden Index that determines the value that provides the highest combined sensitivity and specificity. The cut-off values based on Youden Index were 43 for the original OHCA score, 34 for the sOHCA score, 205 for the original CAHP score, and 165 for the sCAHP score. Goodness-of-fit was assessed using Hosmer-Lemeshow test. A score with P > 0.05 in the test was considered to have good calibration. |
OHCA, out-of-hospital cardiac arrest; CAHP, cardiac arrest hospital prognosis; AUC, area under the receiver operating characteristic curve; CI, confidence interval; PPV, positive predictive value |
Availability of predictive scores
Of the 2,428 patients, the sOHCA score was obtained for 1,359 (56%). Due to one or more missing values in the parameters, the sOHCA score was not available for 1,069 patients.
Of the 1,359 patients whose sOHCA score was obtained, no-flow interval was not recorded for 504 patients, and the original OHCA score was available for 855 (35% of 2,428) patients. Since the collapse of 395 patients was witnessed, the no-flow interval was inherently unavailable for these patients. Score availability was significantly higher for the sOHCA score than the original OHCA score, with a difference of 21% (95% CI, 18–24%).
Of all 2,428 patients, sCAHP score was obtained for 1,834 (76%). Of the 1,834 patients whose sCAHP score was obtained, no-flow interval was not recorded in 704 patients, and the original CAHP score was available for 1,130 (47% of 2,428) patients. No-flow time was unavailable in 529 patients, because the collapse of these patients was not witnessed. Score availability was significantly higher for the sCAHP score than the original CAHP score, with difference of 29% (95% CI, 26–32%). When the sOHCA and sCAHP scores were compared, score availability was significantly higher in the sCAHP score, with a difference of 20% (95% CI, 17–22%).
Discrimination of predictive scores
Figure 2 shows the ROC curves of predictive scores. The AUC of the original and simplified OHCA scores were 0.83 (95% CI, 0.79–0.86) and 0.81 (95% CI, 0.77–0.84), respectively. There was no significant difference between the AUC of these two scores (P = 0.41).
The AUC of the original and simplified CAHP scores were 0.90 (95% CI, 0.87–0.92) and 0.88 (95% CI, 0.85–0.90), respectively. There was no significant difference between the AUC of these two scores (P = 0.47). Comparing the sOHCA and sCAHP scores, the AUC was significantly higher in the sCAHP score (P < 0.001). The cut-off values based on Youden Index and PPV > 0.99, respectively, were as follows: 43 and 91 for the original OHCA score, 34 and 74 for the sOHCA score, 205 and 313 for the original CAHP score, and 165 and 240 for the sCAHP score. The cut-off values with PPV > 0.99 gave a false positive rate of < 0.01 in each score, and identified 14/855 (2%) patients in the original OHCA score, 58/1359 (4%) patients in the sOHCA score, 53/1130 (5%) patients in the original CAHP score, and 241/1834 (13%) patients in the sCAHP score.
Calibration
Figure 3 shows the calibration plots, illustrating how the predictive scores performed in predicting 1-month favourable neurological outcome in the study population. The Hosmer-Lemeshow goodness-of-fit test demonstrated that the sOHCA and original CAHP scores had significantly poor model fits (ν = 8, χ2 = 19.1, P = 0.014 and ν = 8, χ2 = 23.8, P = 0.002; respectively), whereas the original OHCA and sCAHP scores did not (ν = 8, χ2 = 11.1, P = 0.196 and ν = 8, χ2 = 13.5, P = 0.097; respectively). A calibration plot showed that the sOHCA and original CAHP scores overestimated the probability of the 1-month unfavourable neurological outcome in patients with relatively better prognosis.
Sensitivity analysis
In the sensitivity analysis, predictive scores were estimated in each of the multiple imputed datasets and were integrated. The AUC of the original and simplified OHCA scores were 0.83 (95% CI, 0.81–0.85) and 0.82 (95% CI, 0.79–0.84), respectively. The AUC of the original and simplified CAHP scores were 0.92 (95% CI, 0.91–0.94) and 0.91 (95% CI, 0.90–0.93), respectively. Comparing the sOHCA and sCAHP scores, the AUC was significantly higher in the sCAHP score (P < 0.001). The Hosmer-Lemeshow goodness-of-fit test revealed that the original OHCA, original CAHP, and sOHCA scores had significantly poor model fits (ν = 8, χ2 = 72.9, P < 0.001; ν = 8, χ2 = 64.1, P < 0.001; and ν = 8, χ2 = 33.4, P < 0.001; respectively), whereas sCAHP score did not (ν = 8, χ2 = 4.0, P = 0.86)(see Additional file 2).