Predicting In-hospital Death in Pneumonic COPD exacerbation via BAP-65, CURB-65, and Machine Learning

doi:10.21203/rs.3.rs-687032/v1

Introduction

There is no established clinical prediction model for in-hospital death among patients with pneumonic chronic obstructive pulmonary disease (COPD) exacerbation. We aimed to externally validate BAP-65 and CURB-65 and to develop a new model based on the eXtreme Gradient Boosting (XGBoost) algorithm.

Methods

This multicentre cohort study included patients aged ≥40 years with pneumonic COPD exacerbation. The input data were age, sex, activities of daily living, mental status, systolic and diastolic blood pressure, respiratory rate, and heart rate, peripheral blood eosinophil count, and blood urea nitrogen. The primary outcome was in-hospital death. BAP-65 and CURB-65 underwent external validation using the area under the receiver operating characteristic curve (AUROC) in the whole dataset. We used XGBoost to develop a new prediction model. We compared the AUROCs of XGBoost with that of BAP-65 and CURB-65 in the test dataset using bootstrap sampling.

Results

We included 1190 patients with pneumonic COPD exacerbation. The in-hospital mortality was 7% (88/1190). In the external validation of BAP-65 and CURB-65, the AUROCs [95% confidence interval (CI)] of BAP-65 and CURB-65 were 0.69 [0.66–0.72], and 0.69 [0.66–0.72], respectively. XGBoost showed an AUROC of 0.71 [0.62–0.81] in the test dataset. There was no significant difference in the AUROCs of XGBoost vs BAP-65 (absolute difference, 0.054; 95% CI, -0.057–0.16) or vs CURB-65 (absolute difference, 0.0021; 95% CI, -0.091–0.088).

Conclusion

BAP-65, CURB-65, and XGBoost showed low predictive performance for in-hospital death in pneumonic COPD exacerbation. Further large-scale studies including more variables are warranted.

Pulmonology

Chronic obstructive pulmonary disease

Machine Learning

Pneumonia

Prognosis

Chronic obstructive pulmonary disease (COPD) is a common respiratory disease that is characterised by airflow limitation due to chronic inflammation of the airways and lungs [1]. Patients with COPD often experience acute worsening of baseline symptoms, and with coexisting consolidation on chest imaging, mortality is increased compared to non-pneumonic COPD exacerbation [2].

CURB-65 (confusion, blood urea nitrogen > 19 mg/dL, respiratory rate ≥ 30 breaths/min, systolic blood pressure < 90 mmHg or diastolic blood pressure ≤ 60 mmHg, and age ≥ 65 years) is a simple prediction model in patients with community-acquired pneumonia and has been validated internally and externally [3,4]. On the other hand, BAP-65 (blood urea nitrogen ≥ 25 mg/dL, altered mental status, heart rate ≥ 109 beats/min, and age ≥ 65 years) is an easily computable prediction model in patients with COPD exacerbation, which has shown good performance in internal validation and external validation cohorts [5,6]. To date, there is no established clinical prediction model in the specific population with pneumonic COPD exacerbation. It is also unclear whether BAP-65 and CURB-65 can be applied to patients with pneumonic COPD exacerbation [7].

Our study had two purposes: (i) the external validation of BAP-65 and CURB-65 for predicting in-hospital death among patients with pneumonic COPD exacerbation, and (ii) the development of a high-performance clinical prediction model using a modern machine learning algorithm that is gaining ground in the medical field [8].

Study design

Our study was a multicentre retrospective cohort study conducted across five acute care hospitals in Japan. To maximise patient capture, patient data were collected during different periods in each hospital between April 1, 2008, and July 31, 2020.

Patient selection was based on a validated algorithm for selecting patients with pneumonic COPD exacerbation [9]. The algorithm was based on the 10th revision of the International Classification of Diseases and Related Health Problems. First, patients aged ≥ 40 years who had both pneumonia and COPD exacerbation were selected. Patients with other differential diagnoses mimicking pneumonic COPD exacerbation were excluded, including heart failure, pneumothorax, asthma exacerbation, and obstructive pneumonia.

This study was approved by the institutional review board of each hospital (approval number, 200811). This article was reported according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement (eTable1) [10].

Input and output data

The following input data on the day of admission were collected from the data warehouse or electrical medical records in each hospital: age, sex, activity of daily living (full support or not), mental status (altered mental status or not), vital signs (systolic and diastolic blood pressure, respiratory rate, and heart rate), and laboratory results (peripheral blood eosinophil count and blood urea nitrogen). Activities of daily living were defined as full support when the Barthel index was zero, and altered mental status was defined as a Japan Coma Scale score ≥ 1. The Japan Coma Scale has been used extensively in clinical sites in Japan, and it is coded in nationwide administrative claims data [11]. Our variable selection was based on existing clinical prediction models of pneumonia or COPD exacerbation [3,5,12]. The primary outcome was in-hospital death which was derived from the electrical medical records in each hospital.

Statistical analysis

The study process is illustrated in Figure 1. Patient characteristics were summarised as means for continuous variables and as percentages for categorical variables. All statistical analyses were performed using the R software version 4.0.2 (R Foundation for Statistical Computing, Vienna, Austria). The scripts are available in the GitHub repository (https://github.com/AkihiroShiroshita/Prediction-model-for-Pnuemonic-COPD-exacerbation.git).

External validation of BAP-65 and CURB-65

We conducted the external validation of BAP-65 and CURB-65 with respect to the entire dataset to evaluate their performance in a large sample size. We calculated the sensitivity and specificity using each total score as the cut-off point. To assess the calibration ability, we summarised the mortality according to each total risk score. To assess the discriminatory ability of the two models, we calculated the area under the receiver operating characteristic curve (AUROC). We used multiple imputation to cope with missing data [13]. We created a total of 100 datasets using multiple imputation with chained equations and calculated the AUROC within each dataset. Thereafter, we combined the estimates of AUROC using Rubin’s combining rule [14,15].

Model development via machine learning

We used the eXtreme Gradient Boosting (XGBoost) algorithm to develop a clinical prediction model for in-hospital death among patients with pneumonic COPD exacerbation. The XGBoost algorithm is a powerful ensemble method of machine learning that combines a set of weak learners of the decision tree [16]. Its parallel computation enables the efficient and accurate development of a prediction model. Because it extracts variable importance, imputation of missing data, scaling, or normalisation is not required. What is required in the algorithm is the proper tuning of the hyperparameters, which are parameters that control the behaviour of the model. In our study, the original data were first partitioned into training and test datasets. We used the stratified sampling method with a 7:3 ratio for data splitting, which allowed the two datasets to have similar in-hospital mortality. Second, we developed prediction models using a training dataset. We performed a grid search with 4-fold cross-validation to obtain the optimal hyperparameters for maximising the mean AUROC [16]. In the grid search, the hyperparameter candidates for max_depth (maximum tree depth) was {2, 4, 6, 8, 10}, and min_child_weight (minimum degree of impurity needed in a node) was {1, 2, 3, 4, 5}. After fixing max_depth and min_child_weight, we searched the maximum number of trees based on the cross-validation. We set the remaining hyperparameters as default. Third, for external validation, we validated the trained model using the test dataset. We used the AUROC as an index to validate the model. Finally, the importance of the variables based on the impurity metric was plotted.

Model comparison

We compared the model performances of the three prediction models using the test dataset to allow comparison on a one-to-one basis. To evaluate the discriminatory performance, we described the ROC curves of the three prediction models. Thereafter, we estimated the differences in AUROCs using bootstrap sampling (BAP-65 vs XGBoost, and CURB-65 vs XGBoost) [17]. The XGBoost model can take into account missing data while BAP-65 and CURB-65 cannot. The XGBoost model used the whole test dataset while BAP-65 and CURB-65 used the patient data without missing values.

The patient selection flowchart is shown in Figure 1. We initially selected 1431 patients. After excluding 241 patients with other diagnoses, 1190 patients with pneumonic COPD exacerbation were included in our analysis. Patient characteristics are summarised in Table 1. The in-hospital mortality rate was 88/1190 (7%). The number of intratracheal intubations was 16/1190 (1%), and the median length of hospital stay was 12 (interquartile range: 8–18) days.

External validation of BAP-65 and CURB-65

Table 2 presents the summary of the number of patients with each total score. The calibration performances of both prediction models were low. The AUROC of BAP-65 was 0.69 [95% confidence interval (CI): 0.66 to 0.72) and that of CURB-65 was 0.69 [95% CI: 0.66 to 0.72). The discriminatory performance of both prediction models was also low.

Model development via machine learning

Based on the results of the grid search, we set up the hyperparameters as follows: max_depth (maximum tree depth) = 4, min_child_weight (minimum degree of impurity needed in a node) = 2, eta (learning rate) = 0.1, subsample (the proportion of cases to be randomly sampled for each tree) = 0.8, colsample_bytree (the proportion of predictor variables sampled for each tree) = 0.8, gamma (minimal loss to expand on a leaf node) = 0, lambda (L2 regularisation term on weights) = 1, alpha (L1 regularisation term on weights) = 0, scale_pos_weight (balance of positive and negative weights) = 1, and maximum number of trees = 37. Cross-validation of the developed model revealed a mean AUROC of 0.76, and external validation in the test dataset revealed an AUROC of 0.72 [95% CI: 0.62 to 0.82). Feature importance is illustrated in Figure 2, and it revealed that blood urea nitrogen was the most important factor for predicting in-hospital death. On the contrary, activities of daily living and sex showed little importance.

Model comparison

We performed model comparisons using the test data. The XGBoost model used the whole test dataset (N = 357), while BAP-65 and CURB-65 used the data of 314 and 281 patients, respectively, because of missing values. Figure 3 shows the ROC curves of the BAP-65, CURB-65, and XGBoost models. There was no significant difference in AUROCs between the XGBoost model and BAP-65 (absolute difference, 0.054; 95% CI: -0.057 to 0.16) or between the XGBoost model and CURB-65 (absolute difference, 0.0021; 95% CI: -0.091 to 0.088).

Our study revealed that all three models (BAP-65, CURB-65, and XGBoost model) had low discriminatory ability for predicting in-hospital death among patients with pneumonic COPD exacerbation. Further large-scale studies are needed to develop a specific clinical prediction model for pneumonic COPD exacerbation.

The two simple scoring systems, BAP-65 and CURB-65, showed low predictive performance in our dataset of patients with pneumonic COPD exacerbation. Our results were consistent with those of a previous retrospective cohort study that revealed that CURB-65 had poor performance for predicting pneumonic COPD exacerbation [18]. Our target population included patients with a specific category of pneumonic COPD exacerbation. The disease spectrum of pneumonic COPD exacerbation may differ from that of COPD exacerbation and pneumonia. A specific clinical prediction model for pneumonic COPD exacerbation is warranted.

Our new clinical prediction model based on the XGBoost algorithm also had a low predictive performance for in-hospital deaths in pneumonic COPD exacerbation. The XGBoost algorithm is a powerful machine learning technique that can overcome the drawbacks of the development processes of BAP-65 and CURB-65. Contrary to the recursive partition that was used in the development of BAP-65, the XGBoost model avoids model instability [5,19]. In addition, unlike logistic regression, which was used in the development of CURB-65, the XGBoost model is not based on the assumption of linearity and does not require the categorisation of continuous variables [3]. It can also find the optimal interaction terms between variables. Despite its high ability, our XGBoost model did not show high performance.

There may be two reasons for the low predictive performance of the XGBoost model. First, our sample may have been too small to develop internally and externally validated prediction models. As a rule of thumb, the number of events required for model development is at least 10 events per variable [20]. Our input data included 10 variables, and at least 100 events were required; however, there were only about 60 events in our training dataset. Although we used the XGBoost model, which may require a smaller sample size, we could not overcome the problem in our dataset [21]. Second, other missing variables were included in the model. For example, in a previous study, the DECAF score tended to have a higher predictive performance than CURB-65 in pneumonic COPD exacerbation [18]. The Extended Medical Research Council Dyspnea Scale, arterial blood gas analysis results, and atrial fibrillation may be additional candidates. According to a systematic review of prediction models for COPD exacerbation, patients’ baseline characteristics such as body mass index, forced expiratory volume in 1 second, and previous COPD exacerbation were used in studies with a low risk of bias [22]. These could also be additional candidates.

The feature importance plot in our study highlighted the importance of blood urea nitrogen, systolic blood pressure, and altered mental status. These variables should be included in a new clinical prediction model for pneumonic COPD exacerbation. On the contrary, activities of daily living and sex were of little importance in the model. In our study, the activities of daily living were categorised as full support or not, and this might have led to the loss of notable information. Approximately 90% of the included patients were men, which may have led to the unimportance of sex as a variable. Our study revealed some candidates for the included variables in developing a new model.

In our study, we could not conclude which clinical model was superior to the others. In our test dataset, the number of patients was approximately 300, and the in-hospital mortality was 25, which was much smaller than the necessary sample size for precise external validation [23,24]. The results of the external validation of BAP-65 and CURB-65 in the whole dataset showed that the predictive ability for in-hospital death could be similar between CURB-65 and BAP-65. Physicians who use either BAP-65 or CURB-65 will not have to change their practice based on our study.

Our study had several limitations. First, only Japanese patients were included, suggesting a lack of generalisability. Second, the primary outcome in our study was in-hospital mortality, and long-term outcomes could not be evaluated. Third, we could not set aside an additional dataset for external validation before splitting the dataset. Because we performed external validation on the split dataset, the AUROC in the test dataset may have been overestimated. To overcome these limitations, large-scale studies from different regions are needed.

BAP-65, CURB-65, and the XGBoost model showed poor performance in predicting in-hospital death among patients with pneumonic COPD exacerbation. Further large-scale studies with more variables are needed to develop a new prognostic model for pneumonic COPD exacerbation.

Acknowledgement: None

Data availability statement: The datasets generated and/or analysed during the current study are not publicly available due to the privacy issues but are available from the corresponding author on reasonable request.

Conflict of interest: The authors declare no conflicts of interest.

Funding sources: Funding for the English language editing was obtained from the Systematic Review Workshop Peer Support Group (SRWS-PSG: https://community.camp-fire.jp/projects/view/187310). The funder played no role in the study design, study execution, data analyses, data interpretation, or decision to submit the report.

Author contributions: AS, Yuya K, HS, CS, KS, SM, KT, and Yuki K contributed to the conception and design of the work. AS, HS, CS, KS, SM, and KT contributed to data acquisition. AS, Yuya K, and Yuki K contributed to the data analysis and interpretation. AS, Yuya K, and Yuki K drafted the manuscript. All authors revised the manuscript critically and approved the final version of the manuscript. AS, Yuya K, HS, CS, KS, SM, KT, and Yuki K agreed to be accountable for all aspects of any part of the work.

1. 2020 Gold Reports. Global Initiative for Chronic Obstructive Lung Disease - GOLD. https://goldcopd.org/gold-reports. Date last accessed: March 5 2021.

2. Saleh A, López-Campos JL, Hartl S, et al. The effect of incidental consolidation on management and outcomes in COPD exacerbations: Data from the European COPD Audit. PLoS One 2015; 10: e0134004.

3. Lim WS, van der Eerden MM, Laing R, et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax 2003; 58: 377-382.

4. Ilg A, Moskowitz A, Konanki V, et al. Performance of the CURB-65 Score in predicting critical care interventions in patients admitted with community-acquired pneumonia. Ann Emerg Med 2019; 74: 60-68.

5. Tabak YP, Sun X, Johannes RS, et al. Mortality and need for mechanical ventilation in acute exacerbations of chronic obstructive pulmonary disease: development and validation of a simple risk score. Arch Intern Med 2009; 169: 1595-1602.

6. Shorr AF, Sun X, Johannes RS, et al. Validation of a novel risk score for severity of illness in acute exacerbations of COPD. Chest 2011; 140: 1177-1183.

7. Trethewey SP, Hurst JR, Turner AM. Pneumonia in exacerbations of COPD: what is the clinical significance? ERJ Open Res 2020; 6: 00282-2019.

8. Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med 2019; 380: 1347-1358.

9. Shiroshita A, Shiba H, Tanaka Y, et al. Effectiveness of steroid therapy on pneumonic chronic obstructive pulmonary disease exacerbation: a multicenter, retrospective cohort study. Int J Chron Obstruct Pulmon Dis 2020; 15: 2539-2547.

10. Collins GS, Reitsma JB, Altman DG, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med 2015; 162: 55-63.

11. Shigematsu K, Nakano H, Watanabe Y. The eye response test alone is sufficient to predict stroke outcome—reintroduction of Japan Coma Scale: a cohort study. BMJ Open 2013; 3: e002736.

12. Steer J, Gibson J, Bourke SC. The DECAF Score: predicting hospital mortality in exacerbations of chronic obstructive pulmonary disease. Thorax 2012; 67: 970-976.

13. White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med 2011; 30: 377-399.

14. Toutenburg H. Rubin, D.B. Multiple imputation for nonresponse in surveys. Stat Pap 1990; 31: 180-180.

15. Snell KI, Ensor J, Debray TP, et al. Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures? Stat Methods Med Res 2018; 27: 3505-3522.

16. Chen T, Guestrin C. XGBoost: A scalable tree boosting system 2016; arXiv:1603.02754

17. Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med 2000; 19: 1141-1164

18. Echevarria C, Steer J, Heslop-Marshall K, et al. Validation of the DECAF score to predict hospital mortality in acute exacerbations of COPD. Thorax 2016; 71: 133-140.

19. Li R-H, Belford GG. Instability of decision tree classification algorithms. In: Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: Association for Computing Machinery; 2002. p. 570-575. (KDD ’02). https://doi.org/10.1145/775047.775131. Date last accessed: June 16 2021.

20. Riley RD, Snell KI, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med 2019; 38: 1276-1296.

21. Floares AG, Ferisgan M, Onita D, et al. The smallest sample size for the desired diagnosis accuracy. International Journal of Oncology and Cancer Therapy 2017; 2, 13-19.

22. Bellou V, Belbasis L, Konstantinidis AK, et al. Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: systematic review and critical appraisal. BMJ 2019; 367: 15358.

23. Vergouwe Y, Steyerberg EW, Eijkemans MJC, et al. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 2005; 58: 475-483.

24. Steyerberg EW. Validation in prediction research: the waste by data splitting. J Clin Epidemiol 2018; 103:131-133.

Table 1. Patient characteristics

Characteristics	Survivors (N = 1102)	Non-survivors (N = 88)	Total (N = 1190)
Age (years)	77 ± 8	80 ± 7	77 ± 8
Male	974 (88)	85 (97)	1059 (89)
Full support in activities of daily living	188 (17)	23 (26)	211 (18)
Altered mental status	132 (12)	39 (44)	171 (14)
Missing data	8 (1)	0 (0)	8 (1)
Systolic blood pressure (mmHg)	133 ± 26	125 ± 25	132 ± 26
Missing data	149 (14)	5 (6)	154 (13)
Diastolic blood pressure (mmHg)	75 ± 17	72 ± 16	75 ± 17
Missing data	155 (14)	5 (6)	160 (13)
Respiratory rate (breaths/minutes)	25 ± 6	27 ± 7	25 ± 7
Missing data	219 (20)	8 (9)	227 (19)
Heart rate (beats/minutes)	102 ± 19	106 ± 21	102 ± 19
Missing data	142 (13)	3 (3)	145 (12)
Blood urea nitrogen (mg/dL)	20 ± 11	30 ± 21	21 ± 12
Missing data	13 (1)	1 (1)	14 (1)
Blood eosinophil count (counts/µL)	99 ± 169	53 ± 123	96 ± 167
Missing data	409 (37)	41 (47)	450 (38)

Data are presented as mean ± standard deviation or as number (%)

Table 2. Risk scores and in-hospital mortality of BAP-65 and CURB-65

Risk scores	Patients (number, %)	In-hospital mortality (number, %)
BAP-65 Class
1	20	0 (0)
2	455	20 (4)
3	404	27 (7)
4	120	32 (27)
5	22	5 (23)
CURB-65
0	21	0 (0)
1	306	16 (5)
2	351	14 (4)
3	191	31 (16)
4	57	13 (23)
5	8	2 (25)

SupplementERJopen.docx

Predicting In-hospital Death in Pneumonic COPD exacerbation via BAP-65, CURB-65, and Machine Learning

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Methods

Results

Discussions

Conclusion

Declarations

References

Tables

Supplementary Files

Status:

Journal Publication

Version 1