Clinical Application of Machine Learning in the Assessment of Pulmonary Embolism in Patients with Gastrointestinal Cancer

doi:10.21203/rs.3.rs-1635342/v1

Download PDF

Research Article

Clinical Application of Machine Learning in the Assessment of Pulmonary Embolism in Patients with Gastrointestinal Cancer

https://doi.org/10.21203/rs.3.rs-1635342/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Background: Pulmonary thromboembolism (PTE) is one of the most important complications in cancer patients. Gastrointestinal cancers are at increased risk of PTE. However, there were few studies that predict pulmonary embolism using machine learning (ML) in cancer patients. The purpose of this study was to develop an ML based prediction model for PTE in gastrointestinal cancer patients, and to compare its performance with the conventional model.

Methods: In a tertiary hospital, patients who underwent computed tomographic pulmonary angiography (CTPA) were reviewed retrospectively from 2010 to 2020. Demographic and predictor variables including the Wells score and D-dimer were investigated. ML was based on the random forest model. In the model comparison, the area under receiver operating curve (AUROC) was used to evaluate the predictive performance of each model.

Results: A total of 446 gastrointestinal cancer patients were analyzed in this study. The overall incidence of PTE was 30.0%. PTE was the most common in pancreatic cancer (47.2%). Compared with the conventional model (AUROC 0.605), the performance of ML model predicting PTE was improved (0.706, P = 0.002) and was further improved with additional input of further demographic factors including age and sex (0.743, P < 0.001). The number of patients classified as requiring CTPA was significantly reduced according to the prediction with ML (1.8% vs 9.4%, P< 0.001).

Conclusion: Prediction model based on ML might have advantages to improve the diagnostic performance and reduce the number of CTPA compared to the conventional model for PTE in patients with gastrointestinal cancer.

Pulmonary thromboembolism

machine learning

gastrointestinal cancer

Venous thromboembolism (VTE) occurs in 15%-20% of cancer patients (1). In the cancer populations, VTE increases the risk of death (2, 3). Epidemiologic studies reported the highest risk of VTE in intraabdominal cancers including gastrointestinal cancers (4-6). Among VTEs, pulmonary thromboembolism (PTE) is a clinically important disease requiring urgent management. Early detection is important because massive PTE can cause cardiac arrest or circulatory collapse, associated with high mortality rates (7).

Computed tomography pulmonary angiography (CTPA) is the method of choice for the evaluation of pulmonary vasculature in patients with suspected PTE (8). However, CTPA is not appropriate for primary test for PTE because only 15%-25% of presenting patients have PTE (8, 9). Unnecessary CTPA should be avoided because of radiation exposure, side effects of contrast media, and cost.

Many studies have developed diagnostic strategies for PTE. These diagnostic strategies identify low-risk patients for PTE in whom imaging studies and anticoagulation therapy can be safely withheld (9). Assessments of clinical pretest probability (C-PTP) such as the Wells score is most often used in combination with the D-dimer test. It is well established that patients with low C-PTP and D-dimer levels of less than 500ng per milliliter are considered PTE ruled out (8-10). However, these patients account for only approximately 30% of outpatients (11).

There have been studies to reduce unnecessary CTPA in patients with suspected PTE. Several studies attempted to change the cut-off value of the D-dimer test (11, 12). Recently, there have been studies to improve the diagnosis of PTE using artificial intelligence. In one study, machine learning (ML) was used to support clinical decisions for CTPA in moderate to high C-PTP patients (13). Another study conducted risk-stratification for deep vein thrombosis (DVT) by ML using the Wells score and D-dimer (14). However, there are few studies using ML to support the decision on CTPA in cancer patients with suspected PTE.

There is a limit to applying the existing diagnostic methods for PTE to cancer patients. This is because cancer patients usually have high D-dimer level, which can result in false positive (15). Additionally, PTE is relatively common in gastrointestinal cancers (4-6). Therefore, our purpose was to reduce unnecessary CTPA by improving diagnostic strategy for PTE in gastrointestinal cancer patients through ML.

Study design and patients

This retrospective study investigated the patients with gastrointestinal cancer who underwent CTPA at a tertiary center (Seoul National University Hospital, Seoul, Korea) between 2010 and 2020, using an electrical medical records database (Figure 1). The diagnosis of gastrointestinal cancer was confirmed by the pathological results. Hepatocellular carcinoma diagnosed based on imaging without pathological confirmation was also included (16). The patients with exclusion criteria were not included in the analysis. Exclusion criteria were as follow; suspected PTE associated with causes other than cancer or other malignancy, no evidence of malignancy at the time of CTPA, ambiguous diagnosis of PTE on CTPA, missing data. Diagnosis of PTE was confirmed by trained experts based on CTPA.

This study was approved by the institutional review board (IRB) of the Seoul National University Hospital, Korea (IRB No.2009-146-1159). The need for informed consent was waived by the IRB.

Data collection and definition

Patient characteristics were retrospectively collected including age, sex, cancer diagnosis, the Wells score and components of the Wells score, and D-dimer. Gastrointestinal cancers were defined as cancers of gastrointestinal tract from esophagus to anus, cancers of liver and pancreatobiliary system (17). The Wells score was calculated as previously used in patients suspected PTE (18). Components of the Wells score included signs and symptoms of VTE, alternative diagnosis less likely than pulmonary embolism, heart rate > 100 beats per minute, history of VTE, immobilization, malignancy, hemoptysis. Among them, we did not include malignancy because all of the subjects were diagnosed with cancer. History of VTE was defined as previous history of PTE or DVT(18). A low C-PTP was defined as a Wells score of 0 to 1.5, a moderate C-PTP was 2.0 to 6.0 and a high C-PTP was 6.5 or higher (18). Number needed to diagnosis (NND) was defined as the number of patients who need to be examined to diagnosis one patient with the disease. NND is the inverse of Youn index (sensitivity + specificity – 1) (19).

Study outcome measures

The primary outcome of this study was the area under the receiver operating characteristics curve (AUROC) and accuracy of ML model for diagnosis of PTE in patients with gastrointestinal cancer.

As a secondary outcome, we compared the number of performed CTPA for PTE in the ML model with the conventional model. We also investigated NND and feature importance of the ML model.

Statistical analysis

To compare the baseline characteristics, the Student’s t-test and Chi-square test were used for continuous and dichotomous variables, respectively. If any subgroups had less than four subjects, the Fisher’s exact test was used instead of a Chi-square test.

ML was performed in a 10-fold cross-validation method. We classified the subjects as 90% training group and 10% validation group. The model is trained using demographic information and labels from the training group, and prediction is performed based on the demographic information of the validation group. After that, the model parameters are reset, and training and prediction are performed on the newly split data. This process is repeated 10 times, while the data split is performed by setting the entire subject to be included in the validation group only once. As a result, the model makes predictions once for every subject, which can increase data efficiency.

The random forest model showed good performance in regression and classification, especially in the healthcare field, where deep learning is not accessible due to the lack of data (20, 21). Although PTE is an important disease, the number is relatively small, so we used the random forest model (22). The random forest model can track which features the model mainly considered in the process of making decisions. This is a form of explainable artificial intelligence, which can solve the model reliability problem that most deep learning algorithms currently have. In this paper, we used impurity-based feature importance (23). The feature importance score is calculated by the ratio of the information gain of all nodes split by a specific feature divided by the total sum of the information gains of all nodes. Consequently, using this importance score, we determined how each feature is influenced for classifying the data.

P-value lower than 0.05 indicated statistical significance. Statistical calculations were performed with SPSS and scikit-learn's random forest model. To measure the feature importance, the impurity-based feature importance algorithm was used. When comparing prediction ROC curves between models, statistical analysis based on the Delong test was performed.

Baseline characteristics of the study population

Of the 708 patients diagnosed with gastrointestinal cancer, 262 were excluded because of the exclusion criteria thus, a total of 446 patients were analyzed (Fig. 1). Among them, 134 (30.0%) patients were diagnosed with PTE. Table 1 shows the comparison of baseline characteristics according to PTE in the gastrointestinal cancer patients. Pancreatic cancer was the most common type of cancer in patients with PTE (47.2%). Patients with PTE had more high C-PTP (24.0% vs. 43.3%) and patients without PTE had more moderate C-PTP (62.8% vs. 45.5%, P < 0.001). Patients suspected with DVT was higher in patients with PTE (27.9% vs. 47.8%, P < 0.001). History of VTE (3.2% vs. 17.2%, P < 0.001) and hemoptysis (0.3% vs. 3.0%, P = 0.0495) were more common in patients with PTE. There was no significant difference between the two groups in other variables.

Table 1

Baseline characteristics of patients according to the presence of pulmonary thromboembolism.
	All patients	Patients without PTE	Patients with PTE
	(n = 446)	(n = 312)	(n = 134)	P value
Age (mean)	66.3\(\pm\)11.4	66.6\(\pm\)11.1	65.6\(\pm\)12.3	0.385
Male (%)	306 (68.6)	221 (70.8)	85 (63.4)	0.123
Cancer (%)				0.001
Gastric cancer	88 (19.7)	72 (81.8)	16 (18.2)
Colon cancer	92 (20.6)	62 (67.4)	30 (32.6)
Hepatocellular carcinoma	106 (23.8)	80 (75.5)	26 (24.5)
Pancreatic cancer	89 (20.0)	47 (52.8)	42 (47.2)
Cholangiocarcinoma	46 (10.3)	30 (65.2)	16 (34.8)
Others	25 (5.6)	21 (84.0)	4 (16.0)
Components of Wells score (%)
Signs and symptoms of DVT	151 (33.9)	87 (27.9)	64 (47.8)	< 0.001
Alternative diagnosis less likely than PTE	215 (48.2)	143 (45.8)	72 (53.7)	0.1453
Heart rate > 100/min	207 (46.4)	141 (45.2)	66 (49.3)	0.4755
Immobilization	171 (38.3)	116 (37.2)	55 (41.0)	0.4915
History of VTE	33 (7.4)	10 (3.2)	23 (17.2)	< 0.001
Hemoptysis	5 (1.1)	1 (0.3)	4 (3.0)	0.0495
Wells score (%)				< 0.001
Low	56 (12.6)	41 (13.1)	15 (11.2)
Moderate	257 (57.6)	196 (62.8)	61 (45.5)
High	133 (29.8)	75 (24.0)	58 (43.3)
D-dimer \(\ge\) 500 ng/mL (%)	437 (98.0)	304 (97.4)	133 (99.3)	0.211
DVT, deep vein thrombosis; PTE, pulmonary thromboembolism; VTE, venous thromboembolism

Primary study outcomes

The results are summarized as AUROC in Table 2 and Fig. 2. AUROC and accuracy of the conventional model using combination of the Wells score and D-dimer was 0.6049 and 0.6053. AUROC (0.6049 vs. 0.7057, P < 0.001) and accuracy (0.6053 vs. 0.6906, P < 0.001) of the ML model using component of the Wells score and D-dimer was significantly improved. And the performance of the ML model was further improved with additional input of further demographic factors including age and sex (AUROC, 0.6049 vs. 0.7433, P < 0.001; accuracy 0.6053 vs. 0.7534, P < 0.001).

Table 2

Comparison of performance between the logistic regression approach and machine learning.
Model	Conventional model	Machine learning model
Feature	Wells score + D-dimer	WC + D-dimer	Age + Sex + WC + D-dimer
AUROC	0.6049	0.7057*	0.7433*
Accuracy	0.6053	0.6906*	0.7534*
Sensitivity	0.5373	0.5448	0.6343
Specificity	0.6346	0.7532	0.8045
Precision	0.3470	0.4019	0.4792
Recall	0.5860	0.6490	0.7194
AUROC, area under reciever operating curve; WC, components of Wells score
^* Statistically significant difference when compared with the conventional model (p-value < 0.05)

Secondary study outcomes

Figure 3 shows the feature importance of the ML models. D-dimer and heart rate were the most important variables in predicting PTE. Age factor was revealed as one of important factors in the ML model with additional input of further demographic factors in predicting PTE.

Figure 4 indicate that the number of patients classified as requiring CTPA during the diagnosis of PTE was significantly reduced according to the prediction with the ML model while maintaining the accuracy of diagnosis. (1.8% vs. 9.4%, P < 0.001). In the ML model, an additional 34 patients avoided unnecessary CTPA. And NND was also improved using the ML model. (5.8 vs. 2.3)

In gastrointestinal cancer patients, PTE is one of the most important complications requiring early diagnosis and treatment. However, the current diagnostic strategies showed the unsatisfactory diagnostic performance of PTE detection and result in the unnecessary CT scans. In this study, the ML model significantly improved the diagnostic performance of PTE and reduced the unnecessary CTPA scans.

A recent study showed that the neural network model using raw structured EMR data can predict the risk of PTE (13). However, since they analyzed all patients suspected with PTE, there will be limitations in establishing a specific diagnostic strategy for PTE in cancer patients. They differ from this study by using a predicting model for only moderate to high C-PTP. In this study, we investigated patients with gastrointestinal cancer. Among patients diagnosed as cancer in South Korea in 2018, gastric cancer and colorectal cancer were the most common, and gastrointestinal cancer accounted for 36% of all cancer patients (24). Based on this study, the ML model could reduce unnecessary CTPA in thousands of gastrointestinal cancer patients per year in South Korea. Considering that CTPA costs approximately 300 US dollars in South Korea, it can significantly reduce medical expenses. And it can also reduce medical burden by reducing contrast media induced complications such as nephropathy and anaphylaxis.

In this study, pancreatic cancer was the most common type of cancer in patients with PTE. Previous studies reported that pancreatic cancer has the highest risk of VTE among all cancers (6, 25). Signs and symptoms of DVT, history of VTE, hemoptysis are factors known to be related to PTE in previous studies (8, 18, 26). However, although immobilization is a strong risk factor for PTE(18), there was no significant difference in immobilization according to PTE in this study. We analyzed cancer patients, and a bias may have occurred because cancer is a risk factor for PTE(18). And cancer may have affected the performance status of patients. In this study, the Wells score could not discriminate PTE, especially in the moderate C-PTP. On the Wells score, cancer corresponds to 1 score. Because we analyzed cancer patients, all subjects had at least a Wells score of 1 or higher.

Many studies have attempted to increase diagnostic rate by changing the cut-off value of D-dimer. One study increased the diagnostic rate by changing the cut-off value of D-dimer according to age (27). Another study ruled out more patients suspected with PTE by increasing the cut-off value of D-dimer to 1000 ng per milliliter in the low probability group in the Wells score (11). Based on these results, a change in the cut-off value of the D-dimer test could improve the diagnostic rate for PTE. And it is even more necessary in cancer patients with high false positive rate of the D-dimer test. In this study, we did not use the previously known cut-off value for D-dimer, but let the ML model adjust the cut-off values according to other variables. So we improved diagnostic performance and reduce the number of CTPA.

D-dimer was also the important factor in feature importance. This suggests that adjusting the cut-off value of D-dimer using ML was important for prediction of PTE. Additionally, age and tachycardia were important factors in feature importance. Their association with PTE has been reported in previous studies (18, 28).

This study has several limitations. First, since it is a retrospective study, unexpected bias may exist. Because we analyzed patients who had undergone CTPA, there may be a selective bias. Second, AUROC, sensitivity, and specificity of the ML model are relatively low compared to other studies (13, 27). This is because the subjects of this study are cancer patients. Cancer patients have a higher D-dimer level and are more likely to be false-positive (15). We used ML to overcome these limitations of D-dimer. However, since most artificial intelligence such as ML is a black box model, the cut-off value of D-dimer in the ML model is unknown. Nevertheless, we showed the importance of D-dimer using feature importance, an explainable artificial intelligence. Third, the number of patients investigated in this study is relatively small. This is because PTE is relatively rare disease with an annual incidence of about 0.1% (22). And other studies have developed ML models with good performance despite the small sample size (29, 30).

Despite these limitations, this study has several strengths. To the best of our knowledge, this study is the first to develop the ML model to predict PTE in gastrointestinal cancer patients. We applied ML to the simplified model that has already been demonstrated. So it is more easily accessible in the emergent department. And since the weight of each variable in the Wells score and the cut-off value of D-dimer are adjusted in the ML model, the limitations of the existing diagnostic system can be overcome. Therefore, a large prospective study in the future would be warranted to verify these results.

In conclusion, this ML model for predicting PTE might improve diagnostic strategies for PTE and reduce the number of unnecessary CTPA during the diagnostic process of PTE in gastrointestinal cancer patients.

PTE: Pulmonary thromboembolism; ML: Machine learning; CTPA : Computed tomographic pulmonary angiography; VTE: Venous thromboembolism; C-PTP: Clinical pretest probability; DVT: Deep vein thrombosis; NND: Number needed to diagnosis; AUROC: Area under the receiver operating characteristics curve

Ethics approval and consent to participate

This research was carried out in accordance with the Declaration of Helsinki. This study was approved by the institutional review board (IRB) of the Seoul National University Hospital, Korea (IRB No.2009-146-1159). The need for informed consent was waived by the IRB.

Consent for publication

Not applicable.

Availability of data and materials

Data are available from the corresponding author upon reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This study was supported by Seoul National University Hospital Research Fund (Grant Number 0420212090).

Authors’ contributions

J.S.K., S.H.L. and S-B.L. devised and designed the study. D.K., M.W.L., N.P. and J.H.C. collected data and analyzed data. J.S.K. and K.K. wrote the manuscript. K.K., I.R.C., W.H.P, J.K.L., J.K.R. and Y-T.K. edited the manuscript. S.H.L. takes full responsibility for the study. All authors have read and approved the manuscript.

Acknowledgment

Not applicable

Blom JW, Vanderschoot JP, Oostindiër MJ, et al. Incidence of venous thrombosis in a large cohort of 66,329 cancer patients: results of a record linkage study. Journal of thrombosis and haemostasis: JTH. 2006;4:529–35.
Gross CP, Galusha DH, Krumholz HM. The impact of venous thromboembolism on risk of death or hemorrhage in older cancer patients. Journal of general internal medicine. 2007;22:321–6.
Sørensen HT, Mellemkjaer L, Olsen JH, et al. Prognosis of cancers associated with venous thromboembolism. The New England journal of medicine. 2000;343:1846–50.
Ohashi Y, Ikeda M, Kunitoh H, et al. Venous thromboembolism in cancer patients: report of baseline data from the multicentre, prospective Cancer-VTE Registry. Japanese journal of clinical oncology. 2020;50:1246–53.
Mulder FI, Horváth-Puhó E, van Es N, et al. Venous thromboembolism in cancer patients: a population-based cohort study. Blood. 2021;137:1959–69.
Cohen AT, Katholing A, Rietbrock S, et al. Epidemiology of first and recurrent venous thromboembolism in patients with active cancer. A population-based cohort study. Thrombosis and haemostasis. 2017;117:57–65.
Mukai M, Oka T. Mechanism and management of cancer-associated thrombosis. J Cardiol. 2018;72:89–93.
Konstantinides SV, Meyer G, Becattini C, et al. 2019 ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS): The Task Force for the diagnosis and management of acute pulmonary embolism of the European Society of Cardiology (ESC). The European respiratory journal. 2019;54.
van Es N, van der Hulle T, van Es J, et al. Wells Rule and d-Dimer Testing to Rule Out Pulmonary Embolism: A Systematic Review and Individual-Patient Data Meta-analysis. Annals of internal medicine. 2016;165:253–61.
Hendriksen JM, Geersing GJ, Lucassen WA, et al. Diagnostic prediction models for suspected pulmonary embolism: systematic review and independent external validation in primary care. BMJ (Clinical research ed). 2015;351:h4438.
Kearon C, de Wit K, Parpia S, et al. Diagnosis of Pulmonary Embolism with d-Dimer Adjusted to Clinical Probability. The New England journal of medicine. 2019;381:2125–34.
Takach Lapner S, Julian JA, Linkins LA, et al. Comparison of clinical probability-adjusted D-dimer and age-adjusted D-dimer interpretation to exclude venous thromboembolism. Thrombosis and haemostasis. 2017;117:1937–43.
Banerjee I, Sofela M, Yang J, et al. Development and Performance of the Pulmonary Embolism Result Forecast Model (PERFORM) for Computed Tomography Clinical Decision Support. JAMA network open. 2019;2:e198719.
Willan J, Katz H, Keeling D. The use of artificial neural network analysis can improve the risk-stratification of patients presenting with suspected deep vein thrombosis. British journal of haematology. 2019;185:289–96.
Crawford F, Andras A, Welch K, et al. D-dimer test for excluding the diagnosis of pulmonary embolism. The Cochrane database of systematic reviews. 2016;2016:Cd010864.
Bruix J, Sherman M. Management of hepatocellular carcinoma: an update. Hepatology (Baltimore, Md). 2011;53:1020–2.
Lee K, Lee JS, Kim J, et al. Oral health and gastrointestinal cancer: A nationwide cohort study. Journal of clinical periodontology. 2020;47:796–808.
Di Nisio M, van Es N, Büller HR. Deep vein thrombosis and pulmonary embolism. Lancet (London, England). 2016;388:3060–73.
Larner AJ. Number Needed to Diagnose, Predict, or Misdiagnose: Useful Metrics for Non-Canonical Signs of Cognitive Status? Dementia and geriatric cognitive disorders extra. 2018;8:321-7.
Pourghasemi HR, Pouyan S, Heidari B, et al. Spatial modeling, risk mapping, change detection, and outbreak trend analysis of coronavirus (COVID-19) in Iran (days between February 19 and June 14, 2020). International Journal of Infectious Diseases. 2020;98:90–108.
Kaur P, Kumar R, Kumar M. A healthcare monitoring system using random forest and internet of things (IoT). Multimedia Tools and Applications. 2019;78:19905–16.
Keller K, Hobohm L, Ebner M, et al. Trends in thrombolytic treatment and outcomes of acute pulmonary embolism in Germany. European heart journal. 2020;41:522–9.
Nembrini S, König IR, Wright MN. The revival of the Gini importance? Bioinformatics (Oxford, England). 2018;34:3711-8.
Korea Central Cancer Registry NCC. Annual report of cancer statistics in Korea in 2018. Ministry of Health and Welfare. 2020.
Frere C, Bournet B, Gourgou S, et al. Incidence of Venous Thromboembolism in Patients With Newly Diagnosed Pancreatic Cancer and Factors Associated With Outcomes. Gastroenterology. 2020;158:1346–58.e4.
Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to categorize patients probability of pulmonary embolism: increasing the models utility with the SimpliRED D-dimer. Thrombosis and haemostasis. 2000;83:416–20.
Nagel SN, Steffen IG, Schwartz S, et al. Age-dependent diagnostic accuracy of clinical scoring systems and D-dimer levels in the diagnosis of pulmonary embolism with computed tomography pulmonary angiography (CTPA). European radiology. 2019;29:4563–71.
Keller K, Beule J, Coldewey M, et al. Heart rate in pulmonary embolism. Internal and emergency medicine. 2015;10:663–9.
Lu Z, Chen H, Jiao X, et al. Prediction of immune checkpoint inhibition with immune oncology-related gene expression in gastrointestinal cancer using a machine learning classifier. Journal for immunotherapy of cancer. 2020;8.
Yokoyama S, Hamada T, Higashi M, et al. Predicted Prognosis of Patients with Pancreatic Cancer by Machine Learning. Clinical cancer research: an official journal of the American Association for Cancer Research. 2020;26:2411–21.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Clinical Application of Machine Learning in the Assessment of Pulmonary Embolism in Patients with Gastrointestinal Cancer

Status:

Version 1

Abstract

Figures

Introduction

Methods

Results

Baseline characteristics of the study population

Primary study outcomes

Secondary study outcomes

Discussion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1