Determining the Temporal Factors of Survival Associated with Brain and Nervous System Cancer Patients: A Hybrid Machine Learning Methodology

doi:10.21203/rs.3.rs-1945771/v1

Download PDF

Research Article

Determining the Temporal Factors of Survival Associated with Brain and Nervous System Cancer Patients: A Hybrid Machine Learning Methodology

https://doi.org/10.21203/rs.3.rs-1945771/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 01 Apr, 2023

Read the published version in International Journal of Healthcare Management →

Version 1

posted

You are reading this latest preprint version

Purpose.

Although different cancer types have been investigated from the perspective of biomedical sciences, machine learning-based studies have been scant, particularly in addressing the temporal impact of brain and central nervous system (BCNS) cancer survival. The present study aims to fill this gap by proposing a machine learning methodology to investigate the temporal effects of the attributes and the levels at which they are associated with BCNS cancer survival.

Methods.

Following the best practices in health analytics, the proposed methodology utilizes a variety of feature selection, data balancing, and sensitivity analysis methods to optimize the knowledge discovery process and the resultant outcomes.

Results.

The findings can potentially assist medical professionals in identifying and targeting specific subsets of features and levels of attributes associated with sharply decreasing (or increasing) survival rates; thereby implementing better treatment options to improve the survival chances of BCNS cancer patients.

Conclusion.

Although the proposed hybrid methodology is validated on a large and feature-rich BCNS cancer data set, it can be utilized to study survival prognostics of other cancer or chronic disease types.

Temporal effect

machine learning

healthcare analytics

parsimonious model

brain and other nervous system cancer

Cancer is clearly one of the most feared diseases in the world. According to the Centers for Disease Control and Prevention (CDC), cancer was the second leading cause of death in the United States in 2020, resulting in about 602,350 deaths (Centers for Desease Control and Prevention, 2022). Lung cancer, colon and rectum cancer, and pancreas cancer were the top three leading causes of cancer death, accounting for 23%, 9%, and 8% of the total cancer deaths, respectively. According to the American Cancer Society, brain and other nervous system cancers ranked among the top 11 causes of cancer-related deaths (American Cancer Society, 2021), Globally, brain and other central nervous system cancers account for about 3% of cancer, while occurring more frequently among men than women (Farmanfarma et al., 2019). According to the available estimates, 308,102 BCNS cancer cases were diagnosed globally in 2020, leading to 251,329 deaths (International Agency for Research on Cancer, 2020). In 2021, there were about 24,530 new brain and other nervous system cancer cases in the United States, and approximately 18,600 people lost their lives to the disease (National Cancer Institute, 2021).

The term “BCNS cancers” is used to describe various forms of cancers or tumors that grow in the brain or the spinal cord, which are often fatal due to their invasive nature and the tendency to be resistant to typical surgical procedures and therapies (Liu & Zong, 2012). Although brain and other central nervous system cancers are rare, they exert a significant social and economic impact on the affected individuals, their families, and the community (Australian Institute of Health and Welfare, 2017). In addition, brain and other central nervous system cancers pose a huge burden on healthcare systems due to their inherently disabling effects on the patients (GBD 2016 Brain and Other CNS Cancer Collaborators, 2019). Unfortunately, BCNS cancers can emerge at any age. Nonetheless, according to data obtained from the United States Central Brain Tumor Registry (dataset from the National Program of Cancer Registries [NPCR] and Surveillance, Epidemiology, and End Results [SEER] registries), malignant brain tumors are most prevalent among males and non-Hispanic White individuals, while benign brain tumors are most common among females and non-Hispanic Black individuals (Liu & Zong, 2012; Ostrom et al., 2019). Analysis of the SEER program data from 1978 through 2017 reveals that the BCNS cancer incidence has increased over this period, as shown in Fig. 1 (National Cancer Institute, 2021).

However, between 2008 and 2017, incidence rates for malignant BCNS cancers declined annually by 0.8% for all ages combined. Unfortunately, the rates observed among children and adolescents increased annually by 0.5–0.7% over this same period (Miller et al., 2021). Similarly, a significant increase in the number of primary brain and other central nervous system cancers was observed among the elderly over the past few decades (Maher & McKee, 2003). This upward trend has been particularly reflected in a higher burden of glioblastoma—the most common primary brain cancer. Over the past four decades, there have been a few significant advances in the prevention, early detection, and treatment of such cases. For example, the 5-year survival rate for glioblastoma patients increased from 4–7% (Miller et al., 2021).

An essential requirement for successful BCNS cancer treatment is the accurate diagnosis (Wong & Yip, 2018). As can be expected, one of the most difficult tasks in the field of medical image processing is brain tumor segmentation. This challenge arises due to the diverse appearance of these tumors and the similarity between cancerous and normal tissues (Hossain et al., 2019). Thus, it is essential to provide practitioners with tools that would allow them to extract relevant information from clinical data in order to develop treatment plans that are specific to a patient, rather than relying on personal experience, anecdotes, or population-wide risk assessments (Agrawal et al., 2012).

At present, magnetic resonance imaging (MRI) remains the most widely used diagnostic tool, even though interpreting MRI scans requires high expertise and may be affected by personal experience. Consequently, several authors have recommended the use of algorithms when interpreting imaging results (Upadhyay & Waldman, 2011; Dahab et al., 2012). For example, color-based segmentation using K-means clustering was shown to yield better results (Datta & Chakraborty, 2011).

Available studies focusing on disease conditions are typically classified under case-control and cohort studies. A case-control study provides additional information about a wide range of possible risk factors by comparing the histories of people who have and do not have brain tumors, whereas cohort studies compare the risks in people with and without certain characteristics. However, as brain tumors and other central nervous system cancers are relatively rare, most of the analytical studies have been case-control studies (Wrensch et al., 2002). Data from such sources are vital in the development of predictive models.

In recent decades, machine learning algorithms have gained considerable popularity among cancer researchers and practitioners. In studies based on historical datasets, many machine learning algorithms have been employed, including artificial neural networks (ANNs) (Delen, 2009), Bayesian networks (Choi et al., 2009), classification trees, k-nearest neighbors (KNN), and support vector machines (SVMs) (García-Laencina et al., 2015), in order to extract prognosis factors and predict patient survival. Using ANNs and logistic regression (LR) models, Lundin and colleagues were able to predict the 5-, 10-, and 15-year survival of patients with breast cancer with a high degree of accuracy (Lundin et al., 1999). Boughorbel et al. (2016) compared eight predictive models focusing on breast cancer survival over a period of two, five, eight, and eleven years. Using ANNs and LR, Simsek et al. explored the impact of several attributes that contribute to breast cancer survival over a period of one year, five years, and ten years (Simsek et al., 2020).

Although machine learning is increasingly utilized in cancer research, there are only a few articles that directly address machine learning algorithms for modeling BCNS cancer survival and the validation of those models. Most of the existing research in this domain focused on traditional statistical analysis with the aim of predicting patient survival or comparing the survival rates among BCNS patients (Fang et al., 2020; Sun et al., 2014; Zhu et al., 2020; Rosenberg et al., 2005). For example, Bohn et al. (2018) used the information retrieved from the National Cancer Institute's Surveillance Epidemiology and End Results (SEER) database for the 2010 − 2014 period to analyze the glioblastoma outcomes. To compare overall survival periods across race groups and the effect of race on 3-year survival, the authors used Kaplan-Meier and log-rank tests and Cox proportional hazards models. Furthermore, the models included attributes such as age, gender, health insurance coverage, primary site, tumor size, the extent of surgery, and year of diagnosis. Although health insurance coverage and year of diagnosis are associated with survival duration, these variables will not be helpful for practitioners assessing prognostic factors and selecting appropriate treatment options for their patients. An online calculator was developed by Senders et al. to predict survival in patients with glioblastoma using classical and machine learning algorithms (Senders et al., 2020). However, their models incorporated the attribute of Insurance to improve performance, which will not be helpful for practitioners or cancer researchers whose aim is to establish potential associations between cancer survivors.

To address the shortcomings of these investigations, in the present study, the temporal effects of attributes and levels of attributes regarding cancer survival are evaluated. Understanding the effect of predicted attributes over short- and long-term periods would be useful for estimating the contribution of various factors to BCNS cancer prognosis. To the best of our knowledge, the temporal effects of the clinical attributes on brain cancer survival have never been examined. A hybrid method based on clinical data is adopted in order to achieve the proposed goal by (a) applying balancing methods, (b) employing extended attribute selection methods, (c) conducting 5-fold stratified cross-validation, and (d) adopting two different types of classification models, Gradient Boosting (GB) and logistic regression (LR), to investigate the contributions of attributes to BCNS cancer survival over 1-, 5-, and 10-year periods.

The remainder of this paper is organized as follows. In Section 2, the overall analytic methodology used in this study, as well as the data source and preparation, are described. The results obtained from the actual data analysis are presented in Section 3. Finally, in Section 4, discussion and conclusions are provided and some thoughts on the direction of future research are offered.

2.1 Study design

The purpose of this study is to investigate the temporal effects of the attributes and the factors that are associated with BCNS survival. Figure 2 displays the study design, which comprises four steps: preprocessing, pre-modeling, prediction, and temporal effects. In the first step, after receiving approval from SEER, we installed SEER*Stat software to download the dataset. We removed all the observations with missing values and eliminated all the attributes not associated with BCNS cancer survival. Furthermore, to observe any temporal trends in the influence of independent attributes on BCNS cancer survival, three separate datasets were derived based on 1-, 5-, and 10-year survival periods. In the second step, we used a filter (ANOVA F-value), embedded (Random Forest), and wrapper (Sequential Forward Selection) method to determine the best attributes associated with the dependent variable. To reduce bias, we used 5-fold stratified cross-validation technique and Synthetic Minority Oversampling Technique (SMOTE) as the balancing technique to enhance the overall performance of the models. After modeling each of the three time periods (1-year, 5-year, 10-year), we eliminated the attributes with low performance scores. We applied multiple prediction models with the selected attributes to classify BCNS cancer survival for each dataset (1-year, 5-year, and 10-year). In the next step, we compared the models in order to define the final model and presented the final attributes for the aforementioned three time periods to observe their temporal effects on BCNS cancer survival. In addition, we specifically examined the temporal effects of the top four attributes/factors that affected survivability. The following subsections provide additional information about the study design.

2.2 Data source and preprocessing

Data for this study were retrieved from the SEER Program of the National Cancer Institute database after completing a request form and obtaining approval from the SEER Program. A total of 138,231 clinical and histological records and 185 attributes of patients that were diagnosed with brain tumors and other nervous system tumors between 1975 and 2018 were collected from 9 registries (Surveillance, Epidemiology, and End Results (SEER) Program, 1975–2018). During the first phase of the data cleaning process, we removed redundant patient-specific information that had no relevance for making a BCNS cancer prognosis or was not informative for the practitioners. Some feature values were not available for a specific year and were indicated by ‘Blank(s)’. We removed the observations which contained at least one ‘Blank(s)’. According to the SEER dictionary, a tumor's exact size (CS tumor size) ranges from 001 and 988 mm. Therefore, we removed all values beyond 988mm. In addition, we removed 888, which referred to ‘Not applicable’. We also removed from the dataset all observations not associated with BCNS cancer death. The attribute SEER cause-specific death classification indicates the cause of death from the dataset. Thus, we removed all instances of ‘Dead (missing/unknown COD)’ and ‘N/A not seq 0–59’. Similarly, for the attribute survival month (which refers to the length of time a BCNS patient has survived) we removed observations with an 'unknown' survival month.

As a part of this study, we generated three separate datasets based on 1-, 5-, and 10-year survival, and labeled the corresponding dependent attributes as Survival_OneYear, Survival_FiveYear, and Survival_TenYear, in order to observe the temporal effect of the attributes on BCNS survival. Therefore, the following piecewise function was used to generate the dependent attribute Survival_OneYear:

(1)

We repeated this procedure for 5-year and 10-year survival and thus obtained three separate datasets that contain binary flags indicating if a patient has survived longer than 12, 60, and 120 months, respectively. We removed all observations of the attribute SEER cause-specific death classification with values of ‘Alive or Dead of Other Causes’ given that the survival duration was shorter than the selected periods. For example, in the last dataset (10-year survival), if a patient dies after 120 months and the attribute ' SEER other cause of death classification' values equal to 'Alive or Dead of Other Causes', we consider it survival (survival month = 1), but if a patient dies before 120 months, we remove the entire row since we do not know whether the patient will die before 120 months because of BCNS cancer. At the end of preprocessing, 13,003 BCNS patients were identified, and 24 attributes directly related to BCNS cancer were retained. A short overview of the attributes is provided in Table 1, and a more detailed description and related information can be found at http://seer.cancer.gov/seerstat/variables/seer/ajcc-stage.

Table 1

Attributes for survival model.
Attribute	Field Description
Age	The age of the patient at the time of diagnosis of BCNS cancer
AYA site 2020	It is primarily used to evaluate data on adolescents and young adults
CS extension	Information concerning the extent of the tumor
CS Tumor size	The size of a tumor in millimeters
CS original	Refers to the version number used to code the CS fields
CS current	This refers to the version number of the CS after the input fields have been recoded
CS site specific factor	Describe any details required to stage the case, clinically relevant facts, or prognostic details
Grade	The degree to which tumor cells resemble the adjacent tissues
First malignant PI	This binary attribute indicates whether the tumor is malignant or not
Histologic ICD − 0–3	Explains the microscopic composition of cells and tissue for a particular primary
Histology-Brain	Refer to the type of tumor. For instance, glioblastoma, pineal parenchymal tumors, etc.
ICD-0-3 mal. tumors	This is a labeled version of the ICD-O-3 values for various forms of malignant tumors
ICD-0-3 hist/B	A labelled version of the ICD-O-3 values for all behaviors
Laterality	Indicates the reportable tumors that originate from a paired organ or side of the body
ICCC site	The International Classification of Childhood Cancer (ICCC) is utilized to classify childhood cancers.
Primary site	Identify the site where the primary tumor originated
Surgery	Explains why surgery was not performed
RX-surg. Prim.	Refers to a surgical procedure that involves removing and/or destroying tissue from the primary site.
RX-surg. Oth. Reg.	Refers to surgical procedure in which lymph nodes or other tissues or organs are removed from distant sites in addition to the primary site
Sequence number	Provides a description of the number and sequence of all tumors
Rare tumors	Grouping of cancer types to define clinically relevant, clinically defined rare cancers.
Summary stage	A simplified version of the tumor stage.
Total number of malignant tumors	The number of in situ/malignant cancers
Total number of benign tumors	The number of in benign/borderline cancers

2.3 Pre-modeling and prediction models

2.3.1 Attribute selection

This study aimed to find a parsimonious model by determining the most important attributes when predicting BCNS cancer patient survival. This procedure involves selecting attributes from a large set of data to remove those that are redundant. A model takes less time to train when attributes are reduced, and performance accuracy is also increased because insignificant attributes can contain noisy observations that negatively affect classification accuracy (Doraisami et al., 2008). Numerous methodologies have been proposed to deal with these issues. Some researchers choose the attributes based on their practical experience and sense of personal insight. For example, Delen et al. (2005) reduced the breast cancer dataset attributes using experts' opinion. In contrast, Serhat et al. (2020) adapted the least absolute shrinkage and selection operator (LASSO) and Genetic Algorithm (GA) to identify the most important attributes. In this study, we selected the attributes using ANOVA F-value, Random Forest, and Sequential Forward Selection method without making any assumptions about our own or experts' knowledge. In the following section, we have briefly described these three methods for selecting attributes.

The ANOVA F-value is a filter method for evaluating the degree of linearity between independent and dependent variables. Based on the ANOVA principle, the score (F) of the x-th p-gap dipeptide in the benchmark dataset can be defined as follows:

$$F=\frac{MSR\left(x\right)}{MSE\left(x\right)}$$

MSR and MSE represent the mean square of the within-sample and between-sample variation, respectively, and are defined as:

$$MSR\left(x\right)={\sum }_{i=1}^{m}\frac{{\left(\left({\sum }_{j=1}^{{n}_{i}}{h}_{ij}^{p}\left(x\right)/{n}_{i}\right)-\left({\sum }_{i=1}^{m}{\sum }_{j=1}^{{n}_{i}}{h}_{ij}^{p}\left(x\right)/{\sum }_{i=1}^{j=m}{n}_{i}\right)\right)}^{2}}{d{f}_{R}}$$

$$MSE\left(x\right)={\sum }_{i=1}^{m}{\sum }_{j=1}^{{n}_{i}}\frac{{\left(\left({h}_{ij}^{p}\left(x\right)\right)-\left({\sum }_{i=1}^{m}{\sum }_{j=1}^{{n}_{i}}{h}_{ij}\left(x\right)/{\sum }_{i=1}^{j=m}{n}_{i}\right)\right)}^{2}}{d{f}_{E}}$$

where $d{f}_{R}=m-1$ and $d{f}_{E}=N-m$. m is the number of groups; N is the total number of samples. ${h}_{ij}^{p}\left(x\right)$ denotes the frequency of the x-th p-gap dipeptide of th j-th sample in the i-th group. The number of samples in the i-th group is indicated by ${n}_{i}$ (Ding et al., 2013; Lin & Ding, 2011; Nasiri & Alavi, 2022).

Random forest is a decision tree-based method (Embedded method) of making decisions using machine learning that combines a voting process with a machine learning algorithm (Genuer et al., 2010). An essential feature of RF, when used to perform classification tasks, is its ability to calculate the significance of the attributes (Pan & Shen, 2009). Using the following formula, we calculate the importance score for the variable x_i (Deng & Runger, 2013):

$${I}_{i}=\frac{1}{{n}_{tree}}{\sum }_{\eta ϵS}G\left({x}_{i},\eta \right)$$

where S refers to a set of nodes separated by x_i in the RF with n_tree trees.

$$G\left({x}_{i},\eta \right)=g\left({x}_{i},\eta \right)-{w}_{1}g\left({x}_{i},{\eta }_{1}\right)-{w}_{2}g\left({x}_{i},{\eta }_{2}\right)$$

where ${\eta }_{1}$ and ${\eta }_{2}$ are the left and right child nodes of v respectively. The portion of instances assigned to each child node on the left and right is represented by where ${w}_{1}$ and ${w}_{2}$respectively. Again, $g\left(\eta \right)={\sum }_{k=1}^{C}{p}_{k}^{\eta }\left(1-{p}_{k}^{\eta }\right)$; where ${p}_{k}^{\eta }$ is the proportion of class-k observations at node $\eta$. Detailed information about the random forest can be found in Breiman (2001).

Sequential forward selection (SFS) is a wrapper method for selecting attributes. Initially, SFS is composed of an empty attribute set. The attribute set is updated at each iteration by adding one of the non-selected features in a manner that minimizes the mean square error (MSE). The MSE is expressed as follows:

$$MSE=\frac{1}{n}{\sum }_{j=1}^{n}\left({x}_{i}-\stackrel{-}{x}\right)$$

where x and n refer to random variables and the number of samples, respectively. The iterative process continues until the MSE tolerance is attained (Zhu et al., 2009; Yang et al., 2017).

To predict the survival of BCNS patients, we performed several classification models. For the sake of simplicity, we presented only Gradient Boosting and Logistic Regression in this study, which performed better than other approaches

2.3.2 Gradient Boosting

Gradient boosting is a machine learning technique that is primarily used for classification problems and was first proposed by Friedman (2002). Based on an ensemble of several simple decision tree models, GB constructs a predictive model (Schapire & Singer, 1999). The GB model incorporates the advantages of the decision tree model while demonstrating a higher degree of accuracy and robustness (Friedman & Meulman, 2003). It combines the K additive tree models ${(g}_{0},{g}_{1},\dots ,{g}_{K})$ to learn a predictive model (Alzamzami et al., 2020).

$$g\left(x\right)={\sum }_{k=0}^{K}{g}_{k}\left(x\right)$$

The tree ensemble model is optimized by reducing the expected generalization error, E, as follows:

$$E={\sum }_{j}^{n}{\left({t}_{j}-{p}_{j}\right)}^{2}$$

where E is a loss function, ${t}_{j}$is the target value, and ${p}_{j}$ is the predicted value. Researchers are increasingly using gradient boosting to solve classification problems in the fields of medicine, engineering, insurance, and finance (Shin, 2019; Sun et al., 2020; Carmona et al., 2019).

2.3.3 Logistic Regression

In general, logistic regression is used for data with binary outputs. An extension of linear regression is logistic regression (Hastie et al., 2001). For logistic regression of binary attributes, log of odds is defined as follows (Johnson & Wichern, 2013):

$$logit\left(f\right)=ln\left(\frac{f}{1-f}\right)$$

where

$$logit\left(f\right)={\beta }_{0}+{\sum }_{i=1}^{p}{\beta }_{i}{x}_{i}$$

solving (11) for $f$ provides

$$f\left(x\right)=\frac{\text{exp}\left({\beta }_{0}+{\sum }_{i=1}^{p}{\beta }_{i}{x}_{i}\right)}{1+{\beta }_{0}+\text{exp}{\sum }_{i=1}^{p}{\beta }_{i}{x}_{i}}$$

which is referred to as a logistic curve, where ${x}_{1},{x}_{2},\dots ,{x}_{p}$ are the independent attributes.

Using maximum likelihood estimation, coefficients ${\beta }_{i}$ can be estimated. If the probability cutoff for binary classification is 0.5, an object is considered successful if f > 0.5, otherwise, it is considered a failure.

2.3.4 Performance Metrics

This study uses commonly used metrics for models' validation. These are Accuracy, Precision, Recall, F-Scores, and AUC scores and are defined as follows (Wolff et al., 2019; Ahmad et al., 2020):

$$accuracy=\frac{TP+TN}{TP+FN+FP+TN}$$

$$precision=\frac{TP}{TP+FP}$$

$$recall=\frac{TP}{TP+FN}$$

$$F-score=2\frac{precision\times recall}{precision+recall}$$

where TP, TN, FP, and FP represent true positives, true negatives, false positives, and false negatives, respectively.

The receiver operating characteristic curve (ROC curve) represents the relationship between true-positive and false-positive rates. A ROC curve is a plot of the true positive (TP) against the false positive (FP) (Fawcett, 2004). In general, the AUC (area under the ROC curve) value is between 0.5 and 1.0; higher AUC values indicate better performance (Zhang et al., 2021). The AUC-score was mainly used in this study to evaluate the performance of the GB and LR models. Other performance metrics might be misleading because of the imbalance of classes within the datasets. However, they are presented for the purpose of demonstrating the performance of the models.

The primary objective of this study is to develop a parsimonious model while retaining its predictive power. To select the important attributes, we have employed ANOVA F-values (AFFS), Random Forest's selections (RFS), and sequential forward selections (SFS) on three datasets (1-year, 5-year, and 10-year). All of the selected or important attributes (checked signed) for these three datasets are presented in Table 2. We have employed two classification models, GB and LR, based on these sets of selected attributes for the three time-interval datasets (1-year, 5-year, and 10-year). Furthermore, because BCNS cancer patients have a higher mortality rate, the survival and death classes for the three-time intervals are disproportionate. Table 3 indicates that survival classes dominate over death classes for 5-year and 10-year datasets. Additionally, we have applied Synthetic Minority Over-Sampling Technique (SMOTE) to overcome the overfitting problem along with a five-fold stratified cross to reduce the bias related to the random sampling of the train and test data samples. SMOTE involves generating data for the minority class to help balance the classes. It uses k-nearest neighbors to interpolate new data points (Chawla et al., 2002). We only applied SMOTE to training data, so no validation data is synthetic. A five-fold stratified cross-validation procedure involves splitting the dataset into five equal folds randomly, each fold containing the same number of classes (major and minor) (May et al., 2010).

It should be noted that we have tried balancing techniques Random Under-Sampling (RUS) and SMOTE, and in almost all scenarios, SMOTE outperformed RUS. To train the model effectively, the number of observations should also increase exponentially with the number of attributes (Han & Kamber, 2006). Therefore, instead of increasing the observation of majority classes (RUS), increasing the observation of minority classes (SMOTE) may increase the model's predictive ability. To simplify the study and make it easy for the reader to understand the results, we have not provided the model output in this study.

Table 2

Selected features by AFFS, RFS, and SFS.
	1-YEAR				5-YEAR				10-YEAR
Attribute Names	AFS	R F S	AFS U RFS	SFS	A F S	R F S	AFS U RFS	SFS	A F S	R F S	AFS U RFS	S F S
Age	√	√	√	√	√	√	√	√	√	√	√	√
AYA site 2020	√		√	√	√	√	√		√	√	√	√
CS extension	√	√	√
CS Tumor size		√	√			√	√			√	√
CS original	√	√	√		√	√	√	√	√	√	√	√
CS current		√	√		√	√	√		√	√	√	√
CS site specific factor	√	√	√			√	√		√	√	√
Grade		√	√					√
First malignant PI	√		√	√	√		√	√
Histologic ICD − 0–3		√	√			√	√	√		√	√	√
Histology-Brain				√	√	√	√		√	√	√	√
ICD-0-3 mal. tumors		√	√		√	√	√		√	√	√	√
ICD-0-3 hist/B		√	√	√	√	√	√	√	√		√
Laterality		√	√	√				√
ICCC site	√		√	√	√	√	√	√	√	√	√	√
Primary site		√	√			√	√	√		√	√	√
Surgery	√	√	√	√				√				√
RX-surg. Prim.	√	√	√	√	√	√	√		√	√	√
RX-surg. Oth reg.	√	√	√		√	√	√		√	√	√
Sequence number								√				√
Rare tumors	√		√	√	√	√	√	√	√	√	√	√
Summary stage	√		√	√				√
Total number of malignant tumors												√
Total number of benign tumors				√				√

Table 3

Distribution of classes.
Classes		Temporal Intervals
	1-YEAR	5-YEAR	10-YEAR
Deaths	6,899 (53%)	11,101 (85%)	11,591 (89%)
Survival	6,104 (47%)	1,902 (15%)	1,412 (11%)

There seems to be an easily observable pattern in the AUC scores presented in Table 4 (indicating the prediction performance for the models): the AUC score of the GB and LR models increases with increasing survival periods. The AUC for all scenarios for the 1-year survival period is low compared to the other two scenarios (5-year and 10-year).

According to Kim et al. 's study, it takes at least five years for a breast cancer patient record to be marked as ‘survived’ or ‘dead’ (Kim & Shin, 2013); thus, our findings are compatible with their results. Based on the classification results in Table 4, it is clear that GB outperformed LB in all of the scenarios. Additionally, SFS outperformed AFS, RFS, and AFS U RFS in almost all cases. For the 1-year dataset, LR and AFS U RFS had the highest AUC score of 0.731 with 20 attributes; on the other hand, SFS and LR had 0.730 with only 12 attributes. The AUC scores in both cases are close; however, SFS has only chosen 12 attributes, avoiding computational complexity and the curse of dimensions.

Overall, SFS is achieving satisfactory results. As the SFS performed better, we have shaded the SFS columns in Table 2 so that readers can easily distinguish the comparison of the various set of attributes for the various cutoff points of BCNS patient survival year. Table 2 indicates that the attribute CS original was not selected as an important attribute for 1-year survival prediction, but it was selected for 5-year and 10-year survival predictions. Again, CS-current was selected only for 10-year survival prediction. Therefore, CS-current was not a significant factor for predicting 1-year and 5-year survival. Furthermore, Fig. 3 provides a better understanding of the temporal effect of some of the significant attributes for each of the three datasets. Figure 3 illustrates the relative importance of attributes estimated by the SFS method. Again, the figure presents the relative importance of attributes estimated by the SFS method. For brevity, we have only presented an attribute in Fig. 3 if at least one attribute importance estimate for each of the three-time intervals (1-year, 5-year, and 10-year) is greater than 0.02. As an example, the attribute importance's estimated for 'AYA site' for 1-year, 5-year, and 10-year are 0.122, 0.10, and 0.021, respectively. We have included the 'AYA site' in the figure since at least one attribute importance exceeds 0.02.

As shown in Fig. 3, the predicting power of "age" decreases from 36–24% from 1-year to 10-year survival. One possible cause of this phenomenon may be the increased predictive power of other attributes over time, such as 'AYA site 2020', 'CS origin', 'Histology ICD-0-3', 'Histology-Brain', 'ICCC site', 'ICD-0-3 hist/b" and 'ICD-0-3 mal. tumors". A similar trend was found by Simsek et al. (2020) for the attribute 'Age' with another type of cancer. Possibly, this can be explained by the fact that Kim et al. (2013) reported that marking patients' records of 'survival' or 'death' takes at least five years, or the impact of cancer factors decreases over a short period of time as a consequence of the medical intervention.

Table 4

Five-fold stratified cross-validation performance results for GB and LR models.
Model	Interval	Selection technique	No. of attributes	Accu.	Precision	Recall	F-score	AUC score
		AFS	12	0.746	0.746	0.746	0.746	0.744
	1-year	RFS	15	0.736	0.735	0.736	0.735	0.733
		AFS U RFS	20	0.739	0.739	0.739	0.739	0.737
		SFS	12	0.746	0.746	0.746	0.746	0.744
		AFS	12	0.891	0.919	0.891	0.899	0.886
GB	5-year	RFS	15	0.891	0.920	0.891	0.899	0.886
		AFS U RFS	16	0.891	0.919	0.891	0.899	0.884
		SFS	13	0.894	0.921	0.894	0.902	0.889
		AFS	12	0.911	0.936	0.911	0.919	0.891
	10-year	RFS	14	0.919	0.941	0.919	0.926	0.903
		AFS U RFS	15	0.919	0.941	0.919	0.926	0.903
		SFS	13	0.920	0.944	0.920	0.927	0.916
		AFS	12	0.727	0.727	0.727	0.727	0.725
	1-year	RFS	15	0.723	0.723	0.723	0.723	0.722
		AFS U RFS	20	0.733	0.733	0.733	0.733	0.731
		SFS	12	0.732	0.731	0.732	0.731	0.730
LR		AFS	12	0.871	0.911	0.871	0.882	0.871
	5-year	RFS	15	0.868	0.909	0.868	0.880	0.868
		AFS U RFS	16	0.872	0.911	0.872	0.883	0.872
		SFS	13	0.895	0.916	0.895	0.902	0.872
		AFS	12	0.905	0.938	0.905	0.915	0.904
	10-year	RFS	14	0.908	0.939	0.908	0.917	0.906
		AFS U RFS	15	0.871	0.911	0.871	0.883	0.872
		SFS	13	0.906	0.938	0.906	0.916	0.904

An additional interesting finding of this study is the effect of levels of the attributes over time. In Figs. 4 and 5, we presented the distributions of survivors and deaths at each level of the four most important attributes for predicting 1-year, 5-year, and 10-year survival. We have observed that the number of survivors during the transition from 1-year to 10-year declines sharply with increasing age. Thus, the ratio of 0–10-year survivors and deaths gradually decrease from 1-year to 5-year and then 10-year. We can even observe equal numbers of survivors and deaths in a 10-year survival dataset of patients ranging in age from 0 to 10 years. Conversely, when transitioning from a 1-year to a 10-year survival period, we have seen a sharp decline in survivors as compared to deaths of patients over the age of 51. There are about zero survivors over the age of 71 in the 10-year survival dataset.

Again, we found that some kinds of CS original, such as ‘020440’, ‘020550’, ‘020302’, and ‘020200’ have virtually no survivors following the transition from 1-year to 10-year. It should be noted that the first two digits of the CS Original refer to the major version, followed by the next two digits of minor version changes. The last two digits refer to corrections of typographical errors that do not affect coding or derivation. Please see the following link for more information http://seer.cancer.gov/seerstat/variables/seer/ajcc-stage. A similar pattern was found in AYA site 2020 attributes for the ‘Glioblastoma’ kind and in Primary site attributes for the 'Occipital Lobe' kind. Using this pattern, practitioners can determine which levels are more dangerous than others and may target those patients for additional treatment to reduce the death rate over time.

In this study, we have proposed a prediction model for brain and other nervous system cancer patients' survivability based on three different time periods. We employed two popular classification models (GB and LR) for predicting survival. In order to select important attributes for each of the time periods prior to training the model, we utilized AFS, RFS, and SFS methods. Furthermore, we adopted a balancing technique algorithm (SMOTE) to resolve the imbalance problems related to the survival and non-survival classes. After performing all the procedures, we found that the GB classification model along with the SFS attribute selection method performed the best in all scenarios.

In this study, we demonstrate that a machine learning algorithm can be used to construct a model with improved predictive power without relying on expert knowledge or experience. Thus, with this high accuracy in predicting patient survival, especially for 5-year and 10-year forecasts, individuals can adjust their lifestyle more appropriately, while physicians can plan a more effective treatment strategy. Furthermore, this study has achieved all its intended purposes, as (a) we have identified the most significant attributes for each of the three time-intervals and have determined how these characteristics change over time for BCNS cancer patients, and (b) have presented the distribution of survivors and deaths for each of the levels of attributes across three time periods, as well as identified those that are more dangerous for survival compared to others.

For this purpose, we have chosen to employ the hybrid technique of selecting attributes and popular classification models to build the most efficient model. We observed that some attributes gain predictive power, and some lose it over time; therefore, practitioners may select specific groups of attributes to concentrate on at different points in time. Again, we have found that the number of survivors for some of the levels of attributes sharply declines as time progresses, whereas other levels decrease gradually in comparison to the number of deaths. Using this characterization of the level of attributes for different periods, practitioners will be able to determine which levels are more dangerous in comparison to others and may select those patients for additional treatment to reduce their death rate over time. It is hoped that these analyses and significant findings will motivate further research into BCNS cancers. Still, it is important to note that, even though machine learning methods can yield significant findings, without the guidance of clinicians or medical researchers, these findings would be meaningless. Thus, going forward, it would be beneficial to adopt other machine learning techniques to improve the model's accuracy or to validate the model's results with different datasets.

Data Source

SEER Cancer Statistics Review. Surveillance, Epidemiology, and End Results (SEER) program (www.seer.cancer.gov) public-use data (1975—2018). National Cancer Institute, Surveillance Research Program, Cancer Statistics Branch, released April 2003. Based on the November 2020 submission. Diagnosis period 1973—2018, 9 registries.

Funding Declaration

No funding is received for this research.

Author Contribution

Gopal Nath – Conducting the analysis and writing the initial draft of the paper
Austin Coursey – Designing the study and writing the initial draft of the paper
Joseph Ekong – Conceptualizing and designing the study and validating the results
Elham Rastegari – Conducting the analysis and writing of the paper
Saptarshi Sengupta – Designing the study, validating the results, editing the paper
Asli Z. Dag – Conducting the analysis, writing the paper, and validating the results
Dursun Delen – Conceptualizing and designing the study, validating the results, editing the paper, and submitting the manuscript.
Competing Interests: The authors declare that they do not have any financial or non-financial interests that are directly or indirectly related to the work included in this manuscript and submitted for publication consideration.

Agrawal, A., Misra, S., Narayanan, R., Polepeddi, L., & Choudhary, A. (2012). Lung cancer survival prediction using ensemble data mining on SEER data. Scientific Programming, 20, 29–42. https://doi.org/10.3233/SPR-2012-0335
Ahmad, I., Yousaf, M., Yousaf, S., & Ahmad, M. (2020). Fake News Detection Using Machine Learning Ensemble Methods. Complexity, 2020, 11. https://doi.org/10.1155/2020/8885861
Alzamzami, F., Hoda, M., & Saddik, A. E. (2020). Light Gradient Boosting Machine for General Sentiment Classification on Short Texts: A Comparative Evaluation. IEEE Access, 8, 101840–101858. https://doi.org/10.1109/ACCESS.2020.2997330
American Cancer Society. (2021). Cancer Facts and Figs. 2021. Retrieved 2022, from https://www.cancer.org/research/cancer-facts-statistics/all-cancer-facts-figures/cancer-facts-figures-2021.html
Australian Institute of Health and Welfare. (2017). Brain and other central nervous system cancers. Canberra: Australian Institute of Health and Welfare.
Bohn, A., Braley, A., Vega, P., & Barengo, N. (2018). The association between race and survival in glioblastoma patients in the US: A retrospective cohort study. PLoS ONE, 13(6), 1–10. https://doi.org/10.1371/journal.pone.0198581
Boughorbel, S., Al-Ali, R., & Elkum, N. (2016). Model Comparison for Breast Cancer Prognosis Based on Clinical Data. PLoS ONE, 11(1), e0146413. https://doi.org/10.1371/journal.pone.0146413
Breiman, L. (2001). Random Forests. Machine Learning, 45, 5–32. https://doi.org/10.1023/A:1010933404324
Carmona, P., Climent, F., & Momparler, A. (2019). Predicting failure in the U.S. banking sector: An extreme gradient boosting approach. nternational Review of Economics & Finance, 61, 304–323. https://doi.org/10.1016/j.iref.2018.03.008
Centers for Desease Control and Prevention. (2022). An Update on Cancer Deaths in the United States. Retrieved 2022, from https://www.cdc.gov/cancer/dcpc/research/update-on-cancer-deaths/index.htm
Chawla, N., Bower, K., Hall, L., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 321–357. https://doi.org/10.1613/jair.953
Choi, J. P., Han, T. H., & Park, R. W. (2009). A hybrid Bayesian network model for predicting breast cancer prognosis. Journal of Korean Society of Medical Informatics, 15(1), 49–57. https://doi.org/10.4258/jksmi.2009.15.1.49
Dahab, D., Ghoniemy, S., & Selim, G. (2012). Automated Brain Tumor Detection and Identification Using Image Processing and Probabilistic Neural Network Techniques. International Journal of Image Processing and Visual Communication, 1(2), 1–8.
Datta, S., & Chakraborty, M. (2011). Brain Tumor Detection from Pre-Processed MR Images using Segmentation Techniques. Special Issue on 2nd National Conference- Computing, Communication and Sensor Network (CCSN) (pp. 1–5). IJCA.
Delen, D. (2009). Analysis of cancer data: a data mining approach. Expert Systems, 26(1) 100–112. https://doi.org/10.1111/j.1468-0394.2008.00480.x
Delen, D., Walker, G., & Kadam, A. (2005). Predicting breast cancer survivability: a comparison of three data mining methods. Artificial Intelligence in Medicine, 34(2), 113–127. https://doi.org/10.1016/j.artmed.2004.07.002
Deng, H., & Runger, G. (2013). Gene selection with guided regularized random forest. Pattern Recognition, 46(12), 3483–3489. https://doi.org/10.1016/j.patcog.2013.05.018
Ding, H., Guo, S. H., Deng, E. Z., Yuan, L. F., Guo, F. B., Huang, J.,.. . Lin, H. (2013). Prediction of Golgi-resident protein types by using feature selection technique. Chemometrics and Intelligent Laboratory Systems, 124, 9–13. http://dx.doi.org/10.1016/j.chemolab.2013.03.005
Doraisami, S., Golzari, S., Norowi, N., & Sulaiman, M. (2008). A Study on Feature Selection and Classification Techniques for Automatic Genre Classification of Traditional Malay Music. ISMIR, (pp. 331–336). Philadelphia, PA. https://doi.org/10.5281/zenodo.1415124
Fang, W., Yang, Z.-Y., Chen, T.-Y., Shen, X.-F., & Zhang, C. (2020). Ethnicity and survival in bladder cancer: a population-based study based on the SEER database. Journal of Translational Medicine, 18(1), 1–11. https://doi.org/10.1186/s12967-020-02308-w
Farmanfarma, K., Mohammadian, M., Shahabinia, Z., Hassanipour, S., & Salehiniya, H. (2019). Brain cancer in the world: an epidemiological review. World Cancer Research Journal, 1–5. http://dx.doi.org/10.32113/wcrj_20197_1356
Fawcett, T. (2004). ROC graphs: notes and practical considerations for researchers. Machine Learning, 31, 1–38.
Friedman, J. H. (2002). Stochastic gradient boosting. Computational Statistics and Data Analysis, 38(4), 367–378. https://doi.org/10.1016/S0167-9473(01)00065-2
Friedman, J. H., & Meulman, J. J. (2003). Multiple additive regression trees with application in epidemiology. Statistics in Medicine, 22(9), 1365–1381. https://doi.org/10.1002/sim.1501
García-Laencina, P. J., Abreu, P. H., Abreu, M. H., & Afonoso, N. (2015). Missing data imputation on the 5-year survival prediction of breast cancer patients with unknown discrete values. Computers in Biology and Medicine, 59, 125–133. https://doi.org/10.1016/j.compbiomed.2015.02.006
GBD 2016 Brain and Other CNS Cancer Collaborators. (2019). Global, regional, and national burden of brain and other CNS cancer, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. The Lancet Neurology, 18(4), 376–393. https://doi.org/10.1016/S1474-4422(18)30468-X
Genuer, R., Poggi, M., & Tuleau-Malot, C. (2010). Variable selection using random forests. Pattern Recognition Letters, 31(14), 2225–2236. https://doi.org/10.1016/j.patrec.2010.03.014
Han, J., & Kamber, M. (2006). Data mining: concepts and techniques. Waltham, MA, USA: Elsevier.
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The Elements of Statistical Learning. New York: Springer.
Hossain, T., Shishir, F. S., Ashraf, M., Al Nasim, M. A., & Muhammad Shah, F. (2019). Brain Tumor Detection Using Convolutional Neural Network. 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT) (pp. 1–6). IEEE. https://doi.org/10.1109/ICASERT.2019.8934561
International Agency for Research on Cancer. (2020). Cancer Today. Retrieved 2022, from https://gco.iarc.fr/today/data/factsheets/cancers/31-Brain-central-nervous-system-fact-sheet.pdf
Johnson, R. A., & Wichern, D. W. (2013). Applied multivariate statistical analysis. London: Pearson Education, Inc.
Kim, J., & Shin, H. (2013). Breast cancer survivability prediction using labeled, unlabeled, and pseudo-labeled patient data. Journal of the American Medical Informatics Association, 20, 613–618. https://doi.org/10.1136/amiajnl-2012-001570
Lin, H., & Ding, H. (2011). Predicting ion channels and their types by the dipeptide mode of pseudo amino acid composition. Journal of Theoretical Biology, 269, 64–69. https://doi.org/10.1016/j.jtbi.2010.10.019
Liu, C., & Zong, H. (2012). Developmental origins of brain tumors. Current Opinion in Neurobiology, 22(5), 844–849. https://doi.org/10.1016/j.conb.2012.04.012
Lundin, M., Lundin, J., Burke, H., Toikkanen, L., & Joensuu, H. (1999). Artificial Neural Networks Applied to Survival Prediction in Breast Cancer. Oncology, 57, 281–286. https://doi.org/10.1159/000012061
Maher, E. A., & McKee, A. C. (2003). Atlas of Diagnostic Oncology (Vol. 3). (A. T. Skarin, & G. P. Canellos, Eds.) London, United Kingdom: Elsevier Science Ltd.
May, R. J., Maier, H. R., & Dandy, G. C. (2010). Data splitting for artificial neural networks using SOM-based stratified sampling. Neural Networks, 23(2), 283–294. https://doi.org/10.1016/j.neunet.2009.11.009
Miller, K. D., Ostrom, Q. T., Kruchko, C., Patil, N., Tihan, T., Cioffi, G.,.. . Barnholtz-Sloan, J. S. (2021). Brain and other central nervous system tumor statistics, 2021. CA: A Cancer Journal for Clinicians, 71, 381–406. https://doi.org/10.3322/caac.21693
Nasiri, H., & Alavi, S. (2022). A Novel Framework Based on Deep Learning and ANOVA Feature Selection Method for Diagnosis of COVID-19 Cases from Chest X-Ray Images. Computational Intelligence and Neuroscience, 2022, 11. https://doi.org/10.1155/2022/4694567
National Cancer Institute. (2021). Cancer Stat Facts: Brain and Other Nervous System Cancer. Retrieved 2022, from Surveillance, Epidemiology, and End Results (SEER) Program: https://seer.cancer.gov/statfacts/html/brain.html
Ostrom, Q., Cioffi, G., Gittleman, H., Patil, N., Waite, K., Kruchko, C., & Barnholtz-Sloan, J. (2019). CBTRUS Statistical Report: Primary Brain and Other Central Nervous System Tumors Diagnosed in the United States in 2012–2016. Neuro Oncology, 21, 1–100. https://doi.org/10.1093/neuonc/noz150
Pan, X., & Shen, H. (2009). Robust Prediction of B-Factor Profile from Sequence Using Two-Stage SVR Based on Random Forest Feature Selection. Protein and Peptide Letters, 16(12), 1447–1454. https://doi.org/10.2174/092986609789839250
Rosenberg, J., Chia, L., & Plevritis, S. (2005). The effect of age, race, tumor size, tumor grade, and disease stage on invasive ductal breast cancer survival in the u.s. SEER database. Breast Cancer Research and Treatment, 89, 47–54. https://doi.org/10.1007/s10549-004-1470-1
Schapire, R. E., & Singer, Y. (1999). Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37(3), 297–336. https://doi.org/10.1023/A:1007614523901
Senders, J. T., Staples, P., Mehrtash, A., Cote, D. J., Taphoorn, M., Reardon, D.,.. . Arnaout, O. (2020). An Online Calculator for the Prediction of Survival in Glioblastoma Patients Using Classical Statistics and Machine Learning. Neurosurgery, 86(2), 184–192. https://doi.org/10.1093/neuros/nyz403
Shin, Y. (2019). Application of Stochastic Gradient Boosting Approach to Early Prediction of Safety Accidents at Construction Site. Advances in Civil Engineering, 2019, 9. https://doi.org/10.1155/2019/1574297
Simsek, S., Kursuncu, U., Kibis, E., AnisAbdellatif, M., & Dag, A. (2020). A hybrid data mining approach for identifying the temporal effects of variables associated with breast cancer survival. Expert Systems with Applications, 139, 112863. https://doi.org/10.1016/j.eswa.2019.112863
Sun, H., Ma, H., Hong, G., Sun, H., & Wang, J. (2014). Survival improvement in patients with pancreatic cancer by decade: A period analysis of the SEER database. Scientific Reports, 4(1), 1–10. https://doi.org/10.1038/srep06747
Sun, R., Wang, G., Zhang, W., Hsu, L.-T., & Ochieng, W. (2020). A gradient boosting decision tree based GPS signal reception classification algorithm. Applied Soft Computing, 86. https://doi.org/10.1016/j.asoc.2019.105942
Surveillance, Epidemiology, and End Results (SEER) Program. (1975–2018). National Cancer Institute. Retrieved 2022, from Surveillance, Epidemiology, and End Results (SEER) Program: https://seer.cancer.gov/
Upadhyay, N., & Waldman, A. D. (2011). Conventional MRI evaluation of gliomas. The British journal of radiology, 84, 107–111. https://doi.org/10.1259/bjr/65711810
Wolff, P., Graña, M., Ríos, S., & Yarza, M. (2019). Machine Learning Readmission Risk Modeling: A Pediatric Case Study. BioMed research international, 2019, 9. https://doi.org/10.1155/2019/8532892
Wong, D., & Yip, S. (2018). Machine learning classifies cancer. Nature, 555, 469–474. https://doi.org/10.1038/d41586-018-02881-7
Wrensch, M., Minn, Y., Chew, T., Bondy, M., & Berger, M. S. (2002). Epidemiology of primary brain tumors: current concepts and review of the literature. Neuro Oncol, 4(4), 278–299. https://doi.org/10.1093/neuonc/4.4.278
Yang, F., Hamit, M., Yan, C. B., Yao, J., Kutluk, A., Kong, X. M., & Zhang, S. X. (2017). Feature Extraction and Classification on Esophageal X-Ray Images of Xinjiang Kazak Nationality. Data Mining for Biomedicine and Healthcare, 2017, 11. https://doi.org/10.1155/2017/4620732
Zhang, K., Su, H., & Dou, Y. (2021). Beyond AP: a new evaluation index for multiclass classi cation task accuracy. Applied Intelligence, 51, 7166–7176. https://doi.org/10.1007/s10489-021-02223-7
Zhu, J., Lu, W., Liu, L., & Niu, B. (2009). Classification of Src kinase inhibitors based on support vector machine. QSAR and Combinatorial Science, 28(6), 719–727. https://doi.org/10.1002/qsar.200860105
Zhu, L., Sun, X., & Bai, W. (2020). Nomograms for predicting cancer specific and over all survival among patients with endometrial carcinoma: A SEER based study. Frontiers in Oncology, 10, 1–12. https://doi.org/10.3389/fonc.2020.00269

No competing interests reported.

Download PDF

Journal Publication

published 01 Apr, 2023

Read the published version in International Journal of Healthcare Management →

Version 1

posted

You are reading this latest preprint version

Determining the Temporal Factors of Survival Associated with Brain and Nervous System Cancer Patients: A Hybrid Machine Learning Methodology

Status:

Journal Publication

Version 1

Abstract

Purpose.

Methods.

Results.

Conclusion.

Figures

1. Introduction And Background

2. Methodology

2.1 Study design

2.2 Data source and preprocessing

2.3 Pre-modeling and prediction models

2.3.1 Attribute selection

2.3.2 Gradient Boosting

2.3.3 Logistic Regression

2.3.4 Performance Metrics

3. Results And Discussion

4. Summary And Conclusion

Declarations

References

Additional Declarations

Status:

Journal Publication

Version 1