Repeatability of Radiomics Studies in Colorectal Cancer: A Systematic Review

doi:10.21203/rs.3.rs-927114/v1

Download PDF

Research Article

Repeatability of Radiomics Studies in Colorectal Cancer: A Systematic Review

https://doi.org/10.21203/rs.3.rs-927114/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 14 Apr, 2023

Read the published version in BMC Gastroenterology →

You are reading this latest preprint version

Purpose: Recently, radiomics has been widely used in colorectal cancer, but many variable factors affect the repeatability of radiomics research. This review aims to analyze the repeatability of radiomics studies in colorectal cancer and to evaluate the current status of radiomics in the field of colorectal cancer.

Methods: A total of 68 eligible studies was included in this review by searching from the PubMed and EmBase databases. Then each study in our review was evaluated using the Radiomics Quality Score (RQS). We analyzed the factors that may affect the repeatability in the radiomics workflow and discussed the repeatability of the included researches.

Results: A total of 68 articles was included in this review, of which only 2 (3%) articles controlled the influence of individual factors, just 2 (3%) articles considered the variability between scanners. In addition, the median score of RQS was 12(out of 36), range −7 to + 28.

Conclusions: The RQS score was moderately low, and most studies did not consider the repeatability of radiomics features, especially in terms of Intra-individual, scanners, and scanning parameters. In order to improve the generalization of the radiomics model, it is necessary to further control the variable factors of repeatability.

Colorectal cancer

Artificial intelligence

Radiomics

Repeatability

Machine learning

Colorectal cancer (CRC) is one of the most common clinical malignant tumors. According to Global Cancer Statistics 2020, over 1.9 million new cases of colorectal cancer and about 935,000 deaths worldwide, ranking third and second in global cancer morbidity and mortality [1]. Overall survival (OS) with colorectal cancer has improved, with the emergence of neoadjuvant chemoradiotherapy (CRT) and immunotherapy[2], but personalized preoperative diagnosis, prognosis, and treatment options for colorectal cancer patients have so far remained challenging.

Tumors are composed of multiple subpopulations of cancer cells, the process of tumor growth, the same malignant tumor may exhibit temporal and spatial variation in different patients or different parts of the same patient[3], called tumor heterogeneity. At present, the biopsy is the most effective way to diagnose colorectal cancer. However, tissue biopsy only samples a certain part of the tumor, which cannot represent the whole tumor[4], so the information of tumor heterogeneity provided by tissue biopsy is limited. Medical images can analyze the overall situation of tumors at different time points, so it can provide incremental value for evaluating tumor heterogeneity[4]. However, traditional radiology usually relies on the naked eye to diagnose according to size and density. This qualitative method will miss many deep features[5]. In recent years, with the rapid development of image analysis methods and pattern recognition tools, the analysis of medical images has gradually developed from qualitative to quantitative analysis.

As a quantitative analysis tool, radiomics extracts features from medical images through high-throughput computing and applies them to personalized clinical decisions to improve the accuracy of diagnosis and prognosis [4]. In the past decade, radiomics has shown unique advantages in staging, grading, and predicting treatment response. Before applying the radiomics model to the clinical decision, it is necessary to ensure that the radiomics features are repeatable, the model can perform well in different populations, different scanners and different centers[5–8]. "Repeatability" refers to a feature that remains unchanged for more than one image in the same or nearly the same object[9]. It is an indicator of discrimination between different scanners or between different institutions[4, 8]. Radiomics obtains quantitative features from medical images through image acquisition, segmentation and feature extraction, in this process, there are many factors that affect the repeatability of features, such as intra-individual variation, differences between scanners, different scanning schemes, differences in segmentation and so on. Moreover, in the following process, feature reduction and modeling, the choice of methods also change the research results, so it is still challenging to obtain a repeatable and easily applicable radiomics model. In order to improve the generalization ability of the model and promote the clinical application of the research results, it is necessary to control the influence of the above variable factors. In the following, we propose several methods to reduce variation. At present, due to the lack of standardization in the field of radiomics, the results of many studies are not comparable and repeatable. The quality of researches in the field of colorectal cancer radiomics is uneven. It is necessary to evaluate the performance of radiomics as a tool for the diagnosis and prognosis of colorectal cancer. Therefore, this systematic review evaluates the repeatability and reporting quality of radiomics in colorectal cancer, and summarizes its current application.

2.1 Review Strategy

We conducted a systematic review according to the Preferred Reporting items for Systematic review and Meta-Analysis (PRISMA) checklist^[10]. But the review was not registered before. Two reviewers used the following retrieval strategies on PubMed and EmBase databases for systematic search: ("colorectal neoplasms"[MeSH Terms] OR ("colorectal"[All Fields] AND "neoplasms"[All Fields]) OR "colorectal neoplasms"[All Fields] OR ("colorectal"[All Fields] AND "carcinoma"[All Fields]) OR "colorectal carcinoma"[All Fields] OR ("colorectal neoplasms"[MeSH Terms] OR ("colorectal"[All Fields] AND "neoplasms"[All Fields]) OR "colorectal neoplasms"[All Fields] OR ("colorectal"[All Fields] AND "cancer"[All Fields]) OR "colorectal cancer"[All Fields])) AND ("radiomic"[All Fields] OR "radiomics"[All Fields]), the search deadline was March 2020.

Two reviewers included articles that met the following criteria: (1) articles published in full text retrieved on PubMed and EmBase databases, (2) articles related to radiomics in the field of colorectal cancer, (3) articles that used ultrasound (UA) and/or computed tomography (CT) and/or magnetic resonance imaging (MRI) and/or fluorodeoxyglucose positron emission tomography-computed tomography (FDG PET-CT). The exclusion criteria were as follows: (1) non-original article (review, meta-analysis, case report, editorial and conference abstract), (2) the purposes of studies are about methodology or test-retest, (3) non-English language studies.

2.2 Study Extraction

Systematically recorded the following data from each included study: author, year, purpose, type, sample size, imaging modality, acquisition parameters, reconstruction parameters, pretreatment method, feature selection method, modeling method, segmentation method, number of features, verification method, performance index, and clinical utility.

2.3 Data synthesis

There are great differences in the methods used in the included studies, so we did not conduct a meta-analysis.

2.4 Quality Assessment

In addition,in order to evaluate the quality of these included studies, we introduced the RQS[4], RQS is a unique quality assessment tool in radiomics[11]. The score is composed of 16 parts, with a total score of 36. The higher the score, the better the quality of the article.

According to the above retrieval strategies and standards, 218 articles were retrieved, and 68 articles were included in this study after all screening. The include and exclude flowchart is shown in Figure 1. The area under the curve (AUC) values of the included studies ranged from 0.71 to 0.95, implying a moderate to good predictive performance of the radiomics model. The key information of included studies is summarized in Additional Table 1.

From the four aspects of the number of articles, research tasks, research types, and RQS scores, we gave an overall overview of included articles, so as to analyze the current situation of the application of radiomics in colorectal cancer.

Figure 2 shows the published papers in the field of radiomics of colorectal cancer in recent years. From 2016 to now, the number of articles has increased year by year, with the fastest growth from 2017 to 2019, and its growth rate has slowed down by 2020. The overall trend showed that radiomics was in the stage of development in the field of colorectal cancer. So radiomics is a new field in recent years, and it still faces many challenges, such as the study of repeatability, model interpretability, over-fitting and so on.

In the field of colorectal cancer, radiology was mainly used for prognosis (51%), while few studies have carried out staging and grading (9%) (Figure 3A). The imaging methods studied can be roughly divided into CT, MRI, PET, PET-CT, UA, and the combination of the two imaging methods (Figure 3B). These studies were mainly based on two imaging methods: CT (45%) and MRI (41%), while there were a few studies related to PET-CT (3%).

Prospective research may provide better clinical evidence for evidence-based medicine and make up for the shortcomings of retrospective research. However, the prospective study has a long follow-up time and many restrictions, so it is not easy to achieve. Most radiomics of colorectal cancer are based on retrospective data sets (94%), and only a few are prospective studies[12–15](6%).

Multicenter research can fully reflect the overall situation and evaluate the generalization ability of the model. Of all the studies, only 7[12, 16–21] (10%) conducted multicenter studies and 5[22–26] (7%) were dual-center studies.

3.1 RQS score

The included articles were scored by RQS, and the specific scores were shown in Additional Table 2. Its score ranges from-7 to 28 (- 19.44–77.78%), the median was 12 (33.33%), of which 13 studies scored less than 10%.

Figure 4 shows the score of each item in the RQS score, 66 studies (97%) reduced the dimension of features to reduce the risk of overfitting, 53 studies (78%) adopted multiple segmentation, and 65 studies reported the differentiation of the model. However, only 24 reported the calibration of the model. Low scores in the following six items (1) phantom studies (2) multiple time point (3) prospective studies (4) open science data (5) Cut-off analysis (6) cost-effectiveness analysis.

Figure 4. Completion rate of 68 studies in RQS.

Only two studies (3%) adopted test-retest reliability. Only five studies disclosed the code and data, but there were no studies that opened ROI-related data. Most studies used the best threshold method to select the threshold, only six (9%) studies used the median as the threshold. Four studies (6%) were prospective, and none of the studies conducted a cost-benefit analysis of the clinical application.

3.2 Repeatability

The repeatability of radiomics features is directly related to the accuracy of model[27]. And many factors in the workflow of radiomics will affect the repeatability of the radiomics features, Such as scanner[9, 27, 28], acquisition parameters[27, 29-33], pretreatment method[34, 35], segmentation method[36-39], inter/intra-observer variability [33, 34, 36], feature selection method [40], modeling method [40]. The factors and solutions that may affect repeatability in the radiomicsworkflow are shown in Figure 5.

Figure 5. Radiomics workflow and repeatability. Each step has associated factors which may influence the repeatability of the study. Although modelling affect reproducibility, there is still no solution.

3.2.1 Intra-individual repeatability

JE van Timmeren, et al. [41] scanned forty patients with rectal cancer twice with the same scanning scheme at 15-minute intervals, and used a consistent correlation coefficient (CCC) to assess the correlation between the features of the two scans, then 7 of 542 features had a CCC> 0.9, and 9 features had a CCC>0.85. Therefore, only some of the features are repeatable at different times for the same individual. The features with the highest repeatability were the “shape”, and the “wavelet” features appeared to be the least reproducible[41]. Certain features are sensitive to changes in organ motion or expansion or shrinkage of the target volume caused by physiological factors such as respiration, bowel peristalsis, cardiac and cardiac activity, so these features show low reproducibility[4]. However, a set of highly reproducible radiomic features can be obtained using the test-retest based on phantom or patients [42, 43].

However, only 2[44, 45] of the 68 articles carried out retest experiments. X Ma, et al. [44] set a base of intra-class correlation coefficient of 0.6 for the retest analysis, in order to ensure the robustness of the features. And J Wang, et al. [45] selected 40 patients with stage II rectal cancer and scanned twice using the same scanner and imaging protocol before treatment, and then used the Spearman correction coefficient to select repeatable features. Only these two studies considered and took measures to control for Intra-individual repeatability.

3.2.2 Acquisition parameters

There may be differences between different scanners. D Mackin, et al. [28]used phantom to compare the radiomics features obtained from four CT scanners: GE, Philips, Siemens, and Toshiba, and found differences between scanners. Then R Berenguer, et al. [27] used two phantom models (the pelvic phantom and the phantom of different materials) to detect the feature differences of intra-CT analysis(differences between different CT acquisition parameters) and inter-CT analysis(differences between five different scanners), showing that only 71 out of 177 features were reproducible. And using hierarchical cluster analysis, the 10 most representative features were selected, including "60 Percentile", “Global Median”, “Global Minimum”, “Kurtosis”, “Mass”, “Volume”, “Roundness”, “Surface Area Density”, “4-Inverse Difference Normalized” and “4-Auto Correlation”. In addition, R Berenguer, et al. [27] reported that the impact of different scanners could be reduced by standardizing the acquisition parameters.

Of the 31 CT-based imaging studies, except for 5 [25, 45–48] that did not provide scanner parameters, the other studies neither used consistent scanning parameters nor assessed the impact of scanner differences on feature repeatability. Therefore, it could not be ruled out that scanner differences do not affect the results of these studies. Among the 28 studies based on MRI, SP Shayesteh, et al. [49] considered the influence of scanner and scanning parameters on feature repeatability and tried to use image preprocessing (noise reduction, intensity normalization and discretization) to reduce the difference. In the PET-CT based research, J Kang, et al. [50] reduced the SUV measurement difference between the two scanners to less than 10 percent through regular standardization and quality assurance.

L He, et al. [32] demonstrated that acquisition parameters (slice thickness, convolution kernel and enhancement) had affects on the diagnostic performance of radiomics, and that radiomics features constructed based on thin-slice (1.25 mm) have better performance in differential diagnosis than features based on thick-slice (5 mm). The reason for the better performance of thin-slice may be the introduction of larger partial volume artifacts in thick-slice. Similarly, L Lu, et al. [31] demonstrated there exist differences in the values of radiomics feature extracted from CT images with different slice thicknesses and reconstruction methods. And Features associated with tumor size, border morphology, low-order density statistics, and coarse texture were more sensitive to variations in acquisition parameters. Subsequently, a more rigorous experiment [30] showed that 63 of the 213 features were affected by voxels, but 42 features were significantly improved, and 21 features changed greatly after resampling. Therefore, for the image data with different slice thicknesses, the resampling may effectively reduce the influence of layer thickness on the repeatability of the study. Of the 68 studies, 9 [16, 21, 23, 51–56] (13%) reduced the effect of slice thickness by resampling.

3.2.3 Segmentation

Accurate and efficient segmentation of regions of interest is helpful to extract robust quantitative imaging features[9]. The segmentation method can be roughly divided into manual, semi-automatic, and automatic segmentation. Manual segmentation is usually regarded as the gold standard, but it has two problems. According to statistics, it took an average of 18 minutes to delineate the region of interest of a tumor. Therefore, it is unlikely to be implemented in the clinic[36]. Second, there were great subjective differences among the observers, which may affect the repeatability of the target area[57, 58]. Existing studies [36–38, 59, 60] proved that semi-automatic segmentation had better stability and higher efficiency than manual segmentation (the average segmentation time was reduced by 4 minutes). Although automatic segmentation based on deep learning may further improve the accuracy of segmentation[39],automatic segmentation is not yet mature and needs further research before it can be used in the clinic. Of the 68 studies, 13 [12, 26, 61–71](19%) used semi-automatic segmentation, 3 [52, 72, 73](4%) used automatic segmentation. Except for 12 [21, 25, 44, 45, 49, 53, 74–79] studies which did not describe the segmentation methods used, the remaining 40[13–20, 22–24, 46–48, 50, 51, 54–56, 80–100]studies adopted the manual segmentation method.

Manual and semi-automatic segmentation may cause deviation between features and real values because of the variability of the segmentation process [101]. Variability includes subjective differences among multiple observers (inter-observer variation) and subjective differences of the same person at different times (intra-observer variation), so it is necessary to use multi-person segmentation or multiple methods to reduce deviation[4]. Of all the articles, 15 did not use multiple segmentation. 19 of the other articles did not analyze the differences between observers or segmentation methods and did not rule out unstable features. Only 34 articles evaluated the variation, but the evaluation indicators were not consistent: 24 articles evaluated by intraclass correlation coefficient(ICC), 3 articles[16, 20, 21]used Dice similarity Coefficient and/or Jaccard similarity coefficient, 1 article [83] used Bland-Altman plots as evaluation parameter, 1 article[45] used Spearman correlation coefficient, 1 article [73] used automatic segmentation which repeatability was verified,and 4 articles [12, 48, 52, 54] did not describe the evaluation index. In a word, most studies (71%) use ICC to evaluate variability, while I Fotina, et al. [102] preferred to use Jaccard similarity coefficient, conformal number, or generalized conformability index to evaluate inter-observer variability.

3.2.4 Feature selection

Radiomics studies always extract a large number of features, whereas the number of samples is often very small, so it is easy to cause dimension disaster so that the model is over-fitted and lacks generalization ability[8, 103]. To ensure that the model has statistical significance and clinical significance, and reduce the false positive rate,A Chalkidou, et al. [103] proposed the following measures: (1) repeatability of features (2) cross-correlation analysis (3) inclusion of clinically important features (4) at least 10-15 patients with each feature (5) external verification.

The main purpose of feature selection is (1) to select repeatable features between different institutions, (2) to remove redundant features (highly related features between features), and (3) to select features that are strongly related to the result variables. Feature selection can effectively reduce the number of features, but different methods need to be selected according to the needs of the research[104]. In all the studies, the Least Absolute Shrinkage and Selection operator (LASSO) (46%) was the most commonly used feature selection method, followed by correlation analysis (33%). The feature selection method is not unique, it needs to be adjusted according to the number of features and sample size[8]. The most suitable method should be selected by comparing a variety of methods.

The sample size of all studies ranged from 15 to 701, with a median of 111, and 78% of the studies had a sample size of 0-200. To assess the adequacy of the sample size in the study, MA Babyak [105] suggested that at least 10-15 patients were needed for each feature. Based on this standard, 17 (25%) of the included studies did not meet the above conditions except 5 studies[73, 74, 79, 98, 106], which did not establish a model and 4 studies[26, 46, 78, 87] that did not indicate the characteristic quantity(Figure 6).

Figure 6. Sample size of included studies. Adequate sample means the ratio of the sample size to the feature number of the study is more than 10, inadequate sample means the ratio is less than 10, unclear means the study did not establish a model or did not specify the number of features.

3.2.5 Modelling methodology

C Parmar, et al. [40] evaluated the performance and stability of 12 classification methods in predicting overall survival, which showed that random forest classification had the highest prediction performance and stability. However, it is not clear which statistical method or machine learning method is better. The model generated by the simple modeling method is easy to explain, and the complex model improves the performance but needs further verification [107].

3.2.6 Evaluation

The generalization ability of the model can be evaluated by using the verification data. According to the principle of confirmatory analysis, independent data sets are needed to verify the results of the training set[108]. Only 51 articles (75%) used independent dataset validation, including 5[22–26] articles using dual-center validation sets and 7[12, 16–21] articles using multicenter validation sets.

3.3 How to increase repeatability

3.3.1 Standardization protocol

Standardizing the radiomics process is the most reliable way to increase repeatability[7]. So Image Biomarker Standardization Initiative (IBSI) [109]standardizes the definition, naming, and software. The Quantitative Imaging Network (QIN) [110] project initiated by NIC (National Cancer Institute) has also promoted the standardization of imaging methods and imaging protocols. In addition, the Quantitative Imaging Biomarkers Alliance (QIBA) [111] organization sponsored by, Radiological Society of North America (RSNA) has developed a standardized quantitative imaging document "Profiles" to promote clinical trials and practices of quantitative imaging markers. At present, only 4[26, 53, 54, 61] (6%) of the radiomics studies of colorectal cancer comply with the IBSI standard.

3.3.2 Test-retest reliability

Before a unified standard is formed, test-retest reliability tests can be taken to increase the repeatability of the study. The same scanner and the same patient were scanned twice at an interval of 15 minutes to determine the characteristics with high repeatability[112]. Moreover, JE van Timmeren, et al. [41] indicated that appropriate test-retest reliability should be carried out in each step. Also, the effects of hardware, acquisition, reconstruction, tumor segmentation, and feature extraction should be strictly controlled. However, only 2 articles[44, 45] (3%) conducted test-retest reliability.

3.3.3 Phantom

For retrospective data that did not use the same scanning scheme, phantom could remove unstable features due to differences in scanner, scanning, and reconstruction parameters[4]. However, none of the included studies used phantom to analyze its repeatability.

3.3.4 Post-processing

With the continuous emergence of new features, the efficiency of test-retest reliability research and phantom research becomes lower[35]. The following post-processing methods can also reduce the variability of features.

Resampling and normalization: Some studies[29, 30, 113]showed that resampling could effectively improve the feature variation caused by voxel differences. However, resampling alone might not improve the variations of all features, and features need to be normalized according to voxel size[30]. Among the 68 studies, resampling was used in 9 articles [16, 21, 23, 51–56] (13%). In other studies, normalization was used to reduce the influence of different gray ranges or the effects of low frequency and intensity inhomogeneity. Normalization also has some disadvantages, such as introducing noise, blurring the image, and causing the loss of image details [114]. However, these shortcomings of normalization would be avoided by using ComBat.

ComBat: Previously, genomics has been affected by batch effects, that is, systematic technical biases introduced by samples in different batches of processing and measurement that are not related to biological status[115]. WE Johnson, et al. [116] developed and validated a method to deal with the "batch effect"-ComBat. In radiomics, the impact of different scanners or scanning schemes is similar to that of batches. Studies[117–119] showed that ComBat could reduce the feature differences caused by different scanners or scanning schemes, and retain the feature differences formed by biological variation. Although ComBat is practical, convenient and fast, it will be affected by the distribution of validated data sets[35], and it cannot be directly applied to imaging data[120]. So Y Li, et al. [35] developed a normalization method based on deep learning, which may effectively avoid the above problems.

The results of radiomics need to be applied to different environments and people, and the repeatability of the evaluation model is helpful to judge its generalizability and clinical application value. So this review analyzed the details of the methods provided in each study to discuss the repeatability of 68 radiomics studies on colorectal cancer and evaluated the quality of studies by RQS. Although the included studies demonstrated the excellent predictive performance (AUC=0.71~0.95) of radiomics in colorectal cancer, most studies did not take into account the repeatability of radiomics features, especially in terms of Intra-individual repeatability, repeatability between scanners, and repeatability of scanning parameters.

Since 2016, the number of colorectal cancer radiomics has increased year by year, but the median RQS score was 12 (33.33%), indicating that radiomics was an immature new technology in the field of colorectal cancer. Three of the four items with the lowest scores in RQS are related to repeatability, including phantom studies, multiple time point and open science data. The same phantom is scanned with the same scanning scheme in different scanners, and the radiomics features independent of the scanner are obtained [4]. Of all the studies reviewed, only J Kang, et al. [50] used phantom and regular standardization for quality control to keep differences in measurements of the standardized uptake value (SUV) between the two scanners below 10%. P Lambin, et al. [4] explained that the purpose of multiple time point is to eliminate the radiomics features that strongly depend on the organ movement or the expansion or shrinkage of the target volume, and the retest-test can be used to detect unstable features, thus reducing the influence of intra-individual variability [42, 43], but only 2[44, 45] (3%) of all studies used the test-retest. Through the retest test, X Ma, et al. [44] removed the unstable radiomics features whose intra-class correlation coefficient is less than 0.6. J Wang, et al. [45] selected the most repeatable radiomics features based on the test-retest data of 40 patients with rectal cancer. Although the subjects of the retest test may be the patient or the phantom[121, 122], the retest study of the patient may increase unnecessary radiation damage. In addition, JE van Timmeren, et al. [41]believes that extensive retest-test experiments can provide a stable set of imaging features, and emphasizes that retest-test studies should be carried out in each scenario, not only in intra-individual variability, but also in scan acquisition and reconstruction settings, tumor segmentation and feature extraction. Open science data includes opening scanning information, opening segmentations of ROIs, opening source code and opening feature extraction methods (including formulas)[4], but only five studies disclosed the code and data, there were no studies that opened ROI-related data. The open science data is beneficial for other researchers to reproduce the research results (independent researchers use the same technology and different data to repeat the research results and independently verify the results) and promote the application of the research model in clinical practice[4–6].

The standardization of scanning parameters, acquisition parameters and reconstruction parameters could effectively reduce variability, because there are many variable factors and a wide variety of diseases in radiology[5, 6], there is no global standard at present. In addition to standardization, the influence of scanning parameters can be reduced by using the same scanning scheme, unstable features may be removed by retest tests[121], resampling might be used to control the influence of slicer thickness[30], and the "batch effect" could be reduced by combat method[117–119]. Of the 68 studies, 9 (13%) adopted resampling and 1 (1%) adopted combat methods to control variability. In order to balance the deviation of imaging features from four institutions, Z Liu, et al. [19] normalized the data of each organization, adopted combat methods to control the deviation of radiomic features, and weighted the radiomics features with inverse probability of treatment weighting (IPTW) to eliminate the covariant effect among the four cohorts.

With regard to the method of image segmentation, it is proved that semi-automatic segmentation is more efficient than manual segmentation, has less variation between observers, and is more mature than automatic segmentation[36–38, 59, 60]. Of all the studies, 40 (59%) were segmented manually, and only 13 (19%) were segmented semi-automatically. For inter-observer variability, only 34 (44%) studies took into account inter-observer variability and excluded instability. Even though the Jaccard index, Cn and CIgen may be better as evaluation parameters [102], 24 studies were evaluated by ICC.

According to the Harrell criterion[123], the sample size should be more than 10 times the number of variables, and feature selection can reduce redundant features and reduce the risk of model over-fitting[8, 103]. But 17 studies (25%) have the possibility of over-fitting even though feature selection has been carried out. Such as, in the study of C Yang, et al. [76], random forest was used to measure the Gini importance of parameters, and finally 10 important parameters were included in the model. But the study included only 89 patients, there was still a risk of over-fitting according to the Harrell criteria. [124]recommends that at least 50 patients be included in radiomics studies, but there are 9 of all reviewed articles have a sample size of less than 50. Too small sample size will seriously affect the quality of the article and reduce the credibility of the research results[125].

In addition to the repeatability of the study, it is also important to avoid overly optimistic results. In many studies, the combination of optimal cut-off method and multiple hypothesis testing may increase type I errors[103], while the multiple hypothesis test correction (Holm Bonferroni or Benjamin-Hochberg pair) is helpful to reduce class I errors[6]. Therefore, only 11 (14%) of the studies were corrected by multiple hypothesis testing.

The main limitation of this review is that there are great differences in the methods used in the included studies, so we did not conduct a meta-analysis. In addition, some related studies may not be retrieved, such as some grey literature. In this paper, only RQS is used to evaluate the quality of the article, whereas RQS is not the only and perfect evaluation method, and other methods (such as QUADAS-2 or TRIPOD) should be used later.

According to the review of 68 literatures, the key problem in radiomics studies is the lack of repeatability, and the standardization of radiomics processes can help to compare or replicate existing studies[7]. But at present, it is difficult to form global standardization[7], so the repeatability of research can be improved through the open science data[4], retest-test research[121] and post-processing. Before radiomics is applied to the clinic, it is not only necessary to conduct a reliable prospective study with large samples, but also to verify the existing models, preferably independently with multicenter data. Based on the existing literature, some studies[22, 26, 52, 53, 83, 98, 100] use DNA microarrays to directly sequence genes, and then use radiomics to predict sequencing results, combining genomics and radiomics. However, it is noteworthy that there are no studies on the combination of liquid biopsy and radiomics. Studies have shown that the combination of liquid biopsy and imaging can play an early warning role in patients prone to recurrence and metastasis [126]. In the future, it may be possible to add biomarkers from other disciplines to the imaging model. T Cheng and X Zhan [127] proposed that a pattern consisting of three or more biomarkers could improve the accuracy and specificity of tumor prediction, diagnosis and prognosis, that is, pattern recognition[127]. Multi-time imaging may be more helpful to observe the changing process of treatment response and obtain more data. Delta radiomics is to quote the time component in the study, that is, to use the image data of multiple time points for radiomics analysis. This method is expected to improve the diagnosis, prognosis prediction and treatment response evaluation [4]. At present, only 7 studies have used multi-time point imaging, and the purpose of these was to predict the response to treatment. Since 95% of the studies are retrospective, more prospective studies with large samples and multiple centers are needed in the future to promote the development of imaging in the field of colorectal cancer. Cost-effectiveness analysis from an economic point of view is necessary for the clinical application of colorectal cancer imaging in the future[4], while there is no related analysis in any study at present.

Although existing studies showed that radiomics is helpful to the personalized treatment of patients in the field of colorectal cancer, there are still many challenges that remain to be solved. According to the RQS score, the quality of included studies was moderately low. Moreover, we found that the main reason for the low RQS score was the lack of repeatability, most studies did not eliminate the influence of scanners, imaging parameters, and other factors. Therefore, these studies of lower quality and lack of repeatability mean that the results are not universal. In order to improve the clinical use of colorectal cancer radiomics, it is necessary to take measures to control repeatability. For example, carried out appropriate test-retest reliability in each step of radiomics or reduced the influence of variable factors by post-processing. In the future, larger sample and multicenter prospective high-quality studies are needed. In addition, and future researches should focus on building a more stable and repeatable model.

AUC

Area Under the Curve

CRC

Colorectal cancer

CRT

Chemoradiotherapy

Computed Tomography

IBSI

Image Biomarker Standardization Initiative

ICC

Intraclass Correlation Coefficient

LASSO

Least Absolute Shrinkage and Selection operator

MRI

Magnetic Resonance Imaging

PET-CT

fluorodeoxyglucose Positron Emission Tomography-Computed Tomography

PRISMA

Preferred Reporting items for Systematic review and Meta-Analysis

QIBA

Quantitative Imaging Biomarkers Alliance

QIN

Quantitative Imaging Network

RQS

Radiomics Quality Score

RSNA

Radiological Society of North America

SUV

Standardized Uptake Value

Ultrasound

Ethics approval and consent to participate

Not applicable.

Consent to publish

Not applicable.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

Not applicable.

Authors Contributions

YL and XQW contributed equally to the work. YL and XQW contributed to study design and wrote the study protocol. XF and YL performed the literature search. YL and XQW performed data collection and data interpretation and drafted the manuscript. XF, YL and GLF reviewed and critically revised the manuscript. YD were responsible for study supervision. All authors have read and approved the manuscript.

Acknowledgements

Not applicable.

Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021. http://dx.doi.org/10.3322/caac.21660.
Brouwer NPM, Bos ACRK, Lemmens VEPP, Tanis PJ, Hugen N, Nagtegaal ID, de Wilt JHW, Verhoeven RHA. An overview of 25 years of incidence, treatment and outcome of colorectal cancer patients. Int J Cancer. 2018;143(11):2758–66. .http://dx.doi.org/https://doi.org/10.1002/ijc.31785.
Vallières M, Freeman CR, Skamene SR, El Naqa I. A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. Phys Med Biol. 2015;60(14):5471–96. .http://dx.doi.org/10.1088/0031-9155/60/14/5471.
Lambin P, Leijenaar RTH, Deist TM, Peerlings J, De Jong EEC, Van Timmeren J, Sanduleanu S, Larue RTHM, Even AJG, Jochems A, et al. Radiomics: The bridge between medical imaging and personalized medicine. Nature Reviews Clinical Oncology. 2017;14(12):749–62. .http://dx.doi.org/10.1038/nrclinonc.2017.141.
Limkin EJ, Sun R, Dercle L, Zacharaki EI, Robert C, Reuzé S, Schernberg A, Paragios N, Deutsch E, Ferté C. Promises and challenges for the implementation of computational medical imaging (radiomics) in oncology. Ann Oncol. 2017;28(6):1191–206. .http://dx.doi.org/10.1093/annonc/mdx034.
Yip SS, Aerts HJ. Applications and limitations of radiomics. Phys Med Biol. 2016;61(13):R150. 166.http://dx.doi.org/10.1088/0031-9155/61/13/r150.
Gillies RJ, Kinahan PE, Hricak H. Radiomics: Images Are More than Pictures, They Are Data. Radiology. 2016;278(2):563–77. http://dx.doi.org/10.1148/radiol.2015151169.
Avanzo M, Stancanello J, El Naqa I. Beyond imaging: The promise of radiomics. Phys Med. 2017;38:122–39. .http://dx.doi.org/10.1016/j.ejmp.2017.05.071.
Kumar V, Gu Y, Basu S, Berglund A, Eschrich SA, Schabath MB, Forster K, Aerts HJ, Dekker A, Fenstermacher D, et al. Radiomics: the process and the challenges. Magn Reson Imaging. 2012;30(9):1234–48. .http://dx.doi.org/10.1016/j.mri.2012.06.010.
McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM. Group atP-D: Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The PRISMA-DTA Statement. JAMA. 2018;319(4):388–96. .http://dx.doi.org/10.1001/jama.2017.19163.
Chetan MR, Gleeson FV. Radiomics in predicting treatment response in non-small-cell lung cancer: current status, challenges and future perspectives. Eur Radiol. 2021;31(2):1049–58. http://dx.doi.org/10.1007/s00330-020-07141-9.
Dercle L, Lu L, Schwartz LH, Qian M, Tejpar S, Eggleton P, Zhao B, Piessevaux H. Radiomics Response Signature for Identification of Metastatic Colorectal Cancer Sensitive to Therapies Targeting EGFR Pathway. J Natl Cancer Inst. 2020;112(9):902–12. .http://dx.doi.org/10.1093/jnci/djaa017.
Chen LD, Li W, Xian MF, Zheng X, Lin Y, Liu BX, Lin MX, Li X, Zheng YL, Xie XY, et al. Preoperative prediction of tumour deposits in rectal cancer by an artificial neural network-based US radiomics model. Eur Radiol. 2020;30(4):1969–79. http://dx.doi.org/10.1007/s00330-019-06558-1.
Chen LD, Liang JY, Wu H, Wang Z, Li SR, Li W, Zhang XH, Chen JH, Ye JN, Li X, et al. Multiparametric radiomics improve prediction of lymph node metastasis of rectal cancer compared with conventional radiomics. Life Sci. 2018;208:55–63. .http://dx.doi.org/10.1016/j.lfs.2018.07.007.
Bulens P, Couwenberg A, Intven M, Debucquoy A, Vandecaveye V, Van Cutsem E, D'Hoore A, Wolthuis A, Mukherjee P, Gevaert O, et al. Predicting the tumor response to chemoradiotherapy for rectal cancer: Model development and external validation using MRI radiomics. Radiother Oncol. 2020;142:246–52. .http://dx.doi.org/10.1016/j.radonc.2019.07.033.
Alvarez-Jimenez C, Antunes JT, Talasila N, Bera K, Brady JT, Gollamudi J, Marderstein E, Kalady MF, Purysko A, Willis JE, et al: Radiomic Texture and Shape Descriptors of the Rectal Environment on Post-Chemoradiation T2-Weighted MRI are Associated with Pathologic Tumor Stage Regression in Rectal Cancers: A Retrospective, Multi-Institution Study. Cancers (Basel) 2020, 12(8).http://dx.doi.org/10.3390/cancers12082027.
Bibault JE, Giraud P, Housset M, Durdux C, Taieb J, Berger A, Coriat R, Chaussade S, Dousset B, Nordlinger B, et al. Deep Learning and Radiomics predict complete response after neo-adjuvant chemoradiation for locally advanced rectal cancer. Sci Rep. 2018;8(1):12611. http://dx.doi.org/10.1038/s41598-018-30657-6.
Dinapoli N, Barbaro B, Gatta R, Chiloiro G, Casà C, Masciocchi C, Damiani A, Boldrini L, Gambacorta MA, Dezio M, et al. Magnetic Resonance, Vendor-independent, Intensity Histogram Analysis Predicting Pathologic Complete Response After Radiochemotherapy of Rectal Cancer. Int J Radiat Oncol Biol Phys. 2018;102(4):765–74. .http://dx.doi.org/10.1016/j.ijrobp.2018.04.065.
Liu Z, Meng X, Zhang H, Li Z, Liu J, Sun K, Meng Y, Dai W, Xie P, Ding Y, et al. Predicting distant metastasis and chemotherapy benefit in locally advanced rectal cancer. Nat Commun. 2020;11(1):4308. http://dx.doi.org/10.1038/s41467-020-18162-9.
Grosu S, Wesp P, Graser A, Maurus S, Schulz C, Knösel T, Cyran CC, Ricke J, Ingrisch M, Kazmierczak PM. Machine Learning-based Differentiation of Benign and Premalignant Colorectal Polyps Detected with CT Colonography in an Asymptomatic Screening Population: A Proof-of-Concept Study. Radiology 2021:202363.http://dx.doi.org/10.1148/radiol.2021202363.
Antunes JT, Ofshteyn A, Bera K, Wang EY, Brady JT, Willis JE, Friedman KA, Marderstein EL, Kalady MF, Stein SL, et al. Radiomic Features of Primary Rectal Cancers on Baseline T(2) -Weighted MRI Are Associated With Pathologic Complete Response to Neoadjuvant Chemoradiation: A Multisite Study. J Magn Reson Imaging. 2020;52(5):1531–41. .http://dx.doi.org/10.1002/jmri.27140.
Cui Y, Liu H, Ren J, Du X, Xin L, Li D, Yang X, Wang D. Development and validation of a MRI-based radiomics signature for prediction of KRAS mutation in rectal cancer. Eur Radiol. 2020;30(4):1948–58. http://dx.doi.org/10.1007/s00330-019-06572-3.
Nakanishi R, Akiyoshi T, Toda S, Murakami Y, Taguchi S, Oba K, Hanaoka Y, Nagasaki T, Yamaguchi T, Konishi T, et al. Radiomics Approach Outperforms Diameter Criteria for Predicting Pathological Lateral Lymph Node Metastasis After Neoadjuvant (Chemo)Radiotherapy in Advanced Low Rectal Cancer. Ann Surg Oncol. 2020;27(11):4273–83. .http://dx.doi.org/10.1245/s10434-020-08974-w.
Cusumano D, Dinapoli N, Boldrini L, Chiloiro G, Gatta R, Masciocchi C, Lenkowicz J, Casà C, Damiani A, Azario L, et al. Fractal-based radiomic approach to predict complete pathological response after chemo-radiotherapy in rectal cancer. Radiol Med. 2018;123(4):286–95. .http://dx.doi.org/10.1007/s11547-017-0838-3.
Vandendorpe B, Durot C, Lebellec L, Le Deley MC, Sylla D, Bimbai AM, Amroun K, Ramiandrisoa F, Cordoba A, Mirabel X, et al. Prognostic value of the texture analysis parameters of the initial computed tomographic scan for response to neoadjuvant chemoradiation therapy in patients with locally advanced rectal cancer. Radiother Oncol. 2019;135:153–60. http://dx.doi.org/10.1016/j.radonc.2019.03.011.
Shi R, Chen W, Yang B, Qu J, Cheng Y, Zhu Z, Gao Y, Wang Q, Liu Y, Li Z, et al. Prediction of KRAS, NRAS and BRAF status in colorectal cancer patients with liver metastasis using a deep artificial neural network based on radiomics and semantic features. Am J Cancer Res. 2020;10(12):4513–26.
Berenguer R, Pastor-Juan MDR, Canales-Vázquez J, Castro-García M, Villas MV, Mansilla Legorburo F, Sabater S. Radiomics of CT Features May Be Nonreproducible and Redundant: Influence of CT Acquisition Parameters. Radiology. 2018;288(2):407–15. http://dx.doi.org/10.1148/radiol.2018172361.
Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, Rodriguez-Rivera E, Dodge C, Jones AK, Court L. Measuring Computed Tomography Scanner Variability of Radiomics Features. Invest Radiol. 2015;50(11):757–65. .http://dx.doi.org/10.1097/rli.0000000000000180.
Mackin D, Fave X, Zhang L, Yang J, Jones AK, Ng CS, Court L. Harmonizing the pixel size in retrospective computed tomography radiomics studies. PLoS One. 2017;12(9):e0178524. http://dx.doi.org/10.1371/journal.pone.0178524.
Shafiq-Ul-Hassan M, Zhang GG, Latifi K, Ullah G, Hunt DC, Balagurunathan Y, Abdalah MA, Schabath MB, Goldgof DG, Mackin D, et al. Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels. Med Phys. 2017;44(3):1050–62. .http://dx.doi.org/10.1002/mp.12123.
Lu L, Ehmke RC, Schwartz LH, Zhao B. Assessing Agreement between Radiomic Features Computed for Multiple CT Imaging Settings. PLoS One. 2016;11(12):e0166550. http://dx.doi.org/10.1371/journal.pone.0166550.
He L, Huang Y, Ma Z, Liang C, Liang C, Liu Z. Effects of contrast-enhancement, reconstruction slice thickness and convolution kernel on the diagnostic performance of radiomics signature in solitary pulmonary nodule. Sci Rep. 2016;6:34921. http://dx.doi.org/10.1038/srep34921.
Kim H, Park CM, Lee M, Park SJ, Song YS, Lee JH, Hwang EJ, Goo JM. Impact of Reconstruction Algorithms on CT Radiomic Features of Pulmonary Tumors: Analysis of Intra- and Inter-Reader Variability and Inter-Reconstruction Algorithm Variability. PLoS One. 2016;11(10):e0164924. http://dx.doi.org/10.1371/journal.pone.0164924.
Traverso A, Kazmierski M, Shi Z, Kalendralis P, Welch M, Nissen HD, Jaffray D, Dekker A, Wee L. Stability of radiomic features of apparent diffusion coefficient (ADC) maps for locally advanced rectal cancer in response to image pre-processing. Phys Med. 2019;61:44–51. http://dx.doi.org/10.1016/j.ejmp.2019.04.009.
Li Y, Han G, Wu X, Li Z, Zhao K, Zhang Z, Liu Z, Liang C. Normalization of multicenter CT radiomics by a generative adversarial network method. Phys Med Biol. 2020. http://dx.doi.org/10.1088/1361-6560/ab8319.
van Heeswijk MM, Lambregts DM, van Griethuysen JJ, Oei S, Rao SX, de Graaff CA, Vliegen RF, Beets GL, Papanikolaou N, Beets-Tan RG. Automated and Semiautomated Segmentation of Rectal Tumor Volumes on Diffusion-Weighted MRI: Can It Replace Manual Volumetry? Int J Radiat Oncol Biol Phys. 2016;94(4):824–31. .http://dx.doi.org/10.1016/j.ijrobp.2015.12.017.
Parmar C, Rios Velazquez E, Leijenaar R, Jermoumi M, Carvalho S, Mak RH, Mitra S, Shankar BU, Kikinis R, Haibe-Kains B, et al. Robust Radiomics feature quantification using semiautomatic volumetric segmentation. PLoS One. 2014;9(7):e102107. http://dx.doi.org/10.1371/journal.pone.0102107.
Day E, Betler J, Parda D, Reitz B, Kirichenko A, Mohammadi S, Miften M. A region growing method for tumor volume segmentation on PET images for rectal and anal cancer patients. Med Phys. 2009;36(10):4349–58. .http://dx.doi.org/10.1118/1.3213099.
Trebeschi S, van Griethuysen JJM, Lambregts DMJ, Lahaye MJ, Parmar C, Bakers FCH, Peters N, Beets-Tan RGH, Aerts H. Deep Learning for Fully-Automated Localization and Segmentation of Rectal Cancer on Multiparametric MR. Sci Rep. 2017;7(1):5301. http://dx.doi.org/10.1038/s41598-017-05728-9.
Parmar C, Grossmann P, Bussink J, Lambin P, Aerts H. Machine Learning methods for Quantitative Radiomic Biomarkers. Sci Rep. 2015;5:13087. http://dx.doi.org/10.1038/srep13087.
van Timmeren JE, Leijenaar RTH, van Elmpt W, Wang J, Zhang Z, Dekker A, Lambin P. Test-Retest Data for Radiomics Feature Stability Analysis: Generalizable or Study-Specific? Tomography 2016, 2(4):361–365.http://dx.doi.org/10.18383/j.tom.2016.00208.
Balagurunathan Y, Kumar V, Gu Y, Kim J, Wang H, Liu Y, Goldgof DB, Hall LO, Korn R, Zhao B, et al. Test-retest reproducibility analysis of lung CT image features. J Digit Imaging. 2014;27(6):805–23. .http://dx.doi.org/10.1007/s10278-014-9716-x.
Zhao B, Tan Y, Tsai WY, Qi J, Xie C, Lu L, Schwartz LH. Reproducibility of radiomics for deciphering tumor phenotype with imaging. Sci Rep. 2016;6:23428. http://dx.doi.org/10.1038/srep23428.
Ma X, Shen F, Jia Y, Xia Y, Li Q, Lu J. MRI-based radiomics of rectal cancer: preoperative assessment of the pathological features. BMC Med Imaging. 2019;19(1):86. http://dx.doi.org/10.1186/s12880-019-0392-7.
Wang J, Shen L, Zhong H, Zhou Z, Hu P, Gan J, Luo R, Hu W, Zhang Z. Radiomics features on radiotherapy treatment planning CT can predict patient survival in locally advanced rectal cancer patients. Sci Rep. 2019;9(1):15346. http://dx.doi.org/10.1038/s41598-019-51629-4.
Giannini V, Defeudis A, Rosati S, Cappello G, Mazzetti S, Panic J, Regge D, Balestra G. An innovative radiomics approach to predict response to chemotherapy of liver metastases based on CT images. Annu Int Conf IEEE Eng Med Biol Soc. 2020;2020:1339–42. http://dx.doi.org/10.1109/embc44109.2020.9176627.
Cai D, Duan X, Wang W, Huang ZP, Zhu Q, Zhong ME, Lv MY, Li CH, Kou WB, Wu XJ, et al. A Metabolism-Related Radiomics Signature for Predicting the Prognosis of Colorectal Cancer. Front Mol Biosci. 2020;7:613918. http://dx.doi.org/10.3389/fmolb.2020.613918.
Dohan A, Gallix B, Guiu B, Le Malicot K, Reinhold C, Soyer P, Bennouna J, Ghiringhelli F, Barbier E, Boige V, et al. Early evaluation using a radiomic signature of unresectable hepatic metastases to predict outcome in patients with colorectal cancer treated with FOLFIRI and bevacizumab. Gut. 2020;69(3):531–9. http://dx.doi.org/10.1136/gutjnl-2018-316407.
Shayesteh SP, Alikhassi A, Fard Esfahani A, Miraie M, Geramifar P, Bitarafan-Rajabi A, Haddad P. Neo-adjuvant chemoradiotherapy response prediction using MRI based ensemble learning method in rectal cancer patients. Phys Med. 2019;62:111–9. http://dx.doi.org/10.1016/j.ejmp.2019.03.013.
Kang J, Lee JH, Lee HS, Cho ES, Park EJ, Baik SH, Lee KY, Park C, Yeu Y, Clemenceau JR, et al. Radiomics features of18f-fluorodeoxyglucose positron‐emission tomography as a novel prognostic signature in colorectal cancer. Cancers. 2021;13(3):1–17. .http://dx.doi.org/10.3390/cancers13030392.
Wu J, Zhang Q, Zhao Y, Liu Y, Chen A, Li X, Wu T, Li J, Guo Y, Liu A. Radiomics Analysis of Iodine-Based Material Decomposition Images With Dual-Energy Computed Tomography Imaging for Preoperatively Predicting Microsatellite Instability Status in Colorectal Cancer. Front Oncol. 2019;9:1250. http://dx.doi.org/10.3389/fonc.2019.01250.
Wu X, Li Y, Chen X, Huang Y, He L, Zhao K, Huang X, Zhang W, Huang Y, Li Y, et al. Deep Learning Features Improve the Performance of a Radiomics Signature for Predicting KRAS Status in Patients with Colorectal Cancer. Acad Radiol. 2020;27(11):e254–62. http://dx.doi.org/10.1016/j.acra.2019.12.007.
Li Y, Eresen A, Shangguan J, Yang J, Benson AB 3rd, Yaghmai V, Zhang Z. Preoperative prediction of perineural invasion and KRAS mutation in colon cancer using machine learning. J Cancer Res Clin Oncol. 2020;146(12):3165–74. .http://dx.doi.org/10.1007/s00432-020-03354-z.
Petkovska I, Tixier F, Ortiz EJ, Golia Pernicka JS, Paroder V, Bates DD, Horvat N, Fuqua J, Schilsky J, Gollub MJ, et al. Clinical utility of radiomics at baseline rectal MRI to predict complete response of rectal cancer after chemoradiation therapy. Abdom Radiol (NY). 2020;45(11):3608–17. .http://dx.doi.org/10.1007/s00261-020-02502-w.
Eresen A, Li Y, Yang J, Shangguan J, Velichko Y, Yaghmai V, Benson AB 3rd, Zhang Z. Preoperative assessment of lymph node metastasis in Colon Cancer patients using machine learning: a pilot study. Cancer Imaging. 2020;20(1):30. http://dx.doi.org/10.1186/s40644-020-00308-z.
Meng X, Xia W, Xie P, Zhang R, Li W, Wang M, Xiong F, Liu Y, Fan X, Xie Y, et al. Preoperative radiomic signature based on multiparametric magnetic resonance imaging for noninvasive evaluation of biological characteristics in rectal cancer. Eur Radiol. 2019;29(6):3200–9. .http://dx.doi.org/10.1007/s00330-018-5763-x.
Rios Velazquez E, Aerts HJ, Gu Y, Goldgof DB, De Ruysscher D, Dekker A, Korn R, Gillies RJ, Lambin P. A semiautomatic CT-based ensemble segmentation of lung tumors: comparison with oncologists' delineations and with the surgical specimen. Radiother Oncol. 2012;105(2):167–73. .http://dx.doi.org/10.1016/j.radonc.2012.09.023.
van Dam IE, van Sörnsen de Koste JR, Hanna GG, Muirhead R, Slotman BJ, Senan S. Improving target delineation on 4-dimensional CT scans in stage I NSCLC using a deformable registration tool. Radiother Oncol. 2010;96(1):67–72. .http://dx.doi.org/10.1016/j.radonc.2010.05.003.
Heye T, Merkle EM, Reiner CS, Davenport MS, Horvath JJ, Feuerlein S, Breault SR, Gall P, Bashir MR, Dale BM, et al. Reproducibility of dynamic contrast-enhanced MR imaging. Part II. Comparison of intra- and interobserver variability with manual region of interest placement versus semiautomatic lesion segmentation and histogram analysis. Radiology. 2013;266(3):812–21. http://dx.doi.org/10.1148/radiol.12120255.
Egger J, Kapur T, Fedorov A, Pieper S, Miller JV, Veeraraghavan H, Freisleben B, Golby AJ, Nimsky C, Kikinis R. GBM volumetry using the 3D Slicer medical image computing platform. Sci Rep. 2013;3:1364. http://dx.doi.org/10.1038/srep01364.
Badic B, Hatt M, Durand S, Jossic-Corcos CL, Simon B, Visvikis D, Corcos L. Radiogenomics-based cancer prognosis in colorectal cancer. Sci Rep. 2019;9(1):9743. http://dx.doi.org/10.1038/s41598-019-46286-6.
Fan S, Li X, Cui X, Zheng L, Ren X, Ma W, Ye Z. Computed Tomography-Based Radiomic Features Could Potentially Predict Microsatellite Instability Status in Stage II Colorectal Cancer: A Preliminary Study. Acad Radiol. 2019;26(12):1633–40. .http://dx.doi.org/10.1016/j.acra.2019.02.009.
Li M, Zhang J, Dan Y, Yao Y, Dai W, Cai G, Yang G, Tong T. A clinical-radiomics nomogram for the preoperative prediction of lymph node metastasis in colorectal cancer. J Transl Med. 2020;18(1):46. http://dx.doi.org/10.1186/s12967-020-02215-0.
Hu T, Wang S, Huang L, Wang J, Shi D, Li Y, Tong T, Peng W. A clinical-radiomics nomogram for the preoperative prediction of lung metastasis in colorectal cancer patients with indeterminate pulmonary nodules. Eur Radiol. 2019;29(1):439–49. .http://dx.doi.org/10.1007/s00330-018-5539-3.
Li M, Sun K, Dai W, Xiang W, Zhang Z, Zhang R, Wang R, Li Q, Mo S, Han L, et al. Preoperative prediction of peritoneal metastasis in colorectal cancer using a clinical-radiomics model. Eur J Radiol. 2020;132:109326. http://dx.doi.org/10.1016/j.ejrad.2020.109326.
O'Farrell AC, Jarzabek MA, Lindner AU, Carberry S, Conroy E, Miller IS, Connor K, Shiels L, Zanella ER, Lucantoni F, et al: Implementing Systems Modelling and Molecular Imaging to Predict the Efficacy of BCL-2 Inhibition in Colorectal Cancer Patient-Derived Xenograft Models. Cancers (Basel) 2020, 12(10).http://dx.doi.org/10.3390/cancers12102978.
Dai W, Mo S, Han L, Xiang W, Li M, Wang R, Tong T, Cai G. Prognostic and predictive value of radiomics signatures in stage I-III colon cancer. Clin Transl Med. 2020;10(1):288–93. .http://dx.doi.org/10.1002/ctm2.31.
Cai C, Hu T, Gong J, Huang D, Liu F, Fu C, Tong T. Multiparametric MRI-based radiomics signature for preoperative estimation of tumor-stroma ratio in rectal cancer. Eur Radiol. 2020. http://dx.doi.org/10.1007/s00330-020-07403-6.
Giannini V, Mazzetti S, Bertotto I, Chiarenza C, Cauda S, Delmastro E, Bracco C, Di Dia A, Leone F, Medico E, et al. Predicting locally advanced rectal cancer response to neoadjuvant therapy with (18)F-FDG PET and MRI radiomics features. Eur J Nucl Med Mol Imaging. 2019;46(4):878–88. http://dx.doi.org/10.1007/s00259-018-4250-6.
Shu Z, Fang S, Ding Z, Mao D, Cai R, Chen Y, Pang P, Gong X. MRI-based Radiomics nomogram to detect primary rectal cancer with synchronous liver metastases. Sci Rep. 2019;9(1):3374. http://dx.doi.org/10.1038/s41598-019-39651-y.
Daye D, Tabari A, Kim H, Chang K, Kamran SC, Hong TS, Kalpathy-Cramer J, Gee MS. Quantitative tumor heterogeneity MRI profiling improves machine learning–based prognostication in patients with metastatic colon cancer. Eur Radiol. 2021. http://dx.doi.org/10.1007/s00330-020-07673-0.
Li M, Li X, Guo Y, Miao Z, Liu X, Guo S, Zhang H. Development and assessment of an individualized nomogram to predict colorectal cancer liver metastases. Quant Imaging Med Surg. 2020;10(2):397–414. .http://dx.doi.org/10.21037/qims.2019.12.16.
Lovinfosse P, Polus M, Van Daele D, Martinive P, Daenen F, Hatt M, Visvikis D, Koopmansch B, Lambert F, Coimbra C, et al. FDG PET/CT radiomics for predicting the outcome of locally advanced rectal cancer. Eur J Nucl Med Mol Imaging. 2018;45(3):365–75. .http://dx.doi.org/10.1007/s00259-017-3855-5.
Chen SW, Shen WC, Chen WT, Hsieh TC, Yen KY, Chang JG, Kao CH. Metabolic Imaging Phenotype Using Radiomics of [(18)F]FDG PET/CT Associated with Genetic Alterations of Colorectal Cancer. Mol Imaging Biol. 2019;21(1):183–90. .http://dx.doi.org/10.1007/s11307-018-1225-8.
Chu Y, Li J, Zeng Z, Huang B, Zhao J, Liu Q, Wu H, Fu J, Zhang Y, Zhang Y, et al. A Novel Model Based on CXCL8-Derived Radiomics for Prognosis Prediction in Colorectal Cancer. Front Oncol. 2020;10:575422. http://dx.doi.org/10.3389/fonc.2020.575422.
Yang C, Jiang ZK, Liu LH, Zeng MS. Pre-treatment ADC image-based random forest classifier for identifying resistant rectal adenocarcinoma to neoadjuvant chemoradiotherapy. Int J Colorectal Dis. 2020;35(1):101–7. .http://dx.doi.org/10.1007/s00384-019-03455-3.
Zhao Y, Yang J, Luo M, Yang Y, Guo X, Zhang T, Hao J, Yao Y, Ma X. Contrast-Enhanced CT-based Textural Parameters as Potential Prognostic Factors of Survival for Colorectal Cancer Patients Receiving Targeted Therapy. Mol Imaging Biol. 2020. http://dx.doi.org/10.1007/s11307-020-01552-2.
Fu J, Zhong X, Li N, Van Dams R, Lewis J, Sung K, Raldow AC, Jin J, Qi XS. Deep learning-based radiomic features for improving neoadjuvant chemoradiation response prediction in locally advanced rectal cancer. Phys Med Biol. 2020;65(7):075001. http://dx.doi.org/10.1088/1361-6560/ab7970.
Boldrini L, Cusumano D, Chiloiro G, Casà C, Masciocchi C, Lenkowicz J, Cellini F, Dinapoli N, Azario L, Teodoli S, et al. Delta radiomics for rectal cancer response prediction with hybrid 0.35 T magnetic resonance-guided radiotherapy (MRgRT): a hypothesis-generating study for an innovative personalized medicine approach. Radiol Med. 2019;124(2):145–53. .http://dx.doi.org/10.1007/s11547-018-0951-y.
Liang C, Huang Y, He L, Chen X, Ma Z, Dong D, Tian J, Liang C, Liu Z. The development and validation of a CT-based radiomics signature for the preoperative discrimination of stage I-II and stage III-IV colorectal cancer. Oncotarget. 2016;7(21):31401–12. .http://dx.doi.org/10.18632/oncotarget.8919.
Huang YQ, Liang CH, He L, Tian J, Liang CS, Chen X, Ma ZL, Liu ZY. Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer. J Clin Oncol. 2016;34(18):2157–64. .http://dx.doi.org/10.1200/jco.2015.65.9128.
Huang Y, He L, Dong D, Yang C, Liang C, Chen X, Ma Z, Huang X, Yao S, Liang C, et al. Individualized prediction of perineural invasion in colorectal cancer: development and validation of a radiomics prediction model. Chin J Cancer Res. 2018;30(1):40–50. .http://dx.doi.org/10.21147/j.issn.1000-9604.2018.01.05.
Yang L, Dong D, Fang M, Zhu Y, Zang Y, Liu Z, Zhang H, Ying J, Zhao X, Tian J. Can CT-based radiomics signature predict KRAS/NRAS/BRAF mutations in colorectal cancer? Eur Radiol 2018, 28(5):2058–2067.http://dx.doi.org/10.1007/s00330-017-5146-8.
Huang X, Cheng Z, Huang Y, Liang C, He L, Ma Z, Chen X, Wu X, Li Y, Liang C, et al. CT-based Radiomics Signature to Discriminate High-grade From Low-grade Colorectal Adenocarcinoma. Acad Radiol. 2018;25(10):1285–97. .http://dx.doi.org/10.1016/j.acra.2018.01.020.
Golia Pernicka JS, Gagniere J, Chakraborty J, Yamashita R, Nardo L, Creasy JM, Petkovska I, Do RRK, Bates DDB, Paroder V, et al. Radiomics-based prediction of microsatellite instability in colorectal cancer at initial computed tomography evaluation. Abdom Radiol (NY). 2019;44(11):3755–63. .http://dx.doi.org/10.1007/s00261-019-02117-w.
Dou Y, Tang X, Liu Y, Gong Z. T stage prediction of colorectal tumor based on multiparametric functional images. Transl Cancer Res. 2020;9(2):522–8. http://dx.doi.org/10.21037/tcr.2019.11.41.
Liu Y, Dou Y, Lu F, Liu L. A study of radiomics parameters from dual-energy computed tomography images for lymph node metastasis evaluation in colorectal mucinous adenocarcinoma. Medicine. 2020;99(11):e19251. http://dx.doi.org/10.1097/md.0000000000019251.
He K, Liu X, Li M, Li X, Yang H, Zhang H. Noninvasive KRAS mutation estimation in colorectal cancer using a deep learning method based on CT imaging. BMC Med Imaging. 2020;20(1):59. http://dx.doi.org/10.1186/s12880-020-00457-4.
Liang M, Cai Z, Zhang H, Huang C, Meng Y, Zhao L, Li D, Ma X, Zhao X. Machine Learning-based Analysis of Rectal Cancer MRI Radiomics for Prediction of Metachronous Liver Metastasis. Acad Radiol. 2019;26(11):1495–504. .http://dx.doi.org/10.1016/j.acra.2018.12.019.
Shi L, Zhang Y, Nie K, Sun X, Niu T, Yue N, Kwong T, Chang P, Chow D, Chen JH, et al. Machine learning for prediction of chemoradiation therapy response in rectal cancer using pre-treatment and mid-radiation multi-parametric MRI. Magn Reson Imaging. 2019;61:33–40. http://dx.doi.org/10.1016/j.mri.2019.05.003.
Horvat N, Veeraraghavan H, Khan M, Blazic I, Zheng J, Capanu M, Sala E, Garcia-Aguilar J, Gollub MJ, Petkovska I. MR Imaging of Rectal Cancer: Radiomics Analysis to Assess Treatment Response after Neoadjuvant Therapy. Radiology. 2018;287(3):833–43. http://dx.doi.org/10.1148/radiol.2018172300.
Liu H, Zhang C, Wang L, Luo R, Li J, Zheng H, Yin Q, Zhang Z, Duan S, Li X, et al. MRI radiomics analysis for predicting preoperative synchronous distant metastasis in patients with rectal cancer. Eur Radiol. 2019;29(8):4418–26. .http://dx.doi.org/10.1007/s00330-018-5802-7.
Li Y, Liu W, Pei Q, Zhao L, Güngör C, Zhu H, Song X, Li C, Zhou Z, Xu Y, et al. Predicting pathological complete response by comparing MRI-based radiomics pre- and postneoadjuvant radiotherapy for locally advanced rectal cancer. Cancer Med. 2019;8(17):7244–52. .http://dx.doi.org/10.1002/cam4.2636.
Nie K, Shi L, Chen Q, Hu X, Jabbour SK, Yue N, Niu T, Sun X. Rectal Cancer: Assessment of Neoadjuvant Chemoradiation Outcome based on Radiomics of Multiparametric MRI. Clin Cancer Res. 2016;22(21):5256–64. .http://dx.doi.org/10.1158/1078-0432.Ccr-15-2997.
Zhou X, Yi Y, Liu Z, Cao W, Lai B, Sun K, Li L, Zhou Z, Feng Y, Tian J. Radiomics-Based Pretherapeutic Prediction of Non-response to Neoadjuvant Therapy in Locally Advanced Rectal Cancer. Ann Surg Oncol. 2019;26(6):1676–84. .http://dx.doi.org/10.1245/s10434-019-07300-3.
Liu Z, Zhang XY, Shi YJ, Wang L, Zhu HT, Tang Z, Wang S, Li XT, Tian J, Sun YS. Radiomics Analysis for Evaluation of Pathological Complete Response to Neoadjuvant Chemoradiotherapy in Locally Advanced Rectal Cancer. Clin Cancer Res. 2017;23(23):7253–62. .http://dx.doi.org/10.1158/1078-0432.Ccr-17-1038.
Cui Y, Yang X, Shi Z, Yang Z, Du X, Zhao Z, Cheng X. Radiomics analysis of multiparametric MRI for prediction of pathological complete response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer. Eur Radiol. 2019;29(3):1211–20. .http://dx.doi.org/10.1007/s00330-018-5683-9.
Horvat N, Veeraraghavan H, Pelossof RA, Fernandes MC, Arora A, Khan M, Marco M, Cheng CT, Gonen M, Golia Pernicka JS, et al. Radiogenomics of rectal adenocarcinoma in the era of precision medicine: A pilot study of associations between qualitative and quantitative MRI imaging features and genetic mutations. Eur J Radiol. 2019;113:174–81. http://dx.doi.org/10.1016/j.ejrad.2019.02.022.
He B, Ji T, Zhang H, Zhu Y, Shu R, Zhao W, Wang K. MRI-based radiomics signature for tumor grading of rectal carcinoma using random forest model. J Cell Physiol. 2019;234(11):20501–9. http://dx.doi.org/10.1002/jcp.28650.
Oh JE, Kim MJ, Lee J, Hur BY, Kim B, Kim DY, Baek JY, Chang HJ, Park SC, Oh JH, et al. Magnetic Resonance-Based Texture Analysis Differentiating KRAS Mutation Status in Rectal Cancer. Cancer Res Treat. 2020;52(1):51–9. .http://dx.doi.org/10.4143/crt.2019.050.
Balagurunathan Y, Gu Y, Wang H, Kumar V, Grove O, Hawkins S, Kim J, Goldgof DB, Hall LO, Gatenby RA, et al. Reproducibility and Prognosis of Quantitative Features Extracted from CT Images. Transl Oncol. 2014;7(1):72–87. .http://dx.doi.org/10.1593/tlo.13844.
Fotina I, Lütgendorf-Caucig C, Stock M, Pötter R, Georg D. Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy. Strahlenther Onkol. 2012;188(2):160–7. .http://dx.doi.org/10.1007/s00066-011-0027-6.
Chalkidou A, O'Doherty MJ, Marsden PK. False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review. PLoS One. 2015;10(5):e0124165. http://dx.doi.org/10.1371/journal.pone.0124165.
Guyon IM. Andr, #xe, Elisseeff: An introduction to variable and feature selection. The Journal of Machine Learning Research 2003.
Babyak MA. What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med. 2004;66(3):411–21. http://dx.doi.org/10.1097/01.psy.0000127692.23278.a9.
van Helden EJ, Vacher YJL, van Wieringen WN, van Velden FHP, Verheul HMW, Hoekstra OS, Boellaard R. Menke-van der Houven van Oordt CW: Radiomics analysis of pre-treatment [(18)F]FDG PET/CT for patients with metastatic colorectal cancer undergoing palliative systemic treatment. Eur J Nucl Med Mol Imaging. 2018;45(13):2307–17. http://dx.doi.org/10.1007/s00259-018-4100-6.
Zwanenburg A. Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis. Eur J Nucl Med Mol Imaging. 2019;46(13):2638–55. http://dx.doi.org/10.1007/s00259-019-04391-8.
Taylor JM, Ankerst DP, Andridge RR. Validation of biomarker-based risk prediction models. Clin Cancer Res. 2008;14(19):5977–83. http://dx.doi.org/10.1158/1078-0432.Ccr-07-4534.
Zwanenburg A, Vallières M, Abdalah MA, Aerts H, Andrearczyk V, Apte A, Ashrafinia S, Bakas S, Beukinga RJ, Boellaard R, et al. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology. 2020;295(2):328–38. http://dx.doi.org/10.1148/radiol.2020191145.
Clarke LP, Nordstrom RJ, Zhang H, Tandon P, Zhang Y, Redmond G, Farahani K, Kelloff G, Henderson L, Shankar L, et al. The Quantitative Imaging Network: NCI's Historical Perspective and Planned Goals. Transl Oncol. 2014;7(1):1–4. http://dx.doi.org/10.1593/tlo.13832.
Buckler AJ, Bresolin L, Dunnick NR, Sullivan DC. A collaborative enterprise for multi-stakeholder participation in the advancement of quantitative imaging. Radiology. 2011;258(3):906–14. .http://dx.doi.org/10.1148/radiol.10100799.
Aerts HJ, Velazquez ER, Leijenaar RT, Parmar C, Grossmann P, Carvalho S, Bussink J, Monshouwer R, Haibe-Kains B, Rietveld D, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun. 2014;5:4006. http://dx.doi.org/10.1038/ncomms5006.
Shafiq-Ul-Hassan M, Latifi K, Zhang G, Ullah G, Gillies R, Moros E. Voxel size and gray level normalization of CT radiomic features in lung cancer. Sci Rep. 2018;8(1):10545. http://dx.doi.org/10.1038/s41598-018-28895-9.
Zwanenburg A, Leger S, Agolli L, Pilz K, Troost EGC, Richter C, Löck S. Assessing robustness of radiomic features by image perturbation. Sci Rep. 2019;9(1):614. http://dx.doi.org/10.1038/s41598-018-36938-4.
Lazar C, Meganck S, Taminau J, Steenhoff D, Coletta A, Molter C, Weiss-Solís DY, Duque R, Bersini H, Nowé A. Batch effect removal methods for microarray gene expression data integration: a survey. Brief Bioinform. 2013;14(4):469–90. .http://dx.doi.org/10.1093/bib/bbs037.
Johnson WE, Li C, Rabinovic A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics. 2007;8(1):118–27. http://dx.doi.org/10.1093/biostatistics/kxj037.
Orlhac F, Boughdad S, Philippe C, Stalla-Bourdillon H, Nioche C, Champion L, Soussan M, Frouin F, Frouin V, Buvat I. A Postreconstruction Harmonization Method for Multicenter Radiomic Studies in PET. J Nucl Med. 2018;59(8):1321–8. .http://dx.doi.org/10.2967/jnumed.117.199935.
Orlhac F, Frouin F, Nioche C, Ayache N, Buvat I. Validation of A Method to Compensate Multicenter Effects Affecting CT Radiomics. Radiology. 2019;291(1):53–9. .http://dx.doi.org/10.1148/radiol.2019182023.
Fortin JP, Cullen N, Sheline YI, Taylor WD, Aselcioglu I, Cook PA, Adams P, Cooper C, Fava M, McGrath PJ, et al. Harmonization of cortical thickness measurements across scanners and sites. Neuroimage. 2018;167:104–20. http://dx.doi.org/10.1016/j.neuroimage.2017.11.024.
Ibrahim A, Primakov S, Beuque M, Woodruff HC, Halilaj I, Wu G, Refaee T, Granzier R, Widaatalla Y, Hustinx R, et al. Radiomics for precision medicine: Current challenges, future prospects, and the proposal of a new framework. Methods. 2021;188:20–9. http://dx.doi.org/10.1016/j.ymeth.2020.05.022.
Ligero M, Jordi-Ollero O, Bernatowicz K, Garcia-Ruiz A, Delgado-Muñoz E, Leiva D, Mast R, Suarez C, Sala-Llonch R, Calvo N, et al. Minimizing acquisition-related radiomics variability by image resampling and batch effect correction to allow for large-scale data analysis. Eur Radiol. 2021;31(3):1460–70. http://dx.doi.org/10.1007/s00330-020-07174-0.
Peerlings J, Woodruff HC, Winfield JM, Ibrahim A, Van Beers BE, Heerschap A, Jackson A, Wildberger JE, Mottaghy FM, DeSouza NM, et al. Stability of radiomics features in apparent diffusion coefficient maps from a multi-centre test-retest trial. Sci Rep. 2019;9(1):4800. http://dx.doi.org/10.1038/s41598-019-41344-5.
Sunil RJ. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Journal of the American Statistical Association 2003.
Sollini M, Antunovic L, Chiti A, Kirienko M. Towards clinical application of image mining: a systematic review on artificial intelligence and radiomics. Eur J Nucl Med Mol Imaging. 2019;46(13):2656–72. .http://dx.doi.org/10.1007/s00259-019-04372-x.
Moreira JM, Santiago I, Santinha J, Figueiredo N, Marias K, Figueiredo M, Vanneschi L, Papanikolaou N. Challenges and Promises of Radiomics for Rectal Cancer. Curr Colorectal Cancer Rep. 2019;15(6):175–80. .http://dx.doi.org/10.1007/s11888-019-00446-y.
Neri E, Del Re M, Paiar F, Erba P, Cocuzza P, Regge D, Danesi R. Radiomics and liquid biopsy in oncology: the holons of systems medicine. Insights Imaging. 2018;9(6):915–24. .http://dx.doi.org/10.1007/s13244-018-0657-7.
Cheng T, Zhan X. Pattern recognition for predictive, preventive, and personalized medicine in cancer. Epma j. 2017;8(1):51–60. http://dx.doi.org/10.1007/s13167-017-0083-9.

Download PDF

Journal Publication

published 14 Apr, 2023

Read the published version in BMC Gastroenterology →

Reviewers agreed at journal
23 May, 2022
Reviews received at journal
04 May, 2022
Reviewers invited by journal
09 Mar, 2022
First submitted to journal
20 Oct, 2021

You are reading this latest preprint version

Repeatability of Radiomics Studies in Colorectal Cancer: A Systematic Review

Status:

Journal Publication

Version 1

Abstract

Figures

1. Introduction

2. Methods

2.1 Review Strategy

2.2 Study Extraction

2.3 Data synthesis

2.4 Quality Assessment

3. Results

3.1 RQS score

3.2.1 Intra-individual repeatability

3.2.2 Acquisition parameters

3.2.3 Segmentation

3.2.4 Feature selection

3.2.5 Modelling methodology

3.2.6 Evaluation

3.3 How to increase repeatability

3.3.1 Standardization protocol

3.3.2 Test-retest reliability

3.3.3 Phantom

3.3.4 Post-processing

4. Discussion

5. Conclusion

Abbreviations

Declarations

References

Supplementary Files

Status:

Journal Publication

Version 1