Predicting Central Lymph Node Metastasis in Papillary Thyroid Carcinoma Using a Fusion Model of Vision Transformer and Traditional Radiomics Based on Dynamic Dual-Modality Ultrasound

doi:10.21203/rs.3.rs-4858879/v1

Download PDF

Research Article

Predicting Central Lymph Node Metastasis in Papillary Thyroid Carcinoma Using a Fusion Model of Vision Transformer and Traditional Radiomics Based on Dynamic Dual-Modality Ultrasound

https://doi.org/10.21203/rs.3.rs-4858879/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Objective

This study aimed to develop a novel fusion model based on dynamic dual-modality with B-mode ultrasound and superb microvascular imaging (SMI), combining Vision Transformer (ViT) and radiomics features to predict central lymph node metastasis (CLNM) in thyroid cancer patients.

Method

In this retrospective diagnostic study, 310 patients with pathologically confirmed papillary thyroid carcinoma from two hospitals were included. We trained ViT models for B-mode and SMI, then extracted ViT and radiomics features from their video images. Initially, Single-modality models were developed, including the B-mode radiomics model (BMUS_RAD) and the B-mode ViT model (BMUS_ViT). Subsequently, Dual-modality models were constructed, encompassing the Dual-modality radiomics model (DMU_RAD), the Dual-modality ViT model (DMU_ViT), and finally, the integrated model DMU_RAD_ViT, to enhance the prediction of CLNM. The performance of each model was compared, and SHAP was utilized for the visual interpretation of the novel fusion model.

Results

Among all the models, the fusion model DMU_RAD_ViT performed the best (AUC = 0.901, p < 0.05). At the same time, the dual-modality model DMU_RAD(AUC = 0.856) and DMU_ViT(AUC = 0.832) is also higher than the single-modal model BMUS_RAD (AUC = 0.837) and BMUS_ViT (AUC = 0.789), respectively. SHAP analysis revealed that 16 radiomics and ViT features from both modalities contributed to the DMU_RAD_ViT model.

Conclusions

The Dual-modality fusion model, integrating both radiomics and ViT features, can be utilized to predict CLNM.

Dynamic ultrasound

Papillary thyroid carcinoma

Cervical lymph node metastasis Superb Microvascular Imaging

Vision Transformer

Thyroid cancer ranks among the fastest-growing malignancies worldwide, with 85%-95% of cases being papillary thyroid carcinoma(PTC)[1, 2]. Although PTC is an indolent tumor with a generally favorable prognosis compared to other cancers, about 30%-80% of PTC cases are prone to cervical lymph node metastasis (CLNM). Currently, surgery remains the preferred treatment for thyroid cancer[3]. Accurate preoperative assessment of CLNM is crucial for determining the surgical strategy[4]. High-resolution ultrasound is the primary method for preoperative detection of lymph node metastasis (LNM), However, the complex anatomy of the neck, especially in the central region, combined with the heavy workload and time constraints faced by ultrasound physicians, often leads to missed diagnoses of CLNM. Studies have shown that the sensitivity of conventional ultrasound for detecting CLNM is less than 40%[5].

In the past few years, superb microvascular imaging (SMI) has garnered significant attention for its application in organs such as the liver and breast[6]. SMI is a novel ultrasound vascular imaging technique that excels at displaying low-velocity blood flow, allowing for the visualization of small blood vessels within and around the thyroid[7]. Compared to color/power Doppler imaging (CDI/PDI), SMI can provide a more detailed depiction of blood flow within the nodules[8]. Some studies have indicated that SMI demonstrates diagnostic performance in distinguishing between benign and malignant thyroid nodules that are comparable to contrast-enhanced ultrasound (CEUS)[9]. Moreover, when combined with B-mode ultrasound (BMUS), SMI can significantly enhance the diagnostic accuracy for thyroid nodules[10].

Advancements in computational power and AI algorithms have significantly enhanced diagnostic accuracy and efficiency in various medical examinations[11]. Radiomics, an advanced image analysis technique, extracts subtle details from medical images that are not visible to the naked eye[12]. These insights are invaluable for both preoperative predictions and postoperative evaluations of diseases[13]. Notably, in the diagnosis and treatment of thyroid conditions, radiomics serves as a powerful tool, offering detailed auxiliary information for diagnosis, treatment strategies, and prognostic outcomes[14]. Research shows that radiomics can predict CLNM using BMUS[15]. Concurrently, deep learning(DL), an AI technique inspired by human brain function, shines in discerning intricate image details, notably through convolutional neural networks(CNNs)[16]. With its multi-layered architecture, DL autonomously learns diverse features across abstraction levels[17]. Compared to conventional radiomics, DL excels in extracting abstract image features[18]. Research indicates that the fusion models of DL features and radiomics features outperform standalone models due to their complementary strengths[19, 20]. However, CNNs fail to effectively capture long-range dependencies and contextual information[21]. Since Dosovitskiy et al. introduced the Vision Transformer (ViT), it has emerged as a pioneering method in image recognition, challenging conventional CNN architectures[22]. By employing the self-attention mechanism on image patches, ViTs excel at capturing global dependencies, thereby enhancing the understanding of context and relationships within images[23]. This ability has resulted in remarkable success in tasks like image classification, object detection, and image segmentation [24–26].

In previous studies, most research has focused on two-dimensional(2D) images. However, 2D images often lack comprehensive information and may miss critical structural details of lesions. In recent years, advancements in neural network architectures, data augmentation techniques, and high-end graphics processors have significantly expanded the scope of three-dimensional(3D) DL applications across various medical imaging modalities[27]. 3D DL can leverage the strengths of both 1D and 2D DL by simultaneously extracting spectral and spatial features from the input volume. These 3D DL features are particularly valuable for analyzing volumetric data in medical imaging. Nonetheless, 3D models have significantly more parameters than 2D models, necessitating a large amount of training data to avoid overfitting. In response to this challenge, some studies have proposed 2.5-dimensional (2.5D) approaches[28], which can effectively mitigate the risk of overfitting while extracting more information than 2D methods.

Collectively, in this study, we will innovatively combine BMUS and SMI to develop a fusion model to predict CLNM. To enhance the model's predictive performance, we will utilize 2.5D images and comprehensively integrate radiomics and ViT features to capture tumor characteristics.

Study Population

This retrospective study received approval from the Institutional Review Board (No.2024-A 06). 310 patients with PTC across two hospitals in China were collected for this study. The training and internal validation sets comprised 249 patients who underwent surgery at Nantong Tumor Hospital between September 1, 2022, and March 1, 2024. The external validation set included 61 patients at the First People's Hospital of Qinzhou, Guangxi, between September 3, 2021, and March 23, 2023. (Fig. 1).

Inclusion criteria: (1) Pathologically confirmed PTC diagnosis. (2) Postoperative pathology clarified the presence or absence of CLNM. (3) First-time thyroid surgery. (4) Complete and clear ultrasound bimodal video, including BMUS and SMI videos of the entire nodule. (5) Ultrasound examination conducted within two weeks before surgery. (6) Complete clinical baseline data.

Exclusion criteria: (1) Patients who received preoperative therapy, such as radiofrequency ablation. (2) Patients undergoing repeat surgery. (3) Unclear postoperative pathology. (4) Blurred or incomplete ultrasound video. (5) Incomplete clinical baseline data.

Clinical baseline data collection: contains gender, age, Size, Pathological results, and location.

Dual-Modality Ultrasound Image Collection

Ultrasound images were obtained using a Samsung (Samsung RS85 systems, Seoul, Korea) or Canon (Canon Aplio 800 systems, Tokyo, Japan) ultrasound device with the linear array probe at 10 to 15 MHz. The patient was placed in the supine position with the head tilted posteriorly to expose the neck region fully. The probe was used to scan the nodule region uniformly and continuously along the long axis from left to right, and the video images were stored. SMI was then performed by the same radiologist using the same line array probe at the same speed in the same direction in successive uniform sweeps, and the SMI video images were stored.

Data Preprocessing and Nodule Segmentation

In the video, keyframes are extracted from the start to the end of the nodule. Then, two images are evenly taken from each side of the largest cross-sectional area of the nodule, resulting in a total of five images per nodule (Fig. 2). The regions of interest (ROIs) were manually delineated using the open-source software ITKSNAP 3.8.0. Two radiologists with more than 5 years of experience in thyroid ultrasound examination independently delineated the tumor boundaries without knowledge of postoperative pathology, resulting in two sets of ROIs. Afterwards, the reproducibility of radiomics features was assessed by calculating the Interclass Correlation Coefficient (ICC), and features with an ICC value less than 0.75 were excluded. The same radiologist delineated ROIs on the SMI images based on their corresponding BMUS images.

Radiomics Feature Extraction

The radiomics features were automatically extracted using the open-source software Pyradiomics (https://pyradiomics.readthedocs.io/en/latest/index.html). A total of 1567 radiomics features were extracted from the ROI of each ultrasound image, including first-order features, shapebased features, and texture features.

Feature selection and model construction

To eliminate the effect of magnitude between different features, making the model more stable and faster convergence during the training process, we first standardize the features with Z-score. Then we use the U-test to select features with significant differences. Furthermore, to reduce redundant and highly correlated features and remove useless information to improve the classification effectiveness of the model, we use Spearman's correlation coefficient to assess the multicollinearity of the features[29], and if any coefficient value ≥ 0.9 exists in a pair of features, only the features with better diagnostic performance are retained. Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression was used for the final feature selection [30]. We employed Support Vector Machine (SVM) classifiers to develop the prediction model. SVM is a commonly used method in biomedical binary classification problems, aiming to identify the optimal hyperplane that best separates different targets, known as the optimal decision boundary[29]. It has been widely utilized in numerous research studies.

ViT Feature Extraction

In this study, we applied the ViT model to extract features from thyroid ultrasound images. Initially, after cropping, images underwent resizing to a standardized 256 × 256 pixel size using linear interpolation. Data augmentation included random horizontal and vertical flips. Subsequently, the ultrasound images were partitioned into 8x8 patches. Each patch was linearly projected into a 1024-dimensional feature vector using learnable weight matrices. To enhance feature representation, a learnable embedding class token was introduced at the sequence start, and each linearly projected patch received a learnable positional encoding to preserve spatial context.

Following preprocessing, the sequence of image patches and positional encodings was fed into six Transformer encoder layers. Each encoder layer comprised layer normalization and multi-head self-attention with 16 attention heads for parallel computation and aggregation of attention scores. Subsequently, a second layer normalization was followed by two fully connected networks using ReLU activation, with output dimensions of 768 and 1024, respectively. Residual connections between subcomponents facilitated the propagation of feature representations across encoder layers.

In the final Transformer encoder layer, the class token's hidden state was transformed into a flattened feature vector, serving as the definitive representation of thyroid ultrasound image features. Post-training, 1024 ViT features were extracted from the last layer output 'to latent' for each image.

Feature fusion and fusion model construction

In this study, all of our fusion strategies use early fusion, combining features from different modalities into a single feature vector[31]. These features were subsequently normalized using z-score normalization. Then the features of each fusion model were downscaled by Spearman correlation analysis, U-test, and LASSO analysis respectively. Finally, the DMU_RAD, DMU_ViT, and DMU_RAD_ViT models were constructed using SVM classifiers, respectively. The detailed method of model construction is described in the "Feature Selection and Model Construction" section above.

Statistical analysis

The baseline data of patients were analyzed using R language (version 4.3.3, Index of /src/base/R-4 (r-project.org)) and the compareGroups package. Continuous variables were summarized as mean ± standard deviation and categorical variables were described using frequencies and percentages. The normality of continuous variables was assessed using the Shapiro-Wilk test. Differences between groups were evaluated using the Mann–Whitney U test or Student's t-test for continuous variables, and the Chi-squared test or Fisher's exact test for categorical variables. Statistical significance was defined as p < 0.05 (two-sided). The DeLong method was employed to compare the AUCs of different models.

Patient Characteristics

A total of 310 patients (237 females, 73 males, mean age 44.8 years ± 12.8 years [SD]) were analyzed in this study, which contained 199 cases from the internal training set, 50 cases from the internal validation set, and 61 cases from the external validation set. The basic characteristics of the internal training set, internal validation set, and external validation set are shown in Table 1. No statistically significant differences were found in the basic characteristics and lymph node metastasis rates among the three study sets (p > 0.05).

Table 1

Baseline characteristics of study sets.
	Training set	Internal test set	External test set	p-Value
	N = 199	N = 50	N = 61
Gender:				0.282
F	155 (77.9%)	40 (80.0%)	42 (68.9%)
M	44 (22.1%)	10 (20.0%)	19 (31.1%)
Age	44.0 [34.0;55.0]	46.5 [35.2;54.0]	43.0 [37.0;49.0]	0.955
Size	1.00 [0.60;1.50]	0.90 [0.70;1.40]	0.90 [0.60;1.40]	0.804
Pathological Results:				0.913
Non-CLNM	99 (49.7%)	26 (52.0%)	32 (52.5%)
CLNM	100 (50.3%)	24 (48.0%)	29 (47.5%)
Location:				0.265
Left	88 (44.2%)	25 (50.0%)	19 (31.1%)
Right	99 (49.7%)	22 (44.0%)	36 (59.0%)
Isthmus	12 (6.03%)	3 (6.00%)	6 (9.84%)
CLNM: central lymph node metastasis

The feature extraction and selection

We ultimately extracted 3,122 radiomics features (1,561 each for BMUS and SMI) and 2,048 ViT features (1,024 each for BMUS and SMI). To ensure comparability of the models, we first normalized the data. Then we removed features with ICC less than 0.75 to ensure feature reproducibility. For each model, we independently conducted the Mann-Whitney U test, Spearman correlation analysis, and LASSO analysis to reduce the dimensionality of the data. LASSO was used to determine the optimal features for each model. Specifically, BMUS_RAD selected 6 radiomics features, BMUS_ViT selected 5 ViT features, DMU_RAD selected 10 BMUS and 8 SMI radiomics features, DMU_ViT selected 4 BMUS features and 3 SMI features, and DMU_RAD_ViT selected 5 BMUS and 5 radiomics features, as well as 3 BMUS and 3 SMI ViT features. The Spearman correlation coefficients indicate that the majority of correlations are below 0.6 (Fig. 3). Additionally, compared to the correlations between radiomics and ViT features, the correlation between radiomics features and ViT features is relatively low. Similarly, the correlation between BMUS and DMU modalities is lower than the correlations within their respective modalities' internal features.

Radiomics Models of BMUS and DMU

All models were constructed using the SVM algorithm. All results are summarized in Table 2. In the training set, the AUC of the DMU_RAD model(0.856; 95%CI: 0.803–0.909) was slightly higher than that of the BMUS_RAD model(0.837;95%CI༚0.782–0.891), but this difference was not statistically significant (P > 0.05). Similar results were observed in the internal test set, where the AUCs for the DMU_RAD model and the BMUS_RAD model were 0.809(95%CI༚0.686–0.933) and 0.782 (95%CI༚0.652–0.912), respectively. The AUC of the DMU_RAD model was superior to that of the BMUS_RAD model, but the difference was not significant (P > 0.05). In the external test set, both BMUS_RAD and DMU_RAD showed relatively poor performance, with AUCs of 0.673 (95%CI༚0.538–0.809) and 0.653 (95%CI༚0.514–0.793), respectively. Furthermore, while BMUS_RAD exhibited better sensitivity in the training set, its performance deteriorated in the test set (0.583 in the internal validation sets, 0.552 in the external test set).

Table 2

Performances of the predictive models in the training and test sets
Model and metric	AUC	95% CI	ACC (%)	SEN (%)	SPE (%)	PPV (%)	NPV (%)
Training set
BMUS_RAD	0.837	0.782–0.891	76.4	82.0	70.7	73.9	79.5
DMU_RAD	0.856	0.803–0.909	77.9	75.0	80.8	79.8	76.2
BMUS_ViT	0.789	0.726–0.851	70.9	56.0	85.9	80.0	65.9
DMU_ViT	0.832	0.775–0.888	74.9	63.0	86.9	82.9	69.9
DMU_RAD_ViT	0.901	0.860–0.943	82.4	85.0	79.8	81.0	84.0
Internal test set
BMUS_RAD	0.782	0.652–0.912	74.0	58.3	88.5	82.4	69.7
DMU_RAD	0.809	0.686–0.933	76.0	79.2	73.1	73.1	79.2
BMUS_ViT	0.747	0.610–0.883	70.0	58.3	80.8	73.7	67.7
DMU_ViT	0.804	0.681–0.927	74.0	75.0	73.1	72.0	76.0
DMU_RAD_ViT	0.865	0.765–0.966	80.0	83.3	76.9	76.9	83.3
External test set
BMUS_RAD	0.673	0.538–0.809	63.9	55.2	71.9	64.0	63.9
DMU_RAD	0.653	0.514–0.793	62.3	51.7	71.9	62.5	62.2
BMUS_ViT	0.740	0.610–0.871	70.5	72.4	68.7	67.7	73.3
DMU_ViT	0.754	0.631–0.878	70.5	65.5	75.0	70.4	70.6
DMU_RAD_ViT	0.821	0.714–0.928	75.4	75.9	75.0	73.3	77.4

AUC, area under the receiver operating characteristic curve; ACC, accuracy; SEN, sensitivity; SPE, specificity; PPV, positive predictive value; NPV, negative predictive value;

ViT Models of BMUS and DMU

Utilizing the same methodology, we developed the BMUS_ViT and DMU_ViT models. In the training set, the AUC of the DMU_ViT model (0.832; 95%CI: 0.775–0.888) was significantly higher than that of the BMUS_ViT model(0.789; 95%CI༚0.726–0.851), P < 0.05. Both in the internal and external test sets, the DMU_ViT model consistently outperformed the BMUS_ViT model in terms of AUC, but the differences were not statistically significant (P > 0.05). Notably, the BMUS_ViT and DMU_ViT models demonstrated excellent specificity༈0.859, 0.869༉in the training set, while their sensitivity was comparatively lower (0.560, 0.630).

Performance Evaluation of the Final Fusion Model

In the training set, the AUC of DMU_RAD_ViT was 0.91(95%CI: 0.860–0.943), which is the highest AUC among all models. All differences are statistically significant (P < 0.05). In the internal and external tests set, the AUC was 0.865(95%CI༚0.765–0.966) and 0.821(95%CI༚0.714–0.928), respectively, which are higher than those of the other models(Fig. 4A-C). However, most differences are not statistically significant(p>0.05)(Table 2). In terms of sensitivity and specificity, DMU_RAD_ViT also performed well, achieving sensitivity and specificity values of 0.850, 0.798; 0.833, 0.769; 0.759, and 0.750, respectively, in all sets. The calibration curves indicated that DMU_RAD_ViT has a good fit in all sets (Fig. 4D-F). The decision curve analysis showed that the DMU_RAD_ViT model offered a higher benefit compared to the other models (Fig. 4G-I). In addition, to explain the model and assess the impact of different features, we used SHapley Additive exPlanations (SHAP), which provide a unified measure of feature importance[32]. To make the results more intuitive, we created visualization images based on SHAP values (Fig. 5). Although the number of ViT features (6) is fewer than the number of RAD features (10), ViT features greater impact on the model's prediction results compared to RAD features.

In our study, based on dynamic videos of BMUS and DMU, we separately established radiomics and ViT models, as well as a final fusion model that combines both. Additionally, we compared the effectiveness of each model in predicting CLNM and assessed the contribution of radiomics and ViT features to the final fusion model. The results demonstrated that the DMU_RAD_ViT model had the best performance, achieving AUCs of 0.901,0.865,0.821 across all sets. Furthermore, the 16 features selected to construct the DMU_RAD_ViT model were derived from BMUS radiomics and ViT, as well as SMI radiomics and ViT, demonstrating a comprehensive integration of these modalities.

As the preferred screening method for the thyroid, ultrasound is highly sensitive for distinguishing between benign and malignant thyroid nodules and has an important impact on surgical approaches[3, 33]. Unfortunately, ultrasound performs poorly in assessing CLNM[5]. In recent years, artificial intelligence has made significant advancements in the field of medicine[34]. Liu et al developed a radiomics model to predict LNM with an AUC of 0.782 in the training set and 0.727 in the testing set[35]. Sun et al established a model called TC-ViT to classify the benign and malignant thyroid nodules with an AUC of 0.869[26]. This demonstrates that using radiomics and ViT to predict CLNM is feasible. Huang et al developed three integrated models using radiomics and DL features to predict pathological complete response in three breast cancer subtypes: HR+/HER2-, HER2+, and TNBC[36]. These models achieved excellent performance with AUCs of 0.959, 0.974, and 0.958 in the training set, respectively. This demonstrates the great potential of integrated models combining radiomics features and DL features.

Early fusion is one of the most commonly used techniques in many multimodal learning applications[37]. The aim is to improve model performance by exploiting the interactions and correlations between different modal features. In comparison to single-modal models, early fusion techniques allow for a more comprehensive understanding of the data by simultaneously considering information from different modalities during the model learning process. This integrated approach to feature learning helps discover potential correlations hidden in different modalities, leading to more accurate and reliable predictions. In our study, we developed a model that integrates radiomics features and ViT features to take full advantage of both. To the best of our knowledge, no study combines radiomics features and ViT features to predict CLNM. The results showed that the DMU_RAD_ViT model had a higher performance compared to DMU_RAD and DMU_ViT, with AUCs of 0.901, 0.865, and 0.821 across all sets, respectively. After LASSO regression, 10 radiomics features and 6 ViT features were selected for constructing the final model. This selection underscores the critical contributions of both radiomics and ViT features to the final model.

Currently, most studies in the field of ultrasound artificial intelligence are based on 2D images[38–40]. However, in actual clinical practice, tumors are typically three-dimensional, and relying solely on 2D images often leads to the omission of critical information about the tumor, thereby reducing the predictive performance of the model. In contrast, 3D DL can extract more complex information from the tumor. The advantages of 3D DL include the ability to capture the spatial context and volumetric structure of the tumor, which can lead to more accurate and comprehensive assessments. It allows for better feature representation and improves the detection and characterization of abnormalities. However, for 3D DL, effective learning of complex spatial features necessitates a large dataset. Inadequate data may result in suboptimal performance or overfitting[27]. To address these limitations, some have proposed the concept of 2.5-dimension (2.5D). This approach combines the advantages of both 2D and 3D techniques, enabling the capture of spatial information in images while reducing computational parameters. Kim et al. developed the 2.5D DenseNet model and the 3D DenseNet model to differentiate invasive pulmonary adenocarcinomas among subsolid nodules and then compared the performance of the two models[28]. The results showed that the 2.5D DenseNet model exhibited higher diagnostic performances, with an AUC of 0.92, compared to the 3D DenseNet model. In this study, considering the limitation in sample size, we used a 2.5D DL approach to avoid overfitting while utilizing the strengths of 3D models. To capture all the features of each nodule as comprehensively as possible, we extracted 5 images from each nodule.

SMI adaptively filters random motion while preserving directional blood flow, providing more sensitive and higher-resolution blood flow evaluation compared to CDI or PDI[41]. Lu et al. [9]compared SMI and CDI/PDI in displaying microvascular blood flow within thyroid nodules and evaluated the diagnostic performance of SMI combined with conventional ultrasound. The results showed that SMI is significantly superior to CDI/PDI in detecting microvascular blood flow signals. Additionally, the accuracy of SMI combined with BMUS is comparable to that of CEUS. The study by Li et al.[42] demonstrates that the morphological characteristics of different SMI patterns play a significant role in the differential diagnosis of thyroid nodules. In our study, we combined BMUS and SMI to develop the DMU model. The findings indicated that the diagnostic capabilities of the dual-modality models surpassed those of the single-modality models, highlighting the crucial role of SMI in predicting lymph node metastasis.

SHAP makes machine learning interpretable by visualizing the contributions of individual features. It assigns importance values to features, enhancing the transparency and understanding of model predictions. SHAP bar and summary plots show that both BMUS features and SMI features play a significant role in predicting CLMN. In terms of model efficacy, whether constructed with radiomics or ViT features, the DMU model consistently outperforms the BMUS model. However, unexpectedly, this advantage is not apparent in the external test set; indeed, the AUC of DMU_RAD on the test set is lower than that of BMUS_RAD. We infer that this is due to the slightly poorer generalization ability of DMU_RAD, which led to their slightly lower diagnostic performance in the validation sets. Fortunately, the DMU_RAD_ViT model does not have this issue, achieving an AUC of 0.821 in the external test set. This indirectly demonstrates that the DMU_RAD_ViT model is superior to both the DMU_RAD and DMU_ViT models. Additionally, the DMU_RAD_ViT model exhibited high sensitivity and specificity, indicating its potential to accurately identify patients with CLNM while minimizing misdiagnosis. This capability enhances its role in guiding clinical decisions regarding surgical procedures.

This study has several limitations. Firstly, given the retrospective nature of this analysis, it is imperative to conduct prospective multi-center validation to confirm our findings and enhance the model's generalizability and clinical applicability. Secondly, our results may have been influenced by the limited size of our dataset, necessitating further investigation of the integrated models' value in larger, prospective studies. Finally, despite employing SHAP methods to interpret the fusion model in this study, the exact meanings of each DL feature remain unclear. With future advancements in AI technology, we will further explore the interpretability of the model.

This study developed and validated a fusion model that could be used to predict CLNM. The model integrated information from BMUS and SMI dynamic images and combined the strengths of radiomics and ViT, showing the best performance among all models. Through SHAP values, we could intuitively explain the source and contribution of each feature in the model, facilitating clinical understanding and acceptance. As a non-invasive method, the fusion model showed great potential in aiding clinical surgical decision-making.

ETHICAL APPROVAL

This study was approved by the Ethics Committee of Nantong Cancer Hospital (No. 2024-A 06) and conducted following the ethical standards outlined in the 1964 Declaration of Helsinki.

FUNDING STATEMENT

This study was supported in part by the Jingmen Science and Technology Bureau (2022YFZD012).

CREDIT AUTHORSHIP CONTRIBUTION STATEMENT

Co-first authors: P.Z. and X.Z. All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis, and interpretation, or all these areas; took part in drafting, revising, or critically reviewing the article; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

DATA AVAILABILITY

The datasets generated or analyzed during the study are available from the corresponding author upon reasonable request.

DECLARATION OF COMPETING INTEREST

The authors declare no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. The data are not publicly available due to privacy or ethical restrictions.

ACKNOWLEDGMENTS

None.

Chen K, Wang Z, Sun W, Zhang D, Zhang T, He L, et al. Bibliometric insights in advances of papillary thyroid microcarcinoma: Research situation, hot points, and global trends. Frontiers in Endocrinology. 2022;13:949993.
Kowalska A, Walczyk A, Kowalik A, Pałyga I, Trybek T, Kopczyński J, et al. Increase in Papillary Thyroid Cancer Incidence Is Accompanied by Changes in the Frequency of the BRAF ^V600E Mutation: A Single-Institution Study. Thyroid. 2016;26:543–51.
Chen DW, Lang BHH, McLeod DSA, Newbold K, Haymart MR. Thyroid cancer. The Lancet. 2023;401:1531–44.
Haugen BR, Alexander EK, Bible KC, Doherty GM, Mandel SJ, Nikiforov YE, et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid. 2016;26:1–133.
Zhao H, Li H. Meta-analysis of ultrasound for cervical lymph nodes in papillary thyroid cancer: Diagnosis of central and lateral compartment nodal metastases. European Journal of Radiology. 2019;112:14–21.
Luo H, Yin L. Diagnostic value of superb microvascular imaging and color doppler for thyroid nodules: A meta-analysis. Front Oncol. 2023;13:1029936.
Jiang L, Zhang D, Chen Y-N, Yu X-J, Pan M-F, Lian L. The value of conventional ultrasound combined with superb microvascular imaging and color Doppler flow imaging in the diagnosis of thyroid malignant nodules: a systematic review and meta-analysis. Front Endocrinol. 2023;14:1182259.
Cappelli C, Pirola I, Gandossi E, Marini F, Cristiano A, Casella C, et al. Ultrasound Microvascular Blood Flow Evaluation: A New Tool for the Management of Thyroid Nodule? Int J Endocrinol. 2019;2019:7874890.
Lu R, Meng Y, Zhang Y, Zhao W, Wang X, Jin M, et al. Superb microvascular imaging (SMI) compared with conventional ultrasound for evaluating thyroid nodules. BMC Med Imaging. 2017;17:65.
Chen L, Zhan J, Diao X-H, Liu Y-C, Shi Y-X, Chen Y, et al. Additional Value of Superb Microvascular Imaging for Thyroid Nodule Classification with the Thyroid Imaging Reporting and Data System. Ultrasound in Medicine & Biology. 2019;45:2040–8.
Liu P, Lu L, Zhang J, Huo T, Liu S, Ye Z. Application of Artificial Intelligence in Medicine: An Overview. Curr Med Sci. 2021;41:1105–15.
Akinci D’Antonoli T, Cuocolo R, Baessler B, Pinto Dos Santos D. Towards reproducible radiomics research: introduction of a database for radiomics studies. Eur Radiol [Internet]. 2023 [cited 2024 Jan 14]; Available from: https://link.springer.com/10.1007/s00330-023-10095-3
Satake H, Ishigaki S, Ito R, Naganawa S. Radiomics in breast MRI: current progress toward clinical application in the era of artificial intelligence. Radiol med. 2022;127:39–56.
Gao X, Ran X, Ding W. The progress of radiomics in thyroid nodules. Front Oncol. 2023;13:1109319.
Zhou S-C, Liu T-T, Zhou J, Huang Y-X, Guo Y, Yu J-H, et al. An Ultrasound Radiomics Nomogram for Preoperative Prediction of Central Neck Lymph Node Metastasis in Papillary Thyroid Carcinoma. Front Oncol. 2020;10:1591.
Zhou L-Q, Zeng S-E, Xu J-W, Lv W-Z, Mei D, Tu J-J, et al. Deep learning predicts cervical lymph node metastasis in clinically node-negative papillary thyroid carcinoma. Insights Imaging. 2023;14:222.
Castiglioni I, Rundo L, Codari M, Di Leo G, Salvatore C, Interlenghi M, et al. AI applications to medical images: From machine learning to deep learning. Physica Medica. 2021;83:9–24.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–44.
Wang W, Liang H, Zhang Z, Xu C, Wei D, Li W, et al. Comparing three-dimensional and two-dimensional deep-learning, radiomics, and fusion models for predicting occult lymph node metastasis in laryngeal squamous cell carcinoma based on CT imaging: a multicentre, retrospective, diagnostic study. eClinicalMedicine. 2024;67:102385.
Lao J, Chen Y, Li Z-C, Li Q, Zhang J, Liu J, et al. A Deep Learning-Based Radiomics Model for Prediction of Survival in Glioblastoma Multiforme. Sci Rep. 2017;7:10353.
He X, Tan E-L, Bi H, Zhang X, Zhao S, Lei B. Fully transformer network for skin lesion analysis. Medical Image Analysis. 2022;77:102357.
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale [Internet]. arXiv; 2021 [cited 2024 May 23]. Available from: http://arxiv.org/abs/2010.11929
Vafaeezadeh M, Behnam H, Gifani P. Ultrasound Image Analysis with Vision Transformers—Review. Diagnostics. 2024;14:542.
Azad R, Kazerouni A, Heidari M, Aghdam EK, Molaei A, Jia Y, et al. Advances in medical image analysis with vision Transformers: A comprehensive review. Medical Image Analysis. 2024;91:103000.
Zhou G, Mosadegh B. Distilling Knowledge From an Ensemble of Vision Transformers for Improved Classification of Breast Ultrasound. Academic Radiology. 2024;31:104–20.
Sun J, Wu B, Zhao T, Gao L, Xie K, Lin T, et al. Classification for thyroid nodule using ViT with contrastive learning in ultrasound images. Computers in Biology and Medicine. 2023;152:106444.
Singh SP, Wang L, Gupta S, Goli H, Padmanabhan P, Gulyás B. 3D Deep Learning on Medical Images: A Review. Sensors. 2020;20:5097.
Kim H, Lee D, Cho WS, Lee JC, Goo JM, Kim HC, et al. CT-based deep learning model to differentiate invasive pulmonary adenocarcinomas appearing as subsolid nodules among surgical candidates: comparison of the diagnostic performance with a size-based logistic model and radiologists. Eur Radiol. 2020;30:3295–305.
Forghani R, Savadjiev P, Chatterjee A, Muthukrishnan N, Reinhold C, Forghani B. Radiomics and Artificial Intelligence for Biomarker and Prediction Model Development in Oncology. Computational and Structural Biotechnology Journal. 2019;17:995–1008.
Sauerbrei W, Perperoglou A, Schmid M, Abrahamowicz M, Becher H, Binder H, et al. State of the art in selection of variables and functional forms in multivariable analysis—outstanding issues. Diagn Progn Res. 2020;4:3, s41512-020-00074–3.
Stahlschmidt SR, Ulfenborg B, Synnergren J. Multimodal deep learning for biomedical data fusion: a review. Briefings in Bioinformatics. 2022;23:bbab569.
Lundberg S, Lee S-I. A Unified Approach to Interpreting Model Predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems [Internet]. 2017. Available from: https://www.researchgate.net/publication/317062430_A_Unified_Approach_to_Interpreting_Model_Predictions
Kouvaraki MA, Shapiro SE, Fornage BD, Edeiken-Monro BS, Sherman SI, Vassilopoulou-Sellin R, et al. Role of preoperative ultrasonography in the surgical management of patients with thyroid cancer. Surgery. 2003;134:946–54.
Sollini M, Antunovic L, Chiti A, Kirienko M. Towards clinical application of image mining: a systematic review on artificial intelligence and radiomics. Eur J Nucl Med Mol Imaging. 2019;46:2656–72.
Liu T, Zhou S, Yu J, Guo Y, Wang Y, Zhou J, et al. Prediction of Lymph Node Metastasis in Patients With Papillary Thyroid Carcinoma: A Radiomics Method Based on Preoperative Ultrasound Images. Technol Cancer Res Treat. 2019;18:153303381983171.
Huang Y, Zhu T, Zhang X, Li W, Zheng X, Cheng M, et al. Longitudinal MRI-based fusion novel model predicts pathological complete response in breast cancer treated with neoadjuvant chemotherapy: a multicenter, retrospective study. eClinicalMedicine. 2023;58:101899.
Mohsen F, Ali H, El Hajj N, Shah Z. Artificial intelligence-based methods for fusion of electronic health records and imaging data. Sci Rep. 2022;12:17981.
Qin P, Wu K, Hu Y, Zeng J, Chai X. Diagnosis of Benign and Malignant Thyroid Nodules Using Combined Conventional Ultrasound and Ultrasound Elasticity Imaging. IEEE J Biomed Health Inform. 2020;24:1028–36.
Kwon M, Shin JH, Park H, Cho H, Kim E, Hahn SY. Radiomics Based on Thyroid Ultrasound Can Predict Distant Metastasis of Follicular Thyroid Carcinoma. JCM. 2020;9:2156.
Zhu Y, Meng Z, Fan X, Duan Y, Jia Y, Dong T, et al. Deep learning radiomics of dual-modality ultrasound images for hierarchical diagnosis of unexplained cervical lymphadenopathy. BMC Med. 2022;20:269.
Aziz MU, Eisenbrey JR, Deganello A, Zahid M, Sharbidre K, Sidhu P, et al. Microvascular Flow Imaging: A State-of-the-Art Review of Clinical Use and Promise. Radiology. 2022;305:250–64.
Li W, Gao L, Du Y, Wang Y, Yang X, Wang H, et al. Ultrasound microflow patterns help in distinguishing malignant from benign thyroid nodules. Cancer Imaging. 2024;24:18.

No competing interests reported.

Download PDF

Editorial decision: Revision requested
23 Aug, 2024
Editor assigned by journal
05 Aug, 2024
Submission checks completed at journal
05 Aug, 2024
First submitted to journal
04 Aug, 2024

You are reading this latest preprint version

Predicting Central Lymph Node Metastasis in Papillary Thyroid Carcinoma Using a Fusion Model of Vision Transformer and Traditional Radiomics Based on Dynamic Dual-Modality Ultrasound

Status:

Version 1

Abstract

Objective

Method

Results

Conclusions

Figures

INTRODUCTION

MATERIALS AND METHODS

Statistical analysis

RESULTS

DISCUSSION

CONCLUSION

Declarations

References

Additional Declarations

Status:

Version 1