Development of a Multi-view Multi-level Artificial Intelligence System to Stratify Risk Assessment of Mammography

doi:10.21203/rs.3.rs-2489648/v1

Download PDF

Research Article

Development of a Multi-view Multi-level Artificial Intelligence System to Stratify Risk Assessment of Mammography

https://doi.org/10.21203/rs.3.rs-2489648/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background:

Recent artificial intelligence has exhibited great potential in breast imaging, but its value in precise risk stratification of mammography still needs further investigation. This study is to develop an artificial intelligence system (AIS) for accurate malignancy diagnosis and supportive decision-making on mammographic risk stratification.

Methods:

In this retrospective study, 49732 mammograms of 24866 breasts from 12815 women from two Asian clinics between August 2012 and December 2018 were included. We developed an AIS using multi-view mammograms and multi-level convolutional neural network features to diagnosis malignancy and further assess the relative strengths of AIS versus current BI-RADS categorization. We further evaluate AIS by conducting a counterbalance-designed AI-assisted study, where ten radiologists read 1302 cases with/without AIS assistance. The area under the receiver operating characteristic curve (AUC), sensitivity, specificity, accuracy, F1 score were measured.

Results:

The AIS yielded AUC of 0.910 to 0.995 for malignancy diagnosis in the validation and testing sets. Within BI-RADS 3–4 subgroups with pathological results, AIS can downgrade 83.1% of false-positives into benign groups, and upgrade 54.1% of false-negatives into malignant groups. Compared with BI-RADS, AIS performed better sensitivity and specificity in dense and no-calcification subgroups. AIS also can successfully assist radiologists identify 7 out of 43 malignancies initially diagnosed with BI-RADS 0 with specificity of 96.7%. In the counterbalance-designed AI-assisted study, the average AUC across 10 readers was significantly improved with AIS assistance (P = 0.001).

Conclusion:

AIS can identify malignancy on mammography and further serve as a supportive tool for stratifying BI-RADS categorization.

artificial intelligence

mammography

breast cancer

diagnosis

deep learning

BI-RADS

Breast cancer is the leading cause of tumor-related deaths among women and its incidence rate is increasing[1]. Mammography is one of the most common imaging modalities for breast cancer screening, and early diagnosis plays an important role in reducing mortality rates[2–4]. In routine clinical workflows, the Breast Imaging Reporting and Data System (BI-RADS) published by the American College of Radiology provides standardized guidelines for mammographic reports and facilitates breast cancer risk assessment and management[5, 6]. However, the false positive and false negative rates of human manual mammographic interpretation remain high owing to individual differences (particularly breast density) and the lack of experienced radiologists[7–10]. Typically, incorrect readings classified as BI-RADS 0 or 4 categories often lead to additional imaging or unnecessary biopsies[11, 12]. For example, false-positive results (based on pathological reports) for BI-RADS 4 category could be due to the wide range of malignant prediction (2%~95%), limiting its clinical application for individual diagnosis. Therefore, there is a need to establish a clinically applicable breast cancer diagnosis method for assisting decision-making on BI-RADS categorization.

Recent advances in deep learning-based artificial intelligence (AI) methods[13–17] have exhibited great potential in breast cancer diagnosis. Some works[18–21] demonstrated that AI could match or surpass human interpretation in mammography. However, some of the potential roles of AI have yet to be fully addressed. Many studies assessed mammography using forced BI-RADS categorization (only 1–5)[18, 19, 22] and focused on comparing the performance of current clinical criteria and BI-RADS lexicon with those of AI. However, further research is need on whether AI can serve as a supportive tool to improve clinical decision-making on routine BI-RADS categorization, e.g., easier interpretation of BI-RADS 0 and better classification of BI-RADS 3 and 4 groups. Beyond that, previous large-scale AI-based mammographic studies have paid little attention to Asian women due to their high proportion of dense parenchyma and blurred lesions on mammograms. Therefore, the potential application of AI in dense parenchyma obscuring lesions also needs further investigation.

To move beyond the limitations of previous AI approaches and accelerate broader adoption of deep-learning techniques by clinicians, we developed a two-step AI system (AIS) that utilizes multi-view mammograms and multi-level mammographic features to automatically diagnose breast cancer in Asian population. We then explored the benefits of AIS on assisted decision-making for BI-RADS 0, 3, and 4 categories by conducting stratified analysis and counterbalance-designed reader study.

Study sample and outcomes

This retrospective multi-cohort study was approved by the Institutional Review Board of Clinic 1 and Clinic 2. De-identified data were collected with a waiver of written informed consent.

This study included both screening and diagnostic mammograms. The malignant and benign groups were confirmed by biopsy or surgical excision, and the no-finding group defined as those without clinical findings for more than 2 years of follow-up imaging. A total of 14457 women (26447 breasts) with age more than 18 years old and out of the period of pregnancy or lactation, and no biopsy, surgery, radiotherapy, or chemoradiotherapy were performed before mammography examination were enrolled. We excluded breasts lack of craniocaudal (CC) or mediolateral oblique (MLO) mammograms or lack of definite clinical follow-up or pathological results (Fig. 1).

Altogether, we collected 28246 consecutive digital mammograms (14123 breasts) from Clinic 1 between January 2015 and December 2018, and randomly split by a ratio of 8:2 into training and validation sets to develop and validate AIS in breast-level. We also collected 7034 mammograms (3517 breasts) from Clinic 2 between August 2012 and January 2016 as testing set 1 and 12452 consecutive mammograms (6226 breasts) from Clinic 2 between February 2016 and December 2018 as testing set 2 to test AIS in breast-level. To facilitate the training of AIS, another 2000 mammograms with annotation from a subsidiary center between January 2018 and December 2020 were collected as auxiliary set, which were only used in patch-level training and validation. Each breast had two paired views, i.e., CC and MLO images, and all mammograms were acquired on the Hologic system.

Development of AIS

Detailed information about AIS is in Additional file 1: SI. In brief, we proposed a multi-view multi-level AIS for breast cancer diagnosis. It contains two steps: 1) patch-level malignant lesion localization (Fig. 2A); 2) breast-level malignancy diagnosis (Fig. 2B). In step one, we proposed a multi-task patch-level network to find suspicious regions, and merged patch-level outputs into two image-level maps: Malignant Map 1 showed a coarse heatmap of malignant lesion characteristics, and Malignant Map 2 localized the suspicious malignant region. In step two, we used the original CC and MLO images together with Malignant Map 1 and 2 to train a breast-level malignancy prediction model. We referred the breast-level predicted probability as the AIS score, and higher score indicated a higher probability of malignancy.

Assisted decision-making for BI-RADS using AIS

In the subsequent applications of AIS on stratifying BI-RADS categorization, two senior radiologists with more than 10 years’ experience provided a BI-RADS category (0–5), breast density, and lesion type (Fig. 2C) among cases had definitive pathological results. We performed following three experiments:

1) Stratified analysis of BI-RADS 3–4 subgroups.

Due to the high uncertainty for BI-RADS 3–4 categories, we conducted stratified analysis to test the ability of AIS to up-/down-grade BI-RADS 3–4 findings. Specifically, AIS predicted the malignancy with a cutoff AIS score of 0.5. For comparison with BI-RADS reports, a breast was assigned positive if a BI-RADS score ≥ 4A.

2) Analysis study for different subgroups

We analyzed the prediction for different subgroups of gland, lesion, and cancer types using those cases with definitive pathological results in the testing sets, despite BI-RADS 0, because it does not have definite image interpretation. Similarly, 4A was used as the cutoff of BI-RADS. For a fair comparison with BI-RADS prediction, the cutoff for AIS was chosen to match the specificity of BI-RADS when comparing sensitivities, and vice versa, we matched the sensitivity level for comparing specificities.

3) Re-assessment of BI-RADS 0 with AIS assistance

In clinical practice, BI-RADS 0 patients need further evaluation, so that we conducted a separate AI-aided study for BI-RADS 0 subgroup. After more than four-week washout, we recalled all radiologists to re-assess the BI-RADS 0 patients with AIS assistance (Additional file 1: SII).

AIS assistance study

We further conducted a reader study to assess the performance of 10 radiologists with three degrees of expertise (senior, mid-level, and junior) responsible for mammography interpretation by asking each reader to provide a forced BI-RADS score (1–5) and a probability of malignancy score (0-100) with/without AIS assistance (Fig. 2D).

The study was conducted among 1302 cases randomly selected from the testing sets. Readers read 651 cases without AIS and the other 651 cases with AIS during the first session. and crossover during the second session, where a four-week washout period separated. All readers were blinded to clinicopathologic information.

Statistical analysis

Statistical analysis was performed with R, version 4.0.1 software (R Project for Statistical Computing). To evaluate the diagnostic ability of the AIS, we plotted the receiver operating characteristic curves (ROC) and assessed AUC, accuracy, sensitivity, specificity, and F1 score at breast-level. Intraclass correlation coefficient (ICC) was used to evaluate the agreement between radiologists with/without AIS. P < .05 indicated a statistically significant difference.

Patient characteristics

A total of 24866 breasts (mean age 49.8 years ± 10.3) were included, 10745 (43.2%) were confirmed by biopsy or surgical pathology (7266 were malignancy and 3479 were benign), and 14119 (56.8%) were no-findings with more than 2 year of follow-up imaging (Table 1).

Table 1

Dataset.
Dataset	Total	Malignant	Benign	No-findings	Age (mean (SD))	Patch-level	Breast-level	ROI
Training	11299	3515	1091	6693	51.3 (10.5)		Train	×
Validation	2824	844	277	1703	51.2 (10.3)		Validation	×
Testing 1	3517	529	992	1996	46.4 (9.2)		Test	×
Testing 2	6226	1378	1119	3729	48.1 (10.0)		Test	×
Auxiliary	1000	1000				Train/Validation		√
Note.— Data are number of breasts. ROI = region of interesting. BI-RADS = Breast Imaging Reporting and Data System. SD = standard deviation.

Performance of AIS

The performance of AIS is shown in Figure S1 and Table 2. When discriminating breasts with malignant lesions from non-malignant lesions (benign group + no-finding group), AIS achieved an AUC of 0.995 (95% CI, 0.992–0.997), 0.933 (95% CI, 0.919–0.947), and 0.947 (95% CI, 0.939–0.955) in the validation set, testing set 1 and testing set 2, respectively. When discriminating breasts with malignant lesions from benign lesions, the AIS achieved an AUC of 0.988 (95%CI, 0.980–0.995), 0.910 (95%CI, 0.893–0.927), and 0.936 (95%CI, 0.926–0.946) in the validation set, testing set 1 and testing set 2, respectively.

Table 2

The performance of the AIS.
Type	Dataset	AUC	Sensitivity (%)	Specificity (%)	Accuracy (%)	F1
Malignant v.s. Non-Malignant	Validation	0.995 (0.992, 0.997)	96.5 [814/844] (95.0,97.6)	98.9 [1959/1980] (98.4,99.3)	98.2 [2773/2824] (97.6,98.7)	0.970
	Testing 1	0.933 (0.919, 0.947)	81.1 [429/529] (77.5,84.3)	95.0 [2838/2988] (94.1,95.7)	92.9 [3267/3517] (92.0,93.7)	0.774
	Testing 2	0.947 (0.939, 0.955)	84.0 [1158/1378] (82.0,85.9)	96.0 [4654/4848] (95.4,96.5)	93.3 [5812/6226] (92.7,94.0)	0.848
Malignant v.s. Benign	Validation	0.988 (0.980, 0.995)	96.1 [811/844] (94.5,97.3)	97.1 [269/277] (94.4,98.8)	96.3 [1080/1121] (95.1,97.4)	0.975
	Testing 1	0.910 (0.893, 0.927)	80.3 [425/529] (76.7,83.6)	91.1 [904/992] (89.2,92.8)	87.4 [1329/1521] (85.6,89.0)	0.816
	Testing 2	0.936 (0.926, 0.946)	84.4 [1163/1378] (82.4,86.3)	92.9 [1040/1119] (91.3,94.4)	88.2 [2203/2497] (86.9,89.5)	0.888
Note.— Numbers in brackets are numerator/denominator, and numbers in parentheses are 95%CI. AUC = area under the receiver operating characteristic curve. CI = confidence interval. AIS = artificial intelligence system.

The results of AIS vs BI-RADS

Clinical characteristics and mammographic assessments are listed in Table S1.

The benefit of AIS for stratifying BI-RADS 3–4 subgroups. — Based on the pathological results, BI-RADS categorization itself achieved accuracy of 73.15 ~ 84.09%, sensitivity of 85.51 ~ 92.32%, specificity of 58.80 ~ 74.28%, and F1 of 0.716 ~ 0.898, while AIS yielded accuracy of 85.06 ~ 96.82%, sensitivity of 76.64%~96.99%, specificity of 90.55 ~ 96.30%, and F1 of 0.802 ~ 0.979 in the validation and testing sets, respectively (Fig. 3A and Table 3). The AUC values for AIS were significantly higher in validation set (0.828 vs. 0.988, P < 0.001), testing set 1 (0.843 vs. 0.892, P < 0.001), and testing set 2 (0.873 vs. 0.934, P < 0.001).

Table 3

Stratified analysis of BI-RADS 3 and 4.
Type	Dataset	AUC	Sensitivity (%)	Specificity (%)	Accuracy (%)	F1
BI-RADS	Validation	0.828 (0.797,0.859)	92.3 [613/664] (90.0,94.2)	58.8 [127/216] (51.9,65.4)	84.1 [740/880] (81.5,86.5)	0.898
	Testing 1	0.843 (0.821,0.866)	85.5 [366/428] (81.8,88.7)	65.1 [427/656] (61.3,68.7)	73.2 [793/1084] (70.4,75.8)	0.716
	Testing 2	0.873 (0.858,0.888)	89.7 [1041/1161] (87.8,91.4)	74.3 [566/762] (71.0,77.3)	83.6 [1607/1923] (81.8,85.2)	0.868
AIS	Validation	0.988 (0.978,0.997)	97.0 [644/664] (95.4,98.2)	96.3 [208/216] (92.8,98.4)	96.8 [852/880] (95.4,97.9)	0.979
	Testing 1	0.892 (0.871,0.912)	76.6 [328/428] (72.3,80.6)	90.5 [594/656] (88.0,92.7)	85.1 [922/1084] (82.8,87.1)	0.802
	Testing 2	0.934 (0.923,0.945)	84.8 [984/1161] (82.6,86.8)	92.4 [704/762] (90.3,94.2)	87.8 [1688/1923] (86.2,89.2)	0.893
Note.— Numbers in brackets are numerator/denominator, and numbers in parentheses are 95%CI. BI-RADS = Breast Imaging Reporting and Data System. AUC = area under the receiver operating characteristic curve. CI = confidence interval. AIS = artificial intelligence system.

In Fig. 3B, we draw a sunburst plot to visualize the benefits of AIS on BI-RADS categorization. Among 3887 breasts in the validation and testing sets, BI-RADS categorization resulted in 6.0% [233/3887] (2.1% [83/3887] BI-RADS 3 and 3.9% [150/3887] BI-RADS 4A) of false-negatives, and 13.2% [514/3887] (11.6% [449/3887] BI-RADS 4B and 1.7% [65/3887] BI-RADS 4C) of false-positives. In contrast, when we analyzed these patients with AIS, it correctly classified 11.0% [427/3887] of BI-RADS 4B and 4C exams into benign groups, thus avoiding excessive surgical treatment. The AIS also correctly classified 3.2% [126/3887] of BI-RADS 3 and 4A exams into malignant groups, thus avoiding missing the best intervention time. Furthermore, 90.6% [1830/2020] of malignant BI-RADS 4B and 4C cases were classified as malignant to ensure necessary surgery, and 96.3% [1079/1120] of benign BI-RADS 3 and 4A cases were classified as benign to avoid unnecessary surgery. Eventually, only 5.9% [231/3887] of all BI-RADS 3–4 cases were mistakenly stratified compared with BI-RADS categorization.

In general, AIS appropriately assigned 83.1% [427/514] of false-positives (BI-RADS 4B and C) into the benign group, meanwhile, it assigned 54.1% [126/233] of false-negatives (BI-RADS 3 and 4A) into the malignant group. The benefits with AIS are to better prevent over- and delayed-treatments.

The sensitivity and specificity analysis under different subgroups. — In Fig. 4 and Table 4, we compare the sensitivity and specificity of BI-RADS categorization with those of AIS within different subgroups. For gland type subgroup, AIS yielded a 3.7% [39/1047] increase in sensitivity and a 9.4% [141/1500] increase in specificity within dense subgroup, while yielded a slight decrease (0.2% [2/805] of sensitivity and 1.9% [4/210] of specificity) within no-dense subgroup. For lesion type subgroup, AIS increased sensitivity and specificity by 4.2% [44/1052] and 20.8% [153/735] within the soft-tissue subgroup, respectively. As for the calcification subgroup, AIS decreased sensitivity and specificity by 1.2% [9/769] and 7.5% [28/375], respectively. For the no-sign subgroup, BI-RADS classified all breasts as benign group, and resulting in 0.0% [0/31] of sensitivity and 99.7% [598/600] of specificity. However, 4.9% [31/631] of breasts were proven to be malignant. Conversely, AIS appeared to be able to identify 19.4% [6/31] malignant breasts with a specificity of 99.7% [598/600]. For the cancer type subgroup analysis, AIS exhibited higher sensitivity and specificity within the IDC/ILC and DCIS subgroups, and lower sensitivity and specificity within the rare subgroup.

Table 4

Stratified analysis of gland type, lesion type and cancer type.
Type	Sensitivity			Specificity			AIS AUC (95%CI)
Type	AIS	BI-RADS	∆	AIS	BI-RADS	∆	AIS AUC (95%CI)
Gland Type
No-Dense	93.7 [754/805]	93.9 [756/805]	-0.2 [2/805]	67.1 [141/210]	69.0 [145/210]	-1.9 [4/210]	0.937 (0.922,0.952)
Dense	89.8 [940/1047]	86.0 [901/1047]	+ 3.7 [39/1047]	85.1 [1277/1500]	75.7 [1136/1500]	+ 9.4 [141/1500]	0.919 (0.907,0.931)
Lesion Type
Calcifications	92.6 [712/769]	93.8 [721/769]	-1.2 [9/769]	69.1 [259/375]	76.5 [287/375]	-7.5 [28/375]	0.938 (0.925,0.952)
Soft tissue	93.2 [980/1052]	89.0 [936/1052]	+ 4.2 [44/1052]	74.7 [549/735]	53.9 [396/735]	+ 20.8 [153/735]	0.909 (0.895,0.922)
No sign	19.4 [6/31]	0.0 [0/31]	+ 19.4 [6/31]	100.0 [600/600]	99.7 [598/600]	+ 0.3 [2/600]	0.799 (0.690,0.907)
Cancer Type
IDC or ILC	92.8 [1482/1597]	90.3 [1442/1597]	+ 2.5 [40/1597]	85.4 [1456/1704]	74.8 [1275/1704]	+ 10.6 [181/1704]	0.941 (0.932,0.949)
DCIS	83.0 [137/165]	80.6 [133/165]	+ 2.4 [4/165]	77.5 [1321/1704]	74.8 [1275/1704]	+ 2.7 [46/1704]	0.870 (0.834,0.905)
Rare	84.4 [76/90]	91.1 [82/90]	-6.7 [6/90]	54.3 [926/1704]	74.8 [1275/1704]	-20.5 [349/1704]	0.877 (0.831,0.922)
Note.— For sensitivity calculations, the threshold for AIS is chosen to match the specificity of BI-RADS classification. For specificity calculations, the threshold for AIS is chosen to match the sensitivity of BI-RADS classification. Numbers in brackets are numerator/denominator. AIS = artificial intelligence system. BI-RADS = Breast Imaging Reporting and Data System. AUC = area under the receiver operating characteristic curve. CI = confidence interval. IDC/ILC = invasive ductal carcinoma/invasive lobular carcinoma. DCIS = ductal carcinoma in situ

Re-assessment of the BI-RADS 0 with AIS assistance. — After applying AIS on all mammograms classified as BI-RADS 0 in the testing sets, radiologists were able to identify malignant lesions in 7 out of 43 breasts with a specificity of 96.7% [352/364]. (Two representative breasts are shown in Figure S2)

Counterbalance-designed AIS-assisted study

Among 1302 breasts assessed in the AIS-assisted study, AIS standalone achieved AUC of 0.926 (95%CI, 0.911–0.942), sensitivity of 83.4% [529/634], specificity of 93.7% [626/668], and accuracy of 88.7% [1155/1302]. The radiologists without assistance achieved AUC = 0.787 ~ 0.919, sensitivity = 51.0% [323/634] ~ 94.5% [599/634], and specificity = 45.7% [305/668] ~ 96.4% [644/668], accuracy = 69.4% [904/1302] ~ 87.5% [1139/1302], while radiologists with assistance achieved AUC = 0.808 ~ 0.925, sensitivity = 54.1% [343/634] ~ 93.8% [595/634], and specificity = 53.3% [356/668] ~ 96.0% [641/668], accuracy = 73.0% [951/1302] ~ 88.4% [1151/1302]. All 10 readers exhibited an increased AUC ranging from 0.003 to 0.047, where 5 readers exhibited significant improvements (P < 0.05, Fig. 5, Table S2). The average AUC across 10 readers was significantly improved with AIS assistance (0.870 vs. 0.888, P = 0.001). The use of AIS resulted in a trend toward increasing sensitivity in 8 out of 10 radiologists with improvements ranging from 2.2% [14/634] to 10.3% [65/634], while the average specificity slightly increased across mid-level and senior groups. Without AIS, ICC for 10 readers was 0.629 (95%CI, 0.609–0.649), while with the support of AIS, ICC increased to 0.672 (95%CI, 0.653–0.690) (Table S3). We plotted two representative breasts with dense parenchyma (Fig. 6) or mass lesions (Figure S3) that were difficult to identify for radiologists, but with the help of AIS, most readers downgrade BI-RADS in benign cases and upgrade BI-RADS in malignant cases.

We developed a two-step AIS to automatically predict breast cancer risk using multi-view mammograms and multi-level convolutional neural network features, and further evaluated its performance in assisting decision-making on BI-RADS categorization. We collected 49732 digital mammograms from Asian clinics that mainly treat symptomatic breast patients. Results showed satisfied diagnostic performance of AIS with AUC of 0.933 ~ 0.995 in differentiating malignant vs. non-malignant breasts and 0.910 ~ 0.988 in differentiating malignant vs. benign breasts in the validation and testing sets.

Unlike previous large-scale AI-based mammographic studies[20, 23], that mainly established on Caucasian population, our AIS were established among Asians, which typically have a high proportion of dense glands and blurred lesions on mammograms. The sensitivity and specificity analysis showed that AIS performed better in dense and no-calcification subgroups compared to BI-RADS categorization. In addition, mammography screening rate and the use of computer-aided diagnosis (CAD) are relatively low in Asia, thus, our study included both screening and diagnostic mammograms with a high proportion of accurate ground-truth label with biopsy/surgery results.

According to the fifth edition of BI-RADS lexicon, incorrect classification between categories 3 and 4 is the main reason for high false-positive and false-negative rates in clinical practice[7, 8, 24–26]. Current criticisms of mammography focused on false positives, i.e., over-diagnosis and associated biopsies, primarily due to the wide range of predicted malignancies in BI-RADS 4 category[25, 27]. For example, low-risk BI-RADS 4A lesions may be evaluated separately and managed with follow-ups rather than immediate biopsy as is done for 4B and 4C lesions. Our AIS showed more accurate risk stratification than BI-RADS categorization in category 3–4 with significant improvements in AUC in the validation and testing sets (0.828 ~ 0.873 vs. 0.892 ~ 0.988, P < 0.001). In addition, by correctly classifying 83.1% [427/514] of false-positive cases correctly into the benign group, and 54.1% [126/233] of false-negative cases into the malignant group, our AIS can benefit BI-RADS 3 ~ 4 categorization, perhaps avoiding unnecessary biopsies or delayed diagnosis.

We also compared the effects of gland density, lesion types, and cancer types on the performance of AIS and BI-RADS categorization. Surprisingly, the sensitivity and specificity of AIS increased in dense and no-calcification subgroups, rather than in no-dense and calcification subgroups, and the former were often more difficult to be identified by radiologists and are more prevalent in Asian populations.

To illustrate the results, we plotted four representative cases with dense breast or mass lesions, which were difficult to read, but most readers can successfully classify them with the assistance of AIS (Fig. 6 and Figure S3). We also observed an increase in sensitivity and specificity of AIS in IDC/ILC and DCIS subgroups, as well as a decrease in the rare cancer subgroup. The main reason could be insufficient training images in the rare cancer subgroup. Another interesting finding was that AIS identified 6 out of 31 malignant breasts with a high specificity of 99.7% [598/600] in the no-sign subgroup, which was classified primarily as benign based on BI-RADS lexicon. In addition, AIS can successfully assist radiologists in identifying 7 out of 43 malignant lesions diagnosed with BI-RADS 0 with specificity of 96.7% [352/364] to reduce unnecessary BI-RADS 0 mammogram recall. The above results strongly suggested that AIS could be an effective complement to current mammographic interpretation, especially in dense breasts, soft-tissue lesions, and false-negative lesions on mammograms, as well as BI-RADS 0 category[28–30].

To further investigate the potential of human-AI partnerships, we conducted a counterbalance-designed reader study to assess the contribution of AIS to breast imaging interpretation among radiologists across different levels. The average AUC across the 10 readers were significantly improved with AIS assistance (0.870 vs. 0.888, P = 0.001). In addition, the inter-reader agreement increased with AIS assistance. Interestingly, the results showed that, readers can find some atypical DCIS breasts with the help of AIS, and two examples are provided (Figure S4).

Our study had several limitations. First, the patient data were collected from institutions that mainly treated symptomatic breast patients rather than screening population, and mammograms were captured by using a single vendor (Hologic), thus, more studies are needed to determine the performance of AIS in breast cancer screening where most subjects are normal, and mammography devices from diverse vendors should also be examined. Second, the effect of AIS-assisted radiological diagnosis was established using only mammograms without any clinical information, despite pathological results and follow-ups as ground truth. Further prospective AIS assistance study with and without clinical variables will be performed in our next research. Third, our AIS were developed among Asian population, whether it can be generalized among other population still need further studies.

Artificial intelligence system can serve as a clinically applicable tool to identify malignancy on mammography and further assist decision-making on the current BI-RADS lexicon, especially for stratified assessment within BI-RADS 3–4 categories to prevent over and delayed diagnosis.

BI-RADS = Breast Imaging Reporting and Data System

AIS = artificial intelligence system

ROC = receiver operating characteristic

AUC = area under the receiver operating characteristic curve

CI = confidence interval

ICC = intraclass correlation coefficient

IDC/ILC=invasive ductal carcinoma/invasive lobular carcinoma

DCIS = ductal carcinoma in situ

Ethics approval and consent to participate

This retrospective multi-cohort study was approved by the Institutional Review Board of Henan Cancer Hospital (Clinic 1) and Henan Provincial People’s Hospital (Clinic 2). De-identified data were collected with a waiver of written informed consent.

Consent for publication

De-identified data were collected with a waiver of written informed consent.

Availability of data and materials

The datasets were used in this study would be available on reasonable request through contacting the corresponding author. The related code of deep learning models and sample data will be available on GitHub (https://github.com/QingxiaWu21/BreastMG) upon acceptance. Any additional information required to reanalyze the data reported in this work is available from the lead contact upon reasonable request.

Competing interests

Q.W. and P.D. are employees of Beijing United Imaging Research Institute of Intelligent Imaging and United Imaging Intelligence (Beijing) Co., Ltd. Z.X and D.S are employees of Shanghai United Imaging Intelligence Co., Ltd. The companies have no role in designing and performing the surveillance and analyzing and interpreting the data. All other authors report no conflict of interest relevant to this article.

Funding

This work is supported by National Science Foundation of China (81720108021), Medical Science and Technology Research Project of Henan Province (SBGJ202101002), Natural Science Foundation of Henan Province (202300410081), Zhengzhou Collaborative Innovation Major Project (20XTZX05015), Key Project of Henan Province Medical Science and Technology Project (LHGJ20210005,HGJ20210001,LHGJ20220055); Henan Provincial Science and Technology Research Projects (222102310627,212102310689).

Authors' contributions

Guarantors of integrity of entire study, H.T., Q.W., M.W.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, H.T., Q.W., Y.W.; clinical studies, H.T., Y.W., B.Z., J.Z., B.W., Y.C., L.D., J.Z., F.F., H.G., C.F., L.M.; experimental studies, Q.W., P.D., Z.X., D.S.; statistical analysis, Q.W., P.D., Z.X.; and manuscript editing, all authors.

Acknowledgements

The authors are grateful to Yingjie Zuo, BM. (Department of Radiology, Wuzhi County People's Hospital), Yunxia Wang, BM. (Department of Radiology, Henan Provincial People’s Hospital) and Menglu Hai, BM. (Department of Radiology, Henan Cancer Hospital) for their support of this project among the reader study of human AI partnership.

Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. Ca Cancer J Clin. 2021;
Myers ER, Moorman P, Gierisch JM, Havrilesky LJ, Grimm LJ, Ghate S, et al. Benefits and Harms of Breast Cancer Screening: A Systematic Review. Jama. 2015;314:1615–34.
Løberg M, Lousdal ML, Bretthauer M, Kalager M. Benefits and harms of mammography screening. Breast Cancer Res Bcr. 2015;17:63.
Tabár L, Vitak B, Chen HT, Yen M, Duffy SW, Smith RA. Beyond randomized controlled trials: organized mammographic screening substantially reduces breast carcinoma mortality. Cancer. 2001;91:1724–31.
Rao AA, Feneis J, Lalonde C, Ojeda-Fournier H. A Pictorial Review of Changes in the BI-RADS Fifth Edition. Radiographics. 2016;36:623–39.
Spak DA, Plaxco JS, Santiago L, Dryden MJ, Dogan BE. BI-RADS® fifth edition: A summary of changes. Diagn Interv Imag. 2017;98:179–90.
Nelson HD, O’Meara ES, Kerlikowske K, Balch S, Miglioretti D. Factors Associated With Rates of False-Positive and False-Negative Results From Digital Mammography Screening: An Analysis of Registry Data. Ann Intern Med. 2016;164:226.
Huynh PT, Jarolimek AM, Daye S. The false-negative mammogram. Radiographics. 1998;18:1137–54.
Batchu S, Liu F, Amireh A, Waller J, Umair M. A Review of Applications of Machine Learning in Mammography and Future Challenges. Oncology. 2021;99:483–90.
Ong M-S, Mandl KD. National Expenditure For False-Positive Mammograms And Breast Cancer Overdiagnoses Estimated At $4 Billion A Year. Health Affair. 2017;34:576–83.
He T, Puppala M, Ezeana CF, Huang Y, Chou P, Yu X, et al. A Deep Learning–Based Decision Support Tool for Precision Risk Assessment of Breast Cancer. Jco Clin Cancer Informatics. 2019;1–12.
Majid AS, Paredes ES de, Doherty RD, Sharma NR, Salvador X. Missed Breast Carcinoma: Pitfalls and Pearls. Radiographics. 2003;23:881–95.
Yala A, Schuster T, Miles R, Barzilay R, Lehman C. A Deep Learning Model to Triage Screening Mammograms: A Simulation Study. Radiology. 2019;293:38–46.
Zhu X, Wolfgruber TK, Leong L, Jensen M, Scott C, Winham S, et al. Deep Learning Predicts Interval and Screening-detected Cancer from Screening Mammograms: A Case-Case-Control Study in 6369 Women. Radiology. 2021;301:550–8.
Yala A, Mikhael PG, Strand F, Lin G, Smith K, Wan Y-L, et al. Toward robust mammography-based models for breast cancer risk. Sci Transl Med. 2021;13:eaba4373.
Aboutalib SS, Mohamed AA, Berg WA, Zuley ML, Sumkin JH, Wu S. Deep Learning to Distinguish Recalled but Benign Mammography Images in Breast Cancer Screening. Clin Cancer Res. 2018;24:5902–9.
Yala A, Lehman C, Schuster T, Portnoi T, Barzilay R. A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction. Radiology [Internet]. 2019;292:60–6. Available from: https://sci-hub.tw/10.1148/radiol.2019182716
Pacilè S, Lopez J, Chone P, Bertinotti T, Grouin JM, Fillard P. Improving Breast Cancer Detection Accuracy of Mammography with the Concurrent Use of an Artificial Intelligence Tool. Radiology Artif Intell. 2020;2:e190208.
Kim H-E, Kim HH, Han B-K, Kim KH, Han K, Nam H, et al. Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. Lancet Digital Heal. 2020;2:e138–48.
McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, et al. International evaluation of an AI system for breast cancer screening. Nature. 2020;577:89–94.
Shoshan Y, Bakalo R, Gilboa-Solomon F, Ratner V, Barkan E, Ozery-Flato M, et al. Artificial Intelligence for Reducing Workload in Breast Cancer Screening with Digital Breast Tomosynthesis. Radiology. 2022;211105.
Lotter W, Diab AR, Haslam B, Kim JG, Grisot G, Wu E, et al. Robust breast cancer detection in mammography and digital breast tomosynthesis using an annotation-efficient deep learning approach. Nat Med. 2021;27:244–9.
Schaffter T, Buist DSM, Lee CI, Nikulin Y, Ribli D, Guan Y, et al. Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms. Jama Netw Open. 2020;3:e200265.
Lee KA, Talati N, Oudsema R, Steinberger S, Margolies LR. BI-RADS 3: Current and Future Use of Probably Benign. Curr Radiology Reports. 2018;6:5.
Flowers CI, O’Donoghue C, Moore D, Goss A, Kim D, Kim J-H, et al. Reducing false-positive biopsies: a pilot study to reduce benign biopsy rates for BI-RADS 4A/B assessments through testing risk stratification and new thresholds for intervention. Breast Cancer Res Tr. 2013;139:769–77.
Lehman CD, Arao RF, Sprague BL, Lee JM, Buist DSM, Kerlikowske K, et al. National Performance Benchmarks for Modern Screening Digital Mammography: Update from the Breast Cancer Surveillance Consortium. Radiology. 2017;283:49–58.
Hubbard RA, Kerlikowske K, Flowers CI, Yankaskas BC, Zhu W, Miglioretti DL. Cumulative Probability of False-Positive Recall or Biopsy Recommendation After 10 Years of Screening Mammography: A Cohort Study. Ann Intern Med. 2011;155:481.
Melnikow J, Fenton JJ, Whitlock EP, Miglioretti DL, Weyrich MS, Thompson JH, et al. Supplemental Screening for Breast Cancer in Women With Dense Breasts: A Systematic Review for the U.S. Preventive Services Task Force. Ann Intern Med. 2016;164:268.
Yi C, Tang Y, Ouyang R, Zhang Y, Cao Z, Yang Z, et al. The added value of an artificial intelligence system in assisting radiologists on indeterminate BI-RADS 0 mammograms. Eur Radiol. 2022;32:1528–37.
Basha MAA, Safwat HK, Eldin AMA, Dawoud HA, Hassanin AM. The added value of digital breast tomosynthesis in improving diagnostic performance of BI-RADS categorization of mammographically indeterminate breast lesions. Insights Imaging. 2020;11:26.

No competing interests reported.

SupplementaryInformation.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Development of a Multi-view Multi-level Artificial Intelligence System to Stratify Risk Assessment of Mammography

Status:

Version 1

Abstract

Background:

Methods:

Results:

Conclusion:

Figures

Background

Methods

Study sample and outcomes

Development of AIS

Assisted decision-making for BI-RADS using AIS

AIS assistance study

Statistical analysis

Results

Patient characteristics

Performance of AIS

The results of AIS vs BI-RADS

Counterbalance-designed AIS-assisted study

Discussion

Conclusion

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Additional Declarations

Supplementary Files

Status:

Version 1