Treatment Concordance by Use Case: MTB vs. Individual Clinicians.
To our knowledge, this study is one of the first to summarize the performance of an oncology AI-based CDSS, measured by concordance of its therapeutic suggestions with treatment decisions made by cancer-treating physicians in practice for female breast cancer patients in diverse, international, cancer care settings. We found substantial agreement between system-generated therapeutic options and both treatment decisions of MTBs as well as individual clinicians in a large number of patients with breast cancer. Our targeted review demonstrate that the system’s suggested treatment options agreed with therapies selected by cancer-treating physicians in China, India, and Thailand, countries where we identified breast cancer treatment decision concordance studies.
The CDSS exhibited a higher treatment decision concordance with MTBs, as compared to individual clinicians. MTBs provide multidisciplinary team management that generally results in decreased mortality, improved quality of life, and reduced costs in cancer patient care.33,34 The higher concordance between system-generated therapeutic options and treatments agreed upon by MTB experts is consistent with the quality of therapeutic options suggested by this CDSS.35 Furthermore, the lower rate of concordance with decisions made by individual physicians as compared to MTBs supports a role for an AI-based CDSS in aiding individual oncologists during the complex clinical task of breast cancer treatment decision making.
Concordance in Different Countries and Breast Cancer Subgroups
According to system use case and country, we identified a study conducted at a tertiary cancer center in India with a higher breast cancer treatment decision concordance between the CDSS and MTB,25 as compared to similar use case studies in China.23,24,26,27 Likewise, individual clinicians in Thailand had higher concordance with the CDSS in 3 studies28,30,32 as compared to 2 large individual clinicians studies from China.29,31 Differences in breast cancer treatment decision concordance between the system and individual clinicians or MTBs in different countries are multifactorial and likely explained by differences in oncology practice patterns at the institutional and national levels.
Successful implementation of a CDSS in medical practice can be achieved by identifying and addressing barriers to CDSS clinical adoption. Successful CDSS implementation relies on factors such as quality, complexity, usability, learnability, transparency, workflow integration, and cost-effectiveness of a CDSS. Furthermore, there is need for early involvement of end users in the development and enhancement of these systems. Consideration of regional health regulatory requirements and localization efforts to address regional differences in clinical practice are also important.37
A lower concordance between system-generated therapeutic options and MTBs, as well as individual clinicians in breast cancer patients ≥ 70 years of age ,was reported in 2 large studies.25,29 Age-related differences in patient and cancer care across China, India and the US may account for the lower concordance reported in older patients. Breast cancer stage at presentation, patient functional status, co-morbidity burden, socioeconomic support, cultural values and treatment preferences may play a role in concordance, with a need for well-designed studies to promote evidence-informed management of elderly patients with breast cancer.38 Prospective studies are also needed to evaluate the technical performance and clinical impact of the system in different subgroups of breast cancer patients.
Evaluation of CDSS Clinical Decision Quality and Impact
Concordance between CDSS and clinical decisions made in practice is limited as a measure of decision quality, which should be based on evidence and best practices. We selected concordance as a metric because many early adopters of CDSS have performed concordance studies to demonstrate reasonable agreement and build trust with end users. Evaluations of CDSSs often measure treatment decision adherence to guideline recommendations from the National Comprehensive Cancer Network (NCCN), Chinese Society of Clinical Oncology (CSCO) or other established guidelines. A study measured adherence of treatment decisions to guideline recommendations in 57 patients with advanced breast, colon, endometrial, esophageal, hepatic, gastric, ovarian, and rectal cancer in China which found a high adherence of WfO treatment options to NCCN and CSCO guidelines.39 A study of 69 patients with colon and rectal cancers from South Korea found an adherence of WfO treatment options to NCCN guidelines and treatment decision concordance with a local MTB of 88.4% and 87.0% respectively.17
A blinded panel of cancer experts re-evaluating treatment decisions by both humans and CDSS or measuring treatment decision adherence to guideline recommendations helps reduce bias associated with the source of recommendations. A blinded study comparing WfO therapeutic options and treatment recommendations by individual physicians in breast, colon, lung and rectal cancers at a regional referral hospital in Thailand employed an expert panel of 3 oncologists to evaluate the quality of treatment decisions offered by clinicians and the CDSS.32 The expert panel, which was blinded to the source of treatment decision, compared treatments recommended by the individual clinicians and system-generated therapeutic options, rating 71% of these paired options as either identical or acceptable.32
The concordance studies we identified in this targeted literature review reflect the intended use of the CDSS, which is to support cancer-treating physicians by providing therapeutic options that reflect best evidence and current practice. The system was designed according to a premise that humans are more likely to make optimal treatment decisions when supported by a CDSS. For institutions lacking a MTB, a CDSS may help fill this gap by providing individual clinicians with a choice of evidence-informed therapeutic options. Consistent with this idea, use of the system as part of clinicians overall decision-making process significantly impacted treatment decisions in several studies. A large cross-sectional observational study measured the impact of the CDSS in treatment decision-making by individual clinicians in 1197 patients with breast cancer in China.29 Participating physicians, initially blinded to system-generated therapeutic options, saw impact of the system on treatment recommendations in 5.0% of cases after the system’s therapeutic options were disclosed to them. The adherence of breast cancer treatments to NCCN and CSCO guidelines increased from an 89.0% baseline to 97.0% in the 5.0% of cases where the physicians reevaluated treatment recommendations after viewing system-generated therapeutic options, as a part of their clinical decision-making process. Another study performed at a tertiary cancer center in India examining treatment decisions before and after exposure to the CDSS’s therapeutic options in 1000 cases of breast, colon, lung, and rectal cancer cases showed a 13.6% decision impact for treatment decisions made by MTB, which were reevaluated during clinical decision making that included viewing therapeutic options offered by the CDSS. This demonstrates that even treatment decisions of expert MTBs in tertiary cancer care settings may potentially be improved by use of a CDSS.40
This targeted review has several limitations related to risk of bias. The reported concordance of system-generated therapeutic options with oncologists’ treatment decisions from individual studies were combined, analyzed and reported according to CDSS use case. There is an inherent risk of bias introduced by the methodology utilized in the studies included in our targeted review, which were all retrospective. Moreover, the studies we reviewed had different sample sizes, proportions of patients by cancer stage, and used various versions of the CDSS. Our targeted review identified and included in the final analysis 10 eligible studies performed in 3 countries, all in the Asian continent. Therefore, the results of our review may not reflect the system’s performance in the US, other Western countries or other regions of the world such as Africa or Latin America.
We did not include studies evaluating the usability, end user satisfaction, workflow integration, or the clinical impact of the system in breast cancer treatment decision making. These factors are likely to play a key role in the performance, implementation, acceptability, and clinical adoption of a CDSS. Nevertheless, assessing technical performance by measuring clinical decision concordance is an important first step in fostering end-user trust and adoption of a CDSS. Technical performance studies are necessary to address a CDSS accuracy as a potential confounder in future workflow or clinical decision impact studies. Strengths of our study are the inclusion of a large number of breast cancer patients in diverse clinical settings, inclusion of peer-reviewed publications only, and clinically relevant analysis of treatment decision concordance based on CDSS use case (MTB versus individual clinicians) and breast cancer subgroups.