Histopathology and proteomics are synergistic for High-Grade Serous Ovarian Cancer platinum response prediction

doi:10.21203/rs.3.rs-4490790/v1

Download PDF

Article

Histopathology and proteomics are synergistic for High-Grade Serous Ovarian Cancer platinum response prediction

https://doi.org/10.21203/rs.3.rs-4490790/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Patients with High-Grade Serous Ovarian Cancer (HGSOC) exhibit varied responses to treatment, with 20–30% showing de novo resistance to platinum-based chemotherapy. While hematoxylin-eosin (H&E) pathological slides are used for routine diagnosis of cancer type, they may also contain diagnostically useful information about treatment response. Our study demonstrates that combining H&E-stained Whole Slide Images (WSIs) with proteomic signatures using a multimodal deep learning framework significantly improves the prediction of platinum response in both discovery and validation cohorts. This method outperforms the Homologous Recombination Deficiency (HRD) score in predicting platinum response and overall patient survival. The study sets new performance benchmarks and explores the intersection of histology and proteomics, highlighting phenotypes related to treatment response pathways, including homologous recombination, DNA damage response, nucleotide synthesis, apoptosis, and ER stress. This integrative approach has the potential to improve personalized treatment and provide insights into the therapeutic vulnerabilities of HGSOC.

Health sciences/Medical research/Genetics research

Health sciences/Medical research/Outcomes research

Biological sciences/Cancer/Gynaecological cancer/Ovarian cancer

Biological sciences/Cancer/Tumour biomarkers

Biological sciences/Computational biology and bioinformatics/Machine learning

High-Grade Serous Ovarian Cancer

Multi-modal deep learning

Platinum response prediction

20–30% of HGSOC patients have treatment-refractory disease at diagnosis and have a poor prognosis ¹. Patients with refractory disease experience the toxicity of platinum-based chemotherapy without benefit. Due to their rapid disease progression, such cases are commonly excluded from participating in clinical trials. Lack of mechanistic understanding significantly impedes developing treatment strategies to overcome first line treatment resistance. Conversely, improved prediction models for treatment resistance could significantly enhance clinical decision-making, allowing for more effective treatment approaches that minimize exposure to potentially non-effective therapies and side effects, while ensuring that patients who are likely to respond to a treatment are identified.

Homologous recombination-deficiency (HRD) plays a major role in response to platinum-based treatments and PARP-inhibitors. Cases with HRD can be identified using mutational-signature based signatures utilizing next generation sequencing-based methods including the HRD-score ², the FDA-approved methods (Myriad myChoice CDx³ and FoundationFocus CDxBRCA LOH⁴) and WGS-based HRDetect ⁵ or WES-retrained HRDetect ⁶ and functional assays. (e.g. RAD51 foci). Although recent publications investigated the possibility of predicting HRD-status using whole slide images ^7,8 and predicting response to platinum-based treatments using WSI or proteomics data ^9–11, their predictive power has not reached the level of clinical utility.

Investigating the proteome is informative of how tumors respond to platinum-based treatments. Recent studies in proteogenomics have revealed that certain patterns of protein expression and modifications are linked to patient survival rates and outcomes in cases of High-Grade Serous Ovarian Carcinoma (HGSOC) ^10,12,13. Additionally, Chowdhury et.al⁹ described a signature of 64 proteins that can predict with high specificity which patients might develop resistance to initial platinum therapy. However, this approach captures only part of the complex interaction between cancer cells and treatment. The spatial organization of these cells within the tumor microenvironment, a key aspect that proteomic analyses often overlook, also significantly influences treatment response. Understanding the spatial distribution of proteins and their interaction with the tumor environment adds another layer of insight, potentially identifying mechanisms that drive therapy resistance and efficacy.

Recent advances in computational pathology demonstrate great potential for treatment response prediction using weakly supervised deep learning models. These computer vision algorithms can use H&E-stained Whole Slide Images (WSIs) to predict the outcomes of treatments like bevacizumab in ovarian cancer ¹⁴. Additionally, these models can determine Homologous Recombination Deficiency (HRD) status ^7,8,15. This capacity to derive significant clinical insights directly from WSIs suggests that when combined with proteomic data, which offers a detailed molecular landscape of tumor behavior, we can significantly improve predictive models.

Since WSI, proteomic signatures or DNA repair deficiency status are all predictive of various aspects of ovarian cancer biology, it is possible that their “fusion” may produce even superior predictive performance.

Early and late fusion models are two approaches in multimodal data fusion and analysis. Early fusion is similar to a multidisciplinary consultation where an oncologist, radiologist, and genetic counselor collaborate from the start, pooling their expertise to quickly form a comprehensive view of a patient’s health. This approach is useful when the interplay among various medical data types is essential for accurate diagnosis and treatment planning. In contrast, late fusion is akin to sequential specialist consultations. Each expert examines the patient independently, contributing insights from their respective fields in detail and later integrate these findings. This method allows for in-depth analysis of each data type, ensuring that unique insights are captured before they are synthesized into a diagnosis or treatment plan.

Our study evaluates early and late fusion models by benchmarking recent methodological advances in multi-modal integration of histopathology and omics data ^16–20. We compare these models across metrics of performance and interpretability. Furthermore, multi-modal fusion provides opportunities for automatically identifying biomarkers related to treatment resistance. The way these models combine different modalities is designed to be inherently interpretable. Therefore, the integration of biological knowledge into the models facilitates detailed investigations at the patient level into the significance of specific histopathology morphologies, pathways and proteins.

Our study is the first to integrate proteomics and histopathological images and show that these modalities are synergistic for platinum response treatment. Furthermore, we show that these models can be probed to provide fine-grained interpretability to better understand tumor response at a mechanistic level.

1.1 Combination of WSI with proteomics significantly increases the accuracy of response prediction

Both the Cancer Genome Atlas Program (TCGA) ¹² and Proteogenomic Translational Research Centers High-Grade Serous Ovarian Carcinoma (PTRC-HGSOC) ⁹ datasets included patients with paired H&E WSI and proteomics data, along with documented responses to platinum chemotherapy. This enabled an investigation into the effectiveness of multi-modal histology-proteomics deep learning compared to their unimodal counterparts for this task.

We first trained a clustering-constrained attention multiple instance learning (CLAM) ²¹ model to predict response to platinum therapy in ovarian cancer based on WSI data alone. The models were trained on one of the data sets available (TCGA or PTRC-HGSOC, see methods for description of data sets) and validated on the other data set. As Table. 1 shows, the predictive power of this method using only pathological images was rather limited independent of whether primary or metastatic tumors were considered. Similarly, we trained a classical machine learning ensemble model based on the proteomics “Chowdhury Signature” ⁹ (methods 3.6.1) on one of the two data sets (TCGA-OV or PTRC-HGSOC) and evaluated the predictive power on the other data set (full split descriptions in Supplementary Fig. 3.a. The proteomics-based predictor performed better than the WSI-based predictor (Table. 1).

Next, we asked whether we could derive a more accurate predictor if we combine patient-paired H&E WSIs and proteomics data using multi-modal deep learning frameworks ^18–20. The model combining WSI and proteomic data results in a significant increase in model performance. (Table. 1). Multi-modal benchmarks consistently outperformed both WSI-only and proteomics-only models predicting treatment resistance to platinum chemotherapy. This finding was robust across various training and testing setups (See methods section 3.1 and Supplementary Figure. 3.a). This suggests that there exist multi-modal features invariant across cohorts that correlate well with tumor sensitivity to treatment. The test AUC results pertain solely to cohorts from held-out sites, ensuring the mitigation of potential biases and validating the generalization of discovered features ²². Table. 1 presents the best performing PorpoiseMMF ¹⁸ model (Supplementary notes 3.5) performance for the multimodal model. Tables containing the results for all benchmarked models can be found within the Supplementary material (Supplementary Tables 1–4).

The results for primary samples shown in Table 1 demonstrate that overall, proteomics-only models slightly outperform WSI-only models. When training on PTRC-HGSOC ⁹ primary tumor samples and testing on TCGA samples, the multi-modal model achieves an AUC (Area Under the Curve) of 0.752. This is a significant improvement over the proteomics-only model, which has an AUC of 0.61. This represents a 14% increase (t = 6.24, p = 0.00336). When training on TCGA tumor samples and testing on PTRC-HGSOC primary tumor samples, the multi-modal model achieves an AUC of 0.835. This represents a 13.6% increase over the proteomics-only model, which has an AUC of 0.755 (t = 4.14, p = 0.0144). Finally, when training on the FHCRC + Mayo sub-cohorts (figure. 2.) of PTRC-HGSOC primary tumor samples and testing on UBC primary tumor samples, the multi-modal model achieves an AUC of 0.84. This represents a 19.9% increase over the proteomics-only model, which has an AUC of 0.558 (t = 7.0, p = 0.00219).

Focusing on metastatic cases separately, Table 1 demonstrates that overall, proteomics-only models slightly outperform WSI-only models, and all test results are slightly lower than primary trained models, as TCGA samples are only from primary tumor sites. When training on PTRC-HGSOC metastatic tumor samples and testing on TCGA samples, the multi-modal model achieves an AUC of 0.704, representing an 8.2% increase over the proteomics-only model, which has an AUC of 0.54 (t = 2.63, p = 0.058). Training on TCGA tumor samples and testing on PTRC-HGSOC metastatic tumor samples, the multi-modal model achieves an AUC of 0.698, representing a 13.7% increase over the proteomics-only model, which has an AUC of 0.566 (t = 2.58, p = 0.0614). Finally, when training on the FHCRC + Mayo sub-cohorts of PTRC-HGSOC metastatic tumor samples and testing on UBC metastatic tumor samples, the multi-modal model achieves an AUC of 0.798, representing a 16.5% increase over the proteomics-only model, which has an AUC of 0.665 (t = 3.33, p = 0.029).

When modeling histopathology-proteomics data, the experimental setup involves selecting the model architecture, the proteomic input structure (methods 3.6), and the histopathology patch embeddings (methods 3.4). We evaluated how changes in each of these choices impacted the performance in predicting platinum response.

Comparing multimodal model architectures, we found that PorpoiseMMF¹⁹ outperformed MCAT²⁰ and SurvPath¹⁹ in overall model performance. The former being a late fusion model and the latter two being early fusion models. In terms of proteomic groupings, Chowdhury signature (methods 3.6.1) > CPTAC signature (methods 3.6.2) > RM pathways (methods 3.6.4) > PRG pathways (3.6.3). Comparing pretrained histopathology image patch embedding networks, our SSL trained ovarian cancer-specific DINO-OV (Supplementary Table 5 for pre-training data and Supplementary section 3.8 for SSL details), UNI²³ and CTransPath²⁴ performed similarly across the tasks (Supplementary Tables 1–4).

Although primary and metastatic cancers show many similarities, they demonstrate large differences in response and prognosis. We found that training the models separately for the primary and metastatic cohorts yields better performance. Based on these findings we also recommend this practice for similar studies.

1.2 WSI + proteomics compared to genetics-based models

Loss of certain biological functions, such as the capacity of homologous recombination-based DNA repair, has been known to be associated with response to DNA damaging therapy in ovarian cancer ²⁵. We compared our WSI + proteomics-based model predictions against the predictive power of genomics-based methods of detecting homologous recombination deficiency status. The FDA-approved Myriad HRD-score assesses genomic alterations to indicate the presence of HRD. The FoundationOne CDx test offers comprehensive genomic profiling, including TMB and genome-wide LOH (loss of heterozygosity) status, which correlates with HRD status. Research tools like HRDetect ²⁶ utilize whole-genome sequencing and lasso regression to predict HRD with high sensitivity. Finally, SigMA exposes notable mutational signatures associated with HRD where Signature 3 ²⁷ is of special interest.

For our studied cohorts, the HRD-score, Signature 3 and HRDetect scores were available for n = 98 TCGA cases. The WSI + proteomics model outperformed the HRD-score, HRDetect, and Signature 3 for the TCGA cohort in predicting tumor response to platinum-based therapy. We found that a linear combination of our WSI + proteomics model and HRD-score showed a significant increase in performance over the pure HRD-score model (DeLong test Z-statistic: -3.113, P-value: 0.002) (See Fig. 3.a, Fig. 3.b, and Supplementary Fig. 4.c). Furthermore, our model predictions were uncorrelated with the HRD-score predictions implying genomics-based and histo-proteomics information is complementary containing little mutual information regarding therapy response prediction (See Fig. 3.c). Performing a more granular investigation into model predictions over the cohorts, we visualize AUC scores for the cohorts distinguished by various clinical variables (See Fig. 3.d)

1.2.1 Predicting survival – Prognostic evaluation

To further evaluate and validate the utility of our multi-modal WSI + proteomics model we conducted survival analysis, comparing the prognostic and predictive potential of the model to other, previously established measurements. In case of the TCGA-OV cohort Cox proportional hazard models were used to determine hazard ratios and compare survival (Overall Survival (OS), DSS, PFI, PFS) and HRD-score, HRDetect, WSI-only, proteomics-only, WSI + proteomics and ensembled models using median cutoff or best cutoff, and the plots for and Kaplan-Meier analysis was performed with the survival R package and Overall Survival (OS), Disease-Specific Survival (DSS), Progression-Free Interval (PFI), Progression-Free Survival (PFS), and Cox proportional hazards models were utilized to determine Hazard Ratios for the validation and TCGA-OV cohort (See Fig. 3.d). Our WSI + proteomics multi-modal model outperformed HRDetect for survival prediction its ability to separate risk groups. For OS, logrank HRDetect p = 0.04 WSI + proteomics p = 0.01. For PFS logrank HRDetect p = 0.05 WSI + proteomics p = 0.01. This is further shown by comparing C-Index values for PFS prediction across the models (Fig. 3.b). Our models were trained to predict the response to treatment rather than individual survival or risk. Nonetheless, as the model demonstrates prognostic efficacy, it further validates its capacity to distinguish between patients with favorable and unfavorable responses accurately. The full analysis is shown in Supplementary Fig. 2.

1.3 Neural network attention maps reveal interpretable spatial distributions from bulk proteomics

As shown for the WSI in Figure. 4.a, the uni-modal WSI-only CLAM ²¹ allows us to visualize patch importances, or "attentions," on the WSI, creating an "attention heatmap" (Figure. 4.b). This visualization provides insight into the model's decision-making process. Patches with high attention are crucial for classifying the entire slide. For instance, if tumor areas of the WSI are highlighted on the attention map for a task that relies on known tumor localized histopathological features, it may validate that the model aligns with pathologists' thinking. Additionally, when we lack documented histopathological features for a task, these heatmaps can help us discover important features the model considers significant. This is particularly relevant for predicting platinum response, where histopathology is not a primary biomarker for decision-making.

Extending this concept into multimodal models, MCAT ²⁰ and SurvPath ¹⁹ enable the visualization of more complex "co-attention heatmaps." For each biological pathway, the model generates one “gene-guided visual concept” ²⁰for each patch in the WSI. These highlight important visual features integrated with proteomic data. The fact that the “co-attention heatmaps” in Fig. 4.d can reveal spatial features may seem to be surprising at first sight. However, this is alike deep neural network’s ability to recognize objects in images after training without detailed annotations. For an intuitive description of how images are connected to proteomic features see Supplementary notes 4.2.

A key strength of MCAT ²⁰ and SurvPath ¹⁹architectures is the ability to visualize these pathway-specific attention maps over WSIs, thereby potentially discovering important morphological features related to treatment sensitivity or resistance. While the critical role of the tumor microenvironment and stroma in treatment response and disease progression is widely recognized, routine histopathological assessments often overlook these components. We found that the stroma adjacent to tumor cells frequently attracts significant attention in numerous slides and cases, underscoring the presence of valuable information within the non-tumor regions of pathological slides. In Figure. 4 we showcase an example case study. Furthermore, we validate the robustness of these generated heatmaps with an ablation study (Supplementary Fig. 4.a). See Supplementary Figs. 5, 6, and 7 for further case examples.

1.4 Pathways and genes with high attention

The MCAT ²⁰ and SurvPath ¹⁹models are pioneering efforts in integrating multimodal data to enhance model interpretability within the domain of clinical outcomes. These models employ an early fusion technique that incorporates pathways; groups of proteins with interconnected functions, as fundamental components of their architecture. This approach leverages learnable embeddings to represent these pathways, generating a single value or "attention" that defines how specific pathways contribute to clinical endpoints. By incorporating pathway embeddings, these models facilitate a method of ranking and visualizing the relative importance of different pathways in relation to a clinical outcome. This visualization is achieved through the analysis of pathway embeddings’ weights, offering insights into which pathways are most critical for the model’s predictions (Fig. 4.c).

Furthermore, the models assign "feature importances" to individual protein expression values within each pathway group. These importances are derived from integrated gradient values ²⁸, a method detailed in the Supplementary notes 3.7. Using both interpretability measures gives a two-tiered overview or "hierarchical explainability," where both the pathway level and the protein level within pathways are illuminated in terms of their contribution to the model’s decision-making process.

To assess whole cohort trends in pathway and proteomic significance, we aggregate these feature importances across the PTRC-HGSOC cohort. This analysis is conducted separately for tumors classified as sensitive and refractory, providing a broad overview of unique proteomic and pathway dynamics across these two categories. The results, highlight the prominence of the HRD pathway in sensitive tumors. In refractory tumors, pathways related to DDR nucleotide synthesis, apoptosis, ER stress, and P53 signaling are identified as significantly influential.

To further investigate the relative importance of individual domain-specific pathways related to HGSOC platinum response we found that the BIOCARTA ATR-BRCA ²⁹ pathway was the most important in both sensitive and refractory cohorts (Figure. 5.a, 5.b). Non-homologous end joining was ranked second for refractory samples and Base excision repair was second highest for the sensitive cohort. Within the BIOCARTA ATR-BRCA ²⁹ pathway, we found that ATR and CHEK1 had the largest influence on model decision-making for sensitive tumors. For the refractory cohort, the models were most sensitive to CHEK1 and TP53 expression values. See Figure. 5 for the top 5 pathways and top proteins for each tumor response cohort and Supplementary Fig. 1. for the ranking of all pathways and proteins.

Debulking surgery followed by platinum-based chemotherapy has been the standard of care of HGSOC. Patients responding to first-line platinum-based therapy often have a durable remission and the clinical benefit could also be further extended by the administration of targeted agents, such as PARP inhibitors. However, platinum resistance, defined as disease recurrence within 6 months of first-line platinum-based chemotherapy, occurs in approximately 25% of cases and the prognosis of such cases is worse with a median survival of 9–12 months ^30,31. Such cases may benefit from novel therapy such as the recently approved antibody drug conjugate Mirvetuximab Soravtansine ³². Accurate prediction of resistance to platinum-based therapy may allow the early administration of other therapies, without waiting several months of failed platinum therapy. In principle, the surgical, debulking biopsy could be used to predict response to therapy to first-line platinum treatment, but no diagnostic method has reached the desired predictive accuracy so far.

A great number of biomarkers have been evaluated to predict platinum sensitivity in HGSOC ³³. Some of those biomarkers were derived from genomic or transcriptomic profiling of HGSOC of known clinical outcome and others, such as quantifying HR deficiency, were developed based on biological rationale. HR deficiency can be detected by various next generation sequencing based methods such as the HRD-score ³⁴, and the FDA-approved Myriad myChoice CDx ³ and FoundationFocus CDxBRCA LOH ⁴, which are based on similar principles. However, the predictive power of these methods was only validated for PARP inhibitor therapy ³⁵, and they are not used in the decision making for first line platinum treatment.

Recently, digital pathology and associated deep learning-based computer vision algorithms emerged as an alternative diagnostic method to predict HRD status and platinum sensitivity. ^7,8,15,36

Since molecular profiling-based, biological pathway-based and digital pathology-based predictors all showed a certain level of predictive power but none of those had sufficiently high accuracy alone, we were wondering whether these methods all capture to a certain extent different aspects of platinum response/resistance and whether those could be combined. To test this hypothesis, we decided to integrate proteomics and histopathology. We found that our models can capture non-mutual information that spans from molecular mechanisms to tissue-level manifestations of HGSOC. Proteomics provides the "what" in terms of molecular changes and potential targets, while histopathology provides the "where" and "how" these changes manifest in tissue architecture and the tumor microenvironment. Our results indicate that amalgamating information from these projections of the disease is beneficial for machine-learning models, similar to how an oncologist integrates multiple sources of information to make informed clinical decisions. Our results are concordant with other recent literature where multimodal deep learning has been shown to improve model performance in risk stratification of HGSOC using MRI, histopathology, and genomic signatures with a late fusion architecture ^37,38. Furthermore, including clinical variables has also been shown to enhance model performance ³⁹.

Our models are validated not only by various train-test configurations but also by the rediscovery of pathways known to be of high importance for HGSOC platinum response, such as Homologous Recombination Repair. Additionally, tumor regions of WSIs gather high patch attention in the WSI-only CLAM ²¹ model, validating the weakly supervised learning as a modeling decision for WSIs in this context.

We chose to focus on combining proteomic data with WSI data because there was sufficient and appropriate data available for both training and validation. Proteomics is rarely used in the diagnostic setting although reverse-phase protein arrays (RPPA) may provide a diagnostically applicable method to quantify a few hundred proteins ⁴⁰. Combining transcriptomics or epigenomic profiling with WSI by the strategy outlined in this paper is a logical next step that may be easier to incorporate into the existing diagnostic pipelines. It is encouraging, that the combined WSI/proteomics predictor in a linear combination of HRD measures produced a predictor with an accuracy that may merit further serious consideration in the clinical setting if verified by further independent cohorts.

Variations in H&E staining routines, sample preservation, and imaging technologies across institutions pose additional challenges. We addressed these through pathology-specific self-supervised learning pre-training ⁴¹, effectively normalizing across such variations and ensuring our models’ applicability to diverse clinical settings. Similarly, potential discrepancies in LC-MS/MS data arising from institutional protocol differences were mitigated by comprehensive distribution overlap analyses, confirming the consistency of our biological signal capture.

3.1 Patient cohorts

This study aimed to predict response to platinum treatment in high-grade serous ovarian cancer by using supervised deep multi-modal learning to H&E WSIs and proteomic LC-MS/MS data. Based on literature search there were only 2 datasets with these data available: PTRC-HGSOC (158 patients 348 samples with all 3 data types) ⁹ and TCGA (127 patients 159 samples with all 3 data types)¹². Refractory cancers were defined as those that progressed or had stable disease within 6 cycles of initial platinum/taxane therapy after initial debulking surgery⁹. Sensitive tumors were defined as those that responded to initial platinum/taxane therapy and did not progress within 2 years. All samples were treatment-naive at the time of biopsy.

3.2 Multimodal deep learning models

In this study we benchmarked various experimental setups. These corresponded to three histopathology patch embedding networks (methods 3.4), five model architectures (Supplementary notes 3.2–3.5) and 4 proteomics grouping setups (methods 3.6.1–3.6.4). Each experiment was run over 5 patient stratified folds, and three training-validation split setups. Reported test accuracies were from the ensemble of the 5 trained models’ predictions. Error estimates are obtained through bootstrapping with 1,000 samples, providing an assessment of variability.

The multimodal model accuracies for PorpoiseMMF ¹⁷ + Chowdhury signature ⁹ + CTransPath ²⁴ setup is presented in Table. 1 as well as for the more granular TCGA genomics analysis in section 3. We investigated the MCAT ²⁰ model + PRG (methods 3.6.3) + CTransPath ²⁴ for deep exploration into WSI and proteomics interpretability (Supplementary notes 4.1–4.2).

3.3 Whole slide images

The PTRC-HGSOC dataset of 348 FFPE WSIs was downloaded from The Cancer Imaging Archive (TCIA) ⁴² [https://www.cancerimagingarchive.net/collection/ptrc-hgsoc/ representing 158 unique patients. The bookend FFPE slides were cut at 4 µm thickness using a microtome and mounted on glass slides (Leica Biosystems Cat 3800040) for H&E staining. Digital images of the H&E slides were recorded using a ScanScope AT Slide Scanner (Leica Aperio Technologies, Vista, CA, USA) under 20X objective magnification (0.4965 µm per pixel resolution). The 348 of the relevant WSIs of the TCGA-OV dataset that were part of the CPTAC-OV cohort were downloaded from The Cancer Imaging Archive (TCIA)⁴² https://www.cancerimagingarchive.net/collection/tcga-ov/ representing 175 unique patients TS1, TS2 (Fresh Frozen), DX1 and DX2 (FFPE) type slides. The TCGA-OV WSIs are heterogeneous in terms of scanner modalities, manufacturers, and acquisition protocols. In most cases, the images were acquired as part of routine care and not as part of a controlled research study or clinical trial. H&E slides were recorded under 20X objective magnification (0.5040 µm per pixel resolution).

3.4 Segmenting and embedding H&E WSIs

The images were visualized to ensure quality, segmented to remove the slide background or scanner artifacts with the CLAM ²¹ WSI preprocessing pipeline, and patched into 224 X 224 pixel patches representing approximately 112 X 112µm. To reduce the dimensionality of these large gigapixel images, these patches were then packed into tensor "bags" with dimensions (n,3,224,224) where n is the number of patches generated for a WSI. These patches were then embedded by the three feature-extracting vision transformer networks to create three separate sets of embeddings covering all samples and cohorts. The CTransPath ²⁴ embedding model was chosen as was it best performing over a large set of tasks ⁴¹. The UNI²³ model was chosen as it was pre-trained on non-TCGA data. This ensures there is no possible data leak when testing downstream models on TCGA WSI data.

Our DINO-OV model was also used as a third model to investigate if "same-domain" image embedders would provide a benefit for this task. Each model had an associated raw patch RGB space transformation and maps each patch dimensions (3, 224, 224) to a vector shaped (d) where d = 1024 for UNI, d = 384 for DINO-OV, and d = 768 for CTransPath. The embedding stage was performed without stain normalization due to recent findings that this has little to no effect on downstream accuracy ⁴¹.

3.5 Proteomic Data

The TCGA MS proteomics data was downloaded from the Supplemental Information Table.3 of Zhang et al.¹². Following pre-processing and validation from the original manuscript an average of 7,952 proteins were measured per tumor. The PTRC-HGSOC proteomics data was downloaded from the deposited data table: Processed LC-MRM-MS data ⁹. After pre-processing 8,800 proteins were found in over 50% of the measured bio-specimens.

The overlapping set of proteins from the TCGA and PTRC-HGSOC datasets were collected amounting to 7621 shared proteins. TCGA (CPTAC) and PTRC-HGSOC datasets were imputed with the scikit-learn KNNImputer ⁴³, log normalized, and standard scaled to remove any batch effect present in the data. The alignment of datasets was validated by testing the overlap of distributions of each protein. We employed the nonparametric statistical test the Kolmogorov- Smirnov (KS) test, to compare the distribution of each protein between the two independent datasets. We found that they do not show a statistically significant difference: Mean KS Statistic: 0.112, Mean P-Value: 0.128.

3.6 Methodologies for proteomics groupings in multi-modal models

Deliberately selecting and structuring proteomics data to add to multi-modal deep learning models has an impact on final model performance as well as the inherent interpretability of the model once trained. We investigated proteomics selection with three high-level strategies. Signature-based, subtype-based, and pathways-based. A signatures-based strategy involves injecting only one fixed length 1d vector also known as a "proteomic signature" which is a subset of the proteomics data. This vector is found by feature selection methods and could be weights or unweighted. In our study, we investigated the unweighted "Chowdhury signature" (methods 3.6.1) proteomics signature. Subtypes-based grouping also offers another solution where non-overlapping protein "subtype" groups from structures (methods 3.6.2). Finally, pathways represent unique cellular functions, they constitute appropriate basic reasoning units suitable for interpretability. For multi-modal models, MCAT ²⁰ and SurvPath ¹⁹data was organized into such groups (methods 3.6.3–3.6.4). We performed four different groupings to assess the importance and robustness of the multi-modal models to the groupings for this task. PorpoiseMMF models were trained with the same sets of proteins as these groups but not grouped instead as one vector input to be concatenated during model fusion. We experimented with two overlapping and two non-overlapping groupings to generate group-level tokens.

3.6.1 Platinum-response protein signature (“Chowdhury Signature”)

Chowdhury et.al ⁹ describe a 64-gene unweighted proteomic signature that predicts response to platinum treatment. This signature was identified by investigation of 22 pathways based on 31 years of literature ⁴⁴ followed by nonlinear machine-learning-based feature selection. From these proteins, 60 were measured in both our training and validation set. This signature was used as input to the PorpoiseMMF model. It was possible to group these proteins into 5 non-overlapping groups: Drug Metabolism & Biological Oxidation, Metabolic, Hypoxia, NK-kB as was described in Chowdhury et.al ⁹. These sub-groups were used in the MCAT and SurvPath architectures.

3.6.2 CPTAC HGSOC subtypes (“CPTAC grouping”)

Zhang et.al ¹² describe 1664 proteins clustered into 7 proteomic subtypes from the non-overlapping WGCNA- derived modules pathways based on the enrichment of KEGG and Reactome ontologies. These groups are labeled: ’DNA replication’, ’cell-cell communications’, ’cytokine signaling’, ’erythrocyte and platelet’, ’ECM interaction’, ’complement cascade’, and ’metabolism’. This method aimed to inject subtype information into the model explicitly to see if it enhanced model accuracy.

3.6.3 Platinum resistant genes and pathways (“PRG pathways”)

Huang et.al ⁴⁴ curated a list of > 900 proteins associated with platinum resistance in cancer from 30 years of literature. From these proteins, 587 were found for all patient samples in our studied cohorts. This is an appropriate list of pathways and genes to use when exploring HGSOC treatment responses as they cover a broad spectrum of biological processes implicated in drug resistance. Specifically, regulation of drug dynamics: the control over drug entry, exit, accumulation, sequestration, and detoxification within the cells, DNA damage response: enhanced capabilities for repairing and tolerating platinum-induced DNA damage, cell survival pathways: Alterations that allow cancer cells to avoid death despite chemotherapy, Pleiotropic processes: Changes in broad, multifunctional pathways that affect cancer cell behavior and finally, Tumor micro-environment: Variations in the cellular environment surrounding tumors that may contribute to drug resistance.

3.6.4 Repair mechanism pathways (“RM pathways”)

To investigate the role of DNA repair mechanisms-related pathways in predicting response to platinum-treatment we utilized the 1664 DNA repair-related pathways from MSigDB and KEGG. These pathways are more mechanistic than the “PRG pathways” section 3.6.3 and are not constrained to the platinum response. See Figure. 5 for pathways and genes ranked by importance.

3.7 Models

3.7.1 Training

For each experiment, we created a 5-fold split stratified on patient ID to ensure samples from the same patient were never found in the training and validation sets. Each learning task was set up as a binary classification (Sensitive vs Refractory). The proteomics-only baseline was an ensemble of ElasticNet, Random Forest, and XGBoost as described in Chowdhury et.al ⁹.

We trained all the deep learning based binary classification models for 15 epochs with gradient descent and the ADAM optimizer and a learning rate of 10^−4. CLAM and multimodal models were trained with a used a binary cross-entropy loss. Models were checkpointed during training and the checkpoints with the lowest validation loss were used for evaluation on test sets. We used L2 regularization with L = 10^−5 and weighted sampling to make sure models were not biased by class imbalances. We used a binary cross-entropy loss:

Where: N is the number of samples, C is the number of classes, yi. C is a binary indicator of whether class label c is the correct classification for observation i, pi. c is the predicted probability that observation i is of class c, λ is the regularization parameter, M is the number of weights, and wj represents each weight parameter in the model.

The full pipeline was trained on 3 Quadro RTX 8000 GPUS at the Budapest EMK server. The CLAM models were run with an SVM bag clustering loss. Metadata was checked for the PTRC-HGSOC hospital hold-out experiments to see if platinum sensitivity could be predicted from this data. Random forest models were not able to predict the label with AUC = 0.47 for the UAB hold-out and AUC = 0.55 for the Mayo hold-out experiment. This ruled out the learning of co-variate signals that could bias results or allow for shortcut learning.

Data availability

No new data was generated during this study. We release our splits of patients and cohorts in the supplementary information.

Code availability

The code to reproduce all results presented in the study is available at https://github.com/csabaiBio/HGSOC_platinum_response

Acknowledgements

This work was supported by the National Research, Development, and Innovation Office of Hungary within the framework of the MILAB Artificial Intelligence National Laboratory (RRF-2.3.1-21-2022-00004) and the Data-Driven Health Division of National Laboratory for Health Security (RRF-2.3.1-21-2022-00006) and by the European Union (Grant Agreement Nr. 10109571, SECURED Project) and OTKA K128780. Further support was provided by the Stipendium Hungaricum Program under the Tempus Public Foundation. It was also supported by the Research and Technology Innovation Fund (KTIA_NAP_13-2014-0021 and 2017-1.2.1-NKP-2017-00002 to Z.S.), Breast Cancer Research Foundation (BCRF-23-159 to Z.S.), Kræftens Bekæmpelse (R325-A18809 and R342-A19788 to Z.S.), Det Frie Forskningsråd Sundhed og Sygdom (2034-00205B to Z.S.), Basser Foundation (to Z.S.), NIH Grant 1 P01 CA228696-01A1 to Z.S.

Author contributions

Z.S, Zs.S, I.C and O.K conceived the experiments, O.K and Zs.S conducted the experiments. Z.S, Zs.S, I.C and O.K analyzed the results. A.O built out computational infrastructure for experiments. A.B trained the Ovarian-specific SSL WSI patch embedding model. P.P managed the computational infrastructure. L.M, Zs.S and analyzed the generated heatmaps. Z.S, Zs.S, and I.C supervised this project. All authors read and approved the final manuscript.

Competing Interests

Z.S. is an inventor on a patent used in the myChoice HRD assay.

Cannistra, S. A. Cancer of the ovary. New England Journal of Medicine 351, 2519–2529 (2004).
Telli, M. L. et al. Homologous recombination deficiency (HRD) score predicts response to platinum-containing neoadjuvant chemotherapy in patients with triple-negative breast cancer. Clinical cancer research 22, 3764–3773 (2016).
Test, M. Cd. MyChoice CDx Test. US Pharm 48, 43–45 (2023).
Milbury, C. A. et al. Clinical and analytical validation of FoundationOne® CDx, a comprehensive genomic profiling assay for solid tumors. PLoS One 17, e0264138 (2022).
Davies, H. et al. HRDetect is a predictor of BRCA1 and BRCA2 deficiency based on mutational signatures. Nat Med 23, 517–525 (2017).
Sztupinszki, Z. et al. Comparative assessment of diagnostic homologous recombination deficiency–associated mutational signatures in ovarian cancer. Clinical Cancer Research 27, 5681–5687 (2021).
Bergstrom, E. N. et al. Deep learning predicts HRD and platinum response from histology slides in breast and ovarian cancer. medRxiv 2022–2023 (2023).
Loeffler, C. M. L. et al. Direct prediction of Homologous Recombination Deficiency from routine histology in ten different tumor types with attention-based Multiple Instance Learning: a development and validation study. medRxiv (2023).
Chowdhury, S. et al. Proteogenomic analysis of chemo-refractory high-grade serous ovarian cancer. Cell 186, 3476–3498 (2023).
Yu, K.-H. et al. Predicting ovarian cancer patients’ clinical response to platinum-based chemotherapy by their tumor proteomic signatures. J Proteome Res 15, 2455–2465 (2016).
Lu, J. et al. Prediction of platinum-based chemotherapy response in advanced high-grade serous ovarian cancer: ADC histogram analysis of primary tumors. Acad Radiol 28, e77–e85 (2021).
Zhang, H. et al. Integrated proteogenomic characterization of human high-grade serous ovarian cancer. Cell 166, 755–765 (2016).
Bateman, N. W. et al. Proteogenomic analysis of enriched HGSOC tumor epithelium identifies prognostic signatures and therapeutic vulnerabilities. NPJ Precis Oncol 8, 68 (2024).
Wang, C.-W. et al. Weakly supervised deep learning for prediction of treatment effectiveness on ovarian cancer from histopathology images. Computerized Medical Imaging and Graphics 99, 102093 (2022).
Lazard, T. et al. Deep learning identifies morphological patterns of homologous recombination deficiency in luminal breast cancers from whole slide images. Cell Rep Med 3, (2022).
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J. & Shah, S. P. Harnessing multimodal data integration to advance precision oncology. Nat Rev Cancer 22, 114–126 (2022).
Lipkova, J. et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell 40, 1095–1110 (2022).
Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878 (2022).
Jaume, G. et al. Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction. arXiv preprint arXiv:2304.06819 (2023).
Chen, R. J. et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. in Proceedings of the IEEE/CVF International Conference on Computer Vision 4015–4025 (2021).
Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat Biomed Eng 5, 555–570 (2021).
Norgeot, B. et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat Med 26, 1320–1324 (2020).
Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat Med 30, 850–862 (2024).
Wang, X. et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Med Image Anal 81, 102559 (2022).
Sztupinszki, Z. et al. Comparative assessment of diagnostic homologous recombination deficiency–associated mutational signatures in ovarian cancer. Clinical Cancer Research 27, 5681–5687 (2021).
Davies, H. et al. HRDetect is a predictor of BRCA1 and BRCA2 deficiency based on mutational signatures. Nat Med 23, 517–525 (2017).
Polak, P. et al. A mutational signature reveals alterations underlying deficient homologous recombination repair in breast cancer. Nat Genet 49, 1476–1486 (2017).
Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. in International conference on machine learning 3319–3328 (2017).
Subramanian, A. et al. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences 102, 15545–15550 (2005).
Tewari, K. S. et al. Final overall survival of a randomized trial of bevacizumab for primary treatment of ovarian cancer. Journal of Clinical Oncology 37, 2317 (2019).
Davis, A., Tinker, A. V & Friedlander, M. “Platinum resistant” ovarian cancer: what is it, who to treat and how to measure benefit? Gynecol Oncol 133, 624–631 (2014).
Matulonis, U. A. et al. Efficacy and safety of mirvetuximab soravtansine in patients with platinum-resistant ovarian cancer with high folate receptor alpha expression: results from the SORAYA study. Journal of clinical oncology 41, 2436–2445 (2023).
van Zyl, B., Tang, D. & Bowden, N. A. Biomarkers of platinum resistance in ovarian cancer: what can we use to improve treatment. Endocr Relat Cancer 25, R303–R318 (2018).
Telli, M. L. et al. Homologous recombination deficiency (HRD) score predicts response to platinum-containing neoadjuvant chemotherapy in patients with triple-negative breast cancer. Clinical cancer research 22, 3764–3773 (2016).
Ray-Coquard, I. et al. Olaparib plus bevacizumab as first-line maintenance in ovarian cancer. New England Journal of Medicine 381, 2416–2428 (2019).
Bourgade, R. et al. Deep learning for detecting brca mutations in high-grade ovarian cancer based on an innovative tumor segmentation method from whole slide images. Modern Pathology 36, 100304 (2023).
Boehm, K. M. et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat Cancer 3, 723–733 (2022).
Hatamikia, S. et al. Ovarian cancer beyond imaging: integration of AI and multiomics biomarkers. Eur Radiol Exp 7, 50 (2023).
Nguyen, E., Cui, Z., Kokaraki, G., Carlson, J. & Liu, Y. Transferable and interpretable treatment effectiveness prediction for ovarian cancer via multimodal deep learning. in AMIA Annual Symposium Proceedings vol. 2023 550 (2023).
Gallagher, R. I. & Espina, V. Reverse phase protein arrays: mapping the path towards personalized medicine. Mol Diagn Ther 18, 619–630 (2014).
Wölflein, G. et al. A Good Feature Extractor Is All You Need for Weakly Supervised Learning in Histopathology. arXiv preprint arXiv:2311.11772 (2023).
Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 26, 1045–1057 (2013).
Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011).
Huang, D. et al. A highly annotated database of genes associated with platinum resistance in cancer. Oncogene 40, 6395–6405 (2021).
Wang, C.-W. et al. Weakly supervised deep learning for prediction of treatment effectiveness on ovarian cancer from histopathology images. Computerized Medical Imaging and Graphics 99, 102093 (2022).
Caron, M. et al. Emerging properties in self-supervised vision transformers. in Proceedings of the IEEE/CVF international conference on computer vision 9650–9660 (2021).
Antol, S. et al. Vqa: Visual question answering. in Proceedings of the IEEE international conference on computer vision 2425–2433 (2015).

Table 1. The table displays separated reported Area Under the Curve (AUC) scores for primary and metastatic tumors with the multimodal PorpoiseMMF ¹⁸ model + Chowdhury signature (methods 3.6.1). The histopathology model is the CLAM ²¹ architecture, and the proteomics model is from the “Chowdhury signature” trained with an ensemble model as described in Chowdhury et.al ⁹. Multimodal models outperform their unimodal counterparts. Test AUCs represent the mean AUC obtained from the ensembled predictions of the five models. Error estimates are obtained through bootstrapping with 1,000 samples, providing an assessment of variability. For validation scores, we report the average AUCs across the folds, with the standard deviation of these AUCs representing the reported errors.

There is a conflict of interest Z.S. is an inventor on a patent used in the myChoice HRD assay.

Supplementarymaterialfor.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Histopathology and proteomics are synergistic for High-Grade Serous Ovarian Cancer platinum response prediction

Status:

Version 1

Abstract

Figures

0 Introduction

1 Results

1.1 Combination of WSI with proteomics significantly increases the accuracy of response prediction

1.2 WSI + proteomics compared to genetics-based models

1.2.1 Predicting survival – Prognostic evaluation

1.3 Neural network attention maps reveal interpretable spatial distributions from bulk proteomics

1.4 Pathways and genes with high attention

2 Discussion

3 Methods

3.1 Patient cohorts

3.2 Multimodal deep learning models

3.3 Whole slide images

3.4 Segmenting and embedding H&E WSIs

3.5 Proteomic Data

3.6 Methodologies for proteomics groupings in multi-modal models

3.6.1 Platinum-response protein signature (“Chowdhury Signature”)

3.6.2 CPTAC HGSOC subtypes (“CPTAC grouping”)

3.6.3 Platinum resistant genes and pathways (“PRG pathways”)

3.6.4 Repair mechanism pathways (“RM pathways”)

3.7 Models

3.7.1 Training

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1