A Differentially Private Federated Learning Application in Privacy-Preserving Medical Imaging

doi:10.21203/rs.3.rs-3873379/v1

Download PDF

Research Article

A Differentially Private Federated Learning Application in Privacy-Preserving Medical Imaging

https://doi.org/10.21203/rs.3.rs-3873379/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

This research addresses the escalating concerns surrounding privacy, particularly in the context of safeguarding sensitive medical data within the increasingly demanding healthcare landscape. We undertake an experimental exploration of differentially private federated learning systems, employing three benchmark datasets—PathMNIST, BloodMNIST, and OrganAMNIST—for medical image classification. This study pioneers the application of federated learning with differential privacy in healthcare, closely simulating real-world data distribution across twelve hospitals. Additionally, we introduce a novel deep-learning architecture tailored for differentially private models. Our findings demonstrate the superior performance of federated learning models compared to traditional approaches, with accuracy levels approaching those of non-private settings. By leveraging resilient deep learning models, we aim to enhance privacy, efficiency, and effectiveness in healthcare solutions, benefiting patients, healthcare practitioners, and the overall healthcare system through privacy-protected healthcare.

Artificial Intelligence and Machine Learning

Nuclear Medicine & Medical Imaging

Federated Learning

Differential Privacy

Medical Imaging

Privacy-Preserving Deep Learning

Deep learning-based medical imaging analysis is faced with a challenge in fully utilizing the advancements in other fields of computer vision. This is because many advanced applications require massive amounts of data to obtain high levels of accuracy that can be used for diagnostic purposes. However, in the medical field, acquiring such datasets poses a problem as patient privacy must be upheld, and there is a need to minimize the amount of data collected and/or shared.

To address this issue, privacy-enhancing technologies have emerged as a viable solution that allows one to gain insights from sensitive datasets while safeguarding the privacy of individuals. These technologies provide the best chance to date to promote data sharing in a responsible and ethical manner. Federated Learning (FL) has emerged as a promising solution that allows for decentralized training on user devices without compromising the privacy of the underlying data. In this approach, instead of centralizing the data in a single location, the data remains on individual user devices or hospitals. This eliminates the need to transfer raw data to a central location, significantly reducing the risk of data breaches. Moreover, the central server coordinates the training process by sending model updates to the devices and receiving model updates from them, ensuring that the training process remains confidential.

Furthermore, Differential Privacy (DP), which adds noise to the data to obscure sensitive information, can further enhance privacy while maintaining accuracy. This technique works by introducing random perturbations into the data that are proportional to the sensitivity of the information contained in the data. By doing so, it becomes extremely difficult for attackers to identify individual data points or reconstruct the original data, even if they gain access to the trained model.

This powerful combination of federated learning and differential privacy holds immense potential for researchers seeking to train robust and accurate DNNs for medical imaging while safeguarding patient privacy. In this paper, we explore the implementation of the FedAvg (Federated Averaging) algorithm [17] and use the Opacus framework to implement differential privacy on the client side to reduce the risk of data leakage during weights transmission between hospitals/clients and the server. We conduct experiments on the MedMNIST datasets, which is a collection of medical images in the MNIST format, providing a comprehensive review of recent work in the field and a comparative analysis of our methodology's results.

Our findings demonstrate the potential of this technique for achieving high levels of privacy protection while training accurate DNNs for medical imaging. By leveraging federated learning and differential privacy, we can develop robust and accurate models that can be trained on large amounts of decentralized medical data without compromising patient privacy.

Differential privacy has been widely studied for medical imaging analysis in a traditional centralized machine learning setting. Authors in [1] proposed a differentially private stochastic gradient descent (DP-SGD) algorithm for training deep neural networks, which adds controlled noise to the gradients to protect the privacy of sensitive data.

Several studies have proposed the use of federated learning along with differential privacy for medical imaging analysis. For instance, a study by [13] utilized differential privacy for preserving the privacy of individual hospitals' data during the model update aggregation process. However, even with differential privacy, there is still a risk of data leakage during data transmission between hospitals and the server in the federated learning settings. While differential privacy is effective during aggregation, it does not guarantee data privacy during transmission.

Authors in [18] demonstrated the potential of federated learning for data-private multi-institutional collaborations in medicine, achieving 99% of the model quality obtained with centralized data using the BraTS dataset.

Authors in [23] proposed a new approach to personalized federated learning. The authors introduce a method for efficiently calculating optimal weighted model combinations

for each client, based on determining how much a client can benefit from a model trained on other clients' data. The proposed method is called Federated First Order Model Optimization (FedFOMO). Their work shows that FedFOMO outperforms existing methods for personalized federated learning on several benchmark datasets. The authors argue that personalized federated learning is crucial because it can improve the accuracy and efficiency of machine learning models in decentralized settings, where data is distributed across multiple devices or locations.

Another paper [14] proposes a federated learning approach, where a decentralized iterative optimization algorithm is implemented and shared local model weights are altered by a randomization mechanism. The authors also propose two domain adaptation methods in this federated learning formulation to address the systemic differences of fMRI distributions from different sites.

In the field of histopathology, authors in [15] demonstrated DP guarantees for a neural network classifier trained with federated learning. Their method involved the addition of Gaussian noise to trained model weights, however, the weights of neural networks lack bounded sensitivity, rendering their DP guarantee meaningless. A meaningful guarantee necessitates the clipping of model weights prior to the addition of noise. To be used in histopathology, we suggest the more conventional DP-SGD approach, which clips gradient updates and adds noise.

To further improve the performance of federated learning, researchers have proposed personalized cross-silo federated learning frameworks that adapt to each client's personalized model [16]. achieved state-of-the-art performance compared to other personalized FL approaches in the literature by testing their framework (APPLE) on two benchmark datasets and two medical imaging datasets from the MedMNIST datasets collection, PathMNIST [8] and OrganAMNIST [4]. They tested their approach on 12 clients. The researchers conducted evaluations on their models' performance in two types of settings: Pathological Non-IID and Practical Non-IID. The Pathological Non-IID setting is intended to assess how well FL algorithms perform under extremely non-IID conditions, while the Practical Non-IID setting aims to replicate a more realistic scenario seen in medical applications where institutes located in different regions may have datasets with varying sizes and imbalanced distributions due to different demographic distributions of patients and categories of data. (Table 1) compares their results to [23]’s results under practical Non-IID settings.

In a study conducted by [24], techniques for preserving privacy were reviewed in conjunction with machine learning, and subsequently applied in the classification of chest X-rays and segmentation of CT scans.

Authors [9] introduced a new framework called FedSup that integrates federated learning and neural architecture search to address the challenges of deploying deep neural networks on resource-constrained edge devices while preserving data privacy. They also proposed a new normalization technique called Parameteric Normalization (PN) to reduce discrepancies across shared normalization layers from different sub-networks. The authors found that FedSup with PN outperformed static batch normalization in terms of accuracy. Additionally, they developed an efficient algorithm called E-FedSup to reduce communication costs and training overhead by sending sub-models to clients on the broadcast stage. They discussed two settings for federated learning: personalized and initial. In the personalized setting, each client device has its own set of parameters that are learned during the training process, while in the initial setting, a single global model is initially trained on a central server and then fine-tuned on each client device using local data.

To evaluate their approach, they conducted empirical evaluations on several standard benchmarks, including the PathMNIST dataset, and the results demonstrated that FedSup with parameteric normalization outperformed static batch normalization in terms of accuracy. in (Table 2), we report their highly performed models in the initial setting since it is similar to our experiment settings in this paper.

Authors in [7], presented a framework for analyzing medical images using differential privacy and equivariant deep learning. They tested their framework on three datasets from the MedMNIST dataset: Colon Pathology, Dermatoscope, and Blood Cell Microscope. Their results are shown in (Table 3). They utilized privacy parameters $\left({\epsilon } = 7.42, {\delta } = {10}^{-5}\right)$ from [10] to ensure privacy protection.

However, in a federated learning context, where data is decentralized and trained locally on devices, unique privacy-preserving techniques may need to be employed to safeguard data source privacy. This is because the updated model weights are sent to a central server for aggregation, potentially exposing sensitive data. Therefore, specific privacy-preserving techniques intended for federated learning should be considered when working with decentralized data sources. Ensuring appropriate privacy-preserving techniques are utilized during federated learning is critical in minimizing the risk of data leakage. However, it is important to note that [7]'s experiments were conducted on conventional centralized machine learning settings and have not been explicitly tested for federated learning. Consequently, when working with decentralized data sources, testing privacy-preserving techniques for federated learning is crucial in guaranteeing their efficacy in preserving data source privacy during model training.

Authors in [3] used DP-SGD to train a differentially private federated learning model for medical image analysis and compared its performance with centralized and non-collaborative models. They conducted a case study using The Cancer Genome Atlas (TCGA) dataset to train multiple sets of hospitals (i.e. 4, 8, 16, 32) collaborating to train one central model. At the same time, the performance was evaluated on each hospital's internal test set and an external validation set. The authors observed that the model was sensitive to DP-SGD hyperparameters and used specific values. They concluded that differentially private federated learning is a viable and reliable framework for the collaborative development of machine learning models in medical image analysis, as it can achieve similar performance with strong privacy guarantees.

Despite the promising results reported in the literature, there are still several gaps, limitations, and challenges that need to be addressed. One of the main challenges is the trade-off between privacy and model accuracy, as adding noise to the data can negatively affect the model's performance. Moreover, the lack of standardized datasets for evaluating differentially private medical imaging algorithms makes it challenging to compare and reproduce results across different studies.

To address these gaps and limitations, our work aims to investigate the use of federated learning with differential privacy for medical imaging analysis, using the MedMNIST dataset. We aim to explore the trade-off between privacy and model accuracy and evaluate the performance of our approach against prior work. Additionally, we aim to provide insights into our study's limitations and potential extensions and discuss the implications of our findings for privacy in medical imaging.

A. Federated Averaging (FedAvg)

Federated averaging (FedAvg) is a communication-efficient algorithm for distributed training with a large number of clients [17]. In FL, clients keep their data locally for privacy protection, and a central parameter server is used to communicate between clients. The central server distributes the parameters to each client and collects the updated parameters from clients. The central server then aggregates these updated parameters to obtain the updated global model [17, 19] as depicted in (Fig. 1). The FedAvg algorithm consists of alternating between a few local stochastic gradient updates at client nodes, followed by a model averaging update at the server [5].

The FedAvg algorithm is known for its simplicity and efficiency in distributed training [5]. It is equivalent to the local SGD if the clients are homogeneous [19]. Several empirical studies have shown that the model output by FedAvg leads to a model that generalizes well to new unseen tasks after a few fine-tuning steps [5].

To establish the iteration complexity required by the clients for proving such a result, a recent study has formally established that FedAvg learns an expressive representation in any setting where the underlying shared representation is a linear map [5]. The FedAvg algorithm is computationally efficient but requires very large numbers of rounds of training to produce good models. Major speedups are obtained when computation on each client is improved, once a minimum level of parallelism over clients is achieved. The amount of computation is controlled by three parameters: C - Fraction of clients participating in that round, E - number of training passes each client makes over its local dataset each round, and B - Local minibatch size used for client updates.

FedAvg is a communication-efficient algorithm for distributed training with a large number of clients [17]. In FedAvg, clients keep their data locally for privacy protection, and a central parameter server is used to communicate between clients. This central server distributes the parameters to each client and collects the updated parameters from clients.

The FedAvg algorithm can be described as follows:

• The central server initializes the model parameters.

• In each round, a random subset of clients is selected to participate in the training process.

• The selected clients perform multiple local training steps on their local data using the current model parameters.

• The clients send their updated model parameters to the central server.

• The central server aggregates the model parameters received from the clients by computing their weighted average.

• The central server sends the updated model parameters to all clients.

The pseudocode for the FedAvg algorithm is shown below:

Algorithm 1: FederatedAveraging. The K clients are indexed by k; B is the local minibatch size, E is the number of local epochs, and η is the learning rate

Server executes:

Initialize${w}_{0}$

for each round t = 1, 2, … do

$m \leftarrow \text{m}\text{a}\text{x}(C . K, 1)$

${S}_{t}\leftarrow$(random set of m clients)

for each client $k \in {S}_{t}$ in parallel do

${w}_{t+1}^{k}\leftarrow \text{C}\text{l}\text{i}\text{e}\text{n}\text{t}\text{U}\text{p}\text{d}\text{a}\text{t}\text{e}(k, {w}_{t})$

${m}_{t}\leftarrow \sum _{k \in { S}_{t}}{n}_{k}$

${w}_{t+1}\leftarrow \sum _{k \in { S}_{t}}\frac{{n}_{k}}{{m}_{t}}{w}_{t+1}^{k}$

ClientUpdate($k, w$): // run on client$k$

$\beta \leftarrow \left(split {\rho }_{k} into batches of size B\right)$

for each local epoch $i$ from 1 to $E$ do

for batch $b \in \beta$ do

$w \leftarrow w- \eta \nabla \mathcal{l}(w;b)$

return $w$ to server

The FedAvg algorithm has been shown to achieve good performance in various settings [12]. It has also been extended and modified in various ways to improve its performance and efficiency [11][17][21].

To evaluate the effectiveness of Federated Learning with Differential Privacy, we implemented the algorithm in Python 3.9. Our implementation served as the basis for experiments investigating the algorithm's performance with differential privacy on the client's side across various privacy settings and participant numbers.

B. Differential Privacy

Differential privacy (DP) is a mathematical framework for ensuring the privacy of individuals when their data is collected and used by a machine learning algorithm. It was first introduced by [6]. Mathematically, it can be defined as follows:

$$Pr\left[M\right(D) \in S] \le {e}^{\epsilon } Pr\left[M\left({D}^{{\prime }}\right)\in S\right]+ {\delta }$$

A mechanism M satisfies ($ϵ$, δ)-differential privacy if, for all possible data sets D and D' that differ by a single individual's data point, and for all possible subsets S of the output space [6].

Where $ϵ$ controls the privacy guarantee. Smaller $ϵ$ values provide stronger privacy guarantees. When $ϵ$ is 0, it means perfect privacy, but it also means less utility. And δ introduces a small probability of failure to satisfy $ϵ$-differential privacy.

The key idea behind DP is that an algorithm is differentially private if, for any two datasets that differ in only one individual's data, the output of the algorithm is indistinguishable with high probability. This means that it is not possible to learn anything about any individual's data by observing the output of the algorithm. There are a number of different ways to add noise to data to achieve DP. Gaussian noise is the most common method for adding noise to data. It works by adding a random number from a Gaussian distribution to each data point. And Laplace noise, this method works by adding a random number from a Laplace distribution to each data point.

The amount of noise that is added to the data is determined by the privacy budget. The privacy budget is a measure of how much privacy is desired. A larger privacy budget means that less noise is added to the data, which results in a more accurate model. However, a larger privacy budget also makes it easier for an attacker to learn about an individual's data.

In [1], the authors show that it is possible to train deep learning models with DP. They do this by using a technique called differentially private stochastic gradient descent (DP-SGD). DP-SGD is a variant of stochastic gradient descent that adds noise to the gradients during training. This helps to ensure that the model does not learn too much about any individual's data. They also discuss the challenges of applying DP to deep learning. One challenge is that DP can make the training process more difficult and less efficient. Another challenge is that it can be difficult to choose the right privacy budget. Despite these challenges, DP is a promising approach for protecting the privacy of individuals when their data is used to train deep learning models.

In this study, we employed the Differentially-Private Stochastic Gradient Descent (DP-SGD) algorithm. The DP-SGD algorithm adds random noise to the gradients computed by SGD in order to provide differential privacy guarantees. The amount of noise added is controlled by two parameters: the noise multiplier and the privacy budget. The noise multiplier determines the strength of the added noise, while the privacy budget determines the amount of privacy protection provided by the algorithm.

The most two important techniques used in the DP-SGD algorithm are gradient clipping and momentum. Gradient clipping limits the magnitude of the gradients to prevent large updates that could compromise privacy. Momentum, on the other hand, helps the algorithm converge more quickly by adding a fraction of the previous gradient to the current gradient.

To reduce the risk of data leakage during data transmission between hospitals and the global model, we applied the DP-SGD algorithm on the client's side (as depicted in (Fig. 3)) using the Opacus framework [22].

Opacus, a Python library, emerges as a pivotal instrument in this domain, facilitating privacy-preserving training of deep learning models by injecting noise into gradient computations. At the core of Opacus lies the delicate balance between adding noise to gradients. Excessive noise can obscure crucial information, compromising model performance, while inadequate noise fails to provide robust privacy guarantees (notice the noised models in (Fig. 3) before sending updates to the global model). Opacus navigates this balance by examining the norms of gradients, ensuring an optimal trade-off. Opacus acknowledges that within data, outliers are prone to divulge sensitive information due to their disproportionately larger gradients. To mitigate this risk, Opacus adopts an individual sample approach. It computes gradients for each sample within a minibatch, subjecting them to individual gradient clipping. This strategy ensures that no single sample unduly influences the model's learning process, enhancing privacy protection. See (Fig. 2) for an illustration [22]. The library provides a set of utilities for adding differential privacy to PyTorch models, including the make_private_with_epsilon function which automatically sets the noise multiplier based on the desired privacy budget ($ϵ$).

In our experimental setup, we used the make_private_with_epsilon function to add differential privacy to the client-side training process with a desired privacy budget of 3. The function automatically calculated the corresponding noise multiplier based on the specified epsilon value, batch size, and other parameters.

Overall, our experimental setup with differential privacy enabled us to evaluate the effectiveness of the FedAvg algorithm in a decentralized learning task while preserving the privacy of the individual clients' data. By combining FedAvg with differential privacy, we believe that our results can help pave the way for practical applications of federated learning in sensitive domains, such as healthcare or finance, where data privacy is a critical concern.

For this study, we designed a new convolutional neural network (CNN) architecture that consists of several convolutional layers and residual blocks. The network uses group normalization layers to normalize the activations of the convolutional layers. Group normalization divides the channels of a convolutional layer into groups and computes normalization statistics for each group separately, which can be useful in scenarios where the batch size is small. The network's architecture includes two initial convolutional layers, followed by three residual blocks. Each residual block consists of two convolutional layers and a group normalization layer. The network ends with an adaptive average pooling layer, which aggregates the features across spatial locations, and a fully connected layer that produces the final classification output. The network is designed for image classification tasks and takes an input image with a specified number of channels and produces an output with a specified number of classes. A visual representation of the model architecture is depicted in (Fig. 4}.

A. Dataset

In this study, we utilized the MedMNIST v2 dataset collection, which is a large-scale collection of standardized biomedical images designed for lightweight 2D and 3D image classification [20]. Specifically, we used three different data modalities: Colon Pathology, Blood Cell Microscope, and Abdominal CT. The PathMNIST dataset [8] consisted of colon pathology images and had 107,180 pre-processed 28x28 images, divided into a training set (89,996), a validation set (10,004), and a test set (7,180). This dataset had 9 multi-class labels suitable for classification tasks.

The BloodMNIST [2] dataset consisted of blood cell microscope images and has 17,092 pre-processed 28x28 images, divided into a training set (11,959), a validation set (1,712), and a test set (3,421). This dataset had 8 multi-class labels suitable for classification tasks. Finally, the OrganAMNIST dataset [4] consisted of abdominal CT images and had 58,850 pre-processed 28x28 images, divided into a training set (34,581), a validation set (6,491), and a test set (17,778). This dataset had 11 multi-class labels suitable for classification tasks.

The Practical Non-IID setting is designed to emulate a more realistic scenario found in medical applications, where healthcare institutions in different regions may possess datasets of varying sizes and imbalanced distributions due to differences in patient demographics and data categories.

In our experiment, we adopted a simplified approach by dividing datasets among 12 hospitals. Initially, we ensured an equal number of samples in each partition and subsequently adjusted the partitions by periodically redistributing samples according to a predefined ratio. This ratio dictates the minimum number of samples allowed in a partition. In our experiment, the 20% ratio meant that the smallest partition contained at least 80% of the average sample size. (Table 4) shows the distribution of data among each hospital for each dataset.

B. Experimental Results

In our experimental investigation, we undertook the training of two distinct models for each of the datasets under consideration. Specifically, we developed a Non-Private Federated model (hereafter referred to as Non-DP FL) and a Differentially Private Federated model (referred to as DP FL). This dual-model approach allowed us to assess and compare the performance and privacy characteristics of these models within the context of federated learning.

To contextualize and benchmark our findings, we conducted a comparative analysis in Table 5. We contrasted our results with those of prior studies that utilized analogous datasets, drawing upon the APPLE [16], FedFOMO [23], and FedSup [9] frameworks to train Non-DP FL models. This comparative exercise enabled us to gauge the relative effectiveness and privacy implications of our DP FL model in relation to its Non-DP FL counterparts utilized in previous research endeavors.

It is worth noting that in a related experiment conducted in a preceding study [7], a similar dichotomy in model development was employed, but in centralized deep-learning settings. In the experiment, two models were trained: a Non-Private model (Traditional) and a Differentially Private model (hereafter referred to as DP Traditional). By contrasting our methodology and results with those of [7], we sought to further elucidate the advantages and trade-offs associated with the incorporation of federated and differential privacy techniques in the training of machine learning models for privacy-sensitive applications.

A. Comparison with Federated Learning Approaches

Our study presents findings from experiments conducted on a practical non-IID setting utilizing three data modalities from the MedMNIST datasets trained across 12 hospitals/clients. The results, as shown in (Table 5), indicate that the accuracy of our Non-Differentially Private model surpasses that of [16] author's federated learning system's highest performance for PathMNIST dataset. However, the privacy-accuracy trade-off for our model is approximately 5.29%, relative to their model, and approximately 8.3% for the comparison between our DP-FL and Non-DP FL models.

Our experiments on the PathMNIST dataset achieved significantly higher accuracy than [9]'s approach in both Non-DP FL and DP-FL settings. Specifically, our Non-DP FL model achieved an accuracy of 89.4%, which is 9.45% higher than the reported accuracy by [9]. Furthermore, our DP-FL model achieved an accuracy of 81.1%, which is 1.15% higher than their approach's performance. These results demonstrate the superior performance of our approach in preserving data privacy while still achieving higher accuracy on the PathMNIST dataset.

In comparison to [16]'s approach, our Non-DP FL approach outperformed their approach by approximately 3.01% on PathMNIST, achieving an accuracy of 89.4%. However, our approach achieved a slightly lower accuracy of 89.6% on OrganAMNIST, which is approximately 6.1% less than their reported accuracy of 95.70%.

B. Comparison with Differential Privacy Approaches

In contrast to [7]'s traditional machine learning approach, our federated learning experiments on the PathMNIST dataset achieved significantly higher accuracy. Specifically, we achieved an accuracy of 89.4% in the Non-DP FL setting and 81.1% in the DP FL setting. In comparison, [7] achieved an accuracy of 82.12% in the Non-DP setting and 77.91% in the DP setting. Additionally, it is important to note that the privacy budget in [7] was $ϵ$ = 7.42, while ours was 3.0, which means our approach achieves better privacy preservation while still achieving higher accuracy on the Colon Pathology dataset. Authors in [7] also conducted experiments on the Blood Cell Microscope dataset using traditional machine learning methods and achieved an accuracy of 96.13% in the Non-DP setting and 92.51% in the DP setting. In contrast, our approach applied federated learning on the same dataset and achieved an accuracy of 93.9% in the Non-DP FL setting and 85.7% in the DP FL setting, while achieving a privacy budget of $ϵ$ = 3.0. These results demonstrate that federated learning can be a viable alternative to traditional machine learning approaches for privacy-preserving data analysis.

Our results showed that the combination of Federated Averaging with differential privacy can achieve competitive classification accuracy while preserving the privacy of the individual clients' data. This finding is consistent with previous research in federated learning and differential privacy, which has demonstrated the effectiveness of these techniques in medical imagining classification problems. We also showed that our Federated Learning approach outperformed previous approaches on the PathMNIST dataset while achieving comparable results on the OrganMNIST dataset. However, there is still room for improvement in terms of privacy-accuracy trade-offs, and further research is needed to optimize the performance of Federated Learning in non-IID settings. Overall, this study contributes to the growing body of research on Federated Learning and its potential for enabling privacy-preserving data analysis in the medical domain. It is important to note that the optimal model architecture and hyperparameters can vary for different datasets, as we observed in our study. For example, while our model settings achieved higher accuracy on the PathMNIST dataset, it resulted in slightly lower performance on the OrganMNIST dataset. Therefore, researchers should carefully tune their models for each dataset to achieve optimal performance. One limitation of our study is the use of a relatively small number of participants. In real-world applications, the number of participants can be much larger. Future research should explore the scalability of the proposed approach to a larger number of participants and investigate methods to reduce the communication overhead in federated learning. Despite these limitations, our study provides insights into the potential of federated learning with differential privacy in sensitive domains, such as healthcare. The proposed approach can enable multiple hospitals to collaboratively train machine learning models on their local data while preserving the privacy of the individual patient's data. This has the potential to improve the accuracy of machine learning models and enable new applications in healthcare while ensuring data privacy and security.

Federated Learning

FedAvg

Federated averaging

Differential Privacy

Non-IID

Non-Independent and Identically Distributed Data

DPFL

Differentially Private Federated Learning

Non-DPFL

Non-Differentially Private Federated Learning

DP-SGD

Differentially Private Stochastic Gradient Descent

Funding

The authors did not receive support from any organization for the submitted work.

Ethics approval

Ethics Committee permission is not required.

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Consent

Informed consent was obtained from all participants included in this study.

Data

The data utilized in this study is openly available at the following link: https://medmnist.com. Researchers interested in replicating or further examining the results presented in this manuscript can access the data at this web address. Any additional materials or code used in the analysis are also available upon request from the corresponding author.

Authors' contributions

Mohamad HAJ FARES, as the primary researcher, was deeply involved in all aspects of this study. Mohamad conducted the experiments, meticulously prepared and curated the data, developed the coding framework, and took charge of reporting and writing the paper. Mohamad's comprehensive involvement ensured the quality and completeness of the research. Ahmet SERTBAŞ, in the role of supervisor, provided invaluable oversight and guidance throughout the project. Ahmet's expertise was instrumental in refining the overall writing of the manuscript, enhancing its clarity and coherence. Ahmet SERTBAŞ played a crucial role in shaping the study's direction and ensuring that it met the highest standards of scholarly research.

Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, oct 2016.
Andrea Acevedo, Anna Merino, Santiago Alfe´rez, A´ ngel Molina, Laura Bold´u, and Jos´e Rodellar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in Brief, 30:105474, 2020.
Mohammed Adnan, Shivam Kalra, Jesse C Cresswell, Graham W Taylor, and Hamid R Tizhoosh. Federated learning and differential privacy for medical image analysis. Scientific Reports, 12, 2022.
Patrick Bilic, Patrick Ferdinand Christ, et al. The liver tumor segmentation benchmark (lits). CoRR, abs/1901.04056, 2019.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fedavg with fine tuning: Local updates lead to representation learning, 2022.
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Serge Vaudenay, editor, Advances in Cryptology - EUROCRYPT 2006, pages 486–503, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
Florian A. H¨olzl, Daniel Rueckert, and Georgios Kaissis. Bridging the gap: Differentially private equivariant deep learning for medical image analysis, 2022.
Jakob Nikolas Kather, Johannes Krisam, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLOS Medicine, 16(1):1–22, 01 2019.
Taehyeon Kim and Se-Young Yun. Supernet training for federated image classification under system heterogeneity, 2022.
Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis. Differentially private training of residual networks with scale normalisation, 2022.
Jakub Koneˇcn´y, H. Brendan McMahan, Felix X. Yu, Peter Richt´arik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency, 2017.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2018.
Wenqi Li, Fausto Milletar`ı, Daguang Xu, Nicola Rieke, Jonny Hancox, Wentao Zhu, Maximilian Baust, Yan Cheng, S´ebastien Ourselin, M. Jorge Cardoso, and Andrew Feng. Privacy-preserving federated brain tumour segmentation. In Heung-Il Suk, Mingxia Liu, Pingkun Yan, and Chunfeng Lian, editors, Machine Learning in Medical Imaging, pages 133–141, Cham, 2019. Springer International Publishing.
Xiaoxiao Li, Yufeng Gu, Nicha Dvornek, Lawrence H. Staib, Pamela Ventola, and James S. Duncan. Multi-site fmri analysis using privacy-preserving federated learning and domain adaptation: Abide results. Medical Image Analysis, 65:101765, 2020.
Ming Y Lu, Richard J Chen, Dehan Kong, Jana Lipkova, Rajendra Singh, Drew FK Williamson, Tiffany Y Chen, and Faisal Mahmood. Federated learning for computational pathology on gigapixel whole slide images. Medical image analysis, 76:102298, 2022
Jun Luo and Shandong Wu. Adapt to adaptation: Learning personalization for cross-silo federated learning, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
Micah J Sheller, Brandon Edwards, G Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R Colen, et al. Federated learning in medicine: facilitating multiinstitutional collaborations without sharing patient data. Scientific reports, 10(1):12598, 2020.
Tao Sun, Dongsheng Li, and Bao Wang. Decentralized federated averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4289–4301, 2012.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. 10(2), jan 2019.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: Userfriendly differential privacy library in pytorch, 2021.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization, 2020.
Alexander Ziller, Dmitrii Usynin, Rickmer Braren, Marcus Makowski, Daniel Rueckert, and Georgios Kaissis. Medical imaging deep learning with differential privacy. Scientific Reports, 11(1):13524, 2021.

Table 1

Highest Performing Models of Apple and FedFOMO frameworks under the practical Non-IID settings
	PathMNIST	OrganAMNIST
APPLE	86.39%	95.70%
FedFOMO	80.19%	82.86%

Table 2

Highest Performing Models of [9] in Initial Federated Learning Setting: Comparative Analysis with FedSup Approach on the PathMNIST dataset [8].
	FedSup	E-FedSup
Static batch normalization	75.02 ± 4.95	74.30 ± 4.47
Parametric normalization	75.38 ± 4.57	74.59 ± 4.11

Table 3

Best results of [7] validation accuracies of the ResNet-9 for a model layout of 8-16-32 channels. Trained on differentially private and Non-Federated machine learning settings.
Colon Pathology		Blood Cell Microscope		Dermatoscope
Non-DP	DP	Non-DP	DP	Non-DP	DP
82.12%	77.91%	96.13%	92.51%	77.24%	73.29%

Table 4

Data samples distribution of used datasets among 12 hospitals
Hospital No.	PathMNIST	OrganAMNIST	BloodMNIST
1	6373	2779	894
2	8625	2983	1098
3	6040	2446	817
4	8958	3317	1175
5	6640	2612	904
6	8360	3152	1089
7	6206	2776	983
8	8794	2988	1011
9	6370	2811	891
10	8630	2953	1103
11	6405	2862	926
12	8595	2902	1069

Table 5

A comparison between our results and similar researches in the field of federated learning, differentially private federated learning, and non-federated differentially private settings.
Dataset	Reference Authors	Non-DPFL	DPFL	Traditional	DP Traditional
PathMNIST	Our work	89.4%	81.1%	-	-
	Luo et al.	86.39%	-	-	-
	Zhang et al.	80.19%	-	-	-
	Kim et al.	75.38 ± 4.57%	-	-	-
	Hölzl, F. A. et al	-	-	82.12%	77.91%
BloodMNIST	Our work	93.9%	85.7%	-	-
BloodMNIST	Hölzl, F. A. et al			96.13%	92.51%
OrganAMNIST	Our work	89.6%	79.4%	-	-
	Luo et al.	95.70%		-	-
	Zhang et al.	82.86%		-	-

The authors declare no competing interests.

Download PDF

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

A Differentially Private Federated Learning Application in Privacy-Preserving Medical Imaging

Status:

Version 1

Abstract

Figures

1. Introduction

2. Related Works

3. Background

4. Proposed Model Architecture

5. Experiment

6. Comparative Performance

7. Conclusion

Abbreviations

Declarations

Funding

Ethics approval

Competing interests

Consent

Data

Authors' contributions

References

Tables

Additional Declarations

Status:

Version 1