Breast cancer is the second largest cause of female cancer death and one of the most hazardous diseases that leads to a higher mortality rate. Breast cancer is initialized with the malignant stage, where the abnormal growth of cancerous lumps is initiated from the breast cells. Periodic clinical checks and self-tests assist in early identification and thus progress the survival rates considerably. One of the eminent medical approaches is breast cancer recognition, which offers scientists and researchers huge complications. Breast cancer detection at an early stage permits the patients to receive suitable treatment, which increases the chances of survival. Thus, this paper utilizes a new form of artificial intelligence training called Federated Learning (FL), especially for breast cancer detection, the most eminent technique in the last few years. FL permits individual hospitals to benefit from the rich datasets of multiple non-affiliated hospitals without centralizing the data in one place. Hence, FL utilizes numerous collaborators for building a strong deep-learning model using a large dataset. In this paper, a hybridization of this type of training with a meta-heuristic and deep learning is aimed to be proposed for breast cancer diagnosis. This model encloses diverse steps that include (a) image collection, (b) feature extraction, and (c) classification phase. Initially, the mammogram images related to breast cancer are collected with the concept of FL from the affected individuals. The federated learning helps in reducing the processing time and ensures better performance of the proposed model. The obtained images are considered for the feature extraction phase. The Densenet architecture is used to extract the features used in the classification phase with the help of Enhanced Recurrent Neural Networks (E-RNN) for detecting breast cancer. Here, the performance is enhanced by tuning the certain parameter in the RNN network using a hybrid optimization algorithm called Hybrid Dragon-Rider Optimization (HDRO) with Dragonfly Algorithm (DA) and Red deer algorithm (RDA) to achieve accurate classification results. The experimental results demonstrate the effectiveness of the suggested breast cancer diagnosis model compared with conventional approaches using diverse quantitative measures.