In-vitro fertilization (IVF) offers a solution for couples facing infertility challenges. However, the success of IVF, particularly in achieving live-birth outcomes, heavily depends on embryologists to conduct morphological assessments of fertilized embryos, a process that is both time-consuming and labor-intensive. While artificial intelligence (AI) has gained recognition for its potential to automate embryo selection, the application of deep learning (DL) is constrained by privacy concerns associated with the requirement for centralized training on extensive datasets. In this paper, we have developed a distributed DL system, termed ‘FedEmbryo’, tailored for personalized embryo selection while preserving data privacy. Within FedEmbryo, we introduce a Federated Task-Adaptive Learning (FTAL) approach with a hierarchical dynamic weighting adaption (HDWA) mechanism. This approach first uniquely integrates multi-task learning (MTL) with federated learning (FL) by proposing a unified multitask client architecture that consists of shared layers and task-specific layers to accommodate the single- and multi-task learning within each client. Furthermore, the HDWA mechanism mitigates the skewed model performance attributed to data heterogeneity from FTAL. It considers the learning feedback (loss ratios) from the tasks and clients, facilitating a dynamic balance to task attention and client aggregation. Finally, we refine FedEmbryo to address critical clinical scenarios in the IVF processes, including morphology evaluation and live-birth outcomes. We operate each morphological metric as an individual task within the client's model to perform FTAL in morphology evaluation and incorporate embryo images with corresponding clinical factors as multimodal inputs to predict live-birth outcomes. Experimental results indicate that FedEmbryo outperforms both locally trained models and state-of-the-art (SOTA) FL methods. Our research marks a significant advancement in the development of AI in IVF treatments.