The immune checkpoint inhibitors have demonstrated promising clinical efficacy across various tumor types, yet the percentage of patients who benefit from them remains low. The bindings between tumor antigens and HLA-I/TCR molecules determine the antigen presentation and T-cell activation, thereby playing important role in the immunotherapy response. Some computational methods have been developed to predict peptide-HLA (pHLA) or peptide-TCR (pTCR) binding specificity, but they focus solely on one task at a time. In this paper, we propose UnifyImmun, a unified cross-attention transformer model designed to simultaneously predict the bindings of peptides to both receptors, thereby providing more comprehensive evaluation of antigen immunogenicity. We devise a two-phase progressive training strategy that enables these two tasks to mutually reinforce each other, by compelling the encoders to extract more expressive features. We also incorporate virtual adversarial training to enhance the model generalizability. Compared to over ten previously published methods for predicting pHLA and pTCR bindings, our method demonstrates better performance in both tasks on multiple independent and external test sets. Notably, on a large-scale COVID-19 pTCR binding test set, our method outperformed the current state-of-the-art methods by at least 9%. The validation experiments on three clinical cohorts confirm that our approach effectively predicts immunotherapy response and clinical outcomes. Furthermore, the cross-attention scores and integrated gradients reveal the amino-acid sites critical for peptide binding to receptors. In essence, our approach marks a significant step towards comprehensive evaluation of antigen immunogenicity.