Accurate and robust prediction of patient-specific responses to drug treatments is critical for drug development and personalized medicine. However, patient data are often too scarce to train a generalized machine learning model. Although many methods have been developed to utilize cell line data, few of them can reliably predict individual patient clinical responses to new drugs due to data distribution shift and confounding factors. We have developed a novel Context-aware Deconfounding Autoencoder (CODE-AE) that can extract intrinsic biological signals masked by context-specific patterns and confounding factors. Extensive comparative studies demonstrated that CODE-AE effectively alleviated the out-of-distribution problem for the model generalization, significantly improved accuracy and robustness over state-of-the-art methods in predicting patient-specific in vivo drug responses purely from in vitro screens. Using CODE-AE, we screened 59 drugs for 9,808 cancer patients. Our results are consistent with existing clinical observations, suggesting the potential of CODE-AE in developing personalized anti-cancer therapies and drug-response biomarkers.