During its chronic degenerative course, Alzheimer's Disease takes a huge toll on the cognitive abilities of patients. Assessment of current and future cognition is an integral component of a diagnosis of dementia, and therefore an important clinical and scientific goal. Unfortunately, subjective, time-consuming and operator-sensitive clinical surveys or neuropyschiatric batteries remain the only viable method of assessing cognition. Given that MRI is the most prevalent, cost-effective, and clinically important imaging modality, it may be considered a suitable predictor of cognition. Yet, it has hitherto proved very challenging to predict one from the other. Here we propose that an image-based Deep Learning model can be custom-built to achieve this goal. We designed a novel multi-task UNet model to predict the subjects’ current and future cognition (via ADAS-Cog scores), taking as input baseline T1-weighted MRI and demographic risk factors. The key innovation in the model is that it seeks to solve two adjacent but relevant tasks: image segmentation into tissue types; and prediction of cognition. The first task gives a high-accuracy segmentation of the brain, comparable to other cutting edge segmentation methods. The features trained from the segmentation task are used in the cognition task. This combination is shown to be far superior to stand-alone single-shot models of cognition. We achieved excellent accuracy in both baseline and time-series forecast of ADAS-Cog scores. Through further feature map analysis made on the receptive fields, we were able to impart much-needed interpretability to the model, critical for real-world clinical practice. This study constitutes the best-reported performance of any comparable approach, and opens the door towards machine-based tracking of AD progression.