Sepsis is the major cause of death in ICUs and amounts to a large portion of medical expenses. Earlier prediction of sepsis can assist in reducing inpatient mortality and healthcare costs. Despite predictive capabilities of the state-of-the-art approaches, deep models still lack the transparency that provides some sort of reasoning for their predictions. In this work, we studied the predictive performance of an attention-based recurrent neural network and its ability to provide interpretability for the relative contribution of each medical factor to the final prediction. We also qualitatively assess the explainability of the proposed model. This approach sheds light on the opacity issue of deep predictive models. Moreover, it enables the clinical staff to make more reliable decisions based on the status of the patient in ICU thus decreasing the sepsis inpatient mortality.