Deep learning using convolutional neural networks has shown great promise in analyzing neuroimaging data. Identification of Alzheimer's Disease patients from healthy individuals using structural magnetic resonance data is one of the clinical problems that has been widely explored by employing convolutional neural networks and producing very high classification accuracies. However, in most studies, the results were not supported by explainability tools. In this study, an interpretable convolutional neural network model derived from pre-trained VGG-16 is proposed to classify Alzheimer’s Disease patients vs. healthy subjects using MRI data obtained from an Open Access Series of Imaging studies (OASIS) dataset. The model was trained and validated based on a 5-fold cross-validation loop and produced a classification accuracy of 71.62% on the validation set. Moreover, we incorporated four CNN visualization techniques that highlight important brain regions used by the model to identify AD patients: saliency map, gradient class activation mapping, occlusion mapping, and heatmap generated by Shapley additive explanation (SHAP) method. The potential of these explainability tools in identifying biased models that produce inflated erroneous results is also investigated, and the resulting heatmaps were able to indicate a bias in the model’s training procedure.