Spatial transcriptomics produces high-dimensional gene expression measurements while retaining their spatial context within tissues. Obtaining a biologically meaningful low dimensional presentation of the data is a crucial step toward data interpretation and downstream analysis. Here, we present STAMP, an interpretable spatially aware dimension reduction method built on a deep generative model that returns low dimensional topics of biologically relevant spatial domains and associated gene modules. STAMP recovered the anatomical structures of the mouse hippocampus and olfactory bulb with known gene markers highly ranked in the respective gene modules. In a lung cancer sample, it delineated cell states with supporting gene markers at a higher resolution than the original annotation and uncovered a topic of cancer associated fibroblasts. Finally, we expanded STAMP to account for batch effects and identify spatiotemporal patterns across chronologically consecutive samples of mouse embryo development. STAMP is implemented in Python and downloadable at https://github.com/JinmiaoChenLab/scTM.