Intratumor heterogeneity in breast cancer associates with poor patient outcome. Tissue fibrosis and stromal stiffening accompany breast cancer development and associate with the aggressiveness of human breast cancer subtypes. Whether human breast cancers demonstrate stiffness heterogeneity, and if this is linked to breast tumor aggression remains unclear. To answer these questions, we developed a spatial method to measure the stiffness heterogeneity in human breast tumor tissues that also quantifies the local stromal stiffness each cell experiences and permits correlation with biomarkers of tumor aggression. Here, we present Spatially Transformed Inferential Force Maps (STIFMaps) to predict the elasticity across whole tissue sections with micron-resolution. The method exploits computer vision to precisely automate AFM indentation and then uses a trained convolutional neural network to predict matrix elasticity using collagen morphological features and ground truth AFM data. Because STIFMaps is compatible with biomarker staining we used the approach to register high-elasticity regions within sections of human breast tumors with markers of mechanical activation and an epithelial to mesenchymal transition (EMT) that associated with tumor aggression. The findings herein highlight the utility of STIFMaps for assessing the mechanical heterogeneity of human breast tissues across length scales from single cells to whole tissues. The method also reveals, for the first time, a direct association between stromal stiffness and EMT, thereby implicating stromal stiffness as a driver of human breast cancer aggression.