Growing evidence supports the importance of quantifying tumor-immune cell interactions in the tumor microenvironment to enable precision cancer therapy. However, most existing methods rely solely upon immune cell density or nearest neighbor-type analyses and fail to fully characterize spatial heterogeneity. Herein, we describe a computational algorithm, termed Tumor-Immune Partitioning and Clustering (TIPC), that jointly measures immune cell partitioning between tumor epithelial and stromal areas and immune cell clustering versus dispersion. As proof of principle, we apply TIPC to two large colorectal cancer cohorts. TIPC identifies tumor subtypes with unique interaction signatures between tumor cells and T cells that harbor prognostic significance and are associated with distinct tumor molecular features. We extend our findings by applying TIPC to additional immune cell types identified using morphology and supervised machine learning. Spatial heterogeneity quantification and novel tumor subtype identification by TIPC may inform precision cancer immunotherapy and deepen our understanding of tumor immunobiology.