Accurate understanding of spatial relationships between humans and objects is a key for recognizing the Human-Object Interaction (HOI).Due to its inherent simplicity and interpretability, Intersection over Union (IoU) has established itself as the predominant metric for expressing spatial relationships.This paper presents a simple yet effective attention mechanism that leverages the IoU metric.Specifically, we introduce a Chunk Block Attention (CBA) module that computes spatial-aware attention weights of image slices extracted through the IoU-guided Human-Object association method.The CBA uniquely integrates self-attention and cross-attention to capture internal and external relationships within slices.Through extensive experiments on various HOI models, we validate the effectiveness of the proposed method.Experimental results show consistent improvements in qualitative measure and impressive interpretability by visualizing the learned attention weights.Furthermore, we provide detailed analysis and visualizations of the attention weights learned by the CBA module.This reveals insights into how the model focuses on relevant spatial and semantic relationships for recognizing complex human-object interactions.