Infrared imaging technology captures the thermal radiation characteristics of targets, enabling object monitoring and filtering redundant road information in complex road scenarios. This paper proposes a lightweight infrared small object detection algorithm GML-YOLO to address issues such as low accuracy, large parameter volume, dependency on high-performance GPU resources, and slow detection speed in infrared pedestrian and vehicle object detection models. Firstly, we designed a lightweight backbone network, G_HGNet, to improve feature extraction efficiency, enabling accurate and real-time feature extraction. Additionally, we incorporated adaptive downsampling and attention mechanisms in the network fusion part, replacing the simple concatenation used in traditional detectors. This design effectively integrates shallow and deep information. Subsequently, the WIOUv3 loss function was employed to enhance model convergence speed and reduce losses, thereby increasing model robustness. Finally, comparative experiments were conducted on our dataset as well as the public FLIR and Pascal VOC datasets. The results demonstrate that GML-YOLO achieves a mean Average Precision ([email protected]) of 89.7% on our dataset (ISTD), 86.5% on the FLIR dataset, and 82.0% on the Pascal VOC dataset. Additionally, computational load and parameter volume were reduced by 20% and 27%, respectively. The improved GML-YOLO algorithm outperforms YOLOv3, YOLOv5, YOLOv6, YOLOv8s, and YOLOv8n, validating the feasibility of our proposed algorithm.