We propose an IR-YOLOv5 to address the limitations of general target detection approaches in infrared scenes. Specifically, IR-YOLOv5 takes into account common issues present in infrared images, such as low resolution, high noise, low contrast, and small targets. To achieve an efficient interaction of information between different channels and obtain rich semantic information of targets in infrared images, the feature extraction network leverages the attention mechanism. Additionally, the detection scheme of YOLOv5 is enhanced, and a detection head designed for small and weak targets is incorporated to improve the detection probability of these targets in infrared images. Furthermore, the method proposes an efficient PANet target recognition network by designing the Detection Block module to effectively reduce the parameter size and ensure real-time operations of the detector while enhancing the detection accuracy. The proposed method is evaluated on the FLIR infrared autonomous driving dataset and outperforms yolov5 with a 4% improvement in mean Average Precision (mAP) and a reduced model parameter size.