In recent years, deep learning convolutional networks have been gradually applied to metal surface inspection. Steel plate defect detection helps improve the efficiency of steel plate production by locating and classifying surface defects on steel plates. The related methods are challenging to balance the accuracy of localization and classification. There are scenarios in the industrial inspection field where the data set is relatively small, and the distribution of defects is significantly irregular; this paper extends the traditional image detection network by introducing an attention mechanism. At the same time, to meet the requirement of higher real-time performance in industrial inspection, the lightweight Ghost network is introduced to replace the original backbone network CSPDarknet53 in YOLOV4 to improve the detection speed of the network. This paper constructs the high accuracy and low latency defect detection algorithm Ghost-CBAM-YOLOV4 Network (GCB-Net). The GCB-Net model is based on the improved YOLOV4 target detection network. The CBAM module and Ghost lightweight network are introduced to complete the prediction of defect categories without reducing the classification and detection accuracy. The desired experimental results are achieved, which is essential for detecting defects in steel plates and improving product quality.