In the tool processing industry, the problem of tool tooth breakage often occurs, so the location, detection and identification of the damaged tooth are very important for the high-precision machining of equipment with sawtooth. Aiming at the problem of low accuracy of existing algorithms for small targets, this paper designs an improved YOLOv5 damaged tooth detection algorithm to improve the detection accuracy. In this paper, we first make a data set about damaged sawteeth. Then add a small target detection layer to improve the feature extraction ability of the model. Then the RCSOSA backbone network is used to replace the YOLOv5 backbone network to further enhance the feature extraction capability of the model. Finally, ECA attention mechanism is added to enhance the channel features of the feature map. The experimental results show that the accuracy of the improved YOLOv5 algorithm reaches 92.8%, and the recall rate and average accuracy are increased by 9.9% and 7.6% compared with the original YOLOv5.