Safety helmets can effectively prevent miners from accidental head injuries, reduce accident rates during coal mine, and safety helmet-wearing detection is of great significance to the safety management of coal mine, and is an important component of video surveillance systems. This paper proposes a new safety helmet-wearing detection algorithm called Depthwise Separable Multi-factor YOLO (DSM-YOLO). First, the algorithm uses Depthwise Separable Convolution(DSConv) to reduce the number of parameters, deepens the extraction of deep feature information, speed up feature transfer in the model, and improves the speed of helmet detection. Second, in order to make a better match between the predicted box of the target and the corresponding ground truth box, a multi-factor loss function is introduced, and the multi-factor loss function simultaneously takes into account the intersection ratio loss, distance loss, aspect ratio loss, and angle perception loss, which improves the accuracy of helmet detection. The results of the experiments showed that the inference speed of the algorithm is 62.5 Frames/s, which is 19.0 Frames/s faster than the YOLOv7 algorithm; the average precision is 95.1%, which is 3.4% higher than the YOLOv7 algorithm, meeting the real-time detection requirements for safety helmet wearing in coal mine.