4.1 Data sets
In the experiment, sample data is obtained from the monitoring environment of industrial equipment, and one frame of image is captured from all surveillance videos in the last 7 days at 10-minute intervals. From these intercepted video frames, 5000 images related to switching status of industrial equipment, 5000 images related to indicator lights of industrial equipment and 5000 images of numbers of industrial equipment are selected as training samples.
Since some of the datasets are affected by various factors such as environment, distance, etc., the experiment will add data enhancement techniques [23] to increase the scale and diversity of test data. In this study, the use of several data enhancement techniques to a certain extent improved the training set, making the optimized YOLOv8 model perform well in the field of small target detection [24].
In this article, random cropping, random resizing, random rotation, random brightness and random contrast are applied to the training set showing the images after each enhancement. For each image in the training set two enhancements were performed with the expectation that by applying these techniques, a more refined and accurate detection model can be trained.
In the dataset, for each of the three devices, switch, indicator and digital, the sample image set contains 5000 images for a total of 15000 sample images. For training and testing purposes, and also to maintain the rigor of the experiment, the dataset is divided into training set, verification set, and testing set at 3:1:1. The training set is used to train the weight parameters of the fitted model, the verification set is used to adjust the parameters of the model to obtain the optimal model, and the test set uses the obtained optimal model for final output prediction and evaluation. That is, the number of training and test sets for each device category is 1500 images, which makes the data volume of the training set and testing set equal.
4.5 Experimental results and analysis
In this experiment, mean precision ([email protected]), precision, recall rate and F1_curve were used to evaluate the performance of the industrial fault detection model.
(1) Mean Average Precision
To verify the superiority of the YOLO-v8n industrial fault identification method proposed in this article in nine types of judgment and identification effect of indicators, switches and data dashboards, the method is compared with the YOLO-v6n, YOLO-v7n, and Faster R-CNN algorithms.And the comparison results are shown in Table 2.
Table 2
Comparison of industrial fault identification effects under different detection states
Type of testing | Number of states | mAP@0.5 /% |
YOLOv8n | YOLOv6n | YOLOv7n | Faster R-CNN |
Red_On | 150 | 93.3 | 92.4 | 91.3 | 91.8 |
Red_Off | 256 | 99.5 | 93.5 | 95.6 | 95.4 |
Green_On | 76 | 96.5 | 91.6 | 92.5 | 92.3 |
Green_Off | 210 | 96.3 | 94.7 | 94.8 | 96.2 |
Yellow_On | 34 | 92.1 | 90.3 | 91.2 | 91.2 |
Yellow_Off | 48 | 93.4 | 89.6 | 90.0 | 92.3 |
Switch_On | 124 | 95.6 | 94.5 | 93.8 | 92.5 |
Switch_Off | 160 | 97.6 | 96.3 | 94.6 | 93.6 |
Data | 160 | 96.2 | 95.8 | 96.2 | 94.2 |
All | 1218 | 95.6 | 92.1 | 93.3 | 93.6 |
As can be seen from Table 2, the mean average accuracy of this paper's detection system for industrial fault recognition ([email protected]) is 95.6%. The average accuracy of this paper's method for the recognition of indicator lights, switches, and data dashboards is 95.1%, 96.6%, and 96.2%, respectively, which is 3.5%, 2.3%, and 2.0% higher than other algorithms, respectively.
The average accuracy of Yellow_On recognition ([email protected]) is the lowest in YOLOv8's recognition of 9 types of statuses and is only 92.1%, analyzing the reason for this is that the data source's yellow indicator lights and yellow switches are on and off. The reason for this situation is that the difference between the yellow indicators light on and off in the data source is not obvious and easy to confuse, and the yellow indicator light on is wrongly identified as off, which requires a large number of data sets to be added and continuous training and optimization to improve the accuracy rate.
(2) Precision and Recall
By analyzing the normalized confusion matrix of this experiment, the precision and recall rate of the computing class are calculated. Using confusion matrix normalization, it is easier to observe the classification accuracy and error of the model across different categories.
The normalized confusion matrix in Fig. 11 demonstrates the relationship between the predictions of the different industrial fault state identification models and the actual labels. Their predictions are shown in Table 3.The prediction accuracies of the nine categories of industrial fault states range from 0.82 to 1.00, from which it can be inferred that the prediction accuracy cases of Red_On, Red_Off, Green_On, and Switch_Off create an imbalance with the prediction accuracies of Yellow_On and Yellow_Off.The imbalance of the normalized confusion matrix may cause the model to perform inconsistently on the evaluation metrics (e.g., precision, recall, etc.), so this experiment adds samples for categories with a small number of samples to balance the sample distribution.
Table 3
Confusion Matrix Data Table
Type of testing | Predictive accuracy |
Red_On | 0.82 |
Red_Off | 0.83 |
Green_On | 0.83 |
Green_Off | 0.97 |
Yellow_On | 1.00 |
Yellow_Off | 1.00 |
Switch_On | 0.95 |
Switch_Off | 0.87 |
Data | 0.97 |
(3) F1-Score Analysis |
In Figure. 12, the F1-Score curve shows the change of F1-Score of the model under different prediction thresholds, and it can be seen that the F1 of all classes reaches 0.93 at a confidence level of 0.513, which is enough to see the excellent performance of this model.
Four additional groups of experiments were added in this section to analyze the results of different models. The same training parameters were used in each group of experiments. The influence of different models on the detection performance was shown in Table 4. Among them, the detection accuracy rate of the improved YOLOv8n algorithm is 96.84%, the recall rate is 98.73%, and the results of F1-Score was 97.81%. In contrast, the performance of other algorithms is slightly lower, e.g., YOLOv6n has an F1-Score of 85.28%, YOLOv7n has an F1-Score of 89.82%, and Faster R-CNN has an F1-Score of 91.97%. From these data, it can be concluded that the YOLOv8n algorithm, which is embedded with the channel attention mechanism of SeNet, which is superior to other algorithms in terms of comprehensive performance. Overall, YOLOv8n performed well in terms of detection accuracy and recall rate, and the comprehensive evaluation index F1-Score reached the highest level, indicating that it achieved a good balance between accuracy and recall. Therefore, it can be concluded that the YOLOv8n algorithm embedded with the SeNet channel attention mechanism has higher accuracy and stability in the detection task.
Table 4
Comparison of performance evaluation indicators for different models
model | Detection accuracy/% | Recall rate/% | F1 |
YOLOv8n | 96.84% | 98.73% | 97.81% |
YOLOv6n | 89.27% | 95.30% | 85.28% |
YOLOv7n | 91.96% | 90.36% | 89.82% |
Faster R-CNN | 92.36% | 92.03% | 91.97% |
(4) Loss plot
When the improved detection model proposed in this paper is applied to the data set, the detection results are shown in Fig. 13. As you can see from the figure, the model has a high convergence and eventually tends to stabilize, indicating that the equipment fault detection model does not occur over-fitting or under-fitting phenomenon, and the results etected by the model are accurate and reliable.
(5) Visualization of industrial fault detection results
This section will present the results of different state models in the clinical problems of industrial faults to better illustrate the analytical effects of different methods. In order to take into account the generalization and strength of the models, images of industrial disturbances in different environments are selected for this work.
As shown in Fig. 14, the red fault indicator light is lit up in the picture and the green fault indicator light is detected, while the yellow indicator light does not have good enough test results after measurement, found that because the yellow fault lit dataset is too small and lit or not due to the angle, light, lens with or without obstruction of the problem, the gap is not very obvious, so it causes the yellow indicator detection results are not enough, and the subsequent experiment will focus on optimizing the model.
As shown in Figure. 15, the closures of the switches in the picture are detected without any error or omission.
As shown in Fig. 16, the numbers on the electronic display were detected one by one and the threshold detection condition performed well.