This study was conducted at Sichuan Agricultural University. Sichuan Agricultural University has built a teaching and research park with a functional area for teaching and research on livestock and poultry. Data were manually gathered using a camera. The sheep were not touched during the experiment by the experimenter or the data collection equipment, and there was no stress response in the sheep. The method used in this paper enables noncontact animal behavior recognition and improves animal welfare compared to traditional manual detection and the use of electronic ear-tagging devices.
3.2. Dataset production and data processing
We used the Python code to intercept the original video to obtain the images of the dataset required for the experiment. To enhance the performance effect of the subsequent model, we manually screened and removed the images that were blurred because the sheep were captured and the sheep were heavily obscured, and 955 images were obtained. Finally, we manually labelled the object detection dataset, pose estimation dataset, and behavior recognition dataset[datasets]. Figure 1 shows some images of the dataset.
Figure 1. Some images of the dataset.
3.2.1. Object detection dataset
This dataset was created using the PASCAL VOC dataset format 31. The object detection dataset was annotated using the free and open-source LabelImg image annotation tool. Under the supervision of experts, four people annotated the images. As shown in Fig. 2, during the annotation process, we ensured that the annotated horizontal rectangular box framed the target exactly and completely, minimizing the framing of background redundant pixels. Finally, we randomly divided the images into three sets: a training set (733 images), a test set (96 images), and a validation set (86 images).
Figure 2. Diagram of object detection data labeling.
3.2.2. Pose estimation dataset
For the pose estimation dataset, we used the COCO dataset format 32. Labelme is an image annotation tool developed by the Computer Science and Artificial Intelligence Laboratory (CSAIL) at the Massachusetts Institute of Technology (MIT) that is capable of labeling key points for the task. We used Labelme to label a total of 400 images for subsequent experiments. As shown in Fig. 3, we selected a total of 18 key points (marked as 0, 1, 2, ..., 17) representing the mouth, nose, right eye, left eye, back, hips, right front heel, right front knee, right front hoof, left front heel, left front knee, left front hoof, right hind heel, right hind knee, right hind hoof, left hind heel, left hind knee, and left hind hoof. During the annotation process, the key points of each sheep are labeled according to the corresponding markers, and the missing and invisible key points are uniformly placed in the upper left corner of the image.
Figure 3. Schematic diagram of key points for pose estimation.
3.2.3. Behavior recognition dataset
On the basis of the pose estimation dataset, we made datasets for sheep standing and lying down. In Fig. 4, the left side is a schematic diagram of the key points of the standing posture, and the right side is a schematic diagram of the key points of the lying posture. Since there were fewer images in the lying posture, we expanded the dataset by data enhancement methods such as left and right flipping and contrast enhancement and add some blank images (images of sheep lying on their stomachs) as distractors. Finally, we obtained 400 standing posture datasets and 198 lying posture datasets.
Figure 4. Schematic diagram of key points of sheep standing and lying posture.
3.3. Methods
In this paper, basic research on sheep class key point identification was carried out by object detection and pose estimation, and the specific flow chart is shown in Fig. 5. In this section, we describe the specific methods and network models used.
Figure 5. Experimental Flowchart.
3.3.1. Attention Mechanism
The attention mechanism was proposed in 2015 by three pioneers, Yoshua Bengio, Kyunghyun Cho and Dzmitry Bahdanau, and was first applied to machine translation and later applied to computer vision. The attention mechanism can be viewed as a dynamic weight adjustment process based on the input image features 33. In neural networks, the attention mechanism is a resource allocation scheme that allocates computational resources to more important tasks with limited computational power while solving the information overload problem, which can effectively simplify the model and accelerate the computation speed. The common attention mechanisms mainly include the channel attention mechanism, spatial attention mechanism and hybrid domain attention mechanism. The channel attention mechanism concentrates on learning the importance between channels, the spatial attention mechanism focuses attention mainly on the focused regions in space, and the hybrid domain attention mechanism does not focus attention only on space or channels (as the first two do) but instead focuses on both feature space and feature channels. As a result, this paper will attempt to incorporate attention mechanisms into the YOLOv7-tiny network model to improve the model's efficiency and accuracy in perceptual information processing. In this paper, four attention mechanisms are studied, namely, CBAM, GAM, SimAM, and CA. Finally, we decided to add GAM to the YOLOv7-tiny network architecture.
CBAM
CBAM is a lightweight hybrid domain attention mechanism that allows attention in both channel and spatial dimensions and is a simple and effective attention module for feed-forward convolutional neural networks. If given an intermediate feature map, the CBAM module sequentially infers the feature information of the attention map along two separate dimensions, channel and space, and then multiplies both feature information by the input feature map for adaptive feature modification. CBAM is lightweight and general-purpose, so it can be seamlessly integrated into any CNN architecture with negligible overhead and can be used with the base CNN for end-to-end training 34.
Figure 6. The structure of CBAM.
Figure 6 shows the structure of the CBAM module, which is mainly divided into the channel attention module and the spatial attention module. The CBAM module first inputs the input feature map to the channel attention module to generate the corresponding output feature map and then inputs the feature map to the spatial attention module to generate the final feature map, whose equations are shown in Equations (1) and (2).
$${\text{F}}^{{\prime }}={\text{M}}_{\text{C}}\left(\text{F}\right) ⨂ \text{F}$$
1
$${\text{F}}^{"}={\text{M}}_{\text{S}}\left({\text{F}}^{{\prime }}\right) {⨂ \text{F}}^{{\prime }},$$
2
where the input features are F∈RC*H*W, the one-dimensional convolution MC∈RC*1*1 for the channel attention module, and the two-dimensional convolution MS∈R1*H*W for the spatial attention module.
GAM
The global attention mechanism (GAM) is a global attention mechanism proposed in 2021 to improve the performance of deep neural networks by reducing information dispersion and amplifying the global interaction representation 35. GAM focuses on the dimensions across spatial channels and is used to amplify the "global" cross-dimensional interactions by retaining information. Figure 7 shows the structure of GAM, which is similar to the structure of CBAM but has a different treatment of channel attention and spatial attention. In this paper, we finally add this attention mechanism to the YOLOv7-tiny network structure.
Figure 7. Overview of GAM.
The channel attention module of CBAM first performs maximum pooling and average pooling on the input feature map, then it is processed separately by MLP, and finally the output feature map is obtained after sigmoid activation. The channel attention module of GAM first performs dimensional conversion on the input feature map, and the dimensionally converted feature map is input to MLP, then converted to the original dimension, and finally sigmoid processing output the feature image. The spatial attention module of CBAM performs maximum pooling and average pooling on the feature map and then superimposes them together for convolution and finally outputs the final feature map after the sigmoid activation function, while the spatial attention module of GAM first reduces the number of channels by convolution with a convolution kernel size of 7 to reduce the computation and then increases the number of channels after a convolution with a convolution kernel of 7, thus keeping the number of channels the same as the result that is finally output by sigmoid.
SimAM
SimAM is a lightweight 3D attention module that uses an energy function to compute attention weights without increasing the original network parameters. Figure 8 shows a schematic diagram of the 3D attention module. An optimized energy function is proposed in SimAM to discover the importance of each neuron. The solution of this function can be implemented in less than ten lines of code, and the module is able to avoid excessive structural tuning work and is more flexible 36. Among others, the formula for the capability function is Eq. (3), and Equations (4)-(7) are the calculation formulas for the unknowns in Eq. (3.
Figure 8. Structure of the 3D attention module.
\({\text{e}}_{\text{t}}({\text{w}}_{\text{t}} , {\text{b}}_{\text{t}} ,{\text{t} ,\text{x}}_{\text{i}})=\frac{1}{\text{M}-1}\sum _{\text{i}=1}^{\text{M}-1}{(-1-({\text{w}}_{\text{t}}{\text{x}}_{\text{i}} + {\text{b}}_{\text{t}}\left)\right)}^{2}\) \(+{(1-({\text{w}}_{\text{t}}\text{t} + {\text{b}}_{\text{t}}\left)\right)}^{2}{\lambda }{{\text{w}}_{\text{t}}}^{2}\) | (3) |
\({\text{w}}_{\text{t}}=-\frac{2(\text{t}-{{\mu }}_{\text{t}})}{{(\text{t}-{{\mu }}_{\text{t}})}^{2}+2{{{\sigma }}_{\text{t}}}^{2}+2{\lambda }}\) | (4) |
\({\text{b}}_{\text{t}}=-\frac{1}{2}(\text{t}+{{\mu }}_{\text{t}}){\text{w}}_{\text{t}}\) | (5) |
\({\mu }=\frac{1}{\text{M}-1}{\sum }_{\text{i}=1}^{\text{M}-1}{\text{x}}_{\text{i}}\) | (6) |
\({{{\sigma }}_{\text{t}}}^{2}=\frac{1}{\text{M}-1}{\sum }_{\text{i}=1}^{\text{M}-1}{{(\text{x}}_{\text{i}}-{{\mu }}_{\text{t}})}^{2}\) | (7) |
CA
CA is a new attention mechanism for mobile networks proposed in 2021. CA considers the location information between channels to enable mobile networks to pay attention over a large area by embedding location information into channel attention. At the same time, CA is able to avoid incurring significant computational overhead by using two 1D global pooling operations to aggregate the input features along the vertical and horizontal directions into two separate directional feature maps, which are encoded into two attention maps, each capturing the long-range correlation of the input feature maps along one spatial direction so that the location information is stored in the generated attention map37. This attention module takes into account channel information as well as direction-dependent location information and is flexible and lightweight enough to be simply inserted into the core module of a lightweight network, whose architecture is shown in Fig. 9.
Figure 9: The structure of CA.
3.3.2. Soft-NMS
Traditional NMS is a crucial postprocessing step in the object detection algorithm that is used to eliminate duplicate boxes, thereby reducing false detections, and it uses its iterative form to continuously perform IoU operations with other boxes with the highest score and filter those with larger IoUs. However, the biggest problem in the traditional NMS algorithm is that it forces the fraction of adjacent detection frames to zero (i.e., the detection frame whose overlap is greater than the overlap threshold Nt is removed). In this case, if a real object is present in the overlapping region, it will cause the detection of that object to fail, thus reducing the average detection rate of the algorithm. Soft-NMS was proposed in 2017 38, and its design was informed by the lessons of NMS; instead of simply removing the detection frames with an IoU greater than the threshold, the score is reduced during algorithm execution, which has two forms, linear weighting and Gaussian weighting, as shown in Equations (8) and (9), respectively.
$${\text{s}}_{\text{i}}=\left\{\begin{array}{c}{\text{s}}_{\text{i}}, iou(M, {\text{b}}_{\text{i}})<{\text{N}}_{\text{t}}\\ {\text{s}}_{\text{i}}(1-iou(M, {\text{b}}_{\text{i}}\left)\right){\text{s}}_{\text{i}}, iou(M, {\text{b}}_{\text{i}})\ge {\text{N}}_{\text{t}}\end{array}\right.$$
8
$${\text{s}}_{\text{i}} = {\text{s}}_{\text{i}}{\text{e}}^{-\frac{{ \text{i}\text{o}\text{u}(\text{M}, {\text{b}}_{\text{i}})}^{2}}{{{\sigma }}^{2}}},\forall {\text{b}}_{\text{i}}\ne \text{D},$$
9
where B denotes the set of initial boxes, bi is the element in set B, S contains corresponding detection scores, si is the element in set S, Nt is the threshold value of NMS, and D is denoted as the set of final boxes.
We apply this technique to the pose estimation network to improve the accuracy of detecting targets in pose estimation.
3.3.3. GAM-YOLOV7-tiny
In this paper, we use YOLOv7-tiny as the base network model for sheep body object detection for improvement. YOLOv7-tiny is an edge GPU-optimized version of YOLOv7, with a smaller and faster model that provides more advantages when deployed on edge devices such as Raspberry Pi. To focus on more information, YOLOv7-tiny has multiple heads in addition to its single head. The lead head is the title given to the one of them who is in charge of the final product. While the middle layer, known as the auxiliary head, is used to support training, it is updated with the aid of auxiliary loss, enabling deep supervision and improved model learning. The network structure of YOLOv7-tiny is shown in Fig. 10.
Figure 10. YOLOv7-tiny network structure.
Based on YOLOv7-tiny, we made model improvements to achieve accurate and fast recognition of sheep and proposed a lightweight object detection algorithm, GAM-YOLOv7-tiny. As shown in Fig. 10, our main improvements are focused on the head section, as follows:
-
1. We compress the network width to reduce the parameters and computation of the model.
-
2. We add the GAM attention mechanism to ensure the improvement of accuracy with the reduction of model parameters and computation.
-
3. We replace the three convolutional layers before the output with RepConv layers to reduce the complexity of the model while maintaining the predictive performance of the network.
-
4. We use a bilinear interpolation algorithm to avoid the problem that the nearest neighbor algorithm destroys the original image pixel gradient relationship.
Figure 11. GAM-YOLOv7-tiny network structure.
Figure 11 shows the structure of the GAM-YOLOv7-tiny model proposed in this paper, where the meaning of each model is as follows (Fig. 12).
Figure 12. Module details for GAM-YOLOv7-tiny.
CBL module: This module consists of three layers, namely, the convolution layer, the batch normalization layer and the LeakyReLU activation function layer. The YOLOv7-tiny structure has three types, each of which differs from the convolution layers in the main: 1x1 convolution with step size 1, 3x3 convolution with step size 1, and 3x3 convolution with step size 2. The 1x1 convolution is used to change the number of channels, the 3x3 convolution with step 1 is used to extract features, and the 3x3 convolution with step 2 is used for downsampling.
MP module and SP module: These modules are both composed of a max pooling layer, and the difference between them is that the output channels are different. The step size of the SP module is 1, and the step size of the MP module is 2, so the number of SP output channels of is 2 times the number of MP output channels.
ELAN module: This module consists of several CBL modules and is an efficient network structure that enables the network to learn more features with greater robustness by controlling the shortest and longest gradient paths.
UpSample module: This is an upsampling module, and the main algorithm used is nearest neighbor interpolation. This algorithm is an extension of linear interpolation, which performs one linear interpolation in each of the two directions. Unlike the nearest neighbor interpolation algorithm, which finds the nearest point and assigns it directly, the bilinear interpolation algorithm finds the nearest four points and then assigns the value to that point, and the main idea is to perform a linear interpolation in the x and y directions.
RepConv module: This module is a RepConv structure without identity connection, which has two modules, one for training and one for inference.
3.3.4. Soft NMS-AlexNet
For sheep body pose estimation, the network model we used is AlexNet, which has eight layers, with the first five convolutional layers and the last three fully connected layers, and its structure is shown in Fig. 13.
Figure 13. Structure of AlexNet network.
AlexNet has the following major technological innovations:
-
It uses the ReLU activation function, which effectively solves the gradient dispersion problem of the sigmoid function when the network is deep.
-
The dropout method, which allows a specific neuron's activation value to stop functioning with a specific probability p during forward propagation, is used to make the model more general and less reliant on specific local features.
-
The use of maximum pooling avoids the blurring effect of average pooling. Moreover, the step size of the maximum pooling is smaller than the size of the convolution kernel so that there is overlap and coverage between the outputs of the pooling layers, which improves the feature richness.
-
The LRN layer is proposed to create a competition mechanism for the activity of local neurons so that the values in which the response is relatively large become relatively larger and suppress other neurons with less feedback, enhancing the generalization ability of the model.
Based on the existing knowledge, we improve the AlexNet model and propose a lightweight pose estimation network model called Soft-NMS-AlexNet. The FPS of AlexNet is high, but the accuracy is low; therefore, our study focuses on accuracy improvement. Compared with the original network, our model has the following features:
-
1. The inclusion of Soft-NMS reduces the false detection rate of the model and improves the accuracy.
-
2. A combination of activation functions is used, and different activation functions are used for the convolutional and fully connected layers, which improves model detection.
-
Experiment and result
4.1 Object detection
4.1.1 Experimental environment and parameters
The experimental framework we used for the object detection experiment of the sheep body is PyTorch, the experimental environment is Windows 10, and the parameters of the experiment are shown in Table 1:
Table 1
Object detection experimental parameters.
Parameter | Detail |
epoch | 200 |
image size | 640*640 |
Optimizer | SGD |
momentum | 0.9 |
4.1.2 Multiobject detection and evaluation indicators
For the evaluation of object detection algorithms, we first need to know four metrics: true positives (TPs), false-positives (FPs), true negatives (TNs), and false-negatives (FNs), whose meanings are shown in Table 2. For this study, TPs indicate the number of sheep identified as true sheep, FPs indicate the number of other failures incorrectly identified as sheep, FNs indicate the number of unidentified sheep, and TNs indicate the number of things correctly identified as not sheep. Obtaining the values of true positives and false-positives requires the calculation of the IoU, which is the degree of overlap between the predicted boxes and the boxes marked in the original graph. Figure 14 shows the schematic diagram of the IoU with the formula in Eq. (10). In general, the values of TPs and FPs depend on the relationship between the IoU and threshold, and the number of detected boxes with an IoU greater than the threshold is noted as TPs, and the data of detected boxes with an IoU less than or equal to the threshold is noted as FPs.
Table 2
Meaning of TPs, FPs, TNs and FNs.
Real Predict | Positive | Negative |
Positive | TP | FN |
Negative | FP | TN |
Figure 14: Schematic diagram of the IoU.
$$\text{I}\text{o}\text{U}=\frac{\text{G}\cap \text{D}}{\text{G}\cup \text{D}}$$
10
We choose precision, recall, F1-score, [email protected], and FPS as evaluation metrics as follows:
Precision is the proportion of the number of actual positive samples among all results predicted to be positive samples, which is calculated as in Eq. (11):
$$\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{o}\text{n}=\frac{\text{T}\text{P}}{\text{T}\text{P}+\text{F}\text{P}}$$
11
Recall denotes the probability of all actual positive samples in the predicted outcome being detected as positive samples, which is calculated as in Eq. (12):
$$\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}=\frac{\text{T}\text{P}}{\text{T}\text{P}+\text{F}\text{N}}$$
12
The F1-score considers precision and recall equally important, so the values of these two indicators are considered together, and the reconciled average of precision and recall is calculated by the formula in Eq. (13):
$$\text{F}1-\text{s}\text{c}\text{o}\text{r}\text{e}=2\times \frac{\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}\times \text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{o}\text{n}}{\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}+\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{o}\text{n}}$$
13
The mAP (mean average precision) is the average of the average precision (AP) of each class, which is an important measure of how well the model performs on all classes. Since our experiment is a single-category target detection, we added an extra useless class (background) for the sheep class dataset. The mAP value we used in this experiment is the mAP at an IoU of 0.5, where the process of calculating AP requires the use of the precision-recall curve for definite integration to find the area. The mAP is specified in Eq. (14):
$$\text{
[email protected] = }\sum _{\text{i=0}}^{\text{1}}{\int }_{\text{0}}^{\text{1}}\text{p(}{\text{r}}_{\text{i}}\text{)dr}$$
14
FPS (frames per second) is used to evaluate the detection speed of the algorithm and indicates the number of frames that can be processed per second. The same model has different processing speeds for different hardware configurations.
4.1.3 Model selection
We tested six models—EfficientDets, YOLOX, YOLOv4-Tiny, RetinaNet, YOLOv7, and YOLOV7-Tiny—to choose the best object detection network model. The results are shown in Table 3.
Table 3
Comparison experiments of different object detection networks.
Model | Precision | Recall | F1-score | mAP @0.5 | FPS |
EfficientDets | 0.914 | 0.802 | 0.854 | 0.938 | 13.864 |
YOLOX | 0.895 | 0.904 | 0.899 | 0.955 | 15.750 |
YOLOv4-Tiny | 0.918 | 0.675 | 0.778 | 0.825 | 32.302 |
RetinaNet | 0.753 | 0.883 | 0.813 | 0.850 | 8.958 |
YOLOv7 | 0.904 | 0.94 | 0.922 | 0.970 | 6.429 |
YOLOV7-Tiny | 0.911 | 0.934 | 0.922 | 0.968 | 31.985 |
Table 3 shows that the YOLOv7 and YOLOv7-tiny models are both in the top spot for F1-score and [email protected] metrics, while for FPS, the YOLOv4-tiny and YOLOv7-tiny models have essentially similar and significantly higher effects than the other models. For a comprehensive comparison, we selected YOLOv7-tiny with better results for the subsequent experiments.
4.1.4 Model compression
There are two scaling factors in the YOLOv7-tiny configuration file, depth_multiple and width_multiple; depth_multiple is the factor that controls the depth of the channel, which indicates the scaling factor of the channel, and width_multiple is used to control the network width. In this study, lightweighting—that is, comparing a model's size reduction to an increase in its speed—is the main objective. Therefore, we considered an initial improvement by scaling factors. As shown in Table 4, five different width_multiple values (0.5, 0.6, 0.7, 0.8, 0.9) were selected to scale the model size in this paper. Table 4 demonstrates that when width_multiple is set to 0.5, the model's accuracy and FPS perform better in a thorough comparison, so we choose a width_multiple of 0.5.
Table 4
Comparison experimental results of different width_multiple.
Width_ multiple | Precision | Recall | F1-score | [email protected] | FPS |
0.9 | 0.935 | 0.914 | 0.924 | 0.962 | 33.15 |
0.8 | 0.882 | 0.938 | 0.910 | 0.966 | 34.89 |
0.7 | 0.905 | 0.915 | 0.910 | 0.959 | 54.27 |
0.6 | 0.936 | 0.901 | 0.918 | 0.958 | 67.52 |
0.5 | 0.942 | 0.874 | 0.907 | 0.953 | 86.26 |
4.1.5 Attentional mechanisms
To investigate the impact of various attention mechanisms on model effects, we add SimAM, CBAM, CA, and GAM as four additional attention mechanisms to the head section of YOLOv7-tiny. The outcomes are displayed in Table 5. We discovered that adding the CBAM attention mechanism produced the highest F1-score and FPS values, whereas adding the GAM attention mechanism produced the highest [email protected] values. We decided to use the GAM attention mechanism since its FPS value was in the range of 2.0 and its F1-score value was second only to that obtained after using the CBAM attention mechanism.
Table 5
Experimental results of different attention mechanisms. Finally, we improved the accuracy of the model through data enhancement strategies, which we chose as shown in Table 6.
Model | Precision | Recall | F1-score | [email protected] | FPS |
YOLOv7-Tiny + SimAM | 0.900 | 0.932 | 0.916 | 0.956 | 89.251 |
YOLOv7-Tiny + CBAM | 0.929 | 0.913 | 0.921 | 0.957 | 89.336 |
YOLOv7-Tiny + CA | 0.921 | 0.909 | 0.915 | 0.956 | 89.155 |
YOLOv7-Tiny + GAM | 0.920 | 0.918 | 0.919 | 0.963 | 87.885 |
Table 6
Data Enhancement Strategies.
Tricks | Value |
HSV_H(Hue) | 0.015 |
HSV_S(Saturation) | 0.7 |
HSV_V(Value) | 0.4 |
mosaic | 0.1 |
4.1.6 Results of object detection
The final volume of our model, which has 1.58 MB of parameters, is 3.5 G, as shown in Table 7. The volume is reduced to 26.3% of the original size, and the FLOPS is reduced by 74.1% when compared to the original YOLOv7-tiny. The schematic of our model's final results is shown in Fig. 15. The output of our model is contrasted with that of the original network in Table 8. The value of our F1-score is the same as that of the original model, the mAP is 96.4%, and the FPS is 88.232 on the RTX 1650Ti graphics card, which is significantly higher than the original model. Additionally, we compared the frame rates (FPS) of various graphics card models (Table 9) and obtained as high as 173.788 on the RTX 3060 card.
Table 7
Comparison of the number of parameters and size between the final model and YOLOv7-tiny.
Tricks | Value |
HSV_H(Hue) | 0.015 |
HSV_S(Saturation) | 0.7 |
HSV_V(Value) | 0.4 |
mosaic | 0.1 |
Table 8
Results of the final model. Figure 15. Model results.
Model | Precision | Recall | F1-score | [email protected] | FPS |
YOLOV7-Tiny | 0.911 | 0.934 | 0.922 | 0.968 | 31.985 |
Ours | 0.914 | 0.93 | 0.922 | 0.964 | 88.232 |
Table 9
Comparison of FPS with different graphics cards.
Graphics | FPS |
RTX 1650Ti | 88.232 |
RTX 3060 | 121.524 |
RTX 3090 | 173.788 |
4.2 Pose estimation—Detection of key points
4.2.1 Experimental environment and parameters
For the key point detection experiments of the sheep body, we adopted the PyTorch framework as the experimental framework, and the experimental environment was the Ubuntu operating system with an RTX1650Ti graphics card. To ensure the experiment's reliability, we used uniform experimental parameters: The image size was 256*256, and the epoch was set to 210.
4.2.2 Evaluation metrics for pose estimation
OKS is a commonly used evaluation metric for pose estimation algorithms that computes the similarity between predicted and labeled true values of key points, and its formula is shown in Eq. (15).
$$\text{O}\text{K}\text{S} = \frac{{\sum }_{\text{i}}\text{e}\text{x}\text{p}\left[\frac{-{{\text{d}}_{\text{i}}}^{2}}{2{\text{s}}^{2}{{\text{k}}_{\text{i}}}^{2}}{\delta }\right({\text{v}}_{\text{i}}>0\left)\right]}{{\sum }_{\text{i}}{\delta }({\text{v}}_{\text{i}}>0)}$$
15
where i denotes the i-th key point, di2 is the Euclidean distance between the predicted value of each key point and the manually labeled value, s is the scale factor, v is the visibility of the key point, ki denotes the normalization factor of 2 times the normalization factor of the i-th key point, and this factor is the standard deviation of the ground truth key points by manual labeling from the true value present in all sample sets; the larger it is, the more difficult it is to label this type of key point. δ denotes 1 if the condition holds and 0 if it does not. AP is the average of all categories, and AR is the maximum recall detected in each image for a fixed number of images. When the OKS threshold was 0.5, we employed the AP50 and AR50 metrics for model evaluation. The calculation of AP50 requires the precision-recall curve, while the calculation of AR50 requires the recall-IoU curve, which is given in Eq. (16) and Eq. (17).
$$\text{A}\text{P}50 = \int \text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{o}\text{n}\left(\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}\right)\text{d}\left(\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}\right)$$
16
$$\text{A}\text{R}50 = 2{\int }_{0.5}^{1}\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}\left(\text{I}\text{o}\text{U}\right)\text{d}\left(\text{I}\text{o}\text{U}\right)$$
17
4.2.3 Selection of models
In this experiment, mmpose, an open-source toolbox for pose estimation, was chosen for the experiment, and we selected seven lightweight models, as shown in Table 10.
Table 10
Comparison of the parameters and size of the network for pose estimation.
Model | FLOPS | Parameters |
SimpleBaseline | 17.03G | 68.64M |
Pose_MobileNetv2 | 2.12G | 9.57M |
HRNet-W32 | 10.27G | 28.54M |
SCNet | 7.08G | 34.01M |
DarkPose | 10.27G | 28.54M |
LiteHRNet | 0.35G | 1.13M |
AlexNet | 1.97G | 5.62M |
Table 11
Pose estimation comparison network results.
Model | Input Size | AP50 | AR50 | FPS |
SimpleBaseline | 256*256 | 0.823 | 0.877 | 39.69 ± 3.00 |
Pose_MobileNetv2 | 256*256 | 0.826 | 0.867 | 49.97 ± 6.60 |
HRNet-W32 | 256*256 | 0.832 | 0.872 | 14.48 ± 0.53 |
SCNet | 256*256 | 0.836 | 0.872 | 19.40 ± 0.89 |
DarkPose | 256*256 | 0.818 | 0.867 | 42.28 ± 2.38 |
LiteHRNet | 256*256 | 0.804 | 0.856 | 15.49 ± 0.57 |
AlexNet | 256*256 | 0.759 | 0.800 | 205.57 ± 15.62 |
The results of the experiments are shown in Table 11. We selected AlexNet with the highest FPS accuracy for the experiments, and subsequently, we mainly worked on accuracy improvement.
4.2.4 Activation functions
The first five layers of AlexNet's 8-layer structure are convolutional layers, and the final three layers are fully connected layers. We make changes to the activation functions involved in the convolutional and fully connected layer species, which include setting the activation functions involved in the convolutional and fully connected layers to the same and different functions. Table 12 displays the results that were attained.
Table 12
Results of different activation functions.
Model | AP50 | AR50 | FPS |
AlexNet + SiLU | 0.787 | 0.836 | 201.50 ± 14.56 |
AlexNet + Hardswish | 0.786 | 0.831 | 200.43 ± 18.58 |
AlexNet + LeakyReLU | 0.782 | 0.831 | 168.20 ± 10.73 |
AlexNet + Hardswish + SiLU | 0.797 | 0.841 | 205.15 ± 12.16 |
AlexNet + Hardswish + LeakyReLU | 0.785 | 0.846 | 210.37 ± 5.29 |
AlexNet + SiLU + Hardswish | 0.788 | 0.831 | 202.46 ± 17.41 |
AlexNet + LeakyReLU + Hardswish | 0.778 | 0.821 | 200.33 ± 18.15 |
AlexNet + LeakyReLU + SiLU | 0.795 | 0.846 | 207.05 ± 7.18 |
AlexNet + SiLU + LeakyReLU | 0.807 | 0.851 | 206.47 ± 10.45 |
From Table 12, we can see that setting the activation function in the first five layers as the SiLU activation function and the activation function in the last three layers as the LeakyReLU function, where the parameter is 0.01, has the highest AP50 and AR50 values, and the FPS values are at a high level. Therefore, we chose the combination of SiLU and HardSwish.
4.2.5 Pose estimation results
After incorporating the Soft-NMS algorithm into the model, we were able to produce results with the following values: AP50 = 0.820, AR50 = 0.862, and FPS = 190.86 ± 23.97. As shown in Table 13, both AP and AR have made significant improvements when compared to the original AlexNet network, while the value of FPS essentially stays the same. At the same time, our model's performance in the current pose estimation networks far outperforms that of the more well-known lightweight network LiteHRNet.
Table 13
Final results of pose estimation.
Model | AP50 | AR50 | FPS |
AlexNet | 0.759 | 0.800 | 206.47 ± 10.45 |
LiteHRNet | 0.804 | 0.856 | 15.49 ± 0.57 |
Ours | 0.820 | 0.862 | 190.86 ± 23.97 |
4.3 Pose estimation—Behavior recognition
Based on the improved Soft-NMS-AlexNet model, we train the standing and lying posture datasets of sheep and test them on the currently popular lightweight network LiteHRNet and the initial AlexNet network, and the comparison results are shown in Table 14.
Table 14
Behavior recognition comparison results.
Model | Pose | AP50 | AR50 | FPS |
AlexNet | Stand | 0.870 | 0.891 | 196.72 ± 16.79 |
Lying | 0.960 | 0.964 | 194.48 ± 16.81 |
LiteHRNet | Stand | 0.924 | 0.935 | 18.25 ± 0.08 |
Lying | 0.959 | 0.964 | 18.43 ± 0.66 |
Ours | Stand | 0.898 | 0.924 | 193.17 ± 18.80 |
Lying | 0.960 | 0.964 | 205.29 ± 12.98 |
We can see that for the standing posture, our model's AP and FPS are better than the initial AlexNet network, and although the AP is lower than that of LiteHRNet, its FPS far exceeds that of LiteHRNet, so from the perspective of lightweightness, our model is at a better level. For the lying posture, the AP of the three models are closer, and for FPS, our model is the highest. Figure 16 shows the predictions of AlexNet, LiteHRNet, and our model, which together show that our model is generally good.
Figure 16. Behavior recognition predictions for different models.