MAT: An improved Swin Transformer model for counting cultured fish

doi:10.21203/rs.3.rs-2594518/v1

The accurate and automatic counting of cultured fish becomes increasingly important in achieving efficient production management with the expansion of the scale of aquaculture. This study proposed a cultured fish counting model (MAT) based on Multi-column dilated convolution, Attention mechanism and swin Transformer to address the problem of low counting accuracy brought on by high stocking density and complicated environment. The MAT model contains two components: the feature extraction module and the attention module. The feature extraction module is a pyramid-like structured network based on the Swin Transformer block which can integrate feature information from various scales and use patch expanding to accomplish upsampling. The attention module consists of three parallel sets of dilated convolutional blocks in different size and a residual attention structure which are used to extract more critical information from images and generate high-quality density maps. The experimental results showed that the counting accuracy of MAT on the cultured fish dataset was 97.57%, and its mean absolute error (MAE) and the root mean square error (RMSE) were 2.09 and 2.38, respectively. Compared to MCNN and CSRnet, the counting accuracy of the MAT model was improved by 4.48% and 1.22%; MAE lowered by 63.5% and 32.8%; RMSE decreased by 63.5% and 32.8%. Additionally, the density maps generated by MAT were more refined. The proposed model was thus proved to have a high counting accuracy and produced high-quality density maps which can effectively estimate the spatial distribution, regional state and specific location of the fish population.

Aquaculture

Fish counting

Swin Transformer

Attention mechanism

Deep learning

Aquaculture has experienced an average annual growth rate of 5.3% since the turn of the century, making it one of the food industries with the fastest growth worldwide for nearly five decades (Olsen, R.L., & Hasan, M.R., 2012). With the scale expansion of aquaculture, the automatic counting of farmed fish became a critical task for efficient production management of aquaculture. By accurately counting the number of fish in the culture environment, producers might better monitor stocking density, help improve fish survival rate and feed conversion rate, thus boosting the economic efficiency (Li, D., 2020). However, manual and statistical approaches were currently used in aquaculture to count the fish (Li, D., Hao, Y., & Duan, Y., 2019). The manual counting was time-consuming and labor-intensive, and the invasive counting method had an effect on the welfare and health of the fish. Therefore, a non-invasive and accurate counting of farmed fish is needed to scientifically direct production.

Computer-vision-based automatic fish counting can minimize harm to the aquaculture environment and thus can maintain the fish behavior as usual. The aquaculture field is paying increasing attention to this research methodology. Morais et al. (2005) obtained an overall counting accuracy of 81% by using log-likelihood ratios to segment the front and back scenes and Bayesian filtering techniques to track changes in the number of fish fries. The method can work reliably in the face of extreme environmental changes and deal with problems such as occlusion or great inter-frame movements. Toh et al. (2009) labeled the spot position of the fish, filtered the noise and background objects through image processing technology and calculated the spot area to obtain the number of fish, which was effective for counting a small fish population whereas the errors accumulated with the increase of fish number due to the sensitivity of the area threshold. Labuguen et al. (2012) used image processing to detect the pixel area that each fish contour occupied and calculated the total area within the shape after the fish fries’ photos taken in a specially designed container were binarized and edge-detected. This experiment achieved high counting accuracy by limiting the experimental conditions to reduce the overlap and occlusion of the fish fries in the water. Hernández-Ontiveros et al. (2018) made use of a low-cost and high-performance embedded system and realized the counting task of two kinds of marine ornamental fish through digital image processing. As early as 1995, the backpropagation algorithm was used to count fish in images (Newbury et al.,1995). For example, Fan et al. (2013) used the Otsu algorithm for threshold segmentation of the fish connectivity domain and built a counting model using the backpropagation networks and least squares support vector institutions, obtaining an average counting accuracy of 98.73%. All the above methods relied on expert experience for foreground segmentation, which overlooked the segmentation accuracy influenced by artifacts from water fluctuations and overlap and shape changes from continuous motion. The counting accuracy was thus affected. In addition, the mentioned fish images above were collected in an ideal environment, which were quite different from those collected in a cultured environment.

The fish counting method based on deep learning algorithms can realize the automatic extraction of features by utilizing the great learning ability of convolutional neural networks while no human experience is needed to segment the foreground. Previous studies on fish counting by deep learning method can be divided into two categories: fish counting based on target detection and fish counting based on density map regression. The counting method based on target detection firstly detected fish targets and then counted them statistically according to the detection results. Lainez et al. (2019) examined the effectiveness of CNNs for the fish counting by means of detecting and then counting the fish in the pictures. Tseng et al. (2020) used the mask R-CNN to identify and count the fish objects in an electronic monitoring system, obtaining the counting accuracy rate of 77.31%. Connolly et al. (2021) used a faster R-CNN as the network framework to detect the fish data collected by a bait-type remote underwater video system (BRUVS). With the advancement of the single-stage target detection algorithm, the YOLOv3 network and the mask R-CNN network were used by Kandimalla et al. (2021) to count fish targets in the high-resolution visual acoustic dataset DIDSON, and the YOLOv3 network was able to reach a detection speed of 24 fps. Allken et al. (2021) implemented the Keras framework to realize the recognition and counting of the sparse targets in trawl nets. The above studies show that the counting approach based on target detection significantly affects fish counting in non-production environments such as in surveillance systems.

The counting method based on the density map regression algorithm is more suitable for dense fish targets in a factory farming environment. Zhang et al. (2020) used a multi-column convolutional neural network MCNN as the front-end network and an extended convolutional neural network DCNN as the back-end network to establish a hybrid neural network model for counting the fish fries, obtaining the counting accuracy rate of 95.06%. Yu et al. (2021) improved the MCNN based on the attention mechanism to generate the density regression diagram of the fish swarm image and obtained the accurate counting number after integration, with the counting accuracy of 94.69%. Yu et al. (2022) proposed a multiscale dense residual connectivity network based on MCNN which generated density maps that accurately reflected the distribution of fish populations, with an average absolute error reduction of 21.8% and a mean square error reduction of 22.8% under high-density conditions. The density map regression algorithm is well suited for counting fish in situations of industrial farming because it can estimate the number of cultured fish in a dense state more precisely. However, due to the structural characteristics of convolutional neural networks, the density maps generated from the original images after multi-layer feature extraction and downsampling are usually of shallow resolution or poor quality. The low-quality density map is likely to be one of the factors limiting the accuracy.

In order to enhance the quality of the density map and produce precise counting results, this research proposes a farmed fish counting model (MAT) based on multi-column dilated convolution, attention mechanism, and Swin Transformer. The MAT model consists of a feature extraction module and an attention module. The feature extraction module is a pyramid-like network structure based on the Swin Transformer block and it can integrate the multi-layer information and improve the resolution of the output feature map. The attention module is a back-end network made up of a multi-column dilated convolution and a residual attention mechanism. The multi-scale information is further processed by this module and the fish aggregation region is treated as a critical region to generate the final density map. The ultimate goal of this study is to offer an effective and reliable counting scheme for cultured fish, which can provide theoretical support for subsequent tasks such as cultured fish biomass estimation.

The following arrangements of this paper are as follows: Section 2 presents the data acquisition and labeling, and the MAT network design; Section 3 gives the experimental results; Section 4 shows ablation experiments and discussion on the model; Section 5 is the conclusion and prospect.

2.1 Dataset

2.1.1 Data acquisition and characterization

The experimental data were collected at Mingbo Aquatic Co., Ltd. in Laizhou City, Shandong Province, China. The experimental fish was Oplegnathus punctatus, and the weight range of each fish was: 150 ± 10g; the body length range was: 20 ± 5cm. The recirculating water culture tank was 3m in diameter and 0.8m in depth. The dissolved oxygen content was controlled at 7.0 ± 1.0mg/L and the water temperature was kept at 20 ~ 25℃. As shown in Fig. 1, the image acquisition system contained two uniform light sources (16W), a high-definition camera and a computer. The camera took fish swimming video from above with a resolution of 2704 × 1520 and at a frame rate of 25 fps. The image data were extracted every 25 frames to make up the dataset. In order to lower the input to the network and reduce the influence of extraneous factors besides the culture tank, the image data were cropped and resized to produce a final image size of 896 × 896 pixels.

The experimental dataset was composed of two parts: Part A included 200 images which were divided into training and validation sets in a 3:1 ratio; Part B, which contained 90 images, served as the test set. A total of 290 image data from Part A and B were collected in the same culture tank at different periods. A single image contained 79 to 91 fish and the total sample size was nearly 25,000. Figure 2 shows the histogram of the distribution with different quantities in the two parts of the dataset. The experimental data were collected under the conditions that the fish grew normally without any damage to their body structure.

As shown in Fig. 3, the challenges of the dataset in fish counting were as follows: (1) Density: many fish were prone to contact densely in a culture tank as shown in Fig. 3(a), which posed a great challenge for the counting algorithm; (2) Overlapping: fish targets in an active state were overlapped due to the height differences, which leaded to missed detections, as shown in Fig. 3(b); (3) Background confusion: The background color of the tank was close to that of the fish body, which leaded to missed detections or wrong detections.

2.1.2 Dataset labeling and production

In this study, the center locations of the fish bodies were marked to provide sample labels. For the targets that were more than 2/3 concealed, no labeling was done. For the targets that were less than 2/3 obscured, the center points of the exposed body sections were labeled (Zhang et al., 2020). The images with labeled points were transformed into density maps, from which the final counting results could be calculated by the integration calculation of the density maps.

The Gaussian geometric adaptive kernel was used to generate the density map (Zhang et al., 2016), and the process was as follows:

If there is a fish target at pixel ${x}_{i}$, it is expressed as a delta function ${\delta }({\chi }-{{\chi }}_{\text{i}})$. An image labeled with N fish can be expressed as formula (1).

$$\text{H}\left(\text{x}\right)=\sum _{\text{i}=1}^{\text{N}}{\delta }\left({\chi }-{{\chi }}_{\text{i}}\right)$$

1

In order to transform the above formula (1) into a continuous density function, it can be convolved with a Gaussian kernel function G(x) to obtain the density function of F(x) = H(x)*G(x). The density function assumes that the sample ${x}_{i}$ is independent in the plane and each ${x}_{i}$ represents a fish body in 3-dimensional space. The pixels associated with various samples ${x}_{i}$ may correspond to areas of various sizes in the actual scene due to the perspective distortion. Accordingly, the average distance between each sample and the neighboring samples is used to determine the diffusion coefficient in order to accurately estimate the number of fish.

For each ${x}_{i}$ in the image, the average distance between ${x}_{i}$ and its m nearest neighbors is assumed to be $\stackrel{-}{{d}_{i}}=\frac{1}{m}\sum _{j=1}^{m}{d}_{j}^{i}$. H(x) and the Gaussian kernel function with variance ${\sigma }_{i}$ proportional to $\stackrel{-}{{d}_{i}}$ are convolved to estimate the density of fish in the vicinity of pixel ${x}_{i}$. More specifically, the function F(x) should be expressed as formula (2), in which the adaptive parameters β = 0.3 and m = 4. In this experiment, all sample labels are labeled and generated in the same manner.

$$\text{F}\left(\text{x}\right)=\sum _{\text{i}=1}^{\text{N}}{\delta }(\text{x}-{\text{x}}_{\text{i}})\text{*}{\text{G}}_{{{\sigma }}_{\text{i}}}\left(\text{x}\right),\text{w}\text{i}\text{t}\text{h}{{\sigma }}_{\text{i}}={\beta }\stackrel{-}{{d}_{i}}$$

2

2.1.3 Data augmentation

As shown in Fig. 4, four data augmentation methods were used in this study to increase data diversity: brightness variation, image blurring, Gaussian noise, and mosaic augmentation. Mosaic augmentation was to make the random cropping of the original images and then stitch them, which was proved to help the model learn to recognize objects smaller than the normal (Bochkovskiy et al., 2020). The total number of the images in the cultured fish dataset was 1040, among which 900, 50 and 90 images were used for the training, validation and test sets, respectively.

2.2 MAT network design

In recent years, the natural language processing model Transformer(Vaswani et al., 2017) was widely used in computer vision fields such as in target detection (Zhu et al., 2021), image classification (Dosovitskiy et al., 2021), image segmentation (Zhang et al., 2021) and crowd counting (Tian et al., 2021; Liang et al., 2021). These methods have achieved excellent performance. In the field of fish counting, the feature extraction layers of neural networks were usually composed of ordinary convolutional structures. In constructing MAT, a feature extraction structure based on Swin Transformer was proposed in a quite different construction method of the traditional convolution (Liu et al., 2021).

2.2.1 Overview of MAT

The structure of the MAT model is shown in Fig. 5. It contained the feature extraction module and the attention module. In the feature extraction module, the input image was split by a patch partition layer and into non-overlapping patches with 4x4 pixels, each with the feature dimension of $4\times 4\times 3$. A linear embedding layer was used to generate feature outputs with arbitrary number of feature dimensions, and the transformed patches were passed through three sets of Swin Transformer blocks to produce layered feature outputs with three scales. To fuse the features from different output layers, a pyramid-like structure was proposed for the feature fusion of the information extracted from different Swin Transformer blocks, and a patch expanding layer was used as the method of upsampling (Cao et al., 2021). The feature maps from different layers were summed pixel by pixel to obtain the final feature map which was then used as the input of the back-end network. Based on the RFB structure (Liu et al., 2018), a three-column parallel dilated convolution was used by the attention module to flexibly aggregate multi-scale contextual information, and the final output density map thus had a higher spatial resolution. The global attention mechanism was used to select more critical information for the dense counting. With the aid of the attention module, MAT model finally produced high-quality density maps.

2.2.1 Swin Transformer block

The MAT model constructed a CNN-like hierarchical structure using the Swin Transformer block as the basic unit. In the initialization phase, the Swin Transformer block segmented the image into patches with small sizes and the adjacent patches were gradually fused as the layers increase. Besides, this structure allowed MAT to train multi-scale features in stages, as shown in Fig. 6(a). Moreover, the Swin Transformer block was less computationally intensive because the multi-headed attention mechanism was performed only within each patch.

The Swin Transformer block adopted a shifted window to realize information interaction of each patch on the upper layer, as shown in Fig. 6(b). Unlike the traditional multi-headed self-attention module (MSA) (Vaswani et al., 2017), the main structure of the Swin Transformer block consisted of two parts: the window-based multi-headed self-attention module (W-MSA) and the shifted window-based multi-headed self-attention module (SW-MSA). The structure also contained a Layer Norm (LN) layer, a multilayer perceptron (MLP) layer with nonlinear activation function GELU, and a residual connection, as illustrated in Fig. 7. The W-MSA and the SW-MSA which were applied to two consecutive Swin Transformer blocks can be represented as:

$${\widehat{z}}^{l}=W-MSA\left(LN\left({z}^{l-1}\right)\right)+{z}^{l-1}$$

3

$${z}^{l}=MLP\left(LN\left({\widehat{z}}^{l}\right)\right)+{\widehat{z}}^{l}$$

4

$${\widehat{z}}^{l+1}=SW-MSA\left(LN\left({z}^{l}\right)\right)+{z}^{l}$$

5

$${z}^{l+1}=MLP\left(LN\left({\widehat{z}}^{l+1}\right)\right)+{\widehat{z}}^{l+1}$$

6

${\widehat{z}}^{l}$ and ${z}^{l}$ denote the outputs of the (S)W-MSA and MLP modules for the block $l$. Similar to the work (Hu et al., 2018; Hu et al., 2019), self-attention is formulated as follows:

$$Attention\left(Q,K,V\right)=SoftMax(\frac{Q{K}^{T}}{\sqrt{d}}+B)V$$

7

$Q, K, and V\in {\mathbb{R}}^{{M}^{2}\times d}$ represent the query, key and value matrices, respectively; ${M}^{2}$ is the number of patches in the window; $d$ is the dimension of the query or key; the values in B are taken from the smaller-sized bias matrix $\widehat{B}\in {\mathbb{R}}^{(2M-1)(2M+1)}$.

2.2.2 Patch merging and expanding

The patch merging layer is another key module in the Swin Transformer. The downsampling and layering structure of the patch merging layer help the MAT network become better suited for operating fine-grained tasks, such as the small-target counting. However, continuous layering and downsampling can lead to the low resolution of the final generated density map. In order to obtain a higher resolution of the subsequent feature map, patch expanding layer which is similar to the patch merging layer in structure was adopted the in the upsampling process of the feature map, increasing the final resolution of the original map from 1/16 to 1/4. The patch expanding layer in stage 2 is taken as an example, as illustrated in Fig. 5. A linear layer was used to expand the input dimension which is twice that of the original one ($\frac{\text{W}}{8}\times \frac{\text{H}}{8}\times 96\to \frac{W}{8}\times \frac{H}{8}\times 192$). Then the rearrangement method was used to double the input resolution and reduce the feature dimension to a quarter of the original one ($\frac{W}{8}\times \frac{H}{8}\times 192\to \frac{W}{4}\times \frac{H}{4}\times 48$).

2.2.3 Attention module

The feature extraction module based on Swin Transformer was able to capture global feature information, so a back-end network was needed for regressing the exact density map. An attention module with multi-scale receptive fields was constructed in the experiment to detect scale changes and density differences.

The scale perspective deformation of the fish in the images is an unavoidable problem due to the camera’s shooting angle. In order to address the scale variation of the fish images, the MAT model made advantage of the RFB structure’s design (Liu et al., 2018) and used a multi-scale dilated convolution structure based on width stacking. The above method was more suitable for small targets detection and had a faster training speed than the depth-based stacking approach (Szegedy and Ioffe, 2016). A 2D dilated convolution is defined as follows:

$$y\left(m,n\right)=\sum _{i=1}^{M}\sum _{j=1}^{N}x(m+r\times i,n+r\times j)w(i,j)$$

8

$x\left(m,n\right)$ represents the input; $w\left(i,j\right)$ is the convolution kernel; $M$,$N$ are the length and width of the convolution kernel; $y\left(m,n\right)$ is the result of the dilated convolution operation; $r$ is the expansion rate and when $r=1$, the dilated convolution becomes regular convolution.

Since there were density variances in the spatial distribution of fish in culture tanks, the back-end network needed to enhance information extraction for the regions where fish congregated. The residual attention mechanism was used to select more critical information for dense counts. The backbone convolution network and attention network were connected in parallel to form the attention module (Wang et al., 2017). The output $H$ of the attention module is shown in Eq. (9).

$${H}_{i,c}\left(x\right)=\left(1+{M}_{i,c}\left(x\right)\right)\text{*}{F}_{i,c}\left(x\right)$$

9

$i$ denotes all spatial ranges; $c$ denotes the index of the channel; ${M}_{i,c}\left(x\right)$ is the mask factor calculated by the attention structure and its range is $\left[\text{0,1}\right]$. When ${M}_{i,c}\left(x\right)$ is close to 0, the output ${H}_{i,c}\left(x\right)$ will be close to the input feature ${F}_{i,c}\left(x\right)$. ${M}_{i,c}\left(x\right)$ is also used as a feature selector to augment the good features of the backbone network and suppress the unfavorable noise.

2.3 Loss function

The sum of mean squared errors (SSE) was used as a loss function for estimating the difference between the density map and the ground truth in this article. The loss function has been widely used in counting models because it is simple and can be trained quickly. The definition of the loss function is shown in Eq. (10):

$$L\left(\theta \right)=\sum _{i=1}^{N}{\left|F\left({X}_{i};\theta \right)-{F}_{i}\right|}^{2}$$

10

${\theta }$ is a set of learnable parameters in MAT; $n$ is the number of training images; ${X}_{i}$ is the input image; ${F}_{i}$ is the ground truth value of image ${X}_{i}$,; $F\left({X}_{i};{\theta }\right)$ denotes the density map generated by MAT model; $L$ is the loss between the estimated density map and the ground truth density map.

2.4 Evaluation indicator

In recent years, MAE and RMSE have been frequently employed to assess the performance of counting models. To evaluate the MAT’s performance, MAE, RMSE and accuracy were used in this study. Among them, MAE and accuracy were used to assess the counting accuracy of the MAT model and RMSE was used to evaluate the stability and robustness of the model. The definitions of each assessment metric are as follows:

$$MAE=\frac{1}{N}{\sum }_{1}^{N}\left|{z}_{i}-{\widehat{z}}_{i}\right|$$

11

$$RMSE=\sqrt{\frac{1}{N}{\sum }_{1}^{N}{({z}_{i}-{\widehat{z}}_{i})}^{2}}$$

12

$$Accuracy=\left(1-\frac{{\sum }_{1}^{N}\left|{z}_{i}-{\widehat{z}}_{i}\right|}{{\sum }_{1}^{N}{z}_{i}}\right)\times 100\text{%}$$

13

$N$ is the number of test images, ${z}_{i}$ is the actual number of fish in the ${i}_{th}$ image, and ${\widehat{z}}_{i}$ is the estimated number of fish in the ${i}_{th}$ image.

The MAT model was run on a workstation with Intel(R) Core(TM) i7-10700F CPU @ 2.90 GHz and an NVIDIA GeForce RTX 2080 GPU. The training parameters were set in Table 1.

Table 1 The setting of parameters in the training process

Parameter name	Set up
Optimization algorithm	Adam
Initialization of weights	Gauss initialization with 0.01 standard deviation
Epoch	2000
Learning rate	1e-5
Loss function	SSE
Batch size	1(online learning)

3.1 Counting results and performance evaluation

Benefiting from the upsampling and back-end network performance, the density map obtained from the MAT model has a 1/4 resolution of the original map. To intuitively display the density map generated by MAT, the density map and the original map are overlayed, as shown in Fig. 8. On the density map, the higher the probability of fish appearing at the pixel point, the closer the color is to red; on the contrary, the color is closer to blue. Finally, the counting results are obtained by making weighted sum of the density map. As shown in Fig. 8, the spatial distribution and the estimated number of fish as well as the specific location of the fish targets are determined by the MAT model.

The evaluation results of the MAT model on the datasets Part A_val and Part B are showed in Table 2. The counting results for each image are displayed in Fig. 9. For the results in Part A_val, the accuracy of the MAT model reached 98.06% and MAE was 1.67 and RMSE was 2.09. For the results in Part B, the counting accuracy of the MAT model reached 97.57% and MAE was 2.09 and RMSE was 2.38. By the observation of Fig. 9, the counting accuracy of only a small number of images is lower than 95.5%. Therefore, the counting accuracy of the MAT model is high.

Table 2

Results of assessing indicators for datasets
	MAE	RMSE	Mean accuracy (%)
Part A_val	1.67	2.09	98.06
Part B	2.09	2.38	97.57

The results of counting accuracy under different fish population distributions are shown in Table 3. The accuracy of images with less than 83 fish targets is higher than that of the overall correct rate. In Part B, the accuracy of segment (83, 87] was 0.39% lower than that of the segment (79, 83], and the accuracy rate of segment (87, 91] is 0.32% lower than that of the segment (83, 87). An increase in the number of fish in a single image leads to a decrease in the counting accuracy. However, there is no significant changes in the counting accuracy of the MAT model and the maximum accuracy differences between different intervals do not exceed 0.7%. It is thus concluded that the stability of the MAT model is good.

Table 3 The counting accuracy under different fish distribution

	Mean accuracy(%)
	(79,83]	(83,87]	(87,91]
Part A_val	98.22	97.86	97.96
Part B	97.96	97.58	97.26

The counting effectiveness of the MAT model was evaluated in face of challenges of density, overlapping, and confusing background mentioned in Section 2. The original fish images and the density images predicted by the MAT model are shown in Fig. 10(a) and Fig. 10(b). The results for the dense challenge are displayed in Fig. 10(c) and Fig. 10(d), which indicates that the MAT model is excellent in individual response for dense fish population and can precisely count the number of fish in the dense area. The results of the MAT model for confusion background challenges is illustrated Fig. 10(e). The counting results show that the MAT model is effective in identifying such targets, the colors of which are close to the center column. The challenge of orthogonal overlapping caused by depth differences is seen in Fig. 10(f), and most of the targets in this case can be well identified by the MAT model. Therefore, the MAT model is proved to be effective in identifying and counting the fish targets for the density, overlapping, and confusing background challenges.

3.2 Model performance comparison

Two deep learning models, MCNN and CSRnet, were utilized to train the cultured fish dataset so as to validate the performance and verify the superiority of the MAT model on fish counting. The evaluation results by using MAE, RMSE and Mean accuracy on Part B are compared in Table 4. The MAT model achieved the best results in metrics of MAE, RMSE, and average accuracy. Compared with the MCNN model, the MAT model had 63.5%, 64.6%, and 4.48% improvement in the three metrics. Compared with the CSRnet model, it had 32.8%, 38.7%, and 1.22% improvement in the three metrics. Table 5 compares model size, floating point operations(FLOPs), and inference speed computed on one 896×896 input image. It can be observed that the model size and FLOPs of MAT are much smaller than CSRnet, and the inference speed is similar to CSRnet. The running cost of MCNN is the smallest, but it also has the lowest counting accuracy. The accuracy comparison of each image on the three models is shown in Fig. 11. Although the accuracy of counting some images by the MCNN or CSRnet model was higher than that by MAT, the average accuracy by MAT was much higher and there was less accuracy fluctuation. It was concluded that the MAT model was superior in stability and counting accuracy.

Table 4 The results of performance indicators for different methods.

Methods	MAE	RMSE	Mean accuracy(%)
MCNN	5.72	6.73	93.38
CSRnet	3.11	3.88	96.39
MAT	2.09	2.38	97.57

Table 5 Comparison of the model size(M), FLOPs and Inference speed (fps).

Methods	Model Size (M)	GFLOPs	Speed(fps)
MCNN	0.133	21.76	58.27
CSRnet	16.263	332.12	14.19
MAT	2.418	45.21	15.62

Visualization was used to further evaluate the quality of density maps generated by using MCNN, CSRnet and MAT models and it turned out that the density maps generated by the MAT model were of higher quality. The original images and the density maps generated by each model are overlayed, as shown in Fig. 12, from which it can be seen that the specific location of individual fish is better reflected in the density maps generated by MAT and the counting numbers are closer to the actual ones. Specifically, for overlapping and dense challenges, merged identifications of multiple connected fish bodies were performed by MCNN and CSRnet models while more precise location identification of specific fish targets was performed by the MAT model, generating valid and accurate points, as seen in Fig. 12(a) and Fig. 12(b). For the confusion background challenge, there were missed detections by MCNN and CSRnet models in varying degrees while fish targets above the background were accurately detected by the MAT model, as seen in Fig. 12(c). Compared to other counting models, the MAT model can better perform the cultured fish counting task and get more accurate counting numbers.

The results of absolute errors between the ground truth maps and the density maps generated by each model are displayed in Fig. 13, in which the more the red and yellow pixels, are there, the bigger the absolute errors between the predicted and actual maps are produced. The results show that significantly more errors in the density maps produced in dense areas are generated by MCNN and CSRnet models, but relatively fewer errors are caused by the MAT model. Besides, the similarity between the generated density map and the ground truth map is ensured by MAT. So, the absolute error is minor and the quality of the density map produced by MAT is high.

To verify the validity of the two modules working together in the MAT model, the effectiveness of each module are discussed in this section.

(1) Improvement of Swin Transformer block on feature extraction of fish targets.

The Swin Transformer block constitutes the main part of the feature extraction module in the MAT model. Through the global association mechanism of the multi-headed self-attention module, the problem of the global information loss occurred when the CNN extracts features can be avoided. The CNN feature extraction is a weighting operation within a finite-size convolutional kernel to get the feature map of the next layer, while the Swin Transformer block chunks the whole image, and then the tiny patches are correlated with each other to get the feature map of the next layer. In Fig. 14, the feature maps obtained by using the Swin Transformer block, MCNN, and Vgg16 as feature extraction methods are compared. Clearer edges, richer details, and less information loss can be seen in the fish images extracted by the Swin transformer block. However, there are edge shadows and position shifts in the feature images extracted by MCNN. The external contour of the fish cannot be shown in the features extracted by Vgg16 and the morphological features of the fish are lost. By comparing the feature maps obtained by different models and analyzing the density maps in Section 3, it is concluded that the MAT model can not only generate density maps that are closer to the ground truth map but also get finer target features in the feature extraction stage, which makes the counting results more reliable.

(2) The effect of each module in MAT on the improvement of counting results.

The ablation experiments are conducted on the MAT model. Figure 15 shows the accuracy curves during the training of the ablation experiments. Figure 16 shows the loss function curves. The results of whether to use the Swin Transformer block as the feature extraction network and the attention module as the back-end network are compared in Table 5. To ensure the integrity of the overall network structure, The Swin Transformer is replaced with the CNN structure in the feature extraction phase of the MAT model to compare the changes brought by different feature extraction networks. When the back-end network is frozen and only the standard regression head is used to train the model, the Swin Transformer block shows a 7.6% improvement in counting accuracy compared to CNN. When the back-end network is unfrozen and the attention module is used as the regression head, the MAT shows a 7.2% improvement in accuracy compared to the CNN + attention module. The result illustrates that using the Swin Transformer block as the feature extraction module consistently achieves higher counting accuracy on this dataset than that using the traditional CNN as the feature extraction module, regardless of whether the attention module is enabled or not. In addition, the overall accuracy of the network model is improved by using the attention module as the back-end network, regardless of whether CNN or Swin Transformer block is used as the feature extraction module. Therefore, the effectiveness of each of the two modules in the MAT model is illustrated by the ablation experiments.

Table 5 The results of ablation experiments.

Module name	CNN	Swin Transformer	Attention module	MAE	RMSE	Mean accuracy(%)
CNN	√			11.19	11.71	87.04%
Swin Transformer		√		5.46	7.19	93.67%
CNN+Attention	√		√	7.37	8.10	91.49%
MAT		√	√	1.67	2.09	98.06%

This paper proposed the MAT model based on the Swin Transformer block, the multi-column dilated convolution and the attention mechanism to solve the fish counting problems in intensive aquaculture environment. The proposed model can effectively tackle the problems of density, overlapping, and background confusion in fish counting process. The feature extraction module dominated by the Swin Transformer block enhances the global connection between pixels and extracts global contextual semantic information. In the attention module, the width-stacked dilated convolution focuses more on the deformation of fish size and the residual attention mechanism directs the model’s attention to more critical regions to produce high-quality density maps. The results showed that the counting accuracy of the MAT model on the validation and test sets was 98.06% and 97.57%, respectively. Compared with the MCNN model, the MAT model had 63.5%, 64.6% and 4.48% improvement in metrics of MAE, RMSE and average accuracy. Compared with the CSRnet model, the MAT model had 32.8%, 38.7%, and 1.22% improvement in the three metrics. Higher fineness and smaller errors can be found in the density map generated by MAT. In summary, the MAT model can achieve high-precision counting of cultured fish and the generated high-quality density maps can be used as the reference information for subsequent tasks.

In this study, small sample sizes and multiple data augmentation approaches are used to reduce the labeling efforts, because labeling fish targets in intensive culture settings requires a lot of labor. In the follow-up studies, unsupervised or semi-supervised learning approach is advised to lessen the labeling workload and improve the overall generalization ability of the model.

Ethical Approval

The experimental data were collected under the conditions that the fish grew normally without any damage to their body structure.

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Authors’ contributions

Rongxiang Du: Conceptualization, Methodology, Software, Investigation, Writing – original draft, Visualization, Writing – review & editing. Yachao Li: Data curation, Validation, Investigation. Zhiyong Wang: Data curation, Investigation. Chen Xu: Data curation, Investigation. Daoliang Li: Validation, Investigation, Resources. Yingyi Chen: Resources, Validation. Wensheng Li: Resources, Project administration. Chunhong Liu: Conceptualization, Supervision, Funding acquisition, Writing – review & editing.

Funding

This work was supported by the Beijing Digital Agriculture Innovation Consortium Project (BAIC10-2022). We are very grateful to Laizhou Jinyiyuan Aquatic Products Co., Ltd. for supplying the fish sample acquisition environment and their valuable input regarding hands-on experience with constructing and operating RAS in practice.

Availability of data and materials

Data will be made available on request. The code that supports the findings of this study is available from the corresponding author upon reasonable request.

Allken, V., Rosen, S., Handegard, N.O., Malde, K. (2021). A deep learning-based method to identify and count pelagic and mesopelagic fishes from trawl camera images. ICES Journal of Marine Science.
Bochkovskiy, A., Wang, C., & Liao, H.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. ArXiv, abs/2004.10934.
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2021). Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation. ArXiv, abs/2105.05537.
Connolly, R., Fairclough, D., Jinks, E., Ditria, E., Jackson, G., Lopez‐Marcano, S., Olds, A., & Jinks, K. (2021). Improved accuracy for automated counting of a fish in baited underwater videos for stock assessment. bioRxiv.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv, abs/2010.11929.
Fan, L., & Liu, Y. (2013). Automate fry counting using computer vision and multi-class least squares support vector machine. Aquaculture, 380, 91-98.
FAO (2010). The state of the world fisheries and aquaculture 2010.Rome: FAO. Available at. http://www.fao.org/docrep/013/i1820e/i1820e.pdf. Accessed 14.07.11
Hernández-Ontiveros, J.M., Inzunza-González, E., García-Guerrero, E.E., López-Bonilla, O.R., Infante-Prieto, S.O., Cárdenas-Valdez, J.R., & Tlelo-Cuautle, E. (2018). Development and implementation of a fish counter by using an embedded system. Comput. Electron. Agric., 145, 53-62.
Hu, H., Gu, J., Zhang, Z., Dai, J., & Wei, Y. (2018). Relation Networks for Object Detection. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3588-3597.
Hu, H., Zhang, Z., Xie, Z., & Lin, S. (2019). Local Relation Networks for Image Recognition. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 3463-3472.
Kandimalla, V., Richard, M., Smith, F., Quirion, J., Torgo, L., & Whidden, C. (2021). Automated Detection, Classification and Counting of Fish in Fish Passages With Deep Learning. Frontiers in Marine Science.
Labuguen, R.T., Volante, E.J., Causo, A.J., Bayot, R.K., Peren, G.S., Macaraig, R.M., Libatique, N., & Tangonan, G.L. (2012). Automated fish fry counting and schooling behavior analysis using computer vision. 2012 IEEE 8th International Colloquium on Signal Processing and its Applications, 255-260.
Lainez, S.M., & Gonzales, D.B. (2019). Automated Fingerlings Counting Using Convolutional Neural Network. 2019 IEEE 4th International Conference on Computer and Communication Systems (ICCCS), 67-72.
Li, D., Hao, Y., & Duan, Y. (2019). Nonintrusive methods for biomass estimation in aquaculture with emphasis on fish: a review. Reviews in Aquaculture.
Li, D., Miao, Z., Peng, F., Wang, L., Hao, Y., Wang, Z., Chen, T., Li, H., & Zheng, Y. (2020). Automatic counting methods in aquaculture: A review. Journal of The World Aquaculture Society.
Li, Y., Zhang, X., & Chen, D. (2018). CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1091-1100.
Liang, D., Chen, X., Xu, W., Zhou, Y., & Bai, X. (2021). TransCrowd: Weakly-Supervised Crowd Counting with Transformer. ArXiv, abs/2104.09116.
Liu, S., Huang, D., & Wang, Y. (2018). Receptive Field Block Net for Accurate and Fast Object Detection. ArXiv, abs/1711.07767.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 9992-10002.
Morais, E.F., Campos, M.F., Pádua, F.L., & Carceroni, R.L. (2005). Particle Filter-Based Predictive Tracking for Robust Fish Counting. XVIII Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI'05), 367-374.
Newbury, P.F., Culverhouse, P.F., & Pilgrim, D.A. (1995). Automatic fish population counting by artificial neural network. Aquaculture, 133, 45-55.
Olsen, R.L., & Hasan, M.R. (2012). A limited supply of fishmeal: Impact on future increases in global aquaculture production. Trends in Food Science and Technology, 27, 120-128.
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A.A. (2016). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. ArXiv, abs/1602.07261.
Tian, Y., Chu, X., & Wang, H. (2021). CCTrans: Simplifying and Improving Crowd Counting with Transformer. ArXiv, abs/2109.14483.
Toh, Y.H., Ng, T.M., & Liew, B.K. (2009). Automated Fish Counting Using Image Processing. 2009 International Conference on Computational Intelligence and Software Engineering, 1-5.
Tseng, C., & Kuo, Y. (2020). Detecting and counting harvested fish and identifying fish types in electronic monitoring system videos using deep convolutional neural networks. ICES Journal of Marine Science.
Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017). Attention is All you Need. NIPS.
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017). Residual Attention Network for Image Classification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6450-6458.
Yu, J., Jia, R., Li, Y., & Sun, H. (2022). Automatic fish counting via a multi-scale dense residual network. Multim. Tools Appl., 81, 17223-17243.
Yu, X., Wang, Y., An, D., & Wei, Y. (2021). Counting method for cultured fishes based on multi-modules and attention mechanism. Aquacultural Engineering.
Zhang, S., Yang, X., Wang, Y., Zhao, Z., Liu, J., Liu, Y., Sun, C., & Zhou, C. (2020). Automatic Fish Population Counting by Machine Vision and a Hybrid Deep Neural Network Model. Animals : an Open Access Journal from MDPI, 10.
Zhang, Y., Liu, H., & Hu, Q. (2021). TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation. MICCAI.
Zhang, Y., Zhou, D., Chen, S., Gao, S., & Ma, Y. (2016). Single-Image Crowd Counting via Multi-Column Convolutional Neural Network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 589-597.
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2021). Deformable DETR: Deformable Transformers for End-to-End Object Detection. ArXiv, abs/2010.04159.

No competing interests reported.

MAT: An improved Swin Transformer model for counting cultured fish

Status:

Version 1

Abstract

Figures

1. Introduction

2. Materials And Methods