Computer vision application in tracking grooming behaviour in arthropods

Computer vision application is a broad-spectrum sophisticated technique of high accuracy, which can be used efﬁciently to detect objects from a video footage. The principle of feature detection is being implemented through this technique for the recognition of desired behavioural patterns. Our results demonstrate a competent function of SURF (Speeded Up Robust Features), a computer vision algorithm in assessing animal behaviour from a recorded video. In the present study, we have applied this algorithm for accurate quantiﬁcation of grooming behaviour in aquatic arthropods, using a semi-transparent freshwater prawn, Macrobrachium lamarrei as a model organism. Grooming behaviour is considered as an index of neuronal stress in several animals. We predict the effective application of this method in the area of behavioural ecology and neuroethological research in diverse group of animals.


Introduction
Behavioural analysis has been considered as a reliable tool to depict the state of nervous system (brain) in an organism. Hence, assessment of behavioural markers has been comprehensively used by psychologists to analyse neuropsychiatric and neurodegenerative disorders. Auto-grooming (self-grooming) or grooming is one such highly considered behavioural marker of neuro-stress or neural disorders in mammals 1,2 . Biologists, working on neuroethological stress in organisms, assess behavioural alteration by using computational methods. Automated quantification of behavioural patterns in animals from video recordings has gained more importance due to its accuracy and efficacy to work with mass data in very less time. It has been a regular and productive practice of collaboration between behavioural ecologists and computer vision researchers to produce data of excellent quality. In fact, application of machine vision always adds a technical dimension to any image-based or vision-based ecological research, which eventually reduces manual errors during data production 3 . Use of computer vision application is a very useful and contemporary protocol to assess behavioural patterns in animals, which we have tried to reflect through our current research. 3D computer vision technology is an advanced form, due to its capability to deal with three-dimensional visual data 4 . Tracktor is a software used for automatic detection of movement and specified behaviours in animals 5 . In Caenorhabditis elegans, automatic behavioural quantification through computer vision was done in the context of drug testing 6 . Multi-Worm Tracker (MWT), is a real time computer vision system which was designed to analyse behaviours in Caenorhabditis elegans 7 . Motion, intraspecific interaction in fossorial and cryptic crabs in natural habitat were analysed using the vision technique 8 . Vision application showed efficacy in the detection of social interaction in animals 9 . Also, this technique has been used to classify aquatic fauna in marine ecosystem 10 . Three-dimensional flying pattern analysis in Drosophila melanogaster and detection of specific behaviours in honeybee was executed through computer vision 11,12 . This computing method has been widely implemented in entomological research 13 . Conventionally, grooming is the act of cleaning body parts which is a behavioural index of chemical stress in a freshwater prawn, Macrobrachium lamarrei 14 . Arthropods are the most diverse group of animals (aquatic and terrestrial both), considered as models for wide range of organismal research including ecological and neuroethological research. Researchers working on arthropod behaviour can apply this technique in a wider perspective for the analysis of several robust behaviours; conditional to case dependent alterations. To the best of our knowledge, till date there are dearth of explicit reports on the use of this technique for the quantification of grooming behaviour in arthropods (invertebrates). Especially, there are extremely less reports on the implementation of computer vision to detect behaviours in aquatic arthropods. Previously Kalueff and Touhimaa (2004) 15 used grooming analysis algorithm to measure grooming activity in mice. Conventional computer vision techniques that we have discussed so far, have not been applied on any semi-transparent animal. As our experimental model (i.e. Macrobrachium lamarrei) is also semi-transparent and show camouflage in their habitat which makes our task challenging. Edge of the body structure of the prawn is one important feature. However, with only edge detection we may not recognize the postures and their variation along the time-frame. Moreover, the edge information by itself is not spatially scaled. Hence, we shifted to blob detection instead of edge detection. In computer vision, blob detection refers to components that are designed to identify points and/or regions in the image that are contrasting in features like brightness or color compared to the surrounding. SURF (Speeded Up Robust Features) 16 is that kind of computer vision algorithm, which we have applied accordingly for appropriate evaluation of grooming behaviour in a semi-transparent prawn species. Interestingly, SURF is not a specific algorithm for behaviour analysis; however, we have used its fundamental principle and efficacy for the detection of particular behavioural activity in a video footage. This is a protocol-based research, where, we have implemented the algorithm in MATLAB R2019b and successfully applied the code on videos of prawns in vitro condition. The algorithm is not limited to prawns or other aquatic arthropods only, rather it has a broader scope to apply on any other living organisms to detect desired postures or behaviours.

Methods
Macrobrachium lamarrei displays a complicated grooming pattern, which is streamlined into two subtypes: Anterior Grooming (AG) and Posterior Grooming (PG), depending on the body region. The test videos were recorded using prawns in aquarium with clear water. Canon EOS 700D camera was set on a tripod, parallel to the aquarium. Only side view is needed to trace the grooming activity in prawns, as it clearly portrays the movement of appendages during the activity. Prawns were sampled from ponds and after video recording, they were left back to their native environment. Our aim is to assess and quantify the desired grooming patterns in this prawn species from any given video. We have extracted frames at a regular interval from the recorded video. Macrobrachium lamarrei is a semi-transparent animal, so the colour of the image is not a necessary information for our method. Hence, we have converted the frame images to grayscale images. We started with manual selection of few reference images of the concerned postures: AG and PG respectively. We have detected and extracted the SURF features from the reference images first, which are our reference feature points. We have then extracted SURF features from each individual frame images in exactly the same way. Finally, we have compared the reference feature points with the features of individual frames. We have evaluated the total number of matching points. If the match is beyond a threshold value, then we label the time point of that frame as a groomed frame of the particular reference type (AG or PG). Typically, we considered the half of the maximum overall matching count as the threshold. We kept track of the total number of groomed and non-groomed frames during the whole video. This quantifies the grooming behaviour of the prawn in respect to time. Here the labels of the time points have to go through a buffer error correction procedure. Suppose in a horizon of t time points (t > 2) one time point in between is labeled differently than the rest. This abrupt alteration may arise due to error because prawns do not alter their behaviour for one single second only. We have rectified this error by introducing a moving average of t time points. The video recorded by the camera is of 30 frames per second. We have extracted a frame from each second intervals. The flowchart in Figure 1 schematically demonstrates the complete method. Following we describe our method elaboratively. As we have implemented our method in MATLAB R2019b, we provide the MATLAB code snippets alongwith.

Feature extraction
We have manually collected two sets of reference images (k 1 and k 2 in numbers respectively) of two grooming patterns -AG and PG: The reference images could also be collected from any arbitrary video of Macrobrachium lamarrei grooming. The model i.e., the prawn can move in any direction inside the aquarium. Eventually it is a three-dimensional space where x, y and z-axes track top-down, left-right and near-far movements of the model respectively. We converted the reference images to grayscale and use a robust local blob detection algorithm to extract important features from the grayscale reference images. In computer vision, SURF is a potential local feature detector and descriptor. It uses an integer approximation of the Hessian blob detector 17 to detect interest points. We have employed SURF for model recognition because it is scale invariant. Hence, horizontal movement of the prawn along z-axes does not introduce any error. We have utilized SURF extraction method to find

Feature comparison
The extracted features from each frame collected from the video at one second intervals are termed as I f eatures . We have compared extracted features between two sets: set of reference images, R f pts and set of frames from concerned video footage, I f eatures . The comparison leads us to the index pair: Ψ AG and ψ PG : {Ψ(l) = 1 ∨ R f pts (l) = I f eatures (l)}. Magnitude of Ψ represents the similarity between a reference image of grooming and the video frame. From k reference images, the total match is the sum of matches in the index pair and we refer it as τ AG and τ PG : τ = ∑ k Ψ respectively. At each time point i, if τ(i) > η, we can say that the prawn is grooming, and the pattern of grooming is reflected by τ itself. Here η is a predefined threshold value. We present the corresponding MATLAB code in Figure 4.

Error correction
Prawns usually do not alter their behaviour for one single second only. However, the hard thresholding that we have applied for the detection of grooming pattern might introduce both false positive and false negative errors. For example, the method can detect a single random time point as groomed (false positive error) or exclude a groomed time point (false negative error) from actual grooming duration. In order to correct this error, we have applied a t-seconds moving average on the τ. For example, moving average with t = 3, means τ(i) = τ(i−1)+τ(i)+τ(i+1)

3
. This flattens the values of τ and effectively correct any single point error. The value of t can be adjusted with respect to the frequency of movement of the prawn and duration of the video. Typically, it is between 3 and 5. In MATLAB 2019b, movmean(total_match,3) function gives 3 seconds moving average of the total match value, it is τ. If τ =movmean(total_match,3) of the i th second is greater than η, we have considered that time point as groomed. In Figure 5, we have presented the corresponding MATLAB code snippet.

Generating the behavioural pattern graph
Grooming is eventually a vector of binary numbers where, Grooming(i) is set to 1 only if the i th second is groomed, otherwise it is set to 0. Finally, we have counted total number of 0's and 1's to find out total number of groomed and non-groomed seconds consecutively. We represent the Grooming vector using a line diagram and the total groomed duration by bar plot using MATLAB in the Result section. Algorithm We present the complete method in the following algorithm.
Algorithm CSM Input: Reference video: Vid, Set of reference images for AG and PG: {R f AG 1 , R f AG 2 , . . . , R f AG k 1 } and {R f PG 2 , R f PG 3 , . . . , R f PG k 2 }; Output: Time points where prawn has groomed 1: R f AG ← convert all AG reference images into 8bit grayscale 2: R f PG ← convert all PG reference images into 8bit grayscale 3: R f AG pts ← SURFextraction(R f AG ) 4: R f PG pts ← SURFextraction(R f PG ) 5: Extract frames from Vid in one second interval 6: nFrames ← total number of frames extracted 7: for t = 1 to nFrames do 8: img ← readFrame(t) 9: 10: I f eatures ← SURFextraction(I g ) 11: for i = 1 to nFeatures do 12: if R f AG pts (i) == I f eatures (i) then

Results
We have applied the proposed method on separate videos, for AG and PG. However, the method can also be applied on a single video having both kind of grooming patterns (if available). Heuristically by setting η = max i (τ) 2 for both AG and PG, we have found the grooming assessment. In our first recorded video, which was of 40 seconds, the proposed method have detected a total of 29 seconds of AG and remaining 11 seconds were non-groomed. In our second recorded video, which was of 80 seconds, the method has detected a total of 65 seconds of PG and remaining 15 seconds as non-groomed. We demonstrate the result by line diagram and bar-plot. In Figure 6, the graphs establish the patterns or expressions of AG and PG which is a simplified manifestation of the grooming activity represented in terms of groomed and not groomed style in a line diagram format. In fact, this type of graphs is effective in determining the grooming frequency and comparing the pattern in different biological or environmental conditions. This line gram is effective for prolonged video footage. In Figure 7, the plots express AG and PG in a spent time format. Normally, quantification of any behaviour in respect to time represents the rate of that particular behaviour (like grooming in our case). This eventually infers to a comparative assessment of behaviours in several circumstances. Additionally, Macrobrachium lamarrei is a semi-transparent prawn species. The efficacy of the algorithm was verified when it capably detected these prawns in water and assessed desired behavioural pattern.

Conclusion
Feature recognition is the most essential part where computer vision effectively recognizes the behavioural pattern. Our work is an application of real time computing in the area of behavioural ecology. The contemporary research in biology mandatorily implements several aspects of computing. Organismal research and vision-based research in biological sciences have gained new acceleration after gaining technical accomplishment. In fact, it has made an easy way for behavioural ecologists to tackle practical challenges while working with live animals and to generate high-throughput data. We predict, through this method, it will be easier to work with aquatic animals, even if the model organism is semi-transparent. It should have a good impact on neuroethological research in invertebrates. Through this method, we suggest that this application is easy to handle and generates prominent data easily. Researchers working on behavioural ecotoxicology, neurotoxicology or drug testing on animal models can surely apply this method.