Static Gesture Recognition Based on Thermal Imaging Sensors

doi:10.21203/rs.3.rs-4689884/v1

Download PDF

Research Article

Static Gesture Recognition Based on Thermal Imaging Sensors

https://doi.org/10.21203/rs.3.rs-4689884/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Gesture recognition technology has demonstrated its applicability in various scenarios within the field of human-computer interaction, owing to its flexibility and non-contact nature. In particular, thermal imaging technology which is not limited by lighting conditions and effectively reduces the risk of privacy disclosure by capture only thermal radiation rather than detailed features. In this study, a low resolution 32 x 24 pixels end-to-end embedded infrared thermal imaging camera gesture recognition system is developed. A thermal imaging gesture dataset of 4,500 images is constructed to train and evaluate the system. The effects on improve gesture recognition accuracy by research the spatial transform network attention mechanism (STN) is discussed. Thus, a new method combines lightweight Convolutional Neural Networks (CNN) and STN is propose. Additionally, the proposed method achieves 98.5% recognition accuracy and inference time of only 59 milliseconds per frame on embedded devices when tested on self-made infrared thermal imaging sign language gesture dataset, outperforming mainstream lightweight models.

Thermal imaging

Gesture recognition

Convolutional neural networks

Embedded systems

Gestures serve as a silent yet potent form of communication, finding utility in diverse fields such as automotive interfaces, healthcare systems, assistive technologies, entertainment and human-computer interaction [1–2]. This versatility enables effective and contactless communication across various contexts, which not only enables immersive virtual reality (VR) and provides touchless control over smart devices but also assists individuals with hear and speech impairments. It plays a crucial role in advanced domains such as robotic control technology and medical diagnosis [3–10].

The advent of deep learning has propelled advancements in digital gesture recognition, with methodologies predominantly based on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). CNN is widely recognized for their superior ability in image classification and feature extraction, whereas RNN excel in handling sequential data. These attributes allow for the nuanced recognition of complex gestures. V.M., et al. used a low-resolution thermal imaging camera for gesture prediction and introduced a novel detection technique that integrates 2D CNN with Temporal Convolutional Networks (TCN). This approach attained a classification accuracy of 95.9% and a mean Average Precision (mAP) [11]. Wu J., et al. developed a dynamic gesture recognition model that employs data gloves, combine a CNN for local feature capture and Bi-directional Long Short-Term Memory (BiLSTM) networks for temporal feature extraction, culminating in a 95.05% accuracy [12]. Lin Z., et al. proposed a dynamic gesture recognition technique used a CNN with a 3D receptive field, achieves 97.5% accuracy [13]. Zhang X.J., et al. leveraged the AlexNet-based CNN model for gesture recognition, achieves an average accuracy of 98% [14]. Tsironi, et al. tried to use CNN and Long Short-Term Memory (LSTM) for dynamic gesture recognition, with a comprehensive accuracy of 80.10% [15]. Bao P., et al. advocated for a direct classification of seven gesture types via deep CNN, bypassing the segmentation or detection phases, and secured a 97.1% accuracy in simple background settings [16]. Oyedotun, et al. performed static gesture recognition based on deep learning and proposed CNN and stacked denoising autoencoder (SDAE), achieves an accuracy of 91.33% [17]. Molina J., et al. employed Deep Neural Networks (DNN) for classify seven gesture actions via Time-of-Flight (TOF) cameras, with a success rate of 94% [18]. Most of the above recognition models comprise several convolutional layers, max pooling layers, a variety of regularization layers and Transformer architectures. Therefore these models will have larger parameters, require more memory and longer model inference times.

Most of studies employ deep learning models that utilize either RGB cameras to capture visible light or depth cameras to collect depth information that are suitable for a wide range of scenarios. However, gesture recognition accuracy may be affected by the quality of RGB images under different lighting conditions. Additionally, the high cost associated with depth cameras restricts their widespread adoption. [19–21].

Miniature infrared sensors employed thermal imaging technology effectively capture gesture information via temperature differentials in environment where insufficient light. This significantly enhances the adaptability and robustness of gesture recognition systems. By acquire thermal rather than color images, these sensors ensure user privacy at the hardware level, safeguard against the inadvertent disclosure of personal identity information.

Thermal imaging cameras are distinguished by their ability to operate independently of ambient light conditions and their cost-efficiency compared to TOF cameras. This kind of sensor accurately capture thermal fluctuations induced by gestures through differentiating between the ambient background and the warmer human body temperature. After conducted a thorough review, this method has selected the MLX90640 infrared sensor due to its compact size, affordability, and high precision, which is characterized by a 32x24 pixel resolution and an adjustable sampling frequency. This study introduces a novel gesture recognition approach that utilizes a lightweight CNN architecture integrate a Spatial Transformer Network (STN) module, thereby enhancing the model's ability to handle variations in imagery. The effectiveness of this lightweight CNN framework was evaluated against the lightweight classification model FastViT [22], renowned for its superior performance on the ImageNet-1K dataset. The successful deployment of this optimized CNN model on the Raspberry Pi which low-energy computational platform, affirms its practical applicability and operational effectiveness in real-world scenarios.

2.1 Collection Method

The Melexis MLX90640 matrix infrared temperature sensor was utilized to collect thermal imaging data for gesture-based sign language. The sensor is equipped with an Inter-IC (I2C) interface for seamless integration with Raspberry Pi embedded system. A custom expansion board is essential to establish a reliable connection between the matrix infrared temperature sensor and the Raspberry Pi. This expansion board securely mounts the matrix infrared temperature sensor and ensures the smooth operation of the thermal imaging camera. It also facilitates accurate signal transmission through the implementation of pull-up resistors to effectively prevent potential voltage fluctuations or signal interference. One end of the pull-up resistor connects to the power supply, while the other end connects to the data and clock lines. The custom expansion board is thoughtfully designed with appropriate pins to facilitate secure attachment of the connection cables to the Raspberry Pi. Data from the matrix infrared temperature sensor is collected via the I2C interface of the Raspberry Pi. The connection method between the I2C interface of the Raspberry Pi and the expansion board is shown in the Raspberry Pi and the sensor are visualized in Figure 1.

Table 1

Raspberry Pi connected to sensor
Raspberry Pi	MLX90640
3.3V	VIN
GND	GND
SDA(Pin3)	SDA
SCL(Pin5)	SCL

A comprehensive thermal imaging dataset of sign language gestures was created by connecting an MLX90640 matrix infrared temperature sensor to a Raspberry Pi to capture and classify various types of images. An organized folder structure is established, with image folders labeled from 0 to 9 to store the corresponding sign language gesture images. The MLX90640 matrix infrared temperature sensor operates by measure half of the 768 (32x24) pixels at a time, with each measurement referred to as a sub-page. A complete data frame consists of two sub-pages. The merged of these sub-pages is crucial for form the complete set of 768 pixels. Consequently, the two read arrays are normalized and merged into a complete data array after the sensor data is read twice consecutively. As a final step, the merged arrays are converted into grayscale images to forming a single measurement data frame. A Python-based thermal imaging collection program was developed for seamlessly capture thermal images via the Raspberry Pi embedded system to ensure efficient collection of thermal imaging data for sign language gestures. Real-time images are processed used bilinear interpolation and pseudo-color mapping techniques to enhance recognition of sign language gesture types, as opposed to display the original grayscale images directly. The intricate step-by-step process encompassing the entire collection program is depicted in Fig. 2.

2.2 Dataset Construction

A dataset contains 4,500 thermal imaging pictures of sign language gestures was constructed following the steps shown in the Fig. 3 to provide necessary train data for deep learning. These gestures were collected from individuals of various ages, result in a diverse dataset due to variations in the angle and distance of the hand relative to the camera during image capture. Calculations and empirical test of the MLX90640 matrix infrared temperature sensor's Field of View (FoV: 55°x35°) established that gestures should be placed within a range of 12cm to 25cm from the sensor to ensure optimal image collection.

Notably, the MLX90640 sensor has a resolution of merely 32×24 pixels. Consequently, the feature information necessary for traditional machine learning tasks is reduced and thereby increasing the difficulty of classification due to the captured images possess low resolution. The entire dataset of 4,500 images occupies a mere 3.4MB of disk space, which is even smaller than that occupied by a single 1080P image (1080x1920 pixels, 8-bit depth, RGB color space). As a result, the dataset's reduced storage requirements allow for flexibility when training various neural network models.

The visual quality of each sign language gesture image varies depending on the hand's position, posture, and external factors unrelated to the thermal imaging sensor itself. The sensor primarily detects relative temperature differences between the background and foreground objects, which is effectively reflect the temperature differential of the human hand in relation to its background environment. Figure 3 exemplifies sign language gesture images from the same classes captured by the same individual. Specifically, Fig. 3(a) represents an image of high quality: clear visibility of all fingers and the entire palm, with the center of the sign language gesture closely aligned with the center of the image. Such quality is desirable for effective deep learning training processes. It is essential to include images like those in Fig. 3(b), where the posture of the gesture obscures the fingers to enhance the model's generalization ability. Figures 3(c) and 3(d) respectively depict images of poor quality resulting from offset angles of gesture pitch and rotation. Machine learning algorithms require classification of gestures with varying temperatures and postures. Every sign language gesture is contributed by different individuals, wherein hand temperatures, distances, and postures can significantly vary. The dataset exhibits substantial diversification even within the same classes. Consequently, a vast amount of profoundly diverse thermal imaging gesture data were collected for this study, which enabling machine learning algorithms to acquire abundant features and successfully differentiate between distinct sign language gestures.

Standard image classification tasks utilize high-resolution and high-quality images for training and learning purposes. These high-quality train images contribute to stability, reduce variations in image quality, and provide the advantage of faster image acquisition speed. However, when capture images of consistent quality used thermal imaging technology becomes challenge due to temperature differences among individuals within the same background environment. The thermal images employed in this research possess a significantly low resolution, with the representation of temperature difference between the hand and background limited to two color changes, without additional gradients or detail layers. As a result, classification becomes particularly challenging. Figure 4 showcases the 10 types of thermal imaging sign language gestures captured used the MLX90640 matrix infrared temperature sensor. Due to the sensor's low-resolution nature, higher temperature differences are required to distinguish objects from the background. The areas that appear gray in the thermal image shown in Fig. 4 are due to heat leakage where the camera captures heat from the surrounding air and walls. The low resolution also leads to visual similarities between certain images within the dataset, such as images 4(a) and 4(j), as well as images 4(d) and 4(h). While these similarities pose little difficulty for experienced experts in gesture classification, accurate classification by computers necessitates the employment of effective algorithms. Including a set of high-quality images substantially improves the success rate of accurate gesture detection for both humans and machines.

Following the collection of 4,500 images (450 per classes), the dataset was partitioned into training, validation, and test sets according to the proportions specified in Table 2.

Table 2 Data set proportion

Class	Train	Validation	Test
0	320	65	65
1	320	65	65
2	320	65	65
3	320	65	65
4	320	65	65
5	320	65	65
6	320	65	65
7	320	65	65
8	320	65	65
9	320	65	65
Summary	3200	650	650

3.1 Attention Mechanism

The Spatial Transformer Network (STN) utilizes an attention mechanism to parameterize transformations through a localization network, allowing for direct prediction of transformation parameters $\:\theta\:$ from the input data. This process comprises three key components: a localization network that predicts the transformation parameters, a grid generator that facilitates the creation of a sampling grid, and a sampler that applies the transformation to produce the transformed output feature map. The transformation can be mathematically represented as Eq. 1:

$$V_{i}^{c}=\sum\limits_{n}^{H} {\sum\limits_{m}^{W} {U_{{nm}}^{c}} } \cdot k(x_{i}^{s} - m;{\Phi _x}) \cdot k(y_{i}^{s} - n;{\Phi _y})$$

Where $\:{V}_{i}^{c}$ is the output pixel value at position $\:i$ in channel $\:c$, $\:{U}_{nm}^{c}$ is the input pixel value at position $\:(n,m)$ in channel $\:c$, $\:k$ represents the sampling kernel parameterized by $\:{{\Phi\:}}_{x}$ and $\:{{\Phi\:}}_{y}$, and $\:({x}_{i}^{s},{y}_{i}^{s})$ represents the source coordinates in the input feature map. The STN module enables the network to output feature maps based on the input feature maps by spatial transformation. This is particularly critical for dealing with changes in gesture appearance due to scale, rotation, and pose changes in gesture recognition tasks. It can effectively reduce the difficulty of low-resolution infrared image classification because of the STN module actively transforms and magnifies the relevant features of gestures. When the STN attention mechanism is added to the proposed lightweight CNN model, the detection accuracy is increased by 1.5%. Figure 5(a) is the image before the space transformation and Fig. 5(b) is the image after the space transformation.

3.2 Depthwise Separable Convolution

The structure for lightweighting the model is primarily inspired by the principle of depthwise separable convolution, which is found in the renowned MobileNet architecture[23]. Traditional CNN model often demand significant computational resources and memory space when process large-scale data, it is posing challenges in resource-constrained environments such as mobile devices or embedded systems. Therefore, there is a need to design more lightweight model structures that can adapt to such resource-limited scenarios. As an effective operation, the depthwise separable convolution reduces the number of parameters and computational cost. It achieves this by initially increasing dimensions through a 1x1 convolution, followed by utilizing depthwise convolution with higher input and output channels, which allows for the extraction of more information. The shortcut connection module, assists in address the issues of vanishing or exploding gradients during the train process of deep neural networks. By incorporating skip connections that directly pass the input to the output layer, the shortcut connection module minimizes information loss and accelerates the training process. Figure 6 illustrates the bottleneck structure, while the parameters for each layer are presented in Table 3.

Table 3 Bottleneck layer parameters

LayerType		Output Size
Input		(None, M, H, W)
1x1 Conv		(None, MZ, H, W)
3x3 Depthwise Conv		(None, MZ, H/S, W/S)
1x1 Conv		(None, Z, H/S, W/S)
Add		(None, Z, H/S, W/S)
Output		(None, Z, H/S, W/S)

3.3 Improved Gesture Recognition Network

The traditional deep learning models exhibit insufficient robustness when applied to classification tasks involving low-quality thermal images. In order to solve the above problems, the present study introduces a deep learning model that learns image features exclusively from the training dataset. The proposed model is characterized by its lightweight and efficient nature, feature 1,690,138 parameters and a weight size of 6.5 megabytes. Inspired by depthwise separable convolutional networks, the model is composed of multiple convolutional layers and two inverted residual structures. Moreover, the proposed model leverages 3x3 convolutional kernels which employed batch normalization and the ReLU activation function following each convolution, except for the initial convolutional layer.

The overall structure of the model proposed in this study is shown in Fig. 7 while the parameters for each layer are presented in Table 4. The input of the model is 32x24 pixel gray image collected by the thermal imaging sensor. The input size of the model is (None, 32, 32, 1). Where None represents the value of the number of parameters passed to the program for training, 32x32 represents the resolution of the training image, and 1 represents the input image channel as 1, which also signifies it is a grayscale image. The STN attention mechanism performs spatial transformation based on the input image's intrinsic features, resulting in the output of transformed feature maps. Since the spatial transformation does not alter the size and channel count of the image, the output feature maps maintain the same dimensions of 32x32 and a channel count of 1 as the input. In the inverted residual part, the input channel number is expanded through a 1x1 expansion layer used an expansion ratio (expand_ratio) to enhance the feature dimensions. This inverted residual block within an input channel number of 32 and an expansion ratio of 6, and the output channel number becomes 192. Following the inverted residual block, a 3x3 depthwise separable convolution layer (Depthwise Separable Convolution) performs individual convolution operations on each input channel. Finally, a 1x1 compression layer compresses the expanded channel number back to the target output channel number, followed by batch normalization. In the fully connected layer, the model first flattens the feature map of the last layer into a one-dimensional vector. Next, through a fully connected layer, these features are mapped to the target classes or values. The fully connected layer maps the flattened feature vector to corresponding 10 outputs. The thermal gesture images are transformed via the STN attention mechanism, afterwards convolution and a stack of inverted residual blocks, to obtain high-level features. These features are further processed to generate gesture recognition probabilities, which are then converted into probability distributions through the softmax activation function.

Table 4

Lightweight CNN model parameters
Layer Type	Output Size
Input	(None, 32, 32, 1)
STN	(None, 32, 32, 1)
Conv	(None, 16, 16, 64)
Conv	(None, 8, 8, 128)
Bottleneck	(None, 8, 8, 128)
Conv	(None, 4, 4, 256)
Bottleneck	(None, 4, 4, 256)
Conv	(None, 2, 2, 128)
Flatten	(None, 512)
Dense	(None, 10)

4.1 Experimental Setup

The model was tested in a 64-bit Windows 10 environment with Python 3.7 and PyTorch, on a Lenovo laptop with the following hardware specifications: CPU Intel i5-9300H, GPU GTX1650 with 4GB memory, and CUDAv10.2. The dataset used for the experiment is a custom low-resolution infrared sign language gesture dataset, as shown in Table 2. This custom infrared dataset is divided into training set, validation set and test set with 3,200, 650 and 650 images respectively. Each images being with being 32 pixels in height and 24 pixels in width, totaled 768 pixels. During training, the original image size was scaled to 32 pixels in both height and width, with the batch size set to 32. This experiment falls under the classes of image classification tasks, thus the Softmax function was adopted.

4.2 Model Comparison

This research compares the proposed lightweight gesture recognition model with the FastViT model for the task of low-resolution infrared sign language gesture recognition. The FastViT model is a more advanced model for the ImageNet dataset classification task, it is outperforming other competitive architectures in various tasks such as image classification, detection, and segmentation. Moreover, it significantly reduces latency on mobile devices and desktop GPUs. The FastViT model comes in various sizes, each with a different architectural size. For this gesture recognition comparison, only utilizes the lightweight variant of the FastViT model. Table 5 presents a detailed comparison between the proposed lightweight gesture recognition model and the FastViT model.

Table 5

Model comparison
Model	Accuracy	Model size (MB)	parameters
Proposed Model	98.50%	6.53	1690138
FastViT-T8	92.27%	13.08	3427720
FastViT-T12	95.91%	25.46	6675304
FastViT-S12	99.77%	40.78	10689768
FastViT-SA12	97.95%	40.78	10689768

The proposed lightweight CNN gesture recognition model achieved an accuracy of 98.50% on the test dataset. The confusion matrix obtained from testing on the test set is shown in Fig. 8.

Figure 8 is observed that the digits "0," "3," "5," "6," "8," and "9" could be correctly identified with high accuracy for their respective classes. The digits '4' and '7' managed to correctly identify their respective classes samples with 98% accuracy. 96% of digit '4' classes samples identified correctly, and 92% of digit '7' classes samples identified correctly. The average inference time for a test sample was 59ms on the Raspberry Pi edge computing device, less than the 62.5ms each frame of thermal imaging is captured. Comparing the performance of the proposed lightweight gesture recognition model with other gesture recognition models (shown in Table 5), it is evident that the proposed model is more lightweight. The model weight file and the number of parameters are less than half of those in the lightest version of the compared models, while achieves higher accuracy.

Given the prevalence of thermal imaging sensors as standard peripherals in embedded devices, it is imperative to develop suitable models as assistive technologies for their widespread application in embedded devices and smartphones. The lightweight CNN model possesses a clear advantage in terms of seamless deployment in embedded devices and integration into mobile applications for real-time inference on smartphones. The notable enhancement in accuracy when compared to conventional models, can be attributed to the strategic integration of the STN attention mechanism and depth-wise separable convolution in the proposed model.

This study presents a complete end-to-end embedded infrared gesture recognition system which is based on a low-resolution thermal imaging camera with a resolution of 32×24 pixels for gesture recognition. A thermal imaging dataset containing 4,500 images with 450 thermal images per gesture category was created to support this study. The study explores the impact of the spatial transformation network attention mechanism on gesture recognition accuracy, result in an improvement of 1.5% in accuracy for gesture recognition tasks. A gesture recognition method is proposed which combines lightweight CNN with the spatial transformation network attention mechanism, which achieves an accuracy of 98.5% on the test dataset. This approach significantly enhances the real-time performance and accuracy of thermal imaging gesture detection on embedded devices. The inference time for each frame image on the embedded device is only 59 milliseconds, which is less than the time to collect one frame of infrared image. It was considered an ideal surveillance technology for nighttime or low-light environments because the gesture recognition method based on thermal imaging technology is not influenced by background lighting conditions. Moreover, the utilization of thermal imaging technology solely captures thermal radiation instead of detailed personality features, thereby mitigate the risk of privacy leakage at the hardware level and provide substantial privacy protection advantages for users.

Acknowledgments

The authors would like to thank for the support by the National Natural Science Foundation, China (No. 52175256) and the Henan Provincial Science and Technology R&D Joint Fund (Key Program) , China (No. 225200810029).

Author contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by REN Hao, LI Hao, YUAN Kang-hui and ZHU Chu-feng. Supervision of the whole process by ZHANG Zhi-yuan. The first draft of the manuscript was written by ZHANG Zhi-yuan and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

Partial financial support was received from Project supported by the National Natural Science Foundation, China (No. 52175256) and the Henan Provincial Science and Technology R&D Joint Fund (Key Program) , China (No. 225200810029).

Conflict of Interest

The authors have no competing interests to declare that are relevant to the content of this article.

Data Availability

The thermal imaging gesture dataset supporting Table. 2 and Figs. 3–5, are published in the ScienceDB, as part of this record: https://doi.org/10.57760/sciencedb.08344.

Thermal image data were captured from the volunteers at Zhengzhou University of Light Industry stored in .png format on ScienceDB as collection thermal imaging gesture dataset. All thermal image data were obtained from the Thermal Imaging Gesture Dataset (TIGD) are collect from the volunteers.

Compliance with Ethical Standards

Research involving Human Participants

In the study, an infrared camera was used to collect human sign language gesture data. Gestures and postures are placed with the consent of the volunteers. The entire data collection process does not involve contact between experimental equipment and the human body.

Informed consent

All data collected in the study were obtained with the consent of the volunteers. The processing and using of data are carried out with the consent of the volunteers.

Ahmed S, Khan F, Ghaffar A, Hussain F, Cho SH. (2019) Finger-counting-based gesture recognition within cars using impulse radar with convolutional neural network. Sensors, 19(6):1429. https://doi.org/10.3390/s19061429
Nogales RE, Benalcázar ME (2021) Hand gesture recognition using machine learning and infrared information: a systematic literature review. Int. J. Mach. Learn. Cyber 12(10):2859-86. https://doi.org/10.1007/s13042-021-01372-y
Quesada L, López G, Guerrero L (2017) Automatic recognition of the American sign language fingerspelling alphabet to assist people living with speech or hearing impairments. J AMB INTEL HUM COMP 8:625-35. https://doi.org/10.1007/s12652-017-0475-7
Tateno, Shigeyuki, Hongbin Liu, and Junhong Ou (2020) Development of sign language motion recognition system for hearing-impaired people using electromyography signal. Sensors 20(20):5807. https://doi.org/10.3390/s20205807
Obermeier, Christian, Thomas Dolk, Thomas C Gunter (2012) The benefit of gestures during communication: Evidence from hearing and hearing-impaired individuals. CORTEX 48(7):857-870. https://doi.org/10.1016/j.cortex.2011.02.007
Saad, M, Bleakley, CJ, Nigram, V, & Kettle, P (2018) Ultrasonic hand gesture recognition for mobile devices. J MULTIMODAL USER IN, 12:31-39. https://doi.org/10.1007/s12193-017-0257-8
Sagayam, K M, Hemanth, D J (2017) Hand posture and gesture recognition techniques for virtual reality applications: a survey. VIRTUAL REAL-LONDON 21 pp 91-107. https://doi.org/10.1007/s10055-016-0301-0
Stančić I, Musić J, Grujić T (2017) Gesture recognition system for real-time mobile robot control based on inertial sensors and motion strings. ENG APPL ARTIF INTEL 66:33-48. https://doi.org/10.1016/j.engappai.2017.08.013
Hu, Bin, Jiacun Wang (2020) Deep learning based hand gesture recognition and UAV flight controls. INT J AUTOM COMPUT 17(1): 17-29. https://doi.org/10.1007/s11633-019-1194-7
Mahmoud N M, Fouad H, Soliman A M (2021) Smart healthcare solutions using the internet of medical things for hand gesture recognition system. COMPLEX INTELL SYST 7:1253-1264. https://doi.org/10.1007/s40747-020-00194-9
Vandersteegen M, Reusen W, Van Beeck K, Goedemé T (2020) Low-latency hand gesture recognition with a low-resolution thermal imager. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . pp: 98-99.
Wu J, Ren P, Song B, Zhang R, Zhao C, Zhang X (2023) Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism. PLOS ONE 18(11): e0294174. https://doi.org/10.1371/journal.pone.0294174
Lin Zhiwei, Zhu Wenzhang, Chen Hao (2021) Dynamic Gesture Recognition Based on Spatio-Temporal Feature Extraction of Neural Networks. Computer and Modernization 6: 41-47.
Zhang Xingjian, Zhang Jianxin (2020) Gesture Recognition Control System Based on Convolutional Neural Networks. Computer Applications and Software 37(10): 220-224.
Tsironi E, Barros P, Weber C, Wermter S (2017) An analysis of convolutional long short-term memory recurrent neural networks for gesture recognition. NEUROCOMPUTING 268: 76-86. https://doi.org/10.1016/j.neucom.2016.12.088
Bao P, Maqueda AI, del-Blanco CR, García N (2017) Tiny hand gesture recognition without localization via a deep convolutional network. IEEE T CONSUM ELECTR 63(3): 251-257.
Oyedotun, Oyebade K., and Adnan Khashman (2017) Deep learning in vision-based static hand gesture recognition. NEURAL COMPUT APPL 28(12): 3941-3951. https://doi.org/10.1007/s00521-016-2294-8
Molina J, Pajuelo JA, Martínez JM (2017) Real-time motion-based hand gestures recognition from time-of-flight video. J SIGNAL PROCESS SYS 86:17-25. https://doi.org/10.1007/s11265-015-1090-5
Roh MC, Christmas B, Kittler J, Lee SW (2008) Gesture spotting for low-resolution sports video annotation. PATTERN RECOGN 41(3):1124-1137. https://doi.org/10.1016/j.patcog.2007.07.013
Pisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: A review. COMPUT VIS IMAGE UND 141: 152-165. https://doi.org/10.1016/j.cviu.2015.08.004
Vishwakarma DK, Grover V (2017) Hand gesture recognition in low-intensity environment using depth images. International Conference on Intelligent Sustainable Systems (ICISS) pp 429-433. https://doi.org/10.1109/ISS1.2017.8389446
Vasu PK, Gabriel J, Zhu J, Tuzel O, Ranjan A (2023) FastViT: A fast hybrid vision transformer using structural reparameterization. In the Proceedings of the IEEE/CVF International Conference on Computer Vision pp 5785-5795.
Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp 4510-4520.

No competing interests reported.

Download PDF

Reviews received at journal
08 Sep, 2024
Reviews received at journal
06 Sep, 2024
Reviews received at journal
06 Sep, 2024
Reviews received at journal
28 Aug, 2024
Reviewers agreed at journal
26 Aug, 2024
Reviewers agreed at journal
19 Aug, 2024
Reviewers agreed at journal
19 Aug, 2024
Reviewers agreed at journal
19 Aug, 2024
Reviewers agreed at journal
18 Aug, 2024
Reviewers agreed at journal
17 Aug, 2024
Reviewers agreed at journal
17 Aug, 2024
Reviewers invited by journal
17 Aug, 2024
Editor assigned by journal
09 Jul, 2024
Submission checks completed at journal
09 Jul, 2024
First submitted to journal
05 Jul, 2024

You are reading this latest preprint version

Static Gesture Recognition Based on Thermal Imaging Sensors

Status:

Version 1

Abstract

Figures

1 Introduction

2 Sign Language Gesture Collection

2.1 Collection Method

2.2 Dataset Construction

3 Lightweight Gesture Recognition Model

3.1 Attention Mechanism

3.2 Depthwise Separable Convolution

3.3 Improved Gesture Recognition Network

4 Experiments and Results Analysis

4.1 Experimental Setup

4.2 Model Comparison

5 Conclusion

Declarations

References

Additional Declarations

Status:

Version 1