Fusing Deep Separable Networks for Emotion Recognition in Complex Environments

doi:10.21203/rs.3.rs-4717715/v1

Download PDF

Research Article

Fusing Deep Separable Networks for Emotion Recognition in Complex Environments

https://doi.org/10.21203/rs.3.rs-4717715/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

This research is dedicated to addressing the challenges of model training and application implementation in the field of traditional face emotion recognition. The proposed scheme is a lightweight facial emotion recognition system based on deep separable networks. The scheme optimizes the Xception network structure in three stages. Initially, the residual module and the deep separable module are combined in order to enhance the efficiency of feature extraction. Subsequently, global average pooling is employed in lieu of the traditional fully connected layer, thereby reducing the number of parameters. Finally, the channel information is integrated through the 2×2 convolution and the point-by-point convolution, which effectively minimizes the loss of information while retaining more valuable feature information. The solution employs the Fer2013 dataset from the Kaggle platform for training with the objective of enhancing the accuracy of emotion recognition in diverse and complex environments within a simulated real-world setting. Furthermore, the objective is to develop a lightweight model that can be deployed in practical applications. The recognition accuracy has now reached an industry-leading level. Moreover, the project has developed a face emotion recognition system that has been validated through the construction of a real-time vision system. This system is capable of completing the tasks of face detection and emotion classification in complex environments, thus providing a solid foundation for the practical application of emotion recognition technology. The source code for the proposed method will be available at https://github.com/Alex-star-arch/Fusing-Deep-Separable-Networks-for-Emotion-Recognition-in-Complex-Environments.

Emotion recognition

residual module

depth separable

network lightweighting

real-time vision system

Emotion recognition is the process of analyzing human emotional information from multiple channels, including facial expressions, voice signals, and physiological signals, using computer technology to determine human emotional states, such as happiness, sadness, anger, surprise, and so forth. This field represents an extremely important and valuable research direction for artificial intelligence, which has a wide range of applications, including smart home, smart education, smart healthcare, and smart security, etc. The objective is to improve the effect and experience of human-computer interaction and to enhance the intelligence and friendliness of AI systems [1]. Research on emotion recognition is inextricably linked to the development of AI and big data. As algorithms approach the level of human perception, understanding human emotions will become a crucial step for AI to enter the realm of self-emotion. Emotions are not only a unique human perceptual expression, but also an integral part of the psyche, reflecting the internal state and external behavior of humans and influencing thinking and decision-making. Consequently, an understanding and recognition of human emotions is of paramount importance to the development and evolution of AI.

The recognition of emotions through facial expressions is the most intuitive and commonly used method, and the one that most closely approximates human perception of emotions towards each other. Facial expressions are a complex and efficient aspect of human nonverbal communication that involves the movement of multiple facial muscles. These changes can convey a wide range of emotions and psychological states, including happiness, sadness, anger, surprise, fear, and disgust. Given the significance of facial expressions in communication, automated facial expression recognition (FER) technology has garnered increasing attention. FER has the potential to transform various fields, including education and healthcare. For instance, FER can be utilized in educational settings to assess the efficacy and quality of teaching [2,3] or in medical settings to assist in analyzing a patient's psychological condition [4]. The advancement of GPU technology has also facilitated the development of downstream applications of FER, which has contributed to its growing popularity. Compared to other image recognition tasks, the main challenges faced by FER are inter-class similarity and intra-class variability in human facial expressions. Inter-class similarity refers to the subtle differences between facial expressions, which makes it difficult to highlight small differences between facial expressions and recognize them correctly. In contrast, intra-class variability, also known as subject variability, refers to the fact that the images of an expression category in the FER dataset consist of different subjects with different facial structures, gender, age, and ethnicity. This variability may affect the learning performance of the solution, as the model may have difficulty generalizing across different subjects, resulting in lower accuracy and reliability. For instance, the distinction between expressions of anger and disgust may be subtle, whereas the variation between individuals within the same expression category may be considerable. Moreover, existing research has identified additional challenges associated with FER on wild datasets, including the recognition of negative expressions, FER under challenging conditions, and the reliance on large neural networks. The paucity of negative emoji images on the Internet renders the collection of a representative dataset that reflects real and complex scenes a challenging endeavor. Consequently, the imbalance in the representation of negative and positive expressions in the wild FER dataset may result in a lower recognition rate for negative expressions than for positive ones. Additionally, there are instances where facial expression recognition is challenging when the subject is at certain angles or when the subject's face is partially occluded by other objects. It is therefore of great importance to achieve accurate recognition of these samples, particularly in view of the likelihood that the challenging conditions will be the same as those encountered in downstream applications. Additionally, in the pursuit of classification performance, existing work has gradually favored the use of large neural networks to achieve these performance gains. Nevertheless, in light of the downstream application's computational resource constraints, FER methods must be capable of serving these applications without requiring the use of powerful resources.

The field of emotion recognition is currently facing a number of challenges and problems. Firstly, emotion is a complex and diverse psychological phenomenon, which is affected by a variety of factors such as individual differences, cultural backgrounds, social environments, etc. Consequently, it is difficult to define and measure it by a uniform and accurate standard [5]. Second, emotion information obtained from different channels may be inconsistent or incomplete. For instance, facial expressions may be masked or camouflaged, speech signals may be interfered or distorted by noise, and physiological signals may be interfered or changed by other factors. In addition, existing emotion recognition datasets are often collected from laboratories with a single environment. However, emotion recognition in real and complex environments may have different needs and difficulties. These include ensuring the efficiency and robustness of emotion recognition in complex environments subject to variations in noise, light, occlusion, gesture, and intensity of expression. Another challenge is achieving real-time and accurate emotion recognition in dynamically changing environments. Furthermore, there is a need to achieve simple and accurate emotion recognition in lightweight scenarios, as well as simple and fast emotion recognition. In order to address these issues, cutting-edge research methods have been proposed in recent years. [7] proposed DialogueRNN, which employs a memory network-based model to model the emotional state and emotion transfer of each speaker in a conversation. This model utilizes global and local attention mechanisms to enhance the utilization of contextual information. [8] employs a graph neural network-based model and proposes GraphCRF, which is used to model the emotional labels of each sentence in a conversation and the relationship between neighboring sentences. Additionally, conditional random fields are employed for global optimization. [9] employs a pre-training language model based approach, proposing BERT-ERC, to pre-train using large-scale unlabeled text data and fine-tune on the target dataset with the objective of enhancing the model's generalizability and expressive power. In the field of emotion recognition in complex environments, cutting-edge research methods have predominantly focused on textual information. However, in practical application scenarios, the integration of surveillance images is essential. Consequently, research in the direction of machine vision is an inevitable avenue for progress in this field.

This paper addresses the challenges posed by the difficulty of application landing and the single application scene caused by the overly complex model and one-sided training data on this task. It builds upon previous work by integrating the residual connection and the deep separable structure to make the network more lightweight. Furthermore, it selects a data set that is more aligned with the real-life scene for training. Ultimately, it achieves emotion recognition in different complex scenes.

This paper focuses on a lightweight emotion recognition method based on a residual deep separable network. This method fuses two network structures from a lightweight perspective, takes an image in a complex environment as input, and obtains the recognition results of human emotions and the percentage probability of various emotions after the classifier.

2.1 Data and processing in different complex environments

The acquisition of data in disparate complex environments can be addressed by developing a dataset that encompasses data from various complex environments. In this paper, we utilize the Fer2013 dataset for data processing and training. The Fer2013 dataset is a facial expression recognition dataset, and some of its data are presented in Fig. 1, which contains 35,887 images of faces with different expressions in seven categories: angry, disgusted, fearful, happy, neutral, sad, and surprised. The dataset is larger and contains images of faces with multiple expressions and multiple scenes, which provides a rich and diverse range of data samples. This is conducive to improving the model's generalization ability and robustness. Additionally, the images in this dataset are subject to various forms of noise, occlusion, and pose change. Furthermore, they encompass a multitude of scenes, aligning with the necessity for research into diverse and intricate scenarios.

This paper employs a deep learning approach to process the data, utilizing a neural network model to extract and analyze the data features in diverse complex environments. In order to process the data obtained from different complex environments, this paper employs a model based on a single-shot multi-frame detector (SSD), which enhances the ability to capture the information features of the data in different complex environments by introducing an improved receptive field block (RFB). In the emotion classification session, a bespoke Xception model architecture is employed, comprising a convolutional layer augmented with maximum pooling and dropout techniques to extract face expression features and a fully connected layer and softmax output layer to predict different emotion categories.

2.2 Fusion of residual connections with depth separable structures

Residual networks are network structures that address the issue of gradient vanishing and degradation by incorporating cross-layer connections within each convolutional block. This enables the network to learn the residual mapping between inputs and outputs, thereby enhancing its expressive and generalization capabilities. Deep separable convolution decomposes the standard convolution into channel-by-channel convolution and point-by-point convolution, as illustrated in Fig. 2. This approach can significantly reduce the number of parameters and computation while maintaining the feature extraction effect. In this paper, different fusion methods are employed for different complex environments. In the standard mode, a residual connection is added to each depth-separable convolutional block, establishing a direct path between the input and the output. This enhances the information flow and gradient back propagation, effectively resolving the issue of gradient vanishing and degradation. It enables the network to be trained in a deeper and more stable manner. In order to enable the network to better learn the details and relationships of features in scenes requiring more attention to detail, this paper adds two residual connections to each depth-separable convolution block in the fusion method. One of these connects the input and output of the channel-by-channel convolution, while the other connects the input and output of the point-by-point convolution. This enhances the information flow and the gradient back propagation in both spatial and channel dimensions. In the event of more challenging scenes, this paper introduces a reverse residual connection to each depth-separable convolution block, establishing a direct pathway between the output and the input. This enhances the feature reconstruction capability and noise immunity of the residual depth-separable network, and enables the network to better handle images in low-quality or complex environments.

This paper proposes the Better-XCEPTION model, which is based on the Xception network. It fuses the residual structure and depth-separable convolution, combining the advantages of traditional lightweight convolution. In comparison to the traditional Xception network, the Better-XCEPTION model significantly increases the number of filters, and effectively fuses ordinary convolution and depth-separable convolution through the combination of 2×2 convolution and point-by-point convolution. Following the fusion of the two, the core module of residual depth-separable is enhanced by reducing the residual depth-separable structure from four times to two times and at the end of the network. A residual depth-separable module with a lower number of parameters is incorporated in front of the layer, with a filter count of 256 to circumvent the issues of overfitting and loss instability. This design achieves finer feature extraction while adding a smaller number of parameters. Given that our network employs a deeply separable network with a gradually increasing number of channels, the traditional parameter reduction method of point-by-point convolution plus fully connected layers in CNNs is no longer applicable. However, we discovered that the combination of Global Average Pooling (GAP) at the end of the network with Softmax is an optimal solution for reducing the number of parameters and preventing the model from becoming excessively large. Following optimization and enhancement, the final number of network parameters is 410,000, while the model's performance has been significantly enhanced.

2.3 Network improvements adapted to different complex environments

This paper introduces the Mini-XCEPTION architecture in our investigation of network lightweighting. Xception's mini-architecture is based on the original network structure, with the addition of a depth-separable convolution at the end, as illustrated in Fig. 3. The structure further reduces the number of parameters by eliminating the parameters in the convolutional layer based on the elimination of the final fully-connected layer, which has approximately 58,000 parameters compared to approximately 650,000 parameters in conventional convolutional neural networks. The proposed model has 58,000 parameters, in contrast to the approximately 650,000 parameters of a conventional convolutional neural network. Based on these findings, we proceeded to further develop the Tiny-XCEPTION architecture, which is based on the Mini-XCEPTION architecture. The residual depth-separable convolution module, which is used to extract the features of the emotional information obtained from different channels, effectively reduces the number of parameters and computation, while maintaining a good feature extraction capability. Additionally, the number of filters in the convolutional layers is reduced by half. Furthermore, the number of channels in the convolution process is reduced by half, the number of extracted feature types is reduced, and the feature extraction is more inclined to global. The overall network structure contains four residual depth separable structures. For the purpose of lightweighting, the global advection pooling (GAP) is still used instead of the full connectivity layer (FC). The whole network architecture has about 18,000 parameters, which is about 80% less compared to the original architecture (Mini-XCEPTION). The number of parameters has been reduced by approximately 80% in comparison to the original architecture (Mini-XCEPTION), while maintaining the performance of the original network. With regard to the seven types of human facial expression labels, the extraction of the network model may not necessitate a great deal of attention to the details of the features. In fact, some of the more exaggerated expressions can be distinguished without a great deal of detail, which makes improving the network in this way an effective approach in the field. However, in some real-world application scenarios, such as stage performances and circus performances, It is essential that the network be able to discern subtle nuances in human facial expressions, particularly when the psychology of the characters is undergoing rapid changes. This is crucial for accurately portraying the characters' emotions and psychological states.

The results presented in Table 1 demonstrate the enhanced performance of the proposed model in comparison to the existing state-of-the-art single network architecture trained on the FER2013 dataset. Among the listed models, the Better-XCEPTION model proposed in this paper, which combines the residual module and depth-separable convolution, stands out with an accuracy of 81.36%. This is attributed to the distinctive design and architecture of the residual depth-separable structure, which enables the effective capture and emphasis of pertinent facial features, thereby facilitating precise emotion classification.

Table 1

Performance comparison on the Fer2013 dataset. The best result is highlighted in bold.
Network Models	Accuracy (%)
GoogleNet[10]	65.20
Deep Emotion[11]	70.02
Inception[12]	71.60
Ad-Corre[13]	72.03
SE-Net50[14] ResNet50[14] VGG[12] Residual Masking Network[15] LHC-NetC[16] LHC-Net[16] CNNs + BOVW[17] Segmentation VGG-19[18] Better-XCEPTION	72.50 73.20 73.28 74.14 74.28 74.42 75.42 75.97 81.36

Concurrently, in order to fulfill the necessities of assorted intricate environments for emotion recognition, this paper enhances the Xception network with lightweight and depth, thus rendering it more malleable to the scene of variable and high precision requirements. The performance of various types of enhanced network models is demonstrated in Table 2.

Table 2

Comparison of performance of various improved networks.
Network Models	Accuracy (%)	Params
Simple-CNN	57.76	642935
Simpler-CNN Better-XCEPTION Mini-XCEPTION	59.36 81.36 70.53	559671 415719 58423
Tiny-XCEPTION	62.97	18314

The preceding table illustrates that the Xception network exhibits superior performance on the Fer2013 dataset and offers greater potential for lightweighting and deepening compared to the traditional CNN network. The Tiny-XCEPTION network, derived from the Xception network, is optimized for minimal parameter count, while the Mini-XCEPTION network is designed to reduce the number of parameters while maintaining accuracy, rendering it suitable for general scenario applications. The Better-XCEPTION network has been shown to achieve state-of-the-art performance in the context of deepening improvement.

In addition, to ensure that our network model is capable of functioning effectively in a range of generalized scenarios within complex environments, we also tested it on other relatively challenging datasets, and obtained positive results. The performance comparisons on RAF-DB and FERPlus are shown in Table 3.

Table 3

Comparison of the most recent results on the RAF-DB and FERPlus databases. The best result is highlighted in bold.
Models	Params(M)	RAF-DB(AC%)	FERPlus(AC%)
RAN[19]	11.2	86.90	89.16
SCAN-CCI[20] ARM[21] TransFER[22]	70 11.2 65.2	89.02 90.42 90.91	89.42 - 90.83
APViT[23] POSTER[24] Better-XCEPTION Tiny-XCEPTION	65.2 71.8 41.6 1.8	91.98 92.05 92.22 87.03	90.96 91.62 92.18 89.20

The experimental results demonstrate that our two proposed models achieve superior results in terms of accuracy and number of parameters, respectively.

Furthermore, this paper presents the design and implementation of an emotion recognition and analysis system. This system supports the recognition of emotions expressed through facial expressions and the real-time analysis of emotions captured through a camera. Additionally, the system enables the switching of different detection models to accommodate the diverse needs of complex environments. The system operates in accordance with the configuration depicted in Fig. 4.

In the figure, real-time emotion analysis is being carried out. The faces captured by the camera are boxed out, and the percentage of all kinds of possible emotions of the characters in the figure predicted by the model can be seen. The system can be utilized to facilitate the psychological treatment of depressed children, the detection of high-risk criminal motives, and other applications. It can also serve as a reference for the development of emotion recognition technology.

This paper presents Better-XCEPTION, a deep learning framework for facial expression recognition based on a network of residual depth-separable architectures. The Better-XCEPTION model combines residual structures and depth-separable convolution techniques popular in machine vision to capture richer facial features, thereby improving the accuracy of emotion classification and reducing the number of parameters. The experimental results on the FER2013 dataset demonstrate that Better-XCEPTION outperforms existing state-of-the-art models. To address the shortcomings of current large-scale models in edge devices and complex environments, we explore the potential of lightweighting and depth of the network to meet the needs of different scenarios. Furthermore, in order to facilitate the practical application of emotion recognition technology, we have designed and implemented a real-time vision system. This system provides a reference for applications in various complex environments and serves to validate our proposed model and its improvement.

In light of the limited number of images of oriental races in the current emotion recognition dataset and the insufficient coverage of various complex environments, we intend to expand the dataset's resources to incorporate a more diverse range of face images that will cover different types and environments. Consequently, the objective is to enhance the model's resilience to variable application conditions, thereby improving its robustness.

Acknowledgement: The authors would like to thank the editors and reviewers for their valuable work.

Funding Statement: The authors received no specific funding for this study.

Author Contributions: The authors confirm contribution to the paper as follows: study conception and design: Z. -H. W. Author, C. -L. W. Author; data collection: S. -Y. T. Author; analysis and interpretation of results: Z. -H. W. Author, C. -L. W. Author, S. -Y. T. Author; draft manuscript preparation: Z. -H. W. Author, S. -Y. Q. Author. All authors reviewed the results and approved the final version of the manuscript.

Availability of Data and Materials: All data generated or analyzed during this study are included in this article and are available from the corresponding author upon reasonable request.

Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.

H. Liu, Y. Zhang, Y. Li, et al., “Review on Emotion Recognition Based on Electroencephalography,” Frontiers in Computational Neuroscience, vol. 15, 758212, 2021.
Y. Wu, L. Shen, “An adaptive landmark-based attention network for students’ facial expression recognition,” in Proc. 6th Int. Conf. on Communication, Image and Signal Processing (CCISP 2021), pp. 139-144, 2021.
X. Li, R. Yue, W. Jia, et al., “Recognizing students’ emotions based on facial expression analysis,” in Proc. 11th Int. Conf. on Information Technology in Medicine and Education (ITME 2021), pp. 96-100, 2021.
J. Ye, Y. Yu, G. Fu, et al., “Analysis and recognition of voluntary facial expression mimicry based on depressed patients,” IEEE Journal of Biomedical and Health Informatics, 2023.
D. Zhang, F. Gao, “The psychological basis and application of emotion recognition,” Advances in Psychological Science, vol. 22, no. 6, pp. 995-1003, 2014.
P. Zhao, X. Li, “A survey of multimodal emotion recognition,” Journal of Computer Science and Exploration, vol. 13, no. 3, pp. 405-416, 2019.
N. Majumder, D. Hazarika, A. Gelbukh, et al., “DialogueRNN: An attentive RNN for emotion detection in conversations,” in Proc. AAAI Conf. on Artificial Intelligence, vol. 33, pp. 6818-6825, 2019.
Z. Zhang, W. Wu, H. Li, “GraphCRF: A graph neural network based approach for emotion recognition in conversation,” in Proc. EMNLP, pp. 7629-7638, 2020.
S. Ghosh, N. Majumder, “BERT-ERC: A pre-trained model for emotion recognition in conversation,” in Proc. First Workshop on NLP for Conversational AI, pp. 103-108, 2019.
P. Giannopoulos, I. Perikos, I. Hatzilygeroudis, “Deep learning approaches for facial emotion recognition: A case study on fer-2013,” Advances in Hybridization of Intelligent Methods: Models, Systems and Applications, pp. 1-16, 2018.
S. Minaee, M. Minaei, A. Abdolrashidi, “Deep-emotion: Facial expression recognition using attentional convolutional network,” Sensors, vol. 21, no. 9, 3046, 2021.
Y. Khaireddin, Z. Chen, “Facial emotion recognition: State of the art performance on fer2013,” 2021.
A. P. Fard, M. H. Mahoor, “Ad-corre: Adaptive correlation-based loss for facial emotion recognition in the wild,” IEEE Access, vol. 10, pp. 26756-26768, 2022.
A. Khanzada, C. Bai, F. T. Celepcikay, “Facial expression recognition with deep learning,” arXiv preprint, arXiv:2004.11823, 2020.
L. Pham, T. H. Vu, T. A. Tran, “Facial expression recognition using residual masking network,” in Proc. 25th Int. Conf. on Pattern Recognition (ICPR), pp. 4513-4519, 2021.
R. Pecoraro, V. Basile, V. Bono, “Local multi-head channel selfattention for facial expression recognition,” Information, vol. 13, no. 9, 419, 2022.
M. I. Georgescu, R. T. Ionescu, M. Popescu, “Local learning with deep and handcrafted features for facial expression recognition,” IEEE Access, vol. 7, pp. 64827-64836, 2019.
S. Vignesh, M. Savithadevi, M. Sridevi, et al., “A novel facial emotion recognition model using segmentation VGG-19 architecture,” Information, vol. 13, no. 9, 419, 2022.
K. Wang, X. Peng, J. Yang, D. Meng, and Y. Qiao, “Region attention networks for pose and occlusion robust facial expression recognition,” IEEE Transactions on Image Processing, vol. 29, pp. 4057–4069, 2020.
D. Gera and S. Balasubramanian, “Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition,” Pattern Recognition Letters, vol. 145, pp. 58–66, 2021.
J. Shi, S. Zhu, and Z. Liang, “Learning to amend facial expression representation via de-albino and affinity,” arXiv preprint arXiv:2103.10189, 2021.
F. Xue, Q. Wang, and G. Guo, “Transfer: Learning relation-aware facial expression representations with transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3601–3610, 2021.
F. Xue, Q. Wang, Z. Tan, Z. Ma, and G. Guo, “Vision transformer with attentive pooling for robust facial expression recognition,” IEEE Transactions on Affective Computing, 2022.
C. Zheng, M. Mendieta, and C. Chen, “Poster: A pyramid cross-fusion transformer network for facial expression recognition,” arXiv preprint arXiv:2204.04083, 2022.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Fusing Deep Separable Networks for Emotion Recognition in Complex Environments

Status:

Version 1

Abstract

Figures

1 Introduction

2 Emotion Recognition Based on Residual Depth Separable Networks

2.1 Data and processing in different complex environments

2.2 Fusion of residual connections with depth separable structures

2.3 Network improvements adapted to different complex environments

3 Comparative Analysis and Visualization of Experimental Results

4 Conclusion

Declarations

References

Additional Declarations

Status:

Version 1