EEG-based Emotional Valence and Emotion Regulation Classification: A Data-centric and Explainable Approach

doi:10.21203/rs.3.rs-3129216/v1

Download PDF

Article

EEG-based Emotional Valence and Emotion Regulation Classification: A Data-centric and Explainable Approach

https://doi.org/10.21203/rs.3.rs-3129216/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Emotion classification using electroencephalographic (EEG) data is a challenging task in the field of Artificial Intelligence. While many researchers have focused on finding the best model or feature extraction technique to achieve optimal results, few have attempted to select the best methodological steps for working with the dataset. In this study, we applied two different theoretical approaches based on the noise of the dataset: curriculum learning and confident learning. Curriculum learning involves presenting training examples to a machine learning model in a specific order, starting with easier examples and gradually increasing in difficulty. This approach has been shown to improve model performance. Confident learning, on the other hand, is a method for identifying and correcting label errors in datasets. By identifying and correcting these errors, confident learning can improve the performance of machine learning models trained on noisy datasets.

Our aim was to explore the impact of different models and methods on emotion classification performance using EEG data. We used an EEG dataset in which participants rated the emotional valence of pictures while performing an emotion regulation (ER) task, comparing a control condition (Look) with two ER strategies: cognitive reappraisal and expressive suppression. We performed a multilabel classification to identify emotional neutrality or polarization of emotional valence (positive or negative) rated by participants and the emotion regulation strategy adopted during the task. We compared the performance of models trained on three datasets selected based on label noise and evaluated their suitability for this task. We then applied the Integrated Gradient technique to each model in order to assess the explainability of each model. Our results suggest different patterns based on the architecture used for feature importance, highlighting both advantages and criticisms.

Biological sciences/Neuroscience

Biological sciences/Neuroscience/Cognitive neuroscience

Artificial Intelligence for Emotion Recognition in EEG

Artificial intelligence (AI) has made significant progress in recent years, enabling the development of systems that can accurately perform tasks such as speech and image recognition. One area of AI that has received particular attention is emotion recognition, which aims to identify and classify human emotional states from various input modalities, giving birth to a new branch of AI, i.e., affective computing¹.

The first attempts in this field concerned recognizing the emotions of people based on their facial expressions ^{2 3}, or more recently, many natural language processing (NLP) models have been developed to assess the emotional valence of words or sentences ⁴.

Electroencephalography (EEG), a neurophysiological technique that measures the spontaneous electrical activity of the brain, has proven to be useful in the field of emotion recognition⁵ due to its sensitivity to emotional changes⁶. For instance, studying emotions through physiological signals can be particularly useful for individuals who have difficulty expressing emotions through facial expression or speech, such as people with traits in autism spectrum disorders⁷. Thus, having a tool that can translate emotions into feedback that is understandable by therapists or parents could be extremely helpful.

However, classifying EEG data can be challenging due to several factors. EEG signal not only contains actual brain activity but, if not properly preprocessed, is also affected by noise and artifacts⁸. Additionally, the EEG signal is non-linear⁹ meaning that linear equations may have limited effectiveness in modelling it. EEG signal is also non-stationary⁹, meaning that its statistical properties vary over time. This can make it difficult for models trained on temporally limited EEG data to generalize at different times or for different people. Finally, there is high inter-subject variability in the EEG signal, which can drastically affect the performance of a model when evaluating different subjects¹⁰.

The consequence of these features, taken together, is represented by the struggle to have good results with machine or deep learning techniques.

One common approach to simplify the input data is feature extraction. This involves extracting relevant features from the data, such as time domain features like event-related potentials (ERPs)¹¹. For example, some studies have employed the amplitude (P200, P300) and latency (N100, N200) of visual ERPs (positive and negative voltage deflections) as features linked to emotions^12,13. Other commonly used features are those in the frequency domain, which can be identified through power spectrum analysis. Indeed, some useful elements that can be utilized to determine the emotion from the EEG signal have been discovered through power spectrum analysis¹⁴. In one recent paper¹⁵, authors extracted five frequency bands and used a Random Forest (RF) model on those extracted features, achieving an accuracy of 70% in classifying emotional valence. Another study by Pachori and colleagues¹⁶, classified emotions (negative, neutral or positive) achieving a result of 94% exploiting the decomposition of a multivariate emotion EEG signal into sub-band signals using the Fourier-Bessel series expansion (FBSE) based empirical wavelet transform (FBSE-EWT)¹⁷

Deep Learning methods for Emotion Recognition

Although feature extraction can be a promising approach for simplifying EEG data, another more challenging method involves using raw data as input to the model. In this case, the architecture and composition of the model itself are used to extract the most important features. Convolutional neural networks (CNN) are often used for this purpose¹⁸ as they are capable of initially extracting both local, low-level features and global, high-level features from raw input data¹⁹. Results achieved when using this approach in a binary classification of emotional valence are often considered good when the accuracy is less than 70%¹⁸, even when using feature extraction techniques. Another possible approach not often used in the literature is classifying emotional valence by using the combination of CNN and Recurrent Neural Networks (RNN). In fact, RNNs have an internal state that allows them to learn from long-term dependencies and temporal patterns in the data²⁰. In a recent work²¹, the authors used a particular type of RNN, a Long Short Term memory network (LSTM)²², particularly efficient to avoid the vanishing gradient problem, which occurs when the gradients of the loss function with respect to the weights of the earlier layers become very small, and handling long sequences²³. With this method, Researchers reached an accuracy around 70% for the emotional valence.

One promising approach could be using a CNN + GRU (Gated Recurrent Unit), which has been scarcely used in emotion classification. However, analysis with EEG sleep data²⁴ suggests that this type of model could be particularly useful for these purposes.

Recently, researchers have begun to use Transformer neural networks²⁵ for EEG analysis. Originally developed for natural language processing²⁶, these models have also been adapted to time series as input data ²⁷ and, more recently, EEG data. A recent study²⁸ achieved impressive results by using Transformer networks to classify mental overload and predict age and gender from raw EEG data, achieving an accuracy of around 90%. However, there are currently no studies on the use of Transformer for emotion classification.

Both feature extraction and raw signal methods have their advantages and disadvantages. Feature extraction results are usually more accurate, but the signal is extensively processed, making it difficult to use for a future Brain Computer Interface (BCI) machine. BCI could be a great asset for some individuals in particular cases such as interventions for autistic people²⁹ that can express their emotions in a misleading way for the therapist³⁰. On the other hand, using the raw signal produces less accurate results, but this method could be more useful in this context.

Emotion Regulation

In this paper, we used an EEG emotion regulation (ER) task to classify the perceived emotional valence of emotional pictures and the ER strategy adopted. We asked participants to assess the emotional valence of 60 pictures while adopting two ER strategies (and a control condition). We used a novel approach based on identifying the best methodology to classify EEG data after minimal preprocessing.

ER refers to the “extrinsic and intrinsic processes responsible for monitoring, evaluating, and modifying emotional reactions, especially their intensity and duration”³¹. It plays an important role in everyone’s life: many studies highlighted the correlation between healthy ER strategies and social and affective adaptation^{32 33}, how it affects decision-making³⁴ and coping with stress³⁵ or the severity of symptoms in conditions such as Post-traumatic stress disorder (PTSD)³⁶ or Attention deficit hyperactivity disorder (ADHD)³⁷. Also, the typical state that characterizes mood and anxiety disorders often depends on emotion dysregulation³⁸.

Two of the most studied ER strategies are cognitive reappraisal and expressive suppression³⁹. Cognitive reappraisal is an antecedent-focused ER strategy, refers to the attempt to reinterpret an emotion-eliciting situation in a way that changes its meaning and emotional impact³³,⁴⁰. Expressive suppression, on the other hand, is a response-focused strategy and can be defined as the attempt to hide, inhibit, or reduce ongoing emotion-expressive behaviour (such as facial expressions, verbal utterances and gestures)⁴¹.

Data-centric approach

Since emotion and ER classification are still challenging tasks for machine learning models, the first aim of our study is to use a data-centric approach to feed the machine learning model the best possible data based on input EEG data.

For what concerns the data-centric approach, the first technique we used was confident learning. This method identifies and corrects label errors in any dataset using any model⁴². Confident learning can help models avoid learning from unreliable or inconsistent labels, which can degrade their accuracy and generalization. In the same vein, we also used curriculum learning. This is a training strategy that orders data samples from easy to hard, based on several criteria such as label noise or the ambiguity of the examples given⁴². The concept of curriculum learning was first introduced by Bengio and colleagues in a data-centric framework⁴³, who proposed a method for training neural networks by presenting training examples in a predefined order of difficulty. This approach was inspired by the idea of curriculum development in education, which involves designing a sequence of learning experiences that gradually build on one another. Since then, curriculum learning has been extensively studied in the machine learning literature, with researchers exploring a variety of different strategies for selecting the order in which training examples are presented⁴⁴. Curriculum learning can help models focus on simpler concepts first and gradually progress to more complex ones, without being overwhelmed by noise or ambiguity.

Indeed, our assumption is that both assessing the emotional valence and adopting the right ER strategy can be difficult for a naïve subject, so a promising way to approach this problem is studying not only the EEG variation but also the sanity of the data and the label assigned.

Explainable AI (XAI)

The second aim of this study was to use explainable AI (XAI) to study how different models make classification decisions with respect to emotion and ER strategy classification of raw EEG data. With regard to XAI, we employed the Integrated Gradient (IG) approach ⁴⁵ to study the deep learning models we trained. IG requires no modification to the original network (Model Agnostic Approach) and is extremely simple to implement. Its objective is to assign an attribution score that underlines how each input feature is positively or negatively related to each prediction. It is based on two axioms: (i) Sensitivity: If one feature change makes the classification output change, then that feature should have a non-zero attribution. This makes sense because if a feature makes the output change, then it must have played a role. (ii) Implementation Invariance: The attribution method result should not depend on the specificities of the neural network architecture ⁴⁵.

Hypothesis

Our hypothesis is that adopting those two novel approaches for emotion recognition, namely data-centric approach and XAI, could lead to less biased networks, improve their learning, and understand how the predictions are made. Curriculum learning could help models focus on simpler concepts first and gradually progress to more complex ones, without being overwhelmed by noise or ambiguity. Confident learning could help models avoid learning from unreliable or inconsistent labels, which can degrade their accuracy and generalization. IG could enhance the understanding of how the predictions are made by different models. In this study, we compared four different architectures: CNN, CNN + RNN, CNN + bidirectional RNN, and Transformer. In summary, in this paper, we propose a novel approach that combines curriculum learning, confident learning and IG for emotion classification, using different models, while adopting two ER strategies, i.e., expressive suppression and cognitive reappraisal, and a neutral strategy that implies just looking at the stimuli.

Behavioural results

Table 1 shows the results of the mixed effect linear model performed on the emotional valence ratings reported by participants. In particular, the estimated marginal means (emmean), standard errors (SE), asymptotic lower and upper confidence limits (asymp.LCL, asymp.UCL), z-ratio, and p-value for each of the comparisons are shown. The mixed-effects linear model on participants’ ratings showed a significant interaction effect (χ² = 1106.7, df = 40, p = < 0.001) between the mindset and the emotional valence. Post-hoc multiple comparisons (corrected according to Tukey’s HSD) based on the different mindsets showed that looking at negative images was associated with lower (i.e., more negative) ratings than reappraising or suppressing them (see Table 1, statistically significant effects are in bold), while there was no significant difference between reappraising and suppressing negative images. On the other hand, looking at positive images was associated with higher emotional valence than reappraising or suppressing them, while there was no significant difference between reappraising and suppressing positive images. These results showed the overall effectiveness of the two emotional regulation strategies in modulating the emotional impact of the images.

Table 1

Results of the mixed effect linear model.
CONTRAST	emmean	SE	asymp.LCL	asymp.UCL	z.ratio	p.value
LOOK N - REAPPRAISE N	-0,527	0,106	-0,849	-0,204	-4.965	<.0001
LOOK N - SUPPRESS N	-0,469	0,118	-0,826	-0,112	-3.990	0.0009
REAPPRAISE N - SUPPRESS N	0,058	0,179	-0,486	0,602	0.323	0.9995
LOOK P - REAPPRAISE P	0,379	0,129	-0,014	0,771	2.932	0.0395
LOOK P - SUPPRESS P	0,352	0,087	0,088	0,616	4.052	0.0007
REAPPRAISE P - SUPPRESS P	-0,027	0,138	-0,446	0,392	-0.193	1

Figure 1 shows the mean and standard deviation of ratings in each mindset for each emotion.

Confident Learning

The quality of data labels was evaluated using Random Forest (RF) as a benchmark model for the confident learning approach. Separate RF models were trained for emotional valence and ER strategy as target classes. All results reported refer to the dataset after applying the SMOTE algorithm, a data augmentation technique.

Emotional Valence

As shown in Table 3, when classifying participants’ ratings (from 1 to 7) in seven classes, the model performed with a F1-score of 0.15 (random chance = 0.14). (see Table 2). The labels that were best predicted were 4 and 5, with an F1-score of 0.19 and 0.18, followed by label 3 with an F1-score of 0.15. The labels that were least accurately predicted were the most positive ones, 6 and 7, with an F1-score of 0.10 and 0.08.

Table 2

Classification report on the validation set of random forest applied to the emotional valence of the pictures with seven classes (one for each possible rating, from 1 to 7).
	Precision	Recall	F1-score	N.examples
1	0.11	0.14	0.12	70
2	0.11	0.15	0.13	89
3	0.15	0.15	0.15	117
4	0.27	0.14	0.19	183
5	0.18	0.17	0.18	118
6	0.10	0.11	0.10	66
7	0.06	0.12	0.08	32
accuracy			0.14	675
macro avg	0.14	0.14	0.13	675
weighted avg	0.17	0.14	0.15	675

In order to simplify the classification problem and to balance the sample of each class, we then classified the emotional valence dividing it into three classes, i.e., negative, neutral and positive. The results reported in Table 3 of the random forest on the validation set show that the class with higher F1-score was the neutral one, with a F1-score of 0.64 and a general accuracy of 0.48. This may be the result of the unbalanced dataset, having a very represented neutral class and two other classes with fewer examples.

Table 3

Classification report on the validation set of the Random Forest applied to emotional valence of the pictures, with three classes (Negative, Neutral and Positive).
	Precision	Recall	F1-score	N.examples
Negative	0.26	0.20	0.23	159
Neutral	0.65	0.62	0.64	418
Positive	0.17	0.26	0.20	98
accuracy			0.47	675
macro avg	0.36	0.36	0.36	675
weighted avg	0.49	0.47	0.48	675

Given the unbalanced dataset, we then tried to classify data into two classes by merging positive (i.e., 6 and 7) and negative samples (i.e., 1 and 2) in a “polarized” class. Table 4 reports the results of a binary classifier that showed a more balanced result, having an F1-score of 0.39 for the polarised class and 0.64 for the neutral one. Even the precision was more balanced, having respectively 0.40 and 0.63 for the polarised and neutral classes.

Table 4

Classification report of the validation set of random forest applied to the emotional valence of the pictures with two classes (Neutral and Polarised)
	Precision	Recall	F1-score	N.examples
Polarised	0.40	0.38	0.39	257
Neutral	0.63	0.65	0.64	418
accuracy			0.55	675
macro avg	0.51	0.51	0.51	675
weighted avg	0.54	0.55	0.54	675

Emotion Regulation

For what concerns the ER strategy classification, results can be observed in Table 5. The accuracy of 0.33. The precision and F1-score were also similar for all classes, indicating that the model had a balanced performance across the classes. The model was slightly better at predicting the class Reappraise, which had the highest F1-score of 0.36. The model was slightly worse at predicting the class Suppress, which had the lowest F1-score of 0.30. The N.examples column shows that the validation set was balanced, with equal amounts of data for each class.

Table 5

Classification report of the validation set of the Random Forest applied to ER Strategies
	Precision	Recall	F1-score	N.examples
Look	0.33	0.32	0.33	225
Reappraise	0.33	0.39	0.36	229
Suppress	0.33	0.28	0.30	221
accuracy			0.33	675
macro avg	0.33	0.33	0.33	675
weighted avg	0.33	0.33	0.33	675

At this point, we began our analysis by training deep neural networks with different architectures using a curriculum learning approach. With this aim, we divided the dataset into three parts based on label noise computed with Cleanlab and for each deep learning architecture we created three deep learning models, trained, separately, with the easy part of the dataset, the easy and medium parts combined, and on the entire dataset. We then used an ensemble learning approach to combine the predictions of these three models to obtain the final result.

Convolutional Neural Networks

Table 6 shows the classification report for the MINI-VGG (Visual Geometry Group), a convolutive neural network with a shallow architecture.

When classifying valence and ER strategy the model obtained a weighted average F1-score of 0.42 on the test set. The model shows better performance in classifying emotional valence, with the highest F1-score for the Neutral class at 0.55, followed by the Polarised class with an F1-score of 0.51. The model performs poorly in recognizing ER strategies. Indeed, the class Look is the best classified, with a F1-score of 0.36, while Reappraise and Suppress classes showed an F1-score of 0.27 and 0.29 respectively.

Table 6

Classification report on the test set of MINI-VGG.
	Precision	Recall	F1-score	N.examples
Neutral	0.60	0.50	0.55	384
Polarised	0.44	0.60	0.51	232
Look	0.34	0.40	0.36	302
Reappraise	0.30	0.24	0.27	228
Suppress	0.27	0.32	0.29	226
weighted avg	0.42	0.43	0.42	1372

Convolutional Neural Networks + Recurrent Neural Networks

Table 7 shows results of the MINI-VGG with GRU layers. The model showed a weighted average F1-score of 0.43. Once again, the highest F1-score is for the Neutral class at 0.57, while the Polarised class showed an F1-score of 0.41. The ER classes totalized an F1-score of 0.31, 0.39 and 0.39 for Look, Reappraise and Suppress respectively.

Table 7

Classification report on test set of MINI-VGG with GRU
	Precision	Recall	F1-score	N.examples
Neutral	0.54	0.61	0.57	384
Polarised	0.46	0.38	0.41	302
Look	0.30	0.32	0.31	232
Reappraise	0.33	0.48	0.39	228
Suppress	0.32	0.49	0.39	226
weighted avg	0.41	0.47	0.43	1372

Table 8 shows results of the MINI-VGG with bidirectional GRU layers. The model showed a weighted average F1-score of 0.43. In this case the best performance concerned the Polarised class, having 0.50 points of F1-score, followed by Neutral with 0.43 points. The ER classes showed a 0.30, 0.43 and 0.42 F1-score for Look, Reappraise and Suppress respectively.

The ER strategies performed better than every other model tested, having F1-score of 0.44, 0.38 and 0.36 for Look, Reappraise and Suppress, respectively.

Table 8

Classification report on test set of MINI-VGG with bidirectional GRU
	Precision	Recall	F1-score	N.examples
Neutral	0.57	0.34	0.43	384
Polarised	0.44	0.58	0.50	302
Look	0.32	0.70	0.44	232
Reappraise	0.35	0.41	0.38	228
Suppress	0.31	0.41	0.36	226
weighted avg	0.42	0.48	0.43	1372

Transformers

Table 9 shows the results of the Transformer architecture. The model showed a weighted average F1-score of 0.28. In this case the best performance regards the Polarised class, having 0.56 points of F1-score, followed by neutral with 0.53 points. Each of the ER classes showed an F1-score of 0.02.

Table 9

Classification report on test set of Transformer model
	Precision	Recall	F1-score	N.examples
Neutral	0.57	0.50	0.53	384
Polarised	0.46	0.70	0.56	302
Look	0.33	0.01	0.02	232
Reappraise	0.23	0.01	0.02	228
Suppress	0.20	0.01	0.02	226
weighted avg	0.39	0.30	0.28	1372

Explainability (Integrated Gradient)

In Fig. 2, we reported an example of a matrix of the IG attribution score for each EEG feature (channels x time) in every deep learning model considered. We find that each model showed a similar pattern of explainability for every predicted class. The CNN architecture reported a scattered and fuzzy attribution importance pattern across all features. Similarly, for both GRU and bidirectional GRU models, a scattered layout limited to the initial or final time parts of the data was underlined. On the other hand, the Transformer architecture evidenced a defined pattern, with some channels showing a more stable and consistent influence over time. Specifically, the attribution gradient evidenced lower average values compared to the other architectures but more distributed among all features.

The color scale ranges from dark blue to yellow, representing positive and negative attribution scores (feature importance) respectively. All figures for other classes and classifications (i.e., True Negative, False Positive, False Negative) are reported in the supplementary materials.

In this study, we employed a multilabel classification approach to classify the emotional valence of visual stimuli and emotion regulation strategies utilized by participants while looking at those visual stimuli, based on participants’ EEG signal. The aims of our study were to provide a methodological framework to handle raw EEG data and provide reliable results in this context. Therefore, the first aim was to prepare the dataset for the deep learning models training via a data-centric approach. To reach this objective, we combined curriculum learning⁴³ and confident learning ⁴² techniques. Subsequently, in the second aim we compared how different deep learning models took classifying decisions by using an XAI technique, namely IG ⁴⁵.

First of all, we selected the proper labels based on the confident learning approach, thus resulting in two classes for emotional valence classification (i.e., neutral vs. polarised) and three classes for ER strategy (i.e., Look, Reappraise and Suppress).

To adopt the curriculum learning approach, we used an ensemble approach with three models having the same architecture. The only change was the training dataset, giving the three different models different training based on the dataset difficulties.

For the multilabel classification with 5 classes and 6 different combinations (i.e., 2 valence * 3 ER strategy classes) we chose four different models: a MINI-VGG, a MINI-VGG with GRU, a MINI-VGG with bidirectional GRU and a Transformer. In general, the best classes performance was neutral and polarised. Regarding the ER strategies, the models still cannot predict those strategies properly.

Our results are interesting even considering that combining CNN with RNN, especially uni- and bidirectional GRU, is quite a unexplored approach in the domain of raw EEG data classification ⁴⁶.

Our intuition was that the neurons of the long-term memory recurrent neural network have the benefit of memorizing both the long-term and short-term emotional information present in the EEG signal, facilitating the recognition of emotions.

Considering the architectures explainability, (i.e., studying the attributions of the gradients of each model) we realised that RNN tends to memorize shorter sequences, and that an array with 64x500 time points is too long to be learned.

In Fig. 2 it can be seen that unidirectional GRU tend to consider only the last 100 ms of the input, while the bidirectional GRU only the first and last 100 ms. Further studies combining Convolutional and Recurrent architectures should therefore test shorter time series compared to our dataset (e.g., 50–100 ms). The CNNs, instead, seem to consider the whole data, but at the same time, the pattern of the importance of the feature is scattered, and it makes the interpretation very difficult. It looks like there are no specific channels or time points to influence the outcome of the classification. Based on IG, the Transformer is the model showing the most interpretable patterns. This model showed patterns of specific channels that can be positively or negatively correlated with the prediction of the model over time. It is also interesting to notice that the Transformer model has the best score for the emotional valence domain and the worst in the ER strategy one. This could be further evidence suggesting no specific patterns in raw EEG activity for different ER strategies.

We believe that our work is valuable for both research and applied purposes for two main reasons. Firstly, while most research in the literature focuses primarily on finding the best model by evaluating classic performance measures⁴⁶ (e.g., F1-score, accuracy), we trust that research should focus on data first. In this vein, we have provided a robust methodological solution based on a data-centric approach for dealing with raw EEG data, which can be particularly challenging⁹ since artifacts in the signal can affect deep learning models.

Secondly, classic performance measures, which are widely used in the literature to evaluate deep learning models, do not reveal how these black box models make decisions. For this reason, we believe that it is always appropriate to adopt an XAI technique to turn models in explainable white boxes, rather than solely discussing their performance. This approach is valuable not only in choosing the best model for a classification problem, but also in refining the architecture accordingly. For instance, in using the CNN + GRU architecture, we observed a temporal limit in the time series memorization only thanks to the XAI technique. Each step (i.e., data-centric approach and XAI) in our methodological framework could be useful to researchers and practitioners to study a specific dataset and build a personalised model that could be used for a real-time BCI.

Also, it should be considered that there are many reasons for this result, in spite of the strict methodological and theoretical background of our work.

Indeed, choosing to divide the dataset into polarised and neutral is, from our point of view, an interesting approach. As shown in Fig. 1, several ratings for every condition range between 3 and 5, so the dataset itself hinted at studying the neutral values. Besides that, our model may be better suited for a particular kind of emotion recognition, meaning that we do not aim to recognize positive or negative emotions as many studies do⁴⁷,⁴⁸,¹⁸.

Those findings suggest that even after considering possible issues with label noise and the best training for the models, there is still work to do to improve those deep learning models, especially for the emotion regulation strategies adopted. By the way, the performance of emotional valence classification is fairly good, especially in Transformer models, when considering that data were processed only for filtering the signal and rejecting bad epochs to approximate a possible BCI approach and that we were using a multilabel classifier⁴⁹.

The interesting advantage of this approach is that it would be even more useful in everyday life, since a great part of the stimuli do not necessarily have strong emotional valence: everyday objects carry subtle affective valences, defined by some authors⁵⁰ as “micro-valences”, which are intrinsic to their perceptual representations.

Besides, the novel aspect of this study is also the classification of the ER strategy, which has never been investigated in the field of artificial intelligence. However, in spite of all methodological checks, classification performance is poor in ER strategies: there are several reasons for these results. We first need to point out that we did not choose the train, validation and test set by randomly selecting epochs from all the participants, but we split the data based on random subjects. Kamrud⁵¹ and colleagues have mathematically shown that cross-participant models, where samples are randomly taken from any subject, tend to have underestimated error rates between 35% and 3900%, thus overestimating the model’s performance. This is related to several reasons: first, EEG varies across participants due to non-stationarity and individual differences⁵². Thus, splitting the data based on subjects ensures that the training, validation, and test sets are independent and reduces the risk of overfitting. Randomly selecting epochs from all participants implies that data from the same subject could appear in both the training and test sets. This could lead to biased performance, as the model may learn to recognize specific characteristics of individual subjects rather than generalizing to new data. Second, splitting the data based on subjects allows us to evaluate the generalizability of our model to new subjects. By training our model on data from one set of subjects and testing it on data from a different set of subjects, we can assess how well our model is able to classify emotional valence in new individuals.

Still, the main limitation of this work is that our models do not outperform the current state-of-the-art, but we have adopted a challenging and specific dataset (EEG raw data), without any real benchmark in model performance. Since each example is potentially affected by artifacts, it is reasonable to obtain lower performance compared to a pre-processed EEG dataset. However, we obtained fairly good performance in emotional valence classification and this research line is potentially more useful in creating models classifying real-time data with minimal preprocessing (i.e., BCI).

Future directions could consist of applying and comparing the effectiveness of this data-centric and XAI approach to different classification problems using EEG raw data. For example, this work could be a seminal step to find solutions when dealing with large artifacts such as movement. Moreover, using a dataset with more observations, or trying transfer learning from another dataset could increase the models performance.

We also think that the state-of-the-art literature may show biased results considering the problem of cross-participant training, validation, and test split. Indeed, out of six EEG deep learning models currently used in research, five are cross-participant models⁴⁶. However, only one of these five models follows proper dataset partitioning methods to ensure that the model is tested on data from participants it has not evaluated before⁴⁶.

For these reasons, we believe that, despite the improvable performance, this work represents an important advance in the field of emotion classification using EEG, allowing a new methodological approach to disentangle the common issues of label noise and overestimated performance.

Participants

We collected data from thirty paid participants (17 females, mean age 26 ± 6.12. Our exclusion criteria comprised the presence of a history of any neurological or psychiatric disease, use of active drugs, abuse of any drugs (including nicotine within 2 hours preceding the study and alcohol within 24 hours preceding the study), as well as being informed about the aim of the study. Written informed consent was obtained from all participants according to the declaration of Helsinki; the IMT Ethical Committee approved the project.

Materials

We selected sixty stimuli from the Oasis database⁵³. In this database, 900 images are present, rated by 822 participants. Every image was rated via a Likert scale from 1 to 7 according to their valence and arousal.

Our aim was to have a small dataset with positive emotional valence for half of the stimuli and negative for the other half. We selected images that respected three criteria: (i) choosing images with no sensitive contents (i.e., sexual or violent scenes) based on the indications of the Ethical Committee; (ii) selecting images with high arousal values (> 4) in order to clearly elicit emotions; (iii) including images with the highest or lowest validation valence values to use strongly polarized positive and negative stimuli.

Overall, we included 60 stimuli, i.e., 30 positive images with mean ± sd valence and arousal respectively of 5.7 ± 1.06 and 4.7 ± 1.6, and 30 negative images with mean ± sd valence and arousal values of 2.1 ± 1.1 and 4.7 ± 1.8, respectively.

In order to check that the stimuli had the same valence polarization for our participants, we compared valence scores for all images between the validation dataset and scores assigned by participants in this study. Even if 21 images showed a significant difference between the means in independent samples t-tests, it is also shown in Fig. 3 that none of these stimuli showed an inversion of positive vs. negative valence, i.e., positive stimuli with mean valence ratings above 4 (= neutral valence) in the validation dataset showed mean valence ratings above 4 also in our dataset, and vice-versa for negative stimuli below 4.

Procedure

After signing the informed consent module, participants were asked to sit on a comfortable chair in an electrically shielded and soundproof room while not crossing their legs or arms. A 19” monitor was positioned in front of them, 1 mt distant.

The EEG cap was then prepared on the participant’s head.

Resting state activity was then recorded for five minutes, but it will not be used in this study.

Experimenters explained to the subjects the experimental procedure before the experiment started. Moreover, written instructions were shown on the monitor to give them the information they needed.

The experimental design is shown in Fig. 4. The experiment consisted of three blocks in which participants were asked to assess the emotional valence of 60 pictures shown on the monitor, while adopting three different emotion regulation strategies: ‘Look’, ‘Reappraise’ and ‘Suppress’, for a total of 180 stimuli seen during the experiment.

Each mindset was explained with written instructions in the beginning of each of the three blocks, which were shown in a counterbalanced order. Participants were free to ask the experimenters information about the different ER strategies in any moment via a microphone.

Once participants understood what mindset they had to adopt, they could press a key to begin the visualization and rating of the stimuli. The mindset was also reminded during the whole experimental block by showing a small text (i.e., ‘LOOK’, ‘REAPPRAISE’ and ‘SUPPRESS’) above the pictures.

Each stimulus, which was shown for 1000 ms, was preceded by a fixation cross of 1000 ms. After the stimulus visualization, participants were asked to rate the emotional valence of the picture, according to the mindset they adopted in that block, on a Likert scale ranging from 1 (absolutely negative) to 7 (absolutely positive).

Data is available for review at the following link https://osf.io/yv468/?view_only=eda5f52314d24fa88e1a0b3d8a9b401e.

EEG recording and preprocessing

The neurophysiological activity of each participant was recorded with a 64-channel EGI EEG system. Electrodes were positioned according to the 10–20 International System. The online reference was Cz and the sampling frequency was 1000 Hz with an impedance below 50KΩ.

All the pre-processing was performed using the MNE-Python package.

The raw signal was filtered (1–48 Hz bandpass with a 50 Hz notch filter. We used the RANSAC (random sample consensus) method⁵⁴ to detect bad channels. The signal was then epoched: each epoch began 500 ms before the stimulus onset lasting 1000 ms. We then normalized the epoched signal considering a baseline from 500 ms to 200 ms before the stimulus onset and cropped the signal before the appearance of the stimulus. At this point, each epoch lasted 500 ms. The total was 180 epochs for each subject. We used the Autoreject algorithm⁵⁵ to identify and reject bad epochs. We set the consensus from 0.2 to 0.5 in steps of 0.1. The consensus corresponds to the proportion of bad channels that are allowed in order to accept the epoch. If the number of bad channels allowed exceeds the real number of bad channels, the algorithm rejects the epoch, otherwise it can interpolate the signal from 1 to 8 bad electrodes. The algorithm chooses the best value possible for each subject. The average number of rejected epochs was 9.16 ± 9.97 per participant. Then the signal was re-referenced offline to the average of all channels.

Data Preprocessing

To prepare the data for analysis, we randomly assigned data from 22 participants to the training set, 4 participants to the validation set, and 4 to the test set. We also applied Principal Component Analysis (PCA)⁵⁶ to reduce the dimensionality of the training set and balance the classes. We selected 100 principal components that explained 87% of the total variance. We verified that the inverse PCA matrix was highly correlated with the original matrix (i.e., distance correlation coefficient⁵⁷ = 0.99). We then used SMOTE⁵⁸(Synthetic Minority Over-sampling Technique) to augment the data in the training set based on the principal components and reconstructed the original matrix shape by applying inverse PCA.

SMOTE works by selecting examples that are close to the feature space. For each instance in the minority class, SMOTE calculates the k-neatest neighbors. It then selects one or more of these nearest neighbors and generates synthetic examples by interpolating between the selected neighbor and the original instance. In this function, the k parameter has to be set because, when a random example from the minority class is first chosen, k nearest neighbors for that example are selected in order to decide where it can draw the line. We chose k = 5 because, in the literature, it is the most frequently used value⁵⁹.

After preprocessing the data, we included 64 channels and a time window of 500 milliseconds. With a sample frequency of 1000 Hz, the shape of each example was (1, 64, 500).

Data Analysis

Behavioural

Statistical behavioral tests were conducted using R⁶⁰We adopted the standard 0.05 Alpha significance level to test against the null hypotheses. In particular, we used Linear Mixed-Effects Models (lmer) package⁶¹. We fitted a linear mixed-effects model testing how the rating of the images differs between mindsets (baseline, reappraisal or suppression) and emotional valence assessed a priori (negative or positive) while accounting for the variation across subjects and file images. The model has three fixed effects (mindset, emotional valence of the stimulus, and their two-way interaction) and two random effects (subject and file image). We then used Estimated Marginal Means (emmean) package⁶² to calculate the contrasts between each condition and correct for multiple comparisons using the Tukey method⁶³.

Data-centric approach

In order to adopt a data-centric approach with confident and curriculum Learning, we used random forest (RF) algorithms.

A random forest (RF) is an ensemble learning method that consists of a collection of decision trees: it is called a "forest" because it is made up of many decision trees⁶⁴.

A decision tree's fundamental goal is to classify or predict data by iteratively splitting the data based on the values of particular attributes⁶⁵. For instance, in this dataset, the random forest could split the data for each electrode or time point. The model incorporates randomness by randomly choosing the splitter and each tree from various random subsamples of data. Every node in the tree indicates a split in the data, and the leaf node is where the ultimate prediction is formed.

Random forests are often used in the literature because they reduce overfitting due to the fact that every decision tree is trained on different subsets of the data and then the predictions are averaged together. We used this algorithm as benchmark for deep learning to calculate the sanity of the labels (according to the confident learning approach) for several reasons: (i) as mentioned above, the model is very good at reducing overfitting; (ii) the random forest is one of the few algorithms that calculates the class weights, and it usually works properly with non-linear data such as EEG. In particular, we used the Cleanlab package. Cleanlab is an open-source framework for machine learning and analytics with noisy data. It provides methods to identify, quantify, and correct errors in datasets, measure and track dataset quality, and train reliable models with noisy labels. Cleanlab is based on rigorous theoretical foundations based on the work of Northcutt and colleagues⁴².

To create different datasets based on both the difficulty of the samples (curriculum learning) and the correct classification classes (confident learning), we trained two kinds of random forests (RFs). One type of RF was used to classify the emotional valence of the images and the other to classify the ER strategies.

We used three criteria to prepare our dataset in the most methodologically sound way: (i) ensuring that the dataset was as balanced as possible, (ii) making theoretical assumptions about both emotional valence and emotion regulation strategy, and (iii) simplifying the classification problem.

To achieve this, we trained the first RF for emotional valence with 7 classes, corresponding to the 7-point Likert scale used for assessment. The second RF had 3 classes (positive, neutral, and negative) and the third RF had just two classes (neutral and polarised).

The other type of RF was used to classify the ER strategy of the participants into 3 classes, corresponding to the 3 emotion regulation strategies. We then calculated the class overlap for each model to determine the best labels for our dataset for multilabel classification of both emotional valence and emotion regulation. We first selected the best label match for emotional valence and ER strategy based on the above-mentioned criteria. Thus, we decided to use a binary classification for emotional valence (polarized and neutral), while maintaining the three original classes for emotion regulation strategy. Then, we measured the label quality for each example and averaged it with emotional valence and mindset. We divided the dataset into three parts based on the 33rd and 66th percentiles of the distribution: easy, intermediate, and difficult.

Deep Learning Architectures

In this study, we used four different ensemble deep learning architectures for our multilabel classification with five classes (Neutral, Polarised, Look, Reappraise, Suppress): 1) MINI-VGG, 2) MINI-VGG with GRU, 3) MINI-VGG with bidirectional GRU and 4) A Transformer model.

Multilabel classification involves N non-exclusive labels (in our case, N = 5). Each label is considered as a binary classification problem whose predicted probability is independent with respect to the other classes.

This multilabel classification approach allows us to avoid using three separate models (one for each ER strategy) and, therefore, training our data on emotional valence on a dataset split in three parts. Additionally, given our experimental protocol, it was necessary for two labels to be present simultaneously (i.e., one label of emotional valence and one for ER strategy).

To make predictions, we used an ensemble majority voting classifier and trained three different models based on the difficulty of the training data. Each model was trained using the principles of curriculum learning. We started by training one model on easy examples, another one on easy and medium examples, and the third one trained on the entire dataset. We used this approach, as Bengio suggested⁴³, because easy data are more easily learned by the model. Indeed, by training three models on datasets of increasing difficulty, we aimed to provide the ensemble models with a broader view of EEG signals, avoiding biased predictions.

In an ensemble majority voting classifier the prediction with the highest number of votes was selected, thus a prediction was confirmed if at least two out of the three classifiers voted for it. We used a Sigmoid function in the last layer of each model, thus obtaining in output a prediction for each label independently. For each prediction, we set a threshold of 0.5 to classify the presence (i.e., probability > 0.5) or absence of the variable.

Regarding the specific deep learning architectures, we used a variant with 1D CNN of the MINI-VGG of Kranthi and colleagues ⁶⁶ because some previous works⁴⁶ demonstrated that shallow deep learning architectures achieve better performance on EEG data as features for classification tasks.

The MINI-VGG we used consisted of two 1D convolutional blocks followed by a classification block. The convolutional blocks were composed of two convolutional layers, each with ReLU activation. In the first block, the convolutional layers used 32 filters with a kernel size of 3, while in the second block, 64 filters were used with a kernel size of 3. At the end of each block a pooling layer performed max pooling operation over a window of 2. There was always a batch normalization after the convolutional one. After each pooling a dropout layer was used. The dropout value was taken as 0.15. Finally, the output was passed through a Dense layer with sigmoid activation function to produce the final predictions.

With regards to the MINI-VGG + GRU and MINI-VGG + bidirectional GRU, we refer to the modification of the MINI-VGG by including three GRU (unidirectional or bidirectional) layers with 128 units and a tanh activation function after the second convolutional block. Instead of dropout we used Monte Carlo Dropout (MCDropout). The output of the GRU blocks was then passed through a Batch Normalization layer and another MCDropout with a dropout percentage of 0.15. Finally, the output was passed through a Dense layer with sigmoid activation function to produce the final predictions.

For the Transformer, the model consisted of two encoders and a classification block. Each encoder block applies layer normalization and multi-head attention with 64 head size and 2 heads to the inputs, followed by a dropout with rate of 0.4 and a normalization. Then, two convolutional layers with 64 filters with a kernel with size of 1 are applied. Only the first convolutional layer used a ReLU activation function. Between the two convolutional layers a dropout with a dropout percentage of 0.15 was applied. The output of the final transformer block is then passed through a global average pooling layer and a series of dense layers with 128 units, ReLU activation, and a dropout rate of 0.4. The final output is produced by a dense layer with 5 units and a sigmoid activation function.

For every model we used Adam optimizer⁶⁷ with learning rate of 1e-06.

For a graphical representation of each architecture employed, see Supplementary Materials.

We trained the model with a default of 20000 epochs, but then we also used the earlystopping callback from Keras⁶⁸ (with a patience of 20 epochs), taking as a benchmark the loss functions of the validation set.

Convolutional Neural Networks

Convolutional neural networks (CNN), which are a type of neural network used for multidimensional data⁶⁹. The main component of the CNN are the convolutional layers.

A convolutional layer applies a set of filters (also called kernels or weights) to the input data. Every filter is a matrix applied to the data to produce a new feature map of the local region of the data in which has been applied. The process of applying a filter to the input data is called convolution. The input data is usually a multidimensional array. The filters are also multi-dimensional arrays smaller in size than the input data⁷⁰.

The convolutional layer applies each filter to the input data by sliding it over the input data and performing an element-wise multiplication with the values at each position. It then sums the results and stores the result in a new feature map. This process is repeated for each filter, producing multiple feature maps.

The output of the convolutional layer is a multi-dimensional array called a feature map, which has the same number of dimensions as the input data (e.g., height, width, and depth) but with the size of the feature map is usually smaller than the input data because the filters are smaller and do not overlap.

The convolutional layer may also have parameters called stride and padding, which control how filters are applied to the input data. Stride controls the step size with which the filters are applied, and padding controls how the input data is padded with additional values around the border before the filters are applied.

One of the main issues with convolutional layers is that the feature map generated by the filter is dependent on its location. As a result, during training, convolutional neural networks learn to associate the presence of specific features with their location in the input image.

To avoid this problem the pooling layers are generally used in the convolutional networks⁷¹.

The CNN we are going to use is the MINI-VGG that derives from the VGG that is a convolutional neural network architecture that was introduced by the Visual Geometry Group (VGG) at the University of Oxford in 2014 ⁷².

Recurrent Neural Networks

Recurrent neural networks (RNNs) are a type of artificial neural network that can handle sequential data or time series data, such as speech, text, or video ⁷³. Unlike feedforward neural networks, which assume that the input and output data are independent of each other, RNNs have feedback connections that allow them to store and reuse information from previous computations. This means that RNNs have connections between their hidden units that form loops, so that the output of a unit can influence its own input in the next time step^20,73. This allows RNNs to store some information from previous inputs in their hidden state, which can be used to process the current input. This gives RNNs the ability to learn from long-term dependencies and temporal patterns in the data. However, RNNs also face some challenges, such as vanishing or exploding gradients and difficulty in handling long sequences⁷⁴. To address these issues, several variants of RNNs have been proposed, such as long short-term memory (LSTM)⁷⁵ or gated recurrent unit (GRU)⁷⁶. They both use a gating mechanism to control the flow of information and avoid the vanishing gradient problem that affects standard RNNs. The main difference between them is that GRU has two gates (reset and update) while LSTM has three gates (input, output and forget). GRU is simpler and faster than LSTM, but LSTM has more flexibility and accuracy on larger datasets⁷⁷.

Also, GRUs and be unidirectional or bidirectional: a unidirectional GRU processes the input sequence in one direction, while a bidirectional GRU processes the input sequence in both forward and backward directions. This allows the model to capture both past and future context when making predictions.

Transformer networks

The Transformer network²⁵ is a type of neural network that utilizes a self-attention mechanism to achieve high-quality results and lower computational requirements for language translation tasks compared to recurrent and convolutional models. This mechanism allows the model to directly compare all parts of the input, regardless of their position, and assign an attention score to each part based on its relevance to the current task. The Transformer network is composed of multiple encoders and decoders. The encoders are stacked on top of each other in the model, with each encoder containing two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. Residual connections are present around both sub-layers, followed by a normalization layer. Since we are working on time series, the decoder is not used in this case, as in other cases in literature²⁸.

Unlike recurrent neural networks (RNNs) and LSTMs, Transformer networks do not have an inherent way to capture the relative positions of the input elements. To provide this contextual information, positional encoding is used in conjunction with each input vector. Positional encoding is not part of the model architecture itself but rather a pre-processing step. A positional encoding vector is generated for each input element and added to its corresponding embedding vector. This allows the model to learn spatial information from the “injected” pattern in the embedding vector.

Integrated Gradients

In our implementation, we used the Integrated Gradients (IG) method to explain the predictions of our model. IG is an interpretability technique method originally proposed in Sundararajan ⁴⁵ for deep neural networks that visualizes the importance of input features in relation to the model’s predictions.

In particular, integrated gradients defines an attribution value for each feature by considering the integral of the gradients taken along a straight path from a baseline instance x’ to the input instance x. Since we are using classifiers, the gradient usually refers to the output corresponding to the true class or to the class predicted by the model.

We used Alibi⁷⁸, that is an open source library in Python. We used a null baseline and the number of steps we used was 25.

We used the explain method of this instance to compute the attributions for our selected examples, passing in the examples, baselines, and target as arguments. The attributions were then obtained from the attributions attribute of the returned explanation object.

Picard, R. W. Affective computing. (MIT press, 2000).
Chakraborty, A., Konar, A., Chakraborty, U. K. & Chatterjee, A. Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 39, 726–743 (2009).
Wang, Y., Ai, H., Wu, B. & Huang, C. Real time facial expression recognition with adaboost. in Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004. vol. 3 926–929 (IEEE, 2004).
Kang, Y., Cai, Z., Tan, C.-W., Huang, Q. & Liu, H. Natural language processing (NLP) in management research: A literature review. Journal of Management Analytics 7, 139–172 (2020).
Bos, D. O. EEG-based emotion recognition. The influence of visual and auditory stimuli 56, 1–17 (2006).
Apicella, A., Arpaia, P., Mastrati, G. & Moccaldi, N. EEG-based detection of emotional valence towards a reproducible measurement of emotions. Scientific Reports 11, 21615 (2021).
Yirmiya, N., Kasari, C., Sigman, M. & Mundy, P. Facial expressions of affect in autistic, mentally retarded and normal children. Journal of Child Psychology and Psychiatry 30, 725–735 (1989).
Lai, C. Q. et al. Artifacts and noise removal for electroencephalogram (EEG): A literature review. in 2018 IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE) 326–332 (IEEE, 2018).
Klonowski, W. Everything you wanted to ask about EEG but were afraid to get the right answer. Nonlinear biomedical physics 3, 1–5 (2009).
Clerc, M., Bougrain, L. & Lotte, F. Brain-computer interfaces 1: Methods and perspectives. (John Wiley & Sons, 2016).
Jenke, R., Peer, A. & Buss, M. Feature extraction and selection for emotion recognition from EEG. IEEE Transactions on Affective computing 5, 327–339 (2014).
Frantzidis, C. A. et al. Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli. IEEE transactions on Information Technology in Biomedicine 14, 589–597 (2010).
Ding, R., Li, P., Wang, W. & Luo, W. Emotion processing by ERP combined with development and plasticity. Neural plasticity (2017).
Atkinson, J. & Campos, D. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Systems with Applications 47, 35–41 (2016).
Torres, E. P., Torres, E. A., Hernández-Álvarez, M. & Yoo, S. G. Emotion recognition related to stock trading using machine learning algorithms with feature selection. Ieee Access 8, 199719–199732 (2020).
Pachori, R. B. A Novel Multivariate-Multiscale Approach for Computing EEG Spectral and Temporal Complexity for Human Emotion Recognition. (2021).
Bhattacharyya, A., Singh, L. & Pachori, R. B. Fourier–Bessel series expansion based empirical wavelet transform for analysis of non-stationary signals. Digital Signal Processing 78, 185–196 (2018).
Hu, J. et al. ScalingNet: extracting features from raw EEG data for emotion recognition. Neurocomputing 463, 177–184 (2021).
Albawi, S., Mohammed, T. A. & Al-Zawi, S. Understanding of a convolutional neural network. in 2017 international conference on engineering and technology (ICET) 1–6 (Ieee, 2017).
Medsker, L. R. & Jain, L. C. Recurrent neural networks. Design and Applications 5, 64–67 (2001).
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F. & Chandran, V. Automatic emotion recognition using temporal multimodal deep learning. IEEE Access 8, 225463–225474 (2020).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural computation 9, 1735–1780 (1997).
Graves, A. & Graves, A. Long short-term memory. Supervised sequence labelling with recurrent neural networks 37–45 (2012).
SM, I. N., Zhu, X., Chen, Y. & Chen, W. Sleep stage classification based on eeg, eog, and cnn-gru deep learning model. in 2019 IEEE 10th International Conference on Awareness Science and Technology (iCAST) 1–7 (IEEE, 2019).
Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems 30, (2017).
Chernyavskiy, A., Ilvovsky, D. & Nakov, P. Transformers:“the end of history” for natural language processing? in Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part III 21 677–693 (Springer, 2021).
Liu, M. et al. Gated transformer networks for multivariate time series classification. arXiv preprint arXiv:2103.14438 (2021).
Siddhad, G., Gupta, A., Dogra, D. P. & Roy, P. P. Efficacy of transformer networks for classification of raw EEG data. arXiv preprint arXiv:2202.05170 (2022).
Fan, J. et al. A Step towards EEG-based brain computer interface for autism intervention. in 2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC) 3767–3770 (IEEE, 2015).
Eack, S. M., Mazefsky, C. A. & Minshew, N. J. Misinterpretation of facial expressions of emotion in verbal adults with autism spectrum disorder. Autism 19, 308–315 (2015).
Thompson, R. A. Emotion regulation: A theme in search of definition. Monographs of the society for research in child development 25–52 (1994).
Gross, J. J. Emotion regulation in adulthood: Timing is everything. Current directions in psychological science 10, 214–219 (2001).
John, O. P. & Gross, J. J. Healthy and unhealthy emotion regulation: Personality processes, individual differences, and life span development. Journal of personality 72, 1301–1334 (2004).
Heilman, R. M., Crişan, L. G., Houser, D., Miclea, M. & Miu, A. C. Emotion regulation and decision making under risk and uncertainty. Emotion 10, 257 (2010).
Wang, M. & Saudino, K. J. Emotion regulation and stress. Journal of Adult Development 18, 95–103 (2011).
Ehring, T. & Quack, D. Emotion regulation difficulties in trauma survivors: The role of trauma type and PTSD symptom severity. Behavior therapy 41, 587–598 (2010).
Seymour, K. E. et al. Emotion regulation mediates the relationship between ADHD and depressive symptoms in youth. Journal of abnormal child psychology 40, 595–606 (2012).
Thompson, R. A., Lewis, M. D. & Calkins, S. D. Reassessing emotion regulation. Child Development Perspectives 2, 124–131 (2008).
Cutuli, D. Cognitive reappraisal and expressive suppression strategies role in the emotion regulation: an overview on their modulatory effects and neural correlates. Frontiers in systems neuroscience 175 (2014).
Lazarus, R. S. & Alfert, E. Short-circuiting of threat by experimentally altering cognitive appraisal. The Journal of Abnormal and Social Psychology 69, 195 (1964).
Gross, J. J. & Levenson, R. W. Emotional suppression: physiology, self-report, and expressive behavior. Journal of personality and social psychology 64, 970 (1993).
Northcutt, C., Jiang, L. & Chuang, I. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research 70, 1373–1411 (2021).
Bengio, Y., Louradour, J., Collobert, R. & Weston, J. Curriculum learning. in Proceedings of the 26th annual international conference on machine learning 41–48 (2009).
Soviany, P., Ionescu, R. T., Rota, P. & Sebe, N. Curriculum learning: A survey. International Journal of Computer Vision 130, 1526–1565 (2022).
Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. in International conference on machine learning 3319–3328 (PMLR, 2017).
Roy, Y. et al. Deep learning-based electroencephalography analysis: a systematic review. Journal of neural engineering 16, 051001 (2019).
Kasuga, Y., Shin, J., Hasan, M. A. M., Okuyama, Y. & Tomioka, Y. EEG-based Positive-Negative Emotion Classification Using Machine Learning Techniques. in 2021 IEEE 14th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC) 135–139 (IEEE, 2021).
Stikic, M., Johnson, R. R., Tan, V. & Berka, C. EEG-based classification of positive and negative affective states. Brain-Computer Interfaces 1, 99–112 (2014).
Aggarwal, S. & Chugh, N. Review of machine learning techniques for EEG based brain computer interface. Archives of Computational Methods in Engineering 1–20 (2022).
Lebrecht, S., Bar, M., Barrett, L. F. & Tarr, M. J. Micro-valences: perceiving affective valence in everyday objects. Frontiers in psychology 3, 107 (2012).
Kamrud, A., Borghetti, B. & Schubert Kabban, C. The effects of individual differences, non-stationarity, and the importance of data partitioning decisions for training and testing of EEG cross-participant models. Sensors 21, 3225 (2021).
Cohen, M. X. Analyzing neural time series data: theory and practice. (MIT press, 2014).
Kurdi, B., Lozano, S. & Banaji, M. R. Introducing the open affective standardized image set (OASIS). Behavior research methods 49, 457–470 (2017).
Derpanis, K. G. Overview of the RANSAC Algorithm. Image Rochester NY 4, 2–3 (2010).
Jas, M., Engemann, D. A., Bekhti, Y., Raimondo, F. & Gramfort, A. Autoreject: Automated artifact rejection for MEG and EEG data. NeuroImage 159, 417–429 (2017).
Wold, S., Esbensen, K. & Geladi, P. Principal component analysis. Chemometrics and intelligent laboratory systems 2, 37–52 (1987).
Edelmann, D., Móri, T. F. & Székely, G. J. On relationships between the Pearson and the distance correlation coefficients. Statistics & probability letters 169, 108960 (2021).
Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16, 321–357 (2002).
Bennin, K. E., Keung, J. W. & Monden, A. On the relative value of data resampling approaches for software defect prediction. Empirical Software Engineering 24, 602–636 (2019).
R Core Team, R. R: A language and environment for statistical computing. (2013).
Bates, D. Fitting linear mixed models in R. R news 5, 27–30 (2005).
Lenth, R., Singmann, H., Love, J., Buerkner, P. & Herve, M. Package ‘emmeans’. (2019).
Keselman, H. J. & Rogan, J. C. The Tukey multiple comparison test: 1953–1976. Psychological Bulletin 84, 1050 (1977).
Breiman, L. Random forests. Machine learning 45, 5–32 (2001).
Safavian, S. R. & Landgrebe, D. A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics 21, 660–674 (1991).
Kranthi Kumar, K., Bharadwaj, R., Ch, S. & Sujana, S. Effective deep learning approach based on VGG-mini architecture for iris recognition. Annals of the Romanian Society for Cell Biology 4718–4726 (2021).
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
Gulli, A. & Pal, S. Deep learning with Keras. (Packt Publishing Ltd, 2017).
O’Shea, K. & Nash, R. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458 (2015).
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. in Proceedings of the IEEE conference on computer vision and pattern recognition 2818–2826 (2016).
Bailer, C., Habtegebrial, T. & Stricker, D. Fast feature extraction with CNNs with pooling layers. arXiv preprint arXiv:1805.03096 (2018).
Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
Schuster, M. & Paliwal, K. K. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing 45, 2673–2681 (1997).
Bengio, Y., Simard, P. & Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks 5, 157–166 (1994).
Yu, Y., Si, X., Hu, C. & Zhang, J. A review of recurrent neural networks: LSTM cells and network architectures. Neural computation 31, 1235–1270 (2019).
Cho, K. et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).
Mateus, B. C., Mendes, M., Farinha, J. T., Assis, R. & Cardoso, A. M. Comparing LSTM and GRU models to predict the condition of a pulp paper press. Energies 14, 6958 (2021).
Klaise, J., Van Looveren, A., Vacanti, G. & Coca, A. Alibi explain: Algorithms for explaining machine learning models. The Journal of Machine Learning Research 22, 8194–8200 (2021).

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

EEG-based Emotional Valence and Emotion Regulation Classification: A Data-centric and Explainable Approach

Status:

Version 1

Abstract

Figures

Introduction

Artificial Intelligence for Emotion Recognition in EEG

Results

Behavioural results

Confident Learning

Emotional Valence

Emotion Regulation

Convolutional Neural Networks

Convolutional Neural Networks + Recurrent Neural Networks

Transformers

Explainability (Integrated Gradient)

Discussion

Method

Participants

Materials

Procedure

EEG recording and preprocessing

Data Preprocessing

Data Analysis

Behavioural

Data-centric approach

Deep Learning Architectures

Convolutional Neural Networks

Recurrent Neural Networks

Transformer networks

Integrated Gradients

References

Additional Declarations

Supplementary Files

Status:

Version 1