Participants
Twenty-two healthy, right-handed participants (13 f/9 m) were recruited for the study. The average age of the study population was 26.4 ± 2.6 years as the inclusion criteria comprised a narrow age-group from 20 to 40 years to minimize the effect of changes in EEG power due to age [28]. Participants had a usual (work-) day routine of going to bed before 11 p.m. and waking up between 5 and 7 a.m. None of the included participants had worked nightshifts for a year or traveled to a different time zone during the two weeks prior to participation. Volunteers consuming drugs or nicotine were excluded from the study. During the experiment, caffeine consumption was prohibited, hence participants stating to feel a physical or psychological effect of caffeine withdrawal after 24 hours of caffeine absence were not considered in the study. The study was approved by the ethical review board of Graz University of Technology and conducted in accordance with the Declaration of Helsinki. Prior to the study, each participant gave a written informed consent. The study was conducted at the laboratory of the Institute of Neural Engineering of Graz University of Technology.
Study procedure
The study procedure was designed in such a way as to mimic a usual working day including demanding tasks after the first and second measurement sessions. These tasks required active engagement from participants in both geometric and linguistic games. After the third recording session, dinner was served that was standardized for each participant and consisted of a pizza of choice. During the last two breaks between sessions, participants were given tasks to induce some sense of tiredness, as encountered in real-life work scenarios. These tasks involved watching a documentary as well as listening to music. Figure 1a shows the timetable of a study day that started at 12 p.m. and finished at 1.30 a.m. after midnight. The experiment consisted of six consecutive measurement sessions performed every two hours at 2 p.m., 4 p.m., 6 p.m., 8 p.m., 10 p.m. and 12 a.m (midnight).
Each session lasted approximately one hour and was composed of several tasks (see Fig. 1b): (i) the EEG electrode impedance check, (ii) a psychomotor vigilance task (PVT) [29], (iii) three questionnaires targeting emotions using the Positive and Negative Affect Schedule (PANAS) [30], (iv) hunger with the Hunger Level Scale developed by the NEMO (Nutrition Education Materials Online) Management Group of Queensland Government, (v)tiredness by making use of the Tiredness Symptoms Scale (TSS) [31], (vi) a 2 min resting EEG recording, (vii) one 6 min eye run to simultaneously record the participant’s EEG and electrooculography (EOG) while performing blinks and eye movements for later eye artifact attenuation (viii) the experimental gesture EEG paradigm as described below and (ix) another 2 min resting EEG recording.
In total, 20 complete sets of six measurement sessions were obtained. Due to technical incidents, one measurement session had to be discarded for two participants.
Gesture EEG Paradigm
The study investigated four different right-hand gestures. The selection of the number of classes was made with the intention of later assessing the efficacy of a classifier designed for a four-direction decoder as part of the INTRECOM project. Each participant was comfortably seated in front of a computer screen positioned at a distance of 50 to 60 cm while placing the right hand on a table inside a wooden box. The participant was requested to follow the instructions shown on the computer screen. In addition, the participant was asked to avoid eye blinking and swallowing during the duration of a trial period. The gesture EEG paradigm was designed based on work of Ofner et al. [32]. At the beginning of each trial, the class cue was presented for 1 s corresponding to one of four classes: fist, pistol, pincer grasp and Y-gesture of the American sign language (see Fig. 2). The detailed timing of one trial can be seen in Fig. 3. Here, a fixation cross in front of a filled green circle with a smaller inner white circle was displayed after the cue for a random time interval of 0.5 to 1 s. The participant was asked to focus their gaze firmly on the cross to avoid eye movements. The presentation of the fixation cross was followed by the preparation period of 2 to 3 s in which the green circle started to shrink with random speed towards the size of the inner white circle. When the filled green circle touched the inner white circle, the participant was instructed to perform the gesture corresponding to the previously shown class cue. The participant was then asked to hold the indicated gesture for approximately 3 s until the screen went black indicating the end of a trial. Between trials, a break of 1.5 s allowed the participant to go back to the initial position and prepare for the next trial. The total duration of a trial period varied between 8 and 9.5 s.
The gesture EEG paradigm consisted of eight movement runs of approximately 5 min with 30 s of break in between. Each run was composed of 32 trials (random cues, balanced), resulting in 256 recorded trials per session and per participant. During the 30 s breaks the participants were instructed to fill out a visual analogue scale (VAS) to assess their level of fatigue.
During each session and before the gesture EEG paradigm, an eye run was performed to simultaneously collect the participant’s EEG and EOG activity while at rest with eyes open, capturing eye movements such as blinks, vertical and horizontal eye movements [33]. The collected data was then used to train an eye artifact attenuation algorithm as described by Kobler et al. [33, 34].
Questionnaires
During each of the sessions prior to the EEG gesture paradigm, participants were asked to complete questionnaires evaluating their state in terms of emotion, hunger, tiredness, and fatigue. These questionnaires were either given in English or translated to German depending on the participant’s preferences. To assess the participant’s emotions, the Positive and Negative Affect Schedule (PANAS) [30] was used, consisting of twenty different emotional states (e.g., interested, excited, upset) rated on a scale from 1 to 5 (1 being low and 5 being high). The PANAS score was divided into a positive and negative affect score by calculating the average of the ten positive and ten negative emotions, respectively. A non-standardized questionnaire introduced by the NEMO Management Group of Queensland Government was used to evaluate the participants' level of hunger. The questionnaire listed several physical sensations of hunger, and participants were asked to pick the most suitable one on a scale from 1 (indicating starvation) to 10 (indicating a full, uncomfortable feeling in the stomach). Furthermore, various symptoms of tiredness were evaluated using the Tiredness Symptoms Scale (TSS) [31] listing fourteen symptoms rated from 0 to 14 (with 0 indicating no impact and 14 indicating a significant impact). In addition, during each of the seven 30 s breaks between paradigm runs, the participants were instructed to rate their level of fatigue using a visual analogue scale (VAS) in the range of 0 (not at all fatigued) to 10 (extremely fatigued).
Recording
For the EEG measurements, 60 active, gel-based electrodes (actiCAP Brain Products GmbH, Germany) were placed on the scalp according to the 10–10 international electrode standard setup covering frontal, central, parietal, occipital and temporal areas. Four additional active electrodes were used to measure the electrical activity induced by the dipolar behavior of the eyes and were positioned at the outer canthi of the eyes as well as on the inferior and superior of the left eye. The reference and ground electrode were placed on the right mastoid and the forehead at the position of FPz, respectively. The EEG and EOG signals were recorded at 500 Hz using biosignal amplifiers (BrainAmp, Brain Products GmbH, Germany).
A video camera was placed above the participant’s right hand inside a box, with a green marker attached to the nail of the right index finger to enable movement tracking by a motion capture system developed at the institute. The sampling frequency of the camera was 30 Hz.
Preprocessing
The recorded signals were processed using MATLAB R2022b (MathWorks. Massachusetts, USA) and the open source toolbox EEGLAB [35]. At first, the raw EEG signals were inspected visually, channels contaminated by noise were identified and interpolated by the four closest electrodes weighted by their inverse distance to the bad channel. Further, the 50 Hz line noise and its first harmonic at 100 Hz were removed using a Butterworth bandstop filter of 2nd order. A Butterworth highpass filter of 5th order with a cutoff frequency of 0.3 Hz was employed to avoid stationary artifacts such as drifts. High frequency noise above 70 Hz was attenuated using a Butterworth lowpass filter (of order 70). In order to remove EOG artifacts such as blinks or saccades, an eye artifact attenuation model was trained on the pre-recorded and processed eye run data of each participant using the SGEYESUB algorithm, as described by Kobler et al. [34]. The most frontal row of electrodes (AF) was not included in following processing steps to minimize the impact of residual artifacts related to eye movements. Pops and drifts in the EEG signals were attenuated using the HEAR algorithm [36]. Subsequently, temporal electrodes (FT7 and FT8, T7 and T8, TP7 and TP8) were excluded due to their higher noise levels compared to the other channels and their lack of relevant information.
Movement-related cortical potentials
MRCPs were extracted using a lowpass Butterworth filter (4th order) with a cut-off frequency of 3 Hz. The data were segmented into epochs of 5.5 s, ranging from 2.5 s before to 3 s after movement onset. The movement onset was obtained from simultaneous recordings of the movement of the right index finger using a motion capture system that extracted the x- and y-coordinates of the green marker. By calculating the speed and applying thresholding, the onset of the movement was detected. In order to remove bad trials to reduce the impact of transient artifacts, trial rejection based on a threshold of ± 100 µV was performed on the broad-band data. These trials were then removed from the lowpass-filtered epochs. After downsampling the epochs to 9 Hz, the data was re-referenced to a common average reference.
Analysis of MRCPs
In order to examine temporal variations in the dynamics of MRCPs, we first assessed whether MRCPs associated with a particular gesture exhibited significantly greater deviations over time compared to those associated with other gestures. Therefore, we averaged epochs of a session and a specific gesture to obtain the MRCPs at each channel for every participant. The negative peak of the motor potential and the positive peak of the post-motor potential were then extracted. Their relative values were calculated by comparing them to those obtained during the initial measurement session at 2 p.m. In order to investigate variations among gestures and across different sessions, a Kruskal-Wallis test [37] was conducted with a statistical significance level of 5%. To address the issue of multiple comparisons, p-values were adjusted following the Benjamini-Hochberg procedure [38].
Amplitude of motor and post-motor potential
To examine potential deviations in MRCPs over time regardless of the executed gesture, the amplitudes of both the motor and post-motor potentials were isolated from the average MRCPs across all four gestures for each measurement session. A Wilcoxon signed rank test [39] was then employed to assess statistical significance between every pair of sessions, using a significance level of 0.05. As before, the method of Benjamini-Hochberg [38] was applied to correct the p-values for multiple comparisons.
Temporal evolution of MRCP dynamics
For further examination of temporal changes in MRCP dynamics, a Wilcoxon rank sum test [39] was employed to compare MRCP patterns between every pair of sessions. Therefore, epochs of all four gestures and all participants were combined and statistical testing was performed for each channel and at each timepoint of the movement epoch. To control for the false discovery rate introduced by multiple testing of sessions, channels and timepoints, as previously, Benjamini-Hochberg procedure [38] was implemented.
Source Localization
To obtain the brain sources associated with MRCPs, we made use of the Brainstorm toolbox provided by Tadel et al. [40]. We computed the head models by employing boundary element methods in accordance with the MNI/Colin27 anatomy template included in Brainstorm and the general EEG electrode positions embedded in EEGLAB. For the estimation of the noise covariance matrix, we estimated the average covariance of 1 s intervals from the first preprocessed resting EEG recording of each session across conditions. As input, we used epochs within the time interval from − 2.5 to 3 s around the movement onset and within the frequency range of 0.3 to 70 Hz sampled at 500 Hz. These epochs were averaged across conditions and participants. For in-depth analysis of specific frequency bands, we bandpass filtered the signals with a stopband attenuation of 60 dB in the frequency range of delta (0.5–3.5 Hz), theta (4–7.5 Hz), alpha (8–12.5 Hz), beta (13–29.5 Hz) and gamma (30–70 Hz) using Brainstorm. Subsequently, we used the standardized low-resolution brain electromagnetic tomography (sLORETA) to estimate the 15002 brain sources with dipole orientations perpendicular to the cortical surface [27]. As described in Pereira et al. [21], we retrieved the significantly different voxels with respect to a baseline, from − 1.75 to 1.5 s relative to the movement onset, by implementing a parametric t-test. To address the issue of multiple comparisons, Benjamini-Hochberg correction [38][21, 38] was applied for the number of voxels, timepoints, measurement sessions and frequency bands to adjust the p-values.
Classification
For the classification of the four gestures, a multiclass shrinkage linear discriminant analysis (sLDA) [41, 42] classifier was applied to the extracted epochs. As input, causal windows of 1 s were shifted along the epochs in steps of 1/9th of a second. Feature extraction windows of 1 s had previously yielded promising results in Ofner et al. [32]. Classification was performed offline and individually on each participant. To investigate whether variations in EEG during movement tasks over time influence the decoding capability of a classifier, five decoding strategies were employed.
To assess the general decoding capability of a classifier being trained on data collected at various timepoints throughout the day and night, two approaches were investigated. Strategy 1 made use of a 5-fold cross-validation by combining four folds of each session as a training set. The trained classifier was then applied to the remaining fold of each measurement session. Strategy 2 was based on a leave-one-out approach by using five sessions as training set and the remaining session as testing set. This strategy was applied to each one of the six sessions by training on all sessions except for the corresponding one used as a test set. Two of the participants did not have a complete set of measurements, therefore only twenty participants were considered for this approach.
To assess the performance of a classifier in the context of continuous data inflow, three training schemes were employed. Strategy 3 considered a 5-fold cross-validation within each session individually. Strategy 4 used an incremental number of sessions as training set and subsequently applied the model on the next measurement session in chronological order (i.e., train on session 1 and test on session 2, then train on session 1 and 2 and test on session 3 etc.). Strategy 5 combined the first two sessions as a training set and evaluated the model on the remaining four sessions. As stated previously, for the fourth and fifth training scheme two participants had to be excluded as their datasets were incomplete.