Multivariate Multiscale Entropy: An Approach to Estimating Vigilance of Driver

Copyright © 2023 Kawser Ahammed et al., licensed to EAI. This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited.


Introduction
Humans possess different types of the mental states through which they interact with their surrounding complex environments. Vigilance, which means the ability to maintain more careful attention for monotonous task especially to notice possible danger, is one of the mental states of humans. Unfortunately, machines are unable to interact with surroundings and generally interact with users through a process known as Human Computer Interaction (HCI) [1]. The purpose of HCI is to provide information through converting brain activity into control signals for computers so that the machine can respond simultaneously [2]. If a brain-computer interaction system works effectively, it can assess the high vigilance of human brain.
Accidents that occur due to the loss of vigilance by drivers are common phenomena for some occupations such as driving buses, trucks, high-speed trains and air planes [1]. In this case, high vigilance analysis is necessary to prevent the occurrence of drowsiness, sleepiness or fatigue by continuously observing the driver's mental state [3]. This high vigilance can be analysed using the MMSE method.
There are various methods of estimating vigilance in the literature. These methods can be referred to as video-based, multi-sensor based and physiological signals based [4,5]. Since the implementation of both video and multi-sensor based approaches is arduous, various physiological signal based approaches, for example, eye movement [3,6] and autonomic nervous system activity [7,8,9,10], have been studied for vigilance estimation. Moreover, physiological signal based methods are considered as most fruitful, effective, and promising for vigilance estimation because most of the physiological signals are an indicator of transition between wakefulness to sleepiness [11,12].
In addition to the above methods, there are different entropy approaches [13,14] that have been studied to determine the alertness of humans. However, all of the methods have been analysed in the time domain instead of the complexity domain. For this reason, the MMSE method has been introduced and applied to a multimodal dataset [1] to characterize the vigilance of drivers from EEG and forehead EOG signals in the complexity domain.
The organization of the paper is as follows. In section 2, the materials and methods have been represented. The results have been described in section 3. The discussion and conclusion have been demonstrated respectively in section 4 and section 5.

Multivariate Multiscale Entropy
The MMSE evaluates multivariate sample entropy over different time scales and deals with the different embedding dimensions, time lags, and amplitude ranges of data channels in a rigorous and unified way. The calculation of MMSE [15,16] is performed through the following steps: (i) To define temporal scales of increasing length, apply coarse graining process to the c-variate time series {x l,i } N i=1 , l = 1, 2, ..., c, where N denotes the number of samples in each variate (channel). For a scale factor ξ, the elements of the multivariate coarse grained time series are calculated as: where 1 ≤ k ≤ N ξ . (ii) To plot multivariate sample entropy (M S En ) as a function of scale factor ξ, calculate multivariate sample entropy for each coarse grained multivariate u ξ l,k .

Multivariate Sample Entropy Calculation
The M S En is the prerequisite for performing MMSE analysis over a number of data channels. For a c-variate coarse grained time series {U ξ l,k } N ξ k=1 , l = 1, 2, ..., c, the M S En [15,16] is performed through the following steps: (i) Form (N − n) composite delay vectors U m (i) ∈ R m (m = c l=1 m l ), where i = 1, 2, ..., N − n, N = N ξ and n = max{M } × max{τ }. (ii) To determine the distance between any two composite delay vectors U m (i) and U m (j), define the maximum norm as

Selection of Parameter values
During multivariate sample entropy calculation, several parameters have to be chosen introducing their own constraints. For example, for multivariate data, the individual channel exhibits different embedding parameters m l and τ l . In addition, the threshold parameter r is set to some percentage of the standard deviation of the original signal. For multivariate sample entropy, r has been chosen with trial and error as 40% of the standard deviation of the original signal for better separation among MMSE curves.

Complexity Analysis of multivariate time series
The rule for measuring the complexity of multivariate time series [15,16] from the MMSE plots (multivariate sample entropy represented as a function of the scale factor) is given below: (i) If the sample entropy values of a multivariate time series are higher than those of the other time series for the majority of the scale factors, the multivariate time series will be more complex than another one. (ii) If the signal in hand only contains useful information at the smallest scale then the multivariate entropy values of the signal decrease monotonically with respect to the scale factors. Figure 1: Electrode Placements for the EEG setups. 12-channel and 6-channel EEG signals were recorded from the posterior site (blue color) and temporal site (red color) respectively (adapted from [1]).

Experimental Set up
The pre-processed EEG, EOG and eye movement data used in the present study were obtained from the publicly available database [1] for vigilance estimation. To gather EEG and EOG data, the experiments were accomplished in a simulated driving environment for 2 hours. Twenty one participants ( 12 males and 9 females at the age between 20 and 25) were employed to participate in the experiments. All participants were in normal vision and toxic materials that induce fatigue were prohibited before participating in the experiments. Most experiments were performed after launch to induce fatigue. A four lane high way scene was shown on a large LCD screen in front of a real vehicle. The participants were asked to drive the car through this straight and monotonous road to induce fatigue. As a result, 12-channel EEG signals (CP1, CPZ, CP2, P1, PZ, P2, PO3, POZ, PO4, O1, OZ, and O2) from posterior site and 6-channel EEG signals from temporal site (FT7, FT8, T7, T8, TP7, and TP8) were recorded simultaneously using Neuroscan system with a 1000 Hz sampling rate according to the international 10-20 system as shown in fig. 1 [1]. The 12 EEG signals were unipolar. The standard electrode set up of Neuroscan systems was followed, where the GND electrode was located posterior to FPz and the REF electrode was located between Cz and CPz. At the same time, forehead EOG signals were recorded using Neuroscan system with a 1000 Hz sampling rate according to electrode placement as shown in fig. 2 [1] and eye movement data was recorded using SMI ETG eye tracking glasses.

Data Processing
The down sampling rate of EEG signals were 200 Hz to reduce computational complexity. The EEG signals were preprocessed with a band-pass filter between 1 Hz and 75 Hz to reduce noise and artifacts. Short-time Fourier transform with a 8 s non-overlapping Hanning window was used to extract five EEG frequency bands: delta(1-4 Hz), theta(4-8 Hz), alpha (8)(9)(10)(11)(12)(13)(14), beta(14-31 Hz) and gamma(31-50 Hz). For each frequency band, the differential entropy (DE) features [1] (efficient EEG features) were extracted. The differential entropy h(X) of a continuous random variable X with probability density function p(x) can be defined as:  density (PSD) features were calculated. Since DE proposed in [17] for vigilance estimation shows superior performance compared to conventional power spectral density features, we have used it for vigilance estimation. Using the FASTICA algorithm [18], EEG and EOG components were extracted from the four forehead channels (Nos. 4-7) shown in fig. 2(b). To separate EEG and EOG signals from the shared forehead electrodes, a separation technique based on ICA (Independent Component Analysis) was used. The input to ICA was the 4-channel forehead signals. The ICs contained both EOG components and EEG components, which were both used in the data analysis. The EOG features derived from ICA were denoted VEO(vertical EOG)-ICA and HEO(horizontal EOG)-ICA. M S En was calculated from ICs. The whole data processing has been shown diagrammatically in Fig. 3.

Data Analysis Methods
In this study, different EEG/EOG combinations and statistical analysis are presented in section 3. Firstly, we have applied MMSE method to brain EEG, forehead EEG and EOG data in order to determine vigilance of driver in terms of complexity. Secondly, we have applied statistical analysis methods (t-test and ANOVA test) to brain EEG, forehead EEG and EOG data with a view to validating the MMSE method. Thirdly, we have employed MMSE method to PER-CLOS labels for estimating vigilance using cognitive states.

Measurement and Calculation of Vigilance Labeling
Different vigilance annotation methods like lane departure, local error rates were introduced by the researchers in the literature [19,20]. Among different annotation methods, the above two annotation methods were not feasible for dual tasks, particularly in real world driving environments. But there is another method called PERCLOS, which refers to the percentage of eye closure over time. It is a reliable measure of vigilance in the literature [21,22,23]. In this research, an automatic continuous vigilance annotation method has been adopted to perform PERCLOS measurement using eye tracking glasses proposed in [24]. To calculate eye tracking based PERCLOS index, the duration of different eye movements (blink, fixation and saccade) and the duration of the eye closures are measured. PERCLOS index can be calculated as follows: where 'CLOS' refers to the duration of the eye closures.
The PERCLOS values obtained using eye tracking glasses are considered as the index of alertness level. To verify PERCLOS measurement, efficiency of the eye tracking based method for vigilance annotations with the facial videos recorded simultaneously was evaluated and a high correlation between the PERCLOS index and the subject's current cognitive states was found [1].
From fig. 4(a), 4(b), 4(c) and 4(d), it is observed that for majority of the scale factors the values of the multivariate sample entropy for forehead EEG signals are higher than the temporal EEG, posterior EEG, brain EEG (fusion of temporal and posterior) and EOG signals. Due to higher multivariate sample entropy values of forehead EEG signals for majority of the scale factors, it can be said that forehead EEG signals contain correlations across multiple time scales and is, therefore, more complex than the temporal EEG, posterior EEG, brain EEG (fusion of temporal and posterior) and EOG signals.

Statistical Analysis of EEG (Brain and Forehead) and EOG Based Vigilance Estimation
This section discusses the statistical analysis performed on EEG (Brain and Forehead) and EOG data. Firstly, One way ANOVA (analysis of variance) test has been applied to EEG values with a view to finding whether there is statistically significant difference among the EEG signals or not in terms of multivariate sample entropy values. After applying ANOVA test, we have found statistically significant difference because of the ANOVA test gives the result of F = 35.03 > F crit = 3.24, effective size=0.64, p < 0.01 (null hypothesis rejection) at 2 Hz frequency resolution. This test also gives the result of F = 32.21 > F crit = 3.24, effective size=0.62, p < 0.01 (null hypothesis rejection) at 5 Hz frequency resolution.
Secondly, there is statistically significant difference among the forehead EEG, fusion of temporal and posterior EEG and EOG in terms of multivariate sample entropy values as One way ANOVA test provides F = 19.09 > F crit = 3.16, effective size=0.40, p < 0.01 (null hypothesis rejection) at 2Hz frequency resolution and F = 54.91 > F crit = 3.20, effective size=0.42, p < 0.01 (null hypothesis rejection) at 5 Hz frequency resolution.

Vigilance Estimation Based on Cognitive States
(Awake, Tired and Drowsy) This section analyses the EEG data in terms of complexity. To do this, EEG data has been categorized into three cognitive states (Awake, Tired and Drowsy) with two threshold (0.35 and 0.7) according to the PERC-LOS index [1] shown in Table 1. To analyse EEG data Table 1: Splitting of EEG data into three categorizes (awake, tired and drowsy) with two thresholds (0.35 and 0.7) according to the PERCLOS index.  fig. 5, one can state that multivariate sample entropy values of awake state are higher than both tired and drowsy state for majority of the scale factor. Two (awake and tired) of these three cognitive states overlap with each other only for scale factor 2 and 7. This indicates that these three cognitive states can be identified by MMSE algorithm. Since the values of multivariate sample entropy for awake state are higher than both tired and drowsy state except for scale factor 2 and 7, it can be said that, unlike tired and drowsy state, awake state contains correlations across multiple time scales and is, therefore, more complex than both tired and drowsy state. This fact also implies that while driving in a simulated driving environment a driver has higher vigilance in awake state rather than both tired and drowsy state.  Although the MMSE method has classified the cognitive states in complexity domain, the classification ability of the MMSE method can be justified with the SVM. To do this, 7 mean multivariate sample entropy values of awake state, 7 mean multivariate sample entropy values of tired state and 7 mean multivariate sample entropy values of drowsy state are used as the attributes of SVM for classification task. Here, we have used a 5-fold cross validation approach for measuring classification results and to avoid biased classification. To perform classification task using SVM, we have used classification learner APP of MATLAB R2016b. In this research, SVM has been used as a classifier as it provided a promising classification accuracy compared to other classifiers. Since multivariate sample entropy values were the input of the SVM, it provided a confusion matrix as shown in Table 2 for the cognitive states. After analysing confusion matrix, it can be said that cognitive states have been classified with the promising classification accuracy of 76.2%. Moreover, the awake state has been identified with sensitivity of 57.1% and specificity of 92.9%. Similarly, the drowsy state has been recognised with sensitivity of 100% and specificity of 85.7%. The tired state is detected with 71.43% sensitivity and 85.7% specificity. Besides, statistically significant difference (p <0.01) has been observed among cognitive states with One way ANOVA test.

Discussion
In this study, we have developed and applied MMSE method to EEG, EOG and eye movement data for vigilance estimation. Although several researches [25,26] have been done for on-road real driving tests, there are many other scenarios that require vigilance estimation, such as actors performance in the theatre. In future, we will apply our method to real scenarios to justify the feasibility. In addition, the data used in this study was acquired in a context of driving rather monotonous a priori (driving on an highway). The validity of the method applied to this study can be limited to this study context and may be less relevant in driving contexts of higher mental load (e.g. driving in a city). This point is the only limitation of our study.
The experimental results demonstrate that our approach can achieve comparable results with the conventional methods [27,28]. The approaches proposed in [27,28] detected the vigilance of drivers with the accuracy of 83.6% and 88.6% respectively. In this study, vigilance estimation has been performed without considering any neurofeedback. In future, neurofeedback in high vigilance task should be investigated.
In this research, we propose a novel method based on complexity science to characterize traditional EEG, forehead EEG, and EOG. From the results, it can be inferred that forehead EEG signals can give higher complexity curves and easily distinguished in complexity domain. Besides, different mental states like awake, tired or sleep states can also be distinguished in complexity domain. It is also intuitive that cognitive loads are higher in awake states than tired or drowsy states. As a result, the complexity profiles for awake states are higher as in drowsy or tired states the signals become more regular and have less information thus have lower complexity values. This complexity features can give us complementary information for future to build vigilance estimation system.

Conclusion
This paper represents a novel method (MMSE) for vigilance estimation. In this paper, characterization of the brain EEG signals, forehead EEG signals, EOG signals and cognitive states (awake, tired and drowsy) have been performed in terms of complexity and the results have been validated by statistical tests. The MMSE analysis curves have clearly shown that forehead EEG signals reveal higher complexity compared to brain EEG and EOG signals. Moreover, statistical tests (t test and one way ANOVA test) used to justify the MMSE results have shown that M S En values of awake state are significantly different from those of tired state and drowsy state. From the experimental results, we have observed that vigilance of driver decreases when the driver goes to one of the states (tired or drowsy) from awake state. The simulation results also indicate that the MMSE method can be utilized practically to monitor continuous attention.