Machine learning models in Heart Rate Variability based mental fatigue prediction: training on heterogeneous data to obtain robust models

doi:10.21203/rs.3.rs-1633975/v1

Download PDF

Article

Machine learning models in Heart Rate Variability based mental fatigue prediction: training on heterogeneous data to obtain robust models

https://doi.org/10.21203/rs.3.rs-1633975/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Prolonged period of cognitive performance often leads to mental fatigue, a psychobiological state that increases the risk of injury and accidents. Previous studies trained machine learning algorithms on Heart Rate Variability (HRV) data to detect fatigue in order to prevent its consequences. However, the results of these studies can hardly be generalized due to methodological issues including the use of only one type of cognitive task to induce fatigue that potentially leads to task-specific predictions. In this study, we combined the datasets of three experiments each applying different cognitive tasks for fatigue induction to train algorithms that detect fatigue and predict its severity. We also tested different time window lengths and compared algorithms trained on resting and task-related data. We found that classification performance was best when the support vector classifier was trained on task-related HRV calculated for 5-minutes time window (AUC = .843, accuracy = .761). For the prediction of fatigue severity, elastic net regression showed the best performance when trained on 4-min HRV data and self-reported measures (R² = .224, RMSE = 17.114). These results indicate that both detection and prediction of fatigue based on HRV are effective when machine learning models are trained on heterogeneous, multi-task datasets.

mental fatigue

machine learning

Heart Rate Variability

classification

regression

Mental fatigue (henceforward, fatigue) is a psychobiological state that is characterized by a decline in performance and a decreased motivation for effort and is usually caused by engaging in cognitively demanding activities for extended period^1,2. Fatigue is a complex state that involves changes in brain activity (e.g. norepinephrine and dopaminergic systems)^3–5, subjective feelings (e.g., lack of energy) and cognitive performance^6–8. Because fatigue tends to influence cognitive performance, it is known to be a highly relevant topic regarding human safety. For example, the risk of work-related and road accidents is substantially higher when people are fatigued^9,10. Accordingly, the prevention as well as the detection of fatigue is imperative¹¹. In line with this, research on the biomarkers of fatigue has become important and suggests that fatigue can be estimated based on markers that reflect the activity of the cerebral cortex (e.g. theta activity obtained by neuroimaging techniques) or the autonomous nervous system (e.g. Heart Rate Variability) in order to prevent its negative consequences¹².

Machine learning is a relatively novel way of utilizing biomarkers to detect fatigue and this approach has captured the attention of science as well as practice alike^13–15. A few studies used biological signals obtained by electroencephalography (EEG) to train machine learning models that are capable of effectively detecting mental fatigue (i.e.: classification models that successfully distinguish between fatigued and non-fatigued states; for review, see ¹⁶). Even though the high accuracy (> 80–90%) of these models is impressive, EEG has several limitations, such as the difficult and time-consuming procedure of setting up the electrodes and its sensitivity to external electromagnetic fields¹⁷. Due to these limitations and the fact that fatigue has also been associated with changes in the autonomic nervous system¹⁸, other studies have investigated whether fatigue detection is possible based on biological signals obtained by peripheral measures, for example, electrooculography¹⁹ or electrocardiography (ECG)²⁰. Most of the fatigue studies utilizing ECG calculated the variation of consecutive R-wave peaks denoted in the literature as Heart Rate Variability (HRV)²¹. HRV reflects the activity of the autonomic nervous system and is indeed a potential reliable biomarker of fatigue since many previous studies have confirmed the association between HRV and fatigue^18,22−24.

Machine learning studies have shown that models trained exclusively on HRV are capable of effectively detecting fatigue with the best accuracy scores ranging between 75% and 91%^20,25−27. Laurent et al. (2013), for example, showed that the support vector machine (SVM) algorithm trained on HRV data recorded during the performance of a fatiguing switching task (i.e. algorithm trained on task-related HRV data) was able to detect fatigue with an accuracy up to approx. 80%. In contrast to this study using task-related data, another tested the algorithms based on resting HRV data²⁶. In their study, the resting period prior to prolonged task performance was labelled as the non-fatigue state, while the resting period after task performance was labelled as the fatigue state. They found that the k-nearest neighbors (KNN) algorithm was capable of detecting fatigue with an accuracy of approx. 75% based on four HRV indices.

The differences in predictive accuracy reported in these studies is probably caused by various factors. The most important factors might be the methodological differences across the studies such as differences in the cognitive tasks used for fatigue induction, the time window used for HRV calculation, the sample size differences (ranging between 13 and 45) or even in the difference being whether the ECG was recorded during rest or active task performance. In contrast to time windows and sample size factors, the effects of which have been extensively studied in the machine learning literature^28,29, no studies have used multiple cognitive tasks for fatigue induction or have directly compared the predictive performance of models trained on resting and task-related HRV data even though this would have significant implications for both practice and research. Therefore, in the present study, we extend the literature in this field by analyzing a multi-task dataset and comparing models trained on task-related as well as resting HRV data.

Beside the predictive performance, the models are also supposed to demonstrate comprehensive generalizability, that is, to make accurate predictions on previously unseen samples³⁰. To increase the generalizability of machine learning models, it is recommended to use a reasonably large dataset, avoid information leakage (e.g. by performing feature selection only on the training dataset), conduct cross-validation and test the models on previously unseen data^31,32. In addition, one might argue that using more than one cognitive task to induce fatigue might strengthen the generalizability because the different tasks affect different cognitive and affective systems³³ and the models trained on such heterogeneous data might be less sensitive to noise and task-specific characteristics³⁴. With other words, models that accurately detect fatigue irrespective of the type of task at hand are supposed to be of greater value because they could be effectively utilized in a variety of situations, which reflects both higher reliability and usefulness in practice.

In this study, we aimed to train machine learning models to achieve comprehensive generalizability and to become robust to task-specific characteristics. To achieve this goal, we first combined the datasets of three fatigue-related experiments that applied different cognitive tasks requiring different cognitive operations. Thus, the fatigue induction was variable. Second, due to the same reason, we were able to conduct analyses on a relatively larger dataset (n = 85) than previous studies (highest n = 45)²⁵. Third, to avoid information leakage, data was preprocessed after the separation of training and test data. And fourth, the models were trained using cross-validation on the training dataset but the final evaluation was based on previously unseen dataset to see how well the models generalize to new data. This analytic approach differs from the previous studies^20,25−27, because they used cross-validation only and did not test the models on unseen holdout data.

Beside fatigue detection, the prediction of the level of fatigue caused by prolonged cognitive performance is another important question that could be addressed by machine learning. In line with this, a few attempts have been made to train machine learning models on biomarkers and other variables like demographics or psychometric data to predict the consequences of performing a fatiguing task^35–37. Highly relevant to our study, Mun and Geng (2019) utilized machine learning to predict the level of post-experiment subjective fatigue based on various types of self-reported and biological data including resting HRV. The most predictive features were self-reported measures (e.g. pre-experiment fatigue, anxiety) but other indices reflecting cardiac activity such as blood pressure and the low frequency HRV component also contributed to the prediction of post-experiment fatigue. Similarly to studies using machine learning to detect fatigue, fatigue was induced by a single task. Consequently, the question remains unanswered whether post-experiment fatigue induced by different cognitive tasks could be predicted by models trained on pre-experiment variables.

In sum, the present study had three main goals. First, we trained classification algorithms to detect fatigue and regression algorithms to predict the severity of post-experiment subjective fatigue induced by prolonged cognitive performance based on a heterogeneous dataset in terms of fatigue induction. Second, we compared the predictive performance of classification models trained on resting and task-related HRV data. To our knowledge, no previous studies have made such comparison. Third and final, we explored the effects of time-window length to find the shortest time windows that still result in accurate predictions because the use of shorter ECG recordings would be beneficial for research as well as practice.

Data base

For the analysis, we combined the datasets of three fatigue experiments with similar procedures. The final dataset consisted of the data of 85 healthy university students (33 males, M = 21.71, SD = 2.53). All participants provided written consent and the experiments were approved by the local ethical committee (nr. 7698). All experiments were carried out in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). Part of the findings of the first fatigue experiment have been published in a previous article²³, but the original dataset (n = 23) was extended by the data of 15 more participants, which has thus not been published. Part of the findings of the second experiment (n = 20) have also been published²⁴. Importantly, these previous studies had different goals and completely different analyses compared to the present study. Findings of the third experiment (n = 27) have not been published before.

Fatigue induction

Figure 1 schematizes the sequence of trials in each experiment. In the first experiment (i.e. task switching experiment), fatigue induction was achieved by a modality-switching task^23,38, which required participants to make temporal judgements about either an auditory or a visual stimulus. In the second experiment (i.e. 2-back experiment)²⁴, we used a bimodal 2-back task with a game-like character that presents auditory and visual stimuli simultaneously and participants are required to compare the actual stimuli with the ones presented two trials earlier³⁹. Finally, the task used in the third experiment (i.e. Stroop experiment) was a bimodal Stroop task that required participants to make sematic categorizations of written or spoken words. The tasks lasted approx. 1.2–1.5 hours. For an elaborated description of the tasks used for fatigue induction, see the Supplementary Materials.

Before the prolonged task performance, participants practiced the task at hand and a few subjective measurements were administered (see below). ECG signals were recorded (with three chest electrodes; Lead II.) for 5 minutes before (pre-task resting) and after (post-task resting) the prolonged task performance and continuously during the whole course of the task. In terms of the duration of resting ECG recording, however, the 2-back experiment differed from the other two experiments, because both the pre-task-, and post-task resting periods lasted only 4 minutes.

Subjective measures

Subjective fatigue was assessed by visual analogue scales (VAS). That is, participants indicated their actual experience of fatigue on a 100 mm horizontal line with “No fatigue at all” and “Very severe fatigue” printed on the left and right end of the line, respectively. The VAS was administered two times: before and after the prolonged task performance. To further characterize the tasks used in the experiments, the NASA Task Load Inventory (NASA_TLX) was also applied⁴⁰. The NASA_TLX is a self-reported measure that assess different aspects of perceived workload. It consists of six subscales with 21 gradations each: mental demand, physical demand, temporal demand, overall performance level, effort, and frustration level.

HRV analysis

ECG data was sampled at a rate of 1 kHz by using a CED 1401 Micro II analogue-digital converter device (CED, Cambridge, UK). After obtaining the ECG signal, it was further processed in Spike2 software. First, the data was visually inspected, and artifacts were removed. Then, the inter-beat-intervals (i.e. RR-intervals) were extracted and analyzed further in Kubios HRV analysis software⁴¹. The “very low” option for artifact correction was chosen and ectopic intervals were replaced using cubic spline interpolation. Time-domain, frequency-domain (using the Fast Fourier Transform routine) and non-linear analyses were applied to quantify HRV. In the time domain, we extracted the mean and the standard deviation of normal-to-normal NN intervals (mean RR and SDNN, respectively), minimum and maximum heart rate, root mean square of successive differences (RMSSD), the percentage of successive RR intervals that have the difference of more than 50 ms (pNN50) and the triangular index. The frequency-domain variables included the natural logarithms of the total power (TP) and the main frequency components such as the very low frequency (VLF; 0–0.04 Hz), the low frequency (LF; 0.04–0.15 Hz) and the high frequency (HF; 0.15–0.4 Hz). The ratio of LF and HF (LF/HF) was also calculated. Finally, the obtained non-linear indices were the width (SD1) and length (SD2) of the Poincaré plot, their ratio (SD1/SD2), approximate entropy, sample entropy, correlation dimension (D2) and the detrended fluctuation analysis measures DFA1 and DFA2.

In sum, 20 HRV indices were used for the analysis. These indices were calculated separately for the two resting periods (pre-task and post-task resting) and for two task-related periods (the beginning and the end of the task). To investigate the effects of time window length on the performance of the machine learning models, the HRV indices were calculated for different time windows ranging from 1 minute to 5 minutes.

Classification algorithms

All programming was implemented in Python using the scikit-learn package (version 0.23.2.)⁴². The aim of classification was to develop models able to accurately distinguish between fatigue and non-fatigue states. This binary classification problem was addressed by training algorithms on resting HRV data and task-related HRV data, separately. For the algorithms trained on resting data, the pre-experiment resting HRV data was labelled as the “Non-fatigue” state and the post-experiment resting HRV data was labelled as the “Fatigue” state. For the algorithms trained on task-related data, the beginning period (i.e. the first 1–5 minutes depending on the time window) of task performance was labelled as the “Non-fatigue” state, while the end of the task (i.e. the last 1–5 minutes depending on the time window) was labelled as the “Fatigue” state. Fatigue was operationalized as an increase in subjective fatigue (measured by VAS). Hence, the data of three participants that did not have a higher post-experiment subjective fatigue score compared to the pre-experiment score were excluded from the classification analyses. Thus, for the classification problems, the sample size was reduced to 82 participants. For each person, HRV was calculated for two intervals (one with the “Fatigue” label and another with the “Non-fatigue” label). The final dataset that the algorithms used for each classification problem consisted of 164 data points for each HRV variable.

The dataset was split into training (~ 70%) and test sets (~ 30%). Feature selection via recursive feature elimination with 5-fold cross-validation (5-CV) was performed on the training set. Three classification algorithms were used: SVM, KNN and random forest. Prior to the training, the data were standardized by z-transformation (except for the training of random forest). Hyperparameters for each classifier were optimized through grid search with 5-CV (for an elaborated description of hyperparameters tuning and feature selection, see the Supplementary materials). Both the internal validation and the evaluation of the classification performance on the test set were done using the area under the receiver operating characteristic curve (area under the curve, AUC). In addition, accuracy, sensitivity and specificity were also calculated to evaluate the performance showed on the testing dataset. To test whether the fatigue detection performances are higher than chance level, permutation tests with 1000 iterations were carried out⁴³(for further details, see Supplementary materials).

Regression models

Similarly to the classification problems, regression modelling was carried out with the scikit-learn package (version 0.23.2.)⁴². As the dataset used for regression modelling was smaller (because only the pre-experiment resting HRV was used), the ratio used for dividing the dataset into training and testing sets was 80%/20% to have a sufficiently large dataset for training. In the regression models, the outcome measure was the level of subjective fatigue after prolonged task performance measured on a VAS. Potential predictor variables included the 20 pre-experiment resting HRV indices, participants’ sex, age, self-reported sleep duration, pre-experiment subjective fatigue and task duration. The least absolute shrinkage and selection operator (LASSO) method was used to select the most important features⁴⁴. Similarly to the classification models, feature selection for performed separately for each time window.

LASSO and elastic net regression were applied. The hyperparameter alpha was tuned for both algorithms on the training set with 5-CV. The data were standardized by z-transformation. Following parameter optimization, the models predicted the level of subjective fatigue in the previously unseen testing dataset. To evaluate the performance, two metrics, the root mean squared error (RMSE) and the R², were calculated. Permutation tests with 1000 iterations were carried out to assess the significance of the R² values.

Finally, to reduce the effect of statistical fluctuations and potential biases due to the random splitting of the data, the entire procedure (i.e. random allocation of participants into training and test sets, feature selection, model training, parameter tuning and evaluation) was repeated 100 times with different randomization seeds for each of the classification and regression problems. The means and standard deviations of all evaluation metrics across the 100 iterations were used to assess the predictive performance and to obtain the 95% confidence interval (CI). However, for regression, the distributions of model performances were found to be skewed and therefore, we used the median to describe the center of the distribution, and the Q1 and Q3 to describe the dispersion of the distribution. In addition, to help the interpretability of the results and to support the potential practical use, we also inspected the most average (i.e. the one with an AUC/R² score closest to the mean) models to get an insight into the optimal parameters.

Fatigue induction and workload

In each experiment, subjective fatigue was significantly higher after than before the experiment (task-switching experiment: t(37) = 9.34, p < .001, M_diff = 32.41; 2-back experiment: t(19) = 6.81, p < .001, M_diff = 25.4; Stroop experiment 3: t(26) = 8.27, p < .001, M_diff = 27.37). These findings show that the fatigue manipulation was successful regardless of the cognitive task performed by the participants. An univariate ANOVA showed that the three experiments did not significantly differ in subjective fatigue before the task started (F(2,82) = 2.605, p = .08, η_p² = .06). The analysis of NASA_TLX scores, however, revealed that the tasks in the three experiments differed in terms of perceived workload. Univariate ANOVAs yielded significant Experiment main effects for mental demand (F(2,82) = 4.247, p < .05, η_p² = .09), physical demand (F(2,82) = 3.471, p < .05, η_p² = .08), temporal demand (F(2,82) = 3.482, p < .05, η_p² = .08), effort (F(2,82) = 5.991, p < .01, η_p² = .13) and frustration (F(2,82) = 3.836, p < .05, η_p² = .09). Bonferroni-corrected pairwise comparisons showed that the Stroop task was perceived to be less mentally demanding (M = 14.63, SD = 3.91) and required the lower level of effort (M = 12.96, SD = 3.78) than the switching task (mental demand: M = 16.71, SD = 3.35; effort: M = 15.87, SD = 3.96) and the 2-back task (mental demand: M = 17.10, SD = 2.02; effort: M = 15.16, SD = 3.18). In addition, the Stroop task was also perceived to be less temporally demanding (M = 9.59, SD = 4.92) and less frustrating (M = 7.96, SD = 5.42) compared to the switching task (temporal demand: M = 12.53, SD = 4.64; frustration: M = 11.63, SD = 5.84). All corrected p-values < .05.

Classification

Using the HRV data, classification models were trained to successfully differentiate fatigue and non-fatigue states. Figure 2 presents the most important features for each classification problem. Results of cross-validation in the training set are reported in the Supplementary Materials (see Table S1). Predictive performance of the classifiers trained on task-related and resting HRV data for the test set are summarized in Table 1 and Table 2, respectively. Below, we report the most important findings. Permutation tests demonstrated that the classification performance of all classifiers in all classification problems differed significantly from the permuted null-distributions. The performance of the SVM, KNN and RF classifiers produced similar results in case of each classification problem. However, on average across all classification problems, the SVM slightly outperformed the other two classifiers. Irrespective of the classifier and time window used, we observed that compared to the resting HRV data, the algorithms were more accurate in predicting fatigue states in the task-related HRV data (see Fig. 3). In addition, as expected, the length of the time window had an effect on model performance because in most of the cases the training on longer time windows resulted in better performances compared to shorter time windows.

Table 1

Results of the classifiers trained on task-related HRV
Time window/ Algorithm		Evaluation metrics (test set)
Time window/ Algorithm		Mean AUC (95% CI)	Mean accuracy (95% CI)	Mean sensitivity (95% CI)	Mean specificity (95% CI)	p
5-min	SVM	.843 (.833 - .853)	.761 (.750 - .772)	.726 (.712 - .740)	.795 (.782 - .809)	.009
	KNN	.821 (.810 - .832)	.746 (.735 - .757)	.706 (.690 - .721)	.787 (.771 - .802)	.004
	RF	.813 (.803 - .823)	.733 (.724 - .742)	.741 (.726 - .755)	.725 (.708 - .742)	.003
4-min	SVM	.831 (.819 - .843)	.743 (.730 - .756)	.702 (.687 - .716)	.785 (.768 - .801)	.013
	KNN	.811 (.800 - .822)	.737 (.726 - .748)	.666 (.651 - .682)	.807 (.793 - .821)	.005
	RF	.811 (.800 - .822)	.734 (.722 - .746)	.720 (.702 - .739)	.748 (.733 - .763)	.006
3-min	SVM	.815 (.803 - .827)	.735 (.723 - .747)	.697 (.683 - .711)	.772 (.756 - .788)	.015
	KNN	.786 (.774 - .798)	.718 (.706 - .73)	.646 (.629 - .664)	.789 (.773 - .805)	.006
	RF	.781 (.770 - .792)	.716 (.705 - .727)	.706 (.688 - .724)	.726 (.708 - .743)	.006
2-min	SVM	.820 (.808 - .832)	.744 (.731 - .757)	.725 (.710 - .740)	.763 (.748 - .778)	.005
	KNN	.808 (.796 - .820)	.736 (.724 - .748)	.685 (.668 - .702)	.787 (.773 - .802)	.005
	RF	.792 (.780 - .804)	.722 (.709 - .735)	.726 (.708 - .744)	.718 (.697 - .739)	.007
1-min	SVM	.779 (.765 - .793)	.717 (.703 - .731)	.680 (.664 - .696)	.754 (.736 - .771)	.018
	KNN	.764 (.751 - .777)	.712 (.700 - .724)	.656 (.638 - .674)	.768 (.753 - .783)	.009
	RF	.763 (.751 - .775)	.698 (.686 - .71)	.690 (.671 - .709)	.706 (.688 - .725)	.007
Abbreviations: AUC = area under the receiver operating characteristic curve; CI = confidence interval; KNN = k-nearest neighbors; p = permutation test p-value; RF = random forest; SVM = support vector machine

Table 2

Results of the classifiers trained on resting HRV
Time window/ Algorithm		Evaluation metrics (test set)
Time window/ Algorithm		Mean AUC (95% CI)	Mean accuracy (95% CI)	Mean sensitivity (95% CI)	Mean specificity (95% CI)	p
5-min	SVM	.743 (.730 − .756)	.701 (.689 − .713)	.709 (.692 − .726)	.693 (.678 − .709)	.035
	KNN	.738 (.725 − .751)	.701 (.689 − .713)	.671 (.650 − .691)	.731 (.712 − .750)	.026
	RF	.731 (.716 − .746)	.691 (.678 − .704)	.685 (.663 − .706)	.696 (.678 − .715)	.027
4-min	SVM	.746 (.732 − .760)	.699 (.686 − .712)	.719 (.702 − .736)	.678 (.660 − .696)	.014
	KNN	.750 (.738 − .762)	.702 (.690 − .714)	.653 (.634 − .672)	.751 (.735 − .768)	.003
	RF	.730 (.717 − .743)	.678 (.665 − .691)	.657 (.639 − .675)	.698 (.680 − .717)	.013
3-min	SVM	.736 (.723 − .749)	.683 (.670 − .696)	.665 (.649 − .682)	.702 (.684 − .719)	.023
	KNN	.738 (.725 − .751)	.686 (.675 − .697)	.604 (.587 − .621)	.768 (.751 − .786)	.014
	RF	.718 (.706 − .730)	.677 (.666 − .688)	.639 (.620 − .658)	.714 (.698 − .731)	.017
2-min	SVM	.715 (.703 − .727)	.668 (.656 − .680)	.640 (.625 − .655)	.697 (.680 − .714)	.029
	KNN	.700 (.687 − .713)	.666 (.654 − .678)	.600 (.582 − .618)	.732 (.715 − .749)	.017
	RF	.705 (.694 − .716)	.672 (.662 − .682)	.616 (.599 − .633)	.728 (.710 − .745)	.017
1-min	SVM	.723 (.709 − .737)	.661 (.650 − .672)	.588 (.571 − .605)	.731 (.711 − .751)	.018
	KNN	.699 (.686 − .712)	.658 (.647 − .669)	.549 (.529 − .569)	.763 (.747 − .779)	.016
	RF	.685 (.671 − .699)	.643 (.631 − .655)	.601 (.579 − .622)	.683 (.661 − .705)	.033
Abbreviations: AUC = area under the receiver operating characteristic curve; CI = confidence interval; KNN = k-nearest neighbors; p = permutation test p-value; RF = random forest; SVM = support vector machine

The overall best performance was achieved by SVM trained on task-related HRV data with a time window length of 5 minutes. The model (AUC = .843, accuracy = 74%) that best represented the distribution of SVM classifiers in this particular classification problem used the non-linear features, DFA2 and approximate entropy, as well as two frequency-domain indices, VLF and LF/HF ratio, while the optimal hyperparameters were 10¹ for C and 10^− 2 for γ with the radial basis function. This model had a sensitivity of 72% and a specificity of 76%. Regarding the other main classification problem (i.e. training the models on resting HRV data), the performance was the best (AUC = .751, accuracy = 74%) when we used KNN (k = 18) as the classifier with a time window of 4 minutes (i.e. the longest time window available for the complete dataset in case of training on resting HRV data). This model used the features D2, DFA2, VLF and SD2. The sensitivity and specificity of the model were 68% and 80%, respectively.

Regression

Regression algorithms were trained to predict the level of subjective fatigue measured after prolonged cognitive task performance. The most frequently selected features for regression modeling are presented in Fig. 4. Results of cross-validation are reported in the Supplementary Materials (see Table S2). The performances of the LASSO and elastic net regression models are summarized in Table 3. All models differed significantly from the permuted null-distribution, except for the models trained on HRV data calculated for 5-min time window, most probably due to missing data in the 2-back experiment. The best performance was achieved by elastic net regression trained on 4-min HRV data. The elastic net regression (R² = .219; RMSE = 16.231; alpha = .08) model that best represent the distribution used the following features: pre-experiment VAS, sex, the duration of the experiment and two HRV indices, SD1/SD2 ratio and the total power. All predictors were positive, indicating that higher levels of pre-experiment subjective fatigue, HRV and longer experiment duration predicted higher levels of post-experiment subjective fatigue. In addition, female participants were predicted to experience more severe post-experiment subjective fatigue compared to males. The correlation between predicted and true values was moderate (r = .559).

Table 3

Results of the regression models predicting the level of post-experiment subjective fatigue
Time window / algorithm		Evaluation metrics (test set)
Time window / algorithm		Median R² (Q1 – Q3)	Median RMSE (Q1 – Q3)	p
5-min*	Elastic net	.003 (-.261 − .202)	18.730 (17.218–21.221)	.116
	LASSO	− .031 (-.317 − .192)	18.931 (17.289–21.222)	.292
4-min	Elastic net	.224 (-.021 − .321)	17.114 (15.491–18.323)	.006
	LASSO	.205 (-.039 − .327)	17.092 (15.719–18.415)	.012
3-min	Elastic net	.209 (.024 − .369)	16.648 (15.505–18.218)	.006
	LASSO	.206 (.015 − .368)	16.855 (15.510- 18.463)	.011
2-min	Elastic net	.216 (-.028 − .380)	16.613 (15.053–18.450)	.006
	LASSO	.187 (-.049 − .364)	16.854 (15.362–18.648)	.017
1-min	Elastic net	.165 (-.038 − .308)	17.311 (15.528–19.714)	.014
	LASSO	.141 (-.053 − .315)	17.444 (15.703–19.782)	.025
*Please, note that the 5-min time window could not be used in the 2-back experiment
Abbreviations: LASSO: least absolute shrinkage and selection operator; p: permutation test p-value; RMSE: root mean squared error

The main objective of this study was to determine to what extent classification models and regression models trained on HRV data can detect the fatigue state and predict the level of subjective fatigue that resulted from prolonged performance of different cognitively demanding tasks, respectively. By combining the datasets of three different experiments applying different cognitive tasks, we aimed to investigate the predictive power of these models when trained on heterogeneous data in terms of fatigue induction. These tasks had different parameters in terms of stimuli, duration, goals etc., and the perceived levels of workload were also different. Therefore, we argue that the models trained on the combined dataset of these experiments are robust meaning that they are not limited to special task characteristics and show higher levels of generalizability.

Compared to previous studies that used only a single task for fatigue induction, the fatigue classification algorithms in the present study performed at a similar level^20,25–27; the AUC scores and balanced accuracies in this study ranged from .685 to .841, and from 68 to 76%, respectively. The predictive power mainly depended on two factors: the time of ECG recording, and the time window used for HRV calculation. One of our major findings was that the predictive power was higher when the models were trained on task-related compared to resting HRV data. This unique observation suggests that HRV features extracted from resting ECG recording might be more prone to individual differences making it more complicated for the algorithms to effectively learn. The comparison of sensitivity and specificity scores indicated that although both metrics were lower for models trained on resting compared to task-related data, specificity (i.e. the classification accuracy for the non-fatigue label) decreased to a greater extent than sensitivity. This might be due to higher individual differences in the pre-experiment resting period that could derive from different levels of pre-task anxiety or differences in mood prior to task performance that can be reflected by cardiac activity^45,46. Another potential explanation could be that the variance was lower in the task-related dataset obtained at the beginning of task performance (i.e. the data with non-fatigue label) leading to more effective model training and better detection of the non-fatigued state.

This finding of better performance of classification based on task-related HRV has important practical implications. While designing monitoring systems that detect fatigue and warn people about their fatigued state in order to avoid accidents, one should consider the advantage of task-related data and build a system that monitors and acts during cognitive activity. A system like this would certainly be beneficial for individuals to optimize the timing of breaks during task performance as it could warn the individual to take a rest when it detects fatigue. However, it is important to note that the observed asset of algorithms trained on task-related data might only apply for HRV and may not necessarily be generalizable to other physiological measures (e.g. brain activity measured by EEG).

Another important result of the classification analyses was that the predictive power increased as a function of time window length used for HRV. This finding is in line with previous studies^20,28, and suggests that HRV indices obtained from longer periods of recording are more reliable and informative for fatigue detection. However, since predicative power was still relatively good (i.e. AUC score = .82, accuracy = 74.4%) when a shorter, 2-min time window was used on task-related data, researchers as well as practitioners may consider using a 2-min time window when fatigue detection in shorter time periods is required. Although the conventional minimum for HRV calculation is 5 minutes⁴⁷, recent studies provide support for the reliability of time windows even shorter than 2 minutes as well^48,49.

In line with the notion that the predictive power depends on the time and length of ECG recording, the best performance was achieved by the SVM classifier trained on task-related HRV data calculated for a 5-min time window. The AUC score of .84 produced by this model indicates high efficacy and is comparable with previous studies^26,27. The good performance could be explained by several factors such as having a large sample size or using recursive feature elimination for feature selection, which has not been used in previous studies. In addition, the inclusion of non-linear HRV indices is also likely to be the source of good performance given the fact that most of the selected features (e.g. entropies, DFA1 and DFA2) were obtained by non-linear analyses. This latter explanation is further supported by a study that found association between non-linear HRV indices and mental workload⁵⁰, a psychological concept strongly related to fatigue⁶. More specifically, theoretical models suggest that the perceived level of mental workload plays a crucial role in the emergence of fatigue. Increasing levels of workload is likely to increase the feeling of fatigue especially when the perceived benefits of task performance decrease or remain unchanged^6,51,52. Beside the high efficacy of models trained on task-related data, the models trained on resting data performed relatively well too. Interestingly, the best AUC score (.75) achieved by these models in our study (with the KNN classifier) was almost identical to the AUC score (.74) of the KNN classifier reported in Huang et al. (2018) who trained the algorithm on resting HRV data. This suggests that an AUC score of ~.75 might be the upper limit of predictive power for models trained on resting HRV.

Another aim of this study was to predict the level of subjective fatigue that emerged during prolonged cognitive activity based on pre-experiment resting HRV and other variables. Consistently with previous studies investigating the predictive power of resting HRV on subjective fatigue in healthy individuals⁵³ and patients with multiple sclerosis⁵⁴, we found that resting HRV contributed to the prediction of post-experiment subjective fatigue. From a methodological point of view, however, our study was different from the previous studies for four reasons. First, the cognitive tasks used for fatigue induction were variable and thus, we found that the predictive power of resting HRV is not limited to a specific cognitive task but can be generalized to a variety of tasks. Second, we examined several time window lengths, and our findings suggest that shorter periods of ECG recording might also be effective for fatigue prediction. Third and finally, beside the time-, and frequency-domain HRV indices, we also included non-linear HRV indices as potential predictors. This decision was justified because the most frequently selected HRV feature was the SD2/SD1 ratio, a non-linear index, which had relatively high overall importance scores.

Beside HRV indices, the regression models included demographic, subjective and other variables (e.g. sleep and task duration) as well. As expected, and in line with previous findings⁵³, the most important predictor of post-experiment subjective fatigue was the pre-experiment level of subjective fatigue. This indicates that a person is more likely to experience severe fatigue after task performance if their baseline level of fatigue is higher relative to those who feel more rested before task performance. This also points out that systems designed for fatigue prevention should take the baseline level of fatigue into account. We also found that sex was a useful predictor and that, compared to males, female participants reported higher levels of fatigue after task performance. Sex differences in task-related fatigue have received little attention so far and those previous studies found no significant sex effects^53,55 or only marginal effects indicating that - consistently with our results - females tended to rate the level of subjective fatigue higher⁵⁶. Nevertheless, this finding provides support for the notion that sex explains a relevant proportion of the variance in fatigue sensitivity possibly through complex interactions between sex hormones and the dopaminergic systems⁵⁷. Furthermore, the duration of the experiment was also found to be an important predictor. This is in line with other fatigue experiments^5,58 suggesting that the level of subjective fatigue increases in a monotonic way with increasing time spent on a cognitive task.

This study has several limitations. The resting periods in the 2-back experiment were only 4-min long and thus, model training on 5-min resting HRV data was conducted on a smaller dataset. This had the strongest effect in the case of regression because models trained on 5-min HRV data performed remarkably poorer compared to the other models. In addition, similar to most machine learning studies that aimed at the prediction of fatigue, this study only investigated within-site generalizability (i.e. how the models generalize when both the training and testing sets were obtained in the same laboratory) and provide no information about between-site generalizability (i.e. how the models generalize to external datasets obtained by other researchers in different locations). Further studies should be conducted in collaboration with independent researchers from different laboratories to have an external dataset for model testing in order to assess between-site generalizability too.

This study showed that detecting fatigue based on HRV data via machine learning is effective even if fatigue is induced by different cognitive tasks leading to more heterogeneous data for both training and testing. This indicates that the models are not strongly influenced by the specific task characteristics which implies higher levels of generalizability. The classifiers trained on ECG data recorded during task performance outperformed those trained on resting data and are recommended to be used in future settings. Finally, regression models predicted the severity of post-experiment subjective fatigue with moderate predictive accuracy. These models included the predictors pre-experiment fatigue, sex, task duration and HRV indices. Similarly to classification, longer time windows resulted in better predictions. Both types of machine learning models (i.e. classification and regression models) included HRV indices obtained by non-linear analyses (entropies, Poincaré plot, detrended fluctuation analyses) and these indices were especially important predictors of fatigue in the machine learning models.

Acknowledgements

AC and AM were supported by National Research, Development and Innovation Office (NKFIH K120012). AM was supported by the ÚNKP-20-3-II-PTE-876 New National Excellence Program of the Ministry for Innovation and Technology from the source of the National Research, Development and Innovation Fund.

Author Contribution

AM and AC designed and performed the experiments, analysed data and wrote the paper. DL and GD interpreted the data and wrote the paper. All authors discussed the results and implications and commented on the manuscript at all stages.

Ethical approval

The procedure of the experiments were reviewed and approved by the Ethics Committee of the Medical School, University of Pécs (nr. 7698).

Conflict of interest

The authors certify that there is no conflict of interest with any organization regarding the material discussed in the manuscript

Data availability

The data and materials for all experiments are available at https://data.mendeley.com/datasets/kbjgw4msv5/draft?a=714e3730-325c-47b3-856a-21453c6ad80f.

Ackerman, P. Cognitive fatigue: Multidisciplinary perspectives on current research and future applications. (American Psychological Association, 2011). doi:10.1037/12343-000.
Robert, G. & Hockey, J. Compensatory control in the regulation of human performance under stress and high workload: A cognitive-energetical framework. Biological Psychology 45, 73–93 (1997).
Lorist, M. M., Boksem, M. A. S. & Ridderinkhof, K. R. Impaired cognitive control and reduced cingulate activity during mental fatigue. Brain Res Cogn Brain Res 24, 199–205 (2005).
Dobryakova, E., Deluca, J., Genova, H. M. & Wylie, G. R. Neural Correlates of Cognitive Fatigue: Cortico-Striatal Circuitry and Effort–Reward Imbalance. Journal of the International Neuropsychological Society 19, 849–853 (2013).
Hopstaken, J. F., van der Linden, D., Bakker, A. B. & Kompier, M. A. J. A multifaceted investigation of the link between mental fatigue and task disengagement. Psychophysiology 52, 305–315 (2015).
Hockey, G. R., Robert, G. & Hockey, J. A motivational control theory of cognitive fatigue Human Factors in the Design of Safety-Critical Systems View project A motivational control theory of cognitive fatigue. (2011) doi:10.1037/12343-008.
Boksem, M. A. S., Meijman, T. F. & Lorist, M. M. Mental fatigue, motivation and action monitoring. Biological Psychology 72, 123–132 (2006).
Matuz, A., van der Linden, D., Zsidó, A. & Csathó, Á. Visually guided movement with increasing time-on-task: Differential effects on movement preparation and movement execution. Quarterly Journal of Experimental Psychology 75, 565–582 (2022).
Zeller, R., Williamson, A. & Friswell, R. The effect of sleep-need and time-on-task on driver fatigue. Transportation Research Part F: Traffic Psychology and Behaviour 74, 15–29 (2020).
Nachreiner, F. TIME ON TASK EFFECTS ON SAFETY. J. Human Ergol 30, 97–102 (2001).
Hu, X. & Lodewijks, G. Detecting fatigue in car drivers and aircraft pilots by using non-invasive measures: The value of differentiation of sleepiness and mental fatigue. Journal of Safety Research 72, 173–187 (2020).
Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D. & Babiloni, F. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience & Biobehavioral Reviews 44, 58–75 (2014).
2015 International Conference on Computer, Communication and Control (IC4) 1–6 (IEEE, 2015). doi:10.1109/IC4.2015.7375629.
Al-Libawy, H., Al-Ataby, A., Al-Nuaimy, W. & Al-Taee, M. A. HRV-Based Operator Fatigue Analysis and Classification Using Wearable Sensors.
Qin, H., Zhou, X., Ou, X., Liu, Y. & Xue, C. Detection of mental fatigue state using heart rate variability and eye metrics during simulated flight. Human Factors and Ergonomics in Manufacturing & Service Industries 31, 637–651 (2021).
Monteiro, T. G., Skourup, C. & Zhang, H. Using EEG for Mental Fatigue Assessment: A Comprehensive Look into the Current State of the Art. IEEE Transactions on Human-Machine Systems 49, 599–610 (2019).
Puce, A. & Hämäläinen, M. S. A Review of Issues Related to Data Acquisition and Analysis in EEG/MEG Studies. Brain Sciences 2017, Vol. 7, Page 58 7, 58 (2017).
Herlambang, M. B., Taatgen, N. A. & Cnossen, F. The Role of Motivation as a Factor in Mental Fatigue. Human Factors 61, 1171–1185 (2019).
Marandi, R. Z., Madeleine, P., Omland, Ø., Vuillerme, N. & Samani, A. An oculometrics-based biofeedback system to impede fatigue development during computer work: A proof-of-concept study. PLOS ONE 14, e0213704 (2019).
Laurent, F. et al. Multimodal information improves the rapid detection of mental fatigue. Biomedical Signal Processing and Control 8, 400–408 (2013).
Malik, M. & Camm, A. J. Heart rate variability. Clinical Cardiology 13, 570–576 (1990).
Mizuno, K. et al. Mental fatigue caused by prolonged cognitive load associated with sympathetic hyperactivity. Behavioral and Brain Functions: BBF 7, 17 (2011).
Matuz, A., van der Linden, D., Topa, K. & Csathó, Á. Cross-modal conflict increases with time-on-task in a temporal discrimination task. Frontiers in Psychology 10, 2429 (2019).
Matuz, A. et al. Enhanced cardiac vagal tone in mental fatigue: Analysis of heart rate variability in Time-on-Task, recovery, and reactivity. PLOS ONE 16, e0238670 (2021).
Tsunoda, K., Chiba, A., Yoshida, K., Watanabe, T. & Mizuno, O. Predicting changes in cognitive performance using heart rate variability. IEICE Transactions on Information and Systems E100D, 2411–2419 (2017).
Huang, S., Li, J., Zhang, P. & Zhang, W. Detection of mental fatigue state with wearable ECG devices. International Journal of Medical Informatics 119, 39–46 (2018).
Goumopoulos, C. & Potha, N. Mental fatigue detection using a wearable commodity device and machine learning. Journal of Ambient Intelligence and Humanized Computing (2022) doi:10.1007/S12652-021-03674-Z.
Le, A. S. & Aoki, H. Effect of Sliding Window Time on the Classification of Driver Mental Workload Performance Using Near-Infrared Spectroscopy (NIRS). IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC 2018-November, 1815–1819 (2018).
Cui, Z. & Gong, G. The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features. Neuroimage 178, 622–637 (2018).
Leisman, D. E. et al. Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals. Crit Care Med 48, 623–633 (2020).
Dwyer, D. B., Falkai, P. & Koutsouleris, N. Machine Learning Approaches for Clinical Psychology and Psychiatry. Annu Rev Clin Psychol 14, 91–118 (2018).
Chandler, C., Foltz, P. W. & Elvevåg, B. Using Machine Learning in Psychiatry: The Need to Establish a Framework That Nurtures Trustworthiness. Schizophrenia Bulletin 46, 11 (2020).
Smith, M. R., Chai, R., Nguyen, H. T., Marcora, S. M. & Coutts, A. J. Comparing the Effects of Three Cognitive Tasks on Indicators of Mental Fatigue. The Journal of Psychology 153, 759–783 (2019).
Yin, Z. & Zhang, J. Task-generic mental fatigue recognition based on neurophysiological signals and dynamical deep extreme learning machine. Neurocomputing 283, 266–281 (2018).
Qi, P. et al. EEG Functional Connectivity Predicts Individual Behavioural Impairment During Mental Fatigue. IEEE Transactions on Neural Systems and Rehabilitation Engineering 28, 2080–2089 (2020).
Bafna, T., Bækgaard, P. & Hansen, J. P. Mental fatigue prediction during eye-typing. PLOS ONE 16, e0246739 (2021).
Lekkas, D., Price, G. D. & Jacobson, N. C. Using smartphone app use and lagged-ensemble machine learning for the prediction of work fatigue and boredom. Computers in Human Behavior 127, 107029 (2022).
Lukas, S., Philipp, A. M. & Koch, I. Crossmodal attention switching: Auditory dominance in temporal discrimination tasks. Acta Psychologica 153, 139–146 (2014).
Heathcote, A. et al. Multi-tasking in Working Memory.
Hart, S. G. & Field California Lowell Staveland, -Moffett E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in psychology 52, 139–183 (1988).
Tarvainen, M. P., Niskanen, J. P., Lipponen, J. A., Ranta-aho, P. O. & Karjalainen, P. A. Kubios HRV – Heart rate variability analysis software. Computer Methods and Programs in Biomedicine 113, 210–220 (2014).
Pedregosa FABIANPEDREGOSA, F. et al. Scikit-learn: Machine Learning in Python Gaël Varoquaux Bertrand Thirion Vincent Dubourg Alexandre Passos PEDREGOSA, VAROQUAUX, GRAMFORT ET AL. Matthieu Perrot. Journal of Machine Learning Research 12, 2825–2830 (2011).
Boeke, E. A., Holmes, A. J. & Phelps, E. A. Toward Robust Anxiety Biomarkers: A Machine Learning Approach in a Large-Scale Sample. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging 5, 799–807 (2020).
Tibshirani, R. Regression Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological) 58, 267–288 (1996).
Schubert, C. et al. Effects of stress on heart rate complexity—A comparison between short-term and chronic stress. Biological Psychology 80, 325–332 (2009).
Kop, W. J. et al. Autonomic nervous system reactivity to positive and negative mood induction: The role of acute psychological responses and frontal electrocortical activity. Biological Psychology 86, 230–238 (2011).
Malik, M. et al. Heart Rate Variability. Circulation 93, 1043–1065 (1996).
Baek, H. J., Cho, C. H., Cho, J. & Woo, J. M. Reliability of Ultra-Short-Term Analysis as a Surrogate of Standard 5-Min Analysis of Heart Rate Variability. https://home.liebertpub.com/tmj 21, 404–414 (2015).
Laborde, S., Mosley, E. & Thayer, J. F. Heart rate variability and cardiac vagal tone in psychophysiological research - Recommendations for experiment planning, data analysis, and data reporting. Frontiers in Psychology 8, 213 (2017).
Delliaux, S., Delaforge, A., Deharo, J. C. & Chaumet, G. Mental Workload Alters Heart Rate Variability, Lowering Non-linear Dynamics. Frontiers in Physiology 10, 565 (2019).
Boksem, M. A. S. & Tops, M. Mental fatigue: Costs and benefits. Brain Research Reviews 59, 125–139 (2008).
van der Linden, D. The urge to stop: The cognitive and biological nature of acute mental fatigue. in Cognitive fatigue: Multidisciplinary perspectives on current research and future applications. 149–164 (American Psychological Association). doi:10.1037/12343-007.
Mun, E.-Y. & Geng, F. Predicting post-experiment fatigue among healthy young adults: Random forest regression analysis. Psychol Test Assess Model 61, 471 (2019).
Sander, C., Modes, F., Schlake, H. P., Eling, P. & Hildebrandt, H. Capturing fatigue parameters: The impact of vagal processing in multiple sclerosis related cognitive fatigue. Multiple Sclerosis and Related Disorders 32, 13–18 (2019).
Fard, S. J. & Lavender, A. Sub-concussion View project Longevity View project. (2019) doi:10.1080/21641846.2019.1562582.
Hopko, S. et al. Design and control of web lateral dynamics View project Control & Combinatorial Optimization View project Effect of Cognitive Fatigue, Operator Sex, and Robot Assistance on Task Performance Metrics, Workload, and Situation Awareness in Human-Robot Collaboration. IEEE ROBOTICS AND AUTOMATION LETTERS 6, 3049 (2021).
Malyuchenko, N. v et al. GENETICS Effects of Genetic Variations in the Dopaminergic System on Fatigue in Humans: Gender Aspects. Translated from Byulleten’ Eksperimental’noi Biologii i Meditsiny 149, 187–193 (2010).
Ackerman, P. L. & Kanfer, R. Test Length and Cognitive Fatigue: An Empirical Examination of Effects on Performance and Test-Taker Reactions. Journal of Experimental Psychology: Applied 15, 163–181 (2009).

No competing interests reported.

Matuzetal2022SupplementyMaterials.pdf

Download PDF

Editorial decision: Major revision
21 Sep, 2022
Reviews received at journal
03 Sep, 2022
Reviewers agreed at journal
23 Aug, 2022
Reviewers invited by journal
01 Jun, 2022
Editor assigned by journal
20 May, 2022
Editor invited by journal
12 May, 2022
Submission checks completed at journal
12 May, 2022
First submitted to journal
07 May, 2022

You are reading this latest preprint version

Machine learning models in Heart Rate Variability based mental fatigue prediction: training on heterogeneous data to obtain robust models

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods

Data base

Fatigue induction

Subjective measures

HRV analysis

Classification algorithms

Regression models

Results

Fatigue induction and workload

Classification

Regression

Discussion

Limitations

Conclusions

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1