Classification
This paper presents a proof of concept for on-the-spot prediction of cybersickness at resting state baseline and near-instant detection of cybersickness during its onset. The algorithms are based on a fusion of brain inspired SNN architectures and HRV classification. Another study has also demonstrated the predictive capacity of their algorithm for CS at resting baseline with a smaller sample size of n=19[20]. Near-instant detection was demonstrated by Nam et al. [21] but required PCA preprocessing, power spectral analysis for EEG and 7 other biosignals. The present study shows that only 2 seconds of EEG data and 30 seconds of ECG data are required, and both biosignals can be used individually or together to predict and detect CS. The modified deSNN-KNN classification algorithm produced the best results in terms of accuracy, over LDA and light-GBM. It was found that similar classification accuracies can be obtained by using either earlier (30-31s, 76.6%) or later time segments (90-91s, 75%) at baseline. Simplifying feature vectors by removing reservoir – output neuron connections, and leaving the direct connections of input neurons to output neurons increases accuracies (Table 1-3). In addition, reducing redundancy in training data by focusing on key cybersickness relevant areas also has the same positive effect on accuracy. However, in the case where a model is trained on all 32 features, but only the top 5 features are considered, a reduction leads to a decrease in accuracy (75.00% to 68.80%) (Table 3). This highlights that in idealistic scenarios, not just a few but all features a model is trained on should be considered when eliminating redundancy. However, it is important to note that there is a trade-off in considering all features, as computational cost increases when conducting exhaustive searches.
Our analysis did not reveal why some participants were predicted in one baseline segment but not the other. An explanation is that this could be due to differences in the temporal characteristics of the spiking activity of neurons captured by the connection weights between input clusters and between individual reservoir neurons. Another explanation could be due to the nature of clinical studies, where there is interindividual variation between participants.
MSSQ and SSQ scores
In our experiment, the MSSQ-short was not a good predictor of cybersickness induction or sickness ratings. This points towards the need for questionnaires more targeted at visually induced motion sickness[22] to assess susceptibility. SSQ scores were a good adjunct to the subjective cybersickness reports in the separation of cybersick and control groups.
Related spatiotemporal brain dynamics were discovered in the following areas:
Fz Brodmann 8 visual attention and eye movements
T8, T7: Auditory processing
CP6: Auditory processing, speech comprehension
O2: Retinotopic mapping of visual scene, edge detection
P4: Angular gyrus attention, memory retrieval, language number processing, spatial cognition
PO3: Associative visual cortex (V3, V4, V5).
F3: Frontal eye fields, visual attention and eye movements.
FC5, FC6: Brocas speech production and articulation (primarily left hemisphere), language processing.
FP1, FP2: Executive function, decision making
CS is a complex condition with many brain areas involved[23,24]. Presented in this study is functional connectivity of the brain that predicts future CS, meaning that an individual with similar neural maps may be susceptible to cybersickness. A high neuron proportion grouped by connection weight of frontal (FC6, FP1, FP2) regions during the CS event, and temporal regions (T8) during resting baseline are consistent with another study showing changes in these areas well into the CS event. In addition, areas involved in CS include those for visual + attention processing and executive function (CP6, O2, PO3, F3, F4, FP1, FP2). Liu et al. [23] found reduced gravitational frequency means (transition of EEG power spectral density, temporal changes within a frequency band), and gravitational frequency standard deviation (dispersion of brain signal) at FP1, FP2, TP9 and TP1. Power spectral entropy (disorder of time sequence signals and irregularity of multi-frequency component signals) and Kolmogorov complexity (time domain complexity) were all reduced at FP1 and FP2 during VIMS[23]. However, it was noted that these changes may be related to other factors, such as alertness level or various mental conditions, and not limited or specific to VIMS. Our finding of an increase in O2’s interaction with other areas during cybersickness highlights that visual processing is altered beyond just the demands of normal visual processing in VR.
Of interest is the brain activity hub found at Cz, which had altered connectivity at resting-state baseline as well as during the onset of cybersickness when compared with controls. Reduced spike count at Cz before VR immersion may indicate that there is less frequency of communication from this area to other connected areas. Cz interacts with three cortices simultaneously, the somatosensory, motor and also is positioned over the mid cingulate, which has increased functional connectivity with the left V5/MT during cybersickness[25]. Krokos, Varshney [26] found high activity power in the central regions similar to the location of Cz, of average scalp maps according to independent component analysis. Brodmann area 5 corresponds to Cz, which is part of the superior parietal lobule and post central gyrus. It is located immediately posterior to the primary somatosensory cortex. Neuroimaging evidence suggests that this area contributes to movement planning. Furthermore, one study showed a correlation between the activity of area 5 neurons and the starting or final coordinates of limb movement. This suggested that BA5 is involved in processing spatial information for limb movement. Emerging evidence suggests that BA5 is also involved in the inhibition of movement[27]. A transcranial magnetic stimulation study found a causal role for BA5 in the regulation of corticospinal output during preparation that differentiates between whether a movement is withheld or executed[28]. One may think that Cz’s role in movement and also as a marker of future cybersickness at resting baseline lends possible credence to the postural instability theory of motion sickness, which postulates that postural instability is both a marker and a predictor of motion sickness, likely extending as well to cybersickness in virtual reality[29]. Our results, however, suggest that although processes related to motor control are altered during the event, we cannot speak for postural instability itself. Furthermore, a recent study shows that postural instability itself is not a good predictor of cybersickness. For purely visually induced motion sickness (VIMS), increases in functional connectivity were also found between the right MT/V5 and anterior insula. Decreased functional connectivity was also found between the left and right V1[25]. The left MT/V5 in particular is an area important for processing of “what” but not “where”, in priming for motion direction but not spatial position[30]. Nonetheless, cortical areas that control movement and visual processing are clearly involved in cybersickness.
Interestingly, cortical areas for visually induced cybersickness also overlap with areas involved in vestibular processing: Cz and FC6 – premotor and supplementary motor (movement processing, planning and inhibition) and P4 – medial superior temporal (motion detection). In this study, it can be observed that the size of the nodal cluster and strength of connectivity shift to right hemispheric dominance during CS, a preference also observed in vestibular processing.
Overall, there appears to be an alteration of activity and connection in areas related to motor control and planning, as well as visual processing. These areas may become targets of the new therapeutic approaches for cybersickness for future studies.
ECG
Although statistical differences between HRV parameters were not found, it was found that classification algorithms for cybersickness using sympathetic HRV indexes are still viable. This suggests that the differences in sympathetic parameters of HRV in cybersick people versus control are more complex and simpler types of statistical analysis may not pick up on it.
Ultra-short-term RMSSD recordings (30s and 10s) have been statistically reliable in previous studies, but this parameter alone does not yield high accuracies (Table 4 and 5).
Future suggestions and limitations:
Given that HRV is computed using R-R intervals of an ECG wave, it may be the case that other parameters, arising from other aspects of the ECG wave could be helpful as features. Further research could elucidate on this matter. This study used machine learning to extract information about the spatiotemporal processes within the cybersick brain. Future studies could explore the role of the interplay between motor control, motor planning and visual processing in VR. The feature interaction network analysis only showed interactions between cortical areas, but not whether they were increasing or decreasing connections. Future studies could shed light on how key cybersickness centers in the brain act to control the flow of information between cortical areas. Furthermore, the finding that different features can be found at different time segments, but still give similar accuracies, points towards the complexity of the cybersickness condition within the brain. It may therefore be of interest to look at the change in features over time, rather than the features at snapshots in time to understand cybersickness in more detail.