In this study, we proposed an immersive hand rehabilitation system based on VR rhythm games and vibrotactile feedback, which is capable of performing task-oriented, repetitive, and intensive tasks with a high level of engagement. Compared with previous studies that developed VR-based hand rehabilitation systems using contact-free hand-tracking technology, our study has the following three differences [21, 22].
First, the target gestures can be personalized using gesture recognition algorithms, and how precisely the gesture should be performed can be adjusted by changing the classification threshold value. As shown in Table 2, with a recognition threshold value of 2 cm, the gesture matching success rate of the stroke survivor was 79.6% and the average success rate of 11 healthy was 90% (SD = 10.7%). A similar accuracy was found in the study by Yuan et al. [50], in which the accuracy ranged from about 75 to 90% when novice players played a commercial rhythm game four times. This indicates that an appropriate threshold value for gesture recognition was selected and that all participants, including the stroke survivor, performed the gestures correctly according to these criteria. Previous studies have utilized built-in hand recognition functions which could recognize only limited hand gestures, such as poke, pinch, and finger opposition, and didn't consider how accurately the hand movements were being performed. In contrast, for this study three training motions— lateral pinch, medium wrap, and finger extension, among the many possible hand motion options, were selected. These motions have proven to be the most commonly used postures for gripping and releasing objects of various shapes and sizes in daily life [35], and have also been adopted as target movements in recently developed soft and exoskeleton robots [51, 52].
Second, a rhythm game was selected as the VR game content to facilitate intensity control and repetitive training. Previously, Ogun et al. selected daily life behaviors such as handling cubes, picking up vegetables from a bowl, and kitchen experience [21]; Pereira et al. selected farm activities such as harvesting crops, milking the cow, and making cheese [22]. Those studies reported that high user engagement was successfully induced, however, the number of repetitions of tasks was insufficient to achieve neuroplastic change and functional improvement [23]. It has been reported by previous research that the number of repetitions must be at least 300 times a day to induce neuroplastic changes and functional improvement of the upper extremity [53]. Our system induced the stroke survivor to perform 200 gestures for about 420 seconds, so it is considered to be sufficient for inducing a high rehabilitative effect. In addition, the intensity of training could be easily adjusted by manipulating the type of song and the frequency or pattern of the target appearance.
Third, vibrotactile feedback was used to induce a high sense of immersion. It is now widely used in video games and VR games to the extent that it is difficult to find products without vibration tactile feedback among commercial game controllers, and it has been reported to provide a stronger sense of immersion than audio-visual feedback alone [54]. It was attached to the median nerve of the wrist to provide vibration without interfering with hand tracking, and the effect of vibrotactile feedback will be discussed in detail using the brain activation results described later.
For the system validation, fNIRS was used to monitor the activation of the cortical area and time course of the hemodynamic response function in four ROIs (PFC, PMC&SMA, SM1, and SAC) during the VR-based hand rehabilitation task. HbO and HbR, which measure neural activity indirectly by detecting hemodynamic changes in the underlying cerebral cortex, are the most commonly used parameters of fNIRS [55].
As is evident from the significantly activated channels shown in Table 2 and cortical activation maps in Fig. 4, the introduction of our system gave rise to significant activations in both sides of SM1 and SAC, and the right sides of PMC&SMA and PFC. The activation of these areas means that the components of the proposed hand rehabilitation system have effectively stimulated each part of the brain as follows. First, activation of SM1 is considered to be caused by repetitive hand gestures and vibrotactile feedback during VR gameplay. There is ample evidence that the SM1 plays an important role in motor execution and sensorimotor integration during voluntary hand movements [56, 57]. Second, the activation of PMC&SMA is thought to be due to the planning and execution of the sequential hand gestures required to play the VR rhythm game that aims to remove each approaching target gesture in a sequential manner. This explanation is based on previous findings that the PMC&SMA play a critical role in the generation of sequential movements from memory with a precise timing plan [58]. Specifically, stroke survivors with PMC&SMA lesions had difficulty generating rhythmic sequential movements, even though they were provided with rhythmic auditory stimuli before the experiment. Third, vibrotactile feedback provided when the VR rhythm game was properly performed is considered to stimulate SAC activity, along with simultaneously provided audiovisual feedback. The explanation comes from previous research results [59, 60]: Reed et al. reported that SAC activation was observed when subjects performed tactile object recognition, and Yu et al. reported SAC was activated when movement-related multisensory feedback was provided. Finally, it is thought that significant brain activity was observed in PFC since the proposed hand rehabilitation system successfully induced user engagement. As an integrated hub for receiving and processing motivational and attentional information [61], it helps to voluntarily allocate attention to specific features, objects or spatial areas, and in our case, the content in the VR rhythm games [62].
As to the Group-level HRF time course of the healthy group, an increase in HbO and a decrease in HbR, the typical patterns of response to stimuli, were observed during the VR-based hand rehabilitation for all ROIs in Fig. 5. An increase in HbO means an increase in blood inflow from the artery to the activated area, and a concomitant decrease in HbR means an increase in oxygen metabolism [63]. Our findings are in line with previous studies in which the hand rehabilitation task is associated with an increase in cerebral blood flow and brain oxygenation [64].
For the individual HRF time course of the stroke survivor, the same phenomenon was observed in the bilateral SM1, right PFC, left SAC, and left PMC&SMA, as shown in Fig. 6. On the other hand, in the right SAC, HbO decreased shortly after the beginning of the task, and then returned to the baseline level and increased. Our observation of decreases in HbO in the right SAC may be caused by pathophysiological and anatomical changes in the brain after stroke [65]. In the right SAC, which can be assumed to be a degenerative brain region, there may be an insufficient increase in regional cerebral blood flow associated with a reduction of HbO during the VR-based hand rehabilitation for the sake of other brain regions [45].
There are two additional remarks worth mentioning. First, there was a difference in the interpretation of activation between the results of significantly activated channels and those of the cortical activation map. Specifically, there was no significantly activated channel in the left PMC&SMA and left PFC, however, the cortical activation maps showed an increase in HbO response for those areas. The reason for this discrepancy is considered to be high subject variability resulting in wider confidence intervals, as shown in Fig. 5. Second, both HbO and HbR tended to increase in the left PFC and right PMC&SMA of the stroke survivor, as shown in Fig. 6. A possible explanation for this phenomenon is an increase in oxygen supply due to increasing HbO and oxygen consumption due to increasing HbR, which is considered to have occurred at the same time [66]. As a further explanation for the increased HbO and HbR in the left PFC, our result is in line with the previous study that reported that a similar phenomenon occurred when performing a high cognitive task with sustained attention [67].
Although it was possible to develop an immersive hand rehabilitation system and confirm its effectiveness through the validation of brain activation, the present study is limited by the small number of participants included and the limited power of the study. In the future, a clinical intervention study should be conducted to verify the effect of the proposed system on the functional restoration of stroke survivors. Moreover, subject-specific target gestures should be applied to stroke survivors with limited hand conditions, such as range of motion and degree of stiffness for stroke survivors.