The Effect of a Full Upper-Limb Control System and Its Usage in Stroke Rehabilitation


 Backgrounds: Transferring behaviors of a human’s upper-limbs to an avatar is widely used in the field of virtual reality rehabilitation. To realize the transfer, movement tracking technology is required. Traditionally, wearable tracking devices are used to do the tracking, however the devices are expensive and cumbersome. Recently, non-wearable upper-limb tracking solutions are proposed, which are cheaper and more comfortable to interact. But most of the existing products cannot track full upper-limbs including both the arms and all the fingers, which limits the motion paradigm and further may lead to limited rehabilitation effect. Methods: In this paper, a novel method was first proposed for full avatar’s upper-limb control which integrates the fine finger motion and the arm wide range motion. Then, based on the method, a Virtual Reality Rehabilitation System (VRRS) was developed for upper-limb rehabilitation. To test the performance of VRRS, two experiments were designed. First, in order to investigate the effect of VRRS on virtual body ownership, agency, location of the body and usability, we compared it with the partial upper-limb tracking method based on Leap Motion controller (LP) in same virtual environments. Then, to study the feasibility of VRRS in rehabilitation, we recruited 16 stroke patients and split them into two groups: the experimental group and the control group. Each group consisted 8 patients, with and without employing VRRS respectively.Results: The control of full avatar’s upper-limbs improved the users’ senses on body ownership, agency and location of the body. The users preferred to use VRRS. In addition, although the upper-limb motor function of patients from both groups were improved, the difference between the FM scores tested on the first day and the last day of the experimental group was more significant than that of the control group. Conclusions: VRRS based on the proposed method for full avatar’s upper-limbs control was developed, which improved the user experience on embodiment and effectively improved the rehabilitation effect for upper-limbs of stroke patients.Trial registration:The study was registered at the First Affiliated Hospital of Jinan University Identifier: KY-2020-036; Date of registration - June 01, 2020.


Introduction
Stroke is common in all kinds of people, the consequence is obvious muscle weakness on one side of the body [1]. It is estimated that 50-75% of patients suffering stroke have persistent impairment of the affected upper limb and they need to do boring repeated physical training to recover their motor function [2]. To provide a more enjoyable and personalized experience on motor rehabilitation, the usage of virtual environment is gradually popular in the eld for its richness and interesting [3]. The virtual nature of the environment allows the behaviors that cannot be realized or highly costed in reality to be implemented in a low-cost way. Previous works indicate that the strength and activity ability of the affected side of body can be effectively restored by controlling the avatar's upper-limbs to interact with objects in a virtual environment. Trianda lou et al. [4] developed a 3D, networked multi-user Virtual Environment for Rehabilitative, Gaming Exercises (VERGE) system, for home therapy with a low-cost Kinect device.
Cuesta-Gómez et al. [5] used an LMC system and serious games for upper limb rehabilitation. Besides, Sucar et al. [6] introduced a virtual reality-based platform for rehabilitation of the upper-limbs with gesture therapy.
In many applications and studies of virtual reality, avatars are used as the medium of people interacting with the virtual environment [28]. Movement tracking is one of the key technologies for avatar control [29]. For partial upper-limb tracking, consumer devices including hand-held controllers such as Oculus Touch or HTC VIVE controllers and Motion Sensing devices such as Kinect and Leap Motion controller are widely used [27]. Granqvist et al. [12] used HTC VIVE controllers with inverse kinematics for partial upperlimb tracking. However, the hand-held controller can only track the position of the hand and cannot give the position information of the ngers. Collingwoode-Williams et al. [11] built a system to study the effect of lip and arm synchronization on the feeling of body ownership. They used a Kinect device for body tracking and an Oculus Rift for head rotation measuring. But the Kinect device can only track arms and part of nger joints. It is proved that complete upper-limb tracking enhances the realism of avatar's upperlimb behavior which in uences patients' cognition and motor recovery. Generally, in order to have a highquality full upper-limb tracking, the user need to use marker-based tracking systems that require them to wear tracking suits such as with the OptiTrack system [7] [8]. The suits are expensive and cumbersome.
Usage of consumer devices is another solution which is cheaper. Lin et al. [10] used the Oculus Rift headset with a Kinect sensor, a Leap Motion controller and a dance pad to navigate and manipulate objects inside synthetic scenes. In the method, only one Kinect was used and cannot recognize which side of the body is facing the device. Wu et al. [9] introduced a set-up which integrates multiple Kinects for robust and accurate full-body 3D skeleton tracking and a Leap Motion controller for hands. However, the tracking area of the Leap Motion controller was limited to a small range above the device.
In this paper, a low-cost method was proposed which can fully control avatar's upper-limbs by tracking movements of both arms and all ngers in large tracking area. We also developed a Virtual Reality Rehabilitation System (VRRS) based on this method. In the virtual environment of VRRS, movement of the user's upper-limbs is mapped to an avatar's upper-limbs to improve the sense of engagement with the virtual environment. In order to dynamically match the motor function of patients, the di culty of tasks can be modi ed conveniently, which is important for motor learning in general [13] and rehabilitation in particular [14].
We rst performed a preliminary experiment to compare VRRS with the partial tracking method based on the Leap Motion controller [30]. In the experiment, we built a virtual environment customized for upperlimb motion, and measured the senses of body ownership, agency, location of the body and the usability of the systems. Then, in order to evaluate the feasibility of VRRS in rehabilitation training, another experiment is performed. In the experiment, 16 stroke patients are recruited and the Fugl-Meyer (FMA) of the patients' upper-limb motor functions are evaluated.
The hypothesis was that VRRS can elicit higher senses on body ownership, agency and location of the body and users were satis ed with VRRS. We also expected VRRS to enrich motion paradigm and thereby be more effective in rehabilitation training to help stroke patients recover their upper-limb motor function.

VRRS System
Architecture As shown in Fig. 1, VRRS is consisted of two FLIR cameras and two computers, including a server computer (Intel Core i7-6700 at 3.40 GHz, 8GB RAM) and a client computer (Intel Core i9-9900k at 3.60GHz, 32GB RAM, and RTX2080TI). The cameras are xed on two tripods placed on a table such that the cameras are about 2m above the ground. The server computer receives the sensor data of the positions of 6 joints of arms and 42 joints of ngers, which are recognized by the OpenPose platform con gured on the client computer. The data are transfered through TCP/IP network architecture customized in C# (Microsoft Visual Studio). The distance between two cameras is about 1.4m, and they are connected to the client computer through USB cable. We used Unity game engine 2018.3.12f1 to develop the system.

Control method for full upper-limbs
With each time step, the data transfered from OpenPose platform are used to set the avatar's pose. We reorganize the received sensor data into a series of 3D coordinate data and lter the 3D data of each frame by setting thresholds to avoid obvious jitter. The data of wrists, elbows and shoulders are set in avatar's skeleton with inverse kinematics being used to adjust the skeleton of arms. And data of ngers are used to set skeleton under corresponding local coordinate system with forward kinematics. As a result, the avatar's upper-limbs are accurately controlled and can be used to interact with the virtual environment.

Study
The main purpose of this study was to test the effect of VRRS and evaluate the feasibility of VRRS in rehabilitation training. In the study, two experiments were performed. In experiment 1, 8 young adults were recruited to experience two kinds of game-like environments using VRRS and Leap Motion controller. And without VRRS served as the control group. These patients were with different level of disability ranging from Fugl-Meyer [15] score of 2-50 (M = 26.6, SD = 16.4). The patients gave their informed consent to the experimental procedures, which were approved by the local ethics committee. In our rst experiment, to avoid familiarization to the two systems, the volunteers were naive to our tasks and assigned to each experimental condition randomly.
Experimental Procedure Experiment 1 In the experiment, the subjects performed two kinds of tasks in game-like virtual environments: normal motion on a plane and mirror motion on a plane, with VRRS and LP respectively, i.e. totally four tasks to be performed (Fig. 2). To avoid the in uence of familiarity with the interactive environment on the experimental results, each participant performed the four tasks in random order. Before each task started, subjects were positioned in a chair and kept the initial posture the same as avatar's upper body. Subjects' arms were placed on A table in front of their bodies and their hands were naturally stretched with palms faced down. Each task consisted 30 trials and the four tasks took about 20 minutes. After all tasks were implemented, the subjects were asked to ll in a questionnaire to grate both VRRS and LP. Task 1/Task 3: Normal motion on the plane based on VRRS/ LP At the beginning, the subject controlled the avatar to move its left upper limb / virtual left hand to the starting area which was originally displayed in blue color by moving his left upper limb. Then, held for 1.5s after the color of the area turning green.
After that, a blue object was randomly generated in the reaching area which was customized to each subject. Meantime, the color of the starting area turned gray and a blue line appeared connecting the object and the starting area. The subject was required to move their left upper limb to reach the object along the blue line and maintain there for 1.5s after the object and the line turned green. If the object cannot be reached within 4s, it will disappear together with the line. And the trial will be regarded as un nished. Then the subject had 2s to move the left upper limb back to the starting area to prepare for the next trial. On the other side, if the object was reached, a score will be displayed in the upper-left corner of the screen. The score was calculated according to the overlapping between the motion trail of the palm center and the line. Then the subject moved left upper limb back to the starting area within 2s and remained suspended until the color of the starting area turned back to blue which meant a trial is completed and prepare to conduct the next trial. The rst 15 trials were implemented by the left upper limb and the opposite limb completed the rest of the trials in which the subject controlled the movements of the avatar's right upper limb / virtual right hand in the same scenario (Fig. 3).
Task 2/Task 4: Mirror motion on the plane based on VRRS/ LP In these two tasks, the motion of the subject's left upper limb was mapped to the avatar's right upper limb /virtual right hand. The avatar's left upper limb/ virtual left hand was controlled by the opposite limb of the subject. Other procedures were the same as those in the task 1 and task 3.

Experiment 2
In the experiment, every stroke patient in the experimental group participated in a three-day experiment on the feasibility of VRRS which includes six training sessions. These patients were required to conduct the session twice a day with an interval of 2 minutes. Before the rst session, the motor function of the stroke patients were evaluated with FMA. Each session was consisted of two tasks including normal motion on a plane and mirror motion on a plane. Considering the duration and intensity that the patients were able to adapt to, the number of trials of each task was set to 20 and the patients took a break of 30s between two tasks. After the patients completed the three-day experiment, they were evaluated again with FMA (Fig. 4). On contrast, the 8 stroke patients in the control group were only evaluated with FMA on the rst day and the third day without completing the training tasks (Fig. 5).

Outcome measures
Measures were performed using the SPSS statistical software system (version 25.0) and Origin function draw tool (version 2018).
In experiment 1, after the subjects completed all the four tasks they were asked to comoplete a questionnaire which integrates avatar embodiment [18] and System Usability Scale [19]. The questionnaire was used to assess the senses of body ownership, agency and location of the body as well as the usability of VRRS and LP. To test the difference in the questionnaire responses between VRRS and LP, a Mann-Whitney U-test was used. In addition, t-test was also applied to test for difference in the total score, time of completion and un nished rate between two systems. We also recorded kinematic traces of all subjects in each task from starting area to the position of the object randomly generated to analyze whether there is a difference between VRRS and LP.
In experiment 2, 16 patients performed twice evaluation of FMA. And t-test was used to test the difference in the Fugl-Meyer scores between the two evaluations of all the participants. Besides, the difference in Dvalue of the two evaluations between the experimental group and the control group was also tested with t-test.

Experiment 1
Overall, users' responses of VRRS were as we expected. 7 of 8 subjects agreed (n = 4) or strongly agreed (n = 3) that upper-limb behaviors of the avatar were authentic as if virtual upper-limbs were parts of their bodies. Additionally, 5 of 8 subjects indicated that they would like to continue to use VRRS. Results of the survey performed at the end of the rst experiment are shown in Table 1. The mean values of agency, system usability and game performance of the VRRS are higher than those of the LP.
Location of The Body, Agency and Body Ownership Fig. 6 showed the result of agency and body ownership sections of the questionnaire. The senses of agency and body ownership were measured with questions from avatar embodiment questionnaire [18]. We removed two questions about mirrors as there was no mirror in our rst experiment. Note that the senses of location of the body, agency and body ownership were enhanced by using VRRS comparing with using LP. There were signi cant differences (p < 0.05) between using VRRS and LP in terms of location of the body (p = 0.0011), agency (p = 0.0001) and body ownership (p = 0.0001).

System Usability and Game Performance
In the experiment, the system usabilities were measured with the System Usability Scale [20] and the game performance was de ned as the sum of scores of normal motion mode and mirror motion mode.
It can be seen from the gure that there was a signi cant effect improvement with VRRS in term of usability and game performance compared to with LP (p = 0.0011, Fig. 7).
Time of completion, un nished rate t-test was applied to measure the time of completing each task and the incomplete rate of two motion modes. We can see there were no differences (p > 0.05) in terms of duration time of completion and the incomplete rate (Fig. 8).
Kinematic traces of subjects using two systems Given kinematic traces of subjects using two systems, we found that kinematic traces recorded by VRRS were more smooth and stable than those recorded by LP, especially in task 1 and task 3 (Fig. 9).

Experiment 2
In the experiment 2, regarding the results of the two evaluations of the experimental group employing VRRS, the average score of post-training was signi cant higher than the score of pre-training (F(1,7) = 57.600, p < 0.001). In the control group without VRRS, there was also a difference (F(1,7) = 24.307, p < 0.01, Fig. 10) between two evaluations.
In addition, the D-value calculated from the difference between pre-training evaluation and post-training evaluation in the two groups were shown in Fig. 11.
The experimental group had more signi cant effect (F(1,7) = 50.267, p < 0.001) compared to the control group.

Discussion
The ndings in our rst experiment indicate the impact of the control method for full upper-limbs on the interaction between the user and the virtual environment. Using the depth sensor based avatar control system has been proved to result in a higher senses of body ownership and agency compared to controller based avatar control [20]. Here, we show that our control method for full upper-limbs can elicit higher feelings of body ownership, agency, location of the body as well as system usability which supports the above hypothesis. In communication with the subjects, they mentioned that there were two reasons why VRRS was rated with higher scores. The rst reason is its full representation of users in the virtual environment. And the second reason is that the avatar can be controlled relatively more stable in a larger range. We speculated that the control of full upper-limbs and representing users with full avatars can lead to a better sense in VR. This agrees with the previous results which suggested that movement tracking and representation of users in a virtual environment can affect the senses of embodiment of the users and interaction as well as presence of space [16] [17]. We also discussed why the difference in kinematic traces between task 1 and task 3 is more signi cant than that between task 2 and task 4. We believe that the mirror motion is more di cult to adapt and more challenging for the cognition which decreases the difference between task 2 and task 4.
In our second experiment, we evaluated the feasibility of VRRS in rehabilitation. The results show that VRRS increases the motion paradigm and promotes the effect of rehabilitation. In the comparison, although the Fugl-Meyer scores of both experimental group and control group were improved, the D-value of two evaluations of the experimental group has more signi cant difference than that of the control group. This is mainly because virtual reality programs are more interesting and enjoyable than traditional therapy [24] and the control method for full upper-limbs meets the needs for correcting undesired motion behaviours in order to conduct an expected motion by visual feedback [21]. The effect of VR in rehabilitation was also noted in previous studies. Saposnik et al. [24] found that VR and video game applications are novel and potentially useful technologies for upper arm improvement after stroke. In addition, there is a lot of evidence that illustrates how enriched virtual environments can be used as a therapeutic training tool which provides a learning experience tailored to individual clients [23]. We also found similar works indicates that tracking human movements becomes vital and necessary in rehabilitation scheme and the avatar upper-limbs control method are gradually applied in order to extract objective and accurate information. For example, Lupu et al. [25] proposed an affordable system with motion tracking by placing markers on limbs for stroke recovery. Tsekleves et al. [26] developed and assessed an interactive game-based rehabilitation tool for balance training of adults with neurological injury using Microsoft Kinect sensor. Additionally, we speculated that accurate virtual avatar representation in terms of appearance and behavior can affect the patient's cognition and motor recovery [8]. These ndings may be the most important result of our study and may contribute signi cantly to rehabilitation research.
The study presents several limitations. First, the current system can only track upper-limbs of one person and transfer the behaviors to an avatar. If some other people appears in the visual tracking area, they will be regarded as the subjects and their partial upper-limbs will be tracked. These may cause disorder in avatar control. So it is necessary to add the function to lock the tracked target. Another aspect which can be improved of VRRS is the assessment method. In VRRS, upper-limb motor function of the patients were completely evaluated through doctors' professional judgement. Liao et al. [31] presented that offering patient access to a clinician for every single rehabilitation session is economically unjusti ed. so they proposed a deep learning-based framework for automated assessment of the quality of physical rehabilitation exercises. This intelligent technology of assessment may be required for the further research.

Conclusion
In this paper, we introduced VRRS, a virtual reality rehabilitation system, based on the proposed control method for full avatar's upper-limbs. We also investigated the in uence of VRRS on body ownership, agency, location of the body and usability by comparing with the LP system based on partial control method. Further more, we conducted an experiment with 16 stroke patients to evaluate the feasibility of using VRRS in rehabilitation of upper extremity. The results show that the system improves the user experience on body ownership, agency, location of the body and can be employed for further optimizing treatment.