VRRS System
Architecture
As shown in Fig. 1, VRRS is consisted of two FLIR cameras and two computers, including a server computer (Intel Core i7-6700 at 3.40 GHz, 8GB RAM) and a client computer (Intel Core i9-9900k at 3.60GHz, 32GB RAM, and RTX2080TI). The cameras are fixed on two tripods placed on a table such that the cameras are about 2m above the ground. The server computer receives the sensor data of the positions of 6 joints of arms and 42 joints of fingers, which are recognized by the OpenPose platform configured on the client computer. The data are transfered through TCP/IP network architecture customized in C# (Microsoft Visual Studio). The distance between two cameras is about 1.4m, and they are connected to the client computer through USB cable. We used Unity game engine 2018.3.12f1 to develop the system.
Control method for full upper-limbs
With each time step, the data transfered from OpenPose platform are used to set the avatar's pose. We reorganize the received sensor data into a series of 3D coordinate data and filter the 3D data of each frame by setting thresholds to avoid obvious jitter. The data of wrists, elbows and shoulders are set in avatar’s skeleton with inverse kinematics being used to adjust the skeleton of arms. And data of fingers are used to set skeleton under corresponding local coordinate system with forward kinematics. As a result, the avatar's upper-limbs are accurately controlled and can be used to interact with the virtual environment.
Study
The main purpose of this study was to test the effect of VRRS and evaluate the feasibility of VRRS in rehabilitation training. In the study, two experiments were performed. In experiment 1, 8 young adults were recruited to experience two kinds of game-like environments using VRRS and Leap Motion controller. And totally 16 stroke patients, who have performed an occupational therapy and physicotherapeutics, participated in the experiment 2. 8 of these patients used VRRS to perform rehabilitation training. The remaining 8 patients did exercise without VRRS.
Participants
A total of 24 volunteers participated in the study: 8 volunteers (6 male, 2 female) aged 20-27 (M = 23.1, SD = 1.6) from our university through advertisement posted on the social media platform and 16 stroke patients (9 male, 7 female) with normal vision aged 11-90 (M = 47.9, SD = 22.5). Among the 16 patients, 8 patients without severe cognitive impairment formed the experimental group and participated in the tasks of experiment 2 with VRRS. The other 8 patients who participated in the tasks of experiment 2 without VRRS served as the control group. These patients were with different level of disability ranging from Fugl-Meyer [15] score of 2-50 (M = 26.6, SD = 16.4). The patients gave their informed consent to the experimental procedures, which were approved by the local ethics committee. In our first experiment, to avoid familiarization to the two systems, the volunteers were naive to our tasks and assigned to each experimental condition randomly.
Experimental Procedure
Experiment 1
In the experiment, the subjects performed two kinds of tasks in game-like virtual environments: normal motion on a plane and mirror motion on a plane, with VRRS and LP respectively, i.e. totally four tasks to be performed (Fig. 2). To avoid the influence of familiarity with the interactive environment on the experimental results, each participant performed the four tasks in random order. Before each task started, subjects were positioned in a chair and kept the initial posture the same as avatar’s upper body. Subjects’ arms were placed on
A table in front of their bodies and their hands were naturally stretched with palms faced down. Each task consisted 30 trials and the four tasks took about 20 minutes. After all tasks were implemented, the subjects were asked to fill in a questionnaire to grate both VRRS and LP.
Task 1/Task 3: Normal motion on the plane based on VRRS/ LP At the beginning, the subject controlled the avatar to move its left upper limb / virtual left hand to the starting area which was originally displayed in blue color by moving his left upper limb. Then, held for 1.5s after the color of the area turning green. After that, a blue object was randomly generated in the reaching area which was customized to each subject. Meantime, the color of the starting area turned gray and a blue line appeared connecting the object and the starting area. The subject was required to move their left upper limb to reach the object along the blue line and maintain there for 1.5s after the object and the line turned green. If the object cannot be reached within 4s, it will disappear together with the line. And the trial will be regarded as unfinished. Then the subject had 2s to move the left upper limb back to the starting area to prepare for the next trial. On the other side, if the object was reached, a score will be displayed in the upper-left corner of the screen. The score was calculated according to the overlapping between the motion trail of the palm center and the line. Then the subject moved left upper limb back to the starting area within 2s and remained suspended until the color of the starting area turned back to blue which meant a trial is completed and prepare to conduct the next trial. The first 15 trials were implemented by the left upper limb and the opposite limb completed the rest of the trials in which the subject controlled the movements of the avatar's right upper limb / virtual right hand in the same scenario (Fig. 3).
Task 2/Task 4: Mirror motion on the plane based on VRRS/ LP In these two tasks, the motion of the subject’s left upper limb was mapped to the avatar's right upper limb /virtual right hand. The avatar's left upper limb/ virtual left hand was controlled by the opposite limb of the subject. Other procedures were the same as those in the task 1 and task 3.
Experiment 2
In the experiment, every stroke patient in the experimental group participated in a three-day experiment on the feasibility of VRRS which includes six training sessions. These patients were required to conduct the session twice a day with an interval of 2 minutes. Before the first session, the motor function of the stroke patients were evaluated with FMA. Each session was consisted of two tasks including normal motion on a plane and mirror motion on a plane. Considering the duration and intensity that the patients were able to adapt to, the number of trials of each task was set to 20 and the patients took a break of 30s between two tasks. After the patients completed the three-day experiment, they were evaluated again with FMA (Fig. 4). On contrast, the 8 stroke patients in the control group were only evaluated with FMA on the first day and the third day without completing the training tasks (Fig. 5).
Outcome measures
Measures were performed using the SPSS statistical software system (version 25.0) and Origin function draw tool (version 2018).
In experiment 1, after the subjects completed all the four tasks they were asked to comoplete a questionnaire which integrates avatar embodiment [18] and System Usability Scale [19]. The questionnaire was used to assess the senses of body ownership, agency and location of the body as well as the usability of VRRS and LP. To test the difference in the questionnaire responses between
VRRS and LP, a Mann-Whitney U-test was used. In addition, t-test was also applied to test for difference in the total score, time of completion and unfinished rate between two systems. We also recorded kinematic traces of all subjects in each task from starting area to the position of the object randomly generated to analyze whether there is a difference between VRRS and LP.
In experiment 2, 16 patients performed twice evaluation of FMA. And t-test was used to test the difference in the Fugl-Meyer scores between the two evaluations of all the participants. Besides, the difference in D-value of the two evaluations between the experimental group and the control group was also tested with t-test.