Stroke is a leading cause of serious long-term disability in the United States. Projections show that by 2030, an additional 3.4 million people or 3.88% of U.S. adults 18 and older will have had a stroke, a 20.5% increase from 2012 [1]. At six months post-stroke, ~ 65% of affected persons continue to have hand deficits that profoundly affect their ability to perform their usual activities [2-4]. Therapy in an inpatient rehabilitation center in the United States only lasts about two to three weeks, and as outpatients, stroke survivors are typically only seen two to three times a week for short time periods. This volume of intervention falls far short of the volume of rehabilitation needed to re-establish normal hand function. Recently published results of innovative lab-based interventions appear to have a similar problem [5,6]. Additionally, individuals with disabilities post-stroke have difficulty accessing rehabilitation facilities due to transportation, health and mobility issues. This further reduces training volume. It is therefore imperative to develop an intervention that can be delivered at home, over a period sufficient to elicit improvements.
Although home based systems are increasing in popularity, adherence to unsupervised home exercise regimens is poor across all diagnoses and particularly so in persons with stroke [7,8]. Low motivation levels are cited as an important barrier [9,10]. Several small studies have cited higher levels of motivation associated with video game-based rehabilitation activities [11,12]. Studies observing or measuring the number of repetitions of training activities performed by individuals with stroke, describe subjects participating in video game-like training activities performing more repetitions than subjects performing traditionally presented activities [13-15].
Telemedicine has been broadly defined as the use of telecommunication technologies to provide medical information and services [16]. Recently, innovative telerehabilitation systems have been developed using information and communication technologies to provide rehabilitation services at a distance. Many studies have developed video game driven systems from commercially available gaming consoles such as Wii and Microsoft Kinect [17], however, these systems do not address hand rehabilitation. Other groups, including members of our own team, have examined the use of custom-made telerehabilitation systems [18,19]. The ideal home based telerehabilitation system must be low cost, easy to setup, and motivating to the user in order to support consistent use. Additionally, it needs to generate progress reports for the user for self-tracking, as well as provide daily monitoring to remote clinicians. Exciting new technologies have now made this approach possible and hold promise for long-term benefit. These technological advances - for the first time - allow for virtual reality games interfaced with discrete finger and hand tracking to be affordable and easy to use.
The Home based Virtual Rehabilitation System (HoVRS) was designed in an attempt to address the above objectives to provide intense upper extremity rehabilitation at home. It allows subjects to access hand/arm rehabilitation without the cost and transportation challenges associated with outpatient rehabilitation. HoVRS consists of four elements: 1) an affordable and commercially available infrared camera specifically designed to capture finger movements – which are not captured by game consoles like Kinect or Wii, 2) multiple engaging games designed to train the hand and arm using commercial gaming mechanics to optimize players’ motivation to perform these activities for long periods of time, 3) monitoring and archiving software that will allow clinicians to design custom rehabilitation interventions, track a patient’s progress, and modify a patient’s rehabilitation program, in-person or remotely, and 4) a secure wireless data connector to collect detailed information on patient movement in real time. The secure communication channel allows for remote monitoring by clinicians, remote technical support, and remote patient and clinician interaction face to face while the patient uses HoVRS. This paper describes the design of HoVRS and presents proof of concept and feasibility data from the first fifteen persons with stroke that participated in pilot testing of HoVRS in their homes.
Home based Virtual Rehabilitation System (HoVRS) Design
HoVRS has two subsystems to deliver home-based training: 1) a patient-based platform to provide the training and 2) a cloud-based online data logging and reporting system (Figure 1A). In the patient’s home, a cross-platform virtual reality training application runs video games (developed in the Unity 3D game engine using the C# language) on their home computer. The Leap Motion Controller (LMC), a low-cost, commercially available, infrared tracking device is used to capture motion of hand and arm without requiring wearable sensors which may be difficult to put on independently or could potentially restrain movement. This allows the user to interact and control the games with their hand and arm.
A. Hardware
Motion Capture
The LMC consists of three infrared LEDs and two cameras. The functional range of the LMC extends from approximately 25 to 600 millimeters above the device (1 inch to 2 feet). A data validation study showed the LMC to be accurate and reliable when the target is within its visual area (±250 mm of the LMC center) [20]. The device’s USB controller reads the sensor data into its own local memory and performs any necessary resolution adjustments. This data is then streamed via USB to the Leap Motion Image Application Programming Interface (API). Using Unity, we programmed the system to feed tracking data into virtual reality activities by calling the Leap Motion API.
Hand Positioning
Some individuals with severe proximal arm impairment have difficulty maintaining their hand above the optimal LMC capture space for the duration of gameplay. An arm support (Figure 1B) was supplied to these individuals at the initial visit to provide anti-gravity arm support to assist with positioning their hand above the LMC. This also allows patients that are unable to move their UE through space the ability to transport their hand in the LMC’s workspace. The arm support was positioned at the proximal aspect of the forearm. Therefore, wrist flexion/extension and forearm pronation/supination movements were not obstructed. Several patients demonstrated difficulties producing wrist and forearm movements when moving their upper extremities against gravity without assistance. The addition of the arm support facilitated distal movements in these subjects. Arm supports were mounted on the table where the LMC and laptop were located. If a subject’s living environment did not have the appropriate surface to attach the arm support, we provided an LMC stand to mount the arm support and LMC. The LMC stand was made in-house and was constructed of metal with a heavy and stable base. Another accommodation we provided to enable optimal hand positioning above the LMC was a hip wedge (Figure 1C), which was secured between a subject’s legs. This was especially helpful for those with some active elbow stability as well as active shoulder rotation movement. The LMC was attached with Velcro on the top of the hip wedge and the subject’s hand was easily placed above the LMC and within its effective range. A third option was a forearm trough. This was used for subjects who only had distal finger movement, allowing them to interact with the system for hand and wrist training. With their arm supported by the trough, subjects were able to focus on finger movement without using the arm and shoulder to stabilize their hand. These accommodations demonstrate the flexibility of the system with regards to physical space and impairment level of the individual.
B. Software
Games
Rehabilitation games can either be installed by engineers during the initial system setup or downloaded from the HoVRS website at the instruction of an individual’s therapist. After a preliminary configuration session that involves system calibration, subjects start the system for subsequent sessions by choosing a game from a Graphic User Interface (GUI) with a single mouse-click. Games’ initial levels are carried forward from their previous training sessions, eliminating the need for calibration at the start of each session.
Currently, twelve games have been developed which can be grouped into one of four categories: Elbow-Shoulder, Wrist, Hand and Whole Arm (see game descriptions in the Additional file 1). In summary, each game trains a specific movement pattern, such as wrist pronation or finger fractionation. The finger games include Car, Bowling, Finger Flying, and Piano. These games utilize the range of whole hand finger flexion/extension calibrations. With the exception of the Piano and Fruit Picking games, these games encourage hand opening to control the speed of movement of a virtual object in order to reach targets and avoid obstacles. The Piano game encourages finger individuation integrated with reaching. Wrist games include Whack a Mole and Wrist Flying and utilize the range of pronation/supination, range of radial/ulnar deviation or range of wrist extension/flexion calibrations. These games encourage the player to practice pronation/supination and extension/flexion in order to successfully catch or hit targets. The third category is shoulder-elbow games which include the Maze, Brick Break, Arm Flying, and Soccer Goalie games. These games require calibrations of vertical and horizontal arm range and use these arm positions to move the object or character in a game. The fourth category is whole arm games which includes Fruit Picking and Fruit Catching. Games in all categories adjust movement of the virtual objects to the range of player movement, calculated based on their calibrations. They all include either multiple levels that increase in difficulty or a single level that adjusts dynamically by algorithms. There is a configuration window that clinicians can use to set up game conditions for variables such as workspace size, activity speed, accuracy demands, etc. Clinicians can also track patient’s performance such as duration of gameplay and game score daily, weekly, and monthly.
Algorithms
HoVRS streams kinematic data of the hand and wrist for 22 degrees of freedom in addition to hand orientation and position. These include hand palm position, palm orientation, wrist position, three joint positions from each finger, (metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint) and fingertip coordinates. These data are used to control the game progression with the help of various online algorithms. There is a target movement that is shaped by each game, for example, the Flying game shapes either finger extension, wrist extension, or shoulder elevation, depending on the specific version of the game. One of the main objectives of these algorithms is to maintain the difficulty levels for each of the games within a prescribed range [21]. Figure 2 shows an example of such an algorithm for the Piano game. The subject was required to flex his active finger, which was randomly assigned by the game, to press the piano key while keeping the other non-active fingers straight. Actual fractionation was calculated as the metacarpal joint (MCP) flexion angle difference between the active finger (solid black line in Figure 2) and the most flexed non-active finger (dotted line) in real time. A piano key was successfully pressed (green line) when the target fractionation reached the pre-set target fractionation. The algorithm running in the background tracks subject’s successful key press rate. The algorithm increases target individuation if success rate is higher than 80% and decreases it if the rate is lower than 80%. By adjusting the target individuation, the subject is prompted to flex his active finger more if he/she is capable while not frustrating him when he/she is tired.
Testing Games
The following computer-based tests of hand kinematics were performed along with clinical tests to monitor adaptations to the games played during the pilot study.
- Hand Opening Range (HOR): The subject opens their hand as much as possible and closes their hand as tightly as possible. Hand Opening value is calculated as the difference in the average MCP and PIP joint angles across all 4 fingers in these two hand positions. The bigger the HOR value, the better hand opening range.
- Hand Opening Accuracy (HOA): The subject controls a cursor that moves up and down by opening and closing their hand. The subject attempts to trace an irregular wave which moves on the screen from left to right at a constant speed. Accuracy is calculated as the root mean square error between the cursor position and the corresponding target point on the wave. Root mean square error is calculated to quantify the differences in cued movement and actual subject movement (smaller error = better performance). The smaller the HOA, the better control of hand opening.
- Wrist Pitch Range (WPR): The subject extends and flexes their wrist against gravity with their forearm in a fixed position. Angular difference between these two positions is reported as WPR.
- Wrist Pitch Accuracy (WPA): The subject controls a cursor that moves up and down by flexing and extending their wrist. Task and accuracy calculations are similar to HOA.
- Hand Roll Range (HRR): The subject moves and holds their hand in pronation and supination with their elbow fixed. Range calculated in a similar fashion to WPR
- Hand Roll Accuracy (HRA): The subject controls a cursor that moves up and down by pronating and supinating their hand. Task and accuracy calculation are similar to HOA.
In addition to the measures described above, the system is capable of collecting other motions, such as radial and ulnar deviation, that we have not yet piloted with stroke subjects. Additional metrics such as speed of hand opening, finger individuation, and smoothness of spatial trajectory of the hand, are available as well making the system capable of monitoring adaptations to the entire library of games offered and customizable to a wide variety of clinical conditions.