Two participants with tetraplegia were recruited for the study after providing informed consent. The study protocols were approved by the Institutional Review Board of Northwell Health (Great Neck, NY). Participant 1 was a 32 year old male, injured 6 years prior, with a C4/C5 ASIA (American Spinal Injury Association) B injury. He participated in 10 sessions (2 hours/session), out of which 7 sessions were used to record 2D and 3D arm movement trajectories. During the remaining 3 sessions, grasp selections were decoded online (in real-time) and used to drive a custom neuromuscular stimulator with textile-based electrodes housed in a sleeve (16). This in turn allowed the participant to perform functional movements (e.g. eat a granola bar). Participant 2 was a 28 year old male, injured 10 years prior, with a C4/C5 ASIA A injury. He participated in 3 sessions, which involved 2 training and 1 online testing session.
B Experiment Setup and Data Collection
Participants were seated with their hands initially resting on a table. A wireless sensor module was attached to the wrist of their arm using a Velcro strap (Fig. 1A). While both participants were bilaterally impaired, each still possessed residual movement that allowed reaching with at least one of their arms, which was then used for the study. The sensor module consisted of a 32-bit ARM microcontroller unit (MCU) from Adafruit (Feather Huzzah32) and a Bosch SensorTec BNO055 9-axis IMU. The IMU has a built-in processor and algorithms to estimate its orientation and perform gravity compensation in real-time to produce linear acceleration in three orthogonal directions. Linear acceleration along the X, Y, and Z axes was available externally via an I2C interface. A
Fig. 1. A) Experimental setup for utilizing reaching movements and enable grasping through the use of an IMU and textile-based electrodes. B) Closeup of custom-built IMU sensor module with microcontroller (battery not shown). C) Still images showing an SCI participant using the IMU to activate neuromuscular stimulation and eat a granola bar.
flexible printed circuit board was designed to interconnect the IMU with the MCU as shown in Fig. 1B. Data was continuously streamed from the MCU at 50Hz via Bluetooth to MATLAB 2019a running on a desktop PC and stored for offline processing.
During the experiments, verbal cues associated with arbitrary (2D & 3D) and natural reaching (3D) movement trajectories were randomly called out to the participant. For the arbitrary movements, the participants were instructed to perform the movements within approximately 1 second and to start from a location that allowed them to complete the movement. It is important to note that for the natural reaching trajectories the participants were instructed to reach for two different target objects on the table (a water bottle and a pen), normally associated with two different grasps (cylindrical and pincer/claw grasp respectively), and to stop just in front of the object. Due to their hand impairment, they could not complete the grasping action, but their natural trajectory information was collected to determine if there were two repeatable and discriminable patterns associated with each target object and associated grasp.
The two different natural reaching (3D) movements were called: ‘bottle-reach’ and ‘pen-reach’ (refer to Table 1). Under the arbitrary reaching movement category, four 2D movements and one 3D movement, called corkscrew, were trained. The four 2D trajectories (performed in the horizontal X-Y plane) corresponded to well-known English and Greek letters: S, ε (epsilon or E), γ (gamma), and M. Experiments were conducted in blocks of 18-20 trials and sufficient breaks were given between blocks to minimize participant fatigue. Initially, the participants were asked to perform only S and ε trajectories because these were simple to learn and didn’t cause fatigue. Later, once the participants became comfortable with moving their arm, we included additional 2D. Thus, in our final datasets there was a higher percentage of 2D trajectories (especially, S and ε) than the remaining trajectories.
During online (real-time) testing of 2D movement recognition, participants also wore a custom-built fabric sleeve with 128 textile-based electrodes over their forearm to receive neuromuscular stimulation. Neuromuscular stimulation was provided by an 8-channel proprietary, battery-operated, voltage-controlled stimulator. The stimulation parameters were set at 500µs pulse-width, 20Hz pulse frequency, and 0 - 110V output. Additional details of the stimulator design will be presented in a forthcoming publication. The stimulation channels were mapped to individual or multiple electrodes on the fabric sleeve, in order to evoke various finger flexion and extension type movements. The mapping process itself is described in detail in a previous paper (12). By grouping multiple stimulation channels and sequencing their activation profile, we could program different hand movements such as opening and closing (and different grasp types).
In a proof-of-concept demonstration in Fig.1C, an SCI participant uses a single 2D trajectory (M) to initiate a neuromuscular stimulation sequence, which allowed the participant to grasp and eat a granola bar, with his paralyzed hand. The stimulation sequence included the appropriate spatial electrode pattern for opening the hand for 5 seconds, followed by the pattern to evoke a cylindrical grasp for 5 seconds to hold the granola bar. Although not implemented here, a second trajectory (e.g. corkscrew) can be used to replace the object back onto the table. The durations for opening and grasping of the hand were selected, to allow enough time for the participant to place his hand around the object and feed himself.
C Data Processing and Machine Learning
The 3-axis linear acceleration obtained from the IMU was band-pass filtered (Butterworth, 8th order, 0.2 – 6Hz) and processed offline for identifying training samples. The magnitude of the 3-axis acceleration vector was used to identify onset of movement by setting a threshold of 0.95g. The movement onsets were then used to segment the acceleration data over time along the X, Y, and Z axes into windows ranging -0.1s to 0.9s with respect to onset. Each trial was visually confirmed to be free from any noise artifacts or if it exceeded the 1s window and such trials were excluded from further analysis. Next, to determine how discriminative the 2D and 3D trajectories were, two time series classifiers based on either a Dynamic Time Warping (DTW) distance measure or Long Short Term Memory (LSTM) network algorithms were trained separately for 2D and 3D trajectories.
The DTW algorithm optimally aligns a sample trajectory with respect to a previously determined template trajectory such that the Euclidean distance between the two trajectories is minimized. This is achieved by iteratively expanding or shrinking the time axis until an optimal match is obtained. For multivariate data such as acceleration, the algorithm simultaneously minimizes the distance along the different dimensions using dependent time warping (17). In our DTW-based classifier, this algorithm was used to compute the optimal distance between a test sample and pre-defined templates associated with the 2D and 3D trajectories. Ultimately, the template with the smallest optimal distance to the test sample, was selected as the classifier’s output. The template for each trajectory type was chosen as the training sample with the least aggregate DTW score (i.e. sum of individual scores) to every other training sample of the same type.
To implement the LSTM network we used MATLAB R2019b’s Deep Learning Toolbox with default values for most parameters. Specifically, an LSTM network comprising of a single bidirectional layer with 10 hidden units was used. This transformed the 2D or 3D linear acceleration data into inputs for a fully connected layer whose outcome was binary, i.e. 0 or 1. Next, a softmax layer was used to determine the probability of multiple output classes. Finally, the network output mode was set at ‘last’, so as to generate a decision only after the final time step has passed. This allowed the LSTM classifier to behave similarly to DTW and classify trajectory windows. During training of the LSTM network weights, an adaptive moment estimation (ADAM) solver was used with a gradient threshold of 1 and maximum number of epochs of 200. Since all the training and validation data were 1 second long, zero padding was not used.
During real-time classification of arm trajectories, the linear acceleration signals were filtered and processed in real-time using a MATLAB script that looped at 50Hz. Within the loop, the acceleration data was divided into 1 second long segments with 98% overlap. To demonstrate an initial proof-of-concept, only the DTW-based classifier was implemented, due to its simple and computationally efficient implementation in MATLAB. For online prediction, the incoming acceleration windows were compared with 2D template trajectories of each type and if the optimal distance between trajectories were below 10 units (empirically determined), then positive decision was made. This would then trigger our custom neuromuscular stimulator to perform a complete movement sequence of opening and closing of the hand.