Study design
All experiments were carried out as part of the ongoing clinical study MYKI (‘Interventional pilot study on the evaluation of the functionality, the safety, and the reliability of an implantable bi-directional MYoKinetic Interface for the natural control of artificial limbs’), which investigates the feasibility and safety of a novel bi-directional interface for hand prostheses. All surgical and experimental procedures were carried out at AOUP hospital (Azienda Ospedaliero Universitaria Pisana). The study was approved by the Ethical Committee of AOUP and the Italian Ministry of Health (Eudamed CIV-IT-22-03-039173).
Before signing of the informed consent, we verified that the participant met the inclusion criteria of the study. The participant then underwent pre-operative clinical evaluations to assess the functionality of the residual muscles in terms of contraction capabilities and eventual presence of fibrotic tissue or muscle atrophy. For comparison with standard of care solutions, the week before the implantation we fitted the participant with a two-state amplitude modulated myoelectric controller. To this aim a custom resin self-suspending socket was fabricated. It included stainless steel electrodes (Ottobock, Inc.) placed above the flexor and extensor muscles of the wrist, respectively providing the closing and opening of the hand. The participant was administered with state of the art tests commonly used to assess the functionality of upper limb prostheses, namely: the Southampton Hand Assessment Procedure (SHAP), in which the participant grasps a variety of objects24; the Minnesota Manual Dexterity Test (MMDT), in which the participant places a series of cylinders in matched holes25; the Clothespin Relocation Task (CRT), where the participant moves three clothespins from a horizontal to a vertical pole26; the Bimanual Activity Test (BAT), administered by a physiotherapist, which assesses the degree of integration of the impaired limb in bimanual tasks27. In addition, the participant performed the Pick and Lift Test (PLT), a well-established procedure in motor control studies28, used to assess the integration of sensorimotor control paradigms. The NASA TLX was administered at completion of the functional tests to evaluate the physical and mental workload29. The McGill Pain Questionnaire was also administered to provide a quantitative measure of pain related to the phantom limb prior to surgery13. Furthermore, we recorded videos of the participant performing ADLs, including: unscrewing a jar/bottle cap, manipulating a fragile object (plastic glass), extracting pills from a blister pack, slicing a soft object with a knife, holding a rubber ball or an egg while moving the arm in space, and tying the shoes. These videos allowed to qualitatively assess (i) the robustness of the controller against the limb movements in space, and (ii) the ability of the participant to finely control the prosthesis.
The participant subsequently underwent the implantation surgery, followed by a one-week rest period that allowed the surgical wounds to heal. During the following five-weeks, experimental sessions to implement and assess the myokinetic control interface were carried out at AOUP three days a week. Ultrasound imaging was used to monitor magnet position and displacement caused by muscle contraction throughout the whole implantation period. Specifically, an ultrasound scanner (MyLab Omega, Esaote s.p.a.) was used to acquire videos of the FCU, ED, and FPL muscles, both in the longitudinal and the transverse muscle planes, while the participant was performing non-fatiguing movements corresponding to a prevalent activation of each of the three implanted muscles, i.e. ulnar deviation, four-digit extension and thumb flexion, respectively. These videos were analysed offline to retrieve the displacement undergone by each magnet with respect to its rest (i.e. relaxed muscle state) position. Regarding the implementation of the control strategy, the first two weeks were dedicated to prosthetic fitting and to the characterization of the candidate control inputs and potential sources of disturbance. Based on the outcomes of this characterization, the last three weeks were dedicated to the implementation of two different myokinetic controllers (direct and pattern recognition based, see below). The evolution of the pain associated with the phantom limb throughout the implantation period was monitored by administering the McGill Pain Questionnaire on a weekly basis.
Transcutaneous Magnet Localizer
The TML employed in the study and its performance in terms of computation time, power consumption, and localization precision/accuracy were comprehensively described in our recent work12. Briefly, the system included seven Acquisition Units (Aus) each hosting 20 magnetic field sensors, one geomagnetic field compensation sensor and a microprocessor-based computation unit (CU), based on the i.MX RT1060 Real Time Processor running on an Arm Cortex-M7 core at 600 MHz. The AUs were arranged above the implanted muscles in locations empirically selected to ensure accurate and stable localizations. The AUs sampled synchronously, meaning that readings from the 140 sensors occurred all at the same time instant, thus ensuring consistent measurements. After sampling, they sequentially transferred the acquisitions to the CU which retrieved the poses of the magnets by feeding them to the localization algorithm, operating in pipeline (concurrently) with the AU sampling and AUs-CU data transferring. The AU-CU rate, defined as the interval of time at which complete data packages were transmitted to the CU, proved equal to ~ 23.6ms, which coincided with the output rate of the whole system, unless the localization algorithm took more time.
Akin to our previous works, the CU derived the poses of the six magnets by modelling the field at the ith sensor as a linear superposition of that produced by six magnetic dipoles and reversing these equations through numerical approximation methods. Specifically, the CU run the Levenberg-Marquardt algorithm (LMA) for each new data package received from the AUs, and considered the results acceptable only if the magnets were localized within a user defined workspace. Such workspace was set following each donning of the prosthesis through a calibration procedure, in which the participant was asked to contract the implanted muscles while moving the limb in space, e.g. bending the elbow and shoulder. After obtaining a comprehensive set of localizations, the final volumes were determined by adding a 2cm safety bias to the minimum and maximum x, y, and z coordinates at which each magnet was localized. During normal use, in case a magnet was retrieved outside of its volume, the localization was marked as incorrect and the event was notified to the user, while the LMA was re-run until a correct localization was achieved.
Prosthetic fitting
Following the rest week after surgery, the participant was fitted with a custom-made temporary self-suspending resin socket that allowed free placement of a variable number of AUs on its external surface. As mentioned above, we empirically searched for an AU number and placement which ensured a stable localization of the six implanted magnets, and found optimal localization precision (repeatability) using seven AUs placed above the implantation sites. The AU selection was achieved by exploiting a custom graphical interface coded through the Processing graphical library which displayed in real-time the current poses of the magnets and the AUs, and allowed to save localization and acquisition data for further offline analysis. The socket could not be connected to the prosthesis, thus it was employed during the first two experimental weeks to characterize the available control signals (see below) while the final socket was being manufactured by a prosthetist. The latter was a custom-made carbon fibre self-suspending socket which integrated all TML components in dedicated slots. Specifically, it included slots to host the seven AUs in the optimal selected locations, and two additional pockets on the outer layer and accessible from the outside that hosted the CU and the battery used to power both the TML and the robotic hand.
Signal characterization
Different datasets were acquired to investigate the candidate signals for control as well as the type and extent of potential disturbances. For this purpose, we had developed a custom Graphical User Interface (GUI) using C# in Visual Studio (Microsoft Corporation) to instruct the participant to perform specific movements while acquiring synchronized localization data from the TML.
To evaluate the independency of the three myokinetic channels at different limb positions, the GUI showed sequences of steps that the participant had to match by contracting the implanted muscles while performing different movements. Specifically, the participant was asked to perform three movements expected to induce a prevalent activation in ED, FCU and FPL with both extended and flexed elbow. These movements were respectively four-digit extensions for ED, ulnar deviation for FCU, and radial deviation for FPL (after excluding thumb flexion because the induced contraction-shift was poor). To avoid muscle fatiguing, contractions were divided into steps of 3s, separated by relaxation intervals of 8s. The acquisitions were repeated multiple times during the implantation period, in order to assess the stability of the contractions over the weeks as well as the intra- and inter-day variability due to multiple donnings of the prosthesis or arm swelling (Extended Data Fig. 1). The same acquisition protocol was also employed to assess muscle activation induced by common grasps (palmar, precision, lateral, wrist pronation/supination, wrist flexion/extension), flexion/extension of the individual fingers, and ab-adduction of the thumb. The GUI temporized the activation and rest intervals and provided a real-time feedback by displaying the contraction-shift of three user-defined magnet pairs over the step sequences. At the end of each acquisition, localization data were saved in a text file for offline analysis.
To investigate the effect of limb movements on the rest distance between magnet pairs and disentangle it from voluntary muscle activation, we asked the participant to move the elbow and shoulder following sinusoidal waves with different periods (8s, 6s, 4s, 3s, and 2s) displayed on the GUI, while keeping the implanted muscles relaxed. In particular, the participant performed elbow medial/lateral rotation and elbow and shoulder flexion/extension, and the mapping between joint angle and localization data was obtained by concurrently tracking optical markers arranged on the wrist, elbow and shoulder of the impaired limb through a three-camera 6DoF optical tracking system (V120:Trio, OptiTrack, US) (Extended Data Fig. 3). Moreover, in order to evaluate the feasibility of modulating muscle activation and capturing such modulation through localization, we asked the participant to mirror the same sinusoidal waves by gradually contracting the implanted muscles. This exercise was performed only for those movements that had demonstrated sufficient signal-to-noise ratio and robustness against limb movements in the sequence of steps.
This signal characterization procedure led to the identification of the optimal strategy and signals to implement the direct controller. In addition, the same GUI was also employed to acquire the data for training the pattern recognition controller, as described below.
Direct control
After signal characterization, the contraction-shift between FPLd and FCUd induced by wrist pronation and supination was identified as the optimal signal to control the closing and opening of the robotic hand, respectively. In addition, the speed of the contraction-shift between the magnets in ED was used to counteract the artifacts caused by limb movements, particularly by elbow flexion/extension. According to this, the controller was implemented as a four-state machine in which the robotic hand could be in the following states: (i) disabled state, i.e. no control input could be sent to the hand when the slope related to the magnets in ED overcame a set threshold th3; (ii) opening state, when the contraction-shift caused by supination overcame a set threshold th2; (iii) closing state, when the contraction-shift caused by pronation overcame a set threshold th1; (iv) rest state, when none of the previous thresholds was overcome. In case (iv), the value of the rest distance between FPLd and FCUd was updated to take into account slow variations caused by different factors over time (e.g. slow limb movements). Practically, two moving average filters with a 90ms and a 1s window were applied at discrete steps of 50ms, in accordance with the robotic hand control frequency, to update the current and rest magnet distance, respectively. The difference between these two measures, i.e. the contraction-shift between FPLd and FCUd, was subsequently compared against the set thresholds th2 and th3. In addition, proportional speed control was implemented by linearly mapping the retrieved contraction-shift to the opening and closing speed of the hand. The GUI was used to tune the value of th1, th2, and th3 and to transmit the updated values to the CU, which implemented the control algorithm and sent the appropriate command to the hand, i.e. the degree of opening/closing in the current grasp type (palmar, precision, or lateral). If the opening signal was maintained for more than 1s, the hand switched to the next grasp type.
The direct controller was assessed quantitatively by administering the same functional test used to evaluate the EMG controller (except the BAT test), and qualitatively by repeating the same ADLs. Accordingly, the NASA TLX was employed to evaluate the physical and mental workload required to complete each functional test.
Pattern recognition
We employed a SVM classifier with a linear kernel to map the co-activity pattern of a muscle group associated with a virtual movement of the phantom limb to a corresponding movement of the robotic hand. More specifically, multiple classifiers were trained to selectively recognize an equal number of classes, and a one versus all approach was applied at the training stage to determine the class with the highest score. At testing time, the latter was determined based on a majority voting approach, according to which the class that obtained the highest score in at least three out of five previous iterations was ultimately selected as the winner. The same GUI was used to acquire the data for training the models, which consisted of a number of individually acquired movements alternating 5s seconds of muscle contraction with 5s of muscle relaxation (rest). To increase the robustness of the classifier, the movements were acquired while keeping the limb in different positions, including elbow extended on the side, elbow flexed at 90°, reaching front (i.e. as to pick up an object on the table), and shoulder flexed at 135° (as to grasp an object placed on top). The GUI temporized the contraction and relaxation phases and saved the output of the localization in a text file.
Subsequent offline analysis was implemented to extract 30 time series describing the linear and angular distances between all possible magnet pairs. Due to a limitation of the CU hardware resources, 18 out of the 30 series, corresponding to nine magnet pairs which showed observable patterns associated with muscle activation, were empirically selected for model training and testing. To ensure that training was performed on the steady portion of the signal, the central 2.5s of the contraction and rest intervals of each selected time series were extracted and concatenated. The feature set was ultimately obtained by normalizing the median value of the samples contained in windows of 200-ms, with a 150ms overlap. These features were used to train the classifiers on Matlab 2017b (MathWorks Inc.), and subsequently derive the SVM parameters to be transmitted to the CU that implemented the embedded classifier. During real-time operation, the latter provided new predictions every 50ms.
We tested the real-time performance of the classifier by implementing a three-class model able to discriminate hand opening (trained on wrist supination data), hand closing (trained on wrist pronation or radial deviation data), and rest. The trained classifier was used to control the robotic hand and perform the same functional tests (including the BAT) and ADLs carried out with the EMG controller and the myokinetic direct controller, including the NASA TLX at test completion. In addition, the real-time performance of the classifier were assessed through a modified Motion Test carried out in two different configurations: (i) with the participant sitting and resting the arm on a support; (ii) with the participant standing and performing different movements in six different arm positions. The training dataset acquired for configuration (i) included three repetitions of two movements, i.e. radial deviation and wrist supination. The training dataset acquired for configuration (ii) included three repetitions of six movements, i.e. radial deviation, wrist supination, wrist extension, ulnar deviation, middle finger extension, and hand closing. The latter were repeated holding the arm in six different positions, namely: shoulder flexed at 45°, 90°, and 135°, elbow flexed at 90%, and reaching front obliquely to the left and to the right (Extended Data Fig. 2). Different combinations of the movements acquired in (ii) were used to train three classifiers, respectively including: radial deviation and supination; the previous plus middle finger extension and ulnar deviation; the previous with hand closing in place of middle extension. According to this, the test evaluated the feasibility of discriminating in real-time a maximum of classes. During the test, the participant was asked to wear the robotic hand in order to reproduce realistic loading conditions on the stump. In line with previous studies from the literature15,30, the Motion Test required 20 correct predictions within 10s to consider a motion completed. Standard metrics were employed to evaluate the test outcomes, namely: the completion time, consisting of the time between the first prediction different from rest and the 20th correct prediction; the completion rate, consisting of the number of completed motions over the total number of motions attempted. The deviation from the standard test was that, unlike what is generally reported in the literature, the completion time also included the time to reach the target position from rest (i.e. arm extended on the side).
Finally, larger datasets comprising up to 16 movements, including grasps, wrist movements, single finger flexion/extension, and thumb adduction/abduction, were acquired with the arm at rest, to assess offline the feasibility of discriminating many classes.