Error Fields: Robotic training forces that forgive occasional movement mistakes

Control of movement uses error feedback during practice to predict actions for the next movement. We have shown that augmenting error can enhance motor learning, while such findings are encouraging, new methods are needed to accommodate a person’s individual reactions to error. The current study demonstrates the design we call error fields, EF, where we temper the augmentation when errors are less likely. We tested the ability of healthy participants (n=21) to adapt to a visual transformation, and we enhanced the training with error fields. We found that training with error fields led to the fastest learning and greatest reduction in error. EF training reduced error more and faster than controls who practiced without error fields (50% more and 46% faster in the target direction; 21% more and 67% faster in the direction perpendicular to that). Moreover, EF was also significantly greater and faster than our previous error augmentation (EA) technique. Hence, clear advantages exist no matter how this is measured or compared to controls. These findings represent an effective teaching method for enhanced training that leverages the statistics of error.


Introduction
Making mistakes is inherently human, and learning from them may require more salient experiences to enable the best improvements in performance. Prior research has successfully shown enhanced motor learning using error augmentation (EA) as a training technique. This technique artificially exaggerates the visual or haptic feedback of the movement errors. Because error feedback is of particular importance to motor learning, enhanced error drives enhanced changes to the feedforward plan 1,2 . A variety of tasks have been shown to mildly benefit from using error augmentation techniques, including upper [3][4][5] and lower extremity activities [6][7][8][9] . Like other forms of interactive virtual environments, the novel experience of error augmentation could serve to increase overall engagement to promote improvements in performance; after all, attention and reward systems are essential for reinforcing memories of motor experiences. However, sensorimotor manipulations that specifically make errors more noticeable improve the rate and amount of motor learning 10 . In fact, error augmentation promotes learning gains that are not present in assistive approaches 11 . On the other hand, studies suggested that augmenting with gains outside of an appropriate range is detrimental to learning 12,13 . Evidently, excessively large or spurious errors diminish the adaptive response 14 .
The inherent challenge of designing intensive training is to promote learning that is relevant outside of training [15][16][17][18] . The motor system combines multiple sources of information in a statistically optimal fashion 19,20 , so learning from the strangeness of augmented feedback must be somehow reconciled with learning of lifelong experiences. Error augmentation, however, naturally provides a smooth transition since sufficient training should reduce errors enough so that nothing remains to be augmented. Even so, recent studies show that motor learning and successful skill transfer are influenced by the statistics of past experiences of error feedback 21,22 . Hence, unexpectedly large signals during a training session could disrupt learning. Yet these findings suggest an exciting possibility: could the statistics of error provide engineering guidelines for how to focus training? Such streamlined augmentation might reinforce existing statistical processing, while at the same time present only minimal distortion to sensorimotor experiences. b a Figure 1. Participants performed a planar reaching task using the (a) ARMin exoskeleton device restricted to shoulder rotation and elbow extension, (b) Overhead view of the experimental setup with ARMin device locked in a planar configuration with targets displaced on a vertical computer screen rehabilitation, by gradually increasing load 23 , adapting forces as needed 24 , or simply allowing participants to choose therapy models 25 . However, such methods that use a Challenge Point Framework that suggests training difficulty should be enhanced to match the learners initial success level 26,27 .
Similarly in therapy, customization is especially vital to address wide-ranging differences in impairments. Our research group already has shown that the initial patterns of abnormal coordination can be characterized and then reshaped with custom forces [28][29][30] . In a similar manner, we found that the errors seen during goal-directed reaching can be described in terms of simple probability distributions [31][32][33][34] . Characterizing the statistics of an individuals reaching movement effectively allows not only a scientific assessment of specific repeatable deficits, it should allow us to make use of such error likelihood and custom-design an effective training.
Here we evaluated this new form of error augmentation that is customized according to each individuals specific error tendencies. Our approach begins with a statistical profile of error trajectories for a given participant. We then formulate an algorithm that augments error only in regions of high probability of error. We conducted an experiment in which participants learned to make straight-line reaches in a visually distorted space. The motion of the on-screen cursor was driven by joint angles of the shoulder and elbow similar to that of Flanagan et al 35 . This task required participants to improve both spatial and extent (timing) errors during reaching. We hypothesized that this new error field treatment (EF) would provide faster learning and improved skill transfer to non-augmented conditions. Study results were presented in preliminary form 36 , and here we include an examination of learned behaviors customized for each participant.

Participants
This experiment employed a seven degree-of-freedom robotic arm exoskeleton device, the ARMin (Figure 1a), located at the Sensory-Motor Systems Lab at ETH Zurich (Zurich, Switzerland) 37 . Twenty-one right-handed participants (10 female) performed the experiment, with 7 participants per group.

Ethical Approval
All the experimental protocols were approved by the Ethics Board of Canton Zurich. All methods were following guidelines and regulations set by the Ethics Board of the Canton Zurich. All participants provided informed consent in accordance with the Ethics Board of the Canton Zurich.

Experimental Protocol
We investigated motor learning in a novel visuomotor transformation as a test-bed for evaluating the training benefits of augmenting error feedback. Participants were asked to perform movements using the robotic exoskeleton with their right arm. In normal reaching conditions, healthy participants typically exhibit only negligible errors after they have become familiarized with the robotic device.
To facilitate planar reaching, a proportional-derivative controller was used to orient the robot such that only shoulder rotation θ 2 and elbow flexion/extension θ 4 were possible ( Figure 1b). Friction, gravity, and viscosity compensation were applied to counter effects of moving an exoskeleton device.
To introduce a challenging motor task, we selected a novel visuomotor transformation similar to the study by Flanagan 35 . In this environment the cursor position is shown moving in linear (Cartesian) space, but actually represented joint angles of the shoulder and elbow. Visual feedback was presented with a vertical computer screen approximately 1.5 meters away ( Figure  1b). While this task environment required that participants control movement through joint angles, motion was visualized by plotting shoulder vs elbow angles as coordinates on the video screen.
For each trial, the task required participants to move along a straight-line path between two targets located 15 centimeters apart on the screen (See Figure 1b). Participants were instructed that they could begin the reach as soon as the target appeared on the screen. Participants were allowed to rest in the target before initiating the next reach. After completion of a reach, feedback on movement time was given: an ideal reach duration of 1.5 seconds would result in the target turning green when the trial was complete. This time constraint was determined from pilot data of baseline reaching movements in normal conditions. Note that we included an additional task that participants perform reaches with a minimum jerk trajectory 38 , though this goal was not presented explicitly on screen. More details about this task are described in next section.
During Baseline (See Figure 2), participants became familiarized with the device by receiving real-time visual feedback of their endpoint position in normal conditions (Cartesian coordinate mapping). During Intermittent Exposure, we presented participants with a remapping of joint space to Cartesian space, randomly presented one in seven trials. This randomized schedule revealed each participants initial performance prior to learning. During the first half of the Intermittent Exposure, participants practiced with the same target set later used for training. The second half involved a new set of targets that were later used to test generalization of training. All participants experienced a short pause after Baseline prior to commencing training. Participants then trained for seventy trials (Training) where the Error Field (EF) Group and Error Augmentation (EA) Group received motor torques and the Control Group received only visual feedback. Following training, participants were evaluated on their ability to control the cursor with joint angles on the target set used during training.  (4)).

Movement Error Analysis
We used two orthogonal components of error by resolving error vector for instant in time to movement-based coordinates. First, perpendicular error was defined as the perpendicular deviation of the trajectory from the straight-line connecting origin of the movement and target, as shown in figure 3a. Secondly, we measured the extent error defined as the distance away at each instant of movement time t from the "ideal" minimum jerk motion trajectory 14 position x MJ which optimizes both smoothness and accuracy, following a straight-line path between the origin, x o and target x T : The terminal time of ideal movement is t d = 1.5 s and the distance between origin x o and target x T is 15 centimeters. Both error types are illustrated in Figure 3a. Individual and group error analyses were performed using each error measure separately. The maximum extent errors measured from a representative participant during the entire experiment are shown in Figure  The mean µ and standard deviation σ were fit with 7 th order polynomials, which enabled a normal distribution to be formulated at each sample in time. (c) The error field was then designed to apply torque (indicated by red arrows) according to that magnitude and probability of errors in separate functions for extent and perpendicular errors.
to the novel task environment, participants exhibited on screen movements that deviated both leftward and rightward to the target (as shown in Figure 4). Errors were signed positive for rightward deviation and negative is leftward deviation from a straight-line path.

Shoulder Angle
Elbow Angle To quantify error changes across training, we computed the average maximum extent and average maximum perpendicular error from initial exposure and final test trails (post-training). We subtracted the initial average error from the final average errors to determine the change. Note that the Intermittent Exposure and final Test phases both featured the joint-space mapping for both groups, and hence served as fair points of comparison for the change in error.
The maximum extent and perpendicular errors occurring during the training phase ( Figure 4, dashed line) were fit as a function of trial number using a nonlinear Nelder-Mead regression: where ε trial is the trajectory of error across trials of training, A is the amount of learning i.e. change in error due to learning, B is the time constant indicating the number of trials for the error to decrease to 63% of it's steady-state value and C is steady-state value of error, this procedure has been previously discussed 5,15 .

Error Field Treatment Algorithm
Participants were presented with novel visual transformation during intermittent exposure trials. We characterized these trials to formulate a profile for each participant according to the probability of error throughout the trajectory. Such profiles were computed for both measures of error described in previous section perpendicular and extent error. At each time sample, we used a single Gaussian distribution to fit the data, resulting in a time-based function that is continuous across the space of possible errors, this method was described 31 . The following steps describe the technique for determining the probabilities of reaching errors in the joint angle task space: 1. Extent Error (ε ext ) and perpendicular error(ε perp ) were computed along the trajectory, for each movement direction d.
2. Mean (µ) and standard deviation (σ ) of perpendicular, ε perp , and extent error, ε ext , were calculated independently from trials during Intermittent Exposure where participants were first exposed to the joint angle task space.
3. Seventh order polynomials were used to fit the error f µ (t), and standard deviation f σ (t), across the first 1.5 seconds of the trajectory for both dimensions of error, resulting in two polynomials describing the mean error and two describing the standard deviation of error.
4. The probability, p, of a given error occurring at time t using a single Gaussian distribution is then constructed using the polynomial functions described above, as a function of time and error: where d indicates movement direction number, and i is the component of error (extent, perpendicular).

Custom-designed Training
The Error Field (EF) and Error Augmentation (EA) groups received torques from the robotic interface as a function of their real-time errors. For extent error, the interface presented loading as a function of deviation motion along a straight-line path, with respect to the ideal trajectory. For perpendicular error, the interface presented loading as a function of deviation from the path, orthogonal to the direction of movement. Note that while the task was visualized in terms of a Euclidean space, the loading was presented in terms of torque at each joint of the exoskeleton arm.
The scale factor λ was determined for each subject based on the characterization of error from Intermittent Exposure. For both extent and perpendicular error, λ was found such that 80% of estimated torque during the trajectory would range from approximately 5-15 Newton-meters. This was done per subject to maintain similar maximum torques for each participant. This value was constant throughout the trajectory and across the entire learning phase. The torques were generated in real-time for the EF group additionally considered probability of error p as defined in equation (3): For comparison, a contrast group experienced the previous method, EA: Note probability p, torque τ and error ε were treated as vectors with components in the direction to the target ("extent") and its perpendicular. Torques were presented only during the first 1.5 seconds of movement so that participants were able to reach the target and complete the movement. Finally, the resulting maximum torques applied were limited to 15 Newton-meters by software, which was well below the robots hardware safety limits.
Note that there were two ways for participants to experience minimal torques during training errors could be very small, or errors could be unlikely (low probability). Even large errors could result in low torques. The effect of this intervention design is to focus training on repeatable errors. Spurious or infrequent errors were de-emphasized. We expected that this type of training allows the motor system to prioritize adaptation( to frequent errors).
If the EF method truly influences learning, then the shape of the EF should also predict how errors evolve. We hypothesized how torque profiles (rendered according to equation (4)) should diminish with training, and error that is largest and most consistent should reduce the most. Using the initial error distribution, we defined a function γ as the error change from initial to final training, at each time step. Because change should also depend on certainty of error, we also scaled this by its standard deviation: Note that this prediction of the shift (reduction) of error is unique to each individual. We then computed the coefficient of determination (R 2 ) for the predicted versus final error trajectories after training. Because EF training made use of customized forces, we hypothesized that γ would predict the error decrease best for the EF group.
We examined statistical significance of changes in errors using repeated measured analysis of variance (ANOVA) and followed with paired t-tests to test the difference between groups. We used Bonferroni-Holm corrections for post hoc pair-wise comparisons.

Results
We found that indeed, EF participants learned more and faster. Training with custom error-fields resulted in the greatest reductions in error (Figure 5a). The EF Group decreased error by 30 ± 10 and 19 ± 18 degrees for the extent and perpendicular components of error, respectively. This was 50% greater than controls for extent (p=0.010) and 67% greater than controls for perpendicular error (p=0.042).
Examination of error change across training indicated evidence of faster learning for the EF condition (See Figure 3b). For extent error, the estimated time constant was 12 ± 6 for the EF Group, which was lower than the 32 ± 16 trials for the EA Group (p=0.0141) and 36 ± 15 trials for the Control Group (p=0.0406). For perpendicular error, the time constant was 57 ± 13 trials for the EF Group, 84 ± 43 for the EA Group, and 106 ± 43 for the Control Group. There was a significant difference between the EF Group and Control Group (p=0.044), more specifically, on average EF group learned 67% and 46% faster for extent and perpendicular errors, respectively. Again, statistical significance was not observed when compared to EA group.

Change in error (degrees)
Time Constant (trials) a b Comparing the EF to the previous method, EA, also revealed advantages for this new EF method. Most interestingly, post-hoc tests of perpendicular error detected that the EF Group learned 15.7% faster than the EA group (p=0.0392). Other tests, however, failed to detect significance ( Figure 5).
While error field training was not designed to cause changes in variability, it was plausible that a novel environment could introduce new sources of uncertainty in motor planning. To check for such changes, we computed ensemble standard deviation of error time records. We found that changes across training was higher for extent (EF:−0.4 ± 0.4, EA:−3.1 ± 3.2, Control:−6.2 ± 6.1) than for perpendicular error (EF =−0.7 ± 0.9, EA:−0.2 ± 3.2, Control:−0.2 ± 3.4), but did not differ significantly between groups.
All groups showed an ability to generalize what was learned to unpracticed target directions. We evaluated how error on such directions changed across training, between intermittent exposure and post-training trials. All participants had significantly lower reaching errors on these motion. However, we did not detect differences between training groups (EF, EA, or controls that received no torques) on the ability to generalize what was learned.
We found no evidence that group differences was due to training intensity. We compared the average torque received by each group during the start of training to ensure that the EF torques were initially scaled to match the EA torque. During the first five trials, peak torque was similar for both the EA and EF Groups (8.7 ± 1.7 Nm and 8.2 ± 1.8 Nm respectively) but differed in location during the trajectory (570 ± 330 ms and 192 ± 90 ms respectively). During the last five trials peak force for EF Group was 2.8 ± 1.3 N and 6.1 ± 2.6 N for the EA Group. The location of peak force was similar for both the EA and EF Groups (393 ± 160 ms and 322 ± 271 ms, respectively).
As we intended, the error reduction from EF was the most predictable, providing evidence that the training technique leveraged the error statistics processing we hypothesized. Error change divided by the standard deviation of error (γ) showed the best fit the EF Group (γ = 1.3 ± 0.3 for extent error and γ = 1.3 ± 0.4 for perpendicular error). This shift, γ, was consistent across for both error types (Figure 6b). We used the coefficient of determination (R 2 ) to measure how well we could predict the shift in error distributions for participants in the EF (Figure 6c). This R 2 was 0.38 (CI: 0.19,0.57) for extent error and was 0.23

6/11
(CI: -0.16, 0.62) for perpendicular error. While not high, it was significantly lager than the R 2 for the EA and control groups, who had negative R 2 , indicating poorer predictions than their mean. Hence, only the EF data could predict the shift in errorsabout 1.3 standard deviations.

Discussion
This study evaluated a novel method of custom designing of robot-assisted training algorithm for motor skills, which was based on error probability. We found that such training led to faster learning and greater error reduction across training -as much as 50% more and 67% faster, and it also outperformed our previous error augmentation (EA) techniques to enhance motor learning 4-6, 12, 39, 40 . In contrast to these studies, this current approach modulated the augmentation based on the likelihood the error. This design was motivated by the idea that the augmented feedback should bear some similarity to an individuals previous motor learning experiences. The improved performance of customized error augmentation demonstrates the potential of incorporating movement statistics into the design of robotic training. Recent studies suggest probabilistic processing is an inherent part of the motor system 20,41 . In such Bayesian learning models, movement errors that are similar to expectations would drive adaptation while spurious errors would be given little weight. We believe that our intervention supports this learning process by emphasizing useful information (in this case, the most consistent reaching errors). In response to a varying environment, humans can combine an appropriate feedforward and impedance control strategy 42 . Herzfeld et al 22 showed that prior experience of error governs the amount of learning. Furthermore, researchers have shown that providing feedback related to frequency of previous errors can improve the rate of adaptation 43 . In our current paradigm, we are ensuring that the more likely errors are amplified during training. These changes align with recent findings from Takiyama et al. who showed that people learn most when they have made that error before 44 .
Through our analysis (See Fig. 6), we also demonstrated how reductions in error were indeed related to the statistics -the error likelihood. Importantly, the error likelihood profile was unique to each subject and varied dramatically within (i.e., from the start to the end of) each movement. Many studies of goal directed movements focus on models of adaptation based on trial-to-trial error. Our findings suggest that a more comprehensive approach would be to consider how probabilities of error throughout the trajectory changed during learning. Beyond variation between individuals, it was clear that within an individual the error likelihood varies along the trajectories and can be different for each direction of movement (See Fig. 3).

7/11
The differences in learning rate between groups (See Fig. 5B) suggest that the error fields improved the efficiency of training. Our results indicated that the EF Group had the smallest time constant. At the beginning of training, errors that were similar to the most frequently recorded during initial exposure received relatively larger torques. One possible explanation for the more rapid learning is that participants responded to the error field as if it were a virtual wall 45 , and hence avoided these haptic interactions. Participants of the EF group were incentivized to express novel movements, which would have then incurred lower torques. In contrast, participants of the EA group would not have been incentivized to adopt any new behaviors other than to reduce error.
Secondly, it might be supposed that participants adopted impedance control via muscle co contraction to resist the loading. Such actions are required in an unstable force environment 46,47 . However, we chose a task for this study that required proper learning of new joint motion patterns rather than altered forces. Hence, subjects would not have actually benefitted from impedance control.
Another possibility is that the lower overall torques in the EF condition presented advantages in terms of lower workload. However, we scaled the magnitude of applied torques for each group according to the errors observed during intermittent exposure. Consequently, loading should be least for the EF condition. Nevertheless, no significant differences were detected between the groups.
While we established group differences in overall learning, we also determined whether changes in performance reflected the structure of the individual's error field in a predictable fashion. We made predictions of how error would reduce based on the initial distribution of error (Fig. 6). When comparing the initial to final movements, we found EF Group reduced errors by about the same proportion about 1.3 standard deviations of error, and this R 2 was positive only for the EF group. Previously, researchers have found that errors post-training showed evidence of motor learning and internal model formation 30,31 . In the current work, such predictable changes in error trajectories supports this as a possible intervention for motor skill training applications where the result can be forested.
The differences we found between components of error further highlight how augmentation can lead to specific changes from different training environments. Our previous studies have shown that error augmentation accelerates learning of perpendicular component of error 5,15,26 . Here we observed an advantage of the EF treatment over the previous EA technique for extent component of error (Fig. 5a). While the trend was similar for perpendicular error, we failed to detect a significant advantage in learning rate from the EF treatment. The opposite was true for overall error reduction the EF group significantly reduced error more than EA group in perpendicular error, but there was no detectable difference in extent error. Both forms may be used interchangeably to obtain different results or target a specific deficit in the learner.
The fact that error reductions differed by component 48 also suggests a form of performance optimization, where some variables are prioritized at the cost of others 49 . The motor system might prefer straight-line movements given strong visual feedback represented as a Cartesian space 50 . In this task, extent errors are essentially timing errors that do little to change the apparent task outcome, perhaps internalized as movement straightness or final endpoint deviation. However, we believe that the added benefit of our treatment focused subjects' attention on extent error, allowing participants in the EF Group to selectively decrease a distinct component of error that would not otherwise have been possible. In other words, the EF approach might be used to bias learning tendencies away from what is predicted by typical processes, such as the UCM.
While the error field treatment resulted in the best learning in this study, residual errors remained at the end of training (See Fig. 6a). This pattern of incomplete learning was likely due to the fact that the customization was based only on the initial characterization phase and did not reflect changes in error patterns during training. To truly capture the challenge point 27 required for optimal learning and ultimately drive error to zero, we would need to re-characterize error as training progresses. Furthermore, one disadvantage of our current implementation of error fields is that it assumed symmetry in error about the mean trajectory. Future implementation of this technique might feature non-normal distributions, or additional Gaussian components for characterizing error as well as continually updating error fields as the training progresses. Such strategies would ensure that smaller distributions of error that occur later in training will also be further driven towards zero error. Re-characterization techniques have been successful both with adaptive resistance 51 and assistance training 25 .
Another important consideration is that while this study chose to parameters error distributions as a function of time, many other choices could easily be implemented. Our current design was inspired by our previous study that showed better fit across time compared to across state 14 .
This paper represents a critical step in training design methodology. The approach we used can easily be adapted for specialized tasks whenever one desires to accommodate wide differences in movement errors across subjects, even when neurophysiological or biomechanical sources of error are unknown. Note that the error fields considered in this study accommodated differences across the trajectory as well as movement direction. This design provided a detailed level of customization (or personalization), more than simply adapting to each subject. We believe that this strategy of focusing the intervention on error probabilities is applicable to any training situation where error can be measured and characterized. Beyond enhancing training, our findings support the idea that the nervous system relies on statistical error information for learning 8/11 which could extend to additional tasks that have a defined goal or trajectory. Understanding such neural mechanisms of learning could then provide both inspiration and a mathematical algorithm for better training interventions. We hope this work might inspire others in a variety of fields, including neurorehabilitation, robotics, sports training, music performance, education, piloting, or surgical training.

Data Availability
The data sets analysed during the current study are available in the Dryad repository 52 , https://doi.org/10.5061/ dryad.prr4xgxnv.