Performance of dexterous object manipulation is enhanced by grasp force modulation

doi:10.21203/rs.3.rs-1559286/v1

PDF

Article

Performance of dexterous object manipulation is enhanced by grasp force modulation

https://doi.org/10.21203/rs.3.rs-1559286/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted 29 Apr, 2022

You are reading this latest preprint version

Dexterous manipulation relies on the ability to attain two goals: controlling object position and orientation (pose) and preventing object slip. Although manipulation has been extensively studied, previous work focused on the control of digit forces for slip prevention. Therefore, how digit forces are coordinated to simultaneously prevent slip and control pose remains unknown. We developed a task requiring subjects to lift an object at different grip configurations while preventing it from tilting. We then mathematically decomposed digit forces into manipulation and grasp forces for pose control and slip prevention, respectively. We found that exertion of seemingly excessive grasp force was instrumental in enhancing manipulation performance. This novel phenomenon was more pronounced at the onset of lift than during hold, underscoring anticipatory mechanisms of pose control before feedback is acquired following object lift. Our approach and findings open new research avenues for investigating neural mechanisms underlying dexterous manipulation and clinical applications.

Imagine grasping a glass of water and lifting it for a sip. To successfully perform this task, the central nervous system (CNS) has to solve a complex problem: you do not want the glass to slip, and therefore may want to squeeze it hard. However, you also want to accurately tilt the glass so that you can drink from it without spilling its content, which requires dynamically changing the digit force distribution. Importantly, digit forces have to be coordinated to simultaneously attain both of these task goals. Although humans have an exquisite ability to perform dexterous manipulation, how digit forces are coordinated to simultaneously prevent object slip and control object position and orientation (pose) is unknown.

This gap in our knowledge is due to the fact that most previous work has focused on digit force modulation for preventing object slip but were devoid of dexterity requirements. Therefore, the analyses developed to determine object slip prevention mechanisms focused on the coordination of forces that are normal (grip) and tangential (load) to the grasp surfaces.^{1,2,11,12,3–10} However, when manipulation also requires object pose control, the two different roles played by digit forces cannot be separated using conventional digit force analysis. For example, the normal component of contact forces contributes to both object slip prevention and pose control when the normal contact force vectors are not collinear.^13,14 In contrast to studies that focused on object slip prevention, Andrew Gordon and colleagues introduced a dexterity component by requiring subjects to coordinate digit forces to minimize object tilt¹⁵, which we later modified by allowing subjects to choose where to grasp the object, i.e., ‘unconstrained grasping’.^13,16–18 However, as this protocol elicited small trial-to-trial digit force modulation, data could not be analyzed to quantify how digit forces are coordinated to concurrently prevent object slip and control object pose. To address these gaps, in the present work we used an approach that combines (a) a novel application of digit force analysis tools designed for robotic manipulation and (b) a dexterous task that requires simultaneous object slip prevention and tilt minimization at different grip configurations.

In robotics, the set of contact forces required to manipulate an object are determined based on the desired manipulation goal and contact distribution.^19,20 This process relies on mathematically decomposing digit forces into two functional components:²¹ manipulation force (F_M) and grasp force (F_G ). F_M is the force responsible for dynamic control of object pose which, in our task, corresponds to the object tilt minimization requirement. In contrast, F_G – equivalent to the internal force – has no effect on object movement and its role is to prevent object slip. F_G is limited only by digit strength, object fragility, and contact surface condition. This manipulation and grasp force decomposition (MGFD) algorithm can therefore effectively disentangle the multiple roles played by digit forces.

Regarding our dexterous manipulation task, we asked participants to grasp an instrumented inverted-T shape object (Fig. 1A) with thumb and index fingertip, lift it while preventing it from tilting, and hold it. We systematically changed the object’s mass distribution and the vertical distance between the digits (offset; Fig. 1B). The former factor was introduced to add a dexterity component to our manipulation task. The rationale for increasing digit offset was that it changes the moment arm of normal digit forces, and therefore their contribution to manipulation. We analyzed digit forces at two task epochs, object lift onset and hold. As shown by previous work, these two epochs engage distinct sensorimotor mechanisms. At object lift onset, anticipatory control mechanisms dominate and are based on sensorimotor memory of previous manipulations, as no feedback of the object mass distribution is available, yet.^13,15,16 In contrast, sensory feedback is acquired throughout the lift which can be used, if needed, to modify the digit force distribution used at object lift onset, thus leading to a steady-state digit force distribution during object hold.^9,11,22

Decomposing digit forces into grasp and manipulation forces allowed us to address the following question: How do humans coordinate F_G and F_M to minimize the risk of both object slip and tilt? Here we consider two of the possible coordination solutions to attain these two task objectives. The first solution consists of squeezing the object harder for larger digit offsets. This is because increasing digit offset increases the risk of object slip as the angle of the F_G vector becomes more tangential to the contact surface, i.e., as it approaches the boundaries of the friction cone. To address the increasing risk of object slip, exerting larger F_G with increasing digit offset brings the total force vector to be more perpendicular to the contact surface. Furthermore, according to Amontons' friction law of dry friction,^10,23 increasing the magnitude of the normal component of F_G expands the volume of the solution space of F_M. Importantly, this phenomenon makes the control of manipulation more robust to sensorimotor noise, external perturbations and/or execution errors. However, object slip prevention would be implemented through unnecessarily large F_G.

An alternative solution that would be efficient in terms of both energy cost and object slip prevention across grip configurations consists of maintaining a constant ratio between F_G and the minimum F_G required to prevent object slip, i.e., an extended definition of the conventional safety margin (SM).^1,11,22,24 This solution, however, would maintain the volume of the F_M solution space constant across digit offsets, hence resulting in less robust manipulation control than the first strategy. We note that the main difference between these two solutions is that the former is implemented by increasing SM with larger digit offsets, whereas the latter is based on maintaining a constant SM across grip configurations. Therefore, both strategies can be considered as different trade-offs between energy expenditure and robustness of manipulation.

Although SM has been reported to increase as a function of unpredictability of task dynamics,²⁵ in predictable task contexts subjects adopt a constant SM across object properties^9,10,12,26 and grip configurations.¹³ Therefore, we hypothesized that, in our predictable contexts, subjects would prioritize minimizing energy cost over an increasingly large manipulation solution space by maintaining a constant SM across grip configurations.

Subjects quickly learned dexterous manipulation regardless of grip configuration

We asked subjects to perform a dexterous manipulation of a sensorized object using a precision grip at different grasp configurations and object’s mass distributions (Fig. 1). Our task required coordination of grasp and manipulation forces to simultaneously prevent object slip and tilt, respectively (see Materials and Methods). To determine the effects of digit offsets, linear mixed-effects models (LMMs) built with random intercepts for the random-effect factor Subject were fitted for all dependent variables.

To minimize object tilt, subjects have to learn to generate a compensatory torque (T_COM) at object lift onset to counter the external torque (T_EXT) caused by the mass added to the left or right side of the object’s base (Fig. 1B). At the same time, subjects have to exert digit forces to prevent object slip during manipulation. All subjects quickly learned our dexterous manipulation task, as indicated by the attainment of a stable T_COM after the third trial (t₂₈₃₈ = 4.52, p < 0.0001). Figure 2A shows kinetic variables from two trials (trial 4) of right center of mass (CM) conditions performed at zero and large digit offsets (R0 and RL, respectively) by a representative subject.

The top row of Fig. 2A depict time courses of the magnitude of the contact forces vector (F) at each digit (the thumb and index finger, blue and red lines, respectively). The bottom row of Fig. 2A shows the time courses of compensatory torque (T_COM) and object tilt angle. T_COM was computed according to the equation [4] (see Data analysis in Materials and Methods). After contacting the object with both digits, T_COM approached the external torque caused by an added mass (T_EXT) around lift onset, resulting in small deviations from the object’s vertical orientation in both experimental conditions. Figure 2B illustrates the digit force vectors at the two discrete epochs we used for analyses (lift onset and hold, left and right column, respectively). Digit force vectors were larger for smaller digit offset conditions regardless of the side of asymmetrical CM location and epoch. The direction of the force vectors appears to remain constant from lift onset to object hold while their magnitude is modulated according to the experimental condition. This phenomenon suggests that a digit force coordination plan had already been established before the onset of object manipulation, i.e., object lift, to be only fine-tuned throughout object lift and hold. Furthermore, our experimental manipulation of digit offset caused distinct patterns of digit force coordination. For example, the magnitude of the thumb and index finger force vectors was greater for zero than for large digit offsets at object hold (R0 and L0 versus RL and LL, respectively), this difference being smaller at object lift onset.

Subjects quickly learned to generate the required T_COM at both object lift onset and during the hold phase. After the third trial in each condition block, T_COM was slightly lower than the external torque at lift onset but matched more closely T_EXT during object hold (Fig. 2C and Supplementary Figure S1). Task performance was evaluated by computing the difference between T_EXT and T_COM, i.e., the resultant torque, T_RES, where a value of zero denotes optimal task performance (minimum object tilt). The observations of the performance by the representative subject in Fig. 2 were confirmed by analyses of data from all subjects using three-way LMMs on T_RES with factors CM × Digit Offset × Trial at lift onset and object hold. At lift onset, subjects learned to exert T_COM after the third trial according to the CM conditions (significant coefficients: −80.00 N·mm, t_3219.26 = −12.04, p < 0.0001 and 79.86 N·mm, t_3219.99 = 12.96, p < 0.0001 for the left and right CM, respectively). During the hold phase, T_RES was close to zero for all conditions, including the left CM condition for which we found a significant but small coefficient (−7.67 N·mm, t_3219.06 = −3.55, p = 0.0004).

Grasp and manipulation forces are concurrently modulated as a function of grasp configuration

To understand the force coordination strategies used by subjects to simultaneously prevent object slip and control object pose, we decomposed digit forces and moments into grasp (F_G) and manipulation (F_M) forces (see Analysis of grasp and manipulation forces in Materials and Methods).

Figure 3A shows the time course of the magnitude of total digit force (F), F_G and F_M vectors, together with the magnitude of the minimum grasp force vector required to prevent object slip (F_G^min) from the same subject and experimental conditions shown in Fig. 2A. F_G and F_M increased in parallel from contact and throughout the manipulation, although the rate at which F_G increased was greater than F_M. Also, the difference between F_G and F_M was much greater for the zero than large digit offset (R0 and RL, respectively). However, regardless of digit offset, this subject exerted two-fold larger F_G than F_G^min. The patterns described above for one subject (Fig. 3A) were common to all subjects.

Figure 3B plots the magnitude of F_G, F_M and F_G^min as data clusters for each experimental condition as an ellipse measured at object lift onset and hold (Fig. 3B, top and bottom row, respectively). As the measured digit offset in each condition was not identical across trials even within the same subject, for graphical purposes we plotted the data as ellipses to illustrate across-trial and subject variability of digit offsets (horizontal axis) and forces (vertical axis). Each ellipse was computed using principal components analysis applied on data from trials 4–20 of all subjects in each digit offset condition (0, L, M, S, and U) (see Materials and Methods). We found that subjects exerted smaller F_G with increasing digit offset. This negative relation can be explained by the fact that as the moment arm, i.e., the digit offset, increases, so does the contribution of digit normal forces to T_COM (t₂₆₈₅ = − 22.26, p < 0.0001 and t₂₆₈₅ = − 32.38, p < 0.0001 for lift onset and hold, respectively) (left column, Fig. 3B).

As the required T_COM is constant within each asymmetrical CM, the greater contribution of digit normal forces to T_COM leads to a decrease in the contribution of digit tangential forces. In turn, the decrease in digit tangential forces lowers the magnitude of F_G^min. This chain effect can also explain the same negative relation between F_G^min and digit offset (t₂₆₈₅ = − 48.30, p < 0.0001 and t₂₆₈₄ = − 63.64, p < 0.0001 for lift onset and hold, respectively) (left column, Fig. 3B), although the estimated F_G^min was much smaller than F_G across all experimental conditions. The much larger F_G compared to F_G^min suggested that factors other than mechanical task requirements influenced the modulation of F_G.

As object properties remained constant across digit offset conditions, one would have expected F_M to remain constant. Surprisingly, however, subjects exerted larger F_M with increasing digit offset, i.e., an opposite relation with digit offset compared to F_G and F_G^min (t₂₆₈₈ = 33.59, p < 0.0001 and t₂₆₈₇ = 61.61, p < 0.0001 for lift onset and hold, respectively) (right column, Fig. 3B).

In sum, we found opposite and asymmetrical effects of increasing digit offset on F_G and F_G^min relative to F_M (Fig. 3B). To determine the mechanisms underlying these effects, we analyzed the relation between digit offset and the non-mechanically obligatory portion of F_G, i.e., the relative grasp safety margin (SM_G), defined as the ratio between the excessive grasp force above the force required to prevent object slip (F_G^XS) and F_G (see Methods).

Relative Grasp Safety Margin increases with increasing digit offset

A larger SM_G denotes a greater F_G than necessary to prevent object slip, i.e., less efficient force control as a larger SM_G is not mechanically obligatory. We found that SM_G was positively related to the digit offset at lift onset for both right and left CM and during hold for the left CM (lift onset: t₁₃₃₀ = 4.56, p < 0.0001 and t₁₃₄₀ = 11.32, p < 0.0001 for right and left CM, respectively; hold: t₁₃₃₀ = 0.74, p = 0.46 and t₁₃₄₀ = 16.76, p < 0.0001 for right and left CM, respectively; Supplementary Figure S2), i.e., an opposite relation relative to the above-described relation between F_G and digit offset (see Fig. 3B, left column). We note that this relation more accurately captures the CNS’ response to digit offset because, unlike F_G, it isolates the non-mechanically obligatory portion of F_G, i.e., the grasp force that participants chose to prevent object slip.

The increase in SM_G with increasing digit offset (with the exception of R CM during hold) refutes our hypothesis that F_G and F_M are coordinated to maintain a constant SM_G. However, when examining the variables used to compute SM_G, an important finding was that F_G and F_G^min were differentially affected by increasing digit offset. Specifically, F_G decreased at a faster rate than F_G^min as indicated by the significant decrease in the ratio between F_G^min and F_G with increasing digit offset for lift onset (both CM) and hold (left CM) (coefficients: −0.0005, t₁₃₃₀ = − 4.56, p < 0.0001 and − 0.0013, t₁₃₄₀ = − 11.41, p < 0.0001 for right and left CM at lift onset, respectively; coefficients: −0.00008, t₁₃₃₀ = − 0.75, p = 0.45, and − 0.0017, t₁₃₄₀ = − 16.97, p < 0.0001 for right and left CM during hold, respectively).

The differential effect of digit offset on the mechanically-obligatory F_G^min versus F_G chosen by the subjects raises the question of what caused such inefficient grasp force control, which is denoted by increasingly excessive F_G with increasing digit offset. To address this question, we shifted the focus of our analysis from F_G and F_M magnitudes to their normal and tangential components. We reasoned that this analysis might provide further insights because the modulation of F_G and F_M components can affect the direction of F_M and F_G vectors, and therefore the risk of object slip and/or large tilts.

Increased relative grasp safety margin is associated with a decrease in tangential manipulation force

We first analyzed whether the modulation of F_M components was associated with the increase of SM_G in response to larger digit offsets. Specifically, if increasing digit offset caused the tangential component of F_M to increase, SM_G would be expected to increase to lower the risk of object slip. Alternatively, if the tangential component of F_M decreased with digit offset, a parallel increase in SM_G would suggest that the CNS erroneously perceived such change as a potential threat to object slip prevention. To distinguish between these two scenarios, we analyzed the coordination of the normal (F_M,z) and tangential (F_M,y) components of F_M across digit offset conditions.

SM_G is associated with object slip prevention, whereas the coordination of F_M,y and F_M,z, i.e., T_COM, dictates object pose control and therefore object tilt minimization. We note that, for a given digit offset, the magnitudes of the differences of F_M components between two digits (∆F_M,y and ∆F_M,z) are indeterminate (Eq. [4] in Materials and Methods). This is shown graphically in Fig. 4A, where object tilt prevention can be attained through covariation of the two F_M components along two different manifolds of the same T_COM for a small and large digit offset. Participants could have used two alternative but equally valid strategies to modulate F_M components in response to changes in digit offset. In the first strategy, one of the two F_M components (e.g., ∆F_M,z) could have remained invariant and the other one could have been modulated to digit offset (strategy 1, Fig. 4B). Alternatively, the two F_M components could have covaried (strategy 2, Fig. 4C). We note that strategies 1 and 2 correspond to the two interpretations of the positive modulation of SM_G to digit offset we presented above, i.e., an increase and decrease in the tangential component of F_M, respectively.

We found that participants used strategy 2, i.e., F_M components covaried as a function of digit offset during object hold (Fig. 4D, bottom panel) where the object pose was actively controlled. The two components covaried negatively during object hold (t_2686.89 = − 44.50, p < 0.0001). Figure 4D (bottom panel) also shows that the tangential component of F_M decreased with increasing digit offset (t₂₆₈₆ = − 62.73, p < 0.0001). In contrast, the two F_M components covaried positively at lift onset (t_2690.06 = 13.95, p < 0.0001; Fig. 4D, top panel) when manipulation force components are coordinated in an anticipatory fashion, i.e., prior to feedback about performance outcome. At object lift onset, the tangential component of F_M significantly increased with increasing digit offset (t₂₆₈₈ = 4.75, p < 0.0001; Fig. 4D, top panel). Importantly, despite the temporal evolution of strategy 2 from object lift onset to hold leading to a covariation between the two F_M components, we could reject the adoption of strategy 1 at either task epoch.

The decrease in ∆F_M,y with increasing digit offset during object hold (Fig. 4D, bottom panel) implies that maximum F_M,y of the two digits also decreased as the total F_M,y was more equally distributed among the two digit contacts. We note that this decrease in max F_M,y fails to explain why SM_G increased with increasing digit offset from lift onset throughout object hold (Supplementary Fig. S2). These two findings might appear paradoxical, as one would expect that an increase of SM_G to prevent object slip is in response to an increase, not a decrease, in the tangential component of manipulation force. To reconcile these two results, we further assessed whether SM_G might have been modulated as a function of the ratio between the tangential and normal grasp components (F_G,y/F_G,z). We note that this ratio corresponds to the direction of the F_G vector which is modulated by grasp configuration and that lies in the null space of the grasp map G (see Analysis of grasp and manipulation forces, Methods).

Greater relative grasp safety margin effectively increases the manipulation force solution space

To understand why SM_G increased despite a decrease in ∆F_M,y, the next analysis focused on the relation between the direction of the F_G vector and SM_G. The mechanical effect of increasing digit offset is that the F_G vector moves closer to the edge of the friction cone (Fig. 5A). As such, this phenomenon increases the risk of object slip under the force closure assumption (see Analysis of grasp and manipulation forces in Materials and Methods). Figure 5B shows the representative subject responding to this greater risk of object slip by decreasing the magnitude of F_G both at object lift onset and during hold. This accounts for the decrease in F_G already described in Fig. 3B. However, we also found that SM_G increased with increasing digit offset (Supplementary Fig. S2). This suggests that subjects used larger SM_G to counter the increase in object slip risk caused by increasing digit offset. At the same time, subjects used F_G that were increasingly greater than necessary to prevent object slip. Therefore, the question arises as to whether the increase in SM_G with digit offset might have served an additional functional purpose.

We found that, by increasing SM_G, subjects effectively and significantly expanded the space of F_M solutions to minimize object tilt. To graphically show this phenomenon, in Fig. 6A we first plot the three-dimensional vectors of F_G, F_G^min and F_M (red, blue and black arrows, respectively) at object hold from the same representative subject shown in Fig. 2. The inner white cone denotes the friction cone at the thumb-object contact. The blue cone contains all non-slippery F_M vectors based on F_G^min. Note that F_G^min was estimated based on minimizing F_G and two constraints: the measured F_M and the friction coefficient (see Materials and Methods). Therefore, F_M always points at the boundary of the blue cone at one of the two digit contacts. The red cone contains all non-slippery F_M based on the measured F_G. Data from two CM conditions and digit offsets are shown in the same format but projected onto the y-z plane in Fig. 6B. As digit offset increases from 0 to L, the space of F_M solutions enabled by F_G (V_M, area enclosed by black and red boundaries) and space of F_M solutions enabled by excessive F_G (V_M^XS$,$ area enclosed by blue and red boundaries) decreases for both CM conditions. However, V_M^XS decreases at a lower rate than V_M. To quantify this phenomenon, we computed the ratio of the two solution volumes (R_vol, V_M^XS over V_M). Figure 6C shows the logit-transformed R_vol plotted CM conditions, trials and subjects. We performed the analyses without the U digit offset conditions to enable comparison across digit offset intervals that were experimentally controlled (see Materials and Methods).

We found that R_vol increased with increasing digit offset as early as object lift onset and throughout object hold (t₂₁₄₉ = 9.50, p < 0.0001 and t₂₁₄₇ = 13.57, p < 0.0001, respectively). However, R_vol increased more rapidly with increasing digit offset at object lift onset than during hold (t₃₂₃₃ = − 2.70, p = 0.0069). Importantly, the expansion of the volume of the F_M solution space positively impacted the control of dexterous manipulation, as demonstrated by the statistically-significant negative relation between peak object tilt and digit offset at object lift onset (left CM only) and during hold (both CM) (t₁₀₇₁ = − 20.80, p < 0.0001 for left CM, and t₁₀₇₃ = − 1.63, p = 0.1 for right CM; t₂₁₅₂ = − 3.75, p = 0.0002 for both CM).

In sum, we had found that participants used smaller F_G in response to increasing digit offset. Further analysis, however, showed that increasing digit offset caused F_G to decrease at a faster rate than F_G^min, leading to larger SM_G. This phenomenon, which could have been interpreted as inefficient digit force control using traditional analysis, effectively led to the expansion of the F_M solution space and a reduction in dexterous manipulation error (Fig. 6C and Supplementary Figure S3).

By applying robotic grasp analysis to the coordination of digit forces for human dexterous manipulation, we discovered a novel phenomenon: the exertion of seemingly excessive grasp force – up to 3 times the magnitude necessary to prevent object slip – was instrumental to enhance the robustness of manipulation. This resulted in a significant improvement in dexterous manipulation performance. Furthermore, the space of manipulation solutions enabled by increasing grasp force was significantly larger at the onset of object manipulation than during hold. This phenomenon likely underscores anticipatory mechanisms of object pose control before feedback is acquired during object lift to proactively minimize the effects of sensorimotor signal variability, as well as planning and/or execution errors. Together, these findings suggest that the increase in grasp force, although inefficient from the perspective of energy expenditure, was in fact instrumental in improving dexterous manipulation performance. Our results provide, for the first time, a different perspective on the role ascribed to grasp force by previous research on grasping and manipulation by identifying a novel functional role of grasp force beyond object slip prevention: expanding the solution space of manipulation force.

The present work departs from previous research in several ways. Most previous work focused on normal and tangential force modulation for object slip prevention but did not address how these forces may contribute to dexterous object pose control. To address this limitation, we used a task that required subjects to coordinate digit forces to simultaneously satisfy object slip prevention and pose control requirements (Fig. 1). The present task, as the task we used in our earlier work, requires coordination of fingertip forces and position of contacts to control object pose.¹³ However, the key methodological difference between our previous and current work is that here we systematically changed the grip configuration (Fig. 1B) rather than allowing subjects to choose it on a trial-to-trial basis, i.e., the Unconstrained experimental condition. This allowed the exploration of a much larger range of digit force-position relations relative to the grip configuration that subjects may spontaneously choose to use. Had we tested only the unconstrained digit offset condition (U; Fig. 1B), we could not have thoroughly examined digit force coordination patterns, as indicated by the significantly smaller (40%) modulation of digit offset relative to all other conditions (Fig. 3B).

We note that grip configurations with larger vertical fingertip spacing significantly changes the biomechanical effects of digit normal forces on object pose control (Fig. 5A). Therefore, whereas previous work with zero digit-offset grasp configuration minimized the contribution of normal force on pose control,¹⁵ our work expanded its role by allowing it to simultaneously contribute to slip prevention and object pose control. Given this dual role of digit normal force, a non-traditional analysis was required to decompose grasp and manipulation forces (see Eq. [3] in Materials and Methods and Fig. 3A).

Effect of grasp force on manipulation force

One of the main findings from previous research on grasping and manipulation is that humans use grip forces that are only slightly higher (+ 10 to 40%) than the minimum required to prevent object slip, i.e., the so-called “safety margin” (SM).^5,22,27−31 This previous work has also shown that subjects tend to use a constant SM regardless of changes in object properties, e.g., contact friction and object mass (but see exceptions, e.g., lifting very small objects¹⁰ or manipulation objects with unpredictable object mass²⁵). Together, these findings suggest that humans can control digit forces efficiently and consistently to prevent object slip. The present paper significantly extends the traditional approach for computing SM – originally designed for object slip prevention tasks – to SM_G for applications to grasp tasks that include a dexterity component.

In contrast to previous work and our hypothesis, we found that subjects increased SM_G for larger digit offsets, even though object properties (mass, mass distribution and contact friction) were invariant. We note that SM_G did not increase with grip configuration only in one case (during object hold, right CM; Supplementary Fig. S2). The increase in SM_G suggests that subjects might have perceived an increasing risk of object slip with larger digit offsets. If so, the lack of modulation of SM_G for the right CM during object hold implies that factors other than local contact conditions – which were identical for left and right CM – may play a role in digit force coordination for object slip prevention, e.g., anatomical differences in the force generating capabilities of the muscles of the digit acting as ‘pivot’ and the digit primarily involved with exerting force on the object. This interpretation is consistent with our previous findings that subjects choose different digit offsets at object lift onset when manipulating an object with a right versus left CM.¹³

The increase in SM_G with digit offset was caused by subjects exerting disproportionally larger excessive grasp force (Fig. 3B). We note that a traditional analysis of digit forces would have led to the conclusion that humans fail to control digit force efficiently when manipulating objects using different grip configurations. However, robotic grasp analysis ruled out the possibility that the increase in SM_G was not obligatorily driven by an increase in the tangential component of F_M (∆F_M,y, Fig. 4D), and further revealed that the increase in the seemingly excessive F_G acted to enhance the robustness of manipulation, i.e., the larger F_M solution space that is supported by F_G (Fig. 6B). Importantly, the expansion of the volume of manipulation forces solutions led to a reduction in peak object tilt, which was a key requirement in our dexterous manipulation task (Fig. 6C). We note that an increase in manipulation solution space does not necessarily result in an improvement in manipulation performance. However, an increase in the manipulation solution space would accommodate more intrinsic and/or environmental noise, e.g., sensory prediction errors, sensorimotor delays and external disturbances (Fig. 7A). The only exception to this phenomenon was for the small digit offset in the right CM at object lift onset: for this condition, peak object tilt was generally very small and could therefore not significantly improve any further (Fig. 7B). Therefore, we interpret the reduction in manipulation performance error as the main factor responsible for driving the non-obligatory increase in F_G with increasing digit offset.

Anticipatory control of manipulation force

We also found that the effect of grasp force on the expansion of the manipulation force solution space was significantly greater at object lift onset than hold (Fig. 6C). This finding provides significant insights into the mechanisms underlying the functional relation between grasp and manipulation force. Object lift onset marks an important transition across task epochs that begins with object contact and culminates with the development of digit force distributions to start manipulation. Importantly, the digit force distribution used on a given trial is controlled in an anticipatory fashion and is therefore driven by sensorimotor memory of digit forces used in previous manipulations.^9,11,22 This reliance on memory up to object lift onset is critically important because feedback about object mass distribution becomes available only shortly after the object starts to be accelerated.^13,15,16

If the digit force distribution attained at object lift onset is appropriate for the planned manipulation, expected and actual sensory feedback – tactile, visual and proprioceptive – acquired throughout object lift will coincide.^7,24 Conversely, if a mismatch occurs, sensory feedback will trigger digit force adjustments to correct for the motor error.^1,7 This theoretical framework allows us to interpret the greater effect of grasp force on expanding the manipulation force solution space at object lift onset as indicative of greater uncertainty about the effects of digit force distribution on object tilt minimization at this task epoch than at the end of manipulation, i.e., after feedback has been acquired and used to hold the object straight. Feedback-based control mechanisms likely complement anticipatory control of manipulation force and their potential contribution is discussed below.

Neural mechanisms underlying the coordination of grasp and manipulation forces

The mathematical decoupling of grasp and manipulation forces raises important questions about the underlying sensorimotor mechanisms. Do grasp and manipulation forces generate distinct tactile inputs from skin mechanoreceptors during manipulation? If so, how and at what level of the CNS are these distinct channels of information integrated for the coordination of grasp and manipulation forces? Conversely, if tactile inputs arising from manipulation cannot discriminate grasp from manipulation forces, how does the CNS decode this sensory information to coordinate these two forces?

Our behavioral data cannot be used to answer these questions, which will require combining our methodological approach (force analysis and a dexterous manipulation task) with electrophysiological recordings. Nevertheless, we speculate that tactile inputs might be capable of separately encoding grasp and manipulation forces. The role of tactile receptors has been extensively studied but mostly in the context of grasp control, i.e., slip prevention (see recent review by Sobinov and Bensmaia³²). It is therefore unknown how tactile inputs arising from object manipulation might be used to coordinate grasp and manipulation forces. However, it is well known that the response of different classes of tactile mechanoreceptors is tuned to the features of skin mechanical deformation, e.g., pressure, vibration, and force direction.^1,33−35 It is therefore conceivable that the distinct spatiotemporal patterns of tactile inputs associated with different grip configurations,³⁶ hence digit force vector directions, might have been instrumental in signaling the digit force distribution necessary to simultaneously prevent object slip and minimize object tilt. If so, trial-by-trial learning of our manipulation task might have led to associating the tactile inputs associated with greater digit offsets with the greater need to reduce manipulation error than for the zero-digit offset grip configuration. This is because in the zero-digit offset condition, normal digit forces contribute to object slip prevention but not manipulation. However, as digit offset increases, so does the contribution of normal digit forces to manipulation.

The speculation that grasp and manipulation forces might be encoded by separate patterns of tactile inputs leads to the question of how this information might be integrated for modulating digit forces. This integration would happen within cortical association areas, e.g. parietal cortex, responsible for integration of somatosensory inputs and subsequent modulation of descending commands from pre-motor and motor areas.^37–39 Although this proposition remains to be investigated, recent work using theta burst transcranial magnetic stimulation has shown a context-dependent involvement of primary sensory and cortical cortices (S1 and M1, respectively) in dexterous manipulation. When grasping occurs at constrained contacts – which enables using a sensorimotor memory-based control of digit forces^40,41 – M1 but not S1 is involved in storing and retrieving learned digit forces and positions. However, when digit position is allowed to vary across trials (unconstrained grasping), integrity of M1 and S1 is instrumental in ensuring digit force modulation to position.⁴² It is therefore conceivable that distinct patterns of tactile inputs reaching S1 might drive the modulation of digit forces that could enable coordination of grasp and manipulation forces.

In conclusion, our findings significantly extend previous work by uncovering, for the first time, the coordination between grasp and manipulation forces. The present work also lays the foundation for new research avenues, including the application of our approach to a broader variety of hand-object interactions and task scenarios, as well as biomedical applications, e.g., biologically-inspired grasp controllers for human-machine interfaces, neuroprosthetics and assistive devices. Several critical questions that should be addressed by future research include determining the extent to which tactile input patterns might encode grasp and manipulation forces separately and how these patterns are integrated in sensorimotor cortices.

Experimental Design

We applied digit force analysis tools designed for robotic manipulation on the forces exerted by the participants performing a dexterous task that requires simultaneous object slip prevention and tilt minimization at different grip configurations. We asked participants to grasp an instrumented inverted-T shape object (Fig. 1A) with thumb and index fingertip, lift it while preventing it from tilting, and hold it. We systematically changed the object’s mass distribution and the vertical distance between the digits (offset; Fig. 1B). The former factor was introduced to add a dexterity component to our manipulation task. The rationale for increasing digit offset was that it changes the moment arm of normal digit forces, and therefore their contribution to manipulation.

Participants

Sixteen healthy right-handed (self-reported) adults aged 19–38 (23.88 ± 5.50) years (8 males) with normal to corrected vision and no history of neurological disorders were recruited for participation to the study. All individuals were naïve to the purpose of this study and provided written informed consent in accordance with the Declaration of Helsinki. The experiment protocols were approved by the Office of Research Integrity and Assurance at Arizona State University.

Apparatus

We asked subjects to grasp and manipulate a custom-made grip device using the thumb and index fingertip. The inverted T-shape device consisted of a sensorized vertical handle connected to a horizontal structure (Fig. 1A). The handle has two parallel graspable surfaces (length = 80 mm; width = 24 mm; distance = 59.2 mm) covered with sandpaper (100-grit). Two light-emitting diode (LED) arrays matching the length of the graspable surfaces were placed on the frontal plane of the object to cue subjects about the location of digit placement for each experimental condition (Fig. 1B). The LED arrays were controlled via a microcontroller board (Arduino Uno Rev 3, Arduino, Boston, Massachusetts, USA). Each graspable surface was instrumented with one six-component force/torque (F/T) transducer (Nano 25, ATI Industrial Automation, Garner, NC, USA; nominal force resolution: 0.0625 N; nominal torque resolution: 0.076 N-cm). The transducers measure forces and moments of forces exerted by each digit on the graspable surfaces (sampling rate: 1 kHz).

The center of mass (CM) of the object was configured to be symmetrical (Center CM, C) or asymmetrical (Left or Right CM, L or R, respectively). For each of these conditions, we placed a mass (0.3 Kg) in one of the three compartments (center, left or right) of the horizonal base of the grip device (Fig. 1B, CM). The torques caused by the added mass about the x-axis of the handle’s center of geometry in the handle coordinates were − 0.17, 0, and + 0.17 N–m when the mass was located in the left, center or right compartment, respectively. The cables of the force transducers and the LED arrays ran away from the handle with a slight offset to the left of the handle center and created a small torque 0.01 N–m. Therefore, the actual task torques were − 0.18, − 0.01, and 0.16 N–m for CM L, C, and R, respectively. Note that the terms “left” and “right” denote the thumb and index finger side of the vertical handle, respectively. The total mass of the object was 0.7 Kg.

We used five thumb-index fingertip vertical distances (Fig. 1B, offsets): 0 mm (0), 8.84 mm (Small, S), 29.7 mm (Medium, M), and 55.4 mm (Large, L). An additional experimental condition (Unconstrained, U) consisted in allowing subjects to grasp the object anywhere along the vertical graspable surfaces. The rationale for testing combinations of CM and offsets is described in the Effect of digit offset on grasp force procedures section.

The position and orientation of the object and hand were tracked by a ten-cameras active infrared marker motion capture system (Impulse, PhaseSpace Inc., San Leandro, CA, USA). We used four infrared LED markers on the object (Fig. 1A, black dashed circles, sampling rate: 480 Hz; spatial accuracy: ~1 mm; spatial resolution: 0.1 mm). On each trial, the LED arrays and an audio signal cued the participants about the upcoming task event and grasp condition (see Experimental procedures). A customized LabVIEW program (National Instruments, Austin, TX, USA) was used to drive the microcontroller board and stream kinetic and kinematic data to the hard drive of the host computer.

Experimental procedures

Each participant reached and manipulated the object for a total of 12 experimental conditions. These conditions consisted of 12 combinations of three CM locations and five offsets: C0, CU, L0, LS, LM, LL, LU, R0, RS, RM, RL, and RU. These combinations of external torques and offsets were selected by taking into consideration the extent to which participant could perform the task comfortably. Note that we presented participants with opposite offsets when grasping and manipulating the object with the L or R CM. Specifically, for three L CM conditions (LS, LM and LL), the LED cued participants to place the thumb higher than the index fingertip, whereas for the R CM conditions (RS, RM and RL) subjects were required to place the thumb lower than the index fingertip (see example at the bottom left of Fig. 1B). The LED arrays indicated the designated contact locations of the thumb and index fingertip by lighting up according to the offset condition (see examples in Fig. 1B). The distance spanned by each set of three LEDs (18.84 mm) was sufficiently large to enable positioning of the tip of the thumb and index finger. Participants were instructed to position their fingertips as close as possible to the active LED arrays. In the constrained offset blocks (0, S, M, and L), three LEDs lighted up on each side of the graspable surface. For the unconstrained grasping block (U), all LEDs in both arrays lighted up to inform participants they could choose digit placement.¹³

Before the experiment, participants washed their hands with soap and warm water to normalize skin condition. The coefficient of friction between the fingertips and the graspable surfaces was estimated from slip force measurement as the ratio between the minimal finger force normal to the graspable surface required to prevent slip to the tangential finger force measured at the object slip onset.²² This method requires fingertip tangential (vertical) forces to be equal and aligned with the gravitational force. Therefore, we performed the slip force measurements using the object in the symmetrical (center) CM configuration. Participants sat in front of the table and held the object with the mass added to the center slot (0 N–m torque) 5 cm above the table while following the instructions to “slowly move the index finger and thumb apart and let the object drop freely when it slips.” Participants performed three object release trials at the beginning and end of the experimental session.

To familiarize with the object, cues and task, subjects performed three CU trials, followed by 20 consecutive trials for each experimental condition (20⊆12 = 240 trials). The pseudo-randomization of the 12 experimental condition blocks was designed to present a different CM across consecutive blocks. The object was placed at a distance of 30 cm in front of the participant on a leveled tabletop 30 cm below the participant’s shoulder joint. This configuration ensured subjects could perform a comfortable grasp. The first auditory cue (Ready) prompted the subject to prepare for the reach. After 1.5 seconds, the second cue from the LED arrays informed subjects where to place their digits and cued them to start the reach, grasp and lift at a self-paced, natural speed. Participants were instructed to (a) grasp the object by placing, as accurately as possible, the tip of their thumb and index finger of their right hand such that each coincided with the active LEDs while extending the other digits, (b) lift the object while preventing it from tilting, as if the object were a cup filled with liquid, and (c) hold it straight for 2 seconds. We note that fulfilling the object tilt minimization criterion required participants to plan and generate a compensatory torque (T_COM) at object lift onset to compensate for the object’s asymmetrical mass distribution (R and L CM conditions¹³). When the object reached a height of 20 cm above the table, a third auditory cue (Hold) was triggered, after which subjects were required to hold the object stationary and in a vertical orientation for ~ 2 seconds. The next auditory cue (Relax) informed subjects to replace the object on the table and move the hand back to the initial position. Subjects were given ~ 10-s breaks between trials and ~ 30-s breaks between blocks.

As the CM of the object cannot be visually inferred (the added mass is hidden from view), subjects do not exert adequate T_COM at lift onset on the first left or right CM trial, thus causing the object to tilt during the lift. However, we previously found that subjects can learn to generate an anticipatory T_COM to minimize object tilt within the first three trials.¹³ This phenomenon was further confirmed in our data. Therefore, to quantify differences in anticipatory T_COM at lift onset across experimental conditions, we excluded the first three trials in each condition block from analysis.

Data processing

Kinematic and kinetic data were resampled offline to 250 Hz, temporally aligned and processed for analysis by custom software written in MATLAB (MathWorks Inc., Natick, MA, USA). All data were first low-pass filtered at 30 Hz with second order, zero-lag Butterworth filters. A small number of trials (0.3%) was excluded from analyses due to temporal misalignment between kinematic and kinetic data.

Data analysis

Object kinematics and task epochs. Object kinematics was computed from the data obtained through the infrared LED markers (Fig. 1A). Object translation was estimated as the displacement of the origin of the object’s fixed frame, whereas the object orientation was extracted as the Z-Y-X Euler angle from the rotation matrix using the MATLAB function rotm2eul. Object lift onset was identified using the maximum object height as the starting point of our algorithm and defined as the time at which the object’s height was higher than 1 mm above the table and the object’s vertical velocity was greater than 5 mm/s. Object hold phase was defined as a one-second window 0.5 second after the Hold cue. As subjects were asked to minimize object tilt, we quantified the quality of dexterous manipulation by measuring peak object tilt at object lift onset and during object hold. For the former epoch, peak object tilt was quantified as the largest angle of the vertical axis of the grip device relative to the vertical occurring within 250 ms from object lift onset. During object hold, peak object tilt was computed by averaging the object’s angle relative to the vertical throughout the one-second object hold window.

To determine the trial after which task performance became stable, the resultant torque (external torque minus compensatory torque, T_RES; see Analysis of manipulation force components’ contribution to T_COM) was tested by linear mixed-effects models (LMMs) with three-way fixed effects of CM x Digit Offset × Trial. The data and Trial factor were incrementally increased starting from trial 10 to 20, followed by 9 to 20, and so forth, until a significant Trial effect was found. We found task performance became stable from trial 4 to 20 for all subjects and conditions.

Coefficient of friction estimation. The translational and torsional coefficients of friction at each contact were estimated for constructing the friction cone at each contact to approximate the minimum grasp (normal) force required to prevent object slip.⁴³ Slip onset was identified as the time of the peak change of the resultant vertical force rate closest to the first drop in the rate of resultant vertical force. This approach was preferred to defining slip onset as the time at which the first peak change in the resultant vertical force occurred as subjects might reflexively re-grip the object after slip onset. The accuracy of this algorithm was verified by also examining object position data. The translational coefficients of friction at the thumb and index fingertip were quantified as the largest value of the ratio between the vertical tangential and normal forces at the time of slip onset estimated at the thumb and index fingertip. We found no significant difference between the translational coefficient of friction measured at the beginning versus end of the experimental session (t₁₅ = 0.078, p = 0.9389). Therefore, we averaged these values for each subject for the analysis of grasp and manipulation forces (below). The coefficient of translational friction averaged across subjects was 1.56 ± 0.06. The torsional coefficient at contacts was assumed to be 6.59 times the translational coefficient.⁴⁴

Analysis of grasp and manipulation forces. A stable grasp that can resist any force vectors applied on the grasped object (i.e., a grasp is force-closure⁴⁵) requires internal forces that are exerted inside the friction cone. Unlike T_COM, the internal forces have no direct effect on the object kinematics and are only limited by the friction cone, the object rigidity, and the amount of force each digit can exert. Therefore, manipulation and internal forces play very different roles in dexterous manipulation: the former is devoted to controlling object position and orientation (collectively defined as ‘object pose’), whereas the latter is devoted to preventing object slip. The procedure to decompose the contact forces at individual digit contacts into manipulation and internal forces are described below.

All forces measured by the two transducers were first spatially rotated and aligned from individual transducer coordinates to the object coordinate frame fixed at the center of the handle (inset, Fig. 1A) and defined following the recommendations for reporting kinematic data.⁴⁶ Each digit force was assumed to be applied at a point denoted as the center of pressure (CoP, ${\left[\begin{array}{ccc}{\text{CoP}}_{\text{x}}& {\text{CoP}}_{\text{y}}& {\text{CoP}}_{\text{z}}\end{array}\right]}^{\text{T}}$) of the force vector application on each graspable surface and computed from the measured forces ($\stackrel{\text{⃑}}{\text{F}}\text{ }\text{=}{ \left[\begin{array}{ccc}{\text{f}}_{\text{x}}& {\text{f}}_{\text{y}}& {\text{f}}_{\text{z}}\end{array}\right]}^{\text{T}}$) and moment of force ($\stackrel{\text{⃑}}{\text{M}}\text{ }\text{=}{ \left[\begin{array}{ccc}{\text{m}}_{\text{x}}& {\text{m}}_{\text{y}}& {\text{m}}_{\text{z}}\end{array}\right]}^{\text{T}}$) using the cross product equality $\stackrel{\text{⃑}}{\text{r}}\text{ }\text{×}\text{ }\stackrel{\text{⃑}}{\text{F}\text{ }}\text{=}\text{ }\stackrel{\text{⃑}}{\text{M}}$ where $\stackrel{\text{⃑}}{\text{r}}\text{ }\text{=}\text{ }{\left[\begin{array}{ccc}\text{Co}{\text{P}}_{\text{x}}& \text{Co}{\text{P}}_{\text{y}}& \text{Co}{\text{P}}_{\text{z}}\end{array}\right]}^{\text{T}}$, T denotes transpose, and |CoP_z| indicates the distance from the graspable surface to the geometrical center of the sensorized vertical handle, i.e., half of the grasp width normal to the grasp surface. The vertical distance between the tip of the thumb and index finger was defined as $\text{Δ}{\text{CoP}}_{\text{y}}\text{ }\text{=}\text{ }{\text{CoP}}_{\text{y}}^{\text{IN}} \text{–}\text{ }{\text{ CoP}}_{\text{y}}^{\text{TH}}$. We note that ∆CoP_y is equivalent to the digit placement offset that we either systematically changed across experimental conditions or was chosen by the subject (U condition; Fig. 1B).

The soft-finger contact model assumes that the finger can exert forces in three directions and one moment of force normal to the contact surface.⁴⁷ The contact forces and moments of forces exerted by each digit at the contact surface in the digit-contact coordinate frame can be computed from $\stackrel{\text{⃑}}{\text{F}}$ and $\stackrel{\text{⃑}}{\text{M}}$ in the object coordinate frame with $\stackrel{\text{⃑}}{\text{r}}$. The generalized force vector in ℝ⁴: ${\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{i}}\text{ }\text{=}\text{ }{\left[\begin{array}{cccc}{\text{f}}_{\text{x}}& {\text{f}}_{\text{y}}& {\text{f}}_{\text{z}}& {\text{m}}_{\text{z}}\end{array}\right]}^{\text{T}}$ represents the contact force vectors applied on the contact surface at ${\text{CoP}}^{\text{i}}\text{ }\text{=}\text{ }{\left[\begin{array}{cc}\text{Co}{\text{P}}_{\text{x}}^{\text{i}}& \text{Co}{\text{P}}_{\text{y}}^{\text{i}}\text{ }\end{array}\right]}^{\text{T}}$, where i = TH, thumb or IN, index finger, in the digit-contact coordinate frame.

The effect of ${\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{i}}$ on the object can be determined by transforming them to the object coordinate frame using the adjoint transformation matrix and the wrench basis.^21,47,48 The adjoint transformation matrix was

${\text{Ad}}_{{\text{g}}_{\text{i}}^{\text{-1}}}^{\text{T}}\text{=}\left[\begin{array}{cc}{\text{R}}^{\text{i}}& \text{0}\\ {\widehat{\text{p}}}^{\text{i}}{\text{R}}^{\text{i}}& {\text{R}}^{\text{i}}\end{array}\right]$ [1]

where, for each digit, Rⁱ is the rotation matrix from the object frame to the contact frame and $\widehat{\text{p}}$ is the skew-symmetric matrix of $\stackrel{\text{⃑}}{\text{r}}$. The wrench basis was

${\text{B}}^{\text{i}}=\left[\begin{array}{cccc}\text{1}& \text{0}& \text{0}& \text{0}\\ \text{0}& \text{1}& \text{0}& \text{0}\\ \text{0}& \text{0}& \text{1}& \text{0}\\ \text{0}& \text{0}& \text{0}& \text{0}\\ \text{0}& \text{0}& \text{0}& \text{0}\\ \text{0}& \text{0}& \text{0}& \text{1}\end{array}\right]$ [2]

where each column represents elements of ${\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}$ and each row is the element of the generalized force vector acting on the object. Based on the force application locations of the thumb and index fingertip with the soft-finger contact model assumption,²¹ the grasp map (G) can be written as:

$\text{G =}{\text{Ad}}_{{\text{g}}_{\text{i}}^{\text{-1}}}^{\text{T}}{\text{B}}^{\text{i}}\text{= }\left[\begin{array}{cccccccc}\text{1}& \text{0}& \text{0}& \text{0}& \text{-1}& \text{0}& \text{0}& \text{0}\\ \text{0}& \text{1}& \text{0}& \text{0}& \text{0}& \text{1}& \text{0}& \text{0}\\ \text{0}& \text{0}& \text{1}& \text{0}& \text{0}& \text{0}& \text{-1}& \text{0}\\ \text{0}& \text{-}{\text{CoP}}_{\text{z}}^{\text{TH}}& {\text{CoP}}_{\text{y}}^{\text{TH}}& \text{0}& \text{0}& \text{-}{\text{CoP}}_{\text{z}}^{\text{IN}}& \text{-}{\text{CoP}}_{\text{y}}^{\text{IN}}& \text{0}\\ {\text{CoP}}_{\text{z}}^{\text{TH}}& \text{0}& \text{-}{\text{CoP}}_{\text{x}}^{\text{TH}}& \text{0}& \text{-}{\text{CoP}}_{\text{z}}^{\text{IN}}& \text{0}& {\text{CoP}}_{\text{x}}^{\text{IN}}& \text{0}\\ \text{-}{\text{CoP}}_{\text{y}}^{\text{TH}}& {\text{CoP}}_{\text{x}}^{\text{TH}}& \text{0}& \text{1}& {\text{CoP}}_{\text{y}}^{\text{IN}}& {\text{CoP}}_{\text{x}}^{\text{IN}}& \text{0}& \text{-1}\end{array}\right]$ [3]

where ${\text{CoP}}_{\text{i}}^{\text{TH}}$ and ${\text{CoP}}_{\text{i}}^{\text{IN}}$ denote the points of thumb and index force application, respectively.

The grasp map was used to transform ${\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{i}}$ at individual digit contacts to the object coordinate frame and consequently mapped the digit forces to the total resultant forces (${\stackrel{\text{⃑}}{\text{F}}}_{\text{O}}$) acting on the object, ${\stackrel{\text{⃑}}{\text{F}}}_{\text{O}}\text{ =}\text{ }\text{G}{\left[\begin{array}{cc}{\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{TH}}& {\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{IN}}\end{array}\right]}^{\text{T}}$. The null space of G was denoted as G₀. Projecting ${\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{i}}$ on to G₀ resulted in F_G defined as ${\stackrel{\text{⃑}}{\text{F}}}_{\text{G}}\text{ = }{\text{G}}_{\text{0}}{\text{G}}_{\text{0}}^{\text{T}}{\left[\begin{array}{cc}{\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{TH}}& {\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{IN}}\end{array}\right]}^{\text{T}}$, i.e., the internal force that lies in G₀ and F_M can be computed as ${\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{i}}$ minus the grasp force, ${\stackrel{\text{⃑}}{\text{F}}}_{\text{M}}\text{ = }{\left[\begin{array}{cc}{\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{TH}}& {\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{IN}}\end{array}\right]}^{\text{T}}-{\stackrel{\text{⃑}}{\text{F}}}_{\text{G}}$. As the grasp tasks were successfully performed in all trials, we assumed all grasps satisfied the force closure criterion.⁴⁵ Based on the force closure assumption, in order to maintain proper F_M without slipping, there must exist an F_G to guarantee that contact forces satisfy friction cone constraints. The magnitude of all thumb and index finger force vectors was quantified as $\left|\text{F}\right|\text{ }\text{=}\text{ }\sum _{\text{i}\text{ }\text{=}\text{ }\text{TH, IN}}\left|{\stackrel{\text{⃑}}{\text{F}}}_{\text{C}}^{\text{i}}\right|$. The magnitude of F_G and F_M was computed as $\left|{\text{F}}_{\text{G}}\right|\text{ }\text{=}\text{ }\sum _{\text{ }\text{i=}\text{ }\text{TH, IN}}\left|{\stackrel{\text{⃑}}{\text{F}}}_{\text{G}}^{\text{i}}\right|$ and ${\text{|F}}_{\text{M}}\text{|}\text{ }\text{=}\text{ }\sum _{\text{i}\text{ }\text{=}\text{ }\text{TH, IN}}\left|{\stackrel{\text{⃑}}{\text{F}}}_{\text{M}}^{\text{i}}\right|$, respectively. The magnitude of a generalized force vector for each digit was calculated as $\left|{\overrightarrow{F}}_{general}\right|= \sqrt{{f}_{x}^{2}+{f}_{y}^{2}+{f}_{z}^{2}}$. The data clusters of |F_G|, |F_M| and the minimum F_G required to prevent object slip (see below) are plotted for each experimental condition as an ellipse measured at object lift onset and hold (Fig. 3B). The ellipses are based on clusters of data from trials 4–20 from all subjects in each digit offset condition. The length of the principal axes of each ellipse was computed using principal component analysis. Half-length of the two principal axes denote the standard deviation along the corresponding axes.

Effect of digit offset on grasp force. We note that the larger the digit offset, the closer the direction of F_G vector (± 43º relative to z_O axis for the largest digit offsets, L) to the boundary of object slip (± 57º relative to z_O axis; the edge of friction cone computed from the translational friction coefficient averaged across subjects; Fig. 5) due to the mechanically-obligatory requirement of F_G being in the null space of the grasp map G (see Analysis of grasp and manipulation forces). F_G is responsible for preventing object slip by steering the digit force vectors at individual contacts to remain within the friction cone. Therefore, the closer the F_G vector is to the edge of the friction cone, the greater the risk of object slip. At the same time, an asymmetric object CM requires an active control of object orientation during manipulation, i.e., object lift. Therefore, digit forces had to be accurately coordinated to simultaneously address the challenges of increased risk of object slip caused by increasing digit offset and large object tilt caused by the need to counter the external torque caused by the asymmetric CM.

Estimation of minimally-required grasp forces. We used the F_M−F_G decomposition to determine how the CNS coordinates these two functionally distinct aspects of dexterous manipulation: slip prevention and object pose control. With regard to slip prevention, it is well known that humans can efficiently attain a small safety margin by exerting small normal force above the minimum required to prevent object slip.^9,10,12,26 However, how the CNS deals simultaneously with object pose control and slip prevention remains unknown. To address this gap, we first estimated excessive grasp force (F_G^XS) as the difference between the magnitude of F_G and the minimal grasp force required to prevent object slip (F_G^min). F_G^min was estimated by an optimization process based on the manipulation force, the grasp map, and the coefficients of friction.⁴⁹ The optimization objective was set as minimizing the magnitude of F_G with translational and torsional friction cone constraints. The magnitude of F_G^min was computed as $|{\text{F}}_{\text{G}}^{\text{min}}\text{| = }\sum _{\text{i = TH, IN}}\left|{\stackrel{\text{⃑}}{\text{F}}}_{\text{G}}^{\text{min, i}}\right|$.

We defined relative grasp safety margin (SM_G) as the ratio between F_G^XS and F_G to quantify the proportion of excessive force normalized to exerted grasp force. Note that this definition of SM_G is different from how relative safety margin (SM) has been defined in the grasping literature.^27,50 Specifically, the traditional definition is based on decomposing the digit force vector into normal and tangential components. As such, SM denotes how far the digit force vector is from the boundary of its friction cone at the digit contact. We note that previous grasping literature focused primarily on manipulation of symmetrical objects at zero digit offsets, where there are no external torque or torques induced by digit forces acting at non-zero digit offsets. Therefore, the magnitude of normal forces above the minimum normal force required to prevent object slip can take any value up to physiological limits with no risk of changing object orientation. In contrast, our SM_G definition uses the F_G component of the digit force vector obtained from the F_M−F_G decomposition. Therefore, while SM and SM_G both quantify digit force control efficiency, SM_G captures the excessive force above that required to satisfy both object slip prevention and tilt minimization.

Analysis of manipulation force components’ contribution to T _COM. Next, we computed T_COM required to counteract the external torque (T_EXT) caused by the added mass (L or R CM) as the resultant moment of force about the x-axis. T_COM consists of the two moment of forces generated by the difference of the tangential and normal manipulation forces between the thumb and index fingertip:

$${\text{T}}_{\text{COM}}\text{ = }\text{–}{\text{CoP}}_{\text{z}}\text{∙}\left({\text{F}}_{\text{M,y}}^{\text{IN}}\text{ }\text{– }{\text{F}}_{\text{M,y}}^{\text{TH}}\right)\text{ }\text{+}\text{ 0.5 }\text{Δ}{\text{CoP}}_{\text{y}}\text{∙}\left({\text{F}}_{\text{M,z}}^{\text{IN}}\text{ }\text{– }{\text{F}}_{\text{M,z}}^{\text{TH}}\right)$$

$\text{=}{\text{ }\text{–}\text{CoP}}_{\text{z}}\text{∙∆}{\text{F}}_{\text{M,y}}\text{ }\text{+}\text{ 0.5 }\text{Δ}{\text{CoP}}_{\text{y}}\text{∙∆}{\text{F}}_{\text{M,z}}$ [4]

The resultant torque (T_RES) was computed as the sum of T_COM and T_EXT. As the task requirement was to attain a vertical pose from object lift onset to static hold, T_RES should be zero to attain the rotational equilibrium about the x-axis.

It can be noticed from Eq. [4] that there is a constant (CoP_z, i.e.,half-object width) and three variables (∆CoP_y, ∆F_M,y, and ∆F_M,z). For a prescribed ∆CoP_y (digit offset), the CNS has one degree of freedom to produce T_COM through arbitrary ∆F_M,y and ∆F_M,z sharing patterns, i.e., their magnitudes are indeterminate as long as they are within the T_COM manifold (Fig. 4). Our previous work¹³ on unconstrained grasping where subjects used relatively small digit offsets has shown that successful performance of our manipulation task is attained by distributing these F_M components near the task manifold (Fig. 4A; see also Fig. 8 of Fu et al.¹³). In the present work, changes in digit offset lead to changes in the slope of the task T_COM manifold. This is because the slope of the T_COM line increases as a function of digit offset. However, how the two F_M components are coordinated as a function of digit offset is indeterminate. Specifically, subjects could satisfy our task requirements through one of two alternative strategies. In the first scenario, one of the F_M components, e.g., ∆F_M,z, could be invariant to changes in digit offset whereas the other component (∆F_M,y) would have to compensate to ensure attainment of T_COM (Fig. 4B). Alternatively, ∆F_M,y and ∆F_M,z could covary across digit offsets (Fig. 4C).

Solution space of non-slippery manipulation forces. Changes in the magnitude of F_G effectively modify the volume of the solution space of non-slippery F_M .^10,23 Thus, as |F_G| increases, it enables the exertion of a larger tangential component of F_M without slip at individual digit contacts. To quantify these effects, we computed the volume of (a) the F_M solution space enabled by the grasp forces (V_M) in the 3D force space and (b) the F_M solution space enabled by F_G^XS (V_M^XS). We then computed the ratio, R_vol, between V_M and V_M^XS which represents the relative change of the non-slippery F_M solution space. R_vol was computed on the small, medium and large digit offset only. The rationale for excluding the U condition was that participants adopted very small and variable digit offsets (see Fig. 2B), and therefore were not suitable to be pooled with data obtained from systematic changes in digit offset.

Statistical analysis

All statistical analyses were conducted in RStudio (version 1.3.1093, RStudio Team, 2020) with R (version 4.0.3, R Core Team, 2020). To determine the effects of digit spacing on digit force coordination, linear mixed-effects models (LMMs) were fitted for all dependent variables with the restricted maximum likelihood criterion using the ‘lme4’ package.⁵¹ All models were built with random intercepts for the random-effect factor Subject.

Task performance, defined as T_RES, was tested by models with three-way fixed effects of CM (3 levels: C, R, and L) × Digit Offset (continuous variable) × Trial (trial 4 through 20). All models investigating the effects of digit spacing on digit forces modulation were built with two-way fixed-effects of CM (2 levels: R and L) × |Digit Offset| (continuous variable). The relation between ∆F_M,y and ∆F_M,z was examined by testing ∆F_M,y using a LMM with two-way fixed effects of CM (2 levels: R and L) × |∆F_M,z| (continuous variable).

As R_vol values were bounded from 0 to 1, the R_vol values were transformed by a logit function for statistical purpose. The effects of digit offset on R_vol were explored by fitting LMMs to the transformed R_vol values with three-way fixed-effects of Epoch (2 levels: lift onset and object hold) × CM (2 levels: R and L) × |Digit Offset| (continuous variable). As we did not find a three-way interaction (only two-way interactions were significant), we analyzed R_vol using two-way fixed-effects analysis with CM (2 levels: R and L) × |Digit Offset| (continuous variable) for each epoch. The degrees of freedom for the t-tests and corresponding p-values were approximated with the Satterthwaite’s method⁵² for all LMMs. All significant levels were set at α = 0.01. Bonferroni adjusted post-hoc LMMs were performed when there were interactions.

Acknowledgments

We thank Drs. Qiushi Fu and Christopher Buneo for comments on an earlier version of the manuscript.

Funding: This material is based upon work supported by the National Science Foundation under Grant No. BCS-1827752.

Author contributions: Y.W. and M.S. designed the experiments, interpreted the data and wrote the manuscript. Y.W. performed the experiments and analyzed the data.

Competing interests: The authors declare that they have no competing interests.

Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Raw data are available on Open Science Framework (DOI: 10.17605/OSF.IO/U7RQS) with no restrictions.

Johansson, R. S. & Flanagan, J. R. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nat. Rev. Neurosci. 10, 345–359 (2009).
Baugh, L. A., Kao, M., Johansson, R. S. & Flanagan, J. R. Material evidence: interaction of well-learned priors and sensorimotor memory when lifting objects. J. Neurophysiol. 108, 1262–1269 (2012).
Cole, K. & Johansson, R. Friction at the digit-object interface scales the sensorimotor transformation for grip responses to pulling loads. Exp. Brain Res. 95, (1993).
Flanagan, J. R., King, S., Wolpert, D. M. & Johansson, R. S. Sensorimotor prediction and memory in object manipulation. Can. J. Exp. Psychol. Can. Psychol. expérimentale 55, 87–95 (2001).
Kinoshita, H. & Francis, P. R. A comparison of prehension force control in young and elderly individuals. Eur. J. Appl. Physiol. 74, 450–460 (1996).
Jenmalm, P., Birznieks, I., Goodwin, A. W. & Johansson, R. S. Influence of object shape on responses of human tactile afferents under conditions characteristic of manipulation. Eur. J. Neurosci. 18, 164–176 (2003).
Johansson, R. S. & Westling, G. Signals in tactile afferents from the fingers eliciting adaptive motor responses during precision grip. Exp. Brain Res. 66, 141–154 (1987).
Macefield, V. G., Häger-Ross, C. & Johansson, R. S. Control of grip force during restraint of an object held between finger and thumb: responses of cutaneous afferents from the digits. Exp. Brain Res. 108, 155–71 (1996).
Johansson, R. S. & Westling, G. Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Exp. Brain Res. 56, 550–64 (1984).
Hiramatsu, Y., Kimura, D., Kadota, K., Ito, T. & Kinoshita, H. Control of precision grip force in lifting and holding of low-mass objects. PLoS One 10, (2015).
Johansson, R. S. & Westling, G. Influences of Cutaneous Sensory Input on the Motor Coordination During Precision Manipulation. in Somatosensory Mechanisms (eds. von Euler, C., Franzen, O., Lindblom, U. & Ottoson, D.) 249–260 (Palgrave Macmillan UK, 1984). doi:10.1007/978-1-349-07292-7_17
Schneider, T. R. & Hermsdörfer, J. Intention to be force efficient improves high-level anticipatory coordination of finger positions and forces in young and elderly adults. J. Neurophysiol. 1663–1680 (2021). doi:10.1152/jn.00499.2020
Fu, Q., Zhang, W. & Santello, M. Anticipatory planning and control of grasp positions and forces for dexterous two-digit manipulation. J. Neurosci. 30, 9117–26 (2010).
Zhang, W., Gordon, A. M., Fu, Q. & Santello, M. Manipulation After Object Rotation Reveals Independent Sensorimotor Memory Representations of Digit Positions and Forces. J. Neurophysiol. 103, 2953–2964 (2010).
Salimi, I., Hollender, I., Frazier, W. & Gordon, A. M. Specificity of internal representations underlying grasping. J. Neurophysiol. 84, 2390–2397 (2000).
Lukos, J. R., Ansuini, C. & Santello, M. Anticipatory control of grasping: independence of sensorimotor memories for kinematics and kinetics. J. Neurosci. 28, 12765–74 (2008).
Ciocarlie, M., Dang, H., Lukos, J., Santello, M. & Allen, P. Functional analysis of finger contact locations during grasping. Proc. – 3rd Jt. EuroHaptics Conf. Symp. Haptic Interfaces Virtual Environ. Teleoperator Syst. World Haptics 2009 401–405 (2009). doi:10.1109/WHC.2009.4810864
Fu, Q., Hasan, Z. & Santello, M. Transfer of learned manipulation following changes in degrees of freedom. J. Neurosci. 31, 13576–84 (2011).
Kerr, J. R. & Roth, B. Analysis of Multifingered Hands. Int. J. Rob. Res. 4, 3–17 (1986).
Yoshikawa, T. & Nagai, K. Manipulating and Grasping Forces in Manipulation by Multifingered Robot Hands. IEEE Trans. Robot. Autom. 7, 67–77 (1991).
Murray, R. M., Li, Z. & Sastry, S. S. A Mathematical Introduction to Robotic Manipulation. (CRC Press, 1994).
Westling, G. & Johansson, R. S. Factors influencing the force control during precision grip. Exp. Brain Res. 53, 277–84 (1984).
Bicchi, A., Gabiccini, M. & Santello, M. Modelling natural and artificial hands with synergies. Philos. Trans. R. Soc. Lond. B. Biol. Sci. 366, 3153–61 (2011).
Burstedt, M. K., Edin, B. B. & Johansson, R. S. Coordination of fingertip forces during human manipulation can emerge from independent neural networks controlling each engaged digit. Exp. Brain Res. 117, 67–79 (1997).
Hadjiosif, A. M. & Smith, M. A. Flexible control of safety margins for action based on environmental variability. J. Neurosci. 35, 9106–9121 (2015).
Solnik, S., Zatsiorsky, V. M. & Latash, M. L. Internal forces during static prehension: Effects of age and grasp configuration. J. Mot. Behav. 46, 211–222 (2014).
Burstedt, M. K., Flanagan, J. R. & Johansson, R. S. Control of grasp stability in humans under different frictional conditions during multidigit manipulation. J. Neurophysiol. 82, 2393–2405 (1999).
Cole, K. J., Rotella, D. L. & Harper, J. G. Mechanisms for age-related changes of fingertip forces during precision gripping and lifting in adults. J. Neurosci. 19, 3238–3247 (1999).
Edin, B. B., Westling, G. & Johansson, R. S. Independent control of human finger-tip forces at individual digits during precision lifting. J. Physiol. 450, 547–64 (1992).
Kinoshita, H. Effect of gloves on prehensile forces during lifting and holding tasks. Ergonomics 42, 1372–1385 (1999).
McDonnell, M. N., Ridding, M. C., Flavel, S. C. & Miles, T. S. Effect of human grip strategy on force control in precision tasks. Exp. Brain Res. 161, 368–373 (2005).
Sobinov, A. R. & Bensmaia, S. J. The neural mechanisms of manual dexterity. Nat. Rev. Neurosci. 0123456789, (2021).
Birznieks, I., Jenmalm, P., Goodwin, A. W. & Johansson, R. S. Encoding of direction of fingertip forces by human tactile afferents. J. Neurosci. 21, 8222–8237 (2001).
Delhaye, B. P., Long, K. H. & Bensmaia, S. J. Neural Basis of Touch and Proprioception in Primate Cortex. Compr. Physiol. 8, 1575–1602 (2018).
Johansson, R. S. & Flanagan, J. R. Tactile Sensory Control of Object Manipulation in Humans. Senses A Compr. Ref. 6, 67–86 (2008).
Johansson, R. S. & Birznieks, I. First spikes in ensembles of human tactile afferents code complex spatial fingertip events. Nat. Neurosci. 7, 170–177 (2004).
Davare, M., Rothwell, J. C. & Lemon, R. N. Causal Connectivity between the Human Anterior Intraparietal Area and Premotor Cortex during Grasp. Curr. Biol. 20, 176–181 (2010).
Davare, M., Kraskov, A., Rothwell, J. C. & Lemon, R. N. Interactions between areas of the cortical grasping network. Curr. Opin. Neurobiol. 21, 565–570 (2011).
Jenmalm, P., Schmitz, C., Forssberg, H. & Ehrsson, H. H. Lighter or heavier than predicted: Neural correlates of corrective mechanisms during erroneously programmed lifts. J. Neurosci. 26, 9015–9021 (2006).
Chouinard, P. A., Leonard, G. & Paus, T. Role of the primary motor and dorsal premotor cortices in the anticipation of forces during object lifting. J. Neurosci. 25, 2277–84 (2005).
Schabrun, S. M., Ridding, M. C. & Miles, T. S. Role of the primary motor and sensory cortex in precision grasping: A transcranial magnetic stimulation study. Eur. J. Neurosci. 27, 750–756 (2008).
Parikh, P. J., Fine, J. M. & Santello, M. Dexterous Object Manipulation Requires Context-Dependent Sensorimotor Cortical Interactions in Humans. Cereb. Cortex 30, 3087–3101 (2020).
Erdmann, M. On a Representation of Friction in Configuration Space. Int. J. Rob. Res. 13, 240–271 (1994).
Kinoshita, H., Bäckström, L., Flanagan, J. R. & Johansson, R. S. Tangential torque effects on the control of grip forces when holding objects with a precision grip. J. Neurophysiol. 78, 1619–1630 (1997).
Bicchi, A. On the Closure Properties of Robotic Grasping. Int. J. Rob. Res. 14, 319–334 (1995).
Wu, G. & Cavanagh, P. R. ISB Recommendations in the Reporting for Standardization of Kinematic Data. J. Biomech. 28, 1257–1261 (1995).
Mason, M. T. & Salisbury, J. K. Robot Hands and the Mechanics of Manipulation. (The MIT Press, 1985).
Harris, W. F. J., Fulton, W. & Harris, J. Representation Theory: A First Course. (Springer New York, 1991).
Li, Y., Cong, M., Liu, D., Du, Y. & Xu, X. Stable grasp planning based on minimum force for dexterous hands. Intell. Serv. Robot. 13, 251–262 (2020).
Su, Z. et al. Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor. in 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) 297–303 (IEEE, 2015). doi:10.1109/HUMANOIDS.2015.7363558
Bates, D., Mächler, M., Bolker, B. M. & Walker, S. C. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67, (2015).
Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest Package: Tests in Linear Mixed Effects Models. J. Stat. Softw. 82, (2017).

There is NO Competing Interest.

supplementarymaterialsfinal.docx
Supplementary Materials of the Article

PDF

Version 1

posted 29 Apr, 2022

You are reading this latest preprint version

Performance of dexterous object manipulation is enhanced by grasp force modulation

Status:

Version 1

Abstract

Figures

Introduction

Results

Subjects quickly learned dexterous manipulation regardless of grip configuration

Grasp and manipulation forces are concurrently modulated as a function of grasp configuration

Relative Grasp Safety Margin increases with increasing digit offset

Increased relative grasp safety margin is associated with a decrease in tangential manipulation force

Greater relative grasp safety margin effectively increases the manipulation force solution space

Discussion

Effect of grasp force on manipulation force

Anticipatory control of manipulation force

Neural mechanisms underlying the coordination of grasp and manipulation forces

Materials And Methods

Experimental Design

Participants

Apparatus

Experimental procedures

Data processing

Data analysis

Statistical analysis

Declarations

References

Competing Interests

Supplementary Files

Status:

Version 1