New model-based manipulation technique for reshaping deformable linear objects

In this article, we consider the problem of reshaping a deformable linear object (DLO) like wires, cables, ropes, and surgical sutures. The solution to this problem would be useful for many fields, especially industrial manufacturing, where the DLO manipulation is still frequently carried out by human workers. In this work, a new model-based manipulation technique for reshaping a DLO is addressed employing a sequence of grasping and releasing primitives performed by a single-armed robot equipped with a gripper. A decision process selects the optimal grasping point exploiting an error minimization approach and chooses the related releasing point. This decision process performs a spline interpolation between the error values obtained from candidate grasping points and chooses the optimal point that owns a minimum error. The multivariate dynamic spline model of the DLO is exploited for selecting the optimal grasping point and predicting the DLO behavior during the manipulation process. Because of its advantages over other integration methods, the symplectic integrator is utilized for iteratively solving the DLO dynamic model. Simulation results of reshaping a DLO lying on a table are presented to evaluate the proposed technique. These results illustrate the intermediate deformation steps which lead the DLO from its starting state to the desired one. They demonstrate that our proposed technique can efficiently manipulate the DLO into various shapes in few steps.


Introduction
Humans can handle and manipulate deformable objects, such as clothes, sheets, wires, and ropes, smoothly and without any issue. On the other hand, manipulating deformable objects using robots is still challenging despite the several attempts of the robotics community [30]. The main reasons for these limitations are the complexity of deformable object dynamics, limited capability in measuring the deformable object state, and unpredictable initial configuration. Due to these limitations, most of the procedures in the assembly process that contains deformable objects are still performed manually. Deformable object manipulation has a variety of applications in many fields such as domestic facilities, assistive dressing, folding clothes [28], garment sorting, food industry [22], and healthcare (like robotic surgery [14]). Furthermore, it plays a vital role in aerospace [33], automotive [15], manufacturing, and electromechanical industries [11]. Sanchez et al. introduce a very thoughtful survey exploiting the robots for deformable object manipulation in domestic and industrial applications [30].
Manipulating deformable linear objects (DLOs), like ropes, cables, tubes, strings, catheters, and surgical sutures, is considered in this paper. The effective solutions to this manipulation issue will help in implementing the robots in many subfields of the wide world of industrial manufacturing. For example, in the automotive industry, manipulating DLO is essential for the planning and installation of electrical cables inside the vehicle system. Also, in medical surgery, DLO manipulation plays a vital role in the suturing process where a flexible wire needs to be knotted [24]. Likewise, this manipulation issue has an essential role in other fields like power distribution and architecture.
The problem of manipulating DLOs was previously investigated. The most commonly investigated task in DLOs robotic manipulation is tying knots. Yamakawa et al. [37] suggested a trajectory planning scheme whereby the knot could be knotted at high-speed by a single robot arm depending on a precise dynamic model of the robot and the rope. The usage of recurrent neural networks for learning the knotting trajectories has been investigated by Mayer et al. [23]. Lee et al. [17] have suggested learning from demonstration (LfD) for learning a function which maps a couple of correspondence points and reduces a bending cost. Battaglia et al. [2] and Li et al. [18] model ropes by a mass-spring system and employ graph networks for learning the dynamics of the rope. Nevertheless, they suppose that the physical state of the rope is fully observable. A motion planner is proposed in [29] for bimanual manipulation of DLOs. This planner is used to tie multiple knots in an experimental setup. This strategy, however, is restricted to knotting problems. Instead of avoiding contacts, Zhu et al. [40] investigated the problem of robotic manipulation of DLOs with environmental contacts.
Inserting DLO in a hole is another commonly studied task, because of many valuable applications in the assembly operations [39]. A hand-eye system was developed in [13] to insert a cord into a hole through the usage of stereo vision, to compute the relative location between cord tip and hole. A method for inserting string through narrow openings employing an approximate Jacobian for estimating the string motion was proposed in [36]. Zanella et al. [39] have inserted a DLO into a hole through estimating the force that acts on the DLO, employing the analysis of the feedback from a tactile sensor via a recurrent neural network (RNN). A model-based manipulation scheme for linear flexible objects is presented by Cheng et al. [5]. They suggested a 3D geometric model for the DLO which was exposed to gravity in addition to a physical representation with several links combined by revolute joints. They displayed the feasibility of their scheme by achieving the plug task, in which a robot employed to unplug a power cable from a socket and then plug it into another.
Vision or/and tactile data are employed in many approaches to this style of problem. Frequently vision is utilized alone because of its effectiveness in data gathering [31]. Though, this solution may be unsuccessful in the existence of different lighting settings and occlusions. The implementation of tactile sensors increases the success rate through overcoming many environment restrictions [26]. Hence, there have been several articles that combine vision and tactile data in one approach to estimate the object features [3,4].
Programming by demonstration directly utilizes the experience of an expert and transforms it into a technical system. A skill controller is built to resolve a hose inserting task through an event-driven hybrid controller [12]. The parameters of the controller are determined from the human demonstrations. A methodology to create an automatic program for achieving a DLO-in-hole task was introduced in [1]. A series of topological contact DLO states and the environment was obtained from a human demonstration in a virtual setting. One major limitation is the difficulty in handling the positioning inaccuracies. To summarize, programming by demonstration provides a promising path for teaching manipulation skills. On the other hand, in general, it is not robust against environmental location uncertainties and initial state changes between demonstration and autonomous implementation.
Only a few works try to consider the control of DLO shape by a robot. Rambow et al. [27] applied a single teleoperated demonstration for mounting a flexible tube in a desired shape by the usage of a two-armed robot. A learning-based system depending on a human demonstrator was introduced by Nair et al. [25], in which a robot takes a series of images illustrating small deformations of a DLO from a starting to the desired state. Then, by following the provided demonstration steps, the robot performed a series of actions for leading the DLO to the desired shape. Baxter robot was arranged in [25] for gathering interaction data with the DLO for 500 h. These data were employed for learning a dynamic model of DLO manipulation which is then utilized for imitating the human demonstration. Yan et al. [38] presented a model-based and visual robot manipulation of DLOs. They showed DLO state estimation using self-supervised training on actual images. With estimated DLO configurations, they trained a differentiable and quick neural network dynamic model which includes the physics of the mass-spring system. Indeed, the desired outcomes still seem far away, despite the invested sources in the last years. If applicable, classical engineering methods give considerably superior results at the expense of flexibility and generality.
The main objective of this article is to reshape a DLO from an initial configuration to a desired one. We assume that the manipulation will be accomplished using a singlearmed robot equipped with a gripper. The single arm can grasp a DLO from a selected point and move it to another one on the same plane (the table surface). Moreover, the manipulation system is assumed to be able to detect the initial state of the DLO by using an appropriate vision system. For instance, the technique proposed in [7] may be used for providing the initial DLO state for starting the algorithm. Then, the manipulation technique will perform a series of grasping and releasing primitives for reshaping the DLO as desired. The optimal grasping point at each primitive is selected by the decision process which employs an error interpolation and then determines the related releasing point. To this end, a multivariate dynamic spline model is utilized as a mathematical model for the DLO. This model enables a real-time and fast evaluation of the DLO behavior during the manipulation procedure. The symplectic integrator is employed for solving the DLO model iteratively to interpolate the DLO state during the manipulation. This symplectic integrator is selected because it is the most stable and the fastest compared to other integrators as reported in our previous work [16]. This is the first work, to the best of our knowledge, that focuses on the problem of the autonomous reshaping of a DLO from a general shape into a desired one, without the need to illustrate the intermediate deformation steps. Besides, the states of the DLO are directly estimated from the exact dynamic model. There are many benefits of this choice. First, it enables incorporating physics priors concerning the DLO behavior when it is manipulated. Such models of dynamics give more realistic behavior estimations over a longer horizon than pixels-learned dynamic models. Second, the states of the DLO do not depend on the environment or the DLO appearance. Consequently, the dynamics in a setting can be utilized in other visually altered settings directly.
The remainder of this paper is prepared as follows. The main features of the mathematical DLO dynamic model are summarized in Section 2. The symplectic integrator scheme is presented in Section 3. Furthermore, Section 4 introduces the proposed manipulation technique. Additionally, Section 5 presents the simulation results. Finally, Section 6 drafts out the conclusions and future work.

DLO dynamic model
To successfully predict the DLO behavior, its deformation/flexibility model is required. This model implies an accurate representation of both the mechanical behavior and the geometry of the DLO. After defining the model, a computationally effective manner is required to evaluate this model over time and to solve the inquiries of motion planning. DLOs are characterized by their shape variations, due to the impact of forces and various environmental constraints. DLO modeling, in general, is a complex and major challenge with a wide range of purposes. Many methods for the DLOs modeling are discussed previously. These methods include mass-spring [20], elastic rod [19], Multibody [32], finite element [10], dynamic spline [35], and other methods. Lv et al. [21] produced a recent review paper that discussed the approach, advantages, and drawbacks of each DLO modeling method. The dynamic spline is a modeling method that offers a strong theoretical basis, higher authenticity, and a continuous model. A multivariate dynamic spline is utilized to model the DLO in this article. Moreover, a symplectic integration approach is employed to iteratively resolve the model and interpolating the DLO state throughout the manipulation process. We decided to exploit the symplectic integrator due to its advantages over other approaches, as proved previously in [16].
The DLO shape can be represented effectively by a thirdorder spline basis that is a function of a free coordinate u. This coordinate u denotes the position along the DLO. It is equal to zero at one endpoint and equal to L at the other endpoint, given that L is the DLO length. This could be written mathematically as: where For several reasons, this mathematical model for the DLO is very successful. First, the calculation of the spatial derivatives executed in a straightforward manner, i.e., where q (j) is the j th derivative of q. Indeed, it can be identified by the same coefficients besides the simple-tocompute derivatives of b i (u). Second, the proprieties of the spline basis confirm the reduction of the model curvature [6], which characterizes the physical behavior of the DLO. Third, this model enables to represent a general nonlinear function with smoothness features as a linear combination of the nonlinear basis function b i (u), that relies only on the free variable u, by the linear coefficients q i . The DLO dynamic model can be described, by employing the system's Lagrange equations, as a function of the control points q i : where F i denotes the applied external force to the i th control point, K represents the total kinetic energy, and U is the whole potential energy. The effect of stretching, bending, torsion, and gravity on DLO is responsible for producing this potential energy. It is worth mentioning that, in the case of dynamic splines, the control points are used as the number of degrees of freedom.
The DLO overall dynamic model could be written, according to the approach described in [34], by expanding Eq. (3) to all control points.
where F represents the vector that consists of the whole external forces, incorporating gravity. P refers to the vector that includes all elastic forces. The complete details of the overall DLO dynamic model and its related terms, which we used in this work, are existed in [16]. At each simulation step, solving the system can be accomplished by employing a simple LU decomposition. Furthermore, at every time step, the control points' positions and velocities can be obtained by integrating the accelerations. Many integration methods can be utilized, but several problems may arise. These problems include the long integration time, numerical instability, and the inappropriateness for long-time predictions. The use of the symplectic integrator will alleviate these problems, as we proved in our previous work [16]. This symplectic integrator and its specifications are addressed in more detail in the next section.

Symplectic integrator
The symplectic integration scheme is employed for solving the model iteratively and interpolating the state of the DLO throughout the manipulation. This symplectic integrator got particular attention because of its merits over other approaches in resolving the dynamic equations. The previous results in [16] demonstrated the ability of this symplectic integrator to surpass other integration approaches in predicting the behavior of the DLO. These results confirmed that the symplectic integrator is the most stable and the fastest integrator against the other considered integration approaches.
In nonlinear dynamics, symplectic integration is commonly utilized. It is developed as the numerical solution of Hamiltonian equations that given by: where q represents the position coordinates, p refers to the momentum coordinates, and H denotes the Hamiltonian that can be obtained from: where K refers to the kinetic energy, and U represents the potential energy. Collection of position and momentum coordinates (q, p) is known as canonical coordinates. The initial step for utilizing the symplectic integrator, in our case, is to convert the Lagrangian equations into Hamiltonian. This conversion is carried out following the procedure discussed in-depth in [16]. Forest [8,9] produced the general form of the symplectic integrator in the fourth order as follows: where τ denotes the size of the time step. q 0 and p 0 refers to the initial values, and q 4 and p 4 refers to the numerical solution after τ . Also, the numerical coefficients ci and di are uniquely computed from:

The proposed manipulation technique
The new manipulation technique for reshaping a DLO on a table will be presented in detail in this section. We assume utilizing a robot with a single arm to reshape the DLO that is lying on a table. The robot arm has a gripper to interact with the DLO. This gripper has two degrees of freedom (one for opening/closing the two fingers and one rotational for the orientation). The manipulation will be achieved using a sequence of picking (grasping) and releasing primitive motions. The robot arm will move only one point from the DLO current state (grasping primitive) to the related point in the desired or goal state (releasing primitive). These motions will be repeated with other points until reaching a certain level of error (error threshold) between the desired and the current DLO state. A decision process will be responsible for selecting the optimal grasping point and its related releasing point in each iteration. A DLO is lying freely on a table, in front of the robot, at a certain height z*. A fixed camera pose is assumed above the table to obtain the input image initially. The robot's interaction with the DLO is limited to two simple primitive motions which are grasping the DLO at the location (x1, y1) and releasing it at the location (x2, y2), where x1, y1, x2, y2 are the coordinates in the table plane. The grasping (x1, y1, z*) and releasing (x2, y2, z*) coordinates can be estimated in the base frame because both the camera pose and table height are known with respect to the robot base frame. During the grasping motion, the robot firstly will approach the grasping point (x1, y1, z*) from the top, with an offset of 0.05 m along the z-axis, and open the gripper. It descends along the z-axis with a linear motion until it reaches z*, then the gripper will close. Then, it moves towards the releasing point (x2, y2, z*). After reaching the releasing point, the gripper will open before rising back to the same offset. In both grasping and releasing motions, the motion planning of the robot arm will be automatically performed using the native robot's inverse kinematics solver.
A flow chart of the proposed manipulation technique is shown in Fig. 1. The technique is begun by setting the control points, let us name them I 0 , equal to the current DLO configuration. As stated before, we presume that a vision system, like the one described in [7], can be employed at this initial stage to detect the DLO and produce a spline-based representation for the DLO shape. Also, I desired represents the control points of the desired or target DLO configuration. The decision process will select the optimal grasping point and the related releasing point using the initial configuration I 0 and the desired state I desired .
The decision process divides the DLO into small segments. So, the DLO has optional grasping points (n s ) at the beginning and the ending of each segment. The decision process chooses only k points at equal distances over the DLO optional grasping points. k is a small integer number. For example, if k = 6 and n s = 101, the candidate grasping points will be 1, 21, 41, 61, 81, 101. The decision process will use the spline-based dynamic model to simulate the evolution of the DLO according to the applied actions. It begins with the candidate grasping point number 1 and applies the action u 1 to move this point to the related point on the desired DLO state. This action u 1 is simulated by a force vector that is proportional to the difference between the position of the desired point and the actual point. The evolution of the DLO control points will be determined after this action u 1 by the help of the dynamic model. This DLO state is denoted by s 1 . This process will be repeated with the remaining candidate grasping points using the actions u 2 , u 3 , ..., u k , respectively. The produced DLO states will be represented by s 2 , s 3 , ..., s k , respectively.
The stage of selecting the optimal grasping point will compute the norm of the error for each candidate grasping point and its candidate state after applying the candidate action. This error is the difference between the desired state I desired and the candidate states s 1 , s 2 , ..., s k , respectively. Using the candidate grasping points and the related error norm, a spline interpolation will be accomplished. Then, the grasping point with the minimum error norm is selected to be sent to the robot arm. The robot arm will perform the grasping and releasing motion primitives. Then the DLO current state will be updated to be I 1 and the decision process will be repeated. The repetition process will continue until reaching a certain level for the norm of the error (error threshold) which means that the current state and the desired state are nearly the same.
We may wonder if we can simply identify the grasping point by selecting an arbitrary point from those points which do not coincide with the desired state. This is obviously very ineffective because it ignores the interconnection property

Simulation results
In order to assess the proposed manipulation technique that is discussed in the previous section, numerical simulations have been performed on MATLAB. The DLO state evolution is aimed at this section. This DLO is influenced by internal inertial and elastic forces besides the external forces such as the gravitational force, the friction force between the DLO and the table, and the applied action force from the gripper of the robot arm. All the simulations have been achieved utilizing the multivariate dynamic splinebased model in Eq. (4). In these simulations, the symplectic integrator is used to solve the dynamic model iteratively. During this work, the simulations have been implemented on Ubuntu 18.04.5 LTS operating system, processor intel core i5-3210M CPU@2.50GHz x 4, RAM 8GiB. In these simulations, the material of the DLO is selected to be aluminum. The DLO length L is equal to 2 m. Moreover, the numbers of control and sample points along the DLO are set to be equal to 9 and 101, respectively. These numbers are selected to be sufficiently small but produce a good interpolation capability. The model parameters which are utilized in these simulations are presented in Table 1.
The DLO that is lying on a table is simulated using the spline-based dynamic model. While, in these simulations, the table itself is simulated by a spring and a damper. The generated force from the spring and the damper is called the normal force and it will affect the DLO state. This normal force will make a balance with the other forces that influence the DLO such as the gravitational force. So, the DLO will be kept stabilized at a certain height z*= 0 m and did not fall below this height value. The spring constant is equal to 5 kN/m, while the damper constant is equal to 40 Ns/m.
The friction force vector, that affects each control point of the DLO due to contact with the table, is represented by F friction . This friction force can be calculated from: where V is the velocity vector of the control point. β and α are constants and equal 50 and 0.1, respectively. In this work, the robotic arm is just simulated by a force vector and denoted by F robot . This force vector is proportional to the difference between the position of the desired point and the actual point. It is calculated from: where P desired and P current are the coordinate of the desired and the current point of the DLO state, respectively, while γ is a constant and tuned to be equal 250 N/m. Figure 2 shows the sequence of DLO states results from the deformation steps which are carried out by the robot iteratively, from the initial I 0 to the target or desired shape I desired , in 6 successful experiments. These results demonstrate that the proposed model-based manipulation technique can reshape the DLO into a variety of shapes, with different initial configurations, in only a few steps. A sample video to show the numerical simulation of the sixth experiment in Fig. 2 can be found in the link 1 . In this Fig. 2 Sequence of DLO states that resulted from the deformation steps carried out by the robot iteratively, using the new manipulation technique, in 6 successful simulation experiments video, the computing time for the four primitive motions is 128.98 seconds. Also, the grasping points for the successive primitive motions are 51, 91, 2, and 101, respectively. Figure 3 presents the error norm versus the grasping points for each iteration of the first simulation experiment in Fig. 2. In each subfigure, the blue circles represent the error norm at the candidate grasping points (1,21,41,61,81,101). While the spline interpolated curve is represented by the dashed red line. In each iteration, the decision process will choose the grasping point with minimum error norm (the point indicated by a blue star) from the interpolated data and send its value to the robot arm. In this experiment, the Fig. 3 Error norm versus the grasping points for each iteration of the first simulation experiment in Fig. 2 grasping points for the successive primitive motions are 21, 97, 4, 13, and 82, respectively.

Conclusions
This research introduces a new model-based manipulation technique for reshaping a deformable linear object (DLO) by a single-armed robot. This new technique performs the manipulation process employing a series of grasping and releasing primitives. The optimal grasping point and its related releasing point are chosen by a decision process. This process makes a spline interpolation between the acquired error values from candidate grasping points and selects the optimal point that has a minimum error. The proposed technique utilizes the multivariate dynamic spline model for representing the DLO and its behavior during the manipulation process. Also, the symplectic integrator is used for solving the DLO dynamic model iteratively and interpolate the DLO shape. It is selected because it is the most stable and the fastest integrator compared to other considered integration schemes. In order to evaluate the manipulation technique, MATLAB simulations for reshaping a DLO lying on a table are accomplished. The obtained results demonstrate the intermediate deformation steps which lead the DLO from its initial configuration to the desired one. Finally, these results prove that the new technique can manipulate the DLO into various shapes in few steps.
Although the presented work in this paper concentrated on reshaping a DLO lying on a table, our goal in future work is the extension of the study to reshape DLOs, like electric wires, in the assembly and manufacturing processes. The proposed manipulation technique will be validated using the practical experiments. The proposed technique will be extended to manipulate multi-branch DLOs, like the pre-gathered wiring harness involved in the aerospace and automotive industries.