Design and performances of magnetic tactile sensor
Mimicking the layered structure of sensitive mechanoreceptors such as human skin (especially for normal force) and fish lateral line (especially for shear force), we designed and fabricated a flexible magnetic tactile sensor with sandwich-like structure. Figure 1a shows the basic structure of this sensor and the composition of each layer. The "Touch layer" is a flexible magnetic film (20mm×20mm; thickness, 1mm) made by mixing the elastomer matrix and NdFeB micromagnets (Supplementary Information Section 1). The composition of the magnetic film under the micro-CT scan is as shown in Fig. 1b, where the gray part is elastomer matrix and the blue particles are NdFeB micromagnets. The magnetization direction of the magnetic film is from the four sides to the center, which is achieved by folding the square magnetic film into the shape of an arrowhead and placing it vertically in a strong magnetic field (4T) to magnetize (Supplementary Information Section 1). The "Buffer layer" is a flexible layer made of soft silicone elastomer (Ecoflex 00–30; thickness 10mm). By mixing silicone oil (PMX-200; viscosity 5cs) into it, we can change the elastic modulus of this layer, thereby adjusting the measurement range and sensitivity of the sensor to force. The "Transmission layer" is a printed circuit board (PCB; thickness 1mm) with a 3D Hall sensor (MLX90393) mounted, which can measure the magnetic flux density in three axes. Moreover, as shown in Fig, 1a, lower middle, the physical photo of the tactile sensor, taking the spacer plate as the dividing line, the "Touch layer" and "Buffer layer" can be installed separately from the "Transmission layer", in order to meet the wireless working requirement when necessary. When an external force is applied to the "Touch layer" of the sensor, the magnetic field lines of the magnetic film will be displaced due to structural deformation, and then the fixed Hall sensor will sense the displacement of the magnetic film by detecting the change of three-axis magnetic signal. Furthermore, according to the corresponding relationship between force and displacement in the "Buffer layer", the perception of external 3D force is finally realized.
The simulation by Comsol Multiphysics shows that the centripetal magnetization method exhibits larger magnetic signal than the unidirectional magnetization method (Fig. 1c). Here, Bz_method 1 and By_method 2 are the uniaxial magnetic strengths that vary the most at the 3D Hall sensor of these two methods, respectively. The change in magnetic signal is the key parameter that determines the performance and working distance (i.e. the distance between "Touch layer" and " Transmission layer") of the magnetic tactile sensors. Through the centripetal magnetization method, the magnetic signal fluctuates more efficiently when “Tough Layer” has a perpendicular displacement under press, e.g. ∆B1 is approximately twice larger than the value of ∆B2. So, the sensitivity to displacement is higher, which can be expressed as SEN=∆B/∆Z. Therefore, the proposed magnetization direction of the sensor’s magnetic film improves the sensitivity of the sensor to displacement by increasing the magnetic field gradient under the magnetic film.
The centripetal magnetization method also enables the sensor with 3D force decoupling ability. When various forces in different directions were applied to the "Touch layer" of the sensor, the output exhibited corresponding decoupling signals (Fig. 1d). When the magnetic film only produced normal displacement along the z direction (only the force along z direction was applied), the signal S changed but the signals Ry and Rx remained constant. When the magnetic film only produced shear displacement along the y direction (only the force along the y direction was applied), the signal Ry changed but the signal S and Rx remained constant. When the magnetic film only produced shear displacement along the x direction (only the force along the x direction was applied), the signal Rx changed but the signal S and Ry remained constant. In other words, the signal S is independently related to the displacement along the z direction, while Rx and Ry are independently related to the displacements along the x and y directions, respectively, indicating that S, Ry and Rx are decoupled from each other within the calibration range (z:0 ~ 3mm, y:-3 ~ 3mm, x:-3 ~ 3mm). According to the measurement data, it can be obtained that the sensor has an accuracy of 0.83% in the normal direction and 1.67% in shear direction. Here, S, Rx and Ry are defined as
$$\left[ \begin{gathered} \begin{array}{*{20}{c}} S \\ {{R_x}} \end{array} \hfill \\ {R_y} \hfill \\ \end{gathered} \right]=\left[ {\begin{array}{*{20}{c}} {\sqrt {a \cdot {B_X}^{2}+a \cdot {B_Y}^{2}+{B_Z}^{2}} } \\ \begin{gathered} \frac{{{B_X}}}{{{B_Z} - b \cdot {B_c}}} \hfill \\ \frac{{{B_Y}}}{{{B_Z} - b \cdot {B_c}}} \hfill \\ \end{gathered} \end{array}} \right]$$
1
where Bx, By and Bz are the magnetic flux densities along the x, y and z directions, respectively. Bc is the maximum difference of Bz along the central axis of the film within the calibration range. a and b are the compensation coefficients of S and R, respectively. For the magnetic film we fabricated in this paper, a is equal to 2.86 and b is equal to 0.37, which are obtained by analyzing the magnetic distribution corrected model and fitting experimental calibration data. See Supplementary Information Section 2 for detailed information.
In addition, the same magnetic signal outputs (Fig. 1e) could be observed when exerting periodic compression of different frequencies (0.25Hz, 0.5Hz, 1Hz and 2Hz) on the sensor in the normal and shear directions, respectively. The ∆S is around 75 and the ∆R is around 0.1 at a compression of 1mm. A total of 3000 loading/unloading (2mm, 0.5Hz) cycles of data were recorded and the magnetic signal was sketched in Fig. 1f. The S and R amplitude almost maintained the same level after a total of 3000 cycles, revealing the high stability and durability of the flexible magnetic tactile sensor. The good adaptability to external forces of different frequencies and excellent stability over a long period of time enable our sensor to deal with complex force sensing in future applications.
Force Decoupling Mechanism And Calibration Of Sensor
The magnetic distribution under the magnetic film was analyzed through mathematical calculation to study the decoupling of the centripetally magnetized film in the Y-Z plane. According to the molecular circulation hypothesis, we equivalently converted the unidirectional magnetic field in the magnetic film into some current loops in the same direction, and the currents along the Z direction cancelled each other out. Finally, the magnetic field of the centripetally magnetized film can be equal to concentric current loops evenly distributed on the upper and lower surfaces of the film (Fig. 2a). In this case, the magnetic field distribution of the magnetic film can be calculated by Biot-Savart law, which states that at any point P, the magnetic field \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {B}\) due to an element \(d\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {l}\) of a current is given by
$$\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {B} =\int {\frac{{{\mu _0}I}}{{4\pi }}} \frac{{d\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {l} \times \overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {r} }}{{{r^3}}}$$
2
where the constant \({\mu _0}\) is the permeability of vacuum and is exactly 4π × 10− 7 T·m/A, is the current intensity, \(d\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {l}\) is an infinitesimal segment in the same direction as the current and \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {r}\) is a vector that points from \(d\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\rightharpoonup}$}} {l}\) to point P. The detailed calculation process of the magnetic field distribution model can be found in the Supplementary Information Section 2. Since the magnetic film was magnetized by folding method, the crease of the film will bring a certain error. Therefore, the parameter J (equivalent current density on the surface of the magnetic film, 1.76×105A/m) was corrected accordingly (Supplementary Information Section 3). Figure 2b shows the experimental measurements and corrected model theoretical calculations of the magnetic distribution in the y-z plane below the magnetic film. With the maximum root mean square error (RMSE) between them being only 3.90 µT (Bz) and 1.51 µT (By), it was verified that the corrected model can accurately reflect the real magnetic distribution of the centripetally magnetized film.
Based on this magnetic distribution corrected model, we decoupled in 3D space, and obtained three decoupling parameters S, Ry and Rz, which are expressed in Eq. (1). The distribution of decoupling parameters values in the 3D space below the magnetic film can be represented as shown in Fig. 2c, where S, Ry and Rx remain constant along their iso-surfaces, respectively. For isotropic materials, normal stress and shear stress are independent of each other, so the displacement in each direction can be directly linked to the force. Thanks to this, the relationship between the decoupling parameters and the external force could be directly obtained from their relationship to the three-axis displacement, which is more convenient to conduct and present. It could be observed that the parameter S increases when ∆Z (or normal force) increases, but remains constant as ∆Y or ∆X (or shear force) increases (Fig. 2d), which further illustrates that the measurement of normal force is independent of shear force. Similarly, consistent conclusion can be drawn for shear force (Fig. 2e and f), i.e., the measurement of shear force is also independent of normal force. Moreover, the shear force along the x direction and the shear force along the y direction are also independent of each other. In addition, the displacement resolution of the sensor could be obtained here: 0.013 mm in normal direction and 0.10 mm in shear direction.
The elastic modulus of "Buffer layer" can be changed to adjust the measurement range and sensitivity of the sensor by mixing silicone oil (PMX-200; viscosity 5cs) into the silicone. As shown in Fig. 2g, when the mass ratio of silicone oil and silicone is gradually increased, the elastic modulus of the soft silicone elastomer will gradually decrease. Here, we fabricated 5 soft silicone elastomers with different mass ratios, whose elastic modulus can be adjusted from 5.2 kPa to 47.5 kPa without significant change in fracture strain. In this way, our tactile sensor can adapt to different working scenarios, such as measuring large forces with a low sensitivity or small forces with high sensitivity. After measuring the force-displacement relationship of "Buffer layers" with different mass ratios (Supplementary Information Section 4), the final calibration curves of the tactile sensors could be obtained and applied to the perception of 3D forces (Fig. 2h and i). And when the buffer with the minimal elastic modulus (5.2 kPa) was applied, the force resolution of the sensor will reach 0.027 N in normal direction and 0.015 N in shear direction.
Adaptive Grasping And 3d Hybrid Force Sensing
The sensor was mounted on a robotic hand (made by 3D printing) to equip it with capability of tactile sensing, and the elastic modulus of the soft silicone elastomer can be adjusted to adapt to different working scenarios. As shown in Fig. 3a, the robotic hand with a hard-substrate (E ~ 47.5kPa) sensor was prone to damage the fragile object (a tomato), meanwhile, the robotic hand with a soft-substrate (E ~ 5.2kPa) sensor had difficulty stably grasping the heavy object (an aluminum block). However, after exchanging the substrates, they both achieved a non-destructive and stable grip. In addition, it should be emphasized that, through the split-type design, the substrate of the tactile sensor can be easily replaced without moving the Hall sensor. In the process of grasping an object, when the normal force reaches a threshold value, the size of the object can be calculated by reading the rotation angle of the servo (the corresponding relationship between the rotation angle and the size is calibrated in advance). At the same time, according to the magnitude of the shear force, the weight of the grasped object can be calculated. In other words, the robotic hand can sense the size and weight of the object simultaneously when grasping an object. Several objects were grasped and measured (Fig. 3b), the average size deviation was 0.15mm and the average weight deviation was 1.23g. It should be noted that since the robotic hand had only one finger mounted with the sensor, the barycenter position of the object will affect the measurement results of weight.
Using a robot or robotic hand to grasp an object is no longer a novelty, but grasping an object that changes in weight and adapting to external disturbances while grasping is still interesting and worth exploring. Benefiting from the force decoupling function, our tactile sensor can sense normal force and shear force simultaneously, which enables it to conveniently control the friction angle (arctan(Fs/Fn)) during the grasping process of the robotic hand to achieve adaptive grasping.
The robotic hand equipped with tactile sensor was used to grasp a volumetric flask (capacity 250ml, weight 90g) (Fig. 3c), whose weight was changed by continuously filling it with water through a funnel (weight 18g). Figure 3d shows the curves of Fs, Fn, and Fs/Fn during the experiment, at t1 (12s), the robotic hand started to grasp the volumetric flask and raised the positive pressure to more than 3N (preset initial normal force). The volumetric flask was released at t2 (13s), and the tactile sensor began to sense the shear force. After maintaining a stable grip to t3 (17s), the robotic hand entered the state of adaptive grasping, keeping Fs/Fn between 0.33 (close to the limit of too tight) and 0.57 (close to the limit of slipping). The funnel was placed at t4 (21s), which caused a jitter but was quickly stabilized. With the slow pouring of water (from t5, 27s to t6, 50s), the robotic hand could maintain Fs/Fn stably between the two bounds, and remained roughly stable even when the weight was rapidly reduced by the lifting with human hand at t7 (55s) and t8 (60s).
Sensing the magnitude and direction of hybrid forces simultaneously in 3D force space through 3D force decoupling is a unique advantage of our magnetic tactile sensor. As shown in Fig. 3e, when an external force (hybrid force) is applied to the "Touch layer" of the sensor, the magnetic film will have displacements along the Z, Y and X coordinates (∆Z, ∆Y and ∆X) relative to the Hall sensor, respectively. Relatively, the Hall sensor will have displacements along the S, Ry and Rx gradients, according to which the changes of S, Ry and Rx can be obtained to calculate the magnitudes of normal force (Fz) and shear force (Fy and Fx). We used the tactile sensor to sense the magnitude and direction of finger pressure, which were displayed on the monitor in real time (Fig. 3f). The yellow plate under the finger was used to ensure that the magnetic film remained flat when an external force was applied.
The tactile sensor could also be applied to human-machine interaction. We used magnetic-based tactile sensor as a control "joystick" of a tank-like robot, mapping the 3D force applied by the human hand to the different movements of the robot. Fx controlled the robot to move forward and backward, Fy controlled the robot to turn left and right, and Fz controlled the robot to grab and place (Fig. 3g). Through the mapping between force and movement, the robot could complete some simple tasks under control, such as grasping and carrying objects (Fig. 3h). It can be concluded that the split-type flexible magnetic tactile sensor can meet the various sensing needs of existing robots and some special human-machine interactions. Moreover, the split-type design allows the sensor to be wirelessly mounted on various robotic hands and shells to act as bionic skin units.
Specialization Of Shear Sensing And Fluid Velocity Measurement
The tactile sensor can also be applied underwater for shear fluid flow velocity sensing, imitating the lateral line of fish, which further illustrates the advantages of split-type sensing and demonstrates the great application potential of the sensor.
The shear force due to the water flow is very small, which can only cause a tiny shear displacement on the solid soft silicone elastomer. Moreover, the adjustment of the elastic modulus of the soft silicone elastomer by silicone oil is limited and too much silicone oil will prevent the silicone from curing. Consequently, we finally chose to change the structure of the "Buffer layer" to observably reduce its shear elastic coefficient, and thus achieved the specialization of shear sensing. The fangtooth is a deep-sea fish that has inspired us about the structure of the flow velocity sensor. Living in the deep sea of 1,000–5,000 meters all the year round, it has no vision and thus evolves a developed surface tactile sensory organ–lateral line (Fig. 4a), whose slender sensory units enable it to sense changes of the surrounding flow field with great sensitivity, endowing the fangtooth with a superb ability to prey, deal with threats, and identify directions. Mimicking the slender unit array structure of its lateral line, and on the premise of keeping the position and shape of the magnetic film unchanged, a grid-shaped "Buffer layer" was designed (cured in a grid-type mold). Compared with the solid substrate, the grid-shaped substrate could generate larger shear displacements and magnetic signals under the same shear forces (about 38 times), thus exhibiting great sensitivity in the shear direction (Fig. 4b).
The shear displacements of grids with different structural parameters under the same shear force (150 N/m2) were simulated and compared in COMSOL Multiphysics (Fig. 4c) to determine the appropriate structural parameters of the grid. It can be observed that the shear displacement was significantly improved with the reduction of the number and thickness of grids. However, a too small number of grids will bend the magnetic film during the deformation, which will cause a non-negligible error. In addition, a grid with too small thickness is not only fragile and difficult to form, but also produces excessive displacement of the magnetic film under the same shear force, which is far beyond the operating range. Therefore, according to the actual flow velocity measurement requirements, here we selected the substrate with a grid number of 4 and a thickness of 1mm.
A flow velocity measurement experimental device was set up to test and visually demonstrate the wirelessly measurement of the shear fluids flow velocity through split-type mounting (Fig. 4d). The grid-shaped substrate with a magnetic film was a scaled-down version, which was mounted on the inside of the square tube wall, while the Hall sensor was mounted on the outside. The center of centripetally magnetized film and the center of Hall sensor were aligned by judging the position where the magnetic flux density along the Z coordinate (Bz) was maximum. The water flow in the square tube was driven by a centrifugal pump, whose power could be controlled to adjust the velocity of water flow in the tube. Meanwhile, a flowmeter was connected between the pump and the square tube to measure the true velocity of water flow. The experimental photo shows the state of the grid-shaped substrate in the water flow. In addition, since we were only concerned about the sensor's perception of the force in the shear direction, we only needed to compare the magnetic flux density along the Y coordinate (By) at different flow velocities, which varied most significantly in the triaxial magnetic flux density. As shown in Fig. 4e, with the increase of the water flow velocity in the square tube, the grid-shaped substrate gradually exhibited more and more obvious shear deformation, and the magnetic flux density along the Y coordinate (By) also increased ensued. The averages of the magnetic flux densities at various flow velocities were fitted by a quadratic curve with a coefficient of determination over 0.99. However, it can be observed that the magnetic flux density value was fluctuant, and the fluctuation increased with the increase of flow velocity, which was caused by two main reasons: the first is that the centrifugal pump brought non-constant flow velocities, the more important reason is that unstable vortices were prone to appear near the square grid, which will be addressed by structural optimization of flexible substrates in future research.
Navigation Application Of Fluid Velocity Measurement
Now that the feasibility of measuring the velocity of the fluid in tube has been verified, in the same way, the velocity of moving objects in water (such as ships, submarines, autonomous underwater vehicles (AUV), etc.) can also be monitor by measuring the flow field around them. As shown in Fig. 5a, two sets of split-type magnetic tactile sensors were installed separately on the inner and outer sides of the toy ship’s hull (the data from one of them was selected), achieving concentricity by judging the position where the magnetic flux density along the Z coordinate (Bz) was maximum. The film’s magnetic field sensors shown in the blue wireframe is the Hall sensors at the location of the magnetic films, which were used to measure the change of the magnetic flux density caused by the displacement of the magnetic film as well as the movement of the toy ship in the surrounding magnetic field, whose interference could be canceled by the environmental magnetic field sensors shown in the yellow wireframe. The green wireframe shows the positions of two magnetic films with grid-shaped substrates. The split-type tactile sensor enabled the measurement of the toy ship speed wirelessly, and keeping the hull intact at all times. When the ship moves over a small area (the environmental magnetic field remains constant), the direction of the ship can be directly identified through the 3D magnetic flux density of the geomagnetic field (Fig. 5b). Bx-By-Bz is the magnetic field coordinate system of the environmental magnetic field sensor. B0 is the geomagnetic field with a constant direction, B0xy is the projection of the geomagnetic field on the x-y plane, and θ is the steering angle of the ship. When the toy ship rotates θ degrees, it can be equivalent to the geomagnetic field B0 rotating θ degrees in the opposite direction in the Bx-By-Bz coordinate system. Therefore, θ, the steering angle of the ship, can be obtained according to the angle between B0xy before and after the ship’s steering, which can be calculated by:
$$\theta {\text{=}}\arccos \left( {\frac{{{B_{x0}}{B_{x1}}+{B_{y0}}{B_{y1}}}}{{\sqrt {{B_{x0}}^{2}+{B_{y0}}^{2}} \sqrt {{B_{x1}}^{2}+{B_{y1}}^{2}} }}} \right) \cdot \frac{{{B_{x0}}{B_{y1}} - {B_{x1}}{B_{y0}}}}{{\left| {{B_{x0}}{B_{y1}} - {B_{x1}}{B_{y0}}} \right|}}$$
3
where Bx0 and By0 are the magnetic flux densities along the X and Y coordinates before the ship's steering measured by the sensor, respectively, Bx1 and By1 are the magnetic flux densities along the X and Y coordinates after the ship's steering measured by the sensor, respectively.
Figure 5c shows the corresponding magnetic flux density along the Y coordinate (By) when the toy ship sails at different speeds in the water, according to which the corresponding relationship between the speed of the toy ship and By can be obtained by quadratic curve fitting with a coefficient of determination over 0.99.
Combining the steering angle of the toy ship relative to the initial position and the speed along the direction of the ship, we can achieve the path navigation of the toy ship in the small size area. The toy ship was remotely controlled to sail in the swimming pool with the pattern "ZJU" as the route (Fig. 5d), the interval between the positions of every two adjacent toy ship's images is 1.5s. The magnetic flux density signals (Fig. 5e) of the magnetic film after canceling the interference of the environmental magnetic field were recorded. Meanwhile, according to the above steering angle algorithm and speed calibration curve, the steering angle of the toy ship relative to the initial position (perpendicular to the pool boundary) and the sailing speed of the toy ship were calculated in Fig. 5f and Fig. 5g, where the red curves are the angle measured by the gyroscope and the speed obtained by image processing, respectively. The extremely high coincidence of the curves proves that the steering angle and the sailing speed predicted by the tactile sensor have satisfactory accuracy. In addition, it can be observed that the angle of the ship was biased to the positive direction at the beginning, so the small deviation in the initial stage was the measurement error of the gyroscope. The relationship between the displacement and time of the toy ship was obtained through the integration of speed to time (Fig. 5h). Finally, combining the angle and displacement of the toy ship, we can get the predicted route in the x-y displacement plane (Fig. 5i), with a maximum offset less than 0.3 meters compared with the actual route. According to the results, the offset of the predicted route does not exceed 0.65% of the total displacement.
It can be concluded from the above experiment that the tactile sensor has the capability to wirelessly complete high-precision speed measurement of an object in water, meanwhile completing navigation in a small area only by the sensor itself. What's even more noteworthy is that since the tactile sensor measures speed relative to the water, it can also be used to perceive and monitor changes and disturbances of the flow field through arrayed and distributed installation. As we all know, most of the existing underwater navigation methods are based on Inertial Navigation System (INS), because it can provide all the required navigation data independently: acceleration, velocity, position, etc. However, if only the INS is used for navigation, there will be inevitable integration errors, which are usually corrected by additional velocity measurement systems. Compared with existing underwater velocity measurement systems, such as Doppler Velocity Log (DVL), computer vision systems and pressure sensing systems, etc., our tactile sensor can measure velocity wirelessly, keeping the hull or bulkhead intact, with the small volume, light weight and low cost at the same time, which meets the sensing requirement of some small AUVs, especially deep-sea small AUVs.