Behavioral biometric tactile systems are emerging technologies that identify people by analyzing the way they interact with their devices. For example, by measuring how fast the person writes or moves the pen and how much pressure they exert when interacting with the device, behavioral biometric tactile algorithms can authenticate the person signing a document1,2. Because personal habits are unique, behavioral biometric data is difficult to replicate, making these technologies better at preventing theft and fraud. Consequently, there is significant interest in using dynamic pressure sensing technology to identify and measure different types of forces that can be used for behavioral recognition3–14.
Optical tactile sensors are an attractive low-cost solution for interactive devices and tactile displays. While typical optical tactile sensors can measure normal force, information on in-plane motion such as velocity can only be obtained by positional analysis of two or more sequential images15–23. However, the resulting “time-averaged” velocity may not represent the true motion, especially when the directional change of motion is rapid with respect to the available rate of measurement, e.g., frames-per-second (fps) of a camera (Supplementary Fig. 1). For more accurate information on in-plane motion, the optical tactile sensor should ideally also measure “instantaneous” velocity of the in-plane motion. Because instantaneous slip velocity is highly correlated to the shear force applied to the solid-solid interface in contact mechanics, having an optical tactile sensor capable of instantaneously measuring both normal and shear force would be desirable for studying behavioral biometrics. Optical polymers24,25, photonic crystals26, optical light guide27, quantum dots28 and organic dyes29,30 have been used to build optical tactile sensors. However, these systems cannot decompose dynamic elements of forces because they have slow signal response, high limit of detection, and low spatio-temporal resolution. To analyze fast movements such as handwriting in behavioral biometrics, we need optical tactile sensors that can quantitatively decompose applied force into vertical normal and lateral shear forces from a single image.
Here, we report a highly sensitive optical tactile sensor system that can instantaneously decouple dynamic touch signals in a single image for behavioral biometric analysis. Our tactile sensor contains a stress concentration layer that resembles the sensory architecture of the human skin and an optical tactile layer that contains an array of upconversion nanocrystals (UCNs). When force is applied on the sensor, the stress concentration layer amplifies and transfers the dynamic force to the tactile layer that interfaces with a total internal reflection (TIR) dove prism. Our UCNs-based optical tactile sensor generates unique axisymmetric or non-axisymmetric upconversion luminescence signals with spatio-temporal patterns that contain information on the motion of the object such as direction, velocity, and magnitude of normal and shear force. Unlike conventional optical tactile sensors, our system can quantitatively decompose the applied force into individual components of vertical normal and lateral shear force from a single image in real-time. We used our sensor system for surface profiling of objects, high-resolution fingerprint recognition and anti-counterfeiting applications. Using a machine learning framework to analyze the luminescence tactile signals, we applied the sensor system in a small-scale Braille-to-Speech (BTS) translation system. Most importantly, our sensor accurately recognized behavioral biometric handwriting patterns. We expect such a simple sensor design to find ample applications in sophisticated behavioral biometric technologies.
Our tactile sensor system includes a tactile sensor pad, a TIR dove prism and a charge-coupled device (CCD) camera (Fig. 1a, and Supplementary Fig. 2). The tactile sensor pad is composed of a stress concentration layer at the top, a thin (20 nm) anti-reflective platinum (Pt) layer in between, and a signal generating tactile layer at the bottom (Fig. 1b). The stress concentration layer is designed to mimic the undulating interface of hard epidermis and soft dermis in human skin because such an interface is known to amplify applied forces by increasing interfacial area and localizing the applied force.32,33 Further, mechanical mismatch between the rigid epidermis and soft dermis enhances force transmission from the skin to sensory mechanoreceptors34,35. To recreate this microarchitecture, we used stiff thermoplastic polyurethane (TPU, 52.5 MPa) and soft polydimethylsiloxane (PDMS, 1.76 MPa) with an undulated two-dimensional (2D) array of micro-hemispheres patterned at the interface (Fig. 1c, and Supplementary Fig. 3). The Pt layer in between serves to suppress diffuse reflection and improve the accuracy of the images (Supplementary Fig. 4). The bottom tactile layer contains an array of PDMS micro-hemispheres embedded with UCNs (Supplementary Fig. 5 and Supplementary Note 1). For our tactile sensor, we synthesized a hexagonal phase yellow luminescent lanthanide-doped UCNs (β-NaYF4:Yb3+/Er3+/Gd3+; 30/2/30 mol%) using a conventional hydrothermal method (Fig. 1d, and 1e, and Supplementary Methods for details). When homogenously embedded into the PDMS micro-hemispheres (Fig. 1f and Supplementary Fig. 6), their luminescence remained unchanged (Fig. 1g, and Methods for details). UCNs are advantageous for our system because they have a high spectral resolution (Full Width at Half Maximum (FWHM) of ~ 20 nm), low optical signal errors and large anti-Stoke shifts. Further, UCNs do not require optical filters for luminescence color purification, have negligible background auto-fluorescence36 and are optically stable upon environmental perturbations37. Moreover, their luminescence colors can be easily tuned38 by simply adjusting the stoichiometry of lanthanide dopant ions (Supplementary Figs. 7, and 8, and Supplementary Methods for details). UCNs are also amenable to large-scale synthesis39.
In our system, signal generation depends on the contact between the UCNs-embedded micro-hemispheres (UMs) in the tactile layer and the TIR prism. Irradiating the prism with 980 nm near-infrared (NIR) laser generates evanescent waves in the thin (< 100 nm) boundary layer close to the interface of the prism and UMs layer. When an object moves across our sensor pad, the stress concentration layer amplifies and transfers the applied dynamic forces to the UMs layer. As the contact area between the UM layer and TIR prism increases, the UMs produce a luminescence signal measurable by the CCD camera and luminescence spectrometer (Fig. 1h). This phenomenon can be ascribed to the transformation of the evanescence wave field into propagating far-fields at contact points between the UM and prism, where TIR is frustrated due to the decrease in the refractive index contrast according to the higher refractive index of PDMS (n = 1.4) compared to air (n = 1.0) (Supplementary Figs. 9, and 10 and Supplementary Note 2). At the contact points, the propagating far-field components can now penetrate the UM, effectively illuminating the UCNs within the UM volume. Consequently, locally compressed UMs with a more extensive contact area with the prism generate a stronger luminescence signal, while those without applied force do not yield measurable luminescence. It is noteworthy that the non-linear luminescence property of UCNs suppresses the initial luminescence caused by the evanescence wave field from the minimal contact between the UM layer and the TIR prism in the absence of applied force, consequently reducing background noise40. Locally pressed UMs with higher contact area with the prism produce higher luminescence signals while those without applied force do not contact the prism and therefore, do not produce luminescence. When we applied a normal force to the top layer of the tactile sensor by vertically pressing a spherical indenter (5 mm diameter), the resulting luminescence signal was axisymmetric (Fig. 1i). The tactile sensor we fabricated exhibited uniform signal generation at across the entire sensor pad or 10 different sensor pads, and demonstrated highly stable signals under continuous laser excitation (Supplementary Figs. 11–13). However, when we simultaneously applied a lateral shear force in addition to the normal force by moving the pressed indenter along the surface of the top layer, the luminescence signal became non-axisymmetric – the shape of the signal is skewed towards the direction of movement (Fig. 1j). This suggests that detailed information on the surface-parallel motions of the indenter such as the magnitude of normal force, and the velocity, direction and magnitude of lateral shear force can be extracted by analyzing the non-axisymmetric shape of the luminescence signal. These features induced subtle changes in luminescence intensity, and to effectively observe this aspect, we proposed a tactile sensor based on UCNs instead of the conventional mechanoluminescence-based tactile sensors. Optical tactile sensor using UCNs have better sensitivity and response time compared with mechanoluminescence tactile sensors41,42. By comparing the luminescence shape from optical tactile sensor, we proposed a method to determine both the direction and magnitude of normal and shear forces in real-time using a machine learning algorithm for force discrimination (Fig. 1k). By analyzing the luminescence tendency such as axisymmetric and non-axisymmetric for static and dynamic optical tactile sensors, respectively, we proposed machine learning methodologies to discriminate normal and shear forces in a single image frame in real-time.
To understand how normal force affects the luminescence signals, we analyzed the signal intensity at different applied normal forces. When the magnitude of the normal force increased from 0 to 5.0 N, both luminescence intensity and the pressure distribution area increased in an axisymmetric circle (Figs. 2a and b, Supplementary Figs. 14, and 15, Supplementary Movie 1, and Methods). We observed that the stress concentration layer exhibited a strong and wide luminescence area with enhanced intensity due to its modulus difference and undulating interface (Supplementary Fig. 16). To confirm the correlation between grayscale and pressure, we first observed the area beneath the prism (Supplementary Fig. 17a). By combining the grayscale of Fig. 2a corresponding to the observed area, we could exhibit the correlation between pressure and grayscale (Supplementary Fig. 18). We obtained an optimized tactile sensor (1000 µm thick) including the stress concentration layer (200 µm thick) that can detect normal force down to 0.05 N, and later utilized for the dynamic force decoupling process (Fig. 2b, Supplementary Note 3 for optimization, and Supplementary Figs. 19–21). The characteristic luminescence intensity of yellow UCNs at 530 and 660 nm also increased with increasing normal force (Fig. 2c). The relationship between luminescence intensity and the cumulative gray value for each pixel exhibits consistent slopes concerning force magnitudes, providing the possibility of presenting a pressure distribution in a color map (Fig. 2d and Supplementary Fig. 22). Under normal force, our sensor did not show hysteresis in the luminescence intensity, indicating that the adhesion between the PDMS micro-hemispherical array and the TIR prism is negligible (Fig. 2e). Further, our tactile sensor is highly stable; its luminescence intensity remained constant when a normal force of 5.0 N was repeatedly applied over 1,000 cycles (Fig. 2f), and even at the high temperature (Supplementary Fig. 23). Furthermore, we measured the response time by repeatedly tapping the sensor (Figs. 2g, and Supplementary Fig. 24). We found the average response time using Gaussian fitting and it was 9.12 ms based on multiple repeated measurements.
We further examined the temporal change in luminescence signal under applied lateral shear force (Methods in details). We began by pressing a spherical indenter to apply a 1.0 N vertical preload on the tactile sensor pad (Supplementary Note 4). The indented area forms a circular axisymmetric luminescence intensity and pressure contour. Following this, we accelerated the pressed indenter along the top surface of the sensor pad at a constant rate of 1.0 mm/s2 until the velocity reached 1.0 mm/s. As soon as the spherical indenter began gliding, the axisymmetric luminescence circle appeared to stretch into a non-axisymmetric oval along the direction of the lateral shear force (Fig. 3a, and Supplementary Fig. 16). This shape change was recorded as a video clip at 10 frames per second (Supplementary Movie 2 middle), and the luminescence image frames were extracted from the video for analysis (Fig. 3a left). Both the pressure contour color map (Fig. 3a middle) and the luminescence intensity plot (Fig. 3a right and detailed in Supplementary Fig. 25) also clearly showed that the luminescence intensity profile had changed from a circle to an oval while the center maintained an axisymmetric circular luminescence (Fig. 3a middle). The non-axisymmetric luminescence continued to form until the indenter reached a specific velocity (Supplementary Figs. 26, and 27, and Supplementary Movie 2). Using a customized force gauge, we measured the corresponding shear force and recorded the values on each non-axisymmetric luminescence image (Fig. 3a, Supplementary Fig. 28). Unlike conventional tactile sensors where the direction of movement is obtained through coordination analysis of multiple images, we can immediately tell the direction of the applied shear force from the non-axisymmetric luminescence profile alone (Fig. 3b)43,44.
Because shear force induces friction force in the opposite direction45, we evaluated the effect of friction on the luminescence signal (Supplementary Note 4). When shear force is applied to the sensor pad by moving the spherical indenter, the micro-hemispheres are subjected to an asymmetric lateral deformation caused by friction between the micro-hemisphere array layer and the TIR prism below the sensor pad. This results in the observed non-axisymmetric luminescence. When we reduced the friction force by spreading silicone oil between the sensor pad and the prism, the applied shear force no longer changed the shape of the luminescence signal; an axisymmetric signal was obtained even during acceleration (Supplementary Fig. 29). We further analyzed the friction force using a surface forces apparatus (SFA) (Supplementary Fig. 30, and Supplementary Methods in detail). The kinetic friction coefficient (obtained from the slope of friction versus load curve) increased from 0.148 to 0.531 as the driving velocity increased from 11.6 to 58.0 µm/s (Supplementary Fig. 31). These results indicate that increasing applied velocity increases frictional forces (or friction coefficient) and micro-hemisphere deformation, which affects the overall non-axisymmetric shape of the luminescence signal. It is worth noting that the sliding velocity-dependent friction coefficients are commonly observed, especially in complex systems with soft polymers46 or polymer-like surfaces47, which contradicts conventional Amontons’ Laws of Friction48,49.
To further quantify this non-axisymmetric luminescence characteristic of shear force, the degree of non-axisymmetric luminescence was statistically described using the concept of skewness (γ), which is obtained by calculating the median and mean of the luminescence intensity profile (Supplementary Note 5). For a moving indenter, the calculated skewness matched well with the measured shear force (Fig. 3c), indicating that the magnitude of the shear force could be quantitatively estimated directly from the non-axisymmetric luminescence profile. When we analyzed the change in shear force at a given skewness for various normal forces, skewness decreased when the normal force increased from 0.5 to 1.5 N (Fig. 3d, and Supplementary Figs. 32–34). This is because an increase in normal force decreases the relative contribution of shear force (and thus, skewness value) to the luminescence signal, which is formed from both the normal force that creates the axisymmetric profile, and shear force that creates the non-axisymmetric profile (Supplementary Fig. 35). Based on the single image frame analysis of luminescence intensity profile for static normal force and the directional skewness for dynamic shear force, we successfully quantified in real time the direction and magnitude of both normal and shear forces from a single image using force discrimination machine learning algorithm (Fig. 3e, Supplementary Fig. 36, Supplementary Tables 1–3, Supplementary Movies 3, and 4, and Supplementary Notes 6, and 7). Our machine learning model was highly reliable with r2 values of 0.965 and 0.947 for normal and shear forces, respectively. In detail, the r, g, and b values of the detected images were analyzed in histogram format to predict the applied normal force. By utilizing the predicted normal force value, the threshold for displaying the contour of the normal force region was also predicted. As a result, normal and shear forces were successfully decoupled based on the determined threshold value. Note that the trend of skewness change was consistent irrespective of the fps, indicating that the more cost-effective 10-fps camera is suitable for the normal and shear force decoupling (Supplementary Figs. 37, and 38).
We further examined whether and how the undulated interface between the stiff TPU and soft PDMS in the stress concentration layer of the tactile sensor pad affects the non-axisymmetric luminescence (Supplementary Figs. 39, and 40). When an indenter moved at a velocity of 1.0 mm/s with 1.0 mm/s2 acceleration across a pad without undulation at the TPU-PDMS interface, the maximum value of skewness was 0.1 (Supplementary Fig. 41). With undulation, the maximum value of skewness increased to 0.37. When the velocity of the indenter was decreased to velocity of 0.5 mm/s with 0.5 mm/s2, the skewness decreased further to 0.32. Our results indicate that the undulation at the interface and the differences in modulus between TPU and PDMS improved the transmission of the applied shear force to the signal-generating UMs layer, which, in turn, increased the non-axisymmetric of the luminescence signal.
To further understand how the undulation at the TPU-PDMS interface enhances force transmission, we used finite element analysis (FEA) to examine the stress distribution of 2D model sensors with and without interface undulation under 1.0 N applied normal force (Fn) with and without shear force (Fs) ranging from 0 to 1.0 N (Fig. 3f, Supplementary Figs. 42–50, and Supplementary Note 8). Because luminescence signal in our tactile sensor is produced only when UMs contact the TIR prism, we expected the intensity of the luminescence signal in the model sensor to scale with the contact width between the individual UMs and the underlying substrate. Consistent with the experimental intensity profile, our FEA analysis showed that contact width is symmetric when only Fn is applied but is skewed toward the direction of applied Fs (Fig. 3g and Supplementary Figs. 46, and 47). The resulting calculated skewness, γ, increased with the magnitude of Fs and was substantially higher in the model sensor with an undulating interface (Fig. 3h). These calculations corroborate the experimental results that showed higher Fs transmission in sensors with an undulating interface (Supplementary Figs. 39, and 40). We visualized this enhancement by plotting the differences in stress distribution for sensors (σw) and without (σwo) an undulating interface, Δσ (Fig. 3i). Roughly 1 mm anterior to the center of the applied force – especially near the micro-hemispheres array – Δσyx increased strongly, indicating that Fs was more effectively transmitted to the micro-hemispherical array owing to the undulating interface. No such enhancement was observed for Δσyy. We also found that σyy remained symmetric even under Fs, whereas σxy became more asymmetric with increasing Fs (Supplementary Fig. 48). A more careful examination of σxy near the TPU-PDMS interface indicates that the regions beneath the interface have on average 50% higher value of σxy for the undulated interface (Supplementary Fig. 49). This phenomenon was observed to be independent of the indenter shape (Supplementary Fig. 50). Furthermore, similar non-axisymmetric luminescence patterns were identified for indenters with different shape (spherical, elliptical, rectangular, and triangular), spherical with different curvature, and cut cone with different radius geometries (Supplementary Figs. 51–56). Consequently, this behavior can be observed regardless of the indenter shape. Although the stress concentration layer may hinder the precise determination of the tip shape, we employed deep learning techniques to facilitate shape identification (Supplementary Fig. 57 and detailed in Supplementary Note 9). Additionally, asymmetric fluorescence emission and the resulting change in skewness when shear force is applied have been confirmed using various tools within the laboratory (Supplementary Fig. 58).
While the stress concentration layer exhibited an excellent characteristic of shear force, its resolution decreased at low normal forces (Supplementary Figs. 17, and 59). To accommodate various applications, we fabricated 200 µm thickness sensor pads without the stress concentration layer for precise vertical normal force measurements and surface inspection. Along with the high-resolution capability of detecting low pressures of 0.02 N (Supplementary Figs. 60–62), which an order of magnitude better than current optical tactile sensors that detect forces down to 0.2 N (Supplementary Table 4)15–22,41,42 our optical sensor enabled high spatio-temporal tactile recognition of 2D surface textures including multiple objects (Fig. 4a-c, and Supplementary Fig. 63) and fingerprints of different personnel (Fig. 4d, and Supplementary Fig. 64). The maximum spatial resolution of our tactile sensor is 100 µm, which is on the upper bounds of the spatial resolution found in previously studied optical tactile sensors (Supplementary Fig. 65, and Supplementary Table 4) 15–22,41,42. Our optical tactile sensor can also recognize and distinguish different surface textures. For example, small screws having different threads too fine for the human eye to see were identified in a high contrast manner using MATLAB-based automated image analysis code (Figs. 4e, and f, Supplementary Fig. 66, Supplementary Movie 5, and Supplementary Note 10)50. The size and shape of dimples (Fig. 4g) and signs of use (Fig. 4h) of different golf balls were also visualized with high spatial-temporal resolution.
We built a small-scale BTS transformation system that can instantly translate microscopic braille patterns into the corresponding sounds using a machine learning algorithm (Supplementary Note 11). Topographic patterns of micro-sized braille for “Hello World” were converted to luminescence images (Fig. 5a), and instantly translated into the corresponding sounds by extracting 2D patterns of dots (Figs. 5b, and c, Supplementary Figs. 67, and 68, and Supplementary Movie 6). The diameter of individual braille dots in this experiment was two times smaller (500 µm) than standard dots (1 mm), increasing the density of storable information by an order of magnitude (~ 32).
Most notably, using the ability to simultaneously visualize the direction of movement and the intensity change of both normal and shear force, we developed a machine learning (ML) framework that distinguishes dynamic biometric handwriting of individuals (Fig. 5d, Supplementary Movie 7, and detailed in Supplementary Notes 12–14). Decoupled images represented normal and shear forces, and related features extracted from videos of three different individuals writing the letter ‘e’ were fed into our ML-based linear discriminant analysis (LDA) for writing style classification. LDA clustering successfully discriminated the three different handwritten letters ‘e’ (Fig. 5e, Supplementary Figs. 69–73, Supplementary Tables 6–9, and Supplementary Note 15). We further extracted the averaged velocity-related features from multiple image frames and combined them with normal and shear force features for handwriting analysis. Our single-frame touch signal decoupling-based ML framework showed outstanding performance when compared to analyzing the handwriting using normal forces alone or normal forces with velocity-related features (Fig. 5f, Supplementary Figs. 74, and 75, Supplementary Table 8). While the average distance within a specific cluster was the same for all analysis, the average distance between different clusters was greater when the analysis included shear force features than when it included velocity features. Comparing the data cluster analyzed using both velocity and shear force features with those analyzed using velocity only and shear force only, we find that the analysis using combined features had a greater impact on the data clusters of velocity only analysis than on shear force only analysis. Further, between velocity only analysis and shear force only analysis, we find that shear force feature-based handwriting analysis forms a more clear-cut data cluster (Supplementary Fig. 72).
To compare clustering performance, we calculated the Silhouette coefficient, which describes the average distance between points within the same cluster and the average distance between points in the nearest cluster to which the data point does not belong to, and the Calinski-Harabasz index, which is the ratio of dispersion within a cluster to dispersion between clusters. Of all the analysis combinations, the normal/shear force combination showed a Silhouette coefficient closest to 1 (0.867) and a remarkable value Calinski-Harabasz index (5.76×102) (Fig. 5g, Supplementary Fig. 73, and Supplementary Table 9). Adding velocity feature to the normal/shear force feature slightly decreased the Silhouette coefficients (0.858) and Calinski-Harabasz index (4.52×102) and there is no significant advantage in terms of computational cost. From this point of view, we believe that besides normal force, shear force feature is a dominant factor for distinguishing writing style. These results demonstrate that normal/shear force-based handwriting analysis represents the most clear-cut cluster and is well classified with clusters of different force combinations. In conventional handwriting authentication, verification is predominantly grounded in the consistency of static handwriting features such as letter size, inclination, and inter-character spacing, rather than dynamic attributes. Remarkably, our study introduces an innovative methodology for handwriting verification, focused on shear force—a parameter not previously explored in this context. This approach encapsulates a wealth of data, encompassing factors such as the exerted pressure during writing and the speed of inscription. These parameters raise the expectation that our framework can offer valuable insights into the writer’s identity, as they exhibit variations correlated with the writer’s age and gender51. Furthermore, our framework possesses the unique capability to decouple normal and shear forces within a single frame, allowing it to provide real-time feedback unlike previous studies that evaluated handwriting only after all letters were completed. Thus, our system holds the potential to significantly enhance the precision of handwriting authentication process.
In conclusion, we present an upconversion luminescence-based behavioral biometric optical tactile sensor that simultaneously and quantitatively discriminates velocity, direction, normal and shear force in a single frame analysis. Such a unique axisymmetric and non-axisymmetric luminescence signal amplified through the human skin mimetic stress concentration microarchitecture was theoretically studied by FEA simulation, experimentally examined by SFA measurement, and quantitatively analyzed. Our optical sensor is simple to build, and we show they can be used for tactile recognition of surface textures and fingerprints. Using machine learning, we applied the sensor system in a small-scale BTS system. Most importantly, our system has enabled accurate dynamic biometric handwriting analysis. We believe our sensor system will open new avenues in facile optical tactile sensors that can intuitively and sensitively display multiple elements of dynamic forces in a single image frame, forming the basis of flexible behavioral biometric optical tactile sensors in wearable devices.