As a fundamental way of human communication, finger motion has been used to convey intuitive nonverbal information, including expression and emphasis, for centuries1,2. In modern society, various expressions through finger motion (e.g., sign language and gesturing) have been combined with advanced sensing technology to transcend environmental and spatial limits3–10. Remarkably, real-time detection and interpretation of finger motion using a human–machine interface (HMI) endowed with advanced computational intelligence for signal processing appear promising for universal communication, helping people with disabilities adjust to their daily life and actual workspace3,8. When combined with the Internet of Things3,4 and artificial intelligence5–7, finger motion recognition can largely expand virtual societies in platforms such as the Metaverse6,9,10, which is an upcoming industry growth engine.
To meaningfully connect human finger motion with machine operation, HMIs should be carefully designed based on the following specifications: 1) form-factor-free user-oriented soft, lightweight, and mechanically robust sensors and neuromorphic electronic devices compatible with finger motions, 2) imperceptible integrated systems that do not restrict finger motion in three-dimensional (3D) free space, and 3) time-series processing of irregular motion signals and algorithms for accurate recognition regardless of the user and environment11,12. Developing a wearable HMI for the fingers satisfying these design specifications can enhance nonverbal communication, especially in the reality–virtuality continuum4,7,10,13.
Conventional detection and recognition of finger motion in free space generally rely on bulky HMIs that include 3D depth cameras6,14, infrared cameras15, inertial measurement units16, and other devices anchored to specific positions, thus hindering detection owing to restricted fields of view17. Additionally, complex algorithms are required for distinguishing the finger from the background or tracing finger motion by calculating and comparing finger positions over a period15,17. Consequently, conventional methods are time and energy intensive when processing massive data and have limited applicability17.
To prevent time-consuming signal processing and achieve comfortability and portability in HMIs, diverse approaches based on the integration of advanced sensors and artificial intelligence have been proposed5,7,18–23. For instance, on-skin strain sensors integrated with artificial synapses18,19 and/or learning algorithms5,7,20–23 can recognize finger motion for tasks such as evaluating the motility of arms20,22,23, language translation from hand signs7,18, and gesture judgment5,21. Regarding skin-conformable design, many finger motion sensors have been demonstrated on auxiliary equipment, such as bulky gloves24 with rigid circuit modules13. Although such skin-like sensors overcome spatial limitations with high sensitivity25, time-dependent position information from starting to ending points is difficult to acquire, limiting nonverbal communication through time-resolved dynamic finger motion tracking. Additionally, most flexible synaptic devices are fabricated on thick (> 20 μm) plastic substrates with bending radii above 1 mm26,27, which are unsuitable for skin-attachable and wearable devices placed in joints and subjected to displacements28,29. To investigate and recognize finger motion in 3D free space, wearable and ultrathin synapse arrays integrated with form-factor-free soft and lightweight sensors should be devised, and their operating reliability under mechanical deformation should be secured. Moreover, multi-signal or time-series data processing and accurate learning should be achieved. Finger motion recognition faces challenges related to slight displacements and spatial confinement in 3D free space that undermine the universal accessibility of practical HMI applications11,12,17. Other challenges include the simultaneous sensing of coordinates and physical variations generated by finger motion over time and the lack of skin-like synaptic devices for learning and recognizing finger motion11,12,17. Integrated systems should be light and wearable and equipped with learning capabilities for finger motion recognition in 3D free space, different from conventional bulky devices with rigid form factors.
Here, we devised an ultrathin real-time finger motion recognition system for operation in 3D free space. The system is designed to track and recognize physical position changes by integrating an ultrathin titanium oxide (TiO2)-based artificial synapse (TOAS) array and organic proximity sensor (OPS) conformably attachable to the finger skin surface. On a 1.5 μm-thick ultra-flexible substrate, an 8 × 8 TOAS array demonstrated essential and reliable synaptic functions under severe mechanical deformation up to 60% of bending strain with repetitive programming pulses (~6,400 pulses), even when attached to a hand-like replica. The ultra-flexible OPS demonstrated operational stability with a high sensitivity of 90 mV/mm under green and red light and generated multioutput voltage signals over time for correctly tracking finger motion in 3D free space. Optical–electrical conversion signal transmission was demonstrated on an integrated OPS–TOAS array. The recognition of various finger-writing patterns of digits (0–9) following a Euler trail under various attaching conditions (i.e., finger, stick bar, fingers of different persons) was evaluated, achieving an average accuracy of 83.2 ± 10.5% (up to ~95 %), which remained unaffected under different strains (£ 60%) and repeated cycles (100 cycles).
Ultrathin TOAS array and synaptic characteristics
Fig. 1a shows a schematic representation of an ultrathin 8 × 8 TOAS array on a SU-8/parylene substrate peeled off from glass. The ultrathin SU-8/parylene substrate enabled the TOAS device to be freestanding, and it could be rolled and folded (Fig. 1b, Supplementary Videos S1 and S2). This allowed it to be transferred and attached conformably to any surface, including human skin29. The fabrication process is detailed in Methods. Fig. 1c shows the cross-sectional transmission electron microscopy (TEM) image of a fabricated TOAS device. The total device thickness was under approximately 55 nm, verifying conformability when combined with the ultrathin substrate. Fig. 1d shows the potential switching mechanism according to the oxygen vacancy (VO) migration degree in the TiO2 layer when applying a programming voltage30,31. Switching of various Al/TiO2/Al memristors has been extensively studied and is primarily governed by an interface-based mechanism, with the conductance largely depending on the VO concentration at the interface of the top Al/TiO2 layer. Specifically, when a negative bias is applied to the top Al electrode, VO can be generated by diffusion from the bulk TiO2 active layer and migrates toward the top Al electrode, abating the insulating nature of the top Al/TiO2 interfaces (left graph of Fig. 1d). VO migration induces a thinner barrier between the Al and TiO2 layers, increasing conductance (i.e., potentiation)30,31. By contrast, when a positive bias is applied to the top Al electrode, VO near the interface is redistributed to the bulk oxide layer and ruptures the conductive paths formed from the previously applied negative bias (right graph of Fig. 1d), thereby reducing conductance (i.e., depression)30,31. Hence, as VO migrates gradually in the layer, the conductance change can show analog switching by controlling the applied electrical pulses. When the voltage was double-swept from −3 to 3 V at the top Al electrode, the TOAS device exhibited a typical interface-type memristive I–V switching behavior without electroforming (Fig. 1e). It switched back to high conductance when reached a certain negative voltage, being consistent with a previously reported Al/TiO2/Al memristor26,31,32. The states of high and low conductance were defined as ION and IOFF at read voltage VR of −0.5 V, respectively. Even after the device was peeled off from a glass substrate, a nearly identical switching curve and ION and IOFF states were observed, indicating the viability of freestanding operation (right graph of Fig. 1e). Fig. 1f shows the histogram of ION and IOFF for 45 out of the 64 cells in the 8 × 8 TOAS array, demonstrating acceptable uniformity and device yield. The average ON and OFF states occurred at 4.24 ± 1.49 × 10−6 and 1.63 ± 1.11 × 10−7 A, respectively.
The TOAS device could implement essential functions of a biological synapse for learning and recognition in a parallel computing framework. A neuron is the basic functional unit in a biological nervous system, which is connected to other neurons via a small gap called a synapse for communication through electrical or chemical signaling (Fig. 1g). When an external stimulus activates ion transport in the nervous system, the corresponding signal propagates from the pre-neuron to the post-neuron. The connection between neurons can be strengthened (potentiated) or weakened (depressed) according to the condition (e.g., types of charged ions) of a series of spikes and can be maintained even after applying a programming pulse, resulting in long-term potentiation (LTP) and long-term depression (LTD)33,34. These synaptic behaviors can be similarly mimicked by the TOAS device. When negative voltage pulses for potentiation were applied to the device, the conductance, namely, post-synaptic current (PSC), gradually increased, exhibiting LTP. On the other hand, positive polarity in the voltage pulses caused depression, decreasing conductance and leading to LTD (Supplementary Fig. S1). In Figs. 1h and 1i, the PSCs for LTP/LTD were measured at VR of −0.5 V while applying different potentiation (VP) and depression (VD) pulses before peeling off. Fig. 1h shows the LTP/LTD function of the TOAS device according to different pulse widths (PW). Each VP and VD train consisted of 32 pulses of -2.4 and 2.0 V applied over intervals Δt of 0.5 s, respectively. Notably, a larger PW potentiated the PSC, leading to a large dynamic range (DR) defined as the ratio of PSC change (Fig. 1h). Similar to PW, larger VP and VD induced a greater change in the DR of the TOAS device (Fig. 1i). These results indicated the dynamic response of the synaptic connection according to the stimulus degree given by the input spike. Hence, the TOAS device resembled biological synapses that proportionally respond to stronger inputs. Fig. 1j shows the PSC over time for different Δt values of four VP pulses. Generally, Δt between input pulses can determine the degree of potentiation30,31. A short Δt of 0.5 s induced the most rapid PSC increase. This is attributed to the Δt-dependent migration and relaxation of VO in the TiO2 layer. Subsequent VP pulses for small Δt promoted further migration of VO toward the top Al layer before the post-synaptic responses (VO migration) from the previous VP pulse completely relaxed back to the bulk oxide layer. In contrast, a long Δt of 8 s induced a milder increase in PSC compared with the short Δt. This behavior is a main feature in spike-rate-dependent plasticity (SRDP) of biological synapses33,34. Like Δt, PW can be used to control factor to change the degree of potentiation. A large PW increases PSC while maintaining each PSC after applying a pulse. The PSC responses in Figs. 1j and 1k are well-preserved over time following the application of VP pulses, indicating that their updated weights in a neural network can be memorized even without excessive energy.
Electrical and mechanical stability of synaptic functions
After peeling off from supporting glass substrate, the ultrathin TOAS device can be placed conformably on the index finger and retained its original shape even under finger motion like the proximal inter-phalangeal (PIP) joint bending, as shown in Fig. 2a. Figs. 2b shows the LTP/LTD functions even when the TOAS array was placed on a hand-like replica (inset of Fig. 2b), which was preliminary used to prevent a possible electric shock. To evaluate the mechanical stability of the TOAS device, it was transferred from an ultrathin substrate to an elastomer (Supplementary Video S3). This setup imitated placing the TOAS device on malleable human skin, which has inherent tension and presents wrinkles as it moves. The elastomer could be stretched or compressed by turning the handle of a vise, which caused wrinkles when moving. The ultra-flexible TOAS device conformably attached to the surface of the elastomer and was compressed or released through elastomer deformation, as illustrated in Fig. 2d. Initially, the TOAS device was transferred to the fully stretched elastomer to exert no strain with initial distance Di between each edge of the elastomer (top scheme of Fig. 2d). When the handle of in-house screw machine was cautiously turned to compress the elastomer, strain was applied to the elastomer and TOAS device atop until reaching final distance Df (bottom scheme of Fig. 2d). Based on distances Di and Df, global strain S applied to the elastomer and TOAS device was defined as S = (Di − Df)/Di × 100%. Fig. 2e shows optical images of the magnified TOAS device under strain S from 0% to 60%, showing deeper wrinkles as S increased.
Stable cyclability under repeated input pulses and mechanical deformation should be achieved for the practical and robust application of the TOAS device on the finger skin surface. To validate the TOAS operation under flexion and stretching tests resembling skin attachment, the PSCs for LTP/LTD function were repeatedly evaluated while varying strain S. Fig. 2f shows the PSCs for LTP/LTD of the TOAS device under S ranging from 0% to 40% with increments of 10%. Stable synaptic operation of the TOAS device was obtained with slight deviations under different strains. The LTP/LTD responses over 100 cycles under 6,400 repeated input pulses at S of 0%–60% are shown in Supplementary Fig. S2. Considering that the human skin normally exhibits linear elastic response to tensile strain £ 15% and irreversible effects above 30%35,36, the TOAS device may stably and reliably operate under expected mechanical deformation of the skin during motion
To statistically examine the effect of strain S on the synaptic functions of the TOAS device, two representative features, namely, DR and nonlinearity (NL) were obtained according to S, as shown in Fig. 2g. Like DR, NL is a key factor in neuromorphic computing because it indicates the trends of weight update caused by programming input pulses37,38. The NL calculation is described in Supplementary Note S1. While strain S changed from 0% to 40%, both the DR and NL retained their initial values without considerable degradation. Hence, the TOAS device achieved mechanically stable neuromorphic computing with minimal loss of its learning capacity. To compare synaptic characteristics under severe mechanical deformation, the statistical LTP/LTD functions of the TOAS device for S of 0% (black) and 60% (red) were analysed in Fig. 2h (Supplementary Fig. S2). Compared with the results for S of 0% (black dot), the average PSCs for S of 60% (red dot) decreased slightly but mostly preserved the conductance. The maximum deviations of each PSC reduced from 28.63% to 16.04% when S ranged from 0% to 60%. The decreased overall DR under S of 60%, which was attributed to the mechanical deformation and repeated operations, possibly caused a lower deviation in PSC. This trend is also observed in Fig. 2i, which shows the DRs according to LTP/LTD cycles. The black and red dots represent the DR distributions over 100 cycles for S of 0% and 60%, respectively. The DR distribution for S of 60% was more condensed in lower values than that for S of 0%.
The mechanical stability of the TOAS device can also be understood from the TiO2 material standpoint26,39,40. When the TOAS device is sufficiently thinner than the substrate, local strain ε can be alleviated, and conformal attachment to wrinkled skin is facilitated. The local strain can be calculated as ε = s/2R, where s and R are the substrate thickness and bending radius, respectively. Hence, ε for the TOAS device can be estimated according to R, as shown in Fig. 2j. Under strain S of 60%, wrinkles with different R values appeared, as confirmed by the confocal image of the TOAS device (inset of Fig. 2i). The minimum R was ~8.77 μm, and thus the maximum ε was ~3.64%. Since ε was within the range of TiO2 strain durability26,40, robust operation of the TOAS device under S of 60% was expected. The detailed calculation of ε is described in Supplementary Fig. S3 and Note S2.
Skin-compatible light-responsive OPS
To digitalize real-time finger motion in 3D free space, an OPS that transduces motion information into electrical signals in the presence of light without requiring physical contact may be used for light-responsive artificial sensory neurons41. In addition, an OPS must meet the following requirements for application as a finger-attachable motion sensor: 1) ultra-flexible and lightweight design compatible with skin surface, 2) stable and sensitive electrical freestanding performance, 4) digitalization of finger motion in free space, and 4) applicability to integration with the TOAS device for operation compatibility. Fig. 3a shows a schematic of the ultrathin OPS junction. An inverted structure of indium tin oxide (ITO)/ZnO/active layer/MoO3/Ag was fabricated on a 1.5 μm-thick parylene substrate and passivated by the same thick parylene layer. The active layer was composed of poly[4,8-bis(5-(2-ethylhexyl)thiophen-2-yl)benzo[1,2-b;4,5-b']dithiophene-2,6-diyl-alt-(4-(2-ethylhexyl)-3-fluorothieno[3,4-b]thiophene-)-2-carboxylate-2-6-diyl)] (PTB7-Th) and [6,6]-phenyl C71 butyric acid methyl ester (PC71BM) (Supplementary Fig. S4), which provided efficient charge transfer, separation, and transport42. After peeling-off the fabricated device from the supporting glass, a 3 μm-thick OPS was obtained (Fig. 3b). Similar current density (J)–voltage (V) characteristics were observed in Fig. 3c before (black) and after (red) peeling-off regardless of light illumination (AM 1.5G, 100 mW/cm2).
To confirm skin-compatible motion recognition sensing in room light, the open-circuit voltage (VOC) and short-circuit current density (JSC) were measured before and after peeling-off under light intensity (IL) ranging from 0.5 to 100 mW/cm (Fig. 3d and Supplementary Fig. S5a). For a deeper insight into charge recombination kinetics, a slope from the logarithmical linearity of VOC and power law dependency of JSC according to IL can be evaluated. In general, the relation between JSC and VOC with light intensity can be described as JSC ∝ ILα and VOC ∝ bkBT/qln(IL), where α and b are the slopes for recombination ideality factor, kB is the Boltzmann constant, T is the temperature (in kelvins), and q is the elementary charge. If α and b were close to unity, the ideal device exhibited trap-free and weak bimolecular recombination43,44. For the ultrathin OPS, α and β were evaluated to be ~1.04 and ~1.28 under as-fabricated condition and ~1.00 and ~1.26 after peeling-off, respectively, indicating a low internal carrier loss owing to minimal bimolecular or trap-assisted charge recombination (see Supplementary Fig. S5b)45–48. These results suitably agreed with previously reported organic photodiodes using an active layer of PTB7-Th:PC71BM42,48. In addition, the invariant electrical characteristics with different light conditions before and after peeling-off showed that the OPSs were well positioned on the neutral plane to minimize the mechanical strain during delamination via proper passivation49,50.
Before demonstrating a skin-compatible and light-responsive motion recognition sensor, the electrical response should be evaluated for changes in distance DL from a light source. Fig. 3e shows a schematic diagram of the corresponding experimental setup. The output voltage (Vout) of the OPS attached to the finger was measured according to distance DL between the light source and OPS. Green (l = 530 nm) and red (l = 625 nm) light emitting diodes (LEDs) were used as light sources on the x- and y-axis, respectively, and IL varied from 6.3 to 0.22 and from 12.7 to 0.41 mW/cm2 for distance DL changing between 0 and 15 cm, respectively (Supplementary Fig. S6). The measured Vout was inversely proportional to the square of DL regardless of the light wavelength, as shown in the inset of Fig. 3f. Vout varied a few tens of millivolts under the given DL ³ 10 cm, indicating that the OPS was highly light-sensitive even under low IL below 1 mW/cm2 (Fig. 3f and Supplementary Fig. S6).
To demonstrate the motion sensor in 3D free space, 3 μm-thick ultrathin OPSs were attached to the top and side of the index finger, as shown in Fig. 3g. Owing to the high flexibility and deformability of the OPS, conformal skin adhesion was achieved by covering the complex finger skin surface without causing any chemical damage. After the LEDs were fixed to the x- (green) and y-axes (red), a participant drew a number by moving the OPS-attached finger in an area of 25 cm2 (graph paper of 5 cm × 5 cm) following a Eulerian trail. The OPSs attached to the finger slightly floated on the ground and moved in free space while a participant drew a number with the finger. When number 3 was drawn, in-situ time-resolved Vout showed variations in the order of few tens of millivolts along the x- (green) and y-axes (red), as shown in Fig. 3h. Fig. 3i shows a 3D plot of Vout differences (DVX and DVY) along the x- and y-axes over time from initial (0 s) to end (8 s) points according to the Eulerian trail for number 3 and its projection onto the xy plane. Using this method, we constructed a finger-written digit dataset under a light source in free space. Fig. 3j shows the projection plots of DVX and DVY for all finger-written digits from 0 to 9, which exhibited independent and distinctive feature patterns (Supplementary Fig. S7). Hence, the ultra-flexible OPSs attached to the finger could be used for skin-compatible finger motion sensing in free space.
To validate the applicability of an integrated OPS–TOAS array, we constructed a 2 × 1 crossbar array using two integrated OPS–TOAS devices that summed the PSCs along a single row generated by each output signal, as shown in Fig. 3k. The OPS generated Vout depending on IL, and the TOAS device produced PSC depending on the applied input voltage. Therefore, in the OPS–TOAS array, the PSC level could be modulated according to input IL to the OPS and resultant conductance of the individual TOAS devices for a compatible applied voltage. The switching characteristics of each TOAS device and corresponding PSC levels are shown in Supplementary Fig. S8. Fig. 3l shows the PSC sum in the 2 × 1 OPS–TOAS array for three states of the two TOAS devices (i.e., LTD/LTD, LTD/LTP, and LTP/LTP) and different input IL (0, 0.9, and 6.4 mW/cm2) to the first and second OPS devices (IL,1st and IL,2nd). Notably, the PSC of the TOAS device further increased (directed to red color) when stronger IL was delivered through light to the OPSs, thereby increasing the PSC sum. The results demonstrated the operating compatibility between the OPS and TOAS and controlled optical–electrical signal conversion of the integrated artificial sensory neuron–synapse array (i.e., OPS–TOAS array).
Learning capability of ultrathin finger-writing motion recognition system
As a proof of concept, to evaluate the learning capability of a neuromorphic system based on the developed OPS–TOAS array and its applicability to finger-writing motion recognition in 3D free space, we performed the recognition of finger-writing digit patterns for various configurations (i.e., finger, stick bar, fingers of different persons). Fig. 4a illustrates the dataflow in the integrated neuromorphic system attached to a finger under exposure to light while finger-writing digit 3. This process has three steps: 1) finger-writing digit pattern (e.g., digit 3) in free space while wearing skin-compatible integrated OPS–TOAS device under light, 2) OPS light detection and Vout digitization (Vn in input neurons for n = 1, 2, …, 256) in real time (i.e., light–electrical signal conversion), and 3) reshaped two-dimensional image and synaptic weight (w) updates in the TOAS device for learning and recognition.
Once the time-resolved OPS signals for digits 0–9 were individually trained as reshaped digit images using a 16 × 16 × 10 (256 input neurons × 10 output neurons) TOAS array, subsequent random digit images written by the OPS-attached finger could be evaluated and recognized based on the obtained w values. Consider 16 × 16 input signals for digit 3 as an example. DVout (DVX and DVY) of the OPS signal along the x- and y-axes were obtained over 8 s with sampling of 1 ms (i.e., 8,000 datapoints), and the transformed inputs were reshaped into two-dimensional digit images, as shown in Fig. 4b (see Fig. 3j and Supplementary Note S3). The 8,000 datapoints were accumulated sequentially and then reshaped as a specific digit pattern that reflected the real-time motion of the OPS-attached finger. In this way, all finger-writing motions for digit patterns from 0 to 9 could be reproduced as 16 × 16 matrix images with corresponding weight w using conventional batch-mode backpropagation (Fig. 4c and Supplementary Note S4). After 10 training epochs, the updated w values (16 × 16 × 10) in the TOAS array ranged between –0.275 and 0.293 µS. Note that w was determined by differences in the conductance values (G+ − G−) between two neighboring TOAS devices in the array considering only LTP. The combination of positive (upper right graph in Fig. 4c) and negative (lower right graph in Fig. 4c) potentiation values determined w for all the digits. Additionally, a noise signal was considered to optimize the calculated Dw during backpropagation learning to expand the range of w to recognize diverse digit pattern motions between users representing the same digit (Supplementary Note S4 and Fig. S9–S13)51–55.
Figs. 4d and 4e show examples of finger-writing motion for digit 3 in 3D free space and the recognition accuracy for the digit patterns according to strain S from 0% to 60% and repeated 100 cycles (for only S = 0% and 60%) for the TOAS array, respectively. The training set was prepared from five reshaped image sets (Supplementary Fig. S14) acquired from the OPS-attached finger of one person. The insets in Fig. 4e show the 5 × 5 confusion matrices according to S and repeated cycles, with the row and column of each inset representing the training and test sets, respectively. For example, the i-th row and j-th column of the 5 × 5 matrix indicated the accuracy on the j-th image set when training the i-th image set. At S of 0%, five diagonal elements showed perfect accuracy (100%), and 20 off-diagonal elements showed prominent performance (84.3 ± 5.26%). The average accuracy across the 25 cells in the matrix at S of 0% was 87.4 ± 7.85%, which remained unaffected by S varying from 0% to 60% and up to 100 cycles (Supplementary Fig. S15 and S16).
Fig. 4f shows examples of finger (left graph) and stick-bar (right graph) writing for digit 3 in 3D free space with distinctive writing patterns. The training set was prepared from 10 reshaped image sets measured by finger-writing (five sets) and stick-bar-writing (five sets, Supplementary Fig. S17) experiments, where the writing modes were alternated for the training (Supplementary Note S4 and Fig. S18–S20). As shown in Fig. 4g, the average accuracy for the 50 cells in the 5 × 10 confusion matrix at S = 0% was 81.3 ± 10.1% and maintained for different configurations (i.e., finger and stick bar), strains, and numbers of cycles. Figs. 4h and 4i show results for two persons with six training sets (three image sets per person, Supplementary Fig. S21). The average accuracy for the 18 cells in the 3 × 6 matrix at S of 0% was 82.5 ± 13.0% and maintained for different persons, strains, and numbers of cycles. Note that the maximum accuracy was ~95 % of the 1st row and 5th column of the 3 × 6 matrix at S of 20%. The recognition accuracies for all the cases are presented in Supplementary Figs. S22–S24.