Efficient operation of intelligent machines in the real world requires methods that allow them to understand and predict the uncertainties presented by the unstructured environments with good accuracy, scalability and generalization, similar to humans. Current methods rely on pretrained networks instead of continuously learning from the dynamic signal properties of working environments and suffer inherent limitations, such as data-hungry procedures, and limited generalization capabilities. Herein, we present a memristor-based differential neuromorphic computing, perceptual signal processing and learning method for intelligent machines. The main features of environmental information such as amplification (> 720%) and adaptation (< 50%) of mechanical stimuli encoded in memristors, are extracted to obtain human-like processing in unstructured environments. The developed method takes advantage of the intrinsic multi-state property of memristors and exhibits good scalability and generalization, as confirmed by validation in two different application scenarios: object grasping and autonomous driving. In the former, a robot hand experimentally realizes safe and stable grasping through fast learning (in ~ 1 ms) the unknown object features (e.g., sharp corner and smooth surface) with a single memristor. In the latter, the decision-making information of 10 unstructured environments in autonomous driving (e.g., overtaking cars, pedestrians) is accurately (94%) extracted with a 40×25 memristor array. By mimicking the intrinsic nature of human low-level perception mechanisms, the electronic memristive neuromorphic circuit-based method, presented here shows the potential for adapting to diverse sensing technologies and helping intelligent machines generate smart high-level decisions in the real world.
Article
Intelligent machines work in unstructured environments by differential neuromorphic computing
https://doi.org/10.21203/rs.3.rs-3644668/v1
This work is licensed under a CC BY 4.0 License
Journal Publication
published 31 May, 2024
Version 1
posted
You are reading this latest preprint version
Understanding and learning sensory data efficiently to achieve human-like perception of the real world is pivotal for next-generation intelligent machines (1–3). With such capabilities, intelligent machines could truly transit from controlled environments such as factories and laboratories into unstructured environments of home and businesses that offer considerable ‘variations’ (Supplementary Discussion 1 and 2). Traditionally, the adeptness of organisms within unstructured environments has been attributed to the acquisition of diverse types of physical information. However, recent advancements in life sciences have revealed that the most crucial mechanism employed by humans for understanding unstructured environments is sensory information differentiation and learning mechanisms (4–8). For a given stimulus, multiple types of receptors and corresponding neurons participate in differential processing, and they adjust their structure and synaptic weight on the features of external stimuli (Supplementary Discussion 3). Specifically, by extracting the main stimulus features embedded in the signal properties, these receptors and neurons can rapidly form a complex set of intricate network-based perception functions, such as integration and firing, potentiation and depotentiation, associative memory, environmental mapping, and motion control tasks. In contrast, conventional intelligent machines operate within the von Neumann separated storage and computation architecture, which makes it inefficient to differentially process highly dynamic unstructured information (9–13). This can be resolved with memristors, as they integrate storage and computation capabilities, coupled with synapse-like characteristics, and thus bear a striking resemblance to the biological synapse at a foundational level (14–19). This similarity opens the possibility for the realization of human-like perception functionalities in intelligent machines. To this end, herein we present a memristor-based differential neuromorphic computing, perceptual signal processing and learning method for intelligent machines.
Unstructured information encompasses multidimensional features, which undergo the perceptual processing of diverse receptors in biology (2, 20, 21). This type of processing indicates the need for designing different modulation methods for neuromorphic computing. However, existing methods focus on keeping a memristor to a fixed receptor (Fig. 1a), e.g. a nociceptor (22–25), thus restricting their full potential. A potential solution is to operate multiple memristors at different desired states. In fact, the adjustable characteristics of memristors allow them to switch their states, which could be used to focus on multiple receptors. Furthermore, distinct features of sensory data exhibit time domain independence, implying that a memristor can adaptively switch to an appropriate state to process each feature at a time slot. Therefore, drawing inspiration from the sensory information processing model and taking advantage of memristors’ intrinsic multistate property, the proposed differential information processing and learning method involves extracting features from unstructured data and applying memristive modulation schemes (Supplementary Discussion 4 to 6). This approach experimentally facilitates the processing and learning of unstructured data within two complex environments i.e., grasping of objects and autonomous driving. In the former, we emulate nociceptors to achieve amplification (> 720%) for hazardous stimuli and adapting receptors to achieve adaptation (< 50%) for mild stimuli, which play a key role in grasping unknown objects. In the latter, we differentiate visual information by a 40×25 memristor matrix and reach a commendable accuracy of 94% in extracting critical information within 10 autonomous driving scenes, after comparison with human labelled results.
Specific Components of Differentiation Processing and Environmental Learning
Sensory, signal encoding and neuromorphic operation modules construct the differential neuromorphic computing method (Fig. 1b). The first can be of the desired sensor type according to mimicked biological sensory. It operates by converting a physical stimulus into electrical information and can be described as:
where s(t) denotes the physical stimulus, R is the response function of the sensor, and p(t) is the outputted electrical signal for each i th channel/memristor.
The signal encoding module extracts features from p(t) and then creates the associated memristive encoding schemes to process sensor information in different manners. The created encoding scheme is applied to the memristor in the neuromorphic operation module, and the changed resistive values indicate the properties of the suffered stimuli. Through the three steps above, different features are properly processed by the predesigned differential modulation methods for memristors, yielding a multifeature differentiation-based comprehensive understanding of environmental knowledge. The entire process corresponds to organisms’ differential information perception capability in stimuli reception, transduction and processing (26), as expressed below:
As explained, the proposed method provides a human-like information processing pipeline, which extracts key features of undergoing stimuli in real-time, opening the possibility for intelligent machines to learn the main attributes of unstructured tasks automatically, as demonstrated in the following tactile and visual information processing experiments.
Tactile Perception
Realizing safe and stable manipulation of unknown objects by robotic hands is tricky but highly needed (27–31). Unlike lab and factory settings, an unknown object may exhibit sharp or slippery attributes. The former potentially damages the contacting end, and the latter places a heavy burden on the sensing and computing modules when keeping the object balanced between stabilization and deformation. To address this issue, the proposed memristor-based differential neuromorphic computing method is embedded into the sensing and controlling strategy of a robotic hand, as depicted in Fig. 2a.
Here, a piezoresistive force sensing architecture (Figure S1) is assembled for receiving pressure amplitudes, and a self-directed channel effect-based multilayered nonvolatile memristor (KNOWM Inc.) is selected for differential neuromorphic computing (Figure S2). The structure, hysteresis curves, and electrical characteristics under pulse testing of the memristor are given in Figs. 2b to d, respectively. The piezoresistive layers and the memristor offer short-term and long-term force information, respectively, based on which the attributes of an object are first extracted by a status acquisition block and then utilized to generate differential modulation schemes for the memristor. The extraction and modulation scheme generation are both conducted in a field programmable gate array (FPGA) platform, as shown in Fig. 2a. Based on the aims of modulation methods, the schemes can be further classified into 3 groups, allowing the memristor to stay at high (> 250k), middle (~ 170k) and low (< 100k) resistance levels. This arrangement facilitates the adaptive, normalization, and nociception perception of external stimuli, aligning with 3 types of biological perception behaviors, i.e., nociception, adaptation, and recovery (Figure S7, S8 and Supplementary Discussion 7). The detailed mapping relationship between attributes and modulation methods is given in the Supplementary Table 1.
The three mimicked biological functionalities are experimentally validated, by studying the change in force induced electrical signals with and without neuromorphic computing. Here, the neuromorphic computing module tunes the adjustment factor of a voltage amplifier block, whose input is the status of the piezoresistive film. After the attribute of a force is extracted, an associated modulation scheme is selected and then applied to the memristor. The resistance of the memristor is then changed and used as an indicator of the amplification factor. The final yielded voltage outputs illustrate the biological perception behaviors. The experimental results in Fig. 2f and h demonstrate the responses of neuromorphic computing to hazardous and mild stimuli. More than 170% amplification of the hazard signal and more than 50% attenuation of the mild signal are obtained. The curve trends are similar to the biological response strengths in Fig. 2g.
When considering the unstructured information processing in the real world, it often becomes necessary to perform further differentiation on the aforementioned functions. For better adaptation to dynamic changes in hazardous scenarios, we mimic the time window processing mechanism in organisms, thereby distinguishing between sudden threats and persistent threats (24, 32). Specifically, during the encoding process, we additionally consider the state of the memristor. When its state falls below a certain threshold (100 kΩ in this case), we increase the amplitude and pulse duty cycle of the positive pulses (as shown in Fig. 2i), achieving the functionality of sensitized processing for external stimulus information. In our work, a > 720% amplification of tactile stimuli is offered (Fig. 2g), exceeding the state-of-the-art works (22, 25, 33). Regarding the adaptation function, the adaptive speed (the magnitude of the attenuation of response strength over time) is of significance (34–36). Here, we implement two adaptation speeds by adjusting the pulse duty cycle and amplitude of encoding pulses as shown in Figs. 2k and 2l. The developed method is then used in two practical unknown object grasping tasks, to extract and learn the main characteristics of objects for safe and stable manipulation (conceptually depicted in Fig. 3a).
In task 1, an irregular object consisting of a cube and a cone is 3D printed. During the grasping process, the robot might encounter the sharp point that leads to painful pressures and poses a potential danger. It needs the help of a nociceptor to quickly learn the hazard stimulus caused by the sharp point and then changes the grasping strategy, as conceptually depicted in Fig. 3b, in which the resistance information of the piezoresistive film and the memristor during the entire grasping task is shown. At 2.2 s, the piezoresistive sensor attached to the robotic hand experiences a significant force due to contact with the sharp point, resulting in decreased resistance. In this case, the pressure conforms to hazardous characteristics, and the memristor is modulated into the amplification state (low resistance). At 8.2 s, the memristor resistance falls below the threshold of 35 kΩ, and the pressure stimulus is amplified by 500%. This indicates that the robot has been in contact with the sharp point, triggering the pain reflex. Subsequently, the robot learns to change the grasping posture at 14.4 s. By 17.4 s, the pressure perceived by the contact point stabilizes, and the memristor is modulated into an adaptation state (high-resistance state), achieving safe and stable grasping.
Similarly, in task 2, a soap is used to represent slippery objects. The instability of tactile information during the slipping process leads to spikes in reading the resistance of the memristor. The spike signals due to slip events prompt the robot to increase the gripping force to ensure stability. As illustrated in Fig. 3c, during the grasp attempt of the soap, the gripping force stabilizes at 3.8 s, with the memristor switching to a high-resistance state. By 18 s, the adaptation level to external tactile information reaches 75%. At 18.5 s, an external inference causes the object to slip, leading to changes in gripping force and resulting in a spike in the memristor resistance. Upon detecting a slip event onset, the robot prompts an increase of the gripping force to prevent the object from slipping further. This timely increase in contact force prevents the object from falling, and the memristor is modulated into a normal perception state (stable middle resistance state), successfully achieving stable grasping of a slippery object. To our knowledge, this represents the first time that a memristor-based approach is successfully employed for local slip detection and consequent adjustment of the actuator force when grasping unknown objects.
At present, the processing and execution time of the proposed method is 1 ms, which can strongly support safe and stable operations for robots (37–40). The operation time can be further improved when better memristive devices that respond in nanoseconds (ns) or microseconds (µs) are employed (41–44).
Visual Perception
In previous tactile information processing, attributes based on force strength were used to learn environmental knowledge. In contrast, visual frequency-based attributes are more important for autonomous driving, as they imply the relative position change of surrounding objects to the car (45–47) and are vital for real-time decision making (48–50). For instance, the sudden appearance of pedestrians or vehicles can lead to life-threatening collisions. Thus, shortening the attribute extraction time of moving objects is important. In addition, the moving direction of the object can further help to make good decisions (51, 52). To this end, the proposed differential neuromorphic method is employed to learn visual frequency-based attributes, forming functions mimicking biological cone cells for retrieving fast information, and neural excitation to maintain fast information for a while (Supplementary Discussion 8).
In the implementation, a driving recorder of 1920×900 resolution and a 25×40 memristor array are used. The gray CMOS image from the recorder is compressed, by averaging a matrix of m×n into one pixel, based on image spatial redundancy theory (53). Next, filtering circuits are used to extract fast and slow pixel value changes within two adjacent frames, obtaining fast and slow visual information attributes, which are then encoded into electrical pulses to modulate the memristor into low- and high-resistance states. Here, slow visual information is taken to release the memristor from a low-resistance state to a high-resistance state. As the change in resistance is analogue, its value within this period reflects the moving orientation of the object, achieving a neural excitation effect.
In the experiments, the driving operations are divided into designed driving and free driving scenarios. In the former, slow driving takes place on a closed road segment, and a pedestrian runs across the road from different sides (left and right), distances (near, medium and far) and speeds (walking and running), aiming to examine the detection ability of the method in 3 widely occurring danger scenarios for autonomous driving as shown in Fig. 4b: 1. Pedestrian running across the road (moment 1); 2. Nearby pedestrian walks across the road (moment 2); 3. The walking pedestrian (moment 3) suddenly runs to another orientation (moment 4). Figure 4c displays 3 representative scenarios during a pedestrian’s moving path given in Fig. 4b, in which the yellow boxes are the m by n pixel area that is first compressed to a single pixel and then processed by the same memristor. The light intensity changes of the pixel are shown in Fig. 4d. Before moment 1, no pedestrian enters, and the slight light intensity change is solely due to the vehicle movements. Upon reaching moment 1, the pedestrian runs into the area from a medium distance, resulting in strong light intensity change, triggering the extraction and encoding module to generate a positive modulation voltage pulse, thus modulating the corresponding memristor into a low-resistance state. After the pedestrian leaves the area, the corresponding memristors enter the neural excitation status and finally return to the high resistance status because the scene information changes slowly. At moment 2, the pedestrian walks into the yellow box area, giving rise to a strong light intensity change again. Note that although the pedestrian is in a slow-motion mode, the distance between the vehicle and pedestrian is short, hence showing the same effect as a pedestrian running from far. At moment 3, the pedestrian enters the detection area from far, and the slow movement characteristics generate a negative voltage pulse that maintains the memristor in a high-resistance state. At moment 4, the sudden movement of the pedestrian strongly changes the light intensity, hence reducing the resistance value of the memristor. However, due to the distance of the pedestrian and the fact that he has already entered the area before the sudden run, the amplitude of the positive pulse generated by the high-frequency feature is smaller compared to the previous more dangerous moments (moments 1 and 2). For this reason, the resistance value of the memristor does not reach to its lowest resistance state. Overall, the 3 danger scenarios are successfully detected.
When the yellow box expands to the whole image, global attributes are gathered. Figures 4e to 4g are 3 examples taken from daytime and evening, together with their corresponding differential neuromorphic computing results. The fast-running pedestrian is accurately captured, and afterimages are generated, implying the orientation.
In free-driving experiments, we collected over 100 hours of video data in various lighting and weather conditions, as conceptually shown in the middle part of Fig. 5. Ten representative scenarios containing important decision-making associated information for autonomous driving, such as taillight, nearby cars and lane lines, are given as examples in Fig. 5. From the differential neurocomputing processed results, it is clear that the information is successfully retrieved. We then further invite 10 senior drivers to locate dangerous objects they believe are important to safe driving in 10 short videos containing the 10 images given in Fig. 5. Their results are offered as the yellow regions in the processed results, and we can learn that human decisions overlap with the proposed method. For all 10 videos, the overlap rate is more than 94%. The difference arises from the process of important information labeling. The judgment criterion of manual labeling, based on senior drivers’ driving/riding experience, tends to prioritize targets (such as pedestrians and cars) that are likely to cause traffic accidents. In contrast, the criterion of differential processing is based on objective motion situation, which is of greater significance for timely decision-making, assigning relatively stationary or slow-moving objects low priority on current driving. Therefore, when both empirical and objective moving targets are present, a person tends to label empirical targets. However, at times, such as when driving with a vehicle traveling at a relatively close speed nearby, the road marking line is more important and can reflect the yawing condition of driving (Fig. 5b). In most cases the empirical and objective moving targets are the same object, but there are exceptions such as in Fig. 5h, leading to a 6% bias in the detection results. The detailed performance can be found in Figure S15 to S21.
Compared to traditional object detection methods in autonomous driving, the developed differential neurocomputing framework does not rely on prior trained dataset, illustrating its great power in processing unstructured information and hence prompt the implementation of real-time decision-making for autonomous driving.
Memristors, with their ability to efficiently mimic the computation units of biology, have been used to construct human-like environmental perception systems (54, 55). However, current memristive computing methods overlook the characteristic attributes of biological information processing (20, 22, 56, 57), which could lead to information loss in unstructured environments. Drawing inspiration from biology, we demonstrated the memristor based processing of environmental information by exploiting their diverse characteristics and using multiple receptors. Subsequently, we demonstrate the effectiveness and versatility of this method by showcasing its application in both tactile and visual perception. The presented results represent a significant advancement toward achieving fully autonomous and intelligent robot systems capable of effectively operating in real-world scenarios. Due to the small size of memristive devices, the possibility of having high-density memristors over large areas and flexible substrates, and their similarity with the fundamental biological sensory processing mechanisms, the presented method could enable intelligent machines to possess sensory capabilities on a scale comparable to humans when combined with diverse sensors (Supplementary Discussion 9), allowing them to genuinely learn and efficiently comprehend the environment (as depicted in Figure S22 and S23).
Electrical Measurements
Electrical measurements were performed using the RIGOL DG4062 function/arbitrary waveform generator and MSO1074 oscilloscope. During the memristor U-I electrical test in Fig. 2c, the memristor is connected in series with a 10 kΩ resistor. The function waveform generator was used to generate a 10 Hz and 500 mV sine wave voltage across the memristor and the fixed value resistor. One channel of the oscilloscope was used to measure the voltage of the memristor, while another channel was used to measure the voltage across the resistor to calculate the current passing through the memristor. The triggering mode is set to the normal mode. During the pulse test Fig. 2d and 2e, the connection between the signal generator, the oscilloscope, the resistor, and the memristor remains unchanged. Subsequently, voltage pulses are generated, and the oscilloscope is responsible for capturing the voltage across the memristor and resistor respectively under single mode.
Tactile Processing System
In the development of our tactile system, we employ the FPGA platform to achieve the following functions: collection of pressure data, generation of modulation schemes, and data uploading. In every control cycle, the system initially detects the current pressure information using the pressure sensor. Subsequently, the FPGA generates an appropriate modulation scheme based on the obtained pressure data and the current state of the memristor. Following this, a digital-to-analog converter applies the modulation voltage to the memristor. Finally, the system uploads the data of the memristor state and pressure information. The detailed descriptions and schematics are provided in Figure S4 and S5.
The Control Circuit of the Memristor
The control circuit of the memristor is designed around an operational amplifier, which is pivotal in achieving precise modulation of the memristor state. Utilizing this design, the memristor state information can be observed based on the output voltage of this circuit. Detailed schematics and tests of this circuit are documented in Figure S3 and S6.
Robot Experiment
In robot experiments, the FPGA platform is utilized to generate the control commands. Upon activation of the control algorithm (detailed in Figure S11), the FPGA sends a command to the DH gripper, triggering the appropriate reflex response timely. These commands are communicated via the Modbus protocol, using the RS485 interface standard. Each command includes a slave number, function code, operation register, operand, and a CRC check code for verification. Once the DH gripper receives a command, it adjusts its gripping force and position to execute the reflex action.
Visual Processing System
To implement large-scale memristor-based visual differential computing, we employ the SPICE simulation platform. The voltage threshold adaptive memristor (VTEAM) model, selected for its versatility, serves as the fundamental computation unit. Detailed information about the electrical characteristics of this model is available in Figure S12. In visual information processing, images captured by a CMOS sensor in real driving scenarios are converted into analog voltage inputs for the visual differential processing system. The visual process system consists of four main components: filters, analog computing circuit, control switch, and memristor-based computing circuit. Further details are provided in Figure S13.
- J. Cui, J. Trinkle, Toward next-generation learned robot manipulation. Sci. Robot. 6, eabd9461 (2021).
- F. Liu, S. Deswal, A. Christou, Y. Sandamirskaya, M. Kaboli, R. Dahiya, Neuro-Inspired electronic skin for robots. Sci. Robot. 7, eabl7344 (2022).
- M. Davies, A. Wild, G. Orchard, Y. Sandamirskaya, G. A. F. Guerra, P. Joshi, P. Plank, S. R. Risbud, Advancing neuromorphic computing with loihi: a survey of results and outlook. Proc. IEEE 109, 911–934 (2021).
- A. Handler, D. D. Ginty, The mechanosensory neurons of touch and their mechanisms of activation. Nat. Rev. Neurosci. 22, 521–537 (2021).
- J. del Mármol, M. A. Yedlin, V. Ruta, The structural basis of odorant recognition in insect olfactory receptors. Nature 597, 126–131 (2021).
- J. A. Butterwick, J. Del Mármol, K. H. Kim, M. A. Kahlson, J. A. Rogow, T. Walz, V. Ruta, Cryo-EM structure of the insect olfactory receptor Orco. Nature 560, 447–452 (2018).
- N. L. Neubarth, A. J. Emanuel, Y. Liu, M. W. Springel, A. Handler, Q. Zhang, B. P. Lehnert, C. Guo, L. L. Orefice, A. Abdelaziz, M. M. DeLisle, M. Iskols, J. Rhyins, S. J. Kim, S. J. Cattel, W. Regehr, C. D. Harvey, J. Drugowitsch, D. D. Ginty, Meissner corpuscles and their spatially intermingled afferents underlie gentle touch perception. Science 368, eabb2751 (2020).
- L. L. Tan, R. Kuner, Neocortical circuits in pain and pain relief. Nat. Rev. Neurosci. 22, 458–471 (2021).
- W. Jiang, B. Xie, C.-C. Liu, Y. Shi, Integrating memristors and CMOS for better AI. Nat. Electron. 2, 376–377 (2019).
- Z. Liu, J. Tang, B. Gao, P. Yao, X. Li, D. Liu, Y. Zhou, H. Qian, B. Hong, H. Wu, Neural signal analysis with memristor arrays towards high-efficiency brain–machine interfaces. Nat. Commun. 11, 4234 (2020).
- Y. Song, X. Wang, Q. Wu, F. Yang, C. Wang, M. Wang, X. Miao, Reconfigurable and efficient implementation of 16 boolean logics and full-adder functions with memristor crossbar for beyond von neumann in-memory computing. Adv. Sci. 9, 2200036 (2022).
- J. Li, S. Ren, Y. Li, L. Yang, Y. Yu, R. Ni, H. Zhou, H. Bao, Y. He, J. Chen, H. Jia, X. Miao, Sparse matrix multiplication in a record-low power self-rectifying memristor array for scientific computing. Sci. Adv. 9, eadf7474 (2023).
- F. Liu, A. S. Dahiya, R. Dahiya, A flexible chip with embedded intelligence. Nat. Electron. 3, 358–359 (2020).
- S. Kumar, X. Wang, J. P. Strachan, Y. Yang, W. D. Lu, Dynamical memristors for higher-complexity neuromorphic computing. Nat. Rev. Mater. 7, 575–591 (2022).
- S. Wang, L. Song, W. Chen, G. Wang, E. Hao, C. Li, Y. Hu, Y. Pan, A. Nathan, G. Hu, S. Gao, Memristor-based intelligent human-like neural computing. Adv. Electron. Mater. 9, 2200877 (2023).
- K. Sun, J. Chen, X. Yan, The future of memristors: materials engineering and neural networks. Adv. Funct. Mater. 31, 2006773 (2021).
- M. A. Zidan, J. P. Strachan, W. D. Lu, The future of electronics based on memristive systems. Nat. Electron. 1, 22–29 (2018).
- W. Chen, L. Song, S. Wang, Z. Zhang, G. Wang, G. Hu, S. Gao, Essential characteristics of memristors for neuromorphic computing. Adv. Electron. Mater. 9, 2200833 (2023).
- R. Dahiya, N. Yogeswaran, F. Liu, L. Manjakkal, E. Burdet, V. Hayward, H. Jörntell, Large-area soft e-skin: the challenges beyond sensor designs. Proc. IEEE 107, 2016–2033 (2019).
- S. Chun, J.-S. Kim, Y. Yoo, Y. Choi, S. J. Jung, D. Jang, G. Lee, K.-I. Song, K. S. Nam, I. Youn, D. Son, C. Pang, Y. Jeong, H. Jung, Y.-J. Kim, B.-D. Choi, J. Kim, S.-P. Kim, W. Park, S. Park, An artificial neural tactile sensing system. Nat. Electron. 4, 429–438 (2021).
- I. You, D. G. Mackanic, N. Matsuhisa, J. Kang, J. Kwon, L. Beker, J. Mun, W. Suh, T. Y. Kim, J. B.-H. Tok, Z. Bao, U. Jeong, Artificial multimodal receptors based on ion relaxation dynamics. Science 370, 961–965 (2020).
- J. H. Yoon, Z. Wang, K. M. Kim, H. Wu, V. Ravichandran, Q. Xia, C. S. Hwang, J. J. Yang, An artificial nociceptor based on a diffusive memristor. Nat. Commun. 9, 417 (2018).
- X. Zhang, Y. Zhuo, Q. Luo, Z. Wu, R. Midya, Z. Wang, W. Song, R. Wang, N. K. Upadhyay, Y. Fang, F. Kiani, M. Rao, Y. Yang, Q. Xia, Q. Liu, M. Liu, J. J. Yang, An artificial spiking afferent nerve based on Mott memristors for neurorobotics. Nat. Commun. 11, 51 (2020).
- D. Julius, A. I. Basbaum, Molecular mechanisms of nociception. Nature 413, 203–210 (2001).
- Y. Kim, Y. J. Kwon, D. E. Kwon, K. J. Yoon, J. H. Yoon, S. Yoo, H. J. Kim, T. H. Park, J.-W. Han, K. M. Kim, C. S. Hwang, Nociceptive memristor. Adv. Mater. 30, 1704320 (2018).
- B. Wark, B. N. Lundstrom, A. Fairhall, Sensory adaptation. Curr. Opin. Neurobiol. 17, 423–429 (2007).
- A. Billard, D. Kragic, Trends and challenges in robot manipulation. Science 364, eaat8414 (2019).
- S. Gao, Y. Dai, A. Nathan, Tactile and vision perception for intelligent humanoids. Adv. Intell. Syst. 4, 2100074 (2022).
- R. S. Dahiya, G. Metta, M. Valle, G. Sandini, Tactile sensing—from humans to humanoids. IEEE Trans. Robot. 26, 1–20 (2010).
- S. Luo, J. Bimbo, R. Dahiya, H. Liu, Robotic tactile perception of object properties: A review. Mechatronics 48, 54–67 (2017).
- F. Liu, S. Deswal, A. Christou, M. Shojaei Baghini, R. Chirila, D. Shakthivel, M. Chakraborty, R. Dahiya, Printed synaptic transistor–based electronic skin for robots to feel and learn. Sci. Robot. 7, eabl7286 (2022).
- R. A. John, N. Tiwari, M. I. B. Patdillah, M. R. Kulkarni, N. Tiwari, J. Basu, S. K. Bose, Ankit, C. J. Yu, A. Nirmal, S. K. Vishwanath, C. Bartolozzi, A. Basu, N. Mathews, Self healable neuromorphic memtransistor elements for decentralized sensory signal processing in robotics. Nat. Commun. 11, 4030 (2020).
- J. Ge, S. Zhang, Z. Liu, Z. Xie, S. Pan, Flexible artificial nociceptor using a biopolymer-based forming-free memristor. Nanoscale 11, 6591–6601 (2019).
- W. F. Młynarski, A. M. Hermundstad, Efficient and adaptive sensory codes. Nat. Neurosci. 24, 998–1009 (2021).
- Y. Lee, J.-H. Ahn, Biomimetic tactile sensors based on nanomaterials. ACS Nano 14, 1220–1226 (2020).
- A. I. Weber, K. Krishnamurthy, A. L. Fairhall, Coding principles in adaptation. Annu. Rev. Vis. Sci. 5, 427–449 (2019).
- Z. Cheng, B. L. Davies, D. G. Caldwell, L. S. Mattos, A hand-held robot for precise and safe PIVC. IEEE Robot. Autom. Lett. 4, 655–661 (2019).
- T. L. Edwards, K. Xue, H. C. M. Meenink, M. J. Beelen, G. J. L. Naus, M. P. Simunovic, M. Latasiewicz, A. D. Farmery, M. D. de Smet, R. E. MacLaren, First-in-human study of the safety and viability of intraocular robotic surgery. Nat. Biomed. Eng. 2, 649–656 (2018).
- M. S. Yasar, H. Alemzadeh, “Real-time context-aware detection of unsafe events in robot-assisted surgery” in 2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) (2020), pp. 385–397.
- A. Hussain, A. Malik, M. U. Halim, A. M. Ali, The use of robotics in surgery: a review. Int. J. Clin. Pract. 68, 1376–1382 (2014).
- R. Yuan, P. J. Tiw, L. Cai, Z. Yang, C. Liu, T. Zhang, C. Ge, R. Huang, Y. Yang, A neuromorphic physiological signal processing system based on VO2 memristor for next-generation human-machine interface. Nat. Commun. 14, 3695 (2023).
- X. F. Lu, Y. Zhang, N. Wang, S. Luo, K. Peng, L. Wang, H. Chen, W. Gao, X. H. Chen, Y. Bao, G. Liang, K. P. Loh, Exploring low power and ultrafast memristor on p-Type van der Waals SnS. Nano Lett. 21, 8800–8807 (2021).
- N. Youngblood, C. A. Ríos Ocampo, W. H. P. Pernice, H. Bhaskaran, Integrated optical memristors. Nat. Photonics 17, 561–572 (2023).
- C. Ma, Z. Luo, W. Huang, L. Zhao, Q. Chen, Y. Lin, X. Liu, Z. Chen, C. Liu, H. Sun, X. Jin, Y. Yin, X. Li, Sub-nanosecond memristor based on ferroelectric tunnel junction. Nat. Commun. 11, 1439 (2020).
- C.-H. Sung, J.-Z. Chuang, The cell biology of vision. J. Cell Biol. 190, 953–963 (2010).
- G. Chen, H. Cao, J. Conradt, H. Tang, F. Rohrbein, A. Knoll, Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Process. Mag. 37, 34–49 (2020).
- A. F. X. Goldberg, O. L. Moritz, D. S. Williams, Molecular basis for photoreceptor outer segment architecture. Prog. Retin. Eye Res. 55, 52–81 (2016).
- S. Liu, L. Liu, J. Tang, B. Yu, Y. Wang, W. Shi, Edge computing for autonomous driving: opportunities and challenges. Proc. IEEE 107, 1697–1716 (2019).
- I. Yaqoob, L. U. Khan, S. M. A. Kazmi, M. Imran, N. Guizani, C. S. Hong, Autonomous driving cars in smart cities: Recent advances, requirements, and challenges. IEEE Netw. 34, 174–181 (2020).
- H. Gao, B. Cheng, J. Wang, K. Li, J. Zhao, D. Li, Object classification using CNN-based fusion of vision and LIDAR in autonomous vehicle environment. IEEE Trans. Ind. Inform. 14, 4224–4231 (2018).
- M. Siam, H. Mahgoub, M. Zahran, S. Yogamani, M. Jagersand, A. El-Sallab, “MODNet: Motion and appearance based moving object detection network for autonomous driving” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC) (2018), pp. 2859–2864.
- A. Manjunath, Y. Liu, B. Henriques, A. Engstle, “Radar based object detection and tracking for autonomous driving” in 2018 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM) (2018), pp. 1–4.
- Y. Wang, K. Lv, R. Huang, S. Song, L. Yang, G. Huang, “Glance and focus: A dynamic approach to reducing spatial redundancy in image classification” in Advances in Neural Information Processing Systems (2020), vol. 33, pp. 2432–2444.
- T. Fu, X. Liu, H. Gao, J. E. Ward, X. Liu, B. Yin, Z. Wang, Y. Zhuo, D. J. F. Walker, J. Joshua Yang, J. Chen, D. R. Lovley, J. Yao, Bioinspired bio-voltage memristors. Nat. Commun. 11, 1861 (2020).
- V. K. Sangwan, M. C. Hersam, Neuromorphic nanoelectronic materials. Nat. Nanotechnol. 15, 517–528 (2020).
- Z. Wu, J. Lu, T. Shi, X. Zhao, X. Zhang, Y. Yang, F. Wu, Y. Li, Q. Liu, M. Liu, A habituation sensory nervous system with memristors. Adv. Mater. 32, 2004398 (2020).
- Z. Zhang, X. Zhao, X. Zhang, X. Hou, X. Ma, S. Tang, Y. Zhang, G. Xu, Q. Liu, S. Long, In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array. Nat. Commun. 13, 6590 (2022).
There is NO Competing Interest.
- SupplementaryInformation.pdf
Supplementary Information
- SV1GraspingExperiment1SharpObject.mp4
SV1 Grasping Experiment 1
- SV2GraspingExperiment2SlipperyObject.mp4
SV2 Grasping Experiment 2
- SV3VisualInformationProcessingatNight.mp4
SV3 Visual information processing at night
- SV4VisualInformationProcessinginDaytime.mp4
SV4 Visual information processing in daytime
- SV5VisualInformationProcessinginVariousDrivingScenarios.mp4
SV5 Visual information processing in various driving scenarios
- SV6IntroductiontoDifferentialNeuromorphicComputing.mp4
SV6 Introduction to differential neuromorphic computing