Intelligent soft robotic fingers with multi-modality perception ability.

In the context of industry 4.0, automatic sorting is becoming prevalent in production lines. Herein, we developed a bionic sensing system to achieve real-time object recognition. The system consists of 9 single-layer triboelectric nanogenerators (SL-TENGs) as touch sensors and 3 comb-shaped TENGs (CS-TENGs) as bending sensors, with a sensitivity of 110 V/kPa and stable output after 20,000 press cycles. These sensors were attached to a manipulator composed of three soft actuators, serving as soft robotic fingers. An enhanced electrical output of these sensors was achieved successfully, demonstrating their feasibility in detecting grasping location, contact pressure, and bending curvature. A one-dimensional convolutional neural network (1D-CNN) with 98.96% accuracy extracted information from the sensors, enabling the manipulator to serve as an intelligent sensing system with multi-modality perception ability. This robotic manipulator successfully integrated TENG-based self-powered sensors, soft actuators, and artificial intelligence, demonstrating the potential for future digital twin applications, particularly in automatic component sorting.

A Wearable Multidimensional Motion Sensor for AI-Enhanced VR Sports.

Regular exercise paves the way to a healthy life. However, conventional sports events are susceptible to weather conditions. Current motion sensors for home-based sports are mainly limited by operation power consumption, single-direction sensitivity, or inferior data analysis. Herein, by leveraging the 3-dimensional printing technique and triboelectric effect, a wearable self-powered multidimensional motion sensor has been developed to detect both the vertical and planar movement trajectory. By integrating with a belt, this sensor could be used to identify some low degree of freedom motions, e.g., waist or gait motion, with a high accuracy of 93.8%. Furthermore, when wearing the sensor at the ankle position, signals generated from shank motions that contain more abundant information could also be effectively collected. By means of a deep learning algorithm, the kicking direction and force could be precisely differentiated with an accuracy of 97.5%. Toward practical application, a virtual reality-enabled fitness game and a shooting game were successfully demonstrated. This work is believed to open up new insights for the development of future household sports or rehabilitation.

Soft Robotic Perception System with Ultrasonic Auto-Positioning and Multimodal Sensory Intelligence.

Flexible electronics such as tactile cognitive sensors have been broadly adopted in soft robotic manipulators to enable human-skin-mimetic perception. To achieve appropriate positioning for randomly distributed objects, an integrated guiding system is inevitable. Yet the conventional guiding system based on cameras or optical sensors exhibits limited environment adaptability, high data complexity, and low cost effectiveness. Herein, a soft robotic perception system with remote object positioning and a multimodal cognition capability is developed by integrating an ultrasonic sensor with flexible triboelectric sensors. The ultrasonic sensor is able to detect the object shape and distance by reflected ultrasound. Thereby the robotic manipulator can be positioned to an appropriate position to perform object grasping, during which the ultrasonic and triboelectric sensors can capture multimodal sensory information such as object top profile, size, shape, hardness, material, etc. These multimodal data are then fused for deep-learning analytics, leading to a highly enhanced accuracy in object identification (∼100%). The proposed perception system presents a facile, low-cost, and effective methodology to integrate positioning capability with multimodal cognitive intelligence in soft robotics, significantly expanding the functionalities and adaptabilities of current soft robotic systems in industrial, commercial, and consumer applications.

Human Machine Interface with Wearable Electronics Using Biodegradable Triboelectric Films for Calligraphy Practice and Correction.

Letter handwriting, especially stroke correction, is of great importance for recording languages and expressing and exchanging ideas for individual behavior and the public. In this study, a biodegradable and conductive carboxymethyl chitosan-silk fibroin (CSF) film is prepared to design wearable triboelectric nanogenerator (denoted as CSF-TENG), which outputs of Voc ≈ 165 V, Isc ≈ 1.4 µA, and Qsc ≈ 72 mW cm-2. Further, in vitro biodegradation of CSF film is performed through trypsin and lysozyme. The results show that trypsin and lysozyme have stable and favorable biodegradation properties, removing 63.1% of CSF film after degrading for 11 days. Further, the CSF-TENG-based human-machine interface (HMI) is designed to promptly track writing steps and access the accuracy of letters, resulting in a straightforward communication media of human and machine. The CSF-TENG-based HMI can automatically recognize and correct three representative letters (F, H, and K), which is benefited by HMI system for data processing and analysis. The CSF-TENG-based HMI can make decisions for the next stroke, highlighting the stroke in advance by replacing it with red, which can be a candidate for calligraphy practice and correction. Finally, various demonstrations are done in real-time to achieve virtual and real-world controls including writing, vehicle movements, and healthcare.

Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions.

Advancements of virtual reality technology pave the way for developing wearable devices to enable somatosensory sensation, which can bring more comprehensive perception and feedback in the metaverse-based virtual society. Here, we propose augmented tactile-perception and haptic-feedback rings with multimodal sensing and feedback capabilities. This highly integrated ring consists of triboelectric and pyroelectric sensors for tactile and temperature perception, and vibrators and nichrome heaters for vibro- and thermo-haptic feedback. All these components integrated on the ring can be directly driven by a custom wireless platform of low power consumption for wearable/portable scenarios. With voltage integration processing, high-resolution continuous finger motion tracking is achieved via the triboelectric tactile sensor, which also contributes to superior performance in gesture/object recognition with artificial intelligence analysis. By fusing the multimodal sensing and feedback functions, an interactive metaverse platform with cross-space perception capability is successfully achieved, giving people a face-to-face like immersive virtual social experience.

Soft Modular Glove with Multimodal Sensing and Augmented Haptic Feedback Enabled by Materials' Multifunctionalities.

Immersive communications rely on smart perception based on diversified and augmented sensing and feedback technologies. However, the increasing of functional components also raises the issue of increased system complexity. Here, we propose a modular soft glove with multimodal sensing and feedback functions by exploring and utilizing the multiple properties of glove materials. With a single design of basic structure, the main functional unit possesses triboelectric-based sensing of static and dynamic contact, vibration, strain, and pneumatic actuation. Additionally, the same unit is also capable of offering pneumatic tactile haptic feedback and electroresistive thermal haptic feedback. Together with a machine learning algorithm, the proposed glove not only performs real-time detection of dexterous hand motion and direct feedback but also realizes intelligent object recognition and augmented feedback, which significantly enhance the communication and perception of more comprehensive information. In general, this glove utilizes a facile designed sensing and feedback device to achieve dual-way and multimodal communication among humans, machines, and the virtual world via smart perceptions.

Noncontact Human-Machine Interface Using Complementary Information Fusion Based on MEMS and Triboelectric Sensors.

Current noncontact human-machine interfaces (HMIs) either suffer from high power consumption, complex signal processing circuits, and algorithms, or cannot support multidimensional interaction. Here, a minimalist, low-power, and multimodal noncontact interaction interface is realized by fusing the complementary information obtained from a microelectromechanical system (MEMS) humidity sensor and a triboelectric sensor. The humidity sensor composed of a two-port aluminum nitride (AlN) bulk wave resonator operating in its length extensional mode and a layer of graphene oxide (GO) film with uniform and controllable thickness, possesses an ultra-tiny form factor (200 × 400 µm2 ), high signal strength (Q = 1729.5), and low signal noise level (±0.31%RH), and is able to continuously and steadily interact with an approaching finger. Meanwhile, the facile triboelectric sensor made of two annular aluminum electrodes enables the interaction interface to rapidly recognize the multidirectional finger movements. By leveraging the resonant frequency changes of the humidity sensor and output voltage waveforms of the triboelectric sensor, the proposed interaction interface is successfully demonstrated as a game control interface to manipulate a car in virtual reality (VR) space and a password input interface to enter high-security 3D passwords, indicating its great potential in diversified applications in the future Metaverse.

Progress of Advanced Devices and Internet of Things Systems as Enabling Technologies for Smart Homes and Health Care.

In the Internet of Things (IoT) era, various devices (e.g., sensors, actuators, energy harvesters, etc.) and systems have been developed toward the realization of smart homes/buildings and personal health care. These advanced devices can be categorized into ambient devices and wearable devices based on their usage scenarios, to enable motion tracking, health monitoring, daily care, home automation, fall detection, intelligent interaction, assistance, living convenience, and security in smart homes. With the rapidly increasing number of such advanced devices and IoT systems, achieving fully self-sustained and multimodal intelligent systems is becoming more and more important to realize a sustainable and all-in-one smart home platform. Hence, in this Review, we systematically present the recent progress of the development of advanced materials, fabrication techniques, devices, and systems for enabling smart home and health care applications. First, advanced polymer, fiber, and fabric materials as well as their respective fabrication techniques for large-scale manufacturing are discussed. After that, functional devices classified into ambient devices (at home ambiance such as door, floor, table, chair, bed, toilet, window, wall, etc.) and wearable devices (on body parts such as finger, wrist, arm, throat, face, back, etc.) are presented for diverse monitoring and auxiliary applications. Next, the current developments of self-sustained systems and intelligent systems are reviewed in detail, indicating two promising research directions in this field. Last, conclusions and outlook pinpointed on the existing challenges and opportunities are provided for the research community to consider.

Artificial Intelligence of Things (AIoT) Enabled Virtual Shop Applications Using Self-Powered Sensor Enhanced Soft Robotic Manipulator.

Rapid advancements of artificial intelligence of things (AIoT) technology pave the way for developing a digital-twin-based remote interactive system for advanced robotic-enabled industrial automation and virtual shopping. The embedded multifunctional perception system is urged for better interaction and user experience. To realize such a system, a smart soft robotic manipulator is presented that consists of a triboelectric nanogenerator tactile (T-TENG) and length (L-TENG) sensor, as well as a poly(vinylidene fluoride) (PVDF) pyroelectric temperature sensor. With the aid of machine learning (ML) for data processing, the fusion of the T-TENG and L-TENG sensors can realize the automatic recognition of the grasped objects with the accuracy of 97.143% for 28 different shapes of objects, while the temperature distribution can also be obtained through the pyroelectric sensor. By leveraging the IoT and artificial intelligence (AI) analytics, a digital-twin-based virtual shop is successfully implemented to provide the users with real-time feedback about the details of the product. In general, by offering a more immersive experience in human-machine interactions, the proposed remote interactive system shows the great potential of being the advanced human-machine interface for the applications of the unmanned working space.

Low cost exoskeleton manipulator using bidirectional triboelectric sensors enhanced multiple degree of freedom sensory system.

Rapid developments of robotics and virtual reality technology are raising the requirements of more advanced human-machine interfaces for achieving efficient parallel control. Exoskeleton as an assistive wearable device, usually requires a huge cost and complex data processing to track the multi-dimensional human motions. Alternatively, we propose a triboelectric bi-directional sensor as a universal and cost-effective solution to a customized exoskeleton for monitoring all of the movable joints of the human upper limbs with low power consumption. The corresponding movements, including two DOF rotations of the shoulder, twisting of the wrist, and the bending motions, are detected and utilized for controlling the virtual character and the robotic arm in real-time. Owing to the structural consistency between the exoskeleton and the human body, further kinetic analysis offers additional physical parameters without introducing other types of sensors. This exoskeleton sensory system shows a great potential of being an economic and advanced human-machine interface for supporting the manipulation in both real and virtual worlds, including robotic automation, healthcare, and training applications.

Triboelectric Nanogenerators and Hybridized Systems for Enabling Next-Generation IoT Applications.

In the past few years, triboelectric nanogenerator-based (TENG-based) hybrid generators and systems have experienced a widespread and flourishing development, ranging among almost every aspect of our lives, e.g., from industry to consumer, outdoor to indoor, and wearable to implantable applications. Although TENG technology has been extensively investigated for mechanical energy harvesting, most developed TENGs still have limitations of small output current, unstable power generation, and low energy utilization rate of multisource energies. To harvest the ubiquitous/coexisted energy forms including mechanical, thermal, and solar energy simultaneously, a promising direction is to integrate TENG with other transducing mechanisms, e.g., electromagnetic generator, piezoelectric nanogenerator, pyroelectric nanogenerator, thermoelectric generator, and solar cell, forming the hybrid generator for synergetic single-source and multisource energy harvesting. The resultant TENG-based hybrid generators utilizing integrated transducing mechanisms are able to compensate for the shortcomings of each mechanism and overcome the above limitations, toward achieving a maximum, reliable, and stable output generation. Hence, in this review, we systematically introduce the key technologies of the TENG-based hybrid generators and hybridized systems, in the aspects of operation principles, structure designs, optimization strategies, power management, and system integration. The recent progress of TENG-based hybrid generators and hybridized systems for the outdoor, indoor, wearable, and implantable applications is also provided. Lastly, we discuss our perspectives on the future development trend of hybrid generators and hybridized systems in environmental monitoring, human activity sensation, human-machine interaction, smart home, healthcare, wearables, implants, robotics, Internet of things (IoT), and many other fields.

Volatile organic compounds sensing based on Bennet doubler-inspired triboelectric nanogenerator and machine learning-assisted ion mobility analysis.

Ion mobility analysis is a well-known analytical technique for identifying gas-phase compounds in fast-response gas-monitoring systems. However, the conventional plasma discharge system is bulky, operates at a high temperature, and inappropriate for volatile organic compounds (VOCs) concentration detection. Therefore, we report a machine learning (ML)-enhanced ion mobility analyzer with a triboelectric-based ionizer, which offers good ion mobility selectivity and VOC recognition ability with a small-sized device and non-strict operating environment. Based on the charge accumulation mechanism, a multi-switched manipulation triboelectric nanogenerator (SM-TENG) can provide a direct current (DC) bias at the order of a few hundred, which can be further leveraged as the power source to obtain a unique and repeatable discharge characteristic of different VOCs, and their mixtures, with a special tip-plate electrode configuration. Aiming to tackle the grand challenge in the detection of multiple VOCs, the ML-enhanced ion mobility analysis method was successfully demonstrated by extracting specific features automatically from ion mobility spectrometry data with ML algorithms, which significantly enhance the detection ability of the SM-TENG based VOC analyzer, showing a portable real-time VOC monitoring solution with rapid response and low power consumption for future internet of things based environmental monitoring applications.

Triboelectric nanogenerator sensors for soft robotics aiming at digital twin applications.

Designing efficient sensors for soft robotics aiming at human machine interaction remains a challenge. Here, we report a smart soft-robotic gripper system based on triboelectric nanogenerator sensors to capture the continuous motion and tactile information for soft gripper. With the special distributed electrodes, the tactile sensor can perceive the contact position and area of external stimuli. The gear-based length sensor with a stretchable strip allows the continuous detection of elongation via the sequential contact of each tooth. The triboelectric sensory information collected during the operation of soft gripper is further trained by support vector machine algorithm to identify diverse objects with an accuracy of 98.1%. Demonstration of digital twin applications, which show the object identification and duplicate robotic manipulation in virtual environment according to the real-time operation of the soft-robotic gripper system, is successfully created for virtual assembly lines and unmanned warehouse applications.

Deep learning enabled smart mats as a scalable floor monitoring system.

Toward smart building and smart home, floor as one of our most frequently interactive interfaces can be implemented with embedded sensors to extract abundant sensory information without the video-taken concerns. Yet the previously developed floor sensors are normally of small scale, high implementation cost, large power consumption, and complicated device configuration. Here we show a smart floor monitoring system through the integration of self-powered triboelectric floor mats and deep learning-based data analytics. The floor mats are fabricated with unique "identity" electrode patterns using a low-cost and highly scalable screen printing technique, enabling a parallel connection to reduce the system complexity and the deep-learning computational cost. The stepping position, activity status, and identity information can be determined according to the instant sensory data analytics. This developed smart floor technology can establish the foundation using floor as the functional interface for diverse applications in smart building/home, e.g., intelligent automation, healthcare, and security.

Wearable Triboelectric/Aluminum Nitride Nano-Energy-Nano-System with Self-Sustainable Photonic Modulation and Continuous Force Sensing.

Wearable photonics offer a promising platform to complement the thriving complex wearable electronics system by providing high-speed data transmission channels and robust optical sensing paths. Regarding the realization of photonic computation and tunable (de)multiplexing functions based on system-level integration of abundant photonic modulators, it is challenging to reduce the overwhelming power consumption in traditional current-based silicon photonic modulators. This issue is addressed by integrating voltage-based aluminum nitride (AlN) modulator and textile triboelectric nanogenerator (T-TENG) on a wearable platform to form a nano-energy-nano-system (NENS). The T-TENG transduces the mechanical stimulations into electrical signals based on the coupling of triboelectrification and electrostatic induction. The self-generated high-voltage from the T-TENG is applied to the AlN modulator and boosts its modulation efficiency regardless of AlN's moderate Pockels effect. Complementarily, the AlN modulator's capacitive nature enables the open-circuit operation mode of T-TENG, providing the integrated NENS with continuous force sensing capability which is notably uninfluenced by operation speeds. Furthermore, a physical model is proposed to describe the coupled AlN modulator/T-TENG system. With the enhanced photonic modulation and the open-circuit operation mode enabled by synergies between the AlN modulator and the T-TENG, optical Morse code transmission and continuous human motion monitoring are demonstrated for practical wearable applications.

Machine Learning Glove Using Self-Powered Conductive Superhydrophobic Triboelectric Textile for Gesture Recognition in VR/AR Applications.

The rapid progress of Internet of things (IoT) technology raises an imperative demand on human machine interfaces (HMIs) which provide a critical linkage between human and machines. Using a glove as an intuitive and low-cost HMI can expediently track the motions of human fingers, resulting in a straightforward communication media of human-machine interactions. When combining several triboelectric textile sensors and proper machine learning technique, it has great potential to realize complex gesture recognition with the minimalist-designed glove for the comprehensive control in both real and virtual space. However, humidity or sweat may negatively affect the triboelectric output as well as the textile itself. Hence, in this work, a facile carbon nanotubes/thermoplastic elastomer (CNTs/TPE) coating approach is investigated in detail to achieve superhydrophobicity of the triboelectric textile for performance improvement. With great energy harvesting and human motion sensing capabilities, the glove using the superhydrophobic textile realizes a low-cost and self-powered interface for gesture recognition. By leveraging machine learning technology, various gesture recognition tasks are done in real time by using gestures to achieve highly accurate virtual reality/augmented reality (VR/AR) controls including gun shooting, baseball pitching, and flower arrangement, with minimized effect from sweat during operation.

Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications.

Human-machine interfaces (HMIs) experience increasing requirements for intuitive and effective manipulation. Current commercialized solutions of glove-based HMI are limited by either detectable motions or the huge cost on fabrication, energy, and computing power. We propose the haptic-feedback smart glove with triboelectric-based finger bending sensors, palm sliding sensor, and piezoelectric mechanical stimulators. The detection of multidirectional bending and sliding events is demonstrated in virtual space using the self-generated triboelectric signals for various degrees of freedom on human hand. We also perform haptic mechanical stimulation via piezoelectric chips to realize the augmented HMI. The smart glove achieves object recognition using machine learning technique, with an accuracy of 96%. Through the integrated demonstration of multidimensional manipulation, haptic feedback, and AI-based object recognition, our glove reveals its potential as a promising solution for low-cost and advanced human-machine interaction, which can benefit diversified areas, including entertainment, home healthcare, sports training, and medical industry.

Wearable Triboelectric-Human-Machine Interface (THMI) Using Robust Nanophotonic Readout.

With the rapid advances in wearable electronics and photonics, self-sustainable wearable systems are desired to increase service life and reduce maintenance frequency. Triboelectric technology stands out as a promising versatile technology due to its flexibility, self-sustainability, broad material availability, low cost, and good scalability. Various triboelectric-human-machine interfaces (THMIs) have been developed including interactive gloves, eye blinking/body motion-triggered interfaces, voice/breath monitors, and self-induced wireless interfaces. Nonetheless, THMIs conventionally use electrical readout and produce pulse-like signals due to the transient charge flows, leading to unstable and lossy transfer of interaction information. To address this issue, we propose a strategy by equipping THMIs with robust nanophotonic aluminum nitride (AlN) modulators for readout. The electrically capacitive nature of AlN modulators enables THMIs to work in the open-circuit condition with negligible charge flows. Meanwhile, the interaction information is transduced from THMIs' voltage to AlN modulators' optical output via the electro-optic Pockels effect. Thanks to the negligible charge flow and the high-speed optical information carrier, stable, information-lossless, and real-time THMIs are achieved. Leveraging the design flexibility of THMIs and nanophotonic readout circuits, various linear sensitivities independent of force speeds are achieved in different interaction force ranges. Toward practical applications, we develop a smart glove to realize continuous real-time robotics control and virtual/augmented reality interaction. Our work demonstrates a generic approach for developing self-sustainable HMIs with stable, information-lossless, and real-time features for wearable systems.

Development Trends and Perspectives of Future Sensors and MEMS/NEMS.

With the fast development of the fifth-generation cellular network technology (5G), the future sensors and microelectromechanical systems (MEMS)/nanoelectromechanical systems (NEMS) are presenting a more and more critical role to provide information in our daily life. This review paper introduces the development trends and perspectives of the future sensors and MEMS/NEMS. Starting from the issues of the MEMS fabrication, we introduced typical MEMS sensors for their applications in the Internet of Things (IoTs), such as MEMS physical sensor, MEMS acoustic sensor, and MEMS gas sensor. Toward the trends in intelligence and less power consumption, MEMS components including MEMS/NEMS switch, piezoelectric micromachined ultrasonic transducer (PMUT), and MEMS energy harvesting were investigated to assist the future sensors, such as event-based or almost zero-power. Furthermore, MEMS rigid substrate toward NEMS flexible-based for flexibility and interface was discussed as another important development trend for next-generation wearable or multi-functional sensors. Around the issues about the big data and human-machine realization for human beings' manipulation, artificial intelligence (AI) and virtual reality (VR) technologies were finally realized using sensor nodes and its wave identification as future trends for various scenarios.