80 × 120 AI-enhanced LiDAR system based on a lightweight intensity-RGB-dToF sensor fusion neural network deployed on an edge device.

Collecting higher-quality three-dimensional points-cloud data in various scenarios practically and robustly has led to a strong demand for such dToF-based LiDAR systems with higher ambient noise rejection ability and limited optical power consumption, which is a sharp conflict. To alleviate such a clash, an idea of utilizing a strong ambient noise rejection ability of intensity and RGB images is proposed, based on which a lightweight CNN is newly, to the best of our knowledge, designed, achieving a state-of-the-art performance even with 90 × less inference time and 480 × fewer FLOPs. With such net deployed on edge devices, a complete AI-LiDAR system is presented, showing a 100 × fewer signal photon demand in simulation experiments when creating depth images of the same quality.

228 × 304 200-mW lidar based on a single-point global-depth d-ToF sensor and RGB-guided super-resolution neural network.

The cutting-edge imaging system exhibits low output resolution and high power consumption, presenting challenges for the RGB-D fusion algorithm. In practical scenarios, aligning the depth map resolution with the RGB image sensor is a crucial requirement. In this Letter, the software and hardware co-design is considered to implement a lidar system based on the monocular RGB 3D imaging algorithm. A 6.4 × 6.4-mm2 deep-learning accelerator (DLA) system-on-chip (SoC) manufactured in a 40-nm CMOS is incorporated with a 3.6-mm2 TX-RX integrated chip fabricated in a 180-nm CMOS to employ the customized single-pixel imaging neural network. In comparison to the RGB-only monocular depth estimation technique, the root mean square error is reduced from 0.48 m to 0.3 m on the evaluated dataset, and the output depth map resolution matches the RGB input.

Multi-Scale Histogram-Based Probabilistic Deep Neural Network for Super-Resolution 3D LiDAR Imaging.

LiDAR (Light Detection and Ranging) imaging based on SPAD (Single-Photon Avalanche Diode) technology suffers from severe area penalty for large on-chip histogram peak detection circuits required by the high precision of measured depth values. In this work, a probabilistic estimation-based super-resolution neural network for SPAD imaging that firstly uses temporal multi-scale histograms as inputs is proposed. To reduce the area and cost of on-chip histogram computation, only part of the histogram hardware for calculating the reflected photons is implemented on a chip. On account of the distribution rule of returned photons, a probabilistic encoder as a part of the network is first proposed to solve the depth estimation problem of SPADs. By jointly using this neural network with a super-resolution network, 16× up-sampling depth estimation is realized using 32 × 32 multi-scale histogram outputs. Finally, the effectiveness of this neural network was verified in the laboratory with a 32 × 32 SPAD sensor system.