Joint-Confidence-guided Multi-Task Learning for 3D Reconstruction and Understanding from Monocular Camera.

3D reconstruction and understanding from monocular camera is a key issue in computer vision. Recent learning-based approaches, especially multi-task learning, significantly achieve the performance of the related tasks. However a few works still have limitation in drawing loss-spatial-aware information. In this paper, we propose a novel Joint-confidence-guided network (JCNet) to simultaneously predict depth, semantic labels, surface normal, and joint confidence map for corresponding loss functions. In details, we design a Joint Confidence Fusion and Refinement (JCFR) module to achieve multi-task feature fusion in the unified independent space, which can also absorb the geometric-semantic structure feature in the joint confidence map. We use confidence-guided uncertainty generated by the joint confidence map to supervise the multi-task prediction across the spatial and channel dimensions. To alleviate the training attention imbalance among different loss functions or spatial regions, the Stochastic Trust Mechanism (STM) is designed to stochastically modify the elements of joint confidence map in the training phase. Finally, we design a calibrating operation to alternately optimize the joint confidence branch and the other parts of JCNet to avoid overfiting. The proposed methods achieve state-of-the-art performance in both geometric-semantic prediction and uncertainty estimation on NYU-Depth V2 and Cityscapes.

Segment-Based Disparity Refinement With Occlusion Handling for Stereo Matching.

In this paper, we propose a disparity refinement method that directly refines the winner-take-all (WTA) disparity map by exploring its statistical significance. According to the primary steps of the segment-based stereo matching, the reference image is over-segmented into superpixels and a disparity plane is fitted for each superpixel by an improved random sample consensus (RANSAC). We design a two-layer optimization to refine the disparity plane. In the global optimization, mean disparities of superpixels are estimated by Markov random field (MRF) inference, and then, a 3D neighborhood system is derived from the mean disparities for occlusion handling. In the local optimization, a probability model exploiting Bayesian inference and Bayesian prediction is adopted and achieves second-order smoothness implicitly among 3D neighbors. The two-layer optimization is a pure disparity refinement method because no correlation information between stereo image pairs is demanded during the refinement. Experimental results on the Middlebury and KITTI datasets demonstrate that the proposed method can perform accurate stereo matching with a faster speed and handle the occlusion effectively. It can be indicated that the "matching cost computation + disparity refinement" framework is a possible solution to produce accurate disparity map at low computational cost.

Simulation of orthodontic force of archwire applied to full dentition using virtual bracket displacement method.

OBJECTIVE: Orthodontic force simulation of tooth provides important guidance for clinical orthodontic treatment. However, previous studies did not involve the simulation of orthodontic force of archwire applied to full dentition. This study aimed to develop a method to simulate orthodontic force of tooth produced by loading a continuous archwire to full dentition using finite element method. METHOD: A three-dimensional tooth-periodontal ligament-bone complex model of mandible was reconstructed from computed tomography images, and models of brackets and archwire were built. The simulation was completed through two steps. First, node displacements of archwire before and after loading were estimated through moving virtual brackets to drive archwire deformation. Second, the obtained node displacements were loaded to implement the loading of archwire, and orthodontic force was calculated. An orthodontic force tester (OFT) was used to measure orthodontic force in vitro for the validation. RESULTS: After the simulation convergence, archwire was successfully loaded to brackets, and orthodontic force of teeth was obtained. Compared with the measured orthodontic force using the OFT, the absolute difference of the simulation results ranged from 0.5 to 22.7 cN for force component and ranged from 2.2 to 80.0 cN•mm for moment component, respectively. The relative difference of the simulation results ranged from 2.5% to 11.0% for force component, and ranged from 0.6% to 14.7% for moment component, respectively. CONCLUSIONS: The developed orthodontic force simulation method based on virtual bracket displacement can be used to simulate orthodontic force provided by the archwire applied to full dentition.

A Method for Tooth Model Reconstruction Based on Integration of Multimodal Images.

A complete digital tooth model is needed for computer-aided orthodontic treatment. However, current methods mainly use computed tomography (CT) images to reconstruct the tooth model which may require multiple CT scans during orthodontic progress, and the reconstructed model is also inaccurate in crown area. This study developed a tooth model reconstruction method based on integration of CT images and laser scan images to overcome these disadvantages. In the method, crown models and complete tooth models are first reconstructed, respectively, from laser scan images and CT images. Then, crown models from laser scan images and tooth models from CT images are registered. Finally, the crown from laser scan images and root from CT images were fused to obtain a new tooth model. Experimental results verified that the developed method is effective to generate the complete tooth model by integrating CT images and laser scan images. Using the proposed method, the reconstructed models provide more accurate crown than CT images, and it is feasible to obtain complete tooth models at any stage of orthodontic treatment by using one CT scan at the pretreatment stage and one laser scan at that stage to avoid multiple CT scans.

Tooth and Alveolar Bone Segmentation From Dental Computed Tomography Images.

Three-dimensional (3D) models of tooth-alveolar bone complex are needed in treatment planning and simulation for computer-aided orthodontics. Tooth and alveolar bone segmentation from computed tomography (CT) images is a fundamental step in reconstructing their models. Due to less application of alveolar bone in conventional orthodontic treatment which may cause undesired side effects, the previous studies mainly focused on tooth segmentation and reconstruction, and did not consider the alveolar bone. In this study, we proposed a method to implement both tooth and alveolar bone segmentation from dental CT images for reconstructing their 3D models. First, the proposed method extracted the connected region of tooth and alveolar bone from CT images using a global convex level set model. Then, individual tooth and alveolar bone are separated from the connected region based on Radon transform and a local level set model. The experimental results showed that the proposed method could successfully complete both the tooth and alveolar bone segmentation from CT images, and outperformed the state of the art tooth segmentation methods in terms of accuracy. This suggests that the proposed method can be used in reconstructing the 3D models of tooth-alveolar bone complex for precise treatment.

[Three-dimensional tooth model reconstruction based on fusion of dental computed tomography images and laser-scanned images].

Complete three-dimensional(3D) tooth model provides essential information to assist orthodontists for diagnosis and treatment planning. Currently, 3D tooth model is mainly obtained by segmentation and reconstruction from dental computed tomography(CT) images. However, the accuracy of 3D tooth model reconstructed from dental CT images is low and not applicable for invisalign design. And another serious problem also occurs, i.e. frequentative dental CT scan during different intervals of orthodontic treatment often leads to radiation to the patients. Hence, this paper proposed a method to reconstruct tooth model based on fusion of dental CT images and laser-scanned images. A complete3 D tooth model was reconstructed with the registration and fusion between the root reconstructed from dental CT images and the crown reconstructed from laser-scanned images. The crown of the complete 3D tooth model reconstructed with the proposed method has higher accuracy. Moreover, in order to reconstruct complete 3D tooth model of each orthodontic treatment interval, only one pre-treatment CT scan is needed and in the orthodontic treatment process only the laser-scan is required. Therefore, radiation to the patients can be reduced significantly.

Individual tooth segmentation from CT images scanned with contacts of maxillary and mandible teeth.

BACKGROUND AND OBJECTIVE: Tooth segmentation from computed tomography (CT) images is a fundamental step in generating the three-dimensional models of tooth for computer-aided orthodontic treatment. Individual tooth segmentation from CT images scanned with contacts of maxillary and mandible teeth is especially challenging, and no method has been reported previously. This study aimed to develop a method for individual tooth segmentation from these images. METHODS: Tooth contours of maxilla and mandible are first segmented from the volumetric CT images slice-by-slice. For each slice, a line is extracted using the Radon transform to separate neighboring teeth, and each tooth contour is then segmented by a level set model from the corresponding side of the line. Then, each maxillary tooth whose contours overlap with that of mandible ones is detected, and a mesh model is reconstructed from all the contours of these maxillary and mandible teeth with contour overlap. The reconstructed mesh model is segmented using threshold and fast marching watershed method to separate the touched maxillary and mandible teeth. Finally, the separated tooth models are restored to fill the holes to obtain complete tooth models. The proposed method was tested on CT images of ten subjects scanned with natural contacts of maxillary and mandible teeth. RESULTS: For all the tested images, individual tooth regions are extracted successfully, and the segmentation accuracy and efficiency of the proposed method is promising. CONCLUSIONS: The proposed method is effective to segment individual tooth from CT images scanned with contacts of maxillary and mandible teeth.

A motion sensing-based framework for robotic manipulation.

To data, outside of the controlled environments, robots normally perform manipulation tasks operating with human. This pattern requires the robot operators with high technical skills training for varied teach-pendant operating system. Motion sensing technology, which enables human-machine interaction in a novel and natural interface using gestures, has crucially inspired us to adopt this user-friendly and straightforward operation mode on robotic manipulation. Thus, in this paper, we presented a motion sensing-based framework for robotic manipulation, which recognizes gesture commands captured from motion sensing input device and drives the action of robots. For compatibility, a general hardware interface layer was also developed in the framework. Simulation and physical experiments have been conducted for preliminary validation. The results have shown that the proposed framework is an effective approach for general robotic manipulation with motion sensing control.

Toward accurate tooth segmentation from computed tomography images using a hybrid level set model.

PURPOSE: A three-dimensional (3D) model of the teeth provides important information for orthodontic diagnosis and treatment planning. Tooth segmentation is an essential step in generating the 3D digital model from computed tomography (CT) images. The aim of this study is to develop an accurate and efficient tooth segmentation method from CT images. METHODS: The 3D dental CT volumetric images are segmented slice by slice in a two-dimensional (2D) transverse plane. The 2D segmentation is composed of a manual initialization step and an automatic slice by slice segmentation step. In the manual initialization step, the user manually picks a starting slice and selects a seed point for each tooth in this slice. In the automatic slice segmentation step, a developed hybrid level set model is applied to segment tooth contours from each slice. Tooth contour propagation strategy is employed to initialize the level set function automatically. Cone beam CT (CBCT) images of two subjects were used to tune the parameters. Images of 16 additional subjects were used to validate the performance of the method. Volume overlap metrics and surface distance metrics were adopted to assess the segmentation accuracy quantitatively. The volume overlap metrics were volume difference (VD, mm(3)) and Dice similarity coefficient (DSC, %). The surface distance metrics were average symmetric surface distance (ASSD, mm), RMS (root mean square) symmetric surface distance (RMSSSD, mm), and maximum symmetric surface distance (MSSD, mm). Computation time was recorded to assess the efficiency. The performance of the proposed method has been compared with two state-of-the-art methods. RESULTS: For the tested CBCT images, the VD, DSC, ASSD, RMSSSD, and MSSD for the incisor were 38.16 ± 12.94 mm(3), 88.82 ± 2.14%, 0.29 ± 0.03 mm, 0.32 ± 0.08 mm, and 1.25 ± 0.58 mm, respectively; the VD, DSC, ASSD, RMSSSD, and MSSD for the canine were 49.12 ± 9.33 mm(3), 91.57 ± 0.82%, 0.27 ± 0.02 mm, 0.28 ± 0.03 mm, and 1.06 ± 0.40 mm, respectively; the VD, DSC, ASSD, RMSSSD, and MSSD for the premolar were 37.95 ± 10.13 mm(3), 92.45 ± 2.29%, 0.29 ± 0.06 mm, 0.33 ± 0.10 mm, and 1.28 ± 0.72 mm, respectively; the VD, DSC, ASSD, RMSSSD, and MSSD for the molar were 52.38 ± 17.27 mm(3), 94.12 ± 1.38%, 0.30 ± 0.08 mm, 0.35 ± 0.17 mm, and 1.52 ± 0.75 mm, respectively. The computation time of the proposed method for segmenting CBCT images of one subject was 7.25 ± 0.73 min. Compared with two other methods, the proposed method achieves significant improvement in terms of accuracy. CONCLUSIONS: The presented tooth segmentation method can be used to segment tooth contours from CT images accurately and efficiently.