Deep learning-assisted construction of three-demensional facial midsagittal plane / 北京大学学报(医学版)

OBJECTIVE@#To establish a deep learning algorithm that can accurately determine three-dimensional facial anatomical landmarks, multi-view stacked hourglass convolutional neural networks (MSH-CNN) and to construct three-dimensional facial midsagittal plane automatically based on MSH-CNN and weighted Procrustes analysis algorithm.@*METHODS@#One hundred subjects with no obvious facial deformity were collected in our oral clinic. Three-dimensional facial data were scanned by three-dimensional facial scanner. Experts annotated twenty-one facial landmarks and midsagittal plane of each data. Eighty three-dimensional facial data were used as training set, to train the MSH-CNN in this study. The overview of MSH-CNN network architecture contained multi-view rendering and training the MSH-CNN network. The three-dimensional facial data were rendered from ninety-six views that were fed to MSH-CNN and the output was one heatmap per landmark. The result of the twenty-one landmarks was accurately placed on the three-dimensional facial data after a three-dimensional view ray voting process. The remaining twenty three-dimensional facial data were used as test set. The trained MSH-CNN automatically determined twenty-one three-dimensional facial anatomical landmarks of each case of data, and calculated the distance between each MSH-CNN landmark and the expert landmark, which was defined as position error. The midsagittal plane of the twenty subjects' could be automatically constructed, using the MSH-CNN and Procrustes analysis algorithm. To evaluate the effect of midsagittal plane by automatic method, the angle between the midsagittal plane constructed by the automatic method and the expert annotated plane was calculated, which was defined as angle error.@*RESULTS@#For twenty subjects with no obvious facial deformity, the average angle error of the midsagittal plane constructed by MSH-CNN and weighted Procrustes analysis algorithm was 0.73°±0.50°, in which the average position error of the twenty-one facial landmarks automatically determined by MSH-CNN was (1.13±0.24) mm, the maximum position error of the orbital area was (1.31±0.54) mm, and the minimum position error of the nasal area was (0.79±0.36) mm.@*CONCLUSION@#This research combines deep learning algorithms and Procrustes analysis algorithms to realize the fully automated construction of the three-dimensional midsagittal plane, which initially achieves the construction effect of clinical experts. The obtained results constituted the basis for the independent intellectual property software development.

Preliminary study on the method of automatically determining facial landmarks based on three-dimensional face template / 中华口腔医学杂志

Objective: To explore the establishment of an efficient and automatic method to determine anatomical landmarks in three-dimensional (3D) facial data, and to evaluate the effectiveness of this method in determining landmarks. Methods: A total of 30 male patients with tooth defect or dentition defect (with good facial symmetry) who visited the Department of Prosthodontics, Peking University School and Hospital of Stomatology from June to August 2021 were selected, and these participants' age was between 18-45 years. 3D facial data of patients was collected and the size normalization and overlap alignment were performed based on the Procrustes analysis algorithm. A 3D face average model was built in Geomagic Studio 2013 software, and a 3D face template was built through parametric processing. MeshLab 2020 software was used to determine the serial number information of 32 facial anatomical landmarks (10 midline landmarks and 22 bilateral landmarks). Five male patients with no mandibular deviation and 5 with mild mandibular deviation were selected from the Department of Orthodontics or Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology from June to August 2021. 3D facial data of patients was collected as test data. Based on the 3D face template and the serial number information of the facial anatomical landmarks, the coordinates of 32 facial anatomical landmarks on the test data were automatically determined with the help of the MeshMonk non-rigid registration algorithm program, as the data for the template method to determine the landmarks. The positions of 32 facial anatomical landmarks on the test data were manually determined by the same attending physician, and the coordinates of the landmarks were recorded as the data for determining landmarks by the expert method. Calculated the distance value of the coordinates of facial anatomical landmarks between the template method and the expert method, as the landmark localization error, and evaluated the effect of the template method in determining the landmarks. Results: For 5 patients with no mandibular deviation, the landmark localization error of all facial anatomical landmarks by template method was (1.65±1.19) mm, the landmark localization error of the midline facial anatomical landmarks was (1.19±0.45) mm, the landmark localization error of bilateral facial anatomical landmarks was (1.85±1.33) mm. For 5 patients with mild mandibular deviation, the landmark localization error of all facial anatomical landmarks by template method was (2.55±2.22) mm, the landmark localization error of the midline facial anatomical landmarks was (1.85±1.13) mm, the landmark localization error of bilateral facial anatomical landmarks was (2.87±2.45) mm. Conclusions: The automatic determination method of facial anatomical landmarks proposed in this study has certain feasibility, and the determination effect of midline facial anatomical landmarks is better than that of bilateral facial anatomical landmarks. The effect of determining facial anatomical landmarks in patients without mandibular deviation is better than that in patients with mild mandibular deviation.

A method for constructing three-dimensional face symmetry reference plane based on weighted shape analysis algorithm / 北京大学学报(医学版)

OBJECTIVE@#To establish a novel method based on three-dimensional (3D) shape analysis and weighted Procrustes analysis (WPA) algorithm to construct a 3D facial symmetry reference plane (SRP), automatically assigning weight to facial anatomical landmarks. The WPA algorithm suitability for commonly observed clinical cases of mandibular deviation were analysed and evaluated.@*METHODS@#Thirty patients with mandibular deviation were recruited for this study. The 3D facial SRPs were extracted independently based on original-mirror alignment method. Thirty-two anatomical landmarks were selected from the overall region by three times to obtain the mean coordinate. The SRP of experimental groups 1 and 2 were using the standard Procrustes analysis (PA) algorithm and WPA algorithm, respectively. A reference plane defined by experts based on regional iterative closest point (ICP) algorithm, served as the ground truth. Three experts manually selecting facial regions with good symmetry for original model, and common region was included in the study. The angle error values between the SRP of WPA algorithm in the experimental group 1 and the truth plane were evaluated in this study, and the SRP of PA algorithm of experimental group 2 was calculated in the same way. Statistics and measurement analysis were used to comprehensively evaluate the clinical suitability of the WPA algorithm to calculate the SRP. A paired t-test analysis (two-tailed) was conducted to compare the angles.@*RESULTS@#The average angle error between the SRP of WPA algorithm and the ground truth was 1.53°±0.84°, which was smaller than that between the SRP of PA and the ground truth (2.06°±0.86°). There were significant differences in the angle errors among the groups (P < 0.05). For the patients with severe mandibular deviation that the distance between pogonion and facial midline greater than 12 mm, the average angle error of the WPA algorithm was 0.86° smaller than that of the PA algorithm.@*CONCLUSION@#The WPA algorithm, based on weighted shape analysis, can provide a more adaptable SRP than the standard PA algorithm when applied to mandibular deviation patients and preliminarily simulate the diagnosis strategies of clinical experts.