Technology trends and applications of deep learning in ultrasonography: image quality enhancement, diagnostic support, and improving workflow efficiency.

In this review of the most recent applications of deep learning to ultrasound imaging, the architectures of deep learning networks are briefly explained for the medical imaging applications of classification, detection, segmentation, and generation. Ultrasonography applications for image processing and diagnosis are then reviewed and summarized, along with some representative imaging studies of the breast, thyroid, heart, kidney, liver, and fetal head. Efforts towards workflow enhancement are also reviewed, with an emphasis on view recognition, scanning guide, image quality assessment, and quantification and measurement. Finally some future prospects are presented regarding image quality enhancement, diagnostic support, and improvements in workflow efficiency, along with remarks on hurdles, benefits, and necessary collaborations.

Deep learning with ultrasonography: automated classification of liver fibrosis using a deep convolutional neural network.

OBJECTIVES: The aim of this study was to develop a deep convolutional neural network (DCNN) for the prediction of the METAVIR score using B-mode ultrasonography images. METHODS: Datasets from two tertiary academic referral centers were used. A total of 13,608 ultrasonography images from 3446 patients who underwent surgical resection, biopsy, or transient elastography were used for training a DCNN for the prediction of the METAVIR score. Pathological specimens or estimated METAVIR scores derived from transient elastography were used as a reference standard. A four-class model (F0 vs. F1 vs. F23 vs. F4) was developed. Diagnostic performance of the algorithm was validated on a separate internal test set of 266 patients with 300 images and external test set of 572 patients with 1232 images. Performance in classification of cirrhosis was compared between the DCNN and five radiologists. RESULTS: The accuracy of the four-class model was 83.5% and 76.4% on the internal and external test set, respectively. The area under the receiver operating characteristic curve (AUC) for classification of cirrhosis (F4) was 0.901 (95% confidence interval [CI], 0.865-0.937) on the internal test set and 0.857 (95% CI, 0.825-0.889) on the external test set, respectively. The AUC of the DCNN for classification of cirrhosis (0.857) was significantly higher than that of all five radiologists (AUC range, 0.656-0.816; p value < 0.05) using the external test set. CONCLUSIONS: The DCNN showed high accuracy for determining METAVIR score using ultrasonography images and achieved better performance than that of radiologists in the diagnosis of cirrhosis. KEY POINTS: • DCNN accurately classified the ultrasonography images according to the METAVIR score. • The AUROC of this algorithm for cirrhosis assessment was significantly higher than that of radiologists. • DCNN using US images may offer an alternative tool for monitoring liver fibrosis.

Usefulness of Virtual Expiratory CT Images to Compensate for Respiratory Liver Motion in Ultrasound/CT Image Fusion: A Prospective Study in Patients with Focal Hepatic Lesions.

OBJECTIVE: To assess whether virtual expiratory (VE)-computed tomography (CT)/ultrasound (US) fusion imaging is more effective than conventional inspiratory (CI)-CT/US fusion imaging for hepatic interventional procedures. MATERIALS AND METHODS: This prospective study was approved by the Institutional Review Board, and informed consent was obtained from each patient. In total, 62 patients with focal hepatic lesions referred for hepatic interventional procedures were enrolled. VE-CT images were generated from CI-CT images to reduce the effects of respiration-induced liver motion. The two types of CT images were fused with real-time US images for each patient. The operators scored the visual similarity with the liver anatomy upon initial image fusion and the summative usability of complete image fusion using the respective five-point scales. The time required for complete image fusion and the number of point locks used were also compared. RESULTS: In comparison with CI-CT/US fusion imaging, VE-CT/US fusion imaging showed significantly higher visual similarity with the liver anatomy on the initial image fusion (mean score, 3.9 vs. 1.7; p < 0.001) and higher summative usability for complete image fusion (mean score, 4.0 vs. 1.9; p < 0.001). The required time (mean, 11.1 seconds vs. 22.5 seconds; p < 0.001) and the number of point locks (mean, 1.6 vs. 3.0; p < 0.001) needed for complete image fusion using VE-CT/US fusion imaging were significantly lower than those needed for CI-CT/US fusion imaging. CONCLUSION: VE-CT/US fusion imaging is more effective than CI-CT/US fusion imaging for hepatic interventional procedures.

Active contour configuration model for estimating the posterior ablative margin in image fusion of real-time ultrasound and 3D ultrasound or magnetic resonance images for radiofrequency ablation: an experimental study.

PURPOSE: The purpose of this study was to evaluate the accuracy of an active contour model for estimating the posterior ablative margin in images obtained by the fusion of real-time ultrasonography (US) and 3-dimensional (3D) US or magnetic resonance (MR) images of an experimental tumor model for radiofrequency ablation. METHODS: Chickpeas (n=12) and bovine rump meat (n=12) were used as an experimental tumor model. Grayscale 3D US and T1-weighted MR images were pre-acquired for use as reference datasets. US and MR/3D US fusion was performed for one group (n=4), and US and 3D US fusion only (n=8) was performed for the other group. Half of the models in each group were completely ablated, while the other half were incompletely ablated. Hyperechoic ablation areas were extracted using an active contour model from real-time US images, and the posterior margin of the ablation zone was estimated from the anterior margin. After the experiments, the ablated pieces of bovine rump meat were cut along the electrode path and the cut planes were photographed. The US images with the estimated posterior margin were compared with the photographs and post-ablation MR images. The extracted contours of the ablation zones from 12 US fusion videos and post-ablation MR images were also matched. RESULTS: In the four models fused under real-time US with MR/3D US, compression from the transducer and the insertion of an electrode resulted in misregistration between the real-time US and MR images, making the estimation of the ablation zones less accurate than was achieved through fusion between real-time US and 3D US. Eight of the 12 post-ablation 3D US images were graded as good when compared with the sectioned specimens, and 10 of the 12 were graded as good in a comparison with nicotinamide adenine dinucleotide staining and histopathologic results. CONCLUSION: Estimating the posterior ablative margin using an active contour model is a feasible way of predicting the ablation area, and US/3D US fusion was more accurate than US/MR fusion.

A deep learning framework for supporting the classification of breast lesions in ultrasound images.

In this research, we exploited the deep learning framework to differentiate the distinctive types of lesions and nodules in breast acquired with ultrasound imaging. A biopsy-proven benchmarking dataset was built from 5151 patients cases containing a total of 7408 ultrasound breast images, representative of semi-automatically segmented lesions associated with masses. The dataset comprised 4254 benign and 3154 malignant lesions. The developed method includes histogram equalization, image cropping and margin augmentation. The GoogLeNet convolutionary neural network was trained to the database to differentiate benign and malignant tumors. The networks were trained on the data with augmentation and the data without augmentation. Both of them showed an area under the curve of over 0.9. The networks showed an accuracy of about 0.9 (90%), a sensitivity of 0.86 and a specificity of 0.96. Although target regions of interest (ROIs) were selected by radiologists, meaning that radiologists still have to point out the location of the ROI, the classification of malignant lesions showed promising results. If this method is used by radiologists in clinical situations it can classify malignant lesions in a short time and support the diagnosis of radiologists in discriminating malignant lesions. Therefore, the proposed method can work in tandem with human radiologists to improve performance, which is a fundamental purpose of computer-aided diagnosis.

Image Fusion of Real-Time Ultrasonography with Computed Tomography: Factors Affecting the Registration Error and Motion of Focal Hepatic Lesions.

Factors affecting the registration error (RE) and motion of focal hepatic lesions (FHLs) in image fusion of real-time ultrasonography (US) with computed tomography (CT) images were prospectively assessed by focusing on respiratory movement and FHL location. Real-time US and pre-acquired CT images at end-inspiration were fused with FHLs for 103 patients. Three-dimensional US data containing FHLs were obtained during end-inspiratory/expiratory phases. Multivariate analysis revealed that diaphragm motion (p < 0.001), chronic liver disease (p = 0.02) and the absolute difference in distance between the FHL and the central portal vein (CPV) during respiration (p = 0.03) were the independent factors that revealed the maximum effect on RE. In contrast, diaphragm motion (p < 0.001) and distance between the FHL and CPV at inspiration (p = 0.036) revealed the maximum effect on FHL motion. In conclusion, RE and FHL motion are affected by the degree of respiratory movement and the location of the FHL. Therefore, image fusion with CT images should be used with caution if the degree of respiratory motion is significant or if the FHL is located at the periphery of the liver.

Comparison Between CT and MR Images as More Favorable Reference Data Sets for Fusion Imaging-Guided Radiofrequency Ablation or Biopsy of Hepatic Lesions: A Prospective Study with Focus on Patient's Respiration.

PURPOSE: To identify the more accurate reference data sets for fusion imaging-guided radiofrequency ablation or biopsy of hepatic lesions between computed tomography (CT) and magnetic resonance (MR) images. MATERIALS AND METHODS: This study was approved by the institutional review board, and written informed consent was received from all patients. Twelve consecutive patients who were referred to assess the feasibility of radiofrequency ablation or biopsy were enrolled. Automatic registration using CT and MR images was performed in each patient. Registration errors during optimal and opposite respiratory phases, time required for image fusion and number of point locks used were compared using the Wilcoxon signed-rank test. RESULTS: The registration errors during optimal respiratory phase were not significantly different between image fusion using CT and MR images as reference data sets (p = 0.969). During opposite respiratory phase, the registration error was smaller with MR images than CT (p = 0.028). The time and the number of points locks needed for complete image fusion were not significantly different between CT and MR images (p = 0.328 and p = 0.317, respectively). CONCLUSION: MR images would be more suitable as the reference data set for fusion imaging-guided procedures of focal hepatic lesions than CT images.

Automatic image fusion of real-time ultrasound with computed tomography images: a prospective comparison between two auto-registration methods.

Background A major drawback of conventional manual image fusion is that the process may be complex, especially for less-experienced operators. Recently, two automatic image fusion techniques called Positioning and Sweeping auto-registration have been developed. Purpose To compare the accuracy and required time for image fusion of real-time ultrasonography (US) and computed tomography (CT) images between Positioning and Sweeping auto-registration. Material and Methods Eighteen consecutive patients referred for planning US for radiofrequency ablation or biopsy for focal hepatic lesions were enrolled. Image fusion using both auto-registration methods was performed for each patient. Registration error, time required for image fusion, and number of point locks used were compared using the Wilcoxon signed rank test. Results Image fusion was successful in all patients. Positioning auto-registration was significantly faster than Sweeping auto-registration for both initial (median, 11 s [range, 3-16 s] vs. 32 s [range, 21-38 s]; P < 0.001] and complete (median, 34.0 s [range, 26-66 s] vs. 47.5 s [range, 32-90]; P = 0.001] image fusion. Registration error of Positioning auto-registration was significantly higher for initial image fusion (median, 38.8 mm [range, 16.0-84.6 mm] vs. 18.2 mm [6.7-73.4 mm]; P = 0.029), but not for complete image fusion (median, 4.75 mm [range, 1.7-9.9 mm] vs. 5.8 mm [range, 2.0-13.0 mm]; P = 0.338]. Number of point locks required to refine the initially fused images was significantly higher with Positioning auto-registration (median, 2 [range, 2-3] vs. 1 [range, 1-2]; P = 0.012]. Conclusion Positioning auto-registration offers faster image fusion between real-time US and pre-procedural CT images than Sweeping auto-registration. The final registration error is similar between the two methods.

A prospective comparison between auto-registration and manual registration of real-time ultrasound with MR images for percutaneous ablation or biopsy of hepatic lesions.

PURPOSE: To compare the accuracy and required time for image fusion of real-time ultrasound (US) with pre-procedural magnetic resonance (MR) images between positioning auto-registration and manual registration for percutaneous radiofrequency ablation or biopsy of hepatic lesions. METHODS: This prospective study was approved by the institutional review board, and all patients gave written informed consent. Twenty-two patients (male/female, n = 18/n = 4; age, 61.0 ± 7.7 years) who were referred for planning US to assess the feasibility of radiofrequency ablation (n = 21) or biopsy (n = 1) for focal hepatic lesions were included. One experienced radiologist performed the two types of image fusion methods in each patient. The performance of auto-registration and manual registration was evaluated. The accuracy of the two methods, based on measuring registration error, and the time required for image fusion for both methods were recorded using in-house software and respectively compared using the Wilcoxon signed rank test. RESULTS: Image fusion was successful in all patients. The registration error was not significantly different between the two methods (auto-registration: median, 3.75 mm; range, 1.0-15.8 mm vs. manual registration: median, 2.95 mm; range, 1.2-12.5 mm, p = 0.242). The time required for image fusion was significantly shorter with auto-registration than with manual registration (median, 28.5 s; range, 18-47 s, vs. median, 36.5 s; range, 14-105 s, p = 0.026). CONCLUSION: Positioning auto-registration showed promising results compared with manual registration, with similar accuracy and even shorter registration time.

Automatic Registration between Real-Time Ultrasonography and Pre-Procedural Magnetic Resonance Images: A Prospective Comparison between Two Registration Methods by Liver Surface and Vessel and by Liver Surface Only.

The aim of this study was to compare the accuracy of and the time required for image fusion between real-time ultrasonography (US) and pre-procedural magnetic resonance (MR) images using automatic registration by a liver surface only method and automatic registration by a liver surface and vessel method. This study consisted of 20 patients referred for planning US to assess the feasibility of percutaneous radiofrequency ablation or biopsy for focal hepatic lesions. The first 10 consecutive patients were evaluated by an experienced radiologist using the automatic registration by liver surface and vessel method, whereas the remaining 10 patients were evaluated using the automatic registration by liver surface only method. For all 20 patients, image fusion was automatically executed after following the protocols and fused real-time US and MR images moved synchronously. The accuracy of each method was evaluated by measuring the registration error, and the time required for image fusion was assessed by evaluating the recorded data using in-house software. The results obtained using the two automatic registration methods were compared using the Mann-Whitney U-test. Image fusion was successful in all 20 patients, and the time required for image fusion was significantly shorter with the automatic registration by liver surface only method than with the automatic registration by liver surface and vessel method (median: 43.0 s, range: 29-74 s vs. median: 83.0 s, range: 46-101 s; p = 0.002). The registration error did not significantly differ between the two methods (median: 4.0 mm, range: 2.1-9.9 mm vs. median: 3.7 mm, range: 1.8-5.2 mm; p = 0.496). The automatic registration by liver surface only method offers faster image fusion between real-time US and pre-procedural MR images than does the automatic registration by liver surface and vessel method. However, the degree of accuracy was similar for the two methods.

Motion compensation for ultrasound thermal imaging using motion-mapped reference model: an in vivo mouse study.

Ultrasound (US)-based thermal imaging is very sensitive to tissue motion, which is a major obstacle to apply US temperature monitoring to noninvasive thermal therapies of in vivo subjects. In this study, we aim to develop a motion compensation method for stable US thermal imaging in in vivo subjects. Based on the assumption that the major tissue motion is approximately periodic caused by respiration, we propose a motion compensation method for change in backscattered energy (CBE) with multiple reference frames. Among the reference frames, the most similar reference to the current frame is selected to subtract the respiratory-induced motions. Since exhaustive reference searching in all stored reference frames can impede real-time thermal imaging, we improve the reference searching by using a motion-mapped reference model. We tested our method in six tumor-bearing mice with high intensity focused ultrasound (HIFU) sonication in the tumor volume until the temperature had increased by 7°C. The proposed motion compensation was evaluated by root-mean-square-error (RMSE) analysis between the estimated temperature by CBE and the measured temperature by thermocouple. As a result, the mean ±SD RMSE in the heating range was 1.1±0.1°C with the proposed method, while the corresponding result without motion compensation was 4.3±2.6°C. In addition, with the idea of motion-mapped reference frame, total processing time to produce a frame of thermal image was reduced in comparison with the exhaustive reference searching, which enabled the motion-compensated thermal imaging in 15 frames per second with 150 reference frames under 50% HIFU duty ratio.

The smart house for older persons and persons with physical disabilities: structure, technology arrangements, and perspectives.

Smart houses are considered a good alternative for the independent life of older persons and persons with disabilities. Numerous intelligent devices, embedded into the home environment, can provide the resident with both movement assistance and 24-h health monitoring. Modern home-installed systems tend to be not only physically versatile in functionality but also emotionally human-friendly, i.e., they may be able to perform their functions without disturbing the user and without causing him/her any pain, inconvenience, or movement restriction, instead possibly providing him/her with comfort and pleasure. Through an extensive survey, this paper analyzes the building blocks of smart houses, with particular attention paid to the health monitoring subsystem as an important component, by addressing the basic requirements of various sensors implemented from both research and clinical perspectives. The paper will then discuss some important issues of the future development of an intelligent residential space with a human-friendly health monitoring functional system.