Auto-delineation of treatment target volume for radiation therapy using large language model-aided multimodal learning.

PURPOSE: Artificial intelligence (AI)-aided methods have made significant progress in the auto-delineation of normal tissues. However, these approaches struggle with the auto-contouring of radiotherapy target volume. Our goal is to model the delineation of target volume as a clinical decision-making problem, resolved by leveraging large language model-aided multimodal learning approaches. METHODS AND MATERIALS: A vision-language model, termed Medformer, has been developed, employing the hierarchical vision transformer as its backbone, and incorporating large language models to extract text-rich features. The contextually embedded linguistic features are seamlessly integrated into visual features for language-aware visual encoding through the visual language attention module. Metrics, including Dice similarity coefficient (DSC), intersection over union (IOU), and 95th percentile Hausdorff distance (HD95), were used to quantitatively evaluate the performance of our model. The evaluation was conducted on an in-house prostate cancer dataset and a public oropharyngeal carcinoma (OPC) dataset, totaling 668 subjects. RESULTS: Our Medformer achieved a DSC of 0.81 ± 0.10 versus 0.72 ± 0.10, IOU of 0.73 ± 0.12 versus 0.65 ± 0.09, and HD95 of 9.86 ± 9.77 mm versus 19.13 ± 12.96 mm for delineation of gross tumor volume (GTV) on the prostate cancer dataset. Similarly, on the OPC dataset, it achieved a DSC of 0.77 ± 0.11 versus 0.72 ± 0.09, IOU of 0.70 ± 0.09 versus 0.65 ± 0.07, and HD95 of 7.52 ± 4.8 mm versus 13.63 ± 7.13 mm, representing significant improvements (p < 0.05). For delineating the clinical target volume (CTV), Medformer achieved a DSC of 0.91 ± 0.04, IOU of 0.85 ± 0.05, and HD95 of 2.98 ± 1.60 mm, comparable to other state-of-the-art algorithms. CONCLUSIONS: Auto-delineation of the treatment target based on multimodal learning outperforms conventional approaches that rely purely on visual features. Our method could be adopted into routine practice to rapidly contour CTV/GTV.

Large Language Model-Augmented Auto-Delineation of Treatment Target Volume in Radiation Therapy.

Radiation therapy (RT) is one of the most effective treatments for cancer, and its success relies on the accurate delineation of targets. However, target delineation is a comprehensive medical decision that currently relies purely on manual processes by human experts. Manual delineation is time-consuming, laborious, and subject to interobserver variations. Although the advancements in artificial intelligence (AI) techniques have significantly enhanced the auto-contouring of normal tissues, accurate delineation of RT target volumes remains a challenge. In this study, we propose a visual language model-based RT target volume auto-delineation network termed Radformer. The Radformer utilizes a hierarchical vision transformer as the backbone and incorporates large language models to extract text-rich features from clinical data. We introduce a visual language attention module (VLAM) for integrating visual and linguistic features for language-aware visual encoding (LAVE). The Radformer has been evaluated on a dataset comprising 2985 patients with head-and-neck cancer who underwent RT. Metrics, including the Dice similarity coefficient (DSC), intersection over union (IOU), and 95th percentile Hausdorff distance (HD95), were used to evaluate the performance of the model quantitatively. Our results demonstrate that the Radformer has superior segmentation performance compared to other state-of-the-art models, validating its potential for adoption in RT practice.

Android mobile-platform-based image reconstruction for photoacoustic tomography.

Significance: In photoacoustic tomography (PAT), numerous reconstruction algorithms have been utilized to recover initial pressure rise distribution from the acquired pressure waves. In practice, most of these reconstructions are carried out on a desktop/workstation and the mobile-based reconstructions are far-flung. In recent years, mobile phones are becoming so ubiquitous, and most of them encompass a higher computing ability. Hence, realizing PAT image reconstruction on a mobile platform is intrinsic, and it will enhance the adaptability of PAT systems with point-of-care applications. Aim: To implement PAT image reconstruction in Android-based mobile platforms. Approach: For implementing PAT image reconstruction in Android-based mobile platforms, we proposed an Android-based application using Python to perform beamforming process in Android phones. Results: The performance of the developed application was analyzed on different mobile platforms using both simulated and experimental datasets. The results demonstrate that the developed algorithm can accomplish the image reconstruction of in vivo small animal brain dataset in 2.4 s. Furthermore, the developed application reconstructs PAT images with comparable speed and no loss of image quality compared to that on a laptop. Employing a two-fold downsampling procedure could serve as a viable solution for reducing the time needed for beamforming while preserving image quality with minimal degradation. Conclusions: We proposed an Android-based application that achieves image reconstruction on cheap, small, and universally available phones instead of relatively bulky expensive desktop computers/laptops/workstations. A beamforming speed of 2.4 s is achieved without hampering the quality of the reconstructed image.

Ultrasound-guided needle tracking with deep learning: A novel approach with photoacoustic ground truth.

Accurate needle guidance is crucial for safe and effective clinical diagnosis and treatment procedures. Conventional ultrasound (US)-guided needle insertion often encounters challenges in consistency and precisely visualizing the needle, necessitating the development of reliable methods to track the needle. As a powerful tool in image processing, deep learning has shown promise for enhancing needle visibility in US images, although its dependence on manual annotation or simulated data as ground truth can lead to potential bias or difficulties in generalizing to real US images. Photoacoustic (PA) imaging has demonstrated its capability for high-contrast needle visualization. In this study, we explore the potential of PA imaging as a reliable ground truth for deep learning network training without the need for expert annotation. Our network (UIU-Net), trained on ex vivo tissue image datasets, has shown remarkable precision in localizing needles within US images. The evaluation of needle segmentation performance extends across previously unseen ex vivo data and in vivo human data (collected from an open-source data repository). Specifically, for human data, the Modified Hausdorff Distance (MHD) value stands at approximately 3.73, and the targeting error value is around 2.03, indicating the strong similarity and small needle orientation deviation between the predicted needle and actual needle location. A key advantage of our method is its applicability beyond US images captured from specific imaging systems, extending to images from other US imaging systems.

High frame rate (∼3 Hz) circular photoacoustic tomography using single-element ultrasound transducer aided with deep learning.

Significance: In circular scanning photoacoustic tomography (PAT), it takes several minutes to generate an image of acceptable quality, especially with a single-element ultrasound transducer (UST). The imaging speed can be enhanced by faster scanning (with high repetition rate light sources) and using multiple-USTs. However, artifacts arising from the sparse signal acquisition and low signal-to-noise ratio at higher scanning speeds limit the imaging speed. Thus, there is a need to improve the imaging speed of the PAT systems without hampering the quality of the PAT image. Aim: To improve the frame rate (or imaging speed) of the PAT system by using deep learning (DL). Approach: For improving the frame rate (or imaging speed) of the PAT system, we propose a novel U-Net-based DL framework to reconstruct PAT images from fast scanning data. Results: The efficiency of the network was evaluated on both single- and multiple-UST-based PAT systems. Both phantom and in vivo imaging demonstrate that the network can improve the imaging frame rate by approximately sixfold in single-UST-based PAT systems and by approximately twofold in multi-UST-based PAT systems. Conclusions: We proposed an innovative method to improve the frame rate (or imaging speed) by using DL and with this method, the fastest frame rate of ∼ 3 Hz imaging is achieved without hampering the quality of the reconstructed image.

Multi-angle data acquisition to compensate transducer finite size in photoacoustic tomography.

In photoacoustic tomography (PAT) systems, the tangential resolution decreases due to the finite size of the transducer as the off-center distance increases. To address this problem, we propose a multi-angle detection approach in which the transducer used for data acquisition rotates around its center (with specific angles) as well as around the scanning center. The angles are calculated based on the central frequency and diameter of the transducer and the radius of the region-of-interest (ROI). Simulations with point-like absorbers (for point-spread-function evaluation) and a vasculature phantom (for quality assessment), and experiments with ten 0.5 mm-diameter pencil leads and a leaf skeleton phantom are used for evaluation of the proposed approach. The results show that a location-independent tangential resolution is achieved with 150 spatial sampling and central rotations with angles of ±8°/±16°. With further developments, the proposed detection strategy can replace the conventional detection (rotating a transducer around ROI) in PAT.

Photoacoustic imaging aided with deep learning: a review.

Photoacoustic imaging (PAI) is an emerging hybrid imaging modality integrating the benefits of both optical and ultrasound imaging. Although PAI exhibits superior imaging capabilities, its translation into clinics is still hindered by various limitations. In recent years, deeplearning (DL), a new paradigm of machine learning, is gaining a lot of attention due to its ability to improve medical images. Likewise, DL is also widely being used in PAI to overcome some of the limitations of PAI. In this review, we provide a comprehensive overview on the various DL techniques employed in PAI along with its promising advantages.

Deep-learning-based multi-transducer photoacoustic tomography imaging without radius calibration.

Pulsed laser diodes are used in photoacoustic tomography (PAT) as excitation sources because of their low cost, compact size, and high pulse repetition rate. In combination with multiple single-element ultrasound transducers (SUTs) the imaging speed of PAT can be improved. However, during PAT image reconstruction, the exact radius of each SUT is required for accurate reconstruction. Here we developed a novel deep learning approach to alleviate the need for radius calibration. We used a convolutional neural network (fully dense U-Net) aided with a convolutional long short-term memory block to reconstruct the PAT images. Our analysis on the test set demonstrates that the proposed network eliminates the need for radius calibration and improves the peak signal-to-noise ratio by ∼73% without compromising the image quality. In vivo imaging was used to verify the performance of the network.

Hybrid Carbon Dot Assembly as a Reactive Oxygen Species Nanogenerator for Ultrasound-Assisted Tumor Ablation.

The efficacy of reactive oxygen species (ROS)-based therapy is substantially constrained by the limited ROS generation, stern activation conditions, and lack of a straightforward reaction paradigm. Carbon dots (CDs) have been highly sought after for therapeutic applications for their biocompatibility and intrinsic fluorescence imaging capabilities, making them suitable for ROS generation. Herein, we synthesized a CD-based ultrasmall hybrid nanostructure possessing active sites of Mo, Cu, and IR-780 dye. After cooperative self-assembly with 1,2-distearoyl-sn-glycero-3-phosphoethanolamine-poly(ethylene glycol), the obtained assembly (CMIR-CDa) exhibits near-infrared fluorescence imaging and photoacoustic tomography. Interestingly, CMIR-CDa can generate singlet oxygen (1O2), hydroxyl radical (·OH), and superoxide radical anion (O2 • -) upon ultrasound stimulus owing to its sonosensitizing and enzyme-mimicking properties, showing an enhanced efficacy for tumor ablation in vivo. The collective in vitro and in vivo results indicate that CMIR-CDa has a high potency as an ROS nanogenerator under US irradiation, even at a low concentration. The present study offers an approach for engineering hybrid CDs in a bioinspired way for intratumoral ROS augmentation in response to deep tissue penetrable external stimuli.

Label-free high frame rate imaging of circulating blood clots using a dual modal ultrasound and photoacoustic system.

Deep vein thrombosis (DVT) is a disorder when a blood clot (thrombus) is formed in one of the deep veins. These clots detach from the original sites and circulate in the blood stream at high velocities. Diagnosing these blood clots at an early stage is necessary to decide the treatment strategy. For label-free, in vivo, and real-time detection, high framerate photoacoustic imaging can be used. In this work, a dual modal clinical ultrasound and photoacoustic (PA) system is used for the high framerate PA imaging of circulating blood clots in blood at linear velocities up to 107 cm/sec. Blood clot had 1.4 times higher signal-to-noise ratio (SNR) in the static mode and 1.3 times higher SNR compared to blood PA signal in the flow experiments. This work demonstrates that fast-moving circulating blood clots are easy to recognize against the background PA signal and may aid in early diagnosis.

Another decade of photoacoustic imaging.

Photoacoustic imaging-a hybrid biomedical imaging modality finding its way to clinical practices. Although the photoacoustic phenomenon was known more than a century back, only in the last two decades it has been widely researched and used for biomedical imaging applications. In this review we focus on the development and progress of the technology in the last decade (2011-2020). From becoming more and more user friendly, cheaper in cost, portable in size, photoacoustic imaging promises a wide range of applications, if translated to clinic. The growth of photoacoustic community is steady, and with several new directions researchers are exploring, it is inevitable that photoacoustic imaging will one day establish itself as a regular imaging system in the clinical practices.

In vivo detection of venous sinus distension due to intracranial hypotension in small animal using pulsed-laser-diode photoacoustic tomography.

Intracranial hypotension (IH) is a pathophysiological condition of reduced intracranial pressure caused by low cerebrospinal fluid (CSF) volume due to dural injuries from lumbar puncture, surgery, or trauma. Understanding the prognosis of IH in small animal models is important to gain insights on the complications associated with it such as orthostatic headache, cerebral venous thrombosis, coma, and so forth. Photoacoustic tomography (PAT) offers a novel and cost-effective way to perceive and detect IH in small animal models. In this study, a pulsed laser diode (PLD)-based PAT imaging system was used to examine the changes in the venous sinuses of the rat brain due to IH, induced through CSF extraction. After the CSF extraction, an increase in the sagittal sinus area by ~30% and width by 40% ± 5% was observed. These results provide supportive evidence that the PLD-PAT can be employed for detecting changes in sagittal sinus due to IH in rat model.

Deep learning approach to improve tangential resolution in photoacoustic tomography.

In circular scan photoacoustic tomography (PAT), the axial resolution is spatially invariant and is limited by the bandwidth of the detector. However, the tangential resolution is spatially variant and is dependent on the aperture size of the detector. In particular, the tangential resolution improves with the decreasing aperture size. However, using a detector with a smaller aperture reduces the sensitivity of the transducer. Thus, large aperture size detectors are widely preferred in circular scan PAT imaging systems. Although several techniques have been proposed to improve the tangential resolution, they have inherent limitations such as high cost and the need for customized detectors. Herein, we propose a novel deep learning architecture to counter the spatially variant tangential resolution in circular scanning PAT imaging systems. We used a fully dense U-Net based convolutional neural network architecture along with 9 residual blocks to improve the tangential resolution of the PAT images. The network was trained on the simulated datasets and its performance was verified by experimental in vivo imaging. Results show that the proposed deep learning network improves the tangential resolution by eight folds, without compromising the structural similarity and quality of image.

Pulsed Laser Diode-Based Desktop Photoacoustic Tomography for Monitoring Wash-In and Wash-Out of Dye in Rat Cortical Vasculature.

Photoacoustic (PA) tomography (PAT) imaging is an emerging biomedical imaging modality useful in various preclinical and clinical applications. Custom-made circular ring array-based transducers and conventional bulky Nd:YAG/OPO lasers inhibit translation of the PAT system to clinics. Ultra-compact pulsed laser diodes (PLDs) are currently being used as an alternative source of near-infrared excitation for PA imaging. High-speed dynamic in vivo imaging has been demonstrated using a compact PLD-based desktop PAT system (PLD-PAT). A visualized experimental protocol using the desktop PLD-PAT system is provided in this work for dynamic in vivo brain imaging. The protocol describes the desktop PLD-PAT system configuration, preparation of animal for brain vascular imaging, and procedure for dynamic visualization of indocyanine green (ICG) dye uptake and clearance process in rat cortical vasculature.