Universal Image Restoration with Text Prompt Diffusion.

Universal image restoration (UIR) aims to accurately restore images with a variety of unknown degradation types and levels. Existing methods, including both learning-based and prior-based approaches, heavily rely on low-quality image features. However, it is challenging to extract degradation information from diverse low-quality images, which limits model performance. Furthermore, UIR necessitates the recovery of images with diverse and complex types of degradation. Inaccurate estimations further decrease restoration performance, resulting in suboptimal recovery outcomes. To enhance UIR performance, a viable approach is to introduce additional priors. The current UIR methods have problems such as poor enhancement effect and low universality. To address this issue, we propose an effective framework based on a diffusion model (DM) for universal image restoration, dubbed ETDiffIR. Inspired by the remarkable performance of text prompts in the field of image generation, we employ text prompts to improve the restoration of degraded images. This framework utilizes a text prompt corresponding to the low-quality image to assist the diffusion model in restoring the image. Specifically, a novel text-image fusion block is proposed by combining the CLIP text encoder and the DA-CLIP image controller, which integrates text prompt encoding and degradation type encoding into time step encoding. Moreover, to reduce the computational cost of the denoising UNet in the diffusion model, we develop an efficient restoration U-shaped network (ERUNet) to achieve favorable noise prediction performance via depthwise convolution and pointwise convolution. We evaluate the proposed method on image dehazing, deraining, and denoising tasks. The experimental results indicate the superiority of our proposed algorithm.

Deep polyp image enhancement using region of interest with paired supervision.

Endoscopic medical imaging in complex curved intestinal structures are prone to uneven illumination, low contrast and lack of texture information. These problems may lead to diagnostic challenges. This paper described the first supervised deep learning based image fusion framework to enable the polyp region highlight through a global image enhancement and a local region of interest (ROI) with paired supervision. Firstly, we conducted a dual attention based network in global image enhancement. The Detail Attention Maps was used to preserve more image details and the Luminance Attention Maps was used to adjust the global illumination of the image. Secondly, we adopted the advanced polyp segmentation network ACSNet to obtain the accurate mask image of lesion region in local ROI acquisition. Finally, a new image fusion strategy was proposed to realize the local enhancement effect of polyp image. Experimental results show that our method can highlight the local details of the lesion area better and reach the optimal comprehensive performance with comparing with 16 traditional and state-of-the-art enhancement algorithms. And 8 doctors and 12 medical students were asked to evaluate our method for assisting clinical diagnosis and treatment effectively. Furthermore, the first paired image dataset LHI was constructed, which will be made available as an open source to research communities.

Deep unsupervised endoscopic image enhancement based on multi-image fusion.

BACKGROUND AND OBJECTIVE: A deep unsupervised endoscopic image enhancement method is proposed based on multi-image fusion to achieve high quality endoscope images from poorly illuminated, low contrast and color deviated images through an unsupervised mapping and deep learning network without the need for ground truth. METHODS: Firstly, three image enhancement methods are used to process original endoscopic images to obtain three derived images, which are then transformed into HSI color space. Secondly, a deep unsupervised multi-image fusion network (DerivedFuse) is proposed to extract and fuse features of the derived images accurately by utilizing a new no-reference quality metric as loss function. I-channel images of the three derived images are inputted into the DerivedFuse network to enhance the intensity component of the original image. Finally, a saturation adjustment function is proposed to adaptive adjusting the saturation component of HSI color space to enrich the color information of the original input image. RESULTS: Three evaluation metrics: Entropy, Contrast Improvement Index (CII) and Average Gradient (AG) are used to evaluate the performance of the proposed method. The results are compared with that of fourteen state-of-the-art algorithms. Experiments on endoscopic image enhancement show that the Entropy value of our method is 3.27% higher than the optimal entropy value of comparison algorithms. The CII of our proposed method is 6.19% higher than that of comparison algorithms. The AG of our method is 7.83% higher than the optimal AG of comparison algorithms. CONCLUSIONS: The proposed deep unsupervised multi-image fusion method can obtain image information details, enhance endoscopic images with high contrast, rich and natural color information, visual and image quality. Sixteen doctors and medical students have given their assessments on the proposed method for assisting clinical diagnoses.