Swin Transformer and the Unet Architecture to Correct Motion Artifacts in Magnetic Resonance Image Reconstruction.

We present a deep learning-based method that corrects motion artifacts and thus accelerates data acquisition and reconstruction of magnetic resonance images. The novel model, the Motion Artifact Correction by Swin Network (MACS-Net), uses a Swin transformer layer as the fundamental block and the Unet architecture as the neural network backbone. We employ a hierarchical transformer with shifted windows to extract multiscale contextual features during encoding. A new dual upsampling technique is employed to enhance the spatial resolutions of feature maps in the Swin transformer-based decoder layer. A raw magnetic resonance imaging dataset is used for network training and testing; the data contain various motion artifacts with ground truth images of the same subjects. The results were compared to six state-of-the-art MRI image motion correction methods using two types of motions. When motions were brief (within 5 s), the method reduced the average normalized root mean square error (NRMSE) from 45.25% to 17.51%, increased the mean structural similarity index measure (SSIM) from 79.43% to 91.72%, and increased the peak signal-to-noise ratio (PSNR) from 18.24 to 26.57 dB. Similarly, when motions were extended from 5 to 10 s, our approach decreased the average NRMSE from 60.30% to 21.04%, improved the mean SSIM from 33.86% to 90.33%, and increased the PSNR from 15.64 to 24.99 dB. The anatomical structures of the corrected images and the motion-free brain data were similar.

A Systematic Review and Identification of the Challenges of Deep Learning Techniques for Undersampled Magnetic Resonance Image Reconstruction.

Deep learning (DL) in magnetic resonance imaging (MRI) shows excellent performance in image reconstruction from undersampled k-space data. Artifact-free and high-quality MRI reconstruction is essential for ensuring accurate diagnosis, supporting clinical decision-making, enhancing patient safety, facilitating efficient workflows, and contributing to the validity of research studies and clinical trials. Recently, deep learning has demonstrated several advantages over conventional MRI reconstruction methods. Conventional methods rely on manual feature engineering to capture complex patterns and are usually computationally demanding due to their iterative nature. Conversely, DL methods use neural networks with hundreds of thousands of parameters and automatically learn relevant features and representations directly from the data. Nevertheless, there are some limitations to DL-based techniques concerning MRI reconstruction tasks, such as the need for large, labeled datasets, the possibility of overfitting, and the complexity of model training. Researchers are striving to develop DL models that are more efficient, adaptable, and capable of providing valuable information for medical practitioners. We provide a comprehensive overview of the current developments and clinical uses by focusing on state-of-the-art DL architectures and tools used in MRI reconstruction. This study has three objectives. Our main objective is to describe how various DL designs have changed over time and talk about cutting-edge tactics, including their advantages and disadvantages. Hence, data pre- and post-processing approaches are assessed using publicly available MRI datasets and source codes. Secondly, this work aims to provide an extensive overview of the ongoing research on transformers and deep convolutional neural networks for rapid MRI reconstruction. Thirdly, we discuss several network training strategies, like supervised, unsupervised, transfer learning, and federated learning for rapid and efficient MRI reconstruction. Consequently, this article provides significant resources for future improvement of MRI data pre-processing and fast image reconstruction.

Comprehensive High-Quality Three-Dimensional Display System Based on a Simplified Light-Field Image Acquisition Method and a Full-Connected Deep Neural Network.

We propose a high-quality, three-dimensional display system based on a simplified light field image acquisition method, and a custom-trained full-connected deep neural network is proposed. The ultimate goal of the proposed system is to acquire and reconstruct the light field images with possibly the most elevated quality from the real-world objects in a general environment. A simplified light field image acquisition method acquires the three-dimensional information of natural objects in a simple way, with high-resolution/high-quality like multicamera-based methods. We trained a full-connected deep neural network model to output desired viewpoints of the object with the same quality. The custom-trained instant neural graphics primitives model with hash encoding output the overall desired viewpoints of the object within the acquired viewing angle in the same quality, based on the input perspectives, according to the pixel density of a display device and lens array specifications within the significantly short processing time. Finally, the elemental image array was rendered through the pixel re-arrangement from the entire viewpoints to visualize the entire field-of-view and re-constructed as a high-quality three-dimensional visualization on the integral imaging display. The system was implemented successfully, and the displayed visualizations and corresponding evaluated results confirmed that the proposed system offers a simple and effective way to acquire light field images from real objects with high-resolution and present high-quality three-dimensional visualization on the integral imaging display system.

A Hybrid Residual Attention Convolutional Neural Network for Compressed Sensing Magnetic Resonance Image Reconstruction.

We propose a dual-domain deep learning technique for accelerating compressed sensing magnetic resonance image reconstruction. An advanced convolutional neural network with residual connectivity and an attention mechanism was developed for frequency and image domains. First, the sensor domain subnetwork estimates the unmeasured frequencies of k-space to reduce aliasing artifacts. Second, the image domain subnetwork performs a pixel-wise operation to remove blur and noisy artifacts. The skip connections efficiently concatenate the feature maps to alleviate the vanishing gradient problem. An attention gate in each decoder layer enhances network generalizability and speeds up image reconstruction by eliminating irrelevant activations. The proposed technique reconstructs real-valued clinical images from sparsely sampled k-spaces that are identical to the reference images. The performance of this novel approach was compared with state-of-the-art direct mapping, single-domain, and multi-domain methods. With acceleration factors (AFs) of 4 and 5, our method improved the mean peak signal-to-noise ratio (PSNR) to 8.67 and 9.23, respectively, compared with the single-domain Unet model; similarly, our approach increased the average PSNR to 3.72 and 4.61, respectively, compared with the multi-domain W-net. Remarkably, using an AF of 6, it enhanced the PSNR by 9.87 ± 1.55 and 6.60 ± 0.38 compared with Unet and W-net, respectively.

High-Quality 3D Visualization System for Light-Field Microscopy with Fine-Scale Shape Measurement through Accurate 3D Surface Data.

We propose a light-field microscopy display system that provides improved image quality and realistic three-dimensional (3D) measurement information. Our approach acquires both high-resolution two-dimensional (2D) and light-field images of the specimen sequentially. We put forward a matting Laplacian-based depth estimation algorithm to obtain nearly realistic 3D surface data, allowing the calculation of depth data, which is relatively close to the actual surface, and measurement information from the light-field images of specimens. High-reliability area data of the focus measure map and spatial affinity information of the matting Laplacian are used to estimate nearly realistic depths. This process represents a reference value for the light-field microscopy depth range that was not previously available. A 3D model is regenerated by combining the depth data and the high-resolution 2D image. The element image array is rendered through a simplified direction-reversal calculation method, which depends on user interaction from the 3D model and is displayed on the 3D display device. We confirm that the proposed system increases the accuracy of depth estimation and measurement and improves the quality of visualization and 3D display images.

Depth Estimation for Integral Imaging Microscopy Using a 3D-2D CNN with a Weighted Median Filter.

This study proposes a robust depth map framework based on a convolutional neural network (CNN) to calculate disparities using multi-direction epipolar plane images (EPIs). A combination of three-dimensional (3D) and two-dimensional (2D) CNN-based deep learning networks is used to extract the features from each input stream separately. The 3D convolutional blocks are adapted according to the disparity of different directions of epipolar images, and 2D-CNNs are employed to minimize data loss. Finally, the multi-stream networks are merged to restore the depth information. A fully convolutional approach is scalable, which can handle any size of input and is less prone to overfitting. However, there is some noise in the direction of the edge. A weighted median filtering (WMF) is used to acquire the boundary information and improve the accuracy of the results to overcome this issue. Experimental results indicate that the suggested deep learning network architecture outperforms other architectures in terms of depth estimation accuracy.

De-Aliasing and Accelerated Sparse Magnetic Resonance Image Reconstruction Using Fully Dense CNN with Attention Gates.

When sparsely sampled data are used to accelerate magnetic resonance imaging (MRI), conventional reconstruction approaches produce significant artifacts that obscure the content of the image. To remove aliasing artifacts, we propose an advanced convolutional neural network (CNN) called fully dense attention CNN (FDA-CNN). We updated the Unet model with the fully dense connectivity and attention mechanism for MRI reconstruction. The main benefit of FDA-CNN is that an attention gate in each decoder layer increases the learning process by focusing on the relevant image features and provides a better generalization of the network by reducing irrelevant activations. Moreover, densely interconnected convolutional layers reuse the feature maps and prevent the vanishing gradient problem. Additionally, we also implement a new, proficient under-sampling pattern in the phase direction that takes low and high frequencies from the k-space both randomly and non-randomly. The performance of FDA-CNN was evaluated quantitatively and qualitatively with three different sub-sampling masks and datasets. Compared with five current deep learning-based and two compressed sensing MRI reconstruction techniques, the proposed method performed better as it reconstructed smoother and brighter images. Furthermore, FDA-CNN improved the mean PSNR by 2 dB, SSIM by 0.35, and VIFP by 0.37 compared with Unet for the acceleration factor of 5.

High-quality 3D display system for an integral imaging microscope using a simplified direction-inversed computation based on user interaction.

We propose and implement a high-quality three-dimensional (3D) display system for an integral imaging microscope using a simplified direction-inversed computation method based on user interaction. A model of the specimen is generated from the estimated depth information (via the convolutional neural network-based algorithm), the quality of the model is defined by the high-resolution two-dimensional image. The new elemental image arrays are generated from the models via a simplified direction-inversed computation method according to the user interaction and directly displayed on the display device. A high-quality 3D visualization of the specimen is reconstructed and displayed while the lens array is placed in front of the display device. The user interaction enables more viewpoints of the specimen to be reconstructed by the proposed system, within the basic viewing zone. Remarkable quality improvement is confirmed through quantitative evaluations of the experimental results.

Three-dimensional see-through augmented-reality display system using a holographic micromirror array.

It is difficult to find the micromirror array with desired specifications for augmented-reality displays, and the custom fabricating methods are complicated and unstable. We propose a novel, to our knowledge, three-dimensional see-through augmented-reality display system using the holographic micromirror array. Unlike the conventional holographic waveguide-type augmented-reality displays, the proposed system utilizes the holographic micromirror array as an in-coupler, without any additional elements. The holographic micromirror array is fabricated through the simple, effective, and stable method of applying the total internal reflection-based hologram recording using a dual-prism. The optical mirror and microlens array are set as references, and the specifications can be customized. It reconstructs a three-dimensional image from a displayed elemental image set without using any additional device, and the user can observe a three-dimensional virtual image while viewing the real-world objects. Thus, the principal advantages of the existing holographic waveguide-type augmented-reality system are retained. An optical experiment confirmed that the proposed system displays three-dimensional images exploiting the augmented-reality system simply and effectively.

Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy.

The integral imaging microscopy system provides a three-dimensional visualization of a microscopic object. However, it has a low-resolution problem due to the fundamental limitation of the F-number (the aperture stops) by using micro lens array (MLA) and a poor illumination environment. In this paper, a generative adversarial network (GAN)-based super-resolution algorithm is proposed to enhance the resolution where the directional view image is directly fed as input. In a GAN network, the generator regresses the high-resolution output from the low-resolution input image, whereas the discriminator distinguishes between the original and generated image. In the generator part, we use consecutive residual blocks with the content loss to retrieve the photo-realistic original image. It can restore the edges and enhance the resolution by ×2, ×4, and even ×8 times without seriously hampering the image quality. The model is tested with a variety of low-resolution microscopic sample images and successfully generates high-resolution directional view images with better illumination. The quantitative analysis shows that the proposed model performs better for microscopic images than the existing algorithms.

Diffraction efficiency enhancement and optimization in full-color HOE using the inhibition characteristics of the photopolymer.

A novel and effective simultaneous recording method, to the best of our knowledge, is proposed for improving the diffraction efficiency and uniformity of full-color holographic optical elements (HOE) using the Bayfol HX102 photopolymer. To improve the diffraction efficiency of a full-color HOE, it is important to find the optimal recording beam intensity taking into account the initial and late responses of the medium. The range of optimal beam intensity for recording full-color HOE can be found experimentally by analyzing the inhibition period and response characteristics of the recording medium for three wavelengths. Through this method, a full-color HOE with an average diffraction efficiency of about 56.81% and a standard deviation of about 1.7% was implemented in a single layer photopolymer.

Improving the quality of full-color holographic three-dimensional displays using depth-related multiple wavefront recording planes with uniform active areas.

In this paper, a depth-related uniform multiple wavefront recording plane (UM-WRP) method is proposed for enhancing the image quality of point cloud-based holograms. Conventional multiple WRP methods, based on full-color computer-generated holograms, experience a color uniformity problem caused by intensity distributions. To solve this problem, the proposed method generates depth-related WRPs to enhance color uniformity, thereby accelerating hologram generation using a uniform active area. The aim is to calculate depth-related WRPs with designed active area sizes that then propagate to the hologram. Compared with conventional multiple WRP methods, reconstructed images have significantly improved quality, as confirmed by numerical simulations and optical experiments.

Max-depth-range technique for faster full-color hologram generation.

In this paper, a max-depth-range method is proposed to determine the optimum length of depth range for faster generation of full-color holograms. For each color channel, objects are divided by a fixed length to create a temporary depth range, and the wavefront recording plane (WRP) is placed in the middle of all layers within the temporary depth range. The proposed method is used to calculate full-color holograms significantly faster than a conventional multiple-WRP method but with almost the same reconstructed image quality. The feasibility of the proposed method was confirmed using numerical and optical experiments for various scenes containing multiple real objects.

Trajectory-Based Air-Writing Recognition Using Deep Neural Network and Depth Sensor.

Trajectory-based writing system refers to writing a linguistic character or word in free space by moving a finger, marker, or handheld device. It is widely applicable where traditional pen-up and pen-down writing systems are troublesome. Due to the simple writing style, it has a great advantage over the gesture-based system. However, it is a challenging task because of the non-uniform characters and different writing styles. In this research, we developed an air-writing recognition system using three-dimensional (3D) trajectories collected by a depth camera that tracks the fingertip. For better feature selection, the nearest neighbor and root point translation was used to normalize the trajectory. We employed the long short-term memory (LSTM) and a convolutional neural network (CNN) as a recognizer. The model was tested and verified by the self-collected dataset. To evaluate the robustness of our model, we also employed the 6D motion gesture (6DMG) alphanumeric character dataset and achieved 99.32% accuracy which is the highest to date. Hence, it verifies that the proposed model is invariant for digits and characters. Moreover, we publish a dataset containing 21,000 digits; which solves the lack of dataset in the current research.

Implementation of full-color holographic system using non-uniformly sampled 2D images and compressed point cloud gridding.

A multiple-camera holographic system using non-uniformly sampled 2D images and compressed point cloud gridding (C-PCG) is suggested. High-quality, digital single-lens reflex cameras are used to acquire the depth and color information from real scenes; these are then virtually reconstructed by the uniform point cloud using a non-uniform sampling method. The C-PCG method is proposed to generate efficient depth grids by classifying groups of object points with the same depth values in the red, green, and blue channels. Holograms are obtained by applying fast Fourier transform diffraction calculations to the grids. Compared to wave-front recording plane methods, the quality of the reconstructed images is substantially better, and the computational complexity is dramatically reduced. The feasibility of our method is confirmed both numerically and optically.

Fast-switching laterally virtual-moving microlens array for enhancing spatial resolution in light-field imaging system without degradation of angular sampling resolution.

We present an electrically controllable fast-switching virtual-moving microlens array (MLA) consisting of a stacked structure of two polarization-dependent microlens arrays (PDMLAs) with optical orthogonality, where the position of the two stacked PDMLAs is shifted by half the elemental pitch in the diagonal direction. By controlling the polarization of the incident light without the physical movement of the molecules comprising the virtual-moving MLA, the periodic sampling position of the MLA can be switched fast using a polarization-switching layer based on a fast-switching liquid crystal cell. Using the fast-switching virtual-moving MLA, the spatial-resolution-enhanced light-field (LF) imaging system was demonstrated without a decrease in the angular sampling resolution as compared to the conventional LF imaging system comprising a passive MLA; two sets of elemental image arrays were captured quickly owing to the short switching time of the virtual-moving MLA of 450 µs. From the two captured sets of the elemental array image, four-times resolution-enhanced reconstruction images of the directional-view and depth-slice images could be obtained.

Chromatic-dispersion-corrected full-color holographic display using directional-view image scaling method.

A novel directional-view image scaling method that corrects chromatic dispersion and enhances the quality of three-dimensional (3D) images reconstructed by a full-color holographic display system is proposed. When the 3D information of the real scene is acquired through the integral imaging pickup method, the orthographic projection image is reconstructed. Then, each directional-view image is separated and synthesized onto the computer-generated hologram. To correct the chromatic dispersion of the full-color holographic 3D display, each directional-view image is scaled depending on the relation between the different wavelengths of single-channel holograms and resolutions of the sub-holograms. According to the optical experimental results, it can be concluded that the proposed method is an effective way of producing full-color holographic images from an orthographic projection image through a simple process.

Quality enhancement and GPU acceleration for a full-color holographic system using a relocated point cloud gridding method.

The calculation of realistic full-color holographic displays is hindered by the high computational cost. Previously, we suggested a point cloud gridding (PCG) method to calculate monochrome holograms of real objects. In this research, a relocated point cloud gridding (R-PCG) method is proposed to enhance the reconstruction quality and accelerate the calculation speed in GPU for a full-color holographic system. We use a depth camera to acquire depth and color information from the real scene then reconstruct the point cloud model virtually. The R-PCG method allows us to classify groups of object points with the same depth values into grids in the red, green, and blue (RGB) channels. Computer-generated holograms (CGHs) are obtained by applying a fast Fourier transform (FFT) diffraction calculation to the grids. The feasibility of the R-PCG method is confirmed by numerical and optical reconstruction.

Three-dimensional visualization system for ophthalmic microscopes using visible light and near-infrared illumination.

In this paper, we describe a three-dimensional visualization system for ophthalmic microscopes that is aimed at microsurgery without the eyepieces. A three-dimensional visualization system for ophthalmic microscopes using the mixed illumination, which consists of visible light and near-infrared illumination, is established in order to acquire more exact information of object and reduce the amount of light irradiated to the patients, and its usage in microsurgery without eyepieces is herein described. A custom-designed stereoscopic three-dimensional display which is manufactured for the convenience of the surgeons during the long-time surgery, is connected directly to the camera of the ophthalmic microscope in order to eliminate the discomfort of eyepieces to the surgeon and signal delay between the camera, mounted on the microscope, and display device for surgeon. The main features of the established system are the signal delay-free for surgeon and the low level of illumination for patient. In particular, it could significantly reduce the amount of light irradiated on a patient's eye via NIR illumination. Upon comparison with the conventional system during clinical ophthalmology trials, this system is confirmed to require almost the same operation time and reduced discomfort and eyestrain during long periods of observation.

Enhancement of the depth-of-field of integral imaging microscope by using switchable bifocal liquid-crystalline polymer micro lens array.

An integral imaging microscopy (IIM) system with improved depth-of-field (DoF) using a custom-designed bifocal polarization-dependent liquid-crystalline polymer micro lens array (LCP-MLA) is proposed. The implemented MLA has improved electro-optical properties such as a small focal ratio, high fill factor, low driving voltage, and fast switching speed, utilizing a well-aligned reactive mesogen on the imprinted reverse shape of the lens and a polarization switching layer. A bifocal MLA switches its focal length according to the polarization angle and acquires different DoF information of the specimen. After two elemental image arrays are captured, the depth-slices are reconstructed and combined to provide a widened DoF. The fabricated bifocal MLA consists of two identical polarization-dependent LCP-MLAs with 1.6 mm and f/16 focal ratio. Our experimental results confirmed that the proposed system improves the DoF of IIM without the need for mechanical manipulation.