Spectral Reconstruction from RGB Imagery: A Potential Option for Infinite Spectral Data?

Spectral imaging has revolutionisedvarious fields by capturing detailed spatial and spectral information. However, its high cost and complexity limit the acquisition of a large amount of data to generalise processes and methods, thus limiting widespread adoption. To overcome this issue, a body of the literature investigates how to reconstruct spectral information from RGB images, with recent methods reaching a fairly low error of reconstruction, as demonstrated in the recent literature. This article explores the modification of information in the case of RGB-to-spectral reconstruction beyond reconstruction metrics, with a focus on assessing the accuracy of the reconstruction process and its ability to replicate full spectral information. In addition to this, we conduct a colorimetric relighting analysis based on the reconstructed spectra. We investigate the information representation by principal component analysis and demonstrate that, while the reconstruction error of the state-of-the-art reconstruction method is low, the nature of the reconstructed information is different. While it appears that the use in colour imaging comes with very good performance to handle illumination, the distribution of information difference between the measured and estimated spectra suggests that caution should be exercised before generalising the use of this approach.

Improving RGB illuminant estimation exploiting spectral average radiance.

We introduce a method that enhances RGB color constancy accuracy by combining neural network and k-means clustering techniques. Our approach stands out from previous works because we combine multispectral and color information together to estimate illuminants. Furthermore, we investigate the combination of the illuminant estimation in the RGB color and in the spectral domains, as a strategy to provide a refined estimation in the RGB color domain. Our investigation can be divided into three main points: (1) identify the spatial resolution for sampling the input image in terms of RGB color and spectral information that brings the highest performance; (2) determine whether it is more effective to predict the illuminant in the spectral or in the RGB color domain, and finally, (3) assuming that the illuminant is in fact predicted in the spectral domain, investigate if it is better to have a loss function defined in the RGB color or spectral domain. Experimental results are carried out on NUS: a standard dataset of multispectral radiance images with an annotated spectral global illuminant. Among the several considered options, the best results are obtained with a model trained to predict the illuminant in the spectral domain using an RGB color loss function. In terms of comparison with the state of the art, this solution improves the recovery angular error metric by 66% compared to the best tested spectral method, and by 41% compared to the best tested RGB method.

Optical and Electromechanical Design and Implementation of an Advanced Multispectral Device to Capture Material Appearance.

The application of materials with changing visual properties with lighting and observation directions has found broad utility across diverse industries, from architecture and fashion to automotive and film production. The expanding array of applications and appearance reproduction requirements emphasizes the critical role of material appearance measurement and surface characterization. Such measurements offer twofold benefits in soft proofing and product quality control, reducing errors and material waste while providing objective quality assessment. Some image-based setups have been proposed to capture the appearance of material surfaces with spatial variations in visual properties in terms of Spatially Varying Bidirectional Reflectance Distribution Functions (SVBRDF) and Bidirectional Texture Functions (BTF). However, comprehensive exploration of optical design concerning spectral channels and per-pixel incident-reflection direction calculations, along with measurement validation, remains an unexplored domain within these systems. Therefore, we developed a novel advanced multispectral image-based device designed to measure SVBRDF and BTF, addressing these gaps in the existing literature. Central to this device is a novel rotation table as sample holder and passive multispectral imaging. In this paper, we present our compact multispectral image-based appearance measurement device, detailing its design, assembly, and optical considerations. Preliminary measurements showcase the device's potential in capturing angular and spectral data, promising valuable insights into material appearance properties.

Raw Spectral Filter Array Imaging for Scene Recognition.

Scene recognition is the task of identifying the environment shown in an image. Spectral filter array cameras allow for fast capture of multispectral images. Scene recognition in multispectral images is usually performed after demosaicing the raw image. Along with adding latency, this makes the classification algorithm limited by the artifacts produced by the demosaicing process. This work explores scene recognition performed on raw spectral filter array images using convolutional neural networks. For this purpose, a new raw image dataset is collected for scene recognition with a spectral filter array camera. The classification is performed using a model constructed based on the pretrained Places-CNN. This model utilizes all nine channels of spectral information in the images. A label mapping scheme is also applied to classify the new dataset. Experiments are conducted with different pre-processing steps applied on the raw images and the results are compared. Higher-resolution images are found to perform better even if they contain mosaic patterns.

Additivity Constrained Linearisation of Camera Calibration Data.

When characterising a digital camera spectrally or colourimetrically, the camera response to a generally diffusely reflecting colour chart is often employed. The recorded responses to the light incident from each colour patch are typically not linearly related to the power of the irradiance on the chart, and the irradiance varies with position on the chart. This necessitates a linearisation of the responses. We present a new single image colour chart-based estimation method of responses, that are linearly related to camera response values known as ground truth. The method estimates the spatial geometry of the irradiance incident on the chart attenuated by lens vignetting and compensates individually for volumetric and per colour channel non-linearities, including compensation for physical scene and camera properties in a pipeline of successive signal transformations between the estimated linear and the given recorded responses. The estimation is controlled by introducing a novel Additivity Principle of linear responses, which is derived from the spectral reflectances of the coloured surfaces on the colour chart, observing that linear relations of the spectral reflectances are equal to the relations of the corresponding linear responses. Crucially, the additivity principle is not subject to metamerism. The method is fundamentally solely reliant on a one-shot set of one triplet of response values sampled from each patch of a colour chart with known spectral reflectances, where rendition level, gray scale, illuminant, camera sensor curves, irradiance geometry, vignetting, moderate specular reflection, colour space, colour correction, gamut correction and noise level are unknown.

Development and validation of a hybrid simulator for ultrasound-guided laparoscopic common bile duct exploration.

BACKGROUND: Ultrasound-guided laparoscopic common bile duct exploration (LCBDE) is the surgical management of choledocholithiasis. The procedure presents significant benefits to patients but still fails to be generalised because of the complex set of skills it requires. A simulator for ultrasound-guided LCBDE would allow trainee surgeons as well as experienced surgeons who perform this surgery seldomly to practice and gain confidence. METHODS: This article presents the development and validation of an easily reproducible hybrid simulator for ultrasound-guided LCBDE which integrates real and virtual components of the task. We first developed a physical model made of silicone. The fabrication technique is replicable and allows quick and easy production of multiple models. We then applied virtual components onto the model to create training for laparoscopic ultrasound examination. Combined with a commercially available lap-trainer and surgical equipment, the model can be used for training the fundamental steps of the surgery through the trans-cystic or trans-choledochal approaches. The simulator was evaluated through face, content, and construct validation. RESULTS: Two novices, eight middle grades, and three experts were recruited to test the simulator. The results of the face validation showed that the surgeons found the model realistic visually and felt realistic when performing the different steps of the surgery. The content validation indicated the usefulness of having a training system to practice the choledochotomy, the choledochoscopy and stone retrieval, and the suturing. The construct validation highlighted the ability of the simulator to differentiate between surgeons with various levels of expertise. CONCLUSIONS: The hybrid simulator presented is a low-cost yet realistic model which allows the surgeons to practice the technical skills required for trans-cystic and trans-choledochal ultrasound-guided LCBDE.

On the Quantification of Visual Texture Complexity.

Complexity is one of the major attributes of the visual perception of texture. However, very little is known about how humans visually interpret texture complexity. A psychophysical experiment was conducted to visually quantify the seven texture attributes of a series of textile fabrics: complexity, color variation, randomness, strongness, regularity, repetitiveness, and homogeneity. It was found that the observers could discriminate between the textures with low and high complexity using some high-level visual cues such as randomness, color variation, strongness, etc. The results of principal component analysis (PCA) on the visual scores of the above attributes suggest that complexity and homogeneity could be essentially the underlying attributes of the same visual texture dimension, with complexity at the negative extreme and homogeneity at the positive extreme of this dimension. We chose to call this dimension visual texture complexity. Several texture measures including the first-order image statistics, co-occurrence matrix, local binary pattern, and Gabor features were computed for images of the textiles in sRGB, and four luminance-chrominance color spaces (i.e., HSV, YCbCr, Ohta's I1I2I3, and CIELAB). The relationships between the visually quantified texture complexity of the textiles and the corresponding texture measures of the images were investigated. Analyzing the relationships showed that simple standard deviation of the image luminance channel had a strong correlation with the corresponding visual ratings of texture complexity in all five color spaces. Standard deviation of the energy of the image after convolving with an appropriate Gabor filter and entropy of the co-occurrence matrix, both computed for the image luminance channel, also showed high correlations with the visual data. In this comparison, sRGB, YCbCr, and HSV always outperformed the I1I2I3 and CIELAB color spaces. The highest correlations between the visual data and the corresponding image texture features in the luminance-chrominance color spaces were always obtained for the luminance channel of the images, and one of the two chrominance channels always performed better than the other. This result indicates that the arrangement of the image texture elements that impacts the observer's perception of visual texture complexity cannot be represented properly by the chrominance channels. This must be carefully considered when choosing an image channel to quantify the visual texture complexity. Additionally, the good performance of the luminance channel in the five studied color spaces proves that variations in the luminance of the texture, or as one could call the luminance contrast, plays a crucial role in creating visual texture complexity.

Beyond Tristimulus Color Vision with Perovskite-Based Multispectral Sensors.

In this study, optical multispectral sensors based on perovskite semiconductors have been proposed, simulated, and characterized. The perovskite material system combined with the 3D vertical integration of the sensor channels allow for realizing sensors with high sensitivities and a high spectral resolution. The sensors can be applied in several emerging areas, including biomedical imaging, surveillance, complex motion planning of autonomous robots or vehicles, artificial intelligence, and agricultural applications. The sensor elements can be vertically integrated on a readout electronic to realize sensor arrays and multispectral digital cameras. In this study, three- and six-channel vertically stacked perovskite sensors are optically designed, electromagnetically simulated, and colorimetrically characterized to evaluate the color reproduction. The proposed sensors allow for the implementation of snapshot cameras with high sensitivity. The proposed sensor is compared to other sensor technologies in terms of sensitivity and selectivity.

Optimized light source spectral power distribution for RGB camera based spectral reflectance recovery.

The accuracy of recovered spectra from camera responses mainly depends on the spectral estimation algorithm used, the camera and filters selected, and the light source used to illuminate the object. We present and compare different light source spectrum optimization methods together with different spectral estimation algorithms applied to reflectance recovery. These optimization methods include the Monte Carlo (MC) method, particle swarm optimization (PSO) and multi-population genetic algorithm (MPGA). Optimized SPDs are compared with D65, D50 A and three LED light sources in simulation and reality. Results obtained show us that MPGA has superior performance, and optimized light source spectra along with better spectral estimation algorithm can provide a more accurate spectral reflectance estimation of an object surface. Meanwhile, it is found that camera spectral sensitivities weighted by optimized SPDs tend to be mutually orthogonal.

Translucency perception: A review.

Translucency is an optical and a perceptual phenomenon that characterizes subsurface light transport through objects and materials. Translucency as an optical property of a material relates to the radiative transfer inside and through this medium, and translucency as a perceptual phenomenon describes the visual sensation experienced by humans when observing a given material under given conditions. The knowledge about the visual mechanisms of the translucency perception remains limited. Accurate prediction of the appearance of the translucent objects can have a significant commercial impact in the fields such as three-dimensional printing. However, little is known how the optical properties of a material relate to a perception evoked in humans. This article overviews the knowledge status about the visual perception of translucency and highlights the applications of the translucency perception research. Furthermore, this review summarizes current knowledge gaps, fundamental challenges and existing ambiguities with a goal to facilitate translucency perception research in the future.

Colour-Balanced Edge-Guided Digital Inpainting: Applications on Artworks.

The virtual inpainting of artworks provides a nondestructive mode of hypothesis visualization, and it is especially attractive when physical restoration raises too many methodological and ethical concerns. At the same time, in Cultural Heritage applications, the level of details in virtual reconstruction and their accuracy are crucial. We propose an inpainting algorithm that is based on generative adversarial network, with two generators: one for edges and another one for colors. The color generator rebalances chromatically the result by enforcing a loss in the discretized gamut space of the dataset. This way, our method follows the modus operandi of an artist: edges first, then color palette, and, at last, color tones. Moreover, we simulate the stochasticity of the lacunae in artworks with morphological variations of a random walk mask that recreate various degradations, including craquelure. We showcase the performance of our model on a dataset of digital images of wall paintings from the Dunhuang UNESCO heritage site. Our proposals of restored images are visually satisfactory and they are quantitatively comparable to state-of-the-art approaches.

Hyperspectral Texture Metrology Based on Joint Probability of Spectral and Spatial Distribution.

Texture characterization from the metrological point of view is addressed in order to establish a physically relevant and directly interpretable feature. In this regard, a generic formulation is proposed to simultaneously capture the spectral and spatial complexity in hyperspectral images. The feature, named relative spectral difference occurrence matrix (RSDOM) is thus constructed in a multireference, multidirectional, and multiscale context. As validation, its performance is assessed in three versatile tasks. In texture classification on HyTexiLa, content-based image retrieval (CBIR) on ICONES-HSI, and land cover classification on Salinas, RSDOM registers 98.5% accuracy, 80.3% precision (for the top 10 retrieved images), and 96.0% accuracy (after post-processing) respectively, outcompeting GLCM, Gabor filter, LBP, SVM, CCF, CNN, and GCN. Analysis shows the advantage of RSDOM in terms of feature size (a mere 126, 30, and 20 scalars using GMM in order of the three tasks) as well as metrological validity in texture representation regardless of the spectral range, resolution, and number of bands.

ZCAM, a colour appearance model based on a high dynamic range uniform colour space.

A colour appearance model based on a uniform colour space is proposed. The proposed colour appearance model, ZCAM, comprises of comparatively simple mathematical equations, and plausibly agrees with the psychophysical phenomenon of colour appearance perception. ZCAM consists of ten colour appearance attributes including brightness, lightness, colourfulness, chroma, hue angle, hue composition, saturation, vividness, blackness, and whiteness. Despite its relatively simpler mathematical structure, ZCAM performed at least similar to the CIE standard colour appearance model CIECAM02 and its revision, CAM16, in predicting a range of reliable experimental data.

LED primary selection algorithms for simulation of CIE standard illuminants.

Although mixture of LEDs is being considered as a simulator of the CIE daylight series, the performance of the simulations is highly dependent on the SPD of the selected LEDs. An algorithm for selection of the best LEDs for simulation of the CIE daylight series is helpful in this regard. To address this problem, using 200 imaginary light primaries and 40 real LEDs, three algorithms based on concepts of equally spacing of wavelength range ("Equal"), Gram Schmidt orthogonalization in LEDs/light primaries spectral subspace ("Gram") and the generalization of Gram Schmidt orthogonalization in the LEDs/light primaries projections onto the illuminants subspace ("Ortho") were proposed and studied. Algorithms, in simulation and reality, were implemented for the CIE standard illuminants of D50, D55, D65 and D75, C and A. The results showed that the performance of the proposed algorithms generally increase with the higher number of selected LEDs/light primaries, while for the LEDs "Gram" and "Ortho" methods showed superior performance, simulations on the imaginary light primaries showed "Ortho" could be considered as the best algorithm.

Perovskite Color Detectors: Approaching the Efficiency Limit.

Color image sensing by a smartphone or digital camera employs sensor elements with an array of color filters for capturing basic blue, green, and red color information. However, the normalized optical efficiency of such color filter-based sensor elements is limited to only one-third. Optical detectors based on perovskites are described, which can overcome this limitation. An efficient color sensor design has been proposed in this study that uses a vertically stacked arrangement of perovskite diodes. As compared to the conventional color filter-based sensors, the proposed sensor structure can potentially reach normalized optical efficiency approaching 100%. In addition, the proposed sensor design does not exhibit color aliasing or color Moiré effects, which is one of the main limitations for the filter-based sensors. Furthermore, up to our knowledge, for the first time, it could be theoretically shown that both vertically arranged sensor and conventional color filter-based sensor provide almost comparable color errors. The optical properties of the perovskite materials are determined by optical measurements in combination with an energy shift model. The optics of the stacked perovskite sensors is investigated by threedimensional finite-difference timedomain simulations. Finally, colorimetric characterization was carried out to determine the color error of the sensors.

Evaluation of the Data Quality from a Round-Robin Test of Hyperspectral Imaging Systems.

In this study, the results from a round-robin test of hyperspectral imaging systems are presented and analyzed. Fourteen different pushbroom hyperspectral systems from eight different institutions were used to acquire spectral cubes from the visible, near infra-red and short-wave infra-red regions. Each system was used to acquire a common set of targets under their normal operating conditions with the data calibrated and processed using the standard processing pipeline for each system. The test targets consisted of a spectral wavelength standard and of a custom-made pigment panel featuring Renaissance-era pigments frequently found in paintings from that period. The quality and accuracy of the resulting data was assessed with quantitative analyses of the spectral, spatial and colorimetric accuracy of the data. The results provide a valuable insight into the accuracy, reproducibility and precision of hyperspectral imaging equipment when used under routine operating conditions. The distribution and type of error found within the data can provide useful information on the fundamental and practical limits of such equipment when used for applications such as spectral classification, change detection, colorimetry and others.

Perceptually Validated Cross-Renderer Analytical BRDF Parameter Remapping.

Material appearance of rendered objects depends on the underlying BRDF implementation used by rendering software packages. A lack of standards to exchange material parameters and data (between tools) means that artists in digital 3D prototyping and design, manually match the appearance of materials to a reference image. Since their effect on rendered output is often non-uniform and counter intuitive, selecting appropriate parameterisations for BRDF models is far from straightforward. We present a novel BRDF remapping technique, that automatically computes a mapping (BRDF Difference Probe) to match the appearance of a source material model to a target one. Through quantitative analysis, four user studies and psychometric scaling experiments, we validate our remapping framework and demonstrate that it yields a visually faithful remapping among analytical BRDFs. Most notably, our results show that even when the characteristics of the models are substantially different, such as in the case of a phenomenological model and a physically-based one, our remapped renderings are indistinguishable from the original source model.

An Evaluation Framework for Spectral Filter Array Cameras to Optimize Skin Diagnosis.

Comparing and selecting an adequate spectral filter array (SFA) camera is application-specific and usually requires extensive prior measurements. An evaluation framework for SFA cameras is proposed and three cameras are tested in the context of skin analysis. The proposed framework does not require application-specific measurements and spectral sensitivities together with the number of bands are the main focus. An optical model of skin is used to generate a specialized training set to improve spectral reconstruction. The quantitative comparison of the cameras is based on reconstruction of measured skin spectra, colorimetric accuracy, and oxygenation level estimation differences. Specific spectral sensitivity shapes influence the results directly and a 9-channel camera performed best regarding the spectral reconstruction metrics. Sensitivities at key wavelengths influence the performance of oxygenation level estimation the strongest. The proposed framework allows to compare spectral filter array cameras and can guide their application-specific development.

Multispectral camera as spatio-spectrophotometer under uncontrolled illumination.

Multispectral constancy enables the illuminant invariant representation of multi-spectral data. This article proposes an experimental investigation of multispectral constancy through the use of multispectral camera as a spectrophotometer for the reconstruction of surface reflectance. Three images with varying illuminations are captured and the spectra of material surfaces is reconstructed. The acquired images are transformed into canonical representation through the use of diagonal transform and spectral adaptation transform. Experimental results show that use of multispectral constancy is beneficial for both filter-wheel and snapshot multi-spectral cameras. The proposed concept is robust to errors in illuminant estimation and is able to perform well with linear spectral reconstruction method. This work makes us one step closer to the use of multispectral imaging for computer vision.

A Spectral Filter Array Camera for Clinical Monitoring and Diagnosis: Proof of Concept for Skin Oxygenation Imaging.

The emerging technology of spectral filter array (SFA) cameras has great potential for clinical applications, due to its unique capability for real time spectral imaging, at a reasonable cost. This makes such cameras particularly suitable for quantification of dynamic processes such as skin oxygenation. Skin oxygenation measurements are useful for burn wound healing assessment and as an indicator of patient complications in the operating room. Due to their unique design, in which all pixels of the image sensor are equipped with different optical filters, SFA cameras require specific image processing steps to obtain meaningful high quality spectral image data. These steps include spatial rearrangement, SFA interpolations and spectral correction. In this paper the feasibility of a commercially available SFA camera for clinical applications is tested. A suitable general image processing pipeline is proposed. As a 'proof of concept' a complete system for spatial dynamic skin oxygenation measurements is developed and evaluated. In a study including 58 volunteers, oxygenation changes during upper arm occlusion were measured with the proposed SFA system and compared with a validated clinical device for localized oxygenation measurements. The comparison of the clinical standard measurements and SFA results show a good correlation for the relative oxygenation changes. This proposed processing pipeline for SFA cameras shows to be effective for relative oxygenation change imaging. It can be implemented in real time and developed further for absolute spatial oxygenation measurements.