An analysis framework for clustering algorithm selection with applications to spectroscopy.

Cluster analysis is a valuable unsupervised machine learning technique that is applied in a multitude of domains to identify similarities or clusters in unlabelled data. However, its performance is dependent of the characteristics of the data it is being applied to. There is no universally best clustering algorithm, and hence, there are numerous clustering algorithms available with different performance characteristics. This raises the problem of how to select an appropriate clustering algorithm for the given analytical purposes. We present and validate an analysis framework to address this problem. Unlike most current literature which focuses on characterizing the clustering algorithm itself, we present a wider holistic approach, with a focus on the user's needs, the data's characteristics and the characteristics of the clusters it may contain. In our analysis framework, we utilize a softer qualitative approach to identify appropriate characteristics for consideration when matching clustering algorithms to the intended application. These are used to generate a small subset of suitable clustering algorithms whose performance are then evaluated utilizing quantitative cluster validity indices. To validate our analysis framework for selecting clustering algorithms, we applied it to four different types of datasets: three datasets of homemade explosives spectroscopy, eight datasets of publicly available spectroscopy data covering food and biomedical applications, a gene expression cancer dataset, and three classic machine learning datasets. Each data type has discernible differences in the composition of the data and the context within which they are used. Our analysis framework, when applied to each of these challenges, recommended differing subsets of clustering algorithms for final quantitative performance evaluation. For each application, the recommended clustering algorithms were confirmed to contain the top performing algorithms through quantitative performance indices.

Spatially and angularly resolved spectroscopy for in-situ estimation of concentration and particle size in colloidal suspensions.

Successful implementation of process analytical technology (PAT) hinges on the ability to make continuous or frequent measurements in-line or at-line of critical product attributes such as composition and particle size, the latter being an important parameter for particulate processes such as suspensions and emulsions. A novel probe-based spatially and angularly-resolved diffuse reflectance measurement (SAR-DRM) system is proposed. This instrument, along with appropriate calibration models, is designed for online monitoring of concentration of chemical species and particle size of the particulate species in process systems involving colloidal suspensions. This measurement system was investigated using polystyrene suspensions of various particle radius and concentration to evaluate its performance in terms of the information obtained from the novel configuration which allows the measurement of a combination of incident light at different angles and collection fibres at different distances from the source fibres. Different strategies of processing and combining the SAR-DRM measurements were considered in terms of the impact on partial least squares (PLS) model performance. The results were compared with those obtained using a bench-top instrument which was used as the reference (off-line) instrument for comparison purposes. The SAR-DRM system showed similar performance to the bench top reference instrument for estimation of particle radius, and outperforms the reference instrument in estimating particle concentration. The investigation shows that the improvement in PLS regression model performance using the SAR-DRM system is related to the extra information captured by the SAR-DRM configuration. The differences in SAR-DRM spectra collected by the different collection fibres from different angular source fibres are the dominant reason for the significant improvement in the model performance. The promising results from this study suggest the potential of the SAR-DRM system as an online monitoring tool for processes involving suspensions. Graphical abstract A probe designed to acquire diffuse reflectance measurements at different source-detector distances for three incidence angles 0°, 30° and 45° was used to estimate particle size and concentration of polystyrene beads in aqueous suspension using partial least squares calibration models.

A Comparative Investigation of the Combined Effects of Pre-Processing, Wavelength Selection, and Regression Methods on Near-Infrared Calibration Model Performance.

Near-infrared (NIR) spectroscopy is being widely used in various fields ranging from pharmaceutics to the food industry for analyzing chemical and physical properties of the substances concerned. Its advantages over other analytical techniques include available physical interpretation of spectral data, nondestructive nature and high speed of measurements, and little or no need for sample preparation. The successful application of NIR spectroscopy relies on three main aspects: pre-processing of spectral data to eliminate nonlinear variations due to temperature, light scattering effects and many others, selection of those wavelengths that contribute useful information, and identification of suitable calibration models using linear/nonlinear regression . Several methods have been developed for each of these three aspects and many comparative studies of different methods exist for an individual aspect or some combinations. However, there is still a lack of comparative studies for the interactions among these three aspects, which can shed light on what role each aspect plays in the calibration and how to combine various methods of each aspect together to obtain the best calibration model. This paper aims to provide such a comparative study based on four benchmark data sets using three typical pre-processing methods, namely, orthogonal signal correction (OSC), extended multiplicative signal correction (EMSC) and optical path-length estimation and correction (OPLEC); two existing wavelength selection methods, namely, stepwise forward selection (SFS) and genetic algorithm optimization combined with partial least squares regression for spectral data (GAPLSSP); four popular regression methods, namely, partial least squares (PLS), least absolute shrinkage and selection operator (LASSO), least squares support vector machine (LS-SVM), and Gaussian process regression (GPR). The comparative study indicates that, in general, pre-processing of spectral data can play a significant role in the calibration while wavelength selection plays a marginal role and the combination of certain pre-processing, wavelength selection, and nonlinear regression methods can achieve superior performance over traditional linear regression-based calibration.

Alternative Measurement Configurations for Extracting Bulk Optical Properties Using an Integrating Sphere Setup.

The usual approach for estimating bulk optical properties using an integrating sphere measurement setup is by acquiring spectra from three measurement modes namely collimated transmittance (Tc), total transmittance (Td), and total diffuse reflectance (Rd), followed by the inversion of these measurements using the adding-doubling method. At high scattering levels, accurate acquisition of Tc becomes problematic due to the presence of significant amounts of forward-scattered light in this measurement which is supposed to contain only unscattered light. In this paper, we propose and investigate the effectiveness of using alternative sets of integrating sphere measurements that avoid the use of Tc and could potentially increase the upper limit of concentrations of suspensions at which bulk optical property measurements can be obtained in the visible-near-infrared (Vis-NIR) region of the spectrum. We examine the possibility of replacing Tc with one or more reflectance measurements at different sample thicknesses. We also examine the possibility of replacing both the collimated (Tc) and total transmittance (Td) measurements with reflectance measurements taken from different sample thicknesses. The analysis presented here indicates that replacing Tc with a reflectance measurement can reduce the errors in the bulk scattering properties when scattering levels are high. When only multiple reflectance measurements are used, good estimates of the bulk optical properties can be obtained when the absorption levels are low. In addition, we examine whether there is any advantage in using three measurements instead of two to obtain the reduced bulk scattering coefficient and the bulk absorption coefficient. This investigation is made in the context of chemical and biological suspensions which have a much larger range of optical properties compared to those encountered with tissue.

Full correction of scattering effects by using the radiative transfer theory for improved quantitative analysis of absorbing species in suspensions.

Sample-to-sample photon path length variations that arise due to multiple scattering can be removed by decoupling absorption and scattering effects by using the radiative transfer theory, with a suitable set of measurements. For samples where particles both scatter and absorb light, the extracted bulk absorption spectrum is not completely free from nonlinear particle effects, since it is related to the absorption cross-section of particles that changes nonlinearly with particle size and shape. For the quantitative analysis of absorbing-only (i.e., nonscattering) species present in a matrix that contains a particulate species that absorbs and scatters light, a method to eliminate particle effects completely is proposed here, which utilizes the particle size information contained in the bulk scattering coefficient extracted by using the Mie theory to carry out an additional correction step to remove particle effects from bulk absorption spectra. This should result in spectra that are equivalent to spectra collected with only the liquid species in the mixture. Such an approach has the potential to significantly reduce the number of calibration samples as well as improve calibration performance. The proposed method was tested with both simulated and experimental data from a four-component model system.

Insights into information contained in multiplicative scatter correction parameters and the potential for estimating particle size from these parameters.

This paper investigates the nature of information contained in scatter correction parameters. The study had two objectives. The first objective was to examine the nature and extent of information contained in scatter correction parameters. The second objective is to examine whether this information can be effectively extracted by proposing a method to obtain particularly the mean particle diameter from the scatter correction parameters. By using a combination of experimental data and simulated data generated using fundamental light propagation theory, a deeper and more fundamental insight of what information is removed by the multiplicative scatter correction (MSC) method is obtained. It was found that the MSC parameters are strongly influenced not only by particle size but also by particle concentration as well as refractive index of the medium. The possibility of extracting particle size information in addition to particle concentration was considered by proposing a two-step method which was tested using a 2-component and 4-component data set. This method can in principle, be used in conjunction with any scatter correction technique provided that the scatter correction parameters exhibit a systematic dependence with respect to particle size and concentration. It was found that the approach which uses the MSC parameters gave a better estimate of the particle diameter compared to using partial least squares (PLS) regression for the 2-component data. For the 4 component data it was found that PLS regression gave better results but further examination indicated this was due to chance correlations of the particle diameter with the two of the absorbing species in the mixture.

Quantitative spectroscopic analysis of heterogeneous mixtures: the correction of multiplicative effects caused by variations in physical properties of samples.

Spectral measurements of complex heterogeneous types of mixture samples are often affected by significant multiplicative effects resulting from light scattering, due to physical variations (e.g., particle size and shape, sample packing, and sample surface, etc.) inherent within the individual samples. Therefore, the separation of the spectral contributions due to variations in chemical compositions from those caused by physical variations is crucial to accurate quantitative spectroscopic analysis of heterogeneous samples. In this work, an improved strategy has been proposed to estimate the multiplicative parameters accounting for multiplicative effects in each measured spectrum and, hence, mitigate the detrimental influence of multiplicative effects on the quantitative spectroscopic analysis of heterogeneous samples. The basic assumption of the proposed method is that light scattering due to physical variations has the same effects on the spectral contributions of each of the spectroscopically active chemical components in the same sample mixture. On the basis of this underlying assumption, the proposed method realizes the efficient estimation of the multiplicative parameters by solving a simple quadratic programming problem. The performance of the proposed method has been tested on two publicly available benchmark data sets (i.e., near-infrared total diffuse transmittance spectra of four-component suspension samples and near-infrared spectral data of meat samples) and compared with some empirical approaches designed for the same purpose. It was found that the proposed method provided appreciable improvement in quantitative spectroscopic analysis of heterogeneous mixture samples. The study indicates that accurate quantitative spectroscopic analysis of heterogeneous mixture samples can be achieved through the combination of spectroscopic techniques with smart modeling methodology.

Effect of sample thickness on the extracted near-infrared bulk optical properties of Bacillus subtilis in liquid culture.

In order to determine the bulk optical properties of a Bacillus subtilis culture during growth phase we investigated the effect of sample thickness on measurements taken with different measurement configurations, namely total diffuse reflectance and total diffuse transmittance. The bulk optical properties were extracted by inverting the measurements using the radiative transfer theory. While the relationship between reflectance and biomass changes with sample thickness and the intensity (absorbance) levels vary significantly for both reflectance and transmittance measurements, the extracted optical properties show consistent behavior in terms of both the relationship with biomass and magnitude. This observation indicates the potential of bulk optical properties for building models that could be more easily transferable compared to those built using raw measurements.

Extraction of chemical information of suspensions using radiative transfer theory to remove multiple scattering effects: application to a model multicomponent system.

The effectiveness of a scatter correction approach based on decoupling absorption and scattering effects through the use of the radiative transfer theory to invert a suitable set of measurements is studied by considering a model multicomponent suspension. The method was used in conjunction with partial least-squares regression to build calibration models for estimating the concentration of two types of analytes: an absorbing (nonscattering) species and a particulate (absorbing and scattering) species. The performances of the models built by this approach were compared with those obtained by applying empirical scatter correction approaches to diffuse reflectance, diffuse transmittance, and collimated transmittance measurements. It was found that the method provided appreciable improvement in model performance for the prediction of both types of analytes. The study indicates that, as long as the bulk absorption spectra are accurately extracted, no further empirical preprocessing to remove light scattering effects is required.

Extraction of chemical information of suspensions using radiative transfer theory to remove multiple scattering effects: application to a model two-component system.

An approach for removing multiple light scattering effects using the radiative transfer theory (RTE) in order to improve the performance of multivariate calibration models is proposed. This approach is then applied to the problem of building calibration models for predicting the concentration of a scattering (particulate) component. Application of this approach to a simulated four component system showed that it will lead to calibration models which perform appreciably better than when empirically scatter corrected measurements of diffuse transmittance (T(d)) or reflectance (R(d)) are used. The validity of the method was also tested experimentally using a two-component (polystyrene-water) system. While the proposed method led to a model that performed better than the one built using R(d), its performance was worse compared to when T(d) measurements were used. Analysis indicates that this is because the model built using T(d) benefits from the strong secondary correlation between particle concentration and path length traveled by the photons which occurs due to the system containing only two components. On the other hand, the model arising from the proposed methodology uses essentially only the chemical (polystyrene) signal. Thus, this approach can be expected to work better in multicomponent systems where the path length correlation would not exist.

Changes in the absorption and scattering properties in the near-infrared region during the growth of Bacillus subtilis in liquid culture.

Multiple scattering of light by cells poses a significant challenge in the development of near-infrared-based methodologies to reliably extract chemical and physical information contained in the spectra collected during the bacterial growth cycle. The extent of information that can be obtained from NIR spectra could, in principle, be vastly improved if the scattering and absorption effects can be effectively separated. This study focuses on the methodology for extracting the bulk optical properties over the course of the bacterial growth cycle and investigates the nature and extent of changes in the optical properties with time. By inverting the radiative transfer equation (RTE) using three measurements, total diffuse reflectance, total diffuse transmittance, and collimated transmittance, the bulk absorption coefficient (microa), the bulk scattering coefficient (micros), and the anisotropy factor (g) are extracted and their changes during the course of the growth cycle are investigated. In this study, a simple bacterial growth system consisting of Bacillus subtilis growing in an aqueous solution (minimum medium) was investigated. The changes in the optical properties of this system during bacterial growth, stationary, and decline phases were investigated by inverting the measurements using the adding-doubling method to solve the RTE in the wavelength region of 950-1850 nm. This study shows that during growth in liquid culture, the absorption and scattering property changes can be consistently extracted from measurements under multiple light scattering conditions. The estimation of the anisotropy factor was not reliable beyond 1200 nm at low bacterial cell counts, but reliability increased with increasing biomass concentration. At all stages in the growth cycle, the anisotropy factor could not be reliably extracted in the first overtone region. However, this does not appear to adversely affect the estimation of the absorption and scattering coefficients.

Complex refractive index of nonspherical particles in the visible near infrared region: application to Bacillus subtilis spores.

A method is presented for the estimation of optical constants in the ultraviolet-visible-near-infrared (UV-Vis-NIR) region of nonspherical particles in a suspension at concentrations where multiple scattering is significant. The optical constants are obtained by an inversion technique using the adding-doubling method to solve the radiative transfer equation in combination with the single scattering theories for modelling scattering by nonspherical particles. Two methods for describing scattering by single scattering are considered: the T-matrix method and the approximate but computationally simpler Rayleigh-Gans-Debye (RGD) approximation. The method is then applied to obtain the optical constants of Bacillus subtilis spores in the wavelength region 400-1200 nm. It is found that the optical constants obtained using the RGD approximation matches those obtained using the T-matrix method to within experimental error.

Relationship between the Kubelka-Munk scattering and radiative transfer coefficients.

The relationship between the Kubelka-Munk (K-M) and the transport scattering coefficient is obtained through a semi-empirical approach. This approach gives the same result as that given by Gate [Appl. Opt.13, 236 (1974)] when the incident beam is diffuse. This result and those given by Star et al. [Phys. Med. Biol.33, 437 (1988)] and Brinkworth [Appl. Opt.11, 1434 (1972)] are compared with the exact solution of the radiative transfer equation over a large range of optical properties. It is found that the latter expressions, which include an absorption component, do not give accurate results over the range considered. Using the semi-empirical approach, the relationship between the K-M and the transport scattering coefficient is derived for the case where the incident light is collimated. It is shown that although the K-M equation is derived based on diffuse incident light, it can also represent very well the reflectance from a slab of infinite thickness when the incident light is collimated. However, in this case the relationship between the coefficients has to include a function that is dependent on the anisotropy factor. Analysis indicates that the K-M transform achieves the objective of obtaining a measure that gives the ratio of absorption to scattering effects for both diffuse and collimated incident beams over a large range of optical properties.

Optical properties of apple skin and flesh in the wavelength range from 350 to 2200 nm.

Optical measurement of fruit quality is challenging due to the presence of a skin around the fruit flesh and the multiple scattering by the structured tissues. To gain insight in the light-tissue interaction, the optical properties of apple skin and flesh tissue are estimated in the 350-2200 nm range for three cultivars. For this purpose, single integrating sphere measurements are combined with inverse adding-doubling. The observed absorption coefficient spectra are dominated by water in the near infrared and by pigments and chlorophyll in the visible region, whose concentrations are much higher in skin tissue. The scattering coefficient spectra show the monotonic decrease with increasing wavelength typical for biological tissues with skin tissue being approximately three times more scattering than flesh tissue. Comparison to the values from time-resolved spectroscopy reported in literature showed comparable profiles for the optical properties, but overestimation of the absorption coefficient values, due to light losses.

Estimation of optical constants from multiple-scattered light using approximations for single particle scattering characteristics.

The inversion of multiple-scattered light measurements to extract the optical constant (complex refractive index) is computationally intensive. A significant portion of this time is due to the effort required for computing the single particle characteristics (absorption and scattering cross sections, anisotropy factor, and the phase function). We investigate approximations for computing these characteristics so as to significantly speed up the calculations without introducing large inaccuracies. Two suspensions of spherical particles viz., polystyrene and poly(methyl methacrylate) were used for this investigation. It was found that using the exact Mie theory to compute the absorption and scattering cross sections and the anisotropy factor with the phase function computed using the Henyey-Greenstein approximation yielded the best results. Analysis suggests that errors in the phase functions and thus in the estimated optical constants depend mainly on how closely the approximations match the Mie phase function at small scattering angles.