Fast and robust NIRS-based characterization of raw organic waste: Using non-linear methods to handle water effects.

Fast characterization of organic waste using near infrared spectroscopy (NIRS) has been successfully developed in the last decade. However, up to now, an on-site use of this technology has been hindered by necessary sample preparation steps (freeze-drying and grinding) to avoid important water effects on NIRS. Recent research studies have shown that these effects are highly non-linear and relate both to the biochemical and physical properties of samples. To account for these complex effects, the current study compares the use of many different types of non-linear methods such as partial least squares regression (PLSR) based methods (global, clustered and local versions of PLSR), machine learning methods (support vector machines, regression trees and ensemble methods) and deep learning methods (artificial and convolutional neural networks). On an independent test data set, non-linear methods showed errors 28% lower than linear methods. The standard errors of prediction obtained for the prediction of total solids content (TS%), chemical oxygen demand (COD) and biochemical methane potential (BMP) were respectively 8%, 160 mg(O2).gTS-1 and 92 mL(CH4).gTS-1. These latter errors are similar to successful NIRS applications developed on freeze-dried samples. These findings hold great promises regarding the development of at-site and online NIRS solutions in anaerobic digestion plants.

Preprocessing NIR Spectra for Aquaphotomics.

Even though NIR spectroscopy is based on the Beer-Lambert law, which clearly relates the concentration of the absorbing elements with the absorbance, the measured spectra are subject to spurious signals, such as additive and multiplicative effects. The use of NIR spectra, therefore, requires a preprocessing step. This article reviews the main preprocessing methods in the light of aquaphotomics. Simple methods for visualizing the spectra are proposed in order to guide the user in the choice of the best preprocessing. The most common chemometrics preprocessing are presented and illustrated by three real datasets. Some preprocessing aims to produce a spectrum as close as possible to the absorbance that would have been measured under ideal conditions and is very useful for the establishment of an aquagram. Others, dedicated to the improvement of the resolution of the spectra, are very useful for the identification of the peaks. Finally, special attention is given to the problem of reducing multiplicative effects and to the potential pitfalls of some very popular methods in chemometrics. Alternatives proposed in recent papers are presented.

On-site substrate characterization in the anaerobic digestion context: A dataset of near infrared spectra acquired with four different optical systems on freeze-dried and ground organic waste.

The near infrared spectra of thirty-three freeze-dried and ground organic waste samples of various biochemical composition were collected on four different optical systems, including a laboratory spectrometer, a transportable spectrometer with two measurement configurations (an immersed probe, and a polarized light system) and a micro-spectrometer. The provided data contains one file per spectroscopic system including the reflectance or absorbance spectra with the corresponding sample name and wavelengths. A reference data file containing carbohydrates, lipid and nitrogen content, biochemical methane potential (BMP) and chemical oxygen demand (COD) for each sample is also provided. This data enables the comparison of the optical systems for predictive model calibration based for example on Partial Least Squares Regression (PLS-R) [1], but could be used more broadly to test new chemometrics methods. For example, the data could be used to evaluate different transfer functions between spectroscopic systems [2]. This dataset enabled the research work reported by Mallet et al. 2021 [3].

Relating Near-Infrared Light Path-Length Modifications to the Water Content of Scattering Media in Near-Infrared Spectroscopy: Toward a New Bouguer-Beer-Lambert Law.

In near-infrared spectroscopy (NIRS), the linear relationship between absorbance and an absorbing compound concentration has been strictly defined by the Bouguer-Beer-Lambert law only for the case of transmission measurements of nonscattering media. However, various quantitative calibrations have been successfully built both on reflectance measurements and for scattering media. Although the lack of linearity for scattering media has been observed experimentally, the sound multivariate statistics and signal processing involved in chemometrics have allowed us to overcome this problem in most cases. However, in the case of samples with varying water content, important modifications of scattering levels still make calibrations difficult to build due to nonlinearities. Moreover, even when calibration procedures are successfully developed, many preprocessing methods used do not guarantee correct spectroscopic assignments (in the sense of a pure chemical absorbance). In particular, this may prevent correct modeling and interpretation of the structure of water. In this study, dynamic near-infrared spectra acquired during a drying process allow the study of the physical effects of water content variations, with a focus on the first overtone OH absorbance region. A model sample consisting of aluminum pellets mixed with water allowed us to study this specifically, without any other absorbing interaction terms related to the dry mass-absorbing constituents. A new formulation of the Bouguer-Beer-Lambert law is proposed, by expressing path length as a power function of water content. Through this new formulation, it is shown that a better and simpler prediction model of water content may be developed, with more precise and accurate identification of water absorbance bands.

Fast at-line characterization of solid organic waste: Comparing analytical performance of different compact near infrared spectroscopic systems with different measurement configurations.

Fast characterization of solid organic waste using near infrared spectroscopy has been successfully developed in the last decade. However, its adoption in biogas plants for monitoring the feeding substrates remains limited due to the lack of applicability and high costs. Recent evolutions in the technology have given rise to both more compact and more modular low-cost near infrared systems which could allow a larger scale deployment. The current study investigates the relevance of these new systems by evaluating four different Fourier transform near-infrared spectroscopic systems with different compactness (laboratory, portable, micro spectrometer) but also different measurement configurations (polarized light, at distance, in contact). Though the conventional laboratory spectrometer showed the best performance on the various biochemical parameters tested (carbohydrates, lipids, nitrogen, chemical oxygen demand, biochemical methane potential), the compact systems provided very close results. Prediction of the biochemical methane potential was possible using a low-cost micro spectrometer with an independent validation set error of only 91 NmL(CH4).gTS-1 compared to 60 NmL(CH4).gTS-1 for a laboratory spectrometer. The differences in performance were shown to result mainly from poorer spectral sampling; and not from instrument characteristics such as spectral resolution. Regarding the measurement configurations, none of the evaluated systems allowed a significant gain in robustness. In particular, the polarized light system provided better results when using its multi-scattered signal which brings further evidence of the importance of physical light-scattering properties in the success of models built on solid organic waste.

Unveiling non-linear water effects in near infrared spectroscopy: A study on organic wastes during drying using chemometrics.

In the context of organic waste management, near infrared spectroscopy (NIRS) is being used to offer a fast, non-destructive, and cost-effective characterization system. However, cumbersome freeze-drying steps of the samples are required to avoid water's interference on near infrared spectra. In order to better understand these effects, spectral variations induced by dry matter content variations were obtained for a wide variety of organic substrates. This was made possible by the development of a customized near infrared acquisition system with dynamic highly-resolved simultaneous scanning of near infrared spectra and estimation of dry matter content during a drying process at ambient temperature. Using principal components analysis, the complex water effects on near infrared spectra are detailed. Water effects are shown to be a combination of both physical and chemical effects, and depend on both the characteristics of the samples (biochemical type and physical structure) and the moisture content level. This results in a non-linear relationship between the measured signal and the analytical characteristic of interest. A typology of substrates with respect to these water effects is provided and could further be efficiently used as a basis for the development of local quantitative calibration models and correction methods accounting for these water effects.