Testing the performance, adequacy, and applicability of an artificial intelligence model for pediatric pneumonia diagnosis.

BACKGROUND: Community-acquired Pneumonia (CAP) is a common childhood infectious disease. Deep learning models show promise in X-ray interpretation and diagnosis, but their validation should be extended due to limitations in the current validation workflow. To extend the standard validation workflow we propose doing a pilot test with the next characteristics. First, the assumption of perfect ground truth (100% sensitive and specific) is unrealistic, as high intra and inter-observer variability have been reported. To address this, we propose using Bayesian latent class models (BLCA) to estimate accuracy during the pilot. Additionally, assessing only the performance of a model without considering its applicability and acceptance by physicians is insufficient if we hope to integrate AI systems into day-to-day clinical practice. Therefore, we propose employing explainable artificial intelligence (XAI) methods during the pilot test to involve physicians and evaluate how well a Deep Learning model is accepted and how helpful it is for routine decisions as well as analyze its limitations by assessing the etiology. This study aims to apply the proposed pilot to test a deep Convolutional Neural Network (CNN)-based model for identifying consolidation in pediatric chest-X-ray (CXR) images already validated using the standard workflow. METHODS: For the standard validation workflow, a total of 5856 public CXRs and 950 private CXRs were used to train and validate the performance of the CNN model. The performance of the model was estimated assuming a perfect ground truth. For the pilot test proposed in this article, a total of 190 pediatric chest-X-ray (CXRs) images were used to test the CNN model support decision tool (SDT). The performance of the model on the pilot test was estimated using extensions of the two-test Bayesian Latent-Class model (BLCA). The sensitivity, specificity, and accuracy of the model were also assessed. The clinical characteristics of the patients were compared according to the model performance. The adequacy and applicability of the SDT was tested using XAI techniques. The adequacy of the SDT was assessed by asking two senior physicians the agreement rate with the SDT. The applicability was tested by asking three medical residents before and after using the SDT and the agreement between experts was calculated using the kappa index. RESULTS: The CRXs of the pilot test were labeled by the panel of experts into consolidation (124/176, 70.4%) and no-consolidation/other infiltrates (52/176, 29.5%). A total of 31/176 (17.6%) discrepancies were found between the model and the panel of experts with a kappa index of 0.6. The sensitivity and specificity reached a median of 90.9 (95% Credible Interval (CrI), 81.2-99.9) and 77.7 (95% CrI, 63.3-98.1), respectively. The senior physicians reported a high agreement rate (70%) with the system in identifying logical consolidation patterns. The three medical residents reached a higher agreement using SDT than alone with experts (0.66±0.1 vs. 0.75±0.2). CONCLUSIONS: Through the pilot test, we have successfully verified that the deep learning model was underestimated when a perfect ground truth was considered. Furthermore, by conducting adequacy and applicability tests, we can ensure that the model is able to identify logical patterns within the CXRs and that augmenting clinicians with automated preliminary read assistants could accelerate their workflows and enhance accuracy in identifying consolidation in pediatric CXR images.

250-year records of mercury and trace element deposition in two lakes from Cajas National Park, SW Ecuadorian Andes.

Historical records of trace elements in lake sediments provide source-to-sink information about potentially toxic pollutants across space and time. We investigated two lakes located at different elevations in the Ecuadorian Andes to understand how trace element fluxes are related to (i) geology, (ii) erosion in the watersheds, and (iii) local point sources and atmospheric loads. In remote Lake Fondococha (4150 m a.s.l.), total Hg fluxes stay constant between ca. 1760 and 1950 and show an approximately 4.4-fold increase between pre-1950 and post-1950 values. The post-1950 increase in fluxes of other trace elements (V, Cr, Co, Ni, Cu, Zn, As, Cd, and Pb) is lower (2.1-3.0-fold) than for Hg. Mostly lithogenic sources and enhanced soil erosion contribute to their post-1950 increase (lithogenic contribution: > 85%, Hg: ~ 58%). Average post-1950 Hg fluxes are approximately 4.3 times higher in peri-urban Lake Llaviucu (3150 m a.s.l.) than in the remote Lake Fondococha. Post-1950 fluxes of the other trace elements showed larger differences between Lakes Fondococha and Llaviucu (5.2 < 25-29.5-fold increase; Ni < Pb-Cd). The comparison of the post-1950 average trace element fluxes that are derived from point and airborne sources revealed 5-687 (Hg-Pb) times higher values in Lake Llaviucu than in Lake Fondococha suggesting that Lake Llaviucu's proximity to the city of Cuenca strongly influences its deposition record (industrial emissions, traffic, caged fishery). Both lakes responded with temporary drops in trace element accumulations to park regulations in the 1970s and 1990s, but show again increasing trends in recent times, most likely caused by increase in vehicular traffic and openings of copper and gold mines around Cajas National Park.

A 150-year record of polycyclic aromatic compound (PAC) deposition from high Andean Cajas National Park, southern Ecuador.

The temporal profiles of polycyclic aromatic compounds (PACs) in lake sediments reflect past changes in emissions, transport and deposition of these pollutants and, thus, record natural and anthropogenic processes. We document fluxes of PACs [(polycyclic aromatic hydrocarbons (PAHs), oxygenated PAHs (OPAHs) and azaarenes (AZAs)] deposited in two tropical high-altitude lakes in the Cajas National Park (Ecuadorian Andes, 2°50'S, 79°10'W). In remote and high elevation Laguna Fondococha (4130m a.s.l.), the temporal fluxes of OPAHs and AZAs were similar to those of PAHs suggesting similar sources. A significant increase of PAC deposition after the 1950s reflects Ecuador's economic development. PAH fluxes were relatively low (∑27PAHs (without retene and perylene): 0.86-11.21ngcm-2yr-1) with a composition pattern typical for long-range atmospheric transport (high 9-fluorenone/fluorene ratios) and biomass burning (30% low molecular weight PAHs). PAHs diagnostic of high temperature combustion (industry, traffic) make up 20-25% of total PAHs. Perylene concentrations increase linearly with increasing sediment depth suggesting diagenetic in-situ production. At lower elevations (Laguna Llaviucu, 3140m a.s.l.) and closer to urban areas, PAC fluxes in the past decades were 4-5 times higher than in the remote high-elevation lake. Laguna Llaviucu also showed higher concentrations of high molecular weight pyrogenic PAHs and a greater diversity of AZAs. Individual OPAHs and AZAs reflect mainly combustion activities. In Laguna Llaviucu, which is at a lower elevation (3140m a.s.l.) and closer to the city, molecular ratios suggest short-range atmospheric transport and deposition of PACs. A very foggy climate (170 rainy days per year) with the precipitation maximum at 3500m removes PACs very efficiently (by wet deposition) from the atmosphere at very short distances from emission sources. This partly explains why L. Llaviucu shows higher fluxes of PACs than the higher elevation L. Fondococha. This study presents the first historical record of organic pollutants from environmental archives in Ecuador.