Don't be misled: 3 misconceptions about external validation of clinical prediction models.

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing 3 common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

Development and clinical impact assessment of a machine-learning model for early prediction of late-onset sepsis.

BACKGROUND AND AIM: Preterm infants are prone to neonatal infections such as late-onset sepsis (LOS). The consequences of LOS can be severe and potentially life-threatening. Unfortunately, LOS often presents with unspecific symptoms, and early screening laboratory tests have limited diagnostic value and are often late. This study aimed to build a predictive algorithm to aid doctors in the early detection of LOS in very preterm infants. METHODS: In a retrospective cohort study, all consecutively admitted preterm infants (GA ≤ 32 weeks) from 2008 until 2019 were included. They were classified as LOS or control according to blood culture results, currently the gold standard. To generate features, routine and continuously measured oxygen saturation and heart rate data with a minute-by-minute sampling rate were extracted from electronic medical records. Care was taken not to include variables indicative of existing LOS suspicion. The timing of a positive blood culture served as a proxy for LOS-onset. An equivalent timestamp was generated in gestational-age-matched control patients without a positive blood culture. Three machine learning (ML) techniques (generalized additive models, logistic regression, and XGBoost) were used to build a classification algorithm. To simulate the performance of the algorithm in clinical practice, a simulation using multiple alarm thresholds was performed on hourly predictions for the total hospitalization period. RESULTS: 292 infants with LOS were matched to 1497 controls. The median gestational age before matching was 28.1 and 30.3 weeks, respectively. Evaluation of the overall discriminative power of the LR algorithm yielded an AUC of 0.73 (p < 0.05) at the moment of clinical suspicion (t = 0). In the longitudinal simulation, our algorithm detects LOS in at least 47% of the patients before clinical suspicion without exceeding the alarm fatigue threshold of 3 alarms per day. Furthermore, medical experts evaluated the algorithm as clinically relevant regarding the feature contributions in the model explanations. CONCLUSIONS: An ML algorithm was trained for the early detection of LOS. Performance was evaluated on both prediction horizons and in a clinical impact simulation. To the best of our knowledge, our assessment of clinical impact with a retrospective simulation on longitudinal data is the most extensive in the literature on LOS prediction to date. The clinically relevant algorithm, based on routinely collected data, can potentially accelerate clinical decisions in the early detection of LOS, even with limited inputs.

A Perspective on a Quality Management System for AI/ML-Based Clinical Decision Support in Hospital Care.

Although many artificial intelligence (AI) and machine learning (ML) based algorithms are being developed by researchers, only a small fraction has been implemented in clinical-decision support (CDS) systems for clinical care. Healthcare organizations experience significant barriers implementing AI/ML models for diagnostic, prognostic, and monitoring purposes. In this perspective, we delve into the numerous and diverse quality control measures and responsibilities that emerge when moving from AI/ML-model development in a research environment to deployment in clinical care. The Sleep-Well Baby project, a ML-based monitoring system, currently being tested at the neonatal intensive care unit of the University Medical Center Utrecht, serves as a use-case illustrating our personal learning journey in this field. We argue that, in addition to quality assurance measures taken by the manufacturer, user responsibilities should be embedded in a quality management system (QMS) that is focused on life-cycle management of AI/ML-CDS models in a medical routine care environment. Furthermore, we highlight the strong similarities between AI/ML-CDS models and in vitro diagnostic devices and propose to use ISO15189, the quality guideline for medical laboratories, as inspiration when building a QMS for AI/ML-CDS usage in the clinic. We finally envision a future in which healthcare institutions run or have access to a medical AI-lab that provides the necessary expertise and quality assurance for AI/ML-CDS implementation and applies a QMS that mimics the ISO15189 used in medical laboratories.

The Sleep Well Baby project: an automated real-time sleep-wake state prediction algorithm in preterm infants.

STUDY OBJECTIVES: Sleep is an important driver of early brain development. However, sleep is often disturbed in preterm infants admitted to the neonatal intensive care unit (NICU). We aimed to develop an automated algorithm based on routinely measured vital parameters to classify sleep-wake states of preterm infants in real-time at the bedside. METHODS: In this study, sleep-wake state observations were obtained in 1-minute epochs using a behavioral scale developed in-house while vital signs were recorded simultaneously. Three types of vital parameter data, namely, heart rate, respiratory rate, and oxygen saturation, were collected at a low-frequency sampling rate of 0.4 Hz. A supervised machine learning workflow was used to train a classifier to predict sleep-wake states. Independent training (n = 37) and validation datasets were validation n = 9) datasets were used. Finally, a setup was designed for real-time implementation at the bedside. RESULTS: The macro-averaged area-under-the-receiver-operator-characteristic (AUROC) of the automated sleep staging algorithm ranged between 0.69 and 0.82 for the training data, and 0.61 and 0.78 for the validation data. The algorithm provided the most accurate prediction for wake states (AUROC = 0.80). These findings were well validated on an independent sample (AUROC = 0.77). CONCLUSIONS: With this study, to the best of our knowledge, a reliable, nonobtrusive, and real-time sleep staging algorithm was developed for the first time for preterm infants. Deploying this algorithm in the NICU environment may assist and adapt bedside clinical work based on infants' sleep-wake states, potentially promoting the early brain development and well-being of preterm infants.

Efficacy of subcutaneous allergen immunotherapy in atopic dogs: A retrospective study of 664 cases.

BACKGROUND: Canine atopic dermatitis (cAD) is an allergic skin disease affecting approximately 10% of dogs. allergen-specific immunotherapy (ASIT) is currently the only treatment option able to induce tolerance to the causative allergens. OBJECTIVE: To retrospectively establish the efficacy of ASIT in atopic dogs. ANIMALS: Client-owned (n  = 664) dogs with cAD presented between 2008 and 2018 to two dermatology referral clinics. MATERIALS AND METHODS: Clinical records of atopic dogs were reviewed to obtain information including the results of the intradermal skin test and/or allergen-specific immunoglobulin (Ig)E serological results, the allergens included in the ASIT, concurrent symptomatic medications, and ASIT efficacy after at least 9 months. RESULTS: Excellent (ASIT alone controlled clinical signs), good (≥50% reduction of clinical signs) and poor (<50% improvement) responses were seen in 31.5%, 28.5% and 40.1% of the dogs, respectively. No significant differences in efficacy were associated with breed, sex, age at initiation of ASIT, type of allergens in ASIT, and between clinics. Dogs re-examined regularly responded significantly better to ASIT than dogs that did not (>50% improvement in 69.3% and 55.4% of the dogs, respectively). Dogs treated with ASIT and concomitant systemic glucocorticoids showed a significantly poorer response (success rate of >50% improvement of 38.5%). CONCLUSIONS AND CLINICAL IMPORTANCE: In 59.9% of atopic dogs, subcutaneous ASIT can improve clinical signs by ≥50%. The beneficial effect of ASIT is higher if dogs are re-examined regularly and if systemic long-term corticosteroids are avoided, at least during the first 9 months of ASIT.

Contexte - La dermatite atopique canine (DAC) est une maladie cutanée allergique qui touche environ 10 % des chiens. L'immunothérapie allergénique (ASIT) est actuellement la seule option thérapeutique capable d'induire une tolérance aux allergènes responsables. Objectif - Établir rétrospectivement l'efficacité de l'ASIT chez les chiens atopiques. Animaux - Chiens de clients (n = 664) atteints de DAC présentés entre 2008 et 2018 à deux cliniques de référé en dermatologie. Méthodes - Les dossiers cliniques des chiens atopiques ont été examinés pour obtenir des informations, notamment les résultats du test cutané intradermique et / ou les résultats sérologiques de l'immunoglobuline (Ig) E spécifique d'allergène, les allergènes inclus dans l'ASIT, les médicaments symptomatiques concomitants et l'efficacité de l'ASIT après au moins neuf mois. Résultats - Des réponses excellentes (signes cliniques contrôlés par l'ASIT seul), bonnes (réduction ≥ 50 % des signes cliniques) et médiocres (amélioration < 50 %) ont été observées chez 31,5 %, 28,5 % et 40,1 % des chiens, respectivement. Aucune différence significative d'efficacité n'a été associée à la race, au sexe, à l'âge au début de l'ASIT, au type d'allergènes dans l'ASIT et entre les cliniques. Les chiens réexaminés régulièrement ont répondu significativement mieux à l'ASIT que les chiens qui ne l'ont pas fait (amélioration > 50 % chez 69,3 % et 55,4 % des chiens, respectivement). Les chiens traités avec l'ASIT et des glucocorticoïdes systémiques concomitants ont montré une réponse significativement plus faible (taux de réussite > 50 % d'amélioration de 38,5 %). Conclusions et importance clinique - Chez 59,9 % des chiens atopiques, l'ASIT sous-cutanée peut améliorer les signes cliniques de ≥ 50 %. L'effet bénéfique de l'ASIT est plus élevé si les chiens sont réexaminés régulièrement et si les corticostéroïdes systémiques à long terme sont évités, au moins pendant les neuf premiers mois de l'ASIT.

Introducción- la dermatitis atópica canina (cAD) es una enfermedad alérgica de la piel que afecta aproximadamente al 10 % de los perros. La inmunoterapia con alérgenos (ASIT) es actualmente la única opción de tratamiento capaz de inducir tolerancia a los alérgenos causantes. Objetivo- establecer retrospectivamente la eficacia de ASIT en perros atópicos. Animales- perros de propietarios particulares (n=664) con cAD presentados entre 2008 y 2018 a dos clínicas de referencia de dermatología. Métodos- se revisaron las historias clínicas de los perros atópicos para obtener información, incluidos los resultados de la prueba cutánea intradérmica y/o los resultados serológicos de la inmunoglobulina (Ig)E específica para alérgenos, los alérgenos incluidos en el ASIT, los medicamentos sintomáticos concurrentes y la eficacia del ASIT después de al menos menos nueve meses. Resultados- se observaron respuestas excelentes (la ASIT sola controló los signos clínicos), buenas (≥50 % de reducción de los signos clínicos) y malas (<50 % de mejora) en el 31,5 %, 28,5 % y 40,1 % de los perros, respectivamente. No se asociaron diferencias significativas en la eficacia con la raza, el sexo, la edad al inicio de la ASIT, el tipo de alérgenos en la ASIT y entre las clínicas. Los perros reexaminados con regularidad respondieron significativamente mejor a la ASIT que los perros que no lo hicieron (>50 % de mejora en el 69,3 % y el 55,4 % de los perros, respectivamente). Los perros tratados con ASIT y glucocorticoides sistémicos concomitantes mostraron una respuesta significativamente peor (tasa de éxito de >50 % de mejora del 38,5 %). Conclusiones e importancia clínica - En el 59,9% de los perros atópicos, la ASIT subcutánea puede mejorar los signos clínicos en ≥50%. El efecto beneficioso de la ASIT es mayor si los perros se vuelven a examinar con regularidad y si se evitan los corticosteroides sistémicos a largo plazo, al menos durante los primeros nueve meses de la ASIT.

Contexto - A dermatite atópica canina (DAC) é uma doença alérgica que afeta aproximadamente 10% dos cães. A imunoterapia alérgeno-específica (ASIT) é atualmente a única opção terapêutica capaz de induzir tolerância aos alérgenos causadores. Objetivo - Estabelecer retrospectivamente a eficácia da ASIT em cães atópicos. Animais - Cães de proprietários (664) com DAC apresentados a duas clínicas de referência em dermatologia entre 2008 e 2018. Métodos - Os prontuários dos cães atópicos foram revisados para se obter informações incluindo os resultados dos testes intradérmicos e/ou sorológicos para anticorpos (IgE) alérgeno-específicos, os alérgenos incluídos na ASIT, medicações sintomáticas concomitantes, e a eficácia de ASIT após um mínimo de nove meses. Resultados - Resposta excelente (ASIT controlou os sinais clínicos sozinha), boa (redução ≥50% nos sinais clínicos) e pobre (<50% de melhora) foi observada em 31,5%, 28,5% e 40,1% dos cães, respectivamente. Não foram observadas diferenças significativas na eficácia em relação a raça, gênero, idade à iniciação da ASIT, tipo de alérgenos na ASIT e entre as clínicas. Os cães reavaliados periodicamente responderam significativamente melhor à ASIT que cães que não foram reavaliados com frequência (>50% de melhora em 69,3% e 55,4% dos cães, respectivamente). Cães tratados com ASIT e glicocorticoides sistêmicos concomitantemente demonstraram uma resposta significativamente mais pobre (a taxa de sucesso >50% ocorreu em 38,5%). Conclusões e importância clínica - Em 59,9% dos cães atópicos, ASIT subcutânea é capaz de melhorar os sinais clínicos em ≥50%. O efeito benéfico de ASIT foi maior se os cães fossem reavaliados regularmente e se corticoides sistêmicos a longo prazo fossem evitados, ao menos durante os primeiros meses de ASIT.

Identification of parameters and formulation of a statistical and machine learning model to identify Babesia canis infections in dogs using available ADVIA hematology analyzer data.

BACKGROUND: Canine babesiosis is an important tick-borne disease in endemic regions. One of the relevant subspecies in Europe is Babesia canis, and it can cause severe clinical signs such as hemolytic anemia. Apart from acute clinical symptoms dogs can also have a more chronic disease development or be asymptomatic carriers. Our objective was to identify readily available ADVIA hematology analyzer parameters suggestive of B. canis parasitemia in dogs and to formulate a predictive model. METHODS: A historical dataset of complete blood count data from an ADVIA hematology system with blood smear or PCR confirmed parasitemia cases was used to obtain a model by conventional statistics (CS) methods and machine learning (ML) using logistical regression and tree methods. RESULTS: Both methods identified that important parameters were platelet count, mean platelet volume and percentage large unstained cells. We were able to formulate a CS model and ML model to screen for Babesia parasitemia in dogs with a sensitivity of 84.6% (CS) and 100% (ML), a specificity of 97.7% (CS) and 95.7% (ML) and a positive likelihood ratio (LR+) of 36.78 (CS) and 23.2 (ML). CONCLUSIONS: This study introduces two methods of screening for B. canis parasitemia on readily available data from ADVIA hematology systems. The algorithms can easily be introduced in laboratories that use these analyzers. When the algorithm marks a sample as 'suggestive' for Babesia parasitemia, the sample is approximately 37 times more likely to show Babesia merozoites on blood smear analysis.

Strong Support for the Millisecond Pulsar Origin of the Galactic Center GeV Excess.

Using γ-ray data from the Fermi Large Area Telescope, various groups have identified a clear excess emission in the Inner Galaxy, at energies around a few GeV. This excess resembles remarkably well a signal from dark-matter annihilation. One of the most compelling astrophysical interpretations is that the excess is caused by the combined effect of a previously undetected population of dim γ-ray sources. Because of their spectral similarity, the best candidates are millisecond pulsars. Here, we search for this hypothetical source population, using a novel approach based on wavelet decomposition of the γ-ray sky and the statistics of Gaussian random fields. Using almost seven years of Fermi-LAT data, we detect a clustering of photons as predicted for the hypothetical population of millisecond pulsar, with a statistical significance of 10.0σ. For plausible values of the luminosity function, this population explains 100% of the observed excess emission. We argue that other extragalactic or Galactic sources, a mismodeling of Galactic diffuse emission, or the thick-disk population of pulsars are unlikely to account for this observation.