Multicenter Validation of a Machine Learning Algorithm for Diagnosing Pediatric Patients with Multisystem Inflammatory Syndrome and Kawasaki Disease

ABSTRACT

BackgroundMultisystem inflammatory syndrome in children (MIS-C) is a novel disease identified during the COVID-19 pandemic characterized by systemic inflammation following SARS-CoV-2 infection. Delays in diagnosing MIS-C may lead to more severe disease with cardiac dysfunction or death. Most pediatric patients recover fully with anti-inflammatory treatments, but early detection of MIS-C remains a challenge given its clinical similarities to Kawasaki disease (KD) and other acute childhood illnesses. MethodsWe developed KIDMATCH (KawasakI Disease vs Multisystem InflAmmaTory syndrome in CHildren), a deep learning algorithm for screening patients for MIS-C, KD, or other febrile illness, using age, the five classical clinical KD signs, and 17 laboratory measurements prospectively collected within 24 hours of admission to the emergency department from 1448 patients diagnosed with KD or other febrile illness between January 1, 2009 and December 31, 2019 at Rady Childrens Hospital. For MIS-C patients, the same data was collected from 131 patients between May 14, 2020 to June 18, 2021 at Rady Childrens Hospital, Connecticut Childrens Hospital, and Childrens Hospital Los Angeles. We trained a two-stage model consisting of feedforward neural networks to distinguish between MIS-C and non MIS-C patients and then KD and other febrile illness. After internally validating the algorithm using 10-fold cross validation, we incorporated a conformal prediction framework to tag patients with erroneous data or distribution shifts, enhancing the model generalizability and confidence by flagging unfamiliar cases as indeterminate instead of making spurious predictions. We externally validated KIDMATCH on 175 MIS-C patients from 16 hospitals across the United States. FindingsKIDMATCH achieved a high median area under the curve in the 10-fold cross validation of 0.988 [IQR 0.98-0.993] in the first stage and 0.96 [IQR 0.956-0.972] in the second stage using thresholds set at 95% sensitivity to detect positive MIS-C and KD cases respectively during training. External validation of KIDMATCH on MIS-C patients correctly classified 76/83 (2 rejected) patients from the CHARMS consortium, 47/50 (1 rejected) patients from Boston Childrens Hospital, and 36/42 (2 rejected) patients from Childrens National Hospital. InterpretationKIDMATCH has the potential to aid frontline clinicians with distinguishing between MIS-C, KD, and similar febrile illnesses in a timely manner to allow prompt treatment and prevent severe complications. FundingEunice Kennedy Shriver National Institute of Child Health and Human Development, National Heart, Lung, and Blood Institute, Patient-Centered Outcomes Research Institute, National Library of Medicine