Elicitation of domain knowledge for a machine learning model for paediatric critical illness in South Africa.

Objectives: Delays in identification, resuscitation and referral have been identified as a preventable cause of avoidable severity of illness and mortality in South African children. To address this problem, a machine learning model to predict a compound outcome of death prior to discharge from hospital and/or admission to the PICU was developed. A key aspect of developing machine learning models is the integration of human knowledge in their development. The objective of this study is to describe how this domain knowledge was elicited, including the use of a documented literature search and Delphi procedure. Design: A prospective mixed methodology development study was conducted that included qualitative aspects in the elicitation of domain knowledge, together with descriptive and analytical quantitative and machine learning methodologies. Setting: A single centre tertiary hospital providing acute paediatric services. Participants: Three paediatric intensivists, six specialist paediatricians and three specialist anaesthesiologists. Interventions: None. Measurements and main results: The literature search identified 154 full-text articles reporting risk factors for mortality in hospitalised children. These factors were most commonly features of specific organ dysfunction. 89 of these publications studied children in lower- and middle-income countries. The Delphi procedure included 12 expert participants and was conducted over 3 rounds. Respondents identified a need to achieve a compromise between model performance, comprehensiveness and veracity and practicality of use. Participants achieved consensus on a range of clinical features associated with severe illness in children. No special investigations were considered for inclusion in the model except point-of-care capillary blood glucose testing. The results were integrated by the researcher and a final list of features was compiled. Conclusion: The elicitation of domain knowledge is important in effective machine learning applications. The documentation of this process enhances rigour in such models and should be reported in publications. A documented literature search, Delphi procedure and the integration of the domain knowledge of the researchers contributed to problem specification and selection of features prior to feature engineering, pre-processing and model development.

Development of artificial neural network models for paediatric critical illness in South Africa.

Objectives: Failures in identification, resuscitation and appropriate referral have been identified as significant contributors to avoidable severity of illness and mortality in South African children. In this study, artificial neural network models were developed to predict a composite outcome of death before discharge from hospital or admission to the PICU. These models were compared to logistic regression and XGBoost models developed on the same data in cross-validation. Design: Prospective, analytical cohort study. Setting: A single centre tertiary hospital in South Africa providing acute paediatric services. Patients: Children, under the age of 13 years presenting to the Paediatric Referral Area for acute consultations. Outcomes: Predictive models for a composite outcome of death before discharge from hospital or admission to the PICU. Interventions: None. Measurements and main results: 765 patients were included in the data set with 116 instances (15.2%) of the study outcome. Models were developed on three sets of features. Two derived from sequential floating feature selection (one inclusive, one parsimonious) and one from the Akaike information criterion to yield 9 models. All developed models demonstrated good discrimination on cross-validation with mean ROC AUCs greater than 0.8 and mean PRC AUCs greater than 0.53. ANN1, developed on the inclusive feature-et demonstrated the best discrimination with a ROC AUC of 0.84 and a PRC AUC of 0.64 Model calibration was variable, with most models demonstrating weak calibration. Decision curve analysis demonstrated that all models were superior to baseline strategies, with ANN1 demonstrating the highest net benefit. Conclusions: All models demonstrated satisfactory performance, with the best performing model in cross-validation being an ANN model. Given the good performance of less complex models, however, these models should also be considered, given their advantage in ease of implementation in practice. An internal validation study is now being conducted to further assess performance with a view to external validation.

An Artificial Neural Network Model for Pediatric Mortality Prediction in Two Tertiary Pediatric Intensive Care Units in South Africa. A Development Study.

Objectives: The performance of mortality prediction models remain a challenge in lower- and middle-income countries. We developed an artificial neural network (ANN) model for the prediction of mortality in two tertiary pediatric intensive care units (PICUs) in South Africa using free to download and use software and commercially available computers. These models were compared to a logistic regression model and a recalibrated version of the Pediatric Index of Mortality 3. Design: This study used data from a retrospective cohort study to develop an artificial neural model and logistic regression model for mortality prediction. The outcome evaluated was death in PICU. Setting: Two tertiary PICUs in South Africa. Patients: 2,089 patients up to the age of 13 completed years were included in the study. Interventions: None. Measurements and Main Results: The AUROC was higher for the ANN (0.89) than for the logistic regression model (LR) (0.87) and the recalibrated PIM3 model (0.86). The precision recall curve however favors the ANN over logistic regression and recalibrated PIM3 (AUPRC = 0.6 vs. 0.53 and 0.58, respectively. The slope of the calibration curve was 1.12 for the ANN model (intercept 0.01), 1.09 for the logistic regression model (intercept 0.05) and 1.02 (intercept 0.01) for the recalibrated version of PIM3. The calibration curve was however closer to the diagonal for the ANN model. Conclusions: Artificial neural network models are a feasible method for mortality prediction in lower- and middle-income countries but significant challenges exist. There is a need to conduct research directed toward the acquisition of large, complex data sets, the integration of documented clinical care into clinical research and the promotion of the development of electronic health record systems in lower and middle income settings.