Testing domain knowledge and risk of bias of a large-scale general artificial intelligence model in mental health.

Background: With a rapidly expanding gap between the need for and availability of mental health care, artificial intelligence (AI) presents a promising, scalable solution to mental health assessment and treatment. Given the novelty and inscrutable nature of such systems, exploratory measures aimed at understanding domain knowledge and potential biases of such systems are necessary for ongoing translational development and future deployment in high-stakes healthcare settings. Methods: We investigated the domain knowledge and demographic bias of a generative, AI model using contrived clinical vignettes with systematically varied demographic features. We used balanced accuracy (BAC) to quantify the model's performance. We used generalized linear mixed-effects models to quantify the relationship between demographic factors and model interpretation. Findings: We found variable model performance across diagnoses; attention deficit hyperactivity disorder, posttraumatic stress disorder, alcohol use disorder, narcissistic personality disorder, binge eating disorder, and generalized anxiety disorder showed high BAC (0.70 ≤ BAC ≤ 0.82); bipolar disorder, bulimia nervosa, barbiturate use disorder, conduct disorder, somatic symptom disorder, benzodiazepine use disorder, LSD use disorder, histrionic personality disorder, and functional neurological symptom disorder showed low BAC (BAC ≤ 0.59). Interpretation: Our findings demonstrate initial promise in the domain knowledge of a large AI model, with performance variability perhaps due to the more salient hallmark symptoms, narrower differential diagnosis, and higher prevalence of some disorders. We found limited evidence of model demographic bias, although we do observe some gender and racial differences in model outcomes mirroring real-world differential prevalence estimates.

Red-flag sepsis and SOFA identifies different patient population at risk of sepsis-related deaths on the general ward.

Controversy exists regarding the best diagnostic and screening tool for sepsis outside the intensive care unit (ICU). Sequential organ failure assessment (SOFA) score has been shown to be superior to systemic inflammatory response syndrome (SIRS) criteria, however, the performance of "Red Flag sepsis criteria" has not been tested formally.The aim of the study was to investigate the ability of Red Flag sepsis criteria to identify the patients at high risk of sepsis-related death in comparison to SOFA based sepsis criteria. We also investigated the comparison of Red Flag sepsis to quick SOFA (qSOFA), SIRS, and national early warning score (NEWS) scores and factors influencing patient mortality.Patients were recruited into a 24-hour point-prevalence study on the general wards and emergency departments across all Welsh acute hospitals. Inclusion criteria were: clinical suspicion of infection and NEWS 3 or above in-line with established escalation criteria in Wales. Data on Red Flag sepsis and SOFA criteria was collected together with qSOFA and SIRS scores and 90-day mortality.459 patients were recruited over a 24-hour period. 246 were positive for Red Flag sepsis, mortality 33.7% (83/246); 241 for SOFA based sepsis criteria, mortality 39.4% (95/241); 54 for qSOFA, mortality 57.4% (31/54), and 268 for SIRS, mortality 33.6% (90/268). 55 patients were not picked up by any criteria. We found that older age was associated with death with OR (95% CI) of 1.03 (1.02-1.04); higher frailty score 1.24 (1.11-1.40); DNA-CPR order 1.74 (1.14-2.65); ceiling of care 1.55 (1.02-2.33); and SOFA score of 2 and above 1.69 (1.16-2.47).The different clinical tools captured different subsets of the at-risk population, with similar sensitivity. SOFA score 2 or above was independently associated with increased risk of death at 90 days. The sequalae of infection-related organ dysfunction cannot be reliably captured based on routine clinical and physiological parameters alone.

Sepsis-related deaths in the at-risk population on the wards: attributable fraction of mortality in a large point-prevalence study.

OBJECTIVE: Sepsis mortality is reported to be high worldwide, however recently the attributable fraction of mortality due to sepsis (AFsepsis) has been questioned. If improvements in treatment options are to be evaluated, it is important to know what proportion of deaths are potentially preventable or modifiable after a sepsis episode. The aim of the study was to establish the fraction of deaths directly related to the sepsis episode on the general wards and emergency departments. RESULTS: 839 patients were recruited over the two 24-h periods in 2016 and 2017. 521 patients fulfilled SEPSIS-3 criteria. 166 patients (32.4%) with sepsis and 56 patients (17.6%) without sepsis died within 90 days. Out of the 166 sepsis deaths 12 (7.2%) could have been directly related to sepsis, 28 (16.9%) possibly related and 96 (57.8%) were not related to sepsis. Overall AFsepsis was 24.1%. Upon analysis of the 40 deaths likely to be attributable to sepsis, we found that 31 patients (77.5%) had the Clinical Frailty Score ≥ 6, 28 (70%) had existing DNA-CPR order and 17 had limitations of care orders (42.5%).