A toolbox for surfacing health equity harms and biases in large language models.

Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessment of LLM-generated answers for biases and EquityMedQA, a collection of seven datasets enriched for adversarial queries. Both our human assessment framework and our dataset design process are grounded in an iterative participatory approach and review of Med-PaLM 2 answers. Through our empirical study, we find that our approach surfaces biases that may be missed by narrower evaluation approaches. Our experience underscores the importance of using diverse assessment methodologies and involving raters of varying backgrounds and expertise. While our approach is not sufficient to holistically assess whether the deployment of an artificial intelligence (AI) system promotes equitable health outcomes, we hope that it can be leveraged and built upon toward a shared goal of LLMs that promote accessible and equitable healthcare.

Disclosure of Adverse Events and Medical Errors: A Framework for Anesthesiologists.

Ethical disclosure of adverse events (AE) presents opportunities and challenges for physicians and has unique ramifications for anesthesiologists. AE disclosure is supported by patients, regulatory organizations, and physicians. Disclosure is part of a physician's ethical duty toward patients, supports fully informed patient decision making, and is a critical component of root cause analysis. Barriers to AE disclosure include disruption of the doctor-patient relationship, fear of litigation, and inadequate training. Apology laws intended to support disclosure and mitigate concern for adverse legal consequences have not fulfilled that initial promise. Training and institutional communication programs support physicians in providing competent, ethical AE disclosure.

Correction: Evaluating the Clinical Feasibility of an Artificial Intelligence-Powered, Web-Based Clinical Decision Support System for the Treatment of Depression in Adults: Longitudinal Feasibility Study.

[This corrects the article DOI: 10.2196/31862.].

The value of standards for health datasets in artificial intelligence-based applications.

Artificial intelligence as a medical device is increasingly being applied to healthcare for diagnosis, risk stratification and resource allocation. However, a growing body of evidence has highlighted the risk of algorithmic bias, which may perpetuate existing health inequity. This problem arises in part because of systemic inequalities in dataset curation, unequal opportunity to participate in research and inequalities of access. This study aims to explore existing standards, frameworks and best practices for ensuring adequate data diversity in health datasets. Exploring the body of existing literature and expert views is an important step towards the development of consensus-based guidelines. The study comprises two parts: a systematic review of existing standards, frameworks and best practices for healthcare datasets; and a survey and thematic analysis of stakeholder views of bias, health equity and best practices for artificial intelligence as a medical device. We found that the need for dataset diversity was well described in literature, and experts generally favored the development of a robust set of guidelines, but there were mixed views about how these could be implemented practically. The outputs of this study will be used to inform the development of standards for transparency of data diversity in health datasets (the STANDING Together initiative).

Evaluating the perceived utility of an artificial intelligence-powered clinical decision support system for depression treatment using a simulation center.

Aifred is a clinical decision support system (CDSS) that uses artificial intelligence to assist physicians in selecting treatments for major depressive disorder (MDD) by providing probabilities of remission for different treatment options based on patient characteristics. We evaluated the utility of the CDSS as perceived by physicians participating in simulated clinical interactions. Twenty physicians who were either staff or residents in psychiatry or family medicine completed a study in which they had three 10-minute clinical interactions with standardized patients portraying mild, moderate, and severe episodes of MDD. During these scenarios, physicians were given access to the CDSS, which they could use in their treatment decisions. The perceived utility of the CDSS was assessed through self-report questionnaires, scenario observations, and interviews. 60% of physicians perceived the CDSS to be a useful tool in their treatment-selection process, with family physicians perceiving the greatest utility. Moreover, 50% of physicians would use the tool for all patients with depression, with an additional 35% noting that they would reserve the tool for more severe or treatment-resistant patients. Furthermore, clinicians found the tool to be useful in discussing treatment options with patients. The efficacy of this CDSS and its potential to improve treatment outcomes must be further evaluated in clinical trials.

Evaluating the Clinical Feasibility of an Artificial Intelligence-Powered, Web-Based Clinical Decision Support System for the Treatment of Depression in Adults: Longitudinal Feasibility Study.

BACKGROUND: Approximately two-thirds of patients with major depressive disorder do not achieve remission during their first treatment. There has been increasing interest in the use of digital, artificial intelligence-powered clinical decision support systems (CDSSs) to assist physicians in their treatment selection and management, improving the personalization and use of best practices such as measurement-based care. Previous literature shows that for digital mental health tools to be successful, the tool must be easy for patients and physicians to use and feasible within existing clinical workflows. OBJECTIVE: This study aims to examine the feasibility of an artificial intelligence-powered CDSS, which combines the operationalized 2016 Canadian Network for Mood and Anxiety Treatments guidelines with a neural network-based individualized treatment remission prediction. METHODS: Owing to the COVID-19 pandemic, the study was adapted to be completed entirely remotely. A total of 7 physicians recruited outpatients diagnosed with major depressive disorder according to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria. Patients completed a minimum of one visit without the CDSS (baseline) and 2 subsequent visits where the CDSS was used by the physician (visits 1 and 2). The primary outcome of interest was change in appointment length after the introduction of the CDSS as a proxy for feasibility. Feasibility and acceptability data were collected through self-report questionnaires and semistructured interviews. RESULTS: Data were collected between January and November 2020. A total of 17 patients were enrolled in the study; of the 17 patients, 14 (82%) completed the study. There was no significant difference in appointment length between visits (introduction of the tool did not increase appointment length; F2,24=0.805; mean squared error 58.08; P=.46). In total, 92% (12/13) of patients and 71% (5/7) of physicians felt that the tool was easy to use; 62% (8/13) of patients and 71% (5/7) of physicians rated that they trusted the CDSS. Of the 13 patients, 6 (46%) felt that the patient-clinician relationship significantly or somewhat improved, whereas 7 (54%) felt that it did not change. CONCLUSIONS: Our findings confirm that the integration of the tool does not significantly increase appointment length and suggest that the CDSS is easy to use and may have positive effects on the patient-physician relationship for some patients. The CDSS is feasible and ready for effectiveness studies. TRIAL REGISTRATION: ClinicalTrials.gov NCT04061642; http://clinicaltrials.gov/ct2/show/NCT04061642.

REFRESH: A new approach to modeling dimensional biases in perceptual similarity and categorization.

Much categorization behavior can be explained by family resemblance: New items are classified by comparison with previously learned exemplars. However, categorization behavior also shows a variety of dimensional biases, where the underlying space has so-called "separable" dimensions: Ease of learning categories depends on how the stimuli align with the separable dimensions of the space. For example, if a set of objects of various sizes and colors can be accurately categorized using a single separable dimension (e.g., size), then category learning will be fast, while if the category is determined by both dimensions, learning will be slow. To capture these dimensional biases, almost all models of categorization supplement family resemblance with either rule-based systems or selective attention to separable dimensions. But these models do not explain how separable dimensions initially arise; they are presumed to be unexplained psychological primitives. We develop, instead, a pure family resemblance version of the Rational Model of Categorization (RMC), which we term the Rational Exclusively Family RESemblance Hierarchy (REFRESH), which does not presuppose any separable dimensions in the space of stimuli. REFRESH infers how the stimuli are clustered and uses a hierarchical prior to learn expectations about the variability of clusters across categories. We first demonstrate the dimensional alignment of natural-category features and then show how through a lifetime of categorization experience REFRESH will learn prior expectations that clusters of stimuli will align with separable dimensions. REFRESH captures the key dimensional biases and also explains their stimulus-dependence and how they are learned and develop. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Using a simulation centre to evaluate preliminary acceptability and impact of an artificial intelligence-powered clinical decision support system for depression treatment on the physician-patient interaction.

BACKGROUND: Recently, artificial intelligence-powered devices have been put forward as potentially powerful tools for the improvement of mental healthcare. An important question is how these devices impact the physician-patient interaction. AIMS: Aifred is an artificial intelligence-powered clinical decision support system (CDSS) for the treatment of major depression. Here, we explore the use of a simulation centre environment in evaluating the usability of Aifred, particularly its impact on the physician-patient interaction. METHOD: Twenty psychiatry and family medicine attending staff and residents were recruited to complete a 2.5-h study at a clinical interaction simulation centre with standardised patients. Each physician had the option of using the CDSS to inform their treatment choice in three 10-min clinical scenarios with standardised patients portraying mild, moderate and severe episodes of major depression. Feasibility and acceptability data were collected through self-report questionnaires, scenario observations, interviews and standardised patient feedback. RESULTS: All 20 participants completed the study. Initial results indicate that the tool was acceptable to clinicians and feasible for use during clinical encounters. Clinicians indicated a willingness to use the tool in real clinical practice, a significant degree of trust in the system's predictions to assist with treatment selection, and reported that the tool helped increase patient understanding of and trust in treatment. The simulation environment allowed for the evaluation of the tool's impact on the physician-patient interaction. CONCLUSIONS: The simulation centre allowed for direct observations of clinician use and impact of the tool on the clinician-patient interaction before clinical studies. It may therefore offer a useful and important environment in the early testing of new technological tools. The present results will inform further tool development and clinician training materials.

Machine learning for early detection of sepsis: an internal and temporal validation study.

OBJECTIVE: Determine if deep learning detects sepsis earlier and more accurately than other models. To evaluate model performance using implementation-oriented metrics that simulate clinical practice. MATERIALS AND METHODS: We trained internally and temporally validated a deep learning model (multi-output Gaussian process and recurrent neural network [MGP-RNN]) to detect sepsis using encounters from adult hospitalized patients at a large tertiary academic center. Sepsis was defined as the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria, a blood culture order, and at least one element of end-organ failure. The training dataset included demographics, comorbidities, vital signs, medication administrations, and labs from October 1, 2014 to December 1, 2015, while the temporal validation dataset was from March 1, 2018 to August 31, 2018. Comparisons were made to 3 machine learning methods, random forest (RF), Cox regression (CR), and penalized logistic regression (PLR), and 3 clinical scores used to detect sepsis, SIRS, quick Sequential Organ Failure Assessment (qSOFA), and National Early Warning Score (NEWS). Traditional discrimination statistics such as the C-statistic as well as metrics aligned with operational implementation were assessed. RESULTS: The training set and internal validation included 42 979 encounters, while the temporal validation set included 39 786 encounters. The C-statistic for predicting sepsis within 4 h of onset was 0.88 for the MGP-RNN compared to 0.836 for RF, 0.849 for CR, 0.822 for PLR, 0.756 for SIRS, 0.619 for NEWS, and 0.481 for qSOFA. MGP-RNN detected sepsis a median of 5 h in advance. Temporal validation assessment continued to show the MGP-RNN outperform all 7 clinical risk score and machine learning comparisons. CONCLUSIONS: We developed and validated a novel deep learning model to detect sepsis. Using our data elements and feature set, our modeling approach outperformed other machine learning methods and clinical scores.

Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study.

BACKGROUND: Successful integrations of machine learning into routine clinical care are exceedingly rare, and barriers to its adoption are poorly characterized in the literature. OBJECTIVE: This study aims to report a quality improvement effort to integrate a deep learning sepsis detection and management platform, Sepsis Watch, into routine clinical care. METHODS: In 2016, a multidisciplinary team consisting of statisticians, data scientists, data engineers, and clinicians was assembled by the leadership of an academic health system to radically improve the detection and treatment of sepsis. This report of the quality improvement effort follows the learning health system framework to describe the problem assessment, design, development, implementation, and evaluation plan of Sepsis Watch. RESULTS: Sepsis Watch was successfully integrated into routine clinical care and reshaped how local machine learning projects are executed. Frontline clinical staff were highly engaged in the design and development of the workflow, machine learning model, and application. Novel machine learning methods were developed to detect sepsis early, and implementation of the model required robust infrastructure. Significant investment was required to align stakeholders, develop trusting relationships, define roles and responsibilities, and to train frontline staff, leading to the establishment of 3 partnerships with internal and external research groups to evaluate Sepsis Watch. CONCLUSIONS: Machine learning models are commonly developed to enhance clinical decision making, but successful integrations of machine learning into routine clinical care are rare. Although there is no playbook for integrating deep learning into clinical care, learnings from the Sepsis Watch integration can inform efforts to develop machine learning technologies at other health care delivery systems.

Initial Clinical Impressions of the Critical Care of COVID-19 Patients in Seattle, New York City, and Chicago.

Since the first recognition of a cluster of novel respiratory viral infections in China in late December 2019, intensivists in the United States have watched with growing concern as infections with the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus-now named coronavirus disease of 2019 (COVID-19)-have spread to hospitals in the United States. Because COVID-19 is extremely transmissible and can progress to a severe form of respiratory failure, the potential to overwhelm available critical care resources is high and critical care management of COVID-19 patients has been thrust into the spotlight. COVID-19 arrived in the United States in January and, as anticipated, has dramatically increased the usage of critical care resources. Three of the hardest-hit cities have been Seattle, New York City, and Chicago with a combined total of over 14,000 cases as of March 23, 2020.In this special article, we describe initial clinical impressions of critical care of COVID-19 in these areas, with attention to clinical presentation, laboratory values, organ system effects, treatment strategies, and resource management. We highlight clinical observations that align with or differ from already published reports. These impressions represent only the early empiric experience of the authors and are not intended to serve as recommendations or guidelines for practice, but rather as a starting point for intensivists preparing to address COVID-19 when it arrives in their community.

Midsagittal tissue bridges are associated with walking ability in incomplete spinal cord injury: A magnetic resonance imaging case series.

Context: Following spinal cord injury (SCI), early prediction of future walking ability is difficult, due to factors such as spinal shock, sedation, impending surgery, and secondary long bone fracture. Accurate, objective biomarkers used in the acute stage of SCI would inform individualized patient management and enhance both patient/family expectations and treatment outcomes. Using magnetic resonance imaging (MRI) and specifically a midsagittal T2-weighted image, the amount of tissue bridging (measured as spared spinal cord tissue) shows potential to serve as such a biomarker. Ten participants with incomplete SCI received MRI of the spinal cord. Using the midsagittal T2-weighted image, anterior and posterior tissue bridges were calculated as the distance from cerebrospinal fluid to the damage. Then, the midsagittal tissue bridge ratio was calculated as the sum of anterior and posterior tissue bridges divided by the spinal cord diameter. Each participant also performed a 6-minute walk test, where the total distance walked was measured within six minutes.Findings: The midsagittal tissue bridge ratio measure demonstrated a high level of inter-rater reliability (ICC = 0.90). Midsagittal tissue bridge ratios were significantly related to distance walked in six minutes (R = 0.68, P = 0.03).Conclusion/clinical relevance: We uniquely demonstrated that midsagittal tissue bridge ratios were correlated walking ability. These preliminary findings suggest potential for this measure to be considered a prognostic biomarker of residual walking ability following SCI.

Massive computational acceleration by using neural networks to emulate mechanism-based biological models.

For many biological applications, exploration of the massive parametric space of a mechanism-based model can impose a prohibitive computational demand. To overcome this limitation, we present a framework to improve computational efficiency by orders of magnitude. The key concept is to train a neural network using a limited number of simulations generated by a mechanistic model. This number is small enough such that the simulations can be completed in a short time frame but large enough to enable reliable training. The trained neural network can then be used to explore a much larger parametric space. We demonstrate this notion by training neural networks to predict pattern formation and stochastic gene expression. We further demonstrate that using an ensemble of neural networks enables the self-contained evaluation of the quality of each prediction. Our work can be a platform for fast parametric space screening of biological models with user defined objectives.

Author Correction: Do no harm: a roadmap for responsible machine learning for health care.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Do no harm: a roadmap for responsible machine learning for health care.

Interest in machine-learning applications within medicine has been growing, but few studies have progressed to deployment in patient care. We present a framework, context and ultimately guidelines for accelerating the translation of machine-learning-based interventions in health care. To be successful, translation will require a team of engaged stakeholders and a systematic process from beginning (problem formulation) to end (widespread deployment).

Establishing the inter-rater reliability of spinal cord damage manual measurement using magnetic resonance imaging.

Study design: Retrospective study. Objectives: To establish the inter-rater reliability in the quantitative evaluation of spinal cord damage following cervical incomplete spinal cord injury (SCI) utilizing magnetic resonance imaging (MRI). MRI was used to perform manual measurements of the cranial and caudal boundaries of edema, edema length, midsagittal tissue bridge ratio, axial damage ratio, and edema volume in 10 participants with cervical incomplete SCI. Setting: Academic university setting. Methods: Structural MRIs of 10 participants with SCI were collected from Northwestern University's Neuromuscular Imaging and Research Lab. All manual measures were performed using OsiriX (Pixmeo Sarl, Geneva, Switzerland). Intraclass correlation coefficients (ICC) were used to determine inter-rater reliability across seven raters of varying experience. Results: High-to-excellent inter-rater reliability was found for all measures. ICC values for cranial/caudal levels of involvement, edema length, midsagittal tissue bridge ratio, axial damage ratio, and edema volume were 0.99, 0.98, 0.90, 0.84, and 0.93, respectively. Conclusions: Manual MRI measures of spinal cord damage are reliable between raters. Researchers and clinicians may confidently utilize manual MRI measures to quantify cord damage. Future research to predict functional recovery following SCI and better inform clinical management is warranted.

Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study.

BACKGROUND: Pythia is an automated, clinically curated surgical data pipeline and repository housing all surgical patient electronic health record (EHR) data from a large, quaternary, multisite health institute for data science initiatives. In an effort to better identify high-risk surgical patients from complex data, a machine learning project trained on Pythia was built to predict postoperative complication risk. METHODS AND FINDINGS: A curated data repository of surgical outcomes was created using automated SQL and R code that extracted and processed patient clinical and surgical data across 37 million clinical encounters from the EHRs. A total of 194 clinical features including patient demographics (e.g., age, sex, race), smoking status, medications, comorbidities, procedure information, and proxies for surgical complexity were constructed and aggregated. A cohort of 66,370 patients that had undergone 99,755 invasive procedural encounters between January 1, 2014, and January 31, 2017, was studied further for the purpose of predicting postoperative complications. The average complication and 30-day postoperative mortality rates of this cohort were 16.0% and 0.51%, respectively. Least absolute shrinkage and selection operator (lasso) penalized logistic regression, random forest models, and extreme gradient boosted decision trees were trained on this surgical cohort with cross-validation on 14 specific postoperative outcome groupings. Resulting models had area under the receiver operator characteristic curve (AUC) values ranging between 0.747 and 0.924, calculated on an out-of-sample test set from the last 5 months of data. Lasso penalized regression was identified as a high-performing model, providing clinically interpretable actionable insights. Highest and lowest performing lasso models predicted postoperative shock and genitourinary outcomes with AUCs of 0.924 (95% CI: 0.901, 0.946) and 0.780 (95% CI: 0.752, 0.810), respectively. A calculator requiring input of 9 data fields was created to produce a risk assessment for the 14 groupings of postoperative outcomes. A high-risk threshold (15% risk of any complication) was determined to identify high-risk surgical patients. The model sensitivity was 76%, with a specificity of 76%. Compared to heuristics that identify high-risk patients developed by clinical experts and the ACS NSQIP calculator, this tool performed superiorly, providing an improved approach for clinicians to estimate postoperative risk for patients. Limitations of this study include the missingness of data that were removed for analysis. CONCLUSIONS: Extracting and curating a large, local institution's EHR data for machine learning purposes resulted in models with strong predictive performance. These models can be used in clinical settings as decision support tools for identification of high-risk patients as well as patient evaluation and care management. Further work is necessary to evaluate the impact of the Pythia risk calculator within the clinical workflow on postoperative outcomes and to optimize this data flow for future machine learning efforts.

Ketamine Anesthesia Does Not Improve Depression Scores in Electroconvulsive Therapy: A Randomized Clinical Trial.

BACKGROUND: Although interest in ketamine use during electroconvulsive therapy (ECT) has increased, studies have been equivocal with regard to its efficacy. The aims of this clinical trial were to evaluate ketamine's antidepressive effects in ECT as a primary anesthetic, determine ketamine's tolerability when compared with standard anesthesia, and determine if plasma brain-derived neurotrophic factor (BDNF) is necessary for treatment response. MATERIALS AND METHODS: Adults meeting criteria for treatment-resistant depression undergoing index course ECT received either methohexital (1 to 2 mg/kg) or ketamine (1 to 2 mg/kg) anesthesia in this dual-arm double-blinded randomized clinical trial (NCT02752724). The primary outcome of this study is change in depression questionnaire scores before and after ECT. Seizure data, depression severity using self-reported and clinician-assessed questionnaires, cognitive scoring, and plasma BDNF concentrations were obtained before and after completion of ECT. RESULTS: There were no differences in seizure lengths, hemodynamics, or seizure stimuli between the ketamine (n=23;138 ECTs) and methohexital (n=27;159 ECTs) groups. Depression scores improved similarly after ECT in both groups. In the methohexital group, 15% of patients failed to achieve adequate seizures and were switched to ketamine and 26% were converted to bilateral ECT stimulus, whereas all ketamine patients achieved adequate seizures and only 4% required bilateral stimulus. Plasma BDNF increased after ECT only in the ketamine group. CONCLUSIONS: Our data show that ketamine does not significantly improve depression when compared with methohexital as a single induction agent for ECT, increases serum BDNF and does not increase rates of post-ECT agitation. Ketamine use in ECT may have some benefits for some patients that are not captured through standard depression assessment questionnaires alone.