External Validation and Updating of a Statistical Civilian-Based Suicide Risk Model in US Naval Primary Care.

Importance: Suicide remains an ongoing concern in the US military. Statistical models have not been broadly disseminated for US Navy service members. Objective: To externally validate and update a statistical suicide risk model initially developed in a civilian setting with an emphasis on primary care. Design, Setting, and Participants: This retrospective cohort study used data collected from 2007 through 2017 among active-duty US Navy service members. The external civilian model was applied to every visit at Naval Medical Center Portsmouth (NMCP), its NMCP Naval Branch Health Clinics (NBHCs), and TRICARE Prime Clinics (TPCs) that fall within the NMCP area. The model was retrained and recalibrated using visits to NBHCs and TPCs and updated using Department of Defense (DoD)-specific billing codes and demographic characteristics, including expanded race and ethnicity categories. Domain and temporal analyses were performed with bootstrap validation. Data analysis was performed from September 2020 to December 2022. Exposure: Visit to US NMCP. Main Outcomes and Measures: Recorded suicidal behavior on the day of or within 30 days of a visit. Performance was assessed using area under the receiver operating curve (AUROC), area under the precision recall curve (AUPRC), Brier score, and Spiegelhalter z-test statistic. Results: Of the 260 583 service members, 6529 (2.5%) had a recorded suicidal behavior, 206 412 (79.2%) were male; 104 835 (40.2%) were aged 20 to 24 years; and 9458 (3.6%) were Asian, 56 715 (21.8%) were Black or African American, and 158 277 (60.7%) were White. Applying the civilian-trained model resulted in an AUROC of 0.77 (95% CI, 0.74-0.79) and an AUPRC of 0.004 (95% CI, 0.003-0.005) at NBHCs with poor calibration (Spiegelhalter P < .001). Retraining the algorithm improved AUROC to 0.92 (95% CI, 0.91-0.93) and AUPRC to 0.66 (95% CI, 0.63-0.68). Number needed to screen in the top risk tiers was 366 for the external model and 200 for the retrained model; the lower number indicates better performance. Domain validation showed AUROC of 0.90 (95% CI, 0.90-0.91) and AUPRC of 0.01 (95% CI, 0.01-0.01), and temporal validation showed AUROC of 0.75 (95% CI, 0.72-0.78) and AUPRC of 0.003 (95% CI, 0.003-0.005). Conclusions and Relevance: In this cohort study of active-duty Navy service members, a civilian suicide attempt risk model was externally validated. Retraining and updating with DoD-specific variables improved performance. Domain and temporal validation results were similar to external validation, suggesting that implementing an external model in US Navy primary care clinics may bypass the need for costly internal development and expedite the automation of suicide prevention in these clinics.

Screening Health-Related Social Needs in Hospitals: A Systematic Review of Health Care Professional and Patient Perspectives.

Health outcomes are markedly influenced by health-related social needs (HRSN) such as food insecurity and housing instability. Under new Joint Commission requirements, hospitals have recently increased attention to HRSN to reduce health disparities. To evaluate prevailing attitudes and guide hospital efforts, the authors conducted a systematic review to describe patients' and health care providers' perceptions related to screening for and addressing patients' HRSN in US hospitals. Articles were identified through PubMed and by expert recommendations, and synthesized by relevance of findings and basic study characteristics. The review included 22 articles, which showed that most health care providers believed that unmet social needs impact health and that screening for HRSN should be a standard part of hospital care. Notable differences existed between perceived importance of HRSN and actual screening rates, however. Patients reported high receptiveness to screening in hospital encounters, but cautioned to avoid stigmatization and protect privacy when screening. Limited knowledge of resources available, lack of time, and lack of actual resources were the most frequently reported barriers to screening for HRSN. Hospital efforts to screen and address HRSN will likely be facilitated by stakeholders' positive perceptions, but common barriers to screening and referral will need to be addressed to effectively scale up efforts and impact health disparities.

Considering Clinician Competencies for the Implementation of Artificial Intelligence-Based Tools in Health Care: Findings From a Scoping Review.

BACKGROUND: The use of artificial intelligence (AI)-based tools in the care of individual patients and patient populations is rapidly expanding. OBJECTIVE: The aim of this paper is to systematically identify research on provider competencies needed for the use of AI in clinical settings. METHODS: A scoping review was conducted to identify articles published between January 1, 2009, and May 1, 2020, from MEDLINE, CINAHL, and the Cochrane Library databases, using search queries for terms related to health care professionals (eg, medical, nursing, and pharmacy) and their professional development in all phases of clinical education, AI-based tools in all settings of clinical practice, and professional education domains of competencies and performance. Limits were provided for English language, studies on humans with abstracts, and settings in the United States. RESULTS: The searches identified 3476 records, of which 4 met the inclusion criteria. These studies described the use of AI in clinical practice and measured at least one aspect of clinician competence. While many studies measured the performance of the AI-based tool, only 4 measured clinician performance in terms of the knowledge, skills, or attitudes needed to understand and effectively use the new tools being tested. These 4 articles primarily focused on the ability of AI to enhance patient care and clinical decision-making by improving information flow and display, specifically for physicians. CONCLUSIONS: While many research studies were identified that investigate the potential effectiveness of using AI technologies in health care, very few address specific competencies that are needed by clinicians to use them effectively. This highlights a critical gap.

Trust in AI: why we should be designing for APPROPRIATE reliance.

Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.

Before-Visit Questionnaire: A Tool to Augment Communication and Decrease Provider Documentation Burden in Pediatric Diabetes.

OBJECTIVE: To develop and evaluate an electronic tool that collects interval history and incorporates it into a provider summary note. METHODS: A parent-facing online before-visit questionnaire (BVQ) collected information from parents and caregivers of pediatric diabetes patients prior to a clinic encounter. This information was related to interval history and perceived self-management barriers. The BVQ generated a summary note that providers could paste in their own documentation. Parents also completed postvisit experience questionnaires. We assessed the BVQs perceived usefulness to parents and providers and compared provider documentation content and length pre- and post-BVQ rollout. We interviewed providers regarding their experiences with the system-generated note. RESULTS: Seventy-three parents of diabetic children were recruited and completed the BVQ. A total of 79% of parents stated that the BVQ helped with visit preparation and 80% said it improved perceived quality of visits. All 16 participating providers reviewed BVQs prior to patient encounters and 100% considered the summary beneficial. Most providers (81%) desired summaries less than 1 week old. A total of 69% of providers preferred the prose version of the summary; however, 75% also viewed the bulleted version as preferable for provider review. Analysis of provider notes revealed that BVQs increased provider documentation of patients' adherence and barriers. We observed a 50% reduction in typing by providers to document interval histories. Providers not using summaries typed an average of 137 words (standard deviation [SD]: 74) to document interval history compared with 68 words [SD 47] typed with BVQ use. DISCUSSION: Providers and parents of children with diabetes appreciated the use of previsit, parent-completed BVQs that automatically produced provider documentation. Despite the BVQ redistributing work from providers to parents, its use was acceptable to both groups. CONCLUSION: Parent-completed questionnaires on the patient's behalf that generate provider documentation encourage communication between parents and providers regarding disease management and reduce provider workload.

Design thinking in applied informatics: what can we learn from Project HealthDesign?

OBJECTIVE: The goals of this study are to describe the value and impact of Project HealthDesign (PHD), a program of the Robert Wood Johnson Foundation that applied design thinking to personal health records, and to explore the applicability of the PHD model to another challenging translational informatics problem: the integration of AI into the healthcare system. MATERIALS AND METHODS: We assessed PHD's impact and value in 2 ways. First, we analyzed publication impact by calculating a PHD h-index and characterizing the professional domains of citing journals. Next, we surveyed and interviewed PHD grantees, expert consultants, and codirectors to assess the program's components and the potential future application of design thinking to artificial intelligence (AI) integration into healthcare. RESULTS: There was a total of 1171 unique citations to PHD-funded work (collective h-index of 25). Studies citing PHD span medical, legal, and computational journals. Participants stated that this project transformed their thinking, altered their career trajectory, and resulted in technology transfer into the commercial sector. Participants felt, in general, that the approach would be valuable in solving contemporary challenges integrating AI in healthcare including complex social questions, integrating knowledge from multiple domains, implementation, and governance. CONCLUSION: Design thinking is a systematic approach to problem-solving characterized by cooperation and collaboration. PHD generated significant impacts as measured by citations, reach, and overall effect on participants. PHD's design thinking methods are potentially useful to other work on cyber-physical systems, such as the use of AI in healthcare, to propose structural or policy-related changes that may affect adoption, value, and improvement of the care delivery system.

Visualization of aggregate perioperative data improves anesthesia case planning: A randomized, cross-over trial.

STUDY OBJECTIVE: A challenge in reducing unwanted care variation is effectively managing the wide variety of performed surgical procedures. While an organization may perform thousands of types of cases, privacy and logistical constraints prevent review of previous cases to learn about prior practices. To bridge this gap, we developed a system for extracting key data from anesthesia records. Our objective was to determine whether usage of the system would improve case planning performance for anesthesia residents. DESIGN: Randomized, cross-over trial. SETTING: Vanderbilt University Medical Center. MEASUREMENTS: We developed a web-based, data visualization tool for reviewing de-identified anesthesia records. First year anesthesia residents were recruited and performed simulated case planning tasks (e.g., selecting an anesthetic type) across six case scenarios using a randomized, cross-over design after a baseline assessment. An algorithm scored case planning performance based on care components selected by residents occurring frequently among prior anesthetics, which was scored on a 0-4 point scale. Linear mixed effects regression quantified the tool effect on the average performance score, adjusting for potential confounders. MAIN RESULTS: We analyzed 516 survey questionnaires from 19 residents. The mean performance score was 2.55 ± SD 0.32. Utilization of the tool was associated with an average score improvement of 0.120 points (95% CI 0.060 to 0.179; p < 0.001). Additionally, a 0.055 point improvement due to the "learning effect" was observed from each assessment to the next (95% CI 0.034 to 0.077; p < 0.001). Assessment score was also significantly associated with specific case scenarios (p < 0.001). CONCLUSIONS: This study demonstrated the feasibility of developing of a clinical data visualization system that aggregated key anesthetic information and found that the usage of tools modestly improved residents' performance in simulated case planning.

User-Centered Clinical Display Design Issues for Inpatient Providers.

BACKGROUND: Suboptimal information display in electronic health records (EHRs) is a notorious pain point for users. Designing an effective display is difficult, due in part to the complex and varied nature of clinical practice. OBJECTIVE: This article aims to understand the goals, constraints, frustrations, and mental models of inpatient medical providers when accessing EHR data, to better inform the display of clinical information. METHODS: A multidisciplinary ethnographic study of inpatient medical providers. RESULTS: Our participants' primary goal was usually to assemble a clinical picture around a given question, under the constraints of time pressure and incomplete information. To do so, they tend to use a mental model of multiple layers of abstraction when thinking of patients and disease; they prefer immediate pattern recognition strategies for answering clinical questions, with breadth-first or depth-first search strategies used subsequently if needed; and they are sensitive to data relevance, completeness, and reliability when reading a record. CONCLUSION: These results conflict with the ubiquitous display design practice of separating data by type (test results, medications, notes, etc.), a mismatch that is known to encumber efficient mental processing by increasing both navigation burden and memory demands on users. A popular and obvious solution is to select or filter the data to display exactly what is presumed to be relevant to the clinical question, but this solution is both brittle and mistrusted by users. A less brittle approach that is more aligned with our users' mental model could use abstraction to summarize details instead of filtering to hide data. An abstraction-based approach could allow clinicians to more easily assemble a clinical picture, to use immediate pattern recognition strategies, and to adjust the level of displayed detail to their particular needs. It could also help the user notice unanticipated patterns and to fluidly shift attention as understanding evolves.

User-Centered Design of a Machine Learning Intervention for Suicide Risk Prediction in a Military Setting.

Primary care represents a major opportunity for suicide prevention in the military. Significant advances have been made in using electronic health record data to predict suicide attempts in patient populations. With a user-centered design approach, we are developing an intervention that uses predictive analytics to inform care teams about their patients' risk of suicide attempt. We present our experience working with clinicians and staff in a military primary care setting to create preliminary designs and a context-specific usability testing plan for the deployment of the suicide risk indicator.

A taxonomy of impacts on clinical and translational research from community stakeholder engagement.

BACKGROUND: Community engagement is increasingly recognized as a valuable tool in clinical and translational research; however, the impact of engagement is not fully understood. No standard nomenclature yet exists to clearly define how research changes when community stakeholders are engaged across the research spectrum. This severely limits our ability to assess the value of community engagement in research. To address this gap, we developed a taxonomy for characterizing and classifying changes in research due to community engagement. METHODS: Using an iterative process, we (a) identified areas of potential impact associated with community engagement from author experience, (b) categorized these in taxonomic bins based on research stages, (c) conducted semi-structured interviews with researchers and community stakeholders, (d) validated the codebook in a sample dataset and (e) refined the taxonomy based on the validation. Community stakeholders were involved in every step of the process including as members of the primary study team. RESULTS: The final taxonomy catalogues changes into eleven domains corresponding to research phases. Each domain includes 2-4 dimensions depicting concepts within the domain's scope and, within each dimension, 2-10 elements labelling activities through which community engagement could change research. CONCLUSIONS: Community engagement has great potential to enhance clinical and translational research. This taxonomy provides a common vocabulary and framework for understanding the impact of community engagement and suggests metrics for assessing the value of community engagement in research.

Supporting Caregivers in Pregnancy: A Qualitative Study of Their Activities and Roles.

BACKGROUND: The significant role of lay caregivers has been explored in chronic and acute illnesses. In pregnancy, caregivers' (eg, the baby's father, friends, and family) roles in promoting the health of the mother and baby are not well understood. OBJECTIVE: We characterize the activities and roles of pregnancy caregivers and offer opportunities for engaging this important group. METHOD: We conducted interviews with 29 pregnancy caregivers. Interview transcripts were analyzed inductively, resulting in a coding scheme of actions and roles that pregnancy caregivers perform. RESULTS: The most common actions and roles included searching for information (97%), accompanying patients to medical appointments (69%), and being a source of emotional support (76%). Identified actions and roles fit a patient work framework, including work types identified by Corbin and Strauss: illness, everyday life, biographical, articulation, and invisible. CONCLUSION: The patient work framework can be employed to describe the activities and roles of pregnancy caregivers. We have contributed new insights into the experiences of pregnancy caregivers and recommendations for educational and technological interventions.

Physicians' perspectives on receiving unsolicited genomic results.

PURPOSE: Physicians increasingly receive genomic test results they did not order, which we term "unsolicited genomic results" (UGRs). We asked physicians how they think such results will affect them and their patients. METHODS: Semistructured interviews were conducted with adult and pediatric primary care and subspecialty physicians at four sites affiliated with a large-scale return-of-results project led by the Electronic Medical Records and Genomics (eMERGE) Network. Twenty-five physicians addressed UGRs and (1) perceived need for actionability, (2) impact on patients, (3) health care workflow, (4) return of results process, and (5) responsibility for results. RESULTS: Physicians prioritize actionability of UGRs and the need for clear, evidence-based "paths" for action coupled with clinical decision support (CDS). They identified potential harms to patients including anxiety, false reassurance, and clinical disutility. Clinicians worried about anticipated workflow issues including responding to UGRs and unreimbursed time. They disagreed about who was responsible for responding to UGRs. CONCLUSION: The prospect of receiving UGRs for otherwise healthy patients raises important concerns for physicians. Their responses informed development of an in-depth survey for physicians following return of UGRs. Strategic workflow integration of UGRs will likely be necessary to empower physicians to serve their patients effectively.

Feasibility Assessment of a Pre-Hospital Automated Sensing Clinical Documentation System.

Clinical documentation in the pre-hospital setting is challenged by limited resources and fast-paced, high-acuity. Military and civilian medics are responsible for performing procedures and treatments to stabilize the patient, while transporting the injured to a trauma facility. Upon arrival, medics typically give a verbal report from memory or informal source of documentation such as a glove or piece of tape. The development of an automated documentation system would increase the accuracy and amount of information that is relayed to the receiving physicians. This paper discusses the 12-week deployment of an Automated Sensing Clinical Documentation (ASCD) system among the Nashville Fire Department EMS paramedics. The paper examines the data collection methods, operational challenges, and perceptions surrounding real-life deployment of the system. Our preliminary results suggest that the ASCD system is feasible for use in the pre-hospital setting, and it revealed several barriers and their solutions.

Consumer health-related needs of pregnant women and their caregivers.

OBJECTIVES: To build effective applications, technology designers must understand consumer health needs. Pregnancy is a common health condition, and expectant families have unanswered questions. This study examined consumer health-related needs in pregnant women and caregivers and determined the types of needs that were not met. MATERIALS AND METHODS: We enrolled pregnant women <36 weeks' gestational age and caregivers from advanced maternal-fetal and group prenatal care settings. Participant characteristics were collected through surveys, and health-related needs were elicited in semi-structured interviews. Researchers categorized needs by semantic type and whether they were met (ie, met, partially met, or unmet). Inter-rater reliability was measured by Cohen's kappa. RESULTS: Seventy-one pregnant women and 29 caregivers participated and reported 1054 needs, 28% unmet, and 49% partially met. Need types were 66.2% informational, 15.9% logistical, 8.9% social, 8.6% medical, and 0.3% other. Inter-rater reliability was near perfect (κ=0.95, P < 0.001). DISCUSSION: Common topics of unmet needs were prognosis, life management, and need for emotional support. For pregnant women, these unmet needs focused around being healthy, childbirth, infant care, and being a good mother; caregivers' needs involved caring for the mother, the natural course of pregnancy, and life after pregnancy. CONCLUSION: Pregnant women and caregivers have a rich set of health-related needs with many not fully met. Caregivers' needs differed from those of pregnant women and may not be adequately addressed by resources designed for mothers. Many unmet needs involved stress and life management. Knowledge about consumer health needs can inform the design of better technologies for pregnancy.

Advancing In-Hospital Clinical Deterioration Prediction Models.

BACKGROUND: Early warning systems lack robust evidence that they improve patients' outcomes, possibly because of their limitation of predicting binary rather than time-to-event outcomes. OBJECTIVES: To compare the prediction accuracy of 2 statistical modeling strategies (logistic regression and Cox proportional hazards regression) and 2 machine learning strategies (random forest and random survival forest) for in-hospital cardiopulmonary arrest. METHODS: Retrospective cohort study with prediction model development from deidentified electronic health records at an urban academic medical center. RESULTS: The classification models (logistic regression and random forest) had statistical recall and precision similar to or greater than those of the time-to-event models (Cox proportional hazards regression and random survival forest). However, the time-to-event models provided predictions that could potentially better indicate to clinicians whether and when a patient is likely to experience cardiopulmonary arrest. CONCLUSIONS: As early warning scoring systems are refined, they must use the best analytical methods that both model the underlying phenomenon and provide an understandable prediction.

Participatory design of probability-based decision support tools for in-hospital nurses.

OBJECTIVE: To describe nurses' preferences for the design of a probability-based clinical decision support (PB-CDS) tool for in-hospital clinical deterioration. METHODS: A convenience sample of bedside nurses, charge nurses, and rapid response nurses (n = 20) from adult and pediatric hospitals completed participatory design sessions with researchers in a simulation laboratory to elicit preferred design considerations for a PB-CDS tool. Following theme-based content analysis, we shared findings with user interface designers and created a low-fidelity prototype. RESULTS: Three major themes and several considerations for design elements of a PB-CDS tool surfaced from end users. Themes focused on "painting a picture" of the patient condition over time, promoting empowerment, and aligning probability information with what a nurse already believes about the patient. The most notable design element consideration included visualizing a temporal trend of the predicted probability of the outcome along with user-selected overlapping depictions of vital signs, laboratory values, and outcome-related treatments and interventions. Participants expressed that the prototype adequately operationalized requests from the design sessions. CONCLUSIONS: Participatory design served as a valuable method in taking the first step toward developing PB-CDS tools for nurses. This information about preferred design elements of tools that support, rather than interrupt, nurses' cognitive workflows can benefit future studies in this field as well as nurses' practice.