Evaluating a Large Language Model's Ability to Answer Clinicians' Requests for Evidence Summaries.

Objective: This study investigated the performance of a generative artificial intelligence (AI) tool using GPT-4 in answering clinical questions in comparison with medical librarians' gold-standard evidence syntheses. Methods: Questions were extracted from an in-house database of clinical evidence requests previously answered by medical librarians. Questions with multiple parts were subdivided into individual topics. A standardized prompt was developed using the COSTAR framework. Librarians submitted each question into aiChat, an internally-managed chat tool using GPT-4, and recorded the responses. The summaries generated by aiChat were evaluated on whether they contained the critical elements used in the established gold-standard summary of the librarian. A subset of questions was randomly selected for verification of references provided by aiChat. Results: Of the 216 evaluated questions, aiChat's response was assessed as "correct" for 180 (83.3%) questions, "partially correct" for 35 (16.2%) questions, and "incorrect" for 1 (0.5%) question. No significant differences were observed in question ratings by question category (p=0.39). For a subset of 30% (n=66) of questions, 162 references were provided in the aiChat summaries, and 60 (37%) were confirmed as nonfabricated. Conclusions: Overall, the performance of a generative AI tool was promising. However, many included references could not be independently verified, and attempts were not made to assess whether any additional concepts introduced by aiChat were factually accurate. Thus, we envision this being the first of a series of investigations designed to further our understanding of how current and future versions of generative AI can be used and integrated into medical librarians' workflow.

Using a Natural Language Processing Approach to Support Rapid Knowledge Acquisition.

Implementing artificial intelligence to extract insights from large, real-world clinical data sets can supplement and enhance knowledge management efforts for health sciences research and clinical care. At Vanderbilt University Medical Center (VUMC), the in-house developed Word Cloud natural language processing system extracts coded concepts from patient records in VUMC's electronic health record repository using the Unified Medical Language System terminology. Through this process, the Word Cloud extracts the most prominent concepts found in the clinical documentation of a specific patient or population. The Word Cloud provides added value for clinical care decision-making and research. This viewpoint paper describes a use case for how the VUMC Center for Knowledge Management leverages the condition-disease associations represented by the Word Cloud to aid in the knowledge generation needed to inform the interpretation of phenome-wide association studies.

Knowledgebase strategies to aid interpretation of clinical correlation research.

OBJECTIVE: Knowledgebases are needed to clarify correlations observed in real-world electronic health record (EHR) data. We posit design principles, present a unifying framework, and report a test of concept. MATERIALS AND METHODS: We structured a knowledge framework along 3 axes: condition of interest, knowledge source, and taxonomy. In our test of concept, we used hypertension as our condition of interest, literature and VanderbiltDDx knowledgebase as sources, and phecodes as our taxonomy. In a cohort of 832 566 deidentified EHRs, we modeled blood pressure and heart rate by sex and age, classified individuals by hypertensive status, and ran a Phenome-wide Association Study (PheWAS) for hypertension. We compared the correlations from PheWAS to the associations in our knowledgebase. RESULTS: We produced PhecodeKbHtn: a knowledgebase comprising 167 hypertension-associated diseases, 15 of which were also negatively associated with blood pressure (pos+neg). Our hypertension PheWAS included 1914 phecodes, 129 of which were in the PhecodeKbHtn. Among the PheWAS association results, phecodes that were in PhecodeKbHtn had larger effect sizes compared with those phecodes not in the knowledgebase. DISCUSSION: Each source contributed unique and additive associations. Models of blood pressure and heart rate by age and sex were consistent with prior cohort studies. All but 4 PheWAS positive and negative correlations for phecodes in PhecodeKbHtn may be explained by knowledgebase associations, hypertensive cardiac complications, or causes of hypertension independently associated with hypotension. CONCLUSION: It is feasible to assemble a knowledgebase that is compatible with EHR data to aid interpretation of clinical correlation research.

Systematic review of international studies evaluating MDRD and CKD-EPI estimated glomerular filtration rate (eGFR) equations in Black adults.

Use of race adjustment in estimating glomerular filtration rate (eGFR) has been challenged given concerns that it may negatively impact the clinical care of Black patients, as it results in Black patients being systematically assigned higher eGFR values than non-Black patients. We conducted a systematic review to assess how well eGFR, with and without race adjustment, estimates measured GFR (mGFR) in Black adults globally. A search across multiple databases for articles published from 1999 to May 2021 that compared eGFR to mGFR and reported outcomes by Black race was performed. We included studies that assessed eGFR using the Modification of Diet in Renal Disease (MDRD) and Chronic Kidney Disease Epidemiology Collaboration (CKD-EPICr) creatinine equations. Risk of study bias and applicability were assessed with the QUality Assessment of Diagnostic Accuracy Studies-2. Of 13,167 citations identified, 12 met the data synthesis criteria (unique patient cohorts in which eGFR was compared to mGFR with and without race adjustment). The studies included patients with and without kidney disease from Africa (n = 6), the United States (n = 3), Europe (n = 2), and Brazil (n = 1). Of 11 CKD-EPI equation studies, all assessed bias, 8 assessed accuracy, 6 assessed precision, and 5 assessed correlation/concordance. Of 7 MDRD equation studies, all assessed bias, 6 assessed accuracy, 5 assessed precision, and 3 assessed correlation/concordance. The majority of studies found that removal of race adjustment improved bias, accuracy, and precision of eGFR equations for Black adults. Risk of study bias was often unclear, but applicability concerns were low. Our systematic review supports the need for future studies to be conducted in diverse populations to assess the possibility of alternative approaches for estimating GFR. This study additionally provides systematic-level evidence for the American Society of Nephrology-National Kidney Foundation Task Force efforts to pursue other options for GFR estimation.

Using SPI-Hub™ to Promote the Key Role of Prepublishing in Healthcare.

With the need to quickly advance knowledge dissemination in rapid-paced fields, and more recently in response to the urgency of the COVID-19 pandemic, prepublishing has been brought to the forefront. SPI-Hub™, a publicly available journal selection decision support tool, is being strategically enhanced to address prospective authors' critical needs in navigating and selecting the most appropriate preprint or traditional publication venue.

Design thinking in applied informatics: what can we learn from Project HealthDesign?

OBJECTIVE: The goals of this study are to describe the value and impact of Project HealthDesign (PHD), a program of the Robert Wood Johnson Foundation that applied design thinking to personal health records, and to explore the applicability of the PHD model to another challenging translational informatics problem: the integration of AI into the healthcare system. MATERIALS AND METHODS: We assessed PHD's impact and value in 2 ways. First, we analyzed publication impact by calculating a PHD h-index and characterizing the professional domains of citing journals. Next, we surveyed and interviewed PHD grantees, expert consultants, and codirectors to assess the program's components and the potential future application of design thinking to artificial intelligence (AI) integration into healthcare. RESULTS: There was a total of 1171 unique citations to PHD-funded work (collective h-index of 25). Studies citing PHD span medical, legal, and computational journals. Participants stated that this project transformed their thinking, altered their career trajectory, and resulted in technology transfer into the commercial sector. Participants felt, in general, that the approach would be valuable in solving contemporary challenges integrating AI in healthcare including complex social questions, integrating knowledge from multiple domains, implementation, and governance. CONCLUSION: Design thinking is a systematic approach to problem-solving characterized by cooperation and collaboration. PHD generated significant impacts as measured by citations, reach, and overall effect on participants. PHD's design thinking methods are potentially useful to other work on cyber-physical systems, such as the use of AI in healthcare, to propose structural or policy-related changes that may affect adoption, value, and improvement of the care delivery system.

SPI-Hub™: a gateway to scholarly publishing information.

BACKGROUND: Advances in the health sciences rely on sharing research and data through publication. As information professionals are often asked to contribute their knowledge to assist clinicians and researchers in selecting journals for publication, the authors recognized an opportunity to build a decision support tool, SPI-Hub: Scholarly Publishing Information Hub™, to capture the team's collective publishing industry knowledge, while carefully retaining the quality of service. CASE PRESENTATION: SPI-Hub's decision support functionality relies on a data framework that describes journal publication policies and practices through a newly designed metadata structure, the Knowledge Management Journal Record™. Metadata fields are populated through a semi-automated process that uses custom programming to access content from multiple sources. Each record includes 25 metadata fields representing best publishing practices. Currently, the database includes more than 24,000 health sciences journal records. To correctly capture the resources needed for both completion and future maintenance of the project, the team conducted an internal study to assess time requirements for completing records through different stages of automation. CONCLUSIONS: The journal decision support tool, SPI-Hub, provides an opportunity to assess publication practices by compiling data from a variety of sources in a single location. Automated and semi-automated approaches have effectively reduced the time needed for data collection. Through a comprehensive knowledge management framework and the incorporation of multiple quality points specific to each journal, SPI-Hub provides prospective users with both recommendations for publication and holistic assessment of the trustworthiness of journals in which to publish research and acquire trusted knowledge.

Why equating all evidence searches to systematic reviews defies their role in information seeking.

All too often the quality and rigor of topic investigations is inaccurately conveyed to information professionals, resulting in a mischaracterization of the research, which, if left unchecked and published, may in turn mislead potential readers. Accurately understanding and categorizing the types of topic investigation searches that are requested of information professionals is critical to both meeting requestors' needs and reflecting their intended methodological approaches. Information professionals' expertise can be an invaluable resource to guide users through the investigative and publication process.

Collection of social determinants of health in the community clinic setting: a cross-sectional study.

BACKGROUND: Addressing social and behavioral determinants of health (SBDs) may help improve health outcomes of community clinic patients. This cross-sectional study explored how assessing SBDs can be used to complement health data collection strategies and provide clinicians with a more in-depth understanding of their patients. METHODS: Adult patients, ages 18 and older, at an urban community health care clinic in Tennessee, U.S.A., were asked to complete a questionnaire regarding health status, health history and SBDs while waiting for their clinic appointment. The SBD component included items from the National Academy of Medicine, the Protocol for Responding to and Assessing Patient Assets, Risks, and Experiences instrument, and the Survey of Household Economics and Decisionmaking. Data collection and analysis occurred in 2017. RESULTS: One hundred participants completed the study. The questionnaire took approximately 11 min to complete, and the response rate was 90% or higher for all items except annual household income (unanswered by 40 participants). The median number of negative SBDs was 4 (IQR 2.75-7.0), 96 participants had at least one unmet need, and the most common negative SBD was physical activity (75%; 75/100). CONCLUSIONS: The hybrid questionnaire provided insight into a community clinic population's SBDs and allowed for a more complete understanding than a single questionnaire alone. The brief questionnaire administration time and low non-response rate support the questionnaire's feasibility in the community clinic setting, and results can be used by clinicians to further the personalization goals of precision medicine. Next steps include evaluating how to connect patients with appropriate resources for addressing their SBDs.

National Academy of Medicine Social and Behavioral Measures: Associations With Self-Reported Health.

INTRODUCTION: Social and behavioral factors play important roles in physical and mental health; however, they are not routinely assessed in the healthcare system. A brief panel of measures of social and behavioral determinants of health (SBDs) were recommended in a National Academy of Medicine report for use in electronic health records. Initial testing of the panel established feasibility of use and robustness of the measures. This study evaluates their convergent and divergent validity in relation to self-reported physical and mental health and social desirability bias. METHODS: Adults, aged ≥18 years, were recruited through Qualtrics online panel survey platform in 2015 (data analyzed in 2015-2016). Participants completed the (1) panel of SBD measures; (2) 12-Item Short Form Health Survey to assess associations with global physical and mental health; and (3) Marlowe-Crowne Social Desirability Scale short form to assess whether social desirability influenced associations between SBD measures and self-reported health. RESULTS: The sample included 513 participants (mean age, 47.9 [SD=14.2] years; 65.5% female). Several SBD domain measures were associated with physical and mental health. Adjusting for age, poorer physical and mental health were observed among participants reporting higher levels of financial resource strain, stress, depression, physical inactivity, current tobacco use, and a positive score for intimate partner violence. These associations remained significant after adjustment for social desirability bias. CONCLUSIONS: SBD domains were associated with global measures of physical and mental health and were not impacted by social desirability bias. The panel of SBD measures should now be tested in clinical settings.

Institute of Medicine Measures of Social and Behavioral Determinants of Health: A Feasibility Study.

INTRODUCTION: Social and behavioral factors are known to affect health but are not routinely assessed in medical practice. To date, no studies have assessed a parsimonious panel of measures of social and behavioral determinants of health (SBDs). This study evaluated the panel of SBD measures recommended by the Institute of Medicine and examined the effect of question order. METHODS: Adults, aged ≥18 years, were recruited using ResearchMatch.org for this randomized, parallel design study conducted in 2015 (data analyzed in 2015-2016). Three versions of the SBD measures, sharing the same items but in different orders of presentation (Versions 1-3), were developed. Randomized to six groups, participants completed each version at least 1 week apart (Weeks 1-3). Version order was counterbalanced across each administration and randomization was stratified by gender, race, and age. Main outcomes were effect of question order, completion time, and non-response rates. RESULTS: Of 781 participants, 624 (80%) completed the Week 1 questionnaire; median completion time for answering all SBD questions was 5 minutes, 583/624 participants answered all items, and no statistically significant differences associated with question order were observed when comparing responses across all versions. No significant differences in responses within assignment groups over time were found, with the exception of the stress measure for Group 5 (p=0.036). CONCLUSION: Question order did not significantly impact participant responses. Time to complete the questionnaire was brief, and non-response rate was low. Findings support the feasibility of using the Institute of Medicine-recommended questionnaire to capture SBDs.

My Cancer Genome: Evaluating an Educational Model to Introduce Patients and Caregivers to Precision Medicine Information.

This study tested an innovative model for creating consumer-level content about precision medicine based on health literacy and learning style principles. "Knowledge pearl" videos, incorporating multiple learning modalities, were created to explain genetic and cancer medicine concepts. Cancer patients and caregivers (n=117) were randomized to view professional-level content directly from the My Cancer Genome (MCG) website (Group A; control), content from MCG with knowledge pearls embedded (Group B), or a consumer translation, targeted at the sixth grade level, with knowledge pearls embedded (Group C). A multivariate analysis showed that Group C, but not Group B, showed greater knowledge gains immediately after viewing the educational material than Group A. Statistically significant group differences in test performance were no longer observed three weeks later. These findings suggest that adherence to health literacy and learning style principles facilitates comprehension of precision medicine concepts and that ongoing review of the educational information is necessary.

Guiding Oncology Patients Through the Maze of Precision Medicine.

As the role of genomics in health care grows, patients increasingly require adequate genetic literacy to fully engage in their care. This study investigated a model for delivering consumer-friendly genetic information to improve understanding of precision medicine using health literacy and learning style principles. My Cancer Genome (MCG), a freely available cancer decision support tool, was used as a testbed. MCG content on a melanoma tumor mutation, BRAF V600E, was translated to a 6th-grade reading level, incorporating multiple learning modalities. A total of 90 patients and caregivers were recruited from a melanoma clinic at an academic medical center and randomized to 3 groups. Group A (control) received an exact copy of text from MCG. Group B was given the same content with hyperlinks to videos explaining key genetic concepts, identified and labeled by the team as knowledge pearls. Group C received the translated content with the knowledge pearls embedded. Changes in knowledge were measured through pre and post questionnaires. Group C showed the greatest improvement in knowledge. The study results demonstrate that providing information based on health literacy and learning style principles can improve patients' understanding of genetic concepts, thus increasing their likelihood of taking an active role in any decision making concerning their health.

A personalized approach to deliver health care information to diabetic patients in community care clinics.

OBJECTIVE: The study's purpose was to test the generalizability of an individualized information prescription model, which has been previously validated for educating patients about hypertension in emergency department and community health center settings. Study investigators assessed the effects of educational materials targeted to health literacy levels and learning styles on patients' diabetes knowledge in a community clinic setting. METHODS: From May to August 2012, 160 patients were recruited and randomized into intervention (n = 81) and control (n = 79) groups. Inclusion criteria included 18 years or older, English or Spanish speaker, and a type 2 diabetes diagnosis. Measures included modified versions of the Diabetes Knowledge Test and Subjective Numeracy Scale, along with brief health literacy and learning style assessments. Study team members contacted both groups after 2 and 6 weeks to reassess diabetes knowledge. RESULTS: The control group showed no significant change in diabetes knowledge at both follow-ups. In contrast, the mean number of diabetes knowledge questions answered correctly by the intervention group increased significantly after 2 weeks (Δ = 2.66, P = 0.000), which persisted at 6 weeks (Δ = 2.46, P = 0.00). CONCLUSIONS: This study showed that patients' knowledge about diabetes increased significantly after exposure to educational materials targeted to their health literacy levels and learning style preferences and that the model is transferrable among health conditions.

Strategically aligning a mandala of competencies to advance a transformative vision.

OBJECTIVE: This paper offers insight into the processes that have shaped the Eskind Biomedical Library's (EBL's) strategic direction and its alignment to the institution's transformative vision. SETTING: The academic biomedical library has a notable track record for developing and pioneering roles for information professionals focused on a sophisticated level of information provision that draws from and fuels practice evolutions. STRATEGY: The medical center's overall transformative vision informs the creation of a fully aligned library strategic plan designed to effectively contribute to the execution of key organizational goals. Annual goals reflect organizational priorities and contain quantifiable and measurable deliverables. Two strategic themes, facilitating genetic literacy and preserving community history, are described in detail to illustrate the concept of goal setting. CONCLUSION: The strategic planning model reflects EBL's adaptation to the ever-changing needs of its organization. The paper provides a characterization of a workable model that can be replicated by other institutions.

Using health literacy and learning style preferences to optimize the delivery of health information.

Limited patient understanding of hypertension contributes to poor health outcomes. In 2 sequential randomized studies, the authors determined the impact of administering information tailored to health literacy level alone or in combination with preferred learning style on patients' understanding of hypertension. Patients with high blood pressure were recruited in an academic emergency department. In Experiment 1 (N = 85), the control group received only the routine discharge instructions; the intervention group received discharge instructions combined with information consistent with their health literacy level as determined by the Short Test of Functional Health Literacy. In Experiment 2 (N = 87), the information provided to the intervention group was tailored to both health literacy and learning style, as indicated by the VARK™ Questionnaire. To measure learning, the authors compared scores on a hypertension assessment administered during the emergency department visit and 2 weeks after discharge. Participants who received materials tailored to both health literacy level and learning style preference showed greater gains in knowledge than did those receiving information customized for health literacy level only. This study demonstrates that personalizing health information to learning style preferences and literacy level improves patient understanding of hypertension.

A pilot study to evaluate learning style-tailored information prescriptions for hypertensive emergency department patients.

OBJECTIVE: This pilot study explored whether learning style-tailored education materials, "information prescriptions," are effective in increasing hypertension knowledge in emergency room patients. METHODS: In a randomized trial, hypertensive emergency medicine patients received either standard care discharge instructions or discharge instructions in combination with an information prescription individualized to each patient's learning-style preference. Two weeks post-visit, the study team assessed changes in hypertension knowledge via a survey. RESULTS: No significant difference was observed for changes in quiz scores on the hypertension knowledge assessment, though patients receiving the tailored information prescriptions reported higher levels of satisfaction with intervention materials. CONCLUSION: The study demonstrated the workflow feasibility of implementing a learning-style approach to patient education in the emergency department setting. Further research is needed to develop more robust measures of high blood pressure knowledge among the emergency department patient population. This work will contribute to establishing a framework for developing customized information prescriptions that can be broadly adapted for use in varied settings and with varied health care conditions.