Collecting patient-reported outcome measures in the electronic health record: Lessons from the NIH pragmatic trials Collaboratory.

The NIH Pragmatic Trials Collaboratory supports the design and conduct of 27 embedded pragmatic clinical trials, and many of the studies collect patient reported outcome measures as primary or secondary outcomes. Study teams have encountered challenges in the collection of these measures, including challenges related to competing health care system priorities, clinician's buy-in for adoption of patient-reported outcome measures, low adoption and reach of technology in low resource settings, and lack of consensus and standardization of patient-reported outcome measure selection and administration in the electronic health record. In this article, we share case examples and lessons learned, and suggest that, when using patient-reported outcome measures for embedded pragmatic clinical trials, investigators must make important decisions about whether to use data collected from the participating health system's electronic health record, integrate externally collected patient-reported outcome data into the electronic health record, or collect these data in separate systems for their studies.

Guiding principles for technical infrastructure to support computable biomedical knowledge.

Over the past 4 years, the authors have participated as members of the Mobilizing Computable Biomedical Knowledge Technical Infrastructure working group and focused on conceptualizing the infrastructure required to use computable biomedical knowledge. Here, we summarize our thoughts and lay the foundation for future work in the development of CBK infrastructure, including: explaining the difference between computable knowledge and data, and contextualizing the conversation with the Learning Health Systems and the FAIR principles. Specifically, we provide three guiding principles to advance the development of CBK infrastructure: (a) Promote interoperable systems for data and knowledge to be findable, accessible, interoperable, and reusable. (b) Enable stable, trustworthy knowledge representations that are human and machine readable. (c) Computable knowledge resources should, when possible, be open. Standards supporting computable knowledge infrastructures must be open.

Potential bias and lack of generalizability in electronic health record data: reflections on health equity from the National Institutes of Health Pragmatic Trials Collaboratory.

Embedded pragmatic clinical trials (ePCTs) play a vital role in addressing current population health problems, and their use of electronic health record (EHR) systems promises efficiencies that will increase the speed and volume of relevant and generalizable research. However, as the number of ePCTs using EHR-derived data grows, so does the risk that research will become more vulnerable to biases due to differences in data capture and access to care for different subsets of the population, thereby propagating inequities in health and the healthcare system. We identify 3 challenges-incomplete and variable capture of data on social determinants of health, lack of representation of vulnerable populations that do not access or receive treatment, and data loss due to variable use of technology-that exacerbate bias when working with EHR data and offer recommendations and examples of ways to actively mitigate bias.

Equity and bias in electronic health records data.

Embedded pragmatic clinical trials (ePCTs) are conducted during routine clinical care and have the potential to increase knowledge about the effectiveness of interventions under real world conditions. However, many pragmatic trials rely on data from the electronic health record (EHR) data, which are subject to bias from incomplete data, poor data quality, lack of representation from people who are medically underserved, and implicit bias in EHR design. This commentary examines how the use of EHR data might exacerbate bias and potentially increase health inequities. We offer recommendations for how to increase generalizability of ePCT results and begin to mitigate bias to promote health equity.

Evolving a national clinical trials learning health system.

Clinical trials generate key evidence to inform decision making, and also benefit participants directly. However, clinical trials frequently fail, often struggle to enroll participants, and are expensive. Part of the problem with trial conduct may be the disconnected nature of clinical trials, preventing rapid data sharing, generation of insights and targeted improvement interventions, and identification of knowledge gaps. In other areas of healthcare, a learning health system (LHS) has been proposed as a model to facilitate continuous learning and improvement. We propose that an LHS approach could greatly benefit clinical trials, allowing for continuous improvements to trial conduct and efficiency. A robust trial data sharing system, continuous analysis of trial enrollment and other success metrics, and development of targeted trial improvement interventions are potentially key components of a Trials LHS reflecting the learning cycle and allowing for continuous trial improvement. Through the development and use of a Trials LHS, clinical trials could be treated as a system, producing benefits to patients, advancing care, and decreasing costs for stakeholders.

Summary of fourth annual MCBK public meeting: Mobilizing computable biomedical knowledge-metadata and trust.

The exponential growth of biomedical knowledge in computable formats challenges organizations to consider mobilizing artifacts in findable, accessible, interoperable, reusable, and trustable (FAIR+T) ways1. There is a growing need to apply biomedical knowledge artifacts to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations lack the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are insufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community formed in 2016 to address these needs. This report summarizes the main outputs of the Fourth Annual MCBK public meeting, which was held virtually July 20 to July 21, 2021 and convened over 100 participants spanning diverse domains to frame and address important dimensions for mobilizing CBK.

Informatics education for translational research teams: An unrealized opportunity to strengthen the national research infrastructure.

Objective: To identify the informatics educational needs of clinical and translational research professionals whose primary focus is not informatics. Introduction: Informatics and data science skills are essential for the full spectrum of translational research, and an increased understanding of informatics issues on the part of translational researchers can alleviate the demand for informaticians and enable more productive collaborations when informaticians are involved. Identifying the level of interest in different topics among various types of of translational researchers will help set priorities for development and dissemination of informatics education. Methods: We surveyed clinical and translational science researchers in Clinical and Translational Science Award (CTSA) programs about their educational needs and preferences. Results: Researchers from 23 out of the 62 CTSA hubs responded to the survey. 67% of respondents across roles and topics expressed interest in learning about informatics topics. There was high interest in all 30 topics included in the survey, with some variation in interest depending on the role of the respondents. Discussion: Our data support the need to advance training in clinical and biomedical informatics. As the complexity and use of information technology and data science in research studies grows, informaticians will continue to be a limited resource for research collaboration, education, and training. An increased understanding of informatics issues across translational research teams can alleviate this burden and allow for more productive collaborations. To inform a roadmap for informatics education for research professionals, we suggest strategies to use the results of this needs assessment to develop future informatics education.

Enhancing the use of EHR systems for pragmatic embedded research: lessons from the NIH Health Care Systems Research Collaboratory.

OBJECTIVE: We identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research. MATERIALS AND METHODS: Since 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements. RESULTS: We received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data. DISCUSSION: Based on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries. CONCLUSION: We are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.

Desiderata for the development of next-generation electronic health record phenotype libraries.

BACKGROUND: High-quality phenotype definitions are desirable to enable the extraction of patient cohorts from large electronic health record repositories and are characterized by properties such as portability, reproducibility, and validity. Phenotype libraries, where definitions are stored, have the potential to contribute significantly to the quality of the definitions they host. In this work, we present a set of desiderata for the design of a next-generation phenotype library that is able to ensure the quality of hosted definitions by combining the functionality currently offered by disparate tooling. METHODS: A group of researchers examined work to date on phenotype models, implementation, and validation, as well as contemporary phenotype libraries developed as a part of their own phenomics communities. Existing phenotype frameworks were also examined. This work was translated and refined by all the authors into a set of best practices. RESULTS: We present 14 library desiderata that promote high-quality phenotype definitions, in the areas of modelling, logging, validation, and sharing and warehousing. CONCLUSIONS: There are a number of choices to be made when constructing phenotype libraries. Our considerations distil the best practices in the field and include pointers towards their further development to support portable, reproducible, and clinically valid phenotype design. The provision of high-quality phenotype definitions enables electronic health record data to be more effectively used in medical domains.

A Conceptual Framework of Data Readiness: The Contextual Intersection of Quality, Availability, Interoperability, and Provenance.

BACKGROUND: Data readiness is a concept often used when referring to health information technology applications in the informatics disciplines, but it is not clearly defined in the literature. To avoid misinterpretations in research and implementation, a formal definition should be developed. OBJECTIVES: The objective of this research is to provide a conceptual definition and framework for the term data readiness that can be used to guide research and development related to data-based applications in health care. METHODS: PubMed, the National Institutes of Health RePORTER, Scopus, the Cochrane Library, and Duke University Library databases for business and information sciences were queried for formal mentions of the term "data readiness." Manuscripts found in the search were reviewed, and relevant information was extracted, evaluated, and assimilated into a framework for data readiness. RESULTS: Of the 264 manuscripts found in the database searches, 20 were included in the final synthesis to define data readiness. In these 20 manuscripts, the term data readiness was revealed to encompass the constructs of data quality, data availability, interoperability, and data provenance. DISCUSSION: Based upon our review of the literature, we define data readiness as the application-specific intersection of data quality, data availability, interoperability, and data provenance. While these concepts are not new, the combination of these factors in a novel data readiness model may help guide future informatics research and implementation science. CONCLUSION: This analysis provides a definition to guide research and development related to data-based applications in health care. Future work should be done to validate this definition, and to apply the components of data readiness to real-world applications so that specific metrics may be developed and disseminated.

A thematic analysis to examine the feasibility of EHR-based clinical decision support for implementing Choosing Wisely ® guidelines.

OBJECTIVE: To identify important barriers and facilitators relating to the feasibility of implementing clinical practice guidelines (CPGs) as clinical decision support (CDS). MATERIALS AND METHODS: We conducted a qualitative, thematic analysis of interviews from seven interviews with dyads (one clinical expert and one systems analyst) who discussed the feasibility of implementing 10 Choosing Wisely® guidelines at their institutions. We conducted a content analysis to extract salient themes describing facilitators, challenges, and other feasibility considerations regarding implementing CPGs as CDS. RESULTS: We identified five themes: concern about data quality impacts implementation planning; the availability of data in a computable format is a primary factor for implementation feasibility; customized strategies are needed to mitigate uncertainty and ambiguity when translating CPGs to an electronic health record-based tool; misalignment of expected CDS with pre-existing clinical workflows impact implementation; and individual level factors of end-users must be considered when selecting and implementing CDS tools. DISCUSSION: The themes reveal several considerations for CPG as CDS implementations regarding data quality, knowledge representation, and sociotechnical issues. Guideline authors should be aware that using CDS to implement CPGs is becoming increasingly popular and should consider providing clear guidelines to aid implementation. The complex nature of CPG as CDS implementation necessitates a unified effort to overcome these challenges. CONCLUSION: Our analysis highlights the importance of cooperation and co-development of standards, strategies, and infrastructure to address the difficulties of implementing CPGs as CDS. The complex interactions between the concepts revealed in the interviews necessitates the need that such work should not be conducted in silos. We also implore that implementers disseminate their experiences.

Association between evidence-based training and clinician proficiency in electronic health record use.

OBJECTIVES: The purpose of the study was to determine if association exists between evidence-based provider training and clinician proficiency in electronic health record (EHR) use and if so, which EHR use metrics and vendor-defined indices exhibited association. MATERIALS AND METHODS: We studied ambulatory clinicians' EHR use data published in the Epic Systems Signal report to assess proficiency between training participants (n = 133) and nonparticipants (n = 14). Data were collected in May 2019 and November 2019 on nonsurgeon clinicians from 6 primary care, 7 urgent care, and 27 specialty care clinics. EHR use training occurred from August 5 to August 15, 2019, prior to EHR upgrade and organizational instance alignment. Analytics performed were descriptive statistics, paired t-tests, multivariate correlations, and hierarchal multiple regression. RESULTS: For number of appointments per 30-day reporting period, trained clinicians sustained an average increase of 16 appointments (P < .05), whereas nontrained clinicians incurred a decrease of 8 appointments. Only the trained clinician group achieved postevent improvement in the vendor-defined Proficiency score with an effect size characterized as moderate to large (dCohen = 0.625). DISCUSSION: Controversies exist on the return of investment from formal EHR training for clinician users. Previously published literature has mostly focused on qualitative data indicators of EHR training success. The findings of our EHR use training study identified EHR use metrics and vendor-defined indices with the capacity for translation into productivity and generated revenue measurements. CONCLUSIONS: One EHR use metric and 1 vendor-defined index indicated improved proficiency among trained clinicians.

Summary of third annual MCBK public meeting: Mobilizing computable biomedical knowledge-Accelerating the second knowledge revolution.

The volume of biomedical knowledge is growing exponentially and much of this knowledge is represented in computer executable formats, such as models, algorithms, and programmatic code. There is a growing need to apply this knowledge to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations do not yet have the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are not sufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community was formed in 2016 to address these needs. This report summarizes the main outputs of the third annual MCBK public meeting, which was held virtually from June 30 to July 1, 2020 and brought together over 200 participants from various domains to frame and address important dimensions for mobilizing CBK.

A Closer Look at the "Right" Format for Clinical Decision Support: Methods for Evaluating a Storyboard BestPractice Advisory.

(1) Background: The five rights of clinical decision support (CDS) are a well-known framework for planning the nuances of CDS, but recent advancements have given us more options to modify the format of the alert. One-size-fits-all assessments fail to capture the nuance of different BestPractice Advisory (BPA) formats. To demonstrate a tailored evaluation methodology, we assessed a BPA after implementation of Storyboard for changes in alert fatigue, behavior influence, and task completion; (2) Methods: Data from 19 weeks before and after implementation were used to evaluate differences in each domain. Individual clinics were evaluated for task completion and compared for changes pre- and post-redesign; (3) Results: The change in format was correlated with an increase in alert fatigue, a decrease in erroneous free text answers, and worsened task completion at a system level. At a local level, however, 14% of clinics had improved task completion; (4) Conclusions: While the change in BPA format was correlated with decreased performance, the changes may have been driven primarily by the COVID-19 pandemic. The framework and metrics proposed can be used in future studies to assess the impact of new CDS formats. Although the changes in this study seemed undesirable in aggregate, some positive changes were observed at the level of individual clinics. Personalized implementations of CDS tools based on local need should be considered.

Summary of second annual MCBK public meeting: Mobilizing Computable Biomedical Knowledge-A movement to accelerate translation of knowledge into action.

The volume of biomedical knowledge is growing exponentially and much of this knowledge is represented in computer executable formats, such as models, algorithms and programmatic code. There is a growing need to apply this knowledge to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations do not yet have the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are not sufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community formed in 2016 to address these needs. This report summarizes the main outputs of the Second Annual MCBK public meeting, which was held at the National Institutes of Health on July 18-19, 2019 and brought together over 150 participants from various domains to frame and address important dimensions for mobilizing CBK.

Measuring implementation feasibility of clinical decision support alerts for clinical practice recommendations.

OBJECTIVE: The study sought to describe key features of clinical concepts and data required to implement clinical practice recommendations as clinical decision support (CDS) tools in electronic health record systems and to identify recommendation features that predict feasibility of implementation. MATERIALS AND METHODS: Using semistructured interviews, CDS implementers and clinician subject matter experts from 7 academic medical centers rated the feasibility of implementing 10 American College of Emergency Physicians Choosing Wisely Recommendations as electronic health record-embedded CDS and estimated the need for additional data collection. Ratings were combined with objective features of the guidelines to develop a predictive model for technical implementation feasibility. RESULTS: A linear mixed model showed that the need for new data collection was predictive of lower implementation feasibility. The number of clinical concepts in each recommendation, need for historical data, and ambiguity of clinical concepts were not predictive of implementation feasibility. CONCLUSIONS: The availability of data and need for additional data collection are essential to assess the feasibility of CDS implementation. Authors of practice recommendations and guidelines can enable organizations to more rapidly assess data availability and feasibility of implementation by including operational definitions for required data.

Optimization of Nursing-Specific Flu Alerts.

Clinical decision support interventions, such as alerts and reminders, can improve clinician compliance with practice guidelines and patient outcomes. Alerts that trigger at inappropriate times are often dismissed by clinicians, reducing desired actions rather than increasing them. A set of nursing-specific alerts related to influenza screening and vaccination were optimized so that they would "trigger" less often but function adequately to maintain institutional flu vaccination compliance. We analyzed the current triggering criteria for six flu vaccine-related alerts and asked nurse end users for suggestions to increase specificity. Using the "five rights" (of clinical decision support) as a framework, alerts were redesigned to address user needs. New alerts were tested and implemented and their activity compared in two different flu seasons, preoptimization and postoptimization. The redesigned alerts resulted in fewer alerts per encounter (P < .0001), less dismissals of alerts (P < .0001), and a 2.8% point improvement in compliance rates for flu vaccine screening, documentation, and administration. A focus group confirmed that the redesign improved workflow, but some nurses thought they still triggered too often. The five rights model can support improvements in alert design and outcomes.

Learning health systems, embedded research, and data standards-recommendations for healthcare system leaders.

Learning health systems that conduct embedded research require infrastructure for the seamless adoption of clinical interventions; this infrastructure should integrate with electronic health record (EHR) systems and enable the use of existing data. As purchasers of EHR systems, and as critical partners, sponsors, and consumers of embedded research, healthcare organizations should advocate for EHR system functionality and data standards that will increase the capacity for embedded research in clinical settings. As stakeholders and proponents for EHR data standards, healthcare leaders should support standards development and promote local adoption to support quality healthcare, continuous improvement, innovative data-driven interventions, and the generation of new knowledge. "Standards-enabled" health systems will be positioned to address emergent and critical research questions, including those related to coronavirus disease 2019 (COVID-19) and future public health threats. The role of a data standards officer or champion could enable health systems to realize this goal.

Emergency Department Clinician Perspectives on the Data Availability to Implement Clinical Decision Support Tools for Five Clinical Practice Guidelines.

Clinical practice guidelines (CPGs) often serve as the knowledge base for clinical decision support (CDS). While CPGs are rigorously created by medical professional societies, the concepts in each guideline may not be sufficient for translation into CDS applications. In addition, clinicians' perceptions of these concepts may differ greatly, affecting the implementation and impact of CDS within an organization. Five guidelines developed by the American College of Emergency Physicians were systematically explored, generating fifty-one unique clinical concepts. These concepts were presented to two nurses and two physicians, whom were asked to assess and comment on the capture of each clinical concept in the electronic health record (EHR) and the subsequent availability of the data for CDS. Nurses and physicians showed differing perceptions of data availability. These differing perceptions may influence an organizational approach to developing and implementing CDS, potentially informing our understanding of why CDS may not achieve the intended impact.