A survey of computable biomedical knowledge repositories.

Introduction: While data repositories are well-established in clinical and research enterprises, knowledge repositories with shareable computable biomedical knowledge (CBK) are relatively new entities to the digital health ecosystem. Trustworthy knowledge repositories are necessary for learning health systems, but the policies, standards, and practices to promote trustworthy CBK artifacts and methods to share, and safely and effectively use them are not well studied. Methods: We conducted an online survey of 24 organizations in the United States known to be involved in the development or deployment of CBK. The aim of the survey was to assess the current policies and practices governing these repositories and to identify best practices. Descriptive statistics methods were applied to data from 13 responding organizations, to identify common practices and policies instantiating the TRUST principles of Transparency, Responsibility, User Focus, Sustainability, and Technology. Results: All 13 respondents indicated to different degrees adherence to policies that convey TRUST. Transparency is conveyed by having policies pertaining to provenance, credentialed contributors, and provision of metadata. Repositories provide knowledge in machine-readable formats, include implementation guidelines, and adhere to standards to convey Responsibility. Repositories report having Technology functions that enable end-users to verify, search, and filter for knowledge products. Less common TRUST practices are User Focused procedures that enable consumers to know about user licensing requirements or query the use of knowledge artifacts. Related to Sustainability, less than a majority post describe their sustainability plans. Few organizations publicly describe whether patients play any role in their decision-making. Conclusion: It is essential that knowledge repositories identify and apply a baseline set of criteria to lay a robust foundation for their trustworthiness leading to optimum uptake, and safe, reliable, and effective use to promote sharing of CBK. Identifying current practices suggests a set of desiderata for the CBK ecosystem in its continued evolution.

A method for structuring complex clinical knowledge and its representational formalisms to support composite knowledge interoperability in healthcare.

INTRODUCTION: The use and interoperability of clinical knowledge starts with the quality of the formalism utilized to express medical expertise. However, a crucial challenge is that existing formalisms are often suboptimal, lacking the fidelity to represent complex knowledge thoroughly and concisely. Often this leads to difficulties when seeking to unambiguously capture, share, and implement the knowledge for care improvement in clinical information systems used by providers and patients. OBJECTIVES: To provide a systematic method to address some of the complexities of knowledge composition and interoperability related to standards-based representational formalisms of medical knowledge. METHODS: Several cross-industry (Healthcare, Linguistics, System Engineering, Standards Development, and Knowledge Engineering) frameworks were synthesized into a proposed reference knowledge framework. The framework utilizes IEEE 42010, the MetaObject Facility, the Semantic Triangle, an Ontology Framework, and the Domain and Comprehensibility Appropriateness criteria. The steps taken were: 1) identify foundational cross-industry frameworks, 2) select architecture description method, 3) define life cycle viewpoints, 4) define representation and knowledge viewpoints, 5) define relationships between neighboring viewpoints, and 6) establish characteristic definitions of the relationships between components. System engineering principles applied included separation of concerns, cohesion, and loose coupling. RESULTS: A "Multilayer Metamodel for Representation and Knowledge" (M*R/K) reference framework was defined. It provides a standard vocabulary for organizing and articulating medical knowledge curation perspectives, concepts, and relationships across the artifacts created during the life cycle of language creation, authoring medical knowledge, and knowledge implementation in clinical information systems such as electronic health records (EHR). CONCLUSION: M*R/K provides a systematic means to address some of the complexities of knowledge composition and interoperability related to medical knowledge representations used in diverse standards. The framework may be used to guide the development, assessment, and coordinated use of knowledge representation formalisms. M*R/K could promote the alignment and aggregated use of distinct domain-specific languages in composite knowledge artifacts such as clinical practice guidelines (CPGs).

Summary of fourth annual MCBK public meeting: Mobilizing computable biomedical knowledge-metadata and trust.

The exponential growth of biomedical knowledge in computable formats challenges organizations to consider mobilizing artifacts in findable, accessible, interoperable, reusable, and trustable (FAIR+T) ways1. There is a growing need to apply biomedical knowledge artifacts to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations lack the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are insufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community formed in 2016 to address these needs. This report summarizes the main outputs of the Fourth Annual MCBK public meeting, which was held virtually July 20 to July 21, 2021 and convened over 100 participants spanning diverse domains to frame and address important dimensions for mobilizing CBK.

Summary of third annual MCBK public meeting: Mobilizing computable biomedical knowledge-Accelerating the second knowledge revolution.

The volume of biomedical knowledge is growing exponentially and much of this knowledge is represented in computer executable formats, such as models, algorithms, and programmatic code. There is a growing need to apply this knowledge to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations do not yet have the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are not sufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community was formed in 2016 to address these needs. This report summarizes the main outputs of the third annual MCBK public meeting, which was held virtually from June 30 to July 1, 2020 and brought together over 200 participants from various domains to frame and address important dimensions for mobilizing CBK.

Building and maintaining trust in clinical decision support: Recommendations from the Patient-Centered CDS Learning Network.

Knowledge artifacts in digital repositories for clinical decision support (CDS) can promote the use of CDS in clinical practice. However, stakeholders will benefit from knowing which they can trust before adopting artifacts from knowledge repositories. We discuss our investigation into trust for knowledge artifacts and repositories by the Patient-Centered CDS Learning Network's Trust Framework Working Group (TFWG). The TFWG identified 12 actors (eg, vendors, clinicians, and policy makers) within a CDS ecosystem who each may play a meaningful role in prioritizing, authoring, implementing, or evaluating CDS and developed 33 recommendations distributed across nine "trust attributes." The trust attributes and recommendations represent a range of considerations such as the "Competency" of knowledge artifact engineers and the "Organizational Capacity" of institutions that develop and implement CDS. The TFWG findings highlight an initial effort to make trust explicit and embedded within CDS knowledge artifacts and repositories and thus more broadly accepted and used.

Summary of second annual MCBK public meeting: Mobilizing Computable Biomedical Knowledge-A movement to accelerate translation of knowledge into action.

The volume of biomedical knowledge is growing exponentially and much of this knowledge is represented in computer executable formats, such as models, algorithms and programmatic code. There is a growing need to apply this knowledge to improve health in Learning Health Systems, health delivery organizations, and other settings. However, most organizations do not yet have the infrastructure required to consume and apply computable knowledge, and national policies and standards adoption are not sufficient to ensure that it is discoverable and used safely and fairly, nor is there widespread experience in the process of knowledge implementation as clinical decision support. The Mobilizing Computable Biomedical Knowledge (MCBK) community formed in 2016 to address these needs. This report summarizes the main outputs of the Second Annual MCBK public meeting, which was held at the National Institutes of Health on July 18-19, 2019 and brought together over 150 participants from various domains to frame and address important dimensions for mobilizing CBK.

Commercial Interests in Continuing Medical Education: Where Do Electronic Health Record Vendors Fit?

The Accreditation Council for Continuing Medical Education (ACCME) will not accredit an organization that it defines as a commercial interest, that is an entity that produces, markets, resells, or distributes health care goods or services consumed by, or used on, patients. Thus, commercial interests are not eligible to be accredited organizations offering continuing medical education (CME) credit to physicians. This decision is based on the concern that commercial interests may use CME events to market their products or services to physicians, who then might inappropriately prescribe or administer those products or services to patients. Studies have shown that CME events supported by pharmaceutical companies, for example, have influenced physicians' prescribing behaviors.Currently, however, the ACCME does not recognize electronic health record (EHR) vendors, which are part of a multi-billion-dollar business, as commercial interests, and it accredits them to provide or directly influence CME events. Like pharmaceutical company-sponsored CME events, EHR vendor activities, which inherently only focus on use of the sponsoring vendor's EHR system despite its potential intrinsic limitations, can lead to physician reciprocity. Such events also may inappropriately influence EHR system purchases, upgrades, and implementation decisions. These actions can negatively influence patient safety and care. Thus, the authors of this Perspective call on the ACCME to recognize EHR vendors as commercial interests and remove them from the list of accredited CME providers.

The Imperative for Patient-Centered Clinical Decision Support.

This commentary introduces the Patient-Centered Clinical Decision Support (PCCDS) Learning Network, which is collaborating with AcademyHealth to publish "Better Decisions Together" as part of eGEMs. Patient-centered clinical decision support (CDS) is an important vehicle to address broad issues in the U.S. health care system regarding quality and safety while also achieving better outcomes and better patient and provider satisfaction. Defined as CDS that supports individual patients and their care givers and/or care teams in health-related decisions and actions, PCCDS is an important step forward in advancing endeavors to move patient-centered care forward. The PCCDS Learning Network has developed a framework, referred to as the Analytic Framework for Action (AFA), to organize thinking and activities around PCCDS. A wide array of activities the PCCDS Learning Network is engaging in to inform and connect stakeholders is discussed.

Clinical decision support models and frameworks: Seeking to address research issues underlying implementation successes and failures.

Computer-based clinical decision support (CDS) has been pursued for more than five decades. Despite notable accomplishments and successes, wide adoption and broad use of CDS in clinical practice has not been achieved. Many issues have been identified as being partially responsible for the relatively slow adoption and lack of impact, including deficiencies in leadership, recognition of purpose, understanding of human interaction and workflow implications of CDS, cognitive models of the role of CDS, and proprietary implementations with limited interoperability and sharing. To address limitations, many approaches have been proposed and evaluated, drawing on theoretical frameworks, as well as management, technical and other disciplines and experiences. It seems clear, because of the multiple perspectives involved, that no single model or framework is adequate to encompass these challenges. This Viewpoint paper seeks to review the various foci of CDS and to identify aspects in which theoretical models and frameworks for CDS have been explored or could be explored and where they might be expected to be most useful.

How Stakeholder Assessment of E-Prescribing Can Help Determine Incentives to Facilitate Management of Care: A Delphi Study.

BACKGROUND: Little research has been conducted on the quality, benefits, costs, and financial considerations associated with health information technology (HIT), particularly informatics technologies such as e-prescribing, from the perspective of all of its stakeholders. OBJECTIVES: To (a) identify the stakeholders involved in e-prescribing and (b) identify and rank order the positives and negatives of e-prescribing from the perspective of stakeholders in order to create a framework for payers, integrated delivery systems, policymakers and legislators, and those who influence public policy to assist them in the development of incentives and payment mechanisms that result in the better management of care. METHODS: The Delphi method was used to enlist a panel of experts in e-prescribing, informatics, and/or HIT who have published in the field. This panel was presented with the results of initial research and an online survey of questions that sought to prioritize the quality, benefit, cost, and financial effects of e-prescribing from the perspective of each stakeholder. Eleven experts completed the first survey, which contained a list of stakeholders and positives and negatives associated with e-prescribing. Nine of the 11 experts completed the second survey, and 7 experts completed the final survey. From the results of these 3 surveys, a framework was presented to 5 framework experts, who were representatives from payers, integrated delivery systems, policymakers and legislators, and those who influence public policy. These framework experts were interviewed regarding the usefulness of the framework from their perspectives. RESULTS: The experts added stakeholders and many positives and negatives to the initial list and rank ordered the positives and negatives of e-prescribing from the perspective of each stakeholder. The aggregate results were summarized by stakeholder category. The positives and negatives were categorized as health, finance, effort, time, management, or data concerns. The framework experts evaluated the framework and found it useful. CONCLUSIONS: Positives and negatives associated with e-prescribing in the areas of quality, benefits, costs, and financial considerations can be rank ordered from the perspective of each stakeholder. The experts agreed on some points but disagreed on others. For example, they agreed that the main negative of e-prescribing from the perspective of pharmacists and pharmacies was its implementation costs but differed on the importance of providing faster information transfer. A framework was created that could be successfully used by payers, integrated delivery systems, policymakers and legislators, and those who influence public policy for the development of incentives and payment mechanisms. DISCLOSURES: This research was supported by the National Library of Medicine of the National Institutes of Health under Award Number T15LM007088. The authors declare no conflicts of interest in the research. Study concept and design were contributed by DeMuro, Ash, Middleton, and Fletcher. DeMuro took the lead in data collection, along with Ash, and data interpretation was performed by DeMuro, Ash, Madison, Middleton, and Fletcher. The manuscript was written primarily by DeMuro, along with Ash and Middleton, and revised by DeMuro, Madison, and Ash, along with Middleton and Fletcher.

A Quality, Benefit, Cost, and Financial Framework for Health Information Technology, E-Prescribing: A Delphi Study.

BACKGROUND: Little research has been conducted about the quality, benefits, costs, and financial considerations associated with health information technology (HIT), particularly informatics technologies, such as e-prescribing, from the perspective of all its stakeholders. OBJECTIVES: This research effort sought to identify the stakeholders involved in e-prescribing and to identify and rank-order the positives and the negatives from the perspective of the stakeholders to create a framework to assist in the development of incentives and payment mechanisms which result in better managed care. METHODS: The Delphi method was employed by enlisting a panel of experts. They were presented with the results of initial research in an online survey of questions which sought to prioritize the quality, benefit, cost, and financial effects of e-prescribing from the perspective of each stakeholder. From the results of this study, a framework was presented to framework experts. RESULTS: The experts added stakeholders and positives and negatives to the initial lists and rank-ordered the positives and negatives of e-prescribing from the perspective of each stakeholder. The aggregate results were summarized by category of stakeholder. The framework experts evaluated the framework. CONCLUSIONS: Positives and negatives can be rank-ordered from the perspective of each stakeholder. A useful framework was created.

Crossing the health IT chasm: considerations and policy recommendations to overcome current challenges and enable value-based care.

While great progress has been made in digitizing the US health care system, today's health information technology (IT) infrastructure remains largely a collection of systems that are not designed to support a transition to value-based care. In addition, the pursuit of value-based care, in which we deliver better care with better outcomes at lower cost, places new demands on the health care system that our IT infrastructure needs to be able to support. Provider organizations pursuing new models of health care delivery and payment are finding that their electronic systems lack the capabilities needed to succeed. The result is a chasm between the current health IT ecosystem and the health IT ecosystem that is desperately needed.In this paper, we identify a set of focal goals and associated near-term achievable actions that are critical to pursue in order to enable the health IT ecosystem to meet the acute needs of modern health care delivery. These ideas emerged from discussions that occurred during the 2015 American Medical Informatics Association Policy Invitational Meeting. To illustrate the chasm and motivate our recommendations, we created a vignette from the multistakeholder perspectives of a patient, his provider, and researchers/innovators. It describes an idealized scenario in which each stakeholder's needs are supported by an integrated health IT environment. We identify the gaps preventing such a reality today and present associated policy recommendations that serve as a blueprint for critical actions that would enable us to cross the current health IT chasm by leveraging systems and information to routinely deliver high-value care.

The Chief Clinical Informatics Officer (CCIO): AMIA Task Force Report on CCIO Knowledge, Education, and Skillset Requirements.

INTRODUCTION: The emerging operational role of the "Chief Clinical Informatics Officer" (CCIO) remains heterogeneous with individuals deriving from a variety of clinical settings and backgrounds. The CCIO is defined in title, responsibility, and scope of practice by local organizations. The term encompasses the more commonly used Chief Medical Informatics Officer (CMIO) and Chief Nursing Informatics Officer (CNIO) as well as the rarely used Chief Pharmacy Informatics Officer (CPIO) and Chief Dental Informatics Officer (CDIO). BACKGROUND: The American Medical Informatics Association (AMIA) identified a need to better delineate the knowledge, education, skillsets, and operational scope of the CCIO in an attempt to address the challenges surrounding the professional development and the hiring processes of CCIOs. DISCUSSION: An AMIA task force developed knowledge, education, and operational skillset recommendations for CCIOs focusing on the common core aspect and describing individual differences based on Clinical Informatics focus. The task force concluded that while the role of the CCIO currently is diverse, a growing body of Clinical Informatics and increasing certification efforts are resulting in increased homogeneity. The task force advised that 1.) To achieve a predictable and desirable skillset, the CCIO must complete clearly defined and specified Clinical Informatics education and training. 2.) Future education and training must reflect the changing body of knowledge and must be guided by changing day-to-day informatics challenges. CONCLUSION: A better defined and specified education and skillset for all CCIO positions will motivate the CCIO workforce and empower them to perform the job of a 21st century CCIO. Formally educated and trained CCIOs will provide a competitive advantage to their respective enterprise by fully utilizing the power of Informatics science.

Lessons learned from implementing service-oriented clinical decision support at four sites: A qualitative study.

OBJECTIVE: To identify challenges, lessons learned and best practices for service-oriented clinical decision support, based on the results of the Clinical Decision Support Consortium, a multi-site study which developed, implemented and evaluated clinical decision support services in a diverse range of electronic health records. METHODS: Ethnographic investigation using the rapid assessment process, a procedure for agile qualitative data collection and analysis, including clinical observation, system demonstrations and analysis and 91 interviews. RESULTS: We identified challenges and lessons learned in eight dimensions: (1) hardware and software computing infrastructure, (2) clinical content, (3) human-computer interface, (4) people, (5) workflow and communication, (6) internal organizational policies, procedures, environment and culture, (7) external rules, regulations, and pressures and (8) system measurement and monitoring. Key challenges included performance issues (particularly related to data retrieval), differences in terminologies used across sites, workflow variability and the need for a legal framework. DISCUSSION: Based on the challenges and lessons learned, we identified eight best practices for developers and implementers of service-oriented clinical decision support: (1) optimize performance, or make asynchronous calls, (2) be liberal in what you accept (particularly for terminology), (3) foster clinical transparency, (4) develop a legal framework, (5) support a flexible front-end, (6) dedicate human resources, (7) support peer-to-peer communication, (8) improve standards. CONCLUSION: The Clinical Decision Support Consortium successfully developed a clinical decision support service and implemented it in four different electronic health records and four diverse clinical sites; however, the process was arduous. The lessons identified by the Consortium may be useful for other developers and implementers of clinical decision support services.

Multiple perspectives on clinical decision support: a qualitative study of fifteen clinical and vendor organizations.

BACKGROUND: Computerized clinical decision support (CDS) can help hospitals to improve healthcare. However, CDS can be problematic. The purpose of this study was to discover how the views of clinical stakeholders, CDS content vendors, and EHR vendors are alike or different with respect to challenges in the development, management, and use of CDS. METHODS: We conducted ethnographic fieldwork using a Rapid Assessment Process within ten clinical and five health information technology (HIT) vendor organizations. Using an inductive analytical approach, we generated themes from the clinical, content vendor, and electronic health record vendor perspectives and compared them. RESULTS: The groups share views on the importance of appropriate manpower, careful knowledge management, CDS that fits user workflow, the need for communication among the groups, and for mutual strategizing about the future of CDS. However, views of usability, training, metrics, interoperability, product use, and legal issues differed. Recommendations for improvement include increased collaboration to address legal, manpower, and CDS sharing issues. CONCLUSIONS: The three groups share thinking about many aspects of CDS, but views differ in a number of important respects as well. Until these three groups can reach a mutual understanding of the views of the other stakeholders, and work together, CDS will not reach its potential.

Measuring agreement between decision support reminders: the cloud vs. the local expert.

BACKGROUND: A cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data. METHODS: Electronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen's Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients. RESULTS: The cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95% CI 0.24 - 0.42) to 0.99 (95% CI 0.97 - 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders. CONCLUSIONS: Preventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.

A qualitative study of the activities performed by people involved in clinical decision support: recommended practices for success.

OBJECTIVE: To describe the activities performed by people involved in clinical decision support (CDS) at leading sites. MATERIALS AND METHODS: We conducted ethnographic observations at seven diverse sites with a history of excellence in CDS using the Rapid Assessment Process and analyzed the data using a series of card sorts, informed by Linstone's Multiple Perspectives Model. RESULTS: We identified 18 activities and grouped them into four areas. Area 1: Fostering relationships across the organization, with activities (a) training and support, (b) visibility/presence on the floor, (c) liaising between people, (d) administration and leadership, (e) project management, (f) cheerleading/buy-in/sponsorship, (g) preparing for CDS implementation. Area 2: Assembling the system with activities (a) providing technical support, (b) CDS content development, (c) purchasing products from vendors (d) knowledge management, (e) system integration. Area 3: Using CDS to achieve the organization's goals with activities (a) reporting, (b) requirements-gathering/specifications, (c) monitoring CDS, (d) linking CDS to goals, (e) managing data. Area 4: Participation in external policy and standards activities (this area consists of only a single activity). We also identified a set of recommendations associated with these 18 activities. DISCUSSION: All 18 activities we identified were performed at all sites, although the way they were organized into roles differed substantially. We consider these activities critical to the success of a CDS program. CONCLUSIONS: A series of activities are performed by sites strong in CDS, and sites adopting CDS should ensure they incorporate these activities into their efforts.