Perspectives on the role of industry in informatics research and authorship.

OBJECTIVES: Advances in informatics research come from academic, nonprofit, and for-profit industry organizations, and from academic-industry partnerships. While scientific studies of commercial products may offer critical lessons for the field, manuscripts authored by industry scientists are sometimes categorically rejected. We review historical context, community perceptions, and guidelines on informatics authorship. PROCESS: We convened an expert panel at the American Medical Informatics Association 2022 Annual Symposium to explore the role of industry in informatics research and authorship with community input. The panel summarized session themes and prepared recommendations. CONCLUSIONS: Authorship for informatics research, regardless of affiliation, should be determined by International Committee of Medical Journal Editors uniform requirements for authorship. All authors meeting criteria should be included, and categorical rejection based on author affiliation is unethical. Informatics research should be evaluated based on its scientific rigor; all sources of bias and conflicts of interest should be addressed through disclosure and, when possible, methodological mitigation.

Contemporary clinical decision support standards using Health Level Seven International Fast Healthcare Interoperability Resources.

OBJECTIVE: To facilitate the development of standards-based clinical decision support (CDS) systems, we review the current set of CDS standards that are based on Health Level Seven International Fast Healthcare Interoperability Resources (FHIR). Widespread adoption of these standards may help reduce healthcare variability, improve healthcare quality, and improve patient safety. TARGET AUDIENCE: This tutorial is designed for the broad informatics community, some of whom may be unfamiliar with the current, FHIR-based CDS standards. SCOPE: This tutorial covers the following standards: Arden Syntax (using FHIR as the data model), Clinical Quality Language, FHIR Clinical Reasoning, SMART on FHIR, and CDS Hooks. Detailed descriptions and selected examples are provided.

Feasibility of Prioritizing Drug-Drug-Event Associations Found in Electronic Health Records.

BACKGROUND AND OBJECTIVE: Several studies have demonstrated the ability to detect adverse events potentially related to multiple drug exposure via data mining. However, the number of putative associations produced by such computational approaches is typically large, making experimental validation difficult. We theorized that those potential associations for which there is evidence from multiple complementary sources are more likely to be true, and explored this idea using a published database of drug-drug-adverse event associations derived from electronic health records (EHRs). METHODS: We prioritized drug-drug-event associations derived from EHRs using four sources of information: (1) public databases, (2) sources of spontaneous reports, (3) literature, and (4) non-EHR drug-drug interaction (DDI) prediction methods. After pre-filtering the associations by removing those found in public databases, we devised a ranking for associations based on the support from the remaining sources, and evaluated the results of this rank-based prioritization. RESULTS: We collected information for 5983 putative EHR-derived drug-drug-event associations involving 345 drugs and ten adverse events from four data sources and four prediction methods. Only seven drug-drug-event associations (<0.5 %) had support from the majority of evidence sources, and about one third (1777) had support from at least one of the evidence sources. CONCLUSIONS: Our proof-of-concept method for scoring putative drug-drug-event associations from EHRs offers a systematic and reproducible way of prioritizing associations for further study. Our findings also quantify the agreement (or lack thereof) among complementary sources of evidence for drug-drug-event associations and highlight the challenges of developing a robust approach for prioritizing signals of these associations.

Analyzing Information Seeking and Drug-Safety Alert Response by Health Care Professionals as New Methods for Surveillance.

BACKGROUND: Patterns in general consumer online search logs have been used to monitor health conditions and to predict health-related activities, but the multiple contexts within which consumers perform online searches make significant associations difficult to interpret. Physician information-seeking behavior has typically been analyzed through survey-based approaches and literature reviews. Activity logs from health care professionals using online medical information resources are thus a valuable yet relatively untapped resource for large-scale medical surveillance. OBJECTIVE: To analyze health care professionals' information-seeking behavior and assess the feasibility of measuring drug-safety alert response from the usage logs of an online medical information resource. METHODS: Using two years (2011-2012) of usage logs from UpToDate, we measured the volume of searches related to medical conditions with significant burden in the United States, as well as the seasonal distribution of those searches. We quantified the relationship between searches and resulting page views. Using a large collection of online mainstream media articles and Web log posts we also characterized the uptake of a Food and Drug Administration (FDA) alert via changes in UpToDate search activity compared with general online media activity related to the subject of the alert. RESULTS: Diseases and symptoms dominate UpToDate searches. Some searches result in page views of only short duration, while others consistently result in longer-than-average page views. The response to an FDA alert for Celexa, characterized by a change in UpToDate search activity, differed considerably from general online media activity. Changes in search activity appeared later and persisted longer in UpToDate logs. The volume of searches and page view durations related to Celexa before the alert also differed from those after the alert. CONCLUSIONS: Understanding the information-seeking behavior associated with online evidence sources can offer insight into the information needs of health professionals and enable large-scale medical surveillance. Our Web log mining approach has the potential to monitor responses to FDA alerts at a national level. Our findings can also inform the design and content of evidence-based medical information resources such as UpToDate.

Recommendations to improve the usability of drug-drug interaction clinical decision support alerts.

OBJECTIVE: To establish preferred strategies for presenting drug-drug interaction (DDI) clinical decision support alerts. MATERIALS AND METHODS: A DDI Clinical Decision Support Conference Series included a workgroup consisting of 24 clinical, usability, and informatics experts representing academia, health information technology (IT) vendors, healthcare organizations, and the Office of the National Coordinator for Health IT. Workgroup members met via web-based meetings 12 times from January 2013 to February 2014, and two in-person meetings to reach consensus on recommendations to improve decision support for DDIs. We addressed three key questions: (1) what, how, where, and when do we display DDI decision support? (2) should presentation of DDI decision support vary by clinicians? and (3) how should effectiveness of DDI decision support be measured? RESULTS: Our recommendations include the consistent use of terminology, visual cues, minimal text, formatting, content, and reporting standards to facilitate usability. All clinicians involved in the medication use process should be able to view DDI alerts and actions by other clinicians. Override rates are common but may not be a good measure of effectiveness. DISCUSSION: Seven core elements should be included with DDI decision support. DDI information should be presented to all clinicians. Finally, in their current form, override rates have limited capability to evaluate alert effectiveness. CONCLUSION: DDI clinical decision support alerts need major improvements. We provide recommendations for healthcare organizations and IT vendors to improve the clinician interface of DDI alerts, with the aim of reducing alert fatigue and improving patient safety.

Terminology challenges implementing the HL7 context-aware knowledge retrieval ('Infobutton') standard.

Point-of-care information needs are common and frequently unmet. One solution to this problem is the use of Infobuttons, which are context-sensitive links from electronic health records (EHR) to knowledge resources, sometimes involving an intermediate broker known as an Infobutton Manager. Health Level Seven (HL7) has developed the Context-Aware Knowledge Retrieval (Infobutton) standard to standardize the integration between EHR systems and knowledge resources. While the standard specifies a set of context attributes and standard terminologies, it leaves to knowledge resources the flexibility to decide how to use these attributes and terminologies to retrieve the most relevant content. This paper describes some of the challenges faced by knowledge resources in trying to locate the most relevant content based on the attribute values for a given Infobutton request. Various approaches to content retrieval are discussed, including the role of indexing with standardized codes, the role of text-based search engines together with their ranking algorithms, and the role of hybrid approaches. Knowledge resource developers must carefully consider business rules, heuristics, and precision/recall tradeoffs when implementing the HL7 Infobutton standard.

Inter-rater agreement among physicians on the clinical significance of drug-drug interactions.

BACKGROUND: Alert fatigue could potentially be improved if physicians agreed on which alerts were clinically significant. We conducted a study to determine the extent to which physicians agree on which drug-drug interactions are clinically significant. METHODS: Two groups of eight generalist physicians reviewed 100 randomly selected drug-drug interactions from the Medi-Span® Drug Therapy Monitoring System™ database and indicated whether they thought each interaction was clinically significant based on the full-text clinical discussion contained within each interaction monograph and their clinical experience. RESULTS: The Fleiss Kappa measure of inter-rater agreement was 0.19 (0.12, 0.26) for one group, 0.22 (0.14, 0.29) for the second group and 0.21 (0.15, 0.27) for the combined group. CONCLUSION: We found poor agreement among generalist physicians on which drug-drug interactions are clinically significant. Use of a feature to allow physicians to tailor alerts to their needs may be an important component in reducing alert fatigue.

Implementations of the HL7 Context-Aware Knowledge Retrieval ("Infobutton") Standard: challenges, strengths, limitations, and uptake.

UNLABELLED: To support clinical decision-making, computerized information retrieval tools known as "infobuttons" deliver contextually-relevant knowledge resources into clinical information systems. The Health Level Seven International (HL7) Context-Aware Knowledge Retrieval (Infobutton) Standard specifies a standard mechanism to enable infobuttons on a large scale. OBJECTIVE: To examine the experience of organizations in the course of implementing the HL7 Infobutton Standard. METHOD: Cross-sectional online survey and in-depth phone interviews. RESULTS: A total of 17 organizations participated in the study. Analysis of the in-depth interviews revealed 20 recurrent themes. Implementers underscored the benefits, simplicity, and flexibility of the HL7 Infobutton Standard. Yet, participants voiced the need for easier access to standard specifications and improved guidance to beginners. Implementers predicted that the Infobutton Standard will be widely or at least fairly well adopted in the next 5 years, but uptake will depend largely on adoption among electronic health record (EHR) vendors. To accelerate EHR adoption of the Infobutton Standard, implementers recommended HL7-compliant infobutton capabilities to be included in the United States Meaningful Use Certification Criteria for EHR systems. LIMITATIONS: Opinions and predictions should be interpreted with caution, since all the participant organizations have successfully implemented the standard and over half of the organizations were actively engaged in the development of the standard. CONCLUSION: Overall, implementers reported a very positive experience with the HL7 Infobutton Standard. Despite indications of increasing uptake, measures should be taken to stimulate adoption of the Infobutton Standard among EHR vendors. Widespread adoption of the Infobutton Standard has the potential to bring contextually relevant clinical decision support content into the healthcare provider workflow.

Clinical information system services and capabilities desired for scalable, standards-based, service-oriented decision support: consensus assessment of the Health Level 7 clinical decision support Work Group.

A standards-based, service-oriented architecture for clinical decision support (CDS) has the potential to significantly enhance CDS scalability and robustness. To enable such a CDS architecture, the Health Level 7 CDS Work Group reviewed the literature, hosted multi-stakeholder discussions, and consulted domain experts to identify and prioritize the services and capabilities required from clinical information systems (CISs) to enable service-oriented CDS. In addition, relevant available standards were identified. Through this process, ten CIS services and eight CIS capabilities were identified as being important for enabling scalable, service-oriented CDS. In particular, through a survey of 46 domain experts, five services and capabilities were identified as being especially critical: 1) the use of standard information models and terminologies; 2) the ability to leverage a Decision Support Service (DSS); 3) support for a clinical data query service; 4) support for an event subscription and notification service; and 5) support for a user communication service.

A large-scale knowledge management method based on the analysis of the use of online knowledge resources.

UNLABELLED: Online health knowledge resources can be integrated into electronic health record systems using decision support tools known as "infobuttons." In this study we describe a knowledge management method based on the analysis of knowledge resource use via infobuttons in multiple institutions. METHODS: We conducted a two-phase analysis of laboratory test infobutton sessions at three healthcare institutions accessing two knowledge resources. The primary study measure was session coverage, i.e. the rate of infobutton sessions in which resources retrieved relevant content. RESULTS: In Phase One, resources covered 78.5% of the study sessions. In addition, a subset of 38 noncovered tests that most frequently raised questions was identified. In Phase Two, content development guided by the outcomes of Phase One resulted in a 4% average coverage increase. CONCLUSION: The described method is a valuable approach to large-scale knowledge management in rapidly changing domains.

Multi-National, Multi-Institutional Analysis of Clinical Decision Support Data Needs to Inform Development of the HL7 Virtual Medical Record Standard.

An important barrier to the widespread dissemination of clinical decision support (CDS) is the heterogeneity of information models and terminologies used across healthcare institutions, health information systems, and CDS resources such as knowledge bases. To address this problem, the Health Level 7 (HL7) Virtual Medical Record project (an open, international standards development effort) is developing community consensus on the clinical information exchanged between CDS engines and clinical information systems. As a part of this effort, the HL7 CDS Work Group embarked on a multinational, collaborative effort to identify a representative set of clinical data elements required for CDS. Based on an analysis of CDS systems from 20 institutions representing 4 nations, 131 data elements were identified as being currently utilized for CDS. These findings will inform the development of the emerging HL7 Virtual Medical Record standard and will facilitate the achievement of scalable, standards-based CDS.

SKOLAR MD: a model for self-directed, in-context Continuing Medical Education.

UNLABELLED: SKOLAR has implemented a web-based CME program with which physicians can earn AMA Category 1 credit for self-directed learning. METHODS: Physicians researched their questions in SKOLAR and applied for CME. Physician auditors reviewed all requests across two phases of the project. A selection rule set was derived from phase one and used in phase two to flag a subset of requests for detailed review. The selection rule set is described. RESULTS: In phase one, SKOLAR received 1039 CME applications. Applicants frequently found their answer (94%) and would apply it clinically (93%). A linear regression analysis comparing time awarded to time requested (capped at actual time spent) had R2=0.79. DISCUSSION: We believe that that this self-directed approach to CME is effective and an important complement to traditional CME programs. However, selective audit of self-directed CME requests is necessary to ensure validity of credits awarded.

KnowledgeLink update: just-in-time context-sensitive information retrieval.

Medical knowledge expands at a pace that makes it impossible for the individual clinician to keep up, especially for medications. Medication-related queries are the most common type of query that typically go unanswered during the course of providing care.1 Unanswered questions may result in errors, as found in one study evaluating systems failures associated with adverse drug events. This study found that better information might have prevented half of serious medication errors, and that lack of drug-specific knowledge accounted for the single largest proportion of these events (29%).2 While this information was available somewhere (either on paper or electronically), it was not at the providers' fingertips. Information technology should anticipate clinicians' needs, and bring the information they require to the point of care. For this purpose, we developed an application extender called KnowledgeLink, which provides "just-in-time" context-sensitive information retrieval for drug-related queries.