Lessons learned from implementing service-oriented clinical decision support at four sites: A qualitative study.

OBJECTIVE: To identify challenges, lessons learned and best practices for service-oriented clinical decision support, based on the results of the Clinical Decision Support Consortium, a multi-site study which developed, implemented and evaluated clinical decision support services in a diverse range of electronic health records. METHODS: Ethnographic investigation using the rapid assessment process, a procedure for agile qualitative data collection and analysis, including clinical observation, system demonstrations and analysis and 91 interviews. RESULTS: We identified challenges and lessons learned in eight dimensions: (1) hardware and software computing infrastructure, (2) clinical content, (3) human-computer interface, (4) people, (5) workflow and communication, (6) internal organizational policies, procedures, environment and culture, (7) external rules, regulations, and pressures and (8) system measurement and monitoring. Key challenges included performance issues (particularly related to data retrieval), differences in terminologies used across sites, workflow variability and the need for a legal framework. DISCUSSION: Based on the challenges and lessons learned, we identified eight best practices for developers and implementers of service-oriented clinical decision support: (1) optimize performance, or make asynchronous calls, (2) be liberal in what you accept (particularly for terminology), (3) foster clinical transparency, (4) develop a legal framework, (5) support a flexible front-end, (6) dedicate human resources, (7) support peer-to-peer communication, (8) improve standards. CONCLUSION: The Clinical Decision Support Consortium successfully developed a clinical decision support service and implemented it in four different electronic health records and four diverse clinical sites; however, the process was arduous. The lessons identified by the Consortium may be useful for other developers and implementers of clinical decision support services.

Sensitivity analysis of methods for active surveillance of acute myocardial infarction using electronic databases.

BACKGROUND: The validity of conclusions from observational studies depends on decisions regarding design, analysis, data quality, and implementation. Through sensitivity analyses, we explored the impact of such decisions on balance control and risk estimates. METHODS: Using as a template the Mini-Sentinel protocol for the active surveillance of acute myocardial infarction (MI) in association with use of antidiabetic agents, we defined cohorts of new users of metformin and second-generation sulfonylureas, baseline covariates and acute MI events using three combinations of washout and baseline periods. Using propensity-score matching, we assessed balance control and risk estimates using cumulative data for matching all patients compared with not rematching prior matches in quarterly analyses over the follow-up period. RESULTS: A longer washout period increased the confidence in new-user status, but at the expense of sample size; a longer baseline period improved capture of covariates related to pre-existing chronic conditions. When all patients were matched each quarter, balance was improved and risk estimates were more robust, especially in the later quarters. CONCLUSIONS: Durations of washout and baseline periods influence the likelihood of new-user status and sample size. Matching all patients tends to result in better covariate balance than matching only new patients. Decisions regarding the durations of washout and baseline periods depend on the specific research question and availability of longitudinal patient data within the database. This paper demonstrates the importance and utility of sensitivity analysis of methods for evaluating the robustness of results in observational studies.

Measuring agreement between decision support reminders: the cloud vs. the local expert.

BACKGROUND: A cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data. METHODS: Electronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen's Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients. RESULTS: The cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95% CI 0.24 - 0.42) to 0.99 (95% CI 0.97 - 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders. CONCLUSIONS: Preventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.

Regenstrief Institute's Medical Gopher: a next-generation homegrown electronic medical record system.

OBJECTIVE: Regenstrief Institute developed one of the seminal computerized order entry systems, the Medical Gopher, for implementation at Wishard Hospital nearly three decades ago. Wishard Hospital and Regenstrief remain committed to homegrown software development, and over the past 4 years we have fully rebuilt Gopher with an emphasis on usability, safety, leveraging open source technologies, and the advancement of biomedical informatics research. Our objective in this paper is to summarize the functionality of this new system and highlight its novel features. MATERIALS AND METHODS: Applying a user-centered design process, the new Gopher was built upon a rich-internet application framework using an agile development process. The system incorporates order entry, clinical documentation, result viewing, decision support, and clinical workflow. We have customized its use for the outpatient, inpatient, and emergency department settings. RESULTS: The new Gopher is now in use by over 1100 users a day, including an average of 433 physicians caring for over 3600 patients daily. The system includes a wizard-like clinical workflow, dynamic multimedia alerts, and a familiar 'e-commerce'-based interface for order entry. Clinical documentation is enhanced by real-time natural language processing and data review is supported by a rapid chart search feature. DISCUSSION: As one of the few remaining academically developed order entry systems, the Gopher has been designed both to improve patient care and to support next-generation informatics research. It has achieved rapid adoption within our health system and suggests continued viability for homegrown systems in settings of close collaboration between developers and providers.

A pilot study of distributed knowledge management and clinical decision support in the cloud.

OBJECTIVE: Implement and perform pilot testing of web-based clinical decision support services using a novel framework for creating and managing clinical knowledge in a distributed fashion using the cloud. The pilot sought to (1) develop and test connectivity to an external clinical decision support (CDS) service, (2) assess the exchange of data to and knowledge from the external CDS service, and (3) capture lessons to guide expansion to more practice sites and users. MATERIALS AND METHODS: The Clinical Decision Support Consortium created a repository of shared CDS knowledge for managing hypertension, diabetes, and coronary artery disease in a community cloud hosted by Partners HealthCare. A limited data set for primary care patients at a separate health system was securely transmitted to a CDS rules engine hosted in the cloud. Preventive care reminders triggered by the limited data set were returned for display to clinician end users for review and display. During a pilot study, we (1) monitored connectivity and system performance, (2) studied the exchange of data and decision support reminders between the two health systems, and (3) captured lessons. RESULTS: During the six month pilot study, there were 1339 patient encounters in which information was successfully exchanged. Preventive care reminders were displayed during 57% of patient visits, most often reminding physicians to monitor blood pressure for hypertensive patients (29%) and order eye exams for patients with diabetes (28%). Lessons learned were grouped into five themes: performance, governance, semantic interoperability, ongoing adjustments, and usability. DISCUSSION: Remote, asynchronous cloud-based decision support performed reasonably well, although issues concerning governance, semantic interoperability, and usability remain key challenges for successful adoption and use of cloud-based CDS that will require collaboration between biomedical informatics and computer science disciplines. CONCLUSION: Decision support in the cloud is feasible and may be a reasonable path toward achieving better support of clinical decision-making across the widest range of health care providers.

Using a service oriented architecture approach to clinical decision support: performance results from two CDS Consortium demonstrations.

The Clinical Decision Support Consortium has completed two demonstration trials involving a web service for the execution of clinical decision support (CDS) rules in one or more electronic health record (EHR) systems. The initial trial ran in a local EHR at Partners HealthCare. A second EHR site, associated with Wishard Memorial Hospital, Indianapolis, IN, was added in the second trial. Data were gathered during each 6 month period and analyzed to assess performance, reliability, and response time in the form of means and standard deviations for all technical components of the service, including assembling and preparation of input data. The mean service call time for each period was just over 2 seconds. In this paper we report on the findings and analysis to date while describing the areas for further analysis and optimization as we continue to expand our use of a Services Oriented Architecture approach for CDS across multiple institutions.

Variability in drug formularies and implications in decision support.

Formularies are highly variable, which limits physicians ability to prescribe cost effective medications for their patients. In this study we explore the composition of 3,346 formularies in terms of outpatient prescription coverage, medication class coverage, and cost implications. Our analysis revealed that 42% of formularies are duplicative and that the unique formularies contain variability for medications that contribute little in terms of cost or overall use. These results lead us to believe the number and complexities of formularies can be dramatically reduced leading to more intuitive decision support for physicians when writing prescriptions.

Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems.

BACKGROUND: Clinical decision support (CDS) is a valuable tool for improving healthcare quality and lowering costs. However, there is no comprehensive taxonomy of types of CDS and there has been limited research on the availability of various CDS tools across current electronic health record (EHR) systems. OBJECTIVE: To develop and validate a taxonomy of front-end CDS tools and to assess support for these tools in major commercial and internally developed EHRs. STUDY DESIGN AND METHODS: We used a modified Delphi approach with a panel of 11 decision support experts to develop a taxonomy of 53 front-end CDS tools. Based on this taxonomy, a survey on CDS tools was sent to a purposive sample of commercial EHR vendors (n=9) and leading healthcare institutions with internally developed state-of-the-art EHRs (n=4). RESULTS: Responses were received from all healthcare institutions and 7 of 9 EHR vendors (response rate: 85%). All 53 types of CDS tools identified in the taxonomy were found in at least one surveyed EHR system, but only 8 functions were present in all EHRs. Medication dosing support and order facilitators were the most commonly available classes of decision support, while expert systems (eg, diagnostic decision support, ventilator management suggestions) were the least common. CONCLUSION: We developed and validated a comprehensive taxonomy of front-end CDS tools. A subsequent survey of commercial EHR vendors and leading healthcare institutions revealed a small core set of common CDS tools, but identified significant variability in the remainder of clinical decision support content.

Comparison of clinical knowledge management capabilities of commercially-available and leading internally-developed electronic health records.

BACKGROUND: We have carried out an extensive qualitative research program focused on the barriers and facilitators to successful adoption and use of various features of advanced, state-of-the-art electronic health records (EHRs) within large, academic, teaching facilities with long-standing EHR research and development programs. We have recently begun investigating smaller, community hospitals and out-patient clinics that rely on commercially-available EHRs. We sought to assess whether the current generation of commercially-available EHRs are capable of providing the clinical knowledge management features, functions, tools, and techniques required to deliver and maintain the clinical decision support (CDS) interventions required to support the recently defined "meaningful use" criteria. METHODS: We developed and fielded a 17-question survey to representatives from nine commercially available EHR vendors and four leading internally developed EHRs. The first part of the survey asked basic questions about the vendor's EHR. The second part asked specifically about the CDS-related system tools and capabilities that each vendor provides. The final section asked about clinical content. RESULTS: All of the vendors and institutions have multiple modules capable of providing clinical decision support interventions to clinicians. The majority of the systems were capable of performing almost all of the key knowledge management functions we identified. CONCLUSION: If these well-designed commercially-available systems are coupled with the other key socio-technical concepts required for safe and effective EHR implementation and use, and organizations have access to implementable clinical knowledge, we expect that the transformation of the healthcare enterprise that so many have predicted, is achievable using commercially-available, state-of-the-art EHRs.

Querying the National Drug File Reference Terminology (NDFRT) to assign drugs to decision support categories.

INTRODUCTION: The accurate categorization of drugs is a prerequisite for decision support rules. The manual process of creating drug classes can be laborious and error-prone. METHODS: All 142 drug classes currently used at Regenstrief Institute for drug interaction alerts were extracted. These drug classes were replicated as fully-defined concepts in our local instance of the NDFRT knowledge base. The performance of these two strategies (manual classification vs. NDFRT-based queries) was compared, and the sensitivity and specificity of each was calculated. RESULTS: Compared to existing manual classifications, NDFRT-based queries made a greater number of correct class-drug assignments: 1528 vs. 1266. NDFRT queries have greater sensitivity (74.9% vs. 62.1%) to classify drugs. However, they have less specificity (85.6% vs. 99.8%). CONCLUSION: The NDFRT knowledge base shows promise for use in an automated strategy to improve the creation and update of drug classes. The chief disadvantage of our NDFRT-based approach was a greater number of false positive assignments due to the inclusion of non-systemic doseforms.

Continuity of Care Document (CCD) Enables Delivery of Medication Histories to the Primary Care Clinician.

INTRODUCTION: The goal of the Enhanced Medication History (EMH) project is to provide medication histories to ambulatory primary care practices in the Indiana Network for Patient Care. METHODS: Medications were aggregated from three different sources of pharmacy data (Medicaid, SureScripts, and the county health system of Indianapolis). Dispensing events were assembled into the Continuity of Care Document (CCD), and presented to clinicians as RxNorm Clinical Drugs. RESULTS: The EMH project completed 46 weeks of operation in a community health center in Indianapolis. Medication Histories were generated for 10498 office visits for 4449 distinct patients. Seven (of nine) attending physicians responded to a written survey and found the Medication Histories useful (3.9±0.4 on a scale of 1 to 5). CONCLUSION: Implementation of the EMH project demonstrated the successful use (as well as the challenging aspects) of the CCD and the RxNorm terminology in the outpatient clinical setting.

The state of the art in clinical knowledge management: an inventory of tools and techniques.

PURPOSE: To explore the need for, and use of, high-quality, collaborative, clinical knowledge management (CKM) tools and techniques to manage clinical decision support (CDS) content. METHODS: In order to better understand the current state of the art in CKM, we developed a survey of potential CKM tools and techniques. We conducted an exploratory study by querying a convenience sample of respondents about their use of specific practices in CKM. RESULTS: The following tools and techniques should be priorities in organizations interested in developing successful computer-based provider order entry (CPOE) and CDS implementations: (1) a multidisciplinary team responsible for creating and maintaining the clinical content; (2) an external organizational repository of clinical content with web-based viewer that allows anyone in the organization to review it; (3) an online, collaborative, interactive, Internet-based tool to facilitate content development; (4) an enterprise-wide tool to maintain the controlled clinical terminology concepts. Even organizations that have been successfully using computer-based provider order entry with advanced clinical decision support features for well over 15 years are not using all of the CKM tools or practices that we identified. CONCLUSIONS: If we are to further stimulate progress in the area of clinical decision support, we must continue to develop and refine our understanding and use of advanced CKM capabilities.

Using National Drug Codes and drug knowledge bases to organize prescription records from multiple sources.

PURPOSE: The utility of National Drug Codes (NDCs) and drug knowledge bases (DKBs) in the organization of prescription records from multiple sources was studied. METHODS: The master files of most pharmacy systems include NDCs and local codes to identify the products they dispense. We obtained a large sample of prescription records from seven different sources. These records carried a national product code or a local code that could be translated into a national product code via their formulary master. We obtained mapping tables from five DKBs. We measured the degree to which the DKB mapping tables covered the national product codes carried in or associated with the sample of prescription records. RESULTS: Considering the total prescription volume, DKBs covered 93.0-99.8% of the product codes from three outpatient sources and 77.4-97.0% of the product codes from four inpatient sources. Among the in-patient sources, invented codes explained 36-94% of the noncoverage. Outpatient pharmacy sources rarely invented codes, which comprised only 0.11-0.21% of their total prescription volume, compared with inpatient pharmacy sources for which invented codes comprised 1.7-7.4% of their prescription volume. The distribution of prescribed products was highly skewed, with 1.4-4.4% of codes accounting for 50% of the message volume and 10.7-34.5% accounting for 90% of the message volume. CONCLUSION: DKBs cover the product codes used by outpatient sources sufficiently well to permit automatic mapping. Changes in policies and standards could increase coverage of product codes used by inpatient sources.

Building a production-ready infrastructure to enhance medication management: early lessons from the nationwide health information network.

Poor medication management practices can lead to serious erosion of health care quality and safety. The DHHS Medication Management Use Case outlines methods for the exchange of electronic health information to improve medication management practices. In this case report, the authors describe initial development of Nationwide Health Information Network (NHIN) services to support the Medication Management Use Case. The technical approach and core elements of medication management transactions involved in the NHIN are presented. Early lessons suggest the pathway to improvements in quality and safety are achievable, yet there are challenges for the medical informatics community to address through future research and development activities.

Enhancing an ePrescribing system by adding medication histories and formularies: the Regenstrief Medication Hub.

Medication histories improve health care quality and safety; formularies serve to control costs. We describe the implementation of the Regenstrief Medication Hub: a system to provide both histories and formularies to the Gopher ePrescribing application. Currently the Medication Hub aggregates data from two sources: the RxHub consortium of pharmacy benefit managers, and Wishard Health Services. During one month, the system generated 53,764 queries, each representing a patient visit. RxHub responded with 4,012 histories; Wishard responded with 23,421 histories. The Medication Hub aggregated and filtered these histories before delivering them to Gopher. However, clinician users accessed the histories during only 0.6% of prescribing sessions. The Medication Hub also managed drug benefit eligibility data, which enabled formulary-based decision support. However, clinicians heeded only 41% of warnings based on the Wishard Formulary, and 16% of warnings based on commercial formularies. The Medication Hub is scalable to accommodate additional pharmacy data sources.

Medication and indication linkage: A practical therapy for the problem list?

BACKGROUND: Establishing a relationship between medications and diagnoses within a functioning electronic medical record system (EMR) has many valuable applications,such as improving the quality and utility of the problem list to support better decisions. METHODS: We evaluated over 1.6 million de-identified patient records from the Regenstrief Medical Record System (RMRS) with over 90 million diagnoses and 20 million medications. Using RxNorm, the VA National Drug File Reference Terminology, and SNOMED-CT (SCT)standard terminologies and mappings we evaluated the linkage for local concept terms for medications and problems (diagnoses & complaints). RESULTS: We were able to map 24,398 candidates as medication and indication pairs. The overall sensitivity and specificity for term pairs was 67.5% and 86% respectively and 39.5% and 97.4 when adjusted for term pair occurrence within single patient records. CONCLUSIONS: Medications can be mapped by machine to a disease/ disorder using established terminology standards.This mapping may inform many knowledge management and decision support features in an EMR.

Aggregation of pharmacy dispensing data into a unified patient medication history.

The Regenstrief Medication Hub system collects pharmacy data from two different sources: Wishard Health Services, and dispensing claims provided by RxHub. These lists are indexed, aggregated, and filtered, to create a single Medication History for each patient. This history is then provided to the Gopher computerized prescribing system. The Medication Hub is a scalable system, capable of integrating additional sources of pharmacy data.

Extensible Stylesheet Language Formatting Objects (XSL-FO): a tool to transform patient data into attractive clinical reports.

Clinicians at Wishard Hospital in Indianapolis print and carry clinical reports called "Pocket Rounds". This paper describes a new process we developed to improve these clinical reports. The heart of our new process is a World Wide Web Consortium standard: Extensible Stylesheet Language Formatting Objects (XSL-FO). Using XSL-FO stylesheets we generated Portable Document Format (PDF) and PostScript reports with complex formatting: columns, tables, borders, shading, indents, dividing lines. We observed patterns of clinical report printing during a eight month study period on three Medicine wards. Usage statistics indicated that clinicians accepted the new system enthusiastically: 78% of 26,418 reports were printed using the new system. We surveyed 67 clinical users. Respondents gave the new reports a rating of 4.2 (on a 5 point scale); they gave the old reports a rating of 3.4. The primary complaint was that it took longer to print the new reports. We believe that XSL-FO is a promising way to transform text data into functional and attractive clinical reports: relatively easy to implement and modify.