Point-of-Care Knowledge-Based Resource Needs of Clinicians: A Survey from a Large Academic Medical Center.

OBJECTIVE: To better understand the literature searching preferences of clinical providers we conducted an institution-wide survey assessing the most preferred knowledge searching techniques. MATERIALS AND METHODS: A survey regarding literature searching preferences was sent to 1862 unique clinical providers throughout Mayo Clinic. The survey consisted of 25 items asking respondents to select which clinical scenarios most often prompt literature searches as well as identify their most preferred knowledge resources. RESULTS: A total of 450 completed surveys were returned and analyzed (24% response rate). 48% of respondents perform literature searches for more than half of their patient interactions with 91% of all searches occurring either before or within 3 hours of the patient interaction. When a search is performed 57% of respondents prefer synthesized information sources as compared to only 13% who prefer original research. 82% of knowledge searches are performed on a workstation or office computer while just 10% occur on a mobile device or at home. CONCLUSION: Providers in our survey demonstrate a need to answer clinical questions on a regular basis, especially in the diagnosis and therapy domains. Responses suggest that most of these searches occur using synthesized knowledge sources in the patient care setting within a very short time from the patient interaction.

A complementary graphical method for reducing and analyzing large data sets. Case studies demonstrating thresholds setting and selection.

OBJECTIVES: Graphical displays can make data more understandable; however, large graphs can challenge human comprehension. We have previously described a filtering method to provide high-level summary views of large data sets. In this paper we demonstrate our method for setting and selecting thresholds to limit graph size while retaining important information by applying it to large single and paired data sets, taken from patient and bibliographic databases. METHODS: Four case studies are used to illustrate our method. The data are either patient discharge diagnoses (coded using the International Classification of Diseases, Clinical Modifications [ICD9-CM]) or Medline citations (coded using the Medical Subject Headings [MeSH]). We use combinations of different thresholds to obtain filtered graphs for detailed analysis. The thresholds setting and selection, such as thresholds for node counts, class counts, ratio values, p values (for diff data sets), and percentiles of selected class count thresholds, are demonstrated with details in case studies. The main steps include: data preparation, data manipulation, computation, and threshold selection and visualization. We also describe the data models for different types of thresholds and the considerations for thresholds selection. RESULTS: The filtered graphs are 1%-3% of the size of the original graphs. For our case studies, the graphs provide 1) the most heavily used ICD9-CM codes, 2) the codes with most patients in a research hospital in 2011, 3) a profile of publications on "heavily represented topics" in MEDLINE in 2011, and 4) validated knowledge about adverse effects of the medication of rosiglitazone and new interesting areas in the ICD9-CM hierarchy associated with patients taking the medication of pioglitazone. CONCLUSIONS: Our filtering method reduces large graphs to a manageable size by removing relatively unimportant nodes. The graphical method provides summary views based on computation of usage frequency and semantic context of hierarchical terminology. The method is applicable to large data sets (such as a hundred thousand records or more) and can be used to generate new hypotheses from data sets coded with hierarchical terminologies.

The false security of blind dates: chrononymization's lack of impact on data privacy of laboratory data.

BACKGROUND: The reuse of clinical data for research purposes requires methods for the protection of personal privacy. One general approach is the removal of personal identifiers from the data. A frequent part of this anonymization process is the removal of times and dates, which we refer to as "chrononymization." While this step can make the association with identified data (such as public information or a small sample of patient information) more difficult, it comes at a cost to the usefulness of the data for research. OBJECTIVES: We sought to determine whether removal of dates from common laboratory test panels offers any advantage in protecting such data from re-identification. METHODS: We obtained a set of results for 5.9 million laboratory panels from the National Institutes of Health's (NIH) Biomedical Translational Research Information System (BTRIS), selected a random set of 20,000 panels from the larger source sets, and then identified all matches between the sets. RESULTS: We found that while removal of dates could hinder the re-identification of a single test result, such removal had almost no effect when entire panels were used. CONCLUSIONS: Our results suggest that reliance on chrononymization provides a false sense of security for the protection of laboratory test results. As a result of this study, the NIH has chosen to rely on policy solutions, such as strong data use agreements, rather than removal of dates when reusing clinical data for research purposes.

The practical impact of ontologies on biomedical informatics.

OBJECTIVES: To examine recent research work in the development and evaluation of controlled biomedical terminologies - especially, the representation of structured, controlled definitional knowledge about the terms themselves; such terminologies are often referred to as 'ontologies'. METHODS: A review of the published literature using PubMed, as well as full-text searches of recent Medinfo and American Medical Informatics Association (AMIA) Symposia proceedings, searching for the terms 'ontology' and 'ontologies' and for articles discussing specific, prominent ontological work. RESULTS: We summaries the ontologic aspects of twelve current terminology projects: Galen, the Unified Medical Language System (UMLS), the Medical Entities Dictionary (MED), SNOMED-CT, LOINC, the Foundational Model of Anatomy (FMA), the Gene Ontology (GO), ISO Reference Terminology Model for Nursing Diagnosis, NDF-RT, RxNorm, the NCI Thesaurus, and DOLCE+. We discuss the origins, domain, and ontologic representation of each of these and attempt to summarize the impact that each has had on terminologic work and biomedical applications. We also note the contributions of the Protégé tool to many of these efforts. CONCLUSION: Terminologic research and development have advanced significantly in the past 20 years, especially since the recent orientation toward controlled biomedical ontologies. This work has had significant impact on the development of terminologies themselves, their acceptance and dissemination as standards, and their use in supporting biomedical information systems.

Linking molecular imaging terminology to the gene ontology (GO).

The rapidly developing domain of molecular imaging represents the merging of current advances in the fields of molecular biology and imaging research. Despite this merger, an information gap continues to exist between the scientists who discover new gene products and the imaging scientists who can exploit this information. The Gene Ontology (GO) Consortium seeks to provide a set of structured terminologies for the conceptual annotation of gene product function, process and location in databases. However, no such structured set of concept-oriented terminology exists for the molecular imaging domain. Since the purpose of GO is to capture the information about the role of gene products, we propose that the mapping of GO's established ontological concepts to a molecular imaging terminology will provide the necessary bridge to fill the information gap between the two fields. We have extracted terms and definitions from an already published molecular imaging glossary as well as molecular imaging research articles, and developed molecular imaging concepts. We then mapped our molecular imaging concepts to the existing gene ontology concepts as a method to comprehensively represent molecular imaging.

Consistency across the hierarchies of the UMLS Semantic Network and Metathesaurus.

OBJECTIVE: To develop and test a method for automatically detecting inconsistencies between the parent-child is-a relationships in the Metathesaurus and the ancestor-descendant relationships in the Semantic Network of the Unified Medical Language System (UMLS). METHODS: We exploited the fact that each Metathesaurus concept is assigned one or more semantic types from the UMLS Semantic Network and that the semantic types are arranged in a hierarchy. We compared the semantic types of each pair of parent and child concepts to determine if the types "explained" the Metathesaurus is-a relationships. We considered cases where the semantic type of the parent was neither the same as, nor an ancestor of, the semantic type of the child to be "unexplained." We applied this method to the January 2002 release of the UMLS and examined the unexplained cases we discovered to determine their causes. RESULTS: We found that 17022 (24.3%) of the parent-child is-a relationships in the UMLS Metathesaurus could not be explained based on the semantic types of the concepts. Causes for these discrepancies included cases where the parent or child was missing a semantic type, cases where the semantic type of the child was too general or the semantic type of the parent was too specific, cases where the parent-child relationship was incorrect, and cases where an ancestor-descendant relationship should be added to the UMLS Semantic network. In many cases, the specific cause of the discrepancy cannot be resolved without authoritative judgment by the UMLS developers. CONCLUSIONS: Our method successfully detects inconsistencies between the hierarchies of the UMLS Metathesaurus and Semantic Network. We believe that our method should be added to the set of tools that the UMLS developers use to maintain and audit the UMLS knowledge sources.

Building a knowledge base to support a digital library.

As part of an effort to develop a knowledge base to support searching online medical literature according to individual needs, we have studied the possibility of using the co-occurrence of MeSH terms in MEDLINE citations associated with the search strategies optimal for evidence based medicine to automated construction of a knowledge base. This study evaluates the relevance of the relationships between the semantic relationship pairs generated by the process, and the clinical validity of the semantic types involved in the process. From the semantic pairs proposed by our method, a group of clinicians judge sixty percent to be relevant. The remaining forty percent included semantic types considered unimportant by clinicians. The knowledge extraction method showed reasonable results. We believe it can be appropriate for the task of retrieving information from the medical record in order to guide users during a searching and retrieval process. Future directions include the validation of the knowledge, based on an evaluation of system performance.

What do patients do with access to their medical records?

We sought to study the phenomenon of patients having access to their own medical records in order to determine the impact on them and on their relationship with their health care providers. We created the Patient Clinical Information System (PatCIS) to interface with the clinical data repository at New York Presbyterian Hospital to allow patients to add to and review their medical data. We also provided educational resources and automated advice programs. We provided access to the system to thirteen subjects over a nineteen-month period and reviewed their activities in the system's usage log. We also collected data via questionnaire and telephone interview. We found that patients varied in their use of the system, from once a month or less to one or more times per day. All patients primarily used the system to review laboratory results. Both they and their physicians believed that use of the system enhanced the patients' understanding of their conditions and improved their communication with their physicians. There were no adverse events encountered during the study.

Developing tailored theory-based educational content for WEB applications: illustrations from the MI-HEART project.

This paper describes how theory facilitated the development of educational content for the MI-HEART project, a tailored Web-based intervention designed to favorably influence the appropriateness and rapidity of decision-making in patients suffering from symptoms of acute myocardial infarction. There were five steps involved: 1) formulating the behavioral goal, 2) defining intervention objectives based on an analyses of the determinants of behavior, 3) developing an assessment tool to measure a person's status on these determinants, 4) creating tailored content that address individual variation on determinants of the health behavior and, 5) developing algorithms and a computer program that link responses from the assessment to specific tailored communication. The approach we describe largely distinguishes Web-based applications that are designed to change health behavior from those that simply impart information. Developers of Web-based applications that propose to improve health status by modifying health-related behaviors need the understanding that although it is said that we live in an "information age", simply increasing knowledge has not been effective in changing behaviors in most instances. Furthermore, the one-size fits all approach to developing educational content cannot address the needs, concerns and interests of different individuals. With informatics technology, our ability to collect information from individuals and provide educational content tailored to the specific information collected is not only possibly, but practical.

Terminology tools: state of the art and practical lessons.

OBJECTIVES: As controlled medical terminologies evolve from simple code-name-hierarchy arrangements, into rich, knowledge-based ontologies of medical concepts, increased demands are placed on both the developers and users of the terminologies. In response, researchers have begun developing tools to address their needs. The aims of this article are to review previous work done to develop these tools and then to describe work done at Columbia University and New York Presbyterian Hospital (NYPH). METHODS: Researchers working with the Systematized Nomenclature of Medicine (SNOMED), the Unified Medical Language System (UMLS), and NYPH's Medical Entities Dictionary (MED) have created a wide variety of terminology browsers, editors and servers to facilitate creation, maintenance and use of these terminologies. RESULTS: Although much work has been done, no generally available tools have yet emerged. Consensus on requirement for tool functions, especially terminology servers is emerging. Tools at NYPH have been used successfully to support the integration of clinical applications and the merger of health care institutions. CONCLUSIONS: Significant advancement has occurred over the past fifteen years in the development of sophisticated controlled terminologies and the tools to support them. The tool set at NYPH provides a case study to demonstrate one feasible architecture.

Partitioning an object-oriented terminology schema.

Controlled medical terminologies are increasingly becoming strategic components of various healthcare enterprises. However, the typical medical terminology can be difficult to exploit due to its extensive size and high density. The schema of a medical terminology offered by an object-oriented representation is a valuable tool in providing an abstract view of the terminology, enhancing comprehensibility and making it more usable. However, schemas themselves can be large and unwieldy. We present a methodology for partitioning a medical terminology schema into manageably sized fragments that promote increased comprehension. Our methodology has a refinement process for the subclass hierarchy of the terminology schema. The methodology is carried out by a medical domain expert in conjunction with a computer. The expert is guided by a set of three modeling rules, which guarantee that the resulting partitioned schema consists of a forest of trees. This makes it easier to understand and consequently use the medical terminology. The application of our methodology to the schema of the Medical Entities Dictionary (MED) is presented.

A knowledge-based, concept-oriented view generation system for clinical data.

Information overload is a well-known problem for clinicians who must review large amounts of data in patient records. Concept-oriented views, which organize patient data around clinical concepts such as diagnostic strategies and therapeutic goals, may offer a solution to the problem of information overload. However, although concept-oriented views are desirable, they are difficult to create and maintain. We have developed a general-purpose, knowledge-based approach to the generation of concept-oriented views and have developed a system to test our approach. The system creates concept-oriented views through automated identification of relevant patient data. The knowledge in the system is represented by both a semantic network and rules. The key relevant data identification function is accomplished by a rule-based traversal of the semantic network. This paper focuses on the design and implementation of the system; an evaluation of the system is reported separately.

Accessing heterogeneous sources of evidence to answer clinical questions.

The large and rapidly growing number of information sources relevant to health care, and the increasing amounts of new evidence produced by researchers, are improving the access of professionals and students to valuable information. However, seeking and filtering useful, valid information can be still very difficult. An online information system that conducts searches based on individual patient data can have a beneficial influence on the particular patient's outcome and educate the healthcare worker. In this paper, we describe the underlying model for a system that aims to facilitate the search for evidence based on clinicians' needs. This paper reviews studies of information needs of clinicians, describes principles of information retrieval, and examines the role that standardized terminologies can play in the integration between a clinical system and literature resources, as well as in the information retrieval process. The paper also describes a model for a digital library system that supports the integration of clinical systems with online information sources, making use of information available in the electronic medical record to enhance searches and information retrieval. The model builds on several different, previously developed techniques to identify information themes that are relevant to specific clinical data. Using a framework of evidence-based practice, the system generates well-structured questions with the intent of enhancing information retrieval. We believe that by helping clinicians to pose well-structured clinical queries and including in them relevant information from individual patients' medical records, we can enhance information retrieval and thus can improve patient-care.

'Televaluation' of clinical information systems: an integrative approach to assessing Web-based systems.

The World Wide Web provides an unprecedented opportunity for widespread access to health-care applications by both patients and providers. The development of new methods for assessing the effectiveness and usability of these systems is becoming a critical issue. This paper describes the distance evaluation (i.e. 'televaluation') of emerging Web-based information technologies. In health informatics evaluation, there is a need for application of new ideas and methods from the fields of cognitive science and usability engineering. A framework is presented for conducting evaluations of health-care information technologies that integrates a number of methods, ranging from deployment of on-line questionnaires (and Web-based forms) to remote video-based usability testing of user interactions with clinical information systems. Examples illustrating application of these techniques are presented for the assessment of a patient clinical information system (PatCIS), as well as an evaluation of use of Web-based clinical guidelines. Issues in designing, prototyping and iteratively refining evaluation components are discussed, along with description of a 'virtual' usability laboratory.

Studying the human-computer-terminology interface.

OBJECTIVE: To explore the use of an observational, cognitive-based approach for differentiating between successful, suboptimal, and failed entry of coded data by clinicians in actual practice, and to detect whether causes for unsuccessful attempts to capture true intended meaning were due to terminology content, terminology representation, or user interface problems. DESIGN: Observational study with videotaping and subsequent coding of data entry events in an outpatient clinic at New York Presbyterian Hospital. PARTICIPANTS: Eight attending physicians, 18 resident physicians, and 1 nurse practitioner, using the Medical Entities Dictionary (MED) to record patient problems, medications, and adverse reactions in an outpatient medical record system. MEASUREMENTS: Classification of data entry events as successful, suboptimal, or failed, and estimation of cause; recording of system response time and total event time. RESULTS: Two hundred thirty-eight data entry events were analyzed; 71.0 percent were successful, 6.3 percent suboptimal, and 22.7 percent failed; unsuccessful entries were due to problems with content in 13.0 percent of events, representation problems in 10.1 percent of events, and usability problems in 5.9 percent of events. Response time averaged 0.74 sec, and total event time averaged 40.4 sec. Of an additional 209 tasks related to drug dose and frequency terms, 94 percent were successful, 0.5 percent were suboptimal, and 6 percent failed, for an overall success rate of 82 percent. CONCLUSIONS: Data entry by clinicians using the outpatient system and the MED was generally successful and efficient. The cognitive-based observational approach permitted detection of false-positive (suboptimal) and false-negative (failed due to user interface) data entry.

Integration of nursing assessment concepts into the medical entities dictionary using the LOINC semantic structure as a terminology model.

Recent investigations have tested the applicability of various terminology models for the representing nursing concepts including those related to nursing diagnoses, nursing interventions, and standardized nursing assessments as a prerequisite for building a reference terminology that supports the nursing domain. We used the semantic structure of Clinical LOINC (Logical Observations, Identifiers, Names, and Codes) as a reference terminology model to support the integration of standardized assessment terms from two nursing terminologies into the Medical Entities Dictionary (MED), the concept-oriented, metadata dictionary at New York Presbyterian Hospital. Although the LOINC semantic structure was used previously to represent laboratory terms in the MED, selected hierarchies and semantic slots required revisions in order to incorporate the nursing assessment concepts. This project was an initial step in integrating nursing assessment concepts into the MED in a manner consistent with evolving standards for reference terminology models. Moreover, the revisions provide the foundation for adding other types of standardized assessments to the MED.

Battling Scylla and Charybdis: the search for redundancy and ambiguity in the 2001 UMLS metathesaurus.

I previously developed methods for identifying cases of multiple synonymous concepts (redundancy) and concepts with multiple meanings (ambiguity) and applied them to the 1995 UMLS Metathesaurus. These methods use semantic approaches (including knowledge about word synonymy and the semantic types assigned to concepts) to complement the standard lexical approaches. In this paper, I describe the results of their application to the 2001 Metathesaurus and examine their implications for the evolution of the UMLS.

Using narrative reports to support a digital library.

The vast amount of information collected and stored in clinical systems can be a significant challenge in the integration of digital libraries and electronic medical records, especially the selection of clinical data to be used in the search, retrieval, and summarization processes. In this study, we describe the use of information retrieval measures with natural language processor output to identify critical information in narrative reports. Our hypothesis is that clinical data that occur often in narrative reports are less important to clinicians than findings that occur rarely. We used the information retrieval methods to analyze one year of discharge summaries. We then conducted a performance study, using physicians as subject. Results show that the methods can be used for filtering critical information from reports. Further studies need to be done on evaluation of the method based on an evaluation of the system performance in the context of a digital library.

Perceived information needs and communication difficulties of inpatient physicians and nurses.

In order to understand the differing perceptions of information needs and communication patterns of healthcare professionals as they relate to medical errors, we conducted a survey and 5 focus group sessions of inpatient physicians and nurses. Although nurses and physicians stated differing information needs, both groups expressed significant problems with obtaining patient, domain and institution-specific information in a timely manner. Identification of appropriate providers and establishing contact with those people was perceived as the most pressing communication need. All focus group participants felt that communication difficulties were common and could give examples in which such difficulties led to adverse events. Our studies suggest that information needs and communication difficulties are common and can lead to medical errors or near misses. Many of these problems may be amenable to information technology solutions.