Mapping the delineation of practice to the AMIA foundational domains for applied health informatics.

OBJECTIVE: This article reports on the alignment between the foundational domains and the delineation of practice (DoP) for health informatics, both developed by the American Medical Informatics Association (AMIA). Whereas the foundational domains guide graduate-level curriculum development and accreditation assessment, providing an educational pathway to the minimum competencies needed as a health informatician, the DoP defines the domains, tasks, knowledge, and skills that a professional needs to competently perform in the discipline of health informatics. The purpose of this article is to determine whether the foundational domains need modification to better reflect applied practice. MATERIALS AND METHODS: Using an iterative process and through individual and collective approaches, the foundational domains and the DoP statements were analyzed for alignment and eventual harmonization. Tables and Sankey plot diagrams were used to detail and illustrate the resulting alignment. RESULTS: We were able to map all the individual DoP knowledge statements and tasks to the AMIA foundational domains, but the statements within a single DoP domain did not all map to the same foundational domain. Even though the AMIA foundational domains and DoP domains are not in perfect alignment, the DoP provides good examples of specific health informatics competencies for most of the foundational domains. There are, however, limited DoP knowledge statements and tasks mapping to foundational domain 6-Social and Behavioral Aspects of Health. DISCUSSION: Both the foundational domains and the DoP were developed independently, several years apart, and for different purposes. The mapping analyses reveal similarities and differences between the practice experience and the curricular needs of health informaticians. CONCLUSIONS: The overall alignment of both domains may be explained by the fact that both describe the current and/or future health informatics professional. One can think of the foundational domains as representing the broad foci for educational programs for health informaticians and, hence, they are appropriately the focus of organizations that accredit these programs.

Why is biomedical informatics hard? A fundamental framework.

Building on previous work to define the scientific discipline of biomedical informatics, we present a framework that categorizes fundamental challenges into groups based on data, information, and knowledge, along with the transitions between these levels. We define each level and argue that the framework provides a basis for separating informatics problems from non-informatics problems, identifying fundamental challenges in biomedical informatics, and provides guidance regarding the search for general, reusable solutions to informatics problems. We distinguish between processing data (symbols) and processing meaning. Computational systems, that are the basis for modern information technology (IT), process data. In contrast, many important challenges in biomedicine, such as providing clinical decision support, require processing meaning, not data. Biomedical informatics is hard because of the fundamental mismatch between many biomedical problems and the capabilities of current technology.

Caries Risk Documentation And Prevention: eMeasures For Dental Electronic Health Records.

BACKGROUND: Longitudinal patient level data available in the electronic health record (EHR) allows for the development, implementation, and validations of dental quality measures (eMeasures). OBJECTIVE: We report the feasibility and validity of implementing two eMeasures. The eMeasures determined the proportion of patients receiving a caries risk assessment (eCRA) and corresponding appropriate risk-based preventative treatments for patients at elevated risk of caries (appropriateness of care [eAoC]) in two academic institutions and one accountable care organization, in the 2019 reporting year. METHODS: Both eMeasures define the numerator and denominator beginning at the patient level, populations' specifications, and validated the automated queries. For eCRA, patients who completed a comprehensive or periodic oral evaluation formed the denominator, and patients of any age who received a CRA formed the numerator. The eAoC evaluated the proportion of patients at elevated caries risk who received the corresponding appropriate risk-based preventative treatments. RESULTS: EHR automated queries identified in three sites 269,536 patients who met the inclusion criteria for receiving a CRA. The overall proportion of patients who received a CRA was 94.4% (eCRA). In eAoC, patients at elevated caries risk levels (moderate, high, or extreme) received fluoride preventive treatment ranging from 56 to 93.8%. For patients at high and extreme risk, antimicrobials were prescribed more frequently site 3 (80.6%) than sites 2 (16.7%) and 1 (2.9%). CONCLUSION: Patient-level data available in the EHRs can be used to implement process-of-care dental eCRA and AoC, eAoC measures identify gaps in clinical practice. EHR-based measures can be useful in improving delivery of evidence-based preventative treatments to reduce risk, prevent tooth decay, and improve oral health.

Exploring the Hazards of Scaling Up Clinical Data Analyses: A Drug Side Effect Discovery Case Report.

We assessed the scalability of pharmacological signal detection use case from a single-site CDW to a large aggregated clinical data warehouse (single-site database with 754,214 distinct patient IDs vs. multisite database with 49.8M). We aimed to explore whether a larger clinical dataset would provide clearer signals for secondary analyses such as detecting the known relationship between prednisone and weight. We found significant weight gain rate using the single-site data but not from using aggregated data (0.0104 kg/day, p<0.0001 vs. -0.050 kg/day, p<.0001). This rate was also found more consistently across 30 age and gender subgroups using the single-site data than in the aggregated data (26 vs. 18 significant weight gain findings). Contrary to our expectations, analyses of much larger aggregated clinical datasets did not yield stronger signals. Researchers must check the underlying model assumptions and account for greater heterogeneity when analyzing aggregated multisite data to ensure reliable findings.

Assessing the completeness of periodontal disease documentation in the EHR: a first step in measuring the quality of care.

BACKGROUND: Our objective was to measure the proportion of patients for which comprehensive periodontal charting, periodontal disease risk factors (diabetes status, tobacco use, and oral home care compliance), and periodontal diagnoses were documented in the electronic health record (EHR). We developed an EHR-based quality measure to assess how well four dental institutions documented periodontal disease-related information. An automated database script was developed and implemented in the EHR at each institution. The measure was validated by comparing the findings from the measure with a manual review of charts. RESULTS: The overall measure scores varied significantly across the four institutions (institution 1 = 20.47%, institution 2 = 0.97%, institution 3 = 22.27% institution 4 = 99.49%, p-value < 0.0001). The largest gaps in documentation were related to periodontal diagnoses and capturing oral homecare compliance. A random sample of 1224 charts were manually reviewed and showed excellent validity when compared with the data generated from the EHR-based measure (Sensitivity, Specificity, PPV, and NPV > 80%). CONCLUSION: Our results demonstrate the feasibility of developing automated data extraction scripts using structured data from EHRs, and successfully implementing these to identify and measure the periodontal documentation completeness within and across different dental institutions.

Measuring sealant placement in children at the dental practice level.

BACKGROUND: Although sealants are an established and recommended caries-preventive treatment, many children still fail to receive them. In addition, research has shown that existing measures underestimate care by overlooking the sealable potential of teeth before evaluating care. To address this, the authors designed and evaluated 3 novel dental electronic health record-based clinical quality measures that evaluate sealant care only after assessing the sealable potential of teeth. METHODS: Measure I recorded the proportion of patients with sealable teeth who received sealants. Measure II recorded the proportion of patients who had at least 1 of their sealable teeth sealed. Measure III recorded the proportion of patients who received sealant on all of their sealable teeth. RESULTS: On average, 48.1% of 6- through 9-year-old children received 1 or more sealants compared with 32.4% of 10- through 14-year-olds (measure I). The average measure score decreased for patients who received sealants for at least 1 of their sealable teeth (measure II) (43.2% for 6- through 9-year-olds and 28.4% for 10- through 14-year-olds). Fewer children received sealants on all eligible teeth (measure III) (35.5% of 6- through 9-year-olds and 21% of 10- through 14-year-olds received sealant on all eligible teeth). Among the 48.5% who were at elevated caries risk, the sealant rates were higher across all 3 measures. CONCLUSIONS: A valid and actionable practice-based sealant electronic measure that evaluates sealant treatment among the eligible population, both at the patient level and the tooth level, has been developed. PRACTICAL IMPLICATIONS: The measure developed in this work provides practices with patient-centered and actionable sealant quality measures that aim to improve oral health outcomes.

DataGauge: A Practical Process for Systematically Designing and Implementing Quality Assessments of Repurposed Clinical Data.

The well-known hazards of repurposing data make Data Quality (DQ) assessment a vital step towards ensuring valid results regardless of analytical methods. However, there is no systematic process to implement DQ assessments for secondary uses of clinical data. This paper presents DataGauge, a systematic process for designing and implementing DQ assessments to evaluate repurposed data for a specific secondary use. DataGauge is composed of five steps: (1) Define information needs, (2) Develop a formal Data Needs Model (DNM), (3) Use the DNM and DQ theory to develop goal-specific DQ assessment requirements, (4) Extract DNM-specified data, and (5) Evaluate according to DQ requirements. DataGauge's main contribution is integrating general DQ theory and DQ assessment methods into a systematic process. This process supports the integration and practical implementation of existing Electronic Health Record-specific DQ assessment guidelines. DataGauge also provides an initial theory-based guidance framework that ties the DNM to DQ testing methods for each DQ dimension to aid the design of DQ assessments. This framework can be augmented with existing DQ guidelines to enable systematic assessment. DataGauge sets the stage for future systematic DQ assessment research by defining an assessment process, capable of adapting to a broad range of clinical datasets and secondary uses. Defining DataGauge sets the stage for new research directions such as DQ theory integration, DQ requirements portability research, DQ assessment tool development and DQ assessment tool usability.

User needs analysis and usability assessment of DataMed - a biomedical data discovery index.

OBJECTIVE: To present user needs and usability evaluations of DataMed, a Data Discovery Index (DDI) that allows searching for biomedical data from multiple sources. MATERIALS AND METHODS: We conducted 2 phases of user studies. Phase 1 was a user needs analysis conducted before the development of DataMed, consisting of interviews with researchers. Phase 2 involved iterative usability evaluations of DataMed prototypes. We analyzed data qualitatively to document researchers' information and user interface needs. RESULTS: Biomedical researchers' information needs in data discovery are complex, multidimensional, and shaped by their context, domain knowledge, and technical experience. User needs analyses validate the need for a DDI, while usability evaluations of DataMed show that even though aggregating metadata into a common search engine and applying traditional information retrieval tools are promising first steps, there remain challenges for DataMed due to incomplete metadata and the complexity of data discovery. DISCUSSION: Biomedical data poses distinct problems for search when compared to websites or publications. Making data available is not enough to facilitate biomedical data discovery: new retrieval techniques and user interfaces are necessary for dataset exploration. Consistent, complete, and high-quality metadata are vital to enable this process. CONCLUSION: While available data and researchers' information needs are complex and heterogeneous, a successful DDI must meet those needs and fit into the processes of biomedical researchers. Research directions include formalizing researchers' information needs, standardizing overviews of data to facilitate relevance judgments, implementing user interfaces for concept-based searching, and developing evaluation methods for open-ended discovery systems such as DDIs.

Rapid and Reusable Text Visualization and Exploration Development with DELVE.

We present DELVE (Document ExpLoration and Visualization Engine), a framework for developing interactive visualizations as modular Web-applications to assist researchers with exploratory literature search. The goal for web-applications driven by DELVE is to better satisfy the information needs of researchers and to help explore and understand the state of research in scientific liter ature by providing immersive visualizations that both contain facets and are driven by facets derived from the literature. We base our framework on principles from user-centered design and human-computer interaction (HCI). Preliminary evaluations demon strate the usefulness of DELVE's techniques: (1) a clinical researcher immediately saw that her original query was inappropriate simply due to the frequencies displayed via generalized clouds and (2) a muscle biologist quickly learned of vocabulary differences found between two disciplines that were referencing the same idea, which we feel is critical for interdisciplinary work. We dis cuss the underlying category-theoretic model of our framework and show that it naturally encourages the development of reusable visualizations by emphasizing interoperability.

Dashboard visualizations: Supporting real-time throughput decision-making.

Providing timely and effective care in the emergency department (ED) requires the management of individual patients as well as the flow and demands of the entire department. Strategic changes to work processes, such as adding a flow coordination nurse or a physician in triage, have demonstrated improvements in throughput times. However, such global strategic changes do not address the real-time, often opportunistic workflow decisions of individual clinicians in the ED. We believe that real-time representation of the status of the entire emergency department and each patient within it through information visualizations will better support clinical decision-making in-the-moment and provide for rapid intervention to improve ED flow. This notion is based on previous work where we found that clinicians' workflow decisions were often based on an in-the-moment local perspective, rather than a global perspective. Here, we discuss the challenges of designing and implementing visualizations for ED through a discussion of the development of our prototype Throughput Dashboard and the potential it holds for supporting real-time decision-making.

Improving the utility of MeSH® terms using the TopicalMeSH representation.

OBJECTIVE: To evaluate whether vector representations encoding latent topic proportions that capture similarities to MeSH terms can improve performance on biomedical document retrieval and classification tasks, compared to using MeSH terms. MATERIALS AND METHODS: We developed the TopicalMeSH representation, which exploits the 'correspondence' between topics generated using latent Dirichlet allocation (LDA) and MeSH terms to create new document representations that combine MeSH terms and latent topic vectors. We used 15 systematic drug review corpora to evaluate performance on information retrieval and classification tasks using this TopicalMeSH representation, compared to using standard encodings that rely on either (1) the original MeSH terms, (2) the text, or (3) their combination. For the document retrieval task, we compared the precision and recall achieved by ranking citations using MeSH and TopicalMeSH representations, respectively. For the classification task, we considered three supervised machine learning approaches, Support Vector Machines (SVMs), logistic regression, and decision trees. We used these to classify documents as relevant or irrelevant using (independently) MeSH, TopicalMeSH, Words (i.e., n-grams extracted from citation titles and abstracts, encoded via bag-of-words representation), a combination of MeSH and Words, and a combination of TopicalMeSH and Words. We also used SVM to compare the classification performance of tf-idf weighted MeSH terms, LDA Topics, a combination of Topics and MeSH, and TopicalMeSH to supervised LDA's classification performance. RESULTS: For the document retrieval task, using the TopicalMeSH representation resulted in higher precision than MeSH in 11 of 15 corpora while achieving the same recall. For the classification task, use of TopicalMeSH features realized a higher F1 score in 14 of 15 corpora when used by SVMs, 12 of 15 corpora using logistic regression, and 12 of 15 corpora using decision trees. TopicalMeSH also had better document classification performance on 12 of 15 corpora when compared to Topics, tf-idf weighted MeSH terms, and a combination of Topics and MeSH using SVMs. Supervised LDA achieved the worst performance in most of the corpora. CONCLUSION: The proposed TopicalMeSH representation (which combines MeSH terms with latent topics) consistently improved performance on document retrieval and classification tasks, compared to using alternative standard representations using MeSH terms alone, as well as, several standard alternative approaches.

Factors Affecting Accuracy of Data Abstracted from Medical Records.

OBJECTIVE: Medical record abstraction (MRA) is often cited as a significant source of error in research data, yet MRA methodology has rarely been the subject of investigation. Lack of a common framework has hindered application of the extant literature in practice, and, until now, there were no evidence-based guidelines for ensuring data quality in MRA. We aimed to identify the factors affecting the accuracy of data abstracted from medical records and to generate a framework for data quality assurance and control in MRA. METHODS: Candidate factors were identified from published reports of MRA. Content validity of the top candidate factors was assessed via a four-round two-group Delphi process with expert abstractors with experience in clinical research, registries, and quality improvement. The resulting coded factors were categorized into a control theory-based framework of MRA. Coverage of the framework was evaluated using the recent published literature. RESULTS: Analysis of the identified articles yielded 292 unique factors that affect the accuracy of abstracted data. Delphi processes overall refuted three of the top factors identified from the literature based on importance and five based on reliability (six total factors refuted). Four new factors were identified by the Delphi. The generated framework demonstrated comprehensive coverage. Significant underreporting of MRA methodology in recent studies was discovered. CONCLUSION: The framework generated from this research provides a guide for planning data quality assurance and control for studies using MRA. The large number and variability of factors indicate that while prospective quality assurance likely increases the accuracy of abstracted data, monitoring the accuracy during the abstraction process is also required. Recent studies reporting research results based on MRA rarely reported data quality assurance or control measures, and even less frequently reported data quality metrics with research results. Given the demonstrated variability, these methods and measures should be reported with research results.

Using common table expressions to build a scalable Boolean query generator for clinical data warehouses.

We present a custom, Boolean query generator utilizing common-table expressions (CTEs) that is capable of scaling with big datasets. The generator maps user-defined Boolean queries, such as those interactively created in clinical-research and general-purpose healthcare tools, into SQL. We demonstrate the effectiveness of this generator by integrating our study into the Informatics for Integrating Biology and the Bedside (i2b2) query tool and show that it is capable of scaling. Our custom generator replaces and outperforms the default query generator found within the Clinical Research Chart cell of i2b2. In our experiments, 16 different types of i2b2 queries were identified by varying four constraints: date, frequency, exclusion criteria, and whether selected concepts occurred in the same encounter. We generated nontrivial, random Boolean queries based on these 16 types; the corresponding SQL queries produced by both generators were compared by execution times. The CTE-based solution significantly outperformed the default query generator and provided a much more consistent response time across all query types (M = 2.03, SD = 6.64 versus M = 75.82, SD = 238.88 s). Without costly hardware upgrades, we provide a scalable solution based on CTEs with very promising empirical results centered on performance gains. The evaluation methodology used for this provides a means of profiling clinical data warehouse performance.

Errors in after-hours phone consultations: a simulation study.

BACKGROUND: After-hours out-of-hospital phone consultations require physicians to make decisions based on information provided by a nurse over the phone. METHODS: We conducted a simulation study to evaluate physicians' actions following communication of key information. 22 nurses were asked to call physicians with six cases based on the six most common reasons for after-hours phone calls. We evaluated physicians' actions following the communication of key clinical information: A situation cue described a patient's problem (eg, confusion). A background cue described a specific clinical finding regarding the cause of the problem (eg, patient's sodium is low). For each cue we defined a list of indicators, based on the medical literature, to ascertain whether physicians acted upon the provided information (which was defined as addressing at least one of the indicators). RESULTS: A total of 108 phone consultations (containing 88 situation and 93 background cues) were analysed. Situation cues were communicated in 90% (79/88) of the calls and background cues in 33% (31/93). Physician acted upon the provided information in 57% (45/79) and 48% (15/31) of the communicated situation and background cues, respectively. When the background cues were not communicated, physicians asked questions expected to elicit the cue in 12% of the cases. Responding to the situation cue was associated with longer conversations and active inquiry by the physician. CONCLUSIONS: After-hours phone calls are error prone. Both nurse communication and physician decision-making are problematic. Efforts to improve patient safety in this setting must address both communication and decision-making.

Evaluation of a problem-specific SBAR tool to improve after-hours nurse-physician phone communication: a randomized trial.

BACKGROUND: After-hours telephone communications are common in patient management. Patterns of communication of key information during after-hours phone calls were evaluated, and the utility of problem-specific Situation, Background, Assessment, Recommendation (SBAR) forms in improving this communication was assessed. METHODS: In a randomized trial using a simulated on-call setting, 20 nurses called physicians regarding six cases adapted from inpatient records and based on the six most common reasons for after-hours nurse-physician communication. Three of the cases were handled without the SBAR forms (control cases), and three cases were handled with the forms (SBAR cases). Two cue types of communication were evaluated: situation cues, which conveyed the patient's situation (for example, a patient is confused), and background cues, which conveyed problem-specific data indicated on the SBAR forms (for example, the patient has a low sodium level). RESULTS: Ninety-two phone calls were analyzed (43 SBAR/49 controls). Most of the nurses reported the situation cues (SBAR 88%, control 84%, p = .60) but not the background cues. There was a trend toward fewer background cues communicated in the SBAR cases (14% versus 31%, p = .08). In 14% of the cases, on average, nurses omitted information or reported wrong information regarding the background cue. Physicians asked questions that resulted in the communication of the cues in a minority of the cases when the background cues were not originally provided by the nurses (SBAR 6%, control 16%, p = .39). CONCLUSIONS: In after-hours phone communication between physicians and nurses, significant information was often not communicated and physicians did not elicit the necessary information. Simply providing an SBAR-based form did not ensure complete communication of key information.

SYFSA: a framework for systematic yet flexible systems analysis.

Although technological or organizational systems that enforce systematic procedures and best practices can lead to improvements in quality, these systems must also be designed to allow users to adapt to the inherent uncertainty, complexity, and variations in healthcare. We present a framework, called Systematic Yet Flexible Systems Analysis (SYFSA) that supports the design and analysis of Systematic Yet Flexible (SYF) systems (whether organizational or technical) by formally considering the tradeoffs between systematicity and flexibility. SYFSA is based on analyzing a task using three related problem spaces: the idealized space, the natural space, and the system space. The idealized space represents the best practice-how the task is to be accomplished under ideal conditions. The natural space captures the task actions and constraints on how the task is currently done. The system space specifies how the task is done in a redesigned system, including how it may deviate from the idealized space, and how the system supports or enforces task constraints. The goal of the framework is to support the design of systems that allow graceful degradation from the idealized space to the natural space. We demonstrate the application of SYFSA for the analysis of a simplified central line insertion task. We also describe several information-theoretic measures of flexibility that can be used to compare alternative designs, and to measure how efficiently a system supports a given task, the relative cognitive workload, and learnability.

Phrase Based Topic Modeling for Semantic Information Processing in Biomedicine.

Given that unstructured data is increasing exponentially everyday, extracting and understanding the information, themes, and relationships from large collections of documents is increasingly important to researchers in many disciplines including biomedicine. Latent Dirichlet Allocation (LDA) is an unsupervised topic modeling technique based on the "bag-of-words" assumption that has been applied extensively to unveil hidden semantic themes within large sets of textual documents. Recently, it was extended using the "bag-of-n-grams" paradigm to account for word order. In this paper, we present an alternative phrase based LDA model to move from a bag of words or n-grams paradigm to a "bag-of-key-phrases" setting by applying a key phrase extraction technique, the C-value method, to further explore latent themes. We evaluate our approach by using a phrase intrusion user study and demonstrate that our model can help LDA generate better and more interpretable topics than those generated using the bag-of-n-grams approach. Given topic models essentially are statistical tools, an important problem in topic modeling is that of visualizing and interacting with the models to understand and extract new information from a collection. To evaluate our phrase based modeling approach in this context, we incorporate it in an open source interactive topic browser. Qualitative evaluations of this browser with biomedical experts demonstrate that our approach can aid biomedical researchers gain better and faster understanding of their document collections.

Twinlist: novel user interface designs for medication reconciliation.

Medication reconciliation is an important and complex task for which careful user interface design has the potential to help reduce errors and improve quality of care. In this paper we focus on the hospital discharge scenario and first describe a novel interface called Twinlist. Twinlist illustrates the novel use of spatial layout combined with multi-step animation, to help medical providers see what is different and what is similar between the lists (e.g., intake list and hospital list), and rapidly choose the drugs they want to include in the reconciled list. We then describe a series of variant designs and discuss their comparative advantages and disadvantages. Finally we report on a pilot study that suggests that animation might help users learn new spatial layouts such as the one used in Twinlist.

Cross-terminology mapping challenges: a demonstration using medication terminological systems.

Standardized terminological systems for biomedical information have provided considerable benefits to biomedical applications and research. However, practical use of this information often requires mapping across terminological systems-a complex and time-consuming process. This paper demonstrates the complexity and challenges of mapping across terminological systems in the context of medication information. It provides a review of medication terminological systems and their linkages, then describes a case study in which we mapped proprietary medication codes from an electronic health record to SNOMED CT and the UMLS Metathesaurus. The goal was to create a polyhierarchical classification system for querying an i2b2 clinical data warehouse. We found that three methods were required to accurately map the majority of actively prescribed medications. Only 62.5% of source medication codes could be mapped automatically. The remaining codes were mapped using a combination of semi-automated string comparison with expert selection, and a completely manual approach. Compound drugs were especially difficult to map: only 7.5% could be mapped using the automatic method. General challenges to mapping across terminological systems include (1) the availability of up-to-date information to assess the suitability of a given terminological system for a particular use case, and to assess the quality and completeness of cross-terminology links; (2) the difficulty of correctly using complex, rapidly evolving, modern terminologies; (3) the time and effort required to complete and evaluate the mapping; (4) the need to address differences in granularity between the source and target terminologies; and (5) the need to continuously update the mapping as terminological systems evolve.