Clinical Research Informatics for Big Data and Precision Medicine.

OBJECTIVES: To reflect on the notable events and significant developments in Clinical Research Informatics (CRI) in the year of 2015 and discuss near-term trends impacting CRI. METHODS: We selected key publications that highlight not only important recent advances in CRI but also notable events likely to have significant impact on CRI activities over the next few years or longer, and consulted the discussions in relevant scientific communities and an online living textbook for modern clinical trials. We also related the new concepts with old problems to improve the continuity of CRI research. RESULTS: The highlights in CRI in 2015 include the growing adoption of electronic health records (EHR), the rapid development of regional, national, and global clinical data research networks for using EHR data to integrate scalable clinical research with clinical care and generate robust medical evidence. Data quality, integration, and fusion, data access by researchers, study transparency, results reproducibility, and infrastructure sustainability are persistent challenges. CONCLUSION: The advances in Big Data Analytics and Internet technologies together with the engagement of citizens in sciences are shaping the global clinical research enterprise, which is getting more open and increasingly stakeholder-centered, where stakeholders include patients, clinicians, researchers, and sponsors.

Clinical research databases and clinical decision making in chronic diseases.

Chronic diseases are the major source of morbidity, mortality, and resource utilization. Large-scale longitudinal databases are rapidly proliferating in both single- and multi-institutional settings, providing clinical data on a broad range of patients who receive 'real world' management. Although bias and changing medical management may limit the types of questions that can be addressed using the data contained in longitudinal clinical databases, many initial hypotheses can be generated from the data. Because chronic diseases persist over long periods of time, understanding the impact of temporal relationships, and of concurrent clinical events and contexts is critical to meaningful interpretation of clinical data. Adapting techniques initially developed for the physical sciences and for statistical process control can produce visual displays of clinical data that capture complex temporal and contextual information. With these tools, investigators can quickly explore vast quantities of clinical data, and discover new temporal relationships and emerging trends.

Nosocomial penicillin-resistant pneumococcal infections at a Midwestern university hospital.

We analyzed 333 isolates of Streptococcus pneumoniae from adult inpatients over a continuous 30-month period. Twenty-five percent of isolates were resistant to penicillin, and 18% of tested strains were not susceptible to ceftriaxone. For 29 (15%) of 199 evaluable patients, S pneumoniae was isolated from a culture first obtained >48 hours after admission, and chart review satisfied our case definition for nosocomial infection. S pneumoniae isolates were penicillin-resistant in 14 of 29 nosocomial and 34 of 170 community-acquired cases (odds ratio, 3.73; 95% confidence interval, 1.53-9.15; P=.01).

Combining laboratory data sets from multiple institutions using the logical observation identifier names and codes (LOINC).

A standard set of names and codes for laboratory test results is critical for any endeavor requiring automated data pooling, including multi-institutional research and cross-facility patient care. This need has led to the development of the logical observation identifier names and codes (LOINC) database and its test-naming convention. This study is an expansion of a pilot study using LOINC to exchange laboratory data between Columbia University Medical Center in New York and Barnes Hospital at Washington University in St. Louis, where we described complexities and ambiguities that arose in the LOINC coding process (D.M. Baorto, J.J. Cimino, C.A. Parvin, M.G. Kahn, Proc. Am. Med. Inf. Assoc. 1997). For the present study, we required the same two medical centers to again extract raw laboratory data from their local information system for a defined patient population, translate tests into LOINC and provide aggregate data which could then be used to compare laboratory utilization. Here we examine a larger number of tests from each site which have been recoded using an updated version of the LOINC database. We conclude that the coding of local tests into LOINC can often be complex, especially the 'Kind of Property' field and apparently trivial differences in choices made by individual institutions can result in nonmatches in electronically pooled data. In the present study, 75% of failures to match the same tests between different institutions using LOINC codes were due to differences in local coding choices. LOINC has the potential to eliminate the need for detailed human inspection during the pooling of laboratory data from diverse sites and perhaps even a built-in capability to adjust matching stringency by selecting subsets of LOINC fields required to match. However, a quality standard coding procedure is required and examples highlighted in this paper may require special attention while mapping to LOINC.

Three perspectives on integrated clinical databases.

The dramatic transformation of health care organizations from independent local entities into regional and national integrated health care delivery enterprises has forced a reevaluation of the role of information systems. Until recently, nearly all clinical information systems were acquired to support financial and administrative services within single facilities. When independent facilities merge to form an integrated health system (IHS), they find that their unique computer systems do not allow for the sharing or combining of clinical data. However, new technologies are beginning to enable patient-specific data scattered among many information systems in many different hospitals and ambulatory care settings to be unified into a single database called a clinical data repository. The hope is that comprehensive electronic clinical records can enable IHSs to meet their goals of improving the quality and reducing the cost of health care. As the number of technological impediments to forming integrated clinical information systems rapidly decreases, nontechnologic issues surrounding the sharing of patient information become more prominent. The author focuses on the clinician's, the administrator's, and the patient's unique perspectives on the benefits and possible problems associated with clinical data repositories. He then describes how one IHS, together with an academic medical institution, has begun to grapple with many of these concerns through an effort called Project Spectrum. The goal of the project is the successful implementation of a comprehensive clinical data repository.

Automated system for identifying potential dosage problems at a large university hospital.

A hospital's experience with an automated system for screening drug orders for potential dosage problems is described. DoseChecker was developed by the hospital pharmacy department in collaboration with a local university. Pharmacy, laboratory, and patient demographic data are transferred nightly from the hospital's mainframe system to a database server; DoseChecker uses these data and user-defined rules to (1) identify patients receiving any of 35 targeted medications, (2) evaluate the appropriateness of current dosages, and (3) generate alerts for patients potentially needing dosage adjustments. The alert reports are distributed to satellite pharmacists, who evaluate each patient's condition and make recommendations to physicians as needed. One of the system's primary purposes is to calculate creatinine clearance and verify that dosages are properly adjusted for renal function. Between May and October 1995, the system electronically screened 28,528 drug orders and detected potential dosage problems in 2859 (10%). The system recommended a lower daily dose in 1992 cases (70%) and a higher daily dose in 867 (30%). Pharmacists contacted physicians concerning 1163 (41%) of the 2859 alerts; in 868 cases (75%), the physicians agreed to adjust the dosage. The most common dosage problem identified was failure to adjust dosages on the basis of declining renal function. An automated system provided an efficient method of identifying inappropriate dosages at a large university hospital.

Iterative usability testing: ensuring a usable clinical workstation.

Once the users' needs are determined, how does one ensure that the resulting software meets the users' needs? This paper describes our application of a process, usability testing, that is used to measure the usability of systems as well as guide modifications to address usability problems. Usability testing is not a method to elicit opinions about software, but rather a method to determine scientifically a product's level of usability. Our application of usability testing is designed to determine the current usability level of a workstation designed for the clinician's use, determine specific problems with the Clinical Workstation's usability, and then evaluate the effectiveness of changes that address those problems.

Using Logical Observation Identifier Names and Codes (LOINC) to exchange laboratory data among three academic hospitals.

Using a standard set of names and codes to exchange electronic laboratory data would facilitate multiinstitutional research and data pooling. This need has led to the development of the Logical Observation Identifier Names and Codes (LOINC) database and its test naming convention. We conducted a study which required 3 academic hospitals (in 2 separate medical centers) to extract raw laboratory data from their local information system for a defined patient population, translate tests into LOINC, and provide aggregate data which could then be used to compare laboratory utilization. We found that the coding of local tests into LOINC can often be complex, especially the "Kind of Property" field, and apparently trivial differences in choices made by individual institutions can result in nonmatches in electronically pooled data. In our study, 72-86% of the failures of LOINC to match the same tests between different institutions were due to differences in local coding choices. LOINC has tremendous potential to eliminate the needing for detailed human inspection during the pooling of laboratory data from diverse sites, and perhaps even a built-in capability to adjust matching stringency by selecting subsets of LOINC fields required to match. However, a quality, standard coding procedure at all sites is critical.

Statistical process control methods for expert system performance monitoring.

The literature on the performance evaluation of medical expert system is extensive, yet most of the techniques used in the early stages of system development are inappropriate for deployed expert systems. Because extensive clinical and informatics expertise and resources are required to perform evaluations, efficient yet effective methods of monitoring performance during the long-term maintenance phase of the expert system life cycle must be devised. Statistical process control techniques provide a well-established methodology that can be used to define policies and procedures for continuous, concurrent performance evaluation. Although the field of statistical process control has been developed for monitoring industrial processes, its tools, techniques, and theory are easily transferred to the evaluation of expert systems. Statistical process tools provide convenient visual methods and heuristic guidelines for detecting meaningful changes in expert system performance. The underlying statistical theory provides estimates of the detection capabilities of alternative evaluation strategies. This paper describes a set of statistical process control tools that can be used to monitor the performance of a number of deployed medical expert systems. It describes how p-charts are used in practice to monitor the GermWatcher expert system. The case volume and error rate of GermWatcher are then used to demonstrate how different inspection strategies would perform.

Monitoring expert system performance using continuous user feedback.

OBJECTIVE: To evaluate the applicability of metrics collected during routine use to monitor the performance of a deployed expert system. METHODS: Two extensive formal evaluations of the GermWatcher (Washington University School of Medicine) expert system were performed approximately six months apart. Deficiencies noted during the first evaluation were corrected via a series of interim changes to the expert system rules, even though the expert system was in routine use. As part of their daily work routine, infection control nurses reviewed expert system output and changed the output results with which they disagreed. The rate of nurse disagreement with expert system output was used as an indirect or surrogate metric of expert system performance between formal evaluations. The results of the second evaluation were used to validate the disagreement rate as an indirect performance measure. Based on continued monitoring of user feedback, expert system changes incorporated after the second formal evaluation have resulted in additional improvements in performance. RESULTS: The rate of nurse disagreement with GermWatcher output decreased consistently after each change to the program. The second formal evaluation confirmed a marked improvement in the program's performance, justifying the use of the nurses' disagreement rate as an indirect performance metric. CONCLUSIONS: Metrics collected during the routine use of the GermWatcher expert system can be used to monitor the performance of the expert system. The impact of improvements to the program can be followed using continuous user feedback without requiring extensive formal evaluations after each modification. When possible, the design of an expert system should incorporate measures of system performance that can be collected and monitored during the routine use of the system.

Evaluation of an objective plan-evaluation model in the three dimensional treatment of nonsmall cell lung cancer.

PURPOSE: Evaluation of three dimensional (3D) radiotherapy plans is difficult because it requires the review of vast amounts of data. Selecting the optimal plan from a set of competing plans involves making trade-offs among the doses delivered to the target volumes and normal tissues. The purpose of this study was to test an objective plan-evaluation model and evaluate its clinical usefulness in 3D treatment planning for nonsmall cell lung cancer. METHODS AND MATERIALS: Twenty patients with inoperable nonsmall cell lung cancer treated with definitive radiotherapy were studied using full 3D techniques for treatment design and implementation. For each patient, the evaluator (the treating radiation oncologist) initially ranked three plans using room-view dose-surface displays and dose-volume histograms, and identified the issues that needed to be improved. The three plans were then ranked by the objective plan-evaluation model. A figure of merit (FOM) was computed for each plan by combining the numerical score (utility in decision-theoretic terms) for each clinical issue. The utility was computed from a probability of occurrence of the issue and a physician-specific weight indicating its clinical relevance. The FOM was used to rank the competing plans for a patient, and the utility was used to identify issues that needed to be improved. These were compared with the initial evaluations of the physician and discrepancies were analyzed. The issues identified in the best treatment plan were then used to attempt further manual optimization of this plan. RESULTS: For the 20 patients (60 plans) in the study, the final plan ranking produced by the plan-evaluation model had an initial 73% agreement with the ranking provided by the evaluator. After discrepant cases were reviewed by the physician, the model was usually judged more objective or "correct." In most cases the model was also able to correctly identify the issues that needed improvement in each plan. Subsequent replanning confirmed that further manual plan optimization could be achieved in 17 patients. CONCLUSION: The objective plan-evaluation model was able to rank lung cancer radiotherapy plans from best to worst. It was useful in improving plans and may be useful to physicians in defining goals for patients based on the ability to effectively and safely treat their tumors.

The computer-based patient record and Robert Fulghum's 16 principles.

Robert Fulghum proposed 16 principles learned in kindergarten that he claimed should govern all human activities. How do these principles apply to current efforts to create a computer-based patient record? Fulghum's principles, when recast for the computer-based patient record, emphasize cooperation, privacy, security, and standardization--elements that are easily recognized as essential features of an electronic medical record. However, a number of less frequently discussed issues, such as data quality, integrity, abuse, and misuse can also be considered in terms of Fulghum's principles.

Using knowledge maintenance for preference assessment.

Most real-life decisions require the decision maker to make trade-offs in order to fulfill multiple conflicting objectives. This is especially true in medical decision making while selecting the optimal therapy plan from among competing therapy plans for a patient. Multi-attribute utility theory provides a framework to specify these trade-offs for optimal decision making based on the preferences of the decision maker. However traditional preference-assessment techniques are difficult to implement and rarely elicit the true preferences of the decision maker. We describe a new preference-assessment method based on the concept of knowledge maintenance where the preference model is changed each time it makes an incorrect recommendation. The method is implemented in a decision-theoretic system to evaluate competing three-dimensional radiation treatment plans. The preference-assessment method leads to preference models which perform better than preference models elicited using traditional assessment techniques.

Extending a clinical repository to include multiple sites.

With the consolidation of health care organizations and services, a clinical repository comprising data from a single site is no longer sufficient. Individual patient data are now spread across multiple sites comprising a single enterprise. Users require an integrated view, or at least a common view, of these clinical data across multiple sites. Many issues arise when one tries to merge data from multiple, distinct organizations into an existing schema. We have addressed these issues while extending our clinical repository for Barnes Hospital with data from Jewish Hospital, both of which are members of the recently formed BJC Health System. We describe the architecture of our existing repository, approaches and issues in extending this repository to include multiple sites, and the specific issues we addressed in our system.

Creating temporal abstractions in three clinical information systems.

Modern clinical information system developers recognize the need to associate temporal information with clinical data. However, specific clinical systems capture different temporal features using a variety of data modeling techniques. Two commonly used methods to represent temporal information are point-based events and interval-based durations. We recently implemented a rule-based expert system for drug dose monitoring on three clinical information systems. The expert system requires both static drug dosing information (drug name, amount, route, frequency) and temporal dosing information (duration of therapy, renewals, restarts). Our design goal was to use the same expert system code on all three information systems by defining a common database schema to hide differences in the original systems' data models. Although we have been successful in mapping clinical data from these three source systems into a unified temporal data representation, we describe how differences in handling time within the three clinical systems made this goal difficult to achieve.

Contextual inquiry: discovering physicians' true needs.

Gathering user requirements that represent the true needs of the users is a challenge. There are many elicitation methods in use today, but they generally are not successful in identifying a comprehensive set of requirements that reflect the users' true needs. This paper describes the requirements gathering method, Contextual Inquiry, that we used to generate physician requirements for a comprehensive Clinical Information System. We feel that this method has advantages over traditional techniques such as surveys, questionnaires, traditional interviews, and focus groups, in obtaining a more comprehensive analysis of the true needs of the users.

Supporting ad-hoc queries in an integrated clinical database.

Whether caring for patients or conducting research, medical decision-makers need access to clinical data. To fulfill that need, commercial software developers have produced a wide range of database query tools that differ greatly in functionality and cost. Generally, tools that have a greater ability to conceal database complexity from the user also require more effort for administrative setup. We describe a cost-effective, commercially-available query tool that requires no special setup to perform most simple queries, yet can be customized to satisfy users' more complex querying requirements.