A general framework for developing computable clinical phenotype algorithms.

OBJECTIVE: Present a general framework providing high-level guidance to developers of computable algorithms for identifying patients with specific clinical conditions (phenotypes) through a variety of approaches, including but not limited to machine learning and natural language processing methods to incorporate rich electronic health record data. MATERIALS/METHODS: Drawing on extensive prior phenotyping experiences and insights derived from three algorithm development projects conducted specifically for this purpose, our team with expertise in clinical medicine, statistics, informatics, pharmacoepidemiology, and healthcare data science methods conceptualized stages of development and corresponding sets of principles, strategies, and practical guidelines for improving the algorithm development process. RESULTS: We propose five stages of algorithm development and corresponding principles, strategies, and guidelines: 1) assessing fitness-for-purpose, 2) creating gold standard data, 3) feature engineering, 4) model development, and 5) model evaluation. DISCUSSION/CONCLUSION: This framework is intended to provide practical guidance and serve as a basis for future elaboration and extension.

Improving Methods of Identifying Anaphylaxis for Medical Product Safety Surveillance Using Natural Language Processing and Machine Learning.

We sought to determine whether machine learning and natural language processing (NLP) applied to electronic medical records could improve performance of automated health-care claims-based algorithms to identify anaphylaxis events using data on 516 patients with outpatient, emergency department, or inpatient anaphylaxis diagnosis codes during 2015-2019 in 2 integrated health-care institutions in the Northwest United States. We used one site's manually reviewed gold-standard outcomes data for model development and the other's for external validation based on cross-validated area under the receiver operating characteristic curve (AUC), positive predictive value (PPV), and sensitivity. In the development site 154 (64%) of 239 potential events met adjudication criteria for anaphylaxis compared with 180 (65%) of 277 in the validation site. Logistic regression models using only structured claims data achieved a cross-validated AUC of 0.58 (95% CI: 0.54, 0.63). Machine learning improved cross-validated AUC to 0.62 (0.58, 0.66); incorporating NLP-derived covariates further increased cross-validated AUCs to 0.70 (0.66, 0.75) in development and 0.67 (0.63, 0.71) in external validation data. A classification threshold with cross-validated PPV of 79% and cross-validated sensitivity of 66% in development data had cross-validated PPV of 78% and cross-validated sensitivity of 56% in external data. Machine learning and NLP-derived data improved identification of validated anaphylaxis events.

Validation of Acute Pancreatitis Among Adults in an Integrated Healthcare System.

BACKGROUND: Acute pancreatitis is a serious gastrointestinal disease that is an important target for drug safety surveillance. Little is known about the accuracy of ICD-10 codes for acute pancreatitis in the United States, or their performance in specific clinical settings. We conducted a validation study to assess the accuracy of acute pancreatitis ICD-10 diagnosis codes in inpatient, emergency department (ED), and outpatient settings. METHODS: We reviewed electronic medical records for encounters with acute pancreatitis diagnosis codes in an integrated healthcare system from October 2015 to December 2019. Trained abstractors and physician adjudicators determined whether events met criteria for acute pancreatitis. RESULTS: Out of 1,844 eligible events, we randomly sampled 300 for review. Across all clinical settings, 182 events met validation criteria for an overall positive predictive value (PPV) of 61% (95% confidence intervals [CI] = 55, 66). The PPV was 87% (95% CI = 79, 92%) for inpatient codes, but only 45% for ED (95% CI = 35, 54%) and outpatient (95% CI = 34, 55%) codes. ED and outpatient encounters accounted for 43% of validated events. Acute pancreatitis codes from any encounter type with lipase >3 times the upper limit of normal had a PPV of 92% (95% CI = 86, 95%) and identified 85% of validated events (95% CI = 79, 89%), while codes with lipase <3 times the upper limit of normal had a PPV of only 22% (95% CI = 16, 30%). CONCLUSIONS: These results suggest that ICD-10 codes accurately identified acute pancreatitis in the inpatient setting, but not in the ED and outpatient settings. Laboratory data substantially improved algorithm performance.

Measuring problem prescription opioid use among patients receiving long-term opioid analgesic treatment: development and evaluation of an algorithm for use in EHR and claims data.

Objective: Opioid surveillance in response to the opioid epidemic will benefit from scalable, automated algorithms for identifying patients with clinically documented signs of problem prescription opioid use. Existing algorithms lack accuracy. We sought to develop a high-sensitivity, high-specificity classification algorithm based on widely available structured health data to identify patients receiving chronic extended-release/long-acting (ER/LA) therapy with evidence of problem use to support subsequent epidemiologic investigations. Methods: Outpatient medical records of a probability sample of 2,000 Kaiser Permanente Washington patients receiving ≥60 days' supply of ER/LA opioids in a 90-day period from 1 January 2006 to 30 June 2015 were manually reviewed to determine the presence of clinically documented signs of problem use and used as a reference standard for algorithm development. Using 1,400 patients as training data, we constructed candidate predictors from demographic, enrollment, encounter, diagnosis, procedure, and medication data extracted from medical claims records or the equivalent from electronic health record (EHR) systems, and we used adaptive least absolute shrinkage and selection operator (LASSO) regression to develop a model. We evaluated this model in a comparable 600-patient validation set. We compared this model to ICD-9 diagnostic codes for opioid abuse, dependence, and poisoning. This study was registered with ClinicalTrials.gov as study NCT02667262 on 28 January 2016. Results: We operationalized 1,126 potential predictors characterizing patient demographics, procedures, diagnoses, timing, dose, and location of medication dispensing. The final model incorporating 53 predictors had a sensitivity of 0.582 at positive predictive value (PPV) of 0.572. ICD-9 codes for opioid abuse, dependence, and poisoning had a sensitivity of 0.390 at PPV of 0.599 in the same cohort. Conclusions: Scalable methods using widely available structured EHR/claims data to accurately identify problem opioid use among patients receiving long-term ER/LA therapy were unsuccessful. This approach may be useful for identifying patients needing clinical evaluation.

Using natural language processing of clinical text to enhance identification of opioid-related overdoses in electronic health records data.

PURPOSE: To enhance automated methods for accurately identifying opioid-related overdoses and classifying types of overdose using electronic health record (EHR) databases. METHODS: We developed a natural language processing (NLP) software application to code clinical text documentation of overdose, including identification of intention for self-harm, substances involved, substance abuse, and error in medication usage. Using datasets balanced with cases of suspected overdose and records of individuals at elevated risk for overdose, we developed and validated the application using Kaiser Permanente Northwest data, then tested portability of the application using Kaiser Permanente Washington data. Datasets were chart-reviewed to provide a gold standard for comparison and evaluation of the automated method. RESULTS: The method performed well in identifying overdose (sensitivity = 0.80, specificity = 0.93), intentional overdose (sensitivity = 0.81, specificity = 0.98), and involvement of opioids (excluding heroin, sensitivity = 0.72, specificity = 0.96) and heroin (sensitivity = 0.84, specificity = 1.0). The method performed poorly at identifying adverse drug reactions and overdose due to patient error and fairly at identifying substance abuse in opioid-related unintentional overdose (sensitivity = 0.67, specificity = 0.96). Evaluation using validation datasets yielded significant reductions, in specificity and negative predictive values only, for many classifications mentioned above. However, these measures remained above 0.80, thus, performance observed during development was largely maintained during validation. Similar results were obtained when evaluating portability, although there was a significant reduction in sensitivity for unintentional overdose that was attributed to missing text clinical notes in the database. CONCLUSIONS: Methods that process text clinical notes show promise for improving accuracy and fidelity at identifying and classifying overdoses according to type using EHR data.

Development of an algorithm to identify inpatient opioid-related overdoses and oversedation using electronic data.

PURPOSE: To facilitate surveillance and evaluate interventions addressing opioid-related overdoses, algorithms are needed for use in large health care databases to identify and differentiate community-occurring opioid-related overdoses from inpatient-occurring opioid-related overdose/oversedation. METHODS: Data were from Kaiser Permanente Northwest (KPNW), a large integrated health plan. We iteratively developed and evaluated an algorithm for electronically identifying inpatient overdose/oversedation in KPNW hospitals from 1 January 2008 to 31 December 2014. Chart audits assessed accuracy; data sources included administrative and clinical records. RESULTS: The best-performing algorithm used these rules: (1) Include events with opioids administered in an inpatient setting (including emergency department/urgent care) followed by naloxone administration within 275 hours of continuous inpatient stay; (2) exclude events with electroconvulsive therapy procedure codes; and (3) exclude events in which an opioid was administered prior to hospital discharge and followed by readmission with subsequent naloxone administration. Using this algorithm, we identified 870 suspect inpatient overdose/oversedation events and chart audited a random sample of 235. Of the random sample, 185 (78.7%) were deemed overdoses/oversedation, 37 (15.5%) were not, and 13 (5.5%) were possible cases. The number of hours between time of opioid and naloxone administration did not affect algorithm accuracy. When "possible" overdoses/oversedations were included with confirmed events, overall positive predictive value (PPV) was very good (PPV = 84.0%). Additionally, PPV was reasonable when evaluated specifically for hospital stays with emergency/urgent care admissions (PPV = 77.0%) and excellent for elective surgery admissions (PPV = 97.0%). CONCLUSIONS: Algorithm performance was reasonable for identifying inpatient overdose/oversedation with best performance among elective surgery patients.

Identifying and classifying opioid-related overdoses: A validation study.

PURPOSE: The study aims to develop and validate algorithms to identify and classify opioid overdoses using claims and other coded data, and clinical text extracted from electronic health records using natural language processing (NLP). METHODS: Primary data were derived from Kaiser Permanente Northwest (2008-2014), an integrated health care system (~n > 475 000 unique individuals per year). Data included International Classification of Diseases, Ninth Revision (ICD-9) codes for nonfatal diagnoses, International Classification of Diseases, Tenth Revision (ICD-10) codes for fatal events, clinical notes, and prescription medication records. We assessed sensitivity, specificity, positive predictive value, and negative predictive value for algorithms relative to medical chart review and conducted assessments of algorithm portability in Kaiser Permanente Washington, Tennessee State Medicaid, and Optum. RESULTS: Code-based algorithm performance was excellent for opioid-related overdoses (sensitivity = 97.2%, specificity = 84.6%) and classification of heroin-involved overdoses (sensitivity = 91.8%, specificity = 99.0%). Performance was acceptable for code-based suicide/suicide attempt classifications (sensitivity = 70.7%, specificity = 90.5%); sensitivity improved with NLP (sensitivity = 78.7%, specificity = 91.0%). Performance was acceptable for the code-based substance abuse-involved classification (sensitivity = 75.3%, specificity = 79.5%); sensitivity improved with the NLP-enhanced algorithm (sensitivity = 80.5%, specificity = 76.3%). The opioid-related overdose algorithm performed well across portability assessment sites, with sensitivity greater than 96% and specificity greater than 84%. Cross-site sensitivity for heroin-involved overdose was greater than 87%, specificity greater than or equal to 99%. CONCLUSIONS: Code-based algorithms developed to detect opioid-related overdoses and classify them according to heroin involvement perform well. Algorithms for classifying suicides/attempts and abuse-related opioid overdoses perform adequately for use for research, particularly given the complexity of classifying such overdoses. The NLP-enhanced algorithms for suicides/suicide attempts and abuse-related overdoses perform significantly better than code-based algorithms and are appropriate for use in settings that have data and capacity to use NLP.

Long-Term Outcomes From Repeated Smoking Cessation Assistance in Routine Primary Care.

PURPOSE: To test the association between repeated clinical smoking cessation support and long-term cessation. DESIGN: Retrospective, observational cohort study using structured and free-text data from electronic health records. SETTING: Six diverse health systems in the United States. PARTICIPANTS: Patients aged ≥18 years who were smokers in 2007 and had ≥1 primary care visit in each of the following 4 years (N = 33 691). MEASURES: Primary exposure was a composite categorical variable (comprised of documentation of smoking cessation medication, counseling, or referral) classifying the proportions of visits for which patients received any cessation assistance (<25% (reference), 25%-49%, 50%-74%, and ≥75% of visits). The dependent variable was long-term quit (LTQ; yes/no), defined as no indication of being a current smoker for ≥365 days following a visit where nonsmoker or former smoker was indicated. ANALYSIS: Mixed effects logistic regression analysis adjusted for age, sex, race, and comorbidities, with robust standard error estimation to account for within site correlation. RESULTS: Overall, 20% of the cohort achieved LTQ status. Patients with ≥75% of visits with any assistance had almost 3 times the odds of achieving LTQ status compared to those with <25% visits with assistance (odds ratio = 2.84; 95% confidence interval: 1.50-5.37). Results were similar for specific assistance types. CONCLUSIONS: These findings provide support for the importance of repeated assistance at primary care visits to increase long-term smoking cessation.

Smoking-Cessation Assistance: Before and After Stage 1 Meaningful Use Implementation.

INTRODUCTION: Brief smoking-cessation interventions in primary care settings are effective, but delivery of these services remains low. The Centers for Medicare and Medicaid Services' Meaningful Use (MU) of Electronic Health Record (EHR) Incentive Program could increase rates of smoking assessment and cessation assistance among vulnerable populations. This study examined whether smoking status assessment, cessation assistance, and odds of being a current smoker changed after Stage 1 MU implementation. METHODS: EHR data were extracted from 26 community health centers with an EHR in place by June 15, 2009. AORs were computed for each binary outcome (smoking status assessment, counseling given, smoking-cessation medications ordered/discussed, current smoking status), comparing 2010 (pre-MU), 2012 (MU preparation), and 2014 (MU fully implemented) for pregnant and non-pregnant patients. RESULTS: Non-pregnant patients had decreased odds of current smoking over time; odds for all other outcomes increased except for medication orders from 2010 to 2012. Among pregnant patients, odds of assessment and counseling increased across all years. Odds of discussing or ordering of cessation medications increased from 2010 compared with the other 2 study years; however, medication orders alone did not change over time, and current smoking only decreased from 2010 to 2012. Compared with non-pregnant patients, a lower percentage of pregnant patients were provided counseling. CONCLUSIONS: Findings suggest that incentives for MU of EHRs increase the odds of smoking assessment and cessation assistance, which could lead to decreased smoking rates among vulnerable populations. Continued efforts for provision of cessation assistance among pregnant patients is warranted.

Assessing Trends in Tobacco Cessation in Diverse Patient Populations.

INTRODUCTION: This study examined change in tobacco use over 4 years among the general population of patients in six diverse health care organizations using electronic medical record data. METHODS: The study cohort (N = 34 393) included all patients age 18 years or older who were identified as smokers in 2007, and who then had at least one primary care visit in each of the following 4 years. RESULTS: In the 4 years following 2007, this patient cohort had a median of 13 primary care visits, and 38.6% of the patients quit smoking at least once. At the end of the fourth follow-up year, 15.4% had stopped smoking for 1 year or more. Smokers were more likely to become long-term quitters if they were 65 or older (OR = 1.32, 95% CI = [1.16, 1.49]), or had a diagnoses of cancer (1.26 [1.12, 1.41]), cardiovascular disease (1.22 [1.09, 1.37]), asthma (1.15 [1.06, 1.25]), or diabetes (1.17 [1.09, 1.27]). Characteristics associated with lower likelihood of becoming a long-term quitter were female gender (0.90 [0.84, 0.95]), black race (0.84 [0.75, 0.94]) and those identified as non-Hispanic (0.50 [0.43, 0.59]). CONCLUSIONS: Among smokers who regularly used these care systems, one in seven had achieved long-term cessation after 4 years. This study shows the practicality of using electronic medical records for monitoring patient smoking status over time. Similar methods could be used to assess tobacco use in any health care organization to evaluate the impact of environmental and organizational programs.

CER Hub: An informatics platform for conducting comparative effectiveness research using multi-institutional, heterogeneous, electronic clinical data.

OBJECTIVES: Comparative effectiveness research (CER) requires the capture and analysis of data from disparate sources, often from a variety of institutions with diverse electronic health record (EHR) implementations. In this paper we describe the CER Hub, a web-based informatics platform for developing and conducting research studies that combine comprehensive electronic clinical data from multiple health care organizations. METHODS: The CER Hub platform implements a data processing pipeline that employs informatics standards for data representation and web-based tools for developing study-specific data processing applications, providing standardized access to the patient-centric electronic health record (EHR) across organizations. RESULTS: The CER Hub is being used to conduct two CER studies utilizing data from six geographically distributed and demographically diverse health systems. These foundational studies address the effectiveness of medications for controlling asthma and the effectiveness of smoking cessation services delivered in primary care. DISCUSSION: The CER Hub includes four key capabilities: the ability to process and analyze both free-text and coded clinical data in the EHR; a data processing environment supported by distributed data and study governance processes; a clinical data-interchange format for facilitating standardized extraction of clinical data from EHRs; and a library of shareable clinical data processing applications. CONCLUSION: CER requires coordinated and scalable methods for extracting, aggregating, and analyzing complex, multi-institutional clinical data. By offering a range of informatics tools integrated into a framework for conducting studies using EHR data, the CER Hub provides a solution to the challenges of multi-institutional research using electronic medical record data.

Using the CER Hub to ensure data quality in a multi-institution smoking cessation study.

Comparative effectiveness research (CER) studies involving multiple institutions with diverse electronic health records (EHRs) depend on high quality data. To ensure uniformity of data derived from different EHR systems and implementations, the CER Hub informatics platform developed a quality assurance (QA) process using tools and data formats available through the CER Hub. The QA process, implemented here in a study of smoking cessation services in primary care, used the 'emrAdapter' tool programmed with a set of quality checks to query large samples of primary care encounter records extracted in accord with the CER Hub common data framework. The tool, deployed to each study site, generated error reports indicating data problems to be fixed locally and aggregate data sharable with the central site for quality review. Across the CER Hub network of six health systems, data completeness and correctness issues were prevalent in the first iteration and were considerably improved after three iterations of the QA process. A common issue encountered was incomplete mapping of local EHR data values to those defined by the common data framework. A highly automated and distributed QA process helped to ensure the correctness and completeness of patient care data extracted from EHRs for a multi-institution CER study in smoking cessation.

Automating assessment of lifestyle counseling in electronic health records.

BACKGROUND: Numerous population-based surveys indicate that overweight and obese patients can benefit from lifestyle counseling during routine clinical care. PURPOSE: To determine if natural language processing (NLP) could be applied to information in the electronic health record (EHR) to automatically assess delivery of weight management-related counseling in clinical healthcare encounters. METHODS: The MediClass system with NLP capabilities was used to identify weight-management counseling in EHRs. Knowledge for the NLP application was derived from the 5As framework for behavior counseling: Ask (evaluate weight and related disease), Advise at-risk patients to lose weight, Assess patients' readiness to change behavior, Assist through discussion of weight-loss methods and programs, and Arrange follow-up efforts including referral. Using samples of EHR data between January 1, 2007, and March 31, 2011, from two health systems, the accuracy of the MediClass processor for identifying these counseling elements was evaluated in postpartum visits of 600 women with gestational diabetes mellitus (GDM) compared to manual chart review as the gold standard. Data were analyzed in 2013. RESULTS: Mean sensitivity and specificity for each of the 5As compared to the gold standard was at or above 85%, with the exception of sensitivity for Assist, which was 40% and 60% for each of the two health systems. The automated method identified many valid Assist cases not identified in the gold standard. CONCLUSIONS: The MediClass processor has performance capability sufficiently similar to human abstractors to permit automated assessment of counseling for weight loss in postpartum encounter records.

Documentation of the 5 as for smoking cessation by PCPs across distinct health systems.

OBJECTIVES: Physicians can help patients quit smoking using the 5 As of smoking cessation. This study aimed to (1) identify the proportion of known smokers that receive smoking cessation services in the course of routine clinical practice; (2) describe demographic and comorbidity characteristics of patients receiving the 5 As in these systems; and (3) evaluate differences in performance of the 5 As across health systems, gender, and age categories. STUDY DESIGN: Electronic medical records of 200 current smokers from 6 unique health systems (N = 1200) were randomly selected from 2006 to 2010. Primary care encounter progress notes were hand coded for occurrences of the 5 As. METHODS: Bivariate comparisons of delivery of the 3 smoking-cessation services by site, gender, and age category were analyzed using χ² tests. RESULTS: About 50% of smokers were advised to quit smoking, 39% were assessed for their readiness to quit, and 54% received some type of assistance to help them quit smoking. Only 2% had a documented plan for follow-up regarding their quitting efforts (arrange). Significant differences were found among sites for documentation of receiving the 5 As and between age groups receiving assistance with quitting. There was no statistically significant difference between genders in receipt of the 5 As. CONCLUSIONS: Documentation of adherence to the 5 As varied by site and some demographics. Adjustments to protocols for addressing cessation and readiness to quit may be warranted. Health systems could apply the methodology described in this paper to assess their own performance, and then use that as a basis to guide improvement initiatives.

A survey of informatics platforms that enable distributed comparative effectiveness research using multi-institutional heterogenous clinical data.

Comparative effectiveness research (CER) has the potential to transform the current health care delivery system by identifying the most effective medical and surgical treatments, diagnostic tests, disease prevention methods, and ways to deliver care for specific clinical conditions. To be successful, such research requires the identification, capture, aggregation, integration, and analysis of disparate data sources held by different institutions with diverse representations of the relevant clinical events. In an effort to address these diverse demands, there have been multiple new designs and implementations of informatics platforms that provide access to electronic clinical data and the governance infrastructure required for interinstitutional CER. The goal of this manuscript is to help investigators understand why these informatics platforms are required and to compare and contrast 6 large-scale, recently funded, CER-focused informatics platform development efforts. We utilized an 8-dimension, sociotechnical model of health information technology to help guide our work. We identified 6 generic steps that are necessary in any distributed, multi-institutional CER project: data identification, extraction, modeling, aggregation, analysis, and dissemination. We expect that over the next several years these projects will provide answers to many important, and heretofore unanswerable, clinical research questions.