Development of a machine learning model to predict mild cognitive impairment using natural language processing in the absence of screening.

BACKGROUND: Patients and their loved ones often report symptoms or complaints of cognitive decline that clinicians note in free clinical text, but no structured screening or diagnostic data are recorded. These symptoms/complaints may be signals that predict who will go on to be diagnosed with mild cognitive impairment (MCI) and ultimately develop Alzheimer's Disease or related dementias. Our objective was to develop a natural language processing system and prediction model for identification of MCI from clinical text in the absence of screening or other structured diagnostic information. METHODS: There were two populations of patients: 1794 participants in the Adult Changes in Thought (ACT) study and 2391 patients in the general population of Kaiser Permanente Washington. All individuals had standardized cognitive assessment scores. We excluded patients with a diagnosis of Alzheimer's Disease, Dementia or use of donepezil. We manually annotated 10,391 clinic notes to train the NLP model. Standard Python code was used to extract phrases from notes and map each phrase to a cognitive functioning concept. Concepts derived from the NLP system were used to predict future MCI. The prediction model was trained on the ACT cohort and 60% of the general population cohort with 40% withheld for validation. We used a least absolute shrinkage and selection operator logistic regression approach (LASSO) to fit a prediction model with MCI as the prediction target. Using the predicted case status from the LASSO model and known MCI from standardized scores, we constructed receiver operating curves to measure model performance. RESULTS: Chart abstraction identified 42 MCI concepts. Prediction model performance in the validation data set was modest with an area under the curve of 0.67. Setting the cutoff for correct classification at 0.60, the classifier yielded sensitivity of 1.7%, specificity of 99.7%, PPV of 70% and NPV of 70.5% in the validation cohort. DISCUSSION AND CONCLUSION: Although the sensitivity of the machine learning model was poor, negative predictive value was high, an important characteristic of models used for population-based screening. While an AUC of 0.67 is generally considered moderate performance, it is also comparable to several tests that are widely used in clinical practice.

Inferring disease severity in rheumatoid arthritis using predictive modeling in administrative claims databases.

BACKGROUND: Confounding by disease severity is an issue in pharmacoepidemiology studies of rheumatoid arthritis (RA), due to channeling of sicker patients to certain therapies. To address the issue of limited clinical data for confounder adjustment, a patient-level prediction model to differentiate between patients prescribed and not prescribed advanced therapies was developed as a surrogate for disease severity, using all available data from a US claims database. METHODS: Data from adult RA patients were used to build regularized logistic regression models to predict current and future disease severity using a biologic or tofacitinib prescription claim as a surrogate for moderate-to-severe disease. Model discrimination was assessed using the area under the receiver (AUC) operating characteristic curve, tested and trained in Optum Clinformatics® Extended DataMart (Optum) and additionally validated in three external IBM MarketScan® databases. The model was further validated in the Optum database across a range of patient cohorts. RESULTS: In the Optum database (n = 68,608), the AUC for discriminating RA patients with a prescription claim for a biologic or tofacitinib versus those without in the 90 days following index diagnosis was 0.80. Model AUCs were 0.77 in IBM CCAE (n = 75,579) and IBM MDCD (n = 7,537) and 0.75 in IBM MDCR (n = 36,090). There was little change in the prediction model assessing discrimination 730 days following index diagnosis (prediction model AUC in Optum was 0.79). CONCLUSIONS: A prediction model demonstrated good discrimination across multiple claims databases to identify RA patients with a prescription claim for advanced therapies during different time-at-risk periods as proxy for current and future moderate-to-severe disease. This work provides a robust model-derived risk score that can be used as a potential covariate and proxy measure to adjust for confounding by severity in multivariable models in the RA population. An R package to develop the prediction model and risk score are available in an open source platform for researchers.

Comparative effectiveness of canagliflozin, SGLT2 inhibitors and non-SGLT2 inhibitors on the risk of hospitalization for heart failure and amputation in patients with type 2 diabetes mellitus: A real-world meta-analysis of 4 observational databases (OBSERVE-4D).

AIMS: Sodium glucose co-transporter 2 inhibitors (SGLT2i) are indicated for treatment of type 2 diabetes mellitus (T2DM); some SGLT2i have reported cardiovascular benefit, and some have reported risk of below-knee lower extremity (BKLE) amputation. This study examined the real-world comparative effectiveness within the SGLT2i class and compared with non-SGLT2i antihyperglycaemic agents. MATERIALS AND METHODS: Data from 4 large US administrative claims databases were used to characterize risk and provide population-level estimates of canagliflozin's effects on hospitalization for heart failure (HHF) and BKLE amputation vs other SGLT2i and non-SGLT2i in T2DM patients. Comparative analyses using a propensity score-adjusted new-user cohort design examined relative hazards of outcomes across all new users and a subpopulation with established cardiovascular disease. RESULTS: Across the 4 databases (142 800 new users of canagliflozin, 110 897 new users of other SGLT2i, 460 885 new users of non-SGLT2i), the meta-analytic hazard ratio estimate for HHF with canagliflozin vs non-SGLT2i was 0.39 (95% CI, 0.26-0.60) in the on-treatment analysis. The estimate for BKLE amputation with canagliflozin vs non-SGLT2i was 0.75 (95% CI, 0.40-1.41) in the on-treatment analysis and 1.01 (95% CI, 0.93-1.10) in the intent-to-treat analysis. Effects in the subpopulation with established cardiovascular disease were similar for both outcomes. No consistent differences were observed between canagliflozin and other SGLT2i. CONCLUSIONS: In this large comprehensive analysis, canagliflozin and other SGLT2i demonstrated HHF benefits consistent with clinical trial data, but showed no increased risk of BKLE amputation vs non-SGLT2i. HHF and BKLE amputation results were similar in the subpopulation with established cardiovascular disease. This study helps further characterize the potential benefits and harms of SGLT2i in routine clinical practice to complement evidence from clinical trials and prior observational studies.

Risk of lower extremity amputations in people with type 2 diabetes mellitus treated with sodium-glucose co-transporter-2 inhibitors in the USA: A retrospective cohort study.

AIMS: To examine the incidence of amputation in patients with type 2 diabetes mellitus (T2DM) treated with sodium glucose co-transporter 2 (SGLT2) inhibitors overall, and canagliflozin specifically, compared with non-SGLT2 inhibitor antihyperglycaemic agents (AHAs). MATERIALS AND METHODS: Patients with T2DM newly exposed to SGLT2 inhibitors or non-SGLT2 inhibitor AHAs were identified using the Truven MarketScan database. The incidence of below-knee lower extremity (BKLE) amputation was calculated for patients treated with SGLT2 inhibitors, canagliflozin, or non-SGLT2 inhibitor AHAs. Patients newly exposed to canagliflozin and non-SGLT2 inhibitor AHAs were matched 1:1 on propensity scores, and a Cox proportional hazards model was used for comparative analysis. Negative controls (outcomes not believed to be associated with any AHA) were used to calibrate P values. RESULTS: Between April 1, 2013 and October 31, 2016, 118 018 new users of SGLT2 inhibitors, including 73 024 of canagliflozin, and 226 623 new users of non-SGLT2 inhibitor AHAs were identified. The crude incidence rates of BKLE amputation were 1.22, 1.26 and 1.87 events per 1000 person-years with SGLT2 inhibitors, canagliflozin and non-SGLT2 inhibitor AHAs, respectively. For the comparative analysis, 63 845 new users of canagliflozin were matched with 63 845 new users of non-SGLT2 inhibitor AHAs, resulting in well-balanced baseline covariates. The incidence rates of BKLE amputation were 1.18 and 1.12 events per 1000 person-years with canagliflozin and non-SGLT2 inhibitor AHAs, respectively; the hazard ratio was 0.98 (95% confidence interval 0.68-1.41; P = .92, calibrated P = .95). CONCLUSIONS: This real-world study observed no evidence of increased risk of BKLE amputation for new users of canagliflozin compared with non-SGLT2 inhibitor AHAs in a broad population of patients with T2DM.

Incidence of diabetic ketoacidosis among patients with type 2 diabetes mellitus treated with SGLT2 inhibitors and other antihyperglycemic agents.

AIMS: To estimate and compare incidence of diabetes ketoacidosis (DKA) among patients with type 2 diabetes who are newly treated with SGLT2 inhibitors (SGLT2i) versus non-SGLT2i antihyperglycemic agents (AHAs) in actual clinical practice. METHODS: A new-user cohort study design using a large insurance claims database in the US. DKA incidence was compared between new users of SGLT2i and new users of non-SGLT2i AHAs pair-matched on exposure propensity scores (EPS) using Cox regression models. RESULTS: Overall, crude incidence rates (95% CI) per 1000 patient-years for DKA were 1.69 (1.22-2.30) and 1.83 (1.58-2.10) among new users of SGLT2i (n=34,442) and non-SGLT2i AHAs (n=126,703). These rates more than doubled among patients with prior insulin prescriptions but decreased by more than half in analyses that excluded potential autoimmune diabetes (PAD). The hazard ratio (95% CI) for DKA comparing new users of SGLT2i to new users of non-SGLT2i AHAs was 1.91 (0.94-4.11) (p=0.09) among the 30,196 EPS-matched pairs overall, and 1.13 (0.43-3.00) (p=0.81) among the 27,515 EPS-matched pairs that excluded PAD. CONCLUSIONS: This was the first observational study that compared DKA risk between new users of SGLT2i and non-SGLT2i AHAs among patients with type 2 diabetes, and overall no statistically significant difference was detected.

Atypical Antipsychotics and the Risks of Acute Kidney Injury and Related Outcomes Among Older Adults: A Replication Analysis and an Evaluation of Adapted Confounding Control Strategies.

OBJECTIVE: A recently published analysis of population-based claims data from Ontario, Canada reported higher risks of acute kidney injury (AKI) and related outcomes among older adults who were new users of atypical antipsychotics (AAPs) compared with unexposed patients. In light of these findings, the objective of the current study was to further investigate the risks of AKI and related outcomes among older adults receiving AAPs. METHODS: A replication of the previously published analysis was performed using the US Truven MarketScan Medicare Supplemental database (MDCR) among patients aged 65 years and older. Compared with non-users of AAPs, the study compared the risk of AKI and related outcomes with users of AAPs (quetiapine, risperidone, olanzapine, aripiprazole, or paliperidone) using a 1-to-1 propensity score matched analysis. In addition, we performed adapted analyses that: (1) included all covariates used to fit propensity score models in outcome models; and (2) required patients to have a diagnosis of schizophrenia, bipolar disorder, or major depression and a healthcare visit within 90 days prior to the index date. RESULTS: AKI effect estimates [as odds ratios (ORs) with 95% confidence intervals (CIs)] were significantly elevated in our MDCR replication analyses (OR 1.45, 95% CI 1.32-1.60); however, in adapted analyses, associations were not significant (OR 0.91, 95% CI 0.78-1.07)). In analyses of AKI and related outcomes, results were mostly consistent between the previously published and the MDCR replication analyses. The primary change that attenuated associations in adapted analyses was the requirement for patients to have a mental health condition and a healthcare visit prior to the index date. CONCLUSIONS: The MDCR analysis yielded similar results when the methodology of the previously published analysis was replicated, but, in adapted analyses, we did not find significantly higher risks of AKI and related outcomes. The contrast of results between our replication and adapted analyses may be due to the analytic approach used to compare patients (and potential confounding by indication). Further research is warranted to evaluate these associations, while also examining methods to account for differences in older adults who do and do not use these medications.

Depression among people with type 2 diabetes mellitus, US National Health and Nutrition Examination Survey (NHANES), 2005-2012.

BACKGROUND: Depression in people with diabetes can result in increased risk for diabetes-related complications. The prevalence of depression has been estimated to be 17.6 % in people with type 2 diabetes mellitus (T2DM), based on studies published between 1980 and 2005. There is a lack of more recent estimates of depression prevalence among the US general T2DM population. METHODS: The present study used the US National Health and Nutrition Examination Survey (NHANES) 2005-2012 data to provide an updated, population-based estimate for the prevalence of depression in people with T2DM. NHANES is a cross-sectional survey of a nationally representative sample of the civilian, non-institutionalized US population. Starting from 2005, the Patient Health Questionnaire (PHQ-9) was included to measure signs and symptoms of depression. We defined PHQ-9 total scores ≥ 10 as clinically relevant depression (CRD), and ≥ 15 as clinically significant depression (CSD). Self-reported current antidepressant use was also combined to estimate overall burden of depression. Predictors of CRD and CSD were investigated using survey logistic regression models. RESULTS: A total of 2182 participants with T2DM were identified. The overall prevalence of CRD and CSD among people with T2DM is 10.6 % (95 % confidence interval (CI) 8.9-12.2 %), and 4.2 % (95 % CI 3.4-5.1 %), respectively. The combined burden of depressive symptoms and antidepressants may be as high as 25.4 % (95 % CI 23.0-27.9 %). Significant predictors of CRD include age (younger than 65), sex (women), income (lower than 130 % of poverty level), education (below college), smoking (current or former smoker), body mass index (≥30 kg/m(2)), sleep problems, hospitalization in the past year, and total cholesterol (≥200 mg/dl). Significant predictors of CSD also include physical activity (below guideline) and cardiovascular diseases. CONCLUSIONS: The prevalence of CRD and CSD among people with T2DM in the US may be lower than in earlier studies, however, the burden of depression remains high. Further research with longitudinal follow-up for depression in people with T2DM is needed to understand real world effectiveness of depression management.

Seasonality in acute liver injury? Findings in two health care claims databases.

BACKGROUND: Presumed seasonal use of acetaminophen-containing products for relief of cold/influenza ("flu") symptoms suggests that there might also be a corresponding seasonal pattern for acute liver injury (ALI), a known clinical consequence of acetaminophen overdose. OBJECTIVE: The objective of this study was to determine whether there were any temporal patterns in hospitalizations for ALI that would correspond to assumed acetaminophen use in cold/flu season. METHODS: In the period 2002-2010, monthly hospitalization rates for ALI using a variety of case definitions were calculated. Data sources included Truven MarketScan(®) Commercial Claims and Encounters (CCAE) and Medicare Supplemental and Coordination of Benefits (MDCR) databases. We performed a statistical test for seasonality of diagnoses using the periodic generalized linear model. To validate that the test can distinguish seasonal from nonseasonal patterns, we included two positive controls (ie, diagnoses of the common cold [acute nasopharyngitis] and influenza), believed to change with seasons, and two negative controls (female breast cancer and diabetes), believed to be insensitive to season. RESULTS: A seasonal pattern was observed in monthly rates for common cold and influenza diagnoses, but this pattern was not observed for monthly rates of ALI, with or without comorbidities (cirrhosis or hepatitis), breast cancer, or diabetes. The statistical test for seasonality was significant for positive controls (P<0.001 for each diagnosis in both databases) and nonsignificant for ALI and negative controls. CONCLUSION: No seasonal pattern was observed in the diagnosis of ALI. The positive and negative controls showed the expected patterns, strengthening the validity of the statistical and visual tests used for detecting seasonality.

Changes in Body Weight Among People With Type 2 Diabetes Mellitus in the United States, NHANES 2005-2012.

PURPOSE: To understand weight loss strategies, weight changes, goals, and behaviors in people with type 2 diabetes mellitus (T2DM) and whether these differ by ethnicity. METHODS: T2DM was identified by self-reported diagnosis using the NHANES 2005-2012 data, which also included measured and self-reported current body weight and height, self-reported weight the prior year, and self-reported aspired weight. Nineteen weight loss strategies were evaluated for association with ≥5% weight loss or weight gain versus <5% weight change. RESULTS: Among people with T2DM, 88.0% were overweight/obese (body mass index [BMI] ≥25 kg/m(2)) in the prior year and 86.1% the current year. About 60% of the overweight/obese took weight loss actions, mostly using diet-related methods with average weight lost <5%. Two most "effective" methods reported (smoking, taking laxatives/vomiting) are also potentially most harmful. Similar BMI distributions but different goals and behaviors about weight and weight loss were observed across ethnicity. Only physical activity meeting the recommended level and changing eating habits were consistently associated with favorable and statistically significant weight change. CONCLUSIONS: Weight management in T2DM is an ongoing challenge, regardless of ethnicity/race. Among overweight/obese T2DM subjects, recommended level of physical activity and changing eating habits were associated with statistically significant favorable weight change.

Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers.

The vision of creating accessible, reliable clinical evidence by accessing the clincial experience of hundreds of millions of patients across the globe is a reality. Observational Health Data Sciences and Informatics (OHDSI) has built on learnings from the Observational Medical Outcomes Partnership to turn methods research and insights into a suite of applications and exploration tools that move the field closer to the ultimate goal of generating evidence about all aspects of healthcare to serve the needs of patients, clinicians and all other decision-makers around the world.

Switching from risperidone long-acting injectable to paliperidone long-acting injectable or oral antipsychotics: analysis of a Medicaid claims database.

This report examines relapse risk following a switch from risperidone long-acting injectable (RLAI) to another long-acting injectable antipsychotic [paliperidone palmitate (PP)] versus a switch to oral antipsychotics (APs). Truven Health's MarketScan Multistate Medicaid Database compared relapses following switches from RLAI. New user cohorts for these two groups were created on the basis of first incidence of exposure to the 'switched to' drug. Groups were balanced using 1:1 propensity score matching. Time-to-event analysis assessed schizophrenia-related hospital/emergency department visits. A total of 188 patients switched from RLAI to PP, and 131 patients switched from RLAI to oral AP. Propensity score-matched cohort included 109 patients who switched to PP and 109 patients who switched to an oral AP. Patients who switched from RLAI to PP had fewer events (26 vs. 32), longer time to an event (mean 70 vs. 47 days), and lower risk of relapse (hazard ratio, 0.54; 95% confidence interval, 0.32-0.92; P=0.024) compared with those who switched from RLAI to oral AP. Switching from RLAI to PP may be associated with a lower risk for relapse and longer duration of therapy compared with switching to oral AP. Given the limitations of observational studies, these results should be confirmed by other prospective evaluations.

How well can post-traumatic stress disorder be predicted from pre-trauma risk factors? An exploratory study in the WHO World Mental Health Surveys.

Post-traumatic stress disorder (PTSD) should be one of the most preventable mental disorders, since many people exposed to traumatic experiences (TEs) could be targeted in first response settings in the immediate aftermath of exposure for preventive intervention. However, these interventions are costly and the proportion of TE-exposed people who develop PTSD is small. To be cost-effective, risk prediction rules are needed to target high-risk people in the immediate aftermath of a TE. Although a number of studies have been carried out to examine prospective predictors of PTSD among people recently exposed to TEs, most were either small or focused on a narrow sample, making it unclear how well PTSD can be predicted in the total population of people exposed to TEs. The current report investigates this issue in a large sample based on the World Health Organization (WHO)'s World Mental Health Surveys. Retrospective reports were obtained on the predictors of PTSD associated with 47,466 TE exposures in representative community surveys carried out in 24 countries. Machine learning methods (random forests, penalized regression, super learner) were used to develop a model predicting PTSD from information about TE type, socio-demographics, and prior histories of cumulative TE exposure and DSM-IV disorders. DSM-IV PTSD prevalence was 4.0% across the 47,466 TE exposures. 95.6% of these PTSD cases were associated with the 10.0% of exposures (i.e., 4,747) classified by machine learning algorithm as having highest predicted PTSD risk. The 47,466 exposures were divided into 20 ventiles (20 groups of equal size) ranked by predicted PTSD risk. PTSD occurred after 56.3% of the TEs in the highest-risk ventile, 20.0% of the TEs in the second highest ventile, and 0.0-1.3% of the TEs in the 18 remaining ventiles. These patterns of differential risk were quite stable across demographic-geographic sub-samples. These results demonstrate that a sensitive risk algorithm can be created using data collected in the immediate aftermath of TE exposure to target people at highest risk of PTSD. However, validation of the algorithm is needed in prospective samples, and additional work is warranted to refine the algorithm both in terms of determining a minimum required predictor set and developing a practical administration and scoring protocol that can be used in routine clinical practice.

Desideratum for evidence based epidemiology.

BACKGROUND: There is great variation in choices of method and specific analytical details in epidemiological studies, resulting in widely varying results even when studying the same drug and outcome in the same database. Not only does this variation undermine the credibility of the research but it limits our ability to improve the methods. METHODS: In order to evaluate the performance of methods and analysis choices we used standard references and a literature review to identify 164 positive controls (drug-outcome pairs believed to represent true adverse drug reactions), and 234 negative controls (drug-outcome pairs for which we have confidence there is no direct causal relationship). We tested 3,748 unique analyses (methods in combination with specific analysis choices) that represent the full range of approaches to adjusting for confounding in five large observational datasets on these controls. We also evaluated the impact of increasingly specific outcome definitions, and performed a replication study in six additional datasets. We characterized the performance of each method using the area under the receiver operator curve (AUC), bias, and coverage probability. In addition, we developed simulated datasets that closely matched the characteristics of the observational datasets into which we inserted data consistent with known drug-outcome relationships in order to measure the accuracy of estimates generated by the analyses. DISCUSSION: We expect the results of this systematic, empirical evaluation of the performance of these analyses across a moderate range of outcomes and databases to provide important insights into the methods used in epidemiological studies and to increase the consistency with which methods are applied, thereby increasing the confidence in results and our ability to systematically improve our approaches.

Variation in choice of study design: findings from the Epidemiology Design Decision Inventory and Evaluation (EDDIE) survey.

BACKGROUND: Researchers using observational data to understand drug effects must make a number of analytic design choices that suit the characteristics of the data and the subject of the study. Review of the published literature suggests that there is a lack of consistency even when addressing the same research question in the same database. OBJECTIVE: To characterize the degree of similarity or difference in the method and analysis choices made by observational database research experts when presented with research study scenarios. RESEARCH DESIGN: On-line survey using research scenarios on drug-effect studies to capture method selection and analysis choices that follow a dependency branching based on response to key questions. SUBJECTS: Voluntary participants experienced in epidemiological study design solicited for participation through registration on the Observational Medical Outcomes Partnership website, membership in particular professional organizations, or links in relevant newsletters. MEASURES: Description (proportion) of respondents selecting particular methods and making specific analysis choices based on individual drug-outcome scenario pairs. The number of questions/decisions differed based on stem questions of study design, time-at-risk, outcome definition, and comparator. RESULTS: There is little consistency across scenarios, by drug or by outcome of interest, in the decisions made for design and analyses in scenarios using large healthcare databases. The most consistent choice was the cohort study design but variability in the other critical decisions was common. CONCLUSIONS: There is great variation among epidemiologists in the design and analytical choices that they make when implementing analyses in observational healthcare databases. These findings confirm that it will be important to generate empiric evidence to inform these decisions and to promote a better understanding of the impact of standardization on research implementation.

Managing data quality for a drug safety surveillance system.

OBJECTIVE: The objective of this study is to present a data quality assurance program for disparate data sources loaded into a Common Data Model, highlight data quality issues identified and resolutions implemented. BACKGROUND: The Observational Medical Outcomes Partnership is conducting methodological research to develop a system to monitor drug safety. Standard processes and tools are needed to ensure continuous data quality across a network of disparate databases, and to ensure that procedures used to extract-transform-load (ETL) processes maintain data integrity. Currently, there is no consensus or standard approach to evaluate the quality of the source data, or ETL procedures. METHODS: We propose a framework for a comprehensive process to ensure data quality throughout the steps used to process and analyze the data. The approach used to manage data anomalies includes: (1) characterization of data sources; (2) detection of data anomalies; (3) determining the cause of data anomalies; and (4) remediation. FINDINGS: Data anomalies included incomplete raw dataset: no race or year of birth recorded. Implausible data: year of birth exceeding current year, observation period end date precedes start date, suspicious data frequencies and proportions outside normal range. Examples of errors found in the ETL process were zip codes incorrectly loaded, drug quantities rounded, drug exposure length incorrectly calculated, and condition length incorrectly programmed. CONCLUSIONS: Complete and reliable observational data are difficult to obtain, data quality assurance processes need to be continuous as data is regularly updated; consequently, processes to assess data quality should be ongoing and transparent.

A comparison of the empirical performance of methods for a risk identification system.

BACKGROUND: Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown. OBJECTIVES: To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data. RESEARCH DESIGN: The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record). MEASURES: Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability. RESULTS: Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case-control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties. CONCLUSIONS: Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.

Assessment of case definitions for identifying acute liver injury in large observational databases.

BACKGROUND: Determining the aetiology of acute liver injury (ALI) may be challenging to both clinicians and researchers. Observational research is particularly useful in studying rare medical outcomes such as ALI; however, case definitions for ALI in previous observational studies lack consistency and sensitivity. ALI is a clinically important condition with various aetiologies, including drug exposure. OBJECTIVE: The aim of this study was to evaluate four distinct case definitions for ALI across a diverse set of large observational databases, providing a better understanding of ALI prevalence and natural history. DATA SOURCES: Seven healthcare databases: GE Healthcare, MarketScan(®) Lab Database, Humana Inc., Partners HealthCare System, Regenstrief Institute, SDI Health (now IMS Health, Inc.), and the National Patient Care Database of the Veterans Health Administration. METHODS: We evaluated prevalence of ALI through the application of four distinct case definitions across seven observational healthcare databases. We described how laboratory and clinical characteristics of identified case populations varied across definitions and examined the prevalence of other hepatobiliary disorders among identified ALI cases that may decrease suspicion of drug-induced liver injury (DILI) in particular. RESULTS: This study demonstrated that increasing the restrictiveness of the case definition resulted in fewer cases, but greater prevalence of ALI clinical features. Considerable heterogeneity in the frequency of laboratory testing and results observed among cases meeting the most restrictive definition suggests that the clinical features, monitoring patterns and suspicion of ALI are highly variable among patients. CONCLUSIONS: Creation of four distinct case definitions and application across a disparate set of observational databases resulted in significant variation in the prevalence of ALI. A greater understanding of the natural history of ALI through examination of electronic healthcare data can facilitate development of reliable and valid ALI case definitions that may enhance the ability to accurately identify associations between ALI and drug exposures. Considerable heterogeneity in laboratory values and frequency of laboratory testing among individuals meeting the criteria for ALI suggests that the evaluation of ALI is highly variable.

Evaluating the impact of database heterogeneity on observational study results.

Clinical studies that use observational databases to evaluate the effects of medical products have become commonplace. Such studies begin by selecting a particular database, a decision that published papers invariably report but do not discuss. Studies of the same issue in different databases, however, can and do generate different results, sometimes with strikingly different clinical implications. In this paper, we systematically study heterogeneity among databases, holding other study methods constant, by exploring relative risk estimates for 53 drug-outcome pairs and 2 widely used study designs (cohort studies and self-controlled case series) across 10 observational databases. When holding the study design constant, our analysis shows that estimated relative risks range from a statistically significant decreased risk to a statistically significant increased risk in 11 of 53 (21%) of drug-outcome pairs that use a cohort design and 19 of 53 (36%) of drug-outcome pairs that use a self-controlled case series design. This exceeds the proportion of pairs that were consistent across databases in both direction and statistical significance, which was 9 of 53 (17%) for cohort studies and 5 of 53 (9%) for self-controlled case series. Our findings show that clinical studies that use observational databases can be sensitive to the choice of database. More attention is needed to consider how the choice of data source may be affecting results.