A Feasibility Study of Drug-Drug Interaction Signal Detection in Regular Pharmacovigilance.

INTRODUCTION: Adverse drug reactions related to drug-drug interactions cause harm to patients. There is a body of research on signal detection for drug interactions in collections of individual case reports, but limited use in regular pharmacovigilance. OBJECTIVE: The aim of this study was to evaluate the feasibility of signal detection of drug-drug interactions in collections of individual case reports of suspected adverse drug reactions. METHODS: This study was conducted in VigiBase, the WHO global database of individual case safety reports. The data lock point was 31 August 2016, which provided 13.6 million reports for analysis after deduplication. Statistical signal detection was performed using a previously developed predictive model for possible drug interactions. The model accounts for an interaction disproportionality measure, expressed suspicion of an interaction by the reporter, potential for interaction through cytochrome P450 activity of drugs, and reported information indicative of unexpected therapeutic response or altered therapeutic effect. Triage filters focused the preliminary signal assessment on combinations relating to serious adverse events with case series of no more than 30 reports from at least two countries, with at least one report during the previous 2 years. Additional filters sought to eliminate already known drug interactions through text mining of standard literature sources. Preliminary signal assessment was performed by a multidisciplinary group of pharmacovigilance professionals from Uppsala Monitoring Centre and collaborating organizations, whereas in-depth signal assessment was performed by experienced pharmacovigilance assessors. RESULTS: We performed preliminary signal assessment for 407 unique drug pairs. Of these, 157 drug pairs were considered already known to interact, whereas 232 were closed after preliminary assessment for other reasons. Ten drug pairs were subjected to in-depth signal assessment and an additional eight were decided to be kept under review awaiting additional reports. The triage filters had a major impact in focusing our preliminary signal assessment on just 14% of the statistical signals generated by the predictive model for drug interactions. In-depth assessment led to three signals communicated with the broader pharmacovigilance community, six closed signals and one to be kept under review. CONCLUSION: This study shows that signals of adverse drug interactions can be detected through broad statistical screening of individual case reports. It further shows that signal assessment related to possible drug interactions requires more detailed information on the temporal relationship between different drugs and the adverse event. Future research may consider whether interaction signal detection should be performed not for individual adverse event terms but for pairs of drugs across a spectrum of adverse events.

Prospective Evaluation of Adverse Event Recognition Systems in Twitter: Results from the Web-RADR Project.

INTRODUCTION: A large number of studies on systems to detect and sometimes normalize adverse events (AEs) in social media have been published, but evidence of their practical utility is scarce. This raises the question of the transferability of such systems to new settings. OBJECTIVES: The aims of this study were to develop an AE recognition system, prospectively evaluate its performance on an external benchmark dataset and identify potential factors influencing the transferability of AE recognition systems. METHODS: A pipeline based on dictionary lookups and logistic regression classifiers was developed using a proprietary dataset of 196,533 Tweets manually annotated for AE relations and prospectively evaluated the system on the publicly available WEB-RADR reference dataset, exploring different aspects affecting transferability. RESULTS: Our system achieved 0.53 precision, 0.52 recall and 0.52 F1-score on the development test set; however, when applied to the WEB-RADR reference dataset, system performance dropped to 0.38 precision, 0.20 recall and 0.26 F1-score. Similarly, a previously published method aiming at automatically detecting adverse event posts reported 0.5 precision, 0.92 recall and 0.65 F1-score on thus another dataset, while performance on the WEB-RADR reference dataset was reduced to 0.37 precision, 0.63 recall and 0.46 F1-score. We identified four potential factors leading to poor transferability: overfitting, selection bias, label bias and prevalence. CONCLUSION: We warn the community about a potentially large discrepancy between the expected performance of automated AE recognition systems based on published results and the actual observed performance on independent data. This study highlights the difficulty of implementing an all-purpose system for automatic adverse event recognition in Twitter, which could explain the lack of such systems in practical pharmacovigilance settings. Our recommendation is to use benchmark independent datasets, such as the WEB-RADR reference, to investigate the transferability of the adverse event recognition systems and ultimately enforce rigorous comparisons across studies on the task.