Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department.

BACKGROUND AND OBJECTIVES: There is increasing interest in using disproportionality-based signal detection methods to support postmarketing safety surveillance activities. Two commonly used methods, empirical Bayes multi-item gamma Poisson shrinker (MGPS) and proportional reporting ratio (PRR), perform differently with respect to the number and types of signals detected. The goal of this study was to compare and analyse the performance characteristics of these two methods, to understand why they differ and to consider the practical implications of these differences for a large, industry-based pharmacovigilance department. METHODS: We compared the numbers and types of signals of disproportionate reporting (SDRs) obtained with MGPS and PRR using two postmarketing safety databases and a simulated database. We recorded signal counts and performed a qualitative comparison of the drug-event combinations signalled by the two methods as well as a sensitivity analysis to better understand how the thresholds commonly used for these methods impact their performance. RESULTS: PRR detected more SDRs than MGPS. We observed that MGPS is less subject to confounding by demographic factors because it employs stratification and is more stable than PRR when report counts are low. Simulation experiments performed using published empirical thresholds demonstrated that PRR detected false-positive signals at a rate of 1.1%, while MGPS did not detect any statistical false positives. In an attempt to separate the effect of choice of signal threshold from more fundamental methodological differences, we performed a series of experiments in which we modified the conventional threshold values for each method so that each method detected the same number of SDRs for the example drugs studied. This analysis, which provided quantitative examples of the relationship between the published thresholds for the two methods, demonstrates that the signalling criterion published for PRR has a higher signalling frequency than that published for MGPS. DISCUSSION AND CONCLUSION: The performance differences between the PRR and MGPS methods are related to (i) greater confounding by demographic factors with PRR; (ii) a higher tendency of PRR to detect false-positive signals when the number of reports is small; and (iii) the conventional thresholds that have been adapted for each method. PRR tends to be more 'sensitive' and less 'specific' than MGPS. A high-specificity disproportionality method, when used in conjunction with medical triage and investigation of critical medical events, may provide an efficient and robust approach to applying quantitative methods in routine postmarketing pharmacovigilance.

Evolving paradigms in pharmacovigilance.

All medicines have adverse effects as well as benefits. The aim of pharmacovigilance is to protect public health by monitoring medicines to identify and evaluate issues and ensure that the overall benefits outweigh the potential risks. The tools and processes used in pharmacovigilance are continually evolving. Increasingly sophisticated tools are being designed to evaluate safety data from clinical trials to enhance the likelihood of detecting safety signals ahead of product registration. Methods include integration of safety data throughout development, meta-analytical techniques, quantitative and qualitative methods for evaluation of adverse event data and graphical tools to explore laboratory and biometric data. Electronic data capture facilitates monitoring of ongoing studies so that it is possible to promptly identify potential issues and manage patient safety. In addition, GSK employs a number of proactive methods for post-marketing signal detection and knowledge management using state-of-the-art statistical and analytical tools. Using these tools, together with safety data collected through pharmacoepidemiologic studies, literature and spontaneous reporting, potential adverse drug reactions can be better identified in marketed products. In summary, the information outlined in this paper provides a valuable benchmark for risk management and pharmacovigilance in pharmaceutical development.

Perspectives on the use of data mining in pharmaco-vigilance.

In the last 5 years, regulatory agencies and drug monitoring centres have been developing computerised data-mining methods to better identify reporting relationships in spontaneous reporting databases that could signal possible adverse drug reactions. At present, there are no guidelines or standards for the use of these methods in routine pharmaco-vigilance. In 2003, a group of statisticians, pharmaco-epidemiologists and pharmaco-vigilance professionals from the pharmaceutical industry and the US FDA formed the Pharmaceutical Research and Manufacturers of America-FDA Collaborative Working Group on Safety Evaluation Tools to review best practices for the use of these methods.In this paper, we provide an overview of: (i) the statistical and operational attributes of several currently used methods and their strengths and limitations; (ii) information about the characteristics of various postmarketing safety databases with which these tools can be deployed; (iii) analytical considerations for using safety data-mining methods and interpreting the results; and (iv) points to consider in integration of safety data mining with traditional pharmaco-vigilance methods. Perspectives from both the FDA and the industry are provided. Data mining is a potentially useful adjunct to traditional pharmaco-vigilance methods. The results of data mining should be viewed as hypothesis generating and should be evaluated in the context of other relevant data. The availability of a publicly accessible global safety database, which is updated on a frequent basis, would further enhance detection and communication about safety issues.