Monitoring Adverse Drug Events in Web Forums: Evaluation of a Pipeline and Use Case Study.

BACKGROUND: To mitigate safety concerns, regulatory agencies must make informed decisions regarding drug usage and adverse drug events (ADEs). The primary pharmacovigilance data stem from spontaneous reports by health care professionals. However, underreporting poses a notable challenge within the current system. Explorations into alternative sources, including electronic patient records and social media, have been undertaken. Nevertheless, social media's potential remains largely untapped in real-world scenarios. OBJECTIVE: The challenge faced by regulatory agencies in using social media is primarily attributed to the absence of suitable tools to support decision makers. An effective tool should enable access to information via a graphical user interface, presenting data in a user-friendly manner rather than in their raw form. This interface should offer various visualization options, empowering users to choose representations that best convey the data and facilitate informed decision-making. Thus, this study aims to assess the potential of integrating social media into pharmacovigilance and enhancing decision-making with this novel data source. To achieve this, our objective was to develop and assess a pipeline that processes data from the extraction of web forum posts to the generation of indicators and alerts within a visual and interactive environment. The goal was to create a user-friendly tool that enables regulatory authorities to make better-informed decisions effectively. METHODS: To enhance pharmacovigilance efforts, we have devised a pipeline comprising 4 distinct modules, each independently editable, aimed at efficiently analyzing health-related French web forums. These modules were (1) web forums' posts extraction, (2) web forums' posts annotation, (3) statistics and signal detection algorithm, and (4) a graphical user interface (GUI). We showcase the efficacy of the GUI through an illustrative case study involving the introduction of the new formula of Levothyrox in France. This event led to a surge in reports to the French regulatory authority. RESULTS: Between January 1, 2017, and February 28, 2021, a total of 2,081,296 posts were extracted from 23 French web forums. These posts contained 437,192 normalized drug-ADE couples, annotated with the Anatomical Therapeutic Chemical (ATC) Classification and Medical Dictionary for Regulatory Activities (MedDRA). The analysis of the Levothyrox new formula revealed a notable pattern. In August 2017, there was a sharp increase in posts related to this medication on social media platforms, which coincided with a substantial uptick in reports submitted by patients to the national regulatory authority during the same period. CONCLUSIONS: We demonstrated that conducting quantitative analysis using the GUI is straightforward and requires no coding. The results aligned with prior research and also offered potential insights into drug-related matters. Our hypothesis received partial confirmation because the final users were not involved in the evaluation process. Further studies, concentrating on ergonomics and the impact on professionals within regulatory agencies, are imperative for future research endeavors. We emphasized the versatility of our approach and the seamless interoperability between different modules over the performance of individual modules. Specifically, the annotation module was integrated early in the development process and could undergo substantial enhancement by leveraging contemporary techniques rooted in the Transformers architecture. Our pipeline holds potential applications in health surveillance by regulatory agencies or pharmaceutical companies, aiding in the identification of safety concerns. Moreover, it could be used by research teams for retrospective analysis of events.

Users' Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts.

BACKGROUND: Within a few months, the COVID-19 pandemic had spread to many countries and had been a real challenge for health systems all around the world. This unprecedented crisis has led to a surge of online discussions about potential cures for the disease. Among them, vaccines have been at the heart of the debates and have faced lack of confidence before marketing in France. OBJECTIVE: This study aims to identify and investigate the opinions of French Twitter users on the announced vaccines against COVID-19 through sentiment analysis. METHODS: This study was conducted in 2 phases. First, we filtered a collection of tweets related to COVID-19 available on Twitter from February 2020 to August 2020 with a set of keywords associated with vaccine mistrust using word embeddings. Second, we performed sentiment analysis using deep learning to identify the characteristics of vaccine mistrust. The model was trained on a hand-labeled subset of 4548 tweets. RESULTS: A set of 69 relevant keywords were identified as the semantic concept of the word "vaccin" (vaccine in French) and focused mainly on conspiracies, pharmaceutical companies, and alternative treatments. Those keywords enabled us to extract nearly 350,000 tweets in French. The sentiment analysis model achieved 0.75 accuracy. The model then predicted 16% of positive tweets, 41% of negative tweets, and 43% of neutral tweets. This allowed us to explore the semantic concepts of positive and negative tweets and to plot the trends of each sentiment. The main negative rhetoric identified from users' tweets was that vaccines are perceived as having a political purpose and that COVID-19 is a commercial argument for the pharmaceutical companies. CONCLUSIONS: Twitter might be a useful tool to investigate the arguments for vaccine mistrust because it unveils political criticism contrasting with the usual concerns on adverse drug reactions. As the opposition rhetoric is more consistent and more widely spread than the positive rhetoric, we believe that this research provides effective tools to help health authorities better characterize the risk of vaccine mistrust.

Implementing a Microservices Architecture for Predicting the Opinion of Twitter Users on COVID Vaccines.

A strong trend in the software industry is to merge the activities of deployment and operationalization through the DevOps approach, which in the case of artificial intelligence is called Machine Learning Operations (MLOps). We present here a microservices architecture containing the whole pipeline (frontend, backend, data predictions) hosted in Docker containers which exposes a model implemented for opinion prediction in Twitter on the COVID vaccines. This is the first description in the literature of implementing a microservice architecture using TorchServe, a library for serving Pytorch models.

Identification of COVID-19 Vaccines Concerns in Health-Related French Web Forums: A Topic Modelling Approach.

Since December 2019 and the first reported cases of COVID-19 in Wuhan, China, there have been 199,466,211 confirmed cases of COVID-19 in the World. The WHO defined vaccination hesitancy as one of the top ten threats to global health in 2019. Our objective was thus to identify topics and trends about COVID-19 vaccines from French web forums to understand the perception of the French population on these vaccines before the vaccination campaign started. We performed a topic model analysis on 485 web forums' posts. 10 topics were found. We reviewed 120 posts from 6 of these 10 topics. One topic was about vaccine hesitancy, refusal, and mistrust, and two topics were related to what the users think about the government, the political and economic choices made towards this epidemic.

Visualising Patterns Associated with Adverse Drug Reactions in French Forums.

As social media are an interesting source of information for pharmacovigilance, we implemented a novel visualisation method for pharmacovigilance specialists applied to French discussion forums. A word embedding model was trained on posts to facilitate the identification of patterns associated with adverse drug reactions.

How to interact with medical terminologies? Formative usability evaluations comparing three approaches for supporting the use of MedDRA by pharmacovigilance specialists.

BACKGROUND: Medical terminologies are commonly used in medicine. For instance, to answer a pharmacovigilance question, pharmacovigilance specialists (PVS) search in a pharmacovigilance database for reports in relation to a given drug. To do that, they first need to identify all MedDRA terms that might have been used to code an adverse reaction in the database, but terms may be numerous and difficult to select as they may belong to different parts of the hierarchy. In previous studies, three tools have been developed to help PVS identify and group all relevant MedDRA terms using three different approaches: forms, structured query-builder, and icons. Yet, a poor usability of the tools may increase PVS' workload and reduce their performance. This study aims to evaluate, compare and improve the three tools during two rounds of formative usability evaluation. METHODS: First, a cognitive walkthrough was performed. Based on the design recommendations obtained from this evaluation, designers made modifications to their tools to improve usability. Once this re-engineering phase completed, six PVS took part in a usability test: difficulties, errors and verbalizations during their interaction with the three tools were collected. Their satisfaction was measured through the System Usability Scale. The design recommendations issued from the tests were used to adapt the tools. RESULTS: All tools had usability problems related to the lack of guidance in the graphical user interface (e.g., unintuitive labels). In two tools, the use of the SNOMED CT to find MedDRA terms hampered their use because French PVS were not used to it. For the most obvious and common terms, the icons-based interface would appear to be more useful. For the less frequently used MedDRA terms or those distributed in different parts of the hierarchy, the structured query-builder would be preferable thanks to its great power and flexibility. The form-based tool seems to be a compromise. CONCLUSION: These evaluations made it possible to identify the strengths of each tool but also their weaknesses to address them before further evaluation. Next step is to assess the acceptability of tools and the expressiveness of their results to help identify and group MedDRA terms.

Discrepancy Between Personal Experience and Negative Opinion with Human Papillomavirus Vaccine in Web Forums.

While vaccines are intended to protect people from infectious diseases, public confidence in vaccination has evolved as patients have reservation about vaccination, with a major concern about its safety. Social media may help regulatory authorities to better understand opposition to vaccination and make informed decisions for better promotion of vaccines' benefits towards the public. Our objective was to explore French web forums for potential pharmacovigilance signals associated with human papillomavirus infections (HPV) vaccines. Among 138 posts associated with a signal randomly chosen for manual review, 29% were actually adverse drug reactions to the vaccine described in clinical studies, and only 2 were personal experiences. Only 14% of the reviewed posts described positive opinion about the vaccine whereas 46% were neutral and 40% were negative. While few personal experiences of adverse reactions were actually reported by users, our case study showed a large proportion of negative opinions.

Use of Social Media for Pharmacovigilance Activities: Key Findings and Recommendations from the Vigi4Med Project.

The large-scale use of social media by the population has gained the attention of stakeholders and researchers in various fields. In the domain of pharmacovigilance, this new resource was initially considered as an opportunity to overcome underreporting and monitor the safety of drugs in real time in close connection with patients. Research is still required to overcome technical challenges related to data extraction, annotation, and filtering, and there is not yet a clear consensus concerning the systematic exploration and use of social media in pharmacovigilance. Although the literature has mainly considered signal detection, the potential value of social media to support other pharmacovigilance activities should also be explored. The objective of this paper is to present the main findings and subsequent recommendations from the French research project Vigi4Med, which evaluated the use of social media, mainly web forums, for pharmacovigilance activities. This project included an analysis of the existing literature, which contributed to the recommendations presented herein. The recommendations are categorized into three categories: ethical (related to privacy, confidentiality, and follow-up), qualitative (related to the quality of the information), and quantitative (related to statistical analysis). We argue that the progress in information technology and the societal need to consider patients' experiences should motivate future research on social media surveillance for the reinforcement of classical pharmacovigilance.

Pharmacology and social media: Potentials and biases of web forums for drug mention analysis-case study of France.

The aim of this study is to analyze drug mentions in web forums to evaluate the utility of this data source for drug post-marketing studies. We automatically annotated over 60 million posts extracted from 21 French web forums. Drug mentions detected in this corpus were matched to drug names in a French drug database (Theriaque®). Our analysis showed that a high proportion of the most frequent drug mentions in the selected web forums correspond to drugs that are usually prescribed to young women, such as combined oral contraceptives. The most mentioned drugs in our corpus correlated weakly to the most prescribed drugs in France but seemed to be influenced by events widely reported in traditional media. In this article, we conclude that web forums have high potential for post-marketing drug-related studies, such as pharmacovigilance, and observation of drug utilization. However, the bias related to forum selection and the corresponding population representativeness should always be taken into account.

Qualitative and Quantitative Analysis of Web Forums for Adverse Events Detection: "Strontium Ranelate" Case Study.

Social media are proposed as a complementary data source for detection and characterisation of adverse drug reactions. While signal detection algorithms were implemented for generating signals in pharmacovigilance databases, the implementation of a graphical user interface supporting the selection and display of algorithms' results is not documented in the medical literature. Although collecting information on the chronology and the impact of adverse drug reactions is desirable to enable causality and quality assessment of potential signals detected in patients' posts, no tool has been proposed yet to consider such data. We describe here two approaches, and the corresponding tools we implemented for: (1) quantitative approach based on signal detection algorithms, and (2) qualitative approach based on expert review of patient's posts. Future work will focus on implementing other statistical methods, exploring the complementarity of both approaches on a larger scale, and prioritizing the posts to manually evaluate after applying appropriate signal detection methods.

Automated Control of Codes Accuracy in Case-Mix Databases by Evaluating Coherence with Available Information in the Electronic Health Record.

Coding accuracy in case-mix databases enables efficient funding of health facilities and accurate epidemiological statistics based on patients' stays information. We assume that the data collected in the electronic health record, especially drug prescriptions and medical reports are relevant for checking the consistency of the coding of diagnoses. We evaluated a new coding control tool, "TOLBIAC control", embedded in the Web100T coding assistant. This tool interacts with the Vidal Application Programming Interface and the electronic health record of the University Hospital of Saint-Etienne. The micro-average F-measure was 0.76 for drug prescriptions and 0.55 for free text medical reports. This initial evaluation has revealed that drug prescriptions in EHRs can successfully be used to develop an automated ICD-10 code-control tool. Nevertheless the "TOLBIAC control" tool is not yet fully effective for widespread use because of its limited performance in text analysis, a feature that is currently undergoing improvements.

Descriptions of Adverse Drug Reactions Are Less Informative in Forums Than in the French Pharmacovigilance Database but Provide More Unexpected Reactions.

Background: Social media have drawn attention for their potential use in Pharmacovigilance. Recent work showed that it is possible to extract information concerning adverse drug reactions (ADRs) from posts in social media. The main objective of the Vigi4MED project was to evaluate the relevance and quality of the information shared by patients on web forums about drug safety and its potential utility for pharmacovigilance. Methods: After selecting websites of interest, we manually evaluated the relevance of the content of posts for pharmacovigilance related to six drugs (agomelatine, baclofen, duloxetine, exenatide, strontium ranelate, and tetrazepam). We compared forums to the French Pharmacovigilance Database (FPVD) to (1) evaluate whether they contained relevant information to characterize a pharmacovigilance case report (patient's age and sex; treatment indication, dose and duration; time-to-onset (TTO) and outcome of the ADR, and drug dechallenge and rechallenge) and (2) perform impact analysis (nature, seriousness, unexpectedness, and outcome of the ADR). Results: The cases in the FPVD were significantly more informative than posts in forums for patient description (age, sex), treatment description (dose, duration, TTO), and outcome of the ADR, but the indication for the treatment was more often found in forums. Cases were more often serious in the FPVD than in forums (46% vs. 4%), but forums more often contained an unexpected ADR than the FPVD (24% vs. 17%). Moreover, 197 unexpected ADRs identified in forums were absent from the FPVD and the distribution of the MedDRA System Organ Classes (SOCs) was different between the two data sources. Discussion: This study is the first to evaluate if patients' posts may qualify as potential and informative case reports that should be stored in a pharmacovigilance database in the same way as case reports submitted by health professionals. The posts were less informative (except for the indication) and focused on less serious ADRs than the FPVD cases, but more unexpected ADRs were presented in forums than in the FPVD and their SOCs were different. Thus, web forums should be considered as a secondary, but complementary source for pharmacovigilance.

Signal Detection for Baclofen in Web Forums: A Preliminary Study.

Web forums are proposed as a new complementary source of knowledge to spontaneous reports by patients and healthcare professionals due to underreporting of adverse drug reactions (ADRs). Some authors suggest that signal detection could be a convenient method for gathering mentions of ADRs in patients' posts. Signal detection methods were proposed to mine pharmacovigilance databases, but little is known about their applicability to web forums. We describe a method implementing several traditional decision rules on signal detection with baclofen applied to a set of more than 6 million posts. We then cross-validated four unexpected signals applying a logistic regression method. Most adverse effects (AEs) described in the summary of product characteristics of baclofen were detected by signal detection methods. Some unexpected AEs were too. Therefore, web forums are confirmed as a complementary resource for improving current knowledge in pharmacovigilance by detecting unexpected adverse drug reactions.

Vigi4Med Scraper: A Framework for Web Forum Structured Data Extraction and Semantic Representation.

The extraction of information from social media is an essential yet complicated step for data analysis in multiple domains. In this paper, we present Vigi4Med Scraper, a generic open source framework for extracting structured data from web forums. Our framework is highly configurable; using a configuration file, the user can freely choose the data to extract from any web forum. The extracted data are anonymized and represented in a semantic structure using Resource Description Framework (RDF) graphs. This representation enables efficient manipulation by data analysis algorithms and allows the collected data to be directly linked to any existing semantic resource. To avoid server overload, an integrated proxy with caching functionality imposes a minimal delay between sequential requests. Vigi4Med Scraper represents the first step of Vigi4Med, a project to detect adverse drug reactions (ADRs) from social networks founded by the French drug safety agency Agence Nationale de Sécurité du Médicament (ANSM). Vigi4Med Scraper has successfully extracted greater than 200 gigabytes of data from the web forums of over 20 different websites.