Automated Classification of Quality Defect Issues Relating to Substandard Medicines Using a Hybrid Machine Learning and Rule-Based Approach.

BACKGROUND AND OBJECTIVE: Substandard medicines can lead to serious safety issues affecting public health; however, the nature of such issues can be widely heterogeneous. Health product regulators seek to prioritise critical product quality defects for review to ensure that prompt risk mitigation measures are taken. This study aims to classify the nature of issues for substandard medicines using machine learning to augment a risk-based and timely review of cases. METHODS: A combined machine learning algorithm with a keyword-based model was developed to classify quality issues using text relating to substandard medicines (CISTERM). The nature of issues for product defect cases were classified based on Medical Dictionary for Regulatory Activities-Health Sciences Authority (MedDRA-HSA) lowest-level terms. RESULTS: Product defect cases received from January 2010 to December 2021 were used for training (n = 11,082) and for testing (n = 2771). The machine learning model achieved a good recall (precision) of 92% (96%) for 'Product adulterated and/or contains prohibited substance', 86% (90%) for 'Out of specification or out of trend test result' and 90% (91%) for 'Manufacturing non-compliance'. CONCLUSION: Post-market surveillance of substandard medicines remains a key activity for drug regulatory authorities. A combined machine learning algorithm with keyword-based model can help to prioritise the review of product quality defect issues in a timely manner.

A risk classification model for prioritising the management of quality issues relating to substandard medicines in Singapore.

BACKGROUND: Monitoring for substandard medicines by regulatory agencies is a key post-market surveillance activity. It is important to prioritise critical product defects for review to ensure that prompt risk mitigation actions are taken. METHODS: A regulatory risk impact prioritisation model for product defects (RISMED) with 11 factors considering the seriousness and extent of impact of a defect was developed. The model generated an overall score that categorised cases into high, medium or low impact. The model was further developed into a statistical risk scoring model (stat-RISMED) using multivariate logistic regression that classified cases into high and non-high impact. Both models were evaluated against an expert-derived gold standard annotation corpus and tested on an independent dataset. RESULTS: Product defect cases received from January 2011 to June 2020 (n = 660) were used to train stat-RISMED and cases from July 2020 to June 2021 (n = 220) for validation. The stat-RISMED identified four factors associated with high impact cases, namely defect classification based on MedDRA-HSA terms, therapeutic indication of product, detectability of defect and whether any overseas regulatory actions were performed. Compared to RISMED, stat-RISMED achieved an improved sensitivity (94% vs 42%) and positive predictive value (47% vs 43%) for the identification of high impact cases, against the gold standard labels. CONCLUSIONS: This study reported characteristics that predicts cases with high impact, and the use of a statistical model to identify such cases. The model may potentially be applied to prioritise product defect issues and strengthen overall surveillance efforts of substandard medicines.