RESUMO
Background: Despite restrictive opioid management guidelines, opioid use disorder (OUD) remains a major public health concern. Machine learning (ML) offers a promising avenue for identifying and alerting clinicians about OUD, thus supporting better clinical decision-making regarding treatment. Objective: This study aimed to assess the clinical validity of an ML application designed to identify and alert clinicians of different levels of OUD risk by comparing it to a structured review of medical records by clinicians. Methods: The ML application generated OUD risk alerts on outpatient data for 649,504 patients from 2 medical centers between 2010 and 2013. A random sample of 60 patients was selected from 3 OUD risk level categories (n=180). An OUD risk classification scheme and standardized data extraction tool were developed to evaluate the validity of the alerts. Clinicians independently conducted a systematic and structured review of medical records and reached a consensus on a patient's OUD risk level, which was then compared to the ML application's risk assignments. Results: A total of 78,587 patients without cancer with at least 1 opioid prescription were identified as follows: not high risk (n=50,405, 64.1%), high risk (n=16,636, 21.2%), and suspected OUD or OUD (n=11,546, 14.7%). The sample of 180 patients was representative of the total population in terms of age, sex, and race. The interrater reliability between the ML application and clinicians had a weighted kappa coefficient of 0.62 (95% CI 0.53-0.71), indicating good agreement. Combining the high risk and suspected OUD or OUD categories and using the review of medical records as a gold standard, the ML application had a corrected sensitivity of 56.6% (95% CI 48.7%-64.5%) and a corrected specificity of 94.2% (95% CI 90.3%-98.1%). The positive and negative predictive values were 93.3% (95% CI 88.2%-96.3%) and 60.0% (95% CI 50.4%-68.9%), respectively. Key themes for disagreements between the ML application and clinician reviews were identified. Conclusions: A systematic comparison was conducted between an ML application and clinicians for identifying OUD risk. The ML application generated clinically valid and useful alerts about patients' different OUD risk levels. ML applications hold promise for identifying patients at differing levels of OUD risk and will likely complement traditional rule-based approaches to generating alerts about opioid safety issues.
RESUMO
Objectives: To assess the PharmD student's experiences about end year clerkship evaluation conducted using Objective Structured Clinical Exam (OSCE) format at JSS College of Pharmacy, Mysuru, India. Methods: The student's experiences were captured using a newly developed, 14-item, 5-point Likert's scale feedback form. The results were analyzed and presented descriptively. The Mann-Whitney U test was used to compare the Likert's scale responses between the sex, entry level for PharmD and performance in the end year exam, whereas the Spearman's rank correlation coefficient test was used to measure the strength of association between the Likert's' scale responses and these variables. A p value of <0.05 was considered statistically significant. Results: Thirty-seven students of fifth year PharmD attended the end year clerkship evaluation and provided their feedback. Out of the 14-items, the most frequent response in eleven items was strongly agree and in three items it was agree. The Mann-Whitney U test revealed statistically significant differences between regular and postbaccalaureate students with respect to Likert's scale responses in all the domains (p < 0.02). The Spearman's rank correlation test revealed no association between the students' performance and their experiences with OSCE as an assessment tool for the end year clerkship exam. Conclusions: The study results demonstrate that OSCE is an alternative and preferred method of evaluating the clinical skills and competencies of fifth year PharmD students in their end year clerkship exam in India. Innovation: For the first time in India, the JSS College of Pharmacy, Mysuru, had successfully implemented the OSCE method for evaluating PharmD students' clerkship in their end year exam and had assessed their experiences about OSCE as an assessment tool.
