Measurement of provider fidelity to immunization guidelines: a mixed-methods study on the feasibility of documenting patient refusals of the human papillomavirus vaccine.

BACKGROUND: Assessment and feedback is a common implementation strategy to improve healthcare provider fidelity to clinical guidelines. For immunization guidelines, fidelity is often measured with doses administered during eligible visits. Adding a patient refusal measure captures provider fidelity more completely (i.e., all instances of a provider recommending a vaccine, resulting in vaccination or refusal) and enables providers to track patient vaccine hesitancy patterns. However, many electronic health record (EHR) systems have no structured field to document multiple instances of refusals for specific vaccines, and existing billing codes for refusal are not vaccine specific. This study assessed the feasibility of a novel method for refusal documentation used in a study focused on human papillomavirus (HPV) vaccine. METHODS: An observational, descriptive-comparative, mixed-methods study design was used to conduct secondary data analysis from an implementation-effectiveness trial. The parent trial compared coach-based versus web-based practice facilitation, including assessment and feedback, to increase HPV vaccination in 21 community-based private pediatric practices. Providers were instructed to document initial HPV vaccine refusals in the EHR's immunization forms and subsequent refusals using dummy procedure codes, for use in assessment and feedback reports. This analysis examined adoption and maintenance of the refusal documentation method during eligible well visits, identified barriers and facilitators to documentation and described demographic patterns in patient refusals. RESULTS: Seven practices adopted the refusal documentation method. Among adopter practices, documented refusals started at 2.4% of eligible well visits at baseline, increased to 14.2% at the start of implementation, peaked at 24.0%, then declined to 18.8%. Barriers to refusal documentation included low prioritization, workflow integration and complication of the billing process. Facilitators included high motivation, documentation instructions and coach support. Among adopter practices, odds of refusing HPV vaccine were 25% higher for patients aged 15-17 years versus 11-12 years, and 18% lower for males versus females. CONCLUSIONS: We demonstrated the value of patient refusal documentation for measuring HPV vaccination guideline fidelity and ways that it can be improved in future research. Creation of vaccine-specific refusal billing codes or EHR adaptations to enable documenting multiple instances of specific vaccine refusals would facilitate consistent refusal documentation. Trial Registration NCT03399396 Registered in ClinicalTrials.gov on 1/16/2018.

Improving Cervical Precancer Surveillance: Validity of Claims-Based Prediction Models in ICD-9 and ICD-10 Eras.

Background: Human papillomavirus vaccine (HPV) impact on cervical precancer (cervical intraepithelial neoplasia grades 2+ [CIN2+]) is observable sooner than impact on cancer. Biopsy-confirmed CIN2+ is not included in most US cancer registries. Billing codes could provide surrogate metrics; however, the International Classification of Diseases, ninth (ICD-9) to tenth (ICD-10) transition disrupts trends. We built, validated, and compared claims-based models to identify CIN2+ events in both ICD eras. Methods: A database of Davidson County (Nashville), Tennessee, pathology-confirmed CIN2+ from the HPV Vaccine Impact Monitoring Project (HPV-IMPACT) provided gold standard events. Using Tennessee Medicaid 2008-2017, cervical diagnostic procedures (N = 8549) among Davidson County women aged 18-39 years were randomly split into 60% training and 40% testing sets. Relevant diagnosis, procedure, and screening codes were used to build models from CIN2+ tissue diagnosis codes alone, least absolute shrinkage and selection operator (LASSO), and random forest. Model-classified index events were counted to estimate incident events. Results: HPV-IMPACT identified 983 incident CIN2+ events. Models identified 1007 (LASSO), 1245 (CIN2+ tissue diagnosis codes alone), and 957 (random forest) incident events. LASSO performed well in ICD-9 and ICD-10 eras: 77.3% (95% confidence interval [CI] = 72.5% to 81.5%) vs 81.1% (95% CI = 71.5% to 88.6%) sensitivity, 93.0% (95% CI = 91.9% to 94.0%) vs 90.2% (95% CI = 87.2% to 92.7%) specificity, 61.3% (95% CI = 56.6% to 65.8%) vs 60.3% (95% CI = 51.0% to 69.1%) positive predictive value, 96.6% (95% CI = 95.8% to 97.3%) vs 96.3% (95% CI = 94.1% to 97.8%) negative predictive value, 91.0% (95% CI = 89.9% to 92.1%) vs 88.8% (95% CI = 85.9% to 91.2%) accuracy, and 85.1% (95% CI = 82.9% to 87.4%) vs 85.6% (95% CI = 81.4% to 89.9%) C-indices, respectively; performance did not statistically significantly differ between eras (95% confidence intervals all overlapped). Conclusions: Results confirmed model utility with good performance across both ICD eras for CIN2+ surveillance. Validated claims-based models may be used in future CIN2+ trend analyses to estimate HPV vaccine impact where population-based biopsies are unavailable.