Development and Evaluation of the Algorithm CErtaInty Tool (ACE-IT) to Assess Electronic Medical Record and Claims-based Algorithms' Fit for Purpose for Safety Outcomes.

INTRODUCTION: Electronic health record (EHR) or medical claims-based algorithms (i.e., operational definitions) can be used to define safety outcomes using real-world data. However, existing tools do not allow researchers and decision-makers to adequately appraise whether a particular algorithm is fit for purpose (FFP) to support regulatory decisions on drug safety surveillance. Our objective was to develop a tool to enable regulatory decision-makers and other stakeholders to appraise whether a given algorithm is FFP for a specific decision context. METHODS: We drafted a set of 77 generic items informed by regulatory guidance documents, existing instruments, and publications. The outcome of ischemic stroke served as an exemplar to inform the development of draft items. The items were designed to be outcome independent. We conducted a three-round online Delphi panel to develop and refine the tool and achieve consensus on items (> 70% agreement) among panel participants composed of regulators, researchers from pharmaceutical organizations, academic clinicians, methodologists, pharmacoepidemiologists, and cardiologists. We conducted a qualitative analysis of panel responses. Five pairs of reviewers independently evaluated two ischemic stroke algorithm validation studies to test its application. We developed a user guide, with explanation and elaboration for each item, guidance on essential and additional elements for user responses, and an illustrative example of a complete assessment. Furthermore, we conducted a 2-h online stakeholder panel of 16 participants from regulatory agencies, academic institutions, and industry. We solicited input on key factors for an FFP assessment, their general reaction to the Algorithm CErtaInty Tool (ACE-IT), limitations of the tool, and its potential use. RESULTS: The expert panel reviewed and made changes to the initial list of 77 items. The panel achieved consensus on 38 items, and the final version of the ACE-IT includes 34 items after removal of duplicate items. Applying the tool to two ischemic stroke algorithms demonstrated challenges in its application and identified shared concepts addressed by more than one item. The ACE-IT was viewed positively by the majority of stakeholders. They identified that the tool could serve as an educational resource as well as an information-sharing platform. The time required to complete the assessment was identified as an important limitation. We consolidated items with shared concepts and added a preliminary screen section and a summary assessment box based on their input. The final version of the ACE-IT is a 34-item tool for assessing whether algorithm validation studies on safety outcomes are FFP. It comprises the domains of internal validity (24 items), external validity (seven items), and ethical conduct and reporting of the validation study (three items). The internal validity domain includes sections on objectives, data sources, population, outcomes, design and setting, statistical methods, reference standard, accuracy, and strengths and limitations. The external validity domain includes items that assess the generalizability to a proposed target study. The domain on ethics and transparency includes items on ethical conduct and reporting of the validation study. CONCLUSION: The ACE-IT supports a structured, transparent, and flexible approach for decision-makers to appraise whether electronic health record or medical claims-based algorithms for safety outcomes are FFP for a specific decision context. Reliability and validity testing using a larger sample of participants in other therapeutic areas and further modifications to reduce the time needed to complete the assessment are needed to fully evaluate its utility for regulatory decision-making.

Association of SARS-CoV-2 Seropositive Antibody Test With Risk of Future Infection.

Importance: Understanding the effect of serum antibodies to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) on susceptibility to infection is important for identifying at-risk populations and could have implications for vaccine deployment. Objective: The study purpose was to evaluate evidence of SARS-CoV-2 infection based on diagnostic nucleic acid amplification test (NAAT) among patients with positive vs negative test results for antibodies in an observational descriptive cohort study of clinical laboratory and linked claims data. Design, Setting, and Participants: The study created cohorts from a deidentified data set composed of commercial laboratory tests, medical and pharmacy claims, electronic health records, and hospital chargemaster data. Patients were categorized as antibody-positive or antibody-negative according to their first SARS-CoV-2 antibody test in the database. Main Outcomes and Measures: Primary end points were post-index diagnostic NAAT results, with infection defined as a positive diagnostic test post-index, measured in 30-day intervals (0-30, 31-60, 61-90, >90 days). Additional measures included demographic, geographic, and clinical characteristics at the time of the index antibody test, including recorded signs and symptoms or prior evidence of coronavirus 2019 (COVID) diagnoses or positive NAAT results and recorded comorbidities. Results: The cohort included 3 257 478 unique patients with an index antibody test; 56% were female with a median (SD) age of 48 (20) years. Of these, 2 876 773 (88.3%) had a negative index antibody result, and 378 606 (11.6%) had a positive index antibody result. Patients with a negative antibody test result were older than those with a positive result (mean age 48 vs 44 years). Of index-positive patients, 18.4% converted to seronegative over the follow-up period. During the follow-up periods, the ratio (95% CI) of positive NAAT results among individuals who had a positive antibody test at index vs those with a negative antibody test at index was 2.85 (95% CI, 2.73-2.97) at 0 to 30 days, 0.67 (95% CI, 0.6-0.74) at 31 to 60 days, 0.29 (95% CI, 0.24-0.35) at 61 to 90 days, and 0.10 (95% CI, 0.05-0.19) at more than 90 days. Conclusions and Relevance: In this cohort study, patients with positive antibody test results were initially more likely to have positive NAAT results, consistent with prolonged RNA shedding, but became markedly less likely to have positive NAAT results over time, suggesting that seropositivity is associated with protection from infection. The duration of protection is unknown, and protection may wane over time.

Real-world data suggest antibody positivity to SARS-CoV-2 is associated with a decreased risk of future infection.

Importance There is limited evidence regarding whether the presence of serum antibodies to SARS-CoV-2 is associated with a decreased risk of future infection. Understanding susceptibility to infection and the role of immune memory is important for identifying at-risk populations and could have implications for vaccine deployment. Objective The purpose of this study was to evaluate subsequent evidence of SARS-CoV-2 infection based on diagnostic nucleic acid amplification test (NAAT) among individuals who are antibody-positive compared with those who are antibody-negative, using real-world data. Design This was an observational descriptive cohort study. Participants The study utilized a national sample to create cohorts from a de-identified dataset composed of commercial laboratory test results, open and closed medical and pharmacy claims, electronic health records, hospital billing (chargemaster) data, and payer enrollment files from the United States. Patients were indexed as antibody-positive or antibody-negative according to their first SARS-CoV-2 antibody test recorded in the database. Patients with more than 1 antibody test on the index date where results were discordant were excluded. Main Outcomes/Measures Primary endpoints were index antibody test results and post-index diagnostic NAAT results, with infection defined as a positive diagnostic test post-index, as measured in 30-day intervals (0-30, 31-60, 61-90, >90 days). Additional measures included demographic, geographic, and clinical characteristics at the time of the index antibody test, such as recorded signs and symptoms or prior evidence of COVID-19 (diagnoses or NAAT+) and recorded comorbidities. Results We included 3,257,478 unique patients with an index antibody test. Of these, 2,876,773 (88.3%) had a negative index antibody result, 378,606 (11.6%) had a positive index antibody result, and 2,099 (0.1%) had an inconclusive index antibody result. Patients with a negative antibody test were somewhat older at index than those with a positive result (mean of 48 versus 44 years). A fraction (18.4%) of individuals who were initially seropositive converted to seronegative over the follow up period. During the follow-up periods, the ratio (CI) of positive NAAT results among individuals who had a positive antibody test at index versus those with a negative antibody test at index was 2.85 (2.73 - 2.97) at 0-30 days, 0.67 (0.6 - 0.74) at 31-60 days, 0.29 (0.24 - 0.35) at 61-90 days), and 0.10 (0.05 - 0.19) at >90 days. Conclusions Patients who display positive antibody tests are initially more likely to have a positive NAAT, consistent with prolonged RNA shedding, but over time become markedly less likely to have a positive NAAT. This result suggests seropositivity using commercially available assays is associated with protection from infection. The duration of protection is unknown and may wane over time; this parameter will need to be addressed in a study with extended duration of follow up.

Amyotrophic lateral sclerosis among patients with a Medicare Advantage prescription drug plan; prevalence, survival and patient characteristics.

Objective: To estimate amyotrophic lateral sclerosis (ALS) prevalence, 5-year survival, and explore factors associated with survival in a Medicare population. Methods: A validated administrative claims algorithm was used to classify individual's ages 18-89 years at index date (first claim with a diagnosis of motor neuron disease or ALS between 1 January 2007 and 31 December 2011) with Medicare Advantage prescription drug coverage into mutually exclusive categories: ALS, no ALS, and possible ALS. Crude prevalence and cumulative survival from index date to the date of death, disenrollment or end of the study were calculated. Cox-proportional hazards were used to estimate and explore factors associated with survival. Results: Of 2631 eligible individuals, the algorithm identified 1271 (48 %), 1157 (44 %), 203 (8 %) as ALS, no ALS and possible ALS, respectively. The 5-year period prevalence and the 2011 point prevalence of ALS were 20.5 and 11.8 per 100,000, respectively. Evidence of death was documented in 81%, 35%, and 1.6% of the ALS, no ALS or possible ALS groups, respectively. Unadjusted median survival time was 388, 542 and 1473 days for the ALS, no ALS and possible ALS groups, respectively. Seeing a psychiatrist or neurologist at the index visit, having respiratory or genitourinary comorbidities, and the number of pre-index acute inpatient admissions were associated with shorter survival. Conclusions: Surveillance data from a Medicare population demonstrated a higher prevalence of ALS. Results highlight the need for effective ALS treatment options and resources for patients with ALS who will likely face limited therapeutic choices and care options at the end of life.

Understanding Prediabetes in a Medicare Advantage Population Using Data Adaptive Techniques.

The objective was to identify individuals with undiagnosed prediabetes from administrative data using adaptive techniques. The data source was a national Medicare Advantage Prescription Drug (MAPD) plan administrative data set. A retrospective, cross-sectional study developed and evaluated data adaptive logistic regression, decision tree, neural network, and ensemble predictive models for metabolic syndrome and prediabetes using 3 mutually exclusive cohorts (N = 279,903). The misclassification rate (MCR), average squared error (ASE), c-statistics, sensitivity (SN), and false positive (FP) rates were compared to select the final predictive models. MAPD individuals with continuous enrollment from 2013 to 2014 were included. Metabolic syndrome and prediabetes were defined using clinical guidelines, diagnosis, and laboratory data. A total of 512 variables identified through subject matter expertise in addition to utilizing all data available were evaluated for the modeling. The ensemble model demonstrated better discrimination (c-statistics, MCR, and ASE of 0.83, 0.24, and 0.16, respectively), high SN, and low FP rate in predicting metabolic syndrome than the individual data adaptive modeling techniques. Logistic regression demonstrated better discrimination (c-statistics, MCR, and ASE of 0.67, 0.13, and 0.11 respectively), high SN, and low FP rate in predicting prediabetes than the other adaptive modeling techniques or ensemble methods. The scored data predicted prediabetes in 44% of the MAPD population, which is comparable to 2005-2006 National Health and Nutrition Examination Survey prediabetes rates of 41%. The logistic regression model demonstrated good performance in predicting undiagnosed prediabetes in MAPD individuals.

Association of obesity with healthcare resource utilization and costs in a commercial population.

OBJECTIVE: To examine the association of obesity with healthcare resource utilization (HRU) and costs among commercially insured individuals. METHODS: This retrospective observational cohort study used administrative claims from 1 January 2007 to 1 December 2013. The ICD-9-CM status codes (V85 hierarchy) from 2008 to 2012 classified body mass index (BMI) into the World Health Organizations' BMI categories. The date of first observed BMI code was defined as the index date and continuous eligibility for one year pre- and post- index date was ensured. Post-index claims determined individuals' HRU and costs. Sampling weights developed using the entropy balance method and National Health and Nutrition Examination Survey data ensured representation of the US adult commercially insured population. Baseline characteristics were described across BMI classes and associations between BMI categories, and outcomes were examined using multivariable regression. RESULTS: The cohort included 9651 individuals with BMI V85 codes. After weighting, the BMI distribution was: normal (31.1%), overweight (33.4%), obese class I (22.0%), obese class II (8.1%) and obese class III (5.4%). Increasing BMI was associated with greater prevalence of cardiometabolic conditions, including hypertension, type 2 diabetes and metabolic syndrome. The use of antihypertensives, antihyperlipidemics, antidiabetics, analgesics and antidepressants rose with increasing BMI. Greater BMI level was associated with increased inpatient, emergency department and outpatient utilization, and higher total healthcare, medical and pharmacy costs. CONCLUSIONS: Increasing BMI was associated with higher prevalence of cardiometabolic conditions and higher HRU and costs. There is an urgent need to address the epidemic of obesity and its clinical and economic impacts.

Positive predictive value between medical-chart body-mass-index category and obesity versus codes in a claims-data warehouse.

OBJECTIVE: To evaluate the positive predictive value of claims-based V85 codes for identifying individuals with varying degrees of BMI relative to their measured BMI obtained from medical record abstraction. METHODS: This was a retrospective validation study utilizing administrative claims and medical chart data from 1 January 2009 to 31 August 2015. Randomly selected samples of patients enrolled in a Medicare Advantage Prescription Drug (MAPD) or commercial health plan and with a V85 claim were identified. The claims-based BMI category (underweight, normal weight, overweight, obese class I-III) was determined via corresponding V85 codes and compared to the BMI category derived from chart abstracted height, weight and/or BMI. The positive predictive values (PPVs) of the claims-based BMI categories were calculated with the corresponding 95% confidence intervals (CIs). RESULTS: The overall PPVs (95% CIs) in the MAPD and commercial samples were 90.3% (86.3%-94.4%) and 91.1% (87.3%-94.9%), respectively. In each BMI category, the PPVs (95% CIs) for the MAPD and commercial samples, respectively, were: underweight, 71.0% (55.0%-87.0%) and 75.9% (60.3%-91.4%); normal, 93.8% (85.4%-100%) and 87.8% (77.8%-97.8%); overweight, 97.4% (92.5%-100%) and 93.5% (84.9%-100%); obese class I, 96.9 (90.9%-100%) and 97.2% (91.9%-100%); obese class II, 97.0% (91.1%-100%) and 93.0% (85.4%-100%); and obese class III, 85.0% (73.3%-96.1%) and 97.1% (91.4%-100%). CONCLUSIONS: BMI categories derived from administrative claims, when available, can be used successfully particularly in the context of obesity research.

A comparison of entropy balance and probability weighting methods to generalize observational cohorts to a population: a simulation and empirical example.

PURPOSE: We compared methods to control bias and confounding in observational studies including inverse probability weighting (IPW) and stabilized IPW (sIPW). These methods often require iteration and post-calibration to achieve covariate balance. In comparison, entropy balance (EB) optimizes covariate balance a priori by calibrating weights using the target's moments as constraints. METHODS: We measured covariate balance empirically and by simulation by using absolute standardized mean difference (ASMD), absolute bias (AB), and root mean square error (RMSE), investigating two scenarios: the size of the observed (exposed) cohort exceeds the target (unexposed) cohort and vice versa. The empirical application weighted a commercial health plan cohort to a nationally representative National Health and Nutrition Examination Survey target on the same covariates and compared average total health care cost estimates across methods. RESULTS: Entropy balance alone achieved balance (ASMD ≤ 0.10) on all covariates in simulation and empirically. In simulation scenario I, EB achieved the lowest AB and RMSE (13.64, 31.19) compared with IPW (263.05, 263.99) and sIPW (319.91, 320.71). In scenario II, EB outperformed IPW and sIPW with smaller AB and RMSE. In scenarios I and II, EB achieved the lowest mean estimate difference from the simulated population outcome ($490.05, $487.62) compared with IPW and sIPW, respectively. Empirically, only EB differed from the unweighted mean cost indicating IPW, and sIPW weighting was ineffective. CONCLUSION: Entropy balance demonstrated the bias-variance tradeoff achieving higher estimate accuracy, yet lower estimate precision, compared with IPW methods. EB weighting required no post-processing and effectively mitigated observed bias and confounding. Copyright © 2016 John Wiley & Sons, Ltd.

The Influence of Respondent Characteristics on the Validity of Self-Reported Survey Responses.

OBJECTIVE: To examine concordance between member self-reports and the organization's administrative claims data for two key health factors: number of chronic conditions, and number of prescription drugs. DATA: Medicare Advantage plan claims data and member survey data from 2011 to 2012. DESIGN: Mailed surveys to 15,000 members, enrolled minimum 6 months, drawn from a random sample of primary care physician practices with at least 200 members. METHODS: Descriptive statistics were generated for extent of concordance. Multivariable logistic regressions were used to analyze the association of selected respondent characteristics with likelihood of concordance. FINDINGS: Concordance for number of chronic conditions was 58.4 percent, with 27.3 percent under-reporting, 14.2 percent over-reporting. Concordance for number of prescription drugs was 56.6 percent with 38.9 percent under-reporting, 4.5 percent over-reporting. Number of prescriptions and assistance in survey completion were associated with higher likelihood of concordance for chronic conditions. Assistance in survey completion and number of chronic conditions were associated with higher concordance, and age and number of prescriptions were associated with lower concordance, for prescription drugs. CONCLUSIONS: Self-reported number of chronic conditions and prescription medications are not in high concordance with claims data. Health care researchers and policy makers using patient self-reported data should be aware of these potential biases.

Over-the-counter decorative contact lenses: Cosmetic or Medical Devices? A Case Series.

PURPOSE: To illustrate the implications of the unauthorized sale and unmonitored wearing of decorative cosmetic contact lenses resulting in ocular complications and to determine the prevalence of the use of cosmetic contact lenses obtained from unlicensed providers by adolescents. METHODS: Observational case report, structured interview, and survey in a retrospective, observational, clinical practice setting. Reported sources of contact lenses were categorized as provider and nonprovider, and associations within the data were reviewed by using a Pearson correlation and chi-square test. RESULTS: Twelve patients (eight female and four male) were seen urgently for acute eye pain and redness after wearing plano decorative contact lenses. None of the patients had previously worn a contact lens. None of the lenses were dispensed by eye care professionals. Four patients developed blinding infections requiring hospital admission. Causative organisms included staphylococci, Pseudomonas, and acanthamoeba. One patient required a penetrating keratoplasty. One hundred fifty-nine patients were surveyed. Thirty-seven (23%) used decorative contact lenses. Lenses were obtained from an unlicensed provider 51% of the time. Education about lens care and handling was significantly associated with acquiring lenses from licensed providers (R = 0.74, P < 0.0001). CONCLUSIONS: Colored noncorrective contact lenses are being dispensed without a prescription or fitting from unlicensed vendors. Patients who acquire lenses from unauthorized providers are significantly less likely to be instructed on appropriate lens use and care. Consequently, uninformed lens wearers are experiencing acute vision-threatening infections and inflammation.