A Comparison of Logistic Regression Against Machine Learning Algorithms for Gastric Cancer Risk Prediction Within Real-World Clinical Data Streams.

PURPOSE: Noncardia gastric cancer (NCGC) is a leading cause of global cancer mortality, and is often diagnosed at advanced stages. Development of NCGC risk models within electronic health records (EHR) may allow for improved cancer prevention. There has been much recent interest in use of machine learning (ML) for cancer prediction, but few studies comparing ML with classical statistical models for NCGC risk prediction. METHODS: We trained models using logistic regression (LR) and four commonly used ML algorithms to predict NCGC from age-/sex-matched controls in two EHR systems: Stanford University and the University of Washington (UW). The LR model contained well-established NCGC risk factors (intestinal metaplasia histology, prior Helicobacter pylori infection, race, ethnicity, nativity status, smoking history, anemia), whereas ML models agnostically selected variables from the EHR. Models were developed and internally validated in the Stanford data, and externally validated in the UW data. Hyperparameter tuning of models was achieved using cross-validation. Model performance was compared by accuracy, sensitivity, and specificity. RESULTS: In internal validation, LR performed with comparable accuracy (0.732; 95% CI, 0.698 to 0.764), sensitivity (0.697; 95% CI, 0.647 to 0.744), and specificity (0.767; 95% CI, 0.720 to 0.809) to penalized lasso, support vector machine, K-nearest neighbor, and random forest models. In external validation, LR continued to demonstrate high accuracy, sensitivity, and specificity. Although K-nearest neighbor demonstrated higher accuracy and specificity, this was offset by significantly lower sensitivity. No ML model consistently outperformed LR across evaluation criteria. CONCLUSION: Drawing data from two independent EHRs, we find LR on the basis of established risk factors demonstrated comparable performance to optimized ML algorithms. This study demonstrates that classical models built on robust, hand-chosen predictor variables may not be inferior to data-driven models for NCGC risk prediction.

Predicting need for advanced illness or palliative care in a primary care population using electronic health record data.

Timely outreach to individuals in an advanced stage of illness offers opportunities to exercise decision control over health care. Predictive models built using Electronic health record (EHR) data are being explored as a way to anticipate such need with enough lead time for patient engagement. Prior studies have focused on hospitalized patients, who typically have more data available for predicting care needs. It is unclear if prediction driven outreach is feasible in the primary care setting. In this study, we apply predictive modeling to the primary care population of a large, regional health system and systematically examine the impact of technical choices, such as requiring a minimum number of health care encounters (data density requirements) and aggregating diagnosis codes using Clinical Classifications Software (CCS) groupings to reduce dimensionality, on model performance in terms of discrimination and positive predictive value. We assembled a cohort of 349,667 primary care patients between 65 and 90 years of age who sought care from Sutter Health between July 1, 2011 and June 30, 2014, of whom 2.1% died during the study period. EHR data comprising demographics, encounters, orders, and diagnoses for each patient from a 12 month observation window prior to the point when a prediction is made were extracted. L1 regularized logistic regression and gradient boosted tree models were fit to training data and tuned by cross validation. Model performance in predicting one year mortality was assessed using held-out test patients. Our experiments systematically varied three factors: model type, diagnosis coding, and data density requirements. We found substantial, consistent benefit from using gradient boosting vs logistic regression (mean AUROC over all other technical choices of 84.8% vs 80.7% respectively). There was no benefit from aggregation of ICD codes into CCS code groups (mean AUROC over all other technical choices of 82.9% vs 82.6% respectively). Likewise increasing data density requirements did not affect discrimination (mean AUROC over other technical choices ranged from 82.5% to 83%). We also examine model performance as a function of lead time, which is the interval between death and when a prediction was made. In subgroup analysis by lead time, mean AUROC over all other choices ranged from 87.9% for patients who died within 0 to 3 months to 83.6% for those who died 9 to 12 months after prediction time.

Perioperative serum creatinine changes and ureteral injury.

PURPOSE: To illustrate a simple method that screens for ureteral injury in the acute postoperative period after urogynecologic surgeries. METHODS: Serum creatinine measurements in the preoperative (baseline) and postoperative periods of urogynecologic surgeries were determined and the correlation of the change to ureteral injury and/or obstruction analyzed. The sample size calculation showed 7 cases and 28 controls were sufficient to detect significant changes in creatinine. Each of the seven cases was matched for age and type of surgery with a control patient in a 1:4 ratio following standard protocol. RESULTS: Chart review of patients (273 cases) undergoing urogynecologic surgeries from October 2009 to June 2014 were undertaken. There were 7 cases of ureteral injury and 28 matching control cases. All cases had intraoperative cystoscopy confirming bilateral ureteral flow. In the ureteral injury group, blockage of ureter was confirmed by CT scan with IV contrast. There was a 59.8% increase in serum creatinine levels postoperative in the ureteral injury group versus a 3.8% decrease in controls. A difference of creatinine levels greater than or equal to 0.3 mg/dL over baseline was evident in ureteral injury cases. CONCLUSION: A small change in serum creatinine level over baseline after urogynecologic surgery alerted the possibility of ureteral injury or obstruction. A simple and inexpensive evaluation of perioperative creatinine levels can promptly diagnose ureteral damage in the acute postoperative period for gynecologic reconstructive surgeries.

Accommodating diverse clients' needs in evaluation capacity building: a case study of the Tobacco Control Evaluation Center.

In this case study, we detail and analyze how the Tobacco Control Evaluation Center (TCEC), an evaluation technical assistance center that serves approximately 100 local tobacco control organizations in California, endeavors to build capacity among the state-funded local providers it serves by using evaluation capacity building activities with an utilization-focused evaluation framework. We call this a "blended approach" and describe these methods. Satisfaction and demand for TCEC services are documented to provide measurements for evaluation capacity building. Final evaluation report scores from two intervention cycles (2004-2007 and 2007-2010) submitted to the California Health Department, Tobacco Control Division are also assessed and compared. These measures demonstrate an increase in evaluation capacity by local projects under TCEC's purview.

Building the evaluation capacity of California's local tobacco control programs.

Successful evaluation capacity building requires a dynamic balance between responding to local agency needs and ensuring that local staff have appropriate skills to conduct rigorous evaluations. In 2004, the California Tobacco Control Program established the Tobacco Control Evaluation Center (TCEC), based at a public research university, to provide evaluation technical assistance to approximately 100 local agencies implementing tobacco control programs. TCEC has been responsive to local needs, for instance, by answering 512 technical assistance requests in the first 5 years of operation and by tailoring training according to needs assessment results. About 50% of the technical assistance requests were for new data collection instruments (n = 255). TCEC has sought proactively to improve local evaluation skills, most recently in a data analysis and report writing skill building campaign that included a webinar, newsletter, and seven regional training meetings. Preliminary analysis suggests a 20% improvement in scores for the local final evaluation reports as a result of this campaign. It is concluded that evaluation technical assistance can be provided effectively by a university as long as the local context is kept in mind, and a balance of responsive and proactive technical assistance is provided.