Evaluation of ML-Based Clinical Decision Support Tool to Replace an Existing Tool in an Academic Health System: Lessons Learned.

There is increasing application of machine learning tools to problems in healthcare, with an ultimate goal to improve patient safety and health outcomes. When applied appropriately, machine learning tools can augment clinical care provided to patients. However, even if a model has impressive performance characteristics, prospectively evaluating and effectively implementing models into clinical care remains difficult. The primary objective of this paper is to recount our experiences and challenges in comparing a novel machine learning-based clinical decision support tool to legacy, non-machine learning tools addressing potential safety events in the hospitals and to summarize the obstacles which prevented evaluation of clinical efficacy of tools prior to widespread institutional use. We collected and compared safety events data, specifically patient falls and pressure injuries, between the standard of care approach and machine learning (ML)-based clinical decision support (CDS). Our assessment was limited to performance of the model rather than the workflow due to challenges in directly comparing both approaches. We did note a modest improvement in falls with ML-based CDS; however, it was not possible to determine that overall improvement was due to model characteristics.

Self-reported health parameters compared with clinician measurements: methods in practice-based research.

CONTEXT: Self-reported health data are used by health insurance companies to assess risk. Most studies show underreporting compared with clinical measurements. OBJECTIVE: To compare self-reported height, weight, blood pressure, waist circumference, and dietary intake with registered dietitian's (RD's) measures of the same parameters. DESIGN: This is a secondary analysis of data collected in a larger study on the benefits of Medical Nutrition Therapy from an RD for overweight and obese patients when provided free of charge through an insurance benefit. SETTING: Participants completed a health risk assessment survey at home, from which the self-reported measures were extracted. The clinical measurements were taken by an RD in the office during a visit for Medical Nutrition Therapy. PARTICIPANTS: Participants were 81.4% female, with a mean body mass index of 35.0. All were insured and had a least 1 visit with an RD. MAIN OUTCOME MEASURE(S): Main outcomes were correlation between self-reported and RD-measured height, weight, body mass index, blood pressure, and waist circumference. Blood pressure was categorized as normal or high and the κ statistic was used to examine category agreement between the 2 measures. Servings of food groups were compared between the 2 measures by examining cumulative percent within 0, 1, or 2 servings of the RD-measured value. RESULTS: The 2 measures of height and weight were highly correlated (0.974 and 0.986, respectively). Blood pressure was more weakly correlated and when categorized had low κ scores, as did servings of food groups. CONCLUSIONS: Height and weight were more closely correlated than in previous studies. In an insured population enrolled in a weight management program, self-reported measures may be accurate for determining program impact. Blood pressure may be better collected categorically than continuously. The necessity of food intake assessment on a risk assessment should be reconsidered.

Trust in insurers and access to physicians: associated enrollee behaviors and changes over time.

OBJECTIVE: Most studies of trust in the medical arena have focused on trust in physicians rather than trust in health insurers, and have been cross-sectional rather than longitudinal studies. This study examined associations among trust in a managed care insurer, trust in one's primary physician, and subsequent enrollee behaviors relating to source of care. The study also documents changes in trust in the study population following the disclosure of physician incentives. STUDY SETTING: A medium-sized (300,000 member) HMO, located in the southeastern United States. DATA COLLECTION: One to two years after baseline, we randomly resurveyed a quarter (n = 558) of the initial study population of a large intervention study designed to measure the impact of disclosing HMO financial incentives on patient trust. This follow-up study was also designed to measure the effects of trust on source of care. ANALYSES: Multivariate regression analyses of survey data examined associations between baseline levels of trust and subsequent enrollee behaviors such as using a non-PCP physician without a PCP referral, as well as changes in trust since baseline. RESULTS: High baseline insurer trust was associated with a lower probability of a patient seeking care from a non-PCP physician (OR = 0.55, 95 percent CI: 0.33, 0.91). No long-term effects of prior disclosure of financial incentives were observed. Overall, there was a slight increase in overall trust in the insurer (1.8 percent, p < .05) but no change in trust in one's primary physician. The increase in insurer trust was primarily restricted to 23 percent of the enrollees who had changed their PCPs following the baseline survey (6.6 percent, p < .01). In multivariate analyses, changing physicians was the most significant predictor of increased insurer trust (OR = 2.17, 95 percent CI: 1.37, 3.43). CONCLUSIONS. Trust in one's insurer seems to change over time more than trust in one's primary physician, and is predictive of enrollee behaviors such as seeking care from other physicians. The ability to change physicians seems to increase trust in the insurer.