Precision Health-Enabled Machine Learning to Identify Need for Wraparound Social Services Using Patient- and Population-Level Data Sets: Algorithm Development and Validation.

BACKGROUND: Emerging interest in precision health and the increasing availability of patient- and population-level data sets present considerable potential to enable analytical approaches to identify and mitigate the negative effects of social factors on health. These issues are not satisfactorily addressed in typical medical care encounters, and thus, opportunities to improve health outcomes, reduce costs, and improve coordination of care are not realized. Furthermore, methodological expertise on the use of varied patient- and population-level data sets and machine learning to predict need for supplemental services is limited. OBJECTIVE: The objective of this study was to leverage a comprehensive range of clinical, behavioral, social risk, and social determinants of health factors in order to develop decision models capable of identifying patients in need of various wraparound social services. METHODS: We used comprehensive patient- and population-level data sets to build decision models capable of predicting need for behavioral health, dietitian, social work, or other social service referrals within a safety-net health system using area under the receiver operating characteristic curve (AUROC), sensitivity, precision, F1 score, and specificity. We also evaluated the value of population-level social determinants of health data sets in improving machine learning performance of the models. RESULTS: Decision models for each wraparound service demonstrated performance measures ranging between 59.2%% and 99.3%. These results were statistically superior to the performance measures demonstrated by our previous models which used a limited data set and whose performance measures ranged from 38.2% to 88.3% (behavioural health: F1 score P<.001, AUROC P=.01; social work: F1 score P<.001, AUROC P=.03; dietitian: F1 score P=.001, AUROC P=.001; other: F1 score P=.01, AUROC P=.02); however, inclusion of additional population-level social determinants of health did not contribute to any performance improvements (behavioural health: F1 score P=.08, AUROC P=.09; social work: F1 score P=.16, AUROC P=.09; dietitian: F1 score P=.08, AUROC P=.14; other: F1 score P=.33, AUROC P=.21) in predicting the need for referral in our population of vulnerable patients seeking care at a safety-net provider. CONCLUSIONS: Precision health-enabled decision models that leverage a wide range of patient- and population-level data sets and advanced machine learning methods are capable of predicting need for various wraparound social services with good performance.

Multi-center colonoscopy quality measurement utilizing natural language processing.

BACKGROUND: An accurate system for tracking of colonoscopy quality and surveillance intervals could improve the effectiveness and cost-effectiveness of colorectal cancer (CRC) screening and surveillance. The purpose of this study was to create and test such a system across multiple institutions utilizing natural language processing (NLP). METHODS: From 42,569 colonoscopies with pathology records from 13 centers, we randomly sampled 750 paired reports. We trained (n=250) and tested (n=500) an NLP-based program with 19 measurements that encompass colonoscopy quality measures and surveillance interval determination, using blinded, paired, annotated expert manual review as the reference standard. The remaining 41,819 nonannotated documents were processed through the NLP system without manual review to assess performance consistency. The primary outcome was system accuracy across the 19 measures. RESULTS: A total of 176 (23.5%) documents with 252 (1.8%) discrepant content points resulted from paired annotation. Error rate within the 500 test documents was 31.2% for NLP and 25.4% for the paired annotators (P=0.001). At the content point level within the test set, the error rate was 3.5% for NLP and 1.9% for the paired annotators (P=0.04). When eight vaguely worded documents were removed, 125 of 492 (25.4%) were incorrect by NLP and 104 of 492 (21.1%) by the initial annotator (P=0.07). Rates of pathologic findings calculated from NLP were similar to those calculated by annotation for the majority of measurements. Test set accuracy was 99.6% for CRC, 95% for advanced adenoma, 94.6% for nonadvanced adenoma, 99.8% for advanced sessile serrated polyps, 99.2% for nonadvanced sessile serrated polyps, 96.8% for large hyperplastic polyps, and 96.0% for small hyperplastic polyps. Lesion location showed high accuracy (87.0-99.8%). Accuracy for number of adenomas was 92%. CONCLUSIONS: NLP can accurately report adenoma detection rate and the components for determining guideline-adherent colonoscopy surveillance intervals across multiple sites that utilize different methods for reporting colonoscopy findings.

Clinical decision support with natural language processing facilitates determination of colonoscopy surveillance intervals.

BACKGROUND & AIMS: With an increased emphasis on improving quality and decreasing costs, new tools are needed to improve adherence to evidence-based practices and guidelines in endoscopy. We investigated the ability of an automated system that uses natural language processing (NLP) and clinical decision support (CDS) to facilitate determination of colonoscopy surveillance intervals. METHODS: We performed a retrospective study at a single Veterans Administration medical center of patients age 40 years and older who had an index outpatient colonoscopy from 2002 through 2009 for any indication except surveillance of a previous colorectal neoplasia. We analyzed data from 10,798 reports, with 6379 linked to pathology results and 300 randomly selected reports. NLP-based CDS surveillance intervals were compared with those determined by paired, blinded, manual review. The primary outcome was adjusted agreement between manual review and the fully automated system. RESULTS: κ statistical analysis produced a value of 0.74 (P < .001) for agreement between the full text annotation and the NLP-based CDS system. Fifty-five reports (18.3%; 95% confidence interval, 14.1%-23.2%) differed between manual review and CDS recommendations. Of these, NLP error accounted for 30 (54.5%), incomplete resection of adenomatous tissue accounted for 14 (25.5%), and masses observed without biopsy findings of cancer accounted for 4 (7.2%). NLP-based CDS surveillance intervals had higher levels of agreement with the standard (81.7%) than the level agreement between experts (72% agreement between paired reviewers). CONCLUSIONS: A fully automated system that uses NLP and a guideline-based CSD system can accurately facilitate guideline-recommended adherence surveillance for colonoscopy.

Regenstrief Institute's Medical Gopher: a next-generation homegrown electronic medical record system.

OBJECTIVE: Regenstrief Institute developed one of the seminal computerized order entry systems, the Medical Gopher, for implementation at Wishard Hospital nearly three decades ago. Wishard Hospital and Regenstrief remain committed to homegrown software development, and over the past 4 years we have fully rebuilt Gopher with an emphasis on usability, safety, leveraging open source technologies, and the advancement of biomedical informatics research. Our objective in this paper is to summarize the functionality of this new system and highlight its novel features. MATERIALS AND METHODS: Applying a user-centered design process, the new Gopher was built upon a rich-internet application framework using an agile development process. The system incorporates order entry, clinical documentation, result viewing, decision support, and clinical workflow. We have customized its use for the outpatient, inpatient, and emergency department settings. RESULTS: The new Gopher is now in use by over 1100 users a day, including an average of 433 physicians caring for over 3600 patients daily. The system includes a wizard-like clinical workflow, dynamic multimedia alerts, and a familiar 'e-commerce'-based interface for order entry. Clinical documentation is enhanced by real-time natural language processing and data review is supported by a rapid chart search feature. DISCUSSION: As one of the few remaining academically developed order entry systems, the Gopher has been designed both to improve patient care and to support next-generation informatics research. It has achieved rapid adoption within our health system and suggests continued viability for homegrown systems in settings of close collaboration between developers and providers.

Natural language processing accurately categorizes findings from colonoscopy and pathology reports.

BACKGROUND & AIMS: Little is known about the ability of natural language processing (NLP) to extract meaningful information from free-text gastroenterology reports for secondary use. METHODS: We randomly selected 500 linked colonoscopy and pathology reports from 10,798 nonsurveillance colonoscopies to train and test the NLP system. By using annotation by gastroenterologists as the reference standard, we assessed the accuracy of an open-source NLP engine that processed and extracted clinically relevant concepts. The primary outcome was the highest level of pathology. Secondary outcomes were location of the most advanced lesion, largest size of an adenoma removed, and number of adenomas removed. RESULTS: The NLP system identified the highest level of pathology with 98% accuracy, compared with triplicate annotation by gastroenterologists (the standard). Accuracy values for location, size, and number were 97%, 96%, and 84%, respectively. CONCLUSIONS: The NLP can extract specific meaningful concepts with 98% accuracy. It might be developed as a method to further quantify specific quality metrics.