Machine learning to develop a predictive model of pressure injury in persons with spinal cord injury.

STUDY DESIGN: A 5-year longitudinal, retrospective, cohort study. OBJECTIVES: Develop a prediction model based on electronic health record (EHR) data to identify veterans with spinal cord injury/diseases (SCI/D) at highest risk for new pressure injuries (PIs). SETTING: Structured (coded) and text EHR data, for veterans with SCI/D treated in a VHA SCI/D Center between October 1, 2008, and September 30, 2013. METHODS: A total of 4709 veterans were available for analysis after randomly selecting 175 to act as a validation (gold standard) sample. Machine learning models were created using ten-fold cross validation and three techniques: (1) two-step logistic regression; (2) regression model employing adaptive LASSO; (3) and gradient boosting. Models based on each method were compared using area under the receiver-operating curve (AUC) analysis. RESULTS: The AUC value for the gradient boosting model was 0.62 (95% CI = 0.54-0.70), for the logistic regression model it was 0.67 (95% CI = 0.59-0.75), and for the adaptive LASSO model it was 0.72 (95% CI = 0.65-80). Based on these results, the adaptive LASSO model was chosen for interpretation. The strongest predictors of new PI cases were having fewer total days in the hospital in the year before the annual exam, higher vs. lower weight and most severe vs. less severe grade of injury based on the American Spinal Cord Injury Association (ASIA) Impairment Scale. CONCLUSIONS: While the analyses resulted in a potentially useful predictive model, clinical implications were limited because modifiable risk factors were absent in the models.

Comparing Outcomes of the Veterans Health Administration's Traumatic Brain Injury and Mental Health Screening Programs: Types and Frequency of Specialty Services Used.

The Veterans Health Administration (VHA) screens veterans who deployed in support of the wars in Afghanistan and Iraq for traumatic brain injury (TBI) and mental health (MH) disorders. Chronic symptoms after mild TBI overlap with MH symptoms, for which there are already established screens within the VHA. It is unclear whether the TBI screen facilitates treatment for appropriate specialty care over and beyond the MH screens. Our primary objective was to determine whether TBI screening is associated with different types (MH, Physical Medicine & Rehabilitation [PM&R], and Neurology) and frequency of specialty services compared with the MH screens. A retrospective cohort design examined veterans receiving VHA care who were screened for both TBI and MH disorders between Fiscal Year (FY) 2007 and FY 2018 (N = 241,136). We calculated service utilization counts in MH, PM&R, and Neurology in the six months after the screens. Zero-inflated negative binomial regression models of encounters (counts) were fit separately by specialty care type and for a total count of specialty services. We found that screening positive for TBI resulted in 2.38 times more specialty service encounters than screening negative for TBI. Compared with screening positive for MH only, screening positive for both MH and TBI resulted in 1.78 times more specialty service encounters and 1.33 times more MH encounters. The TBI screen appears to increase use of MH, PM&R, and Neurology services for veterans with post-deployment health concerns, even in those also identified as having a possible MH disorder.

Measuring pain care quality in the Veterans Health Administration primary care setting.

ABSTRACT: The lack of a reliable approach to assess quality of pain care hinders quality improvement initiatives. Rule-based natural language processing algorithms were used to extract pain care quality (PCQ) indicators from documents of Veterans Health Administration primary care providers for veterans diagnosed within the past year with musculoskeletal disorders with moderate-to-severe pain intensity across 2 time periods 2013 to 2014 (fiscal year [FY] 2013) and 2017 to 2018 (FY 2017). Patterns of documentation of PCQ indicators for 64,444 veterans and 124,408 unique visits (FY 2013) and 63,427 veterans and 146,507 visits (FY 2017) are described. The most commonly documented PCQ indicators in each cohort were presence of pain, etiology or source, and site of pain (greater than 90% of progress notes), while least commonly documented were sensation, what makes pain better or worse, and pain's impact on function (documented in fewer than 50%). A PCQ indicator score (maximum = 12) was calculated for each visit in FY 2013 (mean = 7.8, SD = 1.9) and FY 2017 (mean = 8.3, SD = 2.3) by adding one point for every indicator documented. Standardized Cronbach alpha for total PCQ scores was 0.74 in the most recent data (FY 2017). The mean PCQ indicator scores across patient characteristics and types of healthcare facilities were highly stable. Estimates of the frequency of documentation of PCQ indicators have face validity and encourage further evaluation of the reliability, validity, and utility of the measure. A reliable measure of PCQ fills an important scientific knowledge and practice gap.

Leveraging Electronic Health Care Record Information to Measure Pressure Ulcer Risk in Veterans With Spinal Cord Injury: A Longitudinal Study Protocol.

BACKGROUND: Pressure ulcers (PrUs) are a frequent, serious, and costly complication for veterans with spinal cord injury (SCI). The health care team should periodically identify PrU risk, although there is no tool in the literature that has been found to be reliable, valid, and sensitive enough to assess risk in this vulnerable population. OBJECTIVE: The immediate goal is to develop a risk assessment model that validly estimates the probability of developing a PrU. The long-term goal is to assist veterans with SCI and their providers in preventing PrUs through an automated system of risk assessment integrated into the veteran's electronic health record (EHR). METHODS: This 5-year longitudinal, retrospective, cohort study targets 12,344 veterans with SCI who were cared for in the Veterans Health Administration (VHA) in fiscal year (FY) 2009 and had no record of a PrU in the prior 12 months. Potential risk factors identified in the literature were reviewed by an expert panel that prioritized factors and determined if these were found in structured data or unstructured form in narrative clinical notes for FY 2009-2013. These data are from the VHA enterprise Corporate Data Warehouse that is derived from the EHR structured (ie, coded in database/table) or narrative (ie, text in clinical notes) data for FY 2009-2013. RESULTS: This study is ongoing and final results are expected in 2017. Thus far, the expert panel reviewed the initial list of risk factors extracted from the literature; the panel recommended additions and omissions and provided insights about the format in which the documentation of the risk factors might exist in the EHR. This list was then iteratively refined through review and discussed with individual experts in the field. The cohort for the study was then identified, and all structured, unstructured, and semistructured data were extracted. Annotation schemas were developed, samples of documents were extracted, and annotations are ongoing. Operational definitions of structured data elements have been created and steps to create an analytic dataset are underway. CONCLUSIONS: To our knowledge, this is the largest cohort employed to identify PrU risk factors in the United States. It also represents the first time natural language processing and statistical text mining will be used to expand the number of variables available for analysis. A major strength of this quantitative study is that all VHA SCI centers were included in the analysis, reducing potential for selection bias and providing increased power for complex statistical analyses. This longitudinal study will eventually result in a risk prediction tool to assess PrU risk that is reliable and valid, and that is sensitive to this vulnerable population.

Improving identification of fall-related injuries in ambulatory care using statistical text mining.

OBJECTIVES: We determined whether statistical text mining (STM) can identify fall-related injuries in electronic health record (EHR) documents and the impact on STM models of training on documents from a single or multiple facilities. METHODS: We obtained fiscal year 2007 records for Veterans Health Administration (VHA) ambulatory care clinics in the southeastern United States and Puerto Rico, resulting in a total of 26 010 documents for 1652 veterans treated for fall-related injury and 1341 matched controls. We used the results of an STM model to predict fall-related injuries at the visit and patient levels and compared them with a reference standard based on chart review. RESULTS: STM models based on training data from a single facility resulted in accuracy of 87.5% and 87.1%, F-measure of 87.0% and 90.9%, sensitivity of 92.1% and 94.1%, and specificity of 83.6% and 77.8% at the visit and patient levels, respectively. Results from training data from multiple facilities were almost identical. CONCLUSIONS: STM has the potential to improve identification of fall-related injuries in the VHA, providing a model for wider application in the evolving national EHR system.

Using information from the electronic health record to improve measurement of unemployment in service members and veterans with mTBI and post-deployment stress.

OBJECTIVE: The purpose of this pilot study is 1) to develop an annotation schema and a training set of annotated notes to support the future development of a natural language processing (NLP) system to automatically extract employment information, and 2) to determine if information about employment status, goals and work-related challenges reported by service members and Veterans with mild traumatic brain injury (mTBI) and post-deployment stress can be identified in the Electronic Health Record (EHR). DESIGN: Retrospective cohort study using data from selected progress notes stored in the EHR. SETTING: Post-deployment Rehabilitation and Evaluation Program (PREP), an in-patient rehabilitation program for Veterans with TBI at the James A. Haley Veterans' Hospital in Tampa, Florida. PARTICIPANTS: Service members and Veterans with TBI who participated in the PREP program (N = 60). MAIN OUTCOME MEASURES: Documentation of employment status, goals, and work-related challenges reported by service members and recorded in the EHR. RESULTS: Two hundred notes were examined and unique vocational information was found indicating a variety of self-reported employment challenges. Current employment status and future vocational goals along with information about cognitive, physical, and behavioral symptoms that may affect return-to-work were extracted from the EHR. The annotation schema developed for this study provides an excellent tool upon which NLP studies can be developed. CONCLUSIONS: Information related to employment status and vocational history is stored in text notes in the EHR system. Information stored in text does not lend itself to easy extraction or summarization for research and rehabilitation planning purposes. Development of NLP systems to automatically extract text-based employment information provides data that may improve the understanding and measurement of employment in this important cohort.

TagLine: Information Extraction for Semi-Structured Text in Medical Progress Notes.

Statistical text mining and natural language processing have been shown to be effective for extracting useful information from medical documents. However, neither technique is effective at extracting the information stored in semi-structure text elements. A prototype system (TagLine) was developed to extract information from the semi-structured text using machine learning and a rule based annotator. Features for the learning machine were suggested by prior work, and by examining text, and selecting attributes that help distinguish classes of text lines. Classes were derived empirically from text and guided by an ontology developed by the VHA's Consortium for Health Informatics Research (CHIR). Decision trees were evaluated for class predictions on 15,103 lines of text achieved an overall accuracy of 98.5 percent. The class labels applied to the lines were then used for annotating semi-structured text elements. TagLine achieved F-measure over 0.9 for each of the structures, which included tables, slots and fillers.

Finding falls in ambulatory care clinical documents using statistical text mining.

OBJECTIVE: To determine how well statistical text mining (STM) models can identify falls within clinical text associated with an ambulatory encounter. MATERIALS AND METHODS: 2241 patients were selected with a fall-related ICD-9-CM E-code or matched injury diagnosis code while being treated as an outpatient at one of four sites within the Veterans Health Administration. All clinical documents within a 48-h window of the recorded E-code or injury diagnosis code for each patient were obtained (n=26 010; 611 distinct document titles) and annotated for falls. Logistic regression, support vector machine, and cost-sensitive support vector machine (SVM-cost) models were trained on a stratified sample of 70% of documents from one location (dataset Atrain) and then applied to the remaining unseen documents (datasets Atest-D). RESULTS: All three STM models obtained area under the receiver operating characteristic curve (AUC) scores above 0.950 on the four test datasets (Atest-D). The SVM-cost model obtained the highest AUC scores, ranging from 0.953 to 0.978. The SVM-cost model also achieved F-measure values ranging from 0.745 to 0.853, sensitivity from 0.890 to 0.931, and specificity from 0.877 to 0.944. DISCUSSION: The STM models performed well across a large heterogeneous collection of document titles. In addition, the models also generalized across other sites, including a traditionally bilingual site that had distinctly different grammatical patterns. CONCLUSIONS: The results of this study suggest STM-based models have the potential to improve surveillance of falls. Furthermore, the encouraging evidence shown here that STM is a robust technique for mining clinical documents bodes well for other surveillance-related topics.

Using ensemble models to classify the sentiment expressed in suicide notes.

In 2007, suicide was the tenth leading cause of death in the U.S. Given the significance of this problem, suicide was the focus of the 2011 Informatics for Integrating Biology and the Bedside (i2b2) Natural Language Processing (NLP) shared task competition (track two). Specifically, the challenge concentrated on sentiment analysis, predicting the presence or absence of 15 emotions (labels) simultaneously in a collection of suicide notes spanning over 70 years. Our team explored multiple approaches combining regular expression-based rules, statistical text mining (STM), and an approach that applies weights to text while accounting for multiple labels. Our best submission used an ensemble of both rules and STM models to achieve a micro-averaged F(1) score of 0.5023, slightly above the mean from the 26 teams that competed (0.4875).

Using ontology network structure in text mining.

Statistical text mining treats documents as bags of words, with a focus on term frequencies within documents and across document collections. Unlike natural language processing (NLP) techniques that rely on an engineered vocabulary or a full-featured ontology, statistical approaches do not make use of domain-specific knowledge. The freedom from biases can be an advantage, but at the cost of ignoring potentially valuable knowledge. The approach proposed here investigates a hybrid strategy based on computing graph measures of term importance over an entire ontology and injecting the measures into the statistical text mining process. As a starting point, we adapt existing search engine algorithms such as PageRank and HITS to determine term importance within an ontology graph. The graph-theoretic approach is evaluated using a smoking data set from the i2b2 National Center for Biomedical Computing, cast as a simple binary classification task for categorizing smoking-related documents, demonstrating consistent improvements in accuracy.