Predictive modeling of initiation and delayed mental health contact for depression.

BACKGROUND: Depression is prevalent among Operation Enduring Freedom and Operation Iraqi Freedom (OEF/OIF) Veterans, yet rates of Veteran mental health care utilization remain modest. The current study examined: factors in electronic health records (EHR) associated with lack of treatment initiation and treatment delay; the accuracy of regression and machine learning models to predict initiation of treatment. METHODS: We obtained data from the VA Corporate Data Warehouse (CDW). EHR data were extracted for 127,423 Veterans who deployed to Iraq/Afghanistan after 9/11 with a positive depression screen and a first depression diagnosis between 2001 and 2021. We also obtained 12-month pre-diagnosis and post-diagnosis patient data. Retrospective cohort analysis was employed to test if predictors can reliably differentiate patients who initiated, delayed, or received no mental health treatment associated with their depression diagnosis. RESULTS: 108,457 Veterans with depression, initiated depression-related care (55,492 Veterans delayed treatment beyond one month). Those who were male, without VA disability benefits, with a mild depression diagnosis, and had a history of psychotherapy were less likely to initiate treatment. Among those who initiated care, those with single and mild depression episodes at baseline, with either PTSD or who lacked comorbidities were more likely to delay treatment for depression. A history of mental health treatment, of an anxiety disorder, and a positive depression screen were each related to faster treatment initiation. Classification of patients was modest (ROC AUC = 0.59 95%CI = 0.586-0.602; machine learning F-measure = 0.46). CONCLUSIONS: Having VA disability benefits was the strongest predictor of treatment initiation after a depression diagnosis and a history of mental health treatment was the strongest predictor of delayed initiation of treatment. The complexity of the relationship between VA benefits and history of mental health care with treatment initiation after a depression diagnosis is further discussed. Modest classification accuracy with currently known predictors suggests the need to identify additional predictors of successful depression management.

Mental health treatment utilization patterns among 108,457 Afghanistan and Iraq veterans with depression.

People with depression often underutilize mental health care. This study was conceived as a first step toward a clinical decision support tool that helps identify patients who are at higher risk of underutilizing care. The primary goals were to (a) describe treatment utilization patterns, early termination, and return to care; (b) identify factors associated with early termination of treatment; and (c) evaluate the accuracy of regression models to predict early termination. These goals were evaluated in a retrospective cohort analysis of 108,457 U.S. veterans who received care from the Veterans Health Administration between 2001 and 2021. Our final sample was 16.5% female with an average age of 34.5. Veterans were included if they had a depression diagnosis, a positive depression screen, and received general health care services at least a year before and after their depression diagnosis. Using treatment quality guidelines, the threshold for treatment underutilization was defined as receiving fewer than four psychotherapy sessions or less than 84 days of antidepressants. Over one fifth of veterans (21.6%) received less than the minimally recommended care for depression. The odds of underutilizing treatment increased with lack of Veterans Administration benefits, male gender, racial/ethnic minority status, and having received mental health treatment in the past (adjusted OR > 1.1). Posttraumatic stress disorder comorbidity correlated with increased depression treatment utilization (adjusted OR < .9). Models with demographic and clinical information from medical records performed modestly in classifying patients who underutilized depression treatment (area under the curve = 0.595, 95% CI [0.588, 0.603]). Most veterans in this cohort received at least the minimum recommended treatment for depression. To improve the prediction of underutilization, patient factors associated with treatment underutilization likely need to be supplemented by additional clinical information. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Machine learning to develop a predictive model of pressure injury in persons with spinal cord injury.

STUDY DESIGN: A 5-year longitudinal, retrospective, cohort study. OBJECTIVES: Develop a prediction model based on electronic health record (EHR) data to identify veterans with spinal cord injury/diseases (SCI/D) at highest risk for new pressure injuries (PIs). SETTING: Structured (coded) and text EHR data, for veterans with SCI/D treated in a VHA SCI/D Center between October 1, 2008, and September 30, 2013. METHODS: A total of 4709 veterans were available for analysis after randomly selecting 175 to act as a validation (gold standard) sample. Machine learning models were created using ten-fold cross validation and three techniques: (1) two-step logistic regression; (2) regression model employing adaptive LASSO; (3) and gradient boosting. Models based on each method were compared using area under the receiver-operating curve (AUC) analysis. RESULTS: The AUC value for the gradient boosting model was 0.62 (95% CI = 0.54-0.70), for the logistic regression model it was 0.67 (95% CI = 0.59-0.75), and for the adaptive LASSO model it was 0.72 (95% CI = 0.65-80). Based on these results, the adaptive LASSO model was chosen for interpretation. The strongest predictors of new PI cases were having fewer total days in the hospital in the year before the annual exam, higher vs. lower weight and most severe vs. less severe grade of injury based on the American Spinal Cord Injury Association (ASIA) Impairment Scale. CONCLUSIONS: While the analyses resulted in a potentially useful predictive model, clinical implications were limited because modifiable risk factors were absent in the models.

Retrospective Analysis of Vision Rehabilitation for Veterans With Traumatic Brain Injury-Related Vision Dysfunction.

INTRODUCTION: Traumatic brain injury (TBI) can trigger vision-based sequelae such as oculomotor and accommodative abnormalities, visual-vestibular integrative dysfunction, visual field loss, and photosensitivity. The need for diagnosis and management of TBI-related vision impairment has increased because of the increasing frequencies of combat warfighters returning from Iraq and Afghanistan with TBIs. The purpose of this research was to learn the sequelae of rehabilitation service delivery to veterans with TBI-related visual dysfunction after they are diagnosed. To accomplish this, we investigated vision rehabilitation assessments and interventions provided to veterans with TBI-related visual dysfunction at the Department of Veterans Affairs (VA) specialty polytrauma facilities for the 2 years following their injury. The research questions asked what assessments, interventions, and prescribed assistive devices were provided by VA specialty clinics (e.g., occupational therapy, polytrauma, and blind rehabilitation) and how service delivery was affected by demographic and clinical variables. MATERIALS AND METHODS: A retrospective design was used to analyze VA data using natural language processing of unstructured clinician notes and logistic regression of structured data. Participants included 350 veterans with TBI who received rehabilitation at one of the five VA Polytrauma Rehabilitation Centers (Tampa, FL; Richmond, VA; Minneapolis, MN; San Antonio, TX; and Palo Alto, CA) between 2008 and 2017 and who were administered the 2008 congressionally mandated "Traumatic Brain Injury Specific Ocular Health and Visual Functioning Exam." The outcome variables were vision assessments, interventions, and prescribed assistive technology discovered via natural language processing of clinician notes as well as the vision rehabilitation specialty clinics providing the clinical care (polytrauma, occupational therapy, outpatient blind rehabilitation, inpatient blind rehabilitation, optometry, and low vision) extracted from VA structured administrative data. RESULTS: Veterans receiving rehabilitation for TBI-related vision dysfunction were most frequently assessed for saccades, accommodation, visual field, and convergence. Intervention was provided most frequently for eye-hand coordination, saccades, accommodation, vergence, and binocular dysfunction. Technology provided included eyeglasses, wheelchair/scooter, walker/cane, aids for the blind, and computer. There was an overlap in the services provided by specialty clinics. Services available and delivered were significantly associated with the comorbidities of each patient and the specialty clinics available at each VA Polytrauma Rehabilitation Center. CONCLUSIONS: The delivery of patient services should be driven by the needs of veterans and not by system-level factors such as the availability of specific vision rehabilitation services at specific locations. Traditional low vision and blind rehabilitation programs were not designed to treat the comorbidities and symptoms associated with TBI. To address this challenge, blind rehabilitation and neurologic recovery cross training is needed. Our findings document how five VA Polytrauma Rehabilitation Centers implemented this training in 2008. The next step is to extend and standardize this new paradigm to community care, where these post-deployment patients now reside.

Extracting Pain Care Quality Indicators from U.S. Veterans Health Administration Chiropractic Care Using Natural Language Processing.

BACKGROUND: Musculoskeletal pain is common in the Veterans Health Administration (VHA), and there is growing national use of chiropractic services within the VHA. Rapid expansion requires scalable and autonomous solutions, such as natural language processing (NLP), to monitor care quality. Previous work has defined indicators of pain care quality that represent essential elements of guideline-concordant, comprehensive pain assessment, treatment planning, and reassessment. OBJECTIVE: Our purpose was to identify pain care quality indicators and assess patterns across different clinic visit types using NLP on VHA chiropractic clinic documentation. METHODS: Notes from ambulatory or in-hospital chiropractic care visits from October 1, 2018 to September 30, 2019 for patients in the Women Veterans Cohort Study were included in the corpus, with visits identified as consultation visits and/or evaluation and management (E&M) visits. Descriptive statistics of pain care quality indicator classes were calculated and compared across visit types. RESULTS: There were 11,752 patients who received any chiropractic care during FY2019, with 63,812 notes included in the corpus. Consultation notes had more than twice the total number of annotations per note (87.9) as follow-up visit notes (34.7). The mean number of total classes documented per note across the entire corpus was 9.4 (standard deviation [SD] = 1.5). More total indicator classes were documented during consultation visits with (mean = 14.8, SD = 0.9) or without E&M (mean = 13.9, SD = 1.2) compared to follow-up visits with (mean = 9.1, SD = 1.4) or without E&M (mean = 8.6, SD = 1.5). Co-occurrence of pain care quality indicators describing pain assessment was high. CONCLUSION: VHA chiropractors frequently document pain care quality indicators, identifiable using NLP, with variability across different visit types.

Complementary and Integrative Health Approaches and Pain Care Quality in the Veterans Health Administration Primary Care Setting: A Quasi-Experimental Analysis.

Background: Complementary and integrative health (CIH) approaches have been recommended in national and international clinical guidelines for chronic pain management. We set out to determine whether exposure to CIH approaches is associated with pain care quality (PCQ) in the Veterans Health Administration (VHA) primary care setting. Methods: We followed a cohort of 62,721 Veterans with newly diagnosed musculoskeletal disorders between October 2016 and September 2017 over 1-year. PCQ scores were derived from primary care progress notes using natural language processing. CIH exposure was defined as documentation of acupuncture, chiropractic or massage therapies by providers. Propensity scores (PSs) were used to match one control for each Veteran with CIH exposure. Generalized estimating equations were used to examine associations between CIH exposure and PCQ scores, accounting for potential selection and confounding bias. Results: CIH was documented for 14,114 (22.5%) Veterans over 16,015 primary care clinic visits during the follow-up period. The CIH exposure group and the 1:1 PS-matched control group achieved superior balance on all measured baseline covariates, with standardized differences ranging from 0.000 to 0.045. CIH exposure was associated with an adjusted rate ratio (aRR) of 1.147 (95% confidence interval [CI]: 1.142, 1.151) on PCQ total score (mean: 8.36). Sensitivity analyses using an alternative PCQ scoring algorithm (aRR: 1.155; 95% CI: 1.150-1.160) and redefining CIH exposure by chiropractic alone (aRR: 1.118; 95% CI: 1.110-1.126) derived consistent results. Discussion: Our data suggest that incorporating CIH approaches may reflect higher overall quality of care for patients with musculoskeletal pain seen in primary care settings, supporting VHA initiatives and the Declaration of Astana to build comprehensive, sustainable primary care capacity for pain management. Future investigation is warranted to better understand whether and to what degree the observed association may reflect the therapeutic benefits patients actually received or other factors such as empowering provider-patient education and communication about these approaches.

Mental Health Diagnoses are Not Associated With Indicators of Lower Quality Pain Care in Electronic Health Records of a National Sample of Veterans Treated in Veterans Health Administration Primary Care Settings.

Prior research has demonstrated disparities in general medical care for patients with mental health conditions, but little is known about disparities in pain care. The objective of this retrospective cohort study was to determine whether mental health conditions are associated with indicators of pain care quality (PCQ) as documented by primary care clinicians in the Veterans Health Administration (VHA). We used natural language processing to analyze electronic health record data from a national sample of Veterans with moderate to severe musculoskeletal pain during primary care visits in the Fiscal Year 2017. Twelve PCQ indicators were annotated from clinician progress notes as present or absent; PCQ score was defined as the sum of these indicators. Generalized estimating equation Poisson models examined associations among mental health diagnosis categories and PCQ scores. The overall mean PCQ score across 135,408 person-visits was 8.4 (SD = 2.3). In the final adjusted model, post-traumatic stress disorder was associated with higher PCQ scores (RR = 1.006, 95%CI 1.002-1.010, P = .007). Depression, alcohol use disorder, other substance use disorder, schizophrenia, and bipolar disorder diagnoses were not associated with PCQ scores. Overall, results suggest that in this patient population, presence of a mental health condition is not associated with lower quality pain care. PERSPECTIVE: This study used a natural language processing approach to analyze medical records to determine whether mental health conditions are associated with indicators of pain care quality as documented by primary care clinicians. Findings suggest that presence of a diagnosed mental health condition is not associated with lower quality pain care.

The Value of Extracting Clinician-Recorded Affect for Advancing Clinical Research on Depression: Proof-of-Concept Study Applying Natural Language Processing to Electronic Health Records.

BACKGROUND: Affective characteristics are associated with depression severity, course, and prognosis. Patients' affect captured by clinicians during sessions may provide a rich source of information that more naturally aligns with the depression course and patient-desired depression outcomes. OBJECTIVE: In this paper, we propose an information extraction vocabulary used to pilot the feasibility and reliability of identifying clinician-recorded patient affective states in clinical notes from electronic health records. METHODS: Affect and mood were annotated in 147 clinical notes of 109 patients by 2 independent coders across 3 pilots. Intercoder discrepancies were settled by a third coder. This reference annotation set was used to test a proof-of-concept natural language processing (NLP) system using a named entity recognition approach. RESULTS: Concepts were frequently addressed in templated format and free text in clinical notes. Annotated data demonstrated that affective characteristics were identified in 87.8% (129/147) of the notes, while mood was identified in 97.3% (143/147) of the notes. The intercoder reliability was consistently good across the pilots (interannotator agreement [IAA] >70%). The final NLP system showed good reliability with the final reference annotation set (mood IAA=85.8%; affect IAA=80.9%). CONCLUSIONS: Affect and mood can be reliably identified in clinician reports and are good targets for NLP. We discuss several next steps to expand on this proof of concept and the value of this research for depression clinical research.

Measuring pain care quality in the Veterans Health Administration primary care setting.

ABSTRACT: The lack of a reliable approach to assess quality of pain care hinders quality improvement initiatives. Rule-based natural language processing algorithms were used to extract pain care quality (PCQ) indicators from documents of Veterans Health Administration primary care providers for veterans diagnosed within the past year with musculoskeletal disorders with moderate-to-severe pain intensity across 2 time periods 2013 to 2014 (fiscal year [FY] 2013) and 2017 to 2018 (FY 2017). Patterns of documentation of PCQ indicators for 64,444 veterans and 124,408 unique visits (FY 2013) and 63,427 veterans and 146,507 visits (FY 2017) are described. The most commonly documented PCQ indicators in each cohort were presence of pain, etiology or source, and site of pain (greater than 90% of progress notes), while least commonly documented were sensation, what makes pain better or worse, and pain's impact on function (documented in fewer than 50%). A PCQ indicator score (maximum = 12) was calculated for each visit in FY 2013 (mean = 7.8, SD = 1.9) and FY 2017 (mean = 8.3, SD = 2.3) by adding one point for every indicator documented. Standardized Cronbach alpha for total PCQ scores was 0.74 in the most recent data (FY 2017). The mean PCQ indicator scores across patient characteristics and types of healthcare facilities were highly stable. Estimates of the frequency of documentation of PCQ indicators have face validity and encourage further evaluation of the reliability, validity, and utility of the measure. A reliable measure of PCQ fills an important scientific knowledge and practice gap.

Veterans with Traumatic Brain Injury-related Ocular Injury and Vision Dysfunction: Recommendations for Rehabilitation.

SIGNIFICANCE: We know the prevalence of traumatic brain injury (TBI)-related vision impairment and ocular injury symptoms. Lacking is an understanding of health care utilization to treat these symptoms. Utilization knowledge is important to structuring access to treatment, identifying clinical training needs, and providing evidence of the effectiveness of treatment. PURPOSE: This article reports rehabilitation, glasses/contacts, and imaging/photography/video recommendations made by optometrists and ophthalmologists as part of the Department of Veterans Affairs-mandated Performance of Traumatic Brain Injury Specific Ocular Health and Visual Functioning Examination administered to veterans with TBI at Department of Veterans Affairs polytrauma specialty facilities. METHODS: Using a retrospective design, natural language processing, and descriptive and regression statistics, data were analyzed for 2458 Operation Enduring Freedom/Operation Iraqi Freedom veterans who were administered the mandated examination between 2008 and 2017. RESULTS: Of the 2458 veterans, vision rehabilitation was recommended for 24%, glasses/contacts were recommended for 57%, and further imaging/photography/video testing was recommended for 58%. Using key words in the referral, we determined that 37% of veterans were referred to blind rehabilitation, 16% to occupational therapy, and 3% to low-vision clinics. More than 50% of the referrals could have been treated by blind rehabilitation, occupational therapy, or low-vision clinics. Rehabilitation referrals were significantly associated with younger age, floaters, photosensitivity, double vision, visual field and balance deficits, dizziness, and difficulty reading. In comparison, prescriptions for glasses and contacts were associated with older age, photosensitivity, blurred vision, decreased visual field and night vision, difficulty reading, and dry eye. Imaging/photography/video testing was associated with floaters, photosensitivity, and headache. CONCLUSIONS: Findings delineate service delivery models available to veterans with TBI-related vision impairment. The challenge these data address is the lack of clear paths from diagnosis of TBI to identification of vision dysfunction deficits to specialized vision rehabilitation, and finally to community reintegration and community based-vision rehabilitation.

Veterans with Traumatic Brain Injury-related Ocular Injury and Vision Dysfunction: Vision Rehabilitation Utilization.

SIGNIFICANCE: Visual dysfunction is frequently associated with traumatic brain injury (TBI). Although evidence regarding the prevalence of symptoms of this population has been published, little is known about health care utilization. A retrospective review of the data derived from the Department of Veterans Affairs (VA)-mandated "Traumatic Brain Injury Specific Ocular Health and Visual Functioning Examination for Polytrauma Rehabilitation Center Patients" provided a unique opportunity to investigate vision rehabilitation utilization. PURPOSE: The purpose of this study was to understand (a) the frequency of vision rehabilitation follow-up visits at 6, 12, and 24 months; (b) the association between follow-up and demographic, comorbidity, and severity of TBI covariates as well as ocular and visual symptoms, geographic access, and evaluating facility; and (c) why some veterans did not follow up with recommendations. METHODS: Retrospective and survey designs were used. The sample included 2458 veterans who served in the Operation Enduring Freedom/Operation Iraqi Freedom conflicts and received care at one of the five VA Polytrauma Rehabilitation Centers between January 1, 2008, and December 31, 2017. Quantitative and qualitative descriptive analyses and stepwise logistic regression were performed. RESULTS: About 60% of veterans followed up with recommended vision rehabilitation with visits equally split between VA Polytrauma Rehabilitation Centers and community VA facilities. For each 10-year increase in age, there was a corresponding reduction of 12% in the odds of follow-up. Veterans with decreased visual field had 50% greater odds of follow-up than those who did not. Veterans with difficulty reading had 59% greater odds of follow-up than those who did not. Those who had a double vision had 45% greater odds of follow-up than those who did not. CONCLUSIONS: Our findings suggest that the need for vision rehabilitation may extend as long as 2 years after TBI. Access to vision rehabilitation is complicated by the paucity of available neuro-optometric services.

Classifying clinical notes with pain assessment using machine learning.

Pain is a significant public health problem, affecting millions of people in the USA. Evidence has highlighted that patients with chronic pain often suffer from deficits in pain care quality (PCQ) including pain assessment, treatment, and reassessment. Currently, there is no intelligent and reliable approach to identify PCQ indicators inelectronic health records (EHR). Hereby, we used unstructured text narratives in the EHR to derive pain assessment in clinical notes for patients with chronic pain. Our dataset includes patients with documented pain intensity rating ratings > = 4 and initial musculoskeletal diagnoses (MSD) captured by (ICD-9-CM codes) in fiscal year 2011 and a minimal 1 year of follow-up (follow-up period is 3-yr maximum); with complete data on key demographic variables. A total of 92 patients with 1058 notes was used. First, we manually annotated qualifiers and descriptors of pain assessment using the annotation schema that we previously developed. Second, we developed a reliable classifier for indicators of pain assessment in clinical note. Based on our annotation schema, we found variations in documenting the subclasses of pain assessment. In positive notes, providers mostly documented assessment of pain site (67%) and intensity of pain (57%), followed by persistence (32%). In only 27% of positive notes, did providers document a presumed etiology for the pain complaint or diagnosis. Documentation of patients' reports of factors that aggravate pain was only present in 11% of positive notes. Random forest classifier achieved the best performance labeling clinical notes with pain assessment information, compared to other classifiers; 94, 95, 94, and 94% was observed in terms of accuracy, PPV, F1-score, and AUC, respectively. Despite the wide spectrum of research that utilizes machine learning in many clinical applications, none explored using these methods for pain assessment research. In addition, previous studies using large datasets to detect and analyze characteristics of patients with various types of pain have relied exclusively on billing and coded data as the main source of information. This study, in contrast, harnessed unstructured narrative text data from the EHR to detect pain assessment clinical notes. We developed a Random forest classifier to identify clinical notes with pain assessment information. Compared to other classifiers, ours achieved the best results in most of the reported metrics. Graphical abstract Framework for detecting pain assessment in clinical notes.

Leveraging Electronic Health Care Record Information to Measure Pressure Ulcer Risk in Veterans With Spinal Cord Injury: A Longitudinal Study Protocol.

BACKGROUND: Pressure ulcers (PrUs) are a frequent, serious, and costly complication for veterans with spinal cord injury (SCI). The health care team should periodically identify PrU risk, although there is no tool in the literature that has been found to be reliable, valid, and sensitive enough to assess risk in this vulnerable population. OBJECTIVE: The immediate goal is to develop a risk assessment model that validly estimates the probability of developing a PrU. The long-term goal is to assist veterans with SCI and their providers in preventing PrUs through an automated system of risk assessment integrated into the veteran's electronic health record (EHR). METHODS: This 5-year longitudinal, retrospective, cohort study targets 12,344 veterans with SCI who were cared for in the Veterans Health Administration (VHA) in fiscal year (FY) 2009 and had no record of a PrU in the prior 12 months. Potential risk factors identified in the literature were reviewed by an expert panel that prioritized factors and determined if these were found in structured data or unstructured form in narrative clinical notes for FY 2009-2013. These data are from the VHA enterprise Corporate Data Warehouse that is derived from the EHR structured (ie, coded in database/table) or narrative (ie, text in clinical notes) data for FY 2009-2013. RESULTS: This study is ongoing and final results are expected in 2017. Thus far, the expert panel reviewed the initial list of risk factors extracted from the literature; the panel recommended additions and omissions and provided insights about the format in which the documentation of the risk factors might exist in the EHR. This list was then iteratively refined through review and discussed with individual experts in the field. The cohort for the study was then identified, and all structured, unstructured, and semistructured data were extracted. Annotation schemas were developed, samples of documents were extracted, and annotations are ongoing. Operational definitions of structured data elements have been created and steps to create an analytic dataset are underway. CONCLUSIONS: To our knowledge, this is the largest cohort employed to identify PrU risk factors in the United States. It also represents the first time natural language processing and statistical text mining will be used to expand the number of variables available for analysis. A major strength of this quantitative study is that all VHA SCI centers were included in the analysis, reducing potential for selection bias and providing increased power for complex statistical analyses. This longitudinal study will eventually result in a risk prediction tool to assess PrU risk that is reliable and valid, and that is sensitive to this vulnerable population.

Classifying Clinical Notes with Pain Assessment.

Pain is a significant public health problem, affecting an estimated 100 million Americans. Evidence has highlighted that patients with chronic pain often suffer from deficits in pain care quality (PCQ). Efforts to improve PCQ hinge on the identification of reliable PCQ indicators such as pain assessment. In this study, we developed a classifier that leverages narratives in clinical notes to derive indicators of pain assessment for patients with chronic pain.

Improving identification of fall-related injuries in ambulatory care using statistical text mining.

OBJECTIVES: We determined whether statistical text mining (STM) can identify fall-related injuries in electronic health record (EHR) documents and the impact on STM models of training on documents from a single or multiple facilities. METHODS: We obtained fiscal year 2007 records for Veterans Health Administration (VHA) ambulatory care clinics in the southeastern United States and Puerto Rico, resulting in a total of 26 010 documents for 1652 veterans treated for fall-related injury and 1341 matched controls. We used the results of an STM model to predict fall-related injuries at the visit and patient levels and compared them with a reference standard based on chart review. RESULTS: STM models based on training data from a single facility resulted in accuracy of 87.5% and 87.1%, F-measure of 87.0% and 90.9%, sensitivity of 92.1% and 94.1%, and specificity of 83.6% and 77.8% at the visit and patient levels, respectively. Results from training data from multiple facilities were almost identical. CONCLUSIONS: STM has the potential to improve identification of fall-related injuries in the VHA, providing a model for wider application in the evolving national EHR system.

Using information from the electronic health record to improve measurement of unemployment in service members and veterans with mTBI and post-deployment stress.

OBJECTIVE: The purpose of this pilot study is 1) to develop an annotation schema and a training set of annotated notes to support the future development of a natural language processing (NLP) system to automatically extract employment information, and 2) to determine if information about employment status, goals and work-related challenges reported by service members and Veterans with mild traumatic brain injury (mTBI) and post-deployment stress can be identified in the Electronic Health Record (EHR). DESIGN: Retrospective cohort study using data from selected progress notes stored in the EHR. SETTING: Post-deployment Rehabilitation and Evaluation Program (PREP), an in-patient rehabilitation program for Veterans with TBI at the James A. Haley Veterans' Hospital in Tampa, Florida. PARTICIPANTS: Service members and Veterans with TBI who participated in the PREP program (N = 60). MAIN OUTCOME MEASURES: Documentation of employment status, goals, and work-related challenges reported by service members and recorded in the EHR. RESULTS: Two hundred notes were examined and unique vocational information was found indicating a variety of self-reported employment challenges. Current employment status and future vocational goals along with information about cognitive, physical, and behavioral symptoms that may affect return-to-work were extracted from the EHR. The annotation schema developed for this study provides an excellent tool upon which NLP studies can be developed. CONCLUSIONS: Information related to employment status and vocational history is stored in text notes in the EHR system. Information stored in text does not lend itself to easy extraction or summarization for research and rehabilitation planning purposes. Development of NLP systems to automatically extract text-based employment information provides data that may improve the understanding and measurement of employment in this important cohort.

TagLine: Information Extraction for Semi-Structured Text in Medical Progress Notes.

Statistical text mining and natural language processing have been shown to be effective for extracting useful information from medical documents. However, neither technique is effective at extracting the information stored in semi-structure text elements. A prototype system (TagLine) was developed to extract information from the semi-structured text using machine learning and a rule based annotator. Features for the learning machine were suggested by prior work, and by examining text, and selecting attributes that help distinguish classes of text lines. Classes were derived empirically from text and guided by an ontology developed by the VHA's Consortium for Health Informatics Research (CHIR). Decision trees were evaluated for class predictions on 15,103 lines of text achieved an overall accuracy of 98.5 percent. The class labels applied to the lines were then used for annotating semi-structured text elements. TagLine achieved F-measure over 0.9 for each of the structures, which included tables, slots and fillers.

Finding falls in ambulatory care clinical documents using statistical text mining.

OBJECTIVE: To determine how well statistical text mining (STM) models can identify falls within clinical text associated with an ambulatory encounter. MATERIALS AND METHODS: 2241 patients were selected with a fall-related ICD-9-CM E-code or matched injury diagnosis code while being treated as an outpatient at one of four sites within the Veterans Health Administration. All clinical documents within a 48-h window of the recorded E-code or injury diagnosis code for each patient were obtained (n=26 010; 611 distinct document titles) and annotated for falls. Logistic regression, support vector machine, and cost-sensitive support vector machine (SVM-cost) models were trained on a stratified sample of 70% of documents from one location (dataset Atrain) and then applied to the remaining unseen documents (datasets Atest-D). RESULTS: All three STM models obtained area under the receiver operating characteristic curve (AUC) scores above 0.950 on the four test datasets (Atest-D). The SVM-cost model obtained the highest AUC scores, ranging from 0.953 to 0.978. The SVM-cost model also achieved F-measure values ranging from 0.745 to 0.853, sensitivity from 0.890 to 0.931, and specificity from 0.877 to 0.944. DISCUSSION: The STM models performed well across a large heterogeneous collection of document titles. In addition, the models also generalized across other sites, including a traditionally bilingual site that had distinctly different grammatical patterns. CONCLUSIONS: The results of this study suggest STM-based models have the potential to improve surveillance of falls. Furthermore, the encouraging evidence shown here that STM is a robust technique for mining clinical documents bodes well for other surveillance-related topics.

Using ensemble models to classify the sentiment expressed in suicide notes.

In 2007, suicide was the tenth leading cause of death in the U.S. Given the significance of this problem, suicide was the focus of the 2011 Informatics for Integrating Biology and the Bedside (i2b2) Natural Language Processing (NLP) shared task competition (track two). Specifically, the challenge concentrated on sentiment analysis, predicting the presence or absence of 15 emotions (labels) simultaneously in a collection of suicide notes spanning over 70 years. Our team explored multiple approaches combining regular expression-based rules, statistical text mining (STM), and an approach that applies weights to text while accounting for multiple labels. Our best submission used an ensemble of both rules and STM models to achieve a micro-averaged F(1) score of 0.5023, slightly above the mean from the 26 teams that competed (0.4875).

Using statistical text mining to supplement the development of an ontology.

Statistical text mining was used to supplement efforts to develop a clinical vocabulary for post-traumatic stress disorder (PTSD) in the VA. A set of outpatient progress notes was collected for a cohort of 405 unique veterans with PTSD and a comparison group of 392 with other psychological conditions at one VA hospital. Two methods were employed: (1) "multi-model term scoring" used stepwise logistic regression to develop 21 separate models by varying three frequency weight and seven term weight options and (2) "iterative term refinement" which used a standard stop list followed by clinical review to eliminate non-clinical terms and terms not related to PTSD. Combined results of the two methods were reviewed by two clinicians resulting in 226 unique PTSD related terms. Results of the statistical text mining methods were compared with ongoing efforts to identify terms based on literature review, focus groups with clinicians treating PTSD and review of an existing vocabulary, lending support to the contributions of the STM analyses.