Applying natural language processing to patient messages to identify depression concerns in cancer patients.

OBJECTIVE: This study aims to explore and develop tools for early identification of depression concerns among cancer patients by leveraging the novel data source of messages sent through a secure patient portal. MATERIALS AND METHODS: We developed classifiers based on logistic regression (LR), support vector machines (SVMs), and 2 Bidirectional Encoder Representations from Transformers (BERT) models (original and Reddit-pretrained) on 6600 patient messages from a cancer center (2009-2022), annotated by a panel of healthcare professionals. Performance was compared using AUROC scores, and model fairness and explainability were examined. We also examined correlations between model predictions and depression diagnosis and treatment. RESULTS: BERT and RedditBERT attained AUROC scores of 0.88 and 0.86, respectively, compared to 0.79 for LR and 0.83 for SVM. BERT showed bigger differences in performance across sex, race, and ethnicity than RedditBERT. Patients who sent messages classified as concerning had a higher chance of receiving a depression diagnosis, a prescription for antidepressants, or a referral to the psycho-oncologist. Explanations from BERT and RedditBERT differed, with no clear preference from annotators. DISCUSSION: We show the potential of BERT and RedditBERT in identifying depression concerns in messages from cancer patients. Performance disparities across demographic groups highlight the need for careful consideration of potential biases. Further research is needed to address biases, evaluate real-world impacts, and ensure responsible integration into clinical settings. CONCLUSION: This work represents a significant methodological advancement in the early identification of depression concerns among cancer patients. Our work contributes to a route to reduce clinical burden while enhancing overall patient care, leveraging BERT-based models.

Don't be misled: 3 misconceptions about external validation of clinical prediction models.

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing 3 common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

Identification of Distinct Brain MRI Phenotypes and Their Association With Long-Term Dementia Risk in Community-Dwelling Older Adults.

BACKGROUND AND OBJECTIVES: Individual brain MRI markers only show at best a modest association with long-term occurrence of dementia. Therefore, it is challenging to accurately identify individuals at increased risk for dementia. We aimed to identify different brain MRI phenotypes by hierarchical clustering analysis based on combined neurovascular and neurodegenerative brain MRI markers and to determine the long-term dementia risk within the brain MRI phenotype subgroups. METHODS: Hierarchical clustering analysis based on 32 combined neurovascular and neurodegenerative brain MRI markers in community-dwelling individuals of the Age-Gene/Environment Susceptibility Reykjavik Study was applied to identify brain MRI phenotypes. A Cox proportional hazards regression model was used to determine the long-term risk for dementia per subgroup. RESULTS: We included 3,056 participants and identified 15 subgroups with distinct brain MRI phenotypes. The phenotypes ranged from limited burden, mostly irregular white matter hyperintensity (WMH) shape and cerebral atrophy, mostly irregularly WMHs and microbleeds, mostly cortical infarcts and atrophy, mostly irregularly shaped WMH and cerebral atrophy to multiburden subgroups. Each subgroup showed different long-term risks for dementia (min-max range hazard ratios [HRs] 1.01-6.18; mean time to follow-up 9.9 ± 2.6 years); especially the brain MRI phenotype with mainly WMHs and atrophy showed a large increased risk (HR 6.18, 95% CI 3.37-11.32). DISCUSSION: Distinct brain MRI phenotypes can be identified in community-dwelling older adults. Our results indicate that distinct brain MRI phenotypes are related to varying long-term risks of developing dementia. Brain MRI phenotypes may in the future assist in an improved understanding of the structural correlates of dementia predisposition.

Predicting Depression Risk in Patients With Cancer Using Multimodal Data: Algorithm Development Study.

BACKGROUND: Patients with cancer starting systemic treatment programs, such as chemotherapy, often develop depression. A prediction model may assist physicians and health care workers in the early identification of these vulnerable patients. OBJECTIVE: This study aimed to develop a prediction model for depression risk within the first month of cancer treatment. METHODS: We included 16,159 patients diagnosed with cancer starting chemo- or radiotherapy treatment between 2008 and 2021. Machine learning models (eg, least absolute shrinkage and selection operator [LASSO] logistic regression) and natural language processing models (Bidirectional Encoder Representations from Transformers [BERT]) were used to develop multimodal prediction models using both electronic health record data and unstructured text (patient emails and clinician notes). Model performance was assessed in an independent test set (n=5387, 33%) using area under the receiver operating characteristic curve (AUROC), calibration curves, and decision curve analysis to assess initial clinical impact use. RESULTS: Among 16,159 patients, 437 (2.7%) received a depression diagnosis within the first month of treatment. The LASSO logistic regression models based on the structured data (AUROC 0.74, 95% CI 0.71-0.78) and structured data with email classification scores (AUROC 0.74, 95% CI 0.71-0.78) had the best discriminative performance. The BERT models based on clinician notes and structured data with email classification scores had AUROCs around 0.71. The logistic regression model based on email classification scores alone performed poorly (AUROC 0.54, 95% CI 0.52-0.56), and the model based solely on clinician notes had the worst performance (AUROC 0.50, 95% CI 0.49-0.52). Calibration was good for the logistic regression models, whereas the BERT models produced overly extreme risk estimates even after recalibration. There was a small range of decision thresholds for which the best-performing model showed promising clinical effectiveness use. The risks were underestimated for female and Black patients. CONCLUSIONS: The results demonstrated the potential and limitations of machine learning and multimodal models for predicting depression risk in patients with cancer. Future research is needed to further validate these models, refine the outcome label and predictors related to mental health, and address biases across subgroups.

MRI phenotypes of glioblastomas early after treatment are suggestive of overall patient survival.

Background: Distinguishing true tumor progression (TP) from treatment-induced abnormalities (eg, pseudo-progression (PP) after radiotherapy) on conventional MRI scans remains challenging in patients with a glioblastoma. We aimed to establish brain MRI phenotypes of glioblastomas early after treatment by combined analysis of structural and perfusion tumor characteristics and assessed the relation with recurrence rate and overall survival time. Methods: Structural and perfusion MR images of 67 patients at 3 months post-radiotherapy were visually scored by a neuroradiologist. In total 23 parameters were predefined and used for hierarchical clustering analysis. Progression status was assessed based on the clinical course of each patient 9 months after radiotherapy (or latest available). Multivariable Cox regression models were used to determine the association between the phenotypes, recurrence rate, and overall survival. Results: We established 4 subgroups with significantly different tumor MRI characteristics, representing distinct MRI phenotypes of glioblastomas: TP and PP rates did not differ significantly between subgroups. Regression analysis showed that patients in subgroup 1 (characterized by having mostly small and ellipsoid nodular enhancing lesions with some hyper-perfusion) had a significant association with increased mortality at 9 months (HR: 2.6 (CI: 1.1-6.3); P = .03) with a median survival time of 13 months (compared to 22 months of subgroup 2). Conclusions: Our study suggests that distinct MRI phenotypes of glioblastomas at 3 months post-radiotherapy can be indicative of overall survival, but does not aid in differentiating TP from PP. The early prognostic information our method provides might in the future be informative for prognostication of glioblastoma patients.

Diagnosis Classification in the Emergency Room Using Natural Language Processing.

Diagnosis classification in the emergency room (ER) is a complex task. We developed several natural language processing classification models, looking both at the full classification task of 132 diagnostic categories and at several clinically applicable samples consisting of two diagnoses that are hard to distinguish.

Predicting Depression Risk in Patients with Cancer Using Multimodal Data.

When patients with cancer develop depression, it is often left untreated. We developed a prediction model for depression risk within the first month after starting cancer treatment using machine learning and Natural Language Processing (NLP) models. The LASSO logistic regression model based on structured data performed well, whereas the NLP model based on only clinician notes did poorly. After further validation, prediction models for depression risk could lead to earlier identification and treatment of vulnerable patients, ultimately improving cancer care and treatment adherence.

Postoperative delirium is associated with grey matter brain volume loss.

Delirium is associated with long-term cognitive dysfunction and with increased brain atrophy. However, it is unclear whether these problems result from or predisposes to delirium. We aimed to investigate preoperative to postoperative brain changes, as well as the role of delirium in these changes over time. We investigated the effects of surgery and postoperative delirium with brain MRIs made before and 3 months after major elective surgery in 299 elderly patients, and an MRI with a 3 months follow-up MRI in 48 non-surgical control participants. To study the effects of surgery and delirium, we compared brain volumes, white matter hyperintensities and brain infarcts between baseline and follow-up MRIs, using multiple regression analyses adjusting for possible confounders. Within the patients group, 37 persons (12%) developed postoperative delirium. Surgical patients showed a greater decrease in grey matter volume than non-surgical control participants [linear regression: B (95% confidence interval) = -0.65% of intracranial volume (-1.01 to -0.29, P < 0.005)]. Within the surgery group, delirium was associated with a greater decrease in grey matter volume [B (95% confidence interval): -0.44% of intracranial volume (-0.82 to -0.06, P = 0.02)]. Furthermore, within the patients, delirium was associated with a non-significantly increased risk of a new postoperative brain infarct [logistic regression: odds ratio (95% confidence interval): 2.8 (0.7-11.1), P = 0.14]. Our study was the first to investigate the association between delirium and preoperative to postoperative brain volume changes, suggesting that delirium is associated with increased progression of grey matter volume loss.

Intensive Care Unit Physicians' Perspectives on Artificial Intelligence-Based Clinical Decision Support Tools: Preimplementation Survey Study.

BACKGROUND: Artificial intelligence-based clinical decision support (AI-CDS) tools have great potential to benefit intensive care unit (ICU) patients and physicians. There is a gap between the development and implementation of these tools. OBJECTIVE: We aimed to investigate physicians' perspectives and their current decision-making behavior before implementing a discharge AI-CDS tool for predicting readmission and mortality risk after ICU discharge. METHODS: We conducted a survey of physicians involved in decision-making on discharge of patients at two Dutch academic ICUs between July and November 2021. Questions were divided into four domains: (1) physicians' current decision-making behavior with respect to discharging ICU patients, (2) perspectives on the use of AI-CDS tools in general, (3) willingness to incorporate a discharge AI-CDS tool into daily clinical practice, and (4) preferences for using a discharge AI-CDS tool in daily workflows. RESULTS: Most of the 64 respondents (of 93 contacted, 69%) were familiar with AI (62/64, 97%) and had positive expectations of AI, with 55 of 64 (86%) believing that AI could support them in their work as a physician. The respondents disagreed on whether the decision to discharge a patient was complex (23/64, 36% agreed and 22/64, 34% disagreed); nonetheless, most (59/64, 92%) agreed that a discharge AI-CDS tool could be of value. Significant differences were observed between physicians from the 2 academic sites, which may be related to different levels of involvement in the development of the discharge AI-CDS tool. CONCLUSIONS: ICU physicians showed a favorable attitude toward the integration of AI-CDS tools into the ICU setting in general, and in particular toward a tool to predict a patient's risk of readmission and mortality within 7 days after discharge. The findings of this questionnaire will be used to improve the implementation process and training of end users.

Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model.

OBJECTIVES: Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration. DESIGN: A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center. SETTING: Two ICUs in tertiary care centers in The Netherlands. PATIENTS: Adult patients who were admitted to the ICU and stayed for longer than 12 hours. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression. CONCLUSIONS: In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.

Functional brain network and trail making test changes following major surgery and postoperative delirium: a prospective, multicentre, observational cohort study.

BACKGROUND: Delirium is a frequent complication after surgery in older adults and is associated with an increased risk of long-term cognitive impairment and dementia. Disturbances in functional brain networks were previously reported during delirium. We hypothesised that alterations in functional brain networks persist after remission of postoperative delirium and that functional brain network alterations are associated with long-term cognitive impairment. METHODS: In this prospective, multicentre, observational cohort study, we included older patients who underwent clinical assessments (including the Trail Making Test B [TMT-B]) and resting-state functional MRI (rs-fMRI) before and 3 months after elective surgery. Delirium was assessed on the first seven postoperative days. RESULTS: Of the 554 enrolled patients, 246 remained after strict motion correction, of whom 38 (16%) developed postoperative delirium. The rs-fMRI functional connectivity strength increased 3 months after surgery in the total study population (ß=0.006; 95% confidence interval [CI]: 0.001-0.011; P=0.013), but it decreased after postoperative delirium (ß=-0.015; 95% CI: -0.028 to 0.002; P=0.023). No difference in TMT-B scores was found at follow-up between patients with and without postoperative delirium. Patients with decreased functional connectivity strength declined in TMT-B scores compared with those who did not (ß=11.04; 95% CI: 0.85-21.2; P=0.034). CONCLUSIONS: Postoperative delirium was associated with decreased brain functional connectivity strength after 3 months, suggesting that delirium has a long-lasting impact on brain networks. The decreased connectivity strength was associated with significant cognitive deterioration after major surgery. CLINICAL TRIAL REGISTRATION: NCT02265263.

Machine learning did not beat logistic regression in time series prediction for severe asthma exacerbations.

Early detection of severe asthma exacerbations through home monitoring data in patients with stable mild-to-moderate chronic asthma could help to timely adjust medication. We evaluated the potential of machine learning methods compared to a clinical rule and logistic regression to predict severe exacerbations. We used daily home monitoring data from two studies in asthma patients (development: n = 165 and validation: n = 101 patients). Two ML models (XGBoost, one class SVM) and a logistic regression model provided predictions based on peak expiratory flow and asthma symptoms. These models were compared with an asthma action plan rule. Severe exacerbations occurred in 0.2% of all daily measurements in the development (154/92,787 days) and validation cohorts (94/40,185 days). The AUC of the best performing XGBoost was 0.85 (0.82-0.87) and 0.88 (0.86-0.90) for logistic regression in the validation cohort. The XGBoost model provided overly extreme risk estimates, whereas the logistic regression underestimated predicted risks. Sensitivity and specificity were better overall for XGBoost and logistic regression compared to one class SVM and the clinical rule. We conclude that ML models did not beat logistic regression in predicting short-term severe asthma exacerbations based on home monitoring data. Clinical application remains challenging in settings with low event incidence and high false alarm rates with high sensitivity.

Different cardiovascular risk factors are related to distinct white matter hyperintensity MRI phenotypes in older adults.

The underlying mechanisms of the association between cardiovascular risk factors and a higher white matter hyperintensity (WMH) burden are unknown. We investigated the association between cardiovascular risk factors and advanced WMH markers in 155 non-demented older adults (mean age: 71 ± 5 years). The association between cardiovascular risk factors and quantitative MRI-based WMH shape and volume markers were examined using linear regression analysis. Presence of hypertension was associated with a more irregular shape of periventricular/confluent WMH (convexity (B (95 % CI)): -0.12 (-0.22--0.03); concavity index: 0.06 (0.02-0.11)), but not with total WMH volume (0.22 (-0.15-0.59)). Presence of diabetes was associated with deep WMH volume (0.89 (0.15-1.63)). Body mass index or hyperlipidemia showed no association with WMH markers. In conclusion, different cardiovascular risk factors seem to be related to a distinct pattern of WMH shape markers in non-demented older adults. These findings may suggest that different underlying cardiovascular pathological mechanisms lead to different WMH MRI phenotypes, which may be valuable for early detection of individuals at risk for stroke and dementia.

Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM).

BACKGROUND: Evaluating patients' experiences is essential when incorporating the patients' perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness. METHODS: We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context. RESULTS: The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. CONCLUSIONS: The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.

Delirium and long-term psychopathology following surgery in older adults.

OBJECTIVE: To describe the risk of postoperative delirium and long-term psychopathology (depression, anxiety or post-traumatic stress syndrome (PTSS)) in older adults. METHODS: 255 elderly patients (≥ 65 years) undergoing major surgery (planned surgical time > 60 min) in a tertiary hospital were compared to 76 non-surgical controls from general practice. Patients were assessed twice daily for postoperative delirium using the Confusion Assessment Method (CAM(-ICU)), nursing delirium screening scale (NuDESC) and validated chart review. Before surgery and 3 and 12 months thereafter, the participants filled in the Hospital Anxiety and Depression Scale (HADS), the Geriatric Depression Scale (GDS-15) and the Post-Traumatic Stress Syndrome-14-Questions Inventory (PTSS-14). Non-surgical controls filled in the same questionnaires with similar follow-up. RESULTS: Patients were more often male, had higher American Society of Anesthesiologists scores and more often had a spouse compared to controls (p < 0.005). Forty-three patients (18%) developed postoperative delirium, who were significantly older, had higher ASA scores and lower estimated IQ scores compared to the patients who did not develop delirium (p < 0.05). There were no differences in psychopathology at baseline and 3-month follow-up between patients and controls. At 12-months, surgical patients less frequently scored positive for depression (7% versus 16%) and anxiety (2% versus 10%) compared to nonsurgical controls (p < 0.05). We did not observe differences in occurrence of psychopathology between patients who had and had not developed postoperative delirium. CONCLUSION: Our results suggest that the older surgical population, with or without postoperative delirium, does not appear to be at greater risk of developing psychopathology. WHY DOES THIS PAPER MATTER?: The older surgical population does not appear to be at greater risk of developing psychopathology, neither seems this risk influenced by the occurrence of postoperative delirium.

Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review.

While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.