A spatio-temporal network for video semantic segmentation in surgical videos.

PURPOSE: Semantic segmentation in surgical videos has applications in intra-operative guidance, post-operative analytics and surgical education. Models need to provide accurate predictions since temporally inconsistent identification of anatomy can hinder patient safety. We propose a novel architecture for modelling temporal relationships in videos to address these issues. METHODS: We developed a temporal segmentation model that includes a static encoder and a spatio-temporal decoder. The encoder processes individual frames whilst the decoder learns spatio-temporal relationships from frame sequences. The decoder can be used with any suitable encoder to improve temporal consistency. RESULTS: Model performance was evaluated on the CholecSeg8k dataset and a private dataset of robotic Partial Nephrectomy procedures. Mean Intersection over Union improved by 1.30% and 4.27% respectively for each dataset when the temporal decoder was applied. Our model also displayed improvements in temporal consistency up to 7.23%. CONCLUSIONS: This work demonstrates an advance in video segmentation of surgical scenes with potential applications in surgery with a view to improve patient outcomes. The proposed decoder can extend state-of-the-art static models, and it is shown that it can improve per-frame segmentation output and video temporal consistency.

Applying Artificial Intelligence Methods for the Estimation of Disease Incidence: The Utility of Language Models.

Background: AI-driven digital health tools often rely on estimates of disease incidence or prevalence, but obtaining these estimates is costly and time-consuming. We explored the use of machine learning models that leverage contextual information about diseases from unstructured text, to estimate disease incidence. Methods: We used a class of machine learning models, called language models, to extract contextual information relating to disease incidence. We evaluated three different language models: BioBERT, Global Vectors for Word Representation (GloVe), and the Universal Sentence Encoder (USE), as well as an approach which uses all jointly. The output of these models is a mathematical representation of the underlying data, known as "embeddings." We used these to train neural network models to predict disease incidence. The neural networks were trained and validated using data from the Global Burden of Disease study, and tested using independent data sourced from the epidemiological literature. Findings: A variety of language models can be used to encode contextual information of diseases. We found that, on average, BioBERT embeddings were the best for disease names across multiple tasks. In particular, BioBERT was the best performing model when predicting specific disease-country pairs, whilst a fusion model combining BioBERT, GloVe, and USE performed best on average when predicting disease incidence in unseen countries. We also found that GloVe embeddings performed better than BioBERT embeddings when applied to country names. However, we also noticed that the models were limited in view of predicting previously unseen diseases. Further limitations were also observed with substantial variations across age groups and notably lower performance for diseases that are highly dependent on location and climate. Interpretation: We demonstrate that context-aware machine learning models can be used for estimating disease incidence. This method is quicker to implement than traditional epidemiological approaches. We therefore suggest it complements existing modeling efforts, where data is required more rapidly or at larger scale. This may particularly benefit AI-driven digital health products where the data will undergo further processing and a validated approximation of the disease incidence is adequate.

Disease Progression Modeling in Chronic Obstructive Pulmonary Disease.

Rationale: The decades-long progression of chronic obstructive pulmonary disease (COPD) renders identifying different trajectories of disease progression challenging.Objectives: To identify subtypes of patients with COPD with distinct longitudinal progression patterns using a novel machine-learning tool called "Subtype and Stage Inference" (SuStaIn) and to evaluate the utility of SuStaIn for patient stratification in COPD.Methods: We applied SuStaIn to cross-sectional computed tomography imaging markers in 3,698 Global Initiative for Chronic Obstructive Lung Disease (GOLD) 1-4 patients and 3,479 controls from the COPDGene (COPD Genetic Epidemiology) study to identify subtypes of patients with COPD. We confirmed the identified subtypes and progression patterns using ECLIPSE (Evaluation of COPD Longitudinally to Identify Predictive Surrogate Endpoints) data. We assessed the utility of SuStaIn for patient stratification by comparing SuStaIn subtypes and stages at baseline with longitudinal follow-up data.Measurements and Main Results: We identified two trajectories of disease progression in COPD: a "Tissue→Airway" subtype (n = 2,354, 70.4%), in which small airway dysfunction and emphysema precede large airway wall abnormalities, and an "Airway→Tissue" subtype (n = 988, 29.6%), in which large airway wall abnormalities precede emphysema and small airway dysfunction. Subtypes were reproducible in ECLIPSE. Baseline stage in both subtypes correlated with future FEV1/FVC decline (r = -0.16 [P < 0.001] in the Tissue→Airway group; r = -0.14 [P = 0.011] in the Airway→Tissue group). SuStaIn placed 30% of smokers with normal lung function at elevated stages, suggesting imaging changes consistent with early COPD. Individuals with early changes were 2.5 times more likely to meet COPD diagnostic criteria at follow-up.Conclusions: We demonstrate two distinct patterns of disease progression in COPD using SuStaIn, likely representing different endotypes. One third of healthy smokers have detectable imaging changes, suggesting a new biomarker of "early COPD."

Pulmonary Lobe Segmentation With Probabilistic Segmentation of the Fissures and a Groupwise Fissure Prior.

A fully automated, unsupervised lobe segmentation algorithm is presented based on a probabilistic segmentation of the fissures and the simultaneous construction of a populationmodel of the fissures. A two-class probabilistic segmentation segments the lung into candidate fissure voxels and the surrounding parenchyma. This was combined with anatomical information and a groupwise fissure prior to drive non-parametric surface fitting to obtain the final segmentation. The performance of our fissure segmentation was validated on 30 patients from the chronic obstructive pulmonary disease COPDGene cohort, achieving a high median F1 -score of 0.90 and showed general insensitivity to filter parameters. We evaluated our lobe segmentation algorithm on the Lobe and Lung Analysis 2011 dataset, which contains 55 cases at varying levels of pathology. We achieved the highest score of 0.884 of the automated algorithms. Our method was further tested quantitatively and qualitatively on 80 patients from the COPDgene study at varying levels of functional impairment. Accurate segmentation of the lobes is shown at various degrees of fissure incompleteness for 96% of all cases. We also show the utility of including a groupwise prior in segmenting the lobes in regions of grossly incomplete fissures.

Manifold Learning of COPD.

Analysis of CT scans for studying Chronic Obstructive Pulmonary Disease (COPD) is generally limited to mean scores of disease extent. However, the evolution of local pulmonary damage may vary between patients with discordant effects on lung physiology. This limits the explanatory power of mean values in clinical studies. We present local disease and deformation distributions to address this limitation. The disease distribution aims to quantify two aspects of parenchymal damage: locally diffuse/dense disease and global homogeneity/heterogeneity. The deformation distribution links parenchymal damage to local volume change. These distributions are exploited to quantify inter-patient differences. We used manifold learning to model variations of these distributions in 743 patients from the COPDGene study. We applied manifold fusion to combine distinct aspects of COPD into a single model. We demonstrated the utility of the distributions by comparing associations between learned embeddings and measures of severity. We also illustrated the potential to identify trajectories of disease progression in a manifold space of COPD.

Multi-scale analysis of imaging features and its use in the study of COPD exacerbation susceptible phenotypes.

We propose a novel framework for exploring patterns of respiratory pathophysiology from paired breath-hold CT scans. This is designed to enable analysis of large datasets with the view of determining relationships between functional measures, disease state and the likelihood of disease progression. The framework is based on the local distribution of image features at various anatomical scales. Principal Component Analysis is used to visualise and quantify the multi-scale anatomical variation of features, whilst the distribution subspace can be exploited within a classification setting. This framework enables hypothesis testing related to the different phenotypes implicated in Chronic Obstructive Pulmonary Disease (COPD). We illustrate the potential of our method on initial results from a subset of patients from the COPDGene study, who are exacerbation susceptible and non-susceptible.