Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology.

Multi-modal medical image segmentation is a crucial task in oncology that enables the precise localization and quantification of tumors. The aim of this work is to present a meta-analysis of the use of multi-modal medical Transformers for medical image segmentation in oncology, specifically focusing on multi-parametric MR brain tumor segmentation (BraTS2021), and head and neck tumor segmentation using PET-CT images (HECKTOR2021). The multi-modal medical Transformer architectures presented in this work exploit the idea of modality interaction schemes based on visio-linguistic representations: (i) single-stream, where modalities are jointly processed by one Transformer encoder, and (ii) multiple-stream, where the inputs are encoded separately before being jointly modeled. A total of fourteen multi-modal architectures are evaluated using different ranking strategies based on dice similarity coefficient (DSC) and average symmetric surface distance (ASSD) metrics. In addition, cost indicators such as the number of trainable parameters and the number of multiply-accumulate operations (MACs) are reported. The results demonstrate that multi-path hybrid CNN-Transformer-based models improve segmentation accuracy when compared to traditional methods, but come at the cost of increased computation time and potentially larger model size.

Automatic Head and Neck Tumor segmentation and outcome prediction relying on FDG-PET/CT images: Findings from the second edition of the HECKTOR challenge.

By focusing on metabolic and morphological tissue properties respectively, FluoroDeoxyGlucose (FDG)-Positron Emission Tomography (PET) and Computed Tomography (CT) modalities include complementary and synergistic information for cancerous lesion delineation and characterization (e.g. for outcome prediction), in addition to usual clinical variables. This is especially true in Head and Neck Cancer (HNC). The goal of the HEad and neCK TumOR segmentation and outcome prediction (HECKTOR) challenge was to develop and compare modern image analysis methods to best extract and leverage this information automatically. We present here the post-analysis of HECKTOR 2nd edition, at the 24th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2021. The scope of the challenge was substantially expanded compared to the first edition, by providing a larger population (adding patients from a new clinical center) and proposing an additional task to the challengers, namely the prediction of Progression-Free Survival (PFS). To this end, the participants were given access to a training set of 224 cases from 5 different centers, each with a pre-treatment FDG-PET/CT scan and clinical variables. Their methods were subsequently evaluated on a held-out test set of 101 cases from two centers. For the segmentation task (Task 1), the ranking was based on a Borda counting of their ranks according to two metrics: mean Dice Similarity Coefficient (DSC) and median Hausdorff Distance at 95th percentile (HD95). For the PFS prediction task, challengers could use the tumor contours provided by experts (Task 3) or rely on their own (Task 2). The ranking was obtained according to the Concordance index (C-index) calculated on the predicted risk scores. A total of 103 teams registered for the challenge, for a total of 448 submissions and 29 papers. The best method in the segmentation task obtained an average DSC of 0.759, and the best predictions of PFS obtained a C-index of 0.717 (without relying on the provided contours) and 0.698 (using the expert contours). An interesting finding was that best PFS predictions were reached by relying on DL approaches (with or without explicit tumor segmentation, 4 out of the 5 best ranked) compared to standard radiomics methods using handcrafted features extracted from delineated tumors, and by exploiting alternative tumor contours (automated and/or larger volumes encompassing surrounding tissues) rather than relying on the expert contours. This second edition of the challenge confirmed the promising performance of fully automated primary tumor delineation in PET/CT images of HNC patients, although there is still a margin for improvement in some difficult cases. For the first time, the prediction of outcome was also addressed and the best methods reached relatively good performance (C-index above 0.7). Both results constitute another step forward toward large-scale outcome prediction studies in HNC.

Overview of the HECKTOR Challenge at MICCAI 2022: Automatic Head and Neck Tumor Segmentation and Outcome Prediction in PET/CT.

This paper presents an overview of the third edition of the HEad and neCK TumOR segmentation and outcome prediction (HECKTOR) challenge, organized as a satellite event of the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2022. The challenge comprises two tasks related to the automatic analysis of FDG-PET/CT images for patients with Head and Neck cancer (H&N), focusing on the oropharynx region. Task 1 is the fully automatic segmentation of H&N primary Gross Tumor Volume (GTVp) and metastatic lymph nodes (GTVn) from FDG-PET/CT images. Task 2 is the fully automatic prediction of Recurrence-Free Survival (RFS) from the same FDG-PET/CT and clinical data. The data were collected from nine centers for a total of 883 cases consisting of FDG-PET/CT images and clinical information, split into 524 training and 359 test cases. The best methods obtained an aggregated Dice Similarity Coefficient (DSCagg) of 0.788 in Task 1, and a Concordance index (C-index) of 0.682 in Task 2.

Development of Radiomic-Based Model to Predict Clinical Outcomes in Non-Small Cell Lung Cancer Patients Treated with Immunotherapy.

Purpose: We aimed to assess the ability of radiomics features extracted from baseline (PET/CT0) and follow-up PET/CT scans, as well as their evolution (delta-radiomics), to predict clinical outcome (durable clinical benefit (DCB), progression, response to therapy, OS and PFS) in non-small cell lung cancer (NSCLC) patients treated with immunotherapy. Methods: 83 NSCLC patients treated with immunotherapy who underwent a baseline PET/CT were retrospectively included. Response was assessed at 6−8 weeks (PET/CT1) using PERCIST criteria and at 3 months with iPERCIST (PET/CT2) or RECIST 1.1 criteria using CT. The predictive performance of clinical parameters (CP), standard PET metrics (SUV, Metabolic Tumor volume, Total Lesion Glycolysis), delta-radiomics and PET and CT radiomics features extracted at baseline and during follow-up were studied. Seven multivariate models with different combinations of CP and radiomics were trained on a subset of patients (75%) using least absolute shrinkage, selection operator (LASSO) and random forest classification with 10-fold cross-validation to predict outcome. Model validation was performed on the remaining patients (25%). Overall and progression-free survival was also performed by Kaplan−Meier survival analysis. Results: Numerous radiomics and delta-radiomics parameters had a high individual predictive value of patient outcome with areas under receiver operating characteristics curves (AUCs) >0.80. Their performance was superior to that of CP and standard PET metrics. Several multivariate models were also promising, especially for the prediction of progression (AUCs of 1 and 0.96 for the training and testing subsets with the PET-CT model (PET/CT0)) or DCB (AUCs of 0.85 and 0.83 with the PET-CT-CP model (PET/CT0)). Conclusions: Delta-radiomics and radiomics features extracted from baseline and follow-up PET/CT images could predict outcome in NSCLC patients treated with immunotherapy and identify patients who would benefit from this new standard. These data reinforce the rationale for the use of advanced image analysis of PET/CT scans to further improve personalized treatment management in advanced NSCLC.

Accurate Tumor Delineation vs. Rough Volume of Interest Analysis for 18F-FDG PET/CT Radiomics-Based Prognostic Modeling inNon-Small Cell Lung Cancer.

BACKGROUND: The aim of this work was to investigate the ability of building prognostic models in non-small cell lung cancer (NSCLC) using radiomic features from positron emission tomography and computed tomography with 2-deoxy-2-[fluorine-18]fluoro-d-glucose (18F-FDG PET/CT) images based on a "rough" volume of interest (VOI) containing the tumor instead of its accurate delineation, which is a significant time-consuming bottleneck of radiomics analyses. METHODS: A cohort of 138 patients with stage II-III NSCLC treated with radiochemotherapy recruited retrospectively (n = 87) and prospectively (n = 51) was used. Two approaches were compared: firstly, the radiomic features were extracted from the delineated primary tumor volumes in both PET (using the automated fuzzy locally adaptive Bayesian, FLAB) and CT (using a semi-automated approach with 3D Slicer™) components. Both delineations were carried out within previously manually defined "rough" VOIs containing the tumor and the surrounding tissues, which were exploited for the second approach: the same features were extracted from this alternative VOI. Both sets for features were then combined with the clinical variables and processed through the same machine learning (ML) pipelines using the retrospectively recruited patients as the training set and the prospectively recruited patients as the testing set. Logistic regression (LR), random forest (RF), and support vector machine (SVM), as well as their consensus through averaging the output probabilities, were considered for feature selection and modeling for overall survival (OS) prediction as a binary classification (either median OS or 6 months OS). The resulting models were compared in terms of balanced accuracy, sensitivity, and specificity. RESULTS: Overall, better performance was achieved using the features from delineated tumor volumes. This was observed consistently across ML algorithms and for the two clinical endpoints. However, the loss of performance was not significant, especially when a consensus of the three ML algorithms was considered (0.89 vs. 0.88 and 0.78 vs. 0.77). CONCLUSION: Our findings suggest that it is feasible to achieve similar levels of prognostic accuracy in radiomics-based modeling by relying on a faster and easier VOI definition, skipping a time-consuming tumor delineation step, thus facilitating automation of the whole radiomics workflow. The associated cost is a loss of performance in the resulting models, although this loss can be greatly mitigated when a consensus of several models is relied upon.

Comparison and Fusion of Machine Learning Algorithms for Prospective Validation of PET/CT Radiomic Features Prognostic Value in Stage II-III Non-Small Cell Lung Cancer.

Machine learning (ML) algorithms for selecting and combining radiomic features into multiparametric prediction models have become popular; however, it has been shown that large variations in performance can be obtained by relying on different approaches. The purpose of this study was to evaluate the potential benefit of combining different algorithms into an improved consensus for the final prediction, as it has been shown in other fields. Methods: The evaluation was carried out in the context of the use of radiomics from 18F-FDG PET/CT images for predicting outcome in stage II-III Non-Small Cell Lung Cancer. A cohort of 138 patients was exploited for the present analysis. Eighty-seven patients had been previously recruited retrospectively for another study and were used here for training and internal validation. We also used data from prospectively recruited patients (n = 51) for testing. Three different machine learning pipelines relying on embedded feature selection were trained to predict overall survival (OS) as a binary classification: Support Vector machines (SVMs), Random Forests (RFs), and Logistic Regression (LR). Two different clinical endpoints were investigated: median OS or OS shorter than 6 months. The fusion of the three approaches was implemented using two different strategies: majority voting on the binary outputs or averaging of the output probabilities. Results: Our results confirm previous findings, highlighting that different ML pipelines select different sets of features and reach different classification performances (accuracy in the testing set ranging between 63% and 67% for median OS, and between 75% and 80% for OS < 6 months). Generating a consensus improved the performance for both endpoints; with the probabilities averaging strategy outperforming the majority voting (accuracy of 78% vs. 71% for median OS and 89 vs. 84% for OS < 6 months). Overall, the performance of these radiomic-based models outperformed the standard clinical staging in both endpoints (accuracy of 58% and 53% accuracy in the testing set for each endpoint). Conclusion: Although obtained in a small cohort of patients, our results suggest that a consensus of machine learning algorithms can improve performance in the context of radiomics. The resulting prognostic stratification in the prospective testing cohort is higher than when relying on the clinical stage. This could be of interest for clinical practice as it could help to identify patients with higher risk amongst stage II and III patients, who could benefit from intensified treatment and/or more frequent follow-up after treatment.

Radiomics in PET/CT: Current Status and Future AI-Based Evolutions.

This short review aims at providing the readers with an update on the current status, as well as future perspectives in the quickly evolving field of radiomics applied to the field of PET/CT imaging. Numerous pitfalls have been identified in study design, data acquisition, segmentation, features calculation and modeling by the radiomics community, and these are often the same issues across all image modalities and clinical applications, however some of these are specific to PET/CT (and SPECT/CT) imaging and therefore the present paper focuses on those. In most cases, recommendations and potential methodological solutions do exist and should therefore be followed to improve the overall quality and reproducibility of published studies. In terms of future evolutions, the techniques from the larger field of artificial intelligence (AI), including those relying on deep neural networks (also known as deep learning) have already shown impressive potential to provide solutions, especially in terms of automation, but also to maybe fully replace the tools the radiomics community has been using until now in order to build the usual radiomics workflow. Some important challenges remain to be addressed before the full impact of AI may be realized but overall the field has made striking advances over the last few years and it is expected advances will continue at a rapid pace.

Can alternative PET reconstruction schemes improve the prognostic value of radiomic features in non-small cell lung cancer?

PURPOSE: To evaluate the potential benefit of using alternative reconstruction schemes of PET images for the prognostic value of radiomic features. METHODS: Patients (n=91) with non-small cell lung cancer were prospectively included. All had a PET/CT examination before treatment. Three different PET images were reconstructed for each patient: the standard clinical protocol (i.e., 4×4×4 mm3 voxels, 5mm Gaussian filter, denoted '200G5'), as well as using smaller voxels (i.e., 2×2×2 mm3 with a larger reconstruction matrix, denoted 400G1) and/or 1mm post-reconstruction Gaussian filter, denoted 200G1). Metabolic volumes of the primary tumors were semi-automatically delineated on the PET images and IBSI compliant radiomic features (intensity, shape, textural) were extracted. First, the distributions of 200G1 and 400G1 features were compared to the reference clinical protocol (200G5) through Bland-Altman tests and the use of linear mixed models. Then, the prognostic value of the features from each of the 3 reconstructions was evaluated in a univariate analysis, through their stratification power in Kaplan-Meier curves through a threshold set at the median. RESULTS: The 3 reconstructions led to different distributions for most of the features. The larger shifts and standard deviations of differences was observed between 200G5 and 400G1, which was also confirmed through linear mixed models. However, these relatively important differences in distributions did not translate into a significant impact on the stratification power of the features in terms of prognosis, although a trend in decreasing prognostic value could be observed (smaller number of features with HR above 2, overall lower HR values). Most prognostic features displayed high correlation with either volume or SUVmax, although there was great variability of prognostic value for similar levels of correlation with these basic metrics. CONCLUSIONS: Using smaller voxels or less strong filtering options in the reconstruction settings of PET images compared to the standard clinical protocols led to different distributions of the resulting radiomic features. However, the hierarchy between patients according to these distributions remained overall the same and therefore the resulting stratification power of the radiomic features was not significantly altered. These results should be compared to those obtained in the context of other pathologies where radiomic features displaying lower correlation with volume or SUVmax may have predictive value, such as in cervical cancer.

Directed Connected Operators: Asymmetric Hierarchies for Image Filtering and Segmentation.

Connected operators provide well-established solutions for digital image processing, typically in conjunction with hierarchical schemes. In graph-based frameworks, such operators basically rely on symmetric adjacency relations between pixels. In this article, we introduce a notion of directed connected operators for hierarchical image processing, by also considering non-symmetric adjacency relations. The induced image representation models are no longer partition hierarchies (i.e., trees), but directed acyclic graphs that generalize standard morphological tree structures such as component trees, binary partition trees or hierarchical watersheds. We describe how to efficiently build and handle these richer data structures, and we illustrate the versatility of the proposed framework in image filtering and image segmentation.

Symbolic representation of brain vascular network with Arteriovenous Malformations from 3DRA images.

Vascular imaging is crucial in the treatment of many diseases. In the case of cerebral ArterioVenous Malformation (AVM), where the vascular network can be deeply altered, an accurate knowledge of its topology is required. For this purpose, after a vessels segmentation and skeletization applied on 3D rotational angiographic images (3DRA), we build a symbolic tree representation of the vascular network thanks to topological descriptors, such as end points, junctions and branches. This leads to an efficient tool to assist the neuroradiologist to understand the feeding and the draining of the AVM and to apprehend its complex architecture in order to determine the best therapeutic strategy before and during embolization interventions.