EndoViT: pretraining vision transformers on a large collection of endoscopic images.

PURPOSE: Automated endoscopy video analysis is essential for assisting surgeons during medical procedures, but it faces challenges due to complex surgical scenes and limited annotated data. Large-scale pretraining has shown great success in natural language processing and computer vision communities in recent years. These approaches reduce the need for annotated data, which is of great interest in the medical domain. In this work, we investigate endoscopy domain-specific self-supervised pretraining on large collections of data. METHODS: To this end, we first collect Endo700k, the largest publicly available corpus of endoscopic images, extracted from nine public Minimally Invasive Surgery (MIS) datasets. Endo700k comprises more than 700,000 images. Next, we introduce EndoViT, an endoscopy-pretrained Vision Transformer (ViT), and evaluate it on a diverse set of surgical downstream tasks. RESULTS: Our findings indicate that domain-specific pretraining with EndoViT yields notable advantages in complex downstream tasks. In the case of action triplet recognition, our approach outperforms ImageNet pretraining. In semantic segmentation, we surpass the state-of-the-art (SOTA) performance. These results demonstrate the effectiveness of our domain-specific pretraining approach in addressing the challenges of automated endoscopy video analysis. CONCLUSION: Our study contributes to the field of medical computer vision by showcasing the benefits of domain-specific large-scale self-supervised pretraining for vision transformers. We release both our code and pretrained models to facilitate further research in this direction: https://github.com/DominikBatic/EndoViT .

CholecTriplet2022: Show me a tool and tell me the triplet - An endoscopic vision challenge for surgical action triplet detection.

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

Comparative immuno-Cerenkov luminescence and -PET imaging enables detection of PSMA+ tumors in mice using 64Cu-radiolabeled monoclonal antibodies.

Here, we describe immuno-Cerenkov luminescence imaging (immuno-CLI) with a specific monoclonal antibody-based tracer for the detection of prostate tumors, which is used in preclinical positron emission tomography (PET) imaging. As PET isotopes generate a continuous spectrum of light in the ultraviolet/visible (UV/vis) wavelength range (Cerenkov luminescence, CL) in dielectric materials and consequently inside living tissues, these isotopes can also be detected by luminescence imaging performed with optical imaging (OI) systems. Imaging tumors with tracers that are specifically binding to a tumor-associated antigen can increase diagnostic accuracy, enables monitoring of treatment efficacy, and can be advantageous compared to radiolabeled small molecules used in PET-oncology such as 2-deoxy-2-[18F]-fluoro-D-glucose ([18F]FDG; glucose metabolism) or [11C]choline (membrane synthesis) which was used to image prostate cancer. In this study, we compared on three consecutive days immuno-CLI and -PET of the applied 64Cu-labeled and well described monoclonal antibody 3/F11 in prostate-specific membrane antigen (PSMA)-positive (C4-2, PSMA+) and -negative (DU 145, PSMA-) prostate tumor xenografts, inoculated in SCID mice. In vivo immuno-CLI and -PET measurements demonstrated linear correlation of both modalities, in line with ex vivo analysis performed with CLI and γ-counting. As CLI is also able to trace radioisotopes used for theranostic approaches, immuno-CLI could be an interesting, low-cost imaging alternative to immuno-PET.