Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences.

PURPOSE: Feature tracking and 3D surface reconstruction are key enabling techniques to computer-assisted minimally invasive surgery. One of the major bottlenecks related to training and validation of new algorithms is the lack of large amounts of annotated images that fully capture the wide range of anatomical/scene variance in clinical practice. To address this issue, we propose a novel approach to obtaining large numbers of high-quality reference image annotations at low cost in an extremely short period of time. METHODS: The concept is based on outsourcing the correspondence search to a crowd of anonymous users from an online community (crowdsourcing) and comprises four stages: (1) feature detection, (2) correspondence search via crowdsourcing, (3) merging multiple annotations per feature by fitting Gaussian finite mixture models, (4) outlier removal using the result of the clustering as input for a second annotation task. RESULTS: On average, 10,000 annotations were obtained within 24 h at a cost of $100. The annotation of the crowd after clustering and before outlier removal was of expert quality with a median distance of about 1 pixel to a publically available reference annotation. The threshold for the outlier removal task directly determines the maximum annotation error, but also the number of points removed. CONCLUSIONS: Our concept is a novel and effective method for fast, low-cost and highly accurate correspondence generation that could be adapted to various other applications related to large-scale data annotation in medical image computing and computer-assisted interventions.

Crowdsourcing for reference correspondence generation in endoscopic images.

Computer-assisted minimally-invasive surgery (MIS) is often based on algorithms that require establishing correspondences between endoscopic images. However, reference annotations frequently required to train or validate a method are extremely difficult to obtain because they are typically made by a medical expert with very limited resources, and publicly available data sets are still far too small to capture the wide range of anatomical/scene variance. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. To our knowledge, this paper is the first to investigate the concept of crowdsourcing in the context of endoscopic video image annotation for computer-assisted MIS. According to our study on publicly available in vivo data with manual reference annotations, anonymous non-experts obtain a median annotation error of 2 px (n = 10,000). By applying cluster analysis to multiple annotations per correspondence, this error can be reduced to about 1 px, which is comparable to that obtained by medical experts (n = 500). We conclude that crowdsourcing is a viable method for generating high quality reference correspondences in endoscopic video images.

Can masses of non-experts train highly accurate image classifiers? A crowdsourcing approach to instrument segmentation in laparoscopic images.

Machine learning algorithms are gaining increasing interest in the context of computer-assisted interventions. One of the bottlenecks so far, however, has been the availability of training data, typically generated by medical experts with very limited resources. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. In this work, we investigate the potential of crowdsourcing for segmenting medical instruments in endoscopic image data. Our study suggests that (1) segmentations computed from annotations of multiple anonymous non-experts are comparable to those made by medical experts and (2) training data generated by the crowd is of the same quality as that annotated by medical experts. Given the speed of annotation, scalability and low costs, this implies that the scientific community might no longer need to rely on experts to generate reference or training data for certain applications. To trigger further research in endoscopic image processing, the data used in this study will be made publicly available.

Viewpoints on Medical Image Processing: From Science to Application.

Medical image processing provides core innovation for medical imaging. This paper is focused on recent developments from science to applications analyzing the past fifteen years of history of the proceedings of the German annual meeting on medical image processing (BVM). Furthermore, some members of the program committee present their personal points of views: (i) multi-modality for imaging and diagnosis, (ii) analysis of diffusion-weighted imaging, (iii) model-based image analysis, (iv) registration of section images, (v) from images to information in digital endoscopy, and (vi) virtual reality and robotics. Medical imaging and medical image computing is seen as field of rapid development with clear trends to integrated applications in diagnostics, treatment planning and treatment.

Calibration of time-of-flight cameras for accurate intraoperative surface reconstruction.

PURPOSE: In image-guided surgery (IGS) intraoperative image acquisition of tissue shape, motion, and morphology is one of the main challenges. Recently, time-of-flight (ToF) cameras have emerged as a new means for fast range image acquisition that can be used for multimodal registration of the patient anatomy during surgery. The major drawbacks of ToF cameras are systematic errors in the image acquisition technique that compromise the quality of the measured range images. In this paper, we propose a calibration concept that, for the first time, accounts for all known systematic errors affecting the quality of ToF range images. Laboratory and in vitro experiments assess its performance in the context of IGS. METHODS: For calibration the camera-related error sources depending on the sensor, the sensor temperature and the set integration time are corrected first, followed by the scene-specific errors, which are modeled as function of the measured distance, the amplitude and the radial distance to the principal point of the camera. Accounting for the high accuracy demands in IGS, we use a custom-made calibration device to provide reference distance data, the cameras are calibrated too. To evaluate the mitigation of the error, the remaining residual error after ToF depth calibration was compared with that arising from using the manufacturer routines for several state-of-the-art ToF cameras. The accuracy of reconstructed ToF surfaces was investigated after multimodal registration with computed tomography (CT) data of liver models by assessment of the target registration error (TRE) of markers introduced in the livers. RESULTS: For the inspected distance range of up to 2 m, our calibration approach yielded a mean residual error to reference data ranging from 1.5±4.3 mm for the best camera to 7.2±11.0 mm. When compared to the data obtained from the manufacturer routines, the residual error was reduced by at least 78% from worst calibration result to most accurate manufacturer data. After registration of the CT data with the ToF data, the mean TRE ranged from 3.7±2.1 mm for point-based and 5.7±1.9 mm for surface-based registration for the best camera to 6.2±3.4 and 11.1±2.8 mm, respectively. Compared to data provided by the manufacturer, the mean TRE decreased by 8%-60% for point-based and by 18%-74% for surface-based registration. CONCLUSIONS: Using the proposed calibration approach improved the measurement accuracy of all investigated ToF cameras. Although evaluated in the context of intraoperative image acquisition, the proposed calibration procedure can easily be applied to other medical applications using ToF cameras, such as patient positioning or respiratory motion tracking in radiotherapy.

MITK-ToF--range data within MITK.

PURPOSE: The time-of-flight (ToF) technique is an emerging technique for rapidly acquiring distance information and is becoming increasingly popular for intra-operative surface acquisition. Using the ToF technique as an intra-operative imaging modality requires seamless integration into the clinical workflow. We thus aim to integrate ToF support in an existing framework for medical image processing. METHODS: MITK-ToF was implemented as an extension of the open-source C++ Medical Imaging Interaction Toolkit (MITK) and provides the basic functionality needed for rapid prototyping and development of image-guided therapy (IGT) applications that utilize range data for intra-operative surface acquisition. This framework was designed with a module-based architecture separating the hardware-dependent image acquisition task from the processing of the range data. RESULTS: The first version of MITK-ToF has been released as an open-source toolkit and supports several ToF cameras and basic processing algorithms. The toolkit, a sample application, and a tutorial are available from http://mitk.org. CONCLUSIONS: With the increased popularity of time-of-flight cameras for intra-operative surface acquisition, integration of range data supports into medical image processing toolkits such as MITK is a necessary step. Handling acquisition of range data from different cameras and processing of the data requires the establishment and use of software design principles that emphasize flexibility, extendibility, robustness, performance, and portability. The open-source toolkit MITK-ToF satisfies these requirements for the image-guided therapy community and was already used in several research projects.