Hybrid Rendering Architecture for Realtime and Photorealistic Simulation of Robot-Assisted Surgery.

In this paper we present a method for combining realtime and non-realtime (photorealistic) rendering with open source software. Realtime rendering provides sufficient realism and is a good choice for most simulation and regression testing purposes in robot-assisted surgery. However, for proper end-to-end testing of the system, some computer vision algorithms require high fidelity images that capture more minute details of the real scene. One of the central practical obstacles to combining both worlds in a uniform way is creating models that are suitable for both kinds of rendering paradigms. We build a modeling pipeline using open source tools that builds on established, open standards for data exchange. The result is demonstrated through a unified model of the medical OpenHELP phantom used in the Gazebo robotics simulator, which can at the same time be rendered with more visual fidelity in the Cycles raytracer.

Time-of-flight-assisted Kinect camera-based people detection for intuitive human robot cooperation in the surgical operating room.

BACKGROUND: Scene supervision is a major tool to make medical robots safer and more intuitive. The paper shows an approach to efficiently use 3D cameras within the surgical operating room to enable for safe human robot interaction and action perception. Additionally the presented approach aims to make 3D camera-based scene supervision more reliable and accurate. METHODS: A camera system composed of multiple Kinect and time-of-flight cameras has been designed, implemented and calibrated. Calibration and object detection as well as people tracking methods have been designed and evaluated. RESULTS: The camera system shows a good registration accuracy of 0.05 m. The tracking of humans is reliable and accurate and has been evaluated in an experimental setup using operating clothing. The robot detection shows an error of around 0.04 m. CONCLUSIONS: The robustness and accuracy of the approach allow for an integration into modern operating room. The data output can be used directly for situation and workflow detection as well as collision avoidance.

Optical coherence tomography guided laser cochleostomy: towards the accuracy on tens of micrometer scale.

Lasers have been proven to be precise tools for bone ablation. Applying no mechanical stress to the patient, they are potentially very suitable for microsurgery on fragile structures such as the inner ear. However, it remains challenging to control the laser-bone ablation without injuring embedded soft tissue. In this work, we demonstrate a closed-loop control of a short-pulsed CO2 laser to perform laser cochleostomy under the monitoring of an optical coherence tomography (OCT) system. A foresighted detection of the bone-endosteum-perilymph boundary several hundred micrometers before its exposure has been realized. Position and duration of the laser pulses are planned based on the residual bone thickness distribution. OCT itself is also used as a highly accurate tracking system for motion compensation between the target area and the optics. During ex vivo experimental evaluation on fresh porcine cochleae, the ablation process terminated automatically when the thickness of the residual tissue layer uniformly reached a predefined value. The shape of the resulting channel bottom converged to the natural curvature of the endosteal layer without injuring the critical structure. Preliminary measurements in OCT scans indicated that the mean absolute accuracy of the shape approximation was only around 20 µm.

Low-distortion perturbative solutions in a camera calibration model including high-distortion radial effects.

PURPOSE: An endoscope fisheye lens camera calibration model was extended using intuitive vector parameters to describe lens distortion with a fourth degree radial polynomial. METHOD: An existing algorithm was expanded to the wider domain of applicability of the high endoscopic distorted lenses. By completely decoupling the computation of the camera distortion center obtained using general properties of distortion lines, even strong radial distortion effects can be perturbatively included into a pure perspective first guess zero-distortion solution. RESULTS: A perturbative and iterative solution used to model small amounts of lens distortion can be maintained in case of strong distortion lenses, such as the endoscope fisheye, with satisfactory results. CONCLUSION: We demonstrated that using an educated guess for the initial solution based on the final location of the lens distortion center, the intuitive vector-based model's full solution can be iteratively obtained starting from a purely perspective distortion-free first guess solution.

Planning and simulation of microsurgical laser bone ablation.

PURPOSE: Laser ablation of hard tissue is not completely understood until now and not modeled for computer-assisted microsurgery. A precise planning and simulation is an essential step toward the usage of microsurgical laser bone ablation in the operating room. METHODS: Planning the volume for laser bone ablation is based on geometrical definitions. Shape and volume of the removed bone by single laser pulses were measured with a confocal microscope for modeling the microsurgical ablation. To remove the planned volume and to achieve smooth surfaces, a simulation of the laser pulse distribution is developed. RESULTS: The confocal measurements show a clear dependency from laser energy and resulting depth. Two-dimensional Gaussian functions are fitting in these craters. Exemplarily three ablation layers were planned, simulated, executed and verified. CONCLUSIONS: To model laser bone ablation in microsurgery the volume and shape of each laser pulse should be known and considered in the process of ablation planning and simulation.

Including parameterization of the discrete ablation process into a planning and simulation environment for robot-assisted laser osteotomy.

Material processing using laser became a widely used method especially in the scope of industrial automation. The systems are mostly based on a precise model of the laser process and the according parameterization. Beside the industrial use the laser as an instrument to treat human tissue has become an integral part in medicine as well. Human tissue as an inhomogeneous material to process, poses the question of how to determine a model, which reflects the interaction processes with a specific laser.Recently it could be shown that the pulsed CO2 laser is suitable to ablate bony and cartilage tissue. Until now this thermo-mechanical bone ablation is not characterized as a discrete process. In order to plan and simulate the ablation process in the correct level of detail, the parameterization is indispensable. We developed a planning and simulation environment, determined parameters by confocal measurements of bony specimen and use these results to transfer planned cutting trajectories into a pulse sequence and corresponding robot locations.

New strategies for high precision surgery of the temporal bone using a robotic approach for cochlear implantation.

The aim of the study was to demonstrate a collision-free trajectory of an instrument through the facial recess to the site of planned cochleostomy guided by a surgery robot. The indication for cochlear implantation is still expanding toward more substantial residual hearing. A cochleostomy as atraumatic as possible will influence the preservation of inner ear function. The employment of a highly precise instrument guidance using a robot could represent a feasible solution for a constant reproducible surgical procedure. Screw markers for a point-based registration were fixed on a human temporal bone specimen prepared with a mastoidectomy and posterior tympanotomy. A DICOM dataset has been generated thereof in a 64-multislice computer tomography (CT). A virtual trajectory in a 3D model has been planned representing the path of instrumentation toward the desired spot of cochleostomy. A 1.9-mm endoscope has been mounted onto the robot system RobaCKa (Staeubli RX90CR) to visualize this trajectory. The target registration error added up to 0.25 mm, which met the desirable tolerance of <0.5 mm. A collision-free propagation of the endoscope into the tympanic cavity via the facial recess has been performed by the robot and the spot of cochleostomy could be visualized through the endoscope. Using a DICOM dataset of a high-resolution CT and a robot as a positioning platform for surgical instruments could be a feasible approach to perform a highly precise and constant reproducible cochleostomy. Furthermore, it could be a crucial step to preserve substantial residual hearing in terms of expanding the indications for cochlear implantation.

System design of a hand-held mobile robot for craniotomy.

This contribution reports the development and initial testing of a Mobile Robot System for Surgical Craniotomy, the Craniostar. A kinematic system based on a unicycle robot is analysed to provide local positioning through two spiked wheels gripping directly onto a patients skull. A control system based on a shared control system between both the Surgeon and Robot is employed in a hand-held design that is tested initially on plastic phantom and swine skulls. Results indicate that the system has substantially lower risk than present robotically assisted craniotomies, and despite being a hand-held mobile robot, the Craniostar is still capable of sub-millimetre accuracy in tracking along a trajectory and thus achieving an accurate transfer of pre-surgical plan to the operating room procedure, without the large impact of current medical robots based on modified industrial robots.

Intraoperative augmented reality: the surgeons view.

Augmented Reality (AR) is a promising tool for intraoperative visualization. Two different AR systems, one projector based, one based on see-through glasses were used on patients. The task was the transfer of preoperative planning into the intraoperative reality, or the visualization of space occupying lesions, respectively. The intraoperative application of both systems is discussed from the surgeons point of view.

Visualization of surgical 3D information with projector-based augmented reality.

For visualizing surgical information (operation plans) directly onto the patient a projector-based augmented reality system is used for cranio-maxillofacial surgery. A prototype is introduced which has been evaluated in the first clinical cases. In a new setup with a second video projector it is now possible to give additionally 3D information for localization and orientation (6DoF). With this method the repositioning of a bone segment is intuitive and exact applicable.

Development and first patient trial of a surgical robot for complex trajectory milling.

OBJECTIVE: Today's surgical robots normally perform "simple" trajectories, e.g., assisting as tool-holding devices in neurosurgery, or milling linear paths for cavities in total hip replacement. From a clinical point of view, it is still a complex undertaking to implement robots in the operating room. Until now, robot systems have not been used in patient trials to mill "complex" trajectories, which involve many positional and orientation changes and are often necessary in cranio-maxillofacial (CMF) surgery. This paper presents the RobaCKa surgical robot system, which allows more precise execution of surgical interventions and milling of "complex" trajectories. MATERIALS AND METHODS: The main components of the RobaCKa system are a (former) CASPAR robot system, a POLARIS system, and a force-torque sensor. RESULTS: In the first patient trial (April 2003) the planned trajectory was executed with an error of 0.66 +/- 0.2 mm. CONCLUSIONS: The use of former industrial robots for surgical applications is possible but complex. The advantages are improved precision and quality and the possibility of documentation. The use of such systems is normally limited to research institutions or large clinics, because it is hardly possible to implement the necessary technical and logistic efforts in routine surgical work.

Creating a statistical atlas of the cranium.

Normative data is very important for simulation procedures in craniofacial surgery. While treating e.g. a malformed skull the surgeon seeks to reconstruct its natural and harmonic shape. Atlas or normative data of the skull could support the surgeon in this effort, as it would provide a standard model of the skull which gives an idea of the natural shape. We create a standard skull by averaging regularly formed skulls in a shape space spanned by spherical harmonics. While state-of-the-art methods use landmarks to define the shape and mean shapes, this method is deterministic, i.e. it manages averaging without landmarks and it provides a complete description of the shape. In addition the shape space can be used to classify shapes to identify different types of an anatomy.

Intraoperative guidance of pre-planned bone deformations with a surface scanning system.

Computer- and robot-based systems to support interventions become more and more important in modem surgery. In general these systems provide methods to plan an intervention pre-operatively and to execute it with support from a autonomous robot-system. Due to the principle restriction of a robot to comparatively simple work steps, there are some complex work steps which the surgeon may plan but which he/she has to execute manually. In craniofacial surgery osteotomised bone segments are deformed by hand to a shape given by the planning system. We support the execution of pre-planned deformation by comparison of the actual shape of an object with the target shape. The actual shape is obtained intra-operatively with a surface scanning device, the deviation from the target shape are visualised by projecting colour-coded error values directly on the object to be deformed. The surgeon uses these projections to adjust further deformation steps. The system is therefore able to validate the correct execution of planned deformations, especially of bony structures.

Atlas-based segmentation of pathological knee joints.

Efficiency, comparability and simplicity are key aspects for user acceptance of surgical planning systems in the long term. Automatic segmentation and identification of geometric reference systems of the anatomical structures are essential to fulfill these requirements. A statistical motivated shape atlas of the knee joint, based on 235 normal and abnormal MR and CT volume sets, is constructed for automatic segmentation of CT image data. In the first step of the atlas construction, the bony structures of the knee were segmented semi-automatically and processed into a dense and a sparse triangulated surface mesh to obtain training data sets. To establish an inter-individual correspondence, a skeleton-based registration method is used. The registered sparse surface meshes are retriangulated to estimate a pointwise inter-individual correspondence. The shape atlas is build upon these correspondences and integrated into a segmentation algorithm. An iterative segmentation scheme is proposed, which consists of a combination of the iterative-closest-point algorithm for spatial registration and of a downhill-simplex optimization procedure for deformation of the statistical motivated shape atlas to the image data. We expect the statistical shape model to be a robust and image modality independent method for the segmentation of pathological knee joints in CT image data.

A new, accurate and easy to implement camera and video projector model.

In 2000, the Institute for Process Control and Robotics/Universität Karlsruhe (TH) has developed a prototype system for projector based augmented reality consisting of a state-of-the-art PC, two CCD cameras and a video projector which is used for registration and projection of surgical planning data. Tracking, registration as well as projection require an accurate calibration process for cameras and video projectors. We have developed a new, flexible, plain and easy to implement model, which can both be used for calibration of cameras and video projectors.

Realtime textured 3D-models for medical applications.

Realistic visualisation becomes more and more important in medicine. Whenever a patient individual 3D-model was generated the aim is to visualise the model as realistic as possible. We use 3D-models in our diagnostic and therapeutic tools for intraoperative visualisation of e.g. CT-scans. Most medical tools uses surface-rendering or volume-rendering for virtual visualisation. The coloration of a visualised model is normally done by using a convenient colour for each surface resp. volume. Our approach for 3D-models generated from motion in image series (e.g. videoendoscopes) is to add textures to the 3D-model. The main problem is, to handle the huge amount of videoimage-data (20Mb/sec.) and render the model in realtime.

3D structure from endoscopic images.

Endoscopy is an important procedure for the diagnostic and therapy of various pathologies. In 2000, the Institute for Process Control and Robotics/Universität Karlsruhe (TH) has developed a basic framework for reconstructing 3D models from image sequences. The framework is able to realise automatic navigation for new colonoscopes with own driving system and on the other hand to offer a virtual 3D endoscopic view during bad visibility conditions caused by e.g. abrupt bleeding. This method generates a 3D-model intraoperatively compared to preoperatively generated 3D-models in virtual endoscopy.

CT, MRI and video based analysis of knee kinematics--a basis for CT based simulation.

With the new computer aided surgery techniques a surgery can be split in two phases, which are pre-operative planning and intervention. The planning is frequently based on three dimensional image data and takes place in the office of the surgeon. Using this approach the surgical feeling, the physical feedback of the patient, is not available during the planning phase. However, this feedback yields information, which is helpful or necessary for the planning. This lack of information can be compensated by a simulation module, which simulates kinematic data from image volume data. To develop such a module first both image volume data and kinematic data have to be captured from a wide range of patients. In this work a new approach for kinematic analysis of the knee is presented which yields an accuracy which is necessary for such simulation. Because the analysis has to be done for many patients the degree of invasiveness is emphasized. This contactless and non invasive approach is based on external fixations, anatomical skin markers, video sequences and computer tomograms. Position tracking of femur and tibia for any motion pattern is possible. Because the individual anatomic structures extracted from the tomogram are tracked, an analysis of any point or coordinate system can be done later using the originally captured data. This gives a new degree of freedom for analysis.