A contextual detector of surgical tools in laparoscopic videos using deep learning.

BACKGROUND: The complexity of laparoscopy requires special training and assessment. Analyzing the streaming videos during the surgery can potentially improve surgical education. The tedium and cost of such an analysis can be dramatically reduced using an automated tool detection system, among other things. We propose a new multilabel classifier, called LapTool-Net to detect the presence of surgical tools in each frame of a laparoscopic video. METHODS: The novelty of LapTool-Net is the exploitation of the correlations among the usage of different tools and, the tools and tasks-i.e., the context of the tools' usage. Towards this goal, the pattern in the co-occurrence of the tools is utilized for designing a decision policy for the multilabel classifier based on a Recurrent Convolutional Neural Network (RCNN), which is trained in an end-to-end manner. In the post-processing step, the predictions are corrected by modeling the long-term tasks' order with an RNN. RESULTS: LapTool-Net was trained using publicly available datasets of laparoscopic cholecystectomy, viz., M2CAI16 and Cholec80. For M2CAI16, our exact match accuracies (when all the tools in one frame are predicted correctly) in online and offline modes were 80.95% and 81.84% with per-class F1-score of 88.29% and 90.53%. For Cholec80, the accuracies were 85.77% and 91.92% with F1-scores if 93.10% and 96.11% for online and offline, respectively. CONCLUSIONS: The results show LapTool-Net outperformed state-of-the-art methods significantly, even while using fewer training samples and a shallower architecture. Our context-aware model does not require expert's domain-specific knowledge, and the simple architecture can potentially improve all existing methods.

Multibody structure-and-motion segmentation by branch-and-bound model selection.

An efficient and robust framework is proposed for two-view multiple structure-and-motion segmentation of unknown number of rigid objects. The segmentation problem has three unknowns, namely the object memberships, the corresponding fundamental matrices, and the number of objects. To handle this otherwise recursive problem, hypotheses for fundamental matrices are generated through local sampling. Once the hypotheses are available, a combinatorial selection problem is formulated to optimize a model selection cost which takes into account the hypotheses likelihoods and the model complexity. An explicit model for outliers is also added for robust segmentation. The model selection cost is minimized through the branch-and-bound technique of combinatorial optimization. The proposed branch-and-bound approach efficiently searches the solution space and guarantees optimality over the current set of hypotheses. The efficiency and the guarantee of optimality of the method is due to its ability to reject solutions without explicitly evaluating them. The proposed approach was validated with synthetic data, and segmentation results are presented for real images.

Multistage branch-and-bound merging for planar surface segmentation in disparity space.

An iterative split-and-merge framework for the segmentation of planar surfaces in the disparity space is presented. Disparity of a scene is modeled by approximating various surfaces in the scene to be planar. In the split phase, the number of planar surfaces along with the underlying plane parameters is assumed to be known from the initialization or from the previous merge phase. Based on these parameters, planar surfaces in the disparity image are labeled to minimize the residuals between the actual disparity and the modeled disparity. The labeled planar surfaces are separated into spatially continuous regions which are treated as candidates for the merging that follows. The regions are merged together under a maximum variance constraint while maximizing the merged area. A multistage branch-and-bound algorithm is proposed to carry out this optimization efficiently. Each stage of the branch-and-bound algorithm separates a planar surface from the set of spatially continuous regions. The multistage merging estimates the number of planar surfaces and their labeling. The splitting and the multistage merging is repeated till convergence is reached or satisfactory results are achieved. Experimental results are presented for variety of stereo image data.

Design and simulation of a visual and haptic assisted biopsy (ViHAB) system.

Core needle biopsy is a non-invasive technique for confirming breast and prostate cancer. Several non-real time image based systems have been developed to guide the needle to the target. X-ray, ultrasound (US), MRI or x-ray fluoroscopy are used to guide the needle during biopsy. However, these methods are non-real time or, the imaging technique is two dimensional or, ionizing. Our broad objective is to develop a visually guided, haptically assisted breast biopsy system (ViHAB) using real time 3D US imaging and haptic guidance. ViHAB will help the radiologist identify suspicious tissue and provide real time guidance for core needle biopsy. The ViHAB simulator developed at the Virtual Environment Laboratory is capable of reading and displaying 3-D US images, keep track of micro-calcification in near real time in a sequence of images as well as provide haptic and visual guidance to a virtual needle via a joystick.

Parameter optimization for 3D mass-spring-damper models.

In this paper, we investigate the physical accuracy of the 3D mass-spring-damper (MSD) model of an isotropic object. The isotropic object should have the same Poisson constant and Young's modulus in different directions, and so should its model. Based on these two properties, we derive a set of constraints on the parameters of the 3D MSD model. From these constraints, the parameters of the MSD model can be obtained by the constrained least square method. For the MSD model with tetrahedral meshes, we show that its tensile stiffness can be achieved very accurately, and the prone irregular Poisson effects can be suppressed below a tolerable level although its Poisson constant generally cannot be precisely achieved. For the MSD model with hexahedral meshes, we find that the parameters of the model can be obtained explicitly in terms of material properties and mesh geometry. In this case, we also demonstrate that both the tensile stiffness and the Poisson constant can be accurately achieved.

Modeling isotropic organs using beam models for the haptic simulation of blunt dissections.

Haptic modeling of organs using existing approaches is still not realistic or real time. We propose and develop the mathematical foundation of a new approach to modeling organs using beams. Beams are well known entities in Civil and Structural engineering. We develop their mathematical properties in the context of organ simulation. The real time advantage arises from the fact that a single beam implementation eliminates hundreds, if not thousands of mass springs from the traditional mass spring models and, thousands of polygons from the finite element method. Even more importantly, our derivation is valid for large deformation. Most previous work has developed equations only for small deflections. Large deformation is important because we set out to simulate blunt cutting which requires models for large deflections. Our new model, when simulated and compared with an FEM model provides comparable accuracy.

Determination of key and driving points of a beam model for tissue simulation.

The simulation of catheter, guide wire, rigid tissues, muscles and blood vessels using conventional methods like mass spring model and FEM are computationally expensive. The former is comparatively faster than the latter but less accurate. Earlier, we proposed a new method of simulating of tubular organs using deformable beam models. This method is not only accurate but also promises to be faster than mass-spring model for the simulation of tissues. This paper focuses on an important aspect of this approach--the determination of key and driving points of a beam model.

Selective tessellation algorithm for modeling interactions between surgical instruments and tissues.

We present a selective spatial tessellation algorithm that is specifically optimized for instrument-to-tissue and instrument-to-instrument collision detection cases, which are the essential part of interaction modeling in surgery simulation with haptic feedback. Virtual surgeries demand haptic rate collision solutions only when instruments are involved in collisions. Other collision cases can be processed at slower rates. The proposed selective tessellation algorithm is capable of differentiating among various collision cases and assigning different priorities to their processing. Without making assumptions about any object movement, the algorithm derives clipping volume as collision detection regions which tightly enclose the objects of interest. Results of implementation of the algorithm in a surgical simulation are provided.

Physically accurate mesh simulation in a laparoscopic hernia surgery simulator.

In this paper we use the 2D angular spring based mass-spring-damper (AMSD) model to simulate the plastic mesh in a laparoscopic hernia surgery simulator. We propose a physically based method to systematically derive the optimal parameters of the 2D AMSD model. While the traditional 2D MSD model lacks resistance against bending, the 2D AMSD model with optimized parameters can provide correct bending resistance as well as stretching resistance. The simulated mesh is demonstrated to be much more realistic.

A novel laparoscopic mesh placement part task trainer.

BACKGROUND: We present a surgical simulator, developed for the training of a laparoscopic surgery and in particular for mesh placement during an inguinal herniorrhaphy. METHODS: Major technical issues related to virtual surgery training systems include virtual patient modelling, collision detection and collision response, haptic and graphic rendering, 3-D motion tracking and some special effects, such as bleeding, cauterizing and so on. Among these problems, real-time deformation modelling and collision detection are the most challenging research topics. RESULTS: In this paper, we describe novel approaches addressing the above issues, which have been successfully adopted in our bimanual hernia repair simulator. CONCLUSION: The implementations of our new collision detection and deformation appear to work well, even at haptic rates for the limited scope of mesh placement training. More sophisticated techniques are needed for full organ deformation especially for blunt dissection simulation.

Haptic herniorrhaphy simulation with robust and fast collision detection algorithm.

Collision detection and soft tissue deformation are two major research challenges in real time VR based simulation, especially when haptic feedback is required. We have developed a real time collision detection algorithm for a prototype laparoscopic surgery trainer. However, this algorithm makes no assumptions about its applications and thus can be a generic solution to complicated collision detection problems. For soft tissue modeling, we use the mass-spring model enhanced with volume constraint and, stability control methods. We use both the new collision detection and tissue modeling algorithms in a bimanual hernia repair simulator which performs a mesh prosthesis stapling operation in real time.

StapSim: a virtual reality-based stapling simulator for laparoscopic hemiorrhaphy.

The growing interest in laparoscopic hernia surgery and in surgical simulators has motivated our current research. In this paper, we present our work in simulating the process of stapling used in laparoscopic herniorrhaphy. By connecting two separate deformable preperitoneal meshes together, our model has simulated the repair process for a bilateral hernia correction. The task of mesh placement and stapling of corners was simulated, to allow surgeons to practice their hand-eye coordination. Various deformable models and numerical methods were researched to comply with the real time requirements. The stapling simulator can either be independently used as a part-task (sub-task) trainer or be integrated as a module of a complete VR-based simulator. A phantom device was used to provide haptic-based force feedback during task rehearsals.

Realistic anatomical texture for laparoscopic surgery simulation.

Visual realism in laparoscopic surgery simulation is very desirable. Previously, much work has been done to extract the organs and textures from the visible human data (VHD) using various rendering techniques. We present here a technique to extract the true texture from visible human data for the laparoscopic herniorrhaphy simulation. A VHD slice provides texture only in the direction in which it was visualized whereas the surgeons visualize the facet of an organ during a laparoscopic surgery. Our paper describes an approach for the extraction and mapping facet texture from the VHD.

Special visual effects for surgical simulation: cauterization, irrigation and suction.

Simulation of cauterization and irrigation forms an important part of a virtual laparoscopic trainer. Typically, they are carried out to stop the intragastric bleeding due to an accidental cut by the surgeon. In this paper, we present a method to simulate these special visual effects in an integrated fashion in real-time. We have simulated cauterization and irrigation using a particle-based system. A physics-based model is used to simulate accumulation and removal of fluids. The integrated special effects were implemented and tested in a prototype environment.

Simulation of a preperitoneal mesh in laparoscopic herniorrhaphy.

During a laparoscopic hernia surgery, a preperitoneal mesh is tacked or stapled to the defect on the inside in order to close the fissure created by the hernia. Here we have proposed and implemented a novel technique for simulating the movement and stapling of the mesh for use in a virtual reality based laparoscopic trainer. The use of particle based system and Mass-Spring model in simulating the movement of the mesh is also discussed.

Simulation of bleeding during laparoscopic herniorrhaphy.

Simulation of intragastric bleeding due to an accidental cut by the surgeon is an important component of a virtual laparoscopic herniorrhaphy trainer. We present a method for simulating bleeding during laparoscopic hemiorrhaphy here. The various approaches used in previous research work are reviewed and our present approach is justified. Physically based fluid models used in computer graphics are used to simulate bleeding.

Adaptive hybrid interpolation techniques for direct Haptic rendering of isosurfaces.

Direct Haptic rendering of voxels from an anatomical dataset provides patient specific haptic feedback vital for diagnosis and surgical planning. Our algorithm uses zero sets of scalar trivariate function for polynomial interpolation with sixty-four neighborhood points to generate isosurfaces on the fly for haptic rendering. This approach gives continuity in surfaces as well as better capture of isosurface features of the medical dataset. The detailed algorithm is presented along with the description of results from haptically rendering medical datasets.