Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study.

BACKGROUND: This study evaluated the use of an augmented reality navigation system that provides a markerless registration system using stereo vision in oral and maxillofacial surgery. METHOD: A feasibility study was performed on a subject, wherein a stereo camera was used for tracking and markerless registration. The computed tomography data obtained from the volunteer was used to create an integral videography image and a 3-dimensional rapid prototype model of the jaw. The overlay of the subject's anatomic site and its 3D-IV image were displayed in real space using a 3D-AR display. Extraction of characteristic points and teeth matching were done using parallax images from two stereo cameras for patient-image registration. RESULTS: Accurate registration of the volunteer's anatomy with IV stereoscopic images via image matching was done using the fully automated markerless system, which recognized the incisal edges of the teeth and captured information pertaining to their position with an average target registration error of < 1 mm. These 3D-CT images were then displayed in real space with high accuracy using AR. Even when the viewing position was changed, the 3D images could be observed as if they were floating in real space without using special glasses. CONCLUSION: Teeth were successfully used for registration via 3D image (contour) matching. This system, without using references or fiducial markers, displayed 3D-CT images in real space with high accuracy. The system provided real-time markerless registration and 3D image matching via stereo vision, which, combined with AR, could have significant clinical applications.

Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study.

To evaluate the feasibility and accuracy of a three-dimensional augmented reality system incorporating integral videography for imaging oral and maxillofacial regions, based on preoperative computed tomography data. Three-dimensional surface models of the jawbones, based on the computed tomography data, were used to create the integral videography images of a subject's maxillofacial area. The three-dimensional augmented reality system (integral videography display, computed tomography, a position tracker and a computer) was used to generate a three-dimensional overlay that was projected on the surgical site via a half-silvered mirror. Thereafter, a feasibility study was performed on a volunteer. The accuracy of this system was verified on a solid model while simulating bone resection. Positional registration was attained by identifying and tracking the patient/surgical instrument's position. Thus, integral videography images of jawbones, teeth and the surgical tool were superimposed in the correct position. Stereoscopic images viewed from various angles were accurately displayed. Change in the viewing angle did not negatively affect the surgeon's ability to simultaneously observe the three-dimensional images and the patient, without special glasses. The difference in three-dimensional position of each measuring point on the solid model and augmented reality navigation was almost negligible (<1 mm); this indicates that the system was highly accurate. This augmented reality system was highly accurate and effective for surgical navigation and for overlaying a three-dimensional computed tomography image on a patient's surgical area, enabling the surgeon to understand the positional relationship between the preoperative image and the actual surgical site, with the naked eye.

Surgical tool alignment guidance by drawing two cross-sectional laser-beam planes.

Conventional surgical navigation requires for surgeons to move their sight and conscious off the surgical field when checking surgical tool's positions shown on the display panel. Since that takes high risks of surgical exposure possibilities to the patient's body, we propose a novel method for guiding surgical tool position and orientation directly in the surgical field by a laser beam. In our navigation procedure, two cross-sectional planar laser beams are emitted from the two laser devices attached onto both sides of an optical localizer, and show surgical tool's entry position on the patient's body surface and its orientation on the side face of the surgical tool. In the experiments, our method gave the surgeons precise and accurate surgical tool adjusting and showed the feasibility to apply to both of open and percutaneous surgeries.

A beam-splitter-type 3-D endoscope for front view and front-diagonal view images.

PURPOSE: In endoscopic surgery, surgeons must manipulate an endoscope inside the body cavity to observe a large field-of-view while estimating the distance between surgical instruments and the affected area by reference to the size or motion of the surgical instruments in 2-D endoscopic images on a monitor. Therefore, there is a risk of the endoscope or surgical instruments physically damaging body tissues. To overcome this problem, we developed a Ø7- mm 3-D endoscope that can switch between providing front and front-diagonal view 3-D images by simply rotating its sleeves. METHODS: This 3-D endoscope consists of a conventional 3-D endoscope and an outer and inner sleeve with a beam splitter and polarization plates. The beam splitter was used for visualizing both the front and front-diagonal view and was set at 25° to the outer sleeve's distal end in order to eliminate a blind spot common to both views. Polarization plates were used to avoid overlap of the two views. We measured signal-to-noise ratio (SNR), sharpness, chromatic aberration (CA), and viewing angle of this 3-D endoscope and evaluated its feasibility in vivo. RESULTS: Compared to the conventional 3-D endoscope, SNR and sharpness of this 3-D endoscope decreased by 20 and 7 %, respectively. No significant difference was found in CA. The viewing angle for both the front and front-diagonal views was about 50°. In the in vivo experiment, this 3-D endoscope can provide clear 3-D images of both views by simply rotating its inner sleeve. CONCLUSIONS: The developed 3-D endoscope can provide the front and front-diagonal view by simply rotating the inner sleeve, therefore the risk of damage to fragile body tissues can be significantly decreased.

3-D endoscope using a single CCD camera and pneumatic vibration mechanism.

BACKGROUND: During endoscopic surgical procedures, surgeons must manipulate an endoscope inside the body cavity to observe a surgical area while estimating the distance between that area and the surgical instruments by reference to a monitor on which the movement and size of the surgical instruments are displayed in 2-D endoscopic images. Therefore, there is a risk of the endoscope or instruments physically damaging body tissues. To overcome this problem, we developed a Ø5-mm, 3-D endoscope using a single 1/10-inch CCD camera and pneumatic vibration mechanism. METHODS: The 3-D endoscope proposed in this paper consists of an outer and inner sleeve, a 1/10-inch CCD camera attached to its distal end, and a pneumatic vibration mechanism attached to its proximal end. This endoscope can acquire left and right endoscopic images for stereovision in synchrony with the periodical motion generated by the vibration mechanism. We measured the displacement at the proximal and distal end of the 3-D endoscope simultaneously, and evaluated the feasibility of its use in vivo. RESULTS: The displacement at the distal end of the endoscope to which the CCD camera is attached was approximately ±0.25 mm. The timing when the displacement of the CCD camera was at maximal amplitude coincided with the timing when the displacement of its proximal end was at maximal amplitude. In the in vivo experiment, this 3-D endoscope can provide clear 3-D images of the surgical area. CONCLUSIONS: The developed 3-D endoscope that uses a single CCD camera and pneumatic vibration mechanism can successfully visualize internal organs inside the body even though the CCD camera is moved by the vibration. Therefore, the risk of damage to fragile body tissues can be significantly decreased.

Magnetic resonance imaging-compatible tactile sensing device based on a piezoelectric array.

Minimally invasive surgery is a widely used medical technique, one of the drawbacks of which is the loss of direct sense of touch during the operation. Palpation is the use of fingertips to explore and make fast assessments of tissue morphology. Although technologies are developed to equip minimally invasive surgery tools with haptic feedback capabilities, the majority focus on tissue stiffness profiling and tool-tissue interaction force measurement. For greatly increased diagnostic capability, a magnetic resonance imaging-compatible tactile sensor design is proposed, which allows minimally invasive surgery to be performed under image guidance, combining the strong capability of magnetic resonance imaging soft tissue and intuitive palpation. The sensing unit is based on a piezoelectric sensor methodology, which conforms to the stringent mechanical and electrical design requirements imposed by the magnetic resonance environment The sensor mechanical design and the device integration to a 0.2 Tesla open magnetic resonance imaging scanner are described, together with the device's magnetic resonance compatibility testing. Its design limitations and potential future improvements are also discussed. A tactile sensing unit based on a piezoelectric sensor principle is proposed, which is designed for magnetic resonance imaging guided interventions.

Developing essential rigid-flexible outer sheath to enable novel multi-piercing surgery.

We have developed a new generation device called rigid-flexible outer sheath with multi-piercing surgery (MPS) to solve the issues of tissue closure, triangulation, and platform stability in natural orifice transluminal endoscopic surgery (NOTES), and the problems of restricted visual field, organ damage, and removing a resected organ from body in needlescopic surgery (NS). The shape of the flexible outer sheath can be selectively locked by a novel pneumatic shapelocking mechanism. Major features include four directional flexion at the distal end, four working channels, and suction and water jet functions. The insertion part of the prototype is 330 mm long with a 25 mm maximum outer diameter. The outer sheath system has successfully preformed in vivo experiment using a swine on partial gastrectomy. The advanced outer sheath system has shown great promise for solving NOTES and NS issues.

Augmented reality system for oral surgery using 3D auto stereoscopic visualization.

We present an augmented reality system for oral and maxillofacial surgery in this paper. Instead of being displayed on a separated screen, three-dimensional (3D) virtual presentations of osseous structures and soft tissues are projected onto the patient's body, providing surgeons with exact knowledge of depth information of high risk tissues inside the bone. We employ a 3D integral imaging technique which produce motion parallax in both horizontal and vertical direction over a wide viewing area in this study. In addition, surgeons are able to check the progress of the operation in real-time through an intuitive 3D based interface which is content-rich, hardware accelerated. These features prevent surgeons from penetrating into high risk areas and thus help improve the quality of the operation. Operational tasks such as hole drilling, screw fixation were performed using our system and showed an overall positional error of less than 1 mm. Feasibility of our system was also verified with a human volunteer experiment.

Nonmetallic rigid-flexible outer sheath with pneumatic shapelocking mechanism and double curvature structure.

Single port access (SPA) surgery is a laparoscopic procedure using only one transumbilical-placed port. Natural orifice transluminal endoscopic surgery (NOTES) offers the possibility of surgery without visible scars. To address the access and stability problems in SPA and NOTES, we developed a device called rigid-flexible outer sheath. This sheath can be switched between flexible and rigid modes by a novel pneumatic shapelocking mechanism, and it has a double curvature structure that enables it to flex in four directions at the distal end and three directions on the rigid-flexible shaft. The insertion part of the prototype is 300 mm long with a 20 mm outer diameter, and the part is equipped with four working channels. In vivo experiments using a swine show that the outer sheath has high potential for solving access and stability problems. We expect that the outer sheath will be useful for SPA and NOTES.

Endoscope system with plasma flushing and coaxial round jet nozzle for off-pump cardiac surgery.

BACKGROUND: To develop a new endoscope for performing simple surgical tasks inside the blood-filled cardiac atrium/chamber, that is, "off-pump" cardiac surgeries. METHODS: We developed the endoscope system with plasma flushing and coaxial round jet nozzle. The "plasma flushing" system was invented to observe the interior of the blood-filled heart by displacing blood cells in front of the endoscope tip. However, some areas could not be observed with simple flushing of the liquid because the flushed liquid mixed with blood. Further, a large amount of liquid had to be flushed, which posed a risk of cardiac damage caused by excess volume. Therefore, to safely capture high-resolution images of the interior of the heart, an endoscope with a coaxial round jet nozzle through which plasma is flushed has been developed. And to reduce the volume of flushed liquid, the synchronization system of heartbeat and the endoscope system with plasma flushing has been developed. RESULTS: We conducted an in vivo experiment to determine whether we could observe intracardiac tissues in swine without the use of a heart-lung machine. As a result, we successfully observed intracardiac tissues without using a heart-lung machine. By using a coaxial nozzle, we could even observe the tricuspid valve. Moreover, we were able to save up to 30% of the flushed liquid by replacing the original system with a synchronization system. And we evaluated the performance of the endoscope with the coaxial round jet nozzle by conducting fluid analysis and an in vitro experiment. CONCLUSION: We successfully observed intracardiac tissues without using a heart-lung machine. By using a coaxial nozzle, we could even observe the tricuspid valve. And by replacing an original system to a synchronization system, we were able to save up to 30% of the flushed liquid. As a follow-up study, we plan to create a surgical flexible device for valve disease that can grasp, staple, and repair cardiac valves by endoscopic visualization.

A new, safer, controllable field-of-view endoscope avoiding movement inside body cavities.

One of the greatest difficulties in endoscopic surgery is the limited field-of-view (FOV) of endoscopes. During endoscopic manipulation in body cavities to expand the FOV, there is the risk of inadvertent damage to body tissues, nerves, and internal organs. The risk increases especially in surgery that is performed inside a very small cavity, or in which body tissues are very fragile. To overcome these issues, we developed a novel endoscope that can provide various FOVs without moving or bending the endoscope itself inside the body cavity and investigated the feasibility of using the new endoscope in vivo. A beam splitter was used to visualize both forward and side views, and two polarization plates and observation windows were used to avoid overlap of the two views. An endoscope having a 7-mm diameter was fabricated through which both views were clearly visualized in vivo. It took only 0.7s to change the FOV with high repeatability, with a maximum distance error of 2.8%. The new endoscope can provide forward and panoramic views without moving the endoscope; therefore, the risk of inadvertent damage to fragile body tissues can be significantly decreased.

Autostereoscopic 3D Display with Long Visualization Depth Using Referential Viewing Area-Based Integral Photography.

We developed an autostereoscopic display for distant viewing of 3D computer graphics (CG) images without using special viewing glasses or tracking devices. The images are created by employing referential viewing area-based CG image generation and pixel distribution algorithm for integral photography (IP) and integral videography (IV) imaging. CG image rendering is used to generate IP/IV elemental images. The images can be viewed from each viewpoint within a referential viewing area and the elemental images are reconstructed from rendered CG images by pixel redistribution and compensation method. The elemental images are projected onto a screen that is placed at the same referential viewing distance from the lens array as in the image rendering. Photographic film is used to record the elemental images through each lens. The method enables 3D images with a long visualization depth to be viewed from relatively long distances without any apparent influence from deviated or distorted lenses in the array. We succeeded in creating an actual autostereoscopic images with an image depth of several meters in front of and behind the display that appear to have 3D even when viewed from a distance.

3-D augmented reality for MRI-guided surgery using integral videography autostereoscopic image overlay.

A 3-D augmented reality navigation system using autostereoscopic images was developed for MRI-guided surgery. The 3-D images are created by employing an animated autostereoscopic image, integral videography (IV), which provides geometrically accurate 3-D spatial images and reproduces motion parallax without using any supplementary eyeglasses or tracking devices. The spatially projected 3-D images are superimposed onto the surgical area and viewed via a half-silvered mirror. A fast and accurate spatial image registration method was developed for intraoperative i.v. image-guided therapy. Preliminary experiments showed that the total system error in patient-to-image registration was 0.90 +/- 0.21 mm, and the procedure time for guiding a needle toward a target was shortened by 75%. An animal experiment was also conducted to evaluate the performance of the system. The feasibility studies showed that augmented reality of the image overlay system could increase the surgical instrument placement accuracy and reduce the procedure time as a result of intuitive 3-D viewing.

Precision-guided surgical navigation system using laser guidance and 3D autostereoscopic image overlay.

This paper describes a precision-guided surgical navigation system for minimally invasive surgery. The system combines a laser guidance technique with a three-dimensional (3D) autostereoscopic image overlay technique. Images of surgical anatomic structures superimposed onto the patient are created by employing an animated imaging method called integral videography (IV), which can display geometrically accurate 3D autostereoscopic images and reproduce motion parallax without the need for special viewing or tracking devices. To improve the placement accuracy of surgical instruments, we integrated an image overlay system with a laser guidance system for alignment of the surgical instrument and better visualization of patient's internal structure. We fabricated a laser guidance device and mounted it on an IV image overlay device. Experimental evaluations showed that the system could guide a linear surgical instrument toward a target with an average error of 2.48 mm and standard deviation of 1.76 mm. Further improvement to the design of the laser guidance device and the patient-image registration procedure of the IV image overlay will make this system practical; its use would increase surgical accuracy and reduce invasiveness.

A coaxial laser endoscope with arbitrary spots in endoscopic view for fetal surgery.

In this paper, we describe a rigid endoscope that transmits a laser beam coaxially to arbitrary points in the endoscopic view, mainly for treatment of twin-to-twin transfusion syndrome. The endoscope consists of a hotmirror for coaxial transmission of visible light and a Nd:YAG laser beam, and galvanometers for controlling the beam irradiation angle. We evaluated the transmission efficiency of the laser power, the spot size through the endoscope and accuracy in positioning the beam. The maximum laser transmission efficiency was 39% and the spot diameter was 2.2-3.2 mm at a distance of 10-20 mm. The positioning accuracy was mostly within 1.0 mm in the endoscopic view at the distance. The average laser power density on the spot was estimated to be 170-370 W/cm2, and a chicken liver was successfully coagulated by changing the laser beam irradiation angle.

Nonmagnetic rigid and flexible outer sheath with pneumatic interlocking mechanism for minimally invasive surgical approach.

We developed a nonmagnetic rigid and flexible outer sheath with pneumatic interlocking mechanism using flexible toothed links and a wire-driven bending distal end. The outer sheath can be switched between rigid and flexible modes easily depending on surgical scenes, and the angle of its distal end can be controlled by three nylon wires. All components of flexible parts are made of MRI-compatible nonmagnetic plastics. We manufactured the device with 300-mm long, 16-mm outer diameter, 7-mm inner diameter and 90-mm bending distal end. Holding power of the device in rigid mode was maximum 3.6 N, which was sufficient for surgical tasks in body cavity. In vivo experiment using a swine, our device performed smooth insertion of a flexible endoscope and a biopsy forceps into reverse side of the liver, intestines and spleen with a curved path. In conclusion, our device shows availability of secure approach of surgical instruments into deep cavity.

Novel endoscope system with plasma flushing for off-pump cardiac surgery.

The purpose of this study is to develop a new endoscope for performing simple surgical tasks inside a cardiac atrium/chamber filled with blood, i.e., for performing "off-pump" cardiac surgeries. In general, it is very difficult to observe the inner wall of the vessels containing circulating blood because the light from the endoscope is scattered by the red blood cells. "Plasma flushing" performed using the separator system is developed to observe the inner side of the heart filled with blood and to remove blood cells from the front of the endoscope tip. The system was used in in vitro quantitative measurement of the device performance and in vivo experiments on a swine. In these experiments, we successfully obtained high-resolution images of the interior of the heart during off-pump surgery.

R-PLUS: a Riemannian anisotropic edge detection scheme for vascular segmentation.

In this paper, detection of edges in oriented fields is addressed. In some applications such as vessel segmentation because of the intrinsic orientation of the structures, edge detection is only demanded in a particular subspace. This is specially usefull when a curve evolution is chosen for segmentation since gradients in parallel to vessel orientation may stop the contour. An anisotropic edge detection scheme is generalized on a Riemannian manifold using the local structure tensor. The method is the generalization of the PLUS operator proposed for accurate curved edge detection. Examples are given and the comparison is made with the state-of-the-art flux maximizing flow which indicates that significant improvements in terms of leakage minimization and thiner vessel delineation is achievable using our methodology.

Fetus-supporting flexible manipulator with balloon-type stabilizer for endoscopic intrauterine surgery.

BACKGROUND: Minimally invasive endoscopic fetal surgery enables intrauterine intervention with reduced risk to the mother and fetus. A novel surgical manipulator is described for stabilizing the fetus and restraining it from floating free during endoscopic intrauterine surgery. METHODS: We designed and fabricated a prototype fetus-supporting manipulator equipped with flexible joint and bending mechanisms and a soft balloon stabilizer. The flexible joint and bending mechanisms enable the stabilizer to reach the target sites within the confined space of the uterus under the guidance of an ultrasound device. The balloon stabilizer could be inserted into the uterus through a small incision. RESULTS: The accuracy evaluation showed that the maximum error of the bending mechanism was as small as 7 mm and the standard deviation of the joint mechanism was just 1.6 degrees. In the experiments using a fetus model, the manipulator could be well controlled with guidance from ultrasound images and its bending mechanism with the balloon stabilizer could be clearly visualized while stabilizing the fetus model. CONCLUSIONS: The manipulator has the potential to be used in minimally invasive intrauterine surgery, although further improvements and experiments remain to be carried out.