Interactive serious game for shoulder rehabilitation based on real-time hand tracking.

BACKGROUND: Virtual reality is becoming popular in the rehabilitation field thanks to the several advantages it can offer to patients and physicians. Indeed, serious games can: motivate and engage the patient; offer different levels of challenge and difficulty based on the patient baseline, and integrate objective measures of the patient's performance during each rehabilitation session. OBJECTIVE: We designed and implemented a serious game for shoulder rehabilitation based on real-time hand tracking. The aim was to maintain the medical benefits of traditional rehabilitation, while reducing human resources and costs and facilitating active patient participation. METHODS: Our software application provides the user with a shoulder horizontal adduction exercise. This exercise takes place in a 2D interactive game environment, controlled by hand movements on a desk pad. The hardware includes a standard desktop computer and screen, and the Leap Motion Controller: a hand tracking system. Changing the desk pad material allows the physiotherapist to vary the friction between the user hand and the supporting surface. RESULTS: Fourteen healthy volunteers and six rehabilitation experts tested our serious game. The results showed that the application is attractive, ergonomic and clinically useful. CONCLUSION: Despite promising results, clinical validation is necessary to demonstrate the efficacy of the serious game.

How to Build a Patient-Specific Hybrid Simulator for Orthopaedic Open Surgery: Benefits and Limits of Mixed-Reality Using the Microsoft HoloLens.

Orthopaedic simulators are popular in innovative surgical training programs, where trainees gain procedural experience in a safe and controlled environment. Recent studies suggest that an ideal simulator should combine haptic, visual, and audio technology to create an immersive training environment. This article explores the potentialities of mixed-reality using the HoloLens to develop a hybrid training system for orthopaedic open surgery. Hip arthroplasty, one of the most common orthopaedic procedures, was chosen as a benchmark to evaluate the proposed system. Patient-specific anatomical 3D models were extracted from a patient computed tomography to implement the virtual content and to fabricate the physical components of the simulator. Rapid prototyping was used to create synthetic bones. The Vuforia SDK was utilized to register virtual and physical contents. The Unity3D game engine was employed to develop the software allowing interactions with the virtual content using head movements, gestures, and voice commands. Quantitative tests were performed to estimate the accuracy of the system by evaluating the perceived position of augmented reality targets. Mean and maximum errors matched the requirements of the target application. Qualitative tests were carried out to evaluate workload and usability of the HoloLens for our orthopaedic simulator, considering visual and audio perception and interaction and ergonomics issues. The perceived overall workload was low, and the self-assessed performance was considered satisfactory. Visual and audio perception and gesture and voice interactions obtained a positive feedback. Postural discomfort and visual fatigue obtained a nonnegative evaluation for a simulation session of 40 minutes. These results encourage using mixed-reality to implement a hybrid simulator for orthopaedic open surgery. An optimal design of the simulation tasks and equipment setup is required to minimize the user discomfort. Future works will include Face Validity, Content Validity, and Construct Validity to complete the assessment of the hip arthroplasty simulator.

Computer guidance system for single-incision bimanual robotic surgery.

The evolution of surgical robotics is following the progress of developments in Minimally Invasive Surgery (MIS), which is moving towards Single-Incision Laparoscopic Surgery (SILS) procedures. The complexity of these techniques has favored the introduction of robotic surgical systems. New bimanual robots, which are completely inserted into the patient's body, have been proposed in order to enhance the surgical gesture in SILS procedures. However, the limited laparoscopic view and the focus on the end-effectors, together with the use of complex robotic devices inside the patient's abdomen, may lead to unexpected collisions, e.g., between the surrounding anatomical organs not involved in the intervention and the surgical robot. This paper describes a computer guidance system, based on patient-specific data, designed to provide intraoperative navigation and assistance in SILS robotic interventions. The navigator has been tested in simulations of some of the surgical tasks involved in a cholecystectomy, using a synthetic anthropomorphic mannequin. The results demonstrate the usability and efficacy of the navigation system, underlining the importance of avoiding unwanted collisions between the robot arms and critical organs. The proposed computer guidance software is able to integrate any bimanual surgical robot design.

Patient-specific surgical simulator for the pre-operative planning of single-incision laparoscopic surgery with bimanual robots.

INTRODUCTION: The trend of surgical robotics is to follow the evolution of laparoscopy, which is now moving towards single-incision laparoscopic surgery. The main drawback of this approach is the limited maneuverability of the surgical tools. Promising solutions to improve the surgeon's dexterity are based on bimanual robots. However, since both robot arms are completely inserted into the patient's body, issues related to possible unwanted collisions with structures adjacent to the target organ may arise. MATERIALS AND METHODS: This paper presents a simulator based on patient-specific data for the positioning and workspace evaluation of bimanual surgical robots in the pre-operative planning of single-incision laparoscopic surgery. RESULTS: The simulator, designed for the pre-operative planning of robotic laparoscopic interventions, was tested by five expert surgeons who evaluated its main functionalities and provided an overall rating for the system. DISCUSSION: The proposed system demonstrated good performance and usability, and was designed to integrate both present and future bimanual surgical robots.

Patient specific surgical simulator for the evaluation of the movability of bimanual robotic arms.

This work presents a simulator based on patient specific data for bimanual surgical robots. Given a bimanual robot with a particular geometry and kinematics, and a patient specific virtual anatomy, the aim of this simulator was to evaluate if a dexterous movability was obtainable to avoid collisions with the surrounding virtual anatomy in order to prevent potential damages to the tissues during the real surgical procedure. In addition, it could help surgeons to find the optimal positioning of the robot before entering the operative room. This application was tested using a haptic device to reproduce the interactions of the robot with deformable organs. The results showed good performances in terms of frame rate for the graphic, haptic, and dynamic processes.

Integration of biomechanical parameters in tetrahedral mass-spring models for virtual surgery simulation.

Surgical simulation requires to have an operating scenario as similar as possible to the real conditions that the surgeon is going to face. Not only visual and geometric patient properties are needed to be reproduced, but also physical and biomechanical properties are theoretically required. In this paper a physically based patient specific simulator for solid organs is described, recalling the underlying theory and providing simulation results and comparisons. The main biomechanical parameters (Young's modulus and density) have been integrated in a Mass-Spring-Damper model (MSDm) based on a tetrahedral structured network. The proposed algorithms allow the automatic setting of node mass and spring stiffness, while the damping coefficient have been modeled using the Rayleigh approach. Moreover, the method automatically detects the organ external layer, allowing the usage of both the surface and internal Young's moduli: for the capsule (or stroma) and for the internal part (or parenchyma). Finally the model can be manually tuned to represent lesions with specific biomechanical properties. The method has beed tested with various material samples. The results have shown a good visual realism ensuring the performance required by an interactive simulation.

EndoCAS navigator platform: a common platform for computer and robotic assistance in minimally invasive surgery.

BACKGROUND: Computer-assisted surgery (CAS) systems are currently used in only a few surgical specialties: ear, nose and throat (ENT), neurosurgery and orthopaedics. Almost all of these systems have been developed as dedicated platforms and work on rigid anatomical structures. The development of augmented reality systems for intra-abdominal organs remains problematic because of the anatomical complexity of the human peritoneal cavity and especially because of the deformability of its organs. The aim of the present work was to develop and implement a highly modular platform (targeted for minimally invasive laparoscopic surgery) generally suitable for CAS, and to produce a prototype for demonstration of its potential clinical application and use in laparoscopic surgery. METHODS: In this paper we outline details of a platform integrating several aspects of CAS and medical robotics into a modular open architecture: the EndoCAS navigator platform, which integrates all the functionalities necessary for provision of computer-based assistance to surgeons during all the management phases (diagnostic work-up, planning and intervention). A specific application for computer-assisted laparoscopic procedures has been developed on the basic modules of the platform. The system provides capabilities for three-dimensional (3D) surface model generation, 3D visualization, intra-operative registration, surgical guidance and robotic assistance during laparoscopic surgery. The description of specific modules and an account of the initial clinical experience with the system are reported. RESULTS: We developed a common platform for computer assisted surgery and implemented a system for intraoperative laparoscopic navigation. The preliminary clinical trials and feedback from the surgeons on its use in laparoscopic surgery have been positive, although experience has been limited to date. CONCLUSIONS: We have successfully developed a system for computer-assisted technologies for use in laparoscopic surgery and demonstrated, by early clinical trials, that the introduction of these technologies in operative laparoscopy, even though they are not yet sufficiently accurate (from an engineering viewpoint) for surgical treatment of intra-abdominal disease, brings added benefits to the execution of interventions by surgeons and hence represents concrete on-going progress in interventional technology.