Concept Graph Neural Networks for Surgical Video Understanding.

Analysis of relations between objects and comprehension of abstract concepts in the surgical video is important in AI-augmented surgery. However, building models that integrate our knowledge and understanding of surgery remains a challenging endeavor. In this paper, we propose a novel way to integrate conceptual knowledge into temporal analysis tasks using temporal concept graph networks. In the proposed networks, a knowledge graph is incorporated into the temporal video analysis of surgical notions, learning the meaning of concepts and relations as they apply to the data. We demonstrate results in surgical video data for tasks such as verification of the critical view of safety, estimation of the Parkland grading scale as well as recognizing instrument-action-tissue triplets. The results show that our method improves the recognition and detection of complex benchmarks as well as enables other analytic applications of interest.

SAGES consensus recommendations on surgical video data use, structure, and exploration (for research in artificial intelligence, clinical quality improvement, and surgical education).

BACKGROUND: Surgery generates a vast amount of data from each procedure. Particularly video data provides significant value for surgical research, clinical outcome assessment, quality control, and education. The data lifecycle is influenced by various factors, including data structure, acquisition, storage, and sharing; data use and exploration, and finally data governance, which encompasses all ethical and legal regulations associated with the data. There is a universal need among stakeholders in surgical data science to establish standardized frameworks that address all aspects of this lifecycle to ensure data quality and purpose. METHODS: Working groups were formed, among 48 representatives from academia and industry, including clinicians, computer scientists and industry representatives. These working groups focused on: Data Use, Data Structure, Data Exploration, and Data Governance. After working group and panel discussions, a modified Delphi process was conducted. RESULTS: The resulting Delphi consensus provides conceptualized and structured recommendations for each domain related to surgical video data. We identified the key stakeholders within the data lifecycle and formulated comprehensive, easily understandable, and widely applicable guidelines for data utilization. Standardization of data structure should encompass format and quality, data sources, documentation, metadata, and account for biases within the data. To foster scientific data exploration, datasets should reflect diversity and remain adaptable to future applications. Data governance must be transparent to all stakeholders, addressing legal and ethical considerations surrounding the data. CONCLUSION: This consensus presents essential recommendations around the generation of standardized and diverse surgical video databanks, accounting for multiple stakeholders involved in data generation and use throughout its lifecycle. Following the SAGES annotation framework, we lay the foundation for standardization of data use, structure, and exploration. A detailed exploration of requirements for adequate data governance will follow.

SAGES video acquisition framework-analysis of available OR recording technologies by the SAGES AI task force.

BACKGROUND: Surgical video recording provides the opportunity to acquire intraoperative data that can subsequently be used for a variety of quality improvement, research, and educational applications. Various recording devices are available for standard operating room camera systems. Some allow for collateral data acquisition including activities of the OR staff, kinematic measurements (motion of surgical instruments), and recording of the endoscopic video streams. Additional analysis through computer vision (CV), which allows software to understand and perform predictive tasks on images, can allow for automatic phase segmentation, instrument tracking, and derivative performance-geared metrics. With this survey, we summarize available surgical video acquisition technologies and associated performance analysis platforms. METHODS: In an effort promoted by the SAGES Artificial Intelligence Task Force, we surveyed the available video recording technology companies. Of thirteen companies approached, nine were interviewed, each over an hour-long video conference. A standard set of 17 questions was administered. Questions spanned from data acquisition capacity, quality, and synchronization of video with other data, availability of analytic tools, privacy, and access. RESULTS: Most platforms (89%) store video in full-HD (1080p) resolution at a frame rate of 30 fps. Most (67%) of available platforms store data in a Cloud-based databank as opposed to institutional hard drives. CV powered analysis is featured in some platforms: phase segmentation in 44% platforms, out of body blurring or tool tracking in 33%, and suture time in 11%. Kinematic data are provided by 22% and perfusion imaging in one device. CONCLUSION: Video acquisition platforms on the market allow for in depth performance analysis through manual and automated review. Most of these devices will be integrated in upcoming robotic surgical platforms. Platform analytic supplementation, including CV, may allow for more refined performance analysis to surgeons and trainees. Most current AI features are related to phase segmentation, instrument tracking, and video blurring.

Artificial intelligence prediction of cholecystectomy operative course from automated identification of gallbladder inflammation.

BACKGROUND: Operative courses of laparoscopic cholecystectomies vary widely due to differing pathologies. Efforts to assess intra-operative difficulty include the Parkland grading scale (PGS), which scores inflammation from the initial view of the gallbladder on a 1-5 scale. We investigated the impact of PGS on intra-operative outcomes, including laparoscopic duration, attainment of the critical view of safety (CVS), and gallbladder injury. We additionally trained an artificial intelligence (AI) model to identify PGS. METHODS: One surgeon labeled surgical phases, PGS, CVS attainment, and gallbladder injury in 200 cholecystectomy videos. We used multilevel Bayesian regression models to analyze the PGS's effect on intra-operative outcomes. We trained AI models to identify PGS from an initial view of the gallbladder and compared model performance to annotations by a second surgeon. RESULTS: Slightly inflamed gallbladders (PGS-2) minimally increased duration, adding 2.7 [95% compatibility interval (CI) 0.3-7.0] minutes to an operation. This contrasted with maximally inflamed gallbladders (PGS-5), where on average 16.9 (95% CI 4.4-33.9) minutes were added, with 31.3 (95% CI 8.0-67.5) minutes added for the most affected surgeon. Inadvertent gallbladder injury occurred in 25% of cases, with a minimal increase in gallbladder injury observed with added inflammation. However, up to a 28% (95% CI - 2, 63) increase in probability of a gallbladder hole during PGS-5 cases was observed for some surgeons. Inflammation had no substantial effect on whether or not a surgeon attained the CVS. An AI model could reliably (Krippendorff's α = 0.71, 95% CI 0.65-0.77) quantify inflammation when compared to a second surgeon (α = 0.82, 95% CI 0.75-0.87). CONCLUSIONS: An AI model can identify the degree of gallbladder inflammation, which is predictive of cholecystectomy intra-operative course. This automated assessment could be useful for operating room workflow optimization and for targeted per-surgeon and per-resident feedback to accelerate acquisition of operative skills.

ADAMTSL2 protein and a soluble biomarker signature identify at-risk non-alcoholic steatohepatitis and fibrosis in adults with NAFLD.

BACKGROUND & AIMS: Identifying fibrosis in non-alcoholic fatty liver disease (NAFLD) is essential to predict liver-related outcomes and guide treatment decisions. A protein-based signature of fibrosis could serve as a valuable, non-invasive diagnostic tool. This study sought to identify circulating proteins associated with fibrosis in NAFLD. METHODS: We used aptamer-based proteomics to measure 4,783 proteins in 2 cohorts (Cohort A and B). Targeted, quantitative assays coupling aptamer-based protein pull down and mass spectrometry (SPMS) validated the profiling results in a bariatric and NAFLD cohort (Cohort C and D, respectively). Generalized linear modeling-logistic regression assessed the ability of candidate proteins to classify fibrosis. RESULTS: From the multiplex profiling, 16 proteins differed significantly by fibrosis in cohorts A (n = 62) and B (n = 98). Quantitative and robust SPMS assays were developed for 8 proteins and validated in Cohorts C (n = 71) and D (n = 84). The A disintegrin and metalloproteinase with thrombospondin motifs like 2 (ADAMTSL2) protein accurately distinguished non-alcoholic fatty liver (NAFL)/non-alcoholic steatohepatitis (NASH) with fibrosis stage 0-1 (F0-1) from at-risk NASH with fibrosis stage 2-4, with AUROCs of 0.83 and 0.86 in Cohorts C and D, respectively, and from NASH with significant fibrosis (F2-3), with AUROCs of 0.80 and 0.83 in Cohorts C and D, respectively. An 8-protein panel distinguished NAFL/NASH F0-1 from at-risk NASH (AUROCs 0.90 and 0.87 in Cohort C and D, respectively) and NASH F2-3 (AUROCs 0.89 and 0.83 in Cohorts C and D, respectively). The 8-protein panel and ADAMTSL2 protein had superior performance to the NAFLD fibrosis score and fibrosis-4 score. CONCLUSION: The ADAMTSL2 protein and an 8-protein soluble biomarker panel are highly associated with at-risk NASH and significant fibrosis; they exhibited superior diagnostic performance compared to standard of care fibrosis scores. LAY SUMMARY: Non-alcoholic fatty liver disease (NAFLD) is one of the most common causes of liver disease worldwide. Diagnosing NAFLD and identifying fibrosis (scarring of the liver) currently requires a liver biopsy. Our study identified novel proteins found in the blood which may identify fibrosis without the need for a liver biopsy.

Distinct Hepatic Gene-Expression Patterns of NAFLD in Patients With Obesity.

Approaches to manage nonalcoholic fatty liver disease (NAFLD) are limited by an incomplete understanding of disease pathogenesis. The aim of this study was to identify hepatic gene-expression patterns associated with different patterns of liver injury in a high-risk cohort of adults with obesity. Using the NanoString Technologies (Seattle, WA) nCounter assay, we quantified expression of 795 genes, hypothesized to be involved in hepatic fibrosis, inflammation, and steatosis, in liver tissue from 318 adults with obesity. Liver specimens were categorized into four distinct NAFLD phenotypes: normal liver histology (NLH), steatosis only (steatosis), nonalcoholic steatohepatitis without fibrosis (NASH F0), and NASH with fibrosis stage 1-4 (NASH F1-F4). One hundred twenty-five genes were significantly increasing or decreasing as NAFLD pathology progressed. Compared with NLH, NASH F0 was characterized by increased inflammatory gene expression, such as gamma-interferon-inducible lysosomal thiol reductase (IFI30) and chemokine (C-X-C motif) ligand 9 (CXCL9), while complement and coagulation related genes, such as C9 and complement component 4 binding protein beta (C4BPB), were reduced. In the presence of NASH F1-F4, extracellular matrix degrading proteinases and profibrotic/scar deposition genes, such as collagens and transforming growth factor beta 1 (TGFB1), were simultaneously increased, suggesting a dynamic state of tissue remodeling. Conclusion: In adults with obesity, distinct states of NAFLD are associated with intrahepatic perturbations in genes related to inflammation, complement and coagulation pathways, and tissue remodeling. These data provide insights into the dynamic pathogenesis of NAFLD in high-risk individuals.

Robotic-assisted minimally invasive esophagectomy (RAMIE) for esophageal cancer training curriculum-a worldwide Delphi consensus study.

BACKGROUND: Structured training protocols can safely improve skills prior initiating complex surgical procedures such as robotic-assisted minimally invasive esophagectomy (RAMIE). As no consensus on a training curriculum for RAMIE has been established so far it is our aim to define a protocol for RAMIE with the Delphi consensus methodology. METHODS: Fourteen worldwide RAMIE experts were defined and were enrolled in this Delphi consensus project. An expert panel was created and three Delphi rounds were performed starting December 2019. Items required for RAMIE included, but were not limited to, virtual reality simulation, wet-lab training, proctoring, and continued monitoring and education. After rating performed by the experts, consensus was defined when a Cronbach alpha of ≥0.80 was reached. If ≥80% of the committee reached a consensus an item was seen as fundamental. RESULTS: All Delphi rounds were completed by 12-14 (86-100%) participants. After three rounds analyzing our 49-item questionnaire, 40 items reached consensus for a training curriculum of RAMIE. CONCLUSION: The core principles for RAMIE training were defined. This curriculum may lead to a wider adoption of RAMIE and a reduction in time to reach proficiency.

SAGES consensus recommendations on an annotation framework for surgical video.

BACKGROUND: The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration. METHODS: Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups. RESULTS: After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established. CONCLUSIONS: While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

Challenges in surgical video annotation.

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Automated operative phase identification in peroral endoscopic myotomy.

BACKGROUND: Artificial intelligence (AI) and computer vision (CV) have revolutionized image analysis. In surgery, CV applications have focused on surgical phase identification in laparoscopic videos. We proposed to apply CV techniques to identify phases in an endoscopic procedure, peroral endoscopic myotomy (POEM). METHODS: POEM videos were collected from Massachusetts General and Showa University Koto Toyosu Hospitals. Videos were labeled by surgeons with the following ground truth phases: (1) Submucosal injection, (2) Mucosotomy, (3) Submucosal tunnel, (4) Myotomy, and (5) Mucosotomy closure. The deep-learning CV model-Convolutional Neural Network (CNN) plus Long Short-Term Memory (LSTM)-was trained on 30 videos to create POEMNet. We then used POEMNet to identify operative phases in the remaining 20 videos. The model's performance was compared to surgeon annotated ground truth. RESULTS: POEMNet's overall phase identification accuracy was 87.6% (95% CI 87.4-87.9%). When evaluated on a per-phase basis, the model performed well, with mean unweighted and prevalence-weighted F1 scores of 0.766 and 0.875, respectively. The model performed best with longer phases, with 70.6% accuracy for phases that had a duration under 5 min and 88.3% accuracy for longer phases. DISCUSSION: A deep-learning-based approach to CV, previously successful in laparoscopic video phase identification, translates well to endoscopic procedures. With continued refinements, AI could contribute to intra-operative decision-support systems and post-operative risk prediction.

Desmoid tumor presenting 2 years after elective Roux-en-Y gastric bypass: a case report and review of the literature.

Desmoid tumors are rare malignancies derived from myofibroblasts, which can cause significant morbidity due to life-threatening invasion of local structures. Risk factors include familial adenomatous polyposis, antecedent surgical trauma and estrogen exposure. We described a previously healthy 27-year-old female in whom a desmoid tumor developed 2 years after a Roux-en-Y gastric bypass was performed for the treatment of obesity. Computed tomography scan demonstrated a 16-cm complex density intra-abdominal mass. Exploratory laparotomy was performed, revealing a mass firmly adherent to the Roux limb, as well as the jejunojejunostomy and distal portion of the bilopancreatic limb. En bloc resection of the mass and the Roux limb was required, as well as reconstruction of the Roux-en-Y anatomy. This case describes a unique, long-term complication of bariatric surgery, in light of a growing population of patients with altered gastric anatomy.

Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy.

OBJECTIVE(S): To develop and assess AI algorithms to identify operative steps in laparoscopic sleeve gastrectomy (LSG). BACKGROUND: Computer vision, a form of artificial intelligence (AI), allows for quantitative analysis of video by computers for identification of objects and patterns, such as in autonomous driving. METHODS: Intraoperative video from LSG from an academic institution was annotated by 2 fellowship-trained, board-certified bariatric surgeons. Videos were segmented into the following steps: 1) port placement, 2) liver retraction, 3) liver biopsy, 4) gastrocolic ligament dissection, 5) stapling of the stomach, 6) bagging specimen, and 7) final inspection of staple line. Deep neural networks were used to analyze videos. Accuracy of operative step identification by the AI was determined by comparing to surgeon annotations. RESULTS: Eighty-eight cases of LSG were analyzed. A random 70% sample of these clips was used to train the AI and 30% to test the AI's performance. Mean concordance correlation coefficient for human annotators was 0.862, suggesting excellent agreement. Mean (±SD) accuracy of the AI in identifying operative steps in the test set was 82% ±â€Š4% with a maximum of 85.6%. CONCLUSIONS: AI can extract quantitative surgical data from video with 85.6% accuracy. This suggests operative video could be used as a quantitative data source for research in intraoperative clinical decision support, risk prediction, or outcomes studies.

Risk factors and prevalence of liver disease in review of 2557 routine liver biopsies performed during bariatric surgery.

BACKGROUND: Obesity is a known risk factor for nonalcoholic fatty liver disease (NAFLD). However, among individuals undergoing bariatric surgery, the prevalence and risk factors for NAFLD, as well as distinct phenotypes of steatosis, nonalcoholic steatohepatitis (NASH), and fibrosis remain incompletely understood. OBJECTIVES: To determine the prevalence and risk factors for steatosis, NASH, and fibrosis in individuals undergoing routine bariatric surgery. SETTING: Academic medical center in the United States. METHODS: Liver wedge biopsies were performed at the time of surgery between 2001 and 2017. Pathology reports were reviewed, and individuals were grouped by NAFLD phenotype. Covariates including demographic characteristics, co-morbidities, and preoperative laboratory values were compared between groups using Student's t test, Pearson's χ2, and logistic regression. RESULTS: Liver biopsies were obtained in 97.7% of first-time bariatric procedures, representing 2557 patients. Mean age was 45.6 years, mean body mass index was 46.7, and most were non-Hispanic white (76.1%) and female (71.6%). On histologic review 61.2% had steatosis and 30.9% NASH. Fibrosis was identified in 29.3% of individuals, and 7.8% had stage ≥2 fibrosis. On logistic regression, elevated aspartate aminotransferase (odds ratio [OR] 1.87; P < .001) and elevated alanine aminotransferase (OR 1.62; P < .001) were independently associated with fibrosis. Elevated hemoglobin A1C of 5.7% to 6.5% (OR 1.29; P < .01) and >6.5% (OR 3.23; P < .001) were also associated with fibrosis. A similar trend was seen for NASH. CONCLUSIONS: NASH and/or fibrosis is present in nearly one third of patients undergoing routine bariatric surgery. Risk factors include diabetes, elevated liver enzymes, and diabetes. Risk assessment and aggressive screening should be considered in patients undergoing bariatric surgery.

Artificial Intelligence in Surgery: Promises and Perils.

OBJECTIVE: The aim of this review was to summarize major topics in artificial intelligence (AI), including their applications and limitations in surgery. This paper reviews the key capabilities of AI to help surgeons understand and critically evaluate new AI applications and to contribute to new developments. SUMMARY BACKGROUND DATA: AI is composed of various subfields that each provide potential solutions to clinical problems. Each of the core subfields of AI reviewed in this piece has also been used in other industries such as the autonomous car, social networks, and deep learning computers. METHODS: A review of AI papers across computer science, statistics, and medical sources was conducted to identify key concepts and techniques within AI that are driving innovation across industries, including surgery. Limitations and challenges of working with AI were also reviewed. RESULTS: Four main subfields of AI were defined: (1) machine learning, (2) artificial neural networks, (3) natural language processing, and (4) computer vision. Their current and future applications to surgical practice were introduced, including big data analytics and clinical decision support systems. The implications of AI for surgeons and the role of surgeons in advancing the technology to optimize clinical effectiveness were discussed. CONCLUSIONS: Surgeons are well positioned to help integrate AI into modern practice. Surgeons should partner with data scientists to capture data across phases of care and to provide clinical context, for AI has the potential to revolutionize the way surgery is taught and practiced with the promise of a future optimized for the highest quality patient care.

Posterior reversible encephalopathy syndrome (PRES) after bariatric surgery--a potential consequence associated with rapid withdrawal of antihypertensive medications.

A 61-year-old woman with a medical history of intracerebral haemorrhage, hypertension, hyperlipidaemia and carotid stenosis presented to the emergency department with altered mental status 3 weeks after undergoing a vertical sleeve gastrectomy for severe obesity. She presented with a hypertensive emergency and a National Institutes of Health Stroke Scale of 4. CT of the head was unrevealing. MRI showed an abnormal signal within the bilateral posterior border-zone areas, with several foci in the parietal and occipital lobes, and thalami, suggestive of posterior reversible encephalopathy syndrome (PRES). The patient was initially placed on a labetalol drip and her preoperative antihypertensive medications--amlodipine, captopril, triamterene and hydrochlorothiazide--were gradually reintroduced. She returned to her baseline and was stable on discharge. Rapid withdrawal of antihypertensive medications in the early postoperative period of bariatric surgery was the aetiology of PRES in this patient. This case report discusses postoperative care of bariatric surgery patients having hypertension.

Broad clinical utilization of NOTES: is it safe?

BACKGROUND: Natural orifice transluminal endoscopic surgery (NOTES) has been the focus of several studies as a less invasive alternative to conventional laparoscopy to access and treat intracavitary organs. For the last 5 years, much has been accomplished with animal studies, yet the clinical utilization of this novel technique is still modest. After 2 years of experience in the laboratory, we started our clinical experience. We report our experience with clinical utilization of NOTES procedures from 2007 to 2010. METHODS: Under UCSD institutional review board-approved trials, 104 patients were enrolled under seven different NOTES protocols from 2007 to 2010, where a NOTES procedure was offered as an alternative to conventional treatments. The treated pathologies were cholelithiasis, biliary dyskinesia, acute and chronic appendicitis, ventral hernias, morbid obesity, and achalasia. The access routes included transgastric (TG), transvaginal (TV), transesophageal (TE), and perirectal (PR). RESULTS: Among the 104 patients enrolled, 103 underwent a surgical procedure starting with diagnostic laparoscopy, and 94 cases were deemed appropriate to proceed via a NOTES approach. There were 9 aborted NOTES procedures at the time of the initial peritoneoscopy before creating a NOTES access route. The reasons to not proceed with a NOTES procedure in the TV cholecystectomy group (n = 5) were a large amount of pelvic adhesions in 4 patients and a severe inflammation of the gallbladder in 1 patient. In the TG cholecystectomy group (n = 1), it was severe inflammation of the gallbladder. In the TG appendectomy group (n = 1), it was the presence of localized peritonitis. In the TE endoscopic myotomy group (n = 2), it was the presence of megaesophagus with an inability to clean the esophagus of food debris. The NOTES procedures performed were 48 TV cholecystectomies, 4 TV appendectomies, 8 TG cholecystectomies, 2 PR peritoneoscopies, 3 TG appendectomies, 3 TV ventral hernia repairs, 5 TE endoscopic myotomies, 3 TV sleeve gastrectomies, and 18 TG sleeve gastrectomies. The average body mass indexes for those in the sleeve gastrectomy group was 42.1 kg/m(2) (TG route) and 40.6 kg/m(2) (TV route). There were no intraoperative complication and no conversions to standard laparoscopy during these procedures. The average hospital stay was 1-2 days. One patient who underwent TV cholecystectomy required an emergency department visit for nausea and vomiting. To date, 3 patients who underwent TV cholecystectomy have become pregnant and delivered normally. CONCLUSIONS: NOTES is safe, feasible, and reproducible with previous training in the laboratory and a consistent team at a high-volume center. Prospective randomized studies of a large patient population are necessary to assess long-term results.