Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation.

Recent immense breakthroughs in generative models such as in GPT4 have precipitated re-imagined ubiquitous usage of these models in all applications. One area that can benefit by improvements in artificial intelligence (AI) is healthcare. The note generation task from doctor-patient encounters, and its associated electronic medical record documentation, is one of the most arduous time-consuming tasks for physicians. It is also a natural prime potential beneficiary to advances in generative models. However with such advances, benchmarking is more critical than ever. Whether studying model weaknesses or developing new evaluation metrics, shared open datasets are an imperative part of understanding the current state-of-the-art. Unfortunately as clinic encounter conversations are not routinely recorded and are difficult to ethically share due to patient confidentiality, there are no sufficiently large clinic dialogue-note datasets to benchmark this task. Here we present the Ambient Clinical Intelligence Benchmark (ACI-BENCH) corpus, the largest dataset to date tackling the problem of AI-assisted note generation from visit dialogue. We also present the benchmark performances of several common state-of-the-art approaches.

Secondary use of electronic medical records for clinical research: Challenges and Opportunities.

With increasingly ubiquitous electronic medical record (EMR) implementation accelerated by the adoption of the HITECH Act, there is much interest in the secondary use of collected data to improve outcomes and promote personalized medicine. A plethora of research has emerged using EMRs to investigate clinical research questions and assess variations in both treatments and outcomes. However, whether because of genuine complexities of modeling disease physiology or because of practical problems regarding data capture, data accuracy, and data completeness, the state of current EMR research is challenging and gives rise to concerns regarding study accuracy and reproducibility. This work explores challenges in how different experimental design decisions can influence results using a specific example of breast cancer patients undergoing excision and reconstruction surgeries from EMRs in an academic hospital and the Veterans Health Administration (VHA) We discuss emerging strategies that will mitigate these limitations, including data sharing, application of natural language processing, and improved EMR user design.

Classification of hepatocellular carcinoma stages from free-text clinical and radiology reports.

Cancer stage information is important for clinical research. However, they are not always explicitly noted in electronic medical records. In this paper, we present our work on automatic classification of hepatocellular carcinoma (HCC) stages from free-text clinical and radiology notes. To accomplish this, we defined 11 stage parameters used in the three HCC staging systems, American Joint Committee on Cancer (AJCC), Barcelona Clinic Liver Cancer (BCLC), and Cancer of the Liver Italian Program (CLIP). After aggregating stage parameters to the patient-level, the final stage classifications were achieved using an expert-created decision logic. Each stage parameter relevant for staging was extracted using several classification methods, e.g. sentence classification and automatic information structuring, to identify and normalize text as cancer stage parameter values. Stage parameter extraction for the test set performed at 0.81 F1. Cancer stage prediction for AJCC, BCLC, and CLIP stage classifications were 0.55, 0.50, and 0.43 F1.

Tumor reference resolution and characteristic extraction in radiology reports for liver cancer stage prediction.

BACKGROUND: Anaphoric references occur ubiquitously in clinical narrative text. However, the problem, still very much an open challenge, is typically less aggressively focused on in clinical text domain applications. Furthermore, existing research on reference resolution is often conducted disjointly from real-world motivating tasks. OBJECTIVE: In this paper, we present our machine-learning system that automatically performs reference resolution and a rule-based system to extract tumor characteristics, with component-based and end-to-end evaluations. Specifically, our goal was to build an algorithm that takes in tumor templates and outputs tumor characteristic, e.g. tumor number and largest tumor sizes, necessary for identifying patient liver cancer stage phenotypes. RESULTS: Our reference resolution system reached a modest performance of 0.66 F1 for the averaged MUC, B-cubed, and CEAF scores for coreference resolution and 0.43 F1 for particularization relations. However, even this modest performance was helpful to increase the automatic tumor characteristics annotation substantially over no reference resolution. CONCLUSION: Experiments revealed the benefit of reference resolution even for relatively simple tumor characteristics variables such as largest tumor size. However we found that different overall variables had different tolerances to reference resolution upstream errors, highlighting the need to characterize systems by end-to-end evaluations.

Tumor information extraction in radiology reports for hepatocellular carcinoma patients.

Hepatocellular carcinoma (HCC) is a deadly disease affecting the liver for which there are many available therapies. Targeting treatments towards specific patient groups necessitates defining patients by stage of disease. Criteria for such stagings include information on tumor number, size, and anatomic location, typically only found in narrative clinical text in the electronic medical record (EMR). Natural language processing (NLP) offers an automatic and scale-able means to extract this information, which can further evidence-based research. In this paper, we created a corpus of 101 radiology reports annotated for tumor information. Afterwards we applied machine learning algorithms to extract tumor information. Our inter-annotator partial match agreement scored at 0.93 and 0.90 F1 for entities and relations, respectively. Based on the annotated corpus, our sequential labeling entity extraction achieved 0.87 F1 partial match, and our maximum entropy classification relation extraction achieved scores 0.89 and 0. 74 F1 with gold and system entities, respectively.

Natural Language Processing in Oncology: A Review.

IMPORTANCE: Natural language processing (NLP) has the potential to accelerate translation of cancer treatments from the laboratory to the clinic and will be a powerful tool in the era of personalized medicine. This technology can harvest important clinical variables trapped in the free-text narratives within electronic medical records. OBSERVATIONS: Natural language processing can be used as a tool for oncological evidence-based research and quality improvement. Oncologists interested in applying NLP for clinical research can play pivotal roles in building NLP systems and, in doing so, contribute to both oncological and clinical NLP research. Herein, we provide an introduction to NLP and its potential applications in oncology, a description of specific tools available, and a review on the state of the current technology with respect to cancer case identification, staging, and outcomes quantification. CONCLUSIONS AND RELEVANCE: More automated means of leveraging unstructured data from daily clinical practice is crucial as therapeutic options and access to individual-level health information increase. Research-minded oncologists may push the avenues of evidence-based research by taking advantage of the new technologies available with clinical NLP. As continued progress is made with applying NLP toward oncological research, incremental gains will lead to large impacts, building a cost-effective infrastructure for advancing cancer care.

Structuring Free-text Microbiology Culture Reports For Secondary Use.

Microbiology lab culture reports are a frequently used diagnostic tool for clinical providers. However, their incorporation into clinical surveillance applications and evidence-based medicine can be severely hindered by the free-text nature of these reports. In this work, we (1) created a microbiology culture template to structure free-text microbiology reports, (2) generated an annotated microbiology report corpus, and (3) built a microbiology information extraction system. Specifically, we combined rule-based, hybrid, and statistical techniques to extract microbiology entities and fill templates for structuring data. System performances were favorable, with entity f1-score 0.889 and relation f1-score 0.795. We plan to incorporate these extractions as features for our ongoing ventilator-associated pneumonia surveillance project, though this tool can be used as an upstream process in other applications. Our newly created corpus includes 1442 unique gram stain and culture microbiology reports generated from a cohort of 715 patients at the University of Washington Medical Facilities.

Parametrically adjustable intubation mannequin with real-time visual feedback.

INTRODUCTION: Training for direct laryngoscopy relies heavily on practice with patients. The necessity for human practice might be supplanted to some extent by an intubation mannequin with accurate airway anatomy, a realistic "feel" during laryngoscopy, the capacity to model many patient configurations, and a means to provide feedback to trainees and instructors. The goals of this project were (1) to build and evaluate an airway simulator with realistic dimensions and haptic sensation that could undergo a range of adjustments in several features that affect laryngoscopy difficulty and (2) to develop a system for displaying information on laryngoscopy force and motion in real time. METHODS: The prototype was an existing 2-dimensional (2D) airway model that closely approximated cephalometric measurements of head, neck, and airway anatomy from the dental and surgical literature. The 2D model was extended in a third dimension by adding layers along the coronal axis. An off-the-shelf airway model provided the tongue, pharynx, larynx, and trachea. Adjustability was built into the face, jaw, mouth, teeth, and spine components. A feedback system was constructed with a force- and motion-sensing laryngoscope and motion sensors incorporated in the mannequin head, jaw, and larynx. Anatomic accuracy was assessed by measuring model dimensions. Realism was evaluated by measuring laryngoscopy force and motion compared with laryngoscopy in patients. RESULTS: The extruded 2.5-dimensional model maintained a close conformity to the anatomic measurements present in the original 2D model. The model could be adjusted through multiple settings for face length, jaw length and tension, mouth opening, and dental condition. The laryngoscopy trajectory had a similar shape to laryngoscopy trajectories in patients, but force was greater, on the order of 50 N, compared with roughly 30 N in patients. The movement of the laryngoscope through the mannequin airway could be displayed in real time during the procedure, establishing a means for feedback. CONCLUSIONS: The model incorporates novel features that could aid in developing mastery of the laryngoscopy procedure. Further work is needed to investigate how adjustability and feedback impact the value of laryngoscopy practice on mannequins.

Grid heterogeneity in in-silico experiments: an exploration of drug screening using DOCK on cloud environments.

Large-scale in-silico screening is a necessary part of drug discovery and Grid computing is one answer to this demand. A disadvantage of using Grid computing is the heterogeneous computational environments characteristic of a Grid. In our study, we have found that for the molecular docking simulation program DOCK, different clusters within a Grid organization can yield inconsistent results. Because DOCK in-silico virtual screening (VS) is currently used to help select chemical compounds to test with in-vitro experiments, such differences have little effect on the validity of using virtual screening before subsequent steps in the drug discovery process. However, it is difficult to predict whether the accumulation of these discrepancies over sequentially repeated VS experiments will significantly alter the results if VS is used as the primary means for identifying potential drugs. Moreover, such discrepancies may be unacceptable for other applications requiring more stringent thresholds. This highlights the need for establishing a more complete solution to provide the best scientific accuracy when executing an application across Grids. One possible solution to platform heterogeneity in DOCK performance explored in our study involved the use of virtual machines as a layer of abstraction. This study investigated the feasibility and practicality of using virtual machine and recent cloud computing technologies in a biological research application. We examined the differences and variations of DOCK VS variables, across a Grid environment composed of different clusters, with and without virtualization. The uniform computer environment provided by virtual machines eliminated inconsistent DOCK VS results caused by heterogeneous clusters, however, the execution time for the DOCK VS increased. In our particular experiments, overhead costs were found to be an average of 41% and 2% in execution time for two different clusters, while the actual magnitudes of the execution time costs were minimal. Despite the increase in overhead, virtual clusters are an ideal solution for Grid heterogeneity. With greater development of virtual cluster technology in Grid environments, the problem of platform heterogeneity may be eliminated through virtualization, allowing greater usage of VS, and will benefit all Grid applications in general.