Multimodal graph attention network for COVID-19 outcome prediction.

When dealing with a newly emerging disease such as COVID-19, the impact of patient- and disease-specific factors (e.g., body weight or known co-morbidities) on the immediate course of the disease is largely unknown. An accurate prediction of the most likely individual disease progression can improve the planning of limited resources and finding the optimal treatment for patients. In the case of COVID-19, the need for intensive care unit (ICU) admission of pneumonia patients can often only be determined on short notice by acute indicators such as vital signs (e.g., breathing rate, blood oxygen levels), whereas statistical analysis and decision support systems that integrate all of the available data could enable an earlier prognosis. To this end, we propose a holistic, multimodal graph-based approach combining imaging and non-imaging information. Specifically, we introduce a multimodal similarity metric to build a population graph that shows a clustering of patients. For each patient in the graph, we extract radiomic features from a segmentation network that also serves as a latent image feature encoder. Together with clinical patient data like vital signs, demographics, and lab results, these modalities are combined into a multimodal representation of each patient. This feature extraction is trained end-to-end with an image-based Graph Attention Network to process the population graph and predict the COVID-19 patient outcomes: admission to ICU, need for ventilation, and mortality. To combine multiple modalities, radiomic features are extracted from chest CTs using a segmentation neural network. Results on a dataset collected in Klinikum rechts der Isar in Munich, Germany and the publicly available iCTCF dataset show that our approach outperforms single modality and non-graph baselines. Moreover, our clustering and graph attention increases understanding of the patient relationships within the population graph and provides insight into the network's decision-making process.

ToxNet: an artificial intelligence designed for decision support for toxin prediction.

BACKGROUND: Artificial intelligences (AIs) are emerging in the field of medical informatics in many areas. They are mostly used for diagnosis support in medical imaging but have potential uses in many other fields of medicine where large datasets are available. AIM: To develop an artificial intelligence (AI) "ToxNet", a machine-learning based computer-aided diagnosis (CADx) system, which aims to predict poisons based on patient's symptoms and metadata from our Poison Control Center (PCC) data. To prove its accuracy and compare it against medical doctors (MDs). METHODS: The CADx system was developed and trained using data from 781,278 calls recorded in our PCC database from 2001 to 2019. All cases were mono-intoxications. Patient symptoms and meta-information (e.g., age group, sex, etiology, toxin point of entry, weekday, etc.) were provided. In the pilot phase, the AI was trained on 10 substances, the AI's prediction was compared to naïve matching, literature matching, a multi-layer perceptron (MLP), and the graph attention network (GAT). The trained AI's accuracy was then compared to 10 medical doctors in an individual and in an identical dataset. The dataset was then expanded to 28 substances and the predictions and comparisons repeated. RESULTS: In the pilot, the prediction performance in a set of 8995 patients with 10 substances was 0.66 ± 0.01 (F1 micro score). Our CADx system was significantly superior to naïve matching, literature matching, MLP, and GAT (p < 0.005). It outperformed our physicians experienced in clinical toxicology in the individual and identical dataset. In the extended dataset, our CADx system was able to predict the correct toxin in a set of 36,033 patients with 28 substances with an overall performance of 0.27 ± 0.01 (F1 micro score), also significantly superior to naïve matching, literature matching, MLP, and GAT. It also outperformed our MDs. CONCLUSION: Our AI trained on a large PCC database works well for poison prediction in these experiments. With further research, it might become a valuable aid for physicians in predicting unknown substances and might be the first step into AI use in PCCs.

AI for Doctors-A Course to Educate Medical Professionals in Artificial Intelligence for Medical Imaging.

Successful adoption of artificial intelligence (AI) in medical imaging requires medical professionals to understand underlying principles and techniques. However, educational offerings tailored to the need of medical professionals are scarce. To fill this gap, we created the course "AI for Doctors: Medical Imaging". An analysis of participants' opinions on AI and self-perceived skills rated on a five-point Likert scale was conducted before and after the course. The participants' attitude towards AI in medical imaging was very optimistic before and after the course. However, deeper knowledge of AI and the process for validating and deploying it resulted in significantly less overoptimism with respect to perceivable patient benefits through AI (p = 0.020). Self-assessed skill ratings significantly improved after the course, and the appreciation of the course content was very positive. However, we observed a substantial drop-out rate, mostly attributed to the lack of time of medical professionals. There is a high demand for educational offerings regarding AI in medical imaging among medical professionals, and better education may lead to a more realistic appreciation of clinical adoption. However, time constraints imposed by a busy clinical schedule need to be taken into account for successful education of medical professionals.

FhSPECT-US guided needle biopsy of sentinel lymph nodes in the axilla: is it feasible?

Until now, core needle biopsy of the axillary sentinel lymph nodes in early stage breast cancer patients is not possible, due to the lack of a proper combination of functional and anatomical information. In this work we present the first fully 3D freehand SPECT--ultrasound fusion, combining the advantages of both modalities. By using spatial positioning either with optical or with electromagnetic tracking for the ultrasound probe, and a mini gamma camera as radiation detector for freehand SPECT reconstructions, we investigate the capability of the introduced multi-model imaging system, where we compare both 3D freehand SPECT and 3D ultrasound to ground truth for a realistic breast mimicking phantom and further analyze the effect of tissue deformation by ultrasound. Finally, we also show its application in a real clinical setting.