Fusing modalities by multiplexed graph neural networks for outcome prediction from medical data and beyond.

With the emergence of multimodal electronic health records, the evidence for diseases, events, or findings may be present across multiple modalities ranging from clinical to imaging and genomic data. Developing effective patient-tailored therapeutic guidance and outcome prediction will require fusing evidence across these modalities. Developing general-purpose frameworks capable of modeling fine-grained and multi-faceted complex interactions, both within and across modalities is an important open problem in multimodal fusion. Generalized multimodal fusion is extremely challenging as evidence for outcomes may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients, due to which simple methods of early, late, or intermediate fusion may be inadequate. In this paper, we present a novel approach that uses the machinery of multiplexed graphs for fusion. This allows for modalities to be represented through their targeted encodings. We model their relationship between explicitly via multiplexed graphs derived from salient features in a combined latent space. We then derive a new graph neural network for multiplex graphs for task-informed reasoning. We compare our framework against several state-of-the-art approaches for multi-graph reasoning and multimodal fusion. As a sanity check on the neural network design, we evaluate the multiplexed GNN on two popular benchmark datasets, namely the AIFB and the MUTAG dataset against several state-of-the-art multi-relational GNNs for reasoning. Second, we evaluate our multiplexed framework against several state-of-the-art multimodal fusion frameworks on two large clinical datasets for two separate applications. The first is the NIH-TB portals dataset for treatment outcome prediction in Tuberculosis, and the second is the ABIDE dataset for Autism Spectrum Disorder classification. Through rigorous experimental evaluation, we demonstrate that the multiplexed GNN provides robust performance improvements in all of these diverse applications.

Improving the Path from Diagnoses to Documentation: A Cognitive Review Tool for Clinical Notes and Administrative Records.

EMR systems are intended to improve patient-centered care management and hospital administrative processing. However, the information stored in EMRs can be disorganized, incomplete, or inconsistent, creating problems at the patient and system level. We present a technology that reconciles inconsistencies between clinical diagnoses and administrative records by analyzing free-text notes, problem lists and recorded diagnoses in real time. A fully integrated pipeline has been developed for efficient, knowledge-driven extraction, normalization, and matching of disease terms among structured and unstructured data, with modular precision of 94-98% on over 1000 patients. This cognitive data review tool improves the path from diagnosis to documentation, facilitating accurate and timely clinical and administrative decision-making.

Metric hashing forests.

In this paper, we propose metric Hashing Forests (mHF) which is a supervised variant of random forests tailored for the task of nearest neighbor retrieval through hashing. This is achieved by training independent hashing trees that parse and encode the feature space such that local class neighborhoods are preserved and encoded with similar compact binary codes. At the level of each internal node, locality preserving projections are employed to project data to a latent subspace, where separability between dissimilar points is enhanced. Following which, we define an oblique split that maximally preserves this separability and facilitates defining local neighborhoods of similar points. By incorporating the inverse-lookup search scheme within the mHF, we can then effectively mitigate pairwise neuron similarity comparisons, which allows for scalability to massive databases with little additional time overhead. Exhaustive experimental validations on 22,265 neurons curated from over 120 different archives demonstrate the superior efficacy of mHF in terms of its retrieval performance and precision of classification in contrast to state-of-the-art hashing and metric learning based methods. We conclude that the proposed method can be utilized effectively for similarity-preserving retrieval and categorization in large neuron databases.