Long-term cognitive decline prediction based on multi-modal data using Multimodal3DSiameseNet: transfer learning from Alzheimer's disease to Parkinson's disease.

PURPOSE: Monitoring and predicting the cognitive state of subjects with neurodegenerative disorders is crucial to provide appropriate treatment as soon as possible. In this work, we present a machine learning approach using multimodal data (brain MRI and clinical) from two early medical visits, to predict the longer-term cognitive decline of patients. Using transfer learning, our model can be successfully transferred from one neurodegenerative disease (Alzheimer's) to another (Parkinson's). METHODS: Our model is a Deep Neural Network with siamese sub-modules dedicated to extracting features from each modality. We pre-train it with data from ADNI (Alzheimer's disease), then transfer it on the smaller PPMI dataset (Parkinson's disease). We show that, even when we do not fine-tune the filters learnt from the ADNI MRIs, the transferred model's results are satisfying on PPMI. RESULTS: The first main result is that our model provides satisfying long-term predictions of cognitive decline from any pair of early visits, with no fixed time delay between these visits (provided the potential decline has started at the second visit). The second main result is that the prediction performance on Parkinson's dataset (PPMI) reaches an AUC of 0.81 on PPMI after transfer learning from Alzheimer's dataset (ADNI), without even having to re-train the image filters, versus an AUC of 0.72 for the model trained from scratch on PPMI. CONCLUSIONS: First, our model is effective for predicting long-term cognitive decline from only two visits, even with irregular intervals of time. When dealing with neurodegenerative diseases, where patients often miss some control visits, this is an important finding. Second, our model is able to transfer the knowledge learnt from one neurodegenerative disease (Alzheimer's) to another (Parkinson's), when using the same imaging modalities (brain MRI) and different clinical variables. This makes it usable even for diseases that are rare or under-studied.

DIVA: Natural Navigation Inside 3D Images Using Virtual Reality.

As three-dimensional microscopy becomes commonplace in biological research, there is an increasing need for researchers to be able to view experimental image stacks in a natural three-dimensional viewing context. Through stereoscopy and motion tracking, commercial virtual reality headsets provide a solution to this important visualization challenge by allowing researchers to view volumetric objects in an entirely intuitive fashion. With this motivation, we present DIVA, a user-friendly software tool that automatically creates detailed three-dimensional reconstructions of raw experimental image stacks that are integrated in virtual reality. In DIVA's immersive virtual environment, users can view, manipulate and perform volumetric measurements on their microscopy images as they would to real physical objects. In contrast to similar solutions, our software provides high-quality volume rendering with native TIFF file compatibility. We benchmark the software with diverse image types including those generated by confocal, light-sheet and electron microscopy. DIVA is available at https://diva.pasteur.fr and will be regularly updated.

Virtual Reality: Beyond Visualization.

Virtual reality (VR) has recently become an affordable technology. A wide range of options are available to access this unique visualization medium, from simple cardboard inserts for smartphones to truly advanced headsets tracked by external sensors. While it is now possible for any research team to gain access to VR, we can still question what it brings to scientific research. Visualization and the ability to navigate complex three-dimensional data are undoubtedly a gateway to many scientific applications; however, we are convinced that data treatment and numerical simulations, especially those mixing interactions with data, human cognition, and automated algorithms will be the future of VR in scientific research. Moreover, VR might soon merit the same level of attention to imaging data as machine learning currently has. In this short perspective, we discuss approaches that employ VR in scientific research based on some concrete examples.