Scipion3: A workflow engine for cryo-electron microscopy image processing and structural biology.

Image-processing pipelines require the design of complex workflows combining many different steps that bring the raw acquired data to a final result with biological meaning. In the image-processing domain of cryo-electron microscopy single-particle analysis (cryo-EM SPA), hundreds of steps must be performed to obtain the three-dimensional structure of a biological macromolecule by integrating data spread over thousands of micrographs containing millions of copies of allegedly the same macromolecule. The execution of such complicated workflows demands a specific tool to keep track of all these steps performed. Additionally, due to the extremely low signal-to-noise ratio (SNR), the estimation of any image parameter is heavily affected by noise resulting in a significant fraction of incorrect estimates. Although low SNR and processing millions of images by hundreds of sequential steps requiring substantial computational resources are specific to cryo-EM, these characteristics may be shared by other biological imaging domains. Here, we present Scipion, a Python generic open-source workflow engine specifically adapted for image processing. Its main characteristics are: (a) interoperability, (b) smart object model, (c) gluing operations, (d) comparison operations, (e) wide set of domain-specific operations, (f) execution in streaming, (g) smooth integration in high-performance computing environments, (h) execution with and without graphical capabilities, (i) flexible visualization, (j) user authentication and private access to private data, (k) scripting capabilities, (l) high performance, (m) traceability, (n) reproducibility, (o) self-reporting, (p) reusability, (q) extensibility, (r) software updates, and (s) non-restrictive software licensing.

Scipion-ED: a graphical user interface for batch processing and analysis of 3D ED/MicroED data.

Three-dimensional electron diffraction (3D ED)/microcrystal electron diffraction (MicroED) techniques are gaining in popularity. However, the data processing often does not fit existing graphical user interface software, instead requiring the use of the terminal or scripting. Scipion-ED, described in this article, provides a graphical user interface and extendable framework for processing of 3D ED/MicroED data. An illustrative project is described, in which multiple 3D ED/MicroED data sets collected on tetragonal lysozyme were processed with DIALS through the Scipion-ED interface. The ability to resolve unmodelled features in the electrostatic potential map was compared between three strategies for merging data sets.

Using RELION software within the Scipion framework.

Scipion is a modular image-processing framework that integrates several software packages under a unified interface while taking care of file formats and conversions. Here, new developments and capabilities of the Scipion plugin for the widely used RELION software package are presented and illustrated with an image-processing pipeline for published data. The user interfaces of Scipion and RELION are compared and the key differences are highlighted, allowing this manuscript to be used as a guide for both new and experienced users of this software. Different on-the-fly image-processing options are also discussed, demonstrating the flexibility of the Scipion framework.

Localized reconstruction in Scipion expedites the analysis of symmetry mismatches in cryo-EM data.

Technological advances in transmission electron microscopes and detectors have turned cryogenic electron microscopy (cryo-EM) into an essential tool for structural biology. A commonly used cryo-EM data analysis method, single particle analysis, averages hundreds of thousands of low-dose images of individual macromolecular complexes to determine a density map of the complex. The presence of symmetry in the complex is beneficial since each projection image can be assigned to multiple views of the complex. However, data processing that applies symmetry can average out asymmetric features and consequently data analysis methods are required to resolve asymmetric structural features. Scipion is a cryo-EM image processing framework that integrates functions from different image processing packages as plugins. To extend its functionality for handling symmetry mismatches, we present here a Scipion plugin termed LocalRec implementing the localized reconstruction method. When tested on an adenovirus data set, the plugin enables resolving the symmetry-mismatched trimeric fibre bound to the five-fold vertices of the capsid. Furthermore, it improves the structure determination of the icosahedral capsid by dealing with the defocus gradient across the particle. LocalRec is expected to be widely applicable in a range of cryo-EM investigations of flexible and symmetry mismatched complexes.

Development of basic building blocks for cryo-EM: the emcore and emvis software libraries.

Image-processing software has always been an integral part of structure determination by cryogenic electron microscopy (cryo-EM). Recent advances in hardware and software are recognized as one of the key factors in the so-called cryo-EM resolution revolution. Increasing computational power has opened many possibilities to consider more demanding algorithms, which in turn allow more complex biological problems to be tackled. Moreover, data processing has become more accessible to many experimental groups, with computations that used to last for many days at supercomputing facilities now being performed in hours on personal workstations. All of these advances, together with the rapid expansion of the community, continue to pose challenges and new demands on the software-development side. In this article, the development of emcore and emvis, two basic software libraries for image manipulation and data visualization in cryo-EM, is presented. The main goal is to provide basic functionality organized in modular components that other developers can reuse to implement new algorithms or build graphical applications. An additional aim is to showcase the importance of following established practices in software engineering, with the hope that this could be a first step towards a more standardized way of developing and distributing software in the field.

Scipion web tools: Easy to use cryo-EM image processing over the web.

Macromolecular structural determination by Electron Microscopy under cryogenic conditions is revolutionizing the field of structural biology, interesting a large community of potential users. Still, the path from raw images to density maps is complex, and sophisticated image processing suites are required in this process, often demanding the installation and understanding of different software packages. Here, we present Scipion Web Tools, a web-based set of tools/workflows derived from the Scipion image processing framework, specially tailored to nonexpert users in need of very precise answers at several key stages of the structural elucidation process.

ScipionCloud: An integrative and interactive gateway for large scale cryo electron microscopy image processing on commercial and academic clouds.

New instrumentation for cryo electron microscopy (cryoEM) has significantly increased data collection rate as well as data quality, creating bottlenecks at the image processing level. Current image processing model of moving the acquired images from the data source (electron microscope) to desktops or local clusters for processing is encountering many practical limitations. However, computing may also take place in distributed and decentralized environments. In this way, cloud is a new form of accessing computing and storage resources on demand. Here, we evaluate on how this new computational paradigm can be effectively used by extending our current integrative framework for image processing, creating ScipionCloud. This new development has resulted in a full installation of Scipion both in public and private clouds, accessible as public "images", with all the required preinstalled cryoEM software, just requiring a Web browser to access all Graphical User Interfaces. We have profiled the performance of different configurations on Amazon Web Services and the European Federated Cloud, always on architectures incorporating GPU's, and compared them with a local facility. We have also analyzed the economical convenience of different scenarios, so cryoEM scientists have a clearer picture of the setup that is best suited for their needs and budgets.

Hybrid Electron Microscopy Normal Mode Analysis graphical interface and protocol.

This article presents an integral graphical interface to the Hybrid Electron Microscopy Normal Mode Analysis (HEMNMA) approach that was developed for capturing continuous motions of large macromolecular complexes from single-particle EM images. HEMNMA was shown to be a good approach to analyze multiple conformations of a macromolecular complex but it could not be widely used in the EM field due to a lack of an integral interface. In particular, its use required switching among different software sources as well as selecting modes for image analysis was difficult without the graphical interface. The graphical interface was thus developed to simplify the practical use of HEMNMA. It is implemented in the open-source software package Xmipp 3.1 (http://xmipp.cnb.csic.es) and only a small part of it relies on MATLAB that is accessible through the main interface. Such integration provides the user with an easy way to perform the analysis of macromolecular dynamics and forms a direct connection to the single-particle reconstruction process. A step-by-step HEMNMA protocol with the graphical interface is given in full details in Supplementary material. The graphical interface will be useful to experimentalists who are interested in studies of continuous conformational changes of macromolecular complexes beyond the modeling of continuous heterogeneity in single particle reconstruction.

Iterative elastic 3D-to-2D alignment method using normal modes for studying structural dynamics of large macromolecular complexes.

This article presents a method to study large-scale conformational changes by combining electron microscopy (EM) single-particle image analysis and normal mode analysis (NMA). It is referred to as HEMNMA, which stands for hybrid electron microscopy normal mode analysis. NMA of a reference structure (atomic-resolution structure or EM volume) is used to predict possible motions that are then confronted with EM images within an automatic iterative elastic 3D-to-2D alignment procedure to identify actual motions in the imaged samples. HEMNMA can be used to extensively analyze the conformational changes and may be used in combination with classic discrete procedures. The identified conformations allow modeling of deformation pathways compatible with the experimental data. HEMNMA was tested with synthetic and experimental data sets of E. coli 70S ribosome, DNA polymerase Pol α and B subunit complex of the eukaryotic primosome, and tomato bushy stunt virus.