Precision requirements and data compression in CryoEM/CryoET.

With larger, higher speed detectors and improved automation, individual CryoEM instruments are capable of producing a prodigious amount of data each day, which must then be stored, processed and archived. While it has become routine to use lossless compression on raw counting-mode movies, the averages which result after correcting these movies no longer compress well. These averages could be considered sufficient for long term archival, yet they are conventionally stored with 32 bits of precision, despite high noise levels. Derived images are similarly stored with excess precision, providing an opportunity to decrease project sizes and improve processing speed. We present a simple argument based on propagation of uncertainty for safe bit truncation of flat-fielded images combined with lossless compression. The same method can be used for most derived images throughout the processing pipeline. We test the proposed strategy on two standard, data-limited CryoEM data sets, demonstrating that these limits are safe for real-world use. We find that 5 bits of precision is sufficient for virtually any raw CryoEM data and that 8-12 bits is sufficient for intermediate averages or final 3-D structures. Additionally, we detail and recommend specific rules for discretization of data as well as a practical compressed data representation that is tuned to the specific needs of CryoEM.

New software tools in EMAN2 inspired by EMDatabank map challenge.

EMAN2 is an extensible software suite with complete workflows for performing high-resolution single particle analysis, 2-D and 3-D heterogeneity analysis, and subtomogram averaging, among other tasks. Participation in the recent CryoEM Map Challenge sponsored by the EMDatabank led to a number of significant improvements to the single particle analysis process in EMAN2. A new convolutional neural network particle picker was developed, which dramatically improves particle picking accuracy for difficult data sets. A new particle quality metric capable of accurately identifying "bad" particles with a high degree of accuracy, no human input, and a negligible amount of additional computation, has been introduced, and this now serves as a replacement for earlier human-biased methods. The way 3-D single particle reconstructions are filtered has been altered to be more comparable to the filter applied in several other popular software packages, dramatically improving the appearance of sidechains in high-resolution structures. Finally, an option has been added to perform local resolution-based iterative filtration, resulting in local resolution improvements in many maps.