TeraChem: Accelerating electronic structure and ab initio molecular dynamics with graphical processing units.

Developed over the past decade, TeraChem is an electronic structure and ab initio molecular dynamics software package designed from the ground up to leverage graphics processing units (GPUs) to perform large-scale ground and excited state quantum chemistry calculations in the gas and the condensed phase. TeraChem's speed stems from the reformulation of conventional electronic structure theories in terms of a set of individually optimized high-performance electronic structure operations (e.g., Coulomb and exchange matrix builds, one- and two-particle density matrix builds) and rank-reduction techniques (e.g., tensor hypercontraction). Recent efforts have encapsulated these core operations and provided language-agnostic interfaces. This greatly increases the accessibility and flexibility of TeraChem as a platform to develop new electronic structure methods on GPUs and provides clear optimization targets for emerging parallel computing architectures.

Quantum Chemistry for Solvated Molecules on Graphical Processing Units Using Polarizable Continuum Models.

The conductor-like polarization model (C-PCM) with switching/Gaussian smooth discretization is a widely used implicit solvation model in chemical simulations. However, its application in quantum mechanical calculations of large-scale biomolecular systems can be limited by computational expense of both the gas phase electronic structure and the solvation interaction. We have previously used graphical processing units (GPUs) to accelerate the first of these steps. Here, we extend the use of GPUs to accelerate electronic structure calculations including C-PCM solvation. Implementation on the GPU leads to significant acceleration of the generation of the required integrals for C-PCM. We further propose two strategies to improve the solution of the required linear equations: a dynamic convergence threshold and a randomized block-Jacobi preconditioner. These strategies are not specific to GPUs and are expected to be beneficial for both CPU and GPU implementations. We benchmark the performance of the new implementation using over 20 small proteins in solvent environment. Using a single GPU, our method evaluates the C-PCM related integrals and their derivatives more than 10× faster than that with a conventional CPU-based implementation. Our improvements to the linear solver provide a further 3× acceleration. The overall calculations including C-PCM solvation require, typically, 20-40% more effort than that for their gas phase counterparts for a moderate basis set and molecule surface discretization level. The relative cost of the C-PCM solvation correction decreases as the basis sets and/or cavity radii increase. Therefore, description of solvation with this model should be routine. We also discuss applications to the study of the conformational landscape of an amyloid fibril.

Ab initio interactive molecular dynamics on graphical processing units (GPUs).

A virtual molecular modeling kit is developed based on GPU-enabled interactive ab initio molecular dynamics (MD). The code uses the TeraChem and VMD programs with a modified IMD interface. Optimization of the GPU accelerated TeraChem program specifically for small molecular systems is discussed, and a robust multiple time step integrator is employed to accurately integrate strong user-supplied pulling forces. Smooth and responsive visualization techniques are developed to allow interactive manipulation at minimum simulation rates below five MD steps per second. Representative calculations at the Hartree-Fock level of theory are demonstrated for molecular systems containing up to a few dozen atoms.

An atomic orbital-based formulation of analytical gradients and nonadiabatic coupling vector elements for the state-averaged complete active space self-consistent field method on graphical processing units.

We recently presented an algorithm for state-averaged complete active space self-consistent field (SA-CASSCF) orbital optimization that capitalizes on sparsity in the atomic orbital basis set to reduce the scaling of computational effort with respect to molecular size. Here, we extend those algorithms to calculate the analytic gradient and nonadiabatic coupling vectors for SA-CASSCF. Combining the low computational scaling with acceleration from graphical processing units allows us to perform SA-CASSCF geometry optimizations for molecules with more than 1000 atoms. The new approach will make minimal energy conical intersection searches and nonadiabatic dynamics routine for molecular systems with O(10(2)) atoms.

Analytic first derivatives of floating occupation molecular orbital-complete active space configuration interaction on graphical processing units.

The floating occupation molecular orbital-complete active space configuration interaction (FOMO-CASCI) method is a promising alternative to the state-averaged complete active space self-consistent field (SA-CASSCF) method. We have formulated the analytic first derivative of FOMO-CASCI in a manner that is well-suited for a highly efficient implementation using graphical processing units (GPUs). Using this implementation, we demonstrate that FOMO-CASCI gradients are of similar computational expense to configuration interaction singles (CIS) or time-dependent density functional theory (TDDFT). In contrast to CIS and TDDFT, FOMO-CASCI can describe multireference character of the electronic wavefunction. We show that FOMO-CASCI compares very favorably to SA-CASSCF in its ability to describe molecular geometries and potential energy surfaces around minimum energy conical intersections. Finally, we apply FOMO-CASCI to the excited state hydrogen transfer reaction in methyl salicylate.

An atomic orbital-based formulation of the complete active space self-consistent field method on graphical processing units.

Despite its importance, state-of-the-art algorithms for performing complete active space self-consistent field (CASSCF) computations have lagged far behind those for single reference methods. We develop an algorithm for the CASSCF orbital optimization that uses sparsity in the atomic orbital (AO) basis set to increase the applicability of CASSCF. Our implementation of this algorithm uses graphical processing units (GPUs) and has allowed us to perform CASSCF computations on molecular systems containing more than one thousand atoms. Additionally, we have implemented analytic gradients of the CASSCF energy; the gradients also benefit from GPU acceleration as well as sparsity in the AO basis.

Multiple time step integrators in ab initio molecular dynamics.

Multiple time-scale algorithms exploit the natural separation of time-scales in chemical systems to greatly accelerate the efficiency of molecular dynamics simulations. Although the utility of these methods in systems where the interactions are described by empirical potentials is now well established, their application to ab initio molecular dynamics calculations has been limited by difficulties associated with splitting the ab initio potential into fast and slowly varying components. Here we present two schemes that enable efficient time-scale separation in ab initio calculations: one based on fragment decomposition and the other on range separation of the Coulomb operator in the electronic Hamiltonian. We demonstrate for both water clusters and a solvated hydroxide ion that multiple time-scale molecular dynamics allows for outer time steps of 2.5 fs, which are as large as those obtained when such schemes are applied to empirical potentials, while still allowing for bonds to be broken and reformed throughout the dynamics. This permits computational speedups of up to 4.4x, compared to standard Born-Oppenheimer ab initio molecular dynamics with a 0.5 fs time step, while maintaining the same energy conservation and accuracy.

Generating Efficient Quantum Chemistry Codes for Novel Architectures.

We describe an extension of our graphics processing unit (GPU) electronic structure program TeraChem to include atom-centered Gaussian basis sets with d angular momentum functions. This was made possible by a "meta-programming" strategy that leverages computer algebra systems for the derivation of equations and their transformation to correct code. We generate a multitude of code fragments that are formally mathematically equivalent, but differ in their memory and floating-point operation footprints. We then select between different code fragments using empirical testing to find the highest performing code variant. This leads to an optimal balance of floating-point operations and memory bandwidth for a given target architecture without laborious manual tuning. We show that this approach is capable of similar performance compared to our hand-tuned GPU kernels for basis sets with s and p angular momenta. We also demonstrate that mixed precision schemes (using both single and double precision) remain stable and accurate for molecules with d functions. We provide benchmarks of the execution time of entire self-consistent field (SCF) calculations using our GPU code and compare to mature CPU based codes, showing the benefits of the GPU architecture for electronic structure theory with appropriately redesigned algorithms. We suggest that the meta-programming and empirical performance optimization approach may be important in future computational chemistry applications, especially in the face of quickly evolving computer architectures.

Ab initio quantum chemistry for protein structures.

Structural properties of over 55 small proteins have been determined using both density-based and wave-function-based electronic structure methods in order to assess the ability of ab initio "force fields" to retain the properties described by experimental structures measured with crystallography or nuclear magnetic resonance. The efficiency of the GPU-based quantum chemistry algorithms implemented in our TeraChem program enables us to carry out systematic optimization of ab initio protein structures, which we compare against experimental and molecular mechanics force field references. We show that the quality of the ab initio optimized structures, as judged by conventional protein health metrics, increases with increasing basis set size. On the other hand, there is little evidence for a significant improvement of predicted structures using density functional theory as compared to Hartree-Fock methods. Although occasional pathologies of minimal basis sets are observed, these are easily alleviated with even the smallest double-ζ basis sets.

Excited-State Electronic Structure with Configuration Interaction Singles and Tamm-Dancoff Time-Dependent Density Functional Theory on Graphical Processing Units.

Excited-state calculations are implemented in a development version of the GPU-based TeraChem software package using the configuration interaction singles (CIS) and adiabatic linear response Tamm-Dancoff time-dependent density functional theory (TDA-TDDFT) methods. The speedup of the CIS and TDDFT methods using GPU-based electron repulsion integrals and density functional quadrature integration allows full ab initio excited-state calculations on molecules of unprecedented size. CIS/6-31G and TD-BLYP/6-31G benchmark timings are presented for a range of systems, including four generations of oligothiophene dendrimers, photoactive yellow protein (PYP), and the PYP chromophore solvated with 900 quantum mechanical water molecules. The effects of double and single precision integration are discussed, and mixed precision GPU integration is shown to give extremely good numerical accuracy for both CIS and TDDFT excitation energies (excitation energies within 0.0005 eV of extended double precision CPU results).

Dynamic Precision for Electron Repulsion Integral Evaluation on Graphical Processing Units (GPUs).

It has recently been demonstrated that novel streaming architectures found in consumer video gaming hardware such as graphical processing units (GPUs) are well-suited to a broad range of computations including electronic structure theory (quantum chemistry). Although recent GPUs have developed robust support for double precision arithmetic, they continue to provide 2-8× more hardware units for single precision. In order to maximize performance on GPU architectures, we present a technique of dynamically selecting double or single precision evaluation for electron repulsion integrals (ERIs) in Hartree-Fock and density functional self-consistent field (SCF) calculations. We show that precision error can be effectively controlled by evaluating only the largest integrals in double precision. By dynamically scaling the precision cutoff over the course of the SCF procedure, we arrive at a scheme that minimizes the number of double precision integral evaluations for any desired accuracy. This dynamic precision scheme is shown to be effective for an array of molecules ranging in size from 20 to nearly 2000 atoms.