TeraChem: Accelerating electronic structure and ab initio molecular dynamics with graphical processing units.

Developed over the past decade, TeraChem is an electronic structure and ab initio molecular dynamics software package designed from the ground up to leverage graphics processing units (GPUs) to perform large-scale ground and excited state quantum chemistry calculations in the gas and the condensed phase. TeraChem's speed stems from the reformulation of conventional electronic structure theories in terms of a set of individually optimized high-performance electronic structure operations (e.g., Coulomb and exchange matrix builds, one- and two-particle density matrix builds) and rank-reduction techniques (e.g., tensor hypercontraction). Recent efforts have encapsulated these core operations and provided language-agnostic interfaces. This greatly increases the accessibility and flexibility of TeraChem as a platform to develop new electronic structure methods on GPUs and provides clear optimization targets for emerging parallel computing architectures.

Structural insights into the aPKC regulatory switch mechanism of the human cell polarity protein lethal giant larvae 2.

Metazoan cell polarity is controlled by a set of highly conserved proteins. Lethal giant larvae (Lgl) functions in apical-basal polarity through phosphorylation-dependent interactions with several other proteins as well as the plasma membrane. Phosphorylation of Lgl by atypical protein kinase C (aPKC), a component of the partitioning-defective (Par) complex in epithelial cells, excludes Lgl from the apical membrane, a crucial step in the establishment of epithelial cell polarity. We present the crystal structures of human Lgl2 in both its unphosphorylated and aPKC-phosphorylated states. Lgl2 adopts a double ß-propeller structure that is unchanged by aPKC phosphorylation of an unstructured loop in its second ß-propeller, ruling out models of phosphorylation-dependent conformational change. We demonstrate that phosphorylation controls the direct binding of purified Lgl2 to negative phospholipids in vitro. We also show that a coil-helix transition of this region that is promoted by phosphatidylinositol 4,5-bisphosphate (PIP2) is also phosphorylation-dependent, implying a highly effective phosphorylative switch for membrane association.

Unsupervised determination of protein crystal structures.

We present a method for automatic solution of protein crystal structures. The method proceeds with a single initial model obtained, for instance, by molecular replacement (MR). If a good-quality search model is not available, as often is the case with MR of distant homologs, our method first can automatically screen a large pool of poorly placed models and single out promising candidates for further processing if there are any. We demonstrate its utility by solving a set of synthetic cases in the 2.9- to 3.45-Å resolution.

Solving the structure of Lgl2, a difficult blind test of unsupervised structure determination.

In the companion paper by Ufimtsev and Levitt [Ufimtsev IS, Levitt M (2019) Proc Natl Acad Sci USA, 10.1073/pnas.1821512116], we presented a method for unsupervised solution of protein crystal structures and demonstrated its utility by solving several test cases of known structure in the 2.9- to 3.45-Å resolution range. Here we apply this method to solve the crystal structure of a 966-amino acid construct of human lethal giant larvae protein (Lgl2) that resisted years of structure determination efforts, at 3.2-Å resolution. The structure was determined starting with a molecular replacement (MR) model identified by unsupervised refinement of a pool of 50 candidate MR models. This initial model had 2.8-Å RMSD from the solution. The solved structure was validated by comparison with a model subsequently derived from an alternative crystal form diffracting to higher resolution. This model could phase an anomalous difference Fourier map from an Hg derivative, and a single-wavelength anomalous dispersion phased density map made from these sites aligned with the refined structure.

Analytic first derivatives of floating occupation molecular orbital-complete active space configuration interaction on graphical processing units.

The floating occupation molecular orbital-complete active space configuration interaction (FOMO-CASCI) method is a promising alternative to the state-averaged complete active space self-consistent field (SA-CASSCF) method. We have formulated the analytic first derivative of FOMO-CASCI in a manner that is well-suited for a highly efficient implementation using graphical processing units (GPUs). Using this implementation, we demonstrate that FOMO-CASCI gradients are of similar computational expense to configuration interaction singles (CIS) or time-dependent density functional theory (TDDFT). In contrast to CIS and TDDFT, FOMO-CASCI can describe multireference character of the electronic wavefunction. We show that FOMO-CASCI compares very favorably to SA-CASSCF in its ability to describe molecular geometries and potential energy surfaces around minimum energy conical intersections. Finally, we apply FOMO-CASCI to the excited state hydrogen transfer reaction in methyl salicylate.

An atomic orbital-based formulation of the complete active space self-consistent field method on graphical processing units.

Despite its importance, state-of-the-art algorithms for performing complete active space self-consistent field (CASSCF) computations have lagged far behind those for single reference methods. We develop an algorithm for the CASSCF orbital optimization that uses sparsity in the atomic orbital (AO) basis set to increase the applicability of CASSCF. Our implementation of this algorithm uses graphical processing units (GPUs) and has allowed us to perform CASSCF computations on molecular systems containing more than one thousand atoms. Additionally, we have implemented analytic gradients of the CASSCF energy; the gradients also benefit from GPU acceleration as well as sparsity in the AO basis.

Generating Efficient Quantum Chemistry Codes for Novel Architectures.

We describe an extension of our graphics processing unit (GPU) electronic structure program TeraChem to include atom-centered Gaussian basis sets with d angular momentum functions. This was made possible by a "meta-programming" strategy that leverages computer algebra systems for the derivation of equations and their transformation to correct code. We generate a multitude of code fragments that are formally mathematically equivalent, but differ in their memory and floating-point operation footprints. We then select between different code fragments using empirical testing to find the highest performing code variant. This leads to an optimal balance of floating-point operations and memory bandwidth for a given target architecture without laborious manual tuning. We show that this approach is capable of similar performance compared to our hand-tuned GPU kernels for basis sets with s and p angular momenta. We also demonstrate that mixed precision schemes (using both single and double precision) remain stable and accurate for molecules with d functions. We provide benchmarks of the execution time of entire self-consistent field (SCF) calculations using our GPU code and compare to mature CPU based codes, showing the benefits of the GPU architecture for electronic structure theory with appropriately redesigned algorithms. We suggest that the meta-programming and empirical performance optimization approach may be important in future computational chemistry applications, especially in the face of quickly evolving computer architectures.

Ab initio quantum chemistry for protein structures.

Structural properties of over 55 small proteins have been determined using both density-based and wave-function-based electronic structure methods in order to assess the ability of ab initio "force fields" to retain the properties described by experimental structures measured with crystallography or nuclear magnetic resonance. The efficiency of the GPU-based quantum chemistry algorithms implemented in our TeraChem program enables us to carry out systematic optimization of ab initio protein structures, which we compare against experimental and molecular mechanics force field references. We show that the quality of the ab initio optimized structures, as judged by conventional protein health metrics, increases with increasing basis set size. On the other hand, there is little evidence for a significant improvement of predicted structures using density functional theory as compared to Hartree-Fock methods. Although occasional pathologies of minimal basis sets are observed, these are easily alleviated with even the smallest double-ζ basis sets.

Excited-State Electronic Structure with Configuration Interaction Singles and Tamm-Dancoff Time-Dependent Density Functional Theory on Graphical Processing Units.

Excited-state calculations are implemented in a development version of the GPU-based TeraChem software package using the configuration interaction singles (CIS) and adiabatic linear response Tamm-Dancoff time-dependent density functional theory (TDA-TDDFT) methods. The speedup of the CIS and TDDFT methods using GPU-based electron repulsion integrals and density functional quadrature integration allows full ab initio excited-state calculations on molecules of unprecedented size. CIS/6-31G and TD-BLYP/6-31G benchmark timings are presented for a range of systems, including four generations of oligothiophene dendrimers, photoactive yellow protein (PYP), and the PYP chromophore solvated with 900 quantum mechanical water molecules. The effects of double and single precision integration are discussed, and mixed precision GPU integration is shown to give extremely good numerical accuracy for both CIS and TDDFT excitation energies (excitation energies within 0.0005 eV of extended double precision CPU results).

Dynamic Precision for Electron Repulsion Integral Evaluation on Graphical Processing Units (GPUs).

It has recently been demonstrated that novel streaming architectures found in consumer video gaming hardware such as graphical processing units (GPUs) are well-suited to a broad range of computations including electronic structure theory (quantum chemistry). Although recent GPUs have developed robust support for double precision arithmetic, they continue to provide 2-8× more hardware units for single precision. In order to maximize performance on GPU architectures, we present a technique of dynamically selecting double or single precision evaluation for electron repulsion integrals (ERIs) in Hartree-Fock and density functional self-consistent field (SCF) calculations. We show that precision error can be effectively controlled by evaluating only the largest integrals in double precision. By dynamically scaling the precision cutoff over the course of the SCF procedure, we arrive at a scheme that minimizes the number of double precision integral evaluations for any desired accuracy. This dynamic precision scheme is shown to be effective for an array of molecules ranging in size from 20 to nearly 2000 atoms.

GPU-accelerated molecular modeling coming of age.

Graphics processing units (GPUs) have traditionally been used in molecular modeling solely for visualization of molecular structures and animation of trajectories resulting from molecular dynamics simulations. Modern GPUs have evolved into fully programmable, massively parallel co-processors that can now be exploited to accelerate many scientific computations, typically providing about one order of magnitude speedup over CPU code and in special cases providing speedups of two orders of magnitude. This paper surveys the development of molecular modeling algorithms that leverage GPU computing, the advances already made and remaining issues to be resolved, and the continuing evolution of GPU technology that promises to become even more useful to molecular modeling. Hardware acceleration with commodity GPUs is expected to benefit the overall computational biology community by bringing teraflops performance to desktop workstations and in some cases potentially changing what were formerly batch-mode computational jobs into interactive tasks.

A multistate empirical valence bond model for solvation and transport simulations of OH- in aqueous solutions.

We describe a new multistate empirical valence bond (MS-EVB) model of OH(-) in aqueous solutions. This model is based on the recently proposed "charged ring" parameterization for the intermolecular interaction of hydroxyl ion with water [Ufimtsev, et al., Chem. Phys. Lett., 2007, 442, 128] and is suitable for classical molecular simulations of OH(-) solvation and transport. The model reproduces the hydration structure of OH(-)(aq) in good agreement with experimental data and the results of ab initio molecular dynamics simulations. It also accurately captures the major structural, energetic, and dynamic aspects of the proton transfer processes involving OH(-) (aq). The model predicts an approximately two-fold increase of the OH(-) mobility due to proton exchange reactions.

Observation of a Zundel-like transition state during proton transfer in aqueous hydroxide solutions.

It is generally accepted that the anomalous diffusion of the aqueous hydroxide ion results from its ability to accept a proton from a neighboring water molecule; yet, many questions exist concerning the mechanism for this process. What is the solvation structure of the hydroxide ion? In what way do water hydrogen bond dynamics influence the transfer of a proton to the ion? We present the results of femtosecond pump-probe and 2D infrared experiments that probe the O-H stretching vibration of a solution of dilute HOD dissolved in NaOD/D(2)O. Upon the addition of NaOD, measured pump-probe transients and 2D IR spectra show a new feature that decays with a 110-fs time scale. The calculation of 2D IR spectra from an empirical valence bond molecular dynamics simulation of a single NaOH molecule in a bath of H(2)O indicates that this fast feature is due to an overtone transition of Zundel-like H(3)O(2)(-) states, wherein a proton is significantly shared between a water molecule and the hydroxide ion. Given the frequency of vibration of shared protons, the observations indicate the shared proton state persists for 2-3 vibrational periods before the proton localizes on a hydroxide. Calculations based on the EVB-MD model argue that the collective electric field in the proton transfer direction is the appropriate coordinate to describe the creation and relaxation of these Zundel-like transition states.

Quantum Chemistry on Graphical Processing Units. 2. Direct Self-Consistent-Field Implementation.

We demonstrate the use of graphical processing units (GPUs) to carry out complete self-consistent-field calculations for molecules with as many as 453 atoms (2131 basis functions). Speedups ranging from 28× to 650× are achieved as compared to a mature third-party quantum chemistry program (GAMESS) running on a traditional CPU. The computational organization used to construct the Coulomb and exchange operators is discussed. We also present results using three GPUs in parallel, combining coarse and fine-grained parallelism.

Quantum Chemistry on Graphical Processing Units. 3. Analytical Energy Gradients, Geometry Optimization, and First Principles Molecular Dynamics.

We demonstrate that a video gaming machine containing two consumer graphical cards can outpace a state-of-the-art quad-core processor workstation by a factor of more than 180× in Hartree-Fock energy + gradient calculations. Such performance makes it possible to run large scale Hartree-Fock and Density Functional Theory calculations, which typically require hundreds of traditional processor cores, on a single workstation. Benchmark Born-Oppenheimer molecular dynamics simulations are performed on two molecular systems using the 3-21G basis set - a hydronium ion solvated by 30 waters (94 atoms, 405 basis functions) and an aspartic acid molecule solvated by 147 waters (457 atoms, 2014 basis functions). Our GPU implementation can perform 27 ps/day and 0.7 ps/day of ab initio molecular dynamics simulation on a single desktop computer for these systems.

Quantum Chemistry on Graphical Processing Units. 1. Strategies for Two-Electron Integral Evaluation.

Modern videogames place increasing demands on the computational and graphical hardware, leading to novel architectures that have great potential in the context of high performance computing and molecular simulation. We demonstrate that Graphical Processing Units (GPUs) can be used very efficiently to calculate two-electron repulsion integrals over Gaussian basis functions [Formula: see text] the first step in most quantum chemistry calculations. A benchmark test performed for the evaluation of approximately 10(6) (ss|ss) integrals over contracted s-orbitals showed that a naïve algorithm implemented on the GPU achieves up to 130-fold speedup over a traditional CPU implementation on an AMD Opteron. Subsequent calculations of the Coulomb operator for a 256-atom DNA strand show that the GPU advantage is maintained for basis sets including higher angular momentum functions.