Quantum Perturbation Theory Using Tensor Cores and a Deep Neural Network.

Time-independent quantum response calculations are performed using Tensor cores. This is achieved by mapping density matrix perturbation theory onto the computational structure of a deep neural network. The main computational cost of each deep layer is dominated by tensor contractions, i.e., dense matrix-matrix multiplications, in mixed-precision arithmetics, which achieves close to peak performance. Quantum response calculations are demonstrated and analyzed using self-consistent charge density-functional tight-binding theory as well as coupled-perturbed Hartree-Fock theory. For linear response calculations, a novel parameter-free convergence criterion is presented that is well-suited for numerically noisy low-precision floating point operations and we demonstrate a peak performance of almost 200 Tflops using the Tensor cores of two Nvidia A100 GPUs.

Quantum-Based Molecular Dynamics Simulations Using Tensor Cores.

Tensor cores, along with tensor processing units, represent a new form of hardware acceleration specifically designed for deep neural network calculations in artificial intelligence applications. Tensor cores provide extraordinary computational speed and energy efficiency but with the caveat that they were designed for tensor contractions (matrix-matrix multiplications) using only low-precision floating-point operations. Despite this perceived limitation, we demonstrate how tensor cores can be applied with high efficiency to the challenging and numerically sensitive problem of quantum-based Born-Oppenheimer molecular dynamics, which requires highly accurate electronic structure optimizations and conservative force evaluations. The interatomic forces are calculated on-the-fly from an electronic structure that is obtained from a generalized deep neural network, where the computational structure naturally takes advantage of the exceptional processing power of the tensor cores and allows for high performance in excess of 100 Tflops on a single Nvidia A100 GPU. Stable molecular dynamics trajectories are generated using the framework of extended Lagrangian Born-Oppenheimer molecular dynamics, which combines computational efficiency with long-term stability, even when using approximate charge relaxations and force evaluations that are limited in accuracy by the numerically noisy conditions caused by the low-precision tensor core floating-point operations. A canonical ensemble simulation scheme is also presented, where the additional numerical noise in the calculated forces is absorbed into a Langevin-like dynamics.

Mixed Precision Fermi-Operator Expansion on Tensor Cores from a Machine Learning Perspective.

We present a second-order recursive Fermi-operator expansion scheme using mixed precision floating point operations to perform electronic structure calculations using tensor core units. A performance of over 100 teraFLOPs is achieved for half-precision floating point operations on Nvidia's A100 tensor core units. The second-order recursive Fermi-operator scheme is formulated in terms of a generalized, differentiable deep neural network structure, which solves the quantum mechanical electronic structure problem. We demonstrate how this network can be accelerated by optimizing the weight and bias values to substantially reduce the number of layers required for convergence. We also show how this machine learning approach can be used to optimize the coefficients of the recursive Fermi-operator expansion to accurately represent the fractional occupation numbers of the electronic states at finite temperatures.

Linear Scaling Pseudo Fermi-Operator Expansion for Fractional Occupation.

Recursive Fermi-operator expansion methods for the calculation of the idempotent density matrix are valid only at zero electronic temperature with integer occupation numbers. We show how such methods can be modified to include fractional occupation numbers of an approximate or pseudo Fermi-Dirac distribution and how the corresponding entropy term of the free energy is calculated. The proposed methodology is demonstrated and evaluated for different electronic structure methods, including density functional tight-binding theory, Kohn-Sham density functional theory using numerical orbitals, and quantum chemistry Hartree-Fock theory using Gaussian basis functions.

On-the-Fly Computation of Frontal Orbitals in Density Matrix Expansions.

We propose a method for computation of frontal (homo and lumo) orbitals in recursive polynomial expansion algorithms for the density matrix. Such algorithms give a computational cost that increases only linearly with system size for sufficiently sparse systems, but a drawback compared to the traditional diagonalization approach is that molecular orbitals are not readily available. Our method is based on the idea to use the polynomial of the density matrix expansion as an eigenvalue filter giving large separation between eigenvalues around homo and lumo [ Rubensson et al. J. Chem. Phys. 2008 , 128 , 176101 ]. This filter is combined with a shift-and-square (folded spectrum) method to move the desired eigenvalue to the end of the spectrum. In this work we propose a transparent way to select recursive expansion iteration and shift for the eigenvector computation that results in a sharp eigenvalue filter. The filter is obtained as a byproduct of the density matrix expansion, and there is no significant additional cost associated either with its construction or with its application. This gives a clear-cut and efficient eigenvalue solver that can be used to compute homo and lumo orbitals with sufficient accuracy in a small fraction of the total recursive expansion time. Our algorithms make use of recent homo and lumo eigenvalue estimates that can be obtained at negligible cost [ Rubensson et al. SIAM J. Sci. Comput . 2014 , 36 , B147 ]. We illustrate our method by performing self-consistent field calculations for large scale systems.

Parameterless Stopping Criteria for Recursive Density Matrix Expansions.

Parameterless stopping criteria for recursive polynomial expansions to construct the density matrix in electronic structure calculations are proposed. Based on convergence-order estimation the new stopping criteria automatically and accurately detect when the calculation is dominated by numerical errors and continued iteration does not improve the result. Difficulties in selecting a stopping tolerance and appropriately balancing it in relation to parameters controlling the numerical accuracy are avoided. Thus, our parameterless stopping criteria stand in contrast to the standard approach to stop as soon as some error measure goes below a user-defined parameter or tolerance. We demonstrate that the stopping criteria work well both in dense and sparse matrix calculations and in large-scale self-consistent field calculations with the quantum chemistry program Ergo ( www.ergoscf.org ) .

Graph-based linear scaling electronic structure theory.

We show how graph theory can be combined with quantum theory to calculate the electronic structure of large complex systems. The graph formalism is general and applicable to a broad range of electronic structure methods and materials, including challenging systems such as biomolecules. The methodology combines well-controlled accuracy, low computational cost, and natural low-communication parallelism. This combination addresses substantial shortcomings of linear scaling electronic structure theory, in particular with respect to quantum-based molecular dynamics simulations.

Canonical density matrix perturbation theory.

Density matrix perturbation theory [Niklasson and Challacombe, Phys. Rev. Lett. 92, 193001 (2004)] is generalized to canonical (NVT) free-energy ensembles in tight-binding, Hartree-Fock, or Kohn-Sham density-functional theory. The canonical density matrix perturbation theory can be used to calculate temperature-dependent response properties from the coupled perturbed self-consistent field equations as in density-functional perturbation theory. The method is well suited to take advantage of sparse matrix algebra to achieve linear scaling complexity in the computational cost as a function of system size for sufficiently large nonmetallic materials and metals at high temperatures.

Assessment of density matrix methods for linear scaling electronic structure calculations.

Purification and minimization methods for linear scaling computation of the one-particle density matrix for a fixed Hamiltonian matrix are compared. This is done by considering the work needed by each method to achieve a given accuracy in terms of the difference from the exact solution. Numerical tests employing orthogonal as well as non-orthogonal versions of the methods are performed using both element magnitude and cutoff radius based truncation approaches. It is investigated how the convergence speed for the different methods depends on the eigenvalue distribution in the Hamiltonian matrix. An expression for the number of iterations required for the minimization methods studied is derived, taking into account the dependence on both the band gap and the chemical potential. This expression is confirmed by numerical tests. The minimization methods are found to perform at their best when the chemical potential is located near the center of the eigenspectrum. The results indicate that purification is considerably more efficient than the minimization methods studied even when a good starting guess for the minimization is available. In test calculations without a starting guess, purification is more than an order of magnitude more efficient than minimization.

Bringing about matrix sparsity in linear-scaling electronic structure calculations.

The performance of linear-scaling electronic structure calculations depends critically on matrix sparsity. This article gives an overview of different strategies for removal of small matrix elements, with emphasis on schemes that allow for rigorous control of errors. In particular, a novel scheme is proposed that has significantly smaller computational overhead compared with the Euclidean norm-based truncation scheme of Rubensson et al. (J Comput Chem 2009, 30, 974) while still achieving the desired asymptotic behavior required for linear scaling. Small matrix elements are removed while ensuring that the Euclidean norm of the error matrix stays below a desired value, so that the resulting error in the occupied subspace can be controlled. The efficiency of the new scheme is investigated in benchmark calculations for water clusters including up to 6523 water molecules. Furthermore, the foundation of matrix sparsity is investigated. This includes a study of the decay of matrix element magnitude with distance between basis function centers for different molecular systems and different methods. The studied methods include Hartree­Fock and density functional theory using both pure and hybrid functionals. The relation between band gap and decay properties of the density matrix is also discussed.

Kohn-Sham Density Functional Theory Electronic Structure Calculations with Linearly Scaling Computational Time and Memory Usage.

We present a complete linear scaling method for hybrid Kohn-Sham density functional theory electronic structure calculations and demonstrate its performance. Particular attention is given to the linear scaling computation of the Kohn-Sham exchange-correlation matrix directly in sparse form within the generalized gradient approximation. The described method makes efficient use of sparse data structures at all times and scales linearly with respect to both computational time and memory usage. Benchmark calculations at the BHandHLYP/3-21G level of theory are presented for polypeptide helix molecules with up to 53 250 atoms. Threshold values for computational approximations were chosen on the basis of their impact on the occupied subspace so that the different parts of the calculations were carried out at balanced levels of accuracy. The largest calculation used 307 204 Gaussian basis functions on a single computer with 72 GB of memory. Benchmarks for three-dimensional water clusters are also included, as well as results using the 6-31G** basis set.

Nonmonotonic Recursive Polynomial Expansions for Linear Scaling Calculation of the Density Matrix.

As it stands, density matrix purification is a powerful tool for linear scaling electronic structure calculations. The convergence is rapid and depends only weakly on the band gap. However, as will be shown in this letter, there is room for improvements. The key is to allow for nonmonotonicity in the recursive polynomial expansion. On the basis of this idea, new purification schemes are proposed that require only half the number of matrix-matrix multiplications compared to previous schemes. The speedup is essentially independent of the location of the chemical potential and increases with decreasing band gap.

Truncation of small matrix elements based on the Euclidean norm for blocked data structures.

Methods for the removal of small symmetric matrix elements based on the Euclidean norm of the error matrix are presented in this article. In large scale Hartree-Fock and Kohn-Sham calculations it is important to be able to enforce matrix sparsity while keeping errors under control. Truncation based on some unitary-invariant norm allows for control of errors in the occupied subspace as described in (Rubensson et al. J Math Phys 49, 032103). The Euclidean norm is unitary-invariant and does not grow intrinsically with system size and is thus suitable for error control in large scale calculations. The presented truncation schemes repetitively use the Lanczos method to compute the Euclidean norms of the error matrix candidates. Ritz value convergence patterns are utilized to reduce the total number of Lanczos iterations.

Automatic Selection of Integral Thresholds by Extrapolation in Coulomb and Exchange Matrix Constructions.

We present a method to compute Coulomb and exchange matrices with predetermined accuracy as measured by a matrix norm. The computation of these matrices is fundamental in Hartree-Fock and Kohn-Sham electronic structure calculations. We show numerically that, when modern algorithms for Coulomb and exchange matrix evaluation are applied, the Euclidean norm of the error matrix ε is related to the threshold value τ as ε = cτ(α). The presented extrapolation method automatically selects the integral thresholds so that the Euclidean norm of the error matrix is at the requested accuracy. This approach is demonstrated for a variety of systems, including protein-like systems, water clusters, and graphene sheets. The proposed method represents an important step toward complete error control throughout the self-consistent field calculation as described in [J. Math. Phys. 2008, 49, 032103].

Hartree-Fock calculations with linearly scaling memory usage.

We present an implementation of a set of algorithms for performing Hartree-Fock calculations with resource requirements in terms of both time and memory directly proportional to the system size. In particular, a way of directly computing the Hartree-Fock exchange matrix in sparse form is described which gives only small addressing overhead. Linear scaling in both time and memory is demonstrated in benchmark calculations for system sizes up to 11 650 atoms and 67 204 Gaussian basis functions on a single computer with 32 Gbytes of memory. The sparsity of overlap, Fock, and density matrices as well as band gaps are also shown for a wide range of system sizes, for both linear and three-dimensional systems.

Computation of interior eigenvalues in electronic structure calculations facilitated by density matrix purification.

Density matrix purification, is in this work, used to facilitate the computation of eigenpairs around the highest occupied and the lowest unoccupied molecular orbitals (HOMO and LUMO, respectively) in electronic structure calculations. The ability of purification to give large separation between eigenvalues close to the HOMO-LUMO gap is used to accelerate convergence of the Lanczos method. Illustrations indicate that a new eigenpair is found more often than every second Lanczos iteration when the proposed methods are used.

Recursive inverse factorization.

A recursive algorithm for the inverse factorization S(-1)=ZZ(*) of Hermitian positive definite matrices S is proposed. The inverse factorization is based on iterative refinement [A.M.N. Niklasson, Phys. Rev. B 70, 193102 (2004)] combined with a recursive decomposition of S. As the computational kernel is matrix-matrix multiplication, the algorithm can be parallelized and the computational effort increases linearly with system size for systems with sufficiently sparse matrices. Recent advances in network theory are used to find appropriate recursive decompositions. We show that optimization of the so-called network modularity results in an improved partitioning compared to other approaches. In particular, when the recursive inverse factorization is applied to overlap matrices of irregularly structured three-dimensional molecules.

Density matrix purification with rigorous error control.

Density matrix purification, although being a powerful tool for linear scaling construction of the density matrix in electronic structure calculations, has been limited by uncontrolled error accumulation. In this article, a strategy for the removal of small matrix elements in density matrix purification is proposed with which the forward error can be rigorously controlled. The total forward error is separated into two parts, the error in eigenvalues and the error in the occupied invariant subspace. We use the concept of canonical angles to measure and control differences between exact and approximate occupied subspaces. We also analyze the conditioning of the density matrix construction problem and propose a method for calculation of interior eigenvalues to be used together with density matrix purification.

A hierarchic sparse matrix data structure for large-scale Hartree-Fock/Kohn-Sham calculations.

A hierarchic sparse matrix data structure for Hartree-Fock/Kohn-Sham calculations is presented. The data structure makes the implementation of matrix manipulations needed for large systems faster, easier, and more maintainable without loss of performance. Algorithms for symmetric matrix square and inverse Cholesky decomposition within the hierarchic framework are also described. The presented data structure is general; in addition to its use in Hartree-Fock/Kohn-Sham calculations, it may also be used in other research areas where matrices with similar properties are encountered. The applicability of the data structure to ab initio calculations is shown with help of benchmarks on water droplets and graphene nanoribbons.

Systematic sparse matrix error control for linear scaling electronic structure calculations.

Efficient truncation criteria used in multiatom blocked sparse matrix operations for ab initio calculations are proposed. As system size increases, so does the need to stay on top of errors and still achieve high performance. A variant of a blocked sparse matrix algebra to achieve strict error control with good performance is proposed. The presented idea is that the condition to drop a certain submatrix should depend not only on the magnitude of that particular submatrix, but also on which other submatrices that are dropped. The decision to remove a certain submatrix is based on the contribution the removal would cause to the error in the chosen norm. We study the effect of an accumulated truncation error in iterative algorithms like trace correcting density matrix purification. One way to reduce the initial exponential growth of this error is presented. The presented error control for a sparse blocked matrix toolbox allows for achieving optimal performance by performing only necessary operations needed to maintain the requested level of accuracy.