Hybrid programming-model strategies for GPU offloading of electronic structure calculation kernels.

To address the challenge of performance portability and facilitate the implementation of electronic structure solvers, we developed the basic matrix library (BML) and Parallel, Rapid O(N), and Graph-based Recursive Electronic Structure Solver (PROGRESS) library. The BML implements linear algebra operations necessary for electronic structure kernels using a unified user interface for various matrix formats (dense and sparse) and architectures (CPUs and GPUs). Focusing on density functional theory and tight-binding models, PROGRESS implements several solvers for computing the single-particle density matrix and relies on BML. In this paper, we describe the general strategies used for these implementations on various computer architectures, using OpenMP target functionalities on GPUs, in conjunction with third-party libraries to handle performance critical numerical kernels. We demonstrate the portability of this approach and its performance in benchmark problems.

A fast, dense Chebyshev solver for electronic structure on GPUs.

Matrix diagonalization is almost always involved in computing the density matrix needed in quantum chemistry calculations. In the case of modest matrix sizes (≲4000), performance of traditional dense diagonalization algorithms on modern GPUs is underwhelming compared to the peak performance of these devices. This motivates the exploration of alternative algorithms better suited to these types of architectures. We newly derive, and present in detail, an existing Chebyshev expansion algorithm [Liang et al., J. Chem. Phys. 119, 4117-4125 (2003)] whose number of required matrix multiplications scales with the square root of the number of terms in the expansion. Focusing on dense matrices of modest size, our implementation on GPUs results in large speed ups when compared to diagonalization. Additionally, we improve upon this existing method by capitalizing on the inherent task parallelism and concurrency in the algorithm. This improvement is implemented on GPUs by using CUDA and HIP streams via the MAGMA library and leads to a significant speed up over the serial-only approach for smaller (≲1000) matrix sizes. Finally, we apply our technique to a model system with a high density of states around the Fermi level, which typically presents significant challenges.

Efficient Computational Modeling of Human Ventricular Activation and Its Electrocardiographic Representation: A Sensitivity Study.

Patient-specific models of the ventricular myocardium, combined with the computational power to run rapid simulations, are approaching the level where they could be used for personalized cardiovascular medicine. A major remaining challenge is determining model parameters from available patient data, especially for models of the Purkinje-myocardial junctions (PMJs): the sites of initial ventricular electrical activation. There are no non-invasive methods for localizing PMJs in patients, and the relationship between the standard clinical ECG and PMJ model parameters is underexplored. Thus, this study aimed to determine the sensitivity of the QRS complex of the ECG to the anatomical location and regional number of PMJs. The QRS complex was simulated using an image-based human torso and biventricular model, and cardiac electrophysiology was simulated using Cardioid. The PMJs were modeled as discrete current injection stimuli, and the location and number of stimuli were varied within initial activation regions based on published experiments. Results indicate that the QRS complex features were most sensitive to the presence or absence of four "seed" stimuli, and adjusting locations of nearby "regional" stimuli provided finer tuning. Decreasing number of regional stimuli by an order of magnitude resulted in virtually no change in the QRS complex. Thus, a minimal 12-stimuli configuration was identified that resulted in physiological excitation, defined by QRS complex feature metrics and ventricular excitation pattern. Overall, the sensitivity results suggest that parameterizing PMJ location, rather than number, be given significantly higher priority in future studies creating personalized ventricular models from patient-derived ECGs.

Large-Scale First-Principles Molecular Dynamics Simulations with Electrostatic Embedding: Application to Acetylcholinesterase Catalysis.

Enzymes are complicated solvated systems that typically require many atoms to simulate their function with any degree of accuracy. We have recently developed numerical techniques for large scale first-principles molecular dynamics simulations and applied them to the study of the enzymatic reaction catalyzed by acetylcholinesterase. We carried out density functional theory calculations for a quantum-mechanical (QM) subsystem consisting of 612 atoms with an O(N) complexity finite-difference approach. The QM subsystem is embedded inside an external potential field representing the electrostatic effect due to the environment. We obtained finite-temperature sampling by first-principles molecular dynamics for the acylation reaction of acetylcholine catalyzed by acetylcholinesterase. Our calculations show two energy barriers along the reaction coordinate for the enzyme-catalyzed acylation of acetylcholine. The second barrier (8.5 kcal/mol) is rate-limiting for the acylation reaction and in good agreement with experiment.

A wrench in the works of human acetylcholinesterase: soman induced conformational changes revealed by molecular dynamics simulations.

Irreversible inactivation of human acetylcholinesterase (hAChE) by organophosphorous pesticides (OPs) and chemical weapon agents (CWA) has severe morbidity and mortality consequences. We present data from quantum mechanics/molecular mechanics (QM/MM) and 80 classical molecular dynamics (MD) simulations of the apo and soman-adducted forms of hAChE to investigate the effects on the dynamics and protein structure when the catalytic Serine 203 is phosphonylated. We find that the soman phosphonylation of the active site Ser203 follows a water assisted addition-elimination mechanism with the elimination of the fluoride ion being the highest energy barrier at 6.5 kcal/mole. We observe soman-dependent changes in backbone and sidechain motions compared to the apo form of the protein. These alterations restrict the soman-adducted hAChE to a structural state that is primed for the soman adduct to be cleaved and removed from the active site. The altered motions and resulting structures provide alternative pathways into and out of the hAChE active site. In the soman-adducted protein both side and back door pathways are viable for soman adduct access. Correlation analysis of the apo and soman adducted MD trajectories shows that the correlation of gorge entrance and back door motion is disrupted when hAChE is adducted. This supports the hypothesis that substrate and product can use two different pathways as entry and exit sites in the apo form of the protein. These alternative pathways have important implications for the rational design of medical countermeasures.

A high-resolution computational model of the deforming human heart.

Modeling of the heart ventricles is one of the most challenging tasks in soft tissue mechanics because cardiac tissue is a strongly anisotropic incompressible material with an active component of stress. In most current approaches with active force, the number of degrees of freedom (DOF) is limited by the direct method of solution of linear systems of equations. We develop a new approach for high-resolution heart models with large numbers of DOF by: (1) developing a hex-dominant finite element mixed formulation and (2) developing a Krylov subspace iterative method that is able to solve the system of linearized equations for saddle-point problems with active stress. In our approach, passive cardiac tissue is modeled as a hyperelastic, incompressible material with orthotropic properties, and mixed pressure-displacement finite elements are used to enforce incompressibility. Active stress is generated by a model with force dependence on length and velocity of muscle shortening. The ventricles are coupled to a lumped circulatory model. For efficient solution of linear systems, we use Flexible GMRES with a nonlinear preconditioner based on block matrix decomposition involving the Schur complement. Three methods for approximating the inverse of the Schur complement are evaluated: inverse of the pressure mass matrix; least squares commutators; and sparse approximate inverse. The sub-matrix corresponding to the displacement variables is preconditioned by a V-cycle of hybrid geometric-algebraic multigrid followed by correction with several iterations of GMRES preconditioned by sparse approximate inverse. The overall solver is demonstrated on a high-resolution two ventricle mesh based on a human anatomy with roughly 130 K elements and 1.7 M displacement DOF. Effectiveness of the numerical method for active contraction is shown. To the best of our knowledge, this solver is the first to efficiently model ventricular contraction using an iterative linear solver for the mesh size demonstrated and therefore opens the possibility for future very high-resolution models. In addition, several relatively simple benchmark problems are designed for a verification exercise to show that the solver is functioning properly and correctly solves the underlying mathematical model. Here, the output of the newly designed solver is compared to that of the mechanics component of Chaste ('Cancer, Heart and Soft Tissue Environment'). These benchmark tests may be used by other researchers to verify their newly developed methods and codes.

Accurate and scalable O(N) algorithm for first-principles molecular-dynamics computations on large parallel computers.

We present the first truly scalable first-principles molecular dynamics algorithm with O(N) complexity and controllable accuracy, capable of simulating systems with finite band gaps of sizes that were previously impossible with this degree of accuracy. By avoiding global communications, we provide a practical computational scheme capable of extreme scalability. Accuracy is controlled by the mesh spacing of the finite difference discretization, the size of the localization regions in which the electronic wave functions are confined, and a cutoff beyond which the components of the overlap matrix can be omitted when computing selected elements of its inverse. We demonstrate the algorithm's excellent parallel scaling for up to 101,952 atoms on 23,328 processors, with a wall-clock time of the order of 1 min per molecular dynamics time step and numerical error on the forces of less than 7×10(-4) Ha/Bohr.

Towards real-time simulation of cardiac electrophysiology in a human heart at high resolution.

We have developed the capability to rapidly simulate cardiac electrophysiological phenomena in a human heart discretised at a resolution comparable with the length of a cardiac myocyte. Previous scientific investigation has generally invoked simplified geometries or coarse-resolution hearts, with simulation duration limited to 10s of heartbeats. Using state-of-the-art high-performance computing techniques coupled with one of the most powerful computers available (the 20 PFlop/s IBM BlueGene/Q at Lawrence Livermore National Laboratory), high-resolution simulation of the human heart can now be carried out over 1200 times faster compared with published results in the field. We demonstrate the utility of this capability by simulating, for the first time, the formation of transmural re-entrant waves in a 3D human heart. Such wave patterns are thought to underlie Torsades de Pointes, an arrhythmia that indicates a high risk of sudden cardiac death. Our new simulation capability has the potential to impact a multitude of applications in medicine, pharmaceuticals and implantable devices.

Stacking of oligo- and polythiophene cations in solution: surface tension and dielectric saturation.

The stacking of positively charged (or doped) terthiophene oligomers and quaterthiophene polymers in solution is investigated applying a recently developed unified electrostatic and cavitation model for first-principles calculations in a continuum solvent. The thermodynamic and structural patterns of the dimerization are explored in different solvents, and the distinctive roles of polarity and surface tension are characterized and analyzed. Interestingly, we discover a saturation in the stabilization effect of the dielectric screening that takes place at rather small values of epsilon(0). Moreover, we address the interactions in trimers of terthiophene cations, with the aim of generalizing the results obtained for the dimers to the case of higher-order stacks and nanoaggregates.

A unified electrostatic and cavitation model for first-principles molecular dynamics in solution.

The electrostatic continuum solvent model developed by [Fattebert and Gygi J. Comput. Chem. 23, 662 (2002); Int. J. Quantum Chem. 93, 139 (2003)] is combined with a first-principles formulation of the cavitation energy based on a natural quantum-mechanical definition for the surface of a solute. Despite its simplicity, the cavitation contribution calculated by this approach is found to be in remarkable agreement with that obtained by more complex algorithms relying on a large set of parameters. Our model allows for very efficient Car-Parrinello simulations of finite or extended systems in solution and demonstrates a level of accuracy as good as that of established quantum-chemistry continuum solvent methods. We apply this approach to the study of tetracyanoethylene dimers in dichloromethane, providing valuable structural and dynamical insights on the dimerization phenomenon.

Optical properties of silicon clusters in the presence of water: a first principles theoretical analysis.

We investigate the impact of water on the optical absorption of prototypical silicon clusters. Our clusters contain 5 silicon atoms, tetrahedrally coordinated and passivated with either hydrogen or oxygen. We approach this complex problem by assessing the contributions of three factors: chemical reactivity, thermal equilibration, and dielectric screening. We find that the silanone (Si=O) functional group is not chemically stable in the presence of water and exclude this as a source of significant red shift in absorption in aqueous environments. We perform first principles molecular dynamics simulations of the solvation of a chemically stable, oxygenated silicon cluster with explicit water molecules at 300 K. We find a systematic 0.7 eV red shift in the absorption gap of this cluster, which we attribute to thermally induced fluctuations in the molecular structure. Surprisingly, we find no observable screening impact of the solvent, in contrast with consistent blue shifts observed for similarly sized organic molecules in polar solvents. The predicted red shift is expected to be significantly smaller for larger Si quantum dots produced experimentally, guaranteeing that their vacuum optical properties are preserved even in aqueous environments.

Density functional theory for efficient ab initio molecular dynamics simulations in solution.

We present a density functional for first-principles molecular dynamics simulations that includes the electrostatic effects of a continuous dielectric medium. It allows for numerical simulations of molecules in solution in a model polar solvent. We propose a smooth dielectric model function to model solvation into water and demonstrate its good numerical properties for total energy calculations and constant energy molecular dynamics.