Best practices for constructing, preparing, and evaluating protein-ligand binding affinity benchmarks [Article v0.1].

Free energy calculations are rapidly becoming indispensable in structure-enabled drug discovery programs. As new methods, force fields, and implementations are developed, assessing their expected accuracy on real-world systems (benchmarking) becomes critical to provide users with an assessment of the accuracy expected when these methods are applied within their domain of applicability, and developers with a way to assess the expected impact of new methodologies. These assessments require construction of a benchmark-a set of well-prepared, high quality systems with corresponding experimental measurements designed to ensure the resulting calculations provide a realistic assessment of expected performance when these methods are deployed within their domains of applicability. To date, the community has not yet adopted a common standardized benchmark, and existing benchmark reports suffer from a myriad of issues, including poor data quality, limited statistical power, and statistically deficient analyses, all of which can conspire to produce benchmarks that are poorly predictive of real-world performance. Here, we address these issues by presenting guidelines for (1) curating experimental data to develop meaningful benchmark sets, (2) preparing benchmark inputs according to best practices to facilitate widespread adoption, and (3) analysis of the resulting predictions to enable statistically meaningful comparisons among methods and force fields. We highlight challenges and open questions that remain to be solved in these areas, as well as recommendations for the collection of new datasets that might optimally serve to measure progress as methods become systematically more reliable. Finally, we provide a curated, versioned, open, standardized benchmark set adherent to these standards (PLBenchmarks) and an open source toolkit for implementing standardized best practices assessments (arsenic) for the community to use as a standardized assessment tool. While our main focus is free energy methods based on molecular simulations, these guidelines should prove useful for assessment of the rapidly growing field of machine learning methods for affinity prediction as well.

Fragment Pose Prediction Using Non-equilibrium Candidate Monte Carlo and Molecular Dynamics Simulations.

Part of early stage drug discovery involves determining how molecules may bind to the target protein. Through understanding where and how molecules bind, chemists can begin to build ideas on how to design improvements to increase binding affinities. In this retrospective study, we compare how computational approaches like docking, molecular dynamics (MD) simulations, and a non-equilibrium candidate Monte Carlo (NCMC)-based method (NCMC + MD) perform in predicting binding modes for a set of 12 fragment-like molecules, which bind to soluble epoxide hydrolase. We evaluate each method's effectiveness in identifying the dominant binding mode and finding additional binding modes (if any). Then, we compare our predicted binding modes to experimentally obtained X-ray crystal structures. We dock each of the 12 small molecules into the apo-protein crystal structure and then run simulations up to 1 µs each. Small and fragment-like molecules likely have smaller energy barriers separating different binding modes by virtue of relatively fewer and weaker interactions relative to drug-like molecules and thus likely undergo more rapid binding mode transitions. We expect, thus, to see more rapid transitions between binding modes in our study. Following this, we build Markov State Models to define our stable ligand binding modes. We investigate if adequate sampling of ligand binding modes and transitions between them can occur at the microsecond timescale using traditional MD or a hybrid NCMC+MD simulation approach. Our findings suggest that even with small fragment-like molecules, we fail to sample all the crystallographic binding modes using microsecond MD simulations, but using NCMC+MD, we have better success in sampling the crystal structure while obtaining the correct populations.

Biomolecular Solvation Structure Revealed by Molecular Dynamics Simulations.

To compare ordered water positions from experiment with those from molecular dynamics (MD) simulations, a number of MD models of water structure in crystalline endoglucanase were calculated. The starting MD model was derived from a joint X-ray and neutron diffraction crystal structure, enabling the use of experimentally assigned protonation states. Simulations were performed in the crystalline state, using a periodic 2 × 2 × 2 supercell with explicit solvent. Water X-ray and neutron scattering density maps were computed from MD trajectories using standard macromolecular crystallography methods. In one set of simulations, harmonic restraints were applied to bias the protein structure toward the crystal structure. For these simulations, the recall of crystallographic waters using strong peaks in the MD water electron density was very good, and there also was substantial visual agreement between the boomerang-like wings of the neutron scattering density and the crystalline water hydrogen positions. An unrestrained simulation also was performed. For this simulation, the recall of crystallographic waters was much lower. For both restrained and unrestrained simulations, the strongest water density peaks were associated with crystallographic waters. The results demonstrate that it is now possible to recover crystallographic water structure using restrained MD simulations but that it is not yet reasonable to expect unrestrained MD simulations to do the same. Further development and generalization of MD water models for force-field development, macromolecular crystallography, and medicinal chemistry applications is now warranted. In particular, the combination of room-temperature crystallography, neutron diffraction, and crystalline MD simulations promises to substantially advance modeling of biomolecular solvation.

Improved ligand geometries in crystallographic refinement using AFITT in PHENIX.

Modern crystal structure refinement programs rely on geometry restraints to overcome the challenge of a low data-to-parameter ratio. While the classical Engh and Huber restraints work well for standard amino-acid residues, the chemical complexity of small-molecule ligands presents a particular challenge. Most current approaches either limit ligand restraints to those that can be readily described in the Crystallographic Information File (CIF) format, thus sacrificing chemical flexibility and energetic accuracy, or they employ protocols that substantially lengthen the refinement time, potentially hindering rapid automated refinement workflows. PHENIX-AFITT refinement uses a full molecular-mechanics force field for user-selected small-molecule ligands during refinement, eliminating the potentially difficult problem of finding or generating high-quality geometry restraints. It is fully integrated with a standard refinement protocol and requires practically no additional steps from the user, making it ideal for high-throughput workflows. PHENIX-AFITT refinements also handle multiple ligands in a single model, alternate conformations and covalently bound ligands. Here, the results of combining AFITT and the PHENIX software suite on a data set of 189 protein-ligand PDB structures are presented. Refinements using PHENIX-AFITT significantly reduce ligand conformational energy and lead to improved geometries without detriment to the fit to the experimental data. For the data presented, PHENIX-AFITT refinements result in more chemically accurate models for small-molecule ligands.

POSIT: Flexible Shape-Guided Docking For Pose Prediction.

We present a new approach to structure-based drug design (POSIT) rigorously built on the simple concept that pose prediction is intimately coupled to the quality and availability of experimental structural data. We demonstrate the feasibility of the approach by performing retrospective analyses on three data sets designed to explore the strengths and weaknesses of POSIT relative to existing methods. We then present results documenting 2.5 years of prospective use of POSIT across a variety of structure-based industrial drug-discovery research projects. We find that POSIT is well-suited to guiding research decision making for structure-based design and, in particular, excels at enabling lead-optimization campaigns. We show that the POSIT framework can drive superior pose-prediction performance and generate results that naturally lend themselves to prospective decision making during lead optimization. We believe the results presented here are (1) the largest prospective validation of a pose prediction method reported to date (71 crystal structures); (2) provide an unprecedented look at the scope of impact of a computational tool; and (3) represent a first-of-its-kind analysis. We hope that this work inspires additional studies that look at the real impact and performance of computational research tools on prospective drug design.

The application of statistical methods to cognate docking: a path forward?

Cognate docking has been used as a test for pose prediction quality in docking engines for decades. In this paper, we report a statistically rigorous analysis of cognate docking performance using tools in the OpenEye docking suite. We address a number of critically important aspects of the cognate docking problem that are often handled poorly: data set quality, methods of comparison of the predicted pose to the experimental pose, and data analysis. The focus of the paper lies in the third problem, extracting maximally predictive knowledge from comparison data. To this end, we present a multistage protocol for data analysis that by combining classical null-hypothesis significance testing with effect size estimation provides crucial information about quantitative differences in performance between methods as well as the probability of finding such differences in future experiments. We suggest that developers of software and users of software have different levels of interest in different parts of this protocol, with users being primarily interested in effect size estimation while developers may be most interested in statistical significance. This protocol is completely general and therefore will provide the basis for method comparisons of many different kinds.

Essential considerations for using protein-ligand structures in drug discovery.

Protein-ligand structures are the core data required for structure-based drug design (SBDD). Understanding the error present in this data is essential to the successful development of SBDD tools. Methods for assessing protein-ligand structure quality and a new set of identification criteria are presented here. When these criteria were applied to a set of 728 structures previously used to validate molecular docking software, only 17% were found to be acceptable. Structures were re-refined to maintain internal consistency in the comparison and assessment of the quality criteria. This process resulted in Iridium, a highly trustworthy protein-ligand structure database to be used for development and validation of structure-based design tools for drug discovery.

Conformer generation with OMEGA: algorithm and validation using high quality structures from the Protein Databank and Cambridge Structural Database.

Here, we present the algorithm and validation for OMEGA, a systematic, knowledge-based conformer generator. The algorithm consists of three phases: assembly of an initial 3D structure from a library of fragments; exhaustive enumeration of all rotatable torsions using values drawn from a knowledge-based list of angles, thereby generating a large set of conformations; and sampling of this set by geometric and energy criteria. Validation of conformer generators like OMEGA has often been undertaken by comparing computed conformer sets to experimental molecular conformations from crystallography, usually from the Protein Databank (PDB). Such an approach is fraught with difficulty due to the systematic problems with small molecule structures in the PDB. Methods are presented to identify a diverse set of small molecule structures from cocomplexes in the PDB that has maximal reliability. A challenging set of 197 high quality, carefully selected ligand structures from well-solved models was obtained using these methods. This set will provide a sound basis for comparison and validation of conformer generators in the future. Validation results from this set are compared to the results using structures of a set of druglike molecules extracted from the Cambridge Structural Database (CSD). OMEGA is found to perform very well in reproducing the crystallographic conformations from both these data sets using two complementary metrics of success.

Identification of 4-(2-(4-amino-1,2,5-oxadiazol-3-yl)-1-ethyl-7-{[(3S)-3-piperidinylmethyl]oxy}-1H-imidazo[4,5-c]pyridin-4-yl)-2-methyl-3-butyn-2-ol (GSK690693), a novel inhibitor of AKT kinase.

Overexpression of AKT has an antiapoptotic effect in many cell types, and expression of dominant negative AKT blocks the ability of a variety of growth factors to promote survival. Therefore, inhibitors of AKT kinase activity might be useful as monotherapy for the treatment of tumors with activated AKT. Herein, we describe our lead optimization studies culminating in the discovery of compound 3g (GSK690693). Compound 3g is a novel ATP competitive, pan-AKT kinase inhibitor with IC 50 values of 2, 13, and 9 nM against AKT1, 2, and 3, respectively. An X-ray cocrystal structure was solved with 3g and the kinase domain of AKT2, confirming that 3g bound in the ATP binding pocket. Compound 3g potently inhibits intracellular AKT activity as measured by the inhibition of the phosphorylation levels of GSK3beta. Intraperitoneal administration of 3g in immunocompromised mice results in the inhibition of GSK3beta phosphorylation and tumor growth in human breast carcinoma (BT474) xenografts.

How to do an evaluation: pitfalls and traps.

The recent literature is replete with papers evaluating computational tools (often those operating on 3D structures) for their performance in a certain set of tasks. Most commonly these papers compare a number of docking tools for their performance in cognate re-docking (pose prediction) and/or virtual screening. Related papers have been published on ligand-based tools: pose prediction by conformer generators and virtual screening using a variety of ligand-based approaches. The reliability of these comparisons is critically affected by a number of factors usually ignored by the authors, including bias in the datasets used in virtual screening, the metrics used to assess performance in virtual screening and pose prediction and errors in crystal structures used.

A critical assessment of docking programs and scoring functions.

Docking is a computational technique that samples conformations of small molecules in protein binding sites; scoring functions are used to assess which of these conformations best complements the protein binding site. An evaluation of 10 docking programs and 37 scoring functions was conducted against eight proteins of seven protein types for three tasks: binding mode prediction, virtual screening for lead identification, and rank-ordering by affinity for lead optimization. All of the docking programs were able to generate ligand conformations similar to crystallographically determined protein/ligand complex structures for at least one of the targets. However, scoring functions were less successful at distinguishing the crystallographic conformation from the set of docked poses. Docking programs identified active compounds from a pharmaceutically relevant pool of decoy compounds; however, no single program performed well for all of the targets. For prediction of compound affinity, none of the docking programs or scoring functions made a useful prediction of ligand binding affinity.

Aminoalkoxybenzyl pyrrolidines as novel human urotensin-II receptor antagonists.

High throughput screening of the corporate compound collection led to the discovery of a novel series of substituted aminoalkoxybenzyl pyrrolidines as human urotensin-II receptor antagonists. The synthesis, initial structure-activity relationships, and optimization of the initial hit that led to the identification of a truncated sub-series, represented by SB-436811 (1a), are described.