Cross-validation pitfalls when selecting and assessing regression and classification models.

BACKGROUND: We address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. A key operational component of the proposed methods is cloud computing which enables routine use of previously infeasible approaches. METHODS: We describe in detail an algorithm for repeated grid-search V-fold cross-validation for parameter tuning in classification and regression, and we define a repeated nested cross-validation algorithm for model assessment. As regards variable selection and parameter tuning we define two algorithms (repeated grid-search cross-validation and double cross-validation), and provide arguments for using the repeated grid-search in the general case. RESULTS: We show results of our algorithms on seven QSAR datasets. The variation of the prediction performance, which is the result of choosing different splits of the dataset in V-fold cross-validation, needs to be taken into account when selecting and assessing classification and regression models. CONCLUSIONS: We demonstrate the importance of repeating cross-validation when selecting an optimal model, as well as the importance of repeating nested cross-validation when assessing a prediction error.

Chemical Descriptors Library (CDL): a generic, open source software library for chemical informatics.

In this article the Chemical Descriptors Library (CDL), a generic, open source software library for chemical informatics is introduced. The library is written using standard-compliant C++ programming language. The CDL provides a generic interface for traversing the structure of a molecular graph and accessing its properties. As a result, the software offers flexibility, reusability, and maintainability. This interface has been used to develop several chemical informatics algorithms, including molecular text format parsers and writers; substructure, pharmacophore, and atom type fingerprints; and both common substructure search and SMARTS search. The algorithms are described and evaluated on 3 data sets comprising 1000, 50000, and 100000 small molecules, respectively. The properties of the algorithms in terms of complexity analysis and processing times are presented and discussed.

Competitive Workflow: novel software architecture for automating drug design.

As industrialization of laboratory processes for drug discovery continues to gather momentum, the bottleneck has moved toward exploitation of this tide of information to enable better quality decisions. The development of information-management systems to automate data and materials management can have a positive impact on productivity, as can increasingly sophisticated computer-aided molecular design approaches. However, as long as key decisions can only be taken by a small number of expert individuals working in a complex social environment, the impact of such innovations will be limited. This review describes Competitive Workflow, a distributed multi-agent system explicitly designed for the automation of decision making, currently the preserve of the expert. The approach builds on workflow architectures that capture best practice in information processing, but aims to extend these to model the tacit knowledge of the expert in the selection of alternative pathways through the workflow. The review also discusses recent developments in related workflow-management systems, particularly for information management and processing services front multiple sources, as well as distributed multi-agent approaches. A specific implementation of Competitive workflow--the Discovery Bus--and its application to meta-quantitative structure-activity relationship analysis is also described.

Integrating in vitro ADMET data through generic physiologically based pharmacokinetic models.

Early estimation of kinetics in man currently relies on extrapolation from experimental data generated in animals. Recent results from the application of a generic physiologically based model, Cloe PK) (Cyprotex), which is parameterised for human and rat physiology, to the estimation of plasma pharmacokinetics, are summarised in this paper. A comparison with predictive methods that involve scaling from in vivo animal data can also be made from recently published data. On average, the divergence of the predicted plasma concentrations from the observed data was 0.47 log units. For the external test set, > 70% of the predicted values of the AUC were within threefold of the observed values. Furthermore, the model was found to match or exceed the performance of three published interspecies scaling methods for estimating clearance, all of which showed a distinct bias towards overprediction. It is concluded that Cloe PK, as a means of integrating readily determined in vitro and/or in silico data, is a powerful, cost-effective tool for estimating exposure and kinetics in drug discovery and risk assessment that should, if widely adopted, lead to major reductions in the need for animal experimentation.

Prediction of in vivo tissue distribution from in vitro data. 2. Influence of albumin diffusion from tissue pieces during an in vitro incubation on estimated tissue-to-unbound plasma partition coefficients (Kpu).

PURPOSE: To determine the extent of albumin diffusion from tissue pieces into medium during in vitro incubations, to develop and assess the utility of mathematical models describing this effect on the estimation of tissue-to-unbound plasma partition coefficients (Kpu) of drug substances and to derive factors to correct for associated errors. METHODS: Twelve separate tissues were obtained from rats sacrificed by cervical dislocation, 48 h after an intravenous dose of 125I-human albumin, and tissue pieces incubated to determine the efflux of albumin into media over 2 to 4 h. A mathematical model was developed to predict and correct for the effect of albumin diffusion on the measured Kpu values of drugs. RESULTS: The model predicted that the effect of albumin diffusion from tissue pieces during in vitro incubation (ranging from 14 to 59% remaining in tissue) on Kpu values was generally minimal, except for compounds that are highly plasma bound and have a low measured Kpu. Under these circumstances, the measured Kpu substantially underestimates the true value. An equation was derived from readily available or measurable parameters to correct for this underestimation. CONCLUSIONS: Albumin diffuses from tissue pieces into protein free media during in vitro incubations until equilibrium is reached, defined by the albumin Kpu. Model predictions indicated that for the majority of compounds albumin diffusion would have a minimal effect on the measured Kpu value and that a correction factor could be calculated to account for any deviation.

Prediction of in vivo tissue distribution from in vitro data. 3. Correlation between in vitro and in vivo tissue distribution of a homologous series of nine 5-n-alkyl-5-ethyl barbituric acids.

PURPOSE: To evaluate the ability to determine accurate in vivo tissue-to-unbound plasma distribution coefficients (Kpue) from in vitro data. METHODS: Fresh pieces of fifteen rat tissues/organs were incubated at 37 degrees C with a homologous series of nine barbiturates covering a wide range of lipophilicity (Log P 0.02 to 4.13). Steady-state in vivo Kpue values were estimated from the tissue and plasma concentrations following simultaneous dosing by constant rate i.v. infusion of all nine barbiturates. Drug concentrations in the tissues and media were determined by HPLC with UV or mass spectrometric detection. RESULTS: The pharmacokinetics of the barbiturate series following constant rate i.v. infusion indicated a range of clearance (0.49 to 30 ml x min(-1) x kg(-1)) and volume of distribution at steady state (0.51 to 1.9 l x kg(-1)) values. Good agreement was observed between the in vitro and in vivo Kpu values, although for the most lipophilic barbiturates the in vitro data underpredicted the in vivo tissue distribution for all tissues. CONCLUSION: The in vitro system for predicting the extent of in vivo tissue distribution works well for compounds of widely differing lipophilicity, although for the most lipophilic drugs it may result in an underprediction of in vivo values.

Progress in simulation modelling for pharmacokinetics.

Simulation models for the prediction of pharmacokinetics in humans and other mammalian species, which are based on the physiology and mechanistic models of absorption, distribution, metabolism and elimination are reviewed. The structure of such models is explained with reference to papers describing the mathematical details and alternative representations of organ flow and distribution. Approaches to the modelling of more complex tissues such as tumours and the liver are also reviewed as well as some specific transport processes such as biliary secretion and methods of ADME property estimation by experimental and in silico models. Specific approaches to the modelling of gastro-intestinal transit are explained as is the extension of the approach to simulating drug-drug interactions following co-administration of more than one drug.