Critical Assessment of Conformal Prediction Methods Applied in Binary Classification Settings.

In recent years there has been an increase in the number of scientific papers that suggest using conformal predictions in drug discovery. We consider that some versions of conformal predictions applied in binary settings are embroiled in pitfalls, not obvious at first sight, and that it is important to inform the scientific community about them. In the paper we first introduce the general theory of conformal predictions and follow with an explanation of the version currently dominant in drug discovery research today. Finally, we provide cases for their critical assessment in binary classification settings.

Missed opportunities in large scale comparison of QSAR and conformal prediction methods and their applications in drug discovery.

Recently Bosc et al. (J Cheminform 11(1): 4, 2019), published an article describing a case study that directly compares conformal predictions with traditional QSAR methods for large-scale predictions of target-ligand binding. We consider this study to be very important. Unfortunately, we have found several issues in the authors' approach as well as in the presentation of their findings.

No association between donor telomere length and outcomes after allogeneic unrelated hematopoietic cell transplant in patients with acute leukemia.

Recent studies suggest improved survival in patients with severe aplastic anemia receiving hematopoietic cell transplant (HCT) from unrelated donors with longer telomeres. Here, we tested whether this effect is generalizable to patients with acute leukemia. From the Center for International Blood and Marrow Transplant Research (CIBMTR®) database, we identified 1097 patients who received 8/8 HLA-matched unrelated HCT for acute myeloid leukemia (AML) or acute lymphocytic leukemia (ALL) between 2004 and 2012 with myeloablative conditioning, and had pre-HCT blood sample from the donor in CIBMTR repository. The median age at HCT for recipients was 40 years (range ≤1-68), and 32 years for donors (range = 18-61). We used qPCR for relative telomere length (RTL) measurement, and Cox proportional hazard models for statistical analyses. In a discovery cohort of 300 patients, longer donor RTL (>25th percentile) was associated with reduced risks of relapse (HR = 0.62, p = 0.05) and acute graft-versus-host disease II-IV (HR = 0.68, p = 0.05), and possibly with a higher probability of neutrophil engraftment (HR = 1.3, p = 0.06). However, these results did not replicate in two validation cohorts of 297 and 488 recipients. There was one exception; a higher probability of neutrophil engraftment was observed in one validation cohort (HR = 1.24, p = 0.05). In a combined analysis of the three cohorts, no statistically significant associations (all p > 0.1) were found between donor RTL and any outcomes.

Cross-validation pitfalls when selecting and assessing regression and classification models.

BACKGROUND: We address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. A key operational component of the proposed methods is cloud computing which enables routine use of previously infeasible approaches. METHODS: We describe in detail an algorithm for repeated grid-search V-fold cross-validation for parameter tuning in classification and regression, and we define a repeated nested cross-validation algorithm for model assessment. As regards variable selection and parameter tuning we define two algorithms (repeated grid-search cross-validation and double cross-validation), and provide arguments for using the repeated grid-search in the general case. RESULTS: We show results of our algorithms on seven QSAR datasets. The variation of the prediction performance, which is the result of choosing different splits of the dataset in V-fold cross-validation, needs to be taken into account when selecting and assessing classification and regression models. CONCLUSIONS: We demonstrate the importance of repeating cross-validation when selecting an optimal model, as well as the importance of repeating nested cross-validation when assessing a prediction error.

Hemodynamic flow modeling through an abdominal aorta aneurysm using data mining tools.

Geometrical changes of blood vessels, called aneurysm, occur often in humans with possible catastrophic outcome. Then, the blood flow is enormously affected, as well as the blood hemodynamic interaction forces acting on the arterial wall. These forces are the cause of the wall rupture. A mechanical quantity characteristic for the blood-wall interaction is the wall shear stress, which also has direct physiological effects on the endothelial cell behavior. Therefore, it is very important to have an insight into the blood flow and shear stress distribution when an aneurysm is developed in order to help correlating the mechanical conditions with the pathogenesis of pathological changes on the blood vessels. This insight can further help in improving the prevention of cardiovascular diseases evolution. Computational fluid dynamics (CFD) has been used in general as a tool to generate results for the mechanical conditions within blood vessels with and without aneurysms. However, aneurysms are very patient specific and reliable results from CFD analyses can be obtained by a cumbersome and time-consuming process of the computational model generation followed by huge computations. In order to make the CFD analyses efficient and suitable for future everyday clinical practice, we have here employed data mining (DM) techniques. The focus was to combine the CFD and DM methods for the estimation of the wall shear stresses in an abdominal aorta aneurysm (AAA) underprescribed geometrical changes. Additionally, computing on the grid infrastructure was performed to improve efficiency, since thousands of CFD runs were needed for creating machine learning data. We used several DM techniques and found that our DM models provide good prediction of the shear stress at the AAA in comparison with full CFD model results on real patient data.

Competitive Workflow: novel software architecture for automating drug design.

As industrialization of laboratory processes for drug discovery continues to gather momentum, the bottleneck has moved toward exploitation of this tide of information to enable better quality decisions. The development of information-management systems to automate data and materials management can have a positive impact on productivity, as can increasingly sophisticated computer-aided molecular design approaches. However, as long as key decisions can only be taken by a small number of expert individuals working in a complex social environment, the impact of such innovations will be limited. This review describes Competitive Workflow, a distributed multi-agent system explicitly designed for the automation of decision making, currently the preserve of the expert. The approach builds on workflow architectures that capture best practice in information processing, but aims to extend these to model the tacit knowledge of the expert in the selection of alternative pathways through the workflow. The review also discusses recent developments in related workflow-management systems, particularly for information management and processing services front multiple sources, as well as distributed multi-agent approaches. A specific implementation of Competitive workflow--the Discovery Bus--and its application to meta-quantitative structure-activity relationship analysis is also described.