Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View.

BACKGROUND: As more and more researchers are turning to big data for new opportunities of biomedical discoveries, machine learning models, as the backbone of big data analysis, are mentioned more often in biomedical journals. However, owing to the inherent complexity of machine learning methods, they are prone to misuse. Because of the flexibility in specifying machine learning models, the results are often insufficiently reported in research articles, hindering reliable assessment of model validity and consistent interpretation of model outputs. OBJECTIVE: To attain a set of guidelines on the use of machine learning predictive models within clinical settings to make sure the models are correctly applied and sufficiently reported so that true discoveries can be distinguished from random coincidence. METHODS: A multidisciplinary panel of machine learning experts, clinicians, and traditional statisticians were interviewed, using an iterative process in accordance with the Delphi method. RESULTS: The process produced a set of guidelines that consists of (1) a list of reporting items to be included in a research article and (2) a set of practical sequential steps for developing predictive models. CONCLUSIONS: A set of guidelines was generated to enable correct application of machine learning models and consistent reporting of model specifications and results in biomedical research. We believe that such guidelines will accelerate the adoption of big data analysis, particularly with machine learning methods, in the biomedical research community.

A note on octonionic support vector regression.

This note presents an analysis of the octonionic form of the division algebraic support vector regressor (SVR) first introduced by Shilton A detailed derivation of the dual form is given, and three conditions under which it is analogous to the quaternionic case are exhibited. It is shown that, in the general case of an octonionic-valued feature map, the usual "kernel trick" breaks down. The cause of this (and its interpretation) is discussed in some detail, along with potential ways of extending kernel methods to take advantage of the distinct features present in the general case. Finally, the octonionic SVR is applied to an example gait analysis problem, and its performance is compared to that of the least squares SVR, the Clifford SVR, and the multidimensional SVR.

A division algebraic framework for multidimensional support vector regression.

In this paper, division algebras are proposed as an elegant basis upon which to extend support vector regression (SVR) to multidimensional targets. Using this framework, a multitarget SVR called epsilon(Z)-SVR is proposed based on an epsilon-insensitive loss function that is independent of the coordinate system or basis used. This is developed to dual form in a manner that is analogous to the standard epsilon-SVR. The epsilon(H)-SVR is compared and contrasted with the least-square SVR (LS-SVR), the Clifford SVR (C-SVR), and the multidimensional SVR (M-SVR). Three practical applications are considered: namely, 1) approximation of a complex-valued function; 2) chaotic time-series prediction in 3-D; and 3) communication channel equalization. Results show that the epsilon(H)-SVR performs significantly better than the C-SVR, the LS-SVR, and the M-SVR in terms of mean-squared error, outlier sensitivity, and support vector sparsity.

A machine learning approach to k-step look-ahead prediction of gait variables from acceleration data.

This paper investigates the use of machine learning to predict a sensitive gait parameter based on acceleration information from previous gait cycles. We investigate a k-step look-ahead prediction which attempts to predict gait variable values based on acceleration information in the current gait cycle. The variable is the minimum toe clearance which has been demonstrated to be a sensitive falls risk predictor. Toe clearance data was collected under normal walking conditions and 9 features consisting of peak acceleration and their normalized occurrences times were extracted. A standard least squares estimator, a generalized regression neural network (GRNN) and a support vector regressor (SVR) were trained using 60% of the data to estimate the minimum toe clearance and the remaining 40% was used to validate the model. It was found that when the training data contained data from all subjects (inter-subject) the best GRNN model provided a root mean square error (RMSE) of 2.8 mm, the best SVR had RMSE of 2.7 mm while the standard least squares linear regression method obtained 3.3 mm. When the training and test data consisted of different subject examples (inter-subject) data, the linear SVR demonstrated superior generalization capability (RMSE=3.3 mm) compared to other competing models. Validation accuracies up to 5-step look-ahead predictions revealed robust performances for both GRNN and SVR models with no clear degradation in prediction accuracy.

Protein topology classification using two-stage support vector machines.

The determination of the first 3-D model of a protein from its sequence alone is a non-trivial problem. The first 3-D model is the key to the molecular replacement method of solving phase problem in x-ray crystallography. If the sequence identity is more than 30%, homology modelling can be used to determine the correct topology (as defined by CATH) or fold (as defined by SCOP). If the sequence identity is less than 25%, however, the task is very challenging. In this paper we address the topology classification of proteins with sequence identity of less than 25%. The input information to the system is amino acid sequence, the predicted secondary structure and the predicted real value relative solvent accessibility. A two stage support vector machine (SVM) approach is proposed for classifying the sequences to three different structural classes (alpha, beta, alpha+beta) in the first stage and 39 topologies in the second stage. The method is evaluated using a newly curated dataset from CATH with maximum pairwise sequence identity less than 25%. An impressive overall accuracy of 87.44% and 83.15% is reported for class and topology prediction, respectively. In the class prediction stage, a sensitivity of 0.77 and a specificity of 0.91 is obtained. Data file, SVM implementation (SVMHEAVY) and result files can be downloaded from http://www.ee.unimelb.edu.au/ISSNIP/downloads/.

Incremental training of support vector machines.

We propose a new algorithm for the incremental training of support vector machines (SVMs) that is suitable for problems of sequentially arriving data and fast constraint parameter variation. Our method involves using a "warm-start" algorithm for the training of SVMs, which allows us to take advantage of the natural incremental properties of the standard active set approach to linearly constrained optimization problems. Incremental training involves quickly retraining a support vector machine after adding a small number of additional training vectors to the training set of an existing (trained) support vector machine. Similarly, the problem of fast constraint parameter variation involves quickly retraining an existing support vector machine using the same training set but different constraint parameters. In both cases, we demonstrate the computational superiority of incremental training over the usual batch retraining method.

Prediction of cystine connectivity using SVM.

One of the major contributors to protein structures is the formation of disulphide bonds between selected pairs of cysteines at oxidized state. Prediction of such disulphide bridges from sequence is challenging given that the possible combination of cysteine pairs as the number of cysteines increases in a protein. Here, we describe a SVM (support vector machine) model for the prediction of cystine connectivity in a protein sequence with and without a priori knowledge on their bonding state. We make use of a new encoding scheme based on physico-chemical properties and statistical features (probability of occurrence of each amino acid residue in different secondary structure states along with PSI-blast profiles). We evaluate our method in SPX (an extended dataset of SP39 (swiss-prot 39) and SP41 (swiss-prot 41) with known disulphide information from PDB) dataset and compare our results with the recursive neural network model described for the same dataset.