Novel Use of Proteomic Profiles in a Convex-Hull Ensemble Classifier to Predict Gynecological Cancer Patients' Susceptibility to Gastrointestinal Mucositis as Side Effect of Radiation Therapy.

BACKGROUND: Whole-pelvis radiation therapy is common practice in the post-surgical treatment of cervical and endometrial cancer. Gastrointestinal mucositis is an adverse side effect of radiation therapy, and is a primary concern in patient management. We investigate whether proteomic information obtained from blood samples drawn from patients scheduled to receive radiation therapy for gynecological cancers could be used to predict which patients are most susceptible to radiation-induced gastrointestinal mucositis, in order to improve the individualization of radiation therapy. METHODS: We use 132 proteins measured on 17 gynecological cancer patients in a convex-hull-based, selective-voting ensemble classifier to classify each patient into one of two classes: patients who would not (class 1) or would (class 2) develop gastrointestinal mucositis. We employ 20 repetitions of 10-fold cross-validation to measure classification accuracy. RESULTS: We achieved a 95% confidence interval on average prediction accuracy of (0.711, 0.771) using pre-radiation proteomic profiles to predict which patients would experience gastrointestinal mucositis. Pathway analysis of the 12 most prominent proteins indicated that they could be assembled into a single interaction network with direct associations. The function associated with the highest number of these 12 proteins was cell-to-cell signaling and interaction. CONCLUSIONS: Pre-radiation proteomic profiles have the potential to classify cervical/endometrial cancer patients with high accuracy as to their susceptibility to gastrointestinal mucositis following radiation therapy. Further study of the network of 12 identified proteins is warranted with a larger patient sample to confirm that these proteins are predictive of gastrointestinal mucositis in this patient population.

Immune Function and Muscle Adaptations to Resistance exercise in Older Adults: Study Protocol for a Randomized Controlled Trial of a Nutritional Supplement.

BACKGROUND: Immune function may influence the ability of older adults to maintain or improve muscle mass, strength, and function during aging. Thus, nutritional supplementation that supports the immune system could complement resistance exercise as an intervention for age-associated muscle loss. The current study will determine the relationship between immune function and exercise training outcomes for older adults who consume a nutritional supplement or placebo during resistance training and post-training follow-up. The supplement was chosen due to evidence suggesting its ingredients [arginine (Arg), glutamine (Gln), and ß-hydroxy ß-methylbutyrate (HMB)] can improve immune function, promote muscle growth, and counteract muscle loss. METHODS/DESIGN: Veterans (age 60 to 80 yrs, N = 50) of the United States military will participate in a randomized double-blind placebo-controlled trial of consumption of a nutritional supplement or placebo during completion of three study objectives: 1) determine if 2 weeks of supplementation improve immune function measured as the response to vaccination and systemic and cellular responses to acute resistance exercise; 2) determine if supplementation during 36 sessions of resistance training boosts gains in muscle size, strength, and function; and 3) determine if continued supplementation for 26 weeks post-training promotes retention of training-induced gains in muscle size, strength, and function. Analyses of the results for these objectives will determine the relationship between immune function and the training outcomes. Participants will undergo nine blood draws and five muscle (vastus lateralis) biopsies so that the effects of the supplement on immune function and the systemic and cellular responses to exercise can be measured. DISCUSSION: Exercise has known effects on immune function. However, the study will attempt to modulate immune function using a nutritional supplement and determine the effects on training outcomes. The study will also examine post-training benefit retention, an important issue for older adults, usually omitted from exercise studies. The study will potentially advance our understanding of the mechanisms of muscle gain and loss in older adults, but more importantly, a nutritional intervention will be evaluated as a complement to exercise for supporting muscle health during aging. TRIAL REGISTRATION: Clinicaltrials.gov identifier: NCT02261961, registration date 10 June 2014, recruitment active.

A diversity index for model space selection in the estimation of benchmark and infectious doses via model averaging.

In chemical and microbial risk assessments, risk assessors fit dose-response models to high-dose data and extrapolate downward to risk levels in the range of 1-10%. Although multiple dose-response models may be able to fit the data adequately in the experimental range, the estimated effective dose (ED) corresponding to an extremely small risk can be substantially different from model to model. In this respect, model averaging (MA) provides more robustness than a single dose-response model in the point and interval estimation of an ED. In MA, accounting for both data uncertainty and model uncertainty is crucial, but addressing model uncertainty is not achieved simply by increasing the number of models in a model space. A plausible set of models for MA can be characterized by goodness of fit and diversity surrounding the truth. We propose a diversity index (DI) to balance between these two characteristics in model space selection. It addresses a collective property of a model space rather than individual performance of each model. Tuning parameters in the DI control the size of the model space for MA.

Practical advice on calculating confidence intervals for radioprotection effects and reducing animal numbers in radiation countermeasure experiments.

The dose of a substance that causes death in P% of a population is called an LDP, where LD stands for lethal dose. In radiation research, a common LDP of interest is the radiation dose that kills 50% of the population by a specified time, i.e., lethal dose 50 or LD50. When comparing LD50 between two populations, relative potency is the parameter of interest. In radiation research, this is commonly known as the dose reduction factor (DRF). Unfortunately, statistical inference on dose reduction factor is seldom reported. We illustrate how to calculate confidence intervals for dose reduction factor, which may then be used for statistical inference. Further, most dose reduction factor experiments use hundreds, rather than tens of animals. Through better dosing strategies and the use of a recently available sample size formula, we also show how animal numbers may be reduced while maintaining high statistical power. The illustrations center on realistic examples comparing LD50 values between a radiation countermeasure group and a radiation-only control. We also provide easy-to-use spreadsheets for sample size calculations and confidence interval calculations, as well as SAS® and R code for the latter.

Subpopulation-specific confidence designation for more informative biomedical classification.

OBJECTIVE: Although classification algorithms are promising tools to support clinical diagnosis and treatment of disease, the usual implicit assumption underlying these algorithms, that all patients are homogeneous with respect to characteristics of interest, is unsatisfactory. The objective here is to exploit the population heterogeneity reflected by characteristics that may not be apparent and thus not controlled, in order to differentiate levels of classification accuracy between subpopulations and further the goal of tailoring therapies on an individual basis. METHODS AND MATERIALS: A new subpopulation-based confidence approach is developed in the context of a selective voting algorithm defined by an ensemble of convex-hull classifiers. Populations of training samples are divided into three subpopulations that are internally homogeneous, with different levels of predictivity. Two different distance measures are used to cluster training samples into subpopulations and assign test samples to these subpopulations. RESULTS: Validation of the new approach's levels of confidence of classification is carried out using six publicly available datasets. Our approach demonstrates a positive correspondence between the predictivity designations derived from training samples and the classification accuracy of test samples. The average difference between highest- and lowest-confidence accuracies for the six datasets is 17.8%, with a minimum of 11.3% and a maximum of 24.1%. CONCLUSION: The classification accuracy increases as the designated confidence increases.

Multinomial logistic regression ensembles.

This article proposes a method for multiclass classification problems using ensembles of multinomial logistic regression models. A multinomial logit model is used as a base classifier in ensembles from random partitions of predictors. The multinomial logit model can be applied to each mutually exclusive subset of the feature space without variable selection. By combining multiple models the proposed method can handle a huge database without a constraint needed for analyzing high-dimensional data, and the random partition can improve the prediction accuracy by reducing the correlation among base classifiers. The proposed method is implemented using R, and the performance including overall prediction accuracy, sensitivity, and specificity for each category is evaluated on two real data sets and simulation data sets. To investigate the quality of prediction in terms of sensitivity and specificity, the area under the receiver operating characteristic (ROC) curve (AUC) is also examined. The performance of the proposed model is compared to a single multinomial logit model and it shows a substantial improvement in overall prediction accuracy. The proposed method is also compared with other classification methods such as the random forest, support vector machines, and random multinomial logit model.

Quantitative benefit-risk analysis for evaluating drug therapies.

Despite the fact that benefit-risk analysis is a necessary component of the review of new drugs for potential regulatory approval in the presence of known adverse side effects, and of the review of already-approved drugs for possible withdrawal from the market when unanticipated adverse events are discovered, formal quantitative tools for benefit-risk analysis are few. This paper proposes a quantitative method that utilizes receiver operating characteristic (ROC) curves to find an optimal dose of a drug that maximizes the differential between the benefit of the intended effect and the risk of adverse side effects, where costs associated with lack of benefit and risk can be incorporated. The method can be applied separately to subpopulations of different sensitivities and to different adverse events to give a full picture of the trade-offs between the benefit afforded by the drug and the risk it incurs, and potentially to allow the drug to be approved only selectively for specific subpopulations, or at different doses for different subpopulations.

Model uncertainty and model averaging in the estimation of infectious doses for microbial pathogens.

Food-borne infection is caused by intake of foods or beverages contaminated with microbial pathogens. Dose-response modeling is used to estimate exposure levels of pathogens associated with specific risks of infection or illness. When a single dose-response model is used and confidence limits on infectious doses are calculated, only data uncertainty is captured. We propose a method to estimate the lower confidence limit on an infectious dose by including model uncertainty and separating it from data uncertainty. The infectious dose is estimated by a weighted average of effective dose estimates from a set of dose-response models via a Kullback information criterion. The confidence interval for the infectious dose is constructed by the delta method, where data uncertainty is addressed by a bootstrap method. To evaluate the actual coverage probabilities of the lower confidence limit, a Monte Carlo simulation study is conducted under sublinear, linear, and superlinear dose-response shapes that can be commonly found in real data sets. Our model-averaging method achieves coverage close to nominal in almost all cases, thus providing a useful and efficient tool for accurate calculation of lower confidence limits on infectious doses.

Estimating misclassification error: a closer look at cross-validation based methods.

BACKGROUND: To estimate a classifier's error in predicting future observations, bootstrap methods have been proposed as reduced-variation alternatives to traditional cross-validation (CV) methods based on sampling without replacement. Monte Carlo (MC) simulation studies aimed at estimating the true misclassification error conditional on the training set are commonly used to compare CV methods. We conducted an MC simulation study to compare a new method of bootstrap CV (BCV) to k-fold CV for estimating clasification error. FINDINGS: For the low-dimensional conditions simulated, the modest positive bias of k-fold CV contrasted sharply with the substantial negative bias of the new BCV method. This behavior was corroborated using a real-world dataset of prognostic gene-expression profiles in breast cancer patients. Our simulation results demonstrate some extreme characteristics of variance and bias that can occur due to a fault in the design of CV exercises aimed at estimating the true conditional error of a classifier, and that appear not to have been fully appreciated in previous studies. Although CV is a sound practice for estimating a classifier's generalization error, using CV to estimate the fixed misclassification error of a trained classifier conditional on the training set is problematic. While MC simulation of this estimation exercise can correctly represent the average bias of a classifier, it will overstate the between-run variance of the bias. CONCLUSIONS: We recommend k-fold CV over the new BCV method for estimating a classifier's generalization error. The extreme negative bias of BCV is too high a price to pay for its reduced variance.

Assessment of performance of survival prediction models for cancer prognosis.

BACKGROUND: Cancer survival studies are commonly analyzed using survival-time prediction models for cancer prognosis. A number of different performance metrics are used to ascertain the concordance between the predicted risk score of each patient and the actual survival time, but these metrics can sometimes conflict. Alternatively, patients are sometimes divided into two classes according to a survival-time threshold, and binary classifiers are applied to predict each patient's class. Although this approach has several drawbacks, it does provide natural performance metrics such as positive and negative predictive values to enable unambiguous assessments. METHODS: We compare the survival-time prediction and survival-time threshold approaches to analyzing cancer survival studies. We review and compare common performance metrics for the two approaches. We present new randomization tests and cross-validation methods to enable unambiguous statistical inferences for several performance metrics used with the survival-time prediction approach. We consider five survival prediction models consisting of one clinical model, two gene expression models, and two models from combinations of clinical and gene expression models. RESULTS: A public breast cancer dataset was used to compare several performance metrics using five prediction models. 1) For some prediction models, the hazard ratio from fitting a Cox proportional hazards model was significant, but the two-group comparison was insignificant, and vice versa. 2) The randomization test and cross-validation were generally consistent with the p-values obtained from the standard performance metrics. 3) Binary classifiers highly depended on how the risk groups were defined; a slight change of the survival threshold for assignment of classes led to very different prediction results. CONCLUSIONS: 1) Different performance metrics for evaluation of a survival prediction model may give different conclusions in its discriminatory ability. 2) Evaluation using a high-risk versus low-risk group comparison depends on the selected risk-score threshold; a plot of p-values from all possible thresholds can show the sensitivity of the threshold selection. 3) A randomization test of the significance of Somers' rank correlation can be used for further evaluation of performance of a prediction model. 4) The cross-validated power of survival prediction models decreases as the training and test sets become less balanced.

A selective voting convex-hull ensemble procedure for personalized medicine.

Genes work in concert as a system as opposed to independent entities and mediate disease states. There has been considerable interest in understanding variations in molecular signatures between normal and disease states. However, a majority of techniques implicitly assume homogeneity between samples within a given group and use a fixed set of genes in discerning the groups. The proposed study overcomes these caveats by using a selective-voting convex-hull ensemble procedure that accommodates molecular heterogeneity within and between groups. The significance of the study is its potential to selectively retrieve sample-specific ensemble sets and investigate variations in the networks corresponding to the ensemble set across these samples. These characteristics fit well within the scope of personalized medicine and comparative effectiveness research that emphasize on patient-tailored interventions. While the results are demonstrated on colon cancer gene expression profiles the approach as such is generic and can be readily extended to other settings.

Statistical analysis of survival data from radiation countermeasure experiments.

We present an introduction to, and examples of, Cox proportional hazards regression in the context of animal lethality studies of potential radioprotective agents. This established method is seldom used to analyze survival data collected in such studies, but is appropriate in many instances. Presenting a hypothetical radiation study that examines the efficacy of a potential radioprotectant both in the absence and presence of a potential modifier, we detail how to implement and interpret results from a Cox proportional hazards regression analysis used to analyze the survival data, and we provide relevant SAS® code. Cox proportional hazards regression analysis of survival data from lethal radiation experiments (1) considers the whole distribution of survival times rather than simply the commonly used proportions of animals that survived, (2) provides a unified analysis when multiple factors are present, and (3) can increase statistical power by combining information across different levels of a factor. Cox proportional hazards regression should be considered as a potential statistical method in the toolbox of radiation researchers.

Selective voting in convex-hull ensembles improves classification accuracy.

OBJECTIVE: Classification algorithms can be used to predict risks and responses of patients based on genomic and other high-dimensional data. While there is optimism for using these algorithms to improve the treatment of diseases, they have yet to demonstrate sufficient predictive ability for routine clinical practice. They generally classify all patients according to the same criteria, under an implicit assumption of population homogeneity. The objective here is to allow for population heterogeneity, possibly unrecognized, in order to increase classification accuracy and further the goal of tailoring therapies on an individualized basis. METHODS AND MATERIALS: A new selective-voting algorithm is developed in the context of a classifier ensemble of two-dimensional convex hulls of positive and negative training samples. Individual classifiers in the ensemble are allowed to vote on test samples only if those samples are located within or behind pruned convex hulls of training samples that define the classifiers. RESULTS: Validation of the new algorithm's increased accuracy is carried out using two publicly available datasets having cancer as the outcome variable and expression levels of thousands of genes as predictors. Selective voting leads to statistically significant increases in accuracy from 86.0% to 89.8% (p<0.001) and 63.2% to 67.8% (p<0.003) compared to the original algorithm. CONCLUSION: Selective voting by members of convex-hull classifier ensembles significantly increases classification accuracy compared to one-size-fits-all approaches.

Is premarket identification of hepatotoxic drugs and sensitive patients possible based on high-dimensional 'omic data?

Idiosyncratic liver toxicity that may lead to post-marketing removal of approved drugs can potentially be explained by the existence of hidden, sensitive subpopulations that are not large enough to affect premarketing toxicity assessments. We consider whether molecular biomarkers of risk and response can be developed to identify sensitive individuals, using classification methods that allow for population heterogeneity represented by characteristics that may not be readily apparent or controlled. If so, drugs that may be hepatotoxic to only a relatively small subpopulation would not be mislabeled as hepatotoxic to the general population and could be prescribed selectively to achieve a maximum health benefit. We outline a possible strategy for identifying individuals who are highly susceptible to drug-related hepatotoxicity and point out the significant challenges.

Determination of sample sizes for demonstrating efficacy of radiation countermeasures.

In response to the ever increasing threat of radiological and nuclear terrorism, active development of nontoxic new drugs and other countermeasures to protect against and/or mitigate adverse health effects of radiation is ongoing. Although the classical LD(50) study used for many decades as a first step in preclinical toxicity testing of new drugs has been largely replaced by experiments that use fewer animals, the need to evaluate the radioprotective efficacy of new drugs necessitates the conduct of traditional LD(50) comparative studies (FDA, 2002, Federal Register 67, 37988-37998). There is, however, no readily available method to determine the number of animals needed for establishing efficacy in these comparative potency studies. This article presents a sample-size formula based on Student's t for comparative potency testing. It is motivated by the U.S. Food and Drug Administration's (FDA's) requirements for robust efficacy data in the testing of response modifiers in total body irradiation experiments where human studies are not ethical or feasible. Monte Carlo simulation demonstrated the formula's performance for Student's t, Wald, and likelihood ratio tests in both logistic and probit models. Importantly, the results showed clear potential for justifying the use of substantially fewer animals than are customarily used in these studies. The present article may thus initiate a dialogue among researchers who use animals for radioprotection survival studies, institutional animal care and use committees, and drug regulatory bodies to reach a consensus on the number of animals needed to achieve statistically robust results for demonstrating efficacy of radioprotective drugs.

A model-free ensemble method for class prediction with application to biomedical decision making.

OBJECTIVE: A classification algorithm that utilizes two-dimensional convex hulls of training-set samples is presented. METHODS AND MATERIAL: For each pair of predictor variables, separate convex hulls of positive and negative samples in the training set are formed, and these convex hulls are used to classify test points according to a nearest-neighbor criterion. An ensemble of these two-dimensional convex-hull classifiers is formed by trimming the (m)C(2) possible classifiers derived from the m predictors to a set of classifiers comprised of only unique predictor variables. Because only two-dimensional spaces are required to be populated by training-set samples, the "curse of dimensionality" is not an issue. At the same time, the power of ensemble voting is exploited by combining the classifications of the unique two-dimensional classifiers to reach a final classification. RESULTS: The algorithm is illustrated by application to three publicly available biomedical data sets with genomic predictors and is shown to have prediction accuracy that is competitive with a number of published classification procedures. CONCLUSION: Because of its superior performance in terms of sensitivity and negative predictive value compared to its competitors, the convex-hull ensemble classifier demonstrates good potential for medical screening, where often the major emphasis is placed on having reliable negative predictions.

Identifying high-dimensional biomarkers for personalized medicine via variable importance ranking.

We apply robust classification algorithms to high-dimensional genomic data to find biomarkers, by analyzing variable importance, that enable a better diagnosis of disease, an earlier intervention, or a more effective assignment of therapies. The goal is to use variable importance ranking to isolate a set of important genes that can be used to classify life-threatening diseases with respect to prognosis or type to maximize efficacy or minimize toxicity in personalized treatment of such diseases. A ranking method and present several other methods to select a set of important genes to use as genomic biomarkers is proposed, and the performance of the selection procedures in patient classification by cross-validation is evaluated. The various selection algorithms are applied to published high-dimensional genomic data sets using several well-known classification methods. For each data set, a set of genes selected on the basis of variable importance that performed the best in classification is reported. That classification algorithm with the proposed ranking method is shown to be competitive with other selection methods for discovering genomic biomarkers underlying both adverse and efficacious outcomes for improving individualized treatment of patients for life-threatening diseases.

Estimation of lag time between onset of and death from an occult tumor.

A new statistical method for estimating the lag time between onset of and death from an occult tumor is proposed for data without cause-of-death information. In this method, the survival function for time to tumor onset, tumor-specific survival function, and competing risks survival function are estimated using the maximum likelihood estimates of the parameters. The proposed method utilizes the estimated survival functions and statistically imputed fatal tumors to estimate the lag time. This approach is developed for rodent tumorigenicity assays that have at least one interim sacrifice and a terminal sacrifice. If the data contain cause-of-death information given by pathologists and it is believed to be reliable, it may be used for estimating the lag time. The proposed method is illustrated using a real data set. The accuracy of the estimation of lag time is evaluated via a Monte Carlo simulation study. This study shows that the estimated lag time is quite reliable.

Analysis of in vivo mutation data can inform cancer risk assessment.

Under the new U.S. Environmental Protection Agency (EPA) Cancer Risk Assessment Guidelines [U.S. EPA, 2005. Guidelines for Carcinogen Risk Assessment. EPA/630/P-03/001B, March 2005], the quantitative model chosen for cancer risk assessment is based on the mode-of-action (MOA) of the chemical under consideration. In particular, the risk assessment model depends on whether or not the chemical causes tumors through a direct DNA-reactive mechanism. It is assumed that direct DNA-reactive carcinogens initiate carcinogenesis by inducing mutations and have low-dose linear dose-response curves, whereas carcinogens that operate through a nonmutagenic MOA may have nonlinear dose-responses. We are currently evaluating whether the analysis of in vivo gene mutation data can inform the risk assessment process by better defining the MOA for cancer and thus influencing the choice of the low-dose extrapolation model. This assessment includes both a temporal analysis of mutation induction and a dose-response concordance analysis of mutation with tumor incidence. Our analysis of published data on riddelliine in rats and dichloroacetic acid in mice indicates that our approach has merit. We propose an experimental design and graphical analysis that allow for assessing time-to-mutation and dose-response concordance, thereby optimizing the potential for in vivo mutation data to inform the choice of the quantitative model used in cancer risk assessment.

Modeling and assaying dioxin-like biological effects for both dioxin-like and certain non-dioxin-like compounds.

13C NMR data have been correlated to Toxic Equivalency Factors (TEFs) of the 29 PCDDs, PCDFs, or PCBs for which non-zero TEFs have been defined. Such correlations are called quantitative spectrometric data-activity relationship (QSDAR) models. An improved QSDAR model predicted TEFs of 0.037 and 0.004, respectively, for 1,3,7,8-tetrachlorodibenzo-p-dioxin (TCDD) and 1,2,3,4,7-pentachlorodibenzo-p-dioxin (PeCDD), both among the 390 congeners for which zero value TEFs are assumed. A QSDAR model of Relative Potency (REP) values estimated the corresponding values as 0.115 and 0.020. Results from both models indicated that these two congeners may exhibit significant dioxin-like toxicity. If other such congeners have non-zero toxicity, TEF-based risk assessments of some dioxin-, furan-, or PCB-contaminated sites or foods may underestimate toxicity. Both models were extensively cross-validated and the TEF model was externally validated. We confirmed the predictions by an independent in vitro method, a luciferase gene expression assay based on mouse liver cells that found REPs of 0.027 and 0.013, respectively, for 1,3,7,8-TCDD and 1,2,3,4,7-PeCDD. The QSDAR-estimated and gene-expression assayed values agreed. The models were used to predict activity for an applicability domain including 108 non-2,3,7,8 dioxin, furan, or PCB congeners and 2,3,7,8-tetrachlorophenothiazine, a dioxin analog proposed as a drug candidate. This study showed that QSDAR prediction followed by a relatively inexpensive in vitro assay could be used to nominate a few candidates among hundreds for further investigation. It suggested that in silico and in vitro nomination protocols may facilitate practical risk assessment when chemical family members exhibit different degrees of toxicity operating via a common mechanism.