Use of ROC curve analysis for prediction gives fallacious results: Use predictivity-based indices.

ABSTRACT: The area under the ROC curve is frequently used for assessing the predictive efficacy of a model, and the Youden index is commonly used to provide the optimal cut-off. Both are misleading tools for predictions. A ROC curve is drawn for the sensitivity of a quantitative test against its (1 - specificity) at different values of the test. Both sensitivity and specificity are retrospective in nature as these are indicators of correct classification of already known conditions. They are not indicators of future events and are not valid for predictions. Predictivity intimately depends on the prevalence which may be ignored by sensitivity and specificity. We explain this fallacy in detail and illustrate with several examples that the actual predictivity could differ greatly from the ROC curve-based predictivity reported by many authors. The predictive efficacy of a test or a model is best assessed by the percentage correctly predicted in a prospective framework. We propose predictivity-based ROC curves as tools for providing predictivities at varying prevalence in different populations. For optimal cut-off for prediction, in place of the Youden index, we propose a P-index where the sum of positive and negative predictivities is maximum after subtracting 1. To conclude, for correctly assessing adequacy of a prediction models, predictivity-based ROC curves should be used instead of the usual sensitivity-specificity-based ROC curves and the P-index should replace the Youden index.

The importance of small samples in medical research.

Almost all bio-statisticians and medical researchers believe that a large sample is always helpful in providing more reliable results. Whereas this is true for some specific cases, a large sample may not be helpful in more situations than we contemplate because of the higher possibility of errors and reduced validity. Many medical breakthroughs have occurred with self-experimentation and single experiments. Studies, particularly analytical studies, may provide more truthful results with a small sample because intensive efforts can be made to control all the confounders, wherever they operate, and sophisticated equipment can be used to obtain more accurate data. A large sample may be required only for the studies with highly variable outcomes, where an estimate of the effect size with high precision is required, or when the effect size to be detected is small. This communication underscores the importance of small samples in reaching a valid conclusion in certain situations and describes the situations where a large sample is not only unnecessary but may even compromise the validity by not being able to exercise full care in the assessments. What sample size is small depends on the context.

Aleatory and epistemic uncertainties can completely derail medical research results.

Aleatory uncertainties are generated by intrinsic factors such as studying a sample rather than the whole population and the source of epistemic uncertainties is extraneous such as limitations of knowledge. These uncertainties inflict all the findings in empirical medical research, but they are rarely appreciated. This article highlights these uncertainties and shows with the help of an example how apparently valid and reliable findings can completely derail due to these uncertainties. We conclude that aleatory and epistemic uncertainties should get due consideration while drawing conclusions and before the results are put into practice. Methods to reduce their impact on results are also presented.

Statistical medicine: An emerging medical specialty.

A large number of statistical tools are now used for medical decision in the core activities of diagnosis, treatment and prognosis. These tools provide undeniable help in improving medical outcomes. Prominent among them are uncertainty measurement by probability, medical indicators and indexes, reference ranges, and scoring systems. In addition are tools such as odds ratio, sensitivity, specificity and predictivities, area under the ROC curve, likelihood ratios, and cost-benefit analysis that are commonly applied in medical research but have implications for day-to-day clinical activities. These tools have so completely integrated into medical practice that statistical medicine by itself can stand alone as a medical specialty. Time has arrived to recognize statistical medicine as a medical specialty.

Smokeless tobacco use: a meta-analysis of risk and attributable mortality estimates for India.

BACKGROUND: Use of smokeless tobacco (SLT) is widely prevalent in India and Indian subcontinent. Cohort and case-control studies in India and elsewhere report excess mortality due to its use. OBJECTIVE: The aim was to estimate the SLT use-attributable deaths in males and females, aged 35 years and older, in India. MATERIALS AND METHODS: Prevalence of SLT use in persons aged 35 years and older was obtained from the Global Adult Tobacco Survey in India and population size and deaths in the relevant age-sex groups were obtained from UN estimates (2010 revision) for 2008. A meta-relative risk (RR) based population attributable fraction was used to estimate attributable deaths in persons aged 35 years and older. A random effects model was used in the meta-analysis on all-cause mortality from SLT use in India including four cohort and one case-control study. The studies included in the meta-analysis were adjusted for smoking, age and education. RESULTS: The prevalence of SLT use in India was 25.2% for men and 24.5% for women aged 35 years and older. RRs for females and males were 1.34 (1.27-1.42) and 1.17 (1.05-1.42), respectively. The number of deaths attributable to SLT use in India is estimated to be 368127 (217,076 women and 151,051 men), with nearly three-fifth (60%) of these deaths occurring among women. CONCLUSION: SLT use caused over 350,000 deaths in India in 2010, and nearly three-fifth of SLT use-attributable deaths were among women in India. This calls for targeted public health intervention focusing on SLT products especially among women.

Reporting quality of multivariable logistic regression in selected Indian medical journals.

BACKGROUND: Use of multivariable logistic regression (MLR) modeling has steeply increased in the medical literature over the past few years. Testing of model assumptions and adequate reporting of MLR allow the reader to interpret results more accurately. AIMS: To review the fulfillment of assumptions and reporting quality of MLR in selected Indian medical journals using established criteria. SETTING AND DESIGN: Analysis of published literature. MATERIALS AND METHODS: Medknow.com publishes 68 Indian medical journals with open access. Eight of these journals had at least five articles using MLR between the years 1994 to 2008. Articles from each of these journals were evaluated according to the previously established 10-point quality criteria for reporting and to test the MLR model assumptions. STATISTICAL ANALYSIS: SPSS 17 software and non-parametric test (Kruskal-Wallis H, Mann Whitney U, Spearman Correlation). RESULTS: One hundred and nine articles were finally found using MLR for analyzing the data in the selected eight journals. The number of such articles gradually increased after year 2003, but quality score remained almost similar over time. P value, odds ratio, and 95% confidence interval for coefficients in MLR was reported in 75.2% and sufficient cases (>10) per covariate of limiting sample size were reported in the 58.7% of the articles. No article reported the test for conformity of linear gradient for continuous covariates. Total score was not significantly different across the journals. However, involvement of statistician or epidemiologist as a co-author improved the average quality score significantly (P=0.014). CONCLUSIONS: Reporting of MLR in many Indian journals is incomplete. Only one article managed to score 8 out of 10 among 109 articles under review. All others scored less. Appropriate guidelines in instructions to authors, and pre-publication review of articles using MLR by a qualified statistician may improve quality of reporting.

The methods of survival analysis for clinicians.

The methods of survival analysis are required to analyze duration data but their use is restricted possibly due to lack of awareness and the intricacies involved. We explain common methods of survival analysis, namely, life table, Kaplan Meier, log rank and Cox model, in a simple and friendly language so that the medical fraternity can use them with confidence where applicable.

A simple nomogram for sample size for estimating sensitivity and specificity of medical tests.

Sensitivity and specificity measure inherent validity of a diagnostic test against a gold standard. Researchers develop new diagnostic methods to reduce the cost, risk, invasiveness, and time. Adequate sample size is a must to precisely estimate the validity of a diagnostic test. In practice, researchers generally decide about the sample size arbitrarily either at their convenience, or from the previous literature. We have devised a simple nomogram that yields statistically valid sample size for anticipated sensitivity or anticipated specificity. MS Excel version 2007 was used to derive the values required to plot the nomogram using varying absolute precision, known prevalence of disease, and 95% confidence level using the formula already available in the literature. The nomogram plot was obtained by suitably arranging the lines and distances to conform to this formula. This nomogram could be easily used to determine the sample size for estimating the sensitivity or specificity of a diagnostic test with required precision and 95% confidence level. Sample size at 90% and 99% confidence level, respectively, can also be obtained by just multiplying 0.70 and 1.75 with the number obtained for the 95% confidence level. A nomogram instantly provides the required number of subjects by just moving the ruler and can be repeatedly used without redoing the calculations. This can also be applied for reverse calculations. This nomogram is not applicable for testing of the hypothesis set-up and is applicable only when both diagnostic test and gold standard results have a dichotomous category.

Elements of medical research.

Most medical research is empirical based on evidence rather than hunches or preferences. It follows a series of specific steps. There are no short cuts. Collection of evidence and its analysis should follow a carefully drawn protocol. Most of the modern medical research requires biostatistical tools to reach to a valid and reliable conclusion. Researcher must have an adequate knowledge and skill to be really effective. The endeavours should be consistent with the accepted medical and research ethics. Medical research can provide immense satisfaction when conducted on scientific lines, and can be occasionally frustrating when years of efforts fail to produce expected results. This article focuses on aspects that can increase the credibility of research. It is addressed to all interested in medical research, and seeking answers to questions such as what actually is research, what are its types, what specific steps should be followed, what a research protocol should contain, and what makes research credible etc.