Challenges and Strategies in the Development of Radiation Biodosimetry Tests for Patient Management.

The public health and medical response to a radiological or nuclear incident requires the capability to sort, assess, treat, triage and ultimately discharge, as well as to refer or transport people to their next step in medical care. The Public Health Emergency Medical Countermeasures Enterprise (PHEMCE), directed by the U.S. Department of Health and Human Services (HHS), facilitates a comprehensive, multi-agency effort to develop and deploy radiation biodosimetry tests. Within HHS, discovery and development of biodosimetry tests includes the National Institute of Allergy and Infectious Diseases (NIAID) National Institutes of Health (NIH), the Office of the Assistant Secretary of Preparedness and Response (ASPR), Biomedical Advanced Research and Development Authority (BARDA), and the Food and Drug Administration (FDA) as primary partners in this endeavor. The study of radiation biodosimetry has advanced significantly, with expansion into the fields of cytogenetics, genomics, proteomics, metabolomics, lipidomics and transcriptomics. In addition, expansion of traditional cytogenetic assessment methods using automated platforms, and development of laboratory surge capacity networks have helped to advance biodefense preparedness. This article describes various programs and coordinating efforts between NIAID, BARDA and FDA in the development of radiation biodosimetry approaches to respond to radiological and nuclear threats.

Concordance Measures and Time-Dependent ROC Methods.

The receiver operating characteristic (ROC) curve displays sensitivity versus 1-specificity over a set of thresholds. The area under the ROC curve (AUC) is a global scalar summary of this curve. In the context of time-dependent ROC methods, we are interested in global scalar measures that summarize sequences of time-dependent AUCs over time. The concordance probability is a candidate for such purposes. The concordance probability can provide a global assessment of the discrimination ability of a test for an event that occurs at random times and may be right censored. If the test adequately differentiates between subjects who survive longer times and those who survive shorter times, this will assist clinical decisions. In this context the concordance probability may support assessment of precision medicine tools based on prognostic biomarkers models for overall survival. Definitions of time-dependent sensitivity and specificity are reviewed. Some connections between such definitions and concordance measures are also reviewed and we establish new connections via new measures of global concordance. We explore the relationship between such measures and their corresponding time-dependent AUC. To illustrate these concepts, an application in the context of Alzheimer's disease is presented.

Generalized linear mixed models for multi-reader multi-case studies of diagnostic tests.

Diagnostic tests are often compared in multi-reader multi-case (MRMC) studies in which a number of cases (subjects with or without the disease in question) are examined by several readers using all tests to be compared. One of the commonly used methods for analyzing MRMC data is the Obuchowski-Rockette (OR) method, which assumes that the true area under the receiver operating characteristic curve (AUC) for each combination of reader and test follows a linear mixed model with fixed effects for test and random effects for reader and the reader-test interaction. This article proposes generalized linear mixed models which generalize the OR model by incorporating a range-appropriate link function that constrains the true AUCs to the unit interval. The proposed models can be estimated by maximizing a pseudo-likelihood based on the approximate normality of AUC estimates. A Monte Carlo expectation-maximization algorithm can be used to maximize the pseudo-likelihood, and a non-parametric bootstrap procedure can be used for inference. The proposed method is evaluated in a simulation study and applied to an MRMC study of breast cancer detection.

Comparing diagnostic tests on benefit-risk.

Comparing diagnostic tests on accuracy alone can be inconclusive. For example, a test may have better sensitivity than another test yet worse specificity. Comparing tests on benefit risk may be more conclusive because clinical consequences of diagnostic error are considered. For benefit-risk evaluation, we propose diagnostic yield, the expected distribution of subjects with true positive, false positive, true negative, and false negative test results in a hypothetical population. We construct a table of diagnostic yield that includes the number of false positive subjects experiencing adverse consequences from unnecessary work-up. We then develop a decision theory for evaluating tests. The theory provides additional interpretation to quantities in the diagnostic yield table. It also indicates that the expected utility of a test relative to a perfect test is a weighted accuracy measure, the average of sensitivity and specificity weighted for prevalence and relative importance of false positive and false negative testing errors, also interpretable as the cost-benefit ratio of treating non-diseased and diseased subjects. We propose plots of diagnostic yield, weighted accuracy, and relative net benefit of tests as functions of prevalence or cost-benefit ratio. Concepts are illustrated with hypothetical screening tests for colorectal cancer with test positive subjects being referred to colonoscopy.

Benefit-risk Evaluation for Diagnostics: A Framework (BED-FRAME).

The medical community needs systematic and pragmatic approaches for evaluating the benefit-risk trade-offs of diagnostics that assist in medical decision making. Benefit-Risk Evaluation of Diagnostics: A Framework (BED-FRAME) is a strategy for pragmatic evaluation of diagnostics designed to supplement traditional approaches. BED-FRAME evaluates diagnostic yield and addresses 2 key issues: (1) that diagnostic yield depends on prevalence, and (2) that different diagnostic errors carry different clinical consequences. As such, evaluating and comparing diagnostics depends on prevalence and the relative importance of potential errors. BED-FRAME provides a tool for communicating the expected clinical impact of diagnostic application and the expected trade-offs of diagnostic alternatives. BED-FRAME is a useful fundamental supplement to the standard analysis of diagnostic studies that will aid in clinical decision making.

Application of futility analysis to refine jitter recordings in myasthenia gravis.

INTRODUCTION: The current practice of single-fiber electromyography (SFEMG) requires that 20 fiber pairs with normal jitter be collected to exclude myasthenia gravis (MG). We applied principles of futility analysis from clinical trials in an attempt to reduce that requirement. METHODS: We utilized conditional power futility analysis to assess the probability of an abnormal 20-pair SFEMG based on ongoing analysis of jitter as each pair is collected. Rules for early test termination in the presence of 0, 1, or 2 abnormal pairs were identified. These rules were then applied to previously collected SFEMG data. RESULTS: SFEMG could be stopped at just 12 pairs if all are normal and at 17 pairs if 1 is abnormal. The rules successfully determined when SFEMG could be stopped in 104 of 106 (98%) studies originally reported to be normal. CONCLUSIONS: If the first 12 SFEMG pairs have normal jitter, the study can be terminated and interpreted as normal.

The SIST-M: predictive validity of a brief structured clinical dementia rating interview.

BACKGROUND: We have previously established the reliability and cross-sectional validity of the SIST-M (Structured Interview and Scoring Tool-Massachusetts Alzheimer's Disease Research Center), a shortened version of an instrument shown to predict progression to Alzheimer disease (AD), even among persons with very mild cognitive impairment (vMCI). OBJECTIVE: To test the predictive validity of the SIST-M. METHODS: Participants were 342 community-dwelling, nondemented older adults in a longitudinal study. Baseline Clinical Dementia Rating (CDR) ratings were determined by either (1) clinician interviews or (2) a previously developed computer algorithm based on 60 questions (of a possible 131) extracted from clinician interviews. We developed age+sex+education-adjusted Cox proportional hazards models using CDR-sum-of-boxes (CDR-SB) as the predictor, where CDR-SB was determined by either a clinician interview or an algorithm; models were run for the full sample (n = 342) and among those jointly classified as vMCI using clinician-based and algorithm-based CDR ratings (n = 156). We directly compared predictive accuracy using time-dependent receiver operating characteristic (ROC) curves. RESULTS: AD hazard ratios (HRs) were similar for clinician-based and algorithm-based CDR-SB: for a 1-point increment in CDR-SB, the respective HRs [95% confidence interval (CI)] were 3.1 (2.5, 3.9) and 2.8 (2.2, 3.5); among those with vMCI, the respective HRs (95% CI) were 2.2 (1.6, 3.2) and 2.1 (1.5, 3.0). Similarly high predictive accuracy was achieved: the concordance probability (weighted average of the area-under-the-ROC curves) over follow-up was 0.78 versus 0.76 using clinician-based versus algorithm-based CDR-SB. CONCLUSION: CDR scores based on items from this shortened interview had high predictive ability for AD-comparable to that using a lengthy clinical interview.