Best practice considerations on the assessment of robotic assisted surgical systems: results from an international consensus expert panel.

BACKGROUND: Health technology assessments (HTAs) of robotic assisted surgery (RAS) face several challenges in assessing the value of robotic surgical platforms. As a result of using different assessment methods, previous HTAs have reached different conclusions when evaluating RAS. While the number of available systems and surgical procedures is rapidly growing, existing frameworks for assessing MedTech provide a starting point, but specific considerations are needed for HTAs of RAS to ensure consistent results. This work aimed to discuss different approaches and produce guidance on evaluating RAS. METHODS: A consensus conference research methodology was adopted. A panel of 14 experts was assembled with international experience and representing relevant stakeholders: clinicians, health economists, HTA practitioners, policy makers, and industry. A review of previous HTAs was performed and seven key themes were extracted from the literature for consideration. Over five meetings, the panel discussed the key themes and formulated consensus statements. RESULTS: A total of ninety-eight previous HTAs were identified from twenty-five total countries. The seven key themes were evidence inclusion and exclusion, patient- and clinician-reported outcomes, the learning curve, allocation of costs, appropriate time horizons, economic analysis methods, and robotic ecosystem/wider benefits. CONCLUSIONS: Robotic surgical platforms are tools, not therapies. Their value varies according to context and should be considered across therapeutic areas and stakeholders. The principles set out in this paper should help HTA bodies at all levels to evaluate RAS. This work may serve as a case study for rapidly developing areas in MedTech that require particular consideration for HTAs.

A novel immunogen to modulate cytokine production and promote immune system reconstitution in HIV-AIDS.

This 'proof of concept' study was implemented in anticipation of identifying and testing a novel antigen of human origin as a potential immunogen in a paradigm that emphasizes immunomodulation and immune system reconstitution as requisites to the development of an effective human immunodeficiency virus (HIV)-acquired immune deficiency syndrome vaccine. Fifteen HIV-infected, highly active antiretroviral therapy (HAART) naive, otherwise healthy male seropositive patients were stratified by [CD4+] into 3 groups of 5 patients: group 1 >500/mm; group 2 > 250/mm but <500/mm; and group 3 < 250/mm. Five healthy male subjects were used as controls. Replicate peripheral blood mononuclear cell (PBMC) [H]thymidine uptake phytohemaglutinin-stimulated proliferation studies, and serum cytokine assays were carried out in the presence or absence of Kveim antigen at dilutions ranging from 0.001 to 100 µg/mL. Serum cytokines [interleukin-2 (IL-2), IL-4, IL-6, interferon gamma, and tumor necrosis factor alpha] were assayed using standardized methodology. Nonparametric statistical analyses and linear regression analysis were used to test for statistical significance and strength of associations. PBMCs harvested from HIV-infected patients and incubated, ex vivo, demonstrated reproducible, antigen concentration-dependent changes in cytokine production over a range of antigen concentrations (0.001-100 µg/mL) in contrast to antigen-naive PBMCs and controls. Significant correlations were demonstrated between antigen concentration and the amount of cytokines secreted. The magnitude of the cytokine response and the patterns of cytokine secretion were HIV group-specific and could be used to identify and distinguish between the 3 groups of HIV-infected subjects. A shift toward the production of type 1-like (Th1) cytokines characteristically seen in systemic sarcoidosis and associated with effective HAART was seen when patterns of cytokine secretion were compared between antigen exposed and antigen-naive PBMCs. PBMCs harvested from seropositive HIV-infected patients and exposed to the Kveim antigen have the following properties: (1) They demonstrate proliferation and exhibit an antigen concentration-dependent secretion of cytokines. The magnitude of the cytokine response can be used to identify and distinguish between groups of seropositive patients stratified by [CD4+]. (2) These PBMCs secrete cytokines in patterns suggestive of a shift to a type 1-like (Th1) response characteristic of HAART and sarcoidosis as opposed to the type 2-like (Th2) cytokine profile characteristic of HIV-acquired immune deficiency syndrome.

Statistical extensions of Ponterotto and Ruckdeschel's (2007) reliability matrix for estimating the adequacy of internal consistency coefficients.

The present article extends work on Ponterotto and Ruckdeschel's Reliability Matrix for estimating the adequacy of internal consistency measures. Specifically, it uses statistical tests to determine whether a calculated coefficient alpha is equal to or greater than the hypothesized population coefficient alpha identified in the Reliability Matrix. The Feldt, Woodruff, and Salih (1987) confidence interval test and Bonett's (2002) approximate z-test and N formula are applied. Select guidelines on reliability assessment and conducting statistical tests to determine adequacy of coefficient alpha are presented.

A comprehensive approach to the interpretation of difference scores.

A comprehensive approach to the interpretation of difference scores is presented. Formulas for the test of statistical significance between two test scores, computed by a confidence interval, and for the calculation of the probabilities for the power of the statistical test, underinterpretation, overinterpretation, and misinterpretation are provided. Definitions and examples of their use in score interpretation are provided.

Differences scores: regression-based reliable difference and the regression-based confidence interval.

Over 50 years ago Payne and Jones (1957) developed what has been labeled the traditional reliable difference formula that continues to be useful as a significance test for the difference between two test scores. The traditional reliable difference is based on the standard error of measurement (SEM) and has been updated to a confidence interval approach. As an alternative to the traditional reliable difference, this article presents the regression-based reliable difference that is based on the standard error of estimate (SEE) and estimated true scores. This new approach should be attractive to clinicians preferring the idea of scores regressing toward the mean. The new approach is also presented in confidence interval form with an interpretation that can be viewed as a statement of all hypotheses that are tenable and consistent with the observed data and has the backing of several authorities. Two well-known conceptualizations for true score confidence intervals are the traditional and regression-based. Now clinicians favoring the regression-based conceptualization are not restricted to the use of traditional model when testing score differences using confidence intervals.

Statistical approaches to achieving sufficiently high test score reliabilities for research purposes.

The author provides statistical approaches to aid investigators in assuring that sufficiently high test score reliabilities are achieved for specific research purposes. The statistical approaches use tests of statistical significance between the obtained reliability and lowest population reliability that an investigator will tolerate. The statistical approaches work for coefficient alpha and related coefficients and for alternate-forms, split-half (2-part alpha), and retest reliabilities. The author shows that, in some instances, a formula can help to estimate the sample size necessary for the statistical test.

Statistical significance and reliability of single-subtest and cluster deviation scores.

Suppose one has a battery of K subtests and a composite for the battery is defined as the mean of the K standardized subtest scores. An individual's single-subtest deviation score is the difference between the individual's score on any single subtest and his composite score. A cluster deviation score is the difference between an examinee's average for a small set (cluster) of subtests and his composite. Formulas are given for the test of statistical significance of the individual's subtest or cluster deviation score and the internal consistency reliability of such deviation scores.

A practical use for the KR-21 reliability coefficient.

KR-21 provides a lower limit for the computed value of KR-20. KR-20 is equivalent to coefficient alpha when a test is composed of dichotomous items scored 0 or 1. Therefore, KR-21 coefficients, computed from simple summary statistics, can be used in cases in which journal authors do not provide the test score reliability. Use of KR-21 in these cases will provide the reader with a lower limit for the value of KR-20.

Score reliability for tests constructed for African-American populations.

Test score reliabilities and sample sizes (N) used to establish the reliabilities are described for a variety of tests constructed for African-American populations. The sample size was 341. The average internal consistency reliability was .74 (SD = .16) with a median value of .77. The median N was 131. The mean internal consistency reliability and median N for tests intended for assessment of individuals were only .72 and 96, respectively.

Psychometric properties of the Folstein Mini-Mental State Examination.

Criterion-referenced (Livingston) and norm-referenced (Gilmer-Feldt) techniques were used to measure the internal consistency reliability of Folstein's Mini-Mental State Examination (MMSE) on a large sample (N = 418) of elderly medical patients. Two administration and scoring variants of the MMSE Attention and Calculation section (Serial 7s only and WORLD only) were investigated. Livingston reliability coefficients (rs) were calculated for a wide range of cutoff scores. As necessary for the calculation of the Gilmer-Feldt r, a factor analysis showed that the MMSE measures three cognitive domains. Livingston's r for the most widely used MMSE cutoff score of 24 was .803 for Serial 7s and .795 for WORLD. The Gilmer-Feldt internal consistency reliability coefficient was .764 for Serial 7s and .747 for WORLD. Item analysis showed that nearly all of the MMSE items were good discriminators, but 12 were too easy. True score confidence intervals should be applied when interpreting MMSE test scores.

Psychometric properties of the benton visual form discrimination test.

Coefficient alpha and an item analysis were calculated for the 16-item Benton Visual Form Discrimination Test (VFDT) using a heterogeneous sample (N = 293) of mostly elderly medical patients who were suspected of having cognitive impairment. The total score reliability was .74. An item analysis found that 15 of the items were within established criteria for item difficulty, however, 5 items were found to be poor discriminators. Through the use of confidence intervals around observed scores, it was shown that the current classification criterion for the VFDT demands a higher reliability coefficient than what was found. Also, evidence for the test's insufficient level of difficulty is presented. It is difficult to recommend this test for clinical use.

Validity and reliability of the Miller Forensic Assessment of Symptoms Test (M-FAST): comment on Veazey, et al. (2005).

Confidence intervals are provided for the validity coefficients calculated by Veazey, et al. for the M-FAST. Two coefficients alpha are also presented along with suggestions for different approaches to calculating the M-FAST internal consistency reliability.

A simple computational alternative to analysis of variance formulas for estimating the k-judge reliability.

Formulas requiring the computation of only three standard deviations are presented for computing the interjudge reliability coefficient for any number of judges. These formulas yield coefficients identical to those obtained from a one-way repeated-measures analysis of variance. Even researchers with small handheld calculators can use this simple approach.

Evidence for aging as the cause of Alzheimer's disease.

Part 1 presents the results of a meta-analytic study on the effects of aging on intelligence. Analysis of a total of 20 longitudinal samples shows that most of the intelligence scores rose before the age of 50 and fell at a progressively increasing rate after the age of 50. An equation describing this rise and fall in intelligence was derived. Part 2 shows the relationship between the predicted prevalence of Alzheimer's disease (from the equation derived in Part 1) and the prevalence of the disease obtained from 10 studies. The predictive curve fit so well with the observed prevalence data that the results can be interpreted as evidence that Alzheimer's is a manifestation of aging.

A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability.

The author presented descriptive statistics for 937 reliability coefficients for various reliability methods (e.g., alpha) and test types (e.g., intelligence). He compared the average reliability coefficients with the reliability standards that are suggested by experts and found that most average reliabilities were less than ideal. Correlations showed that over the past several decades there has been neither a rise nor a decline in the value of internal consistency, retest, or interjudge reliability coefficients. Of the internal consistency approaches, there has been an increase in the use of coefficient alpha, whereas use of the split-half method has decreased over time. Decision analysis and true-score confidence intervals showed how low reliability can result in clinical decision errors.

MMPI-2: Confidence intervals for random responding to the F, F Back, and VRIN scales.

Several studies have investigated random responding to the F, F Back, and VRIN scales. Only one study attempted to provide practical cutoff scores for these scales, but was unable to reach definitive cutoffs. This study uses the normal approximation to the binomial distribution and provides confidence interval bounds for random responding at the 95, 90, and 85% levels for the F, F Back, and VRIN scales. The possibility that humans asked to respond randomly produce F, F Back, and VRIN scores different from computer-generated random scores was investigated. The results show nonsignificant differences between human and computer responses for the F and F Back scales and mixed results for the VRIN scale.

Boston Naming Test: problems with administration and scoring.

The poorly written administration and scoring instructions for the Boston Naming Test allow too wide a range of interpretations. Three different, seemingly correct interpretations of the scoring methods were compared. The results show that these methods can produce large differences in the total score.

A cautionary note on deviation scores for the WISC-III and WAIS-III.

This article highlights some dangers inherent in interpreting individual examinee strengths and weaknesses in WAIS-III and WISC-III profiles. In both manuals, there are tables providing point estimates for determining if a significant difference exists when comparing one subtest to the average of several subtests. However, these point estimates may lead to interpretation errors. A confidence interval approach that provides a solution to the interpretation problem is described and illustrated. These intervals show that differences between a single subtest score and the average subtest score that just reach statistical significance carry the potential of having no clinical significance.

Study samples are too small to produce sufficiently precise reliability coefficients.

In a survey of journal articles, test manuals, and test critique books, the author found that a mean sample size (N) of 260 participants had been used for reliability studies on 742 tests. The distribution was skewed because the median sample size for the total sample was only 90. The median sample sizes for the internal consistency, retest, and interjudge reliabilities were 182, 64, and 36, respectively. The author presented sample size statistics for the various internal consistency methods and types of tests. In general, the author found that the sample sizes that were used in the internal consistency studies were too small to produce sufficiently precise reliability coefficients, which in turn could cause imprecise estimates of examinee true-score confidence intervals. The results also suggest that larger sample sizes have been used in the last decade compared with those that were used in earlier decades.

Estimating the reliability of a test split into two parts of equal or unequal length.

When the reliability of test scores must be estimated by an internal consistency method, partition of the test into just 2 parts may be the only way to maintain content equivalence of the parts. If the parts are classically parallel, the Spearman-Brown formula may be validly used to estimate the reliability of total scores. If the parts differ in their standard deviations but are tau equivalent, Cronbach's alpha is appropriate. However, if the 2 parts are congeneric, that is, they are unequal in functional length or they comprise heterogeneous item types, a less well-known estimate, the Angoff-Feldt coefficient, is appropriate. Guidelines in terms of the ratio of standard deviations are proposed for choosing among Spearman-Brown, alpha, and Angoff-Feldt coefficients.