Bias, coverage, and asymptotic behaviour of random effects meta-analysis: a clinically driven simulation study.

AIMS: The study of foundational features of meta-analysis is incomplete and continues to remain important. Using simulations we study bias and coverage and the asymptotic behaviour of the DerSimonian and Laird (D&L) meta-analysis with varying trial numbers and sizes, levels of risk, and extent of treatment effects. METHODS: With simulated data we model risk of untoward events in randomized controlled trials in meta-analyses. Treatment effect is expressed as relative risk reduction, with effect size estimated by the odds ratio which is then compared with the known population odds ratio. Performance is measured as bias, standardized bias and coverage, with thresholds for desirable results being prespecified. RESULTS: Bias, standardized bias, and coverage varied substantially across meta-analyses of different trial size and number, risk mean and distribution, and relative risk reduction. Although improvements were observed with increasing trial size and number, there was widespread lack of satisfactory performance. Performance using normal risk distributions was worse compared with performance using constant or narrow uniform risk distributions. Asymptotic behaviour using very large trial numbers failed to show bias that appeared to approach zero for any distribution. CONCLUSION: The D&L random effects meta-analysis method performed modestly at best. We were unable to demonstrate asymptotic normality. These results question the validity of the random effects method. The findings need replication and extension, which, if confirmed, would warn against generic use of the D&L method.

LDL-cholesterol differences predicted survival benefit in statin trials by the surrogate threshold effect (STE).

OBJECTIVE: We describe a new statistical method called the surrogate threshold effect (STE) that estimates the threshold level of a surrogate needed in a clinical trial to predict a benefit in the target clinical outcome. In this article, we apply this method to the LDL-cholesterol biomarker surrogate and survival benefit-target outcome in statin trials. STUDY DESIGN AND SETTING: We identified randomized trials comparing statin treatment to placebo treatment or no treatment and reporting all-cause and cardiovascular mortality. Trials with fewer than five all-cause deaths in at least one arm were excluded. Multiple regression modeled the reduction in all-cause and cardiovascular mortality as a function of LDL-cholesterol difference. The 95% confidence and 95% prediction bands were calculated and graphed to determine the minimum LDL-cholesterol difference (the surrogate threshold) below which there would be no predicted survival benefit. RESULTS: In 16 qualifying trials, regression analysis yielded an all-cause mortality model whose prediction bands demonstrated no overall survival gain with LDL-cholesterol difference values below 1.5 mmol/L. The cardiovascular mortality model yielded prediction bands that demonstrated no cardiovascular survival benefit with LDL-cholesterol difference values below 1.4 mmol/L. CONCLUSIONS: In a multitrial setting, the STE approach is a promising yet straightforward statistical method for evaluating the surrogate validity of biomarkers.

Magnetic resonance imaging in rheumatoid arthritis advances and research priorities.

This article updates the work and results of the OMERACT MRI in RA Working Group as presented at the OMERACT 7 meeting in May 2004, focusing on the development of the EULAR-OMERACT rheumatoid arthritis magnetic resonance imaging reference image atlas, and on areas for future research.

The evidence for magnetic resonance imaging as an outcome measure in proof-of-concept rheumatoid arthritis studies.

Magnetic resonance imaging (MRI) has now been used extensively in cross-sectional and observational studies as well as in controlled clinical trials to assess disease activity and joint damage in rheumatoid arthritis (RA). MRI measurements or scores for erosions, bone edema, and synovitis have been developed and validated by several groups. The OMERACT criteria require that outcome measures demonstrate adequate validity, discriminative power, and feasibility if they are to be useful in clinical trials. Specific performance targets for these criteria depend on the scientific, regulatory, logistical, and financial context of the study in question. We review the extent to which MRI assessments of joint erosion, bone edema, and synovitis fulfil these criteria, particularly as they relate to proof-of-concept RA clinical trials.

Musculoskeletal ultrasound including definitions for ultrasonographic pathology.

Ultrasound (US) has great potential as an outcome in rheumatoid arthritis trials for detecting bone erosions, synovitis, tendon disease, and enthesopathy. It has a number of distinct advantages over magnetic resonance imaging, including good patient tolerability and ability to scan multiple joints in a short period of time. However, there are scarce data regarding its validity, reproducibility, and responsiveness to change, making interpretation and comparison of studies difficult. In particular, there are limited data describing standardized scanning methodology and standardized definitions of US pathologies. This article presents the first report from the OMERACT ultrasound special interest group, which has compared US against the criteria of the OMERACT filter. Also proposed for the first time are consensus US definitions for common pathological lesions seen in patients with inflammatory arthritis.

Minimal disease activity for rheumatoid arthritis: a preliminary definition.

Agreement on response criteria in rheumatoid arthritis (RA) has allowed better standardization and interpretation of clinical trial reports. With recent advances in therapy, the proportion of patients achieving a satisfactory state of minimal disease activity (MDA) is becoming a more important measure with which to compare different treatment strategies. The threshold for MDA is between high disease activity and remission and, by definition, anyone in remission will also be in MDA. True remission is still rare in RA; in addition, the American College of Rheumatology definition is difficult to apply in the context of trials. Participants at OMERACT 6 in 2002 agreed on a conceptual definition of minimal disease activity (MDA): "that state of disease activity deemed a useful target of treatment by both the patient and the physician, given current treatment possibilities and limitations." To prepare for a preliminary operational definition of MDA for use in clinical trials, we asked rheumatologists to assess 60 patient profiles describing real RA patients seen in routine clinical practice. Based on their responses, several candidate definitions for MDA were designed and discussed at the OMERACT 7 in 2004. Feedback from participants and additional on-site analyses in a cross-sectional database allowed the formulation of 2 preliminary, equivalent definitions of MDA: one based on the Disease Activity Score 28 (DAS28) index, and one based on meeting cutpoints in 5 out the 7 WHO/ILAR core set measures. Researchers applying these definitions first need to choose whether to use the DAS28 or the core set definition, because although each selects a similar proportion in a population, these are not always the same patients. In both MDA definitions, an initial decision node places all patients in MDA who have a tender joint count of 0 and a swollen joint count of 0, and an erythrocyte sedimentation rate (ESR) no greater than 10 mm. If this condition is not met: * The DAS28 definition places patients in MDA when DAS28 < or = 2.85; * The core set definition places patients in MDA when they meet 5 of 7 criteria: (1) Pain (0-10) < or = 2; (2) Swollen joint count (0-28) < or = 1; (3) Tender joint count (0-28) < or = 1; (4) Health Assessment Questionnaire (HAQ, 0-3) < or = 0.5; (5) Physician global assessment of disease activity (0-10) < or = 1.5; (6) Patient global assessment of disease activity (0-10) < or = 2; (7) ESR < or = 20. This set of 2 definitions gained approval of 73% of the attendees. These (and other) definitions will now be subject to further validation in other databases.

Generic patient self-report and investigator report instruments of therapeutic safety and tolerability.

A patient self-report instrument was designed as a patient event index that maps to a parallel investigator instrument. Event importance (a composite of severity, frequency, and duration) was reported, but attribution was not required. The patient instrument used a checklist but also allowed for spontaneous reporting for new or unusual events. The investigator instrument (also a checklist) includes all events reported by the patient, as well as events such as signs, investigations, and diagnoses that would not generally be known to the patient. Presently, both patient and investigator instruments are to be used alongside current methods of adverse event reporting in clinical trials. The patient instrument would serve as a safety/tolerability index, whereas the investigator instrument would be a fully quantifiable (appropriately weighted), standardized adverse event index. As in many methodological projects in medicine, the overriding problem was the tradeoff between validity (comprehensiveness and accuracy) and feasibility (clarity and short administration time) in instrument development. A summary of pilot studies and results of instrument reliability and validity are presented.

Challenges and progress in adverse event ascertainment and reporting in clinical trials.

Toxicity, safety, and tolerability are integral facets of patient risk/benefit decisions, yet the capacity to define, measure, and compare these aspects is underdeveloped compared to aspects of efficacy. There are many reasons for this, scientific and administrative, but all are surmountable. Probably the greatest primary obstacle is the absence of a measurement instrument designed specifically for this purpose. There are increasing calls from various stakeholders for better evidence, and therefore better ascertainment, in this area, especially in randomized trials, and for these reasons OMERACT began deliberations about these concepts in 1994. A prototype coding instrument (the Rheumatology Common Toxicity Criteria) was developed and discussed at OMERACT 5. In the 2 years before OMERACT 7, a process of concept development and iterative design and testing were conducted to develop a patient self-report and investigator-reported adverse event instruments designed for use in trials at the time of visit. The predominant workload is performed by the patient in a self-report checklist, which is then mapped by the trialist onto a medically sophisticated version. This article presents background on the process of developing a dual adverse event instrument, which was presented and critically discussed in detail at OMERACT 7.

Standardized assessment of adverse events in rheumatology clinical trials: summary of the OMERACT 7 drug safety module update.

A presentation, demonstration, and discussion of recently developed adverse event instruments were the topics for the OMERACT 7 Drug Safety Module. The module began with a plenary introducing the needs and challenges of adverse event ascertainment. It was followed by a review of module work from previous OMERACT meetings on a prototype coding instrument (Rheumatology Common Toxicity Criteria), then a brief description of the process behind the recently developed patient self-report and investigator report adverse event instruments. These current instruments are designed for use in controlled trials although they could be used in other settings. The instruments rely primarily on patient self-reporting using a checklist, which the investigator then folds into a parallel structured but more medically sophisticated instrument. In pilot testing, this innovative dual-use system has shown reliability and acceptability, while preserving validity. A "stakeholder panel" of representatives from 8 sectors followed--patient, nurse investigator, regulator, clinician scientist, industry, OMERACT, global public health/WHO, and Cochrane Collaboration--for their perspectives on the needs, challenges, and potential ways forward for adverse event ascertainment and reporting in clinical trials. At the breakout session small focus groups participated in hands-on interactive testing of one of 3 versions of the instruments, which differ in degree of comprehensiveness. Each focus group had a participatory patient with rheumatoid arthritis. At a second plenary there was group feedback by rapporteurs and presentation of results from pilot studies of iterative testing of validity, reliability, and feasibility of the instruments. During plenary discussion a frequent suggestion for improvement was to refine the process so that event ascertainment could be done entirely using the patient instrument with minimal input from the investigator at the visit, if patient-investigator agreement was high. Most found the patient checklist attractive, particularly if the patient instrument was shown to be reliable and valid. Finally, a future research agenda was discussed.