Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity?

BACKGROUND: As the number of studies in the literature is increasing, orthopaedic surgeons highly depend on meta-analyses as their primary source of scientific evidence. The objectives of this review were to assess the scientific quality and number of published meta-analyses on orthopaedics-related topics over time. METHODS: We conducted, in duplicate and independently, a systematic review of published meta-analyses in orthopaedics in the years 2005 and 2008 and compared them with a previous systematic review of meta-analyses from 1969 to 1999. A search of electronic databases (MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews) was performed to identify meta-analyses published in 2005 and 2008. We searched bibliographies and contacted content experts to identify additional relevant studies. Two investigators independently assessed the quality of the studies, using the Oxman and Guyatt index, and abstracted relevant data. RESULTS: We included forty-five and forty-four meta-analyses from 2005 and 2008, respectively. While the number of meta-analyses increased fivefold from 1999 to 2008, the mean quality score did not change significantly over time (p = 0.067). In the later years, a significantly lower proportion of meta-analyses had methodological flaws (56% in 2005 and 68% in 2008) compared with meta-analyses published prior to 2000 (88%) (p = 0.006). In 2005 and 2008, respectively, 18% and 30% of the meta-analyses had major to extensive flaws in their methodology. Studies from 2008 with positive conclusions used and described appropriate criteria for the validity assessment less often than did those with negative results. The use of random-effects and fixed-effects models as pooling methods became more popular toward 2008. CONCLUSIONS: Although the methodological quality of orthopaedic meta-analyses has increased in the past twenty years, a substantial proportion continues to show major to extensive flaws. As the number of published meta-analyses is increasing, a routine checklist for scientific quality should be used in the peer-review process to ensure methodological standards for publication.

Overlapping systematic reviews of anterior cruciate ligament reconstruction comparing hamstring autograft with bone-patellar tendon-bone autograft: why are they different?

BACKGROUND: Systematic reviews published on the same topic during a similar period of time (i.e., overlapping reviews) on anterior cruciate ligament reconstruction complicate the choice between bone-patellar tendon-bone or hamstring tendon autograft. We aimed to evaluate reasons for differences among the overlapping systematic reviews and to assess the quality of reporting and internal validity. METHODS: We performed a search of MEDLINE, the Cochrane Database of Systematic Reviews, and EMBASE to identify systematic reviews in which bone-patellar tendon-bone graft was compared with hamstring tendon autograft for reconstruction of the anterior cruciate ligament. We evaluated cross-citations among the overlapping reviews and the authors' rationale for repeating the review. The quality of reporting was assessed with the Quality of Reporting of Meta-analyses (QUOROM) statement, and the internal validity was assessed with the Oxman and Guyatt index for methodological quality by at least two assessors. Assessor agreement was evaluated with intraclass correlation coefficients. We evaluated the sensitivity analysis that had been performed in the reviews. RESULTS: We identified eleven overlapping systematic reviews. Three reviews favored the patellar tendon graft for stability, and one favored the hamstring graft. Six reviews favored the hamstring graft to prevent anterior knee pain, and the rest were inconclusive. Only six reviews cited previously published systematic reviews on the same topic, and only two of these reviews cited all available systematic reviews that were available at that time. The quality of reporting ranged from 5 to 18 (median, 12; maximum score, 18). The internal validity ranged from 1 to 7 (median, 2; maximum score, 7). Reviewers reached almost perfect agreement (intraclass correlation coefficients, 0.83 and 0.94). Formal sensitivity analysis was utilized infrequently. The highest-quality review favored hamstring grafts to prevent anterior knee pain and showed weak evidence that bone-patellar tendon-bone grafts yielded better stability. CONCLUSIONS: When overlapping or discordant systematic reviews are encountered, each review must be appraised on the basis of its methodological quality before it can be used to guide clinical decision-making or policy making. The currently available best evidence, derived from a methodologically sound meta-analysis, suggests that hamstring tendon autografts are superior for preventing anterior knee pain, and there is limited evidence that bone-patellar tendon-bone autografts provide better stability.