Reevaluation of statistically significant meta-analyses in advanced cancer patients using the Hartung-Knapp method and prediction intervals-A methodological study.

Using the Hartung-Knapp method and 95% prediction intervals (PIs) in random-effects meta-analyses is recommended by experts but rarely applied. Therefore, we aimed to reevaluate statistically significant meta-analyses using the Hartung-Knapp method and 95% PIs. In this methodological study, three databases were searched from January 2010 to July 2019. We included systematic reviews reporting a statistically significant meta-analysis of at least four randomized controlled trials in advanced cancer patients using either a fixed-effect or random-effects model. We investigated the impact of switching from fixed-effect to random-effects meta-analysis and of using the recommended Hartung-Knapp method in random-effects meta-analyses. Furthermore, we calculated 95% PIs for all included meta-analyses. We identified 6234 hits, of which 261 statistically significant meta-analyses were included. Our recalculations of these 261 meta-analyses produced statistically significant results in 132 of 138 fixed-effect and 114 of 123 random-effects meta-analyses. When switching to a random-effects model, 19 of 132 fixed-effect meta-analyses (14.4%) were no longer statistically significant. Using the Hartung-Knapp method in random-effects meta-analyses resulted in 34 of 114 nonsignificant meta-analyses (29.8%). In the full sample (N = 261), the null effect was included by the 95% PI in 195 (74.7%) and the opposite effect (e.g., hazard ratio 0.5, opposite effect 2) in 98 meta-analyses (37.5%). Using the Hartung-Knapp method and PIs substantially influenced the interpretation of many published, statistically significant meta-analyses. We strongly encourage researchers to check if using the Hartung-Knapp method and reporting 95% PIs is appropriate in random-effects meta-analyses.

Methodological quality was critically low in 9/10 systematic reviews in advanced cancer patients-A methodological study.

OBJECTIVE: To assess the methodological quality and the consideration of heterogeneity in systematic reviews (SRs). STUDY DESIGN AND SETTING: We conducted a methodological study (CRD42019134904) and searched three databases from January 2010 to July 2019. Interventional SRs with a statistically significant meta-analysis of at least four randomized controlled trials in advanced cancer patients were included. A MeaSurement Tool to Assess Systematic Reviews (AMSTAR) 2 was used to evaluate the SRs' methodological quality. The consideration of heterogeneity was categorized in clinical or/and methodological heterogeneity and not explored. RESULTS: From 6234 identified references, 261 SRs were included. Most SRs had a critically low quality (230, 88.1%). The majority of them (209, 80.1%) was classified as critically low because of non-registration (222, 85.1%) combined with the non-reporting of excluded full-texts and missing justifications for exclusion (218, 83.5%). Heterogeneity in trial results was not explored at all in 51 (19.5%) SRs whereas clinical heterogeneity was considered in 117 (44.8%), methodological heterogeneity in 13 (5.0%), and both clinical and methodological heterogeneity in 80 (30.7%) SRs. CONCLUSION: The consideration of these findings in trainings for review authors and peer reviewers could improve the awareness of quality criteria and the quality of future SRs. TRIAL REGISTRATION: PROSPERO-ID: CRD42019134904.