Assessment of quality of systematic reviews in dermatology using AMSTAR 2 Part 2 of 2. Validity and reliability testing of AMSTAR 2 (UP-PGH)

ABSTRACT

@#BACKGROUND AMSTAR 2 enables a more detailed assessment of systematic reviews and includes non-randomised studies of healthcare interventions, compared to its earlier version, AMSTAR. We validated AMSTAR 2 in a group of systematic reviews in dermatology in the Philippines. METHODS We used a cohort of systematic reviews (SRs) in dermatology from the Philippine that were previously described in Part 1 of this 2-part series. The SRs included clinical trials on any intervention for the treatment or prevention of a dermatologic disease or for maintenance of healthy skin, hair or nails. Two reviewers independently extracted data and used AMSTAR 2 to appraise the methodological quality of each included SR. We determined construct validity by comparing the number of critical flaws between a set of non-Cochrane and matched Cochrane reviews, using Wilcoxon rank sum test. We tested for interrater reliability of the AMSTAR 2 tool using Gwet’s AC1 statistic. RESULTS: We included 20 non-Cochrane systematic reviews in dermatology by Philippine-based authors, and a set of 20 reviews from the Cochrane skin group, matched by year and randomly chosen. Construct validity testing showed a significantly greater number of AMSTAR 2 critical flaws (median 4.5 vs 0.0; z=3.64; P=0.000) and non-critical weaknesses (5 vs 2.0; z-score=3.10; P-value=0.001) by non-Cochrane reviews compared to a matched set of Cochrane skin group reviews. There was good interrater reliability (average Gwet’s AC1 statistic = 0.87) with the lowest agreement (0.62) for discussion of heterogeneity (item 14), and the highest agreement (0.97) for study selection criteria (item 3). CONCLUSION The AMSTAR 2 was a valid and reliable tool for assessing systematic reviews using a cohort of reviews by dermatology reviews, both non-Cochrane and Cochrane. Further validation of the AMSTAR 2 is needed to determine if it can be applied to a wide variety of systematic reviews.