Fast-and-frugal decision tree for the rapid critical appraisal of systematic reviews.

Conducting high-quality overviews of reviews (OoR) is time-consuming. Because the quality of systematic reviews (SRs) varies, it is necessary to critically appraise SRs when conducting an OoR. A well-established appraisal tool is A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2, which takes about 15-32 min per application. To save time, we developed two fast-and-frugal decision trees (FFTs) for assessing the methodological quality of SR for OoR either during the full-text screening stage (Screening FFT) or to the resulting pool of SRs (Rapid Appraisal FFT). To build a data set for developing the FFT, we identified published AMSTAR 2 appraisals. Overall confidence ratings of the AMSTAR 2 were used as a criterion and the 16 items as cues. One thousand five hundred and nineteen appraisals were obtained from 24 publications and divided into training and test data sets. The resulting Screening FFT consists of three items and correctly identifies all non-critically low-quality SRs (sensitivity of 100%), but has a positive predictive value of 59%. The three-item Rapid Appraisal FFT correctly identifies 80% of the high-quality SRs and correctly identifies 97% of the low-quality SRs, resulting in an accuracy of 95%. The FFTs require about 10% of the 16 AMSTAR 2 items. The Screening FFT may be applied during full-text screening to exclude SRs with critically low quality. The Rapid Appraisal FFT may be applied to the final SR pool to identify SR that might be of high methodological quality.

Relationship between volume and outcome for gastroschisis: A systematic review.

BACKGROUND: Newborns with gastroschisis need surgery to reduce intestines into the abdominal cavity and to close the abdominal wall. Due to an existing volume-outcome relationship for other high-risk, low-volume procedures, we aimed at examining the relationship between hospital or surgeon volume and outcomes for gastroschisis. METHODS: We conducted a systematic literature search in Medline, Embase, CENTRAL, CINAHL and Biosis Previews in June 2021 and searched for additional literature. We included (cluster-) randomized controlled trials (RCTs) and prospective or retrospective cohort studies analyzing the relationship between hospital or surgeon volume and mortality, morbidity or quality of life. We assessed risk of bias of included studies using ROBINS-I and performed a systematic synthesis without meta-analysis and used GRADE for assessing the certainty of the evidence. RESULTS: We included 12 cohort studies on hospital volume. Higher hospital volume may reduce in-hospital mortality of neonates with gastroschisis, while the evidence is very uncertain for other outcomes. Findings are based on a low certainty of the evidence for in-hospital mortality and a very low certainty of the evidence for all other analyzed outcomes, mainly due to risk of bias and imprecision. We did not identify any study on surgeon volume. CONCLUSION: The evidence suggests that higher hospital volume reduces in-hospital mortality of newborns with gastroschisis. However, the magnitude of this effect seems to be heterogeneous and results should be interpreted with caution. There is no evidence on the relationship between surgeon volume and outcomes.

Measuring test-retest reliability (TRR) of AMSTAR provides moderate to perfect agreement - a contribution to the discussion of the importance of TRR in relation to the psychometric properties of assessment tools.

BACKGROUND: Systematic Reviews (SRs) can build the groundwork for evidence-based health care decision-making. A sound methodological quality of SRs is crucial. AMSTAR (A Measurement Tool to Assess Systematic Reviews) is a widely used tool developed to assess the methodological quality of SRs of randomized controlled trials (RCTs). Research shows that AMSTAR seems to be valid and reliable in terms of interrater reliability (IRR), but the test retest reliability (TRR) of AMSTAR has never been investigated. In our study we investigated the TRR of AMSTAR to evaluate the importance of its measurement and contribute to the discussion of the measurement properties of AMSTAR and other quality assessment tools. METHODS: Seven raters at three institutions independently assessed the methodological quality of SRs in the field of occupational health with AMSTAR. Between the first and second ratings was a timespan of approximately two years. Answers were dichotomized, and we calculated the TRR of all raters and AMSTAR items using Gwet's AC1 coefficient. To investigate the impact of variation in the ratings over time, we obtained summary scores for each review. RESULTS: AMSTAR item 4 (Was the status of publication used as an inclusion criterion?) provided the lowest median TRR of 0.53 (moderate agreement). Perfect agreement of all reviewers was detected for AMSTAR-item 1 with a Gwet's AC1 of 1, which represented perfect agreement. The median TRR of the single raters varied between 0.69 (substantial agreement) and 0.89 (almost perfect agreement). Variation of two or more points in yes-scored AMSTAR items was observed in 65% (73/112) of all assessments. CONCLUSIONS: The high variation between the first and second AMSTAR ratings suggests that consideration of the TRR is important when evaluating the psychometric properties of AMSTAR.. However, more evidence is needed to investigate this neglected issue of measurement properties. Our results may initiate discussion of the importance of considering the TRR of assessment tools. A further examination of the TRR of AMSTAR, as well as other recently established rating tools such as AMSTAR 2 and ROBIS (Risk Of Bias In Systematic reviews), would be useful.

Authors should clearly report how they derived the overall rating when applying AMSTAR 2-a cross-sectional study.

OBJECTIVES: A measurement tool to assess systematic reviews (SRs) 2 (AMSTAR 2) allows for deriving the overall confidence in an SR. We investigated how authors derived the overall confidence rating and whether different schemes lead to different results. STUDY DESIGN AND SETTING: We compared three different schemes (original 7-item scheme, a self-developed 5-item scheme, and the AMSTAR Web site) to derive the overall confidence in AMSTAR 2 using two distinct samples of SRs. Multiple bibliographic databases were searched for articles to analyze how AMSTAR 2 was applied by others. RESULTS: In both samples (n = 60 and n = 58), the Friedman test revealed a significant difference between the schemes (P < 0.001). The Web site scheme was the least strict one, whereas between the 5-item and 7-item scheme, no differences were found in post hoc analyses. We included 53 publications applying AMSTAR 2 identified in our literature search. Only 37 of them (70%) used the original 7-item scheme. Less than half of them (18 of 37) reported how they derived the overall rating. CONCLUSION: Authors should clearly report how they have derived the overall rating when applying AMSTAR 2. Reporting should allow for reproducing the overall ratings for editors, peer reviewers, and readers.

The methodological quality of systematic reviews on the treatment of adult major depression needs improvement according to AMSTAR 2: A cross-sectional study.

BACKGROUND: Several standards have been developed to assess methodological quality of systematic reviews (SR). One widely used tool is the AMSTAR. A recent update - AMSTAR 2 - is a 16 item evaluation tool that enables a detailed assessment of SR that include randomised (RCT) or non-randomised studies (NRS) of healthcare interventions. METHODS: A cross-sectional study of SR on pharmacological or psychological interventions in major depression in adults was conducted. SR published during 2012-2017 were sampled from MEDLINE, EMBASE and the Cochrane Database of SR. Methodological quality was assessed using AMSTAR 2. Potential predictive factors associated with quality were examined. RESULTS: In rating overall confidence in the results of 60 SR four reviews were rated "high", two were "moderate", one was "low" and 53 were "critically low". The mean AMSTAR 2 percentage score was 45.3% (standard deviation 22.6%) in a wide range from 7.1% to 93.8%. Predictors of higher quality were: type of review (higher quality in Cochrane Reviews), SR including only randomized trials and higher journal impact factor. LIMITATIONS: AMSTAR 2 is not intended to be used for the generation of a percentage score. CONCLUSIONS: According to AMSTAR 2 the overall methodological quality of SR on the treatment of adult major depression needs improvement. Although there is a high need for summarized information in the field of mental health, this work demonstrates the need to critically assess SR before using their findings. Better adherence to established reporting guidelines for SR is needed.

Relationship between volume and outcome for gastroschisis: a systematic review protocol.

BACKGROUND: Gastroschisis is a congenital anomaly that needs surgical management for repositioning intestines into the abdominal cavity and for abdominal closure. Higher hospital or surgeon volume has previously been found to be associated with better clinical outcomes for different especially high-risk, low volume procedures. Therefore, we aim to examine the relationship between hospital or surgeon volume and outcomes for gastroschisis. METHODS: We will perform a systematic literature search from inception onwards in Medline, Embase, CENTRAL, CINAHL, and Biosis Previews without applying any limitations. In addition, we will search trial registries and relevant conference proceedings. We will include (cluster-) randomized controlled trials (RCTs) and prospective or retrospective cohort studies analyzing the relationship between hospital or surgeon volume and clinical outcomes. The primary outcomes will be survival and mortality. Secondary outcomes will be different measures of morbidity (e.g., severe gastrointestinal complications, gastrointestinal dysfunctions, and sepsis), quality of life, and length of stay. We will systematically assess risk of bias of included studies using RoB 2 for individually or cluster-randomized trials and ROBINS-I for cohort studies, and extract data on the study design, patient characteristics, case-mix adjustments, statistical methods, hospital and surgeon volume, and outcomes into standardized tables. Title and abstract screening, full text screening, critical appraisal, and data extraction of results will be conducted by two reviewers independently. Other data will be extracted by one reviewer and checked for accuracy by a second one. Any disagreements will be resolved by discussion. We will not pool results statistically as we expect included studies to be clinically and methodologically very diverse. We will conduct a systematic synthesis without meta-analysis and use GRADE for assessing the certainty of the evidence. DISCUSSION: Given the lack of a comprehensive summary of findings on the relationship between hospital or surgeon volume and outcomes for gastroschisis, this systematic review will put things right. Results can be used to inform decision makers or clinicians and to adapt medical care. SYSTEMATIC REVIEW REGISTRATION: Open Science Framework (DOI: https://doi.org/10.17605/OSF.IO/EX34M ; https://doi.org/10.17605/OSF.IO/HGPZ2 ).

Relationship between volume and outcome for surgery on congenital diaphragmatic hernia: A systematic review.

BACKGROUND: Congenital diaphragmatic hernia (CDH) is a rare and life-threatening anomaly that needs surgical therapy after clinical stabilization of the neonate. Given an existing volume-outcome relationship for other high-risk, low volume procedures, we aimed at examining the relationship between hospital or surgeon volume and outcomes for surgery on CDH. METHODS: We conducted a systematic search in multiple databases in September 2019 and searched for additional literature. We assessed risk of bias of included studies using ROBINS-I and synthesized results in a structured narrative way using GRADE. RESULTS: We included 5 cohort studies on hospital volume. Results for in-hospital mortality, one-year mortality and length of stay are inconclusive. The certainty of the evidence was very low for all outcomes, due to risk of bias, inconsistency and imprecision. We did not identify any study on surgeon volume. CONCLUSION: Due to the very low certainty of the evidence it is uncertain whether higher hospital volume is associated with favorable outcomes for neonates undergoing surgery for CDH. There is no evidence on the relationship between surgeon volume and outcomes. Future studies should use more rigorous methodology and analyze additional outcomes to allow for more meaningful inferences. LEVEL OF EVIDENCE: III SYSTEMATIC REVIEW REGISTRATION: PROSPERO (CRD42018090231).

A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool.

OBJECTIVES: The objectives of this study were to determine the interrater reliability (IRR) of assessment of multiple systematic reviews (AMSTAR) 2 for reviews of pharmacological or psychological interventions for the treatment of major depression, to compare it to that of AMSTAR and risk of bias in systematic reviews (ROBIS), and to assess the convergent validity between the appraisal tools. STUDY DESIGN AND SETTING: Two groups of four raters were each assigned one of two samples of 30 systematic reviews. All eight raters applied AMSTAR 2 to their sample. Each group also applied either AMSTAR or ROBIS. Fleiss' kappa and Gwet's AC1 were calculated, and agreement between the tools was assessed. RESULTS: The median kappa values as a measure of IRR indicated a moderate agreement for AMSTAR 2 (median = 0.51), a substantial agreement for AMSTAR (median = 0.62), and a fair agreement for ROBIS (median = 0.27). Validity results showed a positive association for AMSTAR and AMSTAR 2 (r = 0.91) as well as ROBIS and AMSTAR 2 (r = 0.84). For the overall rating, AMSTAR 2 showed a high concordance with ROBIS and a lower concordance with AMSTAR. CONCLUSION: The IRR of AMSTAR 2 was found to be slightly lower than the IRR of AMSTAR and higher than the IRR of ROBIS. Validity measurements indicate that AMSTAR 2 is closely related to both ROBIS and AMSTAR.