Shortest path or random walks? A framework for path weights in network meta-analysis.

Quantifying the contributions, or weights, of comparisons or single studies to the estimates in a network meta-analysis (NMA) is an active area of research. We extend this work to include the contributions of paths of evidence. We present a general framework, based on the path-design matrix, that describes the problem of finding path contributions as a linear equation. The resulting solutions may have negative coefficients. We show that two known approaches, called shortestpath and randomwalk, are special solutions of this equation, and both meet an optimization criterion, as they minimize the sum of absolute path contributions. In general, there is an infinite set of solutions, which can be identified using the generalized inverse (Moore-Penrose pseudoinverse). We consider two further special approaches. For large networks we find that shortestpath is superior with respect to run time and variability, compared to the other approaches, and is thus recommended in practice. The path-weights framework also has the potential to answer more general research questions in NMA.

When does the placebo effect have an impact on network meta-analysis results?

The placebo effect is the 'effect of the simulation of treatment that occurs due to a participant's belief or expectation that a treatment is effective'. Although the effect might be of little importance for some conditions, it can have a great role in others, mostly when the evaluated symptoms are subjective. Several characteristics that include informed consent, number of arms in a study, the occurrence of adverse events and quality of blinding may influence response to placebo and possibly bias the results of randomised controlled trials. Such a bias is inherited in systematic reviews of evidence and their quantitative components, pairwise meta-analysis (when two treatments are compared) and network meta-analysis (when more than two treatments are compared). In this paper, we aim to provide red flags as to when a placebo effect is likely to bias pairwise and network meta-analysis treatment effects. The classic paradigm has been that placebo-controlled randomised trials are focused on estimating the treatment effect. However, the magnitude of placebo effect itself may also in some instances be of interest and has also lately received attention. We use component network meta-analysis to estimate placebo effects. We apply these methods to a published network meta-analysis, examining the relative effectiveness of four psychotherapies and four control treatments for depression in 123 studies.

Network meta-analysis and random walks.

Network meta-analysis (NMA) is a central tool for evidence synthesis in clinical research. The results of an NMA depend critically on the quality of evidence being pooled. In assessing the validity of an NMA, it is therefore important to know the proportion contributions of each direct treatment comparison to each network treatment effect. The construction of proportion contributions is based on the observation that each row of the hat matrix represents a so-called "evidence flow network" for each treatment comparison. However, the existing algorithm used to calculate these values is associated with ambiguity according to the selection of paths. In this article, we present a novel analogy between NMA and random walks. We use this analogy to derive closed-form expressions for the proportion contributions. A random walk on a graph is a stochastic process that describes a succession of random "hops" between vertices which are connected by an edge. The weight of an edge relates to the probability that the walker moves along that edge. We use the graph representation of NMA to construct the transition matrix for a random walk on the network of evidence. We show that the net number of times a walker crosses each edge of the network is related to the evidence flow network. By then defining a random walk on the directed evidence flow network, we derive analytically the matrix of proportion contributions. The random-walk approach has none of the associated ambiguity of the existing algorithm.

Answering complex hierarchy questions in network meta-analysis.

BACKGROUND: Network meta-analysis estimates all relative effects between competing treatments and can produce a treatment hierarchy from the most to the least desirable option according to a health outcome. While about half of the published network meta-analyses present such a hierarchy, it is rarely the case that it is related to a clinically relevant decision question. METHODS: We first define treatment hierarchy and treatment ranking in a network meta-analysis and suggest a simulation method to estimate the probability of each possible hierarchy to occur. We then propose a stepwise approach to express clinically relevant decision questions as hierarchy questions and quantify the uncertainty of the criteria that constitute them. The steps of the approach are summarized as follows: a) a question of clinical relevance is defined, b) the hierarchies that satisfy the defined question are collected and c) the frequencies of the respective hierarchies are added; the resulted sum expresses the certainty of the defined set of criteria to hold. We then show how the frequencies of all possible hierarchies relate to common ranking metrics. RESULTS: We exemplify the method and its implementation using two networks. The first is a network of four treatments for chronic obstructive pulmonary disease where the most probable hierarchy has a frequency of 28%. The second is a network of 18 antidepressants, among which Vortioxetine, Bupropion and Escitalopram occupy the first three ranks with frequency 19%. CONCLUSIONS: The developed method offers a generalised approach of producing treatment hierarchies in network meta-analysis, which moves towards attaching treatment ranking to a clear decision question, relevant to all or a subset of competing treatments.

ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis.

BACKGROUND: Selective outcome reporting and publication bias threaten the validity of systematic reviews and meta-analyses and can affect clinical decision-making. A rigorous method to evaluate the impact of this bias on the results of network meta-analyses of interventions is lacking. We present a tool to assess the Risk Of Bias due to Missing Evidence in Network meta-analysis (ROB-MEN). METHODS: ROB-MEN first evaluates the risk of bias due to missing evidence for each of the possible pairwise comparison that can be made between the interventions in the network. This step considers possible bias due to the presence of studies with unavailable results (within-study assessment of bias) and the potential for unpublished studies (across-study assessment of bias). The second step combines the judgements about the risk of bias due to missing evidence in pairwise comparisons with (i) the contribution of direct comparisons to the network meta-analysis estimates, (ii) possible small-study effects evaluated by network meta-regression, and (iii) any bias from unobserved comparisons. Then, a level of "low risk", "some concerns", or "high risk" for the bias due to missing evidence is assigned to each estimate, which is our tool's final output. RESULTS: We describe the methodology of ROB-MEN step-by-step using an illustrative example from a published NMA of non-diagnostic modalities for the detection of coronary artery disease in patients with low risk acute coronary syndrome. We also report a full application of the tool on a larger and more complex published network of 18 drugs from head-to-head studies for the acute treatment of adults with major depressive disorder. CONCLUSIONS: ROB-MEN is the first tool for evaluating the risk of bias due to missing evidence in network meta-analysis and applies to networks of all sizes and geometry. The use of ROB-MEN is facilitated by an R Shiny web application that produces the Pairwise Comparisons and ROB-MEN Table and is incorporated in the reporting bias domain of the CINeMA framework and software.

Do reporting guidelines have an impact? Empirical assessment of changes in reporting before and after the PRISMA extension statement for network meta-analysis.

BACKGROUND: The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension statement for network meta-analysis (NMA) published in 2015 promotes comprehensive reporting in published systematic reviews with NMA. PRISMA-NMA includes 32 items: 27 core items as indicated in the 2009 PRISMA Statement and five items specific to the reporting of NMAs. Although NMA reporting is improving, it is unclear whether PRISMA-NMA has accelerated this improvement. We aimed to investigate the impact of PRISMA-NMA and highlight key items that require attention and improvement. METHODS: We updated our previous collection of NMAs with articles published between April 2015 and July 2018. We assessed the completeness of reporting for each NMA, including main manuscript and online supplements, using the PRISMA-NMA checklist. The PRISMA-NMA checklist originally includes 32 total items (i.e. a 32-point scale original PRISMA-NMA score). We also prepared a modified version of the PRISMA-NMA checklist with 49 items to evaluate separately at a more granular level all multiple-content items (i.e. a 49-point scale modified PRISMA-NMA score). We compared average reporting scores of articles published until and after 2015. RESULTS: In the 1144 included NMAs the mean modified PRISMA-NMA score was 32.1 (95% CI 31.8-32.4) of a possible 49-excellence-score. For 1-year increase, the mean modified score increased by 0.96 (95% CI 0.32 to 1.59) for 389 NMAs published until 2015 and by 0.53 (95% CI 0.02 to 1.04) for 755 NMAs published after 2015. The mean modified PRISMA-NMA score for NMAs published after 2015 was higher by 0.81 (95% CI 0.23 to 1.39) compared to before 2015 when adjusting for journal impact factor, type of review, funding, and treatment category. Description of summary effect sizes to be used, presentation of individual study data, sources of funding for the systematic review, and role of funders dropped in frequency after 2015 by 6-16%. CONCLUSIONS: NMAs published after 2015 more frequently reported the five items associated with NMA compared to those published until 2015. However, improvement in reporting after 2015 is compatible with that observed on a yearly basis until 2015, and hence, it could not be attributed solely to the publication of the PRISMA-NMA.

Meta-analysis as a system of springs.

Meta-analysis results are usually presented in forest plots, which show the individual study results and the summary effect along with their confidence intervals. In this paper, we propose a system of linear springs as a mechanical analogue of meta-analysis that enables visualization and enhances intuition. The length of a spring corresponds to a study treatment effect and the stiffness of the spring corresponds to its inverse variance. To synthesize study springs we use two main operations: connection in parallel and connection in series. We show the equivalence between meta-analysis and linear springs for fixed effect and random effects pairwise meta-analysis and we also derive indirect treatment effects. We use examples to illustrate the different meta-analytical schemes using the corresponding system of springs. The proposed visualization can serve as an educational tool, especially useful for researchers with no statistical background. The analogy between meta-analysis and springs facilitates intuition for notions such as heterogeneity and the differences between fixed and random effects meta-analysis.

Network meta-analysis results against a fictional treatment of average performance: Treatment effects and ranking metric.

BACKGROUND: Network meta-analysis (NMA) produces complex outputs as many comparisons between interventions are of interest. The estimated relative treatment effects are usually displayed in a forest plot or in a league table and several ranking metrics are calculated and presented. METHODS: In this article, we estimate relative treatment effects of each competing treatment against a fictional treatment of average performance using the "deviation from the means" coding that has been used to parametrize categorical covariates in regression models. We then use this alternative parametrization of the NMA model to present a ranking metric (PreTA: Preferable Than Average) interpreted as the probability that a treatment is better than a fictional treatment of average performance. RESULTS: We illustrate the alternative parametrization of the NMA model using two networks of interventions, a network of 18 antidepressants for acute depression and a network of four interventions for heavy menstrual bleeding. We also use these two networks to highlight differences among PreTA and existing ranking metrics. We further examine the agreement between PreTA and existing ranking metrics in 232 networks of interventions and conclude that their agreement depends on the precision with which relative effects are estimated. CONCLUSIONS: A forest plot with NMA relative treatment effects using "deviation from means" coding could complement presentation of NMA results in large networks and in absence of an obvious reference treatment. PreTA is a viable alternative to existing probabilistic ranking metrics that naturally incorporates uncertainty.

Agreement between ranking metrics in network meta-analysis: an empirical study.

OBJECTIVE: To empirically explore the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA) and to investigate how network characteristics influence the agreement. DESIGN: Empirical evaluation from re-analysis of NMA. DATA: 232 networks of four or more interventions from randomised controlled trials, published between 1999 and 2015. METHODS: We calculated treatment hierarchies from several ranking metrics: relative treatment effects, probability of producing the best value [Formula: see text] and the surface under the cumulative ranking curve (SUCRA). We estimated the level of agreement between the treatment hierarchies using different measures: Kendall's τ and Spearman's ρ correlation; and the Yilmaz [Formula: see text] and Average Overlap, to give more weight to the top of the rankings. Finally, we assessed how the amount of the information present in a network affects the agreement between treatment hierarchies, using the average variance, the relative range of variance and the total sample size over the number of interventions of a network. RESULTS: Overall, the pairwise agreement was high for all treatment hierarchies obtained by the different ranking metrics. The highest agreement was observed between SUCRA and the relative treatment effect for both correlation and top-weighted measures whose medians were all equal to 1. The agreement between rankings decreased for networks with less precise estimates and the hierarchies obtained from [Formula: see text] appeared to be the most sensitive to large differences in the variance estimates. However, such large differences were rare. CONCLUSIONS: Different ranking metrics address different treatment hierarchy problems, however they produced similar rankings in the published networks. Researchers reporting NMA results can use the ranking metric they prefer, unless there are imprecise estimates or large imbalances in the variance estimates. In this case treatment hierarchies based on both probabilistic and non-probabilistic ranking metrics should be presented.

The statistical importance of a study for a network meta-analysis estimate.

BACKGROUND: In pairwise meta-analysis, the contribution of each study to the pooled estimate is given by its weight, which is based on the inverse variance of the estimate from that study. For network meta-analysis (NMA), the contribution of direct (and indirect) evidence is easily obtained from the diagonal elements of a hat matrix. It is, however, not fully clear how to generalize this to the percentage contribution of each study to a NMA estimate. METHODS: We define the importance of each study for a NMA estimate by the reduction of the estimate's variance when adding the given study to the others. An equivalent interpretation is the relative loss in precision when the study is left out. Importances are values between 0 and 1. An importance of 1 means that the study is an essential link of the pathway in the network connecting one of the treatments with another. RESULTS: Importances can be defined for two-stage and one-stage NMA. These numbers in general do not add to one and thus cannot be interpreted as 'percentage contributions'. After briefly discussing other available approaches, we question whether it is possible to obtain unique percentage contributions for NMA. CONCLUSIONS: Importances generalize the concept of weights in pairwise meta-analysis in a natural way. Moreover, they are uniquely defined, easily calculated, and have an intuitive interpretation. We give some real examples for illustration.

In network meta-analysis, most of the information comes from indirect evidence: empirical study.

OBJECTIVES: Network meta-analysis (NMA) may produce more precise estimates of treatment effects than pairwise meta-analysis. We examined the relative contribution of network paths of different lengths to estimates of treatment effects. STUDY DESIGN AND SETTING: We analyzed 213 published NMAs. We categorized network shapes according to the presence or absence of at least one closed loop (nonstar or star network) and derived the graph density, radius, and diameter. We identified paths of different lengths and calculated their percentage contribution to each NMA effect estimate, based on their contribution matrix. RESULTS: Among the 213 NMAs included in analyses, 33% of the information came from paths of length 1 (direct evidence), 47% from paths of length 2 (indirect paths with one intermediate treatment) and 20% from paths of length 3. The contribution of paths of different lengths depended on the size of networks, presence of closed loops, and graph radius, density, and diameter. Longer paths contribute more as the number of treatments and loops and the graph radius and diameter increase. CONCLUSION: The contribution of different paths depends on the size and structure of networks, with important implications for assessing the risk of bias and confidence in NMA results.

CINeMA: An approach for assessing confidence in the results of a network meta-analysis.

BACKGROUND: The evaluation of the credibility of results from a meta-analysis has become an important part of the evidence synthesis process. We present a methodological framework to evaluate confidence in the results from network meta-analyses, Confidence in Network Meta-Analysis (CINeMA), when multiple interventions are compared. METHODOLOGY: CINeMA considers 6 domains: (i) within-study bias, (ii) reporting bias, (iii) indirectness, (iv) imprecision, (v) heterogeneity, and (vi) incoherence. Key to judgments about within-study bias and indirectness is the percentage contribution matrix, which shows how much information each study contributes to the results from network meta-analysis. The contribution matrix can easily be computed using a freely available web application. In evaluating imprecision, heterogeneity, and incoherence, we consider the impact of these components of variability in forming clinical decisions. CONCLUSIONS: Via 3 examples, we show that CINeMA improves transparency and avoids the selective use of evidence when forming judgments, thus limiting subjectivity in the process. CINeMA is easy to apply even in large and complicated networks.

CINeMA: Software for semiautomated assessment of the confidence in the results of network meta-analysis.

Network meta-analysis (NMA) compares several interventions that are linked in a network of comparative studies and estimates the relative treatment effects between all treatments, using both direct and indirect evidence. NMA is increasingly used for decision making in health care, however, a user-friendly system to evaluate the confidence that can be placed in the results of NMA is currently lacking. This paper is a tutorial describing the Confidence In Network Meta-Analysis (CINeMA) web application, which is based on the framework developed by Salanti et al (2014, PLOS One, 9, e99682) and refined by Nikolakopoulou et al (2019, bioRxiv). Six domains that affect the level of confidence in the NMA results are considered: (a) within-study bias, (b) reporting bias, (c) indirectness, (d) imprecision, (e) heterogeneity, and (f) incoherence. CINeMA is freely available and open-source and no login is required. In the configuration step users upload their data, produce network plots and define the analysis and effect measure. The dataset should include assessments of study-level risk of bias and judgments on indirectness. CINeMA calls the netmeta routine in R to estimate relative effects and heterogeneity. Users are then guided through a systematic evaluation of the six domains. In this way reviewers assess the level of concerns for each relative treatment effect from NMA as giving rise to "no concerns," "some concerns," or "major concerns" in each of the six domains, which are graphically summarized on the report page for all effect estimates. Finally, judgments across the domains are summarized into a single confidence rating ("high," "moderate," "low," or "very low"). In conclusion, the user-friendly web-based CINeMA platform provides a transparent framework to evaluate evidence from systematic reviews with multiple interventions.

Estimating the contribution of studies in network meta-analysis: paths, flows and streams.

In network meta-analysis, it is important to assess the influence of the limitations or other characteristics of individual studies on the estimates obtained from the network. The proportion contribution matrix, which shows how much each direct treatment effect contributes to each treatment effect estimate from network meta-analysis, is crucial in this context. We use ideas from graph theory to derive the proportion that is contributed by each direct treatment effect. We start with the 'projection' matrix in a two-step network meta-analysis model, called the H matrix, which is analogous to the hat matrix in a linear regression model. We develop a method to translate H entries to proportion contributions based on the observation that the rows of H  can be interpreted as flow networks, where a stream is defined as the composition of a path and its associated flow. We present an algorithm that identifies the flow of evidence in each path and decomposes it into direct comparisons. To illustrate the methodology, we use two published networks of interventions. The first compares no treatment, quinolone antibiotics, non-quinolone antibiotics and antiseptics for underlying eardrum perforations and the second compares 14 antimanic drugs. We believe that this approach is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the proportion contributions of direct evidence from individual studies to network treatment effects.