Twenty years of network meta-analysis: Continuing controversies and recent developments.

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

Transportability of two heart failure trials to a disease registry using individual patient data.

OBJECTIVES: Randomized controlled trials are the gold-standard for determining therapeutic efficacy, but are often unrepresentative of real-world settings. Statistical transportation methods (hereafter transportation) can partially account for these differences, improving trial applicability without breaking randomization. We transported treatment effects from two heart failure (HF) trials to a HF registry. STUDY DESIGN AND SETTING: Individual-patient-level data from two trials (Carvedilol or Metoprolol European Trial (COMET), comparing carvedilol and metoprolol, and digitalis investigation group trial (DIG), comparing digoxin and placebo) and a Scottish HF registry were obtained. The primary end point for both trials was all-cause mortality; composite outcomes were all-cause mortality or hospitalization for COMET and HF-related death or hospitalization for DIG. We performed transportation using regression-based and inverse odds of sampling weights (IOSW) approaches. RESULTS: Registry patients were older, had poorer renal function and received higher-doses of loop-diuretics than trial participants. For each trial, point estimates were similar for the original and IOSW (e.g., DIG composite outcome: OR 0.75 (0.69, 0.82) vs. 0.73 (0.64, 0.83)). Treatment effect estimates were also similar when examining high-risk (0.64 (0.46, 0.89)) and low-risk registry patients (0.73 (0.61, 0.86)). Similar results were obtained using regression-based transportation. CONCLUSION: Regression-based or IOSW approaches can be used to transport trial effect estimates to patients administrative/registry data, with only moderate reductions in precision.

Using individual participant data to improve network meta-analysis projects.

A network meta-analysis combines the evidence from existing randomised trials about the comparative efficacy of multiple treatments. It allows direct and indirect evidence about each comparison to be included in the same analysis, and provides a coherent framework to compare and rank treatments. A traditional network meta-analysis uses aggregate data (eg, treatment effect estimates and standard errors) obtained from publications or trial investigators. An alternative approach is to obtain, check, harmonise and meta-analyse the individual participant data (IPD) from each trial. In this article, we describe potential advantages of IPD for network meta-analysis projects, emphasising five key benefits: (1) improving the quality and scope of information available for inclusion in the meta-analysis, (2) examining and plotting distributions of covariates across trials (eg, for potential effect modifiers), (3) standardising and improving the analysis of each trial, (4) adjusting for prognostic factors to allow a network meta-analysis of conditional treatment effects and (5) including treatment-covariate interactions (effect modifiers) to allow relative treatment effects to vary by participant-level covariate values (eg, age, baseline depression score). A running theme of all these benefits is that they help examine and reduce heterogeneity (differences in the true treatment effect between trials) and inconsistency (differences in the true treatment effect between direct and indirect evidence) in the network. As a consequence, an IPD network meta-analysis has the potential for more precise, reliable and informative results for clinical practice and even allows treatment comparisons to be made for individual patients and targeted populations conditional on their particular characteristics.

Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis.

BACKGROUND: Network meta-analysis (NMA) and indirect comparisons combine aggregate data (AgD) from multiple studies on treatments of interest but may give biased estimates if study populations differ. Population adjustment methods such as multilevel network meta-regression (ML-NMR) aim to reduce bias by adjusting for differences in study populations using individual patient data (IPD) from 1 or more studies under the conditional constancy assumption. A shared effect modifier assumption may also be necessary for identifiability. This article aims to demonstrate how the assumptions made by ML-NMR can be assessed in practice to obtain reliable treatment effect estimates in a target population. METHODS: We apply ML-NMR to a network of evidence on treatments for plaque psoriasis with a mix of IPD and AgD trials reporting ordered categorical outcomes. Relative treatment effects are estimated for each trial population and for 3 external target populations represented by a registry and 2 cohort studies. We examine residual heterogeneity and inconsistency and relax the shared effect modifier assumption for each covariate in turn. RESULTS: Estimated population-average treatment effects were similar across study populations, as differences in the distributions of effect modifiers were small. Better fit was achieved with ML-NMR than with NMA, and uncertainty was reduced by explaining within- and between-study variation. We found little evidence that the conditional constancy or shared effect modifier assumptions were invalid. CONCLUSIONS: ML-NMR extends the NMA framework and addresses issues with previous population adjustment approaches. It coherently synthesizes evidence from IPD and AgD studies in networks of any size while avoiding aggregation bias and noncollapsibility bias, allows for key assumptions to be assessed or relaxed, and can produce estimates relevant to a target population for decision-making. HIGHLIGHTS: Multilevel network meta-regression (ML-NMR) extends the network meta-analysis framework to synthesize evidence from networks of studies providing individual patient data or aggregate data while adjusting for differences in effect modifiers between studies (population adjustment). We apply ML-NMR to a network of treatments for plaque psoriasis with ordered categorical outcomes.We demonstrate for the first time how ML-NMR allows key assumptions to be assessed. We check for violations of conditional constancy of relative effects (such as unobserved effect modifiers) through residual heterogeneity and inconsistency and the shared effect modifier assumption by relaxing this for each covariate in turn.Crucially for decision making, population-adjusted treatment effects can be produced in any relevant target population. We produce population-average estimates for 3 external target populations, represented by the PsoBest registry and the PROSPECT and Chiricozzi 2019 cohort studies.

A systematic review and network meta-analysis of topical pharmacological, oral pharmacological, physical and combined treatments for acne vulgaris.

BACKGROUND: Various treatments for acne vulgaris exist, but little is known about their comparative effectiveness in relation to acne severity. OBJECTIVES: To identify best treatments for mild-to-moderate and moderate-to-severe acne, as determined by clinician-assessed morphological features. METHODS: We undertook a systematic review and network meta-analysis of randomized controlled trials (RCTs) assessing topical pharmacological, oral pharmacological, physical and combined treatments for mild-to-moderate and moderate-to-severe acne, published up to May 2020. Outcomes included percentage change in total lesion count from baseline, treatment discontinuation for any reason, and discontinuation owing to side-effects. Risk of bias was assessed using the Cochrane risk-of-bias tool and bias adjustment models. Effects for treatments with ≥ 50 observations each compared with placebo are reported below. RESULTS: We included 179 RCTs with approximately 35 000 observations across 49 treatment classes. For mild-to-moderate acne, the most effective options for each treatment type were as follows: topical pharmacological - combined retinoid with benzoyl peroxide (BPO) [mean difference 26·16%, 95% credible interval (CrI) 16·75-35·36%]; physical - chemical peels, e.g. salicylic or mandelic acid (39·70%, 95% CrI 12·54-66·78%) and photochemical therapy (combined blue/red light) (35·36%, 95% CrI 17·75-53·08%). Oral pharmacological treatments (e.g. antibiotics, hormonal contraceptives) did not appear to be effective after bias adjustment. BPO and topical retinoids were less well tolerated than placebo. For moderate-to-severe acne, the most effective options for each treatment type were as follows: topical pharmacological - combined retinoid with lincosamide (clindamycin) (44·43%, 95% CrI 29·20-60·02%); oral pharmacological - isotretinoin of total cumulative dose ≥ 120 mg kg-1 per single course (58·09%, 95% CrI 36·99-79·29%); physical - photodynamic therapy (light therapy enhanced by a photosensitizing chemical) (40·45%, 95% CrI 26·17-54·11%); combined - BPO with topical retinoid and oral tetracycline (43·53%, 95% CrI 29·49-57·70%). Topical retinoids and oral tetracyclines were less well tolerated than placebo. The quality of included RCTs was moderate to very low, with evidence of inconsistency between direct and indirect evidence. Uncertainty in findings was high, in particular for chemical peels, photochemical therapy and photodynamic therapy. However, conclusions were robust to potential bias in the evidence. CONCLUSIONS: Topical pharmacological treatment combinations, chemical peels and photochemical therapy were most effective for mild-to-moderate acne. Topical pharmacological treatment combinations, oral antibiotics combined with topical pharmacological treatments, oral isotretinoin and photodynamic therapy were most effective for moderate-to-severe acne. Further research is warranted for chemical peels, photochemical therapy and photodynamic therapy for which evidence was more limited. What is already known about this topic? Acne vulgaris is the eighth most common disease globally. Several topical, oral, physical and combined treatments for acne vulgaris exist. Network meta-analysis (NMA) synthesizes direct and indirect evidence and allows simultaneous inference for all treatments forming an evidence network. Previous NMAs have assessed a limited range of treatments for acne vulgaris and have not evaluated effectiveness of treatments for moderate-to-severe acne. What does this study add? For mild-to-moderate acne, topical treatment combinations, chemical peels, and photochemical therapy (combined blue/red light; blue light) are most effective. For moderate-to-severe acne, topical treatment combinations, oral antibiotics combined with topical treatments, oral isotretinoin and photodynamic therapy (light therapy enhanced by a photosensitizing chemical) are most effective. Based on these findings, along with further clinical and cost-effectiveness considerations, National Institute for Health and Care Excellence (NICE) guidance recommends, as first-line treatments, fixed topical treatment combinations for mild-to-moderate acne and fixed topical treatment combinations, or oral tetracyclines combined with topical treatments, for moderate-to-severe acne.

Improving the Estimation of Subgroup Effects for Clinical Trial Participants with Multimorbidity by Incorporating Drug Class-Level Information in Bayesian Hierarchical Models: A Simulation Study.

BACKGROUND: There is limited guidance for using common drug therapies in the context of multimorbidity. In part, this is because their effectiveness for patients with specific comorbidities cannot easily be established using subgroup analyses in clinical trials. Here, we use simulations to explore the feasibility and implications of concurrently estimating effects of related drug treatments in patients with multimorbidity by partially pooling subgroup efficacy estimates across trials. METHODS: We performed simulations based on the characteristics of 161 real clinical trials of noninsulin glucose-lowering drugs for diabetes, estimating subgroup effects for patients with a hypothetical comorbidity across related trials in different scenarios using Bayesian hierarchical generalized linear models. We structured models according to an established ontology-the World Health Organization Anatomic Chemical Therapeutic Classifications-allowing us to nest all trials within drugs and all drugs within anatomic chemical therapeutic classes, with effects partially pooled at each level of the hierarchy. In a range of scenarios, we compared the performance of this model to random effects meta-analyses of all drugs individually. RESULTS: Hierarchical, ontology-based Bayesian models were unbiased and accurately recovered simulated comorbidity-drug interactions. Compared with single-drug meta-analyses, they offered a relative increase in precision of up to 250% in some scenarios because of information sharing across the hierarchy. Because of the relative precision of the approaches, a large proportion of small subgroup effects was detectable only using the hierarchical model. CONCLUSIONS: By assuming that similar drugs may have similar subgroup effects, Bayesian hierarchical models based on structures defined by existing ontologies can be used to improve the precision of treatment efficacy estimates in patients with multimorbidity, with potential implications for clinical decision making.

Comparative clinical effectiveness and safety of tobacco cessation pharmacotherapies and electronic cigarettes: a systematic review and network meta-analysis of randomized controlled trials.

AIM: To determine how varenicline, bupropion, nicotine replacement therapy (NRT) and electronic cigarettes compare with respect to their clinical effectiveness and safety. METHOD: Systematic reviews and Bayesian network meta-analyses of randomized controlled trials, in any setting, of varenicline, bupropion, NRT and e-cigarettes (in high, standard and low doses, alone or in combination) in adult smokers and smokeless tobacco users with follow-up duration of 24 weeks or greater (effectiveness) or any duration (safety). Nine databases were searched until 19 February 2019. Primary outcomes were sustained tobacco abstinence and serious adverse events (SAEs). We estimated odds ratios (ORs) and treatment rankings and conducted meta-regression to explore covariates. RESULTS: We identified 363 trials for effectiveness and 355 for safety. Most monotherapies and combination therapies were more effective than placebo at helping participants to achieve sustained abstinence; the most effective of these, estimated with some imprecision, were varenicline standard [OR = 2.83, 95% credible interval (CrI) = 2.34-3.39] and varenicline standard + NRT standard (OR = 5.75, 95% CrI = 2.27-14.88). Estimates were higher in smokers receiving counselling than in those without and in studies with higher baseline nicotine dependence scores than in those with lower scores. Varenicline standard + NRT standard showed a high probability of being ranked best or second-best. For safety, only bupropion at standard dose increased the odds of experiencing SAEs compared with placebo (OR = 1.27, 95% CrI = 1.04-1.58), and we found no evidence of effect modification. CONCLUSIONS: Most tobacco cessation monotherapies and combination therapies are more effective than placebo at helping participants to achieve sustained abstinence, with varenicline appearing to be most effective based on current evidence. There does not appear to be strong evidence of associations between most tobacco cessation pharmacotherapies and adverse events; however, the data are limited and there is a need for improved reporting of safety data.

Short-Term Efficacy of Biologic Therapies in Moderate-to-Severe Plaque Psoriasis: A Systematic Literature Review and an Enhanced Multinomial Network Meta-Analysis.

INTRODUCTION: Many targeted, systemic therapies have been developed for treatment of moderate-to-severe psoriasis (PsO). A network meta-analysis (NMA) allows for comparison between treatments not directly compared in randomized controlled trials (RCT). This study's objective was to compare the short-term (10-16 weeks) clinical efficacy according to the Psoriasis Area and Severity Index (PASI) among approved biologic treatments for moderate-to-severe PsO using a novel (enhanced) NMA model. METHODS: A systematic literature review (SLR) of RCTs for patients with moderate-to-severe PsO was conducted. English publications in MEDLINE, Embase, and The Cochrane Library up to March 2019 were searched. An enhanced multinomial Bayesian NMA was performed to simultaneously adjust for baseline risk and utilize the conditional nature of the PASI (50, 75, 90, and 100) levels. The model relaxes typical constraints that all treatments must have the same ranks across PASI levels. RESULTS: The SLR resulted in 319 relevant publications, of which 72 publications from 73 RCTs reporting 10- to 16-week data for at least one PASI response level (30,314 total patients) were included. Interleukin (IL) inhibitors (risankizumab, ixekizumab, brodalumab, secukinumab, and guselkumab) were the best performing treatments for achieving all PASI levels. Etanercept was outperformed by the other subcutaneous tumor necrosis factor α inhibitors. Application of an enhanced NMA model that allowed treatment rankings to differ by PASI level tested the robustness of results of previous NMAs in PsO. CONCLUSION: The results of this model confirmed that IL inhibitors are likely the best short-term treatment choices for improving all PASI levels.

Cost-Effectiveness Analysis of Smoking Cessation Interventions in the United Kingdom Accounting for Major Neuropsychiatric Adverse Events.

OBJECTIVES: Smoking is a leading cause of death worldwide. Cessation aids include varenicline, bupropion, nicotine replacement therapy (NRT), and e-cigarettes at various doses (low, standard and high) and used alone or in combination with each other. Previous cost-effectiveness analyses have not fully accounted for adverse effects nor compared all cessation aids. The objective was to determine the relative cost-effectiveness of cessation aids in the United Kingdom. METHODS: An established Markov cohort model was adapted to incorporate health outcomes and costs due to depression and self-harm associated with cessation aids, alongside other health events. Relative efficacy in terms of abstinence and major adverse neuropsychiatric events was informed by a systematic review and network meta-analysis. Base case results are reported for UK-licensed interventions only. Two sensitivity analyses are reported, one including unlicensed interventions and another comparing all cessation aids but removing the impact of depression and self-harm. The sensitivity of conclusions to model inputs was assessed by calculating the expected value of partial perfect information. RESULTS: When limited to UK-licensed interventions, varenicline standard-dose and NRT standard-dose were most cost-effective. Including unlicensed interventions, e-cigarette low-dose appeared most cost-effective followed by varenicline standard-dose + bupropion standard-dose combined. When the impact of depression and self-harm was excluded, varenicline standard-dose + NRT standard-dose was most cost-effective, followed by varenicline low-dose + NRT standard-dose. CONCLUSION: Although found to be most cost-effective, combined therapy is currently unlicensed in the United Kingdom and the safety of e-cigarettes remains uncertain. The value-of-information analysis suggested researchers should continue to investigate the long-term effectiveness and safety outcomes of e-cigarettes in studies with active comparators.

Joint Modeling of Individual Trajectories, Within-Individual Variability, and a Later Outcome: Systolic Blood Pressure Through Childhood and Left Ventricular Mass in Early Adulthood.

Within-individual variability of repeatedly measured exposures might predict later outcomes (e.g., blood pressure (BP) variability (BPV) is an independent cardiovascular risk factor above and beyond mean BP). Because 2-stage methods, known to introduce bias, are typically used to investigate such associations, we introduce a joint modeling approach, examining associations of mean BP and BPV across childhood with left ventricular mass (indexed to height; LVMI) in early adulthood with data (collected 1990-2011) from the UK Avon Longitudinal Study of Parents and Children cohort. Using multilevel models, we allowed BPV to vary between individuals (a "random effect") as well as to depend on covariates (allowing for heteroskedasticity). We further distinguished within-clinic variability ("measurement error") from visit-to-visit BPV. BPV was predicted to be greater at older ages, at higher body weights, and in female participants and was positively correlated with mean BP. BPV had a weak positive association with LVMI (10% increase in within-individual BP variance was predicted to increase LVMI by 0.21%, 95% credible interval: -0.23, 0.69), but this association became negative (-0.78%, 95% credible interval: -2.54, 0.22) once the effect of mean BP on LVMI was adjusted for. This joint modeling approach offers a flexible method of relating repeatedly measured exposures to later outcomes.

Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study.

Standard network meta-analysis and indirect comparisons combine aggregate data from multiple studies on treatments of interest, assuming that any factors that interact with treatment effects (effect modifiers) are balanced across populations. Population adjustment methods such as multilevel network meta-regression (ML-NMR), matching-adjusted indirect comparison (MAIC), and simulated treatment comparison (STC) relax this assumption using individual patient data from one or more studies, and are becoming increasingly prevalent in health technology appraisals and the applied literature. Motivated by an applied example and two recent reviews of applications, we undertook an extensive simulation study to assess the performance of these methods in a range of scenarios under various failures of assumptions. We investigated the impact of varying sample size, missing effect modifiers, strength of effect modification and validity of the shared effect modifier assumption, validity of extrapolation and varying between-study overlap, and different covariate distributions and correlations. ML-NMR and STC performed similarly, eliminating bias when the requisite assumptions were met. Serious concerns are raised for MAIC, which performed poorly in nearly all simulation scenarios and may even increase bias compared with standard indirect comparisons. All methods incur bias when an effect modifier is missing, highlighting the necessity of careful selection of potential effect modifiers prior to analysis. When all effect modifiers are included, ML-NMR and STC are robust techniques for population adjustment. ML-NMR offers additional advantages over MAIC and STC, including extending to larger treatment networks and producing estimates in any target population, making this an attractive choice in a variety of scenarios.

Multilevel network meta-regression for population-adjusted treatment comparisons.

Standard network meta-analysis (NMA) and indirect comparisons combine aggregate data from multiple studies on treatments of interest, assuming that any effect modifiers are balanced across populations. Population adjustment methods relax this assumption using individual patient data from one or more studies. However, current matching-adjusted indirect comparison and simulated treatment comparison methods are limited to pairwise indirect comparisons and cannot predict into a specified target population. Existing meta-regression approaches incur aggregation bias. We propose a new method extending the standard NMA framework. An individual level regression model is defined, and aggregate data are fitted by integrating over the covariate distribution to form the likelihood. Motivated by the complexity of the closed form integration, we propose a general numerical approach using quasi-Monte-Carlo integration. Covariate correlation structures are accounted for by using copulas. Crucially for decision making, comparisons may be provided in any target population with a given covariate distribution. We illustrate the method with a network of plaque psoriasis treatments. Estimated population-average treatment effects are similar across study populations, as differences in the distributions of effect modifiers are small. A better fit is achieved than a random effects NMA, uncertainty is substantially reduced by explaining within- and between-study variation, and estimates are more interpretable.

Equivalence of entropy balancing and the method of moments for matching-adjusted indirect comparison.

Indirect comparisons are used to obtain estimates of relative effectiveness between two treatments that have not been compared in the same randomized controlled trial, but have instead been compared against a common comparator in separate trials. Standard indirect comparisons use only aggregate data, under the assumption that there are no differences in effect-modifying variables between the trial populations. Population-adjusted indirect comparisons aim to relax this assumption by using individual patient data (IPD) from one trial to adjust for differences in effect modifiers between populations. At present, the most commonly used approach is matching-adjusted indirect comparison (MAIC), where weights are estimated that match the covariate distributions of the reweighted IPD to the aggregate trial. MAIC was originally proposed using the method of moments to estimate the weights, but more recently entropy balancing has been proposed as an alternative. Entropy balancing has an additional "optimality" property ensuring that the weights are as uniform as possible, reducing the standard error of the estimates. In this brief method note, we show that MAIC weights are mathematically identical whether estimated using entropy balancing or the method of moments. Importantly, this means that the standard MAIC (based on the method of moments) also enjoys the "optimality" property. Moreover, the additional flexibility of entropy balancing suggests several interesting avenues for further research, such as combining population adjustment via MAIC with adjustments for treatment switching or nonparametric covariate adjustment.

Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals.

OBJECTIVES: Indirect comparisons via a common comparator (anchored comparisons) are commonly used in health technology assessment. However, common comparators may not be available, or the comparison may be biased due to differences in effect modifiers between the included studies. Recently proposed population adjustment methods aim to adjust for differences between study populations in the situation where individual patient data are available from at least one study, but not all studies. They can also be used when there is no common comparator or for single-arm studies (unanchored comparisons). We aim to characterise the use of population adjustment methods in technology appraisals (TAs) submitted to the United Kingdom National Institute for Health and Care Excellence (NICE). METHODS: We reviewed NICE TAs published between 01/01/2010 and 20/04/2018. RESULTS: Population adjustment methods were used in 7 percent (18/268) of TAs. Most applications used unanchored comparisons (89 percent, 16/18), and were in oncology (83 percent, 15/18). Methods used included matching-adjusted indirect comparisons (89 percent, 16/18) and simulated treatment comparisons (17 percent, 3/18). Covariates were included based on: availability, expert opinion, effective sample size, statistical significance, or cross-validation. Larger treatment networks were commonplace (56 percent, 10/18), but current methods cannot account for this. Appraisal committees received results of population-adjusted analyses with caution and typically looked for greater cost effectiveness to minimise decision risk. CONCLUSIONS: Population adjustment methods are becoming increasingly common in NICE TAs, although their impact on decisions has been limited to date. Further research is needed to improve upon current methods, and to investigate their properties in simulation studies.

Threshold Analysis as an Alternative to GRADE for Assessing Confidence in Guideline Recommendations Based on Network Meta-analyses.

Guideline development requires the synthesis of evidence on several treatments of interest, typically by using network meta-analysis (NMA). Because treatment effects may be estimated imprecisely or be based on evidence lacking internal or external validity, guideline developers must assess the robustness of recommendations made on the basis of the NMA to potential limitations in the evidence. Such limitations arise because the observed estimates differ from the true effects of interest, for example, because of study biases, sampling variation, or issues of relevance. The widely used GRADE (Grading of Recommendations Assessment, Development and Evaluation) framework aims to assess the quality of evidence supporting a recommendation by using a structured series of qualitative judgments. This article argues that GRADE approaches proposed for NMA are insufficient for the purposes of guideline development, because the influence of the evidence on the final recommendation is not taken into account. It outlines threshold analysis as an alternative approach, demonstrating the method with 2 examples of clinical guidelines from the National Institute for Health and Care Excellence (NICE) in the United Kingdom. Threshold analysis quantifies precisely how much the evidence could change (for any reason, such as potential biases, or simply sampling variation) before the recommendation changes, and what the revised recommendation would be. If it is judged that the evidence could not plausibly change by more than this amount, then the recommendation is considered robust; otherwise, it is sensitive to plausible changes in the evidence. In this manner, threshold analysis directly informs decision makers and guideline developers of the robustness of treatment recommendations.

Sensitivity of treatment recommendations to bias in network meta-analysis.

Network meta-analysis (NMA) pools evidence on multiple treatments to estimate relative treatment effects. Included studies are typically assessed for risk of bias; however, this provides no indication of the impact of potential bias on a decision based on the NMA. We propose methods to derive bias adjustment thresholds which measure the smallest changes to the data that result in a change of treatment decision. The methods use efficient matrix operations and can be applied to explore the consequences of bias in individual studies or aggregate treatment contrasts, in both fixed and random-effects NMA models. Complex models with multiple types of data input are handled by using an approximation to the hypothetical aggregate likelihood. The methods are illustrated with a simple NMA of thrombolytic treatments and a more complex example comparing social anxiety interventions. An accompanying R package is provided.

Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal.

Standard methods for indirect comparisons and network meta-analysis are based on aggregate data, with the key assumption that there is no difference between the trials in the distribution of effect-modifying variables. Methods which relax this assumption are becoming increasingly common for submissions to reimbursement agencies, such as the National Institute for Health and Care Excellence (NICE). These methods use individual patient data from a subset of trials to form population-adjusted indirect comparisons between treatments, in a specific target population. Recently proposed population adjustment methods include the Matching-Adjusted Indirect Comparison (MAIC) and the Simulated Treatment Comparison (STC). Despite increasing popularity, MAIC and STC remain largely untested. Furthermore, there is a lack of clarity about exactly how and when they should be applied in practice, and even whether the results are relevant to the decision problem. There is therefore a real and present risk that the assumptions being made in one submission to a reimbursement agency are fundamentally different to-or even incompatible with-the assumptions being made in another for the same indication. We describe the assumptions required for population-adjusted indirect comparisons, and demonstrate how these may be used to generate comparisons in any given target population. We distinguish between anchored and unanchored comparisons according to whether a common comparator arm is used or not. Unanchored comparisons make much stronger assumptions, which are widely regarded as infeasible. We provide recommendations on how and when population adjustment methods should be used, and the supporting analyses that are required to provide statistically valid, clinically meaningful, transparent and consistent results for the purposes of health technology appraisal. Simulation studies are needed to examine the properties of population adjustment methods and their robustness to breakdown of assumptions.