Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR).

BACKGROUND: Health interventions fall along a spectrum from simple to more complex. There is wide interest in methods for reviewing 'complex interventions', but few transparent approaches for assessing intervention complexity in systematic reviews. Such assessments may assist review authors in, for example, systematically describing interventions and developing logic models. This paper describes the development and application of the intervention Complexity Assessment Tool for Systematic Reviews (iCAT_SR), a new tool to assess and categorise levels of intervention complexity in systematic reviews. METHODS: We developed the iCAT_SR by adapting and extending an existing complexity assessment tool for randomized trials. We undertook this adaptation using a consensus approach in which possible complexity dimensions were circulated for feedback to a panel of methodologists with expertise in complex interventions and systematic reviews. Based on these inputs, we developed a draft version of the tool. We then invited a second round of feedback from the panel and a wider group of systematic reviewers. This informed further refinement of the tool. RESULTS: The tool comprises ten dimensions: (1) the number of active components in the intervention; (2) the number of behaviours of recipients to which the intervention is directed; (3) the range and number of organizational levels targeted by the intervention; (4) the degree of tailoring intended or flexibility permitted across sites or individuals in applying or implementing the intervention; (5) the level of skill required by those delivering the intervention; (6) the level of skill required by those receiving the intervention; (7) the degree of interaction between intervention components; (8) the degree to which the effects of the intervention are context dependent; (9) the degree to which the effects of the interventions are changed by recipient or provider factors; (10) and the nature of the causal pathway between intervention and outcome. Dimensions 1-6 are considered 'core' dimensions. Dimensions 7-10 are optional and may not be useful for all interventions. CONCLUSIONS: The iCAT_SR tool facilitates more in-depth, systematic assessment of the complexity of interventions in systematic reviews and can assist in undertaking reviews and interpreting review findings. Further testing of the tool is now needed.

Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations.

OBJECTIVES: Model specification-what adjusting variables are analytically modeled-may influence results of observational associations. We present a standardized approach to quantify the variability of results obtained with choices of adjustments called the "vibration of effects" (VoE). STUDY DESIGN AND SETTING: We estimated the VoE for 417 clinical, environmental, and physiological variables in association with all-cause mortality using National Health and Nutrition Examination Survey data. We selected 13 variables as adjustment covariates and computed 8,192 Cox models for each of 417 variables' associations with all-cause mortality. RESULTS: We present the VoE by assessing the variance of the effect size and in the -log10(P-value) obtained by different combinations of adjustments. We present whether there are multimodality patterns in effect sizes and P-values and the trajectory of results with increasing adjustments. For 31% of the 417 variables, we observed a Janus effect, with the effect being in opposite direction in the 99th versus the 1st percentile of analyses. For example, the vitamin E variant α-tocopherol had a VoE that indicated higher and lower risk for mortality. CONCLUSION: Estimating VoE offers empirical estimates of associations are under different model specifications. When VoE is large, claims for observational associations should be very cautious.

Testing the PRISMA-Equity 2012 reporting guideline: the perspectives of systematic review authors.

Reporting guidelines can be used to encourage standardised and comprehensive reporting of health research. In light of the global commitment to health equity, we have previously developed and published a reporting guideline for equity-focused systematic reviews (PRISMA-E 2012). The objectives of this study were to explore the utility of the equity extension items included in PRISMA-E 2012 from a systematic review author perspective, including facilitators and barriers to its use. This will assist in designing dissemination and knowledge translation strategies. We conducted a survey of systematic review authors to expose them to the new items in PRISMA-E 2012, establish the extent to which they had historically addressed those items in their own reviews, and gather feedback on the usefulness of the new items. Data were analysed using Microsoft Excel 2008 and Stata (version 11.2 for Mac). Of 151 respondents completing the survey, 18.5% (95% CI: 12.7% to 25.7%) had not heard of the PRISMA statement before, although 83.4% (95% CI: 77.5% to 89.3%) indicated that they plan to use PRISMA-E 2012 in the future, depending on the focus of their review. Most (68.9%; 95% CI: 60.8% to 76.2%) thought that using PRISMA-E 2012 would lead them to conduct their reviews differently. Important facilitators to using PRISMA-E 2012 identified by respondents were journal endorsement and incorporation of the elements of the guideline into systematic review software. Barriers identified were lack of time, word limits and the availability of equity data in primary research. This study has been the first to 'road-test' the new PRISMA-E 2012 reporting guideline and the findings are encouraging. They confirm the acceptability and potential utility of the guideline to assist review authors in reporting on equity in their reviews. The uptake and impact of PRISMA-E 2012 over time on design, conduct and reporting of primary research and systematic reviews should continue to be examined.

Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making.

Assessment of applicability is an essential part of the systematic review process. In the context of systematic reviews of the effects of interventions, applicability is an assessment of whether the findings of a review can be applied in a particular context or population. For more complex interventions, assessing applicability can be challenging because of greater diversity of, and interactions within and between, the intended population, intervention components, comparison conditions, and outcomes as well as a range of further considerations related to intervention context and theoretical basis. We recommend that review authors plan and conduct analyses to explain variations in effect and answer questions about mechanisms of action and influence of different settings, contexts, and populations. We also recommend that review authors provide rich descriptions of the setting, implementation details, resource use, and contexts of included studies and assess applicability for at least one target population, setting, and context. This should facilitate applicability assessments by end users. Consensus on terminology is needed and guidance should be developed for the synthesis of implementation information within reviews as well as the documentation of applicability judgments by review authors.