Robust variance estimation in small meta-analysis with the standardized mean difference.

Conventional random-effects models in meta-analysis rely on large sample approximations instead of exact small sample results. While random-effects methods produce efficient estimates and confidence intervals for the summary effect have correct coverage when the number of studies is sufficiently large, we demonstrate that conventional methods result in confidence intervals that are not wide enough when the number of studies is small, depending on the configuration of sample sizes across studies, the degree of true heterogeneity and number of studies. We introduce two alternative variance estimators with better small sample properties, investigate degrees of freedom adjustments for computing confidence intervals, and study their effectiveness via simulation studies.

ABkPowerCalculator: An App to Compute Power for Balanced (AB)k Single Case Experimental Designs.

Single case experimental designs are an important research design in behavioral and medical research. Although there are design standards prescribed by the What Works Clearinghouse for single case experimental designs, these standards do not include statistically derived power computations. Recently we derived the equations for computing power for (AB)k designs. However, these computations and the software code in R may not be accessible to applied researchers who are most likely to want to compute power for their studies. Therefore, we have developed an (AB)k power calculator Shiny App (https://abkpowercalculator.shinyapps.io/ABkpowercalculator/) that researchers can use with no software training. These power computations assume that the researcher would be interested in fitting multilevel models with autocorrelations or conduct similar analyses. The purpose of this software contribution is to briefly explain how power is derived for balanced (AB)k designs and to elaborate on how to use the Shiny App. The app works well on not just computers but mobile phones without installing the R program. We believe this can be a valuable tool for practitioners and applied researchers who want to plan their single case studies with sufficient power to detect appropriate effect sizes.

Effect sizes in ANCOVA and difference-in-differences designs.

It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate and-in non-randomized designs-its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thereby making it comparable to other interventions and studies. Curiously, the estimation of this effect size, including covariate adjustment, has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.

Meta-analyzing individual participant data from studies with complex survey designs: A tutorial on using the two-stage approach for data from educational large-scale assessments.

Descriptive analyses of socially important or theoretically interesting phenomena and trends are a vital component of research in the behavioral, social, economic, and health sciences. Such analyses yield reliable results when using representative individual participant data (IPD) from studies with complex survey designs, including educational large-scale assessments (ELSAs) or social, health, and economic survey and panel studies. The meta-analytic integration of these results offers unique and novel research opportunities to provide strong empirical evidence of the consistency and generalizability of important phenomena and trends. Using ELSAs as an example, this tutorial offers methodological guidance on how to use the two-stage approach to IPD meta-analysis to account for the statistical challenges of complex survey designs (e.g., sampling weights, clustered and missing IPD), first, to conduct descriptive analyses (Stage 1), and second, to integrate results with three-level meta-analytic and meta-regression models to take into account dependencies among effect sizes (Stage 2). The two-stage approach is illustrated with IPD on reading achievement from the Programme for International Student Assessment (PISA). We demonstrate how to analyze and integrate standardized mean differences (e.g., gender differences), correlations (e.g., with students' socioeconomic status [SES]), and interactions between individual characteristics at the participant level (e.g., the interaction between gender and SES) across several PISA cycles. All the datafiles and R scripts we used are available online. Because complex social, health, or economic survey and panel studies share many methodological features with ELSAs, the guidance offered in this tutorial is also helpful for synthesizing research evidence from these studies.

Power analysis for single-case designs: Computations for (AB)k designs.

Currently, the design standards for single-case experimental designs (SCEDs) are based on validity considerations as prescribed by the What Works Clearinghouse. However, there is a need for design considerations such as power based on statistical analyses. We compute and derive power using computations for (AB)k designs with multiple cases which are common in SCEDs. Our computations show that effect size has the maximum impact on power followed by the number of subjects and then the number of phase reversals. An effect size of 0.75 or higher, at least one set of phase reversals (i.e., where k > 1), and at least three subjects showed high power. The latter two conditions agree with current standards about either having at least an ABAB design or a multiple baseline design with three subjects to meet design standards. An effect size of 0.75 or higher is not uncommon in SCEDs either. Autocorrelations, the number of time-points per phase, and intraclass correlations had a smaller but non-negligible impact on power. In sum, power analyses in the present study show that conditions to meet power requirements are not unreasonable in SCEDs. The software code to compute power is available on GitHub for the use of the reader.

Childhood Obesity Evidence Base Project: A Rationale for Taxonomic versus Conventional Meta-Analysis.

Introduction: There is a great need for analytic techniques that allow for the synthesis of learning across seemingly idiosyncratic interventions. Objectives: The primary objective of this paper is to introduce taxonomic meta-analysis and explain how it is different from conventional meta-analysis. Results: Conventional meta-analysis has previously been used to examine the effectiveness of childhood obesity prevention interventions. However, these tend to examine narrowly defined sections of obesity prevention initiatives, and as such, do not allow the field to draw conclusions across settings, participants, or subjects. Compared with conventional meta-analysis, taxonomic meta-analysis widens the aperture of what can be examined to synthesize evidence across interventions with diverse topics, goals, research designs, and settings. A component approach is employed to examine interventions at the level of their essential features or activities to identify the concrete aspects of interventions that are used (intervention components), characteristics of the intended populations (target population or intended recipient characteristics), and facets of the environments in which they operate (contextual elements), and the relationship of these components to effect size. In addition, compared with conventional meta-analysis methods, taxonomic meta-analyses can include the results of natural experiments, policy initiatives, program implementation efforts and highly controlled experiments (as examples) regardless of the design of the report being analyzed as long as the intended outcome is the same. It also characterizes the domain of interventions that have been studied. Conclusion: Taxonomic meta-analysis can be a powerful tool for summarizing the evidence that exists and for generating hypotheses that are worthy of more rigorous testing.

Childhood Obesity Evidence Base Project: Methods for Taxonomy Development for Application in Taxonomic Meta-Analysis.

Meta-analysis has been used to examine the effectiveness of childhood obesity prevention efforts, yet traditional conventional meta-analytic methods restrict the kinds of studies included, and either narrowly define mechanisms and agents of change, or examine the effectiveness of whole interventions as opposed to the specific actions that comprise interventions. Taxonomic meta-analytic methods widen the aperture of what can be included in a meta-analysis data set, allowing for inclusion of many types of interventions and study designs. The National Collaborative on Childhood Obesity Research Childhood Obesity Evidence Base (COEB) project focuses on interventions intended to prevent childhood obesity in children 2-5 years old who have an outcome measure of BMI. The COEB created taxonomies, anchored in the Social Ecological Model, which catalog specific outcomes, intervention components, intended recipients, and contexts of policies, initiatives, and interventions conducted at the individual, interpersonal, organizational, community, and societal level. Taxonomies were created by discovery from the literature itself using grounded theory. This article describes the process used for a novel taxonomic meta-analysis of childhood obesity prevention studies between the years 2010 and 2019. This method can be applied to other areas of research, including obesity prevention in additional populations.

Childhood Obesity Evidence Base Project: A Systematic Review and Meta-Analysis of a New Taxonomy of Intervention Components to Improve Weight Status in Children 2-5 Years of Age, 2005-2019.

Objective: To evaluate the efficacy of childhood obesity interventions and conduct a taxonomy of intervention components that are most effective in changing obesity-related health outcomes in children 2-5 years of age. Methods: Comprehensive searches located 51 studies from 18,335 unique records. Eligible studies: (1) assessed children aged 2-5, living in the United States; (2) evaluated an intervention to improve weight status; (3) identified a same-aged comparison group; (4) measured BMI; and (5) were available between January 2005 and August 2019. Coders extracted study, sample, and intervention characteristics. Effect sizes [ESs; and 95% confidence intervals (CIs)] were calculated by using random-effects models. Meta-regression was used to determine which intervention components explain variability in ESs. Results: Included were 51 studies evaluating 58 interventions (N = 29,085; mean age = 4 years; 50% girls). Relative to controls, children receiving an intervention had a lower BMI at the end of the intervention (g = 0.10, 95% CI = 0.02-0.18; k = 55) and at the last follow-up (g = 0.17, 95% CI = 0.04-0.30; k = 14; range = 18-143 weeks). Three intervention components moderated efficacy: engage caregivers in praise/encouragement for positive health-related behavior; provide education about the importance of screen time reduction to caregivers; and engage pediatricians/health care providers. Conclusions: Early childhood obesity interventions are effective in reducing BMI in preschool children. Our findings suggest that facilitating caregiver education about the importance of screen time reduction may be an important strategy in reducing early childhood obesity.

Assessing heterogeneity and power in replications of psychological experiments.

In this study, we reanalyze recent empirical research on replication from a meta-analytic perspective. We argue that there are different ways to define "replication failure," and that analyses can focus on exploring variation among replication studies or assess whether their results contradict the findings of the original study. We apply this framework to a set of psychological findings that have been replicated and assess the sensitivity of these analyses. We find that tests for replication that involve only a single replication study are almost always severely underpowered. Among the 40 findings for which ensembles of multisite direct replications were conducted, we find that between 11 and 17 (28% to 43%) ensembles produced heterogeneous effects, depending on how replication is defined. This heterogeneity could not be completely explained by moderators documented by replication research programs. We also find that these ensembles were not always well-powered to detect potentially meaningful values of heterogeneity. Finally, we identify several discrepancies between the results of original studies and the distribution of effects found by multisite replications but note that these analyses also have low power. We conclude by arguing that efforts to assess replication would benefit from further methodological work on designing replication studies to ensure analyses are sufficiently sensitive. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Consistency of effects is important in replication: Rejoinder to Mathur and VanderWeele (2019).

In this rejoinder, we discuss Mathur and VanderWeele's response to our article, "Statistical Analyses for Studying Replication: Meta-Analytic Perspectives," which appears in this current issue. We attempt to clarify a point of confusion regarding the inclusion of an original study in an analysis of replication, and the potential impact of publication bias. We then discuss the methods used by Mathur and VanderWeele to conduct an alternative analysis of the Gambler's Fallacy example from our article. We highlight that there are some potential statistical and conceptual differences to their approach compared to what we propose in our article. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses.

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Statistical analyses for studying replication: Meta-analytic perspectives.

Formal empirical assessments of replication have recently become more prominent in several areas of science, including psychology. These assessments have used different statistical approaches to determine if a finding has been replicated. The purpose of this article is to provide several alternative conceptual frameworks that lead to different statistical analyses to test hypotheses about replication. All of these analyses are based on statistical methods used in meta-analysis. The differences among the methods described involve whether the burden of proof is placed on replication or nonreplication, whether replication is exact or allows for a small amount of "negligible heterogeneity," and whether the studies observed are assumed to be fixed (constituting the entire body of relevant evidence) or are a sample from a universe of possibly relevant studies. The statistical power of each of these tests is computed and shown to be low in many cases, raising issues of the interpretability of tests for replication. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Estimation of Population Average Treatment Effects in the FIRST Trial: Application of a Propensity Score-Based Stratification Approach.

OBJECTIVE/STUDY QUESTION: To estimate and compare sample average treatment effects (SATE) and population average treatment effects (PATE) of a resident duty hour policy change on patient and resident outcomes using data from the Flexibility in Duty Hour Requirements for Surgical Trainees Trial ("FIRST Trial"). DATA SOURCES/STUDY SETTING: Secondary data from the National Surgical Quality Improvement Program and the FIRST Trial (2014-2015). STUDY DESIGN: The FIRST Trial was a cluster-randomized pragmatic noninferiority trial designed to evaluate the effects of a resident work hour policy change to permit greater flexibility in scheduling on patient and resident outcomes. We estimated hierarchical logistic regression models to estimate the SATE of a policy change on outcomes within an intent-to-treat framework. Propensity score-based poststratification was used to estimate PATE. DATA COLLECTION/EXTRACTION METHODS: This study was a secondary analysis of previously collected data. PRINCIPAL FINDINGS: Although SATE estimates suggested noninferiority of outcomes under flexible duty hour policy versus standard policy, the noninferiority of a policy change was inconclusively noninferior based on PATE estimates due to imprecision. CONCLUSIONS: Propensity score-based poststratification can be valuable tools to address trial generalizability but may yield imprecise estimates of PATE when sparse strata exist.

Correction to: Estimating effect size when there is clustering in one treatment group.

Equation (26) is formatted incorrectly in the pdf version. It should appear as follows.

Bayesian unknown change-point models to investigate immediacy in single case designs.

Although immediacy is one of the necessary criteria to show strong evidence of a causal relation in single case designs (SCDs), no inferential statistical tool is currently used to demonstrate it. We propose a Bayesian unknown change-point model to investigate and quantify immediacy in SCD analysis. Unlike visual analysis that considers only 3-5 observations in consecutive phases to investigate immediacy, this model considers all data points. Immediacy is indicated when the posterior distribution of the unknown change-point is narrow around the true value of the change-point. This model can accommodate delayed effects. Monte Carlo simulation for a 2-phase design shows that the posterior standard deviations of the change-points decrease with increase in standardized mean difference between phases and decrease in test length. This method is illustrated with real data. (PsycINFO Database Record

Plausibility and influence in selection models: A comment on Citkowicz and Vevea (2017).

I discuss how methods that adjust for publication selection involve implicit or explicit selection models. Such models describe the relation between the studies conducted and those actually observed. I argue that the evaluation of selection models should include an evaluation of the plausibility of the empirical implications of that model. This includes how many studies would have had to exist to yield the observed sample of studies. I also argue that the amount of influence that one or a small number of studies might have on the overall results is also important to understand. (PsycINFO Database Record

The American Psychological Association Task Force assessment of violent video games: Science in the service of public interest.

A task force of experts was convened by the American Psychological Association (APA) to update the knowledge and policy about the impact of violent video game use on potential adverse outcomes. This APA Task Force on Media Violence examined the existing literature, including the meta-analyses in the field, since the last APA report on media violence in 2005. Because the most recent meta-analyses were published in 2010 and reflected work through 2009, the task force conducted a search of the published studies from 2009-2013. These recently published articles were scored and assessed by a systematic evidentiary review, followed by a meta-analysis of the high utility studies, as documented in the evidentiary review. Consistent with the literature that we reviewed, we found that violent video game exposure was associated with: an increased composite aggression score; increased aggressive behavior; increased aggressive cognitions; increased aggressive affect, increased desensitization, and decreased empathy; and increased physiological arousal. The size of the effects was similar to that in prior meta-analyses, suggesting a stable result. Our task force concluded that violent video game use is a risk factor for adverse outcomes, but found insufficient studies to examine any potential link between violent video game use and delinquency or criminal behavior. Our technical report is the basis of this article. (PsycINFO Database Record