Matched Comparison Group Design Standards in Systematic Reviews of Early Childhood Interventions.

BACKGROUND: Systematic reviews assess the quality of research on program effectiveness to help decision makers faced with many intervention options. Study quality standards specify criteria that studies must meet, including accounting for baseline differences between intervention and comparison groups. We explore two issues related to systematic review standards: covariate choice and choice of estimation method. OBJECTIVE: To help systematic reviews develop/refine quality standards and support researchers in using nonexperimental designs to estimate program effects, we address two questions: (1) How well do variables that systematic reviews typically require studies to account for explain variation in key child and family outcomes? (2) What methods should studies use to account for preexisting differences between intervention and comparison groups? METHODS: We examined correlations between baseline characteristics and key outcomes using Early Childhood Longitudinal Study-Birth Cohort data to address Question 1. For Question 2, we used simulations to compare two methods-matching and regression adjustment-to account for preexisting differences between intervention and comparison groups. RESULTS: A broad range of potential baseline variables explained relatively little of the variation in child and family outcomes. This suggests the potential for bias even after accounting for these variables, highlighting the need for systematic reviews to provide appropriate cautions about interpreting the results of moderately rated, nonexperimental studies. Our simulations showed that regression adjustment can yield unbiased estimates if all relevant covariates are used, even when the model is misspecified, and preexisting differences between the intervention and the comparison groups exist.

Reviewing the Reviews: Examining Similarities and Differences Between Federally Funded Evidence Reviews.

BACKGROUND: The federal government's emphasis on supporting the implementation of evidence-based programs has fueled a need to conduct and assess rigorous evaluations of programs. Through partnerships with researchers, policy makers, and practitioners, evidence reviews-projects that identify, assess, and summarize existing research in a given area-play an important role in supporting the quality of these evaluations and how the findings are used. These reviews encourage the use of sound scientific principles to identify, select, and implement evidence-based programs. The goals and standards of each review determine its conclusions about whether a given evaluation is of high quality or a program is effective. It can be difficult for decision makers to synthesize the body of evidence when faced with results from multiple program evaluations. SAMPLE: This study examined 14 federally funded evidence reviews to identify commonalities and differences in their assessments of evidence of effectiveness. FINDINGS: There were both similarities and significant differences across the reviews. In general, the evidence reviews agreed on the broad critical elements to consider when assessing evaluation quality, such as research design, low attrition, and baseline equivalence. The similarities suggest that, despite differences in topic and the availability of existing research, reviews typically favor evaluations that limit potential bias in their estimates of program effects. However, the way in which some of the elements were assessed, such as what constituted acceptable amounts of attrition, differed. Further, and more substantially, the reviews showed greater variation in how they conceptualized "effectiveness."

External Validity: The Next Step for Systematic Reviews?

BACKGROUND: Systematic reviews-which identify, assess, and summarize existing research-are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions. OBJECTIVES: In this article, we focus on how systematic reviews address external validity. METHODS: We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research. RESULTS: We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics-such as sample demographics or settings-are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married. CONCLUSION: Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

Effectiveness of home visiting in improving child health and reducing child maltreatment.

BACKGROUND AND OBJECTIVE: The Patient Protection and Affordable Care Act established the Maternal, Infant, and Early Childhood Home Visiting Program, which provides $1.5 billion to states over 5 years for home visiting program models serving at-risk pregnant women and children from birth to age 5. The act stipulates that 75% of the funds must be used for programs with evidence of effectiveness based on rigorous evaluation research. Home Visiting Evidence of Effectiveness reviewed the home visiting research literature and provided an assessment of the evidence of effectiveness for program models that serve families with pregnant women and children from birth to age 5. METHODS: Home Visiting Evidence of Effectiveness included a systematic search and screening process, a review of the research quality, and an assessment of program effectiveness. Reviewers rated studies' capacity to provide unbiased estimates of program impacts and determined whether a program met the Department of Health and Human Services' criteria for an evidence-based model. RESULTS: As of July 2012, 32 models were reviewed, of which 12 met the Department of Health and Human Services criteria. Most of these models were shown to have favorable effects on child development. Other common favorable effects included health care usage and reductions in child maltreatment. Less common were favorable effects on birth outcomes. CONCLUSIONS: Home visiting is a promising way to serve families who may be difficult to engage in supportive services. Existing rigorous research indicates that home visiting has the potential for positive results among high-risk families, particularly on health care usage and child development.