Reviewing the Reviews: Examining Similarities and Differences Between Federally Funded Evidence Reviews.

BACKGROUND: The federal government's emphasis on supporting the implementation of evidence-based programs has fueled a need to conduct and assess rigorous evaluations of programs. Through partnerships with researchers, policy makers, and practitioners, evidence reviews-projects that identify, assess, and summarize existing research in a given area-play an important role in supporting the quality of these evaluations and how the findings are used. These reviews encourage the use of sound scientific principles to identify, select, and implement evidence-based programs. The goals and standards of each review determine its conclusions about whether a given evaluation is of high quality or a program is effective. It can be difficult for decision makers to synthesize the body of evidence when faced with results from multiple program evaluations. SAMPLE: This study examined 14 federally funded evidence reviews to identify commonalities and differences in their assessments of evidence of effectiveness. FINDINGS: There were both similarities and significant differences across the reviews. In general, the evidence reviews agreed on the broad critical elements to consider when assessing evaluation quality, such as research design, low attrition, and baseline equivalence. The similarities suggest that, despite differences in topic and the availability of existing research, reviews typically favor evaluations that limit potential bias in their estimates of program effects. However, the way in which some of the elements were assessed, such as what constituted acceptable amounts of attrition, differed. Further, and more substantially, the reviews showed greater variation in how they conceptualized "effectiveness."

External Validity: The Next Step for Systematic Reviews?

BACKGROUND: Systematic reviews-which identify, assess, and summarize existing research-are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions. OBJECTIVES: In this article, we focus on how systematic reviews address external validity. METHODS: We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research. RESULTS: We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics-such as sample demographics or settings-are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married. CONCLUSION: Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

Pathways among exposure to violence, maternal depression, family structure, and child outcomes through parenting: a multigroup analysis.

The present study examined the impact of proximal (maternal depression, family structure) and distal (exposure to violence) risk factors on parenting characteristics (warmth, control), which were in turn hypothesized to affect child social-emotional functioning. Using the Family and Child Experiences Study (FACES) 2000 cohort, findings revealed that study variables were significant predictors of child social-emotional functioning. Despite limited significant pathways in the structural equation models, the cumulative effect of the variables resulted in models accounting for 21%-37% of the outcome. Multigroup analysis revealed that although the amount of variance explained varied, the model held across subgroups. Findings support theories such as the family stress model that suggest that family risk factors negatively influencing children's development through influencing parenting behaviors. Findings also support considering both warmth and control as key parenting dimensions. It may be impractical for practitioners to address the myriad of potential risks encountered by low-income families, but parents can be equipped with mental health services, parent education, and other assistance to help them maintain positive parenting practices in the face of challenges.