The Design of Making Pre-K count and High Fives: Two-Stage, Multiyear Random Assignment at Different Levels.

The Making Pre-K Count and High 5s studies represent a recent application of a phased two-stage, multi-level design, which was used to examine the effects of two aligned math programs implemented in early childhood settings. The purpose of this paper is to describe the challenges encountered in implementing this two-stage design and to describe approaches to resolving them. We then present a set of sensitivity analyses the study team used to examine the robustness of the findings. During the pre-K year, pre-K centers were randomly assigned either to receive an evidence-based early math curriculum and associated professional development (Making Pre-K Count) or to a pre-K-as-usual control condition. In the kindergarten year, students who had been in Making Pre-K Count program classrooms in pre-K were then individually randomly assigned within schools to small-group supplemental math clubs that were designed to sustain the gains from the pre-K program, or to a business-as-usual kindergarten experience. Making Pre-K Count took place in 69 pre-K sites, comprising 173 classrooms across New York City. High 5s took place in the 24 sites that were part of the public school treatment arm of the Making Pre-K Count study and included 613 students. The study focuses on the effect of the Making Pre-K Count and High 5s programs on children's math skills at the end of kindergarten as measured by two instruments, the Research-Based Early Math Assessment-Kindergarten (REMA-K) and the Woodcock-Johnson Applied Problems test. The multi-armed design, while logistically and analytically challenging, balanced multiple considerations of power, the number of research questions that could be answered, and efficiency of resources. Robustness checks suggest that the design created groups that were both meaningfully and statistically equivalent. Decisions to use a phased multi-armed design should consider both its strengths and weaknesses. While the design allows for a more flexible, expansive research study, it also introduces complexities that need to be addressed both logistically and analytically.

Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges.

The last two decades have seen a dramatic increase in randomized controlled trials (RCTs) conducted in community colleges. Yet, there is limited empirical information on the design parameters necessary to plan the sample size for RCTs in this context. For a blocked student-level random assignment research design, key design parameters for the minimum detectable true effect (MDTE) are the within-block outcome standard deviation (σ|S) and the within-block outcome variance explained by baseline covariates like student characteristics (R|S2). We provide empirical estimates of these key design parameters, discussing the pattern of estimates across outcomes (enrollment, credits earned, credential attainment, and grade point average), semesters, and studies. The main analyses use student-level data from 8 to 14 RCTs including 5,649-7,099 students (depending on the outcome) with follow-up data for 3 years. The following patterns are observed: the within-block standard deviation (σ|S) and therefore the MDTE can be much larger in later semesters for enrollment outcomes and cumulative credits earned; there is substantial variation across studies in σ|S for degree attainment; and baseline covariates explain less than 10% of the variation in student outcomes. These findings indicate that when planning the sample size for a study, researchers should be mindful of the follow-up period, use a range of values to calculate the MDTE for outcomes that vary across studies, and assume a value of R|S2 between 0 and 0.05. A public database created for this paper includes parameter estimates for additional RCTs and students.

The Validity of the Comparative Interrupted Time Series Design for Evaluating the Effect of School-Level Interventions.

OBJECTIVE: In this article, we examine whether a well-executed comparative interrupted time series (CITS) design can produce valid inferences about the effectiveness of a school-level intervention. This article also explores the trade-off between bias reduction and precision loss across different methods of selecting comparison groups for the CITS design and assesses whether choosing matched comparison schools based only on preintervention test scores is sufficient to produce internally valid impact estimates. RESEARCH DESIGN: We conduct a validation study of the CITS design based on the federal Reading First program as implemented in one state using results from a regression discontinuity design as a causal benchmark. RESULTS: Our results contribute to the growing base of evidence regarding the validity of nonexperimental designs. We demonstrate that the CITS design can, in our example, produce internally valid estimates of program impacts when multiple years of preintervention outcome data (test scores in the present case) are available and when a set of reasonable criteria are used to select comparison organizations (schools in the present case).

The effect of an integrated multisector model for achieving the Millennium Development Goals and improving child survival in rural sub-Saharan Africa: a non-randomised controlled assessment.

BACKGROUND: Simultaneously addressing multiple Millennium Development Goals (MDGs) has the potential to complement essential health interventions to accelerate gains in child survival. The Millennium Villages project is an integrated multisector approach to rural development operating across diverse sub-Saharan African sites. Our aim was to assess the effects of the project on MDG-related outcomes including child mortality 3 years after implementation and compare these changes to local comparison data. METHODS: Village sites averaging 35,000 people were selected from rural areas across diverse agroecological zones with high baseline levels of poverty and undernutrition. Starting in 2006, simultaneous investments were made in agriculture, the environment, business development, education, infrastructure, and health in partnership with communities and local governments at an annual projected cost of US$120 per person. We assessed MDG-related progress by monitoring changes 3 years after implementation across Millenium Village sites in nine countries. The primary outcome was the mortality rate of children younger than 5 years of age. To assess plausibility and attribution, we compared changes to reference data gathered from matched randomly selected comparison sites for the mortality rate of children younger than 5 years of age. Analyses were done on a per-protocol basis. This trial is registered with ClinicalTrials.gov, number NCT01125618. FINDINGS: Baseline levels of MDG-related spending averaged $27 per head, increasing to $116 by year 3 of which $25 was spent on health. After 3 years, reductions in poverty, food insecurity, stunting, and malaria parasitaemia were reported across nine Millennium Village sites. Access to improved water and sanitation increased, along with coverage for many maternal-child health interventions. Mortality rates in children younger than 5 years of age decreased by 22% in Millennium Village sites relative to baseline (absolute decrease 25 deaths per 1000 livebirths, p=0·015) and 32% relative to matched comparison sites (30 deaths per 1000 livebirths, p=0·033). INTERPRETATION: An integrated multisector approach for addressing the MDGs can produce rapid declines in child mortality in the first 3 years of a long-term effort in rural sub-Saharan Africa. FUNDING: UN Human Security Trust Fund, the Lenfest Foundation, Bill & Melinda Gates Foundation, and Becton Dickinson.

Multisector intervention to accelerate reductions in child stunting: an observational study from 9 sub-Saharan African countries.

BACKGROUND: In sub-Saharan Africa, ~ 40% of children <5 y old are stunted, with levels that have remained largely unchanged over the past 2 decades. Although the complex determinants of undernutrition are well recognized, few studies have evaluated strategies that combine nutrition-specific, health-based approaches with food system- and livelihood-based interventions. OBJECTIVE: We examined changes in childhood stunting and its determinants after 3 y of exposure to an integrated, multisector intervention and compared these changes with national trends. DESIGN: A prospective observational trial was conducted across rural sites in 9 sub-Saharan African countries with baseline levels of childhood stunting >20%. A stratified random sample of households and resident children <2 y old from villages exposed to the program were enrolled in the study. Main outcome measures included principal determinants of undernutrition and childhood stunting, which was defined as a height-for-age z score less than -2. National trends in stunting were generated from demographic and health surveys. RESULTS: Three years after the start of the program in 2005-2006, consistent improvements were observed in household food security and diet diversity, whereas coverage with child care and disease-control interventions improved for most outcomes. The prevalence of stunting in children <2 y old at year 3 of the program (2008-2009) was 43% lower (adjusted OR: 0.57; 95% CI: 0.38, 0.83) than at baseline. The average national stunting prevalence for the countries included in the study had remained largely unchanged over the past 2 decades. CONCLUSION: These findings provide encouraging evidence that a package of multisector interventions has the potential to produce reductions in childhood stunting.