Emergence of a predictive model in the hippocampus.

The brain organizes experiences into memories that guide future behavior. Hippocampal CA1 population activity is hypothesized to reflect predictive models that contain information about future events, but little is known about how they develop. We trained mice on a series of problems with or without a common statistical structure to observe how memories are formed and updated. Mice that learned structured problems integrated their experiences into a predictive model that contained the solutions to upcoming novel problems. Retrieving the model during learning improved discrimination accuracy and facilitated learning. Using calcium imaging to track CA1 activity during learning, we found that hippocampal ensemble activity became more stable as mice formed a predictive model. The hippocampal ensemble was reactivated during training and incorporated new activity patterns from each training problem. These results show how hippocampal activity supports building predictive models by organizing new information with respect to existing memories.

Using Behavioral Insights to Market a Workplace Safety Program: Evidence From a Multi-Armed Experiment.

This paper describes how a multi-armed randomized experiment was used to test multiple variants of a behaviorally informed marketing strategy. In particular, we tested whether specific behavioral messages could be used to increase demand for a safety consultation service offered by the U.S. Occupational Safety and Health Administration. Our experiment used a partial factorial design with 19 study arms and a very large research sample-97,182 establishments-to test the impact of various message, formats, and delivery modes compared with an existing (not behaviorally informed) informational brochure and a no-marketing counterfactual. A secondary research goal was to predict the impact of the most successful marketing strategy (i.e., combination of message, format, and mode) so that OSHA would know what to anticipate if that strategy were implemented at scale. We used two related (but distinct) methods to address these two goals. Both begin with a common mixed (i.e., fixed and random effects) ANOVA model. We addressed the first research goal primarily from the fixed effects; we addressed the second research goal by calculating best linear unbiased predictions (BLUPs) from the full mixed model, where the BLUP involves "shrinkage" as in empirical Bayes (EB) approaches. Marketing via brochures was effective overall, nearly doubling the rate of requests for services. However, the behaviorally informed materials performed no better than OSHA's existing informational brochure. This study also highlights the conditions under which a factorial design can be used to efficiently address questions about which of several program variants are most effective.

Increasing Child Fruit and Vegetable Intake: Findings from the US Department of Agriculture Fresh Fruit and Vegetable Program.

BACKGROUND: Fewer than 10% of US children and adolescents consume the recommended amounts of fruits and vegetables (F/V). The US Department of Agriculture's Fresh Fruit and Vegetable Program (FFVP) is intended to increase child F/V consumption by funding low-income schools to distribute free fresh F/V snacks outside of school mealtimes. OBJECTIVE: The evaluation assessed FFVP effects on student F/V consumption and total energy intake in and out of school. DESIGN: The evaluation employed a regression discontinuity design; that is, cross-sectional comparisons of a sample of students in schools just above and just below the FFVP funding cutoff for the program, which depended on the proportion of students eligible for free or reduced-price meals. During the 2010-2011 school year within a randomly selected sample of states, we selected schools in closest proximity to each state-specific FFVP funding cutoff. Interviewers conducted 24-hour diary-assisted recall interviews to assess dietary intake among children in selected schools. PARTICIPANTS/SETTING: Participants were 4,696 students (grades 4 to 6) from 214 elementary schools in 16 randomly selected states. STATISTICAL ANALYSES PERFORMED: Analysis proceeded via multivariate regression, comparing adjusted mean student intake in schools just above and just below the funding cutoff. RESULTS: Adjusted mean daily F/V intake was one-third of a cup per day higher in FFVP-participating schools than in nonparticipating schools (0.32 cups per day; P<0.001), a difference of 15.5%. This included one-quarter cup higher daily F/V intake during school hours (0.26 cups; P<0.001) among students attending FFVP-participating schools. Fresh (but not total) F/V consumption also increased outside of school. CONCLUSIONS: The FFVP increases child fresh and total F/V intake in school, and fresh F/V intake outside of school.

Independent evaluation: insights from public accounting.

BACKGROUND: Maintaining the independence of contract government program evaluation presents significant contracting challenges. The ideal outcome for an agency is often both the impression of an independent evaluation and a glowing report. In this, independent evaluation is like financial statement audits: firm management wants both a public accounting firm to attest to the fairness of its financial accounts and to be allowed to account for transactions as it sees fit. In both cases, the evaluation or audit is being conducted on behalf of outsiders--the public or shareholders--but is overseen by a party with significant interests at stake in the outcome-the agency being evaluated or executive management of the firm. METHOD: We review the contracting strategies developed to maintain independence in auditing. We examine evidence on the effectiveness of professionalism, reputation, liability and owner oversight in constraining behavior in auditing. We then establish parallels with contracting for evaluations and apply these insights to changes that might maintain and improve evaluator independence. CONCLUSIONS AND RECOMMENDATIONS: By analogy with the Sarbanes Oxley Act of 2002 reforms in auditing, we recommend exploring using a reformulated Technical Working Group to encourage more prompt release of more evaluation results and to help insulate evaluators from inappropriate pressure to change their results or analysis approach.

When is a program ready for rigorous impact evaluation? The role of a falsifiable logic model.

BACKGROUND: Recent reviews suggest that many plausible programs are found to have at best small impacts not commensurate with their cost, and often have no detectable positive impacts at all. Even programs with initial rigorous impact evaluation (RIE) that show them to be effective often fail a second test with an expanded population or at multiple sites. OBJECTIVE: This article argues that more rapid movement to RIE is a partial cause of the low success rate of RIE and proposes a constructive response: process evaluations that compare program intermediate outcomes--in the treatment group, during the operation of the program--against a more falsifiable extension of the conventional logic model. CONCLUSION: Our examples suggest that such process evaluations would allow funders to deem many programs unlikely to show impacts and therefore not ready for random assignment evaluation--without the high cost and long time lines of an RIE. The article then develops the broader implications of such a process analysis step for broader evaluation strategy.

How does Medicaid-managed care impact reporting of Medicaid status?

OBJECTIVE: To resolve a conflict in the literature on whether Medicaid-Managed Care (MMC) impacts the Medicaid Undercount. DATA SOURCES/STUDY SETTING: California county-level data (1995-1997) on MMC penetration, public use data from the Current Population Survey (CPS) (1995-1997), and restricted CPS data matched to administrative records on Medicaid enrollment (2001-2002). STUDY DESIGN: We explore the robustness of previous results from the literature first using aggregate data and alternative models. We then examine CPS data linked to Medicaid enrollment data to estimate models of Medicaid reporting errors related to MMC. DATA COLLECTION/EXTRACTION METHODS: The Census Bureau linked administrative data on Medicaid enrollment to the CPS. Other data used were public use. PRINCIPAL FINDINGS: We find similar results to a previous study using aggregate data that suggest that MMC worsens reporting of Medicaid enrollment. However, using alternative methods we find those results are not statistically significant and can have opposite signs. Our linked CPS microdata analysis suggests that MMC improves reporting. The article concludes with implications of these results for policy makers. CONCLUSION: It is unlikely that increased MMC penetration explains the increased Medicaid Undercount.

Contracting for independent evaluation: approaches to an inherent tension.

There has recently been discussion of whether independent contract evaluation is possible. This article acknowledges the inherent tension in contract evaluation and in response suggests a range of constructive approaches to improving the independence of contract evaluation. In particular, a clear separation between the official evaluation report and a contractor's own publication of analysis from the underlying evaluation appears to be a promising approach. In this approach, the funder would retain almost unfettered rights to the official contract report (including the right never to publish but not the right to change the contractor's text while leaving the contractor's authorship) and the contractor would retain clearly defined rights to publish any findings from the evaluation (subject only to the limitations of human subjects and proprietary data and some minimal notice).

An examination of the Medicaid undercount in the current population survey: preliminary results from record linking.

OBJECTIVE: To assess reasons why survey estimates of Medicaid enrollment are 43 percent lower than raw Medicaid program enrollment counts (i.e., "Medicaid undercount"). DATA SOURCES: Linked 2000-2002 Medicaid Statistical Information System (MSIS) and the 2001-2002 Current Population Survey (CPS). DATA COLLECTION METHODS: Centers for Medicare and Medicaid Services provided the Census Bureau with its MSIS file. The Census Bureau linked the MSIS to the CPS data within its secure data analysis facilities. STUDY DESIGN: We analyzed how often Medicaid enrollees incorrectly answer the CPS health insurance item and imperfect concept alignment (e.g., inclusion in the MSIS of people who are not included in the CPS sample frame and people who were enrolled in Medicaid in more than one state during the year). PRINCIPAL FINDINGS: The extent to which the Medicaid enrollee data were adjusted for imperfect concept alignment reduces the raw Medicaid undercount considerably (by 12 percentage points). However, survey response errors play an even larger role with 43 percent of Medicaid enrollees answering the CPS as though they were not enrolled and 17 percent reported being uninsured. CONCLUSIONS: The CPS is widely used for health policy analysis but is a poor measure of Medicaid enrollment at any time during the year because many people who are enrolled in Medicaid fail to report it and may be incorrectly coded as being uninsured. This discrepancy should be considered when using the CPS for policy research.