Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration.

BACKGROUND: Even a well-designed randomized control trial (RCT) study can produce ambiguous results. This article highlights a case in which full sample results from a large-scale RCT in the United Kingdom differ from results for a subsample of survey respondents. OBJECTIVES: Our objective is to ascertain the source of the discrepancy in inferences across data sources and, in doing so, to highlight important threats to the reliability of the causal conclusions derived from even the strongest research designs. RESEARCH DESIGN: The study analyzes administrative data to shed light on the source of the differences between the estimates. We explore the extent to which heterogeneous treatment impacts and survey nonresponse might explain these differences. We suggest checks which assess the external validity of survey measured impacts, which in turn provides an opportunity to test the effectiveness of different weighting schemes to remove bias. The subjects included 6,787 individuals who participated in a large-scale social policy experiment. RESULTS: Our results were not definitive but suggest nonresponse bias is the main source of the inconsistent findings. CONCLUSIONS: The results caution against overconfidence in drawing conclusions from RCTs and highlight the need for great care to be taken in data collection and analysis. Particularly, given the modest size of impacts expected in most RCTs, small discrepancies in data sources can alter the results. Survey data remain important as a source of information on outcomes not recorded in administrative data. However, linking survey and administrative data is strongly recommended whenever possible.

Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration.

BACKGROUND: Even a well-designed randomized control trial (RCT) study can produce ambiguous results. This article highlights a case in which full sample results from a large-scale RCT in the United Kingdom differ from results for a subsample of survey respondents. OBJECTIVES: Our objective is to ascertain the source of the discrepancy in inferences across data sources and, in doing so, to highlight important threats to the reliability of the causal conclusions derived from even the strongest research designs. RESEARCH DESIGN: The study analyzes administrative data to shed light on the source of the differences between the estimates. We explore the extent to which heterogeneous treatment impacts and survey nonresponse might explain these differences. We suggest checks which assess the external validity of survey measured impacts, which in turn provides an opportunity to test the effectiveness of different weighting schemes to remove bias. The subjects included 6,787 individuals who participated in a large-scale social policy experiment. RESULTS: Our results were not definitive but suggest nonresponse bias is the main source of the inconsistent findings. CONCLUSIONS: The results caution against overconfidence in drawing conclusions from RCTs and highlight the need for great care to be taken in data collection and analysis. Particularly, given the modest size of impacts expected in most RCTs, small discrepancies in data sources can alter the results. Survey data remain important as a source of information on outcomes not recorded in administrative data. However, linking survey and administrative data is strongly recommended whenever possible.

Personal traits, cohabitation, and marriage.

This study examines how personal traits affect the likelihood of entering into a cohabitating or marital relationship using a competing risk survival model with cohabitation and marriage as competing outcomes. The data are from Waves 1, 3, and 4 of the National Longitudinal Study of Adolescent Health, a rich dataset with a large sample of young adults (N=9835). A personal traits index is constructed from interviewer-assessed scores on the respondents' physical attractiveness, personality, and grooming. Having a higher score on the personal traits index is associated with a greater hazard of entering into a marital relationship for men and women, but the score does not have a significant influence on entering into a cohabitating relationship. Numerous sensitivity tests support the core findings.

A multilevel analysis of the impacts of services provided by the U.K. Employment Retention and Advancement Demonstration.

BACKGROUND: The United Kingdom Employment Retention and Advancement (U.K. ERA) demonstration was the largest and most comprehensive social experiment ever conducted in the United Kingdom. It examined the extent to which a combination of postemployment advisory support and financial incentives could help lone parents on welfare to find sustained employment with prospects for advancement. ERA was experimentally tested across more than 50 public employment service offices and, within each office, individuals were randomly assigned to either a program (or treatment) group (eligible for ERA) or a control group (not eligible). METHOD: This article presents the results of a multilevel nonexperimental analysis that examines the variation in office-level impacts and attempts to understand what services provided in the offices tend to be associated with impacts. RESULT: The analysis suggests that impacts were greater in offices that emphasized in-work advancement, support while working and financial bonuses for sustained employment, and also in those offices that assigned more caseworkers to ERA participants. Offices that encouraged further education had smaller employment impacts. CONCLUSION: Plausible results are obtained identifying those particular implementation features that tended to be linked to stronger impacts of ERA. The methodology employed also allows the identification of which services are associated with employment and welfare receipt of control families receiving benefits under the traditional New Deal for Lone Parent program.

Welfare reform and child care: evidence from 10 experimental welfare-to-work programs.

This article examines the employment and child care responses of families participating in 10 experimental welfare reform programs conducted in the United States between 1989 and 2002. For the programs analyzed, child care use increases by about the same amount as the increase in employment. Most of the increased child care comprises informal care by a relative, particularly care by a sibling or a grandparent. Although there are significant differences in the child care responses across the various programs tested, there are no significant differences in the impacts for persons leaving welfare versus persons staying on welfare.