Assessing program fidelity in substance abuse health services research.

This paper addresses how treatment fidelity and related constructs (e.g. program implementation) can be assessed in alcohol, drug abuse and mental health services research. First, it introduces definitions of fidelity and related concepts, and then describes various concepts and tools from program evaluation that have proven useful for assessing fidelity. Next, several of these are illustrated in detail through a case study of a multisite fidelity assessment in substance abuse services research: the process evaluation of the NIAAA Homeless Cooperative Agreement Program. This evaluation included analysis of implementation at the program- and participant-level, the development of scales from the individual services data to estimate intervention strength, fidelity, and "leakage" (i.e. the degree to which services intended exclusively for intervention groups were inadvertently delivered to comparison groups) and the methods with which these data were used to assess whether programs were implemented as planned.

Retention of homeless clients in substance abuse treatment. Findings from the National Institute on Alcohol Abuse and Alcoholism Cooperative Agreement Program.

Retaining clients in treatment who are homeless presents a particular challenge for substance abuse treatment providers. A National Institute on Alcohol Abuse and Alcoholism Cooperative Agreement Program offered the first opportunity to systematically study program retention in a multisite study of interventions for homeless persons with alcohol and other drug problems. This article presents results from analyses conducted across 15 interventions implemented at 8 Cooperative Agreement sites. Both qualitative and quantitative data were collected and analyzed. Key findings were that (a) retention problems with homeless clients are as or more pervasive than in the general addicted population; (b) the provision of housing increases retention, but the increases tend to be nullified when the housing is bundled with high-intensity services; (c) homeless clients leave treatment programs for a multitude of reasons; and (d) midcourse corrections to increase retention are frequently successful. The discussion focuses on service components related to retention, the importance of attending to phase transitions, and the importance of being programmatically responsive when serving this population.

Constructing quantitative implementation scales from categorical services data. Examples from a multisite evaluation.

Like measures of outcome, measures of implementation are most useful and analytically powerful when measured at client-level and are quantitative. However, high-quality, individual-level, quantitative service utilization data can be expensive, intrusive, or otherwise impractical to obtain. Cruder data--for example, presence versus absence of a given service--are often more feasible to collect, as well as more likely to be available. Consequently, evaluators can benefit by finding ways to better exploit such data at the analysis phase to compensate for shortcomings at the collection phase. This article documents one such instance in which this was done. Specifically, it describes how quantitative, client-level implementation scales were derived from qualitative (categorical) data and used to support a cross-site synthesis of implementation and outcome analyses in a multisite evaluation. It also suggests additional scenarios in which quantitative implementation scales might be derived form qualitative services data.

Alcohol and drug abuse treatment of homeless persons: results from the NIAAA Community Demonstration Program.

In a national evaluation, we assessed the implementation and outcomes of a multisite demonstration program for homeless persons with alcohol and other drug problems. We developed comprehensive case studies from data on client characteristics, utilization of services, implementation of interventions, and community systems of care at nine project sites. Client-level outcome data were analyzed to estimate the effectiveness of the interventions in a subset of projects with experimental or quasi-experimental evaluation designs. After controlling for baseline predictors, treatment clients in the majority of sites were significantly more likely than comparison clients to report improvement on one or more outcome dimensions. On alcohol use, for example, under conservative assumptions the average treatment client was drinking less at follow-up than were 57 percent of comparison clients. Analyses of predictor-by-treatment interactions suggested that clients with fewer problems benefited most from the interventions. The implementation analysis yielded a number of lessons for policymakers and program planners.

The subjective well-being of mental patients in nursing homes.

This article reports a survey of self-reports of well-being by chronic mental patients in nursing homes. Patients reported lower levels of well-being than the general population, but not lower than other socially disadvantaged groups including urban renewal blacks. Nursing home mental patients reported levels of well-being that were generally similar to mental patients in other settings, including day hospital patients, participants in an innovative community care program, and patients receiving traditional hospital and follow-up care. These reports were mostly unrelated to levels of symptomatology or social integration, but were strongly related to patient perceptions of the quality of the environment. Results have important implications concerning the justifications for community care goals like noninstitutional care, reduction of symptomatology, and the fostering of social integration.

RRT meets RDD: statistical strategies for assuring response privacy in telephone surveys.

Despite its growing role in survey research, the telephone survey has been largely neglected with respect to the design and testing of statistical procedures for assuring response privacy (e.g., randomized response techniques). This paper demonstrates that such procedures are no less feasible to administer in telephone surveys than in face-to-face surveys. Both randomized response techniques and alternatives to randomized response are considered.