Evaluation methods for social intervention.

Experimental design is the method of choice for establishing whether social interventions have the intended effects on the populations they are presumed to benefit. Experience with field experiments, however, has revealed significant limitations relating chiefly to (a) practical problems implementing random assignment, (b) important uncontrolled sources of variability occurring after assignment, and (c) a low yield of information for explaining why certain effects were or were not found. In response, it is increasingly common for outcome evaluation to draw on some form of program theory and extend data collection to include descriptive information about program implementation, client characteristics, and patterns of change. These supplements often cannot be readily incorporated into standard experimental design, especially statistical analysis. An important advance in outcome evaluation is the recent development of statistical models that are able to represent individual-level change, correlates of change, and program effects in an integrated and informative manner.

Constructing quantitative implementation scales from categorical services data. Examples from a multisite evaluation.

Like measures of outcome, measures of implementation are most useful and analytically powerful when measured at client-level and are quantitative. However, high-quality, individual-level, quantitative service utilization data can be expensive, intrusive, or otherwise impractical to obtain. Cruder data--for example, presence versus absence of a given service--are often more feasible to collect, as well as more likely to be available. Consequently, evaluators can benefit by finding ways to better exploit such data at the analysis phase to compensate for shortcomings at the collection phase. This article documents one such instance in which this was done. Specifically, it describes how quantitative, client-level implementation scales were derived from qualitative (categorical) data and used to support a cross-site synthesis of implementation and outcome analyses in a multisite evaluation. It also suggests additional scenarios in which quantitative implementation scales might be derived form qualitative services data.

Formal education and five-year mortality in rheumatoid arthritis: mediation by helplessness scale score.

OBJECTIVE: To analyze scores on a scale designed to measure helplessness, a cognitive variable, as a possible mediator of the association between formal education level and mortality over 5 years in patients with rheumatoid arthritis (RA). METHODS: A cohort of 1,416 patients with RA from 15 private practices in 6 states and Washington, DC was monitored for over 5 years. Demographic, socioeconomic, therapy, functional status, and psychological variables were analyzed as possible predictors of mortality in invariable and multivariable Cox Proportional Hazards models. RESULTS: In a 5-year followup, 1,384 patients were accounted for (97.3%), including 174 who died versus 111 expected (standardized mortality ratio = 1.54). Higher mortality was associated significantly with low formal education, high age, poor scores for activities of daily living (ADL) on a modified health assessment questionnaire (MHAQ), and poor scores on a helplessness scale (all P < 0.01) in univariable analyses. High age, few years of formal education, and poor MHAQ ADL scores were all significant independent predictors of mortality when analyzed simultaneously in a Cox Proportional Hazards model. When helplessness scale scores were included in a model, scores greater than 2.4 (on a scale of 1 to 4), higher age, male gender, and increased MHAQ ADL difficulty scores were all independently significantly predictive of 5-year mortality (P < 0.05), while years of education was no longer a significant predictor. CONCLUSION: Scores on a helplessness scale appear to mediate a component of the association between formal education level and 5-year mortality in these patients with RA. Health professionals and policy makers might consider interventions directed at modification of helplessness as adjunctive to standard interventions to improve outcomes in RA.

Evaluation of a long-term home care program.

This article reports the outcomes of a 9-month evaluation of the Five Hospital Homebound Elderly Program ( FHHEP ), a model long-term, comprehensive, coordinated home care program in Chicago. Outcomes assessed include the mortality, comprehensive functional status, and rates of hospitalization and of institutionalization of the elderly (mean age, 80.4 years), chronically impaired population served by the FHHEP . The evaluation utilized a quasi-experimental, preposttest design with a nonequivalent control group consisting of similarly elderly and impaired subjects who received OAA Title III-c home-delivered meals. Consecutively accepted experimental (n = 122) and control group clients (n = 123) were interviewed using the Duke/ OARS Multi-dimensional Functional Assessment Questionnaire at the time of acceptance to service and 9 months later. Service utilization data were also obtained for both groups to correlate client outcomes and characteristics with level and type of services used. Data collection took place over a 31-month period. Posttest functional status measures were obtained for 83% of experimental and 81% of control subjects. Multivariate analysis was used to control measured pretest differences. Major findings include a significant reduction in the nursing home admissions (16 vs. 28) and nursing home days (including sheltered care) of experimental group clients. The reported analyses also show an increase in experimental clients' sense of physical health well-being and a decrease in their number of previously unmet needs for community services. Somewhat paradoxically, the experimental sample also demonstrated a decrease in physical activities of daily living ( PADL ) functioning. The mortality and hospitalization rate were equal for both groups. Despite savings in nursing home days of care, average per-capita costs for experimental group clients were 19% higher than for controls. However, this additional cost was accompanied by an increase in quality of life. Longer-range cost and outcomes are being assessed through a 4-year follow-up study currently in progress.