Process evaluation results from the HEALTHY nutrition intervention to modify the total school food environment.

The process evaluation of HEALTHY, a large multi-center trial to decrease type 2 diabetes mellitus in middle school children, monitored the implementation of the intervention to ascertain the extent that components were delivered and received as intended. The purpose of this article is to report the process evaluation findings concerning the extent to which the HEALTHY nutrition intervention was implemented during the HEALTHY trial. Overall, the observed fidelity of implementing nutrition strategies improved from baseline to the end of the study. By the last semester, all but two nutrition process evaluation goals were met. The most challenging goal to implement was serving high fiber foods, including grain-based foods and legumes. The easiest goals to implement were lowering the fat content of foods offered and offering healthier beverages. The most challenging barriers experienced by research dietitians and food service staff were costs, availability of foods and student acceptance. Forming strong relationships between the research dietitians and food service staff was identified as a key strategy to meet HEALTHY nutrition goals.

Rationale, design and methods for process evaluation in the HEALTHY study.

The HEALTHY study was a multi-site randomized trial designed to determine whether a 3-year school-based intervention targeting nutrition and physical activity behaviors could effectively reduce risk factors associated with type 2 diabetes in middle school children. Pilot and formative studies were conducted to inform the development of the intervention components and the process evaluation methods for the main trial. During the main trial, both qualitative and quantitative assessments monitored the fidelity of the intervention and motivated modifications to improve intervention delivery. Structured observations of physical education classes, total school food environments, classroom-based educational modules, and communications and promotional campaigns provided verification that the intervention was delivered as intended. Interviews and focus groups yielded a multidimensional assessment of how the intervention was delivered and received, as well as identifying the barriers to and facilitators of the intervention across and within participating schools. Interim summaries of process evaluation data were presented to the study group as a means of ensuring standardization and quality of the intervention across the seven participating centers. Process evaluation methods and procedures documented the fidelity with which the HEALTHY study was implemented across 21 intervention schools and identified ways in which the intervention delivery might be enhanced throughout the study.

Process evaluation results from a school- and community-linked intervention: the Trial of Activity for Adolescent Girls (TAAG).

Process evaluation is a component of intervention research that evaluates whether interventions are delivered and received as intended. Here, we describe the process evaluation results for the Trial of Activity for Adolescent Girls (TAAG) intervention. The intervention consisted of four synergistic components designed to provide supportive school- and community-linked environments to prevent the decline in physical activity in adolescent girls. Process evaluation results indicate that the intervention components were delivered from intervention staff to teachers with high fidelity (84-97%) to the protocol and with lower fidelity (range: 18-93%) from teachers to students. Physical activity programs for girls, a unique feature of the TAAG intervention, increased from a mean of 10 programs per school to a mean of 16 and 15 in years 1 and 2, respectively, in intervention schools, with no change in control schools. These findings suggest that a multicomponent school- and community-based physical activity intervention can be delivered with fidelity and result in a middle school environment that supports physical activity for girls.

"We're living what we're learning": student perspectives in distance learning degree and certificate programs in public health.

The authors used surveys and interviews to study participants' motivations for enrolling and perceptions of the weaknesses and strengths of two distance learning programs administered by the University of North Carolina at Chapel Hill School of Public Health (SPH): the MPH in Public Health Leadership and a certificate program organized collaboratively with the Mahidol University SPH in Thailand. Chief motivations were career advancement, job performance improvement, convenience, and obtaining a degree from a reputable institution. Strengths included the curriculum, networking opportunities, and administrative and technical support. Concerns included quality of interaction with faculty and instructional methods.

Can health professionals learn qualitative evaluation methods on the World Wide Web? A case example.

The Enhancing Data Utilization Skills through Information Technology (EDUSIT) project trained Maternal and Child Health professionals to collect, analyze and interpret data via a year-long web-based course. The overall goal of the project was to strengthen the technology and analytic skills of the public health workforce. This article describes and analyzes a web-based module for training public health professionals to use qualitative research and evaluation methods that was one of six offered within the EDUSIT project. The qualitative module consisted of six units: overview of qualitative methods, planning qualitative studies, conducting field observations, qualitative interviewing, analyzing qualitative data and presenting qualitative findings. Evaluation results found no statistically significant changes in specific knowledge or beliefs about qualitative methods. However, the change in participants' self-efficacy was statistically significant. Participants' self-reports also showed significant changes in perceived skill levels in 'collecting qualitative data through an interview' and 'analyzing and interpreting qualitative data'. Most participants rated each lesson within the qualitative methods module as valuable, and most found the teaching methods used satisfactory, emphasizing the value of both the didactic teaching and the practical exercises and team project. The most common difficulty reported was finding the time to complete the module requirements while also working full-time. Implications of these findings for web-based teaching of public health professionals are discussed.

Early detection of cervical cancer among Native American women: a qualitative supplement to a quantitative study.

The North Carolina Native American Cervical Cancer Prevention Project was a 5-year (1989-1995) National Cancer Institute-funded, community-based, early detection of cervical cancer intervention implemented among two Native American tribes in North Carolina: the eastern band of the Cherokee Indians and the Lumbee. The initial quantitative analysis of the intervention showed modest effects and found that the intervention had different effects in the two communities. Due to the equivocal findings, a retrospective qualitative study was conducted. The qualitative study found that two types of factors influenced the intervention's results. The first were project and intervention characteristics, and the second were community and cultural factors over which the project had no control. The community and cultural factors took two forms: enhancers, which contributed to greater intervention effect, and attenuators, which created barriers to success. Examples of each factor are presented, and implications for cervical cancer detection among Native American women are discussed.

Process evaluation in a multisite, primary obesity-prevention trial in American Indian schoolchildren.

We describe the development, implementation, and use of the process evaluation component of a multisite, primary obesity prevention trial for American Indian schoolchildren. We describe the development and pilot testing of the instruments, provide some examples of the criteria for instrument selection, and provide examples of how process evaluation results were used to document and refine intervention components. The theoretical and applied framework of the process evaluation was based on diffusion theory, social learning theory, and the desire for triangulation of multiple modes of data collection. The primary objectives of the process evaluation were to systematically document the training process, content, and implementation of 4 components of the intervention. The process evaluation was developed and implemented collaboratively so that it met the needs of both the evaluators and those who would be implementing the intervention components. Process evaluation results revealed that observation and structured interviews provided the most informative data; however, these methods were the most expensive and time consuming and required the highest level of skill to undertake. Although the literature is full of idealism regarding the uses of process evaluation for formative and summative purposes, in reality, many persons are sensitive to having their work evaluated in such an in-depth, context-based manner as is described. For this reason, use of structured, quantitative, highly objective tools may be more effective than qualitative methods, which appear to be more dependent on the skills and biases of the researcher and the context in which they are used.

"What the 'caine was tellin' me to do." Crack users' risk of HIV: an exploratory study of female inmates.

This study describes factors that place crack-addicted female jail inmates at risk for HIV infection. The study provides a portrait of the spheres of influences that directly and indirectly promote HIV risk-taking behaviors, women's efforts toward protecting themselves, and reported sexual behaviors. The study documents the far-reaching effects of crack addiction. One-and-a-half-hour interviews were conducted with 14 inmates recovering from crack addiction. The women were aged 19 to 39, and 13 were African American. The results of this study suggest that women's addictions are greatly shaped by their family and intimate relationships. Addictive behavior often precluded safer sex behaviors and increased a woman's likelihood of engagement in HIV-risky behaviors. Many women were victims of childhood and adulthood sexual and physical victimization. Women sought to protect themselves through sexual self-protection strategies, although these measures were often not effective HIV risk-reduction strategies.

Factors that contribute to effective community health promotion coalitions: a study of 10 Project ASSIST coalitions in North Carolina. American Stop Smoking Intervention Study for Cancer Prevention.

The purpose of this study was to identify factors that contribute to the effectiveness of community health promotion coalitions. Member survey data from 10 coalitions formed as part of North Carolina Project ASSIST were analyzed at the coalition level to identify factors related to member participation, member satisfaction, quality of the action plan, resource mobilization, and implementation. The results suggest that coalitions with good communication and skilled members had higher levels of member participation. Coalitions with skilled staff, skilled leadership, good communication, and more of a task focus had higher levels of member satisfaction. Coalitions with more staff time devoted to them and more complex structures had greater resource mobilization, and coalitions with more staff time, good communication, greater cohesion, and more complex structures had higher levels of implementation. Neither member participation nor member satisfaction correlated with the other measures of coalition effectiveness.

A multiple case study of implementation in 10 local Project ASSIST coalitions in North Carolina.

Community health promotion relies heavily on coalitions to address a multitude of public health issues. In spite of their widespread use, there have been very few studies of coalitions at various stages of coalition development. The purpose of this study was to identify factors that facilitated or impeded coalition effectiveness in the implementation stage of coalition development. The research design was a multiple case study with cross-case comparisons. Each of the 10 local North Carolina Project ASSIST coalitions constituted a case. Data collection included: semi-structured interviews, observation, document review, and surveys of members and staff. Some of the major factors that facilitated implementation included: the ability of the coalition to provide its own vision, staff with the skills and time to work with the coalition, frequent and productive communication, cohesion or a sense of belonging on the coalition, and complexity of the coalition structure during the intervention phase. Barriers to effective implementation included: staff turnover and staff lacking community organization skills, dependence on the state-level staff during the planning phase and lack of member input into the action plan. Conflict contributed to staff turnover, reluctance to conduct certain activities and difficulty in recruiting members, all of which had implications for implementation.

A process evaluation of the National Cancer Institute's Data-based Intervention Research program: a study of organizational capacity building.

This paper reports on a qualitative process evaluation of the Data-based Intervention Research (DBIR) program, that was funded by the National Cancer Institute (NCI) and operated in 21 states and the District of Columbia. The goal of DBIR was to build a foundation within state health agencies to ensure the translation of cancer control science into practice. NCI's objective reflected the readiness of cancer control research for public health application, the paucity of cancer control activity within public health settings and the recognition that state health agencies could play a critical role in the effective transfer of research results into public health practice. The qualitative process evaluation reported in this paper is based on one case study of four DBIR programs. The present study indicates that the four state health agencies executed the DBIR program with fidelity. Also, the four states offered a balanced assessment of NCI's role in enabling the state agency operation of DBIR, providing numerous citations illustrating how NCI successfully facilitated organizational capacity as compared to fewer mentions of ways NCI was less than successful. Thus, in funding the DBIR model, NCI was successful in raising state health agency capacity to implement cancer prevention and control programming. Implications for capacity building in state health departments are discussed.

The impact of the National Cancer Institute's Data-based Intervention Research program on state health agencies.

To assist state health agencies adopt a new role in cancer prevention and control, the National Cancer Institute (NCI) initiated the Data-based Intervention Research (DBIR) program. The goal of DBIR was to stimulate data-driven activities and to build capacity for ongoing programs within state health agencies to ensure the translation of cancer prevention and control science into practice across the US. Each state funded under the DBIR program was required to conduct four phases of activity: identifying and analyzing relevant data, using these data to develop a state cancer control plan, and implementing and evaluating prevention and control interventions at the local level. This paper presents the results of survey of the 22 states that participated in the DBIR program. The survey is intended as a supplement to the case study also reported in this issue of Health Education Research. Results indicated that states were able to implement the DBIR model and they show the process to be useful to their cancer prevention efforts. DBIR had a major impact on how states will use data in future planning for cancer prevention and control. States had a number of recommendations for how NCI could improve its working relationships with state health agencies.

Health education intervention strategies: recommendations for future research.

While the ultimate goal of health education interventions is to positively influence health status, more proximal indicators of success are changes in intermediate outcomes, or impact. Because health education interventions work through intermediate outcomes, the linkage to health status is often assumed to be at a conceptual or theoretical level. The term health education intervention strategy is a heuristic device used to conceptualize and organize a large variety of activities. There is a wide range of studies and reports in the literature that either test specific intervention strategies or report on larger health education efforts combining several strategies. This article organizes the discussion to focus on individual-, community-, and policy-level interventions. Mass communications are also considered, and the authors comment on program planning issues that cut across specific interventions at the individual, community, and policy levels. Eleven recommendations are offered for future health education intervention research.

Diffusion of innovations in schools: a study of adoption and implementation of school-based tobacco prevention curricula.

PURPOSE: The purpose of this study was to determine the extent of implementation of school health education curricula, to identify factors which enhanced or impeded implementation, and to examine the link between the adoption and implementation phases of the diffusion process. DESIGN: The study used an experimental design; 22 school districts were randomly assigned to intervention and control conditions. SETTING: The study was conducted in North Carolina. SUBJECTS: All teachers in the study districts identified by their schools as "eligible to teach health" were included in the sample. INTERVENTION: An in-depth training was conducted on the use of the middle school tobacco prevention curriculum that had been adopted. MEASURES: Three measures were used to assess implementation of the curricula. Independent variables of interest included organizational size and climate, teacher training, how long it took to make the adoption decision, and attitudes toward tobacco use prevention curricula. Response rates for these measures ranged from 44% to 78%. RESULTS: Nonparametric correlations and regression modeling indicated that larger organizational size and teacher training were the strongest predictors of curricula implementation. A favorable organizational climate within school districts also improved implementation. CONCLUSIONS: While interventions to increase adoption of school health education curricula should focus on larger school districts, the majority of efforts to improve implementation should focus on smaller districts.

Teachers' use of health curricula: implementation of growing healthy, project SMART, and the teenage health teaching modules.

This quasi-experimental study assessed impact of factors associated with classroom implementation of health curricula by North Carolina teachers. School representatives selected and implemented one of three tobacco prevention curricula--Project SMART, Growing Healthy, or the Teenage Health Teaching Modules--in either sixth or seventh grades. Prior to implementation, experimental teachers and administrators received extensive curricula training. Implementation data were collected through teacher completed checksheets and classroom observations for two time periods--initial implementation (n = 69) and maintained implementation (n = 136). While training was associated significantly with whether teachers implemented a curriculum (p < .05), other factors also were important. Variables outside of teachers' direct control, such as supportive administrators, context in which health instruction is taught, and turbulence, affected quantity and quality of curricular implementation.

Development of level of institutionalization scales for health promotion programs.

This study was conducted to test an instrument for measuring the level of institutionalization (LoIn) of health promotion programs. Institutionalization occurs when a program becomes an integral part of an organization, and the LoIn instrument is a beginning effort to measure the extent of program integration into organizations. The instrument is based on theory that holds that organizations are composed of production, maintenance, supportive, and managerial subsystems. Institutionalization occurs when a program becomes imbedded into these subsystems. A questionnaire designed to test this construct was mailed to 453 administrators in 141 organizations that operate health promotion programs. Based on 322 usable responses (71%), a confirmatory factor analysis was conducted. The results support the hypothesis of an eight-factor model: four factors concern how routinized the program was in each subsystem and four factors concern the degree of program saturation within each subsystem. Correlations of the eight factors with the number of years the programs had been in operation, and managers' perceptions of program permanency, indicated that the four routinization factors were more highly correlated with program longevity than the four niche saturation factors, and the niche saturation factors were more highly correlated with managers' perceptions of program permanence than the routinization factors. The instrument, which is available from the authors, may be used as both a research instrument and a diagnostic tool in assessing the institutionalization of health promotion programs.