A Review of Public Health Training Needs Assessment Approaches: Opportunities to Move Forward.

CONTEXT: Assessing training needs of the public health workforce is crucial for creating professional development opportunities to improve knowledge, competence, and effectiveness of this workforce. DISSEMINATION: Regional Public Health Training Centers (RPHTCs) assess workforce training needs and deliver training based on identified needs. To determine training priorities, several needs assessment surveys have been administered by RPHTCs and national public health member organizations. EVALUATION: This study identified the types of training questions being asked to public health practitioners in the various assessment surveys implemented by RPHTCs and national membership organizations. Although the surveys measured similar overarching constructs, multiple approaches with limited consistency were used to measure training needs. DISCUSSION: Although successful in responding to the needs of their targeted constituents, the limited consistency among survey types makes generalization of findings difficult. Disseminating common metrics and aggregate survey findings would increase efficiency in determining workforce training needs and developing targeted training.

Driving a public health culture of quality: how far down the highway have local health departments traveled?

CONTEXT: There has been an extensive investment in building public health organizational capacity to improve performance and prepare for accreditation. An evolving perspective has focused not only on the practice of quality improvement (QI) within the health department but also upon the extent the culture of QI is embraced within the agency. OBJECTIVE: No studies have examined the current national baseline of QI culture implementation, nor estimated the degree of QI sophistication local health departments (LHDs) have attained. We attempt to fill this void by aligning the findings from the QI module of the National Association of County & City Health Officials (NACCHO) 2010 Profile of LHDs against the constructs defined by the QI Maturity Tool and the NACCHO QI Roadmap (Roadmap to a Culture of Quality Improvement). DESIGN: Specific questions regarding QI activities from the 2010 Profile Study QI module were used to assign responding LHDs to stages within the Roadmap. We also used data from the QI Maturity Tool administered to all LHDs in the 16 participating Multi-State Learning Collaborative states in 2010 and 2011. On the basis of this matched set, we applied the summative domain scores algorithm, classified agencies into 1 of 5 groups, compared our findings with those of the NACCHO survey, and aligned our categories to those of the Roadmap. RESULTS: Nearly 80% of LHDs classified using the NACCHO Profile data were assigned to group 3 or 4 versus 48% using the QI Maturity Tool. Results from the cross-tabulations of the matched data set between the QI Maturity Tool classifications and the NACCHO Profile classifications revealed exact alignment 30% of the time. Forty-nine of 163 agencies were classified in the same grouping in both schemata. In addition, 84% of the agencies were classified within 1 neighboring category. CONCLUSIONS: The results revealed that half, if not most, LHDs fall within the middle categories of QI maturity and sophistication, regardless of which classification system was deployed.

Measuring quality improvement in public health: the development and psychometric testing of a QI Maturity Tool.

There is growing interest and investment in improving the quality of public health services and outcomes. Following the lead of other sectors, efforts are underway to introduce systematic quality improvement (QI) tools and approaches to state and local public health agencies. Little is known, however, about how to describe and reliably measure the level of QI maturity within a public health agency. The authors describe the development of a QI Maturity Tool using research from the fields of organizational design, psychology, health care, and complexity theory. The 37-item assessment tool is based on four quality domains derived from the literature: (a) organizational culture, (b) capacity and competency, (c) practice, and (d) alignment and spread. The tool was designed to identify features of an organization that may be enhancing or impeding QI; monitor the impact of efforts to create a more favorable environment for QI; and define potential cohorts of public health agencies for evaluation purposes. The article presents initial steps in testing and validating the QI Maturity Tool including: (a) developing a theoretical framework, (b) assuring face and content validity, (c) determining the tool's reliability based on estimates of internal consistency, (d) assessing the dimensionality, and (f) determining the construct validity of the instrument. The authors conclude that there is preliminary evidence that the QI Maturity Tool is a promising instrument. Further work is underway to explore whether self-reported survey results align with an agency's actions and the products of their QI efforts.

Assessing quality improvement in local health departments: results from the Multi-State Learning Collaborative.

OBJECTIVE: This study examined changes in quality improvement: (1) over a 3-year period, (2) among mini-collaborative participants, and (3) among agencies that were classified in the lowest QI performance quartile. METHODS: : A QI Maturity Tool was administered to all local health departments in the Multi-State Learning Collaborative states. Factorial ANOVA was performed to determine differences in composite factor scores and interaction effects were explored. RESULTS: : The results revealed a significant increase in the percent of agencies that reported ever implementing a formal QI process from 2009 to 2011 and agencies that participated in a mini-collaborative were more likely than their non-mini-collaborative counterparts to report QI implementation. The findings also suggested significant changes in QI capacity and competency as well as alignment and spread over the course of the grant, among mini-collaborative participants, and among agencies that were originally identified in the lowest QI maturity quartile. No significant changes in organizational culture were revealed by year or among mini-collaborative participants. However, agencies in the lowest quartile saw significant changes in organizational culture during the project period. CONCLUSIONS: Significant changes in specific QI domains are possible to detect during a 3-year period. Our research gives us confidence that the QI Maturity Tool measures the right domains and is an important contribution to quantifying the adoption and spread of QI in public health. However, further refinement is needed to better standardize language and definitions of the component parts of a QI system.

Quality improvement learning collaboratives in public health: findings from a multisite case study.

OBJECTIVES: This study was designed to (1) test a theory on the key drivers of a successful mini-collaborative, (2) describe the application of quality improvement (QI) approaches and techniques among mini-collaborative participants, and (3) identify key attributes that affect the spread and sustainability of QI efforts within a local public health agency. METHODS: A multisite case study methodology was used to evaluate a subset of mini-collaboratives and their participating local health departments that were working in specific target areas. Data were collected during semistructured interviews and while observing mini-collaborative meetings. Documentation reviews were also conducted. We used standard techniques to code the data on the basis of themes and connections between themes. RESULTS: The findings provide early evidence that support our case study theory on the drivers of a successful mini-collaborative including advanced planning, the selection of faculty, timely training and technical assistance, the role of senior leaders, the application of evidence-based practices, the use of an improvement model, evaluation efforts, communication, the availability of resources, target selection, and prior experience with and application of QI. While the case studies provided limited evidence of sustainability and broad spread of QI within participating states, mini-collaboratives appear to have had a major impact and will likely influence the work of participating local health departments going forward. CONCLUSIONS: Our findings suggest that the mini-collaboratives served as a catalyst for engaging local health departments in the theory and practice of QI.

Evaluating the Multi-State Learning Collaborative.

The third phase of the Multi-State Learning Collaborative (MLC) is a 3-year Robert Wood Johnson Foundation initiative designed to build quality improvement capacity within state and local health departments. This commentary provides a brief overview of the MLC, a methodological description of its evaluation, and preliminary findings. Major evaluation data sources include quarterly reports, surveys, case studies, and key informant interviews. Preliminary survey data and quarterly reports from year 1 provide some early evidence that states are progressing toward MLC goals. Approximately 27 percent of local health departments in the 16 participant states report that they have begun to prepare for national accreditation and approximately 39 percent report implementation of at least one quality improvement project within the past 12 months. Ongoing data collection efforts are underway to more fully address the evaluation questions.

Linking accreditation and public health outcomes: a logic model approach.

Emerging public health standards, performance assessment tools, and accreditation models hold significant promise for defining and standardizing public health practice, yet the lack of empirical research on their relationship to outcomes represents a serious barrier to adoption. Given the growing interest and momentum related to public health agency assessment and accreditation efforts, there is increasing need for evidence that performance standards and associated accreditation programs are effective means for moving public health systems toward the ultimate goal of population and community health improvement. This article provides an overview of accreditation in health and other industries, and its relationship to outcomes. We examine lessons that might have meaningful public health translations, as well as influences in and on public health that pose challenges for research and evaluation in this area. Finally, we propose a logic model framework to help depict the ways in which we can begin to explore the impact accreditation has on various levels of outcomes. This logic model is intended to guide the development of measures and to serve as a tool to help convey the breadth and depth of research needed to link accreditation to health outcomes.

Use of national public health performance standards to assess Maine's diabetes system.

In the fall of 2003, Maine underwent a rigorous assessment of the diabetes public health system using a modified version of the State Public Health System Performance Instrument developed by the Centers for Disease Control and Prevention's National Public Health Performance Standards Program and other national partners. The assessment was intended to serve as the impetus for the development of a statewide improvement plan. This article details the assessment process and provides a case study highlighting the successful application of systems-based model standards for a categorical issue.

Writing community-centered evaluation reports.

Documenting the process, results, impact, and effectiveness of community-based health promotion programs is an important part of any evaluation. This article provides information on how to write community-centered evaluation reports for program stake-holders. Specific prerequisites and principles are provided. In addition, several tips for increasing the use of the results are highlighted.