Incorporating multiple data sources to assess changes in organizational capacity in child welfare systems.

Three federally funded Child Welfare Capacity Building Centers provide services to build the organizational capacity of public child welfare agencies to help meet federal requirements, improve practice, and improve outcomes for children and families. The aim of this study was to explore capacity outcomes in five dimensions - resources, infrastructure, knowledge and skills, culture and climate, and engagement and partnership - achieved by child welfare jurisdictions who received Center services. Analyses describe the capacities targeted for improvement and the amount and type of services provided by Centers; assess the relationship between services and capacity increase; and explore whether that relationship differs depending on the jurisdiction's level of foundational capacity. Data collected through surveys and a service delivery tracking system reflect the perspectives of service recipients and service providers. Results reveal jurisdictions typically targeted capacity outcomes in the dimensions of knowledge and skills and infrastructure and received an average of 28 hours of direct services to support their capacity-building efforts. Dosage of service was positively associated with achievement of capacity outcomes, though no interaction was found between service dosage and foundational capacity in the effect on outcomes. Methodological lessons learned and implications for future evaluations of organizational capacity building efforts are offered.

Usability testing, initial implementation, and formative evaluation of an evidence-based intervention: lessons from a demonstration project to reduce long-term foster care.

The field of child welfare faces an undersupply of evidence-based interventions to address long-term foster care. The Permanency Innovations Initiative is a five-year federal demonstration project intended to generate evidence to reduce long stays in foster care for those youth who encounter the most substantial barriers to permanency. This article describes a systematic and staged approach to implementation and evaluation of a PII project that included usability testing as one of its key activities. Usability testing is an industry-derived practice which analyzes early implementation processes and evaluation procedures before they are finalized. This article describes the iterative selection, testing, and analysis of nine usability metrics that were designed to assess three important constructs of the project's initial implementation and evaluation: intervening early, obtaining consent, and engaging parents. Results showed that seven of nine metrics met a predetermined target. This study demonstrates how findings from usability testing influenced the initial implementation and formative evaluation of an evidence-supported intervention. Implications are discussed for usability testing as a quality improvement cycle that may contribute to better operationalized interventions and more reliable, valid, and replicable evidence.