Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale.

Background: Usability - the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction - may be a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in the current study of the IUS. Method: A survey administered to 136 medical professionals from 11 primary care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS. Results: Analyses indicated that a two-factor solution (with "usable" and "learnable" subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively. Conclusions: This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Measurement Based Care in Child Welfare-Involved Children and Youth: Reliability and Validity of the PSC-17.

This study evaluates whether the psychometric properties of the Pediatric Symptoms Checklist-17 (PSC-17), a common behavioral health measure typically used as a dichotomous screening tool for mental health needs, support its use as a continuous measure for tracking behavioral health over time. A total of 6492 foster parents of children and youth aged 5.5-17 completed the PSC-17. Convergent and discriminant validity was assessed by comparing raw PSC-17 subscale scores with associated outcomes (e.g. psychiatric diagnoses). Long-term test-retest reliability was assessed over 6 months. Scores on the PSC-17 demonstrated good convergent and divergent validity. PSC-17 subscale scores were most strongly associated with analogous diagnoses. Test-retest reliability was moderate, as expected for a time window of this length. This study provides moderate support for the psychometric qualities of the PSC-17 when used with children and youth in the child welfare system as a continuous measure of psychosocial functioning over time.

Identifying Psychosocial Problems Among Children and Youth in the Child Welfare System Using the PSC-17: Exploring Convergent and Discriminant Validity with Multiple Informants.

Youth who enter foster care are at risk of mental health need, but questions arise as to the validity of their self-reported symptomatology. This study examines the screening validity of the youth-report version of the Pediatric Symptom Checklist-17 (PSC-17) in a child welfare population. Data come from 2389 youth who completed a version of the PSC-17 adapted for youth report, and their biological and foster parents who completed the parent-report version. Youth also completed a shortened version of the Screen for Child Anxiety Related Disorders (SCARED). Convergent and discriminant validity of the PSC-17 was assessed using multi-trait multi-method matrices. The PSC-17's internalizing subscale was strongly correlated, attention subscale was moderately correlated, and externalizing subscale was weakly correlated with the SCARED's anxiety and PTSD subscales. Comparing youth and foster parent scores, the PSC-17 had moderate convergent validity and weak/fair discriminant validity. Comparing youth, foster parent, and biological parent scores, the PSC-17 had moderate convergent validity and weak/fair discriminant validity. The current study provides some support for the validity of the PSC-17 for the population of youth in foster care.