A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care.

Background: Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner. Methods: In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes. Results: Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%. Conclusions: These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented. Trial registration: NCT03013309 ClinicalTrials.gov.

Research has shown that evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. However, if they are not implemented with fidelity, there is a potential that they will not produce the same effects. Gold-standard methods of implementation monitoring include observations of program sessions. This is expensive and difficult to implement in delivery settings with limited resources. Using data from a trial of the Family Check-Up 4 Health program in primary care settings that served Latino families, we investigated the potential to make use of a form of machine learning called natural language processing (NLP) to monitor program delivery. NLP-based ratings were significantly associated with independent observer ratings of fidelity and participant engagement outcomes. These results demonstrate the potential for NLP methods to monitor implementation in evidence-based parenting programs delivered at scale.

Categorising implementation determinants and strategies within the US HIV implementation literature: a systematic review protocol.

INTRODUCTION: Despite decreased rates of new infections, HIV/AIDS continues to impact certain US populations. In order to achieve the goals laid out in the Ending the HIV Epidemic (EHE) in the US initiative, implementation science is needed to expand the sustained use of effective prevention and treatment interventions, particularly among priority populations at risk for and living with HIV/AIDS. Over 200 HIV-related implementation studies have been funded by the US National Institutes of Health. Therefore, a comprehensive review of the literature identifying implementation determinants (barriers and facilitators) and categorising implementation strategies across the continuum of HIV prevention and care in the USA is appropriate and needed to enhance current knowledge and help achieve the goals laid out in the EHE national strategic plan. METHODS AND ANALYSIS: This systematic review protocol follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Between November 2020 and January 2022, a broad database search strategy of Ovid MEDLINE, PsycINFO and Web of Science was conducted to capture implementation-related studies along the HIV prevention and care continuum. Articles were eligible for inclusion if they were: conducted in the USA, published after the year 2000, written in English, related to HIV/AIDS, focused on outcomes related to dissemination and implementation (ie, did not test/evaluate/explore implementation determinants or strategies) and were behavioural studies (ie, not basic science). We plan to conduct three systematic reviews to identify and categorise determinants and strategies associated with three HIV focus areas: pre-exposure prophylaxis, testing/diagnosing and linkage to care, and treatment. Determinants will be coded according to an adapted Consolidated Framework for Implementation Research 2.0. Implementation strategies and outcomes will be categorised in accordance with existing taxonomies and frameworks. ETHICS AND DISSEMINATION: Ethics approval is not applicable. No original data will be collected. Results will be disseminated through peer-reviewed publications, conference presentations and via online tools. PROSPERO REGISTRATION NUMBER: CRD42021233089.

Implementation methodology from a social systems informatics and engineering perspective applied to a parenting training program.

OBJECTIVE: For implementation of an evidence-based program to be effective, efficient, and equitable across diverse populations, we propose that researchers adopt a systems approach that is often absent in efficacy studies. To this end, we describe how a computer-based monitoring system can support the delivery of the New Beginnings Program (NBP), a parent-focused evidence-based prevention program for divorcing parents. METHOD: We present NBP from a novel systems approach that incorporates social system informatics and engineering, both necessary when utilizing feedback loops, ubiquitous in implementation research and practice. Examples of two methodological challenges are presented: how to monitor implementation, and how to provide feedback by evaluating system-level changes due to implementation. RESULTS: We introduce and relate systems concepts to these two methodologic issues that are at the center of implementation methods. We explore how these system-level feedback loops address effectiveness, efficiency, and equity principles. These key principles are provided for designing an automated, low-burden, low-intrusive measurement system to aid fidelity monitoring and feedback that can be used in practice. DISCUSSION: As the COVID-19 pandemic now demands fewer face-to-face delivery systems, their replacement with more virtual systems for parent training interventions requires constructing new implementation measurement systems based on social system informatics approaches. These approaches include the automatic monitoring of quality and fidelity in parent training interventions. Finally, we present parallels of producing generalizable and local knowledge bridging systems science and engineering method. This comparison improves our understanding of system-level changes, facilitates a program's implementation, and produces knowledge for the field. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Redesigning Implementation Measurement for Monitoring and Quality Improvement in Community Delivery Settings.

The field of prevention has established the potential to promote child adjustment across a wide array of outcomes. However, when evidence-based prevention programs have been delivered at scale in community settings, declines in implementation and outcomes have resulted. Maintaining high quality implementation is a critical challenge for the field. We describe steps towards the development of a practical system to monitor and support the high-quality implementation of evidence-based prevention programs in community settings. Research on the implementation of an evidence-based parenting program for divorcing families called the "New Beginnings Program" serves as an illustration of the promise of such a system. As a first step, we describe a multidimensional theoretical model of implementation that links aspects of program delivery with improvements in participant outcomes. We then describe research on the measurement of each of these implementation dimensions and test their relations to intended program outcomes. As a third step, we develop approaches to the assessment of these implementation constructs that are feasible to use in community settings and to establish their reliability and validity. We focus on the application of machine learning algorithms and web-based data collection systems to assess implementation and provide support for high quality delivery and positive outcomes. Examples are presented to demonstrate that valid and reliable measures can be collected using these methods. Finally, we envision how these measures can be used to develop an unobtrusive system to monitor implementation and provide feedback and support in real time to maintain high quality implementation and program outcomes.

"Home Practice Is the Program": Parents' Practice of Program Skills as Predictors of Outcomes in the New Beginnings Program Effectiveness Trial.

An examination of the content and processes of evidence-based programs is critical for empirically evaluating theories about how programs work, the "action theory" of the program (West et al. in American Journal of Community Psychology, 21, 571-605, 1993). The New Beginnings Program (NBP; Wolchik et al., 2007), a parenting-after-divorce preventive intervention, theorizes that program-induced improvements in parenting across three domains: positive relationship quality, effective discipline, and protecting children from interparental conflict, will reduce the negative outcomes that are common among children from divorced families. The process theory is that home practice of program skills related to these parenting domains is the primary mechanism leading to positive change in parenting. This theory was tested using multi-rater data from 477 parents in the intervention condition of an effectiveness trial of the NBP (Sandler et al. 2016a, 2016b). Four research questions were addressed: Does home practice of skills predict change in the associated parenting outcomes targeted by the program? Is the effect above and beyond the influence of attendance at program sessions? What indicators of home practice (i.e., attempts, fidelity, efficacy, and competence) are most predictive of improvements in parenting? Do these indicators predict parenting improvements in underserved subpopulations (i.e., fathers and Latinos)? Structural Equation Modeling analyses indicated that parent-reported efficacy and provider-rated parent competence of home practice predicted improvements in the targeted parenting domains according to both parent and child reports. Moreover, indicators of home practice predicted improvements in parenting for fathers and Latinos, although patterns of effects varied by parenting outcome.

The Cascading Effects of Multiple Dimensions of Implementation on Program Outcomes: a Test of a Theoretical Model.

This study tests a theoretical cascade model in which multiple dimensions of facilitator delivery predict indicators of participant responsiveness, which in turn lead to improvements in targeted program outcomes. An effectiveness trial of the 10-session New Beginnings Program for divorcing families was implemented in partnership with four county-level family courts. This study included 366 families assigned to the intervention condition who attended at least one session. Independent observers provided ratings of program delivery (i.e., fidelity to the curriculum and process quality). Facilitators reported on parent attendance and parents' competence in home practice of program skills. At pretest and posttest, children reported on parenting and parents reported child mental health. We hypothesized effects of quality on attendance, fidelity and attendance on home practice, and home practice on improvements in parenting and child mental health. Structural Equation Modeling with mediation and moderation analyses were used to test these associations. Results indicated quality was significantly associated with attendance, and attendance moderated the effect of fidelity on home practice. Home practice was a significant mediator of the links between fidelity and improvements in parent-child relationship quality and child externalizing and internalizing problems. Findings provide support for fidelity to the curriculum, process quality, attendance, and home practice as valid predictors of program outcomes for mothers and fathers. Future directions for assessing implementation in community settings are discussed.

A computational future for preventing HIV in minority communities: how advanced technology can improve implementation of effective programs.

African Americans and Hispanics in the United States have much higher rates of HIV than non-minorities. There is now strong evidence that a range of behavioral interventions are efficacious in reducing sexual risk behavior in these populations. Although a handful of these programs are just beginning to be disseminated widely, we still have not implemented effective programs to a level that would reduce the population incidence of HIV for minorities. We proposed that innovative approaches involving computational technologies be explored for their use in both developing new interventions and in supporting wide-scale implementation of effective behavioral interventions. Mobile technologies have a place in both of these activities. First, mobile technologies can be used in sensing contexts and interacting to the unique preferences and needs of individuals at times where intervention to reduce risk would be most impactful. Second, mobile technologies can be used to improve the delivery of interventions by facilitators and their agencies. Systems science methods including social network analysis, agent-based models, computational linguistics, intelligent data analysis, and systems and software engineering all have strategic roles that can bring about advances in HIV prevention in minority communities. Using an existing mobile technology for depression and 3 effective HIV prevention programs, we illustrated how 8 areas in the intervention/implementation process can use innovative computational approaches to advance intervention adoption, fidelity, and sustainability.

Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: experiences of the prevention science and methodology group.

What progress prevention research has made comes through strategic partnerships with communities and institutions that host this research, as well as professional and practice networks that facilitate the diffusion of knowledge about prevention. We discuss partnership issues related to the design, analysis, and implementation of prevention research and especially how rigorous designs, including random assignment, get resolved through a partnership between community stakeholders, institutions, and researchers. These partnerships shape not only study design, but they determine the data that can be collected and how results and new methods are disseminated. We also examine a second type of partnership to improve the implementation of effective prevention programs into practice. We draw on social networks to studying partnership formation and function. The experience of the Prevention Science and Methodology Group, which itself is a networked partnership between scientists and methodologists, is highlighted.