Knowledge and Attitudes Toward an Artificial Intelligence-Based Fidelity Measurement in Community Cognitive Behavioral Therapy Supervision.

To capitalize on investments in evidence-based practices, technology is needed to scale up fidelity assessment and supervision. Stakeholder feedback may facilitate adoption of such tools. This evaluation gathered stakeholder feedback and preferences to explore whether it would be fundamentally feasible or possible to implement an automated fidelity-scoring supervision tool in community mental health settings. A partially mixed, sequential research method design was used including focus group discussions with community mental health therapists (n = 18) and clinical leadership (n = 12) to explore typical supervision practices, followed by discussion of an automated fidelity feedback tool embedded in a cloud-based supervision platform. Interpretation of qualitative findings was enhanced through quantitative measures of participants' use of technology and perceptions of acceptability, appropriateness, and feasibility of the tool. Initial perceptions of acceptability, appropriateness, and feasibility of automated fidelity tools were positive and increased after introduction of an automated tool. Standard supervision was described as collaboratively guided and focused on clinical content, self-care, and documentation. Participants highlighted the tool's utility for supervision, training, and professional growth, but questioned its ability to evaluate rapport, cultural responsiveness, and non-verbal communication. Concerns were raised about privacy and the impact of low scores on therapist confidence. Desired features included intervention labeling and transparency about how scores related to session content. Opportunities for asynchronous, remote, and targeted supervision were particularly valued. Stakeholder feedback suggests that automated fidelity measurement could augment supervision practices. Future research should examine the relations among use of such supervision tools, clinician skill, and client outcomes.

Automated evaluation of psychotherapy skills using speech and language technologies.

With the growing prevalence of psychological interventions, it is vital to have measures which rate the effectiveness of psychological care to assist in training, supervision, and quality assurance of services. Traditionally, quality assessment is addressed by human raters who evaluate recorded sessions along specific dimensions, often codified through constructs relevant to the approach and domain. This is, however, a cost-prohibitive and time-consuming method that leads to poor feasibility and limited use in real-world settings. To facilitate this process, we have developed an automated competency rating tool able to process the raw recorded audio of a session, analyzing who spoke when, what they said, and how the health professional used language to provide therapy. Focusing on a use case of a specific type of psychotherapy called "motivational interviewing", our system gives comprehensive feedback to the therapist, including information about the dynamics of the session (e.g., therapist's vs. client's talking time), low-level psychological language descriptors (e.g., type of questions asked), as well as other high-level behavioral constructs (e.g., the extent to which the therapist understands the clients' perspective). We describe our platform and its performance using a dataset of more than 5000 recordings drawn from its deployment in a real-world clinical setting used to assist training of new therapists. Widespread use of automated psychotherapy rating tools may augment experts' capabilities by providing an avenue for more effective training and skill improvement, eventually leading to more positive clinical outcomes.

Harnessing Innovative Technologies to Train Nurses in Suicide Safety Planning With Hospitalized Patients: Protocol for Formative and Pilot Feasibility Research.

BACKGROUND: Suicide is the 10th leading cause of death in the United States, with >47,000 deaths in 2019. Most people who died by suicide had contact with the health care system in the year before their death. Health care provider training is a top research priority identified by the National Action Alliance for Suicide Prevention; however, evidence-based approaches that target skill-building are resource intensive and difficult to implement. Advances in artificial intelligence technology hold promise for improving the scalability and sustainability of training methods, as it is now possible for computers to assess the intervention delivery skills of trainees and provide feedback to guide skill improvements. Much remains to be known about how best to integrate these novel technologies into continuing education for health care providers. OBJECTIVE: In Project WISE (Workplace Integrated Support and Education), we aim to develop e-learning training in suicide safety planning, enhanced with novel skill-building technologies that can be integrated into the routine workflow of nurses serving patients hospitalized for medical or surgical reasons or traumatic injury. The research aims include identifying strategies for the implementation and workflow integration of both the training and safety planning with patients, adapting 2 existing technologies to enhance general counseling skills for use in suicide safety planning (a conversational agent and an artificial intelligence-based feedback system), observing training acceptability and nurse engagement with the training components, and assessing the feasibility of recruitment, retention, and collection of longitudinal self-report and electronic health record data for patients identified as at risk of suicide. METHODS: Our developmental research includes qualitative and observational methods to explore the implementation context and technology usability, formative evaluation of the training paradigm, and pilot research to assess the feasibility of conducting a future cluster randomized pragmatic trial. The trial will examine whether patients hospitalized for medical or surgical reasons or traumatic injury who are at risk of suicide have better suicide-related postdischarge outcomes when admitted to a unit with nurses trained using the skill-building technology than those admitted to a unit with untrained nurses. The research takes place at a level 1 trauma center, which is also a safety-net hospital and academic medical center. RESULTS: Project WISE was funded in July 2019. As of September 2021, we have completed focus groups and usability testing with 27 acute care and 3 acute and intensive care nurses. We began data collection for research aims 3 and 4 in November 2021. All research has been approved by the University of Washington institutional review board. CONCLUSIONS: Project WISE aims to further the national agenda to improve suicide prevention in health care settings by training nurses in suicide prevention with medically hospitalized patients using novel e-learning technologies. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/33695.

Supporting collaborative reflection on personal values and health.

People with multiple chronic conditions (MCC) need support to identify and articulate how their personal values relate to their health. We drew on previous research involving people with MCC to develop three prototypes for supporting reflection on relationships between values and health. We tested these prototypes in a qualitative study involving 12 people with MCC. We identified benefits and limitations to building on patients' existing visit-preparation practices; revealed varying levels of comfort with deep, exploratory reflection involving a facilitator; and found that reflection oriented toward the future could elicit hopeful attitudes and plans for change, while reflection on the past elicited strong resistance. We discuss these findings in relation to previous literature on designing for reflection in three areas: shifting between self-guided and facilitator-guided reflection, balancing between outcome-oriented and exploratory reflection, and exploring temporality in reflection.

Design feasibility of an automated, machine-learning based feedback system for motivational interviewing.

Direct observation of psychotherapy and providing performance-based feedback is the gold-standard approach for training psychotherapists. At present, this requires experts and training human coding teams, which is slow, expensive, and labor intensive. Machine learning and speech signal processing technologies provide a way to scale up feedback in psychotherapy. We evaluated an initial proof of concept automated feedback system that generates motivational interviewing quality metrics and provides easy access to other session data (e.g., transcripts). The system automatically provides a report of session-level metrics (e.g., therapist empathy) and therapist behavior codes at the talk-turn level (e.g., reflections). We assessed usability, therapist satisfaction, perceived accuracy, and intentions to adopt. A sample of 21 novice (n = 10) or experienced (n = 11) therapists each completed a 10-min session with a standardized patient. The system received the audio from the session as input and then automatically generated feedback that therapists accessed via a web portal. All participants found the system easy to use and were satisfied with their feedback, 83% found the feedback consistent with their own perceptions of their clinical performance, and 90% reported they were likely to use the feedback in their practice. We discuss the implications of applying new technologies to evaluation of psychotherapy. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

"It's hard to argue with a computer:" Investigating Psychotherapists' Attitudes towards Automated Evaluation.

We present CORE-MI, an automated evaluation and assessment system that provides feedback to mental health counselors on the quality of their care. CORE-MI is the first system of its kind for psychotherapy, and an early example of applied machine-learning in a human service context. In this paper, we describe the CORE-MI system and report on a qualitative evaluation with 21 counselors and trainees. We discuss the applicability of CORE-MI to clinical practice and explore user perceptions of surveillance, workplace misuse, and notions of objectivity, and system reliability that may apply to automated evaluation systems generally.

Creating Conditions for Patients' Values to Emerge in Clinical Conversations: Perspectives of Health Care Team Members.

Eliciting, understanding, and honoring patients' values- the things most important to them in daily life-is a cornerstone of patient-centered care. However, this rarely occurs explicitly as a routine part of clinical practice. This is particularly problematic for individuals with multiple chronic conditions (MCC) because they face difficult choices about how to balance competing demands for self-care in accordance with their values. In this study, we sought to inform the design of interventions to support conversations about patient values between patients with MCC and their health care providers. We conducted a field study that included observations of 21 clinic visits for patients who have MCC, and interviews with 16 care team members involved in those visits. This paper contributes a practice-based account of ways in which providers engage with patient values, and discusses how future work in interactive systems design might extend and enrich these engagements.

Designing Contestability: Interaction Design, Machine Learning, and Mental Health.

We describe the design of an automated assessment and training tool for psychotherapists to illustrate challenges with creating interactive machine learning (ML) systems, particularly in contexts where human life, livelihood, and wellbeing are at stake. We explore how existing theories of interaction design and machine learning apply to the psychotherapy context, and identify "contestability" as a new principle for designing systems that evaluate human behavior. Finally, we offer several strategies for making ML systems more accountable to human actors.

Understanding What Is Most Important to Individuals with Multiple Chronic Conditions: A Qualitative Study of Patients' Perspectives.

BACKGROUND: To improve care for individuals living with multiple chronic conditions, patients and providers must align care planning with what is most important to patients in their daily lives. We have a limited understanding of how to effectively encourage communication about patients' personal values during clinical care. OBJECTIVE: To identify what patients with multiple chronic conditions describe as most important to their well-being and health. DESIGN: We interviewed individuals with multiple chronic conditions in their homes and analyzed results qualitatively, guided by grounded theory. PARTICIPANTS: A total of 31 patients (mean age 68.7 years) participated in the study, 19 of which included the participation of family members. Participants were from Kaiser Permanente Washington, an integrated health care system in Washington state. APPROACH: Qualitative analysis of home visits, which consisted of semi-structured interviews aided by photo elicitation. KEY RESULTS: Analysis revealed six domains of what patients described as most important for their well-being and health: principles, relationships, emotions, activities, abilities, and possessions. Personal values were interrelated and rarely expressed as individual values in isolation. CONCLUSIONS: The domains describe the range and types of personal values multimorbid older adults deem important to well-being and health. Understanding patients' personal values across these domains may be useful for providers when developing, sharing, and following up on care plans.

Factors Influencing the Food Purchases of Early Care and Education Providers.

BACKGROUND: With the majority of US children enrolled in some form of early care and education, the settings for early care and education represent a valuable opportunity to positively impact young children's diets and their interactions with food. Little evidence exists on how early care and education providers make food purchasing and service decisions for this population of young children. OBJECTIVE: Our aim was to explore the factors that influence early care and education providers' food purchasing and service decisions. DESIGN: A qualitative design consisting of individual, in-person, and semi-structured interviews with providers and on-site observations was used. PARTICIPANTS/SETTING: Sixteen early care and education providers-selected across a variety of characteristics that might affect food selection (eg, size of site, participation in reimbursement programs, presence of staff assigned to foodservice) using maximum variation purposive sampling-based in the Puget Sound region, Washington, were interviewed from June to September 2014. MAIN OUTCOME MEASURE: Provider perspectives on food purchasing and service decisions. STATISTICAL ANALYSES PERFORMED: Inductive analysis of transcribed interviews using TAMS Analyzer software (GPL version 2, 2012) to identify themes. RESULTS: Ten main influencers emerged from the data. These were grouped into four categories based on an ecological framework: macro-level environments (ie, regulations; suppliers and vendors, including stores); physical environment and settings (ie, organizational mission, budget, and structure; the facility itself); social environments (ie, professional networks; peers; the site-specific parent and child community); and individual factors at both a provider and child-level (ie, providers' skills, behaviors, motivations, attitudes, knowledge, and values; child food preferences; and, child allergies). A model was then developed to identify potential pathways of intervention and underscore the need for a comprehensive approach to improve early care and education nutrition. CONCLUSIONS: This study suggests that a more system-based understanding and approach-one that accounts for an array of influencers and their interactions-is necessary to take advantage of opportunities and address barriers to improving early care and education-based nutrition.

Eliciting Values of Patients with Multiple Chronic Conditions: Evaluation of a Patient-centered Framework.

Patients with multiple chronic conditions often face competing demands for care, and they often do not agree with physicians on priorities for care. Patients ' values shape their healthcare priorities, but existing methods for eliciting values do not necessarily meet patients ' care planning needs. We developed a patient-centered values framework based on a field study with patients and caregivers. In this paper we report on a survey to evaluate how the framework generalizes beyond field study participants, and how well the framework supports values elicitation. We found that respondents frame values in a way that is consistent with the framework, and that domains of the framework can be used to elicit a breadth of potential values individuals with MCC express. These findings demonstrate how a patient-centered perspective on values can expand on the domains considered in values clarification methods andfacilitate patient-provider communication in establishing shared care priorities.

"It just seems outside my health": How Patients with Chronic Conditions Perceive Communication Boundaries with Providers.

To improve care for the growing number of older adults with multiple chronic conditions, physicians and other healthcare providers need to better understand what is most important in the lives of these patients. In a qualitative study of home visits with patients and family caregivers, we found that patients withhold information from providers when communicating about what they deem important to their health and well-being. We examine the various motivations and factors that explain communication boundaries between patients and their healthcare providers. Patients' disclosures reflected perceptions of what was pertinent to share, assumptions about the consequences of sharing, and the influence of interpersonal relationships with providers. Our findings revealed limitations of existing approaches to support patient-provider communication and identified challenges for the design of systems that honor patient needs and preferences.