Evaluating Intensity, Complexity, and Potential for Causal Inference in Social Needs Interventions: A Review of a Scoping Review.

Importance: Interventions that address needs such as low income, housing instability, and safety are increasingly appearing in the health care sector as part of multifaceted efforts to improve health and health equity, but evidence relevant to scaling these social needs interventions is limited. Objective: To summarize the intensity and complexity of social needs interventions included in randomized clinical trials (RCTs) and assess whether these RCTs were designed to measure the causal effects of intervention components on behavioral, health, or health care utilization outcomes. Evidence Review: This review of a scoping review was based on a Patient-Centered Outcomes Research Institute-funded evidence map of English-language US-based RCTs of social needs interventions published between January 1, 1995, and April 6, 2023. Studies were assessed for features related to intensity (defined using modal values as providing as-needed interaction, 8 participant contacts or more, contacts occurring every 2 weeks or more often, encounters of 30 minutes or longer, contacts over 6 months or longer, or home visits), complexity (defined as addressing multiple social needs, having dedicated staff, involving multiple intervention components or practitioners, aiming to change multiple participant behaviors [knowledge, action, or practice], requiring or providing resources or active assistance with resources, and permitting tailoring), and the ability to assess causal inferences of components (assessing interventions, comparators, and context). Findings: This review of a scoping review of social needs interventions identified 77 RCTs in 93 publications with a total of 135 690 participants. Most articles (68 RCTs [88%]) reported 1 or more features of high intensity. All studies reported 1 or more features indicative of high complexity. Because most studies compared usual care with multicomponent interventions that were moderately or highly dependent on context and individual factors, their designs permitted causal inferences about overall effectiveness but not about individual components. Conclusions and Relevance: Social needs interventions are complex, intense, and include multiple components. Our findings suggest that RCTs of these interventions address overall intervention effectiveness but are rarely designed to distinguish the causal effects of specific components despite being resource intensive. Future studies with hybrid effectiveness-implementation and sequential designs, and more standardized reporting of intervention intensity and complexity could help stakeholders assess the return on investment of these interventions.

Performance of two large language models for data extraction in evidence synthesis.

Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT-4) for extracting pre-specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT-4. GPT-4 required use of a third-party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT-4 with the plug-in was lower (68.8%); however, most of the errors were due to the plug-in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT-4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open-source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.

Literature Review of Pathogen Agnostic Molecular Testing of Clinical Specimens From Difficult-to-Diagnose Patients: Implications for Public Health.

To better identify emerging or reemerging pathogens in patients with difficult-to-diagnose infections, it is important to improve access to advanced molecular testing methods. This is particularly relevant for cases where conventional microbiologic testing has been unable to detect the pathogen and the patient's specimens test negative. To assess the availability and utility of such testing for human clinical specimens, a literature review of published biomedical literature was conducted. From a corpus of more than 4,000 articles, a set of 34 reports was reviewed in detail for data on where the testing was being performed, types of clinical specimens tested, pathogen agnostic techniques and methods used, and results in terms of potential pathogens identified. This review assessed the frequency of advanced molecular testing, such as metagenomic next generation sequencing that has been applied to clinical specimens for supporting clinicians in caring for difficult-to-diagnose patients. Specimen types tested were from cerebrospinal fluid, respiratory secretions, and other body tissues and fluids. Publications included case reports and series, and there were several that involved clinical trials, surveillance studies, research programs, or outbreak situations. Testing identified both known human pathogens (sometimes in new sites) and previously unknown human pathogens. During this review, there were no apparent coordinated efforts identified to develop regional or national reports on emerging or reemerging pathogens. Therefore, development of a coordinated sentinel surveillance system that applies advanced molecular methods to clinical specimens which are negative by conventional microbiological diagnostic testing would provide a foundation for systematic characterization of emerging and underdiagnosed pathogens and contribute to national biodefense strategy goals.

Data extraction for evidence synthesis using a large language model: A proof-of-concept study.

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

Recovery schools for improving behavioral and academic outcomes among students in recovery from substance use disorders: a systematic review.

This Campbell systematic review examines the effects of recovery schools on student behavioral and academic outcomes, compared to the effects of non-recovery schools. The review summarizes evidence from one quasi-experimental study (with a total of 194 participants) that had potential serious risk of bias due to confounding. Sizable portions of youth are in recovery from substance use disorders, and many youth will return to use after receiving substance use treatment. Youth spend most of their waking hours at school, and thus schools are important social environments for youth in recovery from substance use disorders. Recovery schools have been identified as educational programs that may help support youth in recovery from substance use disorders. This review focused on two types of recovery schools: RHSs, which are schools that award secondary school diplomas and offer a range of therapeutic services in addition to standard educational curricula; and CRCs, which offer therapeutic and sober support services on college campuses. This review looked at whether recovery schools (RHSs or CRCs) affect academic success and substance use outcomes among students, compared to similar students who are not enrolled in recovery schools. Plain language summary: There is insufficient evidence to know whether recovery high schools and collegiate recovery communities are effective: Evidence that recovery high schools (RHSs) may improve academic and substance use outcomes is based on the findings from a single study with a serious risk of bias.The review in brief: Very limited evidence addresses the effectiveness of recovery high schools (RHSs). There is no rigorous evidence on the effectiveness of collegiate recovery communities (CRCs).It is unclear whether CRCs are effective in promoting academic success and reducing substance use among college students.What is the aim of this review?: This Campbell systematic review examines the effects of recovery schools on student behavioral and academic outcomes, compared to the effects of non-recovery schools. The review summarizes evidence from one quasi-experimental study (with a total of 194 participants) that had potential serious risk of bias due to confounding.What are the main findings of this review?: Sizable portions of youth are in recovery from substance use disorders, and many youth will return to use after receiving substance use treatment. Youth spend most of their waking hours at school, and thus schools are important social environments for youth in recovery from substance use disorders. Recovery schools have been identified as educational programs that may help support youth in recovery from substance use disorders.This review focused on two types of recovery schools: RHSs, which are schools that award secondary school diplomas and offer a range of therapeutic services in addition to standard educational curricula; and CRCs, which offer therapeutic and sober support services on college campuses.This review looked at whether recovery schools (RHSs or CRCs) affect academic success and substance use outcomes among students, compared to similar students who are not enrolled in recovery schools.What studies are included?: The included study of recovery high schools used a controlled quasi-experimental pretest-posttest design and reported on the following outcomes: grade point average, truancy, school absenteeism, alcohol use, marijuana use, other drug use, and abstinence from alcohol/drugs. The included study focused on a sample of U.S. high school students. There were no eligible studies of CRCs.What do the findings of this review mean?: Findings from this review indicate insufficient evidence on the effects of recovery schools on student well-being. Although there is some indication RHSs may improve academic and substance use outcomes, this is based on the findings from a single study. There is no available evidence on the effects of CRCs.No strong conclusions can be drawn at this time, given the lack of available evidence on RHSs and CRCs, and the serious risk of bias in the one RHS study included in the review. The evidence from this review suggests there is a clear need for additional rigorous evaluations of recovery school effects prior to widespread implementation.How up-to-date is this review?: The review authors searched for studies until September 2018. This Campbell systematic review was published in 2018. Executive Summary/Abstract: BACKGROUND: Substance use disorders (SUDs) among youth are a major public health problem. In the United States, for example, the incidence of SUDs increases steadily after age 12 and peaks among youth ages 18-23 (White, Evans, Ali, Achara-Abrahams, & King, 2009). Although not every youth who experiments with alcohol or illicit drugs is diagnosed with an SUD, approximately 7-9% of 12-24 year olds in the United States were admitted for public SUD treatment in 2013 (Substance Abuse and Mental Health Services Administration [SAMHSA], 2016). Recovery from an SUD involves reduction or complete abstinence of use, defined broadly as "voluntarily sustained control over substance use, which maximises health and wellbeing and participation in the rights, roles and responsibilities of society" (UK Drug Policy Commission, 2008). However, SUDs are often experienced as chronic conditions; among youth who successfully complete substance use treatment, approximately 45-70% return to substance use within months of treatment discharge (Anderson, Ramo, Schulte, Cummins, & Brown, 2007; Brown, D'Amico, McCarthy, & Tapert, 2001; Ramo, Prince, Roesch, & Brown, 2012; White et al., 2004). Thus, multiple treatment episodes and ongoing recovery supports after treatment are often necessary to assist with the recovery process (Brown et al., 2001; Ramo et al., 2012; White et al., 2004).Success and engagement at school and in postsecondary education are critical to healthy youth development. For youth in recovery from SUDs, school attendance, engagement, and achievement build human capital by motivating personal growth, creating new opportunities and social networks, and increasing life satisfaction and meaning (Keane, 2011; Terrion, 2012; 2014). Upon discharge from formal substance use treatment settings, schools become one of the most important social environments in the lives of youth with SUDs. Healthy school peer environments can enable youth to replace substance use behaviors and norms with healthy activities and prosocial, sober peers. Conversely, many school environments may be risky for youth in recovery from SUDs due to perceived substance use among peers, availability of drugs or alcohol, and substance-approving norms on campus (Centers for Disease Control [CDC], 2011; Spear & Skala, 1995; Wambeam, Canen, Linkenbach, & Otto, 2014).Given the many social and environmental challenges faced by youth in recovery from substance use, recovery-specific institutional supports are increasingly being linked to educational settings. The two primary types of education-based continuing care supports for youth in recovery, defined under the umbrella term of "recovery schools" for this review, are recovery high schools (RHSs) and collegiate recovery communities (CRCs). RHSs are secondary schools that provide standard high school education and award secondary school diplomas, but also include therapeutic programming aimed at promoting recovery (e.g., group check-ins, community service, counseling sessions). CRCs also provide recovery oriented support services (e.g., self-help groups, counseling sessions, sober dorms) for students, but are embedded within larger college or university settings. The primary aims of RHSs and CRCs are to promote abstinence and prevent relapse among students, and thus ultimately improve students' academic success.OBJECTIVES: This review summarized and synthesized the available research evidence on the effects of recovery schools for improving academic success and behavioural outcomes among high school and college students who are in recovery from substance use. The specific research questions that guided the review are as follows: 1. What effect does recovery school attendance (versus attending a non-recovery or traditional school setting) have on academic outcomes for students in recovery from substance use? Specifically (by program type): a. For recovery high schools: what are the effects on measures of academic achievement, high school completion, and college enrolment?b. For collegiate recovery communities: what are the effects on measures of academic achievement and college completion?2. What effect does recovery school attendance have on substance use outcomes for students in recovery from substance use? Specifically, what are the effects on alcohol, marijuana, cocaine, or other substance use?3. Do the effects of recovery schools on students' outcomes vary according to the race/ethnicity, gender, or socioeconomic status of the students?4. Do the effects of recovery schools on students' outcomes vary according to existing mental health comorbidity status or juvenile justice involvement of the students? SEARCH METHODS: We aimed to identify all published and unpublished literature on recovery schools by using a comprehensive and systematic literature search. We searched multiple electronic databases, research registers, grey literature sources, and reference lists from prior reviews; and contacted experts in the field.SELECTION CRITERIA: Studies were included in the review if they met the following criteria:Types of studies: Randomized controlled trial (RCT), quasi-randomized controlled trial (QRCT), or controlled quasi-experimental design (QED).Types of participants: Students in recovery from substance use who were enrolled part-time or full-time in secondary (high school) or postsecondary (college or university) educational institutions.Types of interventions: Recoveryschools broadly defined as educational institutions, or programs at educational institutions, developed specifically for students in recovery and that address recovery needs in addition to academic development.Types of comparisons: Traditional educational programs or services that did not explicitly have a substance use recovery focus.Types of outcome measures: The review focused on primary outcomes in the following two domains: academic performance (e.g., achievement test scores, grade-point average, high school completion, school attendance, college enrolment, college completion) and substance use (alcohol, marijuana, cocaine, heroin, stimulant, mixed drug use, or other illicit drug use). Studies that met all other eligibility criteria were considered eligible for the narrative review portion of this review even if they did not report outcomes in one of the primary outcome domains.Other criteria: Studies must have been reported between 1978 and 2016. The search was not restricted by geography, language, publication status, or any other study characteristic.DATA COLLECTION AND ANALYSIS: Two reviewers independently screened all titles and abstracts of records identified in the systematic search. Records that were clearly ineligible or irrelevant were excluded at the title/abstract phase; all other records were retrieved in full-text and screened for eligibility by two independent reviewers. Any discrepancies in eligibility assessments were discussed and resolved via consensus. Studies that met the inclusion criteria were coded by two independent reviewers using a structured data extraction form; any disagreements in coding were resolved via discussion and consensus. If members of the review team had conducted any of the primary studies eligible for the review, external and independent data collectors extracted data from those studies. Risk of bias was assessed using the ROBINS-I tool for non-randomized study designs (Sterne, Higgins, & Reeves, 2016).Inverse variance weighted random effects meta-analyses were planned to synthesize effect sizes across studies, as well as heterogeneity analysis, subgroup analysis, sensitivity analysis, and publication bias analysis. However, these synthesis methods were not used given that only one study met the inclusion criteria for the review. Instead, effect sizes (and their corresponding 95% confidence intervals) were reported for all eligible outcomes reported in the study.RESULTS: Only one study met criteria for inclusion in the review. This study used a QED to examine the effects of RHSs on high school students' academic and substance use outcomes. No eligible studies examining CRCs were identified in the search.The results from the one eligible RHS study indicated that after adjusting for pretest values, students in the RHS condition reported levels of grade point averages (= 0.26, 95% CI [-0.04, 0.56]), truancy (= 0.01, 95% CI [-0.29, 0.31]), and alcohol use (= 0.23, 95% CI [-0.07, 0.53]) similar to participants in the comparison condition. However, students in the RHS condition reported improvements in absenteeism (= 0.56, 95% CI [0.25, 0.87]), abstinence from alcohol/drugs (OR = 4.36, 95% CI [1.19, 15.98]), marijuana use (= 0.51, 95% CI [0.20, 0.82]), and other drug use (= 0.45, 95% CI [0.14, 0.76]).Overall, there was a serious risk of bias in the one included study. The study had a serious risk of bias due to confounding, low risk of bias due to selection of participants into the study, moderate risk of bias due to classification of interventions, inconclusive risk of bias due to deviations from intended interventions, inconclusive risk of bias due to missing data, moderate risk of bias in measurement of outcomes, and low risk of bias in selection of reported results.AUTHORS' CONCLUSIONS: There is insufficient evidence regarding the effectiveness of RHSs and CRCs for improving academic and substance use outcomes among students in recovery from SUDs. Only one identified study examined the effectiveness of RHSs. Although the study reported some beneficial effects, the results must be interpreted with caution given the study's potential risk of bias due to confounding and limited external validity. No identified studies examined the effectiveness of CRCs across the outcomes of interest in this review, so it is unclear what effects these programs may have on students' academic and behavioral outcomes.The paucity of evidence on the effectiveness of recovery schools, as documented in this review, thus suggest the need for caution in the widespread adoption of recovery schools for students in recovery from SUDs. Given the lack of empirical support for these recovery schools, additional rigorous evaluation studies are needed to replicate the findings from the one study included in the review. Furthermore, additional research examining the costs of recovery schools may be needed, to help school administrators determine the potential cost-benefits associated with recovery schools.