Fine-grained effect sizes.

To make transparent individuals' responses to intervention over time in the systematic review of single-case experimental designs, we developed a method of estimating and graphing fine-grained effect sizes. Fine-grained effect sizes are both case- and time-specific and thus provide more nuanced information than effect size estimates that average effects across time, across cases, or both. We demonstrate the method for estimating fine-grained effect sizes under three different baseline stability assumptions: outcome stability, level stability, and trend stability. We then use the method to graph individual effect trajectories from three single-case experimental design studies that examined the impact of self-management interventions on students identified with autism. We conclude by discussing limitations associated with estimating and graphing fine-grained effect sizes and directions for further development. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Single-case intervention research design standards: Additional proposed upgrades and future directions.

Single-case intervention research design standards have evolved considerably over the past decade. These standards serve the dual role of assisting in single-case design (SCD) intervention research methodology and as guidelines for literature syntheses within a particular research domain. In a recent article (Kratochwill et al., 2021), we argued for a need to clarify key features of these standards. In this article we offer additional recommendations for SCD research and synthesis standards that have been either underdeveloped or missing in the conduct of research and in literature syntheses. Our recommendations are organized into three categories: expanding design standards, expanding evidence standards, and expanding the applications and consistency of SCDs. The recommendations we advance are for consideration for future standards, research design training, and they are especially important to guide the reporting of SCD intervention investigations as they enter the literature-synthesis phase of evidence-based practice initiatives.

Randomization Procedures for Changing Criterion Designs.

To strengthen the scientific credibility arguments for single-case intervention studies, randomization design-and-analysis methods have been developed for the multiple-baseline, ABAB, and alternating treatment designs, including options for preplanned designs, wherein the series and phase lengths are established prior to gathering data, as well as options for response-guided designs, wherein ongoing visual analyses guide decisions about when to intervene. Our purpose here is to develop randomization methods for another class of single-case design, the changing criterion design. We first illustrate randomization design-and-analysis methods for preplanned changing criterion designs and then develop and illustrate methods for response-guided changing criterion designs. We discuss the limitations associated with the randomization methods and the validity of the corresponding intervention-effect inferences.

Effects of Explicit Vocabulary Interventions for Preschoolers: An Exploratory Application of the Percent of Goal Obtained Effect Size Metric.

PURPOSE: Systematic reviews of literature are routinely conducted to identify practices that are effective in addressing educational and clinical problems. One complication, however, is how best to combine data from both group experimental design (GED) studies and single-case experimental design (SCED) studies. Percent of Goal Obtained (PoGO) has been developed as a metric to express the size of the effect relative to the distance to a goal, which could have broad applicability. This study sought to augment this descriptive index with estimates of standard errors, which are needed to use PoGO as an effect size metric in meta-analyses of SCED and GED studies. This study investigated the application of PoGO and standard errors to both SCED and GED studies examining a common intervention approach used with a single population. METHOD: Sixteen articles investigating explicit vocabulary instruction applied to pre-K and kindergarten students were identified. PoGO and standard errors were calculated for variations of explicit vocabulary interventions. Evaluated interventions included six studies using exclusively an SCED, nine studies using a GED, and one that used both. RESULTS: PoGO was calculated for each treatment condition when applicable (i.e., alternating treatments designs). Standard errors and confidence interval limits also were calculated. PoGO effect size values ranged from 14.4% to 93.6%. PoGO for single-case experiments was 49.2% with a standard error of 7.26, and for group experiments, it was 30.8% with a standard error of 3.71. CONCLUSION: Despite variation in the percentage of goal obtained across studies, the high degree of overlap in PoGO and standard errors between single-case and group experiments provides an indication that systematic reviews can apply this effect size metric to combine information obtained across experimental designs.

Advancing the Application and Use of Single-Case Research Designs: Reflections on Articles from the Special Issue.

This special issue of Perspective on Behavior Science is a productive contribution to current advances in the use and documentation of single-case research designs. We focus in this article on major themes emphasized by the articles in this issue and suggest directions for improving professional standards focused on the design, analysis, and dissemination of single-case research.

Multilevel Meta-Analysis of Individual Participant Data of Single-Case Experimental Designs: One-Stage versus Two-Stage Methods.

To conduct a multilevel meta-analysis of multiple single-case experimental design (SCED) studies, the individual participant data (IPD) can be analyzed in one or two stages. In the one-stage approach, a multilevel model is estimated based on the raw data. In the two-stage approach, an effect size is calculated for each participant and these effect sizes and their sampling variances are subsequently combined to estimate a meta-analytic multilevel model. The multilevel model in the two-stage approach has fewer parameters to estimate, in exchange for the reduction of information of the raw data to effect sizes. In this paper we explore how the one-stage and two-stage IPD approaches can be applied in the context of meta-analysis of single-case designs. Both approaches are compared for several single-case designs of increasing complexity. Through a simulation study we show that the two-stage approach obtains better convergence rates for more complex models, but that model estimation does not necessarily converge at a faster speed. The point estimates of the fixed effects are unbiased for both approaches across all models, as such confirming results from methodological research on IPD meta-analysis of group-comparison designs. In light of these results, we discuss the implementation of both methods in R.

Masked analysis for small-scale cluster randomized controlled trials.

Researchers conducting small-scale cluster randomized controlled trials (RCTs) during the pilot testing of an intervention often look for evidence of promise to justify an efficacy trial. We developed a method to test for intervention effects that is adaptive (i.e., responsive to data exploration), requires few assumptions, and is statistically valid (i.e., controls the type I error rate), by adapting masked visual analysis techniques to cluster RCTs. We illustrate the creation of masked graphs and their analysis using data from a pilot study in which 15 high school programs were randomly assigned to either business as usual or an intervention developed to promote psychological and academic well-being in 9th grade students in accelerated coursework. We conclude that in small-scale cluster RCTs there can be benefits of testing for effects without a priori specification of a statistical model or test statistic.

Single-case design standards: An update and proposed upgrades.

In this paper, we provide a critique focused on the What Works Clearinghouse (WWC) Standards for Single-Case Research Design (Standards 4.1). Specifically, we (a) recommend the use of visual-analysis to verify a single-case intervention study's design standards and to examine the study's operational issues, (b) identify limitations of the design-comparable effect-size measure and discuss related statistical matters, (c) review the applicability and practicality of Standards 4.1 to single-case designs (SCDs), and (d) recommend inclusion of content pertaining to diversity, equity, and inclusion in future standards. Within the historical context of the WWC Pilot Standards for Single-Case Design (1.0), we suggest that Standards 4.1 may best serve as standards for meta-analyses of SCDs but will need to make clear distinctions among the various types of SCD studies that are included in any research synthesis. In this regard, we argue for transparency in SCD studies that meet design standards and those that do not meet design standards in any meta-analysis emanating from the WWC. The intent of these recommendations is to advance the science of SCD research both in research synthesis and in promoting evidence-based practices.

How to Be RAD: Repeated Acquisition Design Features that Enhance Internal and External Validity.

The Repeated Acquisition Design (RAD) is a type of single-case research design (SCRD) that involves repeated and rapid measurement of irreversible discrete skills or behaviors through pre-and postintervention probes across different sets of stimuli. Researchers interested in the study of learning in animals and humans have used the RAD because of its sensitivity to detect immediate changes in rate or accuracy. Despite its strengths, critics of the RAD have cautioned against its use due to reasonable threats to internal validity like pretest effects, history, and maturation. Furthermore, many methodologists and researchers have neglected the RAD in their SCRD standards (e.g., What Works Clearinghouse [WWC], 2020; Horner et al., 2005). Unless given guidance to address threats to internal validity, researchers may avoid the design altogether or continue to use a weak version of the RAD. Therefore, we propose a set of 15 quality RAD indicators, comprising foundational elements that should be present in all RAD studies and additional features that enhance causal inference and external validity. We review contemporary RAD use and describe how the additional features strengthen the rigor of RAD studies. We end the article with suggested guidelines for interpreting effects and the strength of the evidence generated by RAD studies. We invite researchers to use these initial guidelines as a jumping off point for a more RAD future.

Examining the Impact of and Sensitivity of Fit Indices to Omitting Covariates Interaction Effect in Multilevel Multiple-Indicator Multiple-Cause Models.

This study examined the impact of omitting covariates interaction effect on parameter estimates in multilevel multiple-indicator multiple-cause models as well as the sensitivity of fit indices to model misspecification when the between-level, within-level, or cross-level interaction effect was left out in the models. The parameter estimates produced in the correct and the misspecified models were compared under varying conditions of cluster number, cluster size, intraclass correlation, and the magnitude of the interaction effect in the population model. Results showed that the two main effects were overestimated by approximately half of the size of the interaction effect, and the between-level factor mean was underestimated. None of comparative fit index, Tucker-Lewis index, root mean square error of approximation, and standardized root mean square residual was sensitive to the omission of the interaction effect. The sensitivity of information criteria varied depending majorly on the magnitude of the omitted interaction, as well as the location of the interaction (i.e., at the between level, within level, or cross level). Implications and recommendations based on the findings were discussed.

The impact of the good behavior game on writing quantity and quality.

Results from research indicate writing is a critical skill linked to several academic outcomes. To promote improvements in writing quantity and quality, intervention might target increasing students' academic engagement during time designated to practice writing. The purpose of this study was to implement an evidence-based classwide behavioral intervention, the Good Behavior Game (GBG), during daily writing practice time in two classrooms. Participants (n = 45) included students in a Grade 1 and Grade 2 class enrolled in an elementary school in a large suburb in the northeast U.S. Findings based on visual analysis and multilevel modeling indicate that students, on average, wrote more words (quantity) and more correct writing sequences (quality) when the GBG was played versus when it was not. Implications include the need for replication studies to extend findings and explore how school psychologists might consider the use of behavioral interventions to promote improved engagement and academic output in the classroom. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Different randomized multiple-baseline models for different situations: A practical guide for single-case intervention researchers.

Single-case researchers often implement multiple-baseline designs as their preferred methodology for intervention evaluations. Recent writings and empirical investigations have argued in favor of incorporating various forms of randomization into such designs for the purpose of elevating the intervention study's internal validity and scientific credibility. In this article, we consider a variety of randomized multiple-baseline designs and associated randomization statistical tests, along with their potential strengths and limitations. In what amounts to a practical guide, we refer school psychology researchers to these versatile randomization procedures for planning and executing their intervention studies.

A Motivational Interviewing Intervention for Adolescents in Accelerated High School Curricula: Applicability and Acceptability in a Second Sample.

The paper describes the applicability and acceptability of a selective intervention-Motivation, Assessment, and Planning (MAP)-for high school students that was developed based on the principles of motivational interviewing (MI) and tailored to the unique needs and strengths of students taking accelerated coursework, specifically Advanced Placement (AP) and International Baccalaureate (IB) classes. In addition to detailing the intervention in terms of MI spirit, processes, and relational and technical skills, we report applicability and acceptability data from a second iteration of MAP implementation in eight AP/IB programs in a Southeastern state during spring 2018. We analyzed quantitative and qualitative acceptability data from 121 high school freshmen (97 from AP and 24 from IB courses), as well as the seven MAP coaches who were trained using the Motivational Interview Training and Assessment System (Frey et al. 2017). To gain perspectives from the intended end users of the refined MAP, 12 school counselors and school psychologists who were not trained in MAP evaluated the intervention and provided qualitative and quantitative data on applicability and acceptability. All three stakeholder groups (students, coaches, and school mental health staff) rated and described the intervention as highly acceptable and appropriate for addressing the social-emotional needs of adolescents in AP/IB classes.

Bayesian Analysis for Multiple-baseline Studies Where the Variance Differs across Cases in OpenBUGS.

Objective: There is a growing interest in the potential benefits of applying Bayesian estimation for multilevel models of SCED data. Methodological studies have shown that Bayesian estimation resolves convergence issues, can be adequate for the small sample, and can improve the accuracy of the variance components. Despite the potential benefits, the lack of accessibility to software codes makes it difficult for applied researchers to implement Bayesian estimation in their studies. The purpose of this article is to illustrate a feasible way to implement Bayesian estimation using OpenBUGS software to analyze a complex SCED model where within-participants variability and autocorrelation may differ across cases. Method: By using extracted data from a published study, step-by-step guidance in analyzing the data using OpenBUGS software is provided, including (1) model specification, (2) prior distributions, (3) data entering, (4) model estimation, (5) convergence criteria, and (6) posterior inferences and interpretations. Result: Full codes for the analysis are provided.

Testing Measurement Invariance Across Unobserved Groups: The Role of Covariates in Factor Mixture Modeling.

Factor mixture modeling (FMM) has been increasingly used to investigate unobserved population heterogeneity. This study examined the issue of covariate effects with FMM in the context of measurement invariance testing. Specifically, the impact of excluding and misspecifying covariate effects on measurement invariance testing and class enumeration was investigated via Monte Carlo simulations. Data were generated based on FMM models with (1) a zero covariate effect, (2) a covariate effect on the latent class variable, and (3) covariate effects on both the latent class variable and the factor. For each population model, different analysis models that excluded or misspecified covariate effects were fitted. Results highlighted the importance of including proper covariates in measurement invariance testing and evidenced the utility of a model comparison approach in searching for the correct specification of covariate effects and the level of measurement invariance. This approach was demonstrated using an empirical data set. Implications for methodological and applied research are discussed.

Correction to: How to Be RAD: Repeated Acquisition Design Features that Enhance Internal and External Validity.

[This corrects the article DOI: 10.1007/s40614-021-00301-2.].

Effects of Embedding Decontextualized Language Through Book-Sharing Delivered by Mothers and Fathers in Coparenting Environments.

Purpose This experiment investigated the effects of a book-sharing intervention implemented in coparenting homes on the conversations of preschoolers with their parents. Method A multiple baseline design across behaviors was used to evaluate the effects of embedding decontextualized language utterances during book-sharing delivered by four families. A visual analysis, a two-level mixed-effects model, and a social validity evaluation were used to examine the varying effects of the program on mothers and fathers' storybook conversations. Results Embedding decontextualized language prompts in books not only increased parental decontextualized language utterances, but most parents were able to maintain use of strategies without prompts in the books. The intervention effects were consistently higher for parents than for their children. Social validity results demonstrated parental satisfaction with program delivery and content. Conclusion This study adds to the limited literature on father-child and mother-child decontextualized conversations during book-sharing and illustrates a feasible and adaptable way of promoting language experiences in the home that yield engaging decontextualized conversations in meaningful book-reading contexts. Supplemental Material https://doi.org/10.23641/asha.13210799.

Assessing consistency of effects when applying multilevel models to single-case data.

In the context of single-case experimental designs, replication is crucial. On the one hand, the replication of the basic effect within a study is necessary for demonstrating experimental control. On the other hand, replication across studies is required for establishing the generality of the intervention effect. Moreover, the "replicability crisis" presents a more general context further emphasizing the need for assessing consistency in replications. In the current text, we focus on replication of effects within a study, and we specifically discuss the consistency of effects. Our proposal for assessing the consistency of effects refers to one of the promising data analytical techniques, multilevel models, also known as hierarchical linear models or mixed effects models. One option is to check, for each case in a multiple-baseline design, whether the confidence interval for the individual treatment effect excludes zero. This is relevant for assessing whether the effect is replicated as being non-null. However, we consider that it is more relevant and informative to assess, for each case, whether the confidence interval for the random effects includes zero (i.e., whether the fixed effect estimate is a plausible value for each individual effect). This is relevant for assessing whether the effect is consistent in size, with the additional requirement that the fixed effect itself is different from zero. The proposal for assessing consistency is illustrated with real data and is implemented in free user-friendly software.

Multilevel meta-analysis of multiple regression coefficients from single-case experimental studies.

The focus of the current study is on handling the dependence among multiple regression coefficients representing the treatment effects when meta-analyzing data from single-case experimental studies. We compare the results when applying three different multilevel meta-analytic models (i.e., a univariate multilevel model avoiding the dependence, a multivariate multilevel model ignoring covariance at higher levels, and a multivariate multilevel model modeling the existing covariance) to deal with the dependent effect sizes. The results indicate better estimates of the overall treatment effects and variance components when a multivariate multilevel model is applied, independent of modeling or ignoring the existing covariance. These findings confirm the robustness of multilevel modeling to misspecifying the existing covariance at the case and study level in terms of estimating the overall treatment effects and variance components. The results also show that the overall treatment effect estimates are unbiased regardless of the underlying model, but the between-case and between-study variance components are biased in certain conditions. In addition, the between-study variance estimates are particularly biased when the number of studies is smaller than 40 (i.e., 10 or 20) and the true value of the between-case variance is relatively large (i.e., 8). The observed bias is larger for the between-case variance estimates compared to the between-study variance estimates when the true between-case variance is relatively small (i.e., 0.5).

Classwide Extensions of Vocabulary Intervention Improve Learning of Academic Vocabulary by Preschoolers.

Purpose Many preschoolers, especially those from low-income households, would benefit from instruction to enrich their vocabulary and language repertoires. Yet, explicit instruction of vocabulary and language skills generally occurs infrequently in early childhood education settings. This study investigated the additive effects of teacher-led, classwide review strategies to a previously studied small-group intervention on children's learning of academic vocabulary. Method Participants included 23 children with limited oral language skills at risk for reading difficulties enrolled in single-case experimental designs. Effects of the classroom strategies alone also were examined in 10 children with above-average language abilities from 2 classrooms. Results Visual analyses of the adapted alternating treatments designs showed consistent learning improvements when vocabulary instruction was extended into the classroom for 12 children, ceiling effects were evident for 3 participants regardless of condition, and inconsistent or minimal effects were demonstrated by 8 participants. Multilevel modeling used to evaluate the effects statistically revealed strong treatment effects. In addition, the 10 children with above-average language showed impressive learning of vocabulary words from books subject to teacher review strategies in comparison to words from books to which they were not exposed. Teachers varied in the extent to which they implemented review strategies in their classrooms. Nevertheless, their responses to social validity assessments were positive, supporting the feasibility of this intervention. Conclusions The addition of classwide review and practice opportunities is an effective means of enhancing the effects of an easy-to-implement small-group intervention that teaches challenging vocabulary words within prerecorded stories. This approach holds promise as a way to shrink the pervasive word gap that typically exists when children in high-poverty communities enter school.