Reproducibility in Small-N Treatment Research: A Tutorial Using Examples From Aphasiology.

PURPOSE: Small-N studies are the dominant study design supporting evidence-based interventions in communication science and disorders, including treatments for aphasia and related disorders. However, there is little guidance for conducting reproducible analyses or selecting appropriate effect sizes in small-N studies, which has implications for scientific review, rigor, and replication. This tutorial aims to (a) demonstrate how to conduct reproducible analyses using effect sizes common to research in aphasia and related disorders and (b) provide a conceptual discussion to improve the reader's understanding of these effect sizes. METHOD: We provide a tutorial on reproducible analyses of small-N designs in the statistical programming language R using published data from Wambaugh et al. (2017). In addition, we discuss the strengths, weaknesses, reporting requirements, and impact of experimental design decisions on effect sizes common to this body of research. RESULTS: Reproducible code demonstrates implementation and comparison of within-case standardized mean difference, proportion of maximal gain, tau-U, and frequentist and Bayesian mixed-effects models. Data, code, and an interactive web application are available as a resource for researchers, clinicians, and students. CONCLUSIONS: Pursuing reproducible research is key to promoting transparency in small-N treatment research. Researchers and clinicians must understand the properties of common effect size measures to make informed decisions in order to select ideal effect size measures and act as informed consumers of small-N studies. Together, a commitment to reproducibility and a keen understanding of effect sizes can improve the scientific rigor and synthesis of the evidence supporting clinical services in aphasiology and in communication sciences and disorders more broadly. Supplemental Material and Open Science Form: https://doi.org/10.23641/asha.21699476.

Representation in Aphasia Research: An Examination of U.S. Treatment Studies Published Between 2009 and 2019.

PURPOSE: The external validity of aphasia treatment research relies on diverse and representative participants. The purposes of this study were (a) to examine whether reporting of patient-reported age, sex, and race/ethnicity has improved since Ellis (2009) and (b) to evaluate whether these demographic variables were consistent with population-level estimates of stroke survivor demographics in the United States. METHOD: A scoping review examined U.S.-based aphasia treatment studies published between 2009 and 2019 and characterized the percentage of studies reporting age, sex, and race/ethnicity. Summary statistics for these variables were calculated and compared statistically with a population-based study of stroke survivors. RESULTS: It was found out that 97.1% of studies reported age, 93.5% reported sex, and 28.1% reported race and/or ethnicity. Within reporting studies, participant mean age was 58.04 years, 61.6% of participants were men, and 38.4% were women; 86.5% of participants were White, 11.0% were Black, 2.0% were Hispanic/Latino, and 0.5% fell in other racial categories. All three variables were statistically different from the study of Kissela et al. (2012). DISCUSSION: Despite being highlighted as an issue by Ellis (2009), less than 30% of recent aphasia treatment studies reported race or ethnicity, and participants do not appear to be demographically representative compared with estimates of stroke survivors living in the United States. These issues may negatively impact the ecological validity of aphasia treatment research. Aphasia researchers should more consistently report participant race and ethnicity and follow current guidelines for increasing the demographic representation of women and minorities.

Practice-Related Predictors of Semantic Feature Verification Treatment for Aphasia.

PURPOSE: Specifying the active ingredients in aphasia interventions can inform treatment theory and improve clinical implementation. This secondary analysis examined three practice-related predictors of treatment response in semantic feature verification (SFV) treatment. We hypothesized that (a) successful feature verification practice would be associated with naming outcomes if SFV operates similarly to standard feature generation semantic feature analysis and (b) successful retrieval practice would be associated with naming outcomes for treated, but not semantically related, untreated words if SFV operates via a retrieval practice-oriented lexical activation mechanism. METHOD: Item-level data from nine participants with poststroke aphasia who received SFV treatment reported in the work of Evans, Cavanaugh, Quique, et al. (2021) were analyzed using Bayesian generalized linear mixed-effects models. Models evaluated whether performance on three treatment components (facilitated retrieval, feature verification, and effortful retrieval) moderated treatment response for treated and semantically related, untreated words. RESULTS: There was no evidence for or against a relationship between successful feature verification practice and treatment response. In contrast, there was a robust relationship between the two retrieval practice components and treatment response for treated words only. DISCUSSION: Findings were consistent with the second hypothesis: Retrieval practice, but not feature verification practice, appears to be a practice-related predictor of treatment response in SFV. However, treatment components are likely interdependent, and feature verification may still be an active ingredient in SFV. Further research is needed to evaluate the causal role of treatment components on treatment outcomes in aphasia.

Is There a Research-Practice Dosage Gap in Aphasia Rehabilitation?

Purpose Aphasia intervention research aims to improve communication and quality of life outcomes for people with aphasia. However, few studies have evaluated the translation and implementation of evidence-based aphasia interventions to clinical practice. Treatment dosage may be difficult to translate to clinical settings, and a mismatch between dosage in research and clinical practice threatens to attenuate intervention effectiveness. The purpose of this study is to quantify a potential research-practice dosage gap in outpatient aphasia rehabilitation. Method This study utilized a two-part approach. First, we estimated clinical treatment dosage in an episode of care (i.e., treatment provided from outpatient assessment to discharge) via utilization in a regional provider in the United States. Second, we undertook a scoping review of aphasia interventions published from 2009 to 2019 to estimate the typical dosage used in the current aphasia literature. Results Outpatient clinical episodes of care included a median of 10 treatment sessions and a mean of 14.8 sessions (interquartile range: 5-20 sessions). Sessions occurred 1-2 times a week over 4-14 weeks. The median total hours of treatment was 7.5 hr (interquartile range: 3.75-15 hr). In contrast, published interventions administered a greater treatment dosage, consisting of a median of 20 hr of treatment (interquartile range: 12-30 hr) over the course of 15 sessions (interquartile range: 10-24 sessions) approximately 3 times per week. Conclusions Results demonstrate a meaningful research-practice dosage gap, particularly in total treatment hours and weekly treatment intensity. This gap highlights the potential for attenuation of effectiveness from research to outpatient settings. Future translational research should consider clinical dosage constraints and take steps to facilitate intervention implementation, particularly with regard to dosage. Conversely, health care advocacy and continued development of alternative delivery methods are necessary for the successful implementation of treatments with dosage that is incompatible with current clinical contexts. Pragmatic, implementation-focused trials are recommended to evaluate and optimize treatment effectiveness in outpatient clinical settings. Supplemental Material https://doi.org/10.23641/asha.15161568.

Playing With BEARS: Balancing Effort, Accuracy, and Response Speed in a Semantic Feature Verification Anomia Treatment Game.

Purpose The purpose of this study was to develop and pilot a novel treatment framework called BEARS (Balancing Effort, Accuracy, and Response Speed). People with aphasia (PWA) have been shown to maladaptively balance speed and accuracy during language tasks. BEARS is designed to train PWA to balance speed-accuracy trade-offs and improve system calibration (i.e., to adaptively match system use with its current capability), which was hypothesized to improve treatment outcomes by maximizing retrieval practice and minimizing error learning. In this study, BEARS was applied in the context of a semantically oriented anomia treatment based on semantic feature verification (SFV). Method Nine PWA received 25 hr of treatment in a multiple-baseline single-case series design. BEARS + SFV combined computer-based SFV with clinician-provided BEARS metacognitive training. Naming probe accuracy, efficiency, and proportion of "pass" responses on inaccurate trials were analyzed using Bayesian generalized linear mixed-effects models. Generalization to discourse and correlations between practice efficiency and treatment outcomes were also assessed. Results Participants improved on naming probe accuracy and efficiency of treated and untreated items, although untreated item gains could not be distinguished from the effects of repeated exposure. There were no improvements on discourse performance, but participants demonstrated improved system calibration based on their performance on inaccurate treatment trials, with an increasing proportion of "pass" responses compared to paraphasia or timeout nonresponses. In addition, levels of practice efficiency during treatment were positively correlated with treatment outcomes, suggesting that improved practice efficiency promoted greater treatment generalization and improved naming efficiency. Conclusions BEARS is a promising, theoretically motivated treatment framework for addressing the interplay between effort, accuracy, and processing speed in aphasia. This study establishes the feasibility of BEARS + SFV and provides preliminary evidence for its efficacy. This study highlights the importance of considering processing efficiency in anomia treatment, in addition to performance accuracy. Supplemental Material https://doi.org/10.23641/asha.14935812.

Effects of Semantic Feature Type, Diversity, and Quantity on Semantic Feature Analysis Treatment Outcomes in Aphasia.

Purpose Semantic feature analysis (SFA) is a naming treatment found to improve naming performance for both treated and semantically related untreated words in aphasia. A crucial treatment component is the requirement that patients generate semantic features of treated items. This article examined the role feature generation plays in treatment response to SFA in several ways: It attempted to replicate preliminary findings from Gravier et al. (2018), which found feature generation predicted treatment-related gains for both trained and untrained words. It examined whether feature diversity or the number of features generated in specific categories differentially affected SFA treatment outcomes. Method SFA was administered to 44 participants with chronic aphasia daily for 4 weeks. Treatment was administered to multiple lists sequentially in a multiple-baseline design. Participant-generated features were captured during treatment and coded in terms of feature category, total average number of features generated per trial, and total number of unique features generated per item. Item-level naming accuracy was analyzed using logistic mixed-effects regression models. Results Producing more participant-generated features was found to improve treatment response for trained but not untrained items in SFA, in contrast to Gravier et al. (2018). There was no effect of participant-generated feature diversity or any differential effect of feature category on SFA treatment outcomes. Conclusions Patient-generated features remain a key predictor of direct training effects and overall treatment response in SFA. Aphasia severity was also a significant predictor of treatment outcomes. Future work should focus on identifying potential nonresponders to therapy and explore treatment modifications to improve treatment outcomes for these individuals. Supplemental Material https://doi.org/10.23641/asha.12462596.

How Much Time Do People With Aphasia Need to Respond During Picture Naming? Estimating Optimal Response Time Cutoffs Using a Multinomial Ex-Gaussian Approach.

Purpose Aphasia is a language disorder caused by acquired brain injury, which generally involves difficulty naming objects. Naming ability is assessed by measuring picture naming, and models of naming performance have mostly focused on accuracy and excluded valuable response time (RT) information. Previous approaches have therefore ignored the issue of processing efficiency, defined here in terms of optimal RT cutoff, that is, the shortest deadline at which individual people with aphasia produce their best possible naming accuracy performance. The goals of this study were therefore to (a) develop a novel model of aphasia picture naming that could accurately account for RT distributions across response types; (b) use this model to estimate the optimal RT cutoff for individual people with aphasia; and (c) explore the relationships between optimal RT cutoff, accuracy, naming ability, and aphasia severity. Method A total of 4,021 naming trials across 10 people with aphasia were scored for accuracy and RT onset. Data were fit using a novel ex-Gaussian multinomial RT model, which was then used to characterize individual optimal RT cutoffs. Results Overall, the model fitted the empirical data well and provided reliable individual estimates of optimal RT cutoff in picture naming. Optimal cutoffs ranged between approximately 5 and 10 s, which has important implications for assessment and treatment. There was no direct relationship between aphasia severity, naming RT, and optimal RT cutoff. Conclusion The multinomial ex-Gaussian modeling approach appears to be a promising and straightforward way to estimate optimal RT cutoffs in picture naming in aphasia. Limitations and future directions are discussed.

Speed-Accuracy Trade-Offs and Adaptation Deficits in Aphasia: Finding the "Sweet Spot" Between Overly Cautious and Incautious Responding.

Purpose After stroke, how well do people with aphasia (PWA) adapt to the altered functioning of their language system? When completing a language-dependent task, how well do PWA balance speed and accuracy when the goal is to respond both as quickly and accurately as possible? The current work investigates adaptation theory ( Kolk & Heeschen, 1990 ) in the context of speed-accuracy trade-offs in a lexical decision task. PWA were predicted to set less beneficial speed-accuracy trade-offs than matched controls, and at least some PWA were predicted to present with adaptation deficits, with impaired accuracy or response times attributable to speed-accuracy trade-offs. Method The study used the diffusion model ( Ratcliff, 1978 ), a computational model of response time for simple 2-choice tasks. Parameters of the model can be used to distinguish basic processing efficiency from the overall level of caution in setting response thresholds and were used here to characterize speed-accuracy trade-offs in 20 PWA and matched controls during a lexical decision task. Results Models showed that PWA and matched control groups did not differ overall in how they set response thresholds for speed-accuracy trade-offs. However, case series analyses showed that 40% of the PWA group displayed the predicted adaptation deficits, with impaired accuracy or response time performance directly attributable to overly cautious or overly incautious response thresholds. Conclusions Maladaptive speed-accuracy trade-offs appear to be present in some PWA during lexical decision, leading to adaptation deficits in performance. These adaptation deficits are potentially treatable, and clinical implications and next steps for translational research are discussed.

Acquisition and Generalization Responses in Aphasia Naming Treatment: A Meta-Analysis of Semantic Feature Analysis Outcomes.

Purpose This meta-analysis synthesizes results from published studies that used semantic feature analysis (SFA) treatment to improve naming for people with aphasia. It examines how both person- and treatment-related variables affected the likelihood of correct naming responses in individual probe sessions for both acquisition (treated) and generalization (untreated) stimuli. Method The meta-analysis compiled data from 12 studies analyzing a total of 35 participants with aphasia. It used mixed-effects models as a novel statistical tool to examine the effects of 2 sets of variables on naming performance: treatment-related variables, including treatment phase (baseline vs. treatment), dosage (number of treatment sessions), and stimulus type (treated vs. untreated, semantically related vs. unrelated items), and person-specific variables, including degree of language impairment and demographic variables (age, time poststroke). Results Results of the meta-analysis revealed that SFA intervention promoted increased naming accuracy during naming probes when comparing baseline and treatment phases. In addition, increased dosages of SFA were associated with increased naming accuracy, and treatment-related gains were larger for acquisition (treated) than generalization (untreated) stimuli, likewise for related versus unrelated generalization stimuli. Furthermore, a subset of person-specific variables was predictive of SFA-related gains: Language impairment variables were related to treatment-related changes in naming performance, but demographic variables were not. Conclusion These results provide group-level evidence for the efficacy of SFA as well as preliminary estimates of how much naming performance benefit is engendered by varying dosages of SFA. The results also provide promising and previously unobserved evidence of potential person-level predictors of SFA treatment response.

What Matters in Semantic Feature Analysis: Practice-Related Predictors of Treatment Response in Aphasia.

Purpose: This study investigated the predictive value of practice-related variables-number of treatment trials delivered, total treatment time, average number of trials per hour, and average number of participant-generated features per trial-in response to semantic feature analysis (SFA) treatment. Method: SFA was administered to 17 participants with chronic aphasia daily for 4 weeks. Individualized treatment and semantically related probe lists were generated from items that participants were unable to name consistently during baseline testing. Treatment was administered to each list sequentially in a multiple-baseline design. Naming accuracy for treated and untreated items was obtained at study entry, exit, and 1-month follow-up. Results: Item-level naming accuracy was analyzed using logistic mixed-effect regression models. The average number of features generated per trial positively predicted naming accuracy for both treated and untreated items, at exit and follow-up. In contrast, total treatment time and average trials per hour did not significantly predict treatment response. The predictive effect of number of treatment trials on naming accuracy trended toward significance at exit, although this relationship held for treated items only. Conclusions: These results suggest that the number of patient-generated features may be more strongly associated with SFA-related naming outcomes, particularly generalization and maintenance, than other practice-related variables. Supplemental Materials: https://doi.org/10.23641/asha.5734113.