Evaluating the performance of existing and novel equivalence tests for fit indices in structural equation modelling.

It has been suggested that equivalence testing (otherwise known as negligible effect testing) should be used to evaluate model fit within structural equation modelling (SEM). In this study, we propose novel variations of equivalence tests based on the popular root mean squared error of approximation and comparative fit index fit indices. Using Monte Carlo simulations, we compare the performance of these novel tests to other existing equivalence testing-based fit indices in SEM, as well as to other methods commonly used to evaluate model fit. Results indicate that equivalence tests in SEM have good Type I error control and display considerable power for detecting well-fitting models in medium to large sample sizes. At small sample sizes, relative to traditional fit indices, equivalence tests limit the chance of supporting a poorly fitting model. We also present an illustrative example to demonstrate how equivalence tests may be incorporated in model fit reporting. Equivalence tests in SEM also have unique interpretational advantages compared to other methods of model fit evaluation. We recommend that equivalence tests be utilized in conjunction with descriptive fit indices to provide more evidence when evaluating model fit.

Denouncing the use of field-specific effect size distributions to inform magnitude.

An effect size (ES) provides valuable information regarding the magnitude of effects, with the interpretation of magnitude being the most important. Interpreting ES magnitude requires combining information from the numerical ES value and the context of the research. However, many researchers adopt popular benchmarks such as those proposed by Cohen. More recently, researchers have proposed interpreting ES magnitude relative to the distribution of observed ESs in a specific field, creating unique benchmarks for declaring effects small, medium or large. However, there is no valid rationale whatsoever for this approach. This study was carried out in two parts: (1) We identified articles that proposed the use of field-specific ES distributions to interpret magnitude (primary articles); and (2) We identified articles that cited the primary articles and classified them by year and publication type. The first type consisted of methodological papers. The second type included articles that interpreted ES magnitude using the approach proposed in the primary articles. There has been a steady increase in the number of methodological and substantial articles discussing or adopting the approach of interpreting ES magnitude by considering the distribution of observed ES in that field, even though the approach is devoid of a theoretical framework. It is hoped that this research will restrict the practice of interpreting ES magnitude relative to the distribution of ES values in a field and instead encourage researchers to interpret such by considering the specific context of the study.

The role of cognitive control and emotion regulation in predicting mental health problems in children with neurodevelopmental disorders.

BACKGROUND: Many youth with neurodevelopmental disorders (NDDs) experience mental health problems such as anxiety, depression or anger, and these are often associated with impairments of cognition and emotion regulation. The mechanisms that may be linking cognitive difficulties, emotion regulation and mental health are not known. AIMS: The current study examined whether adaptive and maladaptive (dysregulated) emotion regulation mediated the link between different cognitive control processes (working memory, inhibition and shifting) and internalizing/externalizing symptoms in children with NDDs. METHODS: Participants included 48 children (8-13 years of age) with one or more diagnoses of autism, attention deficit hyperactivity disorder, cerebral palsy and learning disability, who were enrolled in a larger study of cognitive behaviour therapy targeting emotion regulation. Multiple mediation analyses were implemented using the PROCESS macro. The mediation effects of adaptive and maladaptive emotion regulation were examined on the relationships between (1) working memory and internalizing/externalizing symptoms, (2) inhibition and internalizing/externalizing symptoms and (3) shifting and internalizing/externalizing symptoms. All data were collected prior to intervention, at baseline. RESULTS: Shifting, inhibitory control and working memory predicted increased emotion dysregulation, which functioned as a full mediator to both internalizing and externalizing problems in children with NDDs. CONCLUSIONS: In the presence of emotionally triggering situations, children with greater cognitive challenges experience greater maladaptive emotion regulation, which results in both internalizing and externalizing problems. For youth with NDDs, therapeutic plans that include strengthening of working memory, inhibition and shifting abilities in addition to emotion regulation skills training may be helpful in alleviating externalizing and internalizing behaviour.

The role of alexithymia in trauma therapy outcomes: Examining improvements in PTSD, dissociation, and interpersonal problems.

OBJECTIVE: Alexithymia is a personality trait that reflects deficits in the cognitive processing and regulation of emotions (Taylor & Bagby, 2013). It has been closely linked to childhood trauma and reported by individuals presenting with other trauma-related conditions, such as posttraumatic stress disorder (PTSD), dissociation, and interpersonal problems (Powers, Etkin, Gyurak, Bradley, & Jovanovic, 2015). Addressing the emotional deficits associated with alexithymia is fundamental to resolving issues of childhood trauma and, therefore, is at the core of many trauma therapy models (e.g., Cloitre, Koenen, Cohen, & Han, 2002). The current study aims to build upon this foundation by examining the role of alexithymia in the improvements of trauma-specific difficulties prior to and following trauma therapy among treatment-seeking women with histories of childhood abuse. METHOD: Data were collected from 167 participants attending Women Recovering from Abuse Program (WRAP), an 8-week, Stage I, day treatment program using primarily group therapy for women with histories of severe childhood trauma. Participants' level of alexithymia, PTSD, and dissociative symptoms, and interpersonal difficulties were assessed at three time points. RESULTS: Significant positive relationships were found between improvements in alexithymia and improvements on all trauma-specific outcomes over the course of treatment (e.g., baseline to posttreatment) and between distinct stages of WRAP. CONCLUSIONS: These findings underscore the role of alexithymia in trauma therapy, and the need to properly attend to the deficits and issues related to alexithymia at initial stages of therapy with survivors of childhood abuse in order to facilitate improvements in trauma-specific symptoms. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Evaluating Equivalence Testing Methods for Measurement Invariance.

Measurement Invariance (MI) is often concluded from a nonsignificant chi-square difference test. Researchers have also proposed using change in goodness-of-fit indices ([Formula: see text]GOFs) instead. Both of these commonly used methods for testing MI have important limitations. To combat these issues, To combat these issues, it was proposed using an equivalence test (EQ) to replace the chi-square difference test commonly used to test MI. Due to concerns with the EQ's power, and adjusted version (EQ-A) was created, but provides little evaluation of either procedure. The current study evaluated the Type I error and power of both the EQ and EQ-A, and compared their performance to that of the traditional chi-square difference test and [Formula: see text]GOFs. The EQ was the only procedure that maintained empirical error rates below the nominal alpha level. Results also highlight that the EQ requires larger sample sizes than traditional difference-based approaches or using equivalence bounds based on larger than conventional RMSEA values (e.g., > .05) to ensure adequate power rates. We do not recommend the proposed adjustment (EQ-A) over the EQ.

Exploring perceptions of meaningfulness in visual representations of bivariate relationships.

Researchers often need to consider the practical significance of a relationship. For example, interpreting the magnitude of an effect size or establishing bounds in equivalence testing requires knowledge of the meaningfulness of a relationship. However, there has been little research exploring the degree of relationship among variables (e.g., correlation, mean difference) necessary for an association to be interpreted as meaningful or practically significant. In this study, we presented statistically trained and untrained participants with a collection of figures that displayed varying degrees of mean difference between groups or correlations among variables and participants indicated whether or not each relationship was meaningful. The results suggest that statistically trained and untrained participants differ in their qualification of a meaningful relationship, and that there is significant variability in how large a relationship must be before it is labeled meaningful. The results also shed some light on what degree of relationship is considered meaningful by individuals in a context-free setting.

ANOVA and the variance homogeneity assumption: Exploring a better gatekeeper.

Valid use of the traditional independent samples ANOVA procedure requires that the population variances are equal. Previous research has investigated whether variance homogeneity tests, such as Levene's test, are satisfactory as gatekeepers for identifying when to use or not to use the ANOVA procedure. This research focuses on a novel homogeneity of variance test that incorporates an equivalence testing approach. Instead of testing the null hypothesis that the variances are equal against an alternative hypothesis that the variances are not equal, the equivalence-based test evaluates the null hypothesis that the difference in the variances falls outside or on the border of a predetermined interval against an alternative hypothesis that the difference in the variances falls within the predetermined interval. Thus, with the equivalence-based procedure, the alternative hypothesis is aligned with the research hypothesis (variance equality). A simulation study demonstrated that the equivalence-based test of population variance homogeneity is a better gatekeeper for the ANOVA than traditional homogeneity of variance tests.

Testing for negligible interaction: A coherent and robust approach.

Researchers often want to demonstrate a lack of interaction between two categorical predictors on an outcome. To justify a lack of interaction, researchers typically accept the null hypothesis of no interaction from a conventional analysis of variance (ANOVA). This method is inappropriate as failure to reject the null hypothesis does not provide statistical evidence to support a lack of interaction. This study proposes a bootstrap-based intersection-union test for negligible interaction that provides coherent decisions between the omnibus test and post hoc interaction contrast tests and is robust to violations of the normality and variance homogeneity assumptions. Further, a multiple comparison strategy for testing interaction contrasts following a non-significant omnibus test is proposed. Our simulation study compared the Type I error control, omnibus power and per-contrast power of the proposed approach to the non-centrality-based negligible interaction test of Cheng and Shao (2007, Statistica Sinica, 17, 1441). For 2 × 2 designs, the empirical Type I error rates of the Cheng and Shao test were very close to the nominal α level when the normality and variance homogeneity assumptions were satisfied; however, only our proposed bootstrapping approach was satisfactory under non-normality and/or variance heterogeneity. In general a × b designs, although the omnibus Cheng and Shao test, as expected, is the most powerful, it is not robust to assumption violation and results in incoherent omnibus and interaction contrast decisions that are not possible with the intersection-union approach.

The mental health status of ethnocultural minorities in Ontario and their mental health care.

BACKGROUND: Mental disorders are a leading cause of disability and early mortality. The objective of this study was to describe and compare psychosocial indicators and mental health service use among ethnoculturally-diverse Ontarians. METHODS: This is a cross-sectional analysis of the Ontario Health Study pilot investigation. Residents were mailed an invitation to one of 3 assessment centres (urban, rural and northern sites) from March 2009 to July 2010. Participants had an interview with a nurse and completed a questionnaire on a touchscreen kiosk. The questionnaire included sociodemographic items, and scales assessing symptoms of depressive symptoms (CES-D) and anxiety (GAD-7), social support (Lubben Social Network Scale), stressful life events, and mental health service use. RESULTS: Eight thousand two hundred thirty-five residents participated, among whom 6652 (82.4 %) self-reported their ethnocultural background as White, 225 (2.8 %) as South Asian, 222 (2.8 %) East Asian, 214 (2.7 %) Southeast Asian, 197 (2.4 %) Black, and 28 (0.3 %) as Aboriginal. Based on their sociodemographic characteristics, participants from these ethnocultural minority groups were matched to White participants. Black participants reported significantly greater stressful life events than White participants (p = .04), particularly death (p < .05), divorce (p = .002) and financial difficulties (p < .001). East Asian participants reported significantly less social support than their White counterparts (p < .001), and this was not confounded by measurement variance. Mental health service use was significantly lower in all ethnocultural minorities except Aboriginals, when compared to White participants (p = .001). CONCLUSIONS: There is a high burden of psychosocial distress in several preponderant ethnocultural minorities in Ontario; many of whom are not accessing available mental health services.

Increasing Literacy in Quantitative Methods: The Key to the Future of Canadian Psychology.

Quantitative methods (QM) dominate empirical research in psychology. Unfortunately most researchers in psychology receive inadequate training in QM. This creates a challenge for researchers who require advanced statistical methods to appropriately analyze their data. Many of the recent concerns about research quality, replicability, and reporting practices are directly tied to the problematic use of QM. As such, improving quantitative literacy in psychology is an important step towards eliminating these concerns. The current paper will include two main sections that discuss quantitative challenges and opportunities. The first section discusses training and resources for students and presents descriptive results on the number of quantitative courses required and available to graduate students in Canadian psychology departments. In the second section, we discuss ways of improving quantitative literacy for faculty, researchers, and clinicians. This includes a strong focus on the importance of collaboration. The paper concludes with practical recommendations for improving quantitative skills and literacy for students and researchers in Canada.

The relationships between therapeutic alliance and internalizing and externalizing symptoms in Trauma-Focused Cognitive Behavioral Therapy.

Therapeutic alliance has been considered an important factor in child psychotherapy and is consistently associated with positive outcomes. Nevertheless, research on alliance in the context of child trauma therapy is very scarce. This study examined the relationships between child therapeutic alliance and psychopathology in an empirically supported child trauma therapy model designed to address issues related to trauma with children and their caregivers. Specifically, we examined the extent to which the child's psychopathology would predict the establishment of a positive alliance early in treatment, as well as the association between alliance and outcome. Participants were 95 children between the ages of 7 and 12 and their caregivers, who went through a community-based Trauma-Focused Cognitive Behavioral Therapy program in Canada. Caregivers filled out the CBCL prior to assessment and following treatment. Children and therapists completed an alliance measure (TASC) at three time points throughout treatment. Symptomatology and child gender emerged as important factors predicting alliance at the beginning of treatment. Girls and internalizing children developed stronger alliances early in treatment. In addition, a strong early alliance emerged as a significant predictor of improvement in internalizing symptoms at the end of treatment. Our findings indicate that symptomatology and gender influence the development of a strong alliance in trauma therapy. We suggest that clinicians should adjust therapeutic style to better engage boys and highly externalizing children in the early stages of therapy.

Equivalence tests for comparing correlation and regression coefficients.

Equivalence tests are an alternative to traditional difference-based tests for demonstrating a lack of association between two variables. While there are several recent studies investigating equivalence tests for comparing means, little research has been conducted on equivalence methods for evaluating the equivalence or similarity of two correlation coefficients or two regression coefficients. The current project proposes novel tests for evaluating the equivalence of two regression or correlation coefficients derived from the two one-sided tests (TOST) method (Schuirmann, 1987, J. Pharmacokinet. Biopharm, 15, 657) and an equivalence test by Anderson and Hauck (1983, Stat. Commun., 12, 2663). A simulation study was used to evaluate the performance of these tests and compare them with the common, yet inappropriate, method of assessing equivalence using non-rejection of the null hypothesis in difference-based tests. Results demonstrate that equivalence tests have more accurate probabilities of declaring equivalence than difference-based tests. However, equivalence tests require large sample sizes to ensure adequate power. We recommend the Anderson-Hauck equivalence test over the TOST method for comparing correlation or regression coefficients.

Assessing clinical significance using robust normative comparisons.

OBJECTIVE: Clinical significance determines whether an intervention makes a real difference in the everyday life of a client. One of the most recommended approaches for conducting group-level analyses of clinical significance is to evaluate whether the treated clinical group is equivalent to a normal comparison group (normative comparisons). The purpose of this study was to demonstrate the analytical and practical power of assessing clinical significance using normative comparisons that are robust to violations of normality and homogeneity of variance assumptions. METHOD: Six datasets were gleaned from published intervention studies for depression. RESULTS: We found that normative comparisons using a robust Schuirmann-Yuen test determined equivalency for 11% fewer clinical samples compared to original normative comparisons that use a Schuirmann test of equivalence. CONCLUSIONS: We recommend that researchers conducting normative comparisons utilize the Schuirmann-Yuen procedure as it provides the most reliable method available for determining if a treated clinical group is equivalent to a normative comparison group.

Evaluating clinical significance: incorporating robust statistics with normative comparison tests.

The purpose of this study was to evaluate a modified test of equivalence for conducting normative comparisons when distribution shapes are non-normal and variances are unequal. A Monte Carlo study was used to compare the empirical Type I error rates and power of the proposed Schuirmann-Yuen test of equivalence, which utilizes trimmed means, with that of the previously recommended Schuirmann and Schuirmann-Welch tests of equivalence when the assumptions of normality and variance homogeneity are satisfied, as well as when they are not satisfied. The empirical Type I error rates of the Schuirmann-Yuen were much closer to the nominal α level than those of the Schuirmann or Schuirmann-Welch tests, and the power of the Schuirmann-Yuen was substantially greater than that of the Schuirmann or Schuirmann-Welch tests when distributions were skewed or outliers were present. The Schuirmann-Yuen test is recommended for assessing clinical significance with normative comparisons.

Effect of non-normality on test statistics for one-way independent groups designs.

The data obtained from one-way independent groups designs is typically non-normal in form and rarely equally variable across treatment populations (i.e., population variances are heterogeneous). Consequently, the classical test statistic that is used to assess statistical significance (i.e., the analysis of variance F test) typically provides invalid results (e.g., too many Type I errors, reduced power). For this reason, there has been considerable interest in finding a test statistic that is appropriate under conditions of non-normality and variance heterogeneity. Previously recommended procedures for analysing such data include the James test, the Welch test applied either to the usual least squares estimators of central tendency and variability, or the Welch test with robust estimators (i.e., trimmed means and Winsorized variances). A new statistic proposed by Krishnamoorthy, Lu, and Mathew, intended to deal with heterogeneous variances, though not non-normality, uses a parametric bootstrap procedure. In their investigation of the parametric bootstrap test, the authors examined its operating characteristics under limited conditions and did not compare it to the Welch test based on robust estimators. Thus, we investigated how the parametric bootstrap procedure and a modified parametric bootstrap procedure based on trimmed means perform relative to previously recommended procedures when data are non-normal and heterogeneous. The results indicated that the tests based on trimmed means offer the best Type I error control and power when variances are unequal and at least some of the distribution shapes are non-normal.

Equivalence-based measures of clinical significance: assessing treatments for depression.

Treatment efficacy is largely determined by statistical significance testing, and clinical significance testing is often used to quantify or qualify the efficacy of a treatment at the individual or group level. This study applies the equivalence-based clinical significance model proposed by Kendall, Marrs-Garcia, Nath, and Sheldrick (1999) and a revised model proposed by Cribbie and Arpin-Cribbie (2009) to the assessment of treatments for depression. Using several studies that investigated treatments for depression, the authors tested whether the posttreatment means were equivalent to those for a similar normal comparison group. All of the studies had significant improvement from pretest to posttest, although for many of the studies the treated group was not equivalent to a normal comparison group at posttest. Further, there are important differences between the conclusions drawn from the Kendall et al. and Cribbie and Arpin-Cribbie methods for assessing equivalence-based clinical significance.

Psychological well-being in obese inpatients with ischemic heart disease at entry and at discharge from a four-week cardiac rehabilitation program.

The purposes of this observational pre-post study were twofold: 1- to evaluate psychological health in obese patients with ischemic heart disease at admission to cardiac rehabilitation (CR) and 2 - to examine the effectiveness of a 4-week CR residential program in improving obese patients' psychological well-being at discharge from CR. A sample of 177 obese patients completed the Psychological General Well-Being Inventory (PGWBI) at admission to the CR program and at discharge. The equivalence testing method with normative comparisons was used to determine the clinical significance of improvements after having established that baseline mean scores on the PGWBI scales were significantly lower than normal means. Results show that patients scored equally or better than norms on many PGWBI dimensions at admission to CR but scored significantly worse on Global Score, Vitality and Self-control. At discharge, mean scores that were impaired at baseline returned to normal levels at the more conservative equivalence interval. A 4-week CR program was thus effective in improving obese patients' psychological well-being. The equivalence testing method allowed to establish the clinical significance of such improvement.

Detecting a lack of association: an equivalence testing approach.

Researchers often test for a lack of association between variables. A lack of association is usually established by demonstrating a non-significant relationship with a traditional test (e.g., Pearson's r). However, for logical as well as statistical reasons, such conclusions are problematic. In this paper, we discuss and compare the empirical Type I error and power rates of three lack of association tests. The results indicate that large, sometimes very large, sample sizes are required for the test statistics to be appropriate. What is especially problematic is that the required sample sizes may exceed what is practically feasible for the conditions that are expected to be common among researchers in psychology. This paper highlights the importance of using available lack of association tests, instead of traditional tests of association, for demonstrating the independence of variables, and qualifies the conditions under which these tests are appropriate.

Factor structure of the Beck Hopelessness Scale in individuals with advanced cancer.

OBJECTIVE: Although the Beck Hopelessness Scale is often used with the seriously ill, its factor structure has been given relatively little consideration in this context. METHODS: The factor structure of this scale was examined in a sample of 406 ambulatory patients with advanced lung or gastrointestinal cancer, using a sequential exploratory-confirmatory factor analysis procedure. RESULTS: A two-factor model was consistent with the data: The first factor reflected a negative outlook and was labeled 'negative expectations'; the second factor identified a sense of resignation and was labeled 'loss of motivation.' CONCLUSIONS: Implications regarding scoring of the scale in this population are discussed, as are implications of the two-factor structure for our understanding of hopelessness in individuals with advanced cancer.

Evaluating clinical significance through equivalence testing: extending the normative comparisons approach.

The field of psychology, as with many other disciplines, has been increasingly interested in being able to measure the effectiveness of behavioral interventions. This trend has led to a number of different approaches for measuring clinical significance, each addressing a slightly different aspect of the clinical outcome. Recently, clinical psychologists (and clients) have supported the contention that one of the most important therapeutic questions is whether clients are functioning equivalently to normal controls following an intervention. To address this question, Kendall, Marrs-Garcia, Nath, and Sheldrick (1999) presented an approach to measuring clinical significance that utilizes tests of equivalence. The present study clarifies the nature of the hypotheses being conducted in measuring clinical significance with tests of equivalence and extends the approach by incorporating recent advances in equivalence testing. A revised approach for evaluating clinical significance via equivalence testing is proposed, and an empirical example demonstrating this approach is provided.