Evaluation of heterogeneity and heterogeneity interval estimators in random-effects meta-analysis of the standardized mean difference in education and psychology.

Meta-analyses are conducted to synthesize the quantitative results of related studies. The random-effects meta-analysis model is based on the assumption that a distribution of true effects exists in the population. This distribution is often assumed to be normal with a mean and variance. The population variance, also called heterogeneity, can be estimated numerous ways. Research exists comparing subsets of heterogeneity estimators over limited conditions. Additionally, heterogeneity is a parameter estimated with uncertainty. Various methods exist for heterogeneity interval estimation, and similar to heterogeneity estimators, these evaluations are limited. The current simulation study examined the performance of Bayesian (with 5 prior specifications) and non-Bayesian estimators over conditions found after a review of meta-analyses of the standardized mean difference in education and psychology research. Three simulation conditions were varied: (a) number of effect sizes per meta-analysis, (b) true heterogeneity, and (c) sample size per effect size within each meta-analysis. Estimators were evaluated over average bias and means square error. Methods of interval estimation were then evaluated with the estimators found to operate optimally. Interval estimators were evaluated based on coverage probability, interval width, and coverage of the estimated value. Overall, the Paule and Mandel estimator, in conjunction with the Jackson method of interval estimation, is recommended if no knowledge exists with regards to the expected value of heterogeneity when synthesizing the standardized mean difference effect size. If heterogeneity is expected to be small (e.g., < .075), then REML with the profile likelihood interval estimator is recommended. Sensitivity analysis evaluating differences in substantive conclusions with a suite of heterogeneity estimators, such as Paule and Mandel, REML, and Hedges and Olkin, is recommended. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Moving Beyond Univariate Post-Hoc Testing in Exercise Science: A Primer on Descriptive Discriminate Analysis.

There has been a recent call to improve data reporting in kinesiology journals, including the appropriate use of univariate and multivariate analysis techniques. For example, a multivariate analysis of variance (MANOVA) with univariate post hocs and a Bonferroni correction is frequently used to investigate group differences on multiple dependent variables. However, this univariate approach decreases power, increases the risk for Type 1 error, and contradicts the rationale for conducting multivariate tests in the first place. PURPOSE: The purpose of this study was to provide a user-friendly primer on conducting descriptive discriminant analysis (DDA), which is a post-hoc strategy to MANOVA that takes into account the complex relationships among multiple dependent variables. METHOD: A real-world example using the Statistical Package for the Social Sciences syntax and data from 1,095 middle school students on their body composition and body image are provided to explain and interpret the results from DDA. RESULTS: While univariate post hocs increased the risk for Type 1 error to 76%, the DDA identified which dependent variables contributed to group differences and which groups were different from each other. For example, students in the very lean and Healthy Fitness Zone categories for body mass index experienced less pressure to lose weight, more satisfaction with their body, and higher physical self-concept than the Needs Improvement Zone groups. However, perceived pressure to gain weight did not contribute to group differences because it was a suppressor variable. CONCLUSION: Researchers are encouraged to use DDA when investigating group differences on multiple correlated dependent variables to determine which variables contributed to group differences.

The assumption of a reliable instrument and other pitfalls to avoid when considering the reliability of data.

The purpose of this article is to help researchers avoid common pitfalls associated with reliability including incorrectly assuming that (a) measurement error always attenuates observed score correlations, (b) different sources of measurement error originate from the same source, and (c) reliability is a function of instrumentation. To accomplish our purpose, we first describe what reliability is and why researchers should care about it with focus on its impact on effect sizes. Second, we review how reliability is assessed with comment on the consequences of cumulative measurement error. Third, we consider how researchers can use reliability generalization as a prescriptive method when designing their research studies to form hypotheses about whether or not reliability estimates will be acceptable given their sample and testing conditions. Finally, we discuss options that researchers may consider when faced with analyzing unreliable data.

Tools to support interpreting multiple regression in the face of multicollinearity.

While multicollinearity may increase the difficulty of interpreting multiple regression (MR) results, it should not cause undue problems for the knowledgeable researcher. In the current paper, we argue that rather than using one technique to investigate regression results, researchers should consider multiple indices to understand the contributions that predictors make not only to a regression model, but to each other as well. Some of the techniques to interpret MR effects include, but are not limited to, correlation coefficients, beta weights, structure coefficients, all possible subsets regression, commonality coefficients, dominance weights, and relative importance weights. This article will review a set of techniques to interpret MR effects, identify the elements of the data on which the methods focus, and identify statistical software to support such analyses.

Revisiting Interpretation of Canonical Correlation Analysis: A Tutorial and Demonstration of Canonical Commonality Analysis.

In the face of multicollinearity, researchers face challenges interpreting canonical correlation analysis (CCA) results. Although standardized function and structure coefficients provide insight into the canonical variates produced, they fall short when researchers want to fully report canonical effects. This article revisits the interpretation of CCA results, providing a tutorial and demonstrating canonical commonalty analysis. Commonality analysis fully explains the canonical effects produced by using the variables in a given canonical set to partition the variance of canonical variates produced from the other canonical set. Conducting canonical commonality analysis without the aid of software is laborious and may be untenable, depending on the number of noteworthy canonical functions and variables in either canonical set. Commonality analysis software is identified for the canonical correlation case and we demonstrate its use in facilitating model interpretation. Data from Holzinger and Swineford (1939) are employed to test a hypothetical theory that problem-solving skills are predicted by fundamental math ability.

Social validation of evidence-based practices in autism by parents, teachers, and administrators.

Relatively little attention has been devoted to the social validation of potentially effective autism interventions. Thus, it is often difficult to identify and implement evidence-based practices, and programming is often inadequate. The authors identified autism intervention components with reported effectiveness for school settings. The results of a social validation survey completed by parents, teachers, and administrators indicate strong, consistent support for program components falling within five functional areas: (a) individualized programming, (b) data collection, (c) the use of empirically-based strategies, (d) active collaboration, and (e) a focus on long-term outcomes. These socially validated interventions can be used to evaluate existing autism curricula and develop training for professionals, parents, and students in order to improve public school autism programs.

Sample homogeneity and coefficient alpha: comment on MJI and Alkhateeb's reliability generalization study and a theoretical clarification.

Reliability generalization studies are increasingly frequently reported in the literature, including Mji and Alkhateeb's 2005 study of the Conceptions of Mathematics Questionnaire. The present article comments on an issue of reliability generalization as used in that study and clarifies a theoretical point regarding the meaning of coefficient alpha and what data features tend to influence it as a measure of score reliability.

Conducting and interpreting canonical correlation analysis in personality research: a user-friendly primer.

The purpose of this article is to reduce potential statistical barriers and open doors to canonical correlation analysis (CCA) for applied behavioral scientists and personality researchers. CCA was selected for discussion, as it represents the highest level of the general linear model (GLM) and can be rather easily conceptualized as a method closely linked with the more widely understood Pearson r correlation coefficient. An understanding of CCA can lead to a more global appreciation of other univariate and multivariate methods in the GLM. We attempt to demonstrate CCA with basic language, using technical terminology only when necessary for understanding and use of the method. We present an entire example of a CCA analysis using SPSS (Version 11.0) with personality data.

Expanding reliability generalization: confidence intervals and Charter's combined reliability coefficient.

Methods of reliability generalization characterize the reliability of a scale's scores across studies. Charter in 2003 presented useful formulae for computing combined estimates of reliability. The present article illustrates use of Charter's formulae and application of confidence intervals for reliability coefficients, including the combined estimate. Researchers may find these methods for reliability generalization useful.

Evaluating the appropriateness of college-age norms for use with adolescents on the NEO personality inventory-revised.

The NEO Personality Inventory-Revised (NEO PI-R) measures normal personality characteristics and has demonstrated appropriate score reliability and validity. It is normed for two groups of individuals, college-age individuals 17 to 20 years old and adults 21 and older. Often, personality instruments normed on older individuals have been used with adolescent populations. To examine the appropriateness of this decision, the current study explored the differences between an adolescent sample (n = 79) and a college-age sample (n = 80) on the 30 facets and the five domains of the NEO PI-R. Group differences on the facet and domain scales were analyzed using descriptive discriminant analysis. Results indicated that the adolescent and college groups differed on each of the five domains. As expected, the groups also scored differently using the aggregated domain-level variables as the outcome measures. Suggestions for future research include the development of normative data for the adolescent population.