Effects of response format on achievement and aptitude assessment results: multi-level random effects meta-analyses.

Psychological achievement and aptitude tests are fundamental elements of the everyday school, academic and professional lives of students, instructors, job applicants, researchers and policymakers. In line with growing demands for fair psychological assessment tools, we aimed to identify psychometric features of tests, test situations and test-taker characteristics that may contribute to the emergence of test bias. Multi-level random effects meta-analyses were conducted to estimate mean effect sizes for differences and relations between scores from achievement or aptitude measures with open-ended (OE) versus closed-ended (CE) response formats. Results from 102 primary studies with 392 effect sizes revealed positive relations between CE and OE assessments (mean r = 0.67, 95% CI [0.57; 0.76]), with negative pooled effect sizes for the difference between the two response formats (mean d av = -0.65; 95% CI [-0.78; -0.53]). Significantly higher scores were obtained on CE exams. Stem-equivalency of items, low-stakes test situations, written short answer OE question types, studies conducted outside the United States and before the year 2000, and test-takers' achievement motivation and sex were at least partially associated with smaller differences and/or larger relations between scores from OE and CE formats. Limitations and the results' implications for practitioners in achievement and aptitude testing are discussed.

Self-correction in science: The effect of retraction on the frequency of citations.

We investigate the citation frequency of retracted scientific papers in science. For the period of five years before and after retraction, we counted the citations to papers in a sample of over 3,000 retracted, and a matched sample of another 3,000 non-retracted papers. Retraction led to a decrease in average annual citation frequency from about 5 before, to 2 citations after retraction. In contrast, for non-retracted control papers the citation counts were 4, and 5, respectively. Put differently, we found only a limited effect of retraction: retraction decreased citation frequency only by about 60%, as compared to non-retracted papers. Thus, retracted papers often live on. For effective self-correction the scientific enterprise needs to be more effective in removing retracted papers from the scientific record. We discuss recent proposals to do so.

The missing link: Global-local processing relates to number-magnitude processing in women.

Number processing and visual hierarchical processing (global-local processing) have much in common. However, the shared aspects of number processing and global-local processing have not been investigated so far. Most visual stimuli are hierarchical with global structures made up of local parts. Processing of global and local aspects occurs in parallel and the global advantage effect indicates faster reactions to global than local aspects. Likewise, multidigit numbers can be represented holistically (whole number magnitudes) or in a decomposed fashion (single digit magnitudes). During comparison of 2-digit numbers, the unit-decade compatibility effect indicates slower responses when the larger number contains the smaller unit digit and has been suggested as a measure of how strongly participants rely on decomposed number representations. However, this interpretation of the compatibility effect is still controversial and a link between global-local processing and the individual tendency to rely on decomposed representations of multidigit numbers remains to be established. To that end we assessed whether the compatibility effect during number comparison was related to various measures of global advantage. To answer this question we drew upon existing data from participants who had completed both, the number comparison task and 2 global-local tasks. Results show that the compatibility effect is indeed negatively related to several measures of global advantage in women, with no evidence for such a relationship in men. These results demonstrate that global-local processing transcends into the numerical domain but also suggest that the compatibility effect reflects different mechanisms in men and women. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Psychometric Properties of the Multifaceted Gender-Related Attributes Survey (GERAS).

Since the 1920s, psychologists have sought to assess the sex- and gender-related attributes of men and women, including primarily aspects of personality and focusing on positive characteristics. In this paper, we introduce a new questionnaire for assessing gender-related attributes with a broader approach than provided by previous ones. Therefore, the questionnaire includes (a) not only personality traits but also cognitions and interests and (b) not only positive but also negative and neutral characteristics. Two independent datasets were acquired (Study 1: N = 1,466; Study 2: N = 471) for development and psychometric analyses. Factor analysis confirmed a hierarchical structure with two separate dimensions of masculinity and femininity overarching the multiple first-order domains of personality, cognition, and interests. Analyses of reliability and convergence with other gender identity and personality scales revealed sufficient values. The new instrument discriminated between the biological sexes and was related to the gender quotas in participants' occupations and social environments, thus providing evidence for criterion-related validity. Therefore, we propose the Gender-Related Attributes Survey (GERAS) as a useful tool for objectively assessing gender-related attributes across multiple facets in gender and sex-difference research.

How Do Men and Women Perceive a High-Stakes Test Situation?

The results of some high-stakes aptitude tests in Austria have revealed sex differences. We suggest that such discrepancies are mediated not principally by differences in aptitudes, skills, and knowledge but sex differences in test takers' perceptions of the test situation. Furthermore, previous research has indicated that candidates' evaluations of the fairness of the testing tool are of great importance from an institutional point of view because such perceptions are known to influence an organization's attractiveness. In this study, we aimed to investigate how women and men perceive and evaluate certain aspects of a high-stakes test situation by using the results and evaluations of an actual medical school aptitude test (747 applicants; 59% women). Test takers voluntarily evaluated the test situation and rated specific aspects of it (e.g., the fairness of the selection tool) and provided information regarding their test anxiety immediately after they completed the 4-h test. Data analyses indicated small, albeit significant sex differences in participants' perceptions of the test. Men described the test situation as slightly giving more opportunity to socialize and possessing more opportunity to deceive than women did. Furthermore, the perception of the test situation did not directly predict the test results, but it served as a moderator for the indirect effect of test anxiety on test results. By contrast, there were significant direct effects but no indirect effects of situation perception on evaluations of the fairness of the selection tool: The more the test situation was perceived as a high-pressure situation, the lower the fairness ratings of the testing tool. Results were discussed with reference to gender roles and test fairness.

Global-local processing relates to spatial and verbal processing: implications for sex differences in cognition.

Sex differences have been reported for a variety of cognitive tasks and related to the use of different cognitive processing styles in men and women. It was recently argued that these processing styles share some characteristics across tasks, i.e. male approaches are oriented towards holistic stimulus aspects and female approaches are oriented towards stimulus details. In that respect, sex-dependent cognitive processing styles share similarities with attentional global-local processing. A direct relationship between cognitive processing and global-local processing has however not been previously established. In the present study, 49 men and 44 women completed a Navon paradigm and a Kimchi Palmer task as well as a navigation task and a verbal fluency task with the goal to relate the global advantage (GA) effect as a measure of global processing to holistic processing styles in both tasks. Indeed participants with larger GA effects displayed more holistic processing during spatial navigation and phonemic fluency. However, the relationship to cognitive processing styles was modulated by the specific condition of the Navon paradigm, as well as the sex of participants. Thus, different types of global-local processing play different roles for cognitive processing in men and women.

Components of Mathematics Anxiety: Factor Modeling of the MARS30-Brief.

Mathematics anxiety involves feelings of tension, discomfort, high arousal, and physiological reactivity interfering with number manipulation and mathematical problem solving. Several factor analytic models indicate that mathematics anxiety is rather a multidimensional than unique construct. However, the factor structure of mathematics anxiety has not been fully clarified by now. This issue shall be addressed in the current study. The Mathematics Anxiety Rating Scale (MARS) is a reliable measure of mathematics anxiety (Richardson and Suinn, 1972), for which several reduced forms have been developed. Most recently, a shortened version of the MARS (MARS30-brief) with comparable reliability was published. Different studies suggest that mathematics anxiety involves up to seven different factors. Here we examined the factor structure of the MARS30-brief by means of confirmatory factor analysis. The best model fit was obtained by a six-factor model, dismembering the known two general factors "Mathematical Test Anxiety" (MTA) and "Numerical Anxiety" (NA) in three factors each. However, a more parsimonious 5-factor model with two sub-factors for MTA and three for NA fitted the data comparably well. Factors were differentially susceptible to sex differences and differences between majors. Measurement invariance for sex was established.

The significance fallacy in inferential statistics.

BACKGROUND: Statistical significance is an important concept in empirical science. However the meaning of the term varies widely. We investigate into the intuitive understanding of the notion of significance. METHODS: We described the results of two different experiments published in a major psychological journal to a sample of students of psychology, labeling the findings as 'significant' versus 'non-significant.' Participants were asked to estimate the effect sizes and sample sizes of the original studies. RESULTS: Labeling the results of a study as significant was associated with estimations of a big effect, but was largely unrelated to sample size. Similarly, non-significant results were estimated as near zero in effect size. CONCLUSIONS: After considerable training in statistics, students largely equate statistical significance with medium to large effect sizes, rather than with large sample sizes. The data show that students assume that statistical significance is due to real effects, rather than to 'statistical tricks' (e.g., increasing sample size).

Differentiating self-projection from simulation during mentalizing: evidence from fMRI.

We asked participants to predict which of two colors a similar other (student) and a dissimilar other (retiree) likes better. We manipulated if color-pairs were two hues from the same color-category (e.g. green) or two conceptually different colors (e.g. green versus blue). In the former case, the mental state that has to be represented (i.e., the percept of two different hues of green) is predominantly non-conceptual or phenomenal in nature, which should promote mental simulation as a strategy for mentalizing. In the latter case, the mental state (i.e. the percept of green versus blue) can be captured in thought by concepts, which facilitates the use of theories for mentalizing. In line with the self-projection hypothesis, we found that cortical midline areas including vmPFC / orbitofrontal cortex and precuneus were preferentially activated for mentalizing about a similar other. However, activation was not affected by the nature of the color-difference, suggesting that self-projection subsumes simulation-like processes but is not limited to them. This indicates that self-projection is a universal strategy applied in different contexts--irrespective of the availability of theories for mentalizing. Along with midline activations linked to self-projection, we also observed activation in right lateral frontal and dorsal parietal areas showing a theory-like pattern. Taken together, this shows that mentalizing does not operate based on simulation or theory, but that both strategies are used concurrently to predict the choices of others.

Publication bias in psychology: a diagnosis based on the correlation between effect size and sample size.

BACKGROUND: The p value obtained from a significance test provides no information about the magnitude or importance of the underlying phenomenon. Therefore, additional reporting of effect size is often recommended. Effect sizes are theoretically independent from sample size. Yet this may not hold true empirically: non-independence could indicate publication bias. METHODS: We investigate whether effect size is independent from sample size in psychological research. We randomly sampled 1,000 psychological articles from all areas of psychological research. We extracted p values, effect sizes, and sample sizes of all empirical papers, and calculated the correlation between effect size and sample size, and investigated the distribution of p values. RESULTS: We found a negative correlation of r = -.45 [95% CI: -.53; -.35] between effect size and sample size. In addition, we found an inordinately high number of p values just passing the boundary of significance. Additional data showed that neither implicit nor explicit power analysis could account for this pattern of findings. CONCLUSION: The negative correlation between effect size and samples size, and the biased distribution of p values indicate pervasive publication bias in the entire field of psychology.