Effects of response format on achievement and aptitude assessment results: multi-level random effects meta-analyses.

Psychological achievement and aptitude tests are fundamental elements of the everyday school, academic and professional lives of students, instructors, job applicants, researchers and policymakers. In line with growing demands for fair psychological assessment tools, we aimed to identify psychometric features of tests, test situations and test-taker characteristics that may contribute to the emergence of test bias. Multi-level random effects meta-analyses were conducted to estimate mean effect sizes for differences and relations between scores from achievement or aptitude measures with open-ended (OE) versus closed-ended (CE) response formats. Results from 102 primary studies with 392 effect sizes revealed positive relations between CE and OE assessments (mean r = 0.67, 95% CI [0.57; 0.76]), with negative pooled effect sizes for the difference between the two response formats (mean d av = -0.65; 95% CI [-0.78; -0.53]). Significantly higher scores were obtained on CE exams. Stem-equivalency of items, low-stakes test situations, written short answer OE question types, studies conducted outside the United States and before the year 2000, and test-takers' achievement motivation and sex were at least partially associated with smaller differences and/or larger relations between scores from OE and CE formats. Limitations and the results' implications for practitioners in achievement and aptitude testing are discussed.

Modeling Responses and Response Times in Tests With the Hierarchical Model and the Three-Parameter Lognormal Distribution.

The hierarchical model of van der Linden is the most popular model for responses and response times in tests. It is composed of two separate submodels-one for the responses and one for the response times-that are joined at a higher level. The submodel for the response times is based on the lognormal distribution. The lognormal distribution is a skew distribution with a support from zero to infinity. Such a support is unrealistic as the solution process demands a minimal processing time that sets a response time threshold. Ignoring this response time threshold misspecifies the model and threatens the validity of model-based inferences. In this article, the response time model of van der Linden is replaced by a model that is based on the three-parameter lognormal distribution. The three-parameter lognormal distribution extends the lognormal distribution by an additional location parameter that bounds the support away from zero. Two different approaches to model fitting are proposed and evaluated with regard to parameter recovery in a simulation study. The extended model is applied to two data sets. In both data sets, the extension improves the fit of the hierarchical model.

Psychometric Properties of the Multifaceted Gender-Related Attributes Survey (GERAS).

Since the 1920s, psychologists have sought to assess the sex- and gender-related attributes of men and women, including primarily aspects of personality and focusing on positive characteristics. In this paper, we introduce a new questionnaire for assessing gender-related attributes with a broader approach than provided by previous ones. Therefore, the questionnaire includes (a) not only personality traits but also cognitions and interests and (b) not only positive but also negative and neutral characteristics. Two independent datasets were acquired (Study 1: N = 1,466; Study 2: N = 471) for development and psychometric analyses. Factor analysis confirmed a hierarchical structure with two separate dimensions of masculinity and femininity overarching the multiple first-order domains of personality, cognition, and interests. Analyses of reliability and convergence with other gender identity and personality scales revealed sufficient values. The new instrument discriminated between the biological sexes and was related to the gender quotas in participants' occupations and social environments, thus providing evidence for criterion-related validity. Therefore, we propose the Gender-Related Attributes Survey (GERAS) as a useful tool for objectively assessing gender-related attributes across multiple facets in gender and sex-difference research.

How Do Men and Women Perceive a High-Stakes Test Situation?

The results of some high-stakes aptitude tests in Austria have revealed sex differences. We suggest that such discrepancies are mediated not principally by differences in aptitudes, skills, and knowledge but sex differences in test takers' perceptions of the test situation. Furthermore, previous research has indicated that candidates' evaluations of the fairness of the testing tool are of great importance from an institutional point of view because such perceptions are known to influence an organization's attractiveness. In this study, we aimed to investigate how women and men perceive and evaluate certain aspects of a high-stakes test situation by using the results and evaluations of an actual medical school aptitude test (747 applicants; 59% women). Test takers voluntarily evaluated the test situation and rated specific aspects of it (e.g., the fairness of the selection tool) and provided information regarding their test anxiety immediately after they completed the 4-h test. Data analyses indicated small, albeit significant sex differences in participants' perceptions of the test. Men described the test situation as slightly giving more opportunity to socialize and possessing more opportunity to deceive than women did. Furthermore, the perception of the test situation did not directly predict the test results, but it served as a moderator for the indirect effect of test anxiety on test results. By contrast, there were significant direct effects but no indirect effects of situation perception on evaluations of the fairness of the selection tool: The more the test situation was perceived as a high-pressure situation, the lower the fairness ratings of the testing tool. Results were discussed with reference to gender roles and test fairness.

Women Who Emerge as Leaders in Temporarily Assigned Work Groups: Attractive and Socially Competent but Not Babyfaced or Naïve?

The underrepresentation of women in top positions has been in the spotlight of research for decades. Prejudice toward female leaders, which decreases women's chances of emerging as leaders, has been discussed as a potential reason. Aiming to investigate the underlying mechanisms of this prejudice, we focused on the question of how facial characteristics might influence women's leadership emergence. Because other research has related ascribed social competence and ascribed naïveté to attractiveness and babyfacedness, respectively, we hypothesized that ascribed social competence would mediate the impact of ascribed attractiveness on leadership emergence and that ascribed naïveté would mediate the impact of ascribed babyfacedness on leadership emergence. In a pilot study, we analyzed data from 101 participants of a women's leadership contest held in 2015 in Germany. We then confirmed these results in a methodologically improved main study on other women who participated in the contest in one of two other years: 2016 and 2017 (N = 195). Women applied to participate in the contest by recording their answers to several questions in a video interview. In the contest, they were assigned to teams of about ten women each and worked on several assessment-center-like tasks. After each task, each member of each team nominated the three women they believed showed the best leadership potential in their group. We operationalized women's leadership emergence as the number of nominations received. We measured participants' facial attractiveness, babyfacedness, social competence, and naïveté by having raters follow a specifically developed rating manual to rate the answers the women gave in the video interviews. In both studies, the results indicated that women with higher ascribed facial attractiveness had higher ascribed social competence, which significantly predicted leadership emergence in the contest. Likewise, women with higher ascribed babyfacedness had higher ascribed naïveté, which significantly, albeit only slightly, negatively predicted leadership emergence. We discuss the implications of the results for personnel selection.

Asynchronous Video Interviewing as a New Technology in Personnel Selection: The Applicant's Point of View.

The present study aimed to integrate findings from technology acceptance research with research on applicant reactions to new technology for the emerging selection procedure of asynchronous video interviewing. One hundred six volunteers experienced asynchronous video interviewing and filled out several questionnaires including one on the applicants' personalities. In line with previous technology acceptance research, the data revealed that perceived usefulness and perceived ease of use predicted attitudes toward asynchronous video interviewing. Furthermore, openness revealed to moderate the relation between perceived usefulness and attitudes toward this particular selection technology. No significant effects emerged for computer self-efficacy, job interview self-efficacy, extraversion, neuroticism, and conscientiousness. Theoretical and practical implications are discussed.

Interactive effects of culture and sex hormones on the sex role self-concept.

Sex role orientation, i.e., a person's masculinity or femininity, influences cognitive and emotional performance, like biological sex. While it is now widely accepted that sex differences are modulated by the hormonal status of female participants (menstrual cycle, hormonal contraceptive use), the question, whether hormonal status and sex hormones also modulate participants sex role orientation has hardly been addressed previously. The present study assessed sex role orientation and hormonal status as well as sex hormone levels in three samples of participants from two different cultures (Northern American, Middle European). Menstrual cycle phase did not affect participant's masculinity or femininity, but had a significant impact on reference group. While women in their follicular phase (low levels of female sex hormones) determined their masculinity and femininity in reference to men, women in their luteal phase (high levels of female sex hormones) determined their masculinity and femininity in reference to women. Hormonal contraceptive users rated themselves as significantly more feminine and less masculine than naturally cycling women. Furthermore, the impact of biological sex on the factorial structure of sex role orientation as well as the relationship of estrogen to masculinity/femininity was modulated by culture. We conclude that culture and sex hormones interactively affect sex role orientation and hormonal status of participants should be controlled for when assessing masculinity and/or femininity.

Response Time Modeling Based on the Proportional Hazards Model.

Response time data are regularly analyzed in psychology. When several response times are assessed per participant, it is common practice to use latent trait models in order to account for the dependency of the response times from the same participant. One such model is the proportional hazards model with random effects. Despite its popularity in survival analysis, this model is rarely used in psychology because of the difficulty of model estimation when latent variables are present. In this article, a new estimation method is proposed. This method is based on the rank correlation matrix containing Kendall's Tau coefficients and unweighted least squares estimation ( Kendall, 1938 ). Compared with marginal maximum likelihood estimation, the new estimation approach is simple, not computationally intensive, and almost as efficient. Additionally, the approach allows the implementation of a test for model fit. Feasibility of the estimation method and validity of the fit test is demonstrated with a simulation study. An application of the model to a real data set is provided.