Comparing forced-choice and single-stimulus personality scores on a level playing field: A meta-analysis of psychometric properties and susceptibility to faking.

Forced-choice (FC) personality assessments have shown potential in mitigating the effects of faking. Yet despite increased attention and usage, there exist gaps in understanding the psychometric properties of FC assessments, and particularly when compared to traditional single-stimulus (SS) measures. The present study conducted a series of meta-analyses comparing the psychometric properties of FC and SS assessments after placing them on an equal playing field-by restricting to only studies that examined matched assessments of each format, and thus, avoiding the extraneous confound of using comparisons from different contexts (Sackett, 2021). Matched FC and SS assessments were compared in terms of criterion-related validity and susceptibility to faking in terms of mean shifts and validity attenuation. Additionally, the correlation between FC and SS scores was examined to help establish construct validity evidence. Results showed that matched FC and SS scores exhibit strong correlations with one another (ρ = .69), though correlations weakened when the FC measure was faked (ρ = .59) versus when both measures were taken honestly (ρ = .73). Average scores increased from honest to faked samples for both FC (d = .41) and SS scores (d = .75), though the effect was more pronounced for SS measures and with larger effects for context-desirable traits (FC d = .61, SS d = .99). Criterion-related validity was similar between matched FC and SS measures overall. However, when considering validity in faking contexts, FC scores exhibited greater validity than SS measures. Thus, although FC measures are not completely immune to faking, they exhibit meaningful benefits over SS measures in contexts of faking. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Turning words into numbers: Assessing work attitudes using natural language processing.

Researchers and practitioners are often interested in assessing employee attitudes and work perceptions. Although such perceptions are typically measured using Likert surveys or some other closed-end numerical rating format, many organizations also have access to large amounts of qualitative employee data. For example, open-ended comments from employee surveys allow workers to provide rich and contextualized perspectives about work. Unfortunately, there are practical challenges when trying to understand employee perceptions from qualitative data. Given this, the present study investigated whether natural language processing (NLP) algorithms could be developed to automatically score employee comments according to important work attitudes and perceptions. Using a large sample of employees, algorithms were developed to translate text into scores that reflect what comments were about (theme scores) and how positively targeted constructs were described (valence scores) for 28 work constructs. The resulting algorithms and scores are labeled the Text-Based Attitude and Perception Scoring (TAPS) dictionaries, which are made publicly available and were built using a mix of count-based scoring and transformer neural networks. The psychometric properties of the TAPS scores were then investigated. Results showed that theme scores differentiated responses based on their likelihood to discuss specific constructs. Additionally, valence scores exhibited strong evidence of reliability and validity, particularly, when analyzed on text responses that were more relevant to the construct of interest. This suggests that researchers and practitioners should explicitly design text prompts to elicit construct-related information if they wish to accurately assess work attitudes and perceptions via NLP. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Using game-like animations of geometric shapes to simulate social interactions: An evaluation of group score differences.

This study introduces a novel, game-like method for measuring social intelligence: the Social Shapes Test. Unlike other existing video or game-based tests, the Shapes Test uses animations of abstract shapes to represent social interactions. We explore demographic differences in Shapes Test scores compared to a written situational judgment test. Gender and race/ethnicity only had meaningful effects on written SJT scores while no effects were found for Shapes Test scores. This pattern of results remained after controlling for general mental ability and English language exposure. We also found metric invariance between demographic groups for both tests. Our results demonstrate the potential for using animated shape tasks as an alternative to written SJTs when designing future game-based assessments.

Meta-analysis of biodata in employment settings: Providing clarity to criterion and construct-related validity estimates.

Although biodata inventories have long been used to hire job applicants, there are limitations to current biodata knowledge and little in the way of contemporary biodata meta-analytic reviews. This study establishes a precise understanding of biodata validity by conducting an updated meta-analysis that differentiates biodata validity in terms of two important defining features: construct domain and scoring method (rational, hybrid, empirical). Evidence was established in terms of criterion-related validity with job performance and additional work outcomes, as well as convergent validity with common external hiring measures. In total, 180 independent samples of criterion correlations were examined, and 63 samples were analyzed containing correlations with convergent measures. Findings across the meta-analyses revealed that biodata inventories are one of the most predictive assessment methods available, but that the relationship with work outcomes differs by construct domain and scoring method. Empirically scored overall composite scales had superior criterion-related validity (ρ = .44) to rationally scored composite scales (ρ = .24). Scales developed to measure conscientiousness and leadership were generally the most predictive of the job performance of the narrow construct domains, and particularly when empirically keyed. However, when biodata scores were correlated with theoretically aligned performance ratings, rational scoring resulted in similar validity coefficients as empirical scoring. Finally, biodata scales exhibited expected patterns of correlations with external measures and were only modestly correlated with cognitive ability and five-factor model personality scores. Taken together, biodata inventories are highly predictive assessment methods and are likely to provide unique variance over other common predictors. (PsycInfo Database Record (c) 2022 APA, all rights reserved).