Human ratings take time: A hierarchical facets model for the joint analysis of ratings and rating times.

Performance assessments increasingly utilize onscreen or internet-based technology to collect human ratings. One of the benefits of onscreen ratings is the automatic recording of rating times along with the ratings. Considering rating times as an additional data source can provide a more detailed picture of the rating process and improve the psychometric quality of the assessment outcomes. However, currently available models for analyzing performance assessments do not incorporate rating times. The present research aims to fill this gap and advance a joint modeling approach, the "hierarchical facets model for ratings and rating times" (HFM-RT). The model includes two examinee parameters (ability and time intensity) and three rater parameters (severity, centrality, and speed). The HFM-RT successfully recovered examinee and rater parameters in a simulation study and yielded superior reliability indices. A real-data analysis of English essay ratings collected in a high-stakes assessment context revealed that raters exhibited considerably different speed measures, spent more time on high-quality than low-quality essays, and tended to rate essays faster with increasing severity. However, due to the significant heterogeneity of examinees' writing proficiency, the improvement in the assessment's reliability using the HFM-RT was not salient in the real-data example. This discussion focuses on the advantages of accounting for rating times as a source of information in rating quality studies and highlights perspectives from the HFM-RT for future research on rater cognition.

Detecting Differential Rater Functioning in Severity and Centrality: The Dual DRF Facets Model.

Performance assessments heavily rely on human ratings. These ratings are typically subject to various forms of error and bias, threatening the assessment outcomes' validity and fairness. Differential rater functioning (DRF) is a special kind of threat to fairness manifesting itself in unwanted interactions between raters and performance- or construct-irrelevant factors (e.g., examinee gender, rater experience, or time of rating). Most DRF studies have focused on whether raters show differential severity toward known groups of examinees. This study expands the DRF framework and investigates the more complex case of dual DRF effects, where DRF is simultaneously present in rater severity and centrality. Adopting a facets modeling approach, we propose the dual DRF model (DDRFM) for detecting and measuring these effects. In two simulation studies, we found that dual DRF effects (a) negatively affected measurement quality and (b) can reliably be detected and compensated under the DDRFM. Using sample data from a large-scale writing assessment (N = 1,323), we demonstrate the practical measurement consequences of the dual DRF effects. Findings have implications for researchers and practitioners assessing the psychometric quality of ratings.

Gender subgroups and intergroup perception: adolescents' views of own-gender and other-gender groups.

The authors examined intergroup bias and perceived group variability regarding gender subgroups relevant to the adolescent culture. Participants were 126 high school students, between 16 years and 19 years of age, who listed male and female subgroups, performed a series of group perception tasks, and, for each subgroup, indicated whether they themselves belonged to the group. Results showed that adolescents' perceptions of gender subgroups were subject to in-group favoritism and out-group derogation, as well as to group homogeneity effects. In line with predictions derived from Optimal Distinctiveness Theory, male and female participants favored in-groups over own-gender and other-gender out-groups and favored other-gender out-groups over own-gender outgroups. Ambivalence was particularly high with other-gender out-groups. The authors discussed implications for future research on the development of gender prejudice and for interventions to reduce prejudice.

Bad but bold: Ambivalent attitudes toward men predict gender inequality in 16 nations.

A 16-nation study involving 8,360 participants revealed that hostile and benevolent attitudes toward men, assessed by the Ambivalence Toward Men Inventory (P. Click & S.T. Fiske, 1999), were (a) reliably measured across cultures, (b) positively correlated (for men and women, within samples and across nations) with each other and with hostile and benevolent sexism toward women (Ambivalent Sexism Inventory, P. Click & S.T. Fiske, 1996), and (c) negatively correlated with gender equality in cross-national comparisons. Stereotype measures indicated that men were viewed as having less positively valenced but more powerful traits than women. The authors argue that hostile as well as benevolent attitudes toward men reflect and support gender inequality by characterizing men as being designed for dominance.