The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward.

The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a "crisis of confidence" stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current "crisis of confidence" in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.

Effects of violent and nonviolent sexualized media on aggression-related thoughts, feelings, attitudes, and behaviors: A meta-analytic review.

Women are often depicted as sex objects rather than as human beings in the media (e.g., magazines, television programs, films, and video games). Theoretically, media depictions of females as sex objects could lead to negative attitudes and even aggressive behavior toward them in the real world. Using the General Aggression Model (Anderson & Bushman, 2002) as a theoretical framework, this meta-analytic review synthesizes the literature on the effects of sexualized media (both violent and nonviolent) on aggression-related thoughts, attitudes, and behaviors. Our sample includes 166 independent studies involving 124,236 participants, which yielded 321 independent effects. Overall, the effects were "small" to "moderate" in size (r = .16 [.14-.18]). Significant correlations were found in experimental, cross-sectional, and longitudinal studies, indicating a triangulation of evidence. Effects were stronger for violent sexualized media (r = .25 [.19-.31]) than for nonviolent sexualized media (r = .15 [.13-.17]), although the effects of nonviolent sexualized media were still significant and nontrivial in size. Moreover, the effects of violent sexualized media on aggression were greater than the effects of violent non-sexualized media on aggression obtained in previous meta-analyses. Effects were similar for male and female participants, for college students and non-students, and for participants of all ages. The effects were also stable over time. Sensitivity analyses found that effects were not unduly influenced by publication bias and/or outliers. In summary, exposure to sexualized media content, especially in combination with violence, has negative effects on women, particularly on what people think about them and how aggressively they treat them.

Getting along to get ahead: The role of social context in tournament promotion and reward systems.

Tournament theory posits that some organizations are modeled after sports tournaments whereby individuals are incentivized to compete and win against other members of the organization. A persistent criticism of tournament theory is that rank-order success of employees is entirely dependent on non-interacting or at least non-cooperating entities. To address what part, if any, cooperation plays in competitive tournaments, this study examines the role of social networks in tournament-style promotion and reward systems. Specifically, we seek to identify the importance of social relationships, such as group dissimilarity, initial tie formation, and tie strength in predicting tournament success. Bringing two largely independent research streams together (one focused on cooperation and one framed around competition), we examine how individuals' performance interacts with their social relationships-their social networks-to influence their chances of winning a tournament. Using the Survivor television series, we analyze the behaviors of 535 interacting contestants across 30 tournaments. In general, the findings help to illustrate how performance and social networks predict tournament advancement. Interestingly, we find that group dissimilarity based on gender, race, and age, largely does not play a role in advancement in the tournaments. Further, the strength of ties fails to mediate between variables such as group dissimilarity and initial tie formation. We conclude by discussing future directions for theoretical and practical exploration of tournament-style promotion systems. Recommendations include continuing to explore and test the role of social dynamics in compensation and promotion systems.

Effects of Weapons on Aggressive Thoughts, Angry Feelings, Hostile Appraisals, and Aggressive Behavior: A Meta-Analytic Review of the Weapons Effect Literature.

A landmark 1967 study showed that simply seeing a gun can increase aggression-called the "weapons effect." Since 1967, many other studies have attempted to replicate and explain the weapons effect. This meta-analysis integrates the findings of weapons effect studies conducted from 1967 to 2017 and uses the General Aggression Model (GAM) to explain the weapons effect. It includes 151 effect-size estimates from 78 independent studies involving 7,668 participants. As predicted by the GAM, our naïve meta-analytic results indicate that the mere presence of weapons increased aggressive thoughts, hostile appraisals, and aggression, suggesting a cognitive route from weapons to aggression. Weapons did not significantly increase angry feelings. Yet, a comprehensive sensitivity analysis indicated that not all naïve mean estimates were robust to the presence of publication bias. In general, these results suggest that the published literature tends to overestimate the weapons effect for some outcomes and moderators.

Violent video game effects remain a societal concern: Reply to Hilgard, Engelhardt, and Rouder (2017).

A large meta-analysis by Anderson et al. (2010) found that violent video games increased aggressive thoughts, angry feelings, physiological arousal, and aggressive behavior and decreased empathic feelings and helping behavior. Hilgard, Engelhardt, and Rouder (2017) reanalyzed the data of Anderson et al. (2010) using newer publication bias methods (i.e., precision-effect test, precision-effect estimate with standard error, p-uniform, p-curve). Based on their reanalysis, Hilgard, Engelhardt, and Rouder concluded that experimental studies examining the effect of violent video games on aggressive affect and aggressive behavior may be contaminated by publication bias, and these effects are very small when corrected for publication bias. However, the newer methods Hilgard, Engelhardt, and Rouder used may not be the most appropriate. Because publication bias is a potential a problem in any scientific domain, we used a comprehensive sensitivity analysis battery to examine the influence of publication bias and outliers on the experimental effects reported by Anderson et al. We used best meta-analytic practices and the triangulation approach to locate the likely position of the true mean effect size estimates. Using this methodological approach, we found that the combined adverse effects of outliers and publication bias was less severe than what Hilgard, Engelhardt, and Rouder found for publication bias alone. Moreover, the obtained mean effects using recommended methods and practices were not very small in size. The results of the methods used by Hilgard, Engelhardt, and Rouder tended to not converge well with the results of the methods we used, indicating potentially poor performance. We therefore conclude that violent video game effects should remain a societal concern. (PsycINFO Database Record

The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance.

INTRODUCTION: Sensitivity analyses refer to investigations of the degree to which the results of a meta-analysis remain stable when conditions of the data or the analysis change. To the extent that results remain stable, one can refer to them as robust. Sensitivity analyses are rarely conducted in the organizational science literature. Despite conscientiousness being a valued predictor in employment selection, sensitivity analyses have not been conducted with respect to meta-analytic estimates of the correlation (i.e., validity) between conscientiousness and job performance. METHODS: To address this deficiency, we reanalyzed the largest collection of conscientiousness validity data in the personnel selection literature and conducted a variety of sensitivity analyses. RESULTS: Publication bias analyses demonstrated that the validity of conscientiousness is moderately overestimated (by around 30%; a correlation difference of about .06). The misestimation of the validity appears to be due primarily to suppression of small effects sizes in the journal literature. These inflated validity estimates result in an overestimate of the dollar utility of personnel selection by millions of dollars and should be of considerable concern for organizations. CONCLUSION: The fields of management and applied psychology seldom conduct sensitivity analyses. Through the use of sensitivity analyses, this paper documents that the existing literature overestimates the validity of conscientiousness in the prediction of job performance. Our data show that effect sizes from journal articles are largely responsible for this overestimation.

Assessing the validity of sales self-efficacy: a cautionary tale.

We developed a focused, context-specific measure of sales self-efficacy and assessed its incremental validity against the broad Big 5 personality traits with department store salespersons, using (a) both a concurrent and a predictive design and (b) both objective sales measures and supervisory ratings of performance. We found that in the concurrent study, sales self-efficacy predicted objective and subjective measures of job performance more than did the Big 5 measures. Significant differences between the predictability of subjective and objective measures of performance were not observed. Predictive validity coefficients were generally lower than concurrent validity coefficients. The results suggest that there are different dynamics operating in concurrent and predictive designs and between broad and contextualized measures; they highlight the importance of distinguishing between these designs and measures in meta-analyses. The results also point to the value of focused, context-specific personality predictors in selection research.