From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the Journal of Applied Psychology.

The psychometric soundness of measures has been a central concern of articles published in the Journal of Applied Psychology (JAP) since the inception of the journal. At the same time, it isn't clear that investigators and reviewers prioritize psychometric soundness to a degree that would allow one to have sufficient confidence in conclusions regarding constructs. The purposes of the present article are to (a) examine current scale development and evaluation practices in JAP; (b) compare these practices to recommended practices, previous practices, and practices in other journals; and (c) use these comparisons to make recommendations for reviewers, editors, and investigators regarding the creation and evaluation of measures including Excel-based calculators for various indices. Finally, given that model complexity appears to have increased the need for short scales, we offer a user-friendly R Shiny app (https://orgscience.uncc.edu/about-us/resources) that identifies the subset of items that maximize a variety of psychometric criteria rather than merely maximizing alpha. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Reflections on the Journal of Applied Psychology for 1989 to 1994: Changes in major research themes and practices over 25 years.

Informal observations concerning journal content indicates that research investigating organizational behavior topics, including work on the structure of groups and determinants and consequences of group process along with the role of leadership in groups, has increased. Some topics have disappeared (e.g., job analysis, human factors, union-related work, consumer behavior) and others are declining (e.g., research methods, psychometrics). Perhaps the biggest change is in the length of articles, which is mostly a function of the inclusion of greater numbers of references and appendix material. Publishing some of this material in supplementary online materials is now current practice in the Journal of Applied Psychology. Concerns about use of journal space may also be entirely moot, if electronic publishing as opposed to print publishing becomes the norm. (PsycINFO Database Record

Solving the Supreme Problem: 100 years of selection and recruitment at the Journal of Applied Psychology.

This article reviews 100 years of research on recruitment and selection published in the Journal of Applied Psychology. Recruitment and selection research has been present in the Journal from the very first issue, where Hall (1917) suggested that the challenge of recruitment and selection was the Supreme Problem facing the field of applied psychology. As this article shows, the various topics related to recruitment and selection have ebbed and flowed over the years in response to business, legal, and societal changes, but this Supreme Problem has captivated the attention of scientist-practitioners for a century. Our review starts by identifying the practical challenges and macro forces that shaped the sciences of recruitment and selection and helped to define the research questions the field has addressed. We then describe the evolution of recruitment and selection research and the ways the resulting scientific advancements have contributed to staffing practices. We conclude with speculations on how recruitment and selection research may proceed in the future. Supplemental material posted online provides additional depth by including a summary of practice challenges and scientific advancements that affected the direction of selection and recruitment research and an outline of seminal articles published in the Journal and corresponding time line. The 100-year anniversary of the Journal of Applied Psychology is very much the celebration of recruitment and selection research, although predictions about the future suggest there is still much exciting work to be done. (PsycINFO Database Record

On reconciling conflicting meta-analytic findings regarding integrity test validity.

We react to the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012a) meta-analysis of the relationship between integrity test scores and work-related criteria, the earlier Ones, Viswesvaran, and Schmidt (1993) meta-analysis of those relationships, the Harris et al. (2012) and Ones, Viswesvaran, and Schmidt (2012) responses, and the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012b) rebuttal. We highlight differences between the findings of the 2 meta-analyses by focusing on studies that used predictive designs, applicant samples, and non-self-report criteria. We conclude that study exclusion criteria, correction for artifacts, and second order sampling error are not likely explanations for the differences in findings. The lack of detailed documentation of all effect size estimates used in either meta-analysis makes it impossible to ascertain the bases for the differences in findings. We call for increased detail in meta-analytic reporting and for better information sharing among the parties producing and meta-analytically integrating validity evidence.

Impact of measurement invariance on construct correlations, mean differences, and relations with external correlates: an illustrative example using Big Five and RIASEC measures.

The impact of measurement invariance and the provision for partial invariance in confirmatory factor analytic models on factor intercorrelations, latent mean differences, and estimates of relations with external variables is investigated for measures of two sets of widely assessed constructs: Big Five personality and the six Holland interests (RIASEC). In comparing models that include provisions for partial invariance with models that do not, the results indicate quite small differences in parameter estimates involving the relations between factors, one relatively large standardized mean difference in factors between the subgroups compared and relatively small differences in the regression coefficients when the factors are used to predict external variables. The results provide support for the use of partially invariant models, but there does not seem to be a great deal of difference between structural coefficients when the measurement model does or does not include separate estimates of subgroup parameters that differ across subgroups. Future research should include simulations in which the impact of various factors related to invariance is estimated.

Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact on demographic status of admitted students.

This study was conducted to determine the validity of noncognitive and cognitive predictors of the performance of college students at the end of their 4th year in college. Results indicate that the primary predictors of cumulative college grade point average (GPA) were Scholastic Assessment Test/American College Testing Assessment (SAT/ACT) scores and high school GPA (HSGPA) though biographical data and situational judgment measures added incrementally to this prediction. SAT/ACT scores and HSGPA were collected and used in various ways by participating institutions in the admissions process while situational judgment measures and biodata were collected for research purposes only during the first few weeks of the participating students' freshman year. Alternative outcomes such as a self-report of performance on a range of student performance dimensions and a measure of organizational citizenship behavior, as well as class absenteeism, were best predicted by noncognitive measures. The racial composition of a student body selected with only cognitive measures or both cognitive and noncognitive measures under various levels of selectivity as well as the performance of students admitted under these scenarios is also reported. The authors concluded that both the biodata and situational judgment measures could be useful supplements to cognitive indexes of student potential in college admissions.

Dimensionality and psychometric invariance of the Youth Self-Report Form of the Child Behavior Checklist in cross-national settings.

The Youth Self-Report Form's (YSR's) factor model was derived from traditional exploratory factor analytical procedures. Assuming appropriate model specification, psychometrically invariant items, and that its items provide useful psychometric information across nations omitted from its normative samples, the YSR is widely used in cross-national studies of nonreferred children. Item response theory analytical procedures reveal (a) 2 dimensions partly overlapping with the YSR's Internalizing and Externalizing second-order factors; (b) variance (i.e., differential item functioning) in how well a few items discriminate for nonreferred children across two nations; and (c) variance in estimating severity levels in children with identical psychopathological severity cross-nationally. Addressing psychometric variance, limiting redundancy, and matching children's psychopathological severity levels with items measuring this severity might promote more accurate and economical assessment.

The use of background and ability profiles to predict college student outcomes.

To determine whether profiles of predictor variables provide incremental prediction of college student outcomes, the authors 1st applied an empirical clustering method to profiles based on the scores of 2,771 entering college students on a battery of biographical data and situational judgment measures, along with SAT and American College Test scores and high school grade point average, which resulted in 5 student groups. Performance of the students in these clusters was meaningfully different on a set of external variables, including college grade point average, self-rated performance, class absenteeism, organizational citizenship behavior, intent to quit their university, and satisfaction with college. The 14 variables in the profile were all significantly correlated with 1 or more of the outcome measures; however, nonlinear prediction of these outcomes on the basis of cluster membership did not add incrementally to a linear-regression-based combination of these 14 variables as predictors.

The impact of corrections for faking on the validity of noncognitive measures in selection settings.

In selection research and practice, there have been many attempts to correct scores on noncognitive measures for applicants who may have faked their responses somehow. A related approach with more impact would be identifying and removing faking applicants from consideration for employment entirely, replacing them with high-scoring alternatives. The current study demonstrates that under typical conditions found in selection, even this latter approach has minimal impact on mean performance levels. Results indicate about .1 SD change in mean performance across a range of typical correlations between a faking measure and the criterion. Where trait scores were corrected only for suspected faking, and applicants not removed or replaced, the minimal impact the authors found on mean performance was reduced even further. By comparison, the impact of selection ratio and test validity is much larger across a range of realistic levels of selection ratios and validities. If selection researchers are interested only in maximizing predicted performance or validity, the use of faking measures to correct scores or remove applicants from further employment consideration will produce minimal effects.

A dynamic multilevel model of demographic diversity and misfit effects.

In this study, the authors proposed and evaluated the linkages of a dynamic multilevel model of demographic diversity and misfit effects in a large sample of quick-service restaurants. Using a cross-level approach, the authors studied an employee's demographic misfit in relation to coworkers' demographics as a predictor of turnover risk over time. At the business-unit level, they studied changes in restaurant demographic diversity in relation to changes in profitability over time and unit turnover rates in relation to profitability. The authors also studied the impact of the match between the racial compositions of the restaurants and their communities on profitability. The results supported linkages between demographic misfit and turnover and partially supported a negative association between racial diversity and changes in profitability.

Developing a biodata measure and situational judgment inventory as predictors of college student performance.

This article describes the development and validation of a biographical data (biodata) measure and situational judgment inventory (SJI) as useful predictors of broadly defined college student performance outcomes. These measures provided incremental validity when considered in combination with standardized college-entrance tests (i.e., SAT/ACT) and a measure of Big Five personality constructs. Racial subgroup mean differences were much smaller on the biodata and SJI measures than on the standardized tests and college grade point average. Female students tended to outperform male students on most predictors and outcomes with the exception of the SAT/ACT. The biodata and SJI measures show promise for student development contexts and for selecting students on a wide range of outcomes with reduced adverse impact.

Is it prudent to administer all items for each Child Behavior Checklist cross-informant syndrome? Evaluating the psychometric properties of the Youth Self-Report dimensions with confirmatory factor analysis and item response theory.

Through surveying of children in 10 nations with parent, teacher, and Youth Self-Report (YSR) forms of the Child Behavior Checklist (CBCL), cross-informant syndromes (CISs) were derived and cross-validated by sample-dependent methodology. Generalizing CBCL syndromes and norms to nations excluded from its normative sample is problematic. This study used confirmatory factor analyses (CFAs) to test factor model fit for CISs on the YSR responses of 625 Jamaican children ages 11 to 18 years. Item response theory (IRT), a sample-independent methodology, was used to estimate the psychometric properties of individual items on each dimension. CFAs indicated poor to moderate model-to-data fit. Across all syndromes, IRT analyses revealed that more than 3/4 of the cross-informant items yielded little information. Eliminating such items could be cost effective in terms of administration time yet improve the measures discrimination across syndrome severity levels.

Impact of elaboration on socially desirable responding and the validity of biodata measures.

The current study investigated the impact of requiring respondents to elaborate on their answers to a biodata measure on mean scores, the validity of the biodata item composites, subgroup mean differences, and correlations with social desirability. Results of this study indicate that elaborated responses result in scores that are much lower than nonelaborated responses to the same items by an independent sample. Despite the lower mean score on elaborated items, it does not appear that elaboration affects the size of the correlation between social desirability and responses to biodata items or that it affects criterion-related validity or subgroup mean differences in a practically significant way.

An investigation of race and sex similarity effects in interviews: a multilevel approach to relational demography.

This research studied the effects of race and sex similarity on ratings in one-on-one highly structured college recruiting interviews (N = 708 interviewers and 12203 applicants for 7 different job families). A series of hierarchical linear models provided no evidence for similarity effects, although the commonly used D-score and analysis-of-variance-based interaction approaches conducted at the individual level of analysis yielded different results. The disparate results demonstrate the importance of attending to nested data structures and levels of analysis issues more broadly. Practically, the results suggest that organizations using carefully administered highly structured interviews may not need to be concerned about bias due to the mismatch between interviewer and applicant race or sex.

Do Reactions to Tests Produce Changes in the Construct Measured?

Applicants for state trooper positions completed two tests and three measures of test reactions as part of a study that examined their reactions to different types of measures and the impact these reactions had on the construct measured by the test. African American examinees felt a video-based measure that involved their interpretation of job-related incidents was more face valid than a reading comprehension measure based on job-related material though their perceptions of the two tests' fairness and predictive validity were similar. Caucasian examinees reacted less favorably to the video-based measure. Results also indicate that test reactions and race are related minimally to both video and reading comprehension testlets beyond their relationship to the latent factor underlying the measure. The latter finding is taken as evidence that test reactions did not affect the nature of the underlying factor measured by a test in this situation.