On the assessment of predictive bias in selection systems with multiple predictors.

There is a long history of examining assessments used in college admissions or personnel selection for predictive bias, also called differential prediction, to determine whether a selection system predicts comparable levels of performance for individuals from different demographic groups who have the same assessment scores. We expand on previous research that has considered predictive bias in individual predictor variables to (a) examine magnitudes of differential prediction in multipredictor selection systems and (b) explore how differences in prediction generalize across samples. We also share updated methods for computing standardized effect sizes for categorically moderated regression models that facilitate the meta-analysis of differential prediction effects. Our findings highlight the importance of analyzing composite predictors when testing for predictive bias in compensatory selection systems and demonstrate the generalizability of long-observed differential prediction trends by race/ethnicity. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

psychmeta: An R Package for Psychometric Meta-Analysis.

Over the past four decades, psychometric meta-analysis (PMA) has emerged a key way that psychological disciplines build cumulative scientific knowledge. Despite the importance and popularity of PMA, software implementing the method has tended to be closed-source, inflexible, limited in terms of the psychometric corrections available, cumbersome to use for complex analyses, and/or costly. To overcome these limitations, we created the psychmeta R package: a free, open-source, comprehensive program for PMA.

Effects of range restriction and criterion contamination on differential validity of the SAT by race/ethnicity and sex.

We illustrate the effects of range restriction and a form of criterion contamination (individual differences in course-taking patterns) on the validity of SAT scores for predicting college academic performance. College data facilitate exploration of differential validity's determinants because they (a) permit the use multivariate range-restriction corrections to more accurately account for differential range restriction across subgroups and (b) allow for separate examinations of composite performance and specific performance episodes, the latter of which controls for ecological contamination of composite performance due to individuals' choices of performance opportunities. Using data from 363,004 students at 107 U.S. institutions, we found that controlling for course-taking patterns resulted in validity coefficients that were appreciably larger than predictors' correlations with obtained grade point averages (GPAs). The validities of SAT scores for predicting the first-year college performance of Black and Hispanic students were not significantly different from the validity for White students after correcting for both course-taking patterns and differential range restriction, but significant Black-White differences were detected for predicting 4-year cumulative performance. Validity estimates for predicting both first-year and 4-year cumulative performance were significantly smaller among Asian students than White students after making these corrections. The SAT's observed validity for predicting college GPAs was substantially lower for males than females and, unexpectedly, controlling for course-taking patterns increased male-female validity differences. Implications for personnel selection research are discussed. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Predicting Performance of First-Year Residents: Correlations Between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study.

PURPOSE: To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. METHOD: The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013-2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME's core competency domains and overall. RESULTS: SI scores predicted midyear and year-end overall performance (r = 0.18 and 0.19, respectively, P < .05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies (r = 0.23, r = 0.22, and r = 0.20, respectively, P < .05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care (ΔR = 0.05), interpersonal and communication skills (ΔR = 0.09), and professionalism (ΔR = 0.09; all P < .05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. CONCLUSIONS: SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.

Constructed response formats and their effects on minority-majority differences and validity.

The inflow of immigrants challenges organizations to consider alternative selection procedures that reduce potential minority (immigrants)-majority (natives) differences, while maintaining valid predictions of performance. To deal with this challenge, this paper proposes response format as a practically and theoretically relevant factor for situational judgment tests (SJTs). We examine a range of response format categories (from traditional multiple-choice formats to more innovative constructed response formats) and conceptually link these response formats to mechanisms underlying minority-majority differences. Two field experiments are conducted with SJTs. Study 1 (274 job seekers) contrasts minority-majority differences in scores on a multiple-choice versus a written constructed response format. Written constructed responses produce much smaller minority-majority differences (d = .28 vs. d = .92). In Study 2 (269 incumbents), scores on a written constructed versus an audiovisual constructed format are compared. The audiovisual format further reduces minority-majority differences (d = .09 vs. d = .41), with validities remaining the same. Results are suggestive of cognitive load as a contributor to the reduction in minority-majority differences, as are rater effects: Scores of raters evaluating transcribed audiovisual responses, which anonymized test takers, produce larger differences. In sum, altering response modality via more realistic response formats (i.e., the audiovisual constructed format) leads to significant reductions in minority-majority differences without impairing criterion-related validity. Implications for selection theory and practice are discussed. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Changing abilities vs. changing tasks: Examining validity degradation with test scores and college performance criteria both assessed longitudinally.

We explore potential explanations for validity degradation using a unique predictive validation data set containing up to four consecutive years of high school students' cognitive test scores and four complete years of those students' college grades. This data set permits analyses that disentangle the effects of predictor-score age and timing of criterion measurements on validity degradation. We investigate the extent to which validity degradation is explained by criterion dynamism versus the limited shelf-life of ability scores. We also explore whether validity degradation is attributable to fluctuations in criterion variability over time and/or GPA contamination from individual differences in course-taking patterns. Analyses of multiyear predictor data suggest that changes to the determinants of performance over time have much stronger effects on validity degradation than does the shelf-life of cognitive test scores. The age of predictor scores had only a modest relationship with criterion-related validity when the criterion measurement occasion was held constant. Practical implications and recommendations for future research are discussed. (PsycINFO Database Record

The relationship between cognitive-ability saturation and subgroup mean differences across predictors of job performance.

The authors quantify the conventional wisdom that predictors' correlations with cognitive ability are positively related to subgroup mean differences. Using meta-analytic and large-N data from diverse predictors, they found that cognitive saturation correlates .84 with predictors' artifact-corrected Black-White d values and .95 with predictors' artifact-corrected Hispanic-White d values. The authors also investigate the extent to which d values are associated with the use of assessor-based scoring and with predictor domains in which differential investment is likely to occur. As a practical application of these findings, they present a procedure to forecast mean differences on a new predictor based on its cognitive saturation and other attributes. They also present a Bayesian framework that allows one to integrate regression-based forecasts with observed d values to achieve more precise estimates of mean differences. The proposed forecasting techniques based on the relationship between mean differences and cognitive saturation can help to mitigate the difficulties inherent in computing precise local estimates of mean differences. (PsycINFO Database Record

Effects of predictor weighting methods on incremental validity.

It is common to add an additional predictor to a selection system with the goal of increasing criterion-related validity. Research on the incremental validity of a second predictor is generally based on forming a regression-weighted composite of the predictors. However, in practice predictors are commonly used in ways other than regression-weighted composites, and we examine the robustness of incremental validity findings to other ways of using predictors, namely, unit weighting and multiple hurdles. We show that there are settings in which the incremental value of a second predictor disappears, and can even produce lower validity than the first predictor alone, when these alternatives to regression weighting are used. First, we examine conditions under which unit weighting will negate gain in predictive power attainable via regression weights. Second, we revisit Schmidt and Hunter's (1998) summary of incremental validity of predictors over cognitive ability, evaluating whether the reported incremental value of a second predictor is different when predictors are unit weighted rather than regression weighted. Third, we analyze data reported in the published literature to discern the frequency with which unit weighting might affect conclusions about whether there is value in adding a second predictor to a first. Finally, we shift from unit weighting to multiple hurdle selection, examining conditions under which conclusions about incremental validity differ when regression weighting is replaced by multiple-hurdle selection. (PsycINFO Database Record

The prosocial and aggressive driving inventory (PADI): a self-report measure of safe and unsafe driving behaviors.

Surveys of 1217 undergraduate students supported the reliability (inter-item and test-retest) and validity of the Prosocial and Aggressive Driving Inventory (PADI). Principal component analyses on the PADI items yielded two scales: Prosocial Driving (17 items) and Aggressive Driving (12 items). Prosocial Driving was associated with fewer reported traffic accidents and violations, with participants who were older and female, and with lower Boredom Susceptibility and Hostility scores, and higher scores on Agreeableness, Conscientiousness, Openness, and Neuroticism. Aggressive Driving was associated with more frequent traffic violations, with female participants, and with higher scores on Competitiveness, Sensation Seeking, Hostility, and Extraversion, and lower scores on Conscientiousness, Agreeableness, and Openness. The theoretical and practical implications of the PADI's dual focus on safe and unsafe driving are discussed.