Using natural language processing to increase prediction and reduce subgroup differences in personnel selection decisions.

The purpose of this research is to demonstrate how using natural language processing (NLP) on narrative application data can improve prediction and reduce racial subgroup differences in scores used for selection decisions compared to mental ability test scores and numeric application data. We posit there is uncaptured and job-related constructs that can be gleaned from applicant text data using NLP. We test our hypotheses in an operational context across four samples (total N = 1,828) to predict selection into Officer Training School in the U.S. Air Force. Boards of three senior officers make selection decisions using a highly structured rating process based on mental ability tests, numeric application information (e.g., number of past jobs, college grades), and narrative application information (e.g., past job duties, achievements, interests, statements of objectives). Results showed that NLP scores of the narrative application generally (a) predict Board scores when combined with test scores and numeric application information at a level of correlation equivalent to the correlation between human raters (.60), (b) add incremental prediction of Board scores beyond mental ability tests and numeric application information, and (c) reduce subgroup differences between racial minorities and nonracial minorities in Board scores compared to mental ability tests and numeric application information. Moreover, NLP scores predict (a) job (training) performance, (b) job (training) performance beyond mental ability tests and numeric application information, and (c) even job (training) performance beyond Board scores. Scoring of narrative application data using NLP shows promise in addressing the validity-adverse impact dilemma in selection. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Using practice employment tests to improve recruitment and personnel selection outcomes for organizations and job seekers.

This study introduces the use of practice employment tests during recruitment as a tool with the potential to improve outcomes for both an organization and its (potential) applicants during personnel selection. Synthesizing research on recruitment, selection, job search, adverse impact, signaling theory, and human capital theory, we propose that practice tests reduce information asymmetry regarding the nature of an organization's assessment procedures, thereby acting as short-term human capital investment opportunities. Using a large sample of potential applicants and applicants who later decided to apply for jobs within a professional occupation in a large organization, we demonstrate that (a) those who took the practice tests scored higher on the actual tests; (b) score gains between practice tests and actual tests were greater for Blacks and Hispanics when compared to Whites; (c) the practice test exhibited a self-selection effect, encouraging those with higher scores to apply; and (d) score gains between practice tests and actual tests were similar to scores observed for those retesting on the actual tests. These findings suggest practice tests may be capable of simultaneously enhancing organizational outcomes (e.g., increased quality of applicants, reduced cost of testing unqualified applicants, and reduced adverse impact) and applicant outcomes (e.g., increased human capital, increased chances of eventual employment, and reduced disappointment and wasted effort from unsuccessful application). (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Paul W. Thayer (1927-2017).

Presents an obituary for Paul W. Thayer, who died on January 25, 2017, at the age of 89. Thayer was an industrial and organizational psychologist probably most distinguished by his professional service. He was a fellow of the American Psychological Association (APA), American Psychological Society (APS), and Society for Industrial and Organizational Psychology (SIOP). He received multiple awards for his service, including SIOP's Distinguished Professional Contributions Award (1986) and its Distinguished Service Award (1990), as well as the APA Award for Distinguished Service to Psychological Science (2014). Paul's most recognized scientific contribution was in industrial training. He coauthored (with William McGehee) the first scientific book on the topic titled, Training in Business and Industry (New York, NY: Wiley), in 1961, which is considered a classic in the field. (PsycINFO Database Record

Using recruitment source timing and diagnosticity to enhance applicants' occupation-specific human capital.

[Correction Notice: An Erratum for this article was reported in Vol 102(5) of Journal of Applied Psychology (see record 2017-14296-001). In the article, the following headings were inadvertently set at the wrong level: Method, Participants and Procedure, Measures, Occupation specific human capital, Symbolic jobs, Relevant majors, Occupation-specific capital hotspots, Source timing, Source diagnosticity, Results, and Discussion. All versions of this article have been corrected.] This study proposes that reaching applicants through more diagnostic recruitment sources earlier in their educational development (e.g., in high school) can lead them to invest more in their occupation-specific human capital (OSHC), thereby making them higher quality candidates. Using a sample of 78,157 applicants applying for jobs within a desirable professional occupation in the public sector, results indicate that applicants who report hearing about the occupation earlier, and applicants who report hearing about the occupation through more diagnostic sources, have higher levels of OSHC upon application. Additionally, source timing and diagnosticity affect the likelihood of candidates applying for jobs symbolic of the occupation, selecting relevant majors, and attending educational institutions with top programs related to the occupation. These findings suggest a firm's recruiting efforts may influence applicants' OSHC investment strategies. (PsycINFO Database Record

Initial investigation into computer scoring of candidate essays for personnel selection.

[Correction Notice: An Erratum for this article was reported in Vol 101(7) of Journal of Applied Psychology (see record 2016-32115-001). In the article the affiliations for Emily D. Campion and Matthew H. Reider were originally incorrect. All versions of this article have been corrected.] Emerging advancements including the exponentially growing availability of computer-collected data and increasingly sophisticated statistical software have led to a "Big Data Movement" wherein organizations have begun attempting to use large-scale data analysis to improve their effectiveness. Yet, little is known regarding how organizations can leverage these advancements to develop more effective personnel selection procedures, especially when the data are unstructured (text-based). Drawing on literature on natural language processing, we critically examine the possibility of leveraging advances in text mining and predictive modeling computer software programs as a surrogate for human raters in a selection context. We explain how to "train" a computer program to emulate a human rater when scoring accomplishment records. We then examine the reliability of the computer's scores, provide preliminary evidence of their construct validity, demonstrate that this practice does not produce scores that disadvantage minority groups, illustrate the positive financial impact of adopting this practice in an organization (N ∼ 46,000 candidates), and discuss implementation issues. Finally, we discuss the potential implications of using computer scoring to address the adverse impact-validity dilemma. We suggest that it may provide a cost-effective means of using predictors that have comparable validity but have previously been too expensive for large-scale screening. (PsycINFO Database Record

Getting on the same page: The effect of normative feedback interventions on structured interview ratings.

This study explores normative feedback as a way to reduce rating errors and increase the reliability and validity of structured interview ratings. Based in control theory and social comparison theory, we propose a model of normative feedback interventions (NFIs) in the context of structured interviews and test our model using data from over 20,000 interviews conducted by more than 100 interviewers over a period of more than 4 years. Results indicate that lenient and severe interviewers reduced discrepancies between their ratings and the overall normative mean rating after receipt of normative feedback, though changes were greater for lenient interviewers. When various waves of feedback were presented in later NFIs, the combined normative mean rating over multiple time periods was more predictive of subsequent rating changes than the normative mean rating from the most recent time period. Mean within-interviewer rating variance, along with interrater agreement and interrater reliability, increased after the initial NFI, but results from later NFIs were more complex and revealed that feedback interventions may lose effectiveness over time. A second study using simulated data indicated that leniency and severity errors did not impact rating validity, but did affect which applicants were hired. We conclude that giving normative feedback to interviewers will aid in minimizing interviewer rating differences and enhance the reliability of structured interview ratings. We suggest that interviewer feedback might be considered as a potential new component of interview structure, though future research is needed before a definitive conclusion can be drawn. (PsycINFO Database Record

Differential validity for cognitive ability tests in employment and educational settings: not much more than range restriction?

The concept of differential validity suggests that cognitive ability tests are associated with varying levels of validity across ethnic groups, such that validity is lower in certain ethnic subgroups than in others. A recent meta-analysis has revived the viability of this concept. Unfortunately, data were not available in this meta-analysis to correct for range restriction within ethnic groups. We reviewed the differential validity literature and conducted 4 studies. In Study 1, we empirically demonstrated that using a cognitive ability test with a common cutoff decreases variance in test scores of Black subgroup samples more than in White samples. In Study 2, we developed a simulation that examined the effects of range restriction on estimates of differential validity. Results demonstrated that different levels of range restriction for subgroups can explain the apparent observed differential validity results in employment and educational settings (but not military settings) when no differential validity exists in the population. In Study 3, we conducted a simulation in which we examined how one corrects for range restriction affects the accuracy of these corrections. Results suggest that the correction approach using a common range restriction ratio for various subgroups may create or perpetuate the illusion of differential validity and that corrections are most accurate when done within each subgroup. Finally, in Study 4, we conducted a simulation in which we assumed differential validity in the population. We found that range restriction artificially increased the size of observed differential validity estimates when the validity of cognitive ability tests was assumed to be higher among Whites. Overall, we suggest that the concept of differential validity may be largely artifactual and current data are not definitive enough to suggest such effects exist.

Do candidate reactions relate to job performance or affect criterion-related validity? A multistudy investigation of relations among reactions, selection test scores, and job performance.

Considerable evidence suggests that how candidates react to selection procedures can affect their test performance and their attitudes toward the hiring organization (e.g., recommending the firm to others). However, very few studies of candidate reactions have examined one of the outcomes organizations care most about: job performance. We attempt to address this gap by developing and testing a conceptual framework that delineates whether and how candidate reactions might influence job performance. We accomplish this objective using data from 4 studies (total N = 6,480), 6 selection procedures (personality tests, job knowledge tests, cognitive ability tests, work samples, situational judgment tests, and a selection inventory), 5 key candidate reactions (anxiety, motivation, belief in tests, self-efficacy, and procedural justice), 2 contexts (industry and education), 3 continents (North America, South America, and Europe), 2 study designs (predictive and concurrent), and 4 occupational areas (medical, sales, customer service, and technological). Consistent with previous research, candidate reactions were related to test scores, and test scores were related to job performance. Further, there was some evidence that reactions affected performance indirectly through their influence on test scores. Finally, in no cases did candidate reactions affect the prediction of job performance by increasing or decreasing the criterion-related validity of test scores. Implications of these findings and avenues for future research are discussed.

Can I retake it? Exploring subgroup differences and criterion-related validity in promotion retesting.

Despite recent interest in the practice of allowing job applicants to retest, surprisingly little is known about how retesting affects 2 of the most critical factors on which staffing procedures are evaluated: subgroup differences and criterion-related validity. We examined these important issues in a sample of internal candidates who completed a job-knowledge test for a within-job promotion. This was a useful context for these questions because we had job-performance data on all candidates (N = 403), regardless of whether they passed or failed the promotion test (i.e., there was no direct range restriction). We found that retest effects varied by subgroup, such that females and younger candidates improved more upon retesting than did males and older candidates. There also was some evidence that Black candidates did not improve as much as did candidates from other racial groups. In addition, among candidates who retested, their retest scores were somewhat better predictors of subsequent job performance than were their initial test scores (rs = .38 vs. .27). The overall results suggest that retesting does not negatively affect criterion-related validity and may even enhance it. Furthermore, retesting may reduce the likelihood of adverse impact against some subgroups (e.g., female candidates) but increase the likelihood of adverse impact against other subgroups (e.g., older candidates).

If at first you don't succeed, try, try again: understanding race, age, and gender differences in retesting score improvement.

This article explores the intersection of 2 critical and timely concerns in personnel selection-applicant retesting and subgroup differences-by exploring demographic differences in retest effects across multiple assessments. Results from large samples of applicants taking 3 written tests (N = 7,031) and 5 performance tests (N = 2,060) revealed that Whites showed larger retest score improvements than Blacks or Hispanics on several of the assessments. However, the differential improvement of Whites was greater on the written tests than on the performance tests. In addition, women and applicants under 40 years of age showed larger improvements with retesting than did men and applicants over 40. We offer some preliminary theoretical explanations for these demographic differences in retesting gains, including differences in ability, testing attitudes and motivation, and receptivity to feedback. In terms of practical implications, the results suggest that allowing applicants to retake selection tests may, in some cases, exacerbate levels of adverse impact, which can have distinct implications for retesting policy and practices in organizations.

Human resource configurations: investigating fit with the organizational context.

The present study investigated how key organizational contextual factors relate to bundles of human resource (HR) practices. In a two-phase study of a sample of 661 organizations representing a full range of industries and organizational size, the authors found that organizations use 1 of 5 HR bundles: cost minimizers, contingent motivators, competitive motivators, resource makers, and commitment maximizers. In addition, the authors showed that the organizations that use a given type of HR bundle may be distinguished by the organizational values they pursue and their organizational structure, thus suggesting that HR choices are related to the context within which organizations operate.

The team role test: development and validation of a team role knowledge situational judgment test.

The main objectives in this research were to introduce the concept of team role knowledge and to investigate its potential usefulness for team member selection. In Study 1, the authors developed a situational judgment test, called the Team Role Test, to measure knowledge of 10 roles relevant to the team context. The criterion-related validity of this measure was examined in 2 additional studies. In a sample of academic project teams (N = 93), team role knowledge predicted team member role performance (r = .34). Role knowledge also provided incremental validity beyond mental ability and the Big Five personality factors in the prediction of role performance. The results of Study 2 revealed that the predictive validity of role knowledge generalizes to team members in a work setting (N = 82, r = .30). The implications of the results for selection in team environments are discussed.

Measuring faking in the employment interview: development and validation of an interview faking behavior scale.

An Interview Faking Behavior (IFB) scale is developed and validated in 6 studies (N = 1,346). In Study 1, a taxonomy of faking behavior is delineated. The factor structure of a measure is evaluated and refined (Studies 2 and 3). The convergent and discriminant validity of the measure is examined (Study 4). The IFB scale consists of 4 factors (Slight Image Creation, Extensive Image Creation, Image Protection, and Ingratiation) and 11 subfactors (Embellishing, Tailoring, Fit Enhancing, Constructing, Inventing, Borrowing, Masking, Distancing, Omitting, Conforming, and Interviewer Enhancing). A study of actual interviews shows that scores on the IFB scale are related to getting a 2nd interview or a job offer (Study 5). In Study 6, an experiment is conducted to test the usefulness of the new measure for studying methods of reducing faking using structured interviews. It is found that past behavior questions are more resistant to faking than situational questions, and follow-up questioning increases faking. Finally, over 90% of undergraduate job candidates fake during employment interviews; however, fewer candidates engage in faking that is semantically closer to lying, ranging from 28% to 75%.

Self-presentation processes in job analysis: a field experiment investigating inflation in abilities, tasks, and competencies.

Although job analysis is a widely used organizational data collection technique, little research has investigated the extent to which job analysis information is affected by self-presentation processes. This study represents the first direct test of the propositions offered by F. P. Morgeson and M. A. Campion (1997) concerning self-presentation in job analysis measurement. Using an experimental design, the authors examined job incumbent response differences across ability, task, and competency statements. Results indicated that ability statements were more subject to inflation than were task statements across all rating scales. Greater endorsement of nonessential ability statements was responsible for the differences. This produced higher endorsement of ability items but lower mean ratings. Finally, frequency and importance ratings of global competency statements were generally higher than decomposed ability and task scales, but required-at-entry judgments demonstrated the opposite relationship.

Selection fairness information and applicant reactions: a longitudinal field study.

Although laboratory studies have found that selection information can affect applicant perceptions, this has not been tested in the field. The authors followed 2 cohorts of police applicants (N = 274) in a longitudinal study to examine the relationship between information, applicant perceptions, and behavior (e.g., turnover). Information was related to perceived fairness measured at the time of testing and 1 month later when applicants received their results. Information moderated the relationship between outcome favorability and test-taking self-efficacy among African Americans but not among Whites. Information was not related to the behavioral measures. The discussion focuses on why certain findings from previous studies were not replicated and on the use of information when applicants have an investment in getting a job.