History and development of the Schmidt-Hunter meta-analysis methods.

In this article, I provide answers to the questions posed by Will Shadish about the history and development of the Schmidt-Hunter methods of meta-analysis. In the 1970s, I headed a research program on personnel selection at the US Office of Personnel Management (OPM). After our research showed that validity studies have low statistical power, OPM felt a need for a better way to demonstrate test validity, especially in light of court cases challenging selection methods. In response, we created our method of meta-analysis (initially called validity generalization). Results showed that most of the variability of validity estimates from study to study was because of sampling error and other research artifacts such as variations in range restriction and measurement error. Corrections for these artifacts in our research and in replications by others showed that the predictive validity of most tests was high and generalizable. This conclusion challenged long-standing beliefs and so provoked resistance, which over time was overcome. The 1982 book that we published extending these methods to research areas beyond personnel selection was positively received and was followed by expanded books in 1990, 2004, and 2014. Today, these methods are being applied in a wide variety of areas.

A General Theoretical Integrative Model of Individual Differences in Interests, Abilities, Personality Traits, and Academic and Occupational Achievement: A Commentary on Four Recent Articles.

This commentary integrates the contents of four recent articles on individual differences (Nye, Su, Rounds, & Drasgow, 2012; Schmidt, 2011; Valla & Ceci, 2011; von Stumm, Hell, & Chamorro-Premuzic, 2011) in a causal theoretical model. In this model, introversion and fluid intelligence cause interest in general learning (intellectual curiosity), which in turn is a major cause of crystallized intelligence. Certain specific interests and fluid intelligence also contribute to crystallized intelligence. Prenatal testosterone hormone conditioning is postulated to cause sex differences in certain specific interests but not in others. Crystallized intelligence, specific interests, and the personality trait of conscientiousness cause adult academic and occupational performance, whereas crystallized intelligence is the main cause of good mental functioning at older ages. Research is presented supporting each link in the model.

Integrity tests predict counterproductive work behaviors and job performance well: comment on Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012).

Examination of the Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012) meta-analysis reveals a number of problems. They meta-analyzed a partial database of integrity test validities. An examination of their coded database revealed that measures coded as integrity tests and meta-analyzed as such often included scales that are not in fact integrity tests. In addition, there were important deficiencies in their analytic approach relating to application of range restriction corrections and identification of moderators. We found the absence of fully hierarchical moderator analyses to be a serious weakness. We also explain why empirical comparisons between test publishers versus non-publishers cannot unambiguously lead to inferences of bias, as alternate explanations are possible, even likely. In light of the problems identified, it appears that the conclusions about integrity test validity drawn by Van Iddekinge et al. cannot be considered accurate or reliable.

A Theory of Sex Differences in Technical Aptitude and Some Supporting Evidence.

In this article, I present a theory that explains the origin of sex differences in technical aptitudes. The theory takes as proven that there are no sex differences in general mental ability (GMA), and it postulates that sex differences in technical aptitude (TA) stem from differences in experience in technical areas, which is in turn based on sex differences in technical interests. Using a large data set, I tested and found support for four predictions made by this theory: (a) the construct level correlation between technical aptitude and GMA is larger for females than males, (b) the observed and true score variability of technical aptitude is greater among males than females, (c) at every level of GMA females have lower levels of technical aptitude, and (d) technical aptitude measures used as estimates of GMA for decision purposes would result in underestimation of GMA levels for girls and women. Given that GMA carries the weight of prediction of job performance, the support found for this last prediction suggests that, for many jobs, technical aptitude tests may underpredict the job performance of female applicants and employees. Future research should examine this question.

The impact of research synthesis methods on industrial-organizational psychology: The road from pessimism to optimism about cumulative knowledge.

This paper presents an account of the impact that research synthesis methods, in the form of psychometric meta-analysis, has had on industrial/organizational (I/O) psychology. This paper outlines the central contributions of psychometric meta-analysis in providing a method for developing cumulative knowledge. First, this paper describes the concerns and the state of the field before the development of meta-analytic methods. Second, the paper explains how meta-analysis addressed these concerns. Third, the paper details the development of psychometric meta-analysis through VG research and describes how the use of psychometric meta-analysis spread to other topic areas in the field. Finally, the paper presents illustrative example literatures, such as training and leadership, where meta-analysis had crucial impacts. Copyright © 2011 John Wiley & Sons, Ltd.

Causal Impact of Employee Work Perceptions on the Bottom Line of Organizations.

Perceptions of work conditions have proven to be important to the well-being of workers. However, customer loyalty, employee retention, revenue, sales, and profit are essential to the success of any business. It is known that these outcomes are correlated with employee attitudes and perceptions of work conditions, but the research into direction of causality has been inconclusive. Using a massive longitudinal database that included 2,178 business units in 10 large organizations, we found evidence supporting the causal impact of employee perceptions on these bottom-line measures; reverse causality of bottom-line measures on employee perceptions existed but was weaker. Managerial actions and practices can impact employee work conditions and employee perceptions of these conditions, thereby improving key outcomes at the organizational level. Perceptions of specific work conditions that engage employees in their work provide practical guidance in how best to manage people to obtain desired results.

Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results.

Today most conclusions about cumulative knowledge in psychology are based on meta-analysis. We first present an examination of the important statistical differences between fixed-effects (FE) and random-effects (RE) models in meta-analysis and between two different RE procedures, due to Hedges and Vevea, and to Hunter and Schmidt. The implications of these differences for the appropriate interpretation of published meta-analyses are explored by applying the two RE procedures to 68 meta-analyses from five large meta-analytic studies previously published in Psychological Bulletin. Under the assumption that the goal of research is generalizable knowledge, results indicated that the published FE confidence intervals (CIs) around mean effect sizes were on average 52% narrower than their actual width, with similar results being produced by the two RE procedures. These nominal 95% FE CIs were found to be on average 56% CIs. Because most meta-analyses in the literature use FE models, these findings suggest that the precision of meta-analysis findings in the literature has often been substantially overstated, with important consequences for research and practice.

Updating meta-analytic research findings: Bayesian approaches versus the medical model.

The authors examine 3 methods of combining new studies into existing meta-analyses: (a) adding the new study or studies to the database and recalculating the meta-analysis (the medical model); (b) using the Bayesian procedure advocated by F. L. Schmidt and J. E. Hunter (1977) and F. L. Schmidt, J. E. Hunter, K. Pearlman, and G. S. Shane (1979) to update the meta-analysis; and (c) using the Bayesian methods advocated by these authors and M. T. Brannick (2001) and M. T. Brannick, S. M. Hall, and Y. Liu (2002) to estimate study-specific parameters. Method b was found to severely overweight new studies relative to the previous studies contained in the meta-analysis, and Method c was found to do the same while also requiring an assumption with a low prior probability of being correct, causing the method to violate Bayesian principles. The authors present an alternative Bayesian procedure that does not suffer from these drawbacks and yields meta-analytic results very similar to those obtained with the medical model. They recommend use of the medical model or this alternative Bayesian procedure.

Correcting for indirect range restriction in meta-analysis: testing a new meta-analytic procedure.

Using computer simulation, the authors assessed the accuracy of J. E. Hunter, F. L. Schmidt, and H. Le's (2006) procedure for correcting for indirect range restriction, the most common type of range restriction, in comparison with the conventional practice of applying the Thorndike Case II correction for direct range restriction. Hunter et al.'s procedure produced more accurate estimates of both the mean and standard deviation in meta-analysis than the conventional procedure. Even when its key assumption that the effect of selection on a 3rd variable is fully mediated by the independent variable was violated, Hunter et al.'s procedure was still relatively more accurate than the conventional procedure. When applied to data from a previously published meta-analysis, the new procedure yielded results that led to different substantive conclusions.

Implications of direct and indirect range restriction for meta-analysis methods and findings.

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.

Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences.

A database integrating 90 years of empirical studies reporting intercorrelations among rated job performance dimensions was used to test the hypothesis of a general factor in job performance. After controlling for halo error and 3 other sources of measurement error, there remained a general factor in job performance ratings at the construct level accounting for 60% of total variance. Construct-level correlations among rated dimensions of job performance were substantially inflated by halo for both supervisory (33%) and peer (63%) intrarater correlations. These findings have important implications for the measurement of job performance and for theories of job performance.

A counterintuitive hypothesis about employment interview validity and some supporting evidence.

This study found mixed support for the hypothesis that the difference in criterion-related validity between unstructured and structured employment interviews is due solely to the greater reliability of structured interviews. Using data from prior meta-analyses, this hypothesis was tested in 4 data sets by using standard psychometric procedures to remove the effects of measurement error in interview scores from correlations with rated job performance and training performance. In the 1st data set. support was found for this hypothesis. However, in a 2nd data set structured interviews had higher true score correlations with performance ratings, and in 2 other data sets unstructured interviews had higher true score correlations. We also found that averaging across 3 to 4 independent unstructured interviews provides the same level of validity for predicting job performance as a structured interview administered by a single interviewer. Practical and theoretical implications are discussed.

General mental ability in the world of work: occupational attainment and job performance.

The psychological construct of general mental ability (GMA), introduced by C. Spearman (1904) nearly 100 years ago, has enjoyed a resurgence of interest and attention in recent decades. This article presents the research evidence that GMA predicts both occupational level attained and performance within one's chosen occupation and does so better than any other ability, trait, or disposition and better than job experience. The sizes of these relationships with GMA are also larger than most found in psychological research. Evidence is presented that weighted combinations of specific aptitudes tailored to individual jobs do not predict job performance better than GMA alone, disconfirming specific aptitude theory. A theory of job performance is described that explicates the central role of GMA in the world of work. These findings support Spearman's proposition that GMA is of critical importance in human affairs.

Beyond alpha: an empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs.

On the basis of an empirical study of measures of constructs from the cognitive domain, the personality domain, and the domain of affective traits, the authors of this study examine the implications of transient measurement error for the measurement of frequently studied individual differences variables. The authors clarify relevant reliability concepts as they relate to transient error and present a procedure for estimating the coefficient of equivalence and stability (L. J. Cronbach, 1947), the only classical reliability coefficient that assesses all 3 major sources of measurement error (random response, transient, and specific factor errors). The authors conclude that transient error exists in all 3 trait domains and is especially large in the domain of affective traits. Their findings indicate that the nearly universal use of the coefficient of equivalence (Cronbach's alpha; L. J. Cronbach, 1951), which fails to assess transient error, leads to overestimates of reliability and undercorrections for biases due to measurement error.

Business-unit-level relationship between employee satisfaction, employee engagement, and business outcomes: a meta-analysis.

Based on 7,939 business units in 36 companies, this study used meta-analysis to examine the relationship at the business-unit level between employee satisfaction-engagement and the business-unit outcomes of customer satisfaction, productivity, profit, employee turnover, and accidents. Generalizable relationships large enough to have substantial practical value were found between unit-level employee satisfaction-engagement and these business-unit outcomes. One implication is that changes in management practices that increase employee satisfaction may increase business-unit outcomes, including profit.

The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed.