An external stability audit framework to test the validity of personality prediction in AI hiring.

Automated hiring systems are among the fastest-developing of all high-stakes AI systems. Among these are algorithmic personality tests that use insights from psychometric testing, and promise to surface personality traits indicative of future success based on job seekers' resumes or social media profiles. We interrogate the validity of such systems using stability of the outputs they produce, noting that reliability is a necessary, but not a sufficient, condition for validity. Crucially, rather than challenging or affirming the assumptions made in psychometric testing - that personality is a meaningful and measurable construct, and that personality traits are indicative of future success on the job - we frame our audit methodology around testing the underlying assumptions made by the vendors of the algorithmic personality tests themselves. Our main contribution is the development of a socio-technical framework for auditing the stability of algorithmic systems. This contribution is supplemented with an open-source software library that implements the technical components of the audit, and can be used to conduct similar stability audits of algorithmic systems. We instantiate our framework with the audit of two real-world personality prediction systems, namely, Humantic AI and Crystal. The application of our audit framework demonstrates that both these systems show substantial instability with respect to key facets of measurement, and hence cannot be considered valid testing instruments.

A Silicon Valley love triangle: Hiring algorithms, pseudo-science, and the quest for auditability.

In this perspective, we develop a matrix for auditing algorithmic decision-making systems (ADSs) used in the hiring domain. The tool is a socio-technical assessment of hiring ADSs that is aimed at surfacing the underlying assumptions that justify the use of an algorithmic tool and the forms of knowledge or insight they purport to produce. These underlying assumptions, it is argued, are crucial for assessing not only whether an ADS works "as intended," but also whether the intentions with which the tool was designed are well founded. Throughout, we contextualize the use of the matrix within current and proposed regulatory regimes and within emerging hiring practices that incorporate algorithmic technologies. We suggest using the matrix to expose underlying assumptions rooted in pseudo-scientific essentialized understandings of human nature and capability and to critically investigate emerging auditing standards and practices that fail to address these assumptions.