A tutorial on Bayesian single-test reliability analysis with JASP.

The current practice of reliability analysis is both uniform and troublesome: most reports consider only Cronbach's α, and almost all reports focus exclusively on a point estimate, disregarding the impact of sampling error. In an attempt to improve the status quo we have implemented Bayesian estimation routines for five popular single-test reliability coefficients in the open-source statistical software program JASP. Using JASP, researchers can easily obtain Bayesian credible intervals to indicate a range of plausible values and thereby quantify the precision of the point estimate. In addition, researchers may use the posterior distribution of the reliability coefficients to address practically relevant questions such as "What is the probability that the reliability of my test is larger than a threshold value of .80?". In this tutorial article, we outline how to conduct a Bayesian reliability analysis in JASP and correctly interpret the results. By making available a computationally complex procedure in an easy-to-use software package, we hope to motivate researchers to include uncertainty estimates whenever reporting the results of a single-test reliability analysis.

Bayesian Estimation of Single-Test Reliability Coefficients.

Popular measures of reliability for a single-test administration include coefficient α, coefficient λ2, the greatest lower bound (glb), and coefficient ω. First, we show how these measures can be easily estimated within a Bayesian framework. Specifically, the posterior distribution for these measures can be obtained through Gibbs sampling - for coefficients α, λ2, and the glb one can sample the covariance matrix from an inverse Wishart distribution; for coefficient ω one samples the conditional posterior distributions from a single-factor CFA-model. Simulations show that - under relatively uninformative priors - the 95% Bayesian credible intervals are highly similar to the 95% frequentist bootstrap confidence intervals. In addition, the posterior distribution can be used to address practically relevant questions, such as "what is the probability that the reliability of this test is between .70 and .90?", or, "how likely is it that the reliability of this test is higher than .80?" In general, the use of a posterior distribution highlights the inherent uncertainty with respect to the estimation of reliability measures.

Rejoinder: The Future of Reliability.

In this rejoinder, we examine some of the issues Peter Bentler, Eunseong Cho, and Jules Ellis raise. We suggest a methodological solid way to construct a test indicating that the importance of the particular reliability method used is minor, and we discuss future topics in reliability research.

Part II: On the Use, the Misuse, and the Very Limited Usefulness of Cronbach's Alpha: Discussing Lower Bounds and Correlated Errors.

Prior to discussing and challenging two criticisms on coefficient [Formula: see text], the well-known lower bound to test-score reliability, we discuss classical test theory and the theory of coefficient [Formula: see text]. The first criticism expressed in the psychometrics literature is that coefficient [Formula: see text] is only useful when the model of essential [Formula: see text]-equivalence is consistent with the item-score data. Because this model is highly restrictive, coefficient [Formula: see text] is smaller than test-score reliability and one should not use it. We argue that lower bounds are useful when they assess product quality features, such as a test-score's reliability. The second criticism expressed is that coefficient [Formula: see text] incorrectly ignores correlated errors. If correlated errors would enter the computation of coefficient [Formula: see text], theoretical values of coefficient [Formula: see text] could be greater than the test-score reliability. Because quality measures that are systematically too high are undesirable, critics dismiss coefficient [Formula: see text]. We argue that introducing correlated errors is inconsistent with the derivation of the lower bound theorem and that the properties of coefficient [Formula: see text] remain intact when data contain correlated errors.