Detecting Examinees With Item Preknowledge on Real Data.

Recently, Belov & Wollack (2021) developed a method for detecting groups of colluding examinees as cliques in a graph. The objective of this article is to study how the performance of their method on real data with item preknowledge (IP) depends on the mechanism of edge formation governed by a response similarity index (RSI). This study resulted in the development of three new RSIs and demonstrated a remarkable advantage of combining responses and response times for detecting examinees with IP. Possible extensions of this study and recommendations for practitioners were formulated.

Graph Theory Approach to Detect Examinees Involved in Test Collusion.

Test collusion (TC) is sharing of test materials or answers to test questions before or during the test (important special case of TC is item preknowledge). Because of potentially large advantages for examinees involved, TC poses a serious threat to the validity of score interpretations. The proposed approach applies graph theory methodology to response similarity analyses for identifying groups of examinees involved in TC without using any knowledge about parts of test that were affected by TC. The approach supports different response similarity indices (specific to a particular type of TC) and different types of groups (connected components, cliques, or near-cliques). A comparison with an up-to-date method using real and simulated data is presented. Possible extensions and practical recommendations are given.

On the Optimality of the Detection of Examinees With Aberrant Answer Changes.

In standardized multiple-choice testing, examinees may change their answers for various reasons. The statistical analysis of answer changes (ACs) has uncovered multiple testing irregularities on large-scale assessments and is now routinely performed at many testing organizations. This article exploits a recent approach where the information about all previous answers is used only to partition administered items into two disjoint subtests: items where an AC occurred and items where an AC did not occur. Two optimal statistics are described, each measuring a difference in performance between these subtests, where the performance is estimated from the final responses. Answer-changing behavior was simulated, where realistic distributions of wrong-to-right, wrong-to-wrong, and right-to-wrong ACs were achieved under various conditions controlled by the following independent variables: type of test, amount of aberrancy, and amount of uncertainty. Results of computer simulations confirmed the theoretical constructs on the optimal power of both statistics and provided several recommendations for practitioners.

Comparing the Performance of Eight Item Preknowledge Detection Statistics.

Item preknowledge describes a situation in which a group of examinees (called aberrant examinees) have had access to some items (called compromised items) from an administered test prior to the exam. Item preknowledge negatively affects both the corresponding testing program and its users (e.g., universities, companies, government organizations) because scores for aberrant examinees are invalid. In general, item preknowledge is hard to detect due to multiple unknowns: unknown groups of aberrant examinees (at unknown test centers or schools) accessing unknown subsets of items prior to the exam. Recently, multiple statistical methods were developed to detect compromised items. However, the detected subset of items (called the suspicious subset) naturally has an uncertainty due to false positives and false negatives. The uncertainty increases when different groups of aberrant examinees had access to different subsets of items; thus, compromised items for one group are uncompromised for another group and vice versa. The impact of uncertainty on the performance of eight statistics (each relying on the suspicious subset) was studied. The measure of performance was based on the receiver operating characteristic curve. Computer simulations demonstrated how uncertainty combined with various independent variables (e.g., type of test, distribution of aberrant examinees) affected the performance of each statistic.

Distributions of the Kullback-Leibler divergence with applications.

The Kullback-Leibler divergence (KLD) is a widely used method for measuring the fit of two distributions. In general, the distribution of the KLD is unknown. Under reasonable assumptions, common in psychometrics, the distribution of the KLD is shown to be asymptotically distributed as a scaled (non-central) chi-square with one degree of freedom or a scaled (doubly non-central) F. Applications of the KLD for detecting heterogeneous response data are discussed with particular emphasis on test security.