Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators.

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables-text reread, item revisit, and answer change-were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior.

Rapid guessing (RG) is a form of non-effortful responding that is characterized by short response latencies. This construct-irrelevant behavior has been shown in previous research to bias inferences concerning measurement properties and scores. To mitigate these deleterious effects, a number of response time threshold scoring procedures have been proposed, which recode RG responses (e.g., treat them as incorrect or missing, or impute probable values) and then estimate parameters for the recoded dataset using a unidimensional or multidimensional IRT model. To date, there have been limited attempts to compare these methods under the possibility that RG may be misclassified in practice. To address this shortcoming, the present simulation study compared item and ability parameter recovery for four scoring procedures by manipulating sample size, the linear relationship between RG propensity and ability, the percentage of RG responses, and the type and rate of RG misclassifications. Results demonstrated two general trends. First, across all conditions, treating RG responses as incorrect produced the largest degree of combined systematic and random error (larger than ignoring RG). Second, the remaining scoring approaches generally provided equal accuracy in parameter recovery when RG was perfectly identified; however, the multidimensional IRT approach was susceptible to increased error as misclassification rates grew. Overall, the findings suggest that recoding RG as missing and employing a unidimensional IRT model is a promising approach.

A Mixture Fluency model using responses and response times with cognitive diagnosis model framework.

Random guessing behaviors are frequently observed in low-stakes assessments, often attributed to factors such as test-takers lacking motivation or experiencing time constraints and fatigue. Existing research suggests that responses stemming from random guessing behaviors introduce biases into the constructs and relationships of interest. This is particularly problematic when estimating the relationship between speed and ability. This study introduces a Mixture Fluency model designed to account for random guessing behaviors while utilizing valid response accuracy and response time to uncover students' latent attribute profiles. The model directly addresses a limitation present in the Fluency cognitive diagnostic model (Wang & Chen, Psychometrika, 85, 600-629, (2020), which assumes that test-takers consistently employ solution behaviors when answering questions. To investigate the effectiveness of the proposed Mixture Fluency model, we conducted a simulation study encompassing various simulation conditions. Results from this study not only confirm the model's ability to detect potential random guessing behaviors but also demonstrate its capacity to enhance the inference of targeted latent constructs within the assessment. Additionally, we showcase the practical utility of the proposed model through an application to real data.

An Iterative Scale Purification Procedure on lz for the Detection of Aberrant Responses.

Many person-fit statistics have been proposed to detect aberrant response behaviors (e.g., cheating, guessing). Among them, lz is one of the most widely used indices. The computation of lz assumes the item and person parameters are known. In reality, they often have to be estimated from data. The better the estimation, the better lz will perform. When aberrant behaviors occur, the person and item parameter estimations are inaccurate, which in turn degrade the performance of lz. In this study, an iterative procedure was developed to attain more accurate person parameter estimates for improved performance of lz. A series of simulations were conducted to evaluate the iterative procedure under two conditions of item parameters, known and unknown, and three aberrant response styles of difficulty-sharing cheating, random-sharing cheating, and random guessing. The results demonstrated the superiority of the iterative procedure over the non-iterative one in maintaining control of Type-I error rates and improving the power of detecting aberrant responses. The proposed procedure was applied to a high-stake intelligence test.

Contrasting multistage and computer-based testing: score accuracy and aberrant responding.

The goal of the present study was to compare and contrast the efficacy of a multistage testing (MST) design using three paths compared to a traditional computer-based testing (CBT) approach involving items across all ability levels. Participants were n = 627 individuals who were subjected to both a computer-based testing (CBT) instrument and a measure constructed using multistage testing to route individuals of low, middle, and high ability to content that was respective to their ability level. Comparisons between the medium of testing involved person ability accuracy estimates and evaluation of aberrant responding. The results indicated that MST assessments deviated markedly from CBT assessments, especially for low- and high-ability individuals. Test score accuracy was higher overall in MST compared to CBT, although error of measurement was enhanced for high-ability individuals during MST compared to CBT. Evaluating response patterns indicated significant amounts of Guttman-related errors during CBT compared to MST using person-fit aberrant response indicators. It was concluded that MST is associated with significant benefits compared to CBT.

Electroencephalographic oscillations of alpha and beta rhythms during phrase-guessing procedure.

Phrases-guessing is one of the essential reasoning abilities in problem solving for human beings. However, it is still an open question about why individuals perform differently during the same reasoning task. In this study, we utilized a bilingual phrase-guessing task to explore the neural activities under the individually different performances with electroencephalography. Participants who had no knowledge of Greek were required to guess the meaning of a Greek phrase (long or short in length) by making an either-or selection as to which translation-equivalent Chinese word corresponds to Greek word. Names of color were used as experimental stimuli for which two Chinese words denoted the same color with one as a conventional color name and the other as a novel color name. The experiment yielded length of phrases (long vs. short) and novelty of phrases (novel vs. conventional) as variables. The behavioral results revealed significant length-by-novelty interaction on the number of selections. However, neither main effects nor interactive effects were found on response time. Further, the amplitude spectrums of high alpha rhythm, low alpha rhythm, and low beta rhythm during the task were positively associated with the participants' number of selections for a long Greek phrase with a novel and complex Chinese phrase (LNc) and a short Greek phrase with a conventional Chinese phrase (SCo), while negatively correlated with the response time of selections for LNc and SCo. Our findings suggested that the consistency between participants' behavior and electrophysiological oscillations (alpha and beta bands) could be employed as biomarkers for decoding the phrase-guessing procedure.

A Systematic Review on Password Guessing Tasks.

Recently, many password guessing algorithms have been proposed, seriously threatening cyber security. In this paper, we systematically review over thirty methods for password guessing published between 2016 and 2023. First, we introduce a taxonomy for classifying the existing methods into trawling guessing and targeted guessing. Second, we present an extensive benchmark dataset that can assist researchers and practitioners in successive works. Third, we conduct a bibliometric analysis to present trends in this field and cross-citation between reviewed papers. Further, we discuss the open challenges of password guessing in terms of diverse application scenarios, guessing efficiency, and the combination of traditional and deep learning methods. Finally, this review presents future research directions to guide successive research and development of password guessing.

The Interplay between Error, Total Variation, Alpha-Entropy and Guessing: Fano and Pinsker Direct and Reverse Inequalities.

Using majorization theory via "Robin Hood" elementary operations, optimal lower and upper bounds are derived on Rényi and guessing entropies with respect to either error probability (yielding reverse-Fano and Fano inequalities) or total variation distance to the uniform (yielding reverse-Pinsker and Pinsker inequalities). This gives a general picture of how the notion of randomness can be measured in many areas of computer science.

Dissecting knowledge, guessing, and blunder in multiple choice assessments.

Multiple choice results are inherently probabilistic outcomes, as correct responses reflect a combination of knowledge and guessing, while incorrect responses additionally reflect blunder, a confidently committed mistake. To objectively resolve knowledge from responses in an MC test structure, we evaluated probabilistic models that explicitly account for guessing, knowledge and blunder using eight assessments (>9,000 responses) from an undergraduate biotechnology curriculum. A Bayesian implementation of the models, aimed at assessing their robustness to prior beliefs in examinee knowledge, showed that explicit estimators of knowledge are markedly sensitive to prior beliefs with scores as sole input. To overcome this limitation, we examined self-ranked confidence as a proxy knowledge indicator. For our test set, three levels of confidence resolved test performance. Responses rated as least confident were correct more frequently than expected from random selection, reflecting partial knowledge, but were balanced by blunder among the most confident responses. By translating evidence-based guessing and blunder rates to pass marks that statistically qualify a desired level of examinee knowledge, our approach finds practical utility in test analysis and design.

Changes in the Speed-Ability Relation Through Different Treatments of Rapid Guessing.

As researchers in the social sciences, we are often interested in studying not directly observable constructs through assessments and questionnaires. But even in a well-designed and well-implemented study, rapid-guessing behavior may occur. Under rapid-guessing behavior, a task is skimmed shortly but not read and engaged with in-depth. Hence, a response given under rapid-guessing behavior does bias constructs and relations of interest. Bias also appears reasonable for latent speed estimates obtained under rapid-guessing behavior, as well as the identified relation between speed and ability. This bias seems especially problematic considering that the relation between speed and ability has been shown to be able to improve precision in ability estimation. For this reason, we investigate if and how responses and response times obtained under rapid-guessing behavior affect the identified speed-ability relation and the precision of ability estimates in a joint model of speed and ability. Therefore, the study presents an empirical application that highlights a specific methodological problem resulting from rapid-guessing behavior. Here, we could show that different (non-)treatments of rapid guessing can lead to different conclusions about the underlying speed-ability relation. Furthermore, different rapid-guessing treatments led to wildly different conclusions about gains in precision through joint modeling. The results show the importance of taking rapid guessing into account when the psychometric use of response times is of interest.

Parents' knowledge and predictions about the age of menarche: experimental evidence from Honduras.

BACKGROUND: Access to accurate, timely and age-appropriate information about menarche is an essential part of menstrual health. Reliable evidence shows that girls primarily obtain information from their mothers and/or other female family members, therefore, it is important to determine parents' knowledge and their predictions about other parents' knowledge of the age of menarche. METHODS: To this end, we performed a pre-registered study with data collected from 360 households in Santa Rosa de Copán, Honduras. We implemented a novel procedure to avoid social desirability bias whereby participants answered two separated questions: i) their knowledge about the age of menarche (self-report) and ii) to predict or guess the modal response of the other participants regarding the same question (modal guess). Participants were paid according to accuracy. Both questions appeared randomly in the survey. RESULTS: Recent studies indicate the age of menarche at 12 years old and 56.11% of the sample gave the same response while 62.78% hit the modal value. We estimated the impact of different sociodemographic variables and found only marginal differences. Interestingly, people with formal education and women tend to respond with lower predictions. CONCLUSION: Parents' knowledge about the age of menarche is high in the study area. The study also found that there was no social desirability bias.

Modeling Rapid Guessing Behaviors in Computer-Based Testlet Items.

In traditional test models, test items are independent, and test-takers slowly and thoughtfully respond to each test item. However, some test items have a common stimulus (dependent test items in a testlet), and sometimes test-takers lack motivation, knowledge, or time (speededness), so they perform rapid guessing (RG). Ignoring the dependence in responses to testlet items can negatively bias standard errors of measurement, and ignoring RG by fitting a simpler item response theory (IRT) model can bias the results. Because computer-based testing captures response times on testlet responses, we propose a mixture testlet IRT model with item responses and response time to model RG behaviors in computer-based testlet items. Two simulation studies with Markov chain Monte Carlo estimation using the JAGS program showed (a) good recovery of the item and person parameters in this new model and (b) the harmful consequences of ignoring RG (biased parameter estimates: overestimated item difficulties, underestimated time intensities, underestimated respondent latent speed parameters, and overestimated precision of respondent latent estimates). The application of IRT models with and without RG to data from a computer-based language test showed parameter differences resembling those in the simulations.

The art of ignoring others' work among academics: A guessing game model of scholarly information search.

Why do scholars pay attention to some works, but not others? This article explores a theoretical model in which scholars search the literature to make sure that their findings are new to their immediate audience. Within the present model, individuals easily disregard literatures of which their audiences are probably unaware. Institutionally organized audiences thus serve as enforcers of the information search. Their members may tacitly collaborate in maintaining unawareness of intellectual developments outside of their common attention space. This model allows us to explain phenomena on which earlier models fail - for example why academics sometimes ignore apparently relevant sources of information or how groups of scholars turn into bubbles, censoring information about findings made in the outside world.

Identifying Ability and Nonability Groups: Incorporating Response Times Using Mixture Modeling.

The goal of the present study was to address the analytical complexity of incorporating responses and response times through applying the Jeon and De Boeck mixture item response theory model in Mplus 8.7. Using both simulated and real data, we attempt to identify subgroups of responders that are rapid guessers or engage knowledge retrieval strategies. When applying the mixture model to a measure of contextual error in linguistics results pointed to the presence of a knowledge retrieval strategy. That is, a participant either knows the content (morphology, grammar rules) and can identify the error, or lacks the requisite knowledge and cannot benefit from spending more time on an item. In contrast, as item difficulty progressed, the high-ability group utilized the additional time to make informed guesses. The methodology is illustrated using annotated code in Mplus 8.7.

Disengaged response behavior when the response button is blocked: Evaluation of a micro-intervention.

In large-scale assessments, disengaged participants might rapidly guess on items or skip items, which can affect the score interpretation's validity. This study analyzes data from a linear computer-based assessment to evaluate a micro-intervention that blocked the possibility to respond for 2 s. The blocked response was implemented to prevent participants from accidental navigation and as a naive attempt to prevent rapid guesses and rapid omissions. The response process was analyzed by interpreting log event sequences within a finite-state machine approach. Responses were assigned to different response classes based on the event sequence. Additionally, post hoc methods for detecting rapid responses based on response time thresholds were applied to validate the classification. Rapid guesses and rapid omissions could be distinguished from accidental clicks by the log events following the micro-intervention. Results showed that the blocked response interfered with rapid responses but hardly led to behavioral changes. However, the blocked response could improve the post hoc detection of rapid responding by identifying responses that narrowly exceed time-bound thresholds. In an assessment context, it is desirable to prevent participants from accidentally skipping items, which in itself may lead to an increasing popularity of initially blocking responses. If, however, data from those assessments is analyzed for rapid responses, additional log data information should be considered.

Are Guessing, Source Coding and Tasks Partitioning Birds of A Feather?

This paper establishes a close relationship among the four information theoretic problems, namely Campbell source coding, Arikan guessing, Huleihel et al. memoryless guessing and Bunte and Lapidoth tasks' partitioning problems in the IID-lossless case. We first show that the aforementioned problems are mathematically related via a general moment minimization problem whose optimum solution is given in terms of Renyi entropy. We then propose a general framework for the mismatched version of these problems and establish all the asymptotic results using this framework. The unified framework further enables us to study a variant of Bunte-Lapidoth's tasks partitioning problem which is practically more appealing. In addition, this variant turns out to be a generalization of Arikan's guessing problem. Finally, with the help of this general framework, we establish an equivalence among all these problems, in the sense that, knowing an asymptotically optimal solution in one problem helps us find the same in all other problems.

PassTCN-PPLL: A Password Guessing Model Based on Probability Label Learning and Temporal Convolutional Neural Network.

The frequent incidents of password leakage have increased people's attention and research on password security. Password guessing is an essential part of password cracking and password security research. The progression of deep learning technology provides a promising way to improve the efficiency of password guessing. However, the mainstream models proposed for password guessing, such as RNN (or other variants, such as LSTM, GRU), GAN and VAE still face some problems, such as the low efficiency and high repetition rate of the generated passwords. In this paper, we propose a password-guessing model based on the temporal convolutional neural network (PassTCN). To further improve the performance of the generated passwords, we propose a novel password probability label-learning method, which reconstructs labels based on the password probability distribution of the training set and deduplicates the training set when training. Experiments on the RockYou dataset showed that, when generating 108 passwords, the coverage rate of PassTCN with password probability label learning (PassTCN-PPLL) reached 12.6%, which is 87.2%, 72.6% and 42.9% higher than PassGAN (a password-guessing model based on GAN), VAEPass (a password-guessing model based on VAE) and FLA (a password-guessing model based on LSTM), respectively. The repetition rate of our model is 25.9%, which is 45.1%, 31.7% and 17.4% lower than that of PassGAN, VAEPass and FLA, respectively. The results confirm that our approach not only improves the coverage rate but also reduces the repetition rate.

Guessing PINs, One Partial PIN at a Time.

Entering digits of a personal identification number (PIN) is a common form of authentication. One variant of this scheme is to request the digits from a random subset of positions, which is sometimes called a partial PIN. In this paper we consider strategies for guessing the PIN when a partial PIN scheme is in use, which allows the quantification of the strength of this mechanism. We suggest several strategies for guessing the PIN under the assumption that the organisation assigns PINs randomly and requests random positions from the PIN at each login. We present analytic and simulation results from the different strategies and explore their performance when guessing different sizes of PIN and requested subset. We find that the most effective strategies have a reasonable chance of recovering a PIN in tens to hundreds of guesses.

Investigating the Effect of Differential Rapid Guessing on Population Invariance in Equating.

Score equating is an essential tool in improving the fairness of test score interpretations when employing multiple test forms. To ensure that the equating functions used to connect scores from one form to another are valid, they must be invariant across different populations of examinees. Given that equating is used in many low-stakes testing programs, examinees' test-taking effort should be considered carefully when evaluating population invariance in equating, particularly as the occurrence of rapid guessing (RG) has been found to differ across subgroups. To this end, the current study investigated whether differential RG rates between subgroups can lead to incorrect inferences concerning population invariance in test equating. A simulation was built to generate data for two examinee subgroups (one more motivated than the other) administered two alternative forms of multiple-choice items. The rate of RG and ability characteristics of rapid guessers were manipulated. Results showed that as RG responses increased, false positive and false negative inferences of equating invariance were respectively observed at the lower and upper ends of the observed score scale. This result was supported by an empirical analysis of an international assessment. These findings suggest that RG should be investigated and documented prior to test equating, especially in low-stakes assessment contexts. A failure to do so may lead to incorrect inferences concerning fairness in equating.

Estimation of Person Ability under Rapid and Effortful Responding.

The goal of the present study was to extend earlier work on the estimation of person theta using maximum likelihood estimation in R by accounting for rapid guessing. This paper provides a modified R function that accommodates person thetas using the Rasch or 2PL models and implements corrections for the presence of rapid guessing or informed guessing behaviors. Initially, a sample of 200 participants was generated using Mplus in order to demonstrate the use of the function with the full sample and a single participant in particular. Subsequently, the function was applied to data from the General Aptitude Test (GAT) and the measurement of cognitive ability. Using a sample of 8500 participants, the present R function was demonstrated. An illustrative example of a single participant, assumed to be either a rapid responder or a successful guesser, is provided using MLE and BME. It was concluded that the present function can contribute to a more valid estimation of person ability.