R-package LNIRT for joint modeling of response accuracy and times.

In computer-based testing it has become standard to collect response accuracy (RA) and response times (RTs) for each test item. IRT models are used to measure a latent variable (e.g., ability, intelligence) using the RA observations. The information in the RTs can help to improve routine operations in (educational) testing, and provide information about speed of working. In modern applications, the joint models are needed to integrate RT information in a test analysis. The R-package LNIRT supports fitting joint models through a user-friendly setup which only requires specifying RA, RT data, and the total number of Gibbs sampling iterations. More detailed specifications of the analysis are optional. The main results can be reported through the summary functions, but output can also be analysed with Markov chain Monte Carlo (MCMC) output tools (i.e., coda, mcmcse). The main functionality of the LNIRT package is illustrated with two real data applications.

Assessing an alternative for "negative variance components": A gentle introduction to Bayesian covariance structure modeling for negative associations among patients with personalized treatments.

The multilevel model (MLM) is the popular approach to describe dependences of hierarchically clustered observations. A main feature is the capability to estimate (cluster-specific) random effect parameters, while their distribution describes the variation across clusters. However, the MLM can only model positive associations among clustered observations, and it is not suitable for small sample sizes. The limitation of the MLM becomes apparent when estimation methods produce negative estimates for random effect variances, which can be seen as an indication that observations are negatively correlated. A gentle introduction to Bayesian covariance structure modeling (BCSM) is given, which makes it possible to model also negatively correlated observations. The BCSM does not model dependences through random (cluster-specific) effects, but through a covariance matrix. We show that this makes the BCSM particularly useful for small data samples. We draw specific attention to detect effects of a personalized intervention. The effect of a personalized treatment can differ across individuals, and this can lead to negative associations among measurements of individuals who are treated by the same therapist. It is shown that the BCSM enables the modeling of negative associations among clustered measurements and aids in the interpretation of negative clustering effects. Through a simulation study and by analysis of a real data example, we discuss the suitability of the BCSM for small data sets and for exploring effects of individualized treatments, specifically when (standard) MLM software produces negative or zero variance estimates. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Bayesian Covariance Structure Modeling of Responses and Process Data.

A novel Bayesian modeling framework for response accuracy (RA), response times (RTs) and other process data is proposed. In a Bayesian covariance structure modeling approach, nested and crossed dependences within test-taker data (e.g., within a testlet, between RAs and RTs for an item) are explicitly modeled. The local dependences are modeled directly through covariance parameters in an additive covariance matrix. The inclusion of random effects (on person or group level) is not necessary, which allows constructing parsimonious models for responses and multiple types of process data. Bayesian Covariance Structure Models (BCSMs) are presented for various well-known dependence structures. Through truncated shifted inverse-gamma priors, closed-form expressions for the conditional posteriors of the covariance parameters are derived. The priors avoid boundary effects at zero, and ensure the positive definiteness of the additive covariance structure at any layer. Dependences of categorical outcome data are modeled through latent continuous variables. In a simulation study, a BCSM for RAs and RTs is compared to van der Linden's hierarchical model (LHM; van der Linden, 2007). Under the BCSM, the dependence structure is extended to allow variations in test-takers' working speed and ability and is estimated with a satisfying performance. Under the LHM, the assumption of local independence is violated, which results in a biased estimate of the variance of the ability distribution. Moreover, the BCSM provides insight in changes in the speed-accuracy trade-off. With an empirical example, the flexibility and relevance of the BCSM for complex dependence structures in a real-world setting are discussed.

Understanding Therapeutic Change Process Research Through Multilevel Modeling and Text Mining.

Online interventions hold great potential for Therapeutic Change Process Research (TCPR), a field that aims to relate in-therapeutic change processes to the outcomes of interventions. Online a client is treated essentially through the language their counsellor uses, therefore the verbal interaction contains many important ingredients that bring about change. TCPR faces two challenges: how to derive meaningful change processes from texts, and secondly, how to assess these complex, varied, and multi-layered processes? We advocate the use text mining and multi-level models (MLMs): the former offers tools and methods to discovers patterns in texts; the latter can analyse these change processes as outcomes that vary at multiple levels. We (re-)used the data from Lamers et al. (2015) because it includes outcomes and the complete online intervention for clients with mild depressive symptoms. We used text mining to obtain basic text-variables from e-mails, that we analyzed through MLMs. We found that we could relate outcomes of interventions to variables containing text-information. We conclude that we can indeed bridge text mining and MLMs for TCPR as it was possible to relate text-information (obtained through text mining) to multi-leveled TCPR outcomes (using a MLM). Text mining can be helpful to obtain change processes, which is also the main challenge for TCPR. We showed how MLMs and text mining can be combined, but our proposition leaves open how to obtain the most relevant textual operationalization of TCPR concepts. That requires interdisciplinary collaboration and discussion. The future does look bright: based on our proof-of-concept study we conclude that MLMs and text mining can indeed advance TCPR.

Modeling Dependence Structures for Response Times in a Bayesian Framework.

A multivariate generalization of the log-normal model for response times is proposed within an innovative Bayesian modeling framework. A novel Bayesian Covariance Structure Model (BCSM) is proposed, where the inclusion of random-effect variables is avoided, while their implied dependencies are modeled directly through an additive covariance structure. This makes it possible to jointly model complex dependencies due to for instance the test format (e.g., testlets, complex constructs), time limits, or features of digitally based assessments. A class of conjugate priors is proposed for the random-effect variance parameters in the BCSM framework. They give support to testing the presence of random effects, reduce boundary effects by allowing non-positive (co)variance parameters, and support accurate estimation even for very small true variance parameters. The conjugate priors under the BCSM lead to efficient posterior computation. Bayes factors and the Bayesian Information Criterion are discussed for the purpose of model selection in the new framework. In two simulation studies, a satisfying performance of the MCMC algorithm and of the Bayes factor is shown. In comparison with parameter expansion through a half-Cauchy prior, estimates of variance parameters close to zero show no bias and undercoverage of credible intervals is avoided. An empirical example showcases the utility of the BCSM for response times to test the influence of item presentation formats on the test performance of students in a Latin square experimental design.

Bayes Factor Covariance Testing in Item Response Models.

Two marginal one-parameter item response theory models are introduced, by integrating out the latent variable or random item parameter. It is shown that both marginal response models are multivariate (probit) models with a compound symmetry covariance structure. Several common hypotheses concerning the underlying covariance structure are evaluated using (fractional) Bayes factor tests. The support for a unidimensional factor (i.e., assumption of local independence) and differential item functioning are evaluated by testing the covariance components. The posterior distribution of common covariance components is obtained in closed form by transforming latent responses with an orthogonal (Helmert) matrix. This posterior distribution is defined as a shifted-inverse-gamma, thereby introducing a default prior and a balanced prior distribution. Based on that, an MCMC algorithm is described to estimate all model parameters and to compute (fractional) Bayes factor tests. Simulation studies are used to show that the (fractional) Bayes factor tests have good properties for testing the underlying covariance structure of binary response data. The method is illustrated with two real data studies.

Measurement model choice influenced randomized controlled trial results.

OBJECTIVE: In randomized controlled trials (RCTs), outcome variables are often patient-reported outcomes measured with questionnaires. Ideally, all available item information is used for score construction, which requires an item response theory (IRT) measurement model. However, in practice, the classical test theory measurement model (sum scores) is mostly used, and differences between response patterns leading to the same sum score are ignored. The enhanced differentiation between scores with IRT enables more precise estimation of individual trajectories over time and group effects. The objective of this study was to show the advantages of using IRT scores instead of sum scores when analyzing RCTs. STUDY DESIGN AND SETTING: Two studies are presented, a real-life RCT, and a simulation study. Both IRT and sum scores are used to measure the construct and are subsequently used as outcomes for effect calculation. RESULTS: The bias in RCT results is conditional on the measurement model that was used to construct the scores. A bias in estimated trend of around one standard deviation was found when sum scores were used, where IRT showed negligible bias. CONCLUSION: Accurate statistical inferences are made from an RCT study when using IRT to estimate construct measurements. The use of sum scores leads to incorrect RCT results.

Joint Modeling of Ability and Differential Speed Using Responses and Response Times.

With computerized testing, it is possible to record both the responses of test takers to test questions (i.e., items) and the amount of time spent by a test taker in responding to each question. Various models have been proposed that take into account both test-taker ability and working speed, with the many models assuming a constant working speed throughout the test. The constant working speed assumption may be inappropriate for various reasons. For example, a test taker may need to adjust the pace due to time mismanagement, or a test taker who started out working too fast may reduce the working speed to improve accuracy. A model is proposed here that allows for variable working speed. An illustration of the model using the Amsterdam Chess Test data is provided.

Psychological flexibility and catastrophizing as associated change mechanisms during online Acceptance & Commitment Therapy for chronic pain.

The underlying mechanisms of the effectiveness of cognitive behavioural interventions for chronic pain need further clarification. The role of, and associations between, pain-related psychological flexibility (PF) and pain catastrophizing (PC) were examined during a randomized controlled trial on internet-based Acceptance & Commitment Therapy (ACT) for chronic pain. We assessed (1) the unique and combined indirect effects of PF and PC on outcomes, and (2) the causality of relations between PF, PC and the primary outcome pain interference in daily life (MPI) during ACT. A total of 238 pain sufferers were allocated to either ACT, a control condition on Expressive Writing, or a waiting list condition. Non-parametric cross-product of coefficients mediational analyses and cross-lagged panel designs were applied. Compared to control conditions, both baseline to post-intervention changes in PF and PC seemed to uniquely mediate baseline to three-month follow-up changes in pain interference and psychological distress. Only PF mediated changes in pain intensity. Indirect effects were twice as large for PF (κ2 = .09-.19) than for PC (κ² PCS = .05-.10). Further assessment of changes during ACT showed, however, that only PF, and not PC, predicted subsequent changes in MPI, while early and late changes in both PF and PC predicted later changes in each other. In conclusion, only PF functioned as a direct, causal working mechanism during ACT, with larger indirect effects that occurred earlier than changes in PC. Additionally, PC may function as an indirect mechanism of change during ACT for chronic pain via its direct influence on PF.

Why item response theory should be used for longitudinal questionnaire data analysis in medical research.

BACKGROUND: Multi-item questionnaires are important instruments for monitoring health in epidemiological longitudinal studies. Mostly sum-scores are used as a summary measure for these multi-item questionnaires. The objective of this study was to show the negative impact of using sum-score based longitudinal data analysis instead of Item Response Theory (IRT)-based plausible values. METHODS: In a simulation study (varying the number of items, sample size, and distribution of the outcomes) the parameter estimates resulting from both modeling techniques were compared to the true values. Next, the models were applied to an example dataset from the Amsterdam Growth and Health Longitudinal Study (AGHLS). RESULTS: The results show that using sum-scores leads to overestimation of the within person (repeated measurement) variance and underestimation of the between person variance. CONCLUSIONS: We recommend using IRT-based plausible value techniques for analyzing repeatedly measured multi-item questionnaire data.

Bayesian inference for an illness-death model for stroke with cognition as a latent time-dependent risk factor.

Longitudinal data can be used to estimate the transition intensities between healthy and unhealthy states prior to death. An illness-death model for history of stroke is presented, where time-dependent transition intensities are regressed on a latent variable representing cognitive function. The change of this function over time is described by a linear growth model with random effects. Occasion-specific cognitive function is measured by an item response model for longitudinal scores on the Mini-Mental State Examination, a questionnaire used to screen for cognitive impairment. The illness-death model will be used to identify and to explore the relationship between occasion-specific cognitive function and stroke. Combining a multi-state model with the latent growth model defines a joint model which extends current statistical inference regarding disease progression and cognitive function. Markov chain Monte Carlo methods are used for Bayesian inference. Data stem from the Medical Research Council Cognitive Function and Ageing Study in the UK (1991-2005).

Harmonization of Neuroticism and Extraversion phenotypes across inventories and cohorts in the Genetics of Personality Consortium: an application of Item Response Theory.

Mega- or meta-analytic studies (e.g. genome-wide association studies) are increasingly used in behavior genetics. An issue in such studies is that phenotypes are often measured by different instruments across study cohorts, requiring harmonization of measures so that more powerful fixed effect meta-analyses can be employed. Within the Genetics of Personality Consortium, we demonstrate for two clinically relevant personality traits, Neuroticism and Extraversion, how Item-Response Theory (IRT) can be applied to map item data from different inventories to the same underlying constructs. Personality item data were analyzed in >160,000 individuals from 23 cohorts across Europe, USA and Australia in which Neuroticism and Extraversion were assessed by nine different personality inventories. Results showed that harmonization was very successful for most personality inventories and moderately successful for some. Neuroticism and Extraversion inventories were largely measurement invariant across cohorts, in particular when comparing cohorts from countries where the same language is spoken. The IRT-based scores for Neuroticism and Extraversion were heritable (48 and 49 %, respectively, based on a meta-analysis of six twin cohorts, total N = 29,496 and 29,501 twin pairs, respectively) with a significant part of the heritability due to non-additive genetic factors. For Extraversion, these genetic factors qualitatively differ across sexes. We showed that our IRT method can lead to a large increase in sample size and therefore statistical power. The IRT approach may be applied to any mega- or meta-analytic study in which item-based behavioral measures need to be harmonized.

The Joint Multivariate Modeling of Multiple Mixed Response Sources: Relating Student Performances with Feedback Behavior.

The present study concerns a Dutch computer-based assessment, which includes an assessment process about information literacy and a feedback process for students. The assessment is concerned with the measurement of skills in information literacy and the feedback process with item-based support to improve student learning. To analyze students' feedback behavior (i.e. feedback use and attention time), test performance, and speed of working, a multivariate hierarchical latent variable model is proposed. The model can handle multivariate mixed responses from multiple sources related to different processes and comprehends multiple measurement components for responses and response times. A flexible within-subject latent variable structure is defined to explore multiple individual latent characteristics related to students' test performance and feedback behavior. Main results of the computer-based assessment showed that feedback-information pages were less visited by well-performing students when they relate to easy items. Students' attention paid to feedback was positively related to working speed but not to the propensity to use feedback.

Compensatory and non-compensatory multidimensional randomized item response models.

Randomized response (RR) models are often used for analysing univariate randomized response data and measuring population prevalence of sensitive behaviours. There is much empirical support for the belief that RR methods improve the cooperation of the respondents. Recently, RR models have been extended to measure individual unidimensional behaviour. An extension of this modelling framework is proposed to measure compensatory or non-compensatory multiple sensitive factors underlying the randomized item response process. A confirmatory multidimensional randomized item response theory model (MRIRT) is proposed for the analysis of multivariate RR data by modelling the response process and specifying structural relationships between sensitive behaviours and background information. A Markov chain Monte Carlo algorithm is developed to estimate simultaneously the parameters of the MRIRT model. The model extension enables the computation of individual true item response probabilities, estimates of individuals' sensitive behaviour on different domains, and their relationships with background variables. An MRIRT analysis is presented of data from a college alcohol problem scale, measuring alcohol-related socio-emotional and community problems, and alcohol expectancy questionnaire, measuring alcohol-related sexual enhancement expectancies. Students were interviewed via direct or RR questioning. Scores of alcohol-related problems and expectancies are significantly higher for the group of students questioned using the RR technique. Alcohol-related problems and sexual enhancement expectancies are positively moderately correlated and vary differently across gender and universities.

The role of psychological flexibility in a self-help acceptance and commitment therapy intervention for psychological distress in a randomized controlled trial.

This study examined the role of psychological flexibility, as a risk factor and as a process of change, in a self-help Acceptance and Commitment Therapy (ACT) intervention for adults with mild to moderate depression and anxiety. Participants were randomized to the self-help programme with e-mail support (n=250), or to a waiting list control group (n=126). All participants completed measures before and after the intervention to assess depression, anxiety and psychological flexibility. Participants in the experimental condition also completed these measures during the intervention (after three and six weeks) and at a three-month follow-up. With multilevel modelling, it was shown that the effects of the intervention on psychological distress were stronger for participants with higher levels of psychological flexibility. Furthermore, our study showed that improved psychological flexibility mediated the effects of the ACT intervention. With a cross-lagged panel design, it was shown that especially improvements in psychological flexibility in the last three sessions of the intervention were important for further reductions in anxiety. To conclude, our study showed the importance of targeting psychological flexibility during an ACT intervention for a reduction in depressive and anxiety symptoms.

Longitudinal measurement in health-related surveys. A Bayesian joint growth model for multivariate ordinal responses.

Longitudinal surveys measuring physical or mental health status are a common method to evaluate treatments. Multiple items are administered repeatedly to assess changes in the underlying health status of the patient. Traditional models to analyze the resulting data assume that the characteristics of at least some items are identical over measurement occasions. When this assumption is not met, this can result in ambiguous latent health status estimates. Changes in item characteristics over occasions are allowed in the proposed measurement model, which includes truncated and correlated random effects and a growth model for item parameters. In a joint estimation procedure adopting MCMC methods, both item and latent health status parameters are modeled as longitudinal random effects. Simulation study results show accurate parameter recovery. Data from a randomized clinical trial concerning the treatment of depression by increasing psychological acceptance showed significant item parameter shifts. For some items, the probability of responding in the middle category versus the highest or lowest category increased significantly over time. The resulting latent depression scores decreased more over time for the experimental group than for the control group and the amount of decrease was related to the increase in acceptance level.

The dirichlet-multinomial model for multivariate randomized response data and small samples

In survey sampling the randomized response (RR) technique can be used to obtain truthful answers to sensitive questions. Although the individual answers are masked due to the RR technique, individual (sensitive) response rates can be estimated when observing multivariate response data. The beta-binomial model for binary RR data will be generalized to handle multivariate categorical RR data. The Dirichlet- multinomial model for categorical RR data is extended with a linear transformation of the masked individual categorical-response rates to correct for the RR design and to retrieve the sensitive categorical-response rates even for small data samples. This specification of the Dirichlet- multinomial model enables a straightforward empirical Bayes estimation of the model parameters. A constrained-Dirichlet prior will be introduced to identify homogeneity restrictions in response rates across persons and/or categories. The performance of the full Bayes parameter estimation method is verified using simulated data. The proposed model will be applied to the college alcohol problem scale study, where students were interviewed directly or interviewed via the randomized response technique about negative consequences from drinking(AU)

A mixture model for the joint analysis of latent developmental trajectories and survival.

A general joint modeling framework is proposed that includes a parametric stratified survival component for continuous time survival data, and a mixture multilevel item response component to model latent developmental trajectories given mixed discrete response data. The joint model is illustrated in a real data setting, where the utility of longitudinally measured cognitive function as a predictor for survival is investigated in a group of elderly persons. The object is partly to determine whether cognitive impairment is accompanied by a higher mortality rate. Time-dependent cognitive function is measured using the generalized partial credit model given occasion-specific mini-mental state examination response data. A parametric survival model is applied for the survival information, and cognitive function as a continuous latent variable is included as a time-dependent explanatory variable along with other explanatory information. A mixture model is defined, which incorporates the latent developmental trajectory and the survival component. The mixture model captures the heterogeneity in the developmental trajectories that could not be fully explained by the multilevel item response model and other explanatory variables. A Bayesian modeling approach is pursued, where a Markov chain Monte Carlo algorithm is developed for simultaneous estimation of the joint model parameters. Practical issues as model building and assessment are addressed using the DIC and various posterior predictive tests.