On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests.

Large-scale assessments often use a computer adaptive test (CAT) for selection of items and for scoring respondents. Such tests often assume a parametric form for the relationship between item responses and the underlying construct. Although semi- and nonparametric response functions could be used, there is scant research on their performance in a CAT. In this work, we compare parametric response functions versus those estimated using kernel smoothing and a logistic function of a monotonic polynomial. Monotonic polynomial items can be used with traditional CAT item selection algorithms that use analytical derivatives. We compared these approaches in CAT simulations with a variety of item selection algorithms. Our simulations also varied the features of the calibration and item pool: sample size, the presence of missing data, and the percentage of nonstandard items. In general, the results support the use of semi- and nonparametric item response functions in a CAT.

Metric Stability in Item Response Models.

In item response theory, uncertainty associated with estimated item parameters can lead to greater uncertainty in subsequent analyses, such as estimating trait scores for individual examinees. Most existing methods to characterize or correct for item parameter uncertainty implicitly assume that the latent trait continuum is fixed across the posterior distribution of item parameters. However, the latent trait continuum can also be understood as an artifact of the fitted model, such that the location of this continuum is determined with error. In other words, item parameter estimation error implies uncertainty about the location of the metric. This article uses Ramsay's (1996) geometry of the latent trait metric to develop a quantitative measure of metric stability, that is, the sampling variability of the latent trait continuum implied by errors in item parameter estimation. Through a series of illustrations, it is clarified how metric stability is related to other item response model evaluation outcomes (e.g., test information, model fit), and how metric stability can be useful in identifying well-determined regions of the latent trait continuum, making sample size recommendations, and selecting a model. Overall, the proposed measure of metric stability provides meaningful and highly interpretable information to aid in item response model evaluation.

Improving Measurement Precision in Experimental Psychopathology Using Item Response Theory.

Although item response models have grown in popularity in many areas of educational and psychological assessment, there are relatively few applications of these models in experimental psychopathology. In this article, we explore the use of item response models in the context of a computerized cognitive task designed to assess visual working memory capacity in people with psychosis as well as healthy adults. We begin our discussion by describing how item response theory can be used to evaluate and improve unidimensional cognitive assessment tasks in various examinee populations. We then suggest how computerized adaptive testing can be used to improve the efficiency of cognitive task administration. Finally, we explore how these ideas might be extended to multidimensional item response models that better represent the complex response processes underlying task performance in psychopathological populations.

Are we accurately evaluating depression in patients with cancer?

Depression remains poorly managed in oncology, in part because of the difficulty of reliably screening and assessing for depression in the context of medical illness. Whether somatic items really skew the ability to identify "true" depression, or represent meaningful indicators of depression, remains to be determined. This study utilized item response theory (IRT) to compare the performance of traditional depression criteria with Endicott's substitutive criteria (ESC; tearfulness or depressed appearance; social withdrawal; brooding; cannot be cheered up). The Patient Health Questionnaire (PHQ-9), ESC, and Center for Epidemiologic Studies Depression Scale (CES-D) were administered to 558 outpatients with cancer. IRT models were utilized to evaluate global and item fit for traditional PHQ-9 items compared with a modified version replacing the 4 somatic items with ESC. The modified PHQ-9 ESC scale was the best fit using a partial credit model; model fit was improved after collapsing the middle 2 response categories and removing psychomotor agitation/retardation. This improved model showed satisfactory scale precision and internal consistency, and was free from differential item functioning for gender, age, and race. Concurrent and criterion validity were supported. Thus, as many have speculated, utilizing the ESC may result in more accurate identification of depressive symptoms in oncology. Depressed mood, anhedonia, and suicidal ideation retained their expected properties in the modified scale, indicating that the traditional underlying syndrome of depression likely remains the same, but the ESC may provide more specificity when assessing patients with cancer. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

A note on the identification of change detection task models to measure storage capacity and attention in visual working memory.

The change detection task is a common method for assessing the storage capacity of working memory, but estimates of memory capacity from this task can be distorted by lapses of attention. When combined with appropriate mathematical models, some versions of the change detection task make it possible to separately estimate working memory and the probability of attentional lapses. In principle, these models should allow researchers to isolate the effects of experimental manipulations, group differences, and individual differences on working memory capacity and on the rate of attentional lapses. However, the present research found that two variants of a widely accepted model of the change detection task are not mathematically identified.

Working Memory Impairment Across Psychotic disorders.

BACKGROUND: Working memory (WM) has been a central focus of cognitive neuroscience research because WM is a resource that is involved in many different cognitive operations. The goal of this study was to evaluate the clinical utility of WM paradigms developed in the basic cognitive neuroscience literature, including methods designed to estimate storage capacity without contamination by lapses of attention. METHODS: A total of 61 people with schizophrenia, 49 with schizoaffective disorder, 47 with bipolar disorder with psychosis, and 59 healthy volunteers were recruited. Participants received multiple WM tasks, including two versions each of a multiple Change Detection paradigm, a visual Change Localization paradigm, and a Running Span task. RESULTS: Healthy volunteers performed better than the combined patient group on the visual Change Localization and running span measures. The multiple Change Detection tasks provided mixed evidence about WM capacity reduction in the patient groups, but a mathematical model of performance suggested that the patient groups differed from controls in their rate of attention lapsing. The 3 patient groups performed similarly on the WM tasks. Capacity estimates from the Change Detection and Localization tasks showed significant correlations with functional capacity and functional outcome. CONCLUSIONS: The patient groups generally performed in a similarly impaired fashion across tasks, suggesting that WM impairment and attention lapsing are general features of psychotic disorders. Capacity estimates from the Change Localization and Detection tasks were related to functional capacity and outcome, suggesting that these methods may be useful in a clinical context.

Metric Transformations and the Filtered Monotonic Polynomial Item Response Model.

The [Formula: see text] metric in item response theory is often not the most useful metric for score reporting or interpretation. In this paper, I demonstrate that the filtered monotonic polynomial (FMP) item response model, a recently proposed nonparametric item response model (Liang & Browne in J Educ Behav Stat 40:5-34, 2015), can be used to specify item response models on metrics other than the [Formula: see text] metric. Specifically, I demonstrate that any item response function (IRF) defined within the FMP framework can be re-expressed as another FMP IRF by taking monotonic transformations of the latent trait. I derive the item parameter transformations that correspond to both linear and nonlinear transformations of the latent trait metric. These item parameter transformations can be used to define an item response model based on any monotonic transformation of the [Formula: see text] metric, so long as the metric transformation is approximated by a monotonic polynomial. I demonstrate this result by defining an item response model directly on the approximate true score metric and discuss the implications of metric transformations for applied testing situations.

Sources of Error in IRT Trait Estimation.

In item response theory (IRT), item response probabilities are a function of item characteristics and latent trait scores. Within an IRT framework, trait score misestimation results from (a) random error, (b) the trait score estimation method, (c) errors in item parameter estimation, and (d) model misspecification. This study investigated the relative effects of these error sources on the bias and confidence interval coverage rates for trait scores. Our results showed that overall, bias values were close to 0, and coverage rates were fairly accurate for central trait scores and trait estimation methods that did not use a strong Bayesian prior. However, certain types of model misspecifications were found to produce severely biased trait estimates with poor coverage rates, especially at extremes of the latent trait continuum. It is demonstrated that biased trait estimates result from estimated item response functions (IRFs) that exhibit systematic conditional bias, and that these conditionally biased IRFs may not be detected by model or item fit indices. One consequence of these results is that certain types of model misspecifications can lead to estimated trait scores that are nonlinearly related to the data-generating latent trait. Implications for item and trait score estimation and interpretation are discussed.