RESUMO
Automatic Item Generation (AIG) techniques are offering innovative ways to produce test items as they overcome many disadvantages involving standard item writing, such as time-consuming work and resource-intensive demands. Although this field is relatively new, it is progressing at a high speed, and several contributions have been accomplished. Nevertheless, a scarce amount of AIG software evidencing favorable psychometric properties of the generated items has been made accessible to the broad scientific community. This research had two goals: first, to present an empirical study of items produced with the aid of the Item Maker (IMak) package available online and, second, to present IMak itself for the automatic generation of figural analogies. We were particularly interested in assessing whether automatically created figural analogy rules could predict item psychometric difficulty. A total of 23 items were generated and administered to 307 participants, 49.51% from Germany. The mean age was 28.61 (SD = 10.19) and 57.65% of the participants were female. Results reveal adequate psychometric properties including convergent validity, that most of the manipulated rules contribute to item difficulty, and that rule-based difficulty prediction is possible to some extent. In other words, psychometric quality of the generated items is supported, which reveals the utility of the IMak package in assessment settings. Finally, the package is presented and its functions for figural analogy item generation are further described.
RESUMO
As retest effects in cognitive ability tests have been investigated by various primary and meta-analytic studies, most studies from this area focus on score gains as a result of retesting. To the best of our knowledge, no meta-analytic study has been reported that provides sizable estimates of response time (RT) reductions due to retesting. This multilevel meta-analysis focuses on mental speed tasks, for which outcome measures often consist of RTs. The size of RT reduction due to retesting in mental speed tasks for up to four test administrations was analyzed based on 36 studies including 49 samples and 212 outcomes for a total sample size of 21,810. Significant RT reductions were found, which increased with the number of test administrations, without reaching a plateau. Larger RT reductions were observed in more complex mental speed tasks compared to simple ones, whereas age and test-retest interval mostly did not moderate the size of the effect. Although a high heterogeneity of effects exists, retest effects were shown to occur for mental speed tasks regarding RT outcomes and should thus be more thoroughly accounted for in applied and research settings.
RESUMO
En el presente artículo se describen los pasos que permiten considerar la eliminación de dos de los reactivos de un Test de Analogías Figurales (TAF). Se explican las bases generales de su análisis psicométrico tanto desde la Teoría Clásica de Tests (TCT) como la Teoría de Respuesta al Ítem (TRI). Se detalla el proceso de eliminación de reactivos basado en la obtención de los índices de dificultad y discriminación de los ítems desde la TCT, así como el análisis de la unidimensionalidad. También se toman criterios basados en la estimación de los parámetros a, b y c de cada ítem según el Modelo Logístico de Tres Parámetros de la TRI, así como la determinación del ajuste de cada reactivo sobre la base de dicho modelo logístico. Se detallan las características desfavorables de algunos ítems del TAF y se discuten las decisiones que llevan al mencionado proceso de eliminación.
This paper describes the steps taken to eliminate two of the items in a Test of Figural Analogies (TFA). The main guidelines of psychometric analysis concerning Classical Test Theory (CTT) and Item Response Theory (IRT) are explained. The item elimination process was based on both the study of the CTT difficulty and discrimination index, and the unidimensionality analysis. The a, b, and c parameters of the Three Parameter Logistic Model of IRT were also considered for this purpose, as well as the assessment of each item fitting this model. The unfavourable characteristics of a group of TFA items are detailed, and decisions leading to their possible elimination are discussed.
