Components of the item selection algorithm in computerized adaptive testing / 보건의료교육평가

Computerized adaptive testing (CAT) greatly improves measurement efficiency in high-stakes testing operations through the selection and administration of test items with the difficulty level that is most relevant to each individual test taker. This paper explains the 3 components of a conventional CAT item selection algorithm: test content balancing, the item selection criterion, and item exposure control. Several noteworthy methodologies underlie each component. The test script method and constrained CAT method are used for test content balancing. Item selection criteria include the maximized Fisher information criterion, the b-matching method, the a-stratification method, the weighted likelihood information criterion, the efficiency balanced information criterion, and the Kullback-Leibler information criterion. The randomesque method, the Sympson-Hetter method, the unconditional and conditional multinomial methods, and the fade-away method are used for item exposure control. Several holistic approaches to CAT use automated test assembly methods, such as the shadow test approach and the weighted deviation model. Item usage and exposure count vary depending on the item selection criterion and exposure control method. Finally, other important factors to consider when determining an appropriate CAT design are the computer resources requirement, the size of item pools, and the test length. The logic of CAT is now being adopted in the field of adaptive learning, which integrates the learning aspect and the (formative) assessment aspect of education into a continuous, individualized learning experience. Therefore, the algorithms and technologies described in this review may be able to help medical health educators and high-stakes test developers to adopt CAT more actively and efficiently.

Validez en test adaptativos informatizados: alternativa para evaluar población con limitaciones visuales / Validade em testes adaptativos computadorizados: alternativa para avaliar população com deficiência visual / Validity in computerized adaptive testing: an alternative to assess visually impaired individuals

Este artículo centra su interés en la importancia de la validez dentro de los procesos de evaluación psicológica cuando se pretende evaluar a personas invidentes o con baja visión, y en el desarrollo de los Test Adaptativos Informatizados (TAI) como una alternativa para la evaluación de esta población. Se presenta una revisión sobre el concepto de validez a partir de las aproximaciones contemporáneas dominantes, y se habla acerca del desarrollo de los TAI, el problema de su validez y los alcances de esta tecnología como una alternativa para evaluar personas con baja visión o invidentes. Finalmente, este trabajo deja abierta la posibilidad para futuros desarrollos e investigaciones en torno a otras alternativas de evaluación con equidad para poblaciones con discapacidad física.

Este artigo centra o seu interesse na importância da validade dentro dos processos de avaliação psicológica de pessoas cegas ou com baixa visão e no desenvolvimento dos Testes Adaptativos Computadorizados (TAC) como uma alternativa para a avaliação dessa população. Apresenta-se uma revisão sobre o conceito de validade a partir das abordagens contemporâneas dominantes e sobre o desenvolvimento dos TAC, o problema da sua validade e os alcances dessa tecnologia como uma alternativa para avaliar pessoas cegas. Finalmente, o trabalho deixa em aberto a possibilidade de futuros desenvolvimentos e pesquisas sobre alternativas de avaliação com equidade para a população com deficiência física.

This article focuses on the importance of validity in the process of psychological evaluation, especially when the evaluation is intended for people with visual impairment, and about the development of the Computerized Adaptive Testing (CAT) as an alternative for the assessment of this population. First, a review of the main controversies about the concept of validity and the development of Computerized Adaptive Test is presented. Then, topics related to validity in CAT and scope, technological limitations, and the challenges of its use as an alternative to evaluate the visual impairment population are discussed. Finally, possibilities for future developments and research on high quality alternative assessment for populations with physical disabilities are set forth.

Can computerized tests be introduced to the Korean Medical Licensing Examination?

In November 2011, the standing Committee of the Korean Medical Licensing Examination (KMLE) recommended that the National Health Personnel Licensing Examination Board introduce computerized testing to the KMLE. Therefore this article contextualizes and explores the possibility of applying computerized testing to the KMLE. Computerized testing comprises computer-based testing (CBT), ubiquitous-based testing (UBT), internet-based testing (IBT), and computerized adaptive testing (CAT). CBT refers to testing administered via a computer as the user interface, while testing with a smart phone or smart pad as the user interface is known as UBT. IBT is testing done online, and CAT is testing tailored so that each item provided fits the examinee's ability level. The benefits and drawbacks of each computerized testing option were surveyed. Among them, I propose CAT as the final goal for KMLE. In order to implement the computerized testing more effectively, it is recommended that items contain multimedia data and should involve interpretation or problem-solving. More evidence is needed to support the positive impact of computerized testing for undergraduate medical education and primary health care. Since the rapid progress of information technology such as internet bandwidth and human-computer interface methods, the introduction of computerized testing to KMLE will soon be plausible. It is possible to increase the quality of the KMLE with the introduction of computerized testing. Medical schools should prepare for the new testing environment of the KMLE by recruiting or training specialists in this field.

Estimation of an Examinee's Ability in the Web-Based Computerized Adaptive Testing Program IRT-CAT / 보건의료교육평가

We developed a program to estimate an examinee s ability in order to provide freely available access to a web-based computerized adaptive testing (CAT) program. We used PHP and Java Script as the program languages, PostgresSQL as the database management system on an Apache web server and Linux as the operating system. A system which allows for user input and searching within inputted items and creates tests was constructed. We performed an ability estimation on each test based on a Rasch model and 2- or 3-parametric logistic models. Our system provides an algorithm for a web-based CAT, replacing previous personal computer-based ones, and makes it possible to estimate an examinee's ability immediately at the end of test.