Modelación con teoría de grafos para la unidimensionalidad de un instrumento de evaluación / Modeling with graph theory to unidimensionality for an assessment instrument

RESUMEN

En el marco de las políticas públicas en Colombia sobre calidad de la educación superior se encuentran los lineamientos para que el Estado reconozca los programas de pregrado como de alta calidad, entre tales lineamientos se consideran los sistemas de evaluación de los estudiantes basados en políticas institucionales claras y transparentes que propicien una identificación integral de las condiciones académicas y actitudinales de los estudiantes por lo que es imprescindible tecnificar y modernizar los procesos de evaluación para fortalecer la emisión de inferencias confiables sobre el logro académico. En este artículo se describe una aplicación de la Teoría de Grafos usando algunas de sus definiciones y teoremas básicos con la intención de potenciar su uso para el estudio del supuesto de unidimensionalidad de la Teoría de Respuesta al Ítem como alternativa para robustecer la evaluación. La unidimensional fue entendida en el sentido de la homogeneidad escalar y de criterio para las relaciones de orden definidas por la dominación tanto para ítems como para evaluados para los ítems conforme al número de evaluados que los contestaron correctamente y a la dificultad; para los evaluados respecto al número de respuestas correctas y a la habilidad. Se aplicó la combinación de las dos teorías a un caso de estudio que contempla el instrumento de la prueba diagnóstica del Examen de Clasificación de Matemáticas Básicas en la Universidad Jorge Tadeo Lozano, con esto se ofrece un valor agregado porque se evidencian los resultados de forma algorítmica y visual con ayuda de herramientas informáticas especializadas.

ABSTRACT

In Colombia, public policies about quality of higher education involve student assessment systems as a relevant characteristic. These systems should have clear and transparent institutional policies requiring comprehensive identification of attitudinal and academic conditions about examinee performance. Consequently, on the way of continuous improvement of academic services offered by institutions of higher education, an assessment process adequately technified allowing valid inferences about academic achievement is required. In this sense, the Rasch measurement model of Item Response Theory is a modern alternative to strengthen educational measurement estimating the ability of the student and the difficulty of the item on a comparable scale. Unidimensionality, local independence and internal consistency are assumptions made in Rasch measurement model. Unidimensionality assumption has several definitions, one of them is the occurrence of a dominant factor influencing test performance. Another definition is considered in the present study. Here, unidimensionality was interpreted as scalar and criterial homogeneity for the order relations defined by domination for items and assessed applicants. Graph Theory is an ideal mathematical modelling approach to this assumption inasmuch as represent intangible interactions as required. In order to achieve this, Graph Theory and Item Response Theory were combined to examine the qualifying test for the Basic Mathematics subject at Jorge Tadeo Lozano University as a case study. This test was composed by 45 items divided in three blocks. With several combinations of these blocks, three virtual booklets were obtained each one comprising 30 of them. From the application of June 2011, a test sample of 509 responses chains was obtained. Three data bases were processed one for each booklet, the first spanned 175 responses chains, the second 170 and the third one 164. To begin with the exmination on the Rasch measurement model, the parameters reliability, separation, Cronbach Alpha and item residual correlation were estimated to gauge and determine performance test for getting acceptable values in each booklet applied. Tatsuoka was followed to undertake the path to unidimensionality from Graph Theory gathering a real case experience processed with suitable software. Additionally, the sensitivity of the order relation was verified through 1) ordering by number of correct responses per item (1I order) and items difficulty (2I order); 2) ordering by number of correct responses per applicant (1E order) and estimated ability (2E order). Furthermore, the linear models were obtained collating these orders. In like manner, three Guttman scales and their adjacency matrices were schematized one for each booklet. Subsequently, the respective graphs were processed and represented using Gephi just as a specialized tool that enables running some algorithms like Force Atlas. Afterwards, the second power of each adjacency matrix was found using Matlab 2014b and domination matrices were calculated for both items and applicants in the aforementioned orders for a total of 12 matrices. Consistency index developed by Cliff was computed for the domination matrices. As a result, moderate consistency was observed. Significant domination for the entries of these matrices was analyzed through McNemar test in order to have an asymmetric dominance relation. Moreover, a reachable matrix was calculated for each one of these significant domination matrices as a limit ofa sequence of boolean powers. Finally, dominance hierarchies were illustrated with vertex degrees and compared with student maps by the means of Winsteps 3.73. The combination of Graph Theory and Item Response Theory allowed a deeper comprehension of unidimensionality assumption. Thereupon, universities can optimize their resources offering to applicant differential academic options per individual position in the ability scale. The results can be used to outline advantages for the applicants who can evidence their position in the ability scale and identify the different areas to improve.