Investigation of adolescents' developmental stages in deductive reasoning: An application of a specialized confirmatory mixture IRT approach.

In this paper, we propose a specialized confirmatory mixture IRT model to analyze complex cognitive assessment data that is designed to evaluate adolescents' developmental stages in deductive reasoning. The model is specified for the following purposes: (1) to measure multiple deductive reasoning traits, (2) to identify adolescents' differential developmental stages based on their ability levels in the multiple dimensions, (3) to quantify the differences in dimension-specific performance between developmental stages, and (4) to examine the difficulty levels of test design factors. A Bayesian estimation of the model is described. The overall goodness-of-fit of the model is assessed as well as its parameter recovery to validate the application of the model to the data.

Selecting cut scores with a composite of item types: the construct mapping procedure.

In this paper, we describe a new method we have developed for setting cut scores between levels of a test. We outline the wide variety of potential methods that have been used for such a process, and emphasize the need for a coherent conceptual framework under which the variety of methods could be understood. We then describe our particular method, based on an item response modeling framework, which uses the Wright Map, a graphical model of item and threshold difficulties, and a piece of computer software that provides probabilities of various responses for scores under consideration as cut scores. Finally, we describe a study we conducted for the Golden State Examination in Chemistry, in which we investigate the classification agreement for two groups using the method, and also investigate the reactions of the committee members to the procedure and the software, and the lessons we learned from this process.

Concrete, abstract, formal, and systematic operations as observed in a "Piagetian" balance-beam task series.

We performed a Rasch analysis of cross-sectional developmental data gathered from children and adults who were presented with a task series derived from Inhelder's and Piaget's balance beam. The partial credit model situates both participants and items along a single hierarchically ordered dimension. As the Model of Hierarchical Complexity predicted, order of hierarchical complexity accurately predicted item difficulty, with notable exceptions at the formal and systematic levels. Gappiness between items was examined using the saltus model. A two level saltus model, which examined the gap between the concrete/abstract and formal/systematic items, was a better predictor of performance than the Rasch analysis (chi square = 71.91, df = 4, p < .01).

Measuring measuring: toward a theory of proficiency with the constructing measures framework.

This paper is relevant to measurement educators who are interested in the variability of understanding and use of the four building blocks in the Constructing Measures framework (Wilson, 2005). It proposes a uni-dimensional structure for understanding Wilson's framework, and explores the evidence for and against this conceptualization. Constructed and fixed choice response items are utilized to collect responses from 72 participants who range in experience and expertise with constructing measures. The data was scored by two raters was analyzed with the Rasch partial credit model using ConQuest (1998). Guided by the 1999 Testing Standards, analyses of validity and reliability evidence provide support for the construct theory and limited uses of the instrument pending item design modifications.

Mapping multiple dimensions of student learning: the ConstructMap program.

In the past, many assessments, especially standardized assessments, tended to be composed of items with specific right and wrong answers, such as those found in multiple choice, true-false and short response items. Performance-based questions that require students to construct answers rather than select correct responses introduce the complexities of multiple correct answers, dependence on teacher judgment for scoring, and requisite ancillary skills such as language fluency, which are technically difficult to handle, and may even introduce problems such as bias against certain groups of students. Recent developments in assessment design and psychometrics have improved the feasibility of assessing performance-based tasks more efficiently and effectively, thereby providing a rich domain of information from which interpretations can be made about what students know and what they can do when they draw upon that knowledge. We developed the ConstructMap computer program specifically to assist teachers in interpreting and representing this type of performance data. The program accepts as input student scores on items associated with one or multiple performance variables, computes proficiencies using multidimensional item response methods, and produces graphical representations of students' estimated proficiency on each of the variables.

Using Rasch scaled stage scores to validate orders of hierarchical complexity of balance beam task sequences.

These studies examine the relationship between the analytic basis underlying the hierarchies produced by the Model of Hierarchical Complexity and the probabilistic Rasch scales that places both participants and problems along a single hierarchically ordered dimension. A Rasch analysis was performed on data from the balance-beam task series. This yielded scaled stage of performance for each of the items. The items formed a series of clusters along this same dimension, according to their order of hierarchical complexity. We sought to ascertain whether there was a significant relationship between the order of hierarchical complexity (a task property variable) of the tasks and the corresponding Rasch scaled difficulty of those same items (a performance variable). It was found that The Model of Hierarchical Complexity was highly accurate in predicting the Rasch Stage scores of the performed tasks, therefore providing an analytic and developmental basis for the Rasch scaled stages.

The saltus model applied to proportional reasoning data.

This chapter examines an application of the saltus model, a mixture model that was designed for the analysis of developmental data. Some background in the types of research for which such a model might be useful is discussed. The equations defining the model are given, as well as the model's relationship to the Rasch model and to other mixture models. An application of the saltus model to an example data set, collected using Noelting's orange juice mixtures tasks, is examined in detail, along with the control files necessary to run the software, and the output file it produced.