Sociocognitive and Argumentation Perspectives on Psychometric Modeling in Educational Assessment.

Rapid advances in psychology and technology open opportunities and present challenges beyond familiar forms of educational assessment and measurement. Viewing assessment through the perspectives of complex adaptive sociocognitive systems and argumentation helps us extend the concepts and methods of educational measurement to new forms of assessment, such as those involving interaction in simulation environments and automated evaluation of performances. I summarize key ideas for doing so and point to the roles of measurement models and their relation to sociocognitive systems and assessment arguments. A game-based learning assessment SimCityEDU: Pollution Challenge! is used to illustrate ideas.

Evidence centered design framework and dynamic bayesian network for modeling learning progression in online assessment system.

An overarching mission of the educational assessment community today is strengthening the connection between assessment and learning. To support this effort, researchers draw variously on developments across technology, analytic methods, assessment design frameworks, research in learning domains, and cognitive, social, and situated psychology. The study lays out the connection among three such developments, namely learning progressions, evidence-centered assessment design (ECD), and dynamic Bayesian modeling for measuring students' advancement along learning progression in a substantive domain. Their conjunction can be applied in both formative and summative assessment uses. In addition, this study conducted an application study in domain of beginning computer network engineering for illustrating the ideas with data drawn from the Cisco Networking Academy's online assessment system.

Characterizing Interactive Communications in Computer-Supported Collaborative Problem-Solving Tasks: A Conditional Transition Profile Approach.

Communication in a collaborative problem-solving activity plays a pivotal role in the success of the collaboration in both academia and the workplace. Computer-supported collaboration makes it possible to collect large-scale communication data to investigate the process at a finer granularity. In this paper, we introduce a conditional transition profile (CTP) to characterize aspects of each team member's communication. Based on the data from a large-scale empirical study, we found that participants in the same team tend to show similar CTP compared to participants from different teams. We also found that team members who showed more "negotiation" after the partner "shared" information tended to show more improvement after the collaboration while those who continued sharing ideas while their partners were negotiating tended to improve less.

Reweighting Data in the Spirit of Tukey: Using Bayesian Posterior Probabilities as Rasch Residuals for Studying Misfit.

A new variant of the iterative "data = fit + residual" data-analytical approach described by Mosteller and Tukey is proposed and implemented in the context of item response theory psychometric models. Posterior probabilities from a Bayesian mixture model of a Rasch item response theory model and an unscalable latent class are expressed as weights for the original data. The data are weighted by the units' posterior probabilities for the unscalable class and used for further exploration of structures. Factor analysis models are compared with the original data and data as reweighted by the posterior probabilities for the unscalable class. In comparing two weighted data sets, Rasch-weighted data and data considered unscalable, differences were evident. Pattern types are detected for the Rasch baseline with patterns that are different patterns from random or systematic contamination. Rasch baseline patterns are strongest near item difficulties closest to the mean generating value of θs. Patterns in baseline conditions are weaker as they depart from an item difficulty of zero and move toward extreme values. Random contamination patterns are typically flat and near zero regardless of item difficulty. Systematic contamination using reversed Rasch-generated data produced alternate patterns to the Rasch baseline condition and in some conditions showed an opposite effect from Rasch patterns. Differences could be detected within residually weighted data between the Rasch-generated subtest and contaminated subtest. Rasch subtest often had Rasch patterns while contaminated subtest had random/flat or systematic/reversed pattern.

Evidence-centered design for simulation-based assessment.

Simulations provide opportunities for people to learn and to develop skills for situations that are expensive, time-consuming, or dangerous. Careful design can support their learning by tailoring the features of situations to their levels of skill, allowing repeated attempts, and providing timely feedback. The same environments provide opportunities for assessing people's capabilities to act in these situations. This article describes an assessment design framework that can help projects develop effective simulation-based assessments. It reviews the rationale and terminology of the "evidence-centered" assessment design framework, discusses how it aligns with the principles of simulation design, and illustrates ideas with examples from engineering and medicine. Advice is offered for designing a new simulation-based assessment and for adapting an existing simulation system for assessment purposes.

Using the Guttman scale to define and estimate measurement error in items over time: the case of cognitive decline and the meaning of "points lost".

We used a Guttman model to represent responses to test items over time as an approximation of what is often referred to as "points lost" in studies of cognitive decline or interventions. To capture this meaning of "point loss", over four successive assessments, we assumed that once an item is incorrect, it cannot be correct at a later visit. If the loss of a point represents actual decline, then failure of an item to fit the Guttman model over time can be considered measurement error. This representation and definition of measurement error also permits testing the hypotheses that measurement error is constant for items in a test, and that error is independent of "true score", which are two key consequences of the definition of "measurement error"--and thereby, reliability--under Classical Test Theory. We tested the hypotheses by fitting our model to, and comparing our results from, four consecutive annual evaluations in three groups of elderly persons: a) cognitively normal (NC, N = 149); b) diagnosed with possible or probable AD (N = 78); and c) cognitively normal initially and a later diagnosis of AD (converters, N = 133). Of 16 items that converged, error-free measurement of "cognitive loss" was observed for 10 items in NC, eight in converters, and two in AD. We found that measurement error, as we defined it, was inconsistent over time and across cognitive functioning levels, violating the theory underlying reliability and other psychometric characteristics, and key regression assumptions.