Functional Approaches for Modeling Unfolding Data.

The purpose of this study is to introduce a functional approach for modeling unfolding response data. Functional data analysis (FDA) has been used for examining cumulative item response data, but a functional approach has not been systematically used with unfolding response processes. A brief overview of FDA is presented and illustrated within the context of unfolding data. Seven decision parameters are described that can provide a guide to conducting FDA in this context. These decision parameters are illustrated with real data using two scales that are designed to measure attitude toward capital punishment and attitude toward censorship. The analyses suggest that FDA offers a useful set of tools for examining unfolding response processes.

Assessing the Validity and Reliability of the Effects of Youngsters' Eyesight on Quality of Life Questionnaire Among Children With Uveitis.

OBJECTIVE: The Effects of Youngsters' Eyesight on Quality of Life (EYE-Q) questionnaire measures vision-related functioning (VRF) and vision-related quality of life (VRQoL) in children with uveitis. Our aim was to revise the alpha version of the EYE-Q to refine VRF and VRQoL subscales and to assess the validity of the EYE-Q. METHODS: Children with juvenile idiopathic arthritis (JIA), JIA-associated uveitis, and other noninfectious uveitis were enrolled. Patients and parents completed the EYE-Q, Pediatric Quality of Life Inventory (overall quality of life), and Childhood Health Assessment Questionnaire (physical functioning). The development site completed the alpha version of the EYE-Q, and the composite sites completed the beta version. We compared item-subscale correlations, internal consistency, and construct and discriminant validity among the different versions. RESULTS: Of the 644 patients enrolled, 61.6% completed the alpha version, and 38.4% the beta version of the EYE-Q. Mean ± SD patient age was 11.1 ± 4.2 years, and 70% were female. Fewer White patients (73.5%) completed the alpha version compared to the beta version (86.2%; P < 0.001). With the exception of patient-reported VRF, both versions had similar item-subscale correlations. Version comparisons on scale internal consistencies indicated significant differences for parent- and patient-reported VRF, but each scale had a Cronbach's α of >0.80 beta. When data were combined, the EYE-Q showed significant differences between JIA-only and uveitis patients on all parent and patient scores, except for patient-reported VRF. CONCLUSION: The EYE-Q appears to be a valid measure of VRF and VRQoL in pediatric uveitis. Our results suggest it may be used as an outcome measure in multicenter pediatric uveitis studies.

Comprehensive Assessment of Quality of Life, Functioning, and Mental Health in Children With Juvenile Idiopathic Arthritis and Noninfectious Uveitis.

OBJECTIVE: Pediatric uveitis can lead to sight-threatening complications and can impact quality of life (QoL) and functioning. We aimed to examine health-related QoL, mental health, physical disability, vision-related functioning (VRF), and vision-related QoL in children with juvenile idiopathic arthritis (JIA), JIA-associated uveitis (JIA-U), and other noninfectious uveitis. We hypothesized that there will be differences based on the presence of eye disease. METHODS: A multicenter cross-sectional study was conducted at four sites. Patients with JIA, JIA-U, or noninfectious uveitis were enrolled. Patients and parents completed the Pediatric Quality of Life Inventory (PedsQL; health-related QoL), the Revised Childhood Anxiety and Depression Scale (RCADS; anxiety/depression), the Childhood Health Assessment Questionnaire (C-HAQ; physical disability), and the Effects of Youngsters' Eyesight on Quality of Life (EYE-Q) (VRF/vision-related QoL). Clinical characteristics and patient-reported outcome measures were compared by diagnosis. RESULTS: Of 549 patients, 332 had JIA, 124 had JIA-U, and 93 had other uveitis diagnoses. Children with JIA-U had worse EYE-Q scores compared to those with JIA only. In children with uveitis, those with anterior uveitis (JIA-U and uveitis only) had less ocular complications, better EYE-Q scores, and worse C-HAQ and PedsQL physical summary scores compared to those with nonanterior disease. In children with anterior uveitis, those with JIA-U had worse PedsQL physical summary and C-HAQ scores than anterior uveitis only. Further, EYE-Q scores were worse in children with bilateral uveitis and more visual impairment. There were no differences in RCADS scores among groups. CONCLUSION: We provide a comprehensive outcome assessment of children with JIA, JIA-U, and other uveitis diagnoses. Differences in QoL and function were noted based on underlying disease. Our results support the addition of a vision-specific measure to better understand the impact of uveitis.

Evaluating Observer Ratings: The Case of Measuring Neighborhood Disorder.

The purpose of this study is to evaluate the quality of observer ratings of neighborhood disorder using a many-facet Rasch model (MFRM). Our goal is to investigate observer severity and observer consistency. Observers trained in the use of a systematic social observation visited and rated residential neighborhoods. Data for this study are drawn from the Family and Community Health Study (FACHS). The FACHS sample consisted of 673 neighborhoods. Two observers, out of a total of 67 observers used for this study, rated each residential neighborhood. The results of this study suggested that there were statistically significant differences in observer severity, even after observer training, and that the ratings of observers are not consistent. Therefore, more or better observer training is necessary. In addition, the interaction effect between observer and item was significant, indicating significant variance in observer severity across at least one item.

Examining Differential Item Functioning in the Household Food Insecurity Scale: Does Participation in SNAP Affect Measurement Invariance?

The Household Food Security Survey Module (HFSSM) is a scale used by the U.S. Department of Agriculture to measure the severity of food insecurity experienced by U.S. households. In this study, measurement invariance of the HFSSM is examined across households based on participation in the Supplemental Nutrition Assistance Program (SNAP). Households with children who responded to the HFSSM in 2015 and 2016 (N = 3,931) are examined. The Rasch model is used to analyze differential item functioning (DIF) related to SNAP participation. Analyses suggest a small difference in reported food insecurity between SNAP and non-SNAP participants (27% versus 23% respectively). However, the size and direction of the DIF mitigates the impact on overall estimates of household food insecurity. Person-fit indices suggest that the household aberrant response rate is 6.6% and the number of misfitting households is comparable for SNAP (6.80%) and non-SNAP participants (6.30%). Implications for research and policy related to food insecurity are discussed.

Examining Rater Judgements in Music Performance Assessment using Many-Facets Rasch Rating Scale Measurement Model.

The fairness of raters in music performance assessment has become an important concern in the field of music. The assessment of students' music performance depends in a fundamental way on rater judgements. The quality of rater judgements is crucial to provide fair, meaningful and informative assessments of music performance. There are many external factors that can influence the quality of rater judgements. Previous research has used different measurement models to examine the quality of rater judgements (e.g., generalizability theory). There are limitations with the previous analysis methods that are based on classical test theory and its extensions. In this study, we use modern measurement theory (Rasch measurement theory) to examine the quality of rater judgements. The many-facets Rasch rating scale model is employed to investigate the extent of rater-invariant measurement in the context of music performance assessments related to university degrees in Malaysia (159 students rated by 24 raters). We examine the rating scale structure, the severity levels of the raters, and the judged difficulty of the items. We also examine the interaction effects across musical instrument subgroups (keyboard, strings, woodwinds, brass, percussions and vocal). The results suggest that there were differences in severity levels among the raters. The results of this study also suggest that raters had different severity levels when rating different musical instrument subgroups. The implications for research, theory and practice in the assessment of music performance are included in this paper.

Exploring the Impersonal Judgments and Personal Preferences of Raters in Rater-Mediated Assessments With Unfolding Models.

The purpose of this study is to explore the use of unfolding models for evaluating the quality of ratings obtained in rater-mediated assessments. Two different judgmental processes can be used to conceptualize ratings: impersonal judgments and personal preferences. Impersonal judgments are typically expected in rater-mediated assessments, and these ratings reflect a cumulative response process. However, raters may also be influenced by their personal preferences in providing ratings, and these ratings may reflect a noncumulative or unfolding response process. The goal of rater training in rater-mediated assessments is to stress impersonal judgments represented by scoring rubrics and to minimize the personal preferences that may represent construct-irrelevant variance in the assessment system. In this study, we explore the use of unfolding models as a framework for evaluating the quality of ratings in rater-mediated assessments. Data from a large-scale assessment of writing in the United States are used to illustrate our approach. The results suggest that unfolding models offer a useful way to evaluate rater-mediated assessments in order to initially explore the judgmental processes underlying the ratings. The data also indicate that there are significant relationships between some essay features (e.g., word count, syntactic simplicity, word concreteness, and verb cohesion) and essay orderings based on the personal preferences of raters. The implications of unfolding models for theory and practice in rater-mediated assessments are discussed.

Using Household Fit Indices to Examine the Psychometric Quality of Food Insecurity Measures.

This study focuses on model-data fit with a particular emphasis on household-level fit within the context of measuring household food insecurity. Household fit indices are used to examine the psychometric quality of household-level measures of food insecurity. In the United States, measures of food insecurity are commonly obtained from the U.S. Household Food Security Survey Module (HFSSM, 18 items) of the Current Population Survey Food Security Supplement (CPS-FSS). These measures, in various forms, are used to inform national programs and policies related to food insecurity. Data for low-income households with children from recent administrations of the HFSSM (2012-2014) are used in this study (N = 7,324). The results suggest that there are detectable levels of misfit with Infit mean square error (MSE) statistics ranging from 6.73 % to 21.33% and Outfit MSE statistics ranging from 5.31% to 9.68%. The data suggest for Outfit MSE statistics that (a) male respondents, (b) respondents with lower levels of education, and (c) respondents who did not report participating in SNAP (Supplemental Nutrition Assistance Program, formerly the Food Stamp Program) tend to have more misfit. For Infit MSE statistics, lack of homeownership appears to be a predictor of misfit. The implications of this research for future research, theory, and policy related to the measurement of household food insecurity are discussed.

Evaluating Model-Data Fit by Comparing Parametric and Nonparametric Item Response Functions: Application of a Tukey-Hann Procedure.

This study describes an approach for examining model-data fit for the dichotomous Rasch model using Tukey-Hann item response functions (TH-IRFs). The procedure proposed in this paper is based on an iterative version of a smoothing technique proposed by Tukey (1977) for estimating nonparametric item response functions (IRFs). A root integrated squared error (RISE) statistic (Douglas and Cohen, 2001) is used to compare the TH-IRFs to the Rasch IRFs. Data from undergraduate students at a large university are used to demonstrate this iterative smoothing technique. The RISE statistic is used for comparing the item response functions to assess model-data fit. A comparison between the residual based Infit and Outfit statistics and RISE statistics are also examined. The results suggest that the RISE statistic and TH-IRFs provide a useful analytical and graphical approach for evaluating item fit. Implications for research, theory and practice related to model-data fit are discussed.

Incorporating Criterion Ratings Into Model-Based Rater Monitoring Procedures Using Latent-Class Signal Detection Theory.

This study presents a new criterion-referenced approach for exploring rating quality within the framework of latent-class signal detection theory (LC-SDT) that goes beyond commonly used reliability indices, and provides substantively meaningful indicators of rater accuracy that can be used to inform rater training and monitoring at the individual rater level. Specifically, this study illustrates a flexible application of restricted LC-SDT modeling, in which restrictions can be specified for the true latent classification to reflect the unique characteristics of a particular assessment context. While the LC-SDT modeling framework provides immediately useful characterizations of raters' behavior, the restricted LC-SDT offers complementary evidence to further support the monitoring of rater behavior by bringing criterion ratings to bear. This study uses ratings from a large-scale writing assessment, and findings suggest that the criterion (i.e., restricted) LC-SDT provides useful information about rating quality for operational raters relative to criterion ratings, which may ultimately inform rater training and monitoring procedures.

Exploring Aberrant Responses Using Person Fit and Person Response Functions.

Person fit statistics provide equivocal interpretations regarding aberrant responses. This study uses person response functions (PRF) to supplement the interpretation of person fit statistics. Sixty-three multiple-choice items were administered to a sample of persons (N=31) who used guessing strategies to answer them. After answering each item, participants indicated which guessing strategy they used. The data were analyzed with a Rasch (1960) model, where the item calibrations were anchored to values obtained when the items were appropriately administered. The participants showed poor model-data fit as expected. Further examination of person misfit using person response functions suggests that PRF can provide information about absolute person fit to a model, whereas fit statistics provide information about relative fit, given the other persons in the testing group. PRF can also provide information about where and how person responses misfit the model. This additional information can assist practitioners in using and interpreting individual scores appropriately.

Exploring Rating Quality in Rater-Mediated Assessments Using Mokken Scale Analysis.

Mokken scale analysis is a probabilistic nonparametric approach that offers statistical and graphical tools for evaluating the quality of social science measurement without placing potentially inappropriate restrictions on the structure of a data set. In particular, Mokken scaling provides a useful method for evaluating important measurement properties, such as invariance, in contexts where response processes are not well understood. Because rater-mediated assessments involve complex interactions among many variables, including assessment contexts, student artifacts, rubrics, individual rater characteristics, and others, rater-assigned scores are suitable candidates for Mokken scale analysis. The purposes of this study are to describe a suite of indices that can be used to explore the psychometric quality of data from rater-mediated assessments and to illustrate the substantive interpretation of Mokken-based statistics and displays in this context. Techniques that are commonly used in polytomous applications of Mokken scaling are adapted for use with rater-mediated assessments, with a focus on the substantive interpretation related to individual raters. Overall, the findings suggest that indices of rater monotonicity, rater scalability, and invariant rater ordering based on Mokken scaling provide diagnostic information at the level of individual raters related to the requirements for invariant measurement. These Mokken-based indices serve as an additional suite of diagnostic tools for exploring the quality of data from rater-mediated assessments that can supplement rating quality indices based on parametric models.

Evaluating Rater Accuracy in Rater-Mediated Assessments Using an Unfolding Model.

The number of performance assessments continues to increase around the world, and it is important to explore new methods for evaluating the quality of ratings obtained from raters. This study describes an unfolding model for examining rater accuracy. Accuracy is defined as the difference between observed and expert ratings. Dichotomous accuracy ratings (0 = inaccurate, 1 = accurate) are unfolded into three latent categories: inaccurate below expert ratings, accurate ratings, and inaccurate above expert ratings. The hyperbolic cosine model (HCM) is used to examine dichotomous accuracy ratings from a statewide writing assessment. This study suggests that HCM is a promising approach for examining rater accuracy, and that the HCM can provide a useful interpretive framework for evaluating the quality of ratings obtained within the context of rater-mediated assessments.

A dyadic model of living with epilepsy based on the perspectives of adults with epilepsy and their support persons.

Epilepsy is a chronic condition that significantly affects the lives of individuals with epilepsy and their support persons, though few studies have examined the experiences of both. To examine these experiences and explore the interpersonal relationships between dyad members, we conducted in-depth interviews with 22 persons with epilepsy and 16 support persons. Data analysis was guided by a grounded theory perspective. We developed a model that shows how epilepsy impacts the lives of both persons with epilepsy and their support persons and how the experiences of persons with epilepsy and supporters influence one another. The core model elements were seizure and treatment factors, relationship characteristics, self-management, seizure control, support provided, illness intrusiveness, and quality of life. Persons with epilepsy moved through the model in five trajectories depending on seizure control, relationship type, and gender. Support providers followed four trajectories based on seizure control, perception of burden, and support for themselves. Persons with epilepsy and their primary support providers have varied experiences in how epilepsy affects their lives. This model could serve as a basis for future research and intervention efforts focused on ways to reduce illness intrusiveness and improve quality of life for persons with epilepsy and their supporters.

Perspectives of adults with epilepsy and their support persons on self-management support.

Social support is an important mechanism for improving self-management, although little is known about its role in epilepsy self-management. We examined the type of support provided to people with epilepsy and its influence on self-management. We conducted in-depth interviews with 22 people with epilepsy and 16 support persons, representing 14 pairs and 10 unpaired individuals. We analyzed the data using principles of grounded theory. Supporters, who were mainly parents and spouses, aided people with epilepsy in every dimension of self-management. Support for self-management occurred along a continuum from person with epilepsy-led management to support person-led management. Where the pairs fell on the continuum depended on developmental stage, relationship type, and relationship dynamics. Seizure control shaped individuals' experiences with self-management and support within each group. The self-management continuum provides a new aspect that can be integrated into existing models of self- and family management.

A mixed methods analysis of support for self-management behaviors: perspectives of people with epilepsy and their support providers.

Social support is associated with improved self-management for people with chronic conditions, such as epilepsy; however, little is known about the perceived ease or difficulty of receiving and providing support for epilepsy self-management. We examined patterns of epilepsy self-management support from the perspectives of both people with epilepsy and their support persons. Fifty-three people with epilepsy and 48 support persons completed a survey on epilepsy self-management support. Of these individuals, 22 people with epilepsy and 16 support persons completed an in-depth interview. Rasch measurement models were used to evaluate the degree of difficulty of receiving or providing support often for nine self-management tasks. We analyzed model-data fit, person and item location along the support latent variable and differential person and item functioning. Qualitative methods were used to provide context and insight into the quantitative results. The results demonstrated good model-data fit. Help with seizures was the easiest type of support to receive or provide more often, followed by rides to a doctor's appointments and help avoiding seizure triggers. The most difficult types of support to receive or provide more often were reminders, particularly for taking and refilling medications. While most participants' responses fit the model, responses of several individuals misfit the model. Person misfit generally occurred because the scale items did not adequately capture some individuals' behaviors. These results could be useful in designing interventions that use support as a means of improving self-management. Additionally, the results provide information to improve or expand current measures of support for epilepsy self-management to better assess the experiences of people with epilepsy and their support persons.

Examining rating quality in writing assessment: rater agreement, error, and accuracy.

The use of performance assessments in which human raters evaluate student achievement has become increasingly prevalent in high-stakes assessment systems such as those associated with recent policy initiatives (e.g., Race to the Top). In this study, indices of rating quality are compared between two measurement perspectives. Within the context of a large-scale writing assessment, this study focuses on the alignment between indices of rater agreement, error, and accuracy based on traditional and Rasch measurement theory perspectives. Major empirical findings suggest that Rasch-based indices of model-data fit for ratings provide information about raters that is comparable to direct measures of accuracy. The use of easily obtained approximations of direct accuracy measures holds significant implications for monitoring rating quality in large-scale rater-mediated performance assessments.

Exploring the alignment of writing self-efficacy with writing achievement using rasch measurement theory and qualitative methods.

Alignment of writing self-efficacy and writing achievement is defined as the congruence between student confidence regarding writing skills (writing self-efficacy) and the actual performance on these writing skills as reflected in teacher grades (achievement). One purpose of this study is to examine the relationship between these two variables. A second purpose is to demonstrate a mixed-methods approach to investigating relationships between affective variables using Rasch measurement and interviews. Participants were eighth grade students (N = 94) from an ethnically and socioeconomically diverse school in the southeastern United States. Our results suggest that students who struggle with the mechanics of writing yet appreciate the expressive capacity of writing, may have higher senses of writing self-efficacy that are not predictive of performance.

Using Rasch measurement theory to assess three depression scales among adults with epilepsy.

People with chronic conditions, such as epilepsy, are at a high risk for depression; however depression is often under-recognized and undertreated. Depression scales, including one specific to people with epilepsy, have been used for screening in this population, although none have been assessed with Rasch measurement theory. This study used Rasch analyses in order to evaluate and compare the psychometric properties of the modified Beck Depression Inventory, the Patient Health Questionnaire, and the Neurological Disorders Depression Inventory for Epilepsy in a sample of people with epilepsy and depression who participated in an intervention designed to reduce depressive symptoms. A secondary purpose was to assess item functioning across time. The sample includes 44 participants in the Project UPLIFT program who completed the assessments before and after taking part in the intervention. Results of the Rasch analysis indicate that the three depression scales functioned as intended. There was good overall targeting between the items and the sample, acceptable model-data fit, and good reliability of separation for persons, items, and time. The participants experienced a significant decrease in depressive symptoms from pretest to posttest. This study illustrates the value of using model-based measurement with the Rasch model to combine items across the three depression scales. It also demonstrates an approach for analyzing and evaluating the results of small scale intervention programs, such as the UPLIFT program.