Age-related Psychometric Dimensionality Using the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition Opioid Use Disorder Diagnostic Criteria.

OBJECTIVES: Age-related psychometric differences in Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) opioid use disorder (OUD) diagnostic criteria have been hypothesized, but not been tested. This study investigated DSM-5 OUD diagnostic criteria for age-related measurement noninvariance among younger adults (YAs) and middle/older adults (MOAs) with past 12-month nonmedical use of prescription opioids. METHODS: People who participated in the 2012-2013 National Epidemiologic Survey on Alcohol and Related Conditions III and reported past 12-month nonmedical use of prescription opioids were included. YAs were 18-49 years old, and MOAs were 50+ years old. Item response theory, differential item functioning (DIF), and differential test functioning were used to assess for age-related measurement noninvariance. RESULTS: One in 5 people met the DSM-5 OUD diagnostic criteria for OUD within the past 12 months, with the most endorsed criteria being tolerance (17.96%). DIF was identified for 3 criteria, including (1) taking opioids for longer or in larger doses than intended, (2) long periods spent obtaining/using/recovering from use, and (3) withdrawal. DIF was associated with the latent OUD severity needed to correctly endorse the criteria, with criteria being correctly endorsed at less severe levels of latent OUD for MOAs when compared with YAs. Differential test functioning analyses showed collectively the criteria had improved detection in MOAs when compared with YAs (P < 0.01). CONCLUSIONS: These findings suggest that there may be age-related variations in the DSM-5 OUD diagnostic criteria's ability to detect latent OUD. Future research should identify contributing factors and the influence it has on the accuracy of age-specific surveillance estimations.

Development of the Abbreviated Technology Anxiety Scale (ATAS).

The purpose of this research was to develop a short measure of technology anxiety and provide validity and reliability evidence for its use in a variety of studies in the social sciences. Technology anxiety is an emotion oriented towards a negative affect leading to the avoidance of information and communication technology (Wilson, 2018). We developed the Abbreviated Technology Anxiety Scale (ATAS) and applied measurement theory to provide validity and reliability evidence. We implemented the study in multiple phases that included expert panel reviews on the content and quality of the items, and three rounds of data collection and analyses to provide the needed evidence. The scores from the ATAS were found to have an internally consistent structure, as well as to correlate with other known measures of technology and anxiety. Our results support the use of the ATAS for low-stakes purposes in research studies and evaluations. A general discussion is provided looking at the potential applications of the ATAS and its relation to other existing measures.

Semisupervised Learning Method to Adjust Biased Item Difficulty Estimates Caused by Nonignorable Missingness in a Virtual Learning Environment.

In data collected from virtual learning environments (VLEs), item response theory (IRT) models can be used to guide the ongoing measurement of student ability. However, such applications of IRT rely on unbiased item parameter estimates associated with test items in the VLE. Without formal piloting of the items, one can expect a large amount of nonignorable missing data in the VLE log file data, and this is expected to negatively affect IRT item parameter estimation accuracy, which then negatively affects any future ability estimates utilized in the VLE. In the psychometric literature, methods for handling missing data have been studied mostly around conditions in which the data and the amount of missing data are not as large as those that come from VLEs. In this article, we introduce a semisupervised learning method to deal with a large proportion of missingness contained in VLE data from which one needs to obtain unbiased item parameter estimates. First, we explored the factors relating to the missing data. Then we implemented a semisupervised learning method under the two-parameter logistic IRT model to estimate the latent abilities of students. Last, we applied two adjustment methods designed to reduce bias in item parameter estimates. The proposed framework showed its potential for obtaining unbiased item parameter estimates that can then be fixed in the VLE in order to obtain ongoing ability estimates for operational purposes.

Development and Validation of a Questionnaire to Measure Chinese Preschool Teachers' Implementation of Social-Emotional Practices.

We describe the development and validation of the Social-Emotional Teaching Practices Questionnaire-Chinese (SETP-C), a self-report instrument designed to gather information about Chinese preschool teachers' implementation of social-emotional practices. Initially (study 1), 262 items for the SETP-C were generated. Content validation of these items was conducted separately with Chinese practice experts, research experts, and preschool teachers. Significant revisions were made to items based on theoretical evidence and empirical findings from initial content validation activities, which led to a 70-item version of the SETP-C. In study 2, preliminary psychometric integrity evidence and item characteristics of the SETP-C were gathered based on the data from a sample of 1,599 Chinese preschool teacher respondents. Results from confirmatory factor analyses suggested a seven-factor measurement model, and high internal consistency score reliability was documented for each dimension of the SETP-C. Results of item response theory graded response models further indicated adequate psychometric properties at the item level.

Predict or describe? How learning analytics dashboard design influences motivation and statistics anxiety in an online statistics course.

Based on the achievement goal theory, this experimental study explored the influence of predictive and descriptive learning analytics dashboards on graduate students' motivation and statistics anxiety in an online graduate-level statistics course. Participants were randomly assigned into one of three groups: (a) predictive dashboard, (b) descriptive dashboard, or (c) control (i.e., no dashboard). Measures of motivation and statistical anxiety were collected in the beginning and the end of the semester via the Motivated Strategies for Learning Questionnaire and Statistical Anxiety Rating Scale. Individual semi-structured interviews were used to understand learners' perceptions of the course and whether the use of the dashboards influenced the meaning of their learning experiences. Results indicate that, compared to the control group, the predictive dashboard significantly reduced learners' interpretation anxiety and had an effect on intrinsic goal orientation that depended on learners' lower or higher initial levels of intrinsic goal orientation. In comparison to the control group, both predictive and descriptive dashboards reduced worth of anxiety (negative attitudes towards statistics) for learners who started the course with higher levels of worth anxiety. Thematic analysis revealed that learners who adopted a more performance-avoidance goal orientation approach demonstrated higher levels of anxiety regardless of the dashboard used. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s11423-021-09998-z.

The Performance of the Semigeneralized Partial Credit Model for Handling Item-Level Missingness.

The semi-generalized partial credit model (Semi-GPCM) has been proposed as a unidimensional modeling method for handling not applicable scale responses and neutral scale responses, and it has been suggested that the model may be of use in handling missing data in scale items. The purpose of this study is to evaluate the ability of the unidimensional Semi-GPCM to aid in the recovery of person parameters from item response data in the presence of item-level missingness, and to compare the performance of the model with two other proposed methods for handling such missingness: a multidimensional modeling approach for missingness and full information maximum likelihood estimation. The results indicate that the Semi-GPCM performs acceptably in an absolute sense when less than 30% of the item data is missing but does not outperform the other two methods under any particular conditions. We conclude with a discussion about when practitioners may or may not want to use the Semi-GPCM to recover person parameters from item response data with missingness.

Applying Unidimensional Models for Semiordered Data to Scale Data With Neutral Responses.

The purpose of this study is to evaluate whether a recently developed semiordered model can be used to explore the functioning of neutral response options in rating scale data. Huggins-Manley, Algina, and Zhou developed a class of unidimensional models for semiordered data within scale items (i.e., items with both ordered response categories and an additional nominal response category) and found promising results when applying them to scale data with Not Applicable response categories. In this study, we extended the application of the semi-partial credit model (PCM) to evaluate whether the semi-PCM can be used to calibrate potentially unordered neutral responses in rating scale data, and if so, how the approach compares with alternate methods of dealing with the neutral response option. Findings indicate that the semi-PCM can (a) assist practitioners in evaluating the ordered or unordered nature of neutral responses and (b) provide a viable alternative for θ estimation in the presence of an unordered neutral category. The process used in this study also provides a methodological framework for researchers and practitioners to use when dealing with neutral responses in their own data.

Imputation Methods to Deal With Missing Responses in Computerized Adaptive Multistage Testing.

Routing examinees to modules based on their ability level is a very important aspect in computerized adaptive multistage testing. However, the presence of missing responses may complicate estimation of examinee ability, which may result in misrouting of individuals. Therefore, missing responses should be handled carefully. This study investigated multiple missing data methods in computerized adaptive multistage testing, including two imputation techniques, the use of full information maximum likelihood and the use of scoring missing data as incorrect. These methods were examined under the missing completely at random, missing at random, and missing not at random frameworks, as well as other testing conditions. Comparisons were made to baseline conditions where no missing data were present. The results showed that imputation and the full information maximum likelihood methods outperformed incorrect scoring methods in terms of average bias, average root mean square error, and correlation between estimated and true thetas.

Evaluating the Accuracy of the Empirical Item Characteristic Curve Preequating Method in the Presence of Test Speededness.

This study aimed to assess the accuracy of the empirical item characteristic curve (EICC) preequating method given the presence of test speededness. The simulation design of this study considered the proportion of speededness, speededness point, speededness rate, proportion of missing on speeded items, sample size, and test length. After crossing all of the manipulated factors and then normalizing the evaluation criteria (bias and root mean square difference [RMSD]) with regard to test length, the results revealed that (1) when test speededness was present, conversions from the EICC preequating method tended to be positively distorted; (2) no practically meaningful moderation effect associated with sample size was found on the relationship between test speededness and the accuracy of EICC preequating; and (3) the location of the speededness point was the driving factor in terms of its impact on the accuracy of EICC preequating. Implications and suggestions were discussed.

Retrofitting Diagnostic Classification Models to Responses From IRT-Based Assessment Forms.

Developing a diagnostic tool within the diagnostic measurement framework is the optimal approach to obtain multidimensional and classification-based feedback on examinees. However, end users may seek to obtain diagnostic feedback from existing item responses to assessments that have been designed under either the classical test theory or item response theory frameworks. Retrofitting diagnostic classification models to existing assessments designed under other psychometric frameworks could be a plausible approach to obtain more actionable scores or understand more about the constructs themselves. This study (a) discusses the possibility and problems of retrofitting, (b) proposes a step-by-step retrofitting framework, and (c) explores the information one can gain from retrofitting through an empirical application example. While retrofitting may not always be an ideal approach to diagnostic measurement, this article aims to invite discussions through presenting the possibility, challenges, process, and product of retrofitting.

Psychometric Consequences of Subpopulation Item Parameter Drift.

This study defines subpopulation item parameter drift (SIPD) as a change in item parameters over time that is dependent on subpopulations of examinees, and hypothesizes that the presence of SIPD in anchor items is associated with bias and/or lack of invariance in three psychometric outcomes. Results show that SIPD in anchor items is associated with a lack of invariance in dimensionality structure of an anchor test, a lack of invariance in scaling coefficients across subpopulations, and a lack of invariance in ability estimates. It is demonstrated that these effects go beyond what can be understood from item parameter drift or differential item functioning.

The Impact of Q-Matrix Designs on Diagnostic Classification Accuracy in the Presence of Attribute Hierarchies.

There is an increasing demand for assessments that can provide more fine-grained information about examinees. In response to the demand, diagnostic measurement provides students with feedback on their strengths and weaknesses on specific skills by classifying them into mastery or nonmastery attribute categories. These attributes often form a hierarchical structure because student learning and development is a sequential process where many skills build on others. However, it remains to be seen if we can use information from the attribute structure and work that into the design of the diagnostic tests. The purpose of this study is to introduce three approaches of Q-matrix design and investigate their impact on classification results under different attribute structures. Results indicate that the adjacent approach provides higher accuracy in a shorter test length when compared with other Q-matrix design approaches. This study provides researchers and practitioners guidance on how to design the Q-matrix in diagnostic tests, which are in high demand from educators.

SEAsic: R Package for Score Equity Assessment.

SEAsic (score equity assessment-summary index computation) is an R package for computing and graphing a variety of indices that quantify an important aspect of test fairness, that of reported score equity. Historically, test fairness has been statistically defined as a lack of differential predication and/or a presence of measurement invariance at the item level. More recent definitions of fairness include the concept of score equity, which calls for additional subpopulation analysis at the equated, reported test score level. SEAsic allows for efficient calculation and graphing of multiple score equity assessment (SEA) indices. All indices in Huggins and Penfield (2012) can be calculated and plotted in various ways given a user-provided conversion table. SEAsic is freely available on the Comprehensive R Archive Network (CRAN). Mac, Windows, and Linux users have access to the package via downloading the appropriate version of R or RStudio from the CRAN website. Multiple examples of each index computation, variations on each index, and plot options are provided in the package manual on CRAN.