Examining the Impact of Differential Item Functioning on Classification Accuracy in Cognitive Diagnostic Models.

Cognitive diagnostic models (CDMs) are of growing interest in educational research because of the models' ability to provide diagnostic information regarding examinees' strengths and weaknesses suited to a variety of content areas. An important step to ensure appropriate uses and interpretations from CDMs is to understand the impact of differential item functioning (DIF). While methods of detecting DIF in CDMs have been identified, there is a limited understanding of the extent to which DIF affects classification accuracy. This simulation study provides a reference to practitioners to understand how different magnitudes and types of DIF interact with CDM item types and group distributions and sample sizes to influence attribute- and profile-level classification accuracy. The results suggest that attribute-level classification accuracy is robust to DIF of large magnitudes in most conditions, while profile-level classification accuracy is negatively influenced by the inclusion of DIF. Conditions of unequal group distributions and DIF located on simple structure items had the greatest effect in decreasing classification accuracy. The article closes by considering implications of the results and future directions.

Examining DIF in the Context of CDMs When the Q-Matrix Is Misspecified.

The rise in popularity and use of cognitive diagnostic models (CDMs) in educational research are partly motivated by the models' ability to provide diagnostic information regarding students' strengths and weaknesses in a variety of content areas. An important step to ensure appropriate interpretations from CDMs is to investigate differential item functioning (DIF). To this end, the current simulation study examined the performance of three methods to detect DIF in CDMs, with particular emphasis on the impact of Q-matrix misspecification on methods' performance. Results illustrated that logistic regression and Mantel-Haenszel had better control of Type I error than the Wald test; however, high power rates were found using logistic regression and Wald methods, only. In addition to the tradeoff between Type I error control and acceptable power, our results suggested that Q-matrix complexity and item structures yield different results for different methods, presenting a more complex picture of the methods' performance. Finally, implications and future directions are discussed.

Factors associated with practicing evidence-based medicine: a study of family medicine residents.

BACKGROUND: Evidence-based medicine (EBM) plays a critical part in ensuring that practitioners use the soundest available medical procedures while avoiding ineffective ones. As such, it plays a key role in medical residency education. However, little research has shown what factors influence residents' adoption of habits in, self-efficacy in, and skills of EBM. MATERIALS AND METHODS: This study gathered responses from a cross section of family medicine residents and new interns from 40 different residencies across the USA. The survey was based on Taylor et al's survey of EBM attitudes and behaviors and the Fresno test's assessment of EBM knowledge and skills. The study used negative binomial regression, ordinary least squares regression, and nonparametric tests of difference to assess the impact of residents' background (year in residency, type of residency, previous EBM training, and previous research experience) on these EBM outcomes. RESULTS: Residents with previous research experience are associated with stronger EBM habits, more self-efficacy in applying EBM, and greater ability in using EBM skills. Previous research experience had a bigger impact on these outcomes than any other predictor. EBM habits, self-efficacy, and skills did not appear to show even increases by year in residency. Previous EBM training was associated with more hours spent reading the literature and higher EBM skill test scores. CONCLUSION: Our findings suggest the practice of EBM may benefit from medical education increasing research experiences and EBM training. Research experiences provide the practical training, while EBM training provides focused instruction necessary for EBM self-efficacy, habits, and skills. These EBM outcomes are not inherently gained through time in family medicine residency. Future research, particularly longitudinal designs, should continue to pursue this line of inquiry.

Membership categorization analysis as an important qualitative method in evaluation.

This paper presents Membership Categorization Analysis (MCA) as a useful qualitative methodology in evaluation study. MCA is closely connected to Conversation Analysis and is most frequently used in Sociology studies. MCA provides evaluators unique insights into the boundaries different categories (e.g. teachers, students) place on themselves and their actions. The paper includes an illustrated example of the value added of MCA in an evaluation of an early childhood educator improvement program.