Relating violations of measurement invariance to group differences in response times.

Measurement invariance is an assumption underlying the regression of a latent variable on a background variable. It requires the measurement model parameters of the latent variable to be equal across the levels of the background variable. Item-specific violations of this assumption are referred to as differential item functioning and are ideally substantively explainable to warrant theoretically valid and meaningful results. Past research has focused on developing statistical approaches to explain differential item functioning effects in terms of item- or person-specific covariates. In this study, we propose a modeling approach that can be used to test if differences in item response times can be used to statistically explain differential item functioning. To this end, we operationalize a latent response process factor and test if item-specific group differences on this factor can account for the observed differences in item scores. We investigate the properties of the model in a simulation study, and we apply the model to a real data set. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Understanding resident ratings of teaching in the workplace: a multi-centre study.

Providing clinical teachers with feedback about their teaching skills is a powerful tool to improve teaching. Evaluations are mostly based on questionnaires completed by residents. We investigated to what extent characteristics of residents, clinical teachers, and the clinical environment influenced these evaluations, and the relation between residents' scores and their teachers' self-scores. The evaluation and feedback for effective clinical teaching questionnaire (EFFECT) was used to (self)assess clinical teachers from 12 disciplines (15 departments, four hospitals). Items were scored on a five-point Likert scale. Main outcome measures were residents' mean overall scores (MOSs), specific scale scores (MSSs), and clinical teachers' self-evaluation scores. Multilevel regression analysis was used to identify predictors. Residents' scores and self-evaluations were compared. Residents filled in 1,013 questionnaires, evaluating 230 clinical teachers. We received 160 self-evaluations. 'Planning Teaching' and 'Personal Support' (4.52, SD .61 and 4.53, SD .59) were rated highest, 'Feedback Content' (CanMEDS related) (4.12, SD .71) was rated lowest. Teachers in affiliated hospitals showed highest MOS and MSS. Medical specialty did not influence MOS. Female clinical teachers were rated higher for most MSS, achieving statistical significance. Residents in year 1-2 were most positive about their teachers. Residents' gender did not affect the mean scores, except for role modeling. At group level, self-evaluations and residents' ratings correlated highly (Kendall's τ 0.859). Resident evaluations of clinical teachers are influenced by teacher's gender, year of residency training, type of hospital, and to a lesser extent teachers' gender. Clinical teachers and residents agree on strong and weak points of clinical teaching.

Repeated evaluations of the quality of clinical teaching by residents.

Many studies report on the validation of instruments for facilitating feedback to clinical supervisors. There is mixed evidence whether evaluations lead to more effective teaching and higher ratings. We assessed changes in resident ratings after an evaluation and feedback session with their supervisors. Supervisors of three medical specialities were evaluated, using a validated instrument (EFFECT). Mean overall scores (MOS) and mean scale scores were calculated and compared using paired T-tests. 24 Supervisors from three departments were evaluated at two subsequent years. MOS increased from 4.36 to 4.49. The MOS of two scales showed an increase >0.2: 'teaching methodology' (4.34-4.55), and 'assessment' (4.11-4.39). Supervisors with an MOS <4.0 at year 1 (n = 5) all demonstrated a strong increase in the MOS (mean overall increase 0.50, range 0.34-0.64). Four supervisors with an MOS between 4.0 and 4.5 (n = 6) demonstrated an increase >0.2 in their MOS (mean overall increase 0.21, range -0.15 to 53). One supervisor with an MOS >4.5 (n = 13) demonstrated an increase >0.02 in the MOS, two demonstrated a decrease >0.2 (mean overall increase -0.06, range -0.42 to 0.42). EFFECT-S was associated with a positive change in residents' ratings of their supervisors, predominantly in supervisors with relatively low initial scores.

Precision of individual and composite performance scores: the ideal number of indicators in an indicator set.

BACKGROUND: In many countries, quality indicators are used to assess the quality of care of family practice. Such assessments need to have an adequate precision, so that the results can be interpreted correctly. However, a small sample size per practitioner can lead to inadequate precision. A possible solution could be to create composite performance scores. OBJECTIVES: To evaluate the relationship between sample size and precision. We examine whether a composite performance score has an increased precision and how many indicators are needed minimally to achieve this level of precision. RESEARCH DESIGN: We performed a descriptive statistical study on data from the medical records of 455 Dutch practices. We included 3 different conditions: diabetes (12 indicators), chronic obstructive pulmonary disease (4 indicators), and Cardiovascular Disease and Risk Management (9 indicators). RESULTS: For individual quality indicators, patient samples close to 100 are required to achieve even moderate precision (10 percentage points) on the performance scores. This number decreases substantially when a composite score is used. A composite derived from combining 5 to 7 indicators can provide much the same precision of measurement as one made up from a much larger number of indicators. CONCLUSIONS: The added value of a composite score depends on the a priori reasons for measuring quality. Our results indicate that especially for formative quality improvement a small number of carefully selected indicators can provide a sufficiently precise composite measure.

Influence of symptom expectancies on stair-climbing performance in chronic fatigue syndrome: effect of study context.

BACKGROUND: In patients with chronic fatigue syndrome (CFS), performance of physical activities may be affected by an anticipated increase in symptoms after these activities. Nijs et al. previously studied the influence of symptom expectancies and related psychological processes on the performance of an isolated physical activity [Nijs J, Meeus M, Heins M, Knoop H, Moorkens G, Bleijenberg G. Kinesiophobia, catastrophizing and anticipated symptoms before stair climbing in chronic fatigue syndrome: an experimental study. Disabil Rehabil 2012. doi: 10.3109/09638288.2011.641661 .]. PURPOSE: We aimed to validate the previous findings in a larger group of patients in a different setting. We also extended the possible underlying psychological processes studied. METHOD: In 49 CFS patients, we measured performance (duration and increase in heart rate) during self-paced climbing and descending of two floors of stairs. Before this task, patients rated experienced fatigue and anticipated fatigue after stair climbing. In addition, kinesiophobia, catastrophising and focusing on bodily symptoms were measured. Using correlational and regression analyses, we tested whether performance during stair climbing could be explained by experienced and anticipated fatigue and psychological factors. RESULTS: Longer duration of stair climbing correlated with higher anticipated fatigue, independently of sex, age, body mass index and fatigue before stair climbing. Focusing on bodily symptoms and fatigue-related catastrophising were related to anticipated fatigue. CONCLUSION: Symptom expectations affect the performance of physical activity in CFS patients, possibly through focusing on bodily symptoms and catastrophising. These findings partially contradict the findings of the previous study, which stresses the importance of study context in conducting this type of experiments (i.e., patient characteristics, instructions).

Evaluation and feedback for effective clinical teaching in postgraduate medical education: validation of an assessment instrument incorporating the CanMEDS roles.

BACKGROUND: Providing clinical teachers in postgraduate medical education with feedback about their teaching skills is a powerful tool to improve clinical teaching. A systematic review showed that available instruments do not comprehensively cover all domains of clinical teaching. We developed and empirically test a comprehensive instrument for assessing clinical teachers in the setting of workplace learning and linked to the CanMEDS roles. METHODS: In a Delphi study, the content validity of a preliminary instrument with 88 items was studied, leading to the construction of the EFFECT (evaluation and feedback for effective clinical teaching) instrument. The response process was explored in a pilot test and focus group research with 18 residents of 6 different disciplines. A confirmatory factor analyses (CFA) and reliability analyses were performed on 407 evaluations of 117 supervisors, collected in 3 medical disciplines (paediatrics, pulmonary diseases and surgery) of 6 departments in 4 different hospitals. RESULTS: CFA yielded an 11 factor model with a good to excellent fit and internal consistencies ranged from 0.740 to 0.940 per domain; 7 items could be deleted. CONCLUSION: The model of workplace learning showed to be a useful framework for developing EFFECT, which incorporates the CanMEDS competencies and proved to be valid and reliable.