Development and Testing of an Instrument for Summative Assessment of Practical Skill Performance: A Generalizability Theory Approach.

BACKGROUND AND PURPOSE: Many newly graduated nurses lack proficiency in practical skill performance. Presently, nursing students' practical skill is assessed by summative instruments with overarching items. The purpose of this study was to develop a more detailed instrument to use in summative assessment of nursing students' practical skill performance and to assess its psychometric properties. METHODS: A 50-item instrument was developed. Video-recorded performances were rated by experienced clinical supervisors. A multifacet measurement design was developed. Relevant parameters were estimated by generalizability analysis. RESULTS: Findings indicated that error of measurement were mainly caused by raters far more than by items. CONCLUSIONS: The present study suggested that summative assessment in realistic settings may not apply one rater only. Two to three/four raters appear necessary to dependably measure most skills.

The choice of product indicators in latent variable interaction models: post hoc analyses.

The unconstrained product indicator (PI) approach is a simple and popular approach for modeling nonlinear effects among latent variables. This approach leaves the practitioner to choose the PIs to be included in the model, introducing arbitrariness into the modeling. In contrast to previous Monte Carlo studies, we evaluated the PI approach by 3 post hoc analyses applied to a real-world case adopted from a research effort in social psychology. The measurement design applied 3 and 4 indicators for the 2 latent 1st-order variables, leaving the researcher with a choice among more than 4,000 possible PI configurations. Sixty so-called matched-pair configurations that have been recommended in previous literature are of special interest. In the 1st post hoc analysis we estimated the interaction effect for all PI configurations, keeping the real-world sample fixed. The estimated interaction effect was substantially affected by the choice of PIs, also across matched-pair configurations. Subsequently, a post hoc Monte Carlo study was conducted, with varying sample sizes and data distributions. Convergence, bias, Type I error and power of the interaction test were investigated for each matched-pair configuration and the all-pairs configuration. Variation in estimates across matched-pair configurations for a typical sample was substantial. The choice of specific configuration significantly affected convergence and the interaction test's outcome. The all-pairs configuration performed overall better than the matched-pair configurations. A further advantage of the all-pairs over the matched-pairs approach is its unambiguity. The final study evaluates the all-pairs configuration for small sample sizes and compares it to the non-PI approach of latent moderated structural equations.

The relationship between burnout and musculoskeletal pain in seven Norwegian occupational groups.

Occupational and gender differences were investigated in the relationship between burnout and musculoskeletal pain in the head, neck, shoulders, and back. Representative samples of lawyers, physicians, nurses, teachers, church ministers, bus drivers, and information technology workers in Norway (N=4507) were analyzed using structural equation modeling. The exhaustion dimension of burnout was positively associated with musculoskeletal pain in all groups, and the strength of the relationship ranged from moderate to strong. The disengagement dimension of burnout was negatively associated with musculoskeletal pain in five groups and only ranged from -0.15 to -0.42. Professional efficacy was slightly weaker, and inconsistently (i.e., both positively and negatively) associated with musculoskeletal pain in four of the groups. There were larger differences in the strength of the relationships between the seven occupational groups than between males and females within the same profession. Results suggest that burnout and musculoskeletal pain are related, but the strength of the associations varies according to gender and occupation. Overall, occupational factors appear to be stronger predictors of the co-occurrence of burnout and musculoskeletal pain than gender.

Generalizability studies of the Global Assessment of Functioning-Split version.

OBJECTIVE: The study aimed to use the Generalizability Theory to investigate the reliability and precision of the split version of the Global Assessment of Functioning (GAF). MATERIALS: Six case vignettes were assessed by 2 samples; one by 19 experienced and independent raters and another by 58 experienced raters from 8 different day-treatment units, evaluating both symptom and function scores of GAF. METHODS: Generalizability studies were conducted to disentangle relevant variance components accounting for error variance in GAF scores. Furthermore, decision studies were conducted to estimate the reliability of different measurement designs, as well as precision in terms of error tolerance ratio. RESULTS: Both symptom and function scores of GAF were found to be highly generalizable, and a measurement design of 2 raters per subject was found to be most efficient with respect to reliability, precision, and use of resources. CONCLUSION: Both symptom and function scores of GAF seem highly consistent across experienced raters.