Development and Testing of an Instrument for Summative Assessment of Practical Skill Performance: A Generalizability Theory Approach.

BACKGROUND AND PURPOSE: Many newly graduated nurses lack proficiency in practical skill performance. Presently, nursing students' practical skill is assessed by summative instruments with overarching items. The purpose of this study was to develop a more detailed instrument to use in summative assessment of nursing students' practical skill performance and to assess its psychometric properties. METHODS: A 50-item instrument was developed. Video-recorded performances were rated by experienced clinical supervisors. A multifacet measurement design was developed. Relevant parameters were estimated by generalizability analysis. RESULTS: Findings indicated that error of measurement were mainly caused by raters far more than by items. CONCLUSIONS: The present study suggested that summative assessment in realistic settings may not apply one rater only. Two to three/four raters appear necessary to dependably measure most skills.

Change mechanisms in psychotherapy: Both improved insight and improved affective awareness are necessary.

OBJECTIVE: Insight and affect awareness are correlated with outcome in a number of studies across different treatment orientations. In this study, we perform a full mediational analysis to examine whether improvement of both cognitive insight and affect awareness are mediators of the specific effects of transference work in dynamic psychotherapy. METHOD: This was a dismantling randomized controlled clinical trial specifically designed to study long-term effects of transference work (exploration of problematic patterns in the therapeutic relationship). One hundred outpatients were randomly assigned to 1 year of dynamic psychotherapy with and without transference work interventions. The outcome variables were the Interpersonal Functioning scale (clinician rated) and Inventory of Interpersonal Problems (patient self-report). Quality of Object Relations was moderator. Using structural equation modeling, we tested change during treatment (1 year) of Insight and Tolerance for Affects as mediators of long-term outcome (over the 1-year study period). RESULTS: For both outcome measures, the best model supported a mediated moderation model where the effects of transference work, for patients with low Quality of Object Relations, were mediated via both change of Insight and Tolerance for Affects. The effect of Insight on outcome was significantly reduced due to an indirect effect via Tolerance for Affects. A number of alternative models allowed us to rule out alternative pathways with some confidence. CONCLUSIONS: Both improved insight and affect awareness seem to be mechanisms for long-term effects of transference work. Our results bridge the gap between mainstream clinical theory and empirical research. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

The choice of product indicators in latent variable interaction models: post hoc analyses.

The unconstrained product indicator (PI) approach is a simple and popular approach for modeling nonlinear effects among latent variables. This approach leaves the practitioner to choose the PIs to be included in the model, introducing arbitrariness into the modeling. In contrast to previous Monte Carlo studies, we evaluated the PI approach by 3 post hoc analyses applied to a real-world case adopted from a research effort in social psychology. The measurement design applied 3 and 4 indicators for the 2 latent 1st-order variables, leaving the researcher with a choice among more than 4,000 possible PI configurations. Sixty so-called matched-pair configurations that have been recommended in previous literature are of special interest. In the 1st post hoc analysis we estimated the interaction effect for all PI configurations, keeping the real-world sample fixed. The estimated interaction effect was substantially affected by the choice of PIs, also across matched-pair configurations. Subsequently, a post hoc Monte Carlo study was conducted, with varying sample sizes and data distributions. Convergence, bias, Type I error and power of the interaction test were investigated for each matched-pair configuration and the all-pairs configuration. Variation in estimates across matched-pair configurations for a typical sample was substantial. The choice of specific configuration significantly affected convergence and the interaction test's outcome. The all-pairs configuration performed overall better than the matched-pair configurations. A further advantage of the all-pairs over the matched-pairs approach is its unambiguity. The final study evaluates the all-pairs configuration for small sample sizes and compares it to the non-PI approach of latent moderated structural equations.

Latent interaction effects in the theory of planned behaviour applied to quitting smoking.

OBJECTIVES: This study applies three latent interaction models in the theory of planned behaviour (TPB; Ajzen, 1988, Attitudes, personality, and behavior. Homewood, IL: Dorsey Press; Ajzen, 1991, Organ. Behav. Hum. Decis. Process., 50, 179) to quitting smoking: (1) attitude × perceived behavioural control on intention; (2) subjective norms (SN) × attitude on intention; and (3) perceived behavioural control × intention on quitting behaviour. METHODS: The data derive from a longitudinal Internet survey of 939 smokers aged 15-74 over a period of 4 months. Latent interaction effects were estimated using the double-mean-centred unconstrained approach (Lin et al., 2010, Struct. Equ. Modeling, 17, 374) in LISREL. RESULTS: Attitude × SN and attitude × perceived behavioural control both showed a significant interaction effect on intention. No significant interaction effect was found for perceived behavioural control × intention on quitting. CONCLUSIONS: The latent interaction approach is a useful method for investigating specific conditions between TPB components in the context of quitting behaviour. Theoretical and practical implications of the results are discussed.

Observations of families in structured interactions: parenting therapists provide reliable ratings of mothers' parenting.

The reliability of observations of parenting by parenting therapists was assessed. An important predictor of externalizing behavior in children is quality of parenting. Data were videotapes of structured interactions in families with a child age 8-12 years referred to the evidence based Parent Management Training Oregon (PMTO) treatment program for child behavior problems. The therapists had clinical PMTO training but no training in systematic observation. PMTO observational coders with specific coder training were included as a reference for the therapists. Five therapists and two coders observed videotapes of 10 families and performed global evaluations of mothers' parenting skills. They used the coder's impression measure used in PMTO research. Scores were analyzed in a generalizability theory framework for the two groups of observers separately. Both observer types reliably rank-ordered the mothers and assessed the level of parenting skills. PMTO therapists without coder training provided reliable ratings of parenting constructs relevant to the clinical PMTO program in a manner comparable to that of the trained reference coders.

Reliability of observers' subjective impressions of families: a generalizability theory approach.

Parenting was observed in videotaped interactions in 30 families referred for child conduct problems. Generalizability coefficients and the impact of varying numbers of raters were estimated. Two measurement designs were compared: All raters observed all families ("crossed" design) and a different rater observed each family ("nested" design). The crossed design provided higher generalizability coefficients than a nested design, implying inflated generalizability estimates if a crossed estimation model is used for a nested data collection. Three and four raters were needed to obtain generalizability coefficients in the .70-.80 range for monitoring and discipline, respectively. One rater was sufficient for a corresponding estimate for positive involvement and for an estimate in .80-.90 range for problem-solving. Estimates for skill encouragement were non-acceptable.

Psychodynamic group psychotherapy: impact of group length and therapist professional characteristics on development of therapeutic alliance.

OBJECTIVES: Little research has been done on therapeutic alliance in group psychotherapy, especially the impact of treatment duration and therapist professional characteristics. METHODS: Therapeutic alliance was rated by patients on the Working Alliance Inventory-Short Form at three time points (sessions 3, 10 and 17) in a randomized controlled trial of short-term and long-term psychodynamic group psychotherapy. As predictors we selected therapist clinical experience and length of didactic training, which have demonstrated ambiguous results in previous research. Linear latent variable growth curve models (structural equation modeling) were developed for the three Working Alliance Inventory-Short Form subscales bond, task and goal. RESULTS: We found a significant variance in individual growth curves (intercepts and slopes) but no differential development due to group length. Longer therapist formal training had a negative impact on early values of subscale task in both treatments. There was an interaction between length of the therapists' clinical experience and group length on early bond, task and goal: therapists with longer clinical experience were rated lower on initial bond in the long-term group but less so in the short-term group. Longer clinical experience influenced initial task and goal positively in the short-term group but was unimportant for task or significantly negative for goal in the long-term group. CONCLUSION: There was no mean development of alliance, and group length did not differentially impact the alliance during 6 months. Early ratings of the three Working Alliance Inventory-Short Form subscales partly reflected different preparations of patients in the two group formats, partly therapist characteristics, but more research is needed to see how these aspects impact alliance development and outcome. Therapists should pay attention to all three aspects of the alliance, when they prepare patients for group therapy. KEY PRACTITIONER MESSAGE: In psychodynamic groups, length of therapy does not differentiate the overall level or the development of member-leader alliance. Within psychodynamic groups, each individual appear to have their unique perception of the member-leader alliance. Therapists with longer formal psychotherapy training may be less successful in establishing early agreement with patients on the tasks of psychodynamic group psychotherapy. Patients perceive a somewhat lower degree of early emotional bonding with the more clinically experienced therapists in long-term psychodynamics groups. Therapists with more clinical experience may contribute to a stronger degree of initial agreement with patients on the tasks and goals of short-term group psychotherapy.

Level and change in alcohol consumption, depression and dysfunctional attitudes among females treated for alcohol addiction.

AIMS: To examine whether individual changes in alcohol consumption among female alcoholics under treatment are predicted by level of and changes in depression and dysfunctional attitudes. METHOD: A total of 120 women who were treated for alcohol addiction at the Karolinska Hospital in Stockholm (Sweden) were assessed twice over a 2-year period using the Depression scale from the Symptom Checklist-90, the Alcohol Use Inventory and the Dysfunctional Attitude Scale (DAS). Latent growth curve analysis was used. RESULTS: Decrease in alcohol consumption, depression and dysfunctional attitude variables were found at group level. The results also showed significant individual variation in change. Changes in alcohol consumption were predicted by baseline alcohol drinking, as well as by level and changes in depression. Stronger reduction in depression was related to higher level of depression at baseline, and with reduction in dysfunctional attitudes. Different DAS sub-scales resulted in different magnitude of the model relations. Good treatment compliance was related to lower baseline level in depression, but also with higher baseline level in dysfunctional attitudes, and predicted stronger reduction in alcohol consumption. CONCLUSION: This paper shows the importance of incorporating both individual level and change in depression as predictors of change in alcohol consumption among subjects treated for alcohol addiction. Also, dysfunctional attitudes are both indirectly and directly related to treatment outcome. By incorporating alcohol consumption, depression and dysfunctional attitudes as targets of intervention, treatment compliance and outcome may be enhanced.

A prospective study of weight development and behavior problems in toddlers: the Norwegian Mother and Child Cohort Study.

BACKGROUND: Previous research has suggested that overweight children have a higher risk of behavior problems, but the causal direction of this relationship remains unclear. In a large prospective population study, we investigated whether child behavior problems and body mass index are associated in toddlers and whether overweight is a risk for behavior problems or vice versa. METHODS: The study was part of the Norwegian Mother and Child Cohort Study. The sample consisted of 10 860 toddlers, followed up to age 36 months. We used data from maternal questionnaires from gestation week 17 and at child ages 18 and 36 months, and data from the Medical Birth Registry of Norway. Child height and weight were assessed at child health stations and recorded by mothers. Behavior problems were assessed using shortened subscales from the Child Behavior Checklist. Statistical analyses were conducted using structural equation modeling. RESULTS: Behavior problems in toddlers were not associated with higher body mass index cross-sectionally at either age 18 or 36 months, and there was no indication that behavior problems caused increasing body mass index over time or vice versa. CONCLUSIONS: The association between behavior problems and body mass index found in older children did not appear in toddlers up to age 36 months. Future studies should focus on the age span from 3 to 6 years, which includes the period of adiposity rebound.

Handedness does not predict side of onset of motor symptoms in Parkinson's disease.

This study focused on the relationship between the asymmetry of initial motor symptoms of Parkinson's disease (PD) and premorbid handedness of patients. Structural equation modeling has been used for this purpose. The survey consisting of validated items measuring handedness and questions related to side of occurrence of initial symptoms was administered to 472 patients with PD [277 men, 195 women, mean age 66.5 (9.3), mean duration of the disease 10 (6.1) years]. The unidimensional model of handedness fits the data well (chi(2) = 37.86, df = 20, P = 0.009, Root Mean Square Error of Approximation = 0.044, Comparative Fit Index = 1.00, Standardized Root Mean Square Residual = 0.042) and side of initial motor symptoms is not significantly related to the factor of handedness (r =0.11, SE = 0.07, P = 0.14). In contrast to several other studies, the results indicate that the side of first occurrence of PD signs cannot be predicted from premorbid handedness of patients.

The relationship between burnout and musculoskeletal pain in seven Norwegian occupational groups.

Occupational and gender differences were investigated in the relationship between burnout and musculoskeletal pain in the head, neck, shoulders, and back. Representative samples of lawyers, physicians, nurses, teachers, church ministers, bus drivers, and information technology workers in Norway (N=4507) were analyzed using structural equation modeling. The exhaustion dimension of burnout was positively associated with musculoskeletal pain in all groups, and the strength of the relationship ranged from moderate to strong. The disengagement dimension of burnout was negatively associated with musculoskeletal pain in five groups and only ranged from -0.15 to -0.42. Professional efficacy was slightly weaker, and inconsistently (i.e., both positively and negatively) associated with musculoskeletal pain in four of the groups. There were larger differences in the strength of the relationships between the seven occupational groups than between males and females within the same profession. Results suggest that burnout and musculoskeletal pain are related, but the strength of the associations varies according to gender and occupation. Overall, occupational factors appear to be stronger predictors of the co-occurrence of burnout and musculoskeletal pain than gender.

Applying aspects of generalizability theory in preliminary validation of the Multifacet Interprofessional Collaboration Model (PINCOM).

OBJECTIVE: Empirical assessment of the Interprofessional Collaboration (IPC) model within the framework of Generalizability Theory (GT). DESIGN AND PARTICIPANTS: A multifacet data collection design served the purpose of examining the relationships between observed indicators representing a defined construct. Professionals working with children and adolescents (n=134), in the context of mental health care, completed a 48 item questionnaire addressing 12 aspects of interprofessional collaboration. RESULTS: Estimated variance components from two G-studies are presented. The relative impact of different sources of variance was estimated 1) for the full design, and 2) for three groups of informants (schools, primary care and specialist services). Differences between groups were found regarding the relative impact of the level - and context facets with respect to perception of IPC. CONCLUSIONS: The methodology of generalizability theory is well suited for data with a complex facet structure as displayed in the present study. We recommend researchers to give domain specifications thorough attention when measuring IPC.

Difficult temperament, breastfeeding, and their mutual prospective effects: the Norwegian Mother and Child Cohort Study.

OBJECTIVE: (1) To examine the extent to which difficult temperament and breastfeeding are associated at child age 6 and 18 months. (2) To examine longitudinally whether breastfeeding has an influence on temperament development or whether difficult temperament has an influence on continued breastfeeding. METHOD: This prospective study is part of the Norwegian Mother and Child Cohort Study, which targets all women giving birth in Norway; the present sample comprises 30,466 children. Mothers reported on child difficult temperament at child age 6 and 18 months and on breastfeeding from 0 to 6 months and from 6 to 12-14 months. Prospective associations between breastfeeding and difficult temperament were examined using structural equation modeling, with comparison of cross-lagged pathways. All analyses were adjusted for background variables that are relevant for breastfeeding. RESULTS: At 6 months, children with more difficult temperament were significantly less likely to have been "fully" breastfed (exclusively and predominantly breastfed) for the recommended period of 6 months (unadjusted point-biserial correlation r pb = -.15; adjusted r pb = -.11). At 12 to 14 months, there was no longer an association between difficult temperament and (continued) breastfeeding after adjusting for background variables, temperament, and breastfeeding at 6 months. The cross-lagged analyses of the longitudinal pathways yielded negligible effects of difficult temperament on later breastfeeding and of breastfeeding on later temperament. CONCLUSION: Difficult temperament and reduced breastfeeding seem to be associated only during the first half-year of the child's life. After that, we found no evidence of codevelopment between difficult temperament and breastfeeding.

Antecedents and outcomes of intervention program participation and task priority change among school psychology counselors: a latent variable growth framework.

A three-year national intervention program introduced into the School Psychology Service (SPS) in Norway with the aim of increasing systemic level work among SP counselors was investigated. Latent variable growth models based on longitudinal data from 195 SP counselors gave no significant mean level change in systemic level work. This concurred with GLM analyses based on data from a sample of 20 schools. However, retrospective self-reported significant positive mean level change for systemic level work was detected among the SP counselors. Intervention program participation was associated with individual change in systemic level work. Self-efficacy beliefs about systemic level work, and school-related etiology beliefs predicted individual change to a certain degree. Comparison of two rival models gave no support for a hypothesized interaction among intervention program participation and beliefs in their effects on systemic level work. Open-ended questions indicated that individual level workload and the perceived expectations from the schools may have concern for a successful effect of the intervention program in addition to the hypothesized ones. Individual change in systemic level work was positively associated with individual change in job satisfaction.

Generalizability studies of the Global Assessment of Functioning-Split version.

OBJECTIVE: The study aimed to use the Generalizability Theory to investigate the reliability and precision of the split version of the Global Assessment of Functioning (GAF). MATERIALS: Six case vignettes were assessed by 2 samples; one by 19 experienced and independent raters and another by 58 experienced raters from 8 different day-treatment units, evaluating both symptom and function scores of GAF. METHODS: Generalizability studies were conducted to disentangle relevant variance components accounting for error variance in GAF scores. Furthermore, decision studies were conducted to estimate the reliability of different measurement designs, as well as precision in terms of error tolerance ratio. RESULTS: Both symptom and function scores of GAF were found to be highly generalizable, and a measurement design of 2 raters per subject was found to be most efficient with respect to reliability, precision, and use of resources. CONCLUSION: Both symptom and function scores of GAF seem highly consistent across experienced raters.

Circumplex structure and personality disorder correlates of the Interpersonal Problems model (IIP-C): construct validity and clinical implications.

This study assessed the construct validity of the circumplex model of the Inventory of Interpersonal Problems (IIP-C) in Norwegian clinical and nonclinical samples. Structure was examined by evaluating the fit of the circumplex model to data obtained by the IIP-C. Observer-rated personality disorder criteria (DSM-IV, Axis II) were used as external correlates. The reliability of the IIP-C scales was acceptable and in the same range as in the original version. A multisample analysis strategy did not support an invariant circumplex model across the 2 groups. However, the estimated structures reflected mostly the same circular pattern of a quasi-circumplex model in the 2 groups. Departures from the ideal model were of negligible practical significance. The validity results examining personality disorder correlates of the IIP-C generally conformed to predictions, providing direct evidence for agreement between self-report and expert judgments of interpersonal problems.

The reliability of self-image change scores in psychotherapy research: An application of generalizability theory.

The aim of this study was to estimate the reliability of the pre- to posttreatment change scores for 3 different self-image aspects, Attack, Love, and Control. To measure self-image, we used the Norwegian version of the introject surface of Benjamin's (1974) structural analysis of social behavior. The article introduces Generalizability (G-) theory, combined with the recent concept of tolerance for error, as a framework for estimating the reliability and precision of change scores in 1- and 2-facet designs. Data were obtained from the Norwegian Multi-Site Study of Process and Outcome in Psychotherapy, including 291 outpatients. The mean number of treatment sessions was 47. The results show that change scores may be highly reliable. Generalizability coefficients resting on the relative and absolute score interpretations, respectively, for both the Love and Attack change scores reached acceptable levels. The reliability of the Control change score was, however, poor. G-theory combined with the error-tolerance concept proved to be a helpful framework for assessing the dependability of change scores in a psychotherapy research setting. (PsycINFO Database Record (c) 2010 APA, all rights reserved).