Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: an observational study.

BACKGROUND: Research has shown that narrative feedback, (self) reflections and a plan to undertake and evaluate improvements are key factors for effective feedback on clinical performance. We investigated the quantity of narrative comments comprising feedback (by trainers), self-reflections (by trainees) and action plans (by trainer and trainee) entered on a mini-CEX form that was modified for use in general practice training and to encourage trainers and trainees to provide narrative comments. In view of the importance of specificity as an indicator of feedback quality, we additionally examined the specificity of the comments. METHOD: We collected and analysed modified mini-CEX forms completed by GP trainers and trainees. Since each trainee has the same trainer for the duration of one year, we used trainer-trainee pairs as the unit of analysis. We determined for all forms the frequency of the different types of narrative comments and rated their specificity on a three-point scale: specific, moderately specific, not specific. Specificity was compared between trainee-trainer pairs. RESULTS: We collected 485 completed modified mini-CEX forms from 54 trainees (mean of 8.8 forms per trainee; range 1-23; SD 5.6). Trainer feedback was more frequently provided than trainee self-reflections, and action plans were very rare. The comments were generally specific, but showed large differences between trainee-trainer pairs. CONCLUSION: The frequency of self-reflection and action plans varied, all comments were generally specific and there were substantial and consistent differences between trainee-trainer pairs in the specificity of comments. We therefore conclude that feedback is not so much determined by the instrument as by the users. Interventions to improve the educational effects of the feedback procedure should therefore focus more on the users than on the instruments.

Measuring continuity of care: psychometric properties of the Nijmegen Continuity Questionnaire.

BACKGROUND: Recently, the Nijmegen Continuity Questionnaire (NCQ) was developed. It aims to measure continuity of care from the patient perspective across primary and secondary care settings. Initial pilot testing proved promising. AIM: To further examine the validity, discriminative ability, and reliability of the NCQ. DESIGN: A prospective psychometric instrument validation study in primary and secondary care in the Netherlands. METHOD: The NCQ was administered to patients with a chronic disease recruited from general practice (n = 145) and hospital outpatient departments (n = 123) (response rate 76%). A principal component analysis was performed to confirm three subscales that had been found previously. Construct validity was tested by correlating the NCQ score to scores of other scales measuring quality of care, continuity, trust, and satisfaction. Discriminative ability was tested by investigating differences in continuity subscores of different subgroups. Test-retest reliability was analysed in 172 patients. RESULTS: Principal factor analysis confirmed the previously found three continuity subscales - personal continuity, care provider knows me; personal continuity, care provider shows commitment; and team/cross-boundary continuity. Construct validity was demonstrated through expected correlations with other variables and discriminative ability through expected differences in continuity subscores of different subgroups. Test-retest reliability was high (the intraclass correlation coefficient varied between 0.71 and 0.82). CONCLUSION: This study provides evidence for the validity, discriminative ability, and reliability of the NCQ. The NCQ can be of value to identify problems in continuity of care.

The process of feedback in workplace-based assessment: organisation, delivery, continuity.

OBJECTIVES: Feedback in workplace-based clinical settings often relies on expert trainers' judgements of directly observed trainee performance. There is ample literature on effective feedback, but in practice trainees in workplace-based training are not regularly observed. We aimed to examine external conditions that impact feedback in observational workplace-based assessment (WBA). METHODS: Interviews were conducted and the resulting data analysed using a qualitative, phenomenological approach. Between October 2009 and January 2010, we interviewed 22 postgraduate general practice trainees at two institutions in the Netherlands. Three researchers analysed the transcripts of the interviews. RESULTS: A three-step scheme emerged from the data. Feedback as part of WBA is of greater benefit to trainees if: (i) observation and feedback are planned by the trainee and trainer; (ii) the content and delivery of the feedback are adequate, and (iii) the trainee uses the feedback to guide his or her learning by linking it to learning goals. Negative emotions reported by almost all trainees in relation to observation and feedback led to different responses. Some trainees avoided observation, whereas others overcame their apprehension and actively sought observation and feedback. Active trainers were able to help trainees overcome their fears. Four types of trainer-trainee pairs were distinguished according to their engagement in observation and feedback. External requirements set by training institutions may stimulate inactive trainers and trainees. CONCLUSIONS: In line with the literature, our results emphasise the importance of the content of feedback and the way it is provided, as well as the importance of its incorporation in trainees' learning. Moreover, we highlight the step before the actual feedback itself. The way arrangements for feedback are made appears to be important to feedback in formative WBA. Finally, we outline several factors that influence the success or failure of feedback but precede the process of observation and feedback.

Nijmegen Continuity Questionnaire: development and testing of a questionnaire that measures continuity of care.

OBJECTIVE: To develop and pilot test a generic questionnaire to measure continuity of care from the patient's perspective across primary and secondary care settings. STUDY DESIGN AND SETTING: We developed the Nijmegen Continuity Questionnaire (NCQ) based on a systematic literature review and analysis of 30 patient interviews. The questionnaire consisted of 16 items about the patient-provider relationship to be answered for five different care providers and 14 items each on the collaboration between four groups of care providers. The questionnaire was distributed among patients with a chronic disease recruited from general practice. We used principal component analysis (PCA) to identify subscales. We refined the factors by excluding several items, for example, items with a high missing rate. RESULTS: In total, 288 patients filled out the questionnaire (response rate, 72%). PCA yielded three subscales: "personal continuity: care provider knows me," "personal continuity: care provider shows commitment," and "team/cross-boundary continuity." Internal consistency of the subscales ranged from 0.82 to 0.89. Interscale correlations varied between 0.42 and 0.61. CONCLUSION: The NCQ shows to be a comprehensive, reliable, and valid instrument. Further testing of reliability, construct validity, and responsiveness is needed before the NCQ can be more widely implemented.

Hypertension patients participating in trials differ in many aspects from patients treated in general practices.

OBJECTIVE: To determine to what extent participants in randomized controlled hypertension trials (RCTs) could be compared to patients with hypertension in general practices. STUDY DESIGN AND SETTING: We analyzed RCTs that had been used in hypertension guidelines or were available for future hypertension guidelines. The characteristics of the participants of these RCTs were compared with those of the patients with hypertension in general practices. In addition, inclusion and exclusion criteria of the RCTs were applied to the hypertension patients in the general practices. RESULTS: Twenty-one trials were analyzed. Participants' characteristics often differed from those of the patients with hypertension in general practices, in particular in the older trials where the participants showed lower cardiovascular risk. More recent trials showed participants' profiles that better reflected those of the patients under treatment in a general practice. Less than half of the patients undergoing treatment in a general practice would have been included in the hypertension trials. CONCLUSION: Participants taking part in trials differed from patients with hypertension in a general practice in a number of important aspects. This hampers their external validity. Inclusion of participants with comorbidity and other general practice characteristics would improve translation of study findings to daily practice recommendations.

Which data source in clinical performance assessment? A pilot study comparing self-recording with patient records and observation.

OBJECTIVE: A pilot study aimed to determine the extent to which each of three data sources could provide complete and reliable data for valid assessment of clinical performance. DESIGN: Clinical decisions taken in 168 consultations by seven family physicians were reviewed against guidelines for 15 clinical conditions. In total, 206 criteria were reviewed using three sources: medical records, observation in surgery, and structured self-recording by the physicians. SETTING AND PARTICIPANTS: Seven family practices in the Netherlands. MAIN MEASURE: Scores (%) of data recorded/total were obtained for each method. Kappa scores for the agreement between the three data sources were also obtained. RESULTS: Medical record examination provided 40%, observation 72%, and physician self-recording 95% of the data required for the review against guidelines. Nine per cent of the clinical decisions could be reviewed when using medical records, 46% when using observation data, and 69% when using data from prospective self-recording. In particular, decisions in the area of patient education and diagnostic examinations could not be reviewed validly using medical records only. Kappa agreements between the data available from the three sources as well as between the review results appeared to be 0.79. CONCLUSIONS: Medical records alone only supply sufficient information for the review of a very limited set of clinical decisions. Physician self-recording has significantly more potential for valid review of a broad range of clinical decisions. Furthermore, self-recording seems a reliable data collection method that deserves further research.

Characteristics of effective clinical guidelines for general practice.

BACKGROUND: The use of clinical guidelines in general practice is often limited. Research on barriers to guideline adherence usually focuses on attitudinal factors. Factors linked to the guideline itself are much less studied. AIM: To identify characteristics of effective clinical guidelines for general practice, and to explore whether these differ between therapeutic and diagnostic recommendations. DESIGN OF STUDY: Analysis of performance data from an audit study of 200 general practitioners (GPs) in The Netherlands conducted in 1997. SETTING: Panel of 12 GPs in The Netherlands who were familiar with guideline methodology. METHOD: A set of 12 attributes, including six potential facilitators and six potential barriers to guideline use, was formulated. The panel assessed the presence of these attributes in 96 guideline recommendations formulated by the Dutch College of General Practitioners. The attributes of recommendations with high compliance rates (70% to 100%) were compared with those with low compliance rates (0% to 60%). RESULTS: Recommendations with high compliance rates were to a lesser extent those requiring new skills (7% compared with 22% in recommendations with low compliance rates), were less often part of a complex decision tree (12% versus 25%), were more compatible with existing norms and values in practice (87% versus 76%), and more often supported with evidence (47% versus 31%). For diagnostic recommendations, the ease of applying them and the potential (negative) reactions of patients were more relevant than for therapeutic recommendations. CONCLUSION: To bridge the gap between research and practice, the evidence as well as the applicability should be considered when formulating recommendations. If the recommendations are not compatible with existing norms and values, not easy to follow or require new knowledge and skills, appropriate implementation strategies should be designed to ensure change in daily practice.