Factors that might undermine the validity of patient and multi-source feedback.

CONTEXT: Multi-source feedback (MSF) and patient feedback (PF) are used increasingly around the world to assess and quality-assure clinical practice. However, concerns about the evidence for their utility pertain to their ability to identify poor performance, the impact of allowing assessees to select their own assessors and the many confounders that may undermine validity. METHODS: This study was conducted in conjunction with the National Clinical Assessment Service (NCAS) in the UK and used established MSF and PF instruments to assess doctors in potential difficulty. Multi-source feedback assessors were nominated by both the practitioner (Pnom) and the referring body (RBnom). Demographics were collected to elucidate any differences found. Ratings generated by MSF and PF were compared with one another and with findings of a previous study that provided a normative cohort. RESULTS: Using MSF, NCAS-assessed doctors scored significantly lower than the reference cohort. Nineteen (28%) NCAS-assessed doctors achieved scores that were less than satisfactory. This rose to 50% when only RBnom assessors were used. Overall, ratings awarded by RBnom assessors were significantly lower than those awarded by Pnom assessors. Collected demographics did not help to explain the difference. Only one NCAS-assessed doctor scored below average according to PF. Doctors in the NCAS-assessed group did not score significantly lower than the reference cohort in PF. Doctor assessment scores awarded by patients were significantly higher than those awarded by colleagues. CONCLUSIONS: Although colleagues appear to report poor performance using MSF, patients fail to report concurrent findings. This challenges the validity of PF as it is currently constructed. Scores in MSF differ significantly depending on whether they are practitioner- or third party-nominated. Previously recognised confounding factors do not help to explain this difference.

State of the science in health professional education: effective feedback.

BACKGROUND: Effective feedback may be defined as feedback in which information about previous performance is used to promote positive and desirable development. This can be challenging as educators must acknowledge the psychosocial needs of the recipient while ensuring that feedback is both honest and accurate. Current feedback models remain reductionist in their approach. They are embedded in the hierarchical, diagnostic endeavours of the health professions. Even when it acknowledges the importance of two-way interactions, feedback often remains an educator-driven, one-way process. LESSONS FROM THE LITERATURE: An understanding of the various types of feedback and an ability to actively seek an appropriate approach may support feedback effectiveness. Facilitative rather than directive feedback enhances learning for high achievers. High-achieving recipients undertaking complex tasks may benefit from delayed feedback. It is hypothesised that such learners are supported by reducing interruptions during the task. If we accept that medical students and doctors are high achievers, we can draw on some guiding principles from a complex and rarely conclusive literature. Feedback should focus on the task rather than the individual and should be specific. It should be directly linked to personal goals. Self-assessment as a means to identify personal learning requirements has no theoretical basis. Motivated recipients benefit from challenging facilitated feedback from external sources. A NEW MODEL: To achieve truly effective feedback, the health professions must nurture recipient reflection-in-action. This builds on self-monitoring informed by external feedback. An integrated approach must be developed to support a feedback culture. Early training and experience such as peer feedback may over time support the required cultural change. Opportunities to provide feedback must not be missed, including those to impart potentially powerful feedback from high-stakes assessments. Feedback must be conceptualised as a supported sequential process rather than a series of unrelated events. Only this sustained approach will maximise any effect.

Use of SPRAT for peer review of paediatricians in training.

OBJECTIVE: To determine whether a multisource feedback questionnaire, SPRAT (Sheffield peer review assessment tool), is a feasible and reliable assessment method to inform the record of in-training assessment for paediatric senior house officers and specialist registrars. DESIGN: Trainees' clinical performance was evaluated using SPRAT sent to clinical colleagues of their choosing. Responses were analysed to determine variables that affected ratings and their measurement characteristics. SETTING: Three tertiary hospitals and five secondary hospitals across a UK deanery. PARTICIPANTS: 112 paediatric senior house officers and middle grades. MAIN OUTCOME MEASURES: 95% confidence intervals for mean ratings; linear and hierarchical regression to explore potential biasing factors; time needed for the process per doctor. RESULTS: 20 middle grades and 92 senior house officers were assessed using SPRAT to inform their record of in-training assessment; 921/1120 (82%) of their proposed raters completed a SPRAT form. As a group, specialist registrars (mean 5.22, SD 0.34) scored significantly higher (t = - 4.765) than did senior house officers (mean 4.81, SD 0.35) (P < 0.001). The grade of the doctor accounted for 7.6% of the variation in the mean ratings. The hierarchical regression showed that only 3.4% of the variation in the means could be additionally attributed to three main factors (occupation of rater, length of working relationship, and environment in which the relationship took place) when the doctor's grade was controlled for (significant F change < 0.001). 93 (83%) of the doctors in this study would have needed only four raters to achieve a reliable score if the intent was to determine if they were satisfactory. The mean time taken to complete the questionnaire by a rater was six minutes. Just over an hour of administrative time is needed for each doctor. CONCLUSIONS: SPRAT seems to be a valid way of assessing large numbers of doctors to support quality assurance procedures for training programmes. The feedback from SPRAT can also be used to inform personal development planning and focus quality improvements.

Clinical management. Where medicine meets management. On reflection.

There is an urgent need for validated, feasible tools to assess doctors' performance. A peer-rating tool mapped to good medical practice (SPRAT) is feasible, valid and reliable. SPRAT is ideal for revalidation purposes and performs particularly well in areas that are traditionally difficult to assess. It can also inform personal development planning.