Feedback Redefined: Principles and Practice.

Feedback is defined as a regulatory mechanism where the effect of an action is fed back to modify and improve future action. In medical education, newer conceptualizations of feedback place the learner at the center of the feedback loop and emphasize learner engagement in the entire process. But, learners reject feedback if they doubt its credibility or it conflicts with their self-assessment. Therefore, attention has turned to sociocultural factors that influence feedback-seeking, acceptance, and incorporation into performance. Understanding and application of specific aspects of psychosocial theories could help in designing initiatives that enhance the effect of feedback on learning and growth. In the end, the quality and impact of feedback should be measured by its influence on recipient behavior change, professional growth, and quality of patient care and not the skills of the feedback provider. Our objective is to compare and contrast older and newer definitions of feedback, explore existing feedback models, and highlight principles of relevant psychosocial theories applicable to feedback initiatives. Finally, we aim to apply principles from patient safety initiatives to emphasize a safe and just culture within which feedback conversations occur so that weaknesses are as readily acknowledged and addressed as strengths.

Composite reliability of workplace-based assessment of international medical graduates.

OBJECTIVE: The fitness to practise of international medical graduates (IMGs) is usually evaluated with standardised assessment tests. The performance rather than the competency of practising doctors should, however, be assessed, for which reason workplace-based assessment (WBA) has gained increasing attention. Our aim was to assess the composite reliability of WBA instruments for assessing IMGs. DESIGN AND SETTING: Between June 2010 and April 2015, 142 IMGs were assessed by 99 calibrated assessors; each was assessed in the workplace over 6 months. The IMGs completed 970 case-based discussions (CBDs), 1741 mini-clinical examination exercises (mini-CEX), and 1020 multi-source feedback (MSF) assessments. PARTICIPANTS: 103 male and 39 female candidates from 28 countries (Africa, Asia, Europe, South America, South Pacific) in urban and rural hospitals of the Hunter New England Health region. MAIN OUTCOME MEASURES: The composite reliability across the three WBA tools, expressed as the standard error of measurement (SEM). RESULTS: In our WBA program, a combination of five CBD and 12 mini-CEX assessments achieved an SEM of 0.33, greater than the threshold 0.26 of a scale point. Adding six MSF results to the assessment package reduced the SEM to 0.24, which is adequately precise. CONCLUSIONS: Combining data from different WBA assessment instruments achieves acceptable reliability for assessing IMGs, provided that the panel of WBA assessment types are carefully selected and the assessors are calibrated.

Cracking the code: residents' interpretations of written assessment comments.

CONTEXT: Interest is growing in the use of qualitative data for assessment. Written comments on residents' in-training evaluation reports (ITERs) can be reliably rank-ordered by faculty attendings, who are adept at interpreting these narratives. However, if residents do not interpret assessment comments in the same way, a valuable educational opportunity may be lost. OBJECTIVES: Our purpose was to explore residents' interpretations of written assessment comments using mixed methods. METHODS: Twelve internal medicine (IM) postgraduate year 2 (PGY2) residents were asked to rank-order a set of anonymised PGY1 residents (n = 48) from a previous year in IM based solely on their ITER comments. Each PGY1 was ranked by four PGY2s; generalisability theory was used to assess inter-rater reliability. The PGY2s were then interviewed separately about their rank-ordering process, how they made sense of the comments and how they viewed ITERs in general. Interviews were analysed using constructivist grounded theory. RESULTS: Across four PGY2 residents, the G coefficient was 0.84; for a single resident it was 0.56. Resident rankings correlated extremely well with faculty member rankings (r = 0.90). Residents were equally adept at reading between the lines to construct meaning from the comments and used language cues in ways similarly reported in faculty attendings. Participants discussed the difficulties of interpreting vague language and provided perspectives on why they thought it occurs (time, discomfort, memorability and the permanency of written records). They emphasised the importance of face-to-face discussions, the relative value of comments over scores, staff-dependent variability of assessment and the perceived purpose and value of ITERs. They saw particular value in opportunities to review an aggregated set of comments. CONCLUSIONS: Residents understood the 'hidden code' in assessment language and their ability to rank-order residents based on comments matched that of faculty. Residents seemed to accept staff-dependent variability as a reality. These findings add to the growing evidence that supports the use of narrative comments and subjectivity in assessment.

Composite reliability of workplace-based assessment for international medical graduates.

OBJECTIVE: The fitness to practise of international medical graduates (IMGs) is usually evaluated with standardised assessment tests. Practising doctors should, however, be assessed on their performance rather than their competency, for which reason workplace-based assessment (WBA) has gained increasing attention. Our aim was to assess the composite reliability of WBA instruments for assessing the performance of IMGs. DESIGN AND SETTING: Between June 2010 and April 2015, 142 IMGs were assessed by 99 calibrated assessors; each cohort was assessed at their workplace over 6 months. The IMGs completed 970 case-based discussions (CBDs), 1741 Mini-Clinical Examination Exercises (mini-CEX) and 1020 multisource feedback (MSF) sessions. PARTICIPANTS: 103 male and 39 female candidates based in urban and rural hospitals of the Hunter New England Health region, from 28 countries (Africa, Asia, Europe, South America, South Pacific). MAIN OUTCOME MEASURES: The reliability of the three WBA tools; the composite reliability of the tools as a group. RESULTS: The composite reliability of our WBA toolbox program was good: the composite reliability coefficient for five CBDs and 12 mini-CEX was 0.895 (standard error of measurement, 0.138). When the six MSF results were included, the composite reliability coefficient was 0.899 (standard error of measurement, 0.125). CONCLUSIONS: WBA is a reliable method for assessing IMGs when multiple tools and assessors are used over a period of time. This form of assessment meets the criteria for "good assessment" (reliability ≥ 0.8) and can be applied in other settings.

Factors influencing trainers' feedback-giving behavior: a cross-sectional survey.

BACKGROUND: The literature provides some insight into the role of feedback givers, but little information about within-trainer factors influencing 'feedback-giving behaviours'. We looked for relationships between characteristics of feedback givers (self-efficacy, task perception, neuroticism, extraversion, agreeableness and conscientiousness) and elements of observation and feedback (frequency, quality of content and consequential impact). METHODS: We developed and tested several hypotheses regarding the characteristics and elements in a cross-sectional digital survey among GP trainers and their trainees in 2011 and 2012. We conducted bivariate analysis using Pearson correlations and performed multiple regression analysis. RESULTS: Sixty-two trainer-trainee couples from three Dutch institutions for postgraduate GP training participated in the study. Trainer scores on 'task perception' and on a scale of the trait 'neuroticism' correlated positively with frequency of feedback and quality of feedback content. Multiple regression analysis supported positive correlations between task perception and frequency of feedback and between neuroticism and quality of feedback content. No other correlations were found. CONCLUSION: This study contributes to the literature on feedback giving by revealing factors that influence feedback-giving behaviour, namely neuroticism and task perception. Trainers whose task perception included facilitation of observation and feedback (task perception) and trainers who were concerned about the safety of their patients during consultations with trainees (neuroticism) engaged more frequently in observation and feedback and gave feedback of higher quality.

Students' perceptions of anatomy across the undergraduate problem-based learning medical curriculum: a phenomenographical study.

BACKGROUND: To get insight in how theoretical knowledge is transformed into clinical skills, important information may arise from mapping the development of anatomical knowledge during the undergraduate medical curriculum. If we want to gain a better understanding of teaching and learning in anatomy, it may be pertinent to move beyond the question of how and consider also the what, why and when of anatomy education. METHODS: A purposive sample of 78 medical students from the 2nd, 3rd, 4th and 6th year of a PBL curriculum participated in 4 focus groups. Each group came together twice, and all meetings were recorded and transcribed verbatim. Data were analysed with template analysis using a phenomenographical approach. RESULTS: Five major topics emerged and are described covering the students' perceptions on their anatomy education and anatomical knowledge: 1) motivation to study anatomy, 2) the relevance of anatomical knowledge, 3) assessment of anatomical knowledge, 4) students' (in)security about their anatomical knowledge and 5) the use of anatomical knowledge in clinical practice. CONCLUSIONS: Results indicated that a PBL approach in itself was not enough to ensure adequate learning of anatomy, and support the hypothesis that educational principles like time-on-task and repetition, have a stronger impact on students' perceived and actual anatomical knowledge than the educational approach underpinning a curriculum. For example, students state that repetitive studying of the subject increases retention of knowledge to a greater extent than stricter assessment, and teaching in context enhances motivation and transfer. Innovations in teaching and assessment, like spiral curriculum, teaching in context, teaching for transfer and assessment for learning (rewarding understanding and higher order cognitive skills), are required to improve anatomy education.

Doctors in a Southeast Asian country communicate sub-optimally regardless of patients' educational background.

OBJECTIVE: To explore the relationship between the style of doctor-patient communication and patients' educational background in a Southeast Asian teaching hospital setting using the Roter Interaction Analysis System (RIAS). METHODS: We analyzed a total of 245 audio-taped consultations involving 30 internal medicine residents with 7-10 patients each in the internal medicine outpatient clinics. The patients were categorized into a group with a high and a group with a low educational level. We ranked the data into 41 RIAS utterances and RIAS-based composite categories in order of observed frequency during consultations. RESULTS: The residents invariantly used a paternalistic style irrespective of patients' educational background. The RIAS utterances and the composite categories show no significant relationship between communication style and patients' educational level. CONCLUSION: Doctors in a Southeast Asian country use a paternalistic communication style during consultations, regardless of patients' educational background. PRACTICE IMPLICATION: To approach a more partnership doctor-patient communication, culture and clinical environment concern of Southeast Asian should be further investigated.

How to introduce incognito standardized patients into outpatient clinics of specialists in rheumatology.

To know what is going on in physicians' surgery hours, assessment of practice performance is important with regard to quality assessment activities. The incognito standardized patient (SP) method is a powerful method to assess this. However, until now no reports have been published about specialists' performance using this method. In this study, 27 rheumatologists in 16 hospitals were each visited by eight incognito SPs to study the feasibility of sending incognito SPs to specialists working in different hospitals, of follow-up consultations and of simulating additional investigations. SPs recorded performance on case-specific checklists. The different steps needed for these visits are described in detail. A total of 136 first and 32 follow-up visits took place. SPs remained undetected in 98% of the visits. It is concluded that this SP method is a feasible one, however, only to be used for very specific purposes. Use of incognito SPs for larger scale projects is not recommended.

Student perceptions about the occurrence of critical incidents in tutorial groups.

The study reported here examines student perceptions about the occurrence of critical incidents in tutorial groups across years of medical training. The study investigates the following research questions. (1) Which factors underlie the occurrence of critical incidents in the tutorial group? (2) How do students rate the incidents with respect to whether they occur in the tutorial groups? (3) Are there differences in scores on the factors identified across years of medical training? The subjects consisted of a stratified random sample of 200 students at the Medical School of the University of Maastricht. In general, the results of the confirmatory factor analyses indicate that a six-factor model showed a good fit. The results show that students frequently perceive lack of elaboration, lack of interaction and unequal participation in tutorial groups. For most of the factors (lack of elaboration, difficult personalities, lack of cohesion, and lack of motivation) the average scores differ significantly (p<0.01) across the years.

Reliability of clinical oral examinations re-examined.

Many medical schools still use oral examinations for the evaluation of clinical competence of students in their clerkship, although it has been proven that orals have poor reliability. This study investigates the feasibility and reliability of multiple oral examinations. Students in the last week of their Internal Medicine clerkship in an outpatient clinic were given several patient-based oral examinations. The student's performance was rated on a list of items reflecting clinical competence. A global judgement of the student's performance was also given. The results indicate that it is possible to increase the number of orals and the number of examiners in the day-to-day practice of an outpatient clinic moderately. The reliability when using a number of orals is better than the reliability of the common single oral examination. The reliability using global judgements appeared to be better than the reliability of averaged item scores.

Quality of student-generated learning issues in a problem-based curriculum.

In a problem-based curriculum students generate learning issues that are the guidelines for their individual study. In an earlier study it was found that a useful learning issue contains a keyword that demarcates the content of a certain topic to be studied and is formulated concisely and unambiguously for all members of the tutorial group. This study investigates two questions. First, how many learning issues can be classified as being of good or bad quality as evaluated against three criteria? Second, what are examples of well and badly formulated learning issues? Learning issues were collected for two problems in the first year during two tutorial meetings of the Maastricht Medical School in the academic year 1998-99. In the next academic year, 12 first year-students were asked to judge the learning issues for each problem against three criteria. The results showed that 21% of the learning issues for one problem and 32% for the other problem were classified as scoring high on all three characteristics. A very small percentage of learning issues (5% and 2%) were classified as scoring low on all three characteristics. Most learning issues scored differently on each of the characteristics. Although most learning issues contain a keyword, many of them were formulated ambiguously and not concisely. It is concluded that first-year students are not yet able to formulate clear learning issues.