Engaged at the extremes: residents' perspectives on clinical teaching assessment.

PURPOSE: Although academic centers rely on assessments from medical trainees regarding the effectiveness of their faculty as teachers, little is known about how trainees conceptualize and approach their role as assessors of their clinical supervisors. METHOD: In 2010, using a constructivist grounded theory approach, five focus group interviews were conducted with 19 residents from an internal medicine residency program. A constant comparative analysis of emergent themes was conducted. RESULTS: Residents viewed clinical teaching assessment (CTA) as a time-consuming task with little reward. They reported struggling throughout the academic year to meet their CTA obligations and described several shortcut strategies they used to reduce their burden. Rather than conceptualizing their assessments as a conduit for both formative and summative feedback, residents perceived CTA as useful for the surveillance of clinical supervisors at the extremes of the spectrum of teaching effectiveness. They put the most effort, including the crafting of written comments, into the CTAs of these outliers. Trainees desired greater transparency in the CTA process and were skeptical regarding the anonymity and perceived validity of their faculty appraisals. CONCLUSIONS: Individual and system-based factors conspire to influence postgraduate medical trainees' motivation for generating high-quality appraisals of clinical teaching. Academic centers need to address these factors if they want to maximize the usefulness of these assessments.

A mixed-methods analysis of residents' written comments regarding their clinical supervisors.

BACKGROUND: Clinical faculty require feedback to guide their development. Although written comments submitted by residents on clinical teaching assessments (CTAs) are potentially a rich source of feedback, little is known about their information quality. METHOD: Naturalistic study involving thematic content analyses and concordancing of comments submitted in a sample of 1,601 CTAs. RESULTS: About half of the CTAs contained comments, and most were related to perceived teacher strengths. Mixed-methods analysis of comments in the "areas for improvement" field revealed that rather than offering constructive criticism, residents vocalized their perceived learning needs. Specific, behaviorally based comments were uncommon. CONCLUSIONS: In their present format, the written comments analyzed by this study seem unlikely to provide faculty with substantive feedback. Greater insight into residents' understanding of the CTA process, including motivational factors, is necessary if academic centers intend to maximize the formative value of CTAs.

Relying on others' reliability: challenges in clinical teaching assessment.

BACKGROUND: The quality of the data generated from internally created faculty teaching instruments often draws skepticism. Strategies aimed at improving the reliability and validity of faculty teaching assessments tend to revolve around literature searches for a replacement instrument(s). PURPOSE: The purpose was to test this "search-and-apply" method and discuss our experiences with it within the context of observational assessment practice. METHOD: In a naturalistic pilot test, two previously validated faculty assessment instruments were paired with a global question. The reliability of both metrics was estimated. RESULTS: Generalizability analyses indicated that for both pilot tested faculty teaching instruments, the global question was a more reliable measure of perceived clinical teaching effectiveness than a multiple-item inventory. Item analysis with Cronbach's coefficient alpha suggested redundant instrument content. Rater error accounted for the greatest proportion of the variance and straight-line responses occurred in approximately 28% of residents' appraisals. CONCLUSIONS: The results of the present study draw attention to one of the common fallacies surrounding instrument-based assessment in medical education; the solution to improving one's assessment practice primarily involves identifying a previously published instrument from the literature. Academic centers need to invest in ongoing quality control efforts including the pilot testing of any proposed instruments.

The sum of the parts detracts from the intended whole: competencies and in-training assessments.

OBJECTIVES: Despite the fact that Canadian residency programmes are required to assess trainees' performance within the context of the CanMEDS Roles Framework, there has been no inquiry into the potential relationship between residents' perceptions of the framework and their in-training assessments (ITA). Using data collected during the study of ITA, we explored residents' perceptions of these competencies. METHODS: From May 2006-07, a purposive sample of 20 resident doctors from internal medicine, paediatrics, and surgery were interviewed about their ITA experiences. Data collection and analysis proceeded in an iterative fashion consistent with grounded theory. In April 2008, a summary of recurrent themes was presented during a focus group interview of another five residents to afford further elaboration and refinement of thematic findings. RESULTS: The in-training assessment report (ITAR) was perceived as a primary source of residents' information on CanMEDS. Residents' familiarity with the set of competencies appeared to be quite limited and they possessed narrow definitions of the roles. Several trainees questioned the framework's relevance and some appeared confused about the overlapping nature of the roles. Although residents viewed the central Medical Expert role as the most relevant and important competency, they incorrectly perceived it as only involving the acquisition of medical and scientific knowledge. A visual rhetorical analysis of a typical ITAR suggests that the visual features found within this assessment tool may be misrepresenting the framework and the centrality of the Medical Expert role. CONCLUSIONS: Resident doctors' knowledge of CanMEDS was found to be limited. The visual structure of the ITAR appears to be a factor in residents' apparent distortion of the CanMEDS construct from its original holistic philosophy.

Rules of engagement: residents' perceptions of the in-training evaluation process.

BACKGROUND: In-training evaluation reports (ITERs) often fall short of their goals of promoting resident learning and development. Efforts to address this problem through faculty development and assessment-instrument modification have been disappointing. The authors explored residents' experiences and perceptions of the ITER process to gain insight into why the process succeeds or fails. METHOD: Using a grounded theory approach, semistructured interviews were conducted with 20 residents. Constant comparative analysis for emergent themes was conducted. RESULTS: All residents identified aspects of "engagement" in the ITER process as the dominant influence on the success of ITERs. Both external (evaluator-driven, such as evaluator credibility) and internal (resident-driven, such as self-assessment) influences on engagement were elaborated. When engagement was lacking, residents viewed the ITER process as inauthentic. CONCLUSIONS: Engagement is a critical factor to consider when seeking to improve ITER use. Our articulation of external and internal influences on engagement provides a starting point for targeted interventions.

'I don't have time': issues of fragmentation, prioritisation and motivation for education scholarship among medical faculty.

OBJECTIVES: Although lack of time has been frequently cited as a barrier to scholarship, there has been little inquiry into what specific factors medical faculty staff perceive as contributing to this dilemma. The purpose of the present study was to explore, in greater detail, lack of time as a barrier for faculty interested in pursuing education scholarship. METHODS: In 2004, as part of a cross-sectional, mixed-methods needs assessment, 73 (67.6%) medical faculty completed a questionnaire probing areas related to education scholarship. Additionally, one year later, 16 respondents (60% of those invited) each participated in one of three focus groups. RESULTS: Despite their interest and regardless of their background training in education, faculty were able, on average, to devote only negligible amounts of time to education scholarship. The most commonly reported barrier to these pursuits was lack of protected time. Further analysis revealed that the time-related factor appeared to involve three themes: fragmentation (where opportunities to work on education projects are sporadic); prioritisation (where work responsibilities including after-hours work and administrative workload complete for time, and where there is difficulty in securing financially remunerated time), and motivation (where the degree of recognition and support for education work by both the department and colleagues is limited). CONCLUSIONS: With respect to education scholarship, the dilemma caused by lack of time involves a complex, multi-faceted set of issues which extends beyond the number of hours available in a day. Personal interest and having background training in education do not appear to be sufficient to encourage involvement. Multiple institutional support mechanisms are necessary.

Measuring resident well-being: impostorism and burnout syndrome in residency.

BACKGROUND: Assessing resident well-being is becoming increasingly important from a programmatic standpoint. Two measures that have been used to assess this are the Clance Impostor Scale (CIS) and the Maslach Burnout Inventory-Human Services Survey (MBI-HSS). However, little is known about the relationship between the two phenomena. OBJECTIVES: To explore the prevalence and association between impostorism and burnout syndrome in a sample of internal medicine residents. DESIGN: Anonymous, cross-sectional postal survey. PARTICIPANTS: Forty-eight internal medicine residents (postgraduate year [PGY] 1-3) at the Schulich School of Medicine & Dentistry (62.3% response rate). MEASUREMENTS AND MAIN RESULTS: Short demographic questionnaire, CIS and MBI-HSS. Impostorism and burnout syndrome were identified in 43.8% and 12.5% of residents, respectively. With the exception of a negative correlation between CIS scores and the MBI's personal accomplishment subscale (r = -.30; 95% CI -.54 to -.02), no other significant relations were identified. Foreign-trained residents were more likely to score as impostors (odds ratio [OR] 10.7; 95% CI 1.2 to 98.2) while senior residents were more likely to experience burnout syndrome (OR 16.5 95% CI 1.6 to 168.5). CONCLUSIONS: Both impostorism and burnout syndrome appear to be threats to resident well-being in our program. The lack of relationship between the two would suggest that programs and researchers wishing to address the issue of resident distress should consider using both measures. The finding that foreign-trained residents appear to be more susceptible to impostorism warrants further study.

Education scholarship: it's not just a question of 'degree'.

BACKGROUND: Although medical faculty are frequently encouraged to participate in education scholarship, there is a paucity of literature addressing how to support those who wish to do so. AIMS: The purpose of this study was to explore faculty involvement in and support needs for pursuing education scholarship. METHODS: A purposive sample of 108 medical faculty with an interest in medical education were invited to participate in a two-phase, mixed-methods study (survey and focus groups). RESULTS: Seventy-three faculty (67.6%) completed the questionnaire with 16 subsequently participating in focus group sessions. Nearly 40% had enrolled in or completed formal education training. Although the majority had been involved in at least one education project during the past five years, few had received funding or published their work. Three support-related themes emerged: education research support; enhancing colleague interactions; and ongoing development activities. Three related barriers were identified: time, access to support staff, and knowledge of research methodology. No significant differences were identified between those with and without additional education training. CONCLUSIONS: Assisting faculty to participate in education scholarship is a complicated endeavor. Institutional supports should not be limited to those with advanced degrees nor rely on Master-level degree programs to provide all the necessary training.

Does exposure to a procedural video enhance preclinical dental student performance in fixed prosthodontics?

To try to alleviate the issue of dental students having an inadequate field of view during live demonstrations of fixed prosthodontic preparations, an instructional video depicting the step-by-step procedures involved in an all-ceramic tooth preparation and provisional crown fabrication (practical exam 1, PE1) was created. Fifty-five second-year dental students were given a personal copy of the video after a lecture and an in-class viewing of the material. Throughout the course, students watched live demonstrations of tooth preparations and then practiced individually on mannequins. The scores achieved by the students on three practical exams (PE1, PE2, and PE3) were compared to those recorded by a class one year prior to the development of the video. The students exposed to the video performed significantly better on PE1 in comparison to the previous year's class, as well as compared to their own performance on the other two practical exams that had no supplementary teaching aids. A significant, moderate-level correlation was detected between exposure to the video and PE1. Ninety-six percent of the students reported on their end-of-year evaluation that the video helped them to prepare for PE1. The results of this study suggest that instructional videos may aid in the teaching of fixed prosthodontics.

Olfactory sensitivity in medical laboratory workers occupationally exposed to organic solvent mixtures.

BACKGROUND: Published epidemiological information relating the effects of occupational exposure to organic solvents (OS) to olfaction is limited. AIMS: The objectives of this pilot study were to measure the chemosensory abilities of medical laboratory employees occupationally exposed to OS mixtures, to compare these with control workers employed within the same occupational setting and to correlate chemosensory performance with OS exposure history and with employees' hedonic (pleasantness) perceptions about workplace OS odors. METHODS: Twenty-four medical laboratory employees (OS-exposed technicians plus control workers minimally exposed to OS) completed a health-related questionnaire, a test of pyridine odor detection threshold, along with a gustatory detection threshold test involving aqueous quinine solutions. Estimates of cumulative hours of OS exposure (CSI) were calculated from self-reports. RESULTS: OS-exposed laboratory technicians detected weaker concentrations of pyridine odor. Positive correlations were detected between CSI estimates to both pyridine detection and the degree that participants reported that OS odors were present in the workplace. However, no association was detected between pyridine detection and how unpleasant workplace OS odors were perceived. The OS-exposed participants were able to detect weaker concentrations of quinine. Compared to controls, OS-exposed workers complained more of experiencing several symptoms while working, including headaches, nasal irritation and mild cognitive impairment. CONCLUSIONS: The results of this cross-sectional pilot study indicated that, compared to controls, medical laboratory technicians exposed to low-level OS mixtures displayed evidence of elevated olfactory sensitivity (hyperosmia) to pyridine odor. The relation of this study's results to chemical intolerance warrants further investigation.