Are evaluations in simulated medical encounters reliable among rater types? A comparison between standardized patient and outside observer ratings of OSCEs.

Objective: By analyzing Objective Structured Clinical Examination (OSCE) evaluations of first-year interns' communication with standardized patients (SP), our study aimed to examine the differences between ratings of SPs and a set of outside observers with training in healthcare communication. Methods: Immediately following completion of OSCEs, SPs evaluated interns' communication skills using 30 items. Later, two observers independently coded video recordings using the same items. We conducted two-tailed t-tests to examine differences between SP and observers' ratings. Results: Rater scores differed significantly on 21 items (p < .05), with 20 of the 21 differences due to higher SP in-person evaluation scores. Items most divergent between SPs and observers included items related to empathic communication and nonverbal communication. Conclusion: Differences between SP and observer ratings should be further investigated to determine if additional rater training is needed or if a revised evaluation measure is needed. Educators may benefit from adjusting evaluation criteria to decrease the number of items raters must complete and may do so by encompassing more global questions regarding various criteria. Furthermore, evaluation measures may be strengthened by undergoing reliability and validity testing. Innovation: This study highlights the strengths and limitations to rater types (observers or SPs), as well as evaluation methods (recorded or in-person).

Associations Between National Board Exam Performance and Residency Program Emphasis on Patient Safety and Interprofessional Teamwork.

OBJECTIVE: Several aspects of medical training may contribute to the ultimate goal of producing excellent physicians whose patients will have the best possible outcomes. However, the relative importance of education, evaluation and feedback, duty hours, practice structure, and program culture in achieving this goal is unclear. This study assessed associations among in-training exam performance, Accreditation Council for Graduate Medical Education (ACGME) Resident Survey responses, and American Board of Medical Specialties (ABMS) national board exam performance. METHODS: Residency training programs at a university teaching hospital were classified as having 5-year first-time ABMS pass rates above (n=12) or below (n=3) the national average for their specialty. These groups were compared by ACGME Resident Survey data and in-training exam performance. RESULTS: Surveys were collected from 484/543 eligible residents (89%), including 177 surveys from programs with below-average board pass rates and 307 surveys from programs with aboveaverage board pass rates. In-training exam performance was similar between groups. Aboveaverage programs had stronger agreement with statements that their culture reinforced patient safety (4.72 vs. 4.30, p=0.006) and that information was not lost during transitions of care (4.14 vs. 3.63, p=0.001). Although the occurrence of interprofessional teamwork was similar between groups, above-average programs had stronger agreement with the statement that interprofessional teamwork was effective (4.60 vs. 4.17, p=0.003). CONCLUSION: Residency programs emphasizing patient safety and effective interprofessional teamwork had above-average first-time national board pass rates.

Perspectives on death and dying: a study of resident comfort with End-of-life care.

BACKGROUND: Despite the benefits to early palliative care in the treatment of terminal illness, barriers to timely hospice referrals exist. Physicians who are more comfortable having end-of-life (EOL) conversations are more likely to refer to hospice. However, very little is known about what factors influence comfort with EOL care. METHODS: An anonymous survey was sent to all the residents and fellows at a single institution. Self-reported education, experience and comfort with EOL care was assessed. Using multivariate logistic regression analysis, variables that influenced comfort with EOL conversations were analyzed. RESULTS: Most residents (88.1%) reported little to no classroom training on EOL care during residency. EOL conversations during residency were frequent (50.6% reported > 10) and mostly unsupervised (61.9%). In contrast, EOL conversations during medical school were infrequent (3.7% reported >10) and mostly supervised (78.6%). Most (54.3%) reported little to no classroom training on EOL care during medical school. Physicians that reported receiving education on EOL conversations during residency and those who had frequent EOL conversations during residency had significantly higher comfort levels having EOL conversations (p = 0.017 and p = 0.003, respectively). Likewise, residents that felt adequately prepared to have EOL conversations when graduating from medical school were more likely to feel comfortable (p = 0.030). CONCLUSIONS: Most residents had inadequate education in EOL conversation skills during medical school and residency. Despite the lack of training, EOL conversations during residency are common and often unsupervised. Those who reported more classroom training during residency on EOL skills had greater comfort with EOL conversations. Training programs should provide palliative care education to all physicians during residency and fellowship, especially for those specialties that are most likely to encounter patients with advanced terminal disease.

The development of the emergency medicine milestones.

The Accreditation Council for Graduate Medical Education (ACGME) has outlined its "Next Accreditation System" (NAS) that will focus on resident and residency outcome measurements. Emergency medicine (EM) is one of seven specialties that will implement the NAS beginning July 2013. All other specialties will follow in July 2014. A key component of the NAS is the development of assessable milestones, which are explicit accomplishments or behaviors that occur during the process of residency education. Milestones describe competencies more specifically and identify specialty-specific knowledge, skills, attitudes, and behaviors (KSABs) that can be used as outcome measures within the general competencies. The ACGME and the American Board of Emergency Medicine (ABEM) convened an EM milestone working group to develop the EM milestones. This article describes the development, use within the NAS, and challenges of the EM milestones.

Retention of EBM competencies.

This longitudinal follow-up study surveyed the attitudes toward, and skills in, evidence-based medicine (EBM) of medical school graduates who had participated in a formal EBM curriculum during their third-year Internal Medicine (IM) clerkship. The objective was to measure whether competencies were retained over time and to identify the long- term impact on the graduates' attitudes toward, and proficiency in, EBM after graduation. Questionnaire results showed that a group of medical school graduates retained EBM skills and a positive attitude about the importance of applying EBM principles in patient care one to three years after completing an EBM course.

The value of resident teaching to improve student perceptions of surgery clerkships and surgical career choices.

BACKGROUND: A fundamental function of attending faculty is to teach and mentor medical students, but the benefit of the resident's role is recognized increasingly. METHODS: Our Standardized Institutional Clinical Clerkship Assessment allows students to rate 27 factors relative to a clinical clerkship. Scores from 1998 to 2005 were used to evaluate our surgical clerkship program and to compare resident and attending teachers. Student surgery career choices also were monitored. RESULTS: Medical students routinely scored residents more highly than attending faculty. Attendings' scores did not improve; however, residents' teaching and overall clerkship scores improved during the study period and paralleled students' increased selection of a surgical career. CONCLUSIONS: Students perceived residents as teachers more than attendings. Residents may have significant influence over students' career choice by their teaching and mentoring activities, which benefit attending efforts.

Impact of an evidence-based medicine curriculum on medical students' attitudes and skills.

PURPOSE: This evaluation study sought to assess the impact of an evidence-based medicine (EBM) course on students' self-perception of EBM skills, determine their use of EBM skills, and measure their performance in applying EBM skills in a simulated case scenario. METHODS: Pre- and post-surveys and skills tests were developed to measure students' attitudes toward and proficiency in EBM skills. Third-year students completed the voluntary survey and skills test at the beginning and completion of a twelve-week clerkship in internal medicine (IM) co-taught by medical and library faculty. Data were analyzed using the Mann-Whitney U test for a two-tailed test. RESULTS: A statistically significant increase was found in the students' self-assessment of skills. Students reported using the journal literature significantly more frequently during the clerkship than before, although textbooks remained their number one resource. A majority of students reported frequent use of EBM skills during the clerkship. Statistically significant improvement in student performance was also found on the posttest, although the level of improvement was more modest than that found on the post-surveys. CONCLUSION: The introduction of EBM skills to students during a clinical clerkship provides students an opportunity to practice EBM skills and reinforces the use of evidence in making patient-care decisions.

Modified transthecal digital block versus traditional digital block for anesthesia of the finger.

PURPOSE: This study compared the modified transthecal digital block (MTDB) technique with the traditional digital block (TDB) according to the degree of discomfort caused by injection and to the onset and the duration of anesthesia. METHODS: This was a prospective, randomized, double-blinded, and controlled study. The same investigator performed all blocks to the middle fingers of each hand. The hand anesthetized and type of block (TDB or MTDB) received first were both randomized. An orientation was given to the 25 participants detailing how to evaluate their own degree of anesthesia. This orientation included establishing a baseline of sensation with a safety pin, a description and diagram of 12 zones of the finger, an explanation of the 10-cm visual analog pain scale, and an explanation of how to record anesthesia progress in the fingers. On completion of each block, the subjects recorded the degree of pain and time to anesthesia in each finger zone. RESULTS: Twenty-five subjects received 1 TDB and 1 MTDB for a total of 50 blocks. The TDB received a mean rating for pain of 2.972 versus 2.784 for the MTDB (p =.579). The TDB took 3.91 minutes on average to take effect, whereas the MTDB took 7.16 minutes, a difference of 3.25 minutes. This was statistically significant in 11 of the 12 zones. Overall, return to sensation from the MTDB took effect 4.63 minutes sooner than the TDB: on average 85.19 minutes versus 89.82 minutes, respectively. This was statistically significant in 2 of the zones. CONCLUSIONS: The effect of MTDB is equal to that of TDB in terms of pain perception. For the dorsal and radial proximal zones, the TDB appears to have better distribution of anesthesia. The MTDB has slower onset to anesthesia than the TDB.