Impact of a Regional Grant Program Through the Lens of Social Cognitive Career Theory: A Mixed-Method Evaluation.

PURPOSE: Evaluations of educational grant programs have focused on research productivity, with few examining impacts on grantees or effective program characteristics. This evaluation examined the regional grant program sponsored by Group on Educational Affairs to examine if and how grantees' careers were affected by funding, and if these experiences aligned with program goals. METHOD: In this concurrent, mixed-methods theory-driven evaluation, quantitative and qualitative data were analyzed independently and then integrated to examine complementarity. Quantitative data examined differences among 4 geographic regions and included proposal and grantee characteristics abstracted from administrative records of 52 funded proposals from 2010-2015 grant cycles. Qualitative data from 23 interviews conducted from 2018 to 2019 explored the impact on grantees, with Social Cognitive Career Theory (SCCT) serving as a framework for deductive thematic analysis. To facilitate integration of findings, quantitative data were layered onto each interview to permit exploration of associations between the 2 data types. RESULTS: Although significant regional differences existed in project length and amount of funding, there were few regional differences in grantee experiences. Despite small funding amounts, grants were perceived as career launching pads. The SCCT framework accounted for grantee experiences, including researcher identity formation and subsequent research, but did not capture collaboration phenomena. Integration of the 2 data types identified experience patterns unique to different groups of grantees (e.g., more or less research experience). The diversity among grantees suggests that clarification of program goals and stronger alignment with criteria for funding may be warranted. CONCLUSIONS: This evaluation illuminates why small educational grant programs may or may not impact interest and productivity in research. Implications exist for funders, including clarifying program goals and providing support for less experienced grantees. Future research should explore grantee subsets (e.g., underrepresented in medicine) to further identify what fosters or inhibits careers of medical education scholars.

Leveraging Mindfulness to Reduce Stress and Improve Quality of Life Among Pharmacy Students.

OBJECTIVE: Methods to improve stress and well-being for health profession trainees are limited. Mindfulness, elevating awareness to the present moment experience with compassion, has been shown to demonstrate effectiveness to enhance well-being. This research leverages techniques from mindfulness to develop and evaluate a credit-bearing longitudinal mindfulness elective, designed to teach mindfulness to improve stress and quality of life (QoL). METHODS: A mindfulness elective was created for pharmacy students. A longitudinal, case-control, survey-based design was used to compare stress and QoL between mindfulness participants and nonparticipant controls. Stress was assessed by the Perceived Stress Scale (PSS) and QoL by the SF-12 v2 Health-Related QoL Scale (SF-12 v2 QoL). RESULTS: Four weeks after course completion, the average PSS score was lower among participants compared to controls (18.58 SD 5.85 vs 20.79 SD 6.31, Cohen's d = 0.36). The Mental Health Component score of SF-12 v2 QoL was higher among participants versus controls (41.94 SD 8.58 vs 36.93 SD 9.59, Cohen's d = 0.55). The Physical Health Component score of SF-12 v2 QoL was lower among participants than the control group (46.13 SD 5.48 vs 48.62 SD 6.53, Cohen's d = 0.41). CONCLUSION: The results indicate small to moderate effect sizes associated with participation in a mindfulness elective, reducing stress and improving mental QoL among pharmacy students. The structure and potential benefits of the course can be extrapolated to other institutions. By offering course credit for mindfulness practice, health profession schools can support student wellness.

Developing a simulation-based training curriculum in transesophageal ultrasound with the use of the endobronchial ultrasound-endoscope.

There is an increasing need to focus on how best to train respiratory physicians to perform EUS with bronchoscope-guided fine-needle aspiration biopsy (EUS-B-FNA). At current, training is mostly performed in the clinical environment under expert supervision; however, the advent of simulation-based education now provides a low-risk setting for novice trainees to learn and practice basic endosonography skills from identifying and understanding normal anatomy as well as pathology, maneuvering of endoscope, interpretation of images, and mastering of sampling techniques. In this descriptive educational paper, we used a six-step approach as a framework to describe the development of a structured training program combining EUS-B-FNA with the already well-established certification training program in endobronchial ultrasound transbronchial needle aspiration. This comprehensive training curriculum includes a theoretical course to achieve foundational knowledge, followed by simulation-based training until mastery standards are met, and supervised clinical apprenticeship. All steps should end with an objective assessment to achieve certification. This systematic development will hopefully encourage endosonography leaders and educators to collaborate and implement an evidence-based comprehensive endosonography curriculum that aims to provide the trainee with the essential EUS-B competencies to ensure that lung cancer patients are diagnosed and staged correctly.

Patient involvement in assessment of postgraduate medical learners: A scoping review.

CONTEXT: Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent. METHODS: We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment. RESULTS: We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise. CONCLUSION: Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship.

Introduction: Education scholarship requires peer reviewers. For novice scholars, reviewing is an important developmental activity that cultivates deeper participation in the scholarship community. Yet getting started with reviewing is challenging for those not involved with the educational scholarship community. Beyond scientific expertise, reviewers must have a mentoring mindset, skills in providing constructive feedback, and knowledge of common ethical challenges associated with producing and evaluating scholarship. Methods: Our workshop introduced novice health professions educators to peer reviewing. It included four stimulus presentations about the peer reviewer's mindset and skills, followed by reinforcement activities. The workshop could be adapted to variously sized groups. An 8:1 ratio of participants to facilitators was ideal for activities. Topics covered included considerations before accepting an invitation, the review process, the good citizen approach to reviewing, and reviewer ethics. The session concluded with suggestions for continued development of peer reviewer competencies. The workshop was evaluated using a custom survey. Results: Throughout 2019 and early 2020, 58 health professions educators and trainees participated in the workshop across varied venues. Evaluations were obtained from 33 participants (57%). Nearly all rated the workshop as high quality and valuable to peer reviewer preparation. Most (26 of 33; 75%) gained confidence about their qualifications to serve as reviewers. Eighty percent (28 of 33) believed they could recognize ethical dilemmas. Discussion: This workshop provided a springboard for peer reviewing health professions education scholarship. Participants generally praised the experience for introducing them to the world of peer review and preparing them for it.

Defining Successful Practice Within Health Systems Science Among Entering Residents: A Single-Institution Qualitative Study of Graduate Medical Education Faculty Observations.

PURPOSE: The American Medical Association's Accelerating Change in Medical Education consortium defined health systems science (HSS) as the study of how health care is delivered, how health care professionals work together to deliver that care, and how the health system can improve patient care and health care delivery. This framework is increasingly being incorporated into medical school curricula. Graduate medical education (GME) had previously elevated systems-based practice (SBP) as a core competency, but expectations are defined by specialty-specific milestones. The lack of a shared competency framework between undergraduate medical education (UME) and GME makes it challenging to ensure that entering residents are prepared to implement HSS/SBP concepts in the workplace. The authors explored GME faculty observations of residents exemplifying successful practice across HSS domains to inform targets for UME training and assessment. METHOD: Authors performed a single-institution qualitative study using transcribed phone interviews with eligible Vanderbilt residency program directors, associate program directors, and core faculty. Participants described observations of successful residents within each HSS domain. Two researchers independently coded, discussed, and reconciled deidentified transcripts using inductive-deductive approaches to identify themes. RESULTS: Seventeen faculty participated across specialties (17/39, 45%). Faculty responses emphasize precurricular experiences including professional degrees, work experience, extracurriculars, and medical school exposure. Importantly, successful residents exhibit foundational core workforce characteristics including growth mindset, curiosity, and a desire to learn about systems. GME faculty identified HSS domain-specific skills, noting distinctions among learning environments. Outcomes resulting from residents' application of HSS concepts include delivering high-quality, person-centered care and systems improvements. CONCLUSIONS: Descriptions of successful practice within HSS domains highlight preparatory experiences and core workforce characteristics and outline entry-level HSS behaviors. Conceptualized in a logic model framework, these findings describe key inputs, learning activities, outputs, and outcomes for systems-prepared entering residents bridging the UME-GME transition.

The International Literature on Teaching Faculty Development in English-Language Journals: A Scoping Review and Recommendations for Core Topics.

BACKGROUND: With increasing physician mobility, there is interest in how medical schools and postgraduate medical education institutions across the world develop and maintain the competence of medical teachers. Published reviews of faculty development (FD) have predominantly included studies from the United States and Canada. OBJECTIVE: We synthesized the international FD literature (beyond the US and Canada), focusing on FD type, intended audience, study format, effectiveness, differences among countries, and potential unique features. METHODS: We identified English-language publications that addressed FD for medical faculty for teaching and related activities, excluding US and Canadian publications. RESULTS: A search of 4 databases identified 149 publications, including 83 intervention studies. There was significant growth in international FD publications for the most recent decade, and a sizable number of studies were from developing economies and/or resulted from international collaborations. Focal areas echo those in earlier published reviews, suggesting the international FD literature addresses similar faculty needs and organizational concerns. CONCLUSIONS: The growth in publications in recent years and a higher proportion of reporting on participant reactions, coupled with less frequent reporting of results, transfer to practice, and impact on learners and the organization, suggest this is an evolving field. To enhance international FD, educators and researchers should focus on addressing common needs expressed by faculty, including curriculum design and evaluation, small group teaching, assessing professionalism and providing feedback. Future research should focus on approaches for developing comprehensive institutional FD programs that include communities of learning and practice and evaluating their impact.

Conversation Starter: Advancing the Theory of Peer-Assisted Learning.

This Conversation Starter article uses four selected abstracts, one each from the four regional Association of American Medical Colleges (AAMC) Group on Educational Affairs (CGEA) 2018 spring meetings, as a springboard for unpacking the definition of peer-assisted learning (PAL). The aim of this article is to prompt deeper reflection on this phenomenon and, in so doing, to foster scholarly program evaluation of this widely adopted instructional approach. This analysis calls for a more nuanced definition of PAL, one that emphasizes process over structure, one that stimulates examination of "doing" PAL and how this affects the personal and professional development of all involved.

Pilot Qualitative Study of Informal Teachers in Interprofessional Collaboration and Practice.

CONTEXT: Nonphysician members of the interprofessional palliative care team often participate in teaching physicians and others in the context of workplace learning due to the interprofessional collaborative nature of the specialty. OBJECTIVES: This pilot study examines the beliefs of the nonphysician members of the interprofessional team about teaching physicians-in-training, the disciplinary training and expertise that informs their teaching, and approaches to teaching in the workplace. METHODS: Semistructured interviews were conducted. All interviews were audio recorded and transcribed verbatim. Initial open coding by two researchers identified the codes, and then the constant comparative method was used to find patterns by axial coding, categories, and themes within the data. RESULTS: Of the 10 health care professionals involved with palliative medical education at one academic medical center, six enrolled in the pilot. Those who participated included chaplains, nurses, a social worker, and a physician assistant. Three major themes were identified from the informal teachers: 1) using professional identity as a foundation for teaching, 2) teaching through experiential learning or debriefing, and 3) teaching to perceived gaps in physician training. CONCLUSION: Nonphysician members of the interprofessional team interacted with physicians-in-training guided by their discipline-based skills and perspectives on patient care. They directed their informal teaching toward perceived educational gaps using reflection and debriefing. Future studies could explore the educational roles of health care professionals across diverse institutions and specialties.

Twelve tips for developing, implementing, and sustaining medical education fellowship programs: Building on new trends and solid foundations.

Medical education fellowship programs (MEFPs) are a form of faculty development contributing to an organization's educational mission and participants' career development. Building an MEFP requires a systematic design, implementation, and evaluation approach which aligns institutional and individual faculty goals. Implementing an MEFP requires a team of committed individuals who provide expertise, guidance, and mentoring. Qualified MEFP directors should utilize instructional methods that promote individual and institutional short and long term growth. Directors must balance the use of traditional design, implementation, and evaluation methodologies with advancing trends that may support or threaten the acceptability and sustainability of the program. Drawing on the expertise of 28 MEFP directors, we provide twelve tips as a guide to those implementing, sustaining, and/or growing a successful MEFP whose value is demonstrated by its impacts on participants, learners, patients, teaching faculty, institutions, the greater medical education community, and the population's health.

Validity evidence of resident competency ratings and the identification of problem residents.

OBJECTIVES: This study examined validity evidence of end-of-rotation evaluations used to measure progress toward mastery of core competencies in residents. In addition, this study investigated whether end-of-rotation evaluations can be used to detect problem residents during their training. METHODS: Historical data for a 4-year period (2009-2012), containing 4986 observations of 291 internal medicine residents, were examined. Residents were observed and assessed by fellows, faculty members and programme directors on nine domains, including the six Accreditation Council for Graduate Medical Education core competencies, as part of their end-of-rotation evaluations. Descriptive statistics were used to collect evidence of the response process. Correlations between competencies and a generalisability study were used to examine the internal structure of the end-of-rotation evaluations. Hierarchical regression was used to estimate the increase in scores across years of training. Scores on end-of-rotation evaluations were compared with trainees identified as problem residents by programme directors. RESULTS: Compared with fellows, faculty and programme directors had significantly greater variability in assigning scores across different competencies. Correlations between competencies ranged from 0.69 to 0.92. The reliability of end-of-rotation evaluations was adequate (fellows, phi coefficient [φ] = 0.68; faculty [including programme directors], φ = 0.71). Mean scores increased by 0.21 points (95% confidence interval 0.18-0.24) per postgraduate year. Mean scores were significantly correlated with classification as a problem resident (r = 0.33, p < 0.001); problem residents also had significantly lower ratings across all competencies during PGY-1 compared with all other residents. CONCLUSIONS: End-of-rotation evaluations are a useful method of measuring the growth in resident performance associated with core competencies when sufficient numbers of end-of-rotation evaluation scores are used. Furthermore, end-of-rotation evaluation scores provide preliminary evidence with which to detect and predict problem residents in subsequent postgraduate training years.

Clinically discriminating checklists versus thoroughness checklists: improving the validity of performance test scores.

PURPOSE: High-quality checklists are essential to performance test score validity. Prior research found that physical exam checklists of items that clinically discriminated between competing diagnoses provided more generalizable scores than all-encompassing thoroughness checklists. The purpose of this study was to compare validity evidence for clinically discriminating versus thoroughness checklists, hypothesizing that evidence would favor the former. METHOD: Faculty at four Chicago-area medical schools developed six standardized patient (SP) cases with checklists of about 20 items ("thoroughness [long] checklists"). Four clinicians identified a subset of items that clinically discriminated between competing diagnoses of each case ("clinically discriminating [short] checklists"). Cases were administered to 155 University of Illinois at Chicago fourth-year medical students during their 2011 Clinical Skills Examination (CSE). Validity evidence was compared for CSE scores based on thoroughness versus clinically discriminating checklist items. RESULTS: Validity evidence favoring clinically discriminating checklists included response process: greater SP checklist accuracy (kappa = 0.75 for long and 0.84 for short checklists, P < .05); internal structure: better item discrimination (0.28 long, 0.42 short, P < .001); internal consistency reliability (0.80 long, 0.92 short); standard error of measurement (z score 8.87 long, 8.05 short); and generalizability (G = 0.504 long, 0.533 short). There were no significant differences overall in relevance ratings, item difficulty, or cut scores of long versus short checklist items. CONCLUSIONS: Limiting checklist items to those affecting diagnostic decisions resulted in better accuracy and psychometric indices. Thoroughness items performed without thinking do not reflect clinical reasoning ability and contribute construct-irrelevant variance to scores.

Validity evidence for a patient note scoring rubric based on the new patient note format of the United States Medical Licensing Examination.

PURPOSE: This study examines validity evidence for the Patient Note Scoring Rubric, which was developed for a local graduation competency exam (GCE) to assess patient notes written in the new United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills format. The rubric was designed to measure three dimensions: Documentation, justified differential diagnosis (DDX), and Workup. METHOD: Analyses used GCE data from 170 fourth-year medical students who completed five standardized patient (SP) cases in May 2012. Five physician raters each scored all responses for one case. Internal structure was examined using correlations between dimensions and between cases; a generalizability study was also conducted. Relationship to other variables was examined by correlating patient note scores with SP encounter scores. Consequence was assessed by comparing pass-fail rates between the rubric and the previous global rating. Response process was examined using rater feedback. RESULTS: Correlations between scores from different dimensions ranged between 0.33 and 0.44. Reliability of scores based on the phi coefficient was 0.43; 15 cases were required to reach a phi coefficient of 0.70. Evidence of case specificity was found. Documentation scores were moderately correlated with SP scores for data gathering (r = 0.47, P < .001). There was no meaningful change in pass-fail rates. Raters' feedback indicated that they required more training for scoring the DDX and Workup dimensions. CONCLUSIONS: There is initial validity evidence for use of this rubric to score local clinical exams that are based on the new USMLE patient note format.

The use of peer optic nerve photographs for teaching direct ophthalmoscopy.

OBJECTIVE: To use a novel teaching exercise to encourage students to practice ophthalmoscopy and to measure the learning effect both subjectively and objectively. DESIGN: Comparative case series. PARTICIPANTS: One hundred thirty-one fourth-year medical students on their 1-week ophthalmology rotations with 89 in the experimental group and 42 in the control group. METHODS: Those in the experimental group had 1 eye dilated and their optic nerve photographed on the first day. The next day, these students received an unlabeled optic nerve photograph belonging to 1 of their peers (typically 8-10 per group) and were given 3 days to identify the student matching the photograph. The students in the control group were simply encouraged to practice ophthalmoscopy on each other without the use of photographs. MAIN OUTCOME MEASURES: Both objective and subjective changes from the beginning to the end of the rotation were measured and compared between the 2 groups. RESULTS: In the 89 students who used peer optic nerve photographs, 75 (84.3%) showed improvement in direct ophthalmoscopy skills over the course of the week. In contrast, only 12 (28.6%) of the 42 control students demonstrated an objective improvement (P<0.001). The subjective confidence levels likewise were more improved in the students who took part in the optic nerve photograph exercise. CONCLUSIONS: These results suggest that the task of matching an unknown optic nerve photograph to the correct eye of a peer leads to increased self-confidence and more proficient use of the direct ophthalmoscope.

Developing and assessing initiatives designed to improve clinical teaching performance.

To improve the teaching performance of emergency physicians, it is necessary to understand the attributes of expert teachers and the optimal methods to deliver faculty development. A working group of medical educators was formed to review the literature, summarize what is known on the topic, and provide recommendations for future research. This occurred as a track of the 2012 Academic Emergency Medicine (AEM) consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success." The group concluded that the current state of research on these topics is limited. Improvement in understanding will come through research focusing on Kirkpatrick's higher levels of evaluation (behavior and results).