What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study.

PURPOSE: The purpose of this study was to enrich understanding about the perceived benefits and drawbacks of constructed response short-answer questions (CR-SAQs) in preclerkship assessment using Norcini's criteria for good assessment as a framework. METHODS: This multi-institutional study surveyed students and faculty at three institutions. A survey using Likert scale and open-ended questions was developed to evaluate faculty and student perceptions of CR-SAQs using the criteria of good assessment to determine the benefits and drawbacks. Descriptive statistics and Chi-square analyses are presented, and open responses were analyzed using directed content analysis to describe benefits and drawbacks of CR-SAQs. RESULTS: A total of 260 students (19%) and 57 faculty (48%) completed the survey. Students and faculty report that the benefits of CR-SAQs are authenticity, deeper learning (educational effect), and receiving feedback (catalytic effect). Drawbacks included feasibility, construct validity, and scoring reproducibility. Students and faculty found CR-SAQs to be both acceptable (can show your reasoning, partial credit) and unacceptable (stressful, not USMLE format). CONCLUSIONS: CR-SAQs are a method of aligning innovative curricula with assessment and could enrich the assessment toolkit for medical educators.

Development and Implementation of a Medical School Course Integrating Basic, Clinical, and Health Systems Sciences.

OBJECTIVE: In recent years, significant steps have been made in integrating basic science and clinical medicine. There remains a gap in adding the third pillar of education: health systems science (HSS). Core clerkships represent an ideal learning venue to integrate all three. Students can experience the value of integrating basic science as they learn clinical medicine in environments where HSS is occurring all around them. METHODS: We outline the creation of Sciences and Art of Medicine Integrated (SAMI), a course that runs parallel with the clerkship year and integrates basic science and HSS with clinical medicine. A complete description of the planning and implementation of SAMI is provided. We include the participants and educational setting, the goals and objectives, and the structure of each session. To encourage the integration of basic science, HSS, and clinical medicine, students utilize a series of tools, described in detail. Examples of each tool are provided utilizing a case of a patient presenting with obstructive sleep apnea. RESULTS: We successfully implemented this course with positive reception from students. CONCLUSION: This course represents a step not only toward the integration of HSS with basic science and clinical medicine but also an advancement in training future clinicians to provide high-value care. Future curricular development must consider the validation of a measure of clinical reasoning that assesses a student's ability to think in a cognitively integrated fashion about basic science, HSS, and clinical medicine demonstrated by enhanced justification of clinical reasoning and a more holistic approach to planning patient care.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study.

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18 years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

A Multi-institutional Study of the Feasibility and Reliability of the Implementation of Constructed Response Exam Questions.

PROBLEM: Some medical schools have incorporated constructed response short answer questions (CR-SAQs) into their assessment toolkits. Although CR-SAQs carry benefits for medical students and educators, the faculty perception that the amount of time required to create and score CR-SAQs is not feasible and concerns about reliable scoring may impede the use of this assessment type in medical education. INTERVENTION: Three US medical schools collaborated to write and score CR-SAQs based on a single vignette. Study participants included faculty question writers (N = 5) and three groups of scorers: faculty content experts (N = 7), faculty non-content experts (N = 6), and fourth-year medical students (N = 7). Structured interviews were performed with question writers and an online survey was administered to scorers to gather information about their process for creating and scoring CR-SAQs. A content analysis was performed on the qualitative data using Bowen's model of feasibility as a framework. To examine inter-rater reliability between the content expert and other scorers, a random selection of fifty student responses from each site were scored by each site's faculty content experts, faculty non-content experts, and student scorers. A holistic rubric (6-point Likert scale) was used by two schools and an analytic rubric (3-4 point checklist) was used by one school. Cohen's weighted kappa (κw) was used to evaluate inter-rater reliability. CONTEXT: This research study was implemented at three US medical schools that are nationally dispersed and have been administering CR-SAQ summative exams as part of their programs of assessment for at least five years. The study exam question was included in an end-of-course summative exam during the first year of medical school. IMPACT: Five question writers (100%) participated in the interviews and twelve scorers (60% response rate) completed the survey. Qualitative comments revealed three aspects of feasibility: practicality (time, institutional culture, teamwork), implementation (steps in the question writing and scoring process), and adaptation (feedback, rubric adjustment, continuous quality improvement). The scorers' described their experience in terms of the need for outside resources, concern about lack of expertise, and value gained through scoring. Inter-rater reliability between the faculty content expert and student scorers was fair/moderate (κw=.34-.53, holistic rubrics) or substantial (κw=.67-.76, analytic rubric), but much lower between faculty content and non-content experts (κw=.18-.29, holistic rubrics; κw=.59-.66, analytic rubric). LESSONS LEARNED: Our findings show that from the faculty perspective it is feasible to include CR-SAQs in summative exams and we provide practical information for medical educators creating and scoring CR-SAQs. We also learned that CR-SAQs can be reliably scored by faculty without content expertise or senior medical students using an analytic rubric, or by senior medical students using a holistic rubric, which provides options to alleviate the faculty burden associated with grading CR-SAQs.

Preparing Students to Deliver Care to Diverse Populations.

The purpose of this study was to increase student exposure to diverse patients using patient ID cards in problem-based learning (PBL) at Case Western Reserve University (CWRU). The pre-clerkship curriculum capitalizes on facilitated small-group, case-based discussions to promote inquiry and learning of the foundational sciences. Quantitative and qualitative results supported a finding of added value to case-discussions and the humanization of case-patients. The inclusion of the patient ID cards resulted in most students indicating that it helped them learn about and prepare to care for their future population of diverse patients. The patient ID cards will allow us to develop specific learning objectives about the demographics to increase learning about diverse patient care. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-022-01648-0.

Disparities Education Strategies in the Case Comprehensive Cancer Center STEM Focused Youth Enjoy Science Program.

The Youth Enjoy Science/Scientific Enrichment and Opportunity (YES/SEO) Program at Case Western Reserve University (CWRU) School of Medicine and the Case Comprehensive Cancer Center (Case CCC) in Cleveland, OH is an intensive Research, Science, Technology, Engineering, and Mathematics (STEM) program targeted at engaging underrepresented minority high school students to better understand and to pursue careers in cancer research and healthcare. The program's long-range goals are to increase diversity of the cancer professional workforce to contribute to elimination of cancer health inequities. A challenging aspect of this intensive research and STEM education program is how to effectively teach about cancer health disparities and to address the importance of developing strategies for their remediation. We describe herein some innovative approaches utilized to engage students in learning about disparities and thinking about solutions. Overall, feedback from our students indicates the importance of introducing disparities education topics often and using multiple approaches, including small and large meetings as well as lecture and conversational formats. These approaches provide opportunities for frequent student engagement and concept reinforcement. Based on this experience, a series of recommendations are provided for incorporating disparities education into intensive research and STEM programs.

Early Medical Students' Experiences as System Navigators: Results of a Qualitative Study.

PURPOSE: To explore how early meaningful experiential learning in community settings impacted medical students' application of systems thinking, their perceptions of systems navigation, and their professional identity as health system change agents. METHODS: Following an immersive Health Systems Science course, first-year medical students partnered with veterans or newly arrived refugee families and served as health system patient navigators embedded within primary care teams for a year. Across two cohorts, fifty-six students participated in the elective. Three voluntary focus groups were conducted each year for a total of six groups with 50 patient navigator students. Inductive content analysis of focus group transcripts was conducted. RESULTS: Qualitative analysis produced three major themes: program impact on students, student impact on patients, and student perceptions of the role of healthcare providers. Students reported a rich understanding of social determinants of health. By improving patient awareness of health and well-being, building capacity to understand medical issues, and increasing medication adherence through teaching, students recognized their impact on patient care. The importance of interprofessional collaboration with social workers also emerged and helped shape students' understanding of how they as physicians are part of a coordinated team working toward better patient care. CONCLUSION: The Case Western Reserve University WR2 curriculum teaches students how to address complex determinants of health and how to consider their role in dynamic health systems. This study highlights rich themes that emerged from students as they recognized the context that creates health for both individuals and communities. It underscores the role of such experiences in reinforcing systems thinking and development of change agency, both contributing to their professional identity formation as physicians.

A rasch analysis of three socialization and communication measures in 4th-year doctor of pharmacy students.

The impact of communication and anxiety on Doctor of Pharmacy students across three measures was examined. Data were collected (N = 120) from 4th-year Doctor of Pharmacy students at a historically black college/university using the Interprofessional Socialization and Valuing Scale, the Personal Report of Communication Apprehension, and the Social Phobia Inventory. Results of Rasch Differential Item Functioning Analysis indicated statistically significant differences between each ethnic group on subcomponents of anxiety in each measure (20% Caucasian, 43% African American, 31.7% Asian, and 4.2% others). Evidence from this study shows that racial demographics affect different subscales of anxiety across doctoral pharmacy students. While some pedagogical implications exist, issues within the measures and their items must also be addressed.

Sustaining Student Engagement - Successes and Challenges of a Virtual STEM Program for High School Students.

Case Western Reserve University's School of Medicine and Comprehensive Cancer Center coordinate in-depth research immersion STEM programs to engage high school students in biomedical research and encourage pursuit of careers in health-related research and clinical care. Due to COVID-19, the 2020 programs were delivered entirely virtually. Student and faculty perceptions of the virtual experience were evaluated using surveys and focus groups. Ninety percent of students felt the virtual program met expectations. Student rankings for programmatic components that could remain virtual in future years showed a preference for highly interactive activities, especially mentorship and dialogue-based activities like discussions of science in the news. Ninety-seven percent of faculty agreed students' scientific knowledge improved. Faculty commented that certain research projects (e.g., data analysis, literature reviews) were highly appropriate for a virtual program, but that the lack of hands-on laboratory activities was challenging. Increased individual attention, flexibility, and independence were hailed as strengths of the virtual program. These findings identify activities that sustain student interest in biomedical, healthcare, and cancer related research using a virtual medium and indicate mentorship and interactive discussion-based activities enhance virtual education. Moreover, the results support incorporation of interactive online pedagogical approaches to enhance student engagement virtually and in-person.

Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study.

BACKGROUND: Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal. OBJECTIVE: This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators. METHODS: The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results. RESULTS: A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001). CONCLUSIONS: Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.

Is remote near-peer anatomy teaching an effective teaching strategy? Lessons learned from the transition to online learning during the Covid-19 pandemic.

In response to the Covid-19 pandemic, medical educators have transformed pre-clerkship anatomy curricula into online formats. The purpose of this study was to evaluate the effectiveness and student perceptions of an online near-peer anatomy curriculum. The classes of 2022 and 2023 completed identical foundational anatomy curricula in-person, whereas the class of 2024 completed an adapted curriculum for remote online learning. Quantitative and qualitative responses were used to compare attitudes between instructional methods. Assessment scores and evaluation survey responses were collected from the classes of 2022 (n = 185), 2023 (n = 184), and 2024 (n = 183). Mean assessment scores (±SD) for the classes of 2022, 2023, and 2024 were 93.64% (±5.86), 93.75% (±4.09), and 92.04% (±4.83), respectively. Post hoc group comparisons showed the class of 2024 scored significantly lower than the two previous classes [2022: (H(1) = 18.58, P < 0.001), 2023: (H(1) = 18.65, P < 0.001)]. Mean survey results concerning curriculum quality were 4.06/5.00 for the class of 2023 and 3.57/5.0 for the class of 2024 (t(365) = 2.67, P = 0.008). Considering a small effect size (η2 = 0.034), there was no meaningful difference in student assessment scores. A potential drawback of online near-peer anatomy teaching remains in student perceptions of course quality; qualitative feedback suggested technological limitations and perceptions of online course instructors were partly responsible for lower student satisfaction. Following the Covid-19 pandemic, medical educators should incorporate the lessons learned from this unique educational inflection point to improve curricula moving forward.

A retrospective case study of successful translational research: Gazelle Hb variant point-of-care diagnostic device for sickle cell disease.

Evaluation researchers at Clinical and Translational Science Award (CTSA) hubs are conducting retrospective case studies to evaluate the translational research process. The objective of this study was to deepen knowledge of the translational process and identify contributors to successful translation. We investigated the successful translation of the HemeChip, a low-cost point-of-care diagnostic device for sickle cell disease, using a protocol for retrospective translational science case studies of health interventions developed by evaluators at the National Health Institutes (NIH) and CTSA hubs. Development of the HemeChip began in 2013 and evidence of device use and impact on public health is growing. Data collection methods included five interviews and a review of press, publications, patents, and grants. Barriers to translation included proving novelty, manufacturing costs, fundraising, and academic-industry relations. Facilitators to translation were CTSA pilot program funding, university resources, entrepreneurship training, due diligence, and collaborations. The barriers to translation, how they were overcome, and the key facilitators identified in this case study pinpoint areas for consideration in future funding mechanisms and the infrastructure required to enable successful translation.

Youth Enjoy Science Program at the Case Comprehensive Cancer Center: Increasing Engagement and Opportunity for Underrepresented Minority Students.

The Youth Enjoy Science (YES) Program at the Case Comprehensive Cancer Center is a National Cancer Institute (NCI) R25-funded training grant, designed to increase the pipeline of underrepresented minority (URM) students entering college and pursuing biomedical research and health care careers in the Cleveland Metropolitan and surrounding school districts. The three components of the program include: Learn to Beat Cancer, engaging middle school students and their families; Research to Beat Cancer, designed for high school students and college undergraduates; and Teach to Beat Cancer, focused on enhancing science, technology, engineering, and mathematics (STEM) teaching capacity among high school teachers. This study focuses on Research to Beat Cancer, which, in 2018 enrolled 36 URM students as paid summer scholars. Students were assigned to a faculty mentor, were taught laboratory safety, responsible conduct of research and the scientific method, and then immersed in full-time laboratory cancer research during an eight-week period. Twice each week, students participated in Lunch and Learn Seminars where faculty members provided combined motivational and scientific guidance lectures. In a capstone poster session at the end of the program, students presented their research to peers, medical and graduate students, family members, faculty, community members and leaders. Students' perceptions of the program were reported using descriptive statistics and qualitative thematic analyses. Twenty-four of the 2018 YES students (67%) and 19 (53%) mentors completed the online post-program survey. Opportunity was a major qualitative theme from student and mentor responses. Future research will investigate the long-term impacts of YES, including college enrollment.

Near Peer Mentors to Address Socio-Emotional Issues Among Underrepresented Minority High School Students in Research Intensive STEM Programs: Perceptions of Students and Mentors.

A Near Peer Mentoring Program (NPMP) was developed in which Medical Student Training Program (MSTP) students met weekly with small groups of high school students who were participating in an intensive summer biomedical research immersion program. The goal of the NPMP was to provide and engage the high school students with opportunities to express and discuss their research and more importantly, their stresses and concerns. After initial reservations, the NPMP provided a comfortable venue for high school students to engage in discussions of both laboratory and personal topics. Overall, their concerns and stresses were expressed in five categories: 1) College Preparation, 2) Preparation for MD and PhD Training and Careers, 3) Summer Research Programmatic Issues and Laboratory Social Structure, 4) Social Issues, and 5) Health and Wellness. High school students identified the following major factors as contributing to programmatic success: relatability, role models, comfort and approachability, organization, and mentor fit. The Near Peer Mentoring initiative revealed the need for STEM and other programs targeting academic success and career development to be alert to social and emotional concerns of students and to provide opportunities for their expression, discussion and guidance.

Using bibliometrics to evaluate translational science training: evidence for early career success of KL2 scholars.

INTRODUCTION: Evaluating clinical and translational research (CTR) mentored training programs is challenging because no two programs are alike. Careful selection of appropriate metrics is required to make valid comparisons between individuals and between programs. The KL2 program provides mentored-training for early-stage CTR investigators. Clinical and Translational Awards across the country have unique KL2 programs. The evaluation of KL2 programs has begun to incorporate bibliometrics to measure KL2 scholar and program impact. METHODS: This study investigated demographic differences in bibliometric performance and post-K award funding of KL2 scholars and compared the bibliometric performance and post-K award federal funding of KL2 scholars and other mentored-K awardees at the same institution. Data for this study included SciVal and iCite bibliometrics and National Institutions of Health RePORTER grant information for mentored-K awardees (K08, K23, and KL2) at Case Western Reserve University between 2005 and 2013. RESULTS: Results showed no demographics differences within the KL2 program scholars. Bibliometric differences between KL2 and other mentored-K awardee indicated an initial KL2 advantage for the number of publications at 5 years' post-matriculation (i.e., the start of the K award). Regression analyses indicated the number of initial publications was a significant predictor of federal grant funding at the same time point. Analysis beyond the 5-year post-matriculation point did not result in a sustained, significant KL2 advantage. CONCLUSIONS: Factors that contributed to the grant funding advantage need to be determined. Additionally, differences between translational and clinical bibliometrics must be interpreted with caution, and appropriate metrics for translational science must be established.