Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships.

PURPOSE: To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation. METHOD: This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system. RESULTS: Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning. CONCLUSIONS: Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.

Moving toward Mastery: Changes in Student Perceptions of Clerkship Assessment with Pass/Fail Grading and Enhanced Feedback.

ProblemClerkship grades contribute to a summative assessment culture in clerkships and can therefore interfere with students' learning. For example, by focusing on summative, tiered clerkship grades, students often discount accompanying feedback that could inform future learning. This case report seeks to explore whether an assessment system intervention which eliminated tiered grades and enhanced feedback was associated with changes in student perceptions of clerkship assessment and perceptions of the clinical learning environment. Intervention: In January 2019, our institution eliminated tiered clerkship grading (honors/pass/fail) for medical students during the core clerkship year and implemented pass/fail clerkship grading along with required twice weekly, work-based assessments for formative feedback. Context: In this single institution, cross-sectional survey study, we collected data from fourth-year medical students one year after an assessment system intervention. The intervention entailed changing from honors/pass/fail to pass/fail grading in all eight core clerkships and implementing an electronic system to record twice-weekly real-time formative work-based assessments. The survey queried student perceptions on the fairness and accuracy of grading and the clinical learning environment-including whether clerkships were mastery- or performance-oriented. We compared responses from students one year after the assessment intervention to those from the class one year before the intervention. Comparisons were made using unpaired, two-tailed t-tests or chi-squared tests as appropriate with Cohen's d for effect size estimation for score differences. Content analysis was used to analyze responses from two open-ended questions about feedback and grading. Impact: Survey response rates were similar before and after intervention (76% (127/168) vs. 72% (118/163), respectively) with no between-group differences in demographics. The after-intervention group showed statistically significant increases in the following factors: "grades are transparent and fair" (Cohen's d = 0.80), "students receive useful feedback" (d = 0.51), and "resident evaluation procedures are fair" (d = 0.40). After-intervention respondents perceived the clerkship learning environment to be more mastery-oriented (d = 0.52), less performance approach-oriented (d = 0.63), and less performance avoid-oriented (d = 0.49). There were no statistical differences in the factors "attending evaluation procedures are fair," "evaluations are accurate," "evaluations are biased," or "perception of stereotype threat." Open-ended questions revealed student recommendations to improve clerkship summary narratives, burden of work-based assessment, and in-person feedback. Lessons Learned: After an assessment system change to pass/fail grading with work-based assessments, we observed moderate to large improvements in student perceptions of clerkship grading and the mastery orientation of the learning environment. Our intervention did not improve perceptions around bias in assessment in clerkships. Other medical schools may consider similar interventions to begin to address student concerns with clerkship assessment and promote a more adaptive learning environment.

From Grading to Assessment for Learning: A Qualitative Study of Student Perceptions Surrounding Elimination of Core Clerkship Grades and Enhanced Formative Feedback.

Problem: Medical students perceive honors grading during core clerkships as unfair and inequitable, and negatively impacting their learning and wellbeing. Eliminating honors grading, a powerful extrinsic motivator, and emphasizing formative feedback may address these problems and promote intrinsic motivation and learning. However, it is unknown how transitioning from honors to pass/fail grading with enhanced formative feedback in the core clerkship year may affect student learning experiences, wellbeing, and perceptions of the learning environment. Intervention: Core clerkship grading was transitioned from honors/pass/fail to pass/fail at one US medical school. In addition, the requirement for students to obtain formative supervisor feedback was formalized to twice per week. Context: This qualitative study utilized semi-structured interviews to explore the perceptions among core clerkship students of learning and assessment. Interview questions addressed motivation, wellbeing, learning behaviors, team dynamics, feedback, and student and supervisor attitudes regarding assessment changes. The authors analyzed data inductively using thematic analysis informed by sensitizing concepts related to theories of motivation (goal orientation theory and self-determination theory). Impact: Eighteen students participated, including five with experience in both honors-eligible and pass/fail clerkships. The authors identified three major themes in students' descriptions of the change in approach to assessment: student engagement in clerkships, wellbeing, and recognition of learning context. Student engagement subthemes included intrinsic motivation for patient care rather than performing; sense of agency over learning, including ability to set learning priorities, seek and receive feedback, take learning risks, and disagree with supervisors, and collaborative relationships with peers and team members. Positive wellbeing was characterized by low stress, sense of authenticity with team members, prioritized physical health, and attention to personal life. Learning context subthemes included recognition of variability of clerkship contexts with pass/fail grading mitigating fairness and equity concerns, support of the grading change from residents and some attendings, and implications surrounding future stress and residency selection. Lessons Learned: Students perceive a transition from honors grading to pass/fail with increased feedback as supporting their engagement in learning, intrinsic motivation, and wellbeing. Drivers of wellbeing appear to include students' feelings of control, achieved through the ability to seek learning opportunities, teaching, and constructive feedback without the perceived need to focus on impressing others. Ongoing evaluation of the consequences of this shift in assessment is needed.

Integrated PET-MRI for Glioma Surveillance: Perfusion-Metabolism Discordance Rate and Association With Molecular Profiling.

OBJECTIVE: Both 18F-FDG PET and perfusion MRI are commonly used techniques for posttreatment glioma surveillance. Using integrated PET-MRI, we assessed the rate of discordance between simultaneously acquired FDG PET images and dynamic contrast-enhanced (DCE) perfusion MR images and determined whether tumor genetics predicts discordance. MATERIALS AND METHODS: Forty-one consecutive patients with high-grade gliomas (20 with grade IV gliomas and 21 with grade III gliomas) underwent a standardized tumor protocol performed using an integrated 3-T PET-MRI scanner. Quantitative measures of standardized uptake value, plasma volume, and permeability were obtained from segmented whole-tumor volumes of interest and targeted ROIs. ROC curve analysis and the Youden index were used to identify optimal cutoffs for FDG PET and DCE-MRI. Two-by-two contingency tables and percent agreement were used to assess accuracy and concordance. Twenty-six patients (63%) from the cohort underwent next-generation sequencing for tumor genetics. RESULTS: The best-performing FDG PET and DCE-MRI cutoffs achieved sensitivities of 94% and 91%, respectively; specificities of 56% and 89%, respectively; and accuracies of 80% and 83%, respectively. FDG PET and DCE-MRI findings were discordant for 11 patients (27%), with DCE-MRI findings correct for six of these patients (55%). Tumor grade, tumor volume, bevacizumab exposure, and time since radiation predicted discordance between FDG PET and DCE-MRI findings, with an ROC AUC value of 0.78. Isocitrate dehydrogenase gene and receptor tyrosine kinase gene pathway mutations increased the ROC AUC value to 0.83. CONCLUSION: FDG PET and DCE-MRI show comparable accuracy and sensitivity in identifying tumor progression. These modalities were shown to have discordant findings for more than a quarter of the patients assessed. Tumor genetics may contribute to perfusion-metabolism discordance, warranting further investigation.