The reliability of the College of Intensive Care Medicine of Australia and New Zealand "Hot Case" examination.

BACKGROUND: High stakes examinations used to credential trainees for independent specialist practice should be evaluated periodically to ensure defensible decisions are made. This study aims to quantify the College of Intensive Care Medicine of Australia and New Zealand (CICM) Hot Case reliability coefficient and evaluate contributions to variance from candidates, cases and examiners. METHODS: This retrospective, de-identified analysis of CICM examination data used descriptive statistics and generalisability theory to evaluate the reliability of the Hot Case examination component. Decision studies were used to project generalisability coefficients for alternate examination designs. RESULTS: Examination results from 2019 to 2022 included 592 Hot Cases, totalling 1184 individual examiner scores. The mean examiner Hot Case score was 5.17 (standard deviation 1.65). The correlation between candidates' two Hot Case scores was low (0.30). The overall reliability coefficient for the Hot Case component consisting of two cases observed by two separate pairs of examiners was 0.42. Sources of variance included candidate proficiency (25%), case difficulty and case specificity (63.4%), examiner stringency (3.5%) and other error (8.2%). To achieve a reliability coefficient of > 0.8 a candidate would need to perform 11 Hot Cases observed by two examiners. CONCLUSION: The reliability coefficient for the Hot Case component of the CICM second part examination is below the generally accepted value for a high stakes examination. Modifications to case selection and introduction of a clear scoring rubric to mitigate the effects of variation in case difficulty may be helpful. Increasing the number of cases and overall assessment time appears to be the best way to increase the overall reliability. Further research is required to assess the combined reliability of the Hot Case and viva components.

Timing's not everything: Immediate and delayed feedback are equally beneficial for performance in formative multiple-choice testing.

INTRODUCTION: Test-enhanced learning (TEL) is an impactful teaching and learning strategy that prioritises active learner engagement through the process of regular testing and reviewing. While it is clear that meaningful feedback optimises the effects of TEL, the ideal timing of this feedback (i.e. immediate or delayed) in a medical education setting is unclear. METHOD: Forty-one second-year medical students were recruited from the University of Melbourne. Participants were given a multiple-choice question test with a mix of immediate (i.e. post-item) and delayed (i.e. post-item-block) conceptual feedback. Students were then tested on near and far transfer items during an immediate post-test, and at a one-week follow-up. RESULTS: A logistic mixed effects model was used to predict the probability of successful near and far transfer. As expected, participants in our study tended to score lower on far transfer items than they did on near transfer items. In addition, correct initial response on a parent question predicted subsequent correct responding. Contrary to our hypotheses, the feedback timing effect was non-significant-there was no discernible difference between feedback delivered immediately versus delayed feedback. DISCUSSION: The findings of this study suggest that the timing of feedback delivery (post-item versus post-item-block) does not influence the efficacy of TEL in this medical education setting. We therefore suggest that educators may consider practical factors when determining appropriate TEL feedback timing in their setting.

Diversity Audit of Medical School Examination Questions.

Phenomenon: This article reports the under-researched presentation of demographic, social, and economic diversity in medical school examination questions. Approach: The present study audited 3,566 pre-clinical and clinical multiple-choice and short answer examination questions in the same year (2018) from three medical schools in two continents to review the diversity of patients portrayed. The audit was based on an extension of Critical Race Theory beyond race and ethnicity to include pertinent social determinants of health. Findings: Patients were presented in 1,537 (43.1%) of the audited examination questions. Apart from age (89.4%) and binary genders (93.9%), other diversity characteristics were rarely portrayed (ethnicity 7.2%, relationship status 1.9%, sexual identity 1.1%, socio-economic status 0.5%, geographic residence 0.1%, disability 0.1%), or not at all (non-binary genders; residency status; religion/spirituality). Insights: While presenting excessive and unnecessary patient characteristics in examination questions should be avoided, the absence of many diversity aspects may reduce examination authenticity and defeat the teaching of diversity in medicine. Medical schools should consider a routine audit and reasonable improvement of the diversity features of patients in examination questions to support teaching and learning activities addressing patients' diversity.

Football Fandom as a Platform for Digital Health Promotion and Behaviour Change: A Mobile App Case Study.

Background: The last decade has seen a dramatic shift toward the study of fitness surveillance, thanks in part to the emergence of mobile health (mHealth) apps that allow users to track their health through a variety of data-driven insights. This study examines the adoption trends and community mediation of the mobile fitness application 'FanFit', a platform aimed at promoting physical activity among sports fans by creating a fitness app branded to their favourite team for health promotion. Objective: Our study looked at the impact of a specially designed mobile app (FanFit) as a digital health intervention for initiating and maintaining physical activity as part of football club membership. Our analysis indicates that app users will adopt healthier behaviours as a result of the app's sense of fan community and behaviour change. Methods: The findings reported here are based on an implementation of the FanFit app and, in particular, on those who participated in a more in-depth study (n = 30). These participants were Rangers FC supporters with a mix of genders (n = 19 males and n = 11 females). Focus groups and interviews were conducted with participants to ascertain users' perspectives on the most effective methods for nudging users toward adopting and maintaining a pattern of fitness behaviours. Results: The findings show that the user community was interested in fitness and wanted to live a 'healthy lifestyle,' which was augmented and fuelled by the app's competitive architecture design. Furthermore, the data reveal a new fan-health discourse about a person's developing wants, talents, and identities as embodied beings. Conclusions: We have developed and presented valid links between the use of sports club apps and health programmes. The app could be useful for sports programmes and club providers looking for mHealth applications that provide community support through fan discourse with opportunities for both male and female fans.

Too hot to handle? Assessing the validity and reliability of the College of Intensive Care Medicine "Hot Case" examination.

The College of Intensive Care Medicine of Australia and New Zealand is responsible for credentialling trainees for specialist practice in intensive care medicine for the safety of patients and the community. This involves defining trainees' performance standards and testing trainees against those standards to ensure safe practice. The second part examination performed towards the end of the training program is a high-stakes assessment. The two clinical "Hot Cases" performed in the examination have a low pass rate, with most candidates failing at least one of the cases. There is increasing expectation for medical specialist training colleges to provide fair and transparent assessment processes to enable defensible decisions regarding trainee progression. Examinations are a surrogate marker of clinical performance with advantages, disadvantages and inevitable compromises. This article evaluates the Hot Case examination using Kane's validity framework and van der Vleuten's utility equation, and identifies issues with validity and reliability which could be managed through an ongoing improvement process.

Ottawa 2020 consensus statement for programmatic assessment - 1. Agreement on the principles.

INTRODUCTION: In the Ottawa 2018 Consensus framework for good assessment, a set of criteria was presented for systems of assessment. Currently, programmatic assessment is being established in an increasing number of programmes. In this Ottawa 2020 consensus statement for programmatic assessment insights from practice and research are used to define the principles of programmatic assessment. METHODS: For fifteen programmes in health professions education affiliated with members of an expert group (n = 20), an inventory was completed for the perceived components, rationale, and importance of a programmatic assessment design. Input from attendees of a programmatic assessment workshop and symposium at the 2020 Ottawa conference was included. The outcome is discussed in concurrence with current theory and research. RESULTS AND DISCUSSION: Twelve principles are presented that are considered as important and recognisable facets of programmatic assessment. Overall these principles were used in the curriculum and assessment design, albeit with a range of approaches and rigor, suggesting that programmatic assessment is an achievable education and assessment model, embedded both in practice and research. Knowledge on and sharing how programmatic assessment is being operationalized may help support educators charting their own implementation journey of programmatic assessment in their respective programmes.

Ottawa 2020 consensus statements for programmatic assessment - 2. Implementation and practice.

INTRODUCTION: Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts. METHODS: After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis. RESULTS AND DISCUSSION: A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation.

Purpose, Pleasure, Pace and Contrasting Perspectives: Teaching and Learning in the Emergency Department.

OBJECTIVES: Teaching and learning in the clinical setting are vital for the training and development of emergency physicians. Increasing service provision and time pressures in the emergency department (ED) have led to junior trainees' perceptions of a lack of teaching and a lack of support during clinical shifts. We sought to explore the perceptions of learners and supervisors in our ED regarding teaching within this diverse and challenging context. METHODS: Nine ED physicians and eight ED trainees were interviewed to explore perceptions of teaching in the ED. Clinical teaching was described as "on-the-floor" teaching during work shifts. We used a validated clinical teaching assessment instrument to help pilot and develop some of our interview questions, and data were analyzed using qualitative thematic analysis. RESULTS: We identified three major themes in our study: 1) the strong sense of purpose and the pleasure gained through teaching and learning interactions, despite both groups being unsure of each other's engagement and enthusiasm; 2) contrasting perspectives of teaching with registrars holding a traditional knowledge transmission view, yet shared perspectives of teacher as being ED consultants; and 3) the effect of patient acuity and volume, which both facilitated learning until a critical point of busyness beyond which service provision pressures and staffing limitations were perceived to negatively impact learning. CONCLUSIONS: The ED is a complex and fluid working and learning environment. We need to develop a shared understanding of teaching and learning opportunities in the ED, which helps all stakeholders move beyond learning as knowledge acquisition and sees the potential for learning from teachers of a multitude of professional backgrounds.

Rethinking Assessment Design: Evidence-Informed Strategies to Boost Educational Impact in the Anatomical Sciences.

University assessment is in the midst of transformation. Assessments are no longer designed solely to determine that students can remember and regurgitate lecture content, nor in order to rank students to aid with some future selection process. Instead, assessments are expected to drive, support, and enhance learning and to contribute to student self-assessment and development of skills and attributes for a lifetime of learning. While traditional purposes of certifying achievement and determining readiness to progress remain important, these new expectations for assessment can create tensions in assessment design, selection, and deployment. With the recognition of these tensions, three contemporary approaches to assessment in medical education are described. These approaches include careful consideration of the educational impact of assessment-before, during (test or recall enhanced learning) and after assessments; development of student (and staff) assessment literacy; and planning of cohesive systems of assessment (with a range of assessment tools) designed to assess the various competencies demanded of future graduates. These approaches purposefully straddle the cross purposes of assessment in modern health professions education. The implications of these models are explored within the context of medical education and then linked with contemporary work in the anatomical sciences in order to highlight current synergies and potential future innovations when using evidence-informed strategies to boost the educational impact of assessments.

Learning With Patients, Students, and Peers: Continuing Professional Development in the Solo Practitioner Workplace.

Conceptualizations of workplace learning have moved from knowledge acquisition to learning as participation in the practices and cultures of the workplace environment. Along with this has come an appreciation of applicability of sociocultural learning theories, which frame learning as occurring within "communities of practice" or learning being "situated" within a workplace environment where collaboration and social interaction are fundamental to the learning process. These conceptualizations of workplace learning are ideally suited to health professions where learners are supervised in clinical work environments and then continue to work in team-based environments as graduates. However, what happens to workplace learning for novice practitioners who have limited periods of clinical supervision and then graduate into solo or small group practices (which may also be in rural or remote locations) and embark on long working careers without supervision? This paper argues workplace learning needs to be scaffolded and supported to reach its full potential in these environments. Drawing on workplace-based learning theory, we highlight the ubiquitous nature of learning in the workplace, the importance of active engagement, reflection, and individual meaning making. Through this reframing of traditional notions of continuing professional development, we emphasize the importance of patients, students, and other practitioners as partners in workplace learning for solo practitioners. We also focus on the role of educators, professional associations, and regulators in helping solo practitioners recognize, access, and maximize the learning opportunities inherent in relatively isolated practice environments.

Relationship Between Single Word Reading, Connected Text Reading, and Reading Comprehension in Persons With Aphasia.

Purpose This study examined the relationship between single word reading, connected text reading, and comprehension in persons with aphasia. Method Thirteen persons with aphasia read orally from the Arizona Battery for Reading and Spelling real-word and nonword lists and the Gray Oral Reading Tests-Fifth Edition. The comprehension questions following each paragraph of the Gray Oral Reading Tests-Fifth Edition were answered and scored. The Reading Comprehension Battery for Aphasia-Second Edition provided a measure of silent reading comprehension. Descriptive statistics and Spearman correlation were used to examine associations among reading measures. Results Persons with aphasia showed associations between single word reading and connected text reading accuracy; however, single word reading ability was not associated with oral or silent reading comprehension. Conclusions Although preliminary, the findings provide support for word-level reading abilities underlying connected text reading accuracy but suggest additional cognitive mechanisms are involved in text-level reading comprehension that are not explained by single word reading alone. The findings indicate clinicians should use caution when inferring comprehension abilities from single word reading performance as reading comprehension abilities are likely best assessed using text-level comprehension assessments.

Beyond right or wrong: More effective feedback for formative multiple-choice tests.

INTRODUCTION: The role of feedback in test-enhanced learning is an understudied area that has the potential to improve student learning. This study investigates the influence of different forms of post-test feedback on retention and transfer of biomedical knowledge within a test-enhanced learning framework. METHODS: 64 participants from a Canadian and an Australian medical school sat two single-best-answer formative multiple choice tests one week apart. We compared the effects of conceptually focused, response-oriented, and simple right/wrong feedback on a learner's ability to correctly answer new (transfer) questions. On the first test occasion, participants received parent items with feedback, and then attempted items closely related (near transfer) to and more distant (far transfer) from parent items. In a repeat test at 1 week, participants were given different near and far transfer versions of parent items. Feedback type, and near and far transfer items were randomized within and across participants. RESULTS: Analysis demonstrated that response-oriented and conceptually focused feedback were superior to traditional right/wrong feedback for both types of transfer tasks and in both immediate and final retention test performance. However, there was no statistically significant difference between response-orientated and conceptually focused groups on near or far transfer problems, nor any differences in performance between our initial test occasion and the retention test 1 week later. As with most studies of transfer, participants' far transfer scores were lower than for near transfer. DISCUSSION: Right/wrong feedback appears to have limited potential to augment test-enhanced learning. Our work suggests that item-level feedback and feedback that identifies and elaborates on key conceptual knowledge are two important areas for future research on learning, retention and transfer.

Perceived educational impact of the medical student long case: a qualitative study.

BACKGROUND: The long case is a traditional method of clinical assessment which has fallen out of favour in certain contexts, primarily due to psychometric concerns. This study explored the long case's educational impact, an aspect which has been neglected in previous research. METHODS: Three focus groups of medical students (20 in total) and semi-structured interviews of six examiners were conducted. Cook and Lineberry's framework for exploring educational impact was used as a sensitising tool during thematic analysis of the data. RESULTS: Participants described the long case and its scoring as having influence on student learning. Engaging in the activity of a long case had an essential role in fostering students' clinical skills and served as a powerful driving force for them to spend time with patients. The long case was seen as authentic, and the only assessment to promote a holistic approach to patients. Students had concerns about inter-case variability, but there was general consensus that the long case was valuable, with allocation of marks being an important motivator for students. CONCLUSIONS: This study offers a unique focus on the traditional long case's educational consequences; the extent of its positive impact would support its place within a program of assessment.

Exploring Strategies for Using Social Media to Self-Manage Health Care When Living With and Beyond Breast Cancer: In-Depth Qualitative Study.

BACKGROUND: As breast cancer survival rates improve and structural health resources are increasingly being stretched, health providers require people living with and beyond breast cancer (LwBBC) to self-manage aspects of their care. OBJECTIVE: This study aimed to explore how women use and experience social media to self-manage their psychosocial needs and support self-management across the breast cancer continuum. METHODS: The experiences of 21 women (age range 27-64 years) were explored using an in-depth qualitative approach. The women varied in the duration of their experiences of LwBBC, which facilitated insights into how they evolve and change their self-management strategies over time. Semistructured interviews were analyzed inductively using a thematic analysis, a polytextual analysis, and voice-centered relational methods. RESULTS: The use of multiple social media platforms, such as YouTube, Facebook, WhatsApp, and Twitter, enabled women to self-manage aspects of their care by satisfying needs for timely, relevant, and appropriate support, by navigating identities disrupted by diagnosis and treatment and by allowing them to (re)gain a sense of control. Women described extending their everyday use of multiple platforms to self-manage their care. However, women experienced social media as both empowering and dislocating, as their engagement was impacted by their everyday experiences of LwBBC. CONCLUSIONS: Health care professionals (HCPs) need to be more aware, and open to the possibilities, of women using multiple social media resources as self-management tools. It is important for HCPs to initiate value-free discussions and create the space necessary for women to share how social media resources support a tailored and timely self-managed approach to their unique psychosocial needs.

Postpositivism in Health Professions Education Scholarship.

An understanding of the diversity of perspectives within the research paradigms of health professions education (HPE) is essential for rigorous research design and more purposeful engagement with the contributions of others. In this article, the authors explicitly discuss the underlying assumptions, notions of good scholarship, and shortcomings of the postpositivism research paradigm. While postpositivism is likely one of the more familiar paradigms within HPE research, it is rarely formally or explicitly described. Drawing on key literature and contemporary examples, the authors describe the ontology, epistemology, methodologies, axiology, signs of rigor, and common critiques of postpositivism. A case study provides the focus for a practical illustration of how a postpositivist approach to education research could be applied. Suggestions for further reading are provided for those who are keen to delve deeper into the history and key tenants of the postpositivist stance.

Fully online OSCEs: A large cohort case study.

This article was migrated. The article was marked as recommended. Objective Structured Clinical Examinations (OSCEs) are extensively used for clinical assessment in the health professions. However, current social distancing requirements (including on-campus bans) at many universities have made the co-location of participants for large cohort OSCEs impossible. While there is a developing literature on remote OSCEs, particularly in response to the COVID-19 pandemic, this is dominated by approaches dealing with small participant numbers. This paper describes our recent large scale (n = 361 candidates) implementation of a remotely delivered 2 station OSCE. The planning for this OSCE was extensive and involved comprehensive candidate, examiner and simulated patient orientation and training. Our processes were explicitly designed to develop platform familiarity for all participants and included building on remote tutorial experiences and device testing. Our remote OSCE design and logistics made use of using existing enterprise solutions including videoconferencing, survey and collaboration platforms and allowed extra time between candidates in case of technical issues. We describe our process in detail including examiner, simulated patient, and candidate perspectives to provide precise detail, hopefully assisting other institutions to understand and adopt our approach. Although logistically complex, we have demonstrated that it is possible to deliver a remote OSCE assessment involving a large student cohort with a limited number of stations using commonly available enterprise solutions. We recognise it would be ideal to sample more broadly across stations and examiners, yet given the constraints of our current COVID-19 impacted environment, we believe this to be an appropriate compromise for a non-graduating cohort at this time.

Challenging feedback myths: Values, learner involvement and promoting effects beyond the immediate task.

CONTEXT: Research suggests that feedback in the health professions is less useful than we would like. In this paper, we argue that feedback has become reliant on myths that perpetuate unproductive rituals. Feedback often resembles a discrete episode of an educator "telling," rather than an active and iterative involvement of the learner in a future-facing process. With this orientation towards past events, it is not surprising that learners become defensive or disengaged when they are reminded of their deficits. METHODS: We tackle three myths of feedback: (a) feedback needs praise-criticism balancing rules; (b) feedback is a skill residing within the teacher; and (c) feedback is an input only. For each myth we provide a reframing with supporting examples from the literature. CONCLUSIONS: Equipping learners to engage in feedback processes may reduce the emotional burden on both parties, rendering techniques such as the feedback sandwich redundant. We also highlight the benefits for learners and teachers of conceptualising feedback as a relational activity, and of tracing the effects of information exchanges. These effects may be immediate or latent, and may manifest in different forms such as changes in learner evaluative judgement or professional identity.

Diagnostic Approaches to Neuroendocrine Neoplasms of Unknown Primary Site.

Neuroendocrine neoplasms (NENs) arise from cells of neuronal and endocrine differentiation. While they are a rare entity, an increasing proportion of patients with NEN present with metastatic disease and no evident primary site using routine imaging or histopathology. NENs of unknown primary site have a poorer prognosis, often due to the challenge of selecting appropriate evidence-based management. We review the available literature and guidelines for the management of NENs of unknown primary site including clinical features, biochemical tests, histopathology, imaging, surgical exploration and localised and systemic treatments. We also discuss novel molecular techniques currently under investigation to aid primary site identification.

Dose-response relationship between physical activity and mortality in people with non-communicable diseases: a study protocol for the systematic review and meta-analysis of cohort studies.

INTRODUCTION: This study protocol outlines our planned systematic review and dose-response meta-analysis of postdiagnosis physical activity and mortality in people with non-communicable diseases (NCDs). METHODS AND ANALYSIS: This study is based on the Preferred Reporting Items for Systematic Reviews and Meta-analysis for Protocols. A systematic literature search will be conducted in various databases-namely, PubMed, Scopus and Web of Science-by two researchers in order to identify prospective observational studies that investigate postdiagnosis physical activity or activity-related energy expenditure and mortality in individuals with NCDs. The target population is adults (≥18 years of age) with one of the following nine NCDs: low back pain, type 2 diabetes mellitus, osteoarthritis, depressive disorder, chronic obstructive pulmonary disease, breast cancer, lung cancer, stroke or ischaemic heart disease. We will focus on all-cause mortality as the primary outcome and investigate indication-specific mortality as the secondary outcome. For each study identified as a result of the literature search, we will conduct graphical dose-response analyses of mortality as a function of activity-related energy consumption. If more than two studies are available for one disease, we will perform linear and non-linear dose-response meta-analyses for said disease using random-effects models. We will investigate the heterogeneity of the studies and publication bias. To assess the risk of bias and the quality of the included studies, we will use the Risk Of Bias In Non-randomised Studies - of Interventions tool, which is a Cochrane tool. ETHICS AND DISSEMINATION: This systematic review will be conducted in compliance with ethical precepts. As the systematic review is based on published studies, approval from an ethics committee is not required. The systematic review and meta-analysis will be published in a peer-reviewed journal. PROSPERO REGISTRATION NUMBER: CRD42018103357.