Inter-school variations in the standard of examiners' graduation-level OSCE judgements.

INTRODUCTION: Ensuring equivalence in high-stakes performance exams is important for patient safety and candidate fairness. We compared inter-school examiner differences within a shared OSCE and resulting impact on students' pass/fail categorisation. METHODS: The same 6 station formative OSCE ran asynchronously in 4 medical schools, with 2 parallel circuits/school. We compared examiners' judgements using Video-based Examiner Score Comparison and Adjustment (VESCA): examiners scored station-specific comparator videos in addition to 'live' student performances, enabling 1/controlled score comparisons by a/examiner-cohorts and b/schools and 2/data linkage to adjust for the influence of examiner-cohorts. We calculated score impact and change in pass/fail categorisation by school. RESULTS: On controlled video-based comparisons, inter-school variations in examiners' scoring (16.3%) were nearly double within-school variations (8.8%). Students' scores received a median adjustment of 5.26% (IQR 2.87-7.17%). The impact of adjusting for examiner differences on students' pass/fail categorisation varied by school, with adjustment reducing failure rate from 39.13% to 8.70% (school 2) whilst increasing failure from 0.00% to 21.74% (school 4). DISCUSSION: Whilst the formative context may partly account for differences, these findings query whether variations may exist between medical schools in examiners' judgements. This may benefit from systematic appraisal to safeguard equivalence. VESCA provided a viable method for comparisons.

A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness.

INTRODUCTION: Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances. METHODS: We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity. RESULTS: Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity. CONCLUSIONS: The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.

How do medical schools influence their students' career choices? A realist evaluation.

INTRODUCTION: The career choices of medical graduates vary widely between medical schools in the UK and elsewhere and are generally not well matched with societal needs. Research has found that experiences in medical school including formal, informal and hidden curricula are important influences. We conducted a realist evaluation of how and why these various social conditions in medical school influence career thinking. METHODS: We interviewed junior doctors at the point of applying for speciality training. We selected purposively for a range of career choices. Participants were asked to describe points during their medical training when they had considered career options and how their thinking had been influenced by their context. Interview transcripts were coded for context-mechanism-outcome (CMO) configurations to test initial theories of how career decisions are made. RESULTS: A total of 26 junior doctors from 12 UK medical schools participated. We found 14 recurring CMO configurations in the data which explained influences on career choice occurring during medical school. DISCUSSION: Our initial theories about career decision-making were refined as follows: It involves a process of testing for fit of potential careers. This process is asymmetric with multiple experiences needed before deciding a career fits ('easing in') but sometimes only a single negative experience needed for a choice to be ruled out. Developing a preference for a speciality aligns with Person-Environment-Fit decision theories. Ruling out a potential career can however be a less thought-through process than rationality-based decision theories would suggest. Testing for fit is facilitated by longer and more authentic undergraduate placements, allocation of and successful completion of tasks, being treated as part of the team and enthusiastic role models. Informal career guidance is more influential than formal. We suggest some implications for medical school programmes.

How well do UK assistantships equip medical students for graduate practice? Think EPAs.

The goal of better medical student preparation for clinical practice drives curricular initiatives worldwide. Learning theory underpins Entrustable Professional Activities (EPAs) as a means of safe transition to independent practice. Regulators mandate senior assistantships to improve practice readiness. It is important to know whether meaningful EPAs occur in assistantships, and with what impact. Final year students at one UK medical school kept learning logs and audio-diaries for six one-week periods during a year-long assistantship. Further data were also obtained through interviewing participants when students and after three months as junior doctors. This was combined with data from new doctors from 17 other UK schools. Realist methods explored what worked for whom and why. 32 medical students and 70 junior doctors participated. All assistantship students reported engaging with EPAs but gaps in the types of EPAs undertaken exist, with level of entrustment and frequency of access depending on the context. Engagement is enhanced by integration into the team and shared understanding of what constitutes legitimate activities. Improving the shared understanding between student and supervisor of what constitutes important assistantship activity may result in an increase in the amount and/or quality of EPAs achieved.

Enhancing authenticity, diagnosticity and equivalence (AD-Equiv) in multicentre OSCE exams in health professionals education: protocol for a complex intervention study.

INTRODUCTION: Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners' judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees' focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners' judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students' performances. METHODS AND ANALYSIS: The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students' scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data. ETHICS AND DISSEMINATION: The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.

'What do we do, doctor?' Transitions of identity and responsibility: a narrative analysis.

Transitioning from student to doctor is notoriously challenging. Newly qualified doctors feel required to make decisions before owning their new identity. It is essential to understand how responsibility relates to identity formation to improve transitions for doctors and patients. This multiphase ethnographic study explores realities of transition through anticipatory, lived and reflective stages. We utilised Labov's narrative framework (Labov in J Narrat Life Hist 7(1-4):395-415, 1997) to conduct in-depth analysis of complex relationships between changes in responsibility and development of professional identity. Our objective was to understand how these concepts interact. Newly qualified doctors acclimatise to their role requirements through participatory experience, perceived as a series of challenges, told as stories of adventure or quest. Rules of interaction within clinical teams were complex, context dependent and rarely explicit. Students, newly qualified and supervising doctors felt tensions around whether responsibility should be grasped or conferred. Perceived clinical necessity was a common determinant of responsibility rather than planned learning. Identity formation was chronologically mismatched to accepting responsibility. We provide a rich illumination of the complex relationship between responsibility and identity pre, during, and post-transition to qualified doctor: the two are inherently intertwined, each generating the other through successful actions in practice. This suggests successful transition requires a supported period of identity reconciliation during which responsibility may feel burdensome. During this, there is a fine line between too much and too little responsibility: seemingly innocuous assumptions can have a significant impact. More effort is needed to facilitate behaviours that delegate authority to the transitioning learner whilst maintaining true oversight.

The role of purple pens in learning to prescribe.

BACKGROUND: Medical doctors are required to prescribe drugs safely and effectively upon qualification, a skill that many feel poorly prepared to undertake. To better prepare doctors, a whole-task approach that develops knowledge and skills, but that also considers the effect of the complex clinical workplace on prescribing, is optimal. We describe an evaluation of an experiential learning programme that allows senior medical students to gain experience with inpatient prescribing during their hospital assistantship. METHODS: A standard operating procedure (SOP) for medical student transcribing was implemented by the teaching hospitals associated with a single medical school. This included medical student prescriptions being written in purple ink. The evaluation consisted of an audit of transcribing activity and a student survey. We evaluated the usage of the initiative, adherence to the SOP and the propensity for error. RESULTS: The survey was completed by 38 out of a possible 108 fifth-year students. All respondents agreed that the programme was helpful in aiding them to learn about prescribing. A total of 247 prescriptions for 50 patients were audited: 25.1% of the prescriptions written by students required some form of amendment by the supervising doctor or pharmacist; three (1.2%) prescription errors remained unidentified; and none presented a patient safety risk. CONCLUSIONS: The purple-pen scheme affords medical students the opportunity to prescribe in the workplace, where they face authentic challenges when safely contributing to patient care. The identification of prescribing errors, feedback and the learners' own reflections helped the learners to focus on areas for improvement in prescribing prior to qualification.

Taking on the doctor role in whole-task simulation.

BACKGROUND: Untimed simulated primary care consultations focusing on safe and effective clinical outcomes were first introduced into undergraduate medical education in Otago, New Zealand, in 2004. We extended this concept and included a secondary care version for final-year students. We offer students opportunities to manage entire consultations, which include making and implementing clinical decisions with simulated patients (SPs). Formative feedback is given by SPs on the achievement of pre-determined outcomes and by faculty members on clinical decision making, medical record keeping and case presentation. METHODS: We explored students' perceptions of the educational value of the sessions using post-session questionnaires (n = 194) and focus groups (n = 36 participants overall). Students are offered opportunities to manage entire consultations with simulated patients RESULTS: Students perceived that the sessions were useful, enjoyable and relevant to early postgraduate practice. They identified useful learning in time management, communication, decision making, prescribing and managing uncertainty. Students identified gaps in their knowledge and recognised that they had been offered opportunities to develop decision-making skills by having to take responsibility for whole consultations and all the decisions included within them. Most students reported positive impacts on learning, although a small minority reported negative impacts on their perceptions of their ability to cope as a junior doctor. DISCUSSION: These simulated consultation sessions appear to lead to the effective learning of a range of skills that students need in order to work as junior doctors. Facilitators leading such sessions must be alert to the possibility of educational harm arising from such simulations, and the need to address this during the debriefing.

Qualitative research using realist evaluation to explain preparedness for doctors' memorable 'firsts'.

CONTEXT: Doctors must be competent from their first day of practice if patients are to be safe. Medical students and new doctors are acutely aware of this, but describe being variably prepared. OBJECTIVES: This study aimed to identify causal chains of the contextual factors and mechanisms that lead to a trainee being capable (or not) of completing tasks for the first time. METHODS: We studied three stages of transition: anticipation; lived experience, and post hoc reflection. In the anticipation stage, medical students kept logbooks and audio diaries and were interviewed. Consenting participants were followed into their first jobs as doctors, during which they made audio diaries to capture the lived experiences of transition. Reflection was captured using interviews and focus groups with other postgraduate trainee doctors. All materials were transcribed and references to first experiences ('firsts') were analysed through the lens of realist evaluation. RESULTS: A total of 32 medical students participated. Eleven participants were followed through the transition to the role of doctor. In addition, 70 postgraduate trainee doctors from three local hospitals who were graduates of 17 UK medical schools participated in 10 focus groups. We identified three categories of firsts (outcomes): firsts that were anticipated and deliberately prepared for in medical school; firsts for which total prior preparedness is not possible as a result of the step change in responsibility between the student and doctor identities, and firsts that represented experiences of failure. Helpful interventions in preparation (context) were opportunities for rehearsal and being given responsibility as a student in the clinical team. Building self-efficacy for tasks was an important mechanism. During transition, the key contextual factor was the provision of appropriate support from colleagues. CONCLUSIONS: Transition is a step change in responsibility for which total preparedness is not achievable. This transition is experienced as a rite of passage when the newly qualified doctor first makes decisions alone. This study extends the existing literature by explaining the mechanisms involved in preparedness for firsts.

What's in a name? A discussion of what medical students and junior doctors call their senior colleagues, and why it matters.

This article was migrated. The article was marked as recommended. How should a medical student address their clinical tutor? Sociolinguistic ideas such as politeness theory tell us that the choice of formal or informal terms of address is determined by the positions of those communicating on two axes; relative status and degree of intimacy. This positioning is influenced by the interaction of personal characteristics of the individuals involved, but there are cross-cultural variations to these rules which are also changing as the world changes. The purpose of the communication will also influence terms of address. There is evidence that reducing social distance within teams improves team-working and that the perception of hierarchy prevents medical students asking for help. Such evidence forces us to take an honest look at how we train our junior colleagues to address us. Students may discover that the etiquette of the medical school classroom differs from that of the clinical placement and find themselves uncertain about how to address their colleagues appropriately. We suggest that it may be helpful in such a quandary to 'mind the gap' rather than ignoring it or trying to close it by imposing a blanket rule on it. We conclude by calling for sociological study with healthcare professionals and their students to discover whether formal or informal forms of address help or hinder aspects of learning and clinical teamwork.

Development and face validation of strategies for improving consultation skills.

While formative workplace based assessment can improve learners' skills, it often does not because the procedures used do not facilitate feedback which is sufficiently specific to scaffold improvement. Provision of pre-formulated strategies to address predicted learning needs has potential to improve the quality and automate the provision of written feedback. To systematically develop, validate and maximise the utility of a comprehensive list of strategies for improvement of consultation skills through a process involving both medical students and their clinical primary and secondary care tutors. Modified Delphi study with tutors, modified nominal group study with students with moderation of outputs by consensus round table discussion by the authors. 35 hospital and 21 GP tutors participated in the Delphi study and contributed 153 new or modified strategies. After review of these and the 205 original strategies, 265 strategies entered the nominal group study to which 46 year four and five students contributed, resulting in the final list of 249 validated strategies. We have developed a valid and comprehensive set of strategies which are considered useful by medical students. This list can be immediately applied by any school which uses the Calgary Cambridge Framework to inform the content of formative feedback on consultation skills. We consider that the list could also be mapped to alternative skills frameworks and so be utilised by schools which do not use the Calgary Cambridge Framework.

Real-time situation awareness assessment in critical illness management: adapting the situation present assessment method to clinical simulation.

BACKGROUND: Situation awareness (SA) is a human factor of critical importance to patient safety. Simulation training aims to examine and debrief human factors; however, SA cannot be directly observed. This has led to the development of SA measurement tools. The Situation Present Assessment Method (SPAM) measures SA in real-time without the need to pause the scenario. The SPAM process involves the delivery of queries to the participant who must answer them accurately and quickly. The latency between the query being asked and answer being received represents SA. METHOD: Two query delivery procedures are described in the literature: query delivery by telephone and in person. These procedures were piloted in simulation teaching with final-year medical students. The scenarios were videotaped and reviewed by the investigators to evaluate each procedure. Our evaluation of the existing SPAM procedures led us to adapt the method by developing a bespoke application, which delivers queries via a personal digital assistant (PDA), calculates the latency data and presents it to the instructor. RESULTS: Presented by telephone, queries tended to disrupt the 'flow' of the simulation. The 'in person' procedure was not disruptive; however, participants found it difficult to distinguish queries from other dialogue. The PDA represented a compromise between these two techniques: generating data without disrupting the scenario. CONCLUSIONS: The use of SPAM is feasible in clinical simulation. By using handheld technology, SA data are made available to the instructor for use in debrief; this expands the utility of SPAM to the field of medical education.