Exploring the use of Rasch modelling in "common content" items for multi-site and multi-year assessment.

Rasch modelling is a powerful tool for evaluating item performance, measuring drift in difficulty over time, and comparing students who sat assessments at different times or at different sites. Here, we use data from thirty UK medical schools to describe the benefits of Rasch modelling in quality assurance and the barriers to using it. Sixty "common content" multiple choice items were offered to all UK medical schools in 2016-17, and a further sixty in 2017-18, with five available in both years. Thirty medical schools participated, for sixty total datasets across two sessions, and 14,342 individual sittings. Schools selected items to embed in written assessment near the end of their programmes. We applied Rasch modelling to evaluate unidimensionality, model fit statistics and item quality, horizontal equating to compare performance across schools, and vertical equating to compare item performance across time. Of the sixty sittings, three provided non-unidimensional data, and eight violated goodness of fit measures. Item-level statistics identified potential improvements in item construction and provided quality assurance. Horizontal equating demonstrated large differences in scores across schools, while vertical equating showed item characteristics were stable across sessions. Rasch modelling provides significant advantages in model- and item- level reporting compared to classical approaches. However, the complexity of the analysis and the smaller number of educators familiar with Rasch must be addressed locally for a programme to benefit. Furthermore, due to the comparative novelty of Rasch modelling, there is greater ambiguity on how to proceed when a Rasch model identifies misfitting or problematic data.

Medical students' perception of e-learning approach (MeSPeLA) - a mixed method research.

There is a discrepancy between the research exploring e-learning at medical universities in Central/Eastern and Western European countries. The aim of the MeSPeLA study was to explore the understanding, experience and expectations of Polish medical students in terms of e-learning. Questionnaire containing open-ended and closed questions supplemented by focus group discussion was validated and performed among 204 medical students in Poland before COVID-19 pandemia. Several domains: understanding of e-learning definitions; students' experience, preferences, expectations and perceptions of e-learning usefulness, advantages and disadvantages were addressed. The qualitative data were analyzed using an inductive approach. 46.0% of students chose a communication-oriented definition as the most appropriate. 7.4% claimed not to have any experience with e-learning. 76.8% of respondents indicated they had contact with e-learning. The main reported e-learning advantages were time saving and easier time management. The most common drawback was limited social interactions. The acceptance of the usage of e-learning was high. Medical undergraduates in Poland regardless of the year of studies, gender or choice of future specialization showed positive attitudes towards e-learning. Students with advanced IT skills showed a better understanding of the e-learning definition and perceived e-learning to be a more useful approach. The expectations and perceptions about e-learning in Polish medical schools seems similar to some extent to that in Western European and the United States so we can be more confident about applying some lessons from these research to Poland or other post-communist countries. Such application has been accelerated due to COVID-19 pandemia.

The Impact of ERASMUS Exchanges on the Professional and Personal Development of Medical Students.

The ERASMUS program is one of the most popular student exchange projects, particularly among the students of Central and Eastern European countries. However, limited research is available with regard to its influence on the professional and personal development of its participants. The study aimed at investigating the experiences and impact of the ERASMUS program on different domains of the personal and professional life of medical students. A questionnaire containing closed and open-ended questions was distributed among 269 former participants of the ERASMUS program from the Poznan University of Medical Sciences to collect qualitative and quantitative data regarding the topic. The response rate was 41%. Mastering professional foreign language skills was the most frequently reported benefit of ERASMUS (94%), followed by a change of approach towards learning by exposure to innovative teaching techniques, character, professionalism and cultural competency development, impact on the migration decisions of the students, as well as the opportunity to compare healthcare and educational systems across countries. Additionally, 57% of respondents stated that ERASMUS impacted their career plans, and few indicated that it had affected their specialty choice. Approximately 28% of respondents have worked abroad in healthcare or research since graduating. Participation in the ERASMUS program proved to be a unique opportunity for professional and personal development.

Variation in performance on common content items at UK medical schools.

BACKGROUND: Due to differing assessment systems across UK medical schools, making meaningful cross-school comparisons on undergraduate students' performance in knowledge tests is difficult. Ahead of the introduction of a national licensing assessment in the UK, we evaluate schools' performances on a shared pool of "common content" knowledge test items to compare candidates at different schools and evaluate whether they would pass under different standard setting regimes. Such information can then help develop a cross-school consensus on standard setting shared content. METHODS: We undertook a cross-sectional study in the academic sessions 2016-17 and 2017-18. Sixty "best of five" multiple choice 'common content' items were delivered each year, with five used in both years. In 2016-17 30 (of 31 eligible) medical schools undertook a mean of 52.6 items with 7,177 participants. In 2017-18 the same 30 medical schools undertook a mean of 52.8 items with 7,165 participants, creating a full sample of 14,342 medical students sitting common content prior to graduation. Using mean scores, we compared performance across items and carried out a "like-for-like" comparison of schools who used the same set of items then modelled the impact of different passing standards on these schools. RESULTS: Schools varied substantially on candidate total score. Schools differed in their performance with large (Cohen's d around 1) effects. A passing standard that would see 5 % of candidates at high scoring schools fail left low-scoring schools with fail rates of up to 40 %, whereas a passing standard that would see 5 % of candidates at low scoring schools fail would see virtually no candidates from high scoring schools fail. CONCLUSIONS: Candidates at different schools exhibited significant differences in scores in two separate sittings. Performance varied by enough that standards that produce realistic fail rates in one medical school may produce substantially different pass rates in other medical schools - despite identical content and the candidates being governed by the same regulator. Regardless of which hypothetical standards are "correct" as judged by experts, large institutional differences in pass rates must be explored and understood by medical educators before shared standards are applied. The study results can assist cross-school groups in developing a consensus on standard setting future licensing assessment.

Predicting failure before it happens: A 5-year, 1042 participant prospective study.

PURPOSE OF THE ARTICLE: Students who fail assessments are at risk of negative consequences, including emotional distress and cessation of studies. Identifying students at risk of failure before they experience difficulties may considerably improve their outcomes. METHODS: Using a prospective design, we collected simple measures of engagement (formative assessment scores, compliance with routine administrative tasks, and attendance) over the first 6 weeks of Year 1. These measures were combined to form an engagement score which was used to predict a summative examination sat 14 weeks after the start of medical school. The project was repeated for five cohorts, giving a total sample size of 1042. RESULTS: Simple linear regression showed engagement predicted performance (R2adj = 0.03, F(1,1040) = 90.09, p < 0.001) with a small effect size. More than half of failing students had an engagement score in the lowest two deciles. CONCLUSIONS: At-risk medical students can be identified with some accuracy immediately after starting medical school using routinely collected, easily analysed data, allowing for tailored interventions to support students. The toolkit provided here can reproduce the predictive model in any equivalent educational context. Medical educationalists must evaluate how the advantages of early detection are balanced against the potential invasiveness of using student data.

Student perspective on outcomes and process - Recommendations for implementing competency-based medical education.

PURPOSE: Competency-based medical education (CBME) seeks to prepare undergraduate and postgraduate trainees for clinical practice. Its major emphasis is on outcomes, but questions about how best to reach these remain. One key issue is the need to integrate what matters most to students when setting educational goals: this is crucial if we are to design curricula that trainees understand and engage with, and that promote successful achievement of competencies. METHOD: We interviewed medical students in years 4 and 6 of a 6-year medical degree and used thematic analysis to understand their main educational priorities and how these fit with the aims of CBME. RESULTS: Two major themes emerged: features of content and process. For content, students wanted clear guidance on what constitutes competence, finding broad outcome statements abstract and difficult to understand as novices. They also attach critical importance to features of process such as being welcomed, included in clinical teams and being known personally - these promote motivation, understanding, and professional development. CONCLUSIONS: We present recommendations for those designing CBME curricula to emphasize the student perspective: what kind of guidance on outcomes is required, and features of process that must not be neglected if competence is to be achieved.

Medical graduate views on statistical learning needs for clinical practice: a comprehensive survey.

BACKGROUND: This paper seeks to contribute to a reputable evidence base for required competencies across different topics in statistics and probability (statistical topics) in preparing medical graduates for clinical practice. This is in order to inform the prioritization of statistical topics within future undergraduate medical curricula, while exploring the need for preparing tomorrow's doctors to be producers, and not merely consumers, of statistics. METHODS: We conducted a comprehensive online survey from July 2013 to August 2014 for a target group of 462 medical graduates with current or prior experience of teaching undergraduate medical students of the University of Edinburgh of whom 278 (60.2%) responded. Statistical topics were ranked by proportion of respondents who identified the practice of statistics, performing statistical procedures or calculations using appropriate data, as a required competency for medical schools to provide in preparing undergraduate medical students for clinical practice. Mixed effects analyses were used to identify potential predictors for selection of the above competency and to compare the likelihood of this selection for a range of statistical topics versus critical appraisal. RESULTS: Evidence was gleaned from medical graduates' experiences of clinical practice for the need for, not only a theoretical understanding of statistics and probability but also, the ability to practice statistics. Nature of employment and statistical topic were highly significant predictors of choice of the practice of statistics as a required competency ((F = 3.777, p < 0.0005) and (F = 45.834, p < 0.0005), respectively). The most popular topic for this competency was graphical presentation of data (84.3% of respondents) in contrast to cross-over trials for the competency understanding the theory only (70.5% of respondents). Several topics were found to be more popular than critical appraisal for competency in the practice of statistics. CONCLUSIONS: The model of medical graduates as mere consumers of statistics is oversimplified. Contrary to what has been suggested elsewhere, statistical learning opportunities in undergraduate medicine should not be restricted to development of critical appraisal skills. Indeed, our findings support development of learning opportunities for undergraduate medical students as producers of statistics across a wide range of statistical topics.

Analysing aggregate clinical data to support evaluation of the Tackling Indigenous Smoking program, 2014-16.

Objective The aim of this study was to assess the change in recording of client population smoking attributes (smoking status recorded and smoking status) in Tackling Indigenous Smoking (TIS)-funded services compared with non-funded services for Aboriginal and Torres Strait Islander people, 2014-16. Methods The study included a cohort of 152 Aboriginal-community controlled services with aggregate client smoking data from 2014 to 2016. Negative binomial regression was used to assess change in smoking status recorded and smoking status between TIS and non-TIS funded organisations. The models controlled for size of client population, jurisdiction and remoteness. Results From 2014 to 2016, the overall reporting rate (change in recording of smoking status) of client smoking status was 1.58-fold higher (95% confidence interval (CI) 1.30-1.91; P<0.001) in TIS-funded than non-TIS-funded services after controlling for year, remoteness and their interaction. The highest change in reporting of client smoking status was for TIS-funded services in remote areas (reporting ratio 6.55; 95% CI 5.18-8.27; P<0.001). In 2016, TIS-funded services reported higher overall levels of recording client smoking status (current, ex- and non-smokers) than non-TIS funded services (RR 1.11; 95% CI 1.00-1.28; P<0.001). There was no significant change in the reporting of smokers, ex-smokers or non-smokers over the three reporting periods. Conclusion The analysis shows higher reporting of the proportion of the service client population for services funded under the TIS program compared with non-TIS-funded services. Existing evidence suggests that following-up smokers with targeted clinical interventions once they have had smoking status recorded could reduce smoking rates in the long term. The public health contribution of this study has defined one method for assessing smoking attributes when using aggregate health service data. This method could be applied to future tobacco control programs in health services. What is known about the topic? Aboriginal and Torres Strait Islander smoking prevalence is high. The Aboriginal and Torres Strait Islander primary healthcare providers' national key performance indicators (nKPIs) are one data source that can track changes over time in smoking in clients of these services. What does this paper add? This paper presents the first analytical study and evaluation of the nKPI dataset items on smoking. What are the implications for practitioners? There is value in analysing routinely collected data in program evaluations. The method used in this paper demonstrates one approach that could be used to assess smoking indicators and their changes over time in TIS program evaluation.

Improving global health: measuring the success of capacity building outreach programs: a view from the International Society of Nephrology.

Capacity building is key to the advance of health care in the developing world, but capacity building can take several forms, and it is not yet clear which forms and funding models are most effective. The International Society of Nephrology (ISN) has developed and sustained a portfolio of global outreach programs that make a unique contribution at low cost to the appropriate training and continuing support of nephrologists throughout the developing world. We describe the programs and present encouraging findings of their outcomes. Finally, we reflect on how the ISN interventions and evaluations measure up against recommended approaches in the literature and consider lessons for ISN and other organizations involved in planning, evaluating, and benchmarking similar programs.

Which non-technical skills do junior doctors require to prescribe safely? A systematic review.

AIMS: Prescribing errors are a major source of avoidable morbidity and mortality. Junior doctors write most in-hospital prescriptions and are the least experienced members of the healthcare team. This puts them at high risk of error and makes them attractive targets for interventions to improve prescription safety. Error analysis has shown a background of complex environments with multiple contributory conditions. Similar conditions in other high risk industries, such as aviation, have led to an increased understanding of so-called human factors and the use of non-technical skills (NTS) training to try to reduce error. To date no research has examined the NTS required for safe prescribing. The aim of this review was to develop a prototype NTS taxonomy for safe prescribing, by junior doctors, in hospital settings. METHODS: A systematic search identified 14 studies analyzing prescribing behaviours and errors by junior doctors. Framework analysis was used to extract data from the studies and identify behaviours related to categories of NTS that might be relevant to safe and effective prescribing performance by junior doctors. Categories were derived from existing literature and inductively from the data. RESULTS: A prototype taxonomy of relevant categories (situational awareness, decision making, communication and team working, and task management) and elements was constructed. CONCLUSIONS: This prototype will form the basis of future work to create a tool that can be used for training and assessment of medical students and junior doctors to reduce prescribing error in the future.

Initial experiences of a student assistantship.

BACKGROUND: Evidence suggests that medical graduates are underprepared to work as junior doctors. To ease transition in the UK, the General Medical Council (GMC) recommended the introduction of a student 'assistantship'. This is a period of training where final-year students take on duties of a foundation doctor under supervision. This study explored the experiences of the first cohort of students and junior doctors participating in the assistantship in one UK medical school in 2012. METHODS: All 248 students and their supervisors were asked to complete an online feedback questionnaire. All students who went on to work locally were also invited to participate in focus groups as recent graduates. Evidence suggests that medical graduates are underprepared to work as junior doctors RESULTS: Questionnaire response rates were 49 per cent for students and 43 per cent for supervisors. Fifteen new graduates participated in focus groups. Aspects of the assistantship considered important to participants frequently mapped to areas specified by the GMC and the locally identified learning outcomes. Additional themes identified included the importance of having meaningful responsibility for patient care, a placement in a general medical or surgical ward and receiving effective feedback. DISCUSSION: The assistantship seems to have been highly valued by students, but could be improved by ensuring that all students are given relevant placements and clinical responsibility.

Exploring patterns of error in acute care using framework analysis.

BACKGROUND: Junior doctors are often the first responders to deteriorating patients in hospital. In the high-stakes and time-pressured context of acute care, the propensity for error is high. This study aimed to identify the main subject areas in which junior doctors' acute care errors occur, and cross-reference the errors with Reason's Generic Error Modelling System (GEMS). GEMS categorises errors according to the underlying cognitive processes, and thus provides insight into the causative factors. The overall aim of this study was to identify patterns in junior doctors' acute care errors in order to enhance understanding and guide the development of educational strategies. METHODS: This observational study utilised simulated acute care scenarios involving junior doctors dealing with a range of emergencies. Scenarios and the subsequent debriefs were video-recorded. Framework analysis was used to categorise the errors according to eight inductively-developed key subject areas. Subsequently, a multi-dimensional analysis was performed which cross-referenced the key subject areas with an earlier categorisation of the same errors using GEMS. The numbers of errors in each category were used to identify patterns of error. RESULTS: Eight key subject areas were identified; hospital systems, prioritisation, treatment, ethical principles, procedural skills, communication, situation awareness and infection control. There was a predominance of rule-based mistakes in relation to the key subject areas of hospital systems, prioritisation, treatment and ethical principles. In contrast, procedural skills, communication and situation awareness were more closely associated with skill-based slips and lapses. Knowledge-based mistakes were less frequent but occurred in relation to hospital systems and procedural skills. CONCLUSIONS: In order to improve the management of acutely unwell patients by junior doctors, medical educators must understand the causes of common errors. Adequate knowledge alone does not ensure prompt and appropriate management and referral. The teaching of acute care skills may be enhanced by encouraging medical educators to consider the range of potential error types, and their relationships to particular tasks and subjects. Rule-based mistakes may be amenable to simulation-based training, whereas skill-based slips and lapses may be reduced using strategies designed to raise awareness of the interplay between emotion, cognition and behaviour.

Examiners are most lenient at the start of a two-day OSCE.

BACKGROUND: OSCEs can be both reliable and valid but are subject to sources of error. Examiners become more hawkish as their experience grows, and recent research suggests that in clinical contexts, examiners are influenced by the ability of recently observed candidates. In OSCEs, where examiners test many candidates over a short space of time, this may introduce bias that does not reflect a candidate's true ability. AIMS: Test whether examiners marked more or less stringently as time elapsed in a summative OSCE, and evaluate the practical impact of this bias. METHODS: We measured changes in examiner stringency in a 13 station OSCE sat by 278 third year MBChB students over the course of two days. RESULTS: Examiners were most lenient at the start of the OSCE in the clinical section (ß = -0.14, p = 0.018) but not in the online section where student answers were machine marked (ß = -0.003, p = 0.965). CONCLUSIONS: The change in marks was likely caused by increased examiner stringency over time derived from a combination of growing experience and exposure to an increasing number of successful candidates. The need for better training and for reviewing standards during the OSCE is discussed.

Midwives benefit from good postnatal care, too.

Appropriate, timely and responsive postnatal care can help women and families negotiate the major life transition that childbirth brings. However, women's experiences of postnatal care are often negative and our increasingly biomedical approach to birth means that greater emphasis is placed on antenatal and intrapartum care at the expense of postnatal care. Good postnatal care is essential not only for women, but for midwives too, and our failure to acknowledge the significance of birth, and our contribution to that event can diminish us as people and midwives.

The effects of contributing to patient care on medical students' workplace learning.

CONTEXT: Previous research has suggested that as medical students become more senior, they should increasingly take on the roles they will enact as newly qualified doctors by contributing to patient care. However, student contribution to patient care carries inherent risks to patient safety. This study aimed to provide students with a new opportunity to contribute to patient care and to use this as a platform from which to explore the influence of contributing to patient care on medical student learning. METHODS: This study took place in the context of final-year medical student prescribing education at the University of Edinburgh, Edinburgh, UK. Students on attachment at a district general hospital were afforded a unique opportunity to learn prescribing by completing in-patient drug charts in a process termed 'pre-prescribing'. All students were invited to participate in focus groups conducted by the principal researcher. Focus group discussions were audio-recorded, transcribed verbatim and thematically analysed. RESULTS: Six focus groups, each lasting 20-50 minutes, were conducted with four to seven participants (33 students in total). The emerging themes took the form of developmental outcomes and learning processes. Developmental outcomes included ability to perform the task, modification of attitudes towards the task, formation of a professional identity, and development of relationships within the team. The central feature of the experience which influenced all developmental outcomes, was making mistakes. The themes interact in complex ways and all contribute towards development as a professional. CONCLUSIONS: This study has demonstrated that contributing to patient care enhances students' development as professionals. Some of these developmental outcomes, such as improvements in knowledge and skills, may be achievable to some extent within the classroom. Other changes, such as developing relationships, forming a sense of professional identity and modifying attitudes, might arguably be achievable only within the context of contributing to patient care.

New South Wales and Australian Capital Territory Researcher Development Program 2005-07: modest investment, considerable outcomes.

This evaluation of the Researcher Development Program (RDP) in NSW and ACT aimed to determine whether the RDP was effective in assisting novice researchers to undertake primary health care research. In mid-2008, 47 participants of the NSW and ACT RDP during 2005-07 were invited to participate in a postal survey. The survey included questions regarding previous research training and experience, outcomes during and after participation in the program, and organisational aspects of the program. Follow-up interviews were conducted with selected participants. Interview questions covered time in the program, supervision, organisational support and placement outcomes. Thirty-seven participants responded to the survey and 23 (62%) participants took part in the semi-structured interviews. Seventy-eight per cent of survey respondents felt that the RDP helped them move from novice to a more experienced researcher with effective supervision identified by participants as a key element in determining the success of the program. Many felt that time allocation was inadequate and 20% thought their capacity to maintain their workload was adversely affected by participating. Outcomes were considerable given the modest nature of the program. Notable outcomes were that most participants published their research and presented their research at a conference. Furthermore, one-fifth of survey respondents had enrolled in higher degrees. Several interviewees reported that their research led to changes in practice. Most respondents found the RDP valuable and considered that undertaking the program increased their research knowledge.

The distracted intravenous access (DIVA) test.

BACKGROUND: The General Medical Council states that all medical graduates must be able to carry out practical procedures, including peripheral venous cannulation, safely and effectively. Teaching and assessments within primary medical training tend to focus on safety rather than 'effectiveness' or technical competence. This study aimed to develop and evaluate a test of automaticity of peripheral venous cannulation skill, appropriate to the level of a medical student. METHODS: Two researchers developed the distracted intravenous access (DIVA) test. Three components are assessed simultaneously: ability to cannulate a plastic arm manikin, performance in an arithmetic test and speed of completion. Volunteers were recruited from three groups: novices (fourth-year medical students), intermediates (foundation year-1 doctors) and experts (anaesthetic and emergency medicine consultants and trainees). Immediately following the test, volunteers completed a questionnaire. RESULTS: Mean scores differed between the three groups: novices (47.7%), intermediates (73.4%), experts (84.4%). Differences were statistically significant (p < 0.0005). The majority of participants found the test to be enjoyable (78%) and useful (76%). More students agreed that the test was fair (78%) than doctors (38%), but this difference was not statistically significant. DISCUSSION: This study presents a feasible method of testing medical students' automaticity when performing peripheral venous cannulation. It has provided evidence of the construct validity and acceptability of the test. The authors suggest that clinical tutors consider offering the test as a formative assessment to final-year medical students in order to familiarise them with the level of proficiency required to perform peripheral venous cannulation when distracted.

Designing a purposeful student assistantship.

BACKGROUND: The student assistantship, during which medical students act as assistants to junior doctors, is currently being integrated into all UK medical school curricula. However, there is a dearth of evidence to guide design and implementation, particularly in relation to the priorities for student learning. This study aimed to explore the specific learning opportunities within an assistantship that are perceived as important by foundation year-1 doctors (FY1s) and their educational supervisors. METHODS: A questionnaire study of University of Edinburgh graduates, and their educational supervisors, was undertaken in January 2011. The survey investigated the perceived importance of 16 predefined learning opportunities, and asked respondents to indicate the learning opportunity that they regarded as most important. Free-text responses relating to other beneficial components were sought and thematically analysed. RESULTS: All 16 learning opportunities were regarded as useful by over 80 per cent of FY1s, and over 50 per cent of educational supervisors. The opportunities considered most important by both FY1s and educational supervisors were prescribing drugs and fluids, providing emergency care and prioritisation of tasks. Free-text responses suggested that experience of out-of-hours working, administrative tasks and the theatre environment were also important. DISCUSSION: By asking FY1s and educational supervisors to consider the most useful learning opportunity, it has been possible to use the respective rankings to guide the design and implementation of the assistantship. Our future challenge is to develop ways of allowing students to gain experience in the areas considered most important, whilst ensuring the safety of patients.

Exploring error in team-based acute care scenarios: an observational study from the United kingdom.

PURPOSE: To investigate the errors made by junior doctors (first year after primary medical qualification) in simulated acute care settings, using (and, for some purposes, amplifying) a previously published generic error-modeling system (GEMS). Possible error types were skill-based slips and lapses, rule-based mistakes, knowledge-based mistakes, and violations. METHOD: In August 2010, 38 junior doctors participated in high-fidelity simulated acute care scenarios in NHS Lothian, Scotland. Each video-recorded scenario was immediately followed by an audio-recorded debrief that encouraged articulation of underlying cognitive processes. Two researchers used evidence from the scenario, debrief, and field notes to determine which errors were attributable to a single underlying cause. In such cases, the errors were coded by template analysis into the GEMS framework. Errors for which a single cause could be identified but which did not fit the framework were coded inductively. RESULTS: A total of 243 errors were identified, with sufficient evidence available to identify a single cause in 190. Skill-based slips and lapses, rule-based mistakes, and knowledge-based mistakes were all clearly identified within the data. Two error types not originally included in the GEMS framework were identified: compound errors and submission errors. CONCLUSIONS: Amplification of GEMS provides a valid framework for categorization of the errors made by junior doctors in simulated acute care contexts. In addition, the amplified framework may be transferable to other, team-based contexts. An improved understanding of the knowledge and skills that are most vulnerable to each specific type of error will allow tailored educational strategies to be developed.