Assuring the quality of high-stakes undergraduate assessments of clinical competence.

In the UK, and in many Commonwealth countries, a university degree is accepted by registration bodies as an indication of competence to practice as a PRHO or intern. Concerns have been raised that the quality of university examinations may not always be sufficient for such high-stakes decision-making. Assessments of clinical competence are subject to many potential sources of error. The search for standardization, and high validity and reliability, demands the identification and reduction of measurement errors and biases due to poor test design or variation in test items, judges, patients or examination procedures. Generalizability and other research studies have identified where the likely sources of error might arise and have been taken into account in the development of published guidelines on international best practice, which institutions should strive to follow. The purpose of this paper is to describe the development of the integrated final-year assessment of clinical competence at the University of Sheffield. The aim was to introduce a range of strategies to ensure the examination met the best practice guidelines. These included blueprinting the assessment to achieve a high degree of content validity; lengthening the examination by adding a written component to the OSCE component to ensure an adequate level of reliability; providing training and feedback for examiners and simulated patients; paying attention to item development; and providing statistical information to assist the examination committee in standard setting and decision-making. This evidence-based approach should be readily achievable by all medical schools.

Discovering professionalism through guided reflection.

Doctors need to identify and understand the professional behaviours of both themselves and others. In order for students to think critically about these issues we encouraged them to use the tenets of the General Medical Council's Duties of a Doctor as a framework in which to reflect on the actions of healthcare professionals at work. Although the critical incident technique is a well-known process for encouraging reflection, little is known about its usefulness for assessment purposes in this setting. We aimed to discover the validity, feasibility and educational impact of the critical incident as an assessment method for first year students undertaking guided reflection in the context of their first exposure to multi-professional health and social care experiences. First year medical students submitted two critical incidents they had observed during multi-professional health and social care attachments and an evaluation of their experiences. Students engaged in the reflective cycle on the professional behaviours of others providing evidence of a varied range of situations. With adequate preparation, junior students are able to reflect on social and healthcare experiences using the Duties of a Doctor as a framework. Critical incidents are a valid and feasible method for assessing students' reflections on professionalism, with good educational impact.

The introduction of large class problem-based learning into an undergraduate medical curriculum: an evaluation.

When considering implementing integrated curriculum models, such as problem-based learning (PBL), concerns may be expressed about the need for increased staff resources required to deliver tutor-led small group PBL. Less staff intensive ways of supporting PBL need to be explored. We compared the outcomes of a PBL module conducted in a large class format within a lecture theatre with a module having the same defined learning outcomes delivered in small group PBL format, both supported by e-learning resources. The pre-existing 27 small groups within the whole class (n = 246) of first year students undertaking a cardiovascular basic science module at Sheffield undergraduate medical school, UK, were randomized to 22 groups undertaking the large class Integrated Learning Activity (ILA) and 5 groups to traditional small group facilitated PBL sessions. Outcome measures were: a pre-post knowledge based test, a student educational effectiveness questionnaire, and assessment of student group work and presentations. There seemed to be no significant differences in learning outcomes between the methods although it is recognized that students would prefer the small group teaching format. Within institutions where resources to support small group PBL are limited, the large group ILA format supported with e-learning techniques may be a useful alternative approach.

Developing assessments of surgical skills for the GMC Performance Procedures.

INTRODUCTION: The objectives were to: (i) establish how 'typical' consultant surgeons perform on 'generic' (non-specialist) surgical simulations before their use in the General Medical Council's Performance Procedures (PPs); (ii) measure any differences in performance between specialties; and (iii) compare the performance of group of surgeons in the PPs with the 'typical' group. VOLUNTEERS AND METHODS: Seventy-four consultant volunteers in gastrointestinal surgery (n=21), vascular surgery (n=11), urology (n=10), orthopaedics (n=15), cardiothoracic surgery (n=10) and plastic surgery (n=7), plus 9 surgeons undertaking phase 2 of the PPs undertook 7 simple simulations in the skills laboratory. The scores of the volunteers were analysed by simulation and specialty using ANOVA. The scores of the volunteers were then compared with the scores of the surgeons in the PPs. RESULTS: There were significant differences between simulations, but most volunteers achieved scores of 75-100%. There was a significant simulation by specialty interaction indicating that the scores of some specialties differed on some simulations. The scores of the group of surgeons in the PPs were significantly lower than the reference group for most simulations. CONCLUSIONS: Simple simulations can be used to assess the basic technical skills of consultant surgeons. The simulation by specialty interaction suggests that whilst some skills may be generic, others are not. The lower scores of the surgeons in the PPs suggest that these tests possess criterion validity, i.e. they may help to determine when poor performance is due to lack of technical competence.

Developing an outcome-focused core curriculum.

BACKGROUND: Many UK medical schools have modified their curricula to meet the requirements of the General Medical Council and other external agencies. In particular, efforts have been focused on increasing integration and reducing factual overload through the definition of a core curriculum. Various approaches to curriculum change have been adopted in an attempt to meet such demands. PURPOSE: This paper describes a curriculum development process, which commences with a clear vision, adopts an outcome-based approach and identifies clear statements of learning outcomes. The process led to the development of an outcome-focused core curriculum structured around clinical problems, which is available to all students and staff. CONCLUSION: A model of curriculum development has evolved which is relatively simple in concept, and appears to be easy to comprehend by students, teaching staff and visitors from other institutions. It provides a practical framework for managing the difficult problems of integration and factual overload. It should be of general interest and applicability to other schools with health professional programmes looking for a realistic and acceptable way of defining a core curriculum.

Student-selected components in the undergraduate medical curriculum: a multi-institutional consensus on assessable key tasks.

This paper describes further progress by the Northern Universities SSC Consortium in achieving consensus on the contribution of the Student Selected Components (SSCs) to undergraduate medical courses. Following the identification of common purposes and outcomes, the group has matched these to assessable key tasks which students may undertake in order to demonstrate competence on a range of attributes not normally fully assessed in the core curriculum. In some schools, SSCs may be the predominant route through which students gain exposure to research methods and reflective practice. In addition to opportunities to acquire knowledge and skills outside the core curriculum, SSCs can contribute towards enhancement of core skills. SSCs may also contribute to the overall longitudinal assessment of personal and professional development. The potential role of portfolios in maximizing student learning from SSCs is discussed, and future areas for development identified.

Competence in the musculoskeletal system: assessing the progression of knowledge through an undergraduate medical course.

BACKGROUND: Professional bodies have expressed concerns that medical students lack appropriate knowledge in musculoskeletal medicine despite its high prevalence of use within the community. Changes in curriculum and teaching strategies may be contributing factors to this. There is little evidence to evaluate the degree to which these concerns are justified. OBJECTIVES: To design and evaluate an assessment procedure that tests the progress of medical students in achieving a core level of knowledge in musculoskeletal medicine during the course. PARTICIPANTS AND SETTING: A stratified sample of 136 volunteer students from all 5 years of the medical course at Sheffield University. METHODS: The progress test concept was adapted to provide a cross-sectional view of student knowledge gain during each year of the course. A test was devised which aimed to provide an assessment of competence set at the standard required of the newly qualified doctor in understanding basic and clinical sciences relevant to musculoskeletal medicine. The test was blueprinted against internal and external guidelines. It comprised 40 multiple-choice and extended matching questions administered by computer. Six musculoskeletal practitioners set the standard using a modified Angoff procedure. RESULTS: Test reliability was 0.6 (Cronbach's alpha). Mean scores of students increased from 41% in Year 1 to 84% by the final year. Data suggest that, from a baseline score in Year 1, there is a disparate experience of learning in Year 2 that evens out in Year 3, with knowledge progression becoming more consistent thereafter. All final year participants scored above the standard predicted by the Angoff procedure. CONCLUSIONS: This short computer-based test was a feasible method of estimating student knowledge acquisition in musculoskeletal medicine across the undergraduate curriculum. Tested students appear to have acquired a satisfactory knowledge base by the end of the course. Knowledge gain seemed relatively independent of specialty-specific clinical training. Proposals from specialty bodies to include long periods of disciplinary teaching may be unnecessary.

Techniques for measuring clinical competence: objective structured clinical examinations.

The traditional clinical examination has been shown to have serious limitations in terms of its validity and reliability. The OSCE provides some answers to these limitations and has become very popular. Many variants on the original OSCE format now exist and much research has been done on various aspects of their use. Issues to be addressed relate to organization matters and to the quality of the assessment. This paper focuses particularly on the latter with respect to ways of ensuring content validity and achieving acceptable levels of reliability. A particular concern has been the demonstrable need for long examinations if high levels of reliability are to be achieved. Strategies for reducing the practical difficulties this raises are discussed. Standard setting methods for use with OSCEs are described.

Student-selected components in the undergraduate medical curriculum: a multi-institutional consensus on purpose.

Student-selected components (SSCs) are an established part of undergraduate medical curricula in the UK. Implementation has been against a background of differing educational approaches to curricular change, together with a lack of clarity about their purpose, relationship with the core curriculum and contribution to overall assessment. This has resulted in a diversity of programmes with perceived differing importance between medical schools. This paper documents the approach used by a consortium of medical schools with diverse curricula to develop consensus on the educational purpose and learning outcomes of SSCs. Agreement on common purposes and outcomes was achieved, and consensus documents are reported. These may be valuable for other medical schools implementing any form of student-selected project work. This work will now be the starting point for further work on producing recommendations for assessment of SSCs, which will be applicable across different medical schools.

Managing the learning environment in undergraduate medical education: the Sheffield approach.

This paper describes one possible model for e-learning in undergraduate medical education with an emphasis on supporting and managing curriculum development. The University of Sheffield School of Medicine is undergoing a major revision of is undergraduate medical curriculum, prompted by requirements of the professional regulatory body, the General Medical Council (GMC), and the Quality Assurance Agency for Higher Education (QAA). A computer-based system was developed to provide more efficient administration of the current course and more effective delivery of educational materials to students. The Sheffield Networked Learning Environment (NLE) has been developed in collaboration with other medical schools. A new 'revised' curriculum, due to start in 2003, will be intensively supported by an NLE which has been extensively tested and modified through pilot studies in the current curriculum.