Assessing the technical skills of surgical trainees.

BACKGROUND: The aims were to determine whether tests of technical skill on simple simulations can predict competence in the operating theatre and whether objective assessment in the operating theatre by direct observation and video recording is feasible and reliable. METHODS: Thirty-three general surgical trainees undertook five simple skill simulations (knotting, skin incision and suturing, tissue dissection, vessel ligation and small bowel anastomosis). The operative competence of each trainee was then assessed during two or three saphenofemoral disconnections (SFDs) by a single surgeon. Video recordings of the operations were also assessed by two surgeons. RESULTS: The inter-rater reliability between direct observation and blinded videotape assessment was high (alpha = 0.96 (95 per cent confidence interval 0.92 to 0.98)). Backward stepwise regression analysis revealed that the best predictors of operative competence were the number of SFDs performed previously plus the simulation scores for dissection and ligation, the key components of SFD (64 per cent of variance explained; P = 0.001). CONCLUSION: Deconstruction of operations into their component parts enables trainees to practise on simple simulations representing each component, and be assessed as competent, before undertaking the actual operation. Assessment of surgical competence by direct observation and video recording is feasible and reliable; such assessments could be used for both formative and summative assessment.

Comparing and contrasting undergraduate competence in musculoskeletal medicine with cardiovascular medicine and neurology.

OBJECTIVES: With an increasing prevalence of musculoskeletal conditions within the UK, specialty bodies are concerned that graduating medical students may lack appropriate knowledge in this system. We investigated the knowledge base of final year Sheffield medical students in the musculoskeletal system, compared with other major body systems. METHODS: A computer-based assessment was designed covering core topics that a pre-registration house officer should know about in musculoskeletal medicine, cardiology and neurology, using a predesigned testing format. The test was blueprinted against internal and external guidelines. It comprised 24 extended matching questions, each with three stems. A sample of 74 volunteer students from the final year (year 5) of the medical course at the University of Sheffield took part in the assessment. RESULTS: Overall scores of students on the test ranged from a baseline of 45% to a maximum of 85%. Test reliability was 0.75 (Cronbach's alpha). On stratifying the overall percentages into marks for individual systems, it was found that there were no significant differences between scores in musculoskeletal medicine, cardiovascular medicine or neurology. CONCLUSIONS: Despite the disparity of teaching between musculoskeletal medicine and other major organ systems within Sheffield's integrated medical curriculum, the knowledge base of medical students in the basic and clinical musculoskeletal sciences appears to be similar to that for cardiovascular medicine and neurology by the time of graduation. Nevertheless, several important issues must be addressed before these findings can be generalized.

Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists.

BACKGROUND: There is increasing emphasis on performance-based assessment of clinical competence. The High Fidelity Patient Simulator (HPS) may be useful for assessment of clinical practice in anaesthesia, but needs formal evaluation of validity, reliability, feasibility and effect on learning. We set out to assess the reliability of a global rating scale for scoring simulator performance in crisis management. METHODS: Using a global rating scale, three judges independently rated videotapes of anaesthetists in simulated crises in the operating theatre. Five anaesthetists then independently rated subsets of these videotapes. RESULTS: There was good agreement between raters for medical management, behavioural attributes and overall performance. Agreement was high for both the initial judges and the five additional raters. CONCLUSIONS: Using a global scale to assess simulator performance, we found good inter-rater reliability for scoring performance in a crisis. We estimate that two judges should provide a reliable assessment. High fidelity simulation should be studied further for assessing clinical performance.

Selecting performance assessment methods for experienced physicians.

BACKGROUND: While much is now known about how to assess the competence of medical practitioners in a controlled environment, less is known about how to measure the performance in practice of experienced doctors working in their own environments. The performance of doctors depends increasingly on how well they function in teams and how well the health care system around them functions. METHODS: This paper reflects the combined experiences of a group of experienced education researchers and the results of literature searches on performance assessment methods. CONCLUSION: Measurement of competence is different to measurement of performance. Components of performance could be re-conceptualised within a different domain structure. Assessment methods may be of a different utility to that in competence assessment and, indeed, of different utility according to the purpose of the assessment. An exploration of the utility of potential performance assessment methods suggests significant gaps that indicate priority areas for research and development.

The use of portfolios for assessment of the competence and performance of doctors in practice.

BACKGROUND: The use of portfolios can potentially provide flexibility in the summative assessment of doctors in practice. An assessment system should reflect and reinforce the active and planned professional development goals of individual doctors. This paper discusses some of the issues involved in developing such a system. RESULTS: To provide a complete picture of an individual doctor's practice, we suggest that a portfolio should encompass: (1) evidence covering all three domains of patient care, personal development and context management; (2) evidence that the person continuously undertakes critical assessment of their own performance, identifies and prioritises areas requiring enhanced performance and takes action to improve them as appropriate; (3) evidence that has been generated by assessments that are acceptably reliable, and (4) evidence which, taken in its entirety, is sufficient, valid, current and authentic. We include a suggested outline of the components of such a portfolio and suggest some criteria to determine the effectiveness of learning cycles. Portfolio reliability and validity requires sufficient evidence on which to base a judgement combined with reliable processes. CONCLUSION: Carefully specified portfolios can contribute to a system that ensures all doctors take an active part in identifying and meeting their own learning needs. Such a system, if properly implemented, would have a greatly beneficial impact on continuous quality improvement for the profession in general.

Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score.

BACKGROUND: Objective structured clinical examination (OSCE) standard-setting procedures are not well developed and are often time-consuming and complex. We report an evaluation of a simple 'contrasting groups' method, applied to an OSCE conducted simultaneously in three separate schools. SUBJECTS: Medical students undertaking an end-of-fifth year multidisciplinary OSCE. METHODS: Using structured marking sheets, pairs of examiners independently scored student performance at each OSCE station. Examiners also provided a global rating of overall performance. The actual scores of any borderline candidates at each station were averaged to provide a passing score for each station. The passing scores for all stations were combined to become the passing score for the whole exam. Validity was determined by making comparisons with performance on other fifth-year assessments. Reliability measures comprised interschool agreement, interexaminer agreement and interstation variability. RESULTS: The approach was simple and had face validity. There was a stronger association between the performance of borderline candidates on the OSCE and their in-course assessments than with their performance on the written exam, giving a weak measure of construct validity in the absence of a better 'gold standard'. There was good agreement between examiners in identifying borderline candidates. There were significant differences between schools in the borderline score for some stations, which disappeared when more than three stations were aggregated. CONCLUSION: This practical method provided a valid and reliable competence-based pass mark. Combining marks from all stations before determining the pass mark was more reliable than making decisions based on individual stations.

Sheffield Assessment Instrument for Letters (SAIL): performance assessment using outpatient letters.

CONTEXT: Well-designed assessments of performance are urgently required for training doctors, and to provide indicators of the quality of practice. OBJECTIVES: To design an assessment process that uses routine outpatient letters, and to evaluate its validity, feasibility, reliability (reproducibility and discrimination) and potential educational impact. PARTICIPANTS: All 26 paediatric registrars in North Trent attending annual assessment panel in 1999 participated. STUDY DESIGN: An assessment instrument (SAIL) was developed from a consensus framework for good practice in written communication. It comprises an 18-point checklist and a global rating scale. Three judges applied the instrument to 260 letters from the routine clinical practice of the 26 participants. Results We achieved consensus on good practice in written communication. This was in keeping with the published literature. All participants completed the assessment. Scoring took 3-6 min per judge per letter. The reliability coefficient in this test situation is 0.72. Modelling predicts that a coefficient of 0.8 (the threshold for high-stakes judgements about performance) can be achieved with more judges or letters. The assessment results are well suited to formative feedback. CONCLUSIONS: SAIL uses letters as a face valid indicator of written communication performance. The instrument is feasible to use, and produces reliable results when applied to paediatric registrars to inform the annual Record of In-Training Assessment (RITA). Feedback from the assessment should help doctors to improve their written communication. Its use may extend to other specialities and settings including revalidation.

Development of a three-centre simultaneous objective structured clinical examination.

OBJECTIVE: To describe the development, organization, implementation and evaluation of a yearly multicentre, identical and simultaneous objective structured clinical examination (OSCE). SUBJECTS: All fifth-year medical students in a 6-year undergraduate medical programme. SETTING: The Christchurch, Dunedin and Wellington Schools of Medicine of the University of Otago, New Zealand. METHOD: One practice and two full 18-station OSCEs have been completed over 2 years, for up to 72 students per centre, in three centres. The process of development and logistics is described. Data are presented on validity, reliability and fairness. RESULTS: Face and content validity were established. Internal consistency was 0.83-0. 86 and interexaminer reliability, as assessed by the coefficient of correlation, averaged 0.78. Students rated the OSCE highly on relevance. Of the total variance in total OSCE marks, the schools contributed 6.9%, and the students 93.1%, in the first year. In the second year the schools contributed 6.2% and the students 93.8%. CONCLUSION: Implementation of a psychometrically sound, multicentre, simultaneous and identical OSCE is possible with a low level of interschool variation.

Strategic planning in medical education: enhancing the learning environment for students in clinical settings.

BACKGROUND: The 1999 Cambridge Conference was held in Northern Queensland, Australia, on the theme of clinical teaching and learning. It provided an opportunity for groups of academic medical educators to consider some of the challenges posed by recent changes to health care delivery and medical education across a number of countries. PURPOSE: This paper describes the issues raised by the practical challenges posed by the current environment and how they might be addressed in ways that could promote more effective learning in clinical settings. METHOD: A SWOT analysis is a tool that can help in forward planning by identifying the strengths, weaknesses, opportunities and threats presented by any situation. Our SWOT analysis was used to generate a list of items, from which we chose those most feasible and most likely to promote positive change. RESULTS: Twenty different issues were identified, with four of them chosen by consensus for further elaboration. The discussion gave rise to four main recommended strategies: ensuring that clinical teachers thoroughly understand the purpose and process of learning in clinical settings; equipping learners with 'survival skills'; making the best use of learning resources within different clinical environments and making judicious use of information technology to enhance learning efficiency. CONCLUSIONS: The four strategies were selected not only because of their inherent importance, but also because of their feasibility. Modest changes can motivate students to feel part of a clinical team and a 'community of practice' and enhance their capacity for self-regulated practice.

The parallel rural community curriculum: an integrated clinical curriculum based in rural general practice.

INTRODUCTION: In an attempt to address the rural medical workforce maldistribution and the concurrent inappropriate caseload at the urban tertiary teaching hospitals, Flinders University and the Riverland Division of General Practice decided to pilot, in 1997, an entire year of undergraduate clinical curriculum in Australian rural general practice. This program is called the Parallel Rural Community Curriculum (PRCC). This paper is a discussion of the aims of the programme; student selection; practice recruitment; curriculum structure, and academic content, together with lessons learnt from the evaluation of the first cohort of students' experience of the course. METHODS: Independent external evaluators undertook a thematic analysis of a series of structured interviews of students and faculty involved in both the PRCC and the traditional curriculum. The mean examination results were determined and a rank order comparison of student academic performance was undertaken. RESULTS: The eight selected volunteer students reported greater access to patients and clinical learning opportunities than their mainstream counterparts and learned clinical decision making in the context of the whole patient, their family, and the available community resources. They identified patients with 'core' clinical conditions and had a longitudinal exposure to common diseases, whereas hospital-based peers had a cross-sectional exposure to highly filtered illness. The PRCC students' academic performance improved in comparison with that of their tertiary hospital peers' and in comparison to their own results in previous years. CONCLUSION: The PRCC curriculum has cut across the traditional clinical discipline boundaries by teaching in an integrated way in rural general practice. It has affirmed the potential role of true generalist physicians in undergraduate medical education.

Assessing clinical competence at the undergraduate level.

This booklet aims to provide relevant background information and guidelines for medical school teachers in clinical departments charged with assessing the clinical competence of undergraduate students. It starts by emphasizing the difference between clinical competence and clinical performance. An approach to defining what should be assessed is outlined. The technical considerations of validity, reliability and practicability are discussed with reference to the ward- or practice-based setting and to the examination setting. The various methods available to assess aspects of competence are described and their strengths and weaknesses reviewed. The paper concludes with a discussion of the important issues of scoring and standard setting. The conclusion is reached that the quality of many current assessments could be improved. To do so will require a multi-format approach using both the practice and examination settings. Some of the traditional methods will have to be abandoned or modified and new methods introduced.

The observed long-case in clinical assessment.

The University of Adelaide has developed a comprehensive approach to final-year clinical assessment through the use of ward ratings during student internships and the administration of an objective examination of clinical competence at the end of the year. The traditional clinical viva has been retained only to make pass-fail decisions on borderline students. Problems in making these decisions prompted a modification of the viva so that each student undertook two long-cases, each being observed by a pair of examiners. Better agreement among examiners and more effective decision-making seemed to ensue. The success of this approach led to the introduction of observed long-cases into the ward setting. This move has been strongly supported by staff and students. The opportunities for remedial action during the course were regarded as particularly valuable. This approach may offer a practical alternative to more sophisticated assessment techniques, such as those involving standardized patients, without the need for additional resources. However, psychometric studies will have to be performed before the approach can be fully recommended.