Teaching and assessing clinical skills: a competency-based programme in China.

The objective of this study was to develop a competency-based clinical skills teaching and assessment programme in China utilizing modern teaching techniques. Medical teachers from three schools agreed on items for inclusion in the complete physical examination of an asymptomatic adult, an outline for an adult and paediatric history, and important interviewing skills. Lesson plans, performance checklists, and written and videotape training materials were developed. Standardized patients were trained at one school to assist with the teaching at that school and with the assessment at all three schools. A national, a provincial, and a local medical school in China were used. Before beginning the new curriculum for students in their first year of clinical training, baseline data were collected on skills of students at various levels of training in the previous curriculum at all three schools. Although in the previous curriculum there was some improvement in clinical skills among advanced compared to more junior students, performance was lower than expected by staff. One year after implementation of the new curriculum, students were evaluated. These students significantly outperformed their counterparts as well as the more senior level students tested the previous year. This project has established a competency-based teaching and assessment programme in China that allows for rapid improvement in the clinical skills of students. Within a short time, a sophisticated group of medical educators has been formed, who now function as consultants to other educators in their own country. Many aspects of this programme are being adapted throughout China and are applicable to medical schools throughout the world.

Pilot study of the use of the ECFMG clinical competence assessment to provide profiles of clinical competencies of graduates of foreign medical schools for residency directors. Educational Commission for Foreign Medical Graduates.

PURPOSE: To conduct the first of a series of pilot projects of the clinical competence assessment (CCA) of the Educational Commission for Foreign Medical Graduates (ECFMG) in order to provide profiles of clinical competencies of graduates of foreign medical schools for residency directors in the United States and for governments and institutions in other countries. METHOD AND RESULTS: In September 1992 the first pilot project of the ECFMG CCA was conducted for a program director who wanted to evaluate ten first-year residents in a midwestern U.S. program. The CCA consists of integrated clinical encounters with ten standardized patients, 60 laser videodisc pictorials, and analysis of test items of previously completed ECFMG certification examinations. Profiles of the following clinical competencies were provided to the program director: data gathering (history and physical examination), interviewing and interpersonal skills, diagnosis and management skills, interpretation of diagnostic and laboratory procedures, written communication of information to the health care team, and spoken-English proficiency. The profiles were provided as individual scores compared with mean scores of a reference group of 525 first-year residents who took the CCA at four U.S. assessment centers, and as percentile scores with a range of one standard error of measurement. CONCLUSION: The individual performance data in this first pilot project were valuable to the program director, who used them to supplement scores on a written examination during the first residency year. The pilot project has shown the ECFMG CCA to be a useful tool for program directors to evaluate applicants and residents who are graduates of foreign medical schools.

ECFMG assessment of clinical competence of graduates of foreign medical schools. Educational Commission for Foreign Medical Graduates.

OBJECTIVE: To develop an assessment of clinical competence of graduates of foreign medical schools and to determine the reliability and validity of the assessment and the feasibility of large-scale administration. DESIGN: The Educational Commission for Foreign Medical Graduates (ECFMG) clinical competence study included (1) clinical encounters with standardized patients to assess history taking, physical examination, and communication skills; (2) laser videodisk pictorials to assess identification and interpretation of diagnostic procedures; (3) written clinical vignettes to assess diagnosis and management skills; and (4) assessment of spoken English. A uniform method of operating the test centers and of training the standardized patients was developed. SETTING: Medical schools and their primary teaching hospitals and affiliated hospitals. PARTICIPANTS: Six hundred twenty-four first-year residents, of whom 525 are graduates of foreign medical schools. MAIN OUTCOME MEASURES: Scores, reliability coefficients, validity measures, feasibility of multisite administration, trends of scores over time, and acceptability by examinees. RESULTS: The ECFMG clinical competence assessment was conducted at four geographically separate test centers. Reliability coefficients were high (.85) for the integrated clinical encounter and were in a reasonable range (.71 to .82) for all test components. The assessment adds to the predictability of the residents' performance in the hospital over that of current ECFMG certification examinations. Test security was addressed by demonstrating no consistent pattern of change in scores over testing dates. Virtually all examinees thought the assessment was appropriate. Standardized patients were able to assess spoken English accurately. CONCLUSION: The feasibility of conducting a reliable and valid test of clinical competence for graduates of foreign medical schools was demonstrated for this test population.

Technical issues: logistics. AAMC.

The author states that she became interested in standardized patients (SPs) around 20 years ago as a means of developing a more uniform and effective way to provide instruction and evaluation of basic clinical skills. She reflects upon in detail: (1) the logistics of using SPs in teaching; (2) how SPs are used in assessment; (3) what aspects of performance SPs can be trained to record and evaluate; (4) issues concerning checklists; (5) evaluation of interviewing skills; (6) evaluation of written communication skills; (7) importance of defining what is being tested; (8) various kinds and uses of inter-station exercises and problems of scoring them; (9) case development and the various sources for case material; (10) ways to generate scores; (11) selecting and training SPs; (12) role of the faculty and primary importance of bedside training with real patients; and (13) pros and cons of national versus single-school efforts to use SPs. She concludes by cautioning that further research must be done before SPs can be used for high-stakes certifying and licensing examinations.

A new program to enhance the teaching and assessment of clinical skills in the People's Republic of China.

The authors describe the history and current state of development of an ongoing and still-evolving collaboration--between three Chinese medical schools, the China Medical Board of New York, Inc. (CMB), and one of the authors (PLS) and her staff at the University of Massachusetts Medical School--to enhance the teaching and assessment of clinical skills in the schools. The program grew out of a request by the three schools to the CMB for assistance in improving their process of evaluating students. Its goal is to develop a uniform and comprehensive course for teaching and assessing basic clinical skills. Standardized patients are also being used as instructors and evaluators. The program is co-sponsored by the three schools and the CMB. Background is given about the CMB, its involvement in Chinese medical education, and its American perspective on the nature of education in Chinese institutions of modern medicine and the factors that can assist and impede educational change. The authors present the mechanisms of the program, the benefits that have occurred so far, and the problems encountered in creating the model curriculum to teach clinical skills that is now being implemented by the three schools. Future plans are outlined, including the hope that the three schools will serve as a resource and disseminate some or all aspects of the new approach to similar medical schools in China, and that the program can serve as a model for implementation of similar programs in other countries.

Positive effects of a clinical performance assessment program.

Since 1986, there has been a clinical performance assessment program for fourth-year students at the University of Massachusetts Medical School. Students interact with several standardized patients (SPs) and complete other tasks such as interpretation of electrocardiograms and interpretation of X-rays. Scores are generated both by checklists and rating forms completed by the SPs and by paperwork completed by the students at the end of each encounter. Since 1986, students have been asked how frequently they have been observed by faculty and residents as they interacted with actual patients; the students report that such observations have markedly increased. Since 1989, there has been increased feedback to students by the attending faculty during and following clinical rotations. Although it is difficult to claim cause and effect, it is clear that since the inception of this exercise, the faculty have made a conscious effort to improve students' clinical skills by providing increased observation and feedback.

The association between students' research involvement in medical school and their postgraduate medical activities.

The authors examined the impact of students' research involvement during medical school on their postresidency medical activities. The three medical schools involved--The Pennsylvania State University College of Medicine (PSU), The University of Connecticut School of Medicine (UCONN), and The University of Massachusetts Medical School (UMASS)--have nearly indistinguishable applicant, matriculant, and curriculum profiles. However, at PSU a research project is a curriculum requirement for students who did not do medical research prior to entering medical school. Questionnaires were sent to all graduates from the classes of 1980, 1981, and 1982. A total of 567 graduates completed the questionnaires, an overall response rate of approximately 76%. Medical school research experience was reported by 83% (183) of the PSU graduates, 34% (52) of the UCONN graduates, and 28% (54) of the UMASS graduates. When compared on a school-by-school basis, the graduates from the three schools did not differ with respect to residency specialty training, fellowship training, academic appointments, career practice choices, or postgraduate research involvement. However, when all the graduates studied were examined as a single group, medical school research experience was found to be strongly associated with postgraduate research involvement.

Results of a survey on the use of standardized patients to teach and evaluate clinical skills.

In 1989, a survey was sent to each U.S. and Canadian medical school requesting information about how standardized patients are used for teaching and evaluating clinical skills, and 95% of the schools responded. Although there was widespread use of standardized patients throughout the curricula, the role and training of these patients varied markedly within a given school as well as across schools. One outcome of this survey is the development of a network to share resources, protocols, and training material to enhance the development of this educational strategy.

An assessment of the clinical skills of fourth-year students at four New England medical schools.

This paper describes a collaborative effort among five New England medical schools to assess important clinical skills of fourth-year medical students graduating in the class of 1988; results are presented from the four schools that provided sufficient data. Faculty from each school developed 36 patient cases representing a variety of common ambulatory-care problems. Over the course of a day, each student, on average, interacted with 16 different standardized patients, who were nonphysicians trained to accurately and consistently portray a patient in a simulated clinical setting. The students obtained focused histories, performed relevant physical examinations, and provided the patients with education and counseling. At each school, the performance of a small number of the students fell below standards set by the faculty. These deficiencies were not detected with the evaluation strategies currently being used. Although the use of standardized patients should not substitute for the process of faculty observing students as they interact with real patients, it appears that standardized patients can provide faculty with important information, not readily available from other sources, about students' performances of essential clinical activities and the levels of their clinical skills.

The Medical Education Evaluation Program of the state of Ohio.

In October 1988, seven foreign medical graduates participated in the first administration of the examination devised by the Medical Education Evaluation Program (MEEP) mandated by the State Medical Board of Ohio. The MEEP was established to provide an objective evaluation of an applicant's clinical competencies; passing the MEEP examination was intended to certify that the applicant's clinical skills were comparable to those of a medical student graduating from a school accredited by the Liaison Committee on Medical Education. An applicant who successfully passed the MEEP examination and fulfilled the other Ohio licensure requirements would be eligible to take the Federation Licensing Examination (FLEX) and apply for an unrestricted license to practice medicine in Ohio. The paper describes the origin and development of the MEEP examination and the testing modalities selected (multiple-choice examinations and the use of standardized patients). Four fundamental areas were tested; these are named and described, along with the method for calculating scores for each area and the criteria for passing the different components of the examination. Although the small sample size prohibited meaningful data analysis for the performance of the first group of MEEP candidates, the MEEP examination appears to meet psychometric standards of certifying and licensing examinations, based on data from comparable tests taken by beginning fourth-year medical students in New England and NBME Part III examination examinees. Some potential pitfalls of the MEEP examination are mentioned, as well as the fact that it presents a challenge to boards of medical examiners of other states to implement performance-based assessments of physicians who graduate from non-accredited medical schools.

Evaluating diagnostic pattern recognition: the psychometric characteristics of a new item format.

The issues related to measuring pattern recognition are similar to measuring other clinical skills; how can testing time be used most efficiently to obtain reliable and valid scores? How should tests be constructed to obtain scores that validly reflect individual performance in making diagnoses? Generalizability analyses indicated that performance in one topic area does not predict performance in other areas very well. For example, students who were relatively expert in diagnosing patients with headaches tend not to be expert in diagnosing patients with chest pain, joint pain, etc. Therefore, to evaluate diagnostic pattern recognition skills in general, it is preferable to sample more presenting complaints with fewer items directed at each one, rather than to sample more items within a small number of presenting complaints. Approximately one hour of testing time would be required to generate a reasonably reliable score (ie, with a generalizability coefficient greater than 0.80). For diagnostic or remedial purposes, performance can be examined by content area to determine specific areas of weakness for individual students. The next phase of this study will be directed at determining whether there are benefits to using the current matching format with a relatively long list of response alternatives rather than a traditional multiple choice format with five choices. It is hypothesized that the shorter list differentially benefits the lower ability students and the more junior students. Efforts will also be directed to determining the applicability of the item format to other content areas such as diagnostic testing and therapy.

A diagnostic fourth-year performance assessment.

The University of Massachusetts Medical Center, Worcester, attempted to develop a standardized, performance-based test battery aimed directly at assessing the critical aspects of clinical competence required for graduation from medical school. The battery used a blend of standardized patient-based and written test materials and was designed to yield a profile of scores, providing a "diagnosis" of student strengths and weaknesses on a skill-by-skill basis. Results indicate that a stable, reproducible assessment of clinical skills can be achieved in a one- to two-day test battery, depending on the specific skills measured. The resulting score profile provides faculty with important information about the clinical competence of students that is not readily available from other sources, thus improving the breadth and accuracy of student assessment. A long-term goal is that performance-based testing techniques will be incorporated into the licensure process to evaluate clinical skills and ensure the competence of graduating physicians.

Ensuring the clinical competence of medical school graduates through standardized patients.

There are substantial problems with the clinical training provided to medical students and with the assessment procedure used by medical schools to ensure that students have acquired the clinical skills necessary for graduate medical education. These skills are not evaluated carefully nor systematically at any point in training or licensure. This article describes the use of standardized patients to help resolve some of these shortcomings. Standardized patients are non-physicians highly trained to function in the multiple roles of patient, teacher, and evaluator while realistically replicating a patient encounter. They are effective teachers of interviewing and physical examination skills. They can help to provide a controlled exposure to common ambulatory and difficult patient communication situations. Initial studies indicate the promise of this approach for ensuring the competence of medical school graduates.