Educational benefits of student self-marking of short-answer questions.

A pilot study was undertaken to evaluate the feasibility and reliability of undergraduate medical student selfmarking of degree written examinations, and to survey student opinion regarding the process. The correlation between student and faculty staff scores for individual questions and the total examination was high (correlation coefficient ranged from 0.77 to 0.91: p < 0.001). There were no significant differences between the mean student and mean faculty staff scores for individual questions or the total examination: 98% (97199) of student scores fell within ± 15% of the faculty staff score, with 92% (91199) of students falling within ± 10%. Although the approach was demonstrated to be reliable, students generally failed to acknowledge the potential value of self-marking in terms of feedback and as a learning opponunity, and found the process stressful.

The role of assessment in expanding professional horizons.

This paper explores assessment innovations which have a system-wide effect on medical education and the medical profession. Important assessment approaches such as the objective structured clinical examination (OSCE), the portfolio, and hi-tech simulations are examples of reform-driven developments. A detailed account is provided on assessment areas that require further developments. The identified areas reflect current thinking in the Centre for Medical Education, University of Dundee Medical School.The assessment innovations are being developed alongside the implementation of the outcome-based curriculum. Areas that require extensive work are: assessment of progression towards defined outcomes, assessment of integrated abilities, assessment of different forms of medical knowledge, assessment of on-the-job learning, learning through assessment, assessment of error management and assessment of portfolio evidence. The identified areas for further assessment development are discussed and where appropriate a theoretical framework is provided.

The performance of foreign medical graduates on the National Board of Medical Examiners (NBME) standardized patient examination prototype: a collaborative study of the NBME and the Educational Commission for Foreign Medical Graduates (ECFMG).

OBJECTIVES: The purpose of the study was to explore foreign medical graduates' (FMGs) performance on a clinical skills (SPX) examination. The National Board of Medical Examiners (NBME) is in the process of developing an SPX for potential use in the United States Medical Licensing Examination (USMLE). The Educational Commission for Foreign Medical Graduates (ECFMG) is developing the Clinical Skills Assessment (CSA) as an additional requirement for FMGs who wish to be certified by ECFMG. DESIGN: Thirty-three FMGs and 151 United States medical students (USMSs) took the SPX during the winter of 1996 as part of the ongoing pilot studies conducted by the NBME. Four clinical skill areas were assessed: history-taking, physical examination, communication and interpersonal skills. The examination used in this research consisted of 12 cases. The examination utilizes standardized patients (SPs) who are trained to document examinee behaviours and evaluate the communication component of the test. The SPs were also trained to evaluate the English proficiency of the candidates. Candidates were also administered the Test of Spoken English developed by the Educational Testing Services (ETS). SETTING: The examination was conducted in one medical school which served as an SPX centre for NBME pilot studies. SUBJECTS: Thirty-three foreign medical students and 151 US medical students. RESULTS: The indications were that the majority of candidates in both groups felt the examination was moderately fair but 78% of FMGs felt moderately pressed for time, vs. 80% of the USMSs who did not feel pressed for time. Reliabilities obtained for the various SPX components were somewhat higher for the FMGs reflecting the heterogeneity of this group. CONCLUSIONS: The NBME-ECFMG collaborative study yielded important information regarding the NBME SPX prototype as a performance measure for FMGs.

[Assessment of clinical competence of medical school graduates in Italy with standardized patients. The opinion of the examinees]. / Sperimentazione in Italia del metodo del paziente standardizzato per la valutazione della competenza clinica. L'opinione dei neolaureati.

An international collaborative project for the evaluation of clinical competence at the end of the Medical School curriculum using the ECFMG-CSA (Educational Commission for Foreign Medical Graduates--Clinical Skills Assessment) prototype was started in Italy in April 1996. Faculty representatives from Italian Medical Schools and experts from the ECFMG in Philadelphia participated in the Project. The CSA consists of integrated clinical encounters with 10 standardized patients during which the examinee is asked to obtain a focused history, perform a relevant physical examination and communicate initial diagnoses and management plan to the Standardized Patient (SP). The SP then completes checklists that are scored by Faculty members. The project was concluded in Spring 1998 and a total of 173 new graduates were examined. The data elaborated by the primary site in Chieti University will be available in the Fall 1998 by the ECFMG in Philadelphia. This preliminary communication reports the opinions of the examinees on the ECFMG-CSA, contained in the questionnaires administered after the test. Most of the examinees considered this new methodology as a valid tool for the assessment of clinical competence, especially history-taking and interpersonal skills and stated that the SP simulations were realistic. The 72% of examinees indicated that the Medical School curriculum does not adequately prepare for the CSA examination. Lastly, 68% was in favour of including the SP in the Medical Licensing Examination.

Lessons learned from six years of international administrations of the ECFMG's SP-based clinical skills assessment.

PURPOSE: The Educational Commission on Foreign Medical Graduates (ECFMG) conducted international clinical skills assessments (CSAs) to evaluate the readiness of foreign medical graduates to enter U.S. residency programs, to validate national medical examinations in other countries, and to introduce other countries to new methods of evaluating medical students. METHOD: The ECFMG conducted CSA studies in the United States, Israel, Spain (Madrid and Barcelona), Ukraine, and Brazil between 1989 and 1995. ECFMG staff worked with local teams in following a seven-phase implementation process. The CSAs were conducted in each country's native language, and clinical cases were translated from English and culturally adapted. A total of 636 examinees took a ten-station standardized patient-based examination. RESULTS: A comparison of test results indicated stable psychometric properties and similar patterns of relationships among test components across all six countries. In every country, physical-examination and patient-note mean scores were lower than were history-taking scores, indicating the possibility of common skill deficiencies. CONCLUSION: The successful completion of international CSA projects in six countries suggests that high-quality standardized CSA projects are feasible and can be implemented from long distances.

Graduates of foreign medical schools: progression to certification by the Educational Commission for Foreign Medical Graduates.

The application process leading to certification by the Education Commission for Foreign Medical Graduates (ECFMG) was studied using the group of 9,491 graduates of foreign medical schools who initiated their applications in 1988. Using the ECFMG's database, these applicants' countries of citizenship, examination histories, certification status, and exchange visitor status were determined for a period of seven years and 9 months, ending in September 1995. Within that time, 45% of these applicants became ECFMG-certified, and 26% of that group entered residency programs accredited by the Accreditation Council for Graduate Medical Education. Of the total number of non-U.S. citizens (2,243) who entered such programs, 61% did so as exchange visitor physicians. The remainder had other kinds of visa status, such as permanent resident, or had become U.S. citizens. Although U.S. citizens and non-U.S. citizens achieved certification at the same rate (45%), the U.S. citizens had greater success in obtaining positions in graduate medical education (GME). Seventy-four percent of the ECFMG-certified certified U.S. citizens entered GME, versus 57% of the ECFMG-certified non-U.S. citizens. The status of the 55% of the 9,491 applicants who did not obtain certification is discussed; a portion of this group of this group continues to pursue certification. Details and requirements of the ECFMG certification process are also described.

Reliability of performance-based clinical skill assessment of emergency medicine residents.

OBJECTIVE: To test the overall reliability of a performance-based clinical skill assessment for entering emergency medicine (EM) residents. Also, to investigate the reliability of separate reporting of diagnostic and management scores for a standardized patient case, subjective scoring of patient notes, and interstation exercise scores. METHODS: Thirty-four first-year EM residents were tested using a 10-station standardized patient (SP) examination. Following each 10-minute encounter, the residents completed a patient note that included differential diagnosis and management. The residents also were asked to read an ECG or chest x-ray (CXR) associated with each case. History, physical examination, and interpersonal skills were scored by the SPs. The patient note, CXR, and ECG readings were scored by faculty emergency physicians. Intercase reliability was determined for the residents. RESULTS: Global score reliability, Cronbach's alpha = 0.85. Reliabilities for the other components were: history, 0.77; physical examination, 0.83; and interpersonal skills, 0.80. Differential diagnosis and management reliabilities were 0.61 and 0.66, respectively. Subjective scoring of the patient note resulted in acceptable reliability for legibility (0.80), history completeness (0.80), and history organization (0.81). Physical examination completeness and organization reliabilities were 0.74 and 0.73. For ECG and CXR readings, alpha = 0.74 and 0.34, respectively. CONCLUSIONS: SPs can be used to reliably assess bedside clinical skills of EM residents. While component reliability levels are slightly lower than the global clinical skill reliability coefficient, they are still high enough to use for identification of individual strengths and weaknesses.