Validity of simple approach to scoring and standard setting for standardized-patient cases in an examination of clinical competence.

The results are disappointing, providing little support for the validity of the case-passing decisions based on this simple approach to scoring and standard setting. The case-passing decisions predicted what the case author intended for about only 73% or 74% of the students on average and, with agreement expected by chance removed, predicted what the case author intended for about only 25% of the students. Even with the use of the optimal pass/fail cutoffs and the dropping of students with ambiguous borderline global ratings, the case-passing decisions failed to agree with the case authors' global ratings for 15% to 30% of the students. The findings might be dismissed as simply due to low reliabilities of passing decisions and global ratings based on a single case. Although this concern would apply to intercase reliabilities, which would be subject to case specificity, the appropriate reliabilities here would seem to be intracase (i.e., intrarater), which should be fairly high (if they could be computed). Nevertheless, it seems reasonable to expect much better agreement between results of case scoring and of standard setting developed by the case author and the case author's global ratings of performance on that case, given that the case author might recall the checklist, assign a weight to each item, and so forth. Also, case-passing decisions would possibly agree more with global ratings of live or videotaped performances than with ratings of written summaries of performance; however, that question remains a challenge for further research. In conclusion, the study provides only weak evidence, at best, for the validity of the scoring and standard setting commonly used with SP assessment. The results do not undermine claims about the realism of the SP approach, however, nor do they call into question the standardization afforded by this method of assessing clinical competence. The results do raise serious concerns about this simple approach to scoring and standard setting for SP-based assessments and suggest that we should focus more on the observation and evaluation of actual student performance on SP cases in the development of valid scoring and standard setting.

An overview of the uses of standardized patients for teaching and evaluating clinical skills. AAMC.

The author defines the term standardized patient (SP), the umbrella term for both a simulated patient (a well person trained to simulate a patient's illness in a standardized way) and an actual patient (who is trained to present his or her own illness in a standardized way). He first discusses the many values of simulated patients over actual patients as teaching and assessment tools in the classroom and refutes a few myths about the use of SPs. Then he recounts the origin and development of SPs over a three-decade period, beginning with his work as a neurologist at the Los Angeles County Hospital, where he trained a model from the art department to simulate a neurological patient and assist in the assessment of clinical clerks. He then describes additional roles of SPs that have developed, including: (1) their use in the Clinical Practice Examination created at Southern Illinois University School of Medicine and (2) the major use that has come into being over the last 10-15 years; facilitating the comprehensive assessment of clinical competence using multiple stations in examinations such as the objective structured clinical examination. He concludes with information about recent and current work on SPs, who are becoming more and more accepted in the assessment process, and urges skeptics not to make judgments about the value of SPs until they have experienced the technique firsthand and reviewed the literature concerning the extensive and often high-quality research about this assessment tool.

Screening test length for sequential testing with a standardized-patient examination: a receiver operating characteristic (ROC) analysis.

The feasibility of using sequential testing (i.e., using a screening test) to reduce the length and expense of a performance-based examination with standardized-patient cases was demonstrated previously. In the present study, quantitative criteria rather than practical considerations were used to determine optimal values for the length of the screening test (i.e., number of cases) and the location of the screening pass-fail cutoff (i.e., its relation to the mean of the pass levels for the different cases). Data were derived from five classes of senior students at the Southern Illinois University School of Medicine, 1987-1991. Specifically, receiver operating characteristic (ROC) curves were plotted for screening tests of varying lengths, with the points on each ROC curve corresponding to different pass-fail cutoffs on the screening test. The results showed that good accuracy can be attained with a screening test that is only one-third the length of the full examination and that the cutoff for this screen should be set slightly above the mean of the case pass levels to maximize sensitivity and specificity. The authors conclude that their study demonstrates the value of an ROC analysis in evaluating the psychometric properties of a screening test in sequential testing.

Standardized (simulated) patients' accuracy in recording clinical performance check-list items.

In large-scale performance-based assessment of medical professionals' clinical competence, simulated patients (standardized patients-SPs) are used not only to simulate case problems but also to record on check-list the examinees' clinical performance during their encounter with the SPs. The purposes of this study were to determine the SPs' overall accuracy in recording check-list items, and whether their accuracy was affected by the various characteristics of the check-list, and by the different times during a day and the different days within the examination when the recordings were obtained. Results showed that the SPs' accuracy in recording check-list items was good to very good and was affected by the length of the check-list, as well as by the type and the clarity of the check-list item. It was further found that the SPs' accuracy was consistent and did not vary over the course of a one-day or a 15-day examination.

Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine.

By the end of 1990-91, the Southern Illinois University School of Medicine had had six years of experience with comprehensive, performance-based examinations of senior medical students' levels of clinical competence; this report assesses the psychometric aspects of the six examinations given during that period. The examinations were aimed at determining the students' readiness for postgraduate training. Compared with other clinical performance-based assessments that use standardized patients (SPs), these examinations had two important and unique features: (1) the examinations assessed a comprehensive range of clinical skills and reasoning; and (2) they approximated the challenges of real clinical practice wherein a practitioner's skills need to be orchestrated and prioritized in order to meet the challenges of the case encountered. Each year, the performance-based assessment given was an intensive clinical examination requiring each student to work up 13 to 18 SP problems over a three-day period. To administer an examination to an entire class of students took three weeks. Because all students after the first year of administration (1986) were required to pass these examinations, the fairness of test design and scoring and the setting of performance standards for the examinations became important issues for the faculty. The results, accumulated over six years and based on a total of 6,804 student-patient encounters involving 405 students, indicate that this kind of clinical performance-based examination can discriminate a wide range of students' clinical performances. The results provide evidence for the examinations' test security, content validity, construct validity, and reliability.

Test security in examinations that use standardized-patient cases at one medical school.

The use of performance-based examinations consisting of standardized-patient (SP) cases has increased greatly in recent years. These examinations are typically long and thus require the presentation of the same SP cases to several consecutive examinee groups. Consequently, concerns have arisen about the potential for violations of test security whereby students who were tested early in the examination period pass on information to students tested later. These concerns are addressed using data from the SP-based examinations administered to five classes (1986-1990) of senior medical students at Southern Illinois University School of Medicine. Because of the length of the examinations, each class was randomly divided into five groups and the examination was administered to one group at a time, requiring three days of testing time per group and three weeks of testing time per class. The results showed no consistent, systematic changes in case means across the five groups tested at different times throughout the examination period, and thus provide no evidence of serious, widespread violations of test security.

Clinical behaviors and skills that faculty from 12 institutions judged were essential for medical students to acquire.

This paper describes a 1988-1989 collaborative mail survey of faculty opinion about clinical behaviors and skills that students should be expected to demonstrate prior to graduation from undergraduate medical school (hereafter called "exit objectives"). Selected faculty from 12 American and Canadian medical schools indicated whether each of 77 objectives was essential for every student to know or demonstrate prior to graduation; useful but not essential at the undergraduate level; or not applicable to their undergraduate program. Their responses provide a glimpse into faculty expectations regarding some of the exit behaviours and skills they deemed essential. Forty-two percent (32) of the 77 objectives were regarded as essential by 75% or more of the faculty members who responded. Essential objectives involved conducting organ system examinations, formulating problems and hypotheses, and gathering fundamental interview, physical, and screening examination data, including emergency examinations of the airway and circulatory systems. Other essential objectives involved collaboration and communication, demonstrating concern for legal and ethical values, and keeping abreast of current information within the discipline. Exit objectives related to the diagnosis and management of specific conditions were regarded as useful but not essential at the undergraduate level. Implications for medical education are discussed.

Inquiry: the pedagogical importance of a skill central to clinical practice.

The skill of inquiry is central to the task of the doctor confronted with a patient problem. Despite its importance this skill is not given appropriate emphasis in the education of medical students or in research concerning the clinical reasoning skills of doctors.

Methylmercury and the health of autochthons in northwest Quebec.

We conducted a cross-sectional epidemiologic and clinical study to evaluate the nature, magnitude, and frequency of unfavourable health states among residents of northwest Quebec. Of particular interest were the possible occurrence of Minamata disease and any other neurological abnormalities. We also sought to determine whether industrial or naturally-occurring mercury in the region's environment were causally associated with any disorders that might be detected. A total of 321 subjects were studied in four groups with contrasting exposure to point-source mercury from a chlor-alkali plant in the area. Methylmercury in hair served as the marker of subject's ingestion of any organic mercury (natural or industrial). Ten pre-chosen target variables, seven clinical and three neurophysiological, were assessed in all subjects. Relationships between mercury in hair and each target variable were calculated. We determined whether gradients in target variables corresponded to gradients in potential exposure of the subjects to the aquatic emissions of the chlor-alkali plant. Minamata disease was not found. Diabetes and alcohol-related health problems assessed as important clinically plausible confounders, were common. Differences in frequency or magnitude of clinical, toxicological, and physiological findings among the contrast groups were small and not of clinical or statistical significance. There were definite relationships between mercury in hair and four target variables despite the low values of mercury measured and the mild nature of the health problems observed. This would suggest that verifiable biological threshold levels for clinical effects, if such exist, may be substantially lower among Canadians than those currently adopted as policy for adults elsewhere. A gradient was found only for two of the ten target variables. Clinically meaningful alternate hypotheses for observed health effects emerge in the data and agree with observations in the field made by consulting clinical experts.

The use of performance-based assessment scores and traditional measures in predicting first-year residency performance.

The purpose of this study is to examine the relationship between students' performance on a performance-based clinical examination and their subsequent performance during the first year of residency. Additionally, its purpose is to further examine the relationship between the performance-based clinical examination and existing measures of clinical competence (clerkship ratings, and NBME Part I and II scores).

Direct, standardized assessment of clinical competence.

Doctor ratings of clerkship performance are often discounted as not accurately reflecting clinical competence. Such ratings are influenced by the following uncontrolled variables: case difficulty; differing rater focus and standards; lack of agreement on what constitutes acceptable performance; and collective patient care responsibility masks individual contributions. Standardized direct measures of clinical competence were developed to control these factors and allow direct comparisons of student performance. Students saw 18 patients representing frequently occurring and important patient problems. Student actions and decisions were recorded and subsequent responses to questions revealed knowledge of pathophysiology, basis for actions, use and interpretation of laboratory investigations, and management. Actions and responses were graded using a pre-set key. The examination covered 73% of designated clinical competencies. Examinations scores corresponded with independent measures of clinical competence. Reliability studies indicated that new cases can be substituted in subsequent years with confidence that scores will maintain similar meaning. Costs are +6.95 per student per case, which is modest considering the quality and quantity of information acquired. Methods described are practical for evaluation of clerks and residents and for licensing and specialty certification examinations.

A comprehensive performance-based assessment of fourth-year students' clinical skills.

Written examinations are widely used for assessment in clinical clerkships and for licensure and specialty board certification, as opposed to assessment based on actual performance with patients. This reliance on written examinations is due to their ease of use and perceived objectivity and occurs despite the fact that the examinations assess few components of clinical competence. Simulated patients can standardize the presentation of a patient problem; and, if the patients are employed in an assessment in a manner parallel to the design of written test items, the assessment can have an objectivity similar to that enjoyed by written tests. Such an assessment allows the major components of clinical competence to be tested. The results and feasibility of using simulated patients in a multiple-station assessment of an entire senior class in January 1986 are described. A second assessment was administered to a different senior class in December 1986. This latter assessment was designed in collaboration with another medical school that administered the same assessment to its senior students in March 1987.

Use of a medical reasoning aptitude test to help predict performance in medical school.

A medical reasoning aptitude test (MRAT) was designed to assess aptitude for clinical problem-solving in medical school applicants. The purpose of the study reported here was to determine whether the information provided by this test, when used in conjunction with college grade-point averages (GPAs) and scores on the Medical College Admission Test (MCAT), would improve the prediction of medical school performance. Specifically, the authors investigated the incremental predictive value of the MRAT relative to students' overall performance in medical school and, more specifically, to their knowledge and clinical reasoning during preclinical years and clinical performance during the clerkship year. Overall, it was found that, with the exception of the first year, the addition of MRAT scores to the GPAs and MCAT scores increased precision in identifying students who performed poorly or exceptionally well in the second year and the clinical clerkship year. This last finding is especially useful, since few other tests have provided that information. On the basis of the preliminary findings, the authors propose further use and validation of the MRAT.

The clinical reasoning process.

Medical school teachers must have an accurate idea of the doctor's clinical reasoning process (CRP) in order to provide students with learning experiences and evaluations that will ensure their acquisition of an effective and efficient CRP. It is difficult to derive this understanding from much that has been written on the subject. It is important to recognize that clinical problems are ill-structured and that the doctor's reasoning is built around a temporal unfolding of information. A model for the CRP is described along with a critique of other models that have been suggested. The results of research that examines components of the process must be seen in relation to the overall process.