Administration of open-ended test questions by computer in a clerkship final examination.

PURPOSE: To evaluate the reliability, efficiency, and cost of administering open-ended test questions by computer. METHODS: A total of 1,194 students in groups of approximately 30 were tested at the end of a required surgical clerkship from 1993 through 1998. For the academic years 1993--94 and 1994--95, the administration of open-ended test questions by computer was compared experimentally with administration by paper-and-pencil for two years. The paper-and-pencil mode of the test was discontinued in 1995, and the administration of the test by computer was evaluated for all students through 1998. Computerized item analysis of responses was added to the students' post-examination review session in 1996. RESULTS: There was no significant difference in the performances of 440 students (1993--94 and 1994--95) on the different modes of test administration. Alpha reliability estimates were comparable. Most students preferred the computer administration, which the faculty judged to be efficient and cost-effective. The immediate availability of item-analysis data strengthened the post-examination review sessions. CONCLUSION: Routine administration of open-ended test questions by computer is practical, and it enables faculty to provide feedback to students immediately after the examination.

Patients don't present with five choices: an alternative to multiple-choice tests in assessing physicians' competence.

PURPOSE: To evaluate an open-ended, computer-scored testing format designed to overcome certain limitations of multiple-choice questions. METHOD: Test items covering content in family medicine were administered in two different formats to 7,036 resident physicians in 380 training programs, and to 35 experienced, board-certified physicians in conjunction with the In-training Examination of the American Board of Family Practice. Examinees completed a booklet of 40 open-ended, uncued (UnQ) test items by selecting the answer to each item from a list of over 500 responses. Similar items were administered using the standard multiple-choice question (MCQ) format. One year later, another test of 40 UnQ test items dealing with core content in family medicine was administered to 7,138 residents. RESULTS: Examinees completed over 560,000 UnQ responses with high compliance and few errors. Both reliability and validity for the UnQ format were higher than for the MCQ format, and the UnQ items discriminated more accurately among levels of physicians' experience. The UnQ format almost eliminated the possibility that the physicians could answer questions by sight recognition or random guessing, and it was particularly effective in measuring knowledge of core content. CONCLUSIONS: This study supports the feasibility of administering open-ended test items to enhance tests of physicians' competence.

A statewide system to track medical students' careers: the Pennsylvania model.

In 1994 the Commonwealth of Pennsylvania announced a statewide Generalist Physician Initiative (GPI) modeled after The Robert Wood Johnson Foundation's GPI. Three-year grants totaling more than $9 million were awarded to seven of Pennsylvania's medical schools, including two that had already received GPI grants from the foundation. Stimulated by these initiatives, the state's six allopathic and two osteopathic medical schools decided to work together to develop a collaborative longitudinal tracking system to follow the careers of all their students from matriculation into their professional careers. This statewide data system, which includes information for more than 18,000 students and graduates beginning with the entering class of 1982, can be used to evaluate the impact of the Pennsylvania GPI, and it also yielded a local longitudinal tracking system for each medical school. This paper outlines the concept of the system, its technical implementation, and the corresponding implications for other medical schools considering the development of similar outcomes assessment systems.

Who is a generalist? An analysis of whether physicians trained as generalists practice as generalists.

Accurate data on the number of generalist physicians are needed to monitor the physician workforce and to plan for future requirements in the changing health care system. This study assessed the relationship between two frequently used definitions of a generalist physician: completion of graduate medical education (GME) in only a generalist discipline and physician's self-report of practicing as a generalist. Data for 4,808 physician graduates from six Pennsylvania medical schools from 1986 to 1991 were analyzed using information from the GME tracking census of the Association of American Medical Colleges and the Physician Masterfile of the American Medical Association. Of 1,291 physicians trained in a generalist discipline, 1,205 (93%) reported practicing as generalists. Conversely, of the 3,517 not trained in a generalist discipline, 3,358 (95%) were not practicing as generalists. These results indicate GME training is a valid predictor of self-reported practice and provide baseline data to monitor future changes.

Relationship of students' perceptions of faculty to scholastic achievement: are popular instructors better educators?

Student evaluation of the faculty is a standard practice in most medical schools. Implied in these evaluations is the motion that popular instructors (ie, those considered outstanding by the students) are better educators, whose teaching translates into higher scores for their students on examinations. We tested this hypothesis by comparing students' evaluations of the faculty with levels of academic achievement in a second-year pathology course. Objective measures of academic achievement included scores on final comprehensive examinations, final course grade, and performance on the United States Medical Licensing Examination (USMLE). During the 4 years studied (1990 to 1995), students belonging to groups with the highest ratings for their instruction performed no better than those with the poorest ratings. There was no correlation between students' perceptions of quality in teaching and their academic achievement. Our results indicate that students' evaluations of the faculty are subjective and do not correlate with objective results used in the assessment of student knowledge. Popular instructors are not necessarily better educators.

The virtues of extended matching and uncued tests as alternatives to multiple choice questions.

The objectives of this study were to compare the reliability and validity of written test formats that are widely used in medical education (multiple choice, uncued, extended matching, and true/false) and evaluate the effects of uncued examinations on long-term retention of medical knowledge. Uncued tests were introduced into a traditional course in general and systemic pathology (six interim tests). In the following year, students were given eight tests written in the four formats, each being used twice. The academic achievement of students in these 2 years was compared with that of students in 2 previous years, in which multiple choice tests were used. Measures of academic achievement included performance on a final comprehensive examination and the United States Medical Licensing Examination (USMLE). Student performance on uncued tests was consistent over time (i.e., there was no learning curve). Mean scores ranged from 77% to 84%, and coefficient alpha reliability estimates on 100-item tests were excellent (0.79 to 0.90). Extended matching tests were also reliable, with a mean coefficient alpha of 0.90. There was no significant relationship between test format and student performance on subsequent comprehensive examinations. Our results indicate that extended matching and uncued tests have considerable advantages over multiple choice and true/false examinations. They are more reliable, better able to discriminate the well-prepared from the marginal student, and well suited for tested core knowledge. Contrary to our expectation, extended matching questions with 20 choices presented to the student were as statistically reliable and valid as uncued queries with several hundred choices.

Assessment of physicians' interest in primary care training/retraining.

PURPOSE: To assess generalists and specialists interest in primary care training and the factors associated with this interest. METHOD: The study sample was drawn from the alumni of the Jefferson Medical College of Thomas Jefferson University (classes of 1970-1990) who were practicing in Pennsylvania. Family practitioners and general internists were defined as generalists; obstetrician-gynecologists (ob-gyns) and internal medicine subspecialists were defined as specialists. In 1995 a questionnaire was mailed consisting of 46 items assessing the physicians' interest in participating in primary care educational programs, reasons for any such interest, and preferences for content. Two items on the specialists' questionnaire asked about changing careers from specialist to generalist, and two items on the generalists' questionnaire asked about broadening the scope of their practices. RESULTS: The response rate was 54% (381/707). In all, 78% of the physicians expressed interest in primary care training. The generalists were more interested in primary care training than were the specialists (p < .001). The ob-gyns were more interested in primary care training than were the medical subspecialists (p = .01). Few of the medical subspecialists and no ob-gyns were influenced by plans to change careers to primary care. More of the ob-gyns than the medical subspecialists were motivated by plans to shift emphasis to provide more primary care. CONCLUSION: The results suggest (1) that although many specialists have an interest in primary care training, it is rarely motivated by plans to change to primary care practice, and (2) that generalists are very interested in expanding their abilities. Both of these findings should be considered in workforce planning.

Retrospective study of 2,200 involuntary psychiatric admissions and readmissions.

OBJECTIVE: The authors describe demographic data, the distribution of diagnoses, and comorbid psychoactive substance use in a large sample of patients involuntarily admitted to a psychiatric hospital from multiple crisis centers and explore the relative roles these variables may play in service utilization and admission rates. METHOD: Data on demographic characteristics and comorbid psychoactive substance use in 2,200 consecutive involuntary hospital admissions of 1,755 psychiatric patients were gathered. Pertinent demographic and comorbidity data at first admission for the 1,755 patients, 314 of whom were admitted more than once, were analyzed; then the data for the 1,441 single-admission patients and the data at first admission for the 314 patients who had multiple admissions were compared. Finally, the diagnostic distribution and comorbid psychoactive substance use in all 2,200 admissions were investigated, with attention to a subgroup of 88 high-risk patients (those with three or more admissions) who represented a total of 307 admissions. RESULTS: Specific demographic characteristics were represented in the patient group at a high level of statistical significance. The diagnosis of schizophrenia was significantly overrepresented. Schizophrenia and psychosis not elsewhere classified clustered in the subgroup with a high risk of readmission. CONCLUSIONS: The results suggest a specific profile for the patient with heightened risk of hospital admission: a young, unmarried, African American male who has schizophrenia without comorbid substance abuse. An effect size data analysis identified marital status and a diagnosis of schizophrenia as the variables associated with the greatest likelihood of admission. Unexpectedly, the impact of comorbid psychoactive substance use was relatively modest and showed a uniform distribution among diagnostic groups.

Gender differences among civilly committed schizophrenia subjects.

The authors describe a sample of 815 acutely ill schizophrenia patients hospitalized under a petition for involuntary inpatient treatment and illustrate how this group of patients can be considered as more representative of involuntary commitment status than more traditional "involuntary" patients drawn from State hospitals' wards. Available demographics of the general population from which the sample was drawn are also presented for comparison. The authors then report the gender distribution of several variables--age, marital status, psychoactive substance use, comorbidity, length of stay, and frequency of readmissions--among the study sample and discuss the relevance of these findings.

Primary care and nonprimary care physicians' concerns in practice and perceptions of medical school curriculum.

The purpose of the present study is to address the issue of physicians' concerns in practice and their perception of a medical school's curriculum with an emphasis on comparisons between primary and nonprimary care physicians. The sample consisted of 663 physicians who graduated from Jefferson Medical College (JMC) between 1982 and 1986, and also responded to a mailed questionnaire. Comparisons were made between physicians in primary care (n=234) and in nonprimary care (n=429) specialties on their responses regarding concerns in medical practice and evaluation of the medical school curriculum. Primary care physicians were more concerned about the time for their professional development whereas nonprimary care physicians were more concerned about an oversupply of physicians in their specialties, prospective hospital payment, and malpractice litigation. Regardless of the specialties, the physicians overall seemed very concerned about their personal time. Interpersonal skills were regarded by all respondents as an important aspect of the medical school's curriculum. The importance of psychological, social, and cultural factors in the curriculum was strongly supported by these physicians' responses, particularly among primary care and women physicians.

A solution to the cueing effects of multiple choice questions: the Un-Q format.

Although the cueing effects inherent in conventional multiple choice questions (MCQs) present serious limitations, this format continues to dominate testing programmes. The present study was undertaken to estimate the effects of cueing when MCQs are used to test medical students, and to evaluate the reliability, validity and feasibility of an alternative testing format. Equivalent items in both MCQ and open-ended, or uncued (Un-Q), formats were administered to 34 third- and fourth-year medical students. The students' mean % correct score on the MCQs was 11 percentage points higher than their mean level of performance on equivalent Un-Qs. When a second set of more difficult items was administered to 16 of these students, their mean performance on the MCQ items was 22 percentage points higher than their performance on equivalent Un-Qs. The results support the feasibility of large group administration of tests constructed in an open-ended format that can be scored by computer. Not only is this format equally reliable and economical when compared with the MCQ, but it also provides important advantages that strengthen its face validity. The Un-Q format can be used to test either simple recall or certain higher level problem-solving skills that cannot be tested by MCQs. Even more important, the results also suggest that the Un-Q format may be a more effective discriminator of academically marginal examinees.

Medical students who enter general surgery residency programs: a follow-up between 1972 and 1986.

Although a few surgical residents do leave the surgical field, little research has been done into the eventual whereabouts of all surgical residents. We undertook this follow-up to find out what proportion of them enter the surgical field and to identify changing patterns over the past 15 years. We compared 593 medical school graduates on demographics, academic credentials, and surgery program directors' rating of their performance in the first postgraduate year (PGY-1). Sixty percent remained in general surgery, and about 26% moved to another specialty within the surgical field. During the mid-1970s, those who remained within the surgical field had better academic credentials on average than those who switched out of the surgical field. However, more recently, the trend has reversed. There were no differences between the groups on age or sex. These results suggest that some good students are being recruited into surgical programs but are later lost in major career switches. Perhaps these changes are related to residents' preferences for specialties that offer more controllable lifestyles than the surgical field.

Cueing in multiple choice questions: a reliable, valid and economical solution.

Test questions dealing with diagnosis and management were administered to medical students in both multiple-choice format and an open-ended form which can be scored by computer. Open-ended questions are feasible to administer, equally reliable and may be more valid in certain domains than multiple-choice questions.