Score reporting on NBME examinations.

Because the National Board of Medical Examiners (NBME) will introduce new comprehensive Part I and Part II examinations in 1991, a review has been made of score reporting methods to be used in the new examinations. The review was conducted also because of concern expressed by some that NBME examination scores are misused in medical schools and in resident selection. In this paper, selected aspects of score reporting are defined, the uses of score reports outlined, and the potential for misuse described. It should be noted that the NBME is obliged to make available numerical scores to state medical boards and to examinees. Individual scores are reported to others only with the permission of the examinee. The results of an opinion poll conducted by the NBME of medical educators and medical students are presented. The range of opinion is broad but favors numerical score reporting and a designated pass/fail score.

A survey of medical school teachers to identify basic biomedical concepts medical students should understand.

Insights from the cognitive sciences indicate a continuing need for physicians to understand conceptual knowledge from the basic sciences, despite recent concerns regarding the increasing amount of information in medicine and the growing emphasis on performance skills. A 1987 survey of selected basic science and clinical teachers in North American medical schools was undertaken to identify basic biomedical concepts that are important in the practice of medicine and to specify how difficult these are for students to learn, apply, or both. Responses from faculty (nominated by their deans to answer the survey) from 82% of the medical schools indicated considerable agreement between the basic science teachers and clinical teachers on the relative importance of a set of biomedical concepts, and showed relatively minor levels of disagreement on how difficult these concepts are. The judgments of these teachers could prove extremely useful in (1) determining concepts that--because of their importance--should receive special attention in curriculum efforts, and (2) determining concepts that--because of their difficulty--need "special handling."

History of not completing courses as predictor of academic difficulty among first-year students.

The study reported here was undertaken to determine whether the prediction of academic difficulty in the first year of medical school is enhanced by a consideration of the number of courses withdrawn from, repeated courses, and incomplete courses on a student's undergraduate academic record. All students enrolled from 1981 through 1985 at Southern Illinois University School of Medicine who experienced academic difficulty in the first year were selected for the study. Successful students were matched with these students in terms of minority or majority status and served as a control group. Discriminant and classification analyses were performed in a hierarchical stepwise manner to predict success or difficulty in the first year. The variables that were significant in discriminating between minority students who had academic difficulty and those who did not were the science grade-point average (grades in biology, chemistry, physics, and mathematics), the score on the reading subtest of the Medical College Admission Test (MCAT), and the number of withdrawals from courses. For majority students, the significant discriminating variables were the score on the MCAT biology subtest and the number of incompletes taken for courses. The results of this study have implications for medical school admissions committees, premedical advisers, and premedical students.

Use of a medical reasoning aptitude test to help predict performance in medical school.

A medical reasoning aptitude test (MRAT) was designed to assess aptitude for clinical problem-solving in medical school applicants. The purpose of the study reported here was to determine whether the information provided by this test, when used in conjunction with college grade-point averages (GPAs) and scores on the Medical College Admission Test (MCAT), would improve the prediction of medical school performance. Specifically, the authors investigated the incremental predictive value of the MRAT relative to students' overall performance in medical school and, more specifically, to their knowledge and clinical reasoning during preclinical years and clinical performance during the clerkship year. Overall, it was found that, with the exception of the first year, the addition of MRAT scores to the GPAs and MCAT scores increased precision in identifying students who performed poorly or exceptionally well in the second year and the clinical clerkship year. This last finding is especially useful, since few other tests have provided that information. On the basis of the preliminary findings, the authors propose further use and validation of the MRAT.

The validity of clerkship performance evaluations.

The validity of narrative evaluations of clinical performance in medical school was investigated for three successive classes of medical school graduates. The criteria were ratings obtained from residency supervisors one year after entering the programme. Bivariate correlational analysis, multiple regression analysis and canonical redundancy analysis were performed to examine the associations between the evaluations by teachers of competence in medical school clerkships and the subsequent ratings by residency supervisors. The results, best summarized in the canonical analysis, showed two relationships between the two sets of measures (lambda 1 = 0.54 and lambda 2 = 0.47). The first relationship revealed an overall-competence-in-medicine dimension; the second suggested an interpersonal-skills/professional-maturity dimension. Overall, knowledge of clerkship performance accounted for 15% of the variance in residency ratings. The findings suggest the need to improve the assessment of professional maturity and behaviour during medical school.

Instruction in medical decision making. A report of two surveys.

Results of two surveys (D and E) that assessed the current structure and status of instruction in medical decision making are reported. Both samples (Survey D, N = 80; Survey E, N = 92) consisted of members of the Society for Medical Decision Making. A consensus was obtained on topics considered important for teaching an introduction to clinical decision analysis to medical professionals. These topics were Bayes' theorem, decision trees, 2 X 2 tables, test sensitivity and specificity, utility, and ROC analysis. There was little agreement on course structure, level, or the preferred method for teaching decision analysis within medical settings. It was concluded that medical educators are in the process of constructing a knowledge base in decision-analytic techniques within academic medicine. It will soon be time to consider the place of more advanced topics within the continuum of medical education.

A comparison of objective and subjective measures of clinical competence.

The purpose of this study was to determine if objective performance measures from a simulated patient practical examination were related to faculty subjective assessments of surgery students' performance. Another purpose was to determine if those students performing well on a simulated patient practical examination were the same students awarded honor ratings by faculty. Faculty ratings of students (n = 72) on history taking ability, physical exam skills, problem identification, use of investigations, problem integration, patient management, and patient relations skills were correlated with objective measures on these same criteria achieved on a practical exam employing three simulated patient cases. Correlations were also calculated to study the relationship between practical exam scores and honor rating recipients. Results indicated significant relationships between faculty subjective ratings and practical exam outcomes on all practical examination skills except physical exam skills and patient management skills. Findings also indicated the performance of honor students on the practical examination was significantly better than non-honor students on two skills: history-taking and patient relations. The implications of these research findings are discussed.

Content knowledge and problem-solving skill in reviewing medical charts.

Skills in reviewing medical charts have been demonstrated components of clinical competence related to knowledge base, level of clinical experience, and basic observational skills. A study of the generalizability of performance on chart review exercises, which controlled for knowledge in the content area, was undertaken to determine their potential in evaluating students' problem-solving ability. Results of the study indicate that the case specificity which has characterized simulated problem-solving tasks is largely case, rather than content, specificity: correlations between scores on single charts demonstrated no consistent relationships for measures of proficiency, efficiency, and diagnostic accuracy. However, averaging the scores on two charts and then computing correlations resulted in significant positive measures for both proficiency and efficiency. The effects of length and difficulty level on the generalizability of tests of problem-solving ability are areas suggested for future research.

A study of the information-seeking skills of medical students and physician faculty.

In the research reported here, the authors compared (a) the abilities of third- and fourth-year students and physicians to solve problems requiring review of current literature with (b) the information-seeking procedures of the three groups in researching those problems. The subjects were given a patient care question to answer, and logs were maintained to document their information-seeking processes. The findings indicated no differences among the three groups in accurately solving patient care problems; a difference in information-seeking processes between third-year students and physicians; and no differences between third-year and fourth-year students or between physicians and fourth-year students in their approaches to seeking information. Mean scores indicated that all three groups need additional training on accessing the literature in making patient care decisions.

An alternative method to predict performance: canonical redundancy analysis.

The relationships between predictors of performance and subsequent measures of clinical performance in medical school were examined for two classes of students at Southern Illinois University School of Medicine. A relatively new statistical technique, canonical redundancy analysis, was used to evaluate the association between six academic and three biographical preselection characteristics and four measures clinical competence. While producing results consistent with the univariate methods traditionally used, the technique extends understanding of the total relationship between the two sets of measures. The preselection variable found to be most related to clinical competence was the undergraduate nonscience grade-point average. Overall, 16 percent of the variation in clinical performance was accounted for on the basis of knowledge of the preselection characteristics.