Shortcomings in the evaluation of students' clinical skills and behaviors in medical school.

The authors review the methods by which U.S. medical schools have evaluated student achievement during the twentieth century, especially for the assessment of noncognitive abilities, including clinical skills and behaviors. With particular reference to the current decade, information collected by the Liaison Committee on Medical Education (LCME) is used to examine the congruence of assessment methods with the rising tide of understanding--and accreditation requirements--that knowledge, competence, and behavioral objectives require different methods of assessment to measure the extent of students' learning in each domain. Amongst 97 medical schools having accreditation surveys between July 1993 and June 1998, only 186 of 751 basic science courses tested students' noncognitive achievements in things such as the preparation for and participation in small-group conferences, the quality of case-based discussion, library research and literature reviews, and research projects, despite staking out scholarship, habits of life-long learning, and reasoned thinking as educational objectives. In the clerkships of these schools, structured and observed assessments of clinical skills--with standardized patients and/or OSCEs--contributed 7.4-23.1% to a student's grade (depending on the clerkship discipline), while the predominant contribution (50-70% across the clerkships) was made by resident and faculty ratings that were based largely on recollections of case presentations and discussions having little relationship to interpersonal skills, rapport with patients, and logical and sequenced history taking and physical examination. On a more optimistic note, the results show that the number of schools using standardized patients in one or more clerkships increased between 1993 and 1998 from 34.1% to 50.4% of the 125 schools in the United States, and the number of schools using standardized patients in comprehensive fourth-year examinations increased from 19.1% to 48% of the total. Despite such progress, this study shows that too many medical schools still fail to employ evaluation methods that specifically assess students' achievement of the skills and behaviors they need to learn to practice medicine. The findings of this article explain why accreditors are paying closer attention to how well schools provide measured assurances that students learn what the faculties set out to teach.

On the culture of student abuse in medical school.

The abuse of students is ingrained in medical education, and has shown little amelioration despite numerous publications and righteous declarations by the academic community over the past decade. The culture of abuse conflicts with the renewed commitments of medical educators and practice professionals to imbue students with a higher degree of professionalism and cultural sensitivity. The authors describe the profiles of student abuse, drawn from recent national surveys of medical students using the AAMC Medical School Graduation Questionnaire, and focus on the most common forms of reported mistreatment--public belittlement and humiliation--that appear to be misguided efforts to reinforce learning. Along with others, the authors believe that the use of aversive methods to make students learn and behave is likely to foster insensitive and punitive behaviors that are passed down from teacher to learner, a "transgenerational legacy" that leads to future mistreatment of others by those who themselves have been mistreated. The undesirable result is compounded when these behaviors are adopted and directed toward patients and colleagues. The authors advocate more concerted action to curtail the abuse of medical students, citing current and proposed accreditation standards that will be employed more stringently by the LCME, and propose a series of more assertive actions that schools should take. The authors stress that the attitudes, behaviors, and values that students acquire in medical school are as much the products of their socialization as the outcomes of curricular design and pedagogy, and implore medical educators to tidy up the environment for learning.

On the importance and validity of medical accreditation standards.

In late 1997, the authors conducted a national survey of communities of interest about the importance and clarity of 44 accreditation standards applied to teaching, learning, and evaluation in medical schools by the Liaison Committee on Medical Education (LCME). Questionnaires were mailed to deans and educational administrators at U.S. medical schools; current LCME members and surveyors and those who had served during the preceding five years; a random selection of residency program directors drawn from both general practice and speciality disciplines; sample groups of medical students and residents; and a cohort of practicing physicians not affiliated with academic medical institutions. Altogether 1,659 questionnaires were mailed, and 701 responses were received (42%). The recipients were asked to use a five-point Likert scale to rate each of the 44 standards both for its perceived importance as an indicator of the quality of undergraduate medical education and for the clarity with which the standard's intent was conveyed. Although the mean ratings of importance all fell in the "moderately important" and "highly important" areas across the respondent groups, the ratings divided into three groups, semantically and statistically. At the high end for importance are standards dealing with fundamental qualities of students, instruction, and the structuring of resources. At the low end of the importance scale are standards dealing largely with matters of process. The ratings for clarity were systematically lower than the ratings for importance, and in some cases the rating for clarity were even more widely discrepant with the ratings for importance. Individual comments by respondents about certain standards were critical of their complicated construction and of confusion about their meaning and measures of compliance. One or more of these hallmarks--being rated of lower importance or clarity, and being the target of criticism by survey respondents--distinguished most of the standards that earlier study had shown are often neglected by surveyors. The predictive validity of each of a number of standards was examined by testing the association between the standard (or its derivative) and outcomes expressed in annual student and school questionnaires and compiled in databases of the Association of American Medical Colleges and the American Medical Association. The result was a mixed bag, confounded by the absence of specific dimensions of many accreditation standards (independent variables) and the lack of discriminating measures of outcome (dependent variables). Nevertheless, the LCME's accreditation standards are believed to be important by those most affected by them. And beyond validating that medical accreditation is guided by relevant standards for teaching, learning, and evaluation, the results of this study point to ways by which the process can be made more precise and useful.

The meaning and application of medical accreditation standards.

The authors examined the operational meaning of the 48 items that state the accreditation standards for teaching, learning, and evaluation in medical school, and determined the extent to which these standards were applied by schools and by on-site evaluators for 59 programs surveyed by the Liaison Committee on Medical Education (LCME) in 1994-1996. In this study, "application" meant that evidence was offered, not necessarily that it proved compliance with the standard. The data sources employed were the medical education databases and self-studies prepared by schools undergoing accreditation surveys, and the reports prepared by ad hoc teams of surveyors. The frequency with which evidence of compliance was offered by the schools and cited by evaluators was determined for each of the 48 accreditation requirements. In addition, the authors compared the patterns of surveyors' concerns about noncompliance at schools surveyed during 1984-1986 and at those visited during 1994-1996. In 1994-1996, schools addressed 42 of the 48 accreditation requirements in 90% of instances of more. The areas of particularly low attention dealt with the definition and communication of educational objectives (47% of schools provided evidence); faculty authority and control of academic programs in clinical affiliates (12%); and the faculty's commitment to being effective teachers and their understanding of pedagogy, curricular design, and methods of evaluation (8%). Survey teams, in contrast, accounted in their reports for only 26 (55%) of the standards during the same time period. Among those least frequently addressed were the definition and communication of educational objectives by schools (accounted for in 59% of the reports); assessment of students' problem-solving ability (51%); comparability of educational experiences and student evaluation across dispersed teaching sites (49%); faculty understanding of pedagogy, curriculum construction, and the evaluation of students (8%); faculty authority and control of academic programs in clinical affiliates (7%); and knowledge of the administration and faculty about methods for measuring student performance (2%). Over the past decade, surveyors' most frequently cited concerns about schools' noncompliance with accreditation standards dealt with student counseling and health services, institutional financial and space/facilities resources, faculty issues, and vacant decanal and department chair positions. Next in order were concerns about various aspects of the educational program leading to the MD degree. Among the high-profile concerns about the educational program that increased significantly over the decade were those about curriculum design, management, and evaluation; primary/ambulatory care experiences; and student advancement policies and due-process issues. Schools paid high attention to most of the 48 standards, in large part because they were prompted by the formatting of the medical education database and self-study guidelines. In those instances of lesser attention, the fault lies as much or more with ambiguities in the construction and meaning of the standards as with institutional laxity. The surveyors' inattention to accreditation standards is more troubling. In some cases it can be attributed to uncertainties about the meaning of the requirements and the quantities that need to be audited; or surveyors may be comfortable reaching a "substantial compliance" threshold without adducing all the evidence. The authors argue that many of the standards given scant attention on surveys are important to educational program development and quality control. The LCME will need to consider whether more prominent definition and highlighting should be given to neglected standards, or whether some of the requirements are at the margin as quality indicators. A planned survey of communities of interest-educators, practitioners, students, graduates, and residency program directors, among others-may help confirm

The objectives of medical education: reflections in the accreditation looking glass.

The authors explored the extent to which medical schools have established institutional and departmentalized educational objectives, by examining the accreditation databases, institutional self-studies, and site visit reports of 59 schools surveyed by the Liaison Committee on medical Education (LCME) in 1994-1996. In this study, the individual school was the unit of analysis, and the dependent variables were statements--in outcomes' terms--of institutional and departmental learning objectives. Objectives were classified as "robust" when they were expressed as measurable learning outcomes in the domains of knowledge and skills and behaviors. Departmental objectives were assessed separately for core basic science courses and clinical clerkships. A number of independent variables were studied for their association with the elaboration of outcomes-based educational objectives: centralized management of the curriculum, curricular reform, an office of education, an institutionalized faculty development program, problem-based learning, interdisciplinary teaching, systematized assessment of educational outcomes, and structured clinical skills assessment. Of the 59 schools, 15 (25%) had robust institutional objectives that identified items of measurable knowledge, skills, and behavior, while 44 (75%) had modest objectives, usually descriptions of the purpose of the curriculum, global statements about the knowledge and behavior qualities that students should acquire, and the subjects that instructors intended to tech. Schools with well-articulated learning outcomes were more likely to have centralized management of the curriculum, a record of curricular reform, a program of problem-based learning, rich basic science course and clerkship objectives, and structured assessment of students' clinical skills. Well-stated and diverse educational objectives were found for basic science courses in only 13 of the 59 schools (22%). Thirty of the schools (51%) had clerkships with robust learning objectives, more commonly in the generalist specialties of pediatrics, family medicine, and internal medicine. Clerkships with stout objectives more often employed structured assessments of students' clinical skills. Schools that paid attention to the explication of learning objectives were more likely to link specific institutional and departmentalized outcomes with methods of evaluating students, in turn guiding the content and methods of instruction. Schools with vapid objectives attracted 40% more accreditation citations for shortcomings in curricular management, course and clerkship quality, and the evaluation of student achievement, especially in the clinical skills domain. In the future, accreditors should focus more closely on this association.

The influence of accreditation on educational change in U.S. medical schools.

The authors examined the influence of accreditation on educational change and reform in U.S. medical schools in the past decade, by reviewing the survey databases and site visit reports of 90 schools that had comprehensive accreditation surveys by the Liaison Committee on Medical Education (LCME) between July 1992 and June 1997. In this study, substantive change was defined as centralizing the design and management of the curriculum, as well as one or more of the following reforms: integrating basic and clinical science instruction and/or conversion to interdisciplinary courses; implementing methods of active, small-group, independent, and hypothesis-based learning; and substantially increasing students' exposure to ambulatory and primary care. Accreditation reports were reviewed to determine the extent to which the LCME previously had admonished schools for shortcomings in their educational programs and advised curricular changes. Notice was taken of grant support by national foundations promoting educational reform, in relation both to the correction of accreditation deficiencies and to curricular reform undertaken by schools on their own initiative. The study also scrutinized the evolution of accreditation standards promoting educational reform, and the LCME's support of initiatives for shortening the period of medical education and promoting performance-based teaching and the assessment of clinical skills. On entering the 1990s, the LCME toughened the standards for design and management of the medical curriculum and for the evaluation of educational program effectiveness that schools must conduct. The greater assessment rigor identified educational shortcomings in 61 of 90 medical schools coming up for accreditation surveys during 1992-1997. On those occasions, 34 of the 61 schools had instituted reforms or were on the verge of doing so. Twenty-five of the schools carrying out reforms (73%) had received major foundation grants, compared with ten of the 27 schools (37%) that had accomplished little. Fifteen schools that had not been reproached earlier were found on the 1992-1997 surveys to have undertaken substantial innovation on their own initiative, five with the help of major foundation awards. The study also shows that a number of schools implemented parts of more sweeping reforms with the help of smaller foundation grants for more discrete purposes. In some instances, it has not been possible to differentiate the influence of the LCME as a force for educational reform from the incentives for change created by national foundations. Overall, the LCME, through its standards and assessment practices, and in synergy with schools and kindred agencies promoting change, is now on the leading edge of improved education and evaluation in the nation's medical schools.

On rising medical student debt: in for a penny, in for a pound.

Using national databases of the Association of American Medical Colleges, the authors have examined reasons for the rising indebtedness of U.S. medical students, looking across the past decade at the influence of tuition and fees (tuition-fees) alone and the total costs of attending school, the effects of the changing demographics of medical school enrollments and lengthened graduation times, the relationship between the availability of school-funded scholarships and the amount of student loan disbursements, the pattern of student financial aid, and the reliance on borrowing to cover the costs of medical education. In constant dollars, the average indebtedness of students graduating from public schools increased 59.2% between 1985 and 1995, and that for graduates of private schools increased 64.2%. The fraction of graduates bringing debt with them when they entered medical school declined from 42.1% in 1985 to 33.6% in 1995. Premedical debt as a fraction of total debt declined at public schools from 9% in 1985 to 7% in 1995, and at private schools from 7.8% in 1985 to 5.9% in 1995. For public schools, tuition-fees increased 60.1% between 1985 and 1995, and average medical school debt increased 60.9%; for private schools, tuition-fees increased 30.1% over that period, while average medical school debt increased 66.2%. On average, public school graduates accrued debt greater than their four-year tuition-fee payments, while the average debt accrued by private school graduates was less than tuition-fee amounts. In 1995, graduates of public schools had debt accumulations representing 62% of the average total cost of attendance (tuition, fees, books, supplies, equipment, and living expenses), and the indebtedness of private school graduates was 55% of the average total cost, findings suggesting that total costs were the stronger driver of the amounts borrowed. On a national scale, the influences on medical school debt of longer graduation times, the growing number of women students, greater racial-ethnic diversity, and the admission of more older students age were negligible or small. The average parental income, adjusted to constant dollars, actually increased between 1985 and 1995. For public schools, the aggregate amounts of student aid have climbed at a steeper rate than schools' tuition-fee revenues during the past decade. For public schools, tuition-fee revenues rose 66.7% between 1985 and 1995, while the amount of loans to students at public schools increased 92.7%. For private schools, tuition-fee revenues went up 36.5%, and the amount of loans to students rose 57.9% during the same period. Federal Stafford Loans represented the major financing source, increasing from 71.5% of public schools' tuition-fee revenue in 1985 to 92.2% in 1995, and from 23% of private schools' tuition-fee revenue in 1985 to 38% in 1995. Over the decade, scholarship support kept pace with tuition-fee increases at public schools, but lagged behind the increases at private schools. The recent escalation of student debt has coincided with the lifting of the federal loan borrowing limits under the Higher Education Act. In parallel, entering medical students have declared their intentions to rely more heavily on loans as a means of financing. These findings, although based on national data and trends, provide a framework for exploration of the factors affecting educational costs and financing at individual medical schools. The importance of doing so is mounting, as students may be throwing caution to the winds in the more favorable climate for borrowing, ignoring indicators of changing practice opportunities and incomes ahead.

Determinants of the generalist career intentions of 1995 graduating medical students.

Using national databases of the Association of American Medical College, the authors employed logistic regression analysis to show the relative predictive influences of selected demographic, structural, attitudinal, and educational variables on the specialty careers choices of 1995 U.S. medical school graduates. Plans to pursue certification in family practice or an unspecified generalist career could be predicted with moderate success, while choices of general internal medicine and general pediatrics could not. The intentions of the 1995 graduates to pursue generalist specialty, were significantly associated with demographic factors such as female gender, older student age, and rural hometown; early interest in the generalist specialties; attitudes favoring helping people over seeking opportunities for leadership, intellectual challenge, or research; the presence of a department of family medicine in the medical school; and ambulatory care experiences in the third and fourth years. In the multiple-regression models used in this study, a number of factors widely touted as important to the cultivation of generalism were not significant predictors of generalist decisions; an institutional mission statement expressly addressing the cultivation of generalist careers; giving admission preferences to applicants who vowed an interest in generalism; public (versus private) school sponsorship; discrete organization units for general internal medical or general pediatrics; the proportion of institutional faculty in the general specialty of medicine and pediatrics; the level of educational debt; the students; clinical experiences in the first and second years of medical school. The authors acknowledge the danger of inferring causal relationships from analyses of this kind, and described how the power of previous associations--e.g., that between a required third-year clerkship in family medicine and graduates' family practice career choices--may be weakened when the independent variable spreads across institutional cultures that at present are less conductive to primary care. The findings of this analysis add to the evidence that generalist career intentions are largely carried on the tide of students' interests and experiences in family medicine and ambulatory primary care. In terms of the predictive values of the input variable in this study, career decisions for the other two generalist specialties--general internal medicine and general pediatrics--were essentially a crapshoot, either because the tactics to promote interest in these fields were ineffective (or confounded), or because the efforts were underdeveloped. Moreover, the statistical models of this study employed quantifiable variables that can be discerned and manipulated to guide the result, whereas medical students tend to identify less tangible elements as more powerful factors influencing their career choices. The results sharpen the strategic focus, but must be combined with those of other, descriptive analysis for a more complete understanding of graduating students' career decisions.

Specialty intentions of 1995 U.S. medical school graduates and patterns of generalist career choice and decision making.

The authors report on the specialty intentions that graduating students declared on the 1995 AAMC Medical School Graduation Questionnaire (GQ) and compare the pattern of career choices in 1995 with that in 1992. Family practice was the leading choice of graduates in 1995, followed by internal medicine subspecialties and general internal medicine. These choices represented significant gains over those made in these specialties in 1992 and were at the expense of declines in the interest of 1995 graduates for internal medicine specialties, radiology, anesthesiology, obstetrics-gynecology subspecialties, and some other fields. In 1992, 14.6% of graduating students declared plans to pursue careers in one of the generalist specialties; in 1995, 27.6% declared such plans. In 1992, no school graduated 50% or more students with generalist intentions, and only one school reached 40%; in 1995, five schools graduated more than 50%, and another 15 graduated more than 40% who favored generalist careers. Medical schools with significant GQ response rates (110 out of 125) were aggregated by level of generalist production (top 25%, middle 50%, and bottom 25%) according to the percentages of their 1995 graduates selecting careers in the individual generalist specialties of family practice, general internal medicine, and general pediatrics, and in these generalist specialties in toto. Within these groups, the linking of GQ responses to declarations given by the same students on the Matriculating Student Questionnaire (MSQ) made it possible to determine the extent to which graduates' specialty choices represented early interests that were retained or interests acquired later during medical school.(ABSTRACT TRUNCATED AT 250 WORDS)

Medical students' career indecision and specialty rejection: roads not taken.

The authors used data from the AAMC Matriculating Student Questionnaire and Medical School Graduation Questionnaire to ascertain how closely the specialty or subspecialty choices of the 1991 and 1994 graduates of U.S. medical schools matched the preferences they had declared when they were matriculated; the extent to which these students strongly considered and then rejected choices that arose during medical school; and the graduation choices of the substantial number of students in both cohorts who were undecided about their careers when they entered medical school. Approximately 80% of the graduates in both classes rejected the specialty intentions they had declared when they began medical school. However, matriculation interests in the generalist specialties--family practice, general pediatrics, and general internal medicine--were more enduring for the 1994 cohort, while interests in the medical, surgical, and support specialties were less so. Large percentages of the 1991 and 1994 cohorts were undecided about their careers at matriculation (20.8% and 26.2%, respectively), and nearly the same proportions remained undecided at graduation. However, more of the graduates in the 1994 cohort who had initially been undecided reached decisions favoring one of the generalist specialties than was true for the 1991 cohort. Nearly half the 1994 graduates had strongly considered and then rejected an alternative to their matriculation interest that arose during medical school. Within the generalist specialties, both early and later interests in family practice were more durable than were those in general pediatrics and general internal medicine: for every student who retreated from tentative interest in family practice, another student's interest was reinforced or kindled.(ABSTRACT TRUNCATED AT 250 WORDS)

The research career interests of graduating medical students.

In this baseline study, the authors analyze in detail many of the factors that influenced the research career intentions of the 1994 U.S. graduates of MD-only programs. Studies of the research interests of the nation's medical school graduates are important because MD-PhD programs do not produce sufficient numbers of physician-scientists, and the remainder must come from the regular population of medical graduates. Data on school characteristics and medical students' demographics, research career intentions, and educational experiences were derived from the AAMC's Institutional Profile System (IPS), Student Application and Information Management System (SAIMS), Matriculating Student Questionnaire (MSQ), and Medical School Graduation Questionnaire (GQ). The 1994 GQ was used as the index instrument to make the correlations reported in this article. A number of findings emerged concerning the 1994 graduates. A greater percentage of these students who began medical school with strong research career intentions and maintained these intentions had entered private medical schools. The lower rate of research interest amongst the students enrolled in public medical schools was compounded by the significantly greater loss of earlier research intentions of those in public schools compared with those in private schools.(ABSTRACT TRUNCATED AT 250 WORDS)

The decline and rise of the medical school applicant pool.

The authors characterize the demographic changes that transpired with the decline and rise of the medical school applicant pool over the past decade, and describe the variations in academic antecedents, attrition, and graduation rates of students matriculated during that time. Data over the ten-year cycle, derived from the AAMC's Student and Applicant Information Management System (SAIMS), were examined in the context of published education and employment statistics. The contraction and expansion of the applicant pool were related to changes in the number and pattern of undergraduate majors and to changes in the employment conditions for college-educated youth. Furthermore, a significant part of the variations in size of the applicant pool is an artifact of changes in the number of repeat applications. Matriculants' pre-medical grades and MCAT scores dropped slightly during the period of applicant decline, and rebounded as admission committees were able to exercise greater selection when the pool expanded. The attrition of medical students rose and fell during this time, but the changes were small and of little discrete influence on graduation rates during the period. The downturn and rebound in applications over the past decade appear to be more related to cycles in the employment market for college graduates than to applicants' perceptions of unfavorable/favorable conditions in medical education and practice.

The longer road to medical school graduation.

The authors followed the academic progress and graduation rates of students matriculating in U.S. medical schools each year between 1976 and 1988. The four-year graduation rates of medical students in MD programs declined from 91.4% for students who matriculated in 1976 to 81.2% for 1988 matriculants. During the same period, the percentage graduating in five years increased from 5.5% to 10.6%, while those still enrolled or on leaves of absence after five years rose from 1.9% to 4.1%. The lengthening of undergraduate medical education is associated with students' spending extra time on special studies or research, but to a greater extent graduation has been delayed for remediation of academic difficulty or slowing of the pace of education to overcome handicaps in academic preparation and learning skills. Underrepresented minority students, particularly black Americans, have lower four-year graduation rates than majority students, and the rate has fallen steadily across successive classes matriculating between 1976 and 1988. The longer periods of undergraduate medical education since 1984, however, have been accompanied by a reversal of the growing attrition rate for black-American students, probably the result of more appropriate pacing of education over the first year or two and the wider availability of student academic support. The longer road to graduation also accumulates greater educational costs, contributing to the escalation in student indebtedness that has been out of proportion to increases in tuition and fees.(ABSTRACT TRUNCATED AT 250 WORDS)

Factors influencing the specialty choices of 1993 medical school graduates.

The authors describe how the 8,128 medical school seniors who completed the 1993 AAMC Medical School Graduation Questionnaire (GQ) rated the influence that each of 36 factors (such as interest in helping people) had on their specialty decisions, using a 0 to 4 Likert-type scale where 4 indicated a major influence on specialty choice. The list of factors and their mean ratings are presented to show those of greatest influence (e.g., type of patient problems encountered, rated 3.44), of moderate influence (e.g., type of patients encountered; 2.75), and of limited influence (e.g., length of residency; 1.18). Factors' ratings also are compared across specialty groups. While the seven most influential factors were given about the same weight by the seniors regardless of their specialty choice, there were significant differences in the ratings of other factors across specialties (e.g., patient contact factors were rated higher by future generalists than by graduates favoring other specialties). The GQ respondents' ratings of factors over time are compared for 1991, 1992, and 1993. Last, those respondents who had changed their minds about a specialty were asked to rate the 36 factors according to how strongly they had influenced the rejection of the previous specialty choice; most of the factors were rated as having been of minor influence. As a general rule, income prospects and educational debt were not strong influences, either on specialty choice or rejection. The authors state that the influences of the factors they have reported are those that set the initial direction of graduates' residency training.(ABSTRACT TRUNCATED AT 250 WORDS)