Offices of research in medical education: accomplishments and added value contributions.

PURPOSE: Offices of research in medical education have been in existence since the 1st one was begun by Hale Hamm at Case Western Reserve in 1958. There are now 61 medical schools in North America that have a formal office of medical education (OME) and are part of the Society of Directors of Research in Medical Education (SDRME). The purpose of this study was to report how SDRME and OMEs have contributed to the research in medical education (RIME) efforts. SUMMARY: This study reports specific initiatives that have been begun by the SDRME culled from historical documents and business meeting minutes and profiles results of the 2000 biannual membership survey that describe what the individual units are doing in support of their institutions. As an organization, SDRME sponsors literature reviews, 5 of which have been published in peer-reviewed journals. It has also collaborated with the American Medical Association, the National Institutes of Health Bureau of the Health Professions (NIH BHP), and the Association of American Medical Colleges in offering conferences and workshops. SDRME worked with the Macy Foundation on a grant initiative which by November 1993 had established 6 consortia to develop regional centers on clinical competence and with the NIH BHP in the development of a grants program that established centers for RIME. Results from the 2000 membership survey indicate that individual units are heavily tied into evaluation efforts at both the institutional and national levels. Research efforts being engaged by these units have led to a reasonably high, but variable, level of publication activity. The current wave of new curricula being implemented in medical education centers are being heavily supported by these units, with all units being involved in curriculum planning and administration. CONCLUSIONS: SDRME has been effective in promoting RIME through sponsoring literature reviews and collaborating with various national organizations. OME units have been effective in serving the needs of their institutions as well as contributing to national research efforts.

Toward setting a research agenda for systematic reviews of evidence of the effects of medical education.

PURPOSE: To provide an update on, and a preliminary research agenda for, best evidence medical education (BEME). SUMMARY: Efforts related to evidence-based medical education are summarized briefly, including BEME, the newly formed Campbell Collaboration, and the Cochrane Collaboration's Effective Practice and Organization of Care review group. A list of topics and priorities for which evidence of effectiveness in medical education should be systematically reviewed is provided based on the results of a session at the July 2000 annual meeting of the Society of Directors of Research in Medical Education. The highest ranked topics clustered around four major conceptual areas: (a) curricular design, (b) learning and instructional methods, (c) testing and assessment, and (d) outcomes. CONCLUSIONS: BEME is gaining momentum with growing numbers of people becoming involved as well as an increased number of pertinent workshops, publications, and Web sites. The work of creating pertinent systematic reviews of the medical education literature is at hand.

Journal publications in radiologic education: a review of the literature, 1987-1997.

RATIONALE AND OBJECTIVES: This study was designed to analyze articles on radiologic education quantitatively and qualitatively, comparing those published in 1987-1997 with those published in 1966-1986. MATERIALS AND METHODS: An initial literature search used four major databases to identify and retrieve articles related to radiologic education. Additional articles were identified through manual cross-checking of references from the original articles. All articles were reviewed by two radiologists as to type of article (editorial, expository, survey, correlational, or experimental, including preexperimental, quasi-experimental, or true experimental), statistics used (inferential or descriptive), educational emphasis (medical student, resident, postgraduate, or other), and topic of article (philosophical or political, technology, program evaluation, program description, examinations, or career decisions). Interrater agreement was estimated by means of the kappa statistic. A chi2 test for independence was used to assess whether the relative distribution of articles was similar for the two periods. RESULTS: More articles per year were published in 1987-1997 (n = 12.6, P < .01) compared with 1966-1986 (n = 9.2). Articles pertinent to radiologic resident education predominated (50.7% vs 29.9% in the prior study, P < .01). In both periods, most articles were expository (37.7% vs 49.5%), and the most common topic was program description (34% vs 35%). Editorials decreased from 35.5% to 18.1%. Experimental studies accounted for 12.3%, increased from 8.7%. The fastest-growing topic of study was technology (30.4% vs 17.5%, P < .01). CONCLUSION: The increased number of articles addressing radiologic education is encouraging. Although the percentage of experimental studies increased slightly in this period, there is still little empirical research in radiologic education.

Challenges in using rater judgements in medical education.

Changes in the healthcare environment are putting increasing pressure on medical schools to make faculty accountable and to document the quality of the medical education they provide. Faculty's ratings of students' performances and students' ratings of faculty's teaching are important elements in these efforts to document educational quality. This article discusses selected research related to factors affecting raters' judgements, analyses how changes in the health care environment are influencing such judgements, offers some suggestions to moderate some of the effects and links these influences to the system that upholds professional standards. Ratings are known to have a positive bias (generosity error), provide limited discrimination and often fail to document serious deficits. The potential sources of these problems relate to the mechanics of the rating task, the system used to obtain ratings and factors affecting rater judgement. As managed care demands reduce the time faculty have for teaching, as system-wide disincentives to provide negative ratings proliferate and as social engineering challenges, such as the Americans with Disabilities Act, impose differential standards for students, the natural tendency to avoid giving negative ratings becomes even harder to resist. Ultimately, these forces compromise the capability of faculty to uphold the standards of the profession. The author calls for a national effort to stem the erosion of those standards.

Problem-based learning in ophthalmology: a pilot program for curricular renewal.

OBJECTIVES: To gain experience with problem-based learning as a demonstration project in a medical school's curriculum renewal effort and determine if using a single facilitator to circulate among the small groups would yield positive results. DESIGN: We developed 16 cases around 4 ophthalmic problems that were used in 3-hour small-group sessions during the Introduction to Clinical Medicine semester of the second-year curriculum. A single faculty member facilitated the small groups of 4 students each that were created by self-division at each of 5 sessions. SETTING: A state-supported large Midwestern medical school. PARTICIPANTS: All students (N = 75) enrolled in the Introduction to Clinical Medicine course prior to their standard introductory ophthalmology lectures. MAIN OUTCOME MEASURES: A 5-item pretest, related to each of that day's clinical problems, was administered at the beginning and again at the end of the session as a posttest. A satisfaction questionnaire with Likert-type questions was also completed by the students at the close of the session. RESULTS: Knowledge scores showed statistically significant gains with a mean of 1.7 points. Student satisfaction was very positive--85% stated that they learned more than they would have in the traditional format and 93% agreed that they enjoyed the problem-based learning format. CONCLUSIONS: A single facilitator successfully managed small groups of students in a modified problem-based learning format that produced significant knowledge gains and high student satisfaction. This positive experience was one of the factors that led to adoption of problem-based learning into the curriculum.

Teaching a screening musculoskeletal examination: a randomized, controlled trial of different instructional methods.

PURPOSE: To develop and test a program to teach a rapid screening musculoskeletal examination. METHOD: In 1995, 191 medical and physician assistant students were randomized to four intervention groups: written materials only (n = 47), written materials and videotape (n = 46), written materials and small-group sessions facilitated by fourth-year medical students (n = 55), and all three methods (n = 43). Assessments, in the form of a written test and standardized patient examinations, were conducted before the interventions (n = 40 randomly selected students), seven to ten days and again three months after the interventions (n = all 191 students), and 16 months after the interventions (n = 103 students). RESULTS: While the four intervention groups' written test scores were approximately equal, their scores on the standardized patient examination differed significantly. The students taught in small groups demonstrated significantly superior examination skills compared with the students taught with written material or videotape at seven to ten days and retained this relative superiority after three and 16 months (p < .0001). CONCLUSION: Small-group instruction with hands-on supervised practice is superior to more passive instructional methods for teaching musculoskeletal examination skills and can be successfully delivered by trained senior medical student facilitators with minimal direct expenditure of faculty time.

Chest radiology case exchange program: a paradigm for resident teaching and independent resident learning.

RATIONALE AND OBJECTIVES: The purpose of this study was to test the effectiveness of resident-prepared, independent learning cases in teaching residents chest radiology. MATERIALS AND METHODS: Three 2nd-year residents (one each from the University of Wisconsin, the Oregon Health Sciences University, and the University of Michigan) prepared four chest radiology teaching cases each (total, 12 cases). Radiology residents from each institution were randomly divided into control (n = 30) and experimental (n = 35) groups. Residents from both groups took a pretest of 36 multiple-choice questions covering the material from the 12 teaching cases. Residents in the experimental group reviewed these cases independently, and both groups took the same test (posttest) immediately after the teaching cases had been reviewed and again 3 months later (final test). RESULTS: Test scores were similar across institutions (P > .05) but differed across time and treatment groups (experimental vs control) (P < .0001). Mean differences in test scores between the experimental and control groups at pretest, posttest, and final test were -0.4, +9.0, +4.0, respectively, demonstrating increased performance at posttesting that was still present (though somewhat attenuated) 3 months later at final testing. CONCLUSION: Independent study of resident-prepared chest radiology teaching cases increases the resident's knowledge for as long as 3 months after instruction.

Medical students and radiology residents: can they learn as effectively with the same educational materials?

RATIONALE AND OBJECTIVES: The purpose of this study was to evaluate the effectiveness of resident-prepared, independent-learning materials for teaching chest radiology to medical students. MATERIALS AND METHODS: Students from three U.S. medical schools enrolled in radiology clerkships between March 1998 and June 1998 were randomly divided into control (n = 27) and experimental (n = 31) groups. The experimental group studied 12 chest radiology independent-learning cases (intervention) used to teach radiology residents in a previous study. Both groups took a 36-item, multiple-choice test (previously used to test radiology residents) on three occasions (before intervention [pretest], 1 day after intervention [posttest], and 2-4 weeks after intervention [final examination]). Student scores were then compared with resident scores. RESULTS: Mean scores were similar across institutions at pretest, but increases at posttest and final examination scores differed across time, school, and group (P < .005). Mean differences in scores between experimental and control groups at pretest, posttest, and 2-4-week final examination were -0.22, 9.79, and 9.14, respectively, demonstrating increased performance at posttesting that remained present (though slightly attenuated) 2-4 weeks later. Comparing performance, residents had mean pretest scores of 19.2 and students of 14.1, a five-point difference attributable to the residents' greater experience. Both residents and students gained approximately nine points at posttest. At final examination, the difference between residents and students was only 1.4 points, suggesting the experimental program (teaching materials) brought students close to the long-term retention shown by residents. CONCLUSION: Independent study of resident-prepared chest radiology teaching cases increased medical student knowledge for at least 2 or 4 weeks after instruction. Although starting at lower knowledge levels, students experienced gains in knowledge comparable to those of residents, suggesting the same materials can be used to teach both students and residents.

The effects of attrition on implementing and evaluating a mental health continuing education program for nursing personnel in long-term care facilities.

BACKGROUND: The effects of attrition on a continuing education program for nursing personnel within long-term care facilities are described. METHODS: Allowing flexible participation can enhance the impact of a continuing education program designed for paraprofessional staff. Increasing the number of nursing staff who experience only a portion of the training has implications for the instructional design and the evaluation. Recommendations to improve accommodation of staff and enhance dissemination of training are offered as well as appropriate evaluation techniques. CONCLUSION: Self-contained short units of instruction allow those with minimal time to benefit from the training. Effective evaluation requires special sampling and measurement strategies. Cross-sectional methods make good use of training resources but usually yield measurable changes only in lower-level program objectives.

Radiology faculty evaluation: bi-institutional implementation of an experimental appraisal instrument.

RATIONALE AND OBJECTIVES: The authors evaluated the reliability and validity of an experimental radiology faculty appraisal instrument. MATERIALS AND METHODS: Residents from the University of Wisconsin (UW) and Indiana University (IU) were asked to use a previously developed behaviorally based, 53-item experimental faculty appraisal instrument. Twenty UW residents evaluated 29 UW faculty members and 37 IU residents evaluated 31 IU faculty members by using the experimental instrument. Residents also evaluated faculty by using their institution's existing appraisal instrument. RESULTS: Correlations between existing and experimental forms were .69 and .87 for UW and IU, respectively. Existing form reliabilities were .89 and .94 and experimental form reliabilities were .98 and .98 for UW and IU, respectively. Experimental form length was reduced to 30 items by eliminating the questions that correlated the least with section scores. Reliabilities of scores on the shortened form were .97 and .98 and correlated .65 and .88 with scores on the longer form for UW and IU, respectively. CONCLUSION: Ratings obtained with the existing forms correlated substantially with the experimental form, attesting to the experimental form's validity. Shortening the experimental form had a minimal effect on the reliability and validity of the data obtained. The behavior-based form was used to rate behaviors that residents believed discriminated between effective and ineffective instructors, enabling an objective and relevant assessment to be made.

Development of a radiology faculty appraisal instrument by using critical incident interviewing.

RATIONALE AND OBJECTIVES: To develop a valid and reliable radiology faculty appraisal instrument based on scientific methods. MATERIALS AND METHODS: Fifteen radiology residents participated in critical incident interviewing. During a 1-hour interview, a resident was asked to describe five incidents each of effective and ineffective faculty behavior. Two investigators independently listened to the tape-recorded interviews, and two different investigators sorted the incidents into broad categories. A faculty appraisal instrument was developed by listing similar incidents under broad categories. A five-point rating scale was applied to each item. Content validity was assessed by resident and faculty critique of the appraisal instrument. RESULTS: A total of 168 incidents of faculty behavior were generated. The frequency with which similar incidents were reported was recorded. The most common behaviors reported were related to staff expertise and teaching. Interjudge reliability was good, as determined by computing K indices of agreement (overall K = 0.59). There was good agreement regarding instrument content validity among residents but not among faculty. CONCLUSION: Residents supported the use of the new appraisal instrument, but further tests of validity and reliability and faculty acceptance of the instrument will determine its usefulness as a tool for monitoring faculty teaching performance and making decisions regarding faculty promotion.

Resident learning and knowledge retention from resident-prepared chest radiology conferences.

RATIONALE AND OBJECTIVES: The authors assessed resident learning and retention of knowledge from resident-prepared chest radiology conferences. MATERIALS AND METHODS: Radiology residents presented five chest conferences to their peers during a 5-month interval; the conferences were modeled on a case presentation format. Tests were given 5 minutes before each conference (pretest) and immediately after each conference (posttest). The tests were readministered as a final examination 6 months later, at which time the residents were asked to evaluate the conference format. RESULTS: Conference attendance ranged from six to 11 residents. Mean posttest scores were statistically significantly higher than mean pretest scores (P < .0001). Six-month retention scores were higher than pretest scores (P < .05) but lower than posttest scores (P < .05). On a scale of 1-6, with 1 representing strongly disagree and 6 strongly agree, residents strongly agreed that the conferences provided an excellent learning experience (mean score, 5.27). CONCLUSION: Resident-prepared conferences are effective for teaching residents chest radiology. Resident testing at 6 months demonstrated retention of knowledge above pretest levels but lower than posttest levels.

Resident conference at the viewbox: an experimental approach.

RATIONALE AND OBJECTIVES: To develop and test an experimental radiology resident conference format designed to incorporate more cases, improve visibility of cases, increase resident participation, decrease resident anxiety, and provide residents with feedback on their oral presentation skills. METHODS: Sixteen radiology residents were paired into eight teams. Each team was given an identical packet of eight chest radiograph cases, assigned a viewbox, and given 16 minutes to review the cases. Each case was then discussed in front of the group, by one resident from a team, such that all teams participated in oral presentation. The conference moderator provided a written handout, discussion of the findings, and diagnosis for each case. Residents anonymously evaluated each other's performance, the experimental conference format, and 15 traditional conferences. RESULTS: Residents subjectively found the experimental conference to be a statistically significant improvement over the traditional conferences in certain areas (P < .05). CONCLUSION: The experimental conference serves as a valuable alternative to the traditional "hot-seat" conference.

Student health policies of U.S. medical schools.

BACKGROUND: Medical students are at risk of exposure to bloodborne pathogens, yet few data are available about U.S. medical schools' policies to protect students. METHOD: A cross-sectional survey of the student affairs deans at the 126 U.S. medical schools was conducted in May 1994. A confidential questionnaire inquired about policies regarding vaccination for hepatitis B virus (HBV), blood and body-fluid exposures, universal precautions training, and health and disability insurance for students. RESULTS: A total of 108 (86%) of the schools participated in the survey. Most (99, 92%) required either HBV vaccination, evidence of immunity, or a signed waiver refusing vaccination. Nearly all (94, 87%) required health insurance, and almost all (101, 94%) offered a plan (at a mean cost of $690 annually), but fewer schools (69, 64%) offered disability insurance. The schools frequently held students responsible for the costs of HBV vaccination (73, 68%), postexposure serologic testing (22, 20%), and treatment of training-related medical problems (43, 40%). CONCLUSION: Most medical schools comply with current recommendations for preventing training-related exposures to bloodborne pathogens, illness, and injury, but students face a substantial financial responsibility for these services at a time when many have large debts. Many schools do not have disability insurance readily available for students. Medical schools should review their student health policies to protect students adequately.

Blood and body fluid exposures during clinical training: relation to knowledge of universal precautions.

To investigate the relation between knowledge of universal precautions and rates of exposure to blood and body fluid during clinical training, a cohort of 155 students was surveyed following training in universal precautions and 18 months later. A total of 127 students (82%) participated; 58 (46%) experienced at least one exposure during the first clinical training year. Knowledge of universal precautions was inversely associated with the frequency of mucous membrane exposures (p .001); an apparent "dose-response" effect was evident (one-way analysis of variance; F = 5.2, p - 0.007). Students are frequently exposed to blood and body fluid during clinical training. Higher levels of retained knowledge about universal precautions are associated with a decreased risk of mucous membrane exposure.

Universal precautions training of preclinical students: impact on knowledge, attitudes, and compliance.

BACKGROUND: Little information exists regarding the impact of universal precautions training programs on preclinical students' knowledge, attitudes, and behavior. METHODS: We developed, implemented, and assessed an educational program in universal precautions for 2nd-year medical and preclinical physician assistant students. Students (n = 170) completed pre- and post-training questionnaires to assess universal precautions knowledge and to evaluate attitudes about their perceived risk for bloodborne pathogen infection, the importance of universal precautions procedures, and their willingness to provide care for human immunodeficiency virus (HIV)-positive or acquired immune deficiency syndrome (AIDS) patients. Phlebotomy, intravenous catheter insertion, and arterial blood gas sampling techniques were demonstrated, practiced, and evaluated during practical training sessions. Outcome measures included changes in pre- and posttraining knowledge scores and attitudes, as well as observed compliance with universal precautions during practical training. RESULTS: Universal precautions knowledge scores increased significantly after training (P < 0.0001). Personal assessments of the risk of developing HIV due to patient care significantly decreased (P < 0.0001) and willingness to provide care for AIDS patients increased (P = 0.004) following training. Importantly, students reported that high expected rates of contact with HIV-positive and other patient groups would not significantly affect their specialty choice. Observed compliance with universal precautions procedures during practical training ranged from 95 to 99% for glove use, 76 to 77% for direct sharps disposal without needle recapping, and 56 to 78% for handwashing after glove removal during phlebotomy and intravenous catheter insertion. CONCLUSIONS: This program is effective in increasing students' knowledge of universal precautions. Training favorably affects students' willingness to care for HIV-positive patients and their assessed risk of developing occupational bloodborne infection.