Creative professional activity: an additional platform for promotion of faculty.

Academic promotion has traditionally been based on research and teaching, but faculty members' contributions to the profession may not be fully captured in those dimensions. Faculty members may influence the practice of medicine and improve the care of patients yet not obtain traditional measures of achievement through publications, grants, or teaching awards. With this problem in mind, at the University of Toronto Faculty of Medicine, the promotions committee developed and implemented a promotions criterion called Creative Professional Activity (CPA) to recognize and reward a variety of types of academic endeavors that have a demonstrable impact on medical practice and care. CPA comprises three activities: professional innovation, exemplary practice, and contributions to the development of the discipline. In this article, the authors define CPA, provide illustrative case examples, describe how faculty members document CPA, and report the use of this promotions criterion in the Department of Medicine over the last decade. The challenges of implementing CPA as a promotion criterion are described. CPA is consistent with the Department of Medicine's goal of achieving excellence through original research, education, or creative work that advances the care of patients.

Consistency of standards and stability of pass/fail decisions with examinee-based standard-setting methods in a small-scale objective structured clinical examination.

PROBLEM STATEMENT AND BACKGROUND: The apparent feasibility and the face validity of the examinee-based methods such as the borderline-group methods provide support for their increasing adoption by health profession schools. Before that can occur, however, more information on the quality of the standards produced by these techniques is required. The purpose of the present study was to assess the quality of the standards produced on a small-scale objective structured clinical examination (OSCE) by the borderline-group and contrasting-groups examinee-based standard-setting procedures. METHOD: These two examinee-based criterion-referenced standard-setting procedures were done for an undergraduate fourth-year surgical OSCE and the consistency of the standards and the decisions arising from the standards were assessed. RESULTS: Both techniques provided consistent and realistic standards. CONCLUSIONS: There is sufficient theoretical and empirical evidence to support the use of examinee-based standard-setting techniques in small-scale OSCEs that use expert examiners.

Effect of an undergraduate medical curriculum on students' self-directed learning.

PURPOSE: Lifelong, self-directed learning (SDL) has been identified as an important ability for medical graduates. To evaluate the effect of the University of Toronto Faculty of Medicine's revised undergraduate medical curriculum on students' SDL, a cross-sectional study was conducted. METHOD: A questionnaire package was mailed to 280 randomly selected students, 70 from each of the four years of the curriculum. The package contained the two most widely recognized, extensively used, and validated instruments of SDL (Guglielmino's 58-item Self-Directed Learning Readiness Scale and Oddi's 24-item Continuous Learning Inventory) and Ryan's two-part Self-Assessment Questionnaire. An identification number and sociodemographic questions were included with the questionnaires. Data analysis was completed using chi-square for differences of proportions, analysis of variance for differences between means, and linear regression for trends. RESULTS: A total of 250 (89.3%) complete questionnaire packages were returned. No significant trend in SDL was evident by curriculum year, and similar SDL levels were observed for women and men. However, a significant positive trend in SDL was found with the highest level of premedical education achieved (undergraduate only, masters, or doctoral). Further, students' perceptions concerning the importance of SDL decreased according to year in the curriculum. CONCLUSION: This study found no evidence that students' self-reported SDL is positively influenced by the current undergraduate medical curriculum at the University of Toronto Faculty of Medicine.

The effect of incorporating normative data into a criterion-referenced standard setting in medical education.

PURPOSE: Determining standards for assessing clinical performance is a controversial issue. Purely item-based methods such as the Angoff method often produce unrealistic judgments, even when used by experienced judges. The rather unstudied compromise methods combine absolute and relative judgments and thereby incorporate normative data into criterion-based standard-setting judgments. The purpose of this study was to compare the quality and implications of standards set by three methods used for the Objective Structured Clinical Examination (OSCE). METHOD: Ninety-six judges set standards for 36 surgical year-4 undergraduate OSCE stations. All judges had normative student performance data when judgments were made with the Angoff, Ebel, or Hofstee methods. RESULTS: The Hofstee method gave more realistic cutoff scores and standard errors and better Meskauskas and Jaeger indices than the Angoff and Ebel methods. CONCLUSIONS: Medical educators setting standards for an OSCE should consider adopting the Hofstee method.

A sequenced OSCE for licensure: administrative issues, results and myths.

In response to stakeholder demands for a more cost-effective clinical examination, the Medical Council of Canada adopted a sequenced format for the OSCE component of its licensure examination. The sequenced OSCE was administered in 1997 at 14 sites and assessed 1,796 test takers. The 10-station screening test had an alpha = 0.66 (M = 61.3%, SD = 4.7%). The fail rate of 1.7% for Canadian first-time takers was the lowest since 1992. Significant savings were realized despite costs incurred by the new format. The logistical problems that were encountered were largely due to the constraints of administering a multi-site OSCE. Although the sequenced format was cost effective and psychometrically acceptable, the response of many test takers and faculty members was negative. Consequently, it is the logistical and political lessons that may be the most generalizable.

Evaluating medical grand rounds.

INTRODUCTION: Since January 2000, standard presenter evaluation forms have been made available to grand rounds organizers in the Department of Medicine, University of Toronto. During the 2000-2001 academic year, effort was directed at the accumulation of evidence for the validity of the results generated. METHODS: Two issues were addressed: the integrity or coherence of the form itself and the number of forms or evaluations required to achieve a stable estimate of the construct "presenter effectiveness" for an individual presenter. RESULTS: Positive evidence relating to the integrity of the form is presented and the number of evaluations or ratings required to provide a stable estimate of presenter effectiveness is suggested. DISCUSSION: Most presenters' ratings were distributed in a narrow range. Ranking of individual presentations would require exceptionally high precision. Separation into groups requires less precision. This type of classification appears sufficient to enable planning decisions.

Using the Judgments of Physician Examiners in Setting the Standards for a National Multi-center High Stakes OSCE.

In 1994 and 1995, the Medical Council of Canada used an innovative approach to set the pass mark on its large scale, multi-center national OSCE which is designed to assess basic clinical and communication skills in physicians in Canada after 15 months of post-graduate medical training. The goal of this article is to describe the new approach and to present the experience with the method during its first two years of operation. The approach utilizes the global judgments of the physician examiners at each station to identify the candidates with borderline performances. The scores of the candidates whose performances are judged to be borderline are summed for each station, yielding an initial passing score for all stations and then the examination as a whole. The latter score is then adjusted upward one standard error of measurement for the final passing score and is used as one of the criteria to pass the examination. Based on the results to date, the new approach has worked well. The advantages, disadvantages and areas of possible refinement for the approach are reviewed.