Identifying the unauthorized use of examination material.

Item disclosure is one of the most serious threats to the validity of high stakes examinations, and identifying examinees that may have had unauthorized access to material is an important step in ensuring the integrity of an examination. A procedure was developed to identify examinees that potentially had unauthorized prior access to examination content. A standardized difference score is created by comparing examinee ability estimates for potentially exposed items to ability estimates for unexposed items. Outliers in this distribution are then flagged for further review. The steps associated with this procedure are described and followed by an example of applying the procedure. In addition, the use of this procedure is supported by the results of a simulation that models the use of unauthorized access to examination material.

A comparison of physician examiners and trained assessors in a high-stakes OSCE setting.

BACKGROUND: The Medical Council of Canada (MCC) administers an objective structured clinical examination for licensure. Traditionally, physician examiners (PE) have evaluated these examinees. Recruitment of physicians is becoming more difficult. Determining if alternate scorers can be used is of increasing importance. METHOD: In 2003, the MCC ran a study using trained assessors (TA) simultaneously with PEs. Four examination centers and three history-taking stations were selected. Health care workers were recruited as the TAs. RESULTS: A 3x2x4 mixed analyses of variance indicated no significant difference between scorers (F1,462=.01, p=.94). There were significant interaction effects, which were, localized to site 1/station 3, site 3/station 2, and site 4/station1. Pass/fail decisions would have misclassified 14.4-25.01% of examinees. CONCLUSION: Trained assessors may be a valid alternative to PE for completing checklists in history-taking stations, but their role in completing global ratings is not supported by this study.

A sequenced OSCE for licensure: administrative issues, results and myths.

In response to stakeholder demands for a more cost-effective clinical examination, the Medical Council of Canada adopted a sequenced format for the OSCE component of its licensure examination. The sequenced OSCE was administered in 1997 at 14 sites and assessed 1,796 test takers. The 10-station screening test had an alpha = 0.66 (M = 61.3%, SD = 4.7%). The fail rate of 1.7% for Canadian first-time takers was the lowest since 1992. Significant savings were realized despite costs incurred by the new format. The logistical problems that were encountered were largely due to the constraints of administering a multi-site OSCE. Although the sequenced format was cost effective and psychometrically acceptable, the response of many test takers and faculty members was negative. Consequently, it is the logistical and political lessons that may be the most generalizable.

Using the Judgments of Physician Examiners in Setting the Standards for a National Multi-center High Stakes OSCE.

In 1994 and 1995, the Medical Council of Canada used an innovative approach to set the pass mark on its large scale, multi-center national OSCE which is designed to assess basic clinical and communication skills in physicians in Canada after 15 months of post-graduate medical training. The goal of this article is to describe the new approach and to present the experience with the method during its first two years of operation. The approach utilizes the global judgments of the physician examiners at each station to identify the candidates with borderline performances. The scores of the candidates whose performances are judged to be borderline are summed for each station, yielding an initial passing score for all stations and then the examination as a whole. The latter score is then adjusted upward one standard error of measurement for the final passing score and is used as one of the criteria to pass the examination. Based on the results to date, the new approach has worked well. The advantages, disadvantages and areas of possible refinement for the approach are reviewed.