OSCE Standard Setting: Three Borderline Group Methods.

High-stakes assessments must discriminate between examinees who are sufficiently competent to practice in the health professions and examinees who are not. In these settings, criterion-referenced standard-setting methods are strongly preferred over norm referenced methods. While there are many criterion-referenced options, few are feasible or cost effective for objective structured clinical examinations (OSCEs). The human and financial resources required to organize OSCEs alone are often significant, leaving little in an institution's budget for additional resource-intensive standard-setting methods. The modified borderline group method introduced by Dauphinee et al. for a large-scale, multi-site OSCE is a very feasible option but is not as defensible for smaller scale OSCEs. This study compared the modified borderline group method to two adaptations that address its limitations for smaller scale OSCEs while retaining its benefits, namely feasibility. We evaluated decision accuracy and consistency of calculated cut scores derived from (1) modified, (2) regression-based, and (3) 4-facet Rasch model borderline group methods. Data were from a 12-station OSCE that assessed 112 nurses for entry to practice in a Canadian context. The three cut scores (64-65%) all met acceptable standards of accuracy and consistency; however, the modified borderline group method was the most influenced by lower scores within the borderline group, leading to the lowest cut score. The two adaptations may be more defensible than modified BGM in the context of a smaller (n < 100-150) OSCE.

Augmenting physician examiner scoring in objective structured clinical examinations: including the standardized patient perspective.

In Canada, high stakes objective structured clinical examinations (OSCEs) administered by the Medical Council of Canada have relied exclusively on physician examiners (PEs) for scoring. Prior research has looked at using SPs to replace PEs. This paper reports on two studies that implement and evaluate a standardized patient (SP) scoring tool to augment PE scoring. The unique aspect of this study is that it explores the benefits of combining SP and PE scores. SP focus groups developed rating scales for four dimensions they labelled: Listening, Communication, Empathy/Rapport, and Global Impression. In Study I, 43 SPs from one site of a national PE-scored OSCE rated 60 examinees with the initial SP rating scales. In Study II, 137 SPs used slightly revised rating scales with optional narrative comments to score 275 examinees at two sites. Examinees were blinded to SP scoring and SP ratings did not count. Separate PE and SP scoring was examined using descriptive statistics and correlations. Combinations of SP and PE scoring were assessed using pass-rates, reliability, and decision consistency and accuracy indices. In Study II, SP and PE comments were examined. SPs showed greater variability in their scoring, and rated examinees lower than PEs on common elements, resulting in slightly lower pass rates when combined. There was a moderate tendency for both SPs and PEs to make negative comments for the same examinee but for different reasons. We argue that SPs and PE assess performance from different perspectives, and that combining scores from both augments overall reliability of scores and pass/fail decisions. There is potential to provide examinees with feedback comments from each group.

Core principles of assessment in competency-based medical education.

The meaningful assessment of competence is critical for the implementation of effective competency-based medical education (CBME). Timely ongoing assessments are needed along with comprehensive periodic reviews to ensure that trainees continue to progress. New approaches are needed to optimize the use of multiple assessors and assessments; to synthesize the data collected from multiple assessors and multiple types of assessments; to develop faculty competence in assessment; and to ensure that relationships between the givers and receivers of feedback are appropriate. This paper describes the core principles of assessment for learning and assessment of learning. It addresses several ways to ensure the effectiveness of assessment programs, including using the right combination of assessment methods and conducting careful assessor selection and training. It provides a reconceptualization of the role of psychometrics and articulates the importance of a group process in determining trainees' progress. In addition, it notes that, to reach its potential as a driver in trainee development, quality care, and patient safety, CBME requires effective information management and documentation as well as ongoing consideration of ways to improve the assessment system.

Development and testing of an objective structured clinical exam (OSCE) to assess socio-cultural dimensions of patient safety competency.

BACKGROUND: Patient safety (PS) receives limited attention in health professional curricula. We developed and pilot tested four Objective Structured Clinical Examination (OSCE) stations intended to reflect socio-cultural dimensions in the Canadian Patient Safety Institute's Safety Competency Framework. SETTING AND PARTICIPANTS: 18 third year undergraduate medical and nursing students at a Canadian University. METHODS: OSCE cases were developed by faculty with clinical and PS expertise with assistance from expert facilitators from the Medical Council of Canada. Stations reflect domains in the Safety Competency Framework (ie, managing safety risks, culture of safety, communication). Stations were assessed by two clinical faculty members. Inter-rater reliability was examined using weighted κ values. Additional aspects of reliability and OSCE performance are reported. RESULTS: Assessors exhibited excellent agreement (weighted κ scores ranged from 0.74 to 0.82 for the four OSCE stations). Learners' scores varied across the four stations. Nursing students scored significantly lower (p<0.05) than medical students on three stations (nursing student mean scores=1.9, 1.9 and 2.7; medical student mean scores=2.8, 2.9 and 3.5 for stations 1, 2 and 3, respectively where 1=borderline unsatisfactory, 2=borderline satisfactory and 3=competence demonstrated). 7/18 students (39%) scored below 'borderline satisfactory' on one or more stations. CONCLUSIONS: Results show (1) four OSCE stations evaluating socio-cultural dimensions of PS achieved variation in scores and (2) performance on this OSCE can be evaluated with high reliability, suggesting a single assessor per station would be sufficient. Differences between nursing and medical student performance are interesting; however, it is unclear what factors explain these differences.

A method for identifying extreme OSCE examiners.

BACKGROUND: Performance assessments rely on human judgment, and are vulnerable to rater effects (e.g. leniency or harshness). Making valid inferences from performance ratings for high-stakes decisions requires the management of rater effects. A simple method for detecting extreme raters that does not require sophisticated statistical knowledge or software has been developed as part of the quality assurance process for objective structured clinical examinations (OSCEs). We believe it is applicable to a range of examinations that rely on human raters. METHODS: The method has three steps. First, extreme raters are identified by comparing individual rater means with the mean of all raters. A rater is deemed extreme if their mean was three standard deviations below (hawks) or above (doves) the overall mean. This criterion is adjustable. Second, the distribution of an extreme rater's scores was compared with the overall distribution for the station. This step mitigates a station effect. Third, the cohort of candidates seen by the rater is examined to ensure that any cohort effect is ruled out. RESULTS AND IMPLICATIONS: Of 3000+ raters, fewer than 0.3% have been identified as being extreme using the proposed criteria. Rater performance is being monitored on a regular basis, and the impact of these raters on candidate results will be considered before results are finalised. Extreme raters are contacted by the organisation to review their rating style. If this intervention fails to modify the rater's scoring pattern, the rater is no longer invited back. As more data are collected the organisation will assess them to inform the development of approaches to improve extreme rater performance.

Influence of physicians' management and communication ability on patients' persistence with antihypertensive medication.

BACKGROUND: Less than 75% of people prescribed antihypertensive medication are still using treatment after 6 months. Physicians determine treatment, educate patients, manage side effects, and influence patient knowledge and motivation. Although physician communication ability likely influences persistence, little is known about the importance of medical management skills, even though these abilities can be enhanced through educational and practice interventions. The purpose of this study was to determine whether a physician's medical management and communication ability influence persistence with antihypertensive treatment. METHODS: This was a population-based study of 13,205 hypertensive patients who started antihypertensive medication prescribed by a cohort of 645 physicians entering practice in Quebec, Canada, between 1993 and 2007. Medical Council of Canada licensing examination scores were used to assess medical management and communication ability. Population-based prescription and medical services databases were used to assess starting therapy, treatment changes, comorbidity, and persistence with antihypertensive treatment in the first 6 months. RESULTS: Within 6 months after starting treatment, 2926 patients (22.2%) had discontinued all antihypertensive medication. The risk of nonpersistence was reduced for patients who were treated by physicians with better medical management (odds ratio per 2-SD increase in score, 0.74; 95% confidence interval, 0.63-0.87) and communication (0.88; 0.78-1.00) ability and with early therapy changes (odds ratio, 0.45; 95% confidence interval, 0.37-0.54), more follow-up visits, and nondiuretics as the initial choice of therapy. Medical management ability was responsible for preventing 15.8% (95% confidence interval, 7.5%-23.3%) of nonpersistence. CONCLUSION: Better clinical decision-making and data collection skills and early modifications in therapy improve persistence with antihypertensive therapy.

Quality assurance best practices for simulation-based examinations.

Whether used for formative or summative evaluation, health professions schools, residency programs, continuing medical education programs, and specialty boards using simulation-based assessment must consider quality assurance methods to ensure reliable results. This article addresses the content, training, and administrative protocols that are necessary for findings based on valid, reliable, and fair assessments.

Doctor scores on national qualifying examinations predict quality of care in future practice.

OBJECTIVES: This study aimed to determine if national licensing examinations that measure medical knowledge (QE1) and clinical skills (QE2) predict the quality of care delivered by doctors in future practice. METHODS: Cohorts of doctors who took the Medical Council of Canada Qualifying Examinations Part I (QE1) and Part II (QE2) between 1993 and 1996 and subsequently entered practice in Ontario, Canada (n = 2420) were followed for their first 7-10 years in practice. The 208 of these doctors who were randomly selected for peer assessment of quality of care were studied. Main outcome measures included quality of care (acceptable/unacceptable) as assessed by doctor peer-examiners using a structured chart review and interview. Multivariate logistic regression was used to determine if qualifying examination scores predicted the outcome of the peer assessments while controlling for age, sex, training and specialty, and if the addition of the QE2 scores provided additional prediction of quality of care. RESULTS: Fifteen (7.2%) of the 208 doctors assessed were considered to provide unacceptable quality of care. Doctors in the bottom quartile of QE1 scores had a greater than three-fold increase in the risk of an unacceptable quality-of-care assessment outcome (odds ratio [OR] 3.41, 95% confidence interval [CI] 1.14-10.22). Doctors in the bottom quartile of QE2 scores were also at higher risk of being assessed as providing unacceptable quality of care (OR 4.24, 95% CI 1.32-13.61). However, QE2 results provided no significant improvement in predicting peer assessment results over QE1 results (likelihood ratio test: chi(2) = 3.21, P-value((1 d.f.)) = 0.07). CONCLUSIONS: Doctor scores on qualifying examinations are significant predictors of quality-of-care problems based on regulatory, practice-based peer assessment.

Does the gender of the standardised patient influence candidate performance in an objective structured clinical examination?

CONTEXT: The objective structured clinical examination (OSCE) requires the use of standardised patients (SPs). Recruitment of SPs can be challenging and factors assumed to be neutral may vary between SPs. On stations that are considered gender-neutral, either male or female SPs may be used. This may lead to an increase in measurement error. Prior studies on SP gender have often confounded gender with case. OBJECTIVE: The objective of this study was to assess whether a variation in SP gender on the same case resulted in a systematic difference in student scores. METHODS: At the University of Ottawa, 140 Year 3 medical students participated in a 10-station OSCE. Two physical examination stations were selected for study because they were perceived to be 'gender-neutral'. One station involved the physical examination of the back and the other of the lymphatic system. On each of the study stations, male and female SPs were randomly allocated. RESULTS: There was no difference in mean scores on the back examination station for students with female (6.96/10.00) versus male (7.04/10.00) SPs (P = 0.713). However, scores on the lymphatic system examination station showed a significant difference, favouring students with female (8.30/10.00) versus male (7.41/10.00) SPs (P < 0.001). Results were not dependent on student gender. CONCLUSIONS: The gender of the SP may significantly affect student performance in an undergraduate OSCE in a manner that appears to be unrelated to student gender. It would be prudent to use the same SP gender for the same case, even on seemingly gender-neutral stations.

The use of standardized patient assessments for certification and licensure decisions.

Although standardized patients have been employed for formative assessment for over 40 years, their use in high-stakes medical licensure examinations has been a relatively recent phenomenon. As part of the medical licensure process in the United States and Canada, the clinical skills of medical students, medical school graduates, and residents are evaluated in a simulated clinical environment. All of the evaluations attempt to provide the public with some assurance that the person who achieves a passing score has the knowledge and/or requisite skills to provide safe and effective medical services. Although the various standardized patient-based licensure examinations differ somewhat in terms of purpose, content, and scope, they share many commonalities. More important, given the extensive research that was conducted to support these testing initiatives, combined with their success in promoting educational activities and in identifying individuals with clinical skills deficiencies, they provide a framework for validating new simulation modalities and extending simulation-based assessment into other areas.

Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities.

CONTEXT: Poor patient-physician communication increases the risk of patient complaints and malpractice claims. To address this problem, licensure assessment has been reformed in Canada and the United States, including a national standardized assessment of patient-physician communication and clinical history taking and examination skills. OBJECTIVE: To assess whether patient-physician communication examination scores in the clinical skills examination predicted future complaints in medical practice. DESIGN, SETTING, AND PARTICIPANTS: Cohort study of all 3424 physicians taking the Medical Council of Canada clinical skills examination between 1993 and 1996 who were licensed to practice in Ontario and/or Quebec. Participants were followed up until 2005, including the first 2 to 12 years of practice. MAIN OUTCOME MEASURE: Patient complaints against study physicians that were filed with medical regulatory authorities in Ontario or Quebec and retained after investigation. Multivariate Poisson regression was used to estimate the relationship between complaint rate and scores on the clinical skills examination and traditional written examination. Scores are based on a standardized mean (SD) of 500 (100). RESULTS: Overall, 1116 complaints were filed for 3424 physicians, and 696 complaints were retained after investigation. Of the physicians, 17.1% had at least 1 retained complaint, of which 81.9% were for communication or quality-of-care problems. Patient-physician communication scores for study physicians ranged from 31 to 723 (mean [SD], 510.9 [91.1]). A 2-SD decrease in communication score was associated with 1.17 more retained complaints per 100 physicians per year (relative risk [RR], 1.38; 95% confidence interval [CI], 1.18-1.61) and 1.20 more communication complaints per 100 practice-years (RR, 1.43; 95% CI, 1.15-1.77). After adjusting for the predictive ability of the clinical decision-making score in the traditional written examination, the patient-physician communication score in the clinical skills examination remained significantly predictive of retained complaints (likelihood ratio test, P < .001), with scores in the bottom quartile explaining an additional 9.2% (95% CI, 4.7%-13.1%) of complaints. CONCLUSION: Scores achieved in patient-physician communication and clinical decision making on a national licensing examination predicted complaints to medical regulatory authorities.

Is it feasible to include a technical skill station on a national licensing examination?

The purpose of the current study was to assess the feasibility and validity of including a technical skill station on a national licensing examination. At the 2003 Medical Council of Canada Qualifying Examination, 745 test takers participated in a pilot station assessing the ability to perform a technical procedure. Checklists and rating scales were used for scoring. Validity was investigated by comparing surgery-trained to non-surgery-trained test takers. The mean for the pilot station was 72.4%. The pilot station was moderately correlated to the rest of the examination (item-total correlation .43). The mean score for surgery test takers was higher than for other test takers (P < .001). Inclusion of a technical skill station on a high-stakes examination is feasible, and at many levels, there is evidence of the validity of including this station.

A comparison of physician examiners and trained assessors in a high-stakes OSCE setting.

BACKGROUND: The Medical Council of Canada (MCC) administers an objective structured clinical examination for licensure. Traditionally, physician examiners (PE) have evaluated these examinees. Recruitment of physicians is becoming more difficult. Determining if alternate scorers can be used is of increasing importance. METHOD: In 2003, the MCC ran a study using trained assessors (TA) simultaneously with PEs. Four examination centers and three history-taking stations were selected. Health care workers were recruited as the TAs. RESULTS: A 3x2x4 mixed analyses of variance indicated no significant difference between scorers (F1,462=.01, p=.94). There were significant interaction effects, which were, localized to site 1/station 3, site 3/station 2, and site 4/station1. Pass/fail decisions would have misclassified 14.4-25.01% of examinees. CONCLUSION: Trained assessors may be a valid alternative to PE for completing checklists in history-taking stations, but their role in completing global ratings is not supported by this study.

A sequenced OSCE for licensure: administrative issues, results and myths.

In response to stakeholder demands for a more cost-effective clinical examination, the Medical Council of Canada adopted a sequenced format for the OSCE component of its licensure examination. The sequenced OSCE was administered in 1997 at 14 sites and assessed 1,796 test takers. The 10-station screening test had an alpha = 0.66 (M = 61.3%, SD = 4.7%). The fail rate of 1.7% for Canadian first-time takers was the lowest since 1992. Significant savings were realized despite costs incurred by the new format. The logistical problems that were encountered were largely due to the constraints of administering a multi-site OSCE. Although the sequenced format was cost effective and psychometrically acceptable, the response of many test takers and faculty members was negative. Consequently, it is the logistical and political lessons that may be the most generalizable.

Using the Judgments of Physician Examiners in Setting the Standards for a National Multi-center High Stakes OSCE.

In 1994 and 1995, the Medical Council of Canada used an innovative approach to set the pass mark on its large scale, multi-center national OSCE which is designed to assess basic clinical and communication skills in physicians in Canada after 15 months of post-graduate medical training. The goal of this article is to describe the new approach and to present the experience with the method during its first two years of operation. The approach utilizes the global judgments of the physician examiners at each station to identify the candidates with borderline performances. The scores of the candidates whose performances are judged to be borderline are summed for each station, yielding an initial passing score for all stations and then the examination as a whole. The latter score is then adjusted upward one standard error of measurement for the final passing score and is used as one of the criteria to pass the examination. Based on the results to date, the new approach has worked well. The advantages, disadvantages and areas of possible refinement for the approach are reviewed.