Does spaced education improve clinical knowledge among Family Medicine residents? A cluster randomized controlled trial.

Spaced education is a learning strategy to improve knowledge acquisition and retention. To date, no robust evidence exists to support the utility of spaced education in the Family Medicine residency. We aimed to test whether alerts to encourage spaced education can improve clinical knowledge as measured by scores on the Canadian Family Medicine certification examination. METHOD: We conducted a cluster randomized controlled trial to empirically and pragmatically test spaced education using two versions of the Family Medicine Study Guide mobile app. 12 residency training programs in Canada agreed to participate. At six intervention sites, we consented 335 of the 654 (51%) eligible residents. Residents in the intervention group were sent alerts through the app to encourage the answering of questions linked to clinical cases. At six control sites, 299 of 586 (51%) residents consented. Residents in the control group received the same app but with no alerts. Incidence rates of case completion between trial arms were compared using repeated measures analysis. We linked residents in both trial arms to their knowledge scores on the certification examination of the College of Family Physicians of Canada. RESULTS: Over 67 weeks, there was no statistically significant difference in the completion of clinical cases by participants. The difference in mean exam scores and the associated confidence interval did not exceed the pre-defined limit of 4 percentage points. CONCLUSION: Further research is recommended before deploying spaced educational interventions in the Family Medicine residency to improve knowledge.

Validación transcultural del Cuestionario de Enseñanza Clínica de Maastricht / Transcultural validation of the Maastricht Clinical Teaching Questionnaire

La evaluación del desempeño del docente clínico permite proporcionar feedback para motivarlo a mejorar sus habilidades docentes. Se realizó la validación transcultural del Cuestionario de Enseñanza Clínica de Maastricht, mediante la guía internacional para la adaptación de los cuestionarios. Se investigó validez de contenido, proceso de respuesta, constructo y confiabilidad. Luego de la adaptación, residentes de dos hospitales universitarios evaluaron a 187 docentes clínicos. La validación del contenido y del proceso de respuesta fue adecuada. En el análisis factorial confirmatorio, todos los índices y criterios se ajustaron bien con el modelo de 5 factores y 14 ítems. El alfa de Cronbach fue de 0,80. El coeficiente G fue > 0,70, con bajos valores de varianza del error absoluto. Fueron necesarias 6 evaluaciones por docente para una evaluación confiable en cada dominio y en el desempeño global. La versión en castellano del Cuestionario es válida y confiable para evaluar a los docentes.

The evaluation of the clinical teacher's performance provides feedback to motivate them to improve their teaching skills. To perform the cross-cultural adaptation of the Maastricht Clinical Teaching Questionnaire, the International Guide for the Adaptation of the Questionnaires was followed. The validity of content, response process, construct and reliability were investigated. After cross-cultural adaption, residents of two University hospitals evaluated 187 clinical teachers. Content and answering process were validated. In the confirmatory factor analysis, all indexes and criteria for a good fit suited the 5 factors and 14 items model. The Cronbach's alpha coefficient was 0.80. The G coefficient was > 0.70, with low variance of the absolute error. Every clinical teacher should receive at least 6 evaluations to achieve a reliable evaluation of every domain and of their global performance. The Spanish version of the questionnaire is a valid, reliable instrument for medical residents to evaluate teachers.

Transcultural validation of the Maastricht Clinical Teaching Questionnaire. / Validación transcultural del Cuestionario de Enseñanza Clínica de Maastricht.

The evaluation of the clinical teacher's performance provides feedback to motivate them to improve their teaching skills. To perform the cross-cultural adaptation of the Maastricht Clinical Teaching Questionnaire, the International Guide for the Adaptation of the Questionnaires was followed. The validity of content, response process, construct and reliability were investigated. After cross-cultural adaption, residents of two University hospitals evaluated 187 clinical teachers. Content and answering process were validated. In the confirmatory factor analysis, all indexes and criteria for a good fit suited the 5 factors and 14 items model. The Cronbach's alpha coefficient was 0.80. The G coefficient was > 0.70, with low variance of the absolute error. Every clinical teacher should receive at least 6 evaluations to achieve a reliable evaluation of every domain and of their global performance. The Spanish version of the questionnaire is a valid, reliable instrument for medical residents to evaluate teachers.

La evaluación del desempeño del docente clínico permite proporcionar feedback para motivarlo a mejorar sus habilidades docentes. Se realizó la validación transcultural del Cuestionario de Enseñanza Clínica de Maastricht, mediante la guía internacional para la adaptación de los cuestionarios. Se investigó validez de contenido, proceso de respuesta, constructo y confiabilidad. Luego de la adaptación, residentes de dos hospitales universitarios evaluaron a 187 docentes clínicos. La validación del contenido y del proceso de respuesta fue adecuada. En el análisis factorial confirmatorio, todos los índices y criterios se ajustaron bien con el modelo de 5 factores y 14 ítems. El alfa de Cronbach fue de 0,80. El coeficiente G fue > 0,70, con bajos valores de varianza del error absoluto. Fueron necesarias 6 evaluaciones por docente para una evaluación confiable en cada dominio y en el desempeño global. La versión en castellano del Cuestionario es válida y confiable para evaluar a los docentes.

Experts' responses in script concordance tests: a response process validity investigation.

CONTEXT: The script concordance test (SCT), designed to measure clinical reasoning in complex cases, has recently been the subject of several critical research studies. Amongst other issues, response process validity evidence remains lacking. We explored the response processes of experts on an SCT scoring panel to better understand their seemingly divergent beliefs about how new clinical data alter the suitability of proposed actions within simulated patient cases. METHODS: A total of 10 Argentine gastroenterologists who served as the expert panel on an existing SCT re-answered 15 cases 9 months after their original panel participation. They then answered questions probing their reasoning and reactions to other experts' perspectives. RESULTS: The experts sometimes noted they would not ordinarily consider the actions proposed for the cases at all (30/150 instances [20%]) or would collect additional data first (54/150 instances [36%]). Even when groups of experts agreed about how new clinical data in a case affected the suitability of a proposed action, there was often disagreement (118/133 instances [89%]) about the suitability of the proposed action before the new clinical data had been introduced. Experts reported confidence in their responses, but showed limited consistency with the responses they had given 9 months earlier (linear weighted kappa = 0.33). Qualitative analyses showed nuanced and complex reasons behind experts' responses, revealing, for example, that experts often considered the unique affordances and constraints of their varying local practice environments when responding. Experts generally found other experts' alternative responses moderately compelling (mean ± standard deviation 2.93 ± 0.80 on a 5-point scale, where 3 = moderately compelling). Experts switched their own preferred responses after seeing others' reasoning in 30 of 150 (20%) instances. CONCLUSIONS: Expert response processes were not consistent with the classical interpretation and use of SCT scores. However, several fruitful and justifiable alternatives for the use of SCT-like methods are proposed, such as to guide assessments for learning.

Admission factors associated with international medical graduate certification success: a collaborative retrospective review of postgraduate medical education programs in Ontario.

BACKGROUND: The failure rate on certification examinations of The College of Family Physicians of Canada (CFPC) and the Royal College of Physicians and Surgeons of Canada (RCPSC) is significantly higher for international medical graduates than for Canadian medical school graduates. The purpose of the current study was to generate evidence that supports or refutes the validity of hypotheses proposed to explain the lower success rates. METHODS: We conducted retrospective analyses of admissions and certification data to determine the factors associated with success of international medical graduate residents on the certification examinations. International medical graduates who entered an Ontario residency program between 2005 and 2012 and had written a certification examination by the time of the analysis (2015) were included in the study. Data available at the time of admission for each resident, including demographic characteristics, previous experiences and previous professional experiences, were collected from each of the 6 Ontario medical schools and matched with certification examination results provided by The CFPC and the RCPSC. We developed logistic regression models to determine the association of each factor with success on the examinations. RESULTS: Data for 900 residents were analyzed. The models revealed resident age to be strongly associated with performance across all examinations. Fluency in English, female sex and the Human Development Index value associated with the country of medical school training had differential associations across the examinations. INTERPRETATION: The findings should contribute to an improved understanding of certification success by international medical graduates, help residency programs identify at-risk residents and underpin the development of specific educational and remedial interventions. In considering the results, it should be kept in mind that some variables are not amenable to changes in selection criteria.

Practice simulated office orals as a predictor of Certification examination performance in family medicine.

OBJECTIVE: To determine if performance on practice simulated office orals (SOOs) conducted during residency training could predict residents' performance on the SOO component of the College of Family Physicians of Canada's (CFPC's) final Certification examination. DESIGN: Prospective cohort study. SETTING: University of Ottawa in Ontario. PARTICIPANTS: Family medicine residents enrolled in the University of Ottawa's Family Medicine Residency program between July 1, 2012, and June 30, 2014, who were eligible to write the CFPC Certification examination in the spring of 2014 and who had participated in all 4 practice SOO examination sessions; 23 residents met these criteria. MAIN OUTCOME MEASURES: Scores on practice SOO sessions during fall 2012, spring 2013, fall 2013, and spring 2014; and the SOO component score on the spring 2014 administration of the CFPC Certification examination. RESULTS: Weighted least squares regression analysis using the 4 practice SOO session scores significantly predicted the final Certification examination SOO score (P < .05), with an adjusted R2 value of 0.29. Additional analysis revealed that the mean scores for the cohort generated at each time point were statistically different from each other (P < .001) and that the relationship over time could be represented by either a linear relationship or a quadratic relationship. A generalizability study generated a relative generalizability coefficient of 0.63. CONCLUSION: Our results confirm the usefulness of practice SOOs as a progress test and demonstrate the feasibility of using them to predict final scores on the SOO component of the CFPC's Certification examination.

Do Weekly Alerts From a Mobile Application Influence Reading During Residency?

BACKGROUND: The benefits of "spaced education" have been documented for residents in highly focused specialties. We found no published studies of spaced education in family medicine. In this study, we report on the feasibility of delivering weekly alerts from a mobile application (app) developed for exam preparation, to increase the reading of clinical information in the family medicine residency. DESIGN: This is a 2-phase mixed methods study. Phase one is a quasi-experimental study of resident reading of information related to priority topics in family medicine. Reading was documented by page views in a noncommercial mobile app. PARTICIPANTS: All incoming first-year residents at two university training programs in Canada. The intervention group received one alert per week to priority topics on the app, beginning in their second month of residency. The control group was given access to the same app, but received no alerts. RESULTS: In this paper, we report the phase one preliminary findings. In the intervention group, 81 of 96 first year residents consented. At the control site, 79 of 85 residents consented. After 100 days, intervention group residents had viewed more pages of clinical information across all 99 priority topics (1,546 versus 900) and per topic (15.7 versus 9.1 pages, P < 0.0003). On average, each increase of one visit to the app following a weekly alert was associated with an increase of 3.2 visits to pages of clinical information in the app. CONCLUSION: A weekly alert delivered via mobile app shows promise with respect to reading in the family medicine residency.

Progress testing in family medicine--A novel use for simulated office oral exams.

BACKGROUND: Simulated office orals (SOOs) are used by the College of Family Physicians of Canada as a method to evaluate family medicine resident readiness for clinical practice. The use of SOOs as a progress test would provide residency programs with useful information to determine resident readiness for challenging the certification exam. The data from a progress test could then be easily manipulated to generate a risk assessment plot. METHODS: During a prospective cohort study conducted at the University of Ottawa, the feasibility of using practice SOO sessions, a structured clinical exam, as a progress test was explored. Twenty-three residents participated in all four practice SOO sessions and their results were entered into a risk assessment plot. RESULTS: Repeated measures analysis of the data using ANOVA demonstrated that the residents' scores at each time point were statistically different from each other, generating an F(3, 66) = 27.52, p < 0.001, η(2) = 0.55.and that the relationship over time was linear with an F(1, 22) = 123.80, p < 0.001, η(2) = 0.85. At the final time point, a risk assessment resulted in no learners mapping to quadrants III or IV. CONCLUSIONS: Our results demonstrate the feasibility of utilizing a SOO exam, a clinical exam, as a progress test. In addition, we propose generating a risk assessment plot, using the data from the Fall 2013 and Spring 2014 practice SOO sessions, as a means of identifying residents at risk. Further studies will be needed to confirm the utility of this analysis. Combined with other measures acquired during in-training evaluation, the utilization of practice SOOs as a progress test will provide program directors with valuable information on resident progression.

The practicum script concordance test: an online continuing professional development format to foster reflection on clinical practice.

INTRODUCTION: Judgment in the face of uncertainty is an important dimension of expertise and clinical competence. However, it is challenging to conceive continuing professional development (CPD) initiatives aimed at helping physicians enhance their clinical judgment skills in ill-defined situations. We present an online script concordance-based CPD program (the Practicum Script Concordance Test, copyright ©2006 by Practicum Foundation), a tool that can be used to support health professionals in the development of their reflective clinical reasoning ability. We describe the rationale and principles and report on the implementation of 2 online programs based on this new CPD initiative. METHOD: The Practicum Script Concordance Test program consists of daily testing and feedback over the course of a year using SCT items. Feedback is both global (eg, health professionals are told their cumulative mean score) and specific (eg, they can view the expert panel's responses together with their justifications for their answers). Participants have the option of contacting a personal tutor, to whom they can send questions. Data regarding feasibility, participation, and acceptability were collected. RESULTS: Initial implementation took place in Mexico where 1901 physicians (1349 paediatricians, 552 cardiologists) were enrolled in Practicum programs. Around 70% of those enrolled pursued the program and were very satisfied with its format and content. The online format was an important factor in the development and maintenance of the programs. Dropouts had issues with the SCT concept and the time required to participate. DISCUSSION: The on-line Practicum Script Concordance Test program was designed to foster expertise development based on practice, reflection and feedback. Although further research is needed to examine its impact physicians' practice and ultimately on patient outcomes, it is an original and promising development in CPD.

Defining competency-based evaluation objectives in family medicine: professionalism.

OBJECTIVE: To develop and describe observable evaluation objectives for assessing competence in professionalism, which are grounded in the experience of practising physicians. DESIGN: Modified nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using an iterative process, the expert group defined a list of observable behaviours that are indicative of professionalism, or not, in the family medicine setting. Themes relate to professional behaviour in family medicine; specific observable behaviours are those that family physicians believe are indicative of professionalism for each theme. MAIN FINDINGS: The expert group identified 12 themes and 140 specific observable behaviours to assist in the observation and discussion of professional behaviour in family medicine workplace settings. CONCLUSION: Competency-based education literature emphasizes the importance of formative evaluation and feedback. Such feedback is particularly challenging in the domain of professionalism because of its personal nature and the potential for emotional reactions. Effective dialogue between learners and teachers begins with clear expectations and reference to descriptions of relevant, specific behaviour. This research has generated a competency-based resource to assist the assessment of professional behaviour in family medicine educational programs.

Clerkship pathway: a factor in certification success for international medical graduates.

OBJECTIVE: To identify factors that help predict success for international medical graduates (IMGs) who train in Canadian residency programs and pass the Canadian certification examinations. DESIGN: A retrospective analysis of 58 variables in the files of IMGs who applied to the Collège des médecins du Québec between 2000 and 2008. SETTING: Quebec. PARTICIPANTS: Eight hundred ten IMGs who applied to the Collège des médecins du Québec through either the "equivalency pathway" (ie, starting training at a residency level) or the "clerkship pathway" (ie, relearning at the level of a medical student in the last 2 years of the MD diploma). MAIN OUTCOME MEASURES: Success factors in achieving certification. Data were analyzed using descriptive statistics and ANOVA (analysis of variance). RESULTS: International medical graduates who chose the "clerkship pathway" had greater success on certification examinations than those who started at the residency level did. CONCLUSION: There are several factors that influence IMGs' success on certification examinations, including integration issues, the acquisition of clinical decision-making skills, and the varied educational backgrounds. These factors perhaps can be better addressed by a regular clerkship pathway, in which IMGs benefit from learner-centred teaching and have more time for reflection on and understanding of the North American approach to medical education. The clerkship pathway is a useful strategy for assuring the integration of IMGs in the North American health care system. A 2-year relearning period in medical school at a clinical clerkship level deserves careful consideration.

Defining competency-based evaluation objectives in family medicine: procedure skills.

OBJECTIVE: To develop evaluation objectives for assessing competence in procedure skills using a key-features approach. This was part of a multiyear project to develop competency-based evaluation objectives for Certification in Family Medicine. DESIGN: Nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using a nominal group technique, the expert group developed the general key features for procedure skills. The expert group also linked the key features to already established skill dimensions in the domain of competence, to the 4 principles of family medicine, and to the CanMEDS roles. MAIN FINDINGS: The general key features were developed after 5 iterations. Ten key features were outlined and were shown to reflect all the essential skill dimensions in the domain of competence for family medicine. The key features were linked to 2 of the 4 principles of family medicine and to 4 of the CanMEDS roles. CONCLUSION: The general key features for procedure skills were developed to assess competence in procedure skills in family medicine.

Effect of field notes on confidence and perceived competence: survey of faculty and residents.

OBJECTIVE: To evaluate the effectiveness of field notes in assessing teachers' confidence and perceived competence, and the effect of field notes on residents' perceptions of their development of competence. DESIGN: A faculty and resident survey completed 5 years after field notes were introduced into the program. SETTING: Five Dalhousie University family medicine sites--Fredericton, Moncton, and Saint John in New Brunswick, and Halifax and Sydney in Nova Scotia. PARTICIPANTS: First- and second-year family medicine residents (as of May 2009) and core family medicine faculty. MAIN OUTCOME MEASURES: Residents' outcome measures included beliefs about the effects of field notes on performance, learning, reflection, clinical skills development, and feedback received. Faculty outcome measures included beliefs about the effect of field notes on guiding feedback, teaching, and reflection on clinical practice. RESULTS: Forty of 88 residents (45.5%) participated. Fifteen of 50 faculty (30.0%) participated, which only permitted a discussion of trends for faculty. Residents believed field note-directed feedback reinforced their performance (81.1%), helped them learn (67.6%), helped them reflect on practice and learning (66.7%), and focused the feedback they received, making it more useful (62.2%) (P < .001 for all); 63.3% believed field note-directed feedback helped with clinical skills development (P < .01). Faculty believed field notes helped to provide more focused (86.7%) and effective feedback (78.6%), improved teaching (75.0%), and encouraged reflection on their own clinical practice (73.3%). CONCLUSION: Most surveyed residents believed field note use improved the feedback they received and helped them to develop competence through improved performance, learning, reflection, and clinical skills development. The trends from faculty information suggested faculty believed field notes were an effective teaching, feedback, and reflection tool.

Defining competency-based evaluation objectives in family medicine: communication skills.

OBJECTIVE: To provide a pragmatic approach to the evaluation of communication skills using observable behaviours, as part of a multiyear project to develop competency-based evaluation objectives for Certification in family medicine. DESIGN: A nominal group technique was used to develop themes and subthemes and to identify positive and negative observable behaviours that demonstrate competence in communication in family medicine. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: The group used the nominal group technique to derive a list of observable behaviours that would constitute a detailed operational definition of competence in communication skills; multiple iterations were used until saturation was achieved. The group met several times a year, and membership remained unchanged during the 4 years in which the work was conducted. The iterative process was undertaken twice--once for communication with patients and once for communication with colleagues. MAIN FINDINGS: Five themes, 5 subthemes, and 106 positive and negative observable behaviours were generated. The subtheme of charting skills was defined using a key-features analysis. CONCLUSION: Communication skills were defined in terms of themes and observable behaviours. These definitions were intended to help assess family physicians' competence at the start of independent practice.

Defining competency-based evaluation objectives in family medicine: key-feature approach.

OBJECTIVE: To develop key features for priority topics previously identified by the College of Family Physicians of Canada that, together with skill dimensions and phases of the clinical encounter, broadly describe competence in family medicine. DESIGN: Modified nominal group methodology, which was used to develop key features for each priority topic through an iterative process. SETTING: The College of Family Physicians of Canada. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian family medicine context with respect to region, sex, language, community type, and experience. METHODS: The group used a modified Delphi process to derive a detailed operational definition of competence, using multiple iterations until consensus was achieved for the items under discussion. The group met 3 to 4 times a year from 2000 to 2007. MAIN FINDINGS: The group analyzed 99 topics and generated 773 key features. There were 2 to 20 (average 7.8) key features per topic; 63% of the key features focused on the diagnostic phase of the clinical encounter. CONCLUSION: This project expands previous descriptions of the process of generating key features for assessment, and removes this process from the context of written examinations. A key-features analysis of topics focuses on higher-order cognitive processes of clinical competence. The project did not define all the skill dimensions of competence to the same degree, but it clearly identified those requiring further definition. This work generates part of a discipline-specific, competency-based definition of family medicine for assessment purposes. It limits the domain for assessment purposes, which is an advantage for the teaching and assessment of learners. A validation study on the content of this work would ensure that it truly reflects competence in family medicine.

Defining competency-based evaluation objectives in family medicine: dimensions of competence and priority topics for assessment.

OBJECTIVE: To develop a definition of competence in family medicine sufficient to guide a review of Certification examinations by the Board of Examiners of the College of Family Physicians of Canada. DESIGN: Delphi analysis of responses to a 4-question postal survey. SETTING: Canadian family practice. PARTICIPANTS: A total of 302 family physicians who have served as examiners for the College of Family Physicians of Canada's Certification examination. METHODS: A survey comprising 4 short-answer questions was mailed to the 302 participating family physicians asking them to list elements that define competence in family medicine among newly certified family physicians beginning independent practice. Two expert groups used a modified Delphi consensus process to analyze responses and generate 2 basic components of this definition of competence: first, the problems that a newly practising family physician should be competent to handle; second, the qualities, behaviour, and skills that characterize competence at the start of independent practice. MAIN FINDINGS: Response rate was 54%; total number of elements among all responses was 5077, for an average 31 per respondent. Of the elements, 2676 were topics or clinical situations to be dealt with; the other 2401 were skills, behaviour patterns, or qualities, without reference to a specific clinical problem. The expert groups identified 6 essential skills, the phases of the clinical encounter, and 99 priority topics as the descriptors used by the respondents. More than 20% of respondents cited 30 of the topics. CONCLUSION: Family physicians define the domain of competence in family medicine in terms of 6 essential skills, the phases of the clinical encounter, and priority topics. This survey represents the first level of definition of evaluation objectives in family medicine. Definition of the interactions among these elements will permit these objectives to become detailed enough to effectively guide assessment.

Examination outcomes for international medical graduates pursuing or completing family medicine residency training in Quebec.

OBJECTIVE: To review the success of international medical graduates (IMGs) who are pursuing or have completed a Quebec residency training program and examinations. DESIGN: We retrospectively reviewed IMGs' success rates on the pre-residency Collège des médecins du Québec medical clinical sciences written examination and objective structured clinical examination, as well as on the post-residency Certification Examination in Family Medicine. SETTING: Quebec. PARTICIPANTS: All IMGs taking their examinations between 2001 and 2008, inclusive, and Canadian and American graduates taking their examinations during this same period. MAIN OUTCOME MEASURES: Success rates for IMGs on the pre-residency and post-residency examinations, compared with success rates for Canadian and American graduates. RESULTS: Success rates on the pre-residency clinical examinations remained below 50% from 2001 to 2008 for IMGs. Similarly, during the same period, the average success rate on the Certification examination was 56.0% for IMGs, compared with 93.5% for Canadian and American medical graduates. CONCLUSION: Despite pre-residency competency screening and in-program orientation and supports, a substantial number of IMGs in Quebec are not passing their Certification examinations. Another study is under way to analyze reasons for some IMGs' lack of success and to find ways to help IMGs complete residency training successfully and pass the Certification examination.

The presence and impact of local item dependence on objective structured clinical examinations scores and the potential use of the polytomous, many-facet Rasch model.

OBJECTIVE: The purpose of this research project was to extend the research on the robustness of the dichotomous Rasch model to violations of the local independence assumption to the polytomous many-facet Rasch model (MFRM). Candidate scores from oral examinations and objective structured clinical examinations (OSCEs) have been shown to contain variance due to rater error/bias. If the MFRM is robust to local item dependence (LID), then the MFRM could theoretically be applied to medical OSCEs. METHODS: Five OSCEs were used in the study: 3 chiropractic licensure OSCEs and 2 nursing licensure OSCEs. Items were assigned to split-halves based on common stimulus. Split-half correlations were compared with Spearman-Brown estimates of reliability based on Cronbach alpha with all items contributing. Two- and 3-facet MFRM analyses were performed, first with individual items contributing and second with station totals contributing. Correlations were estimated between the 2 MFRM estimates. RESULTS: Cronbach alpha estimates with all items contributing were all very high (>.87). Spearman-Brown estimates were all considerably higher than split-half correlations. Correlations between MFRM by items and by stations were all very high (>.993). CONCLUSIONS: The research project provided evidence that OSCEs violate the local item independence assumption. The project also showed that the MFRM is quite robust to such violations. The authors recommend that the MFRM be applied to OSCEs by station totals for estimates of candidate ability, and by items for item performance measures and quality control programs.

Effect of a community oriented problem based learning curriculum on quality of primary care delivered by graduates: historical cohort comparison study.

OBJECTIVE: To assess whether the transition from a traditional curriculum to a community oriented problem based learning curriculum at Sherbrooke University is associated with the expected improvements in preventive care and continuity of care without a decline in diagnosis and management of disease. DESIGN: Historical cohort comparison study. SETTING: Sherbrooke University and three traditional medical schools in Quebec, Canada. PARTICIPANTS: 751 doctors from four graduation cohorts (1988-91); three before the transition to community based problem based learning (n = 600) and one after the transition (n = 151). OUTCOME MEASURES: Annual performance in preventive care (mammography screening rate), continuity of care, diagnosis (difference in prescribing rates for specific diseases and relief of symptoms), and management (prescribing rate for contraindicated drugs) assessed using provincial health databases for the first 4-7 years of practice. RESULTS: After transition to a community oriented problem based learning curriculum, graduates of Sherbrooke University showed a statistically significant improvement in mammography screening rates (55 more women screened per 1000, 95% confidence interval 10.6 to 99.3) and continuity of care (3.3% more visits coordinated by the doctor, 0.9% to 5.8%) compared with graduates of a traditional medical curriculum. Indicators of diagnostic and management performance did not show the hypothesised decline. Sherbrooke graduates showed a significant fourfold increase in disease specific prescribing rates compared with prescribing for symptom relief after the transition. CONCLUSION: Transition to a community oriented problem based learning curriculum was associated with significant improvements in preventive care and continuity of care and an improvement in indicators of diagnostic performance.