Association of a Competency-Based Assessment System With Identification of and Support for Medical Residents in Difficulty.

Importance: Competency-based medical education is now established in health professions training. However, critics stress that there is a lack of published outcomes for competency-based medical education or competency-based assessment tools. Objective: To determine whether competency-based assessment is associated with better identification of and support for residents in difficulty. Design, Setting, and Participants: This cohort study of secondary data from archived files on 458 family medicine residents (2006-2008 and 2010-2016) was conducted between July 5, 2016, and March 2, 2018, using a large, urban family medicine residency program in Canada. Exposures: Introduction of the Competency-Based Achievement System (CBAS). Main Outcomes and Measures: Proportion of residents (1) with at least 1 performance or professionalism flag, (2) receiving flags on multiple distinct rotations, (3) classified as in difficulty, and (4) with flags addressed by the residency program. Results: Files from 458 residents were reviewed (pre-CBAS: n = 163; 81 [49.7%] women; 90 [55.2%] aged >30 years; 105 [64.4%] Canadian medical graduates; post-CBAS: n = 295; 144 [48.8%] women; 128 [43.4%] aged >30 years; 243 [82.4%] Canadian medical graduates). A significant reduction in the proportion of residents receiving at least 1 flag during training after CBAS implementation was observed (0.38; 95% CI, 0.377-0.383), as well as a significant decrease in the numbers of distinct rotations during which residents received flags on summative assessments (0.24; 95% CI, 0.237-0.243). There was a decrease in the number of residents in difficulty after CBAS (from 0.13 [95% CI, 0.128-0.132] to 0.17 [95% CI, 0.168-0.172]) depending on the strictness of criteria defining a resident in difficulty. Furthermore, there was a significant increase in narrative documentation that a flag was discussed with the resident between the pre-CBAS and post-CBAS conditions (0.18; 95% CI, 0.178-0.183). Conclusions and Relevance: The CBAS approach to assessment appeared to be associated with better identification of residents in difficulty, facilitating the program's ability to address learners' deficiencies in competence. After implementation of CBAS, residents experiencing challenges were better supported and their deficiencies did not recur on later rotations. A key argument for shifting to competency-based medical education is to change assessment approaches; these findings suggest that competency-based assessment may be useful.

Involving users in the refinement of the competency-based achievement system: an innovative approach to competency-based assessment.

BACKGROUND: Competency-based assessment innovations are being implemented to address concerns about the effectiveness of traditional approaches to medical training and the assessment of competence. AIM: Integrating intended users' perspectives during the piloting and refinement process of an innovation is necessary to ensure the innovation meets users' needs. Failure to do so results in no opportunity for users to influence the innovation, nor for developers to assess why an innovation works or does not work in different contexts. METHODS: A qualitative participatory action research approach was used. Sixteen first-year residents participated in three focus groups and two interviews during piloting. Verbatim transcripts were analyzed individually and then across all transcripts using a constant comparison approach. RESULTS: The analysis revealed three key characteristics related to the impact on the residents' acceptance of the innovation as being a worthwhile investment of time and effort: access to frequent, timely, and specific feedback from preceptors. Findings were used to refine the innovation further. CONCLUSION: This study highlights the necessary conditions for assessing the success of implementation of educational innovations. Reciprocal communication between users and developers is vital. This reflects the approaches recommended in the Ottawa Consensus Statement on research in assessment published in Medical Teacher in March 2011.

Improving the Quality and Capacity of Canada's Health Services: Primary Care Physician Perspectives.

OBJECTIVE: This study set out to identify the perspectives of family physicians (FP/GPs) on the quality and capacity of the services they provide and of the system in which they work, to assess their responsiveness to potential changes and to determine their suggestions for future directions to enhance primary care services. METHODS: Thematic results from prior focus groups with FP/GPs provided direction for a questionnaire sent to practitioners in the urban study area. Seventy-four questions, most using a five-point Likert scale, were grouped into 10 sections: physician issues (based on themes from the focus groups), access to specialist services, workload, scope of practice, primary care physician networks, interdisciplinary collaborative practice, complexities and challenges of family practice, future directions, comments and demographics. RESULTS: Five hundred and eighty-three FP/GPs were surveyed, and 300 responses (52%) were analyzed for frequencies and comparisons using SPSS. In addition to informative responses to the various survey sections noted above, specific physician suggestions for future directions to improve quality and capacity were identified. These included access to specialists/consultants, teamwork/collaborative practice, access to diagnostics, electronic records/technology, time and remuneration. CONCLUSIONS: The identified suggestions by FP/GPs to enhance the quality and capacity of health services contribute to a framework for policy development at national, provincial/territorial and regional levels and can be used as a reference point for the progress of primary care reform initiatives.

The influence of residents training level on their evaluation of clinical teaching faculty.

BACKGROUND: Evaluations by learners are the most common sources of information on teaching. There is some debate about the role of these assessments, but the overall evaluation of faculty by learners was found to be valid and reliable. PURPOSE: The purpose of this study was to examine the relationship between the level of training of family medicine residents and their evaluation of emergency medicine clinical teachers over time. METHODS: A prospective cohort analysis of 6 years of faculty evaluation of 115 teachers was conducted. RESULTS: The 562 residents returned 3,046 valid individual evaluations. There was no significant association between the level of residents' training and the ratings for clinical instruction (p > .05). Resident evaluations did not vary by time of year (p > .05); however, they did significantly differ by year of evaluation, showing that ratings increased over the 6 years of the study (p < .0001). CONCLUSIONS: Neither the residents' level of training nor the timing during the academic year were significant independent predictors of perceived superior teaching performance, although ratings increased over the 6 years of the study.

Resident evaluation of clinical teachers based on teachers' certification.

OBJECTIVE: To examine the influence of emergency medicine (EM) certification of clinical teaching faculty on evaluations provided by residents. METHODS: A prospective cohort analysis was conducted of assessments between July 1994 and July 2000 on residents' evaluations of EM faculty at the University of Alberta, Edmonton, Canada. Resident- and faculty-related variables were entered anonymously using the validated evaluation tool (ER Scale). Credentialing and demographic information on EM faculty was supplemented by data obtained through a nine-question survey. Groups were compared using ANOVA. RESULTS: The 562 residents returned 705 (91%) valid evaluation sheets on 115 EM faculty members. The four domains of didactic teaching, clinical teaching, approachability, and helpfulness were assessed. The majority of ratings were in the very good or superb categories for each domain. Instructors with certification in EM had higher scores in didactic, clinical teaching compared with others, and teachers without national certification scored lower in the helpful and approachable categories (p < 0.05). The route of obtaining EM certifications either through training or practice eligibility did not affect scores. Instructors under the age of 40 years had higher scores than the older age groups in three of four categories (p < 0.05). Instructors working at the teaching sites on a half-time basis received higher scores than those working full-time, and scores varied based on site. Overall, teaching ratings improved over the study period (p < 0.05). CONCLUSIONS: Significant differences exist among instructors in the EM setting that affect their teaching rating scores. National certification in EM, academic track, rotation year, and site are all correlated with better teaching performance.