Development, Implementation, and Meta-Evaluation of a National Approach to Programmatic Assessment in Canadian Family Medicine Residency Training.

The growing international adoption of competency-based medical education has created a desire for descriptions of innovative assessment approaches that generate appropriate and sufficient information to allow for informed, defensible decisions about learner progress. In this article, the authors provide an overview of the development and implementation of the approach to programmatic assessment in postgraduate family medicine training programs in Canada, called Continuous Reflective Assessment for Training (CRAFT). CRAFT is a principles-guided, high-level approach to workplace-based assessment that was intentionally designed to be adaptable to local contexts, including size of program, resources available, and structural enablers and barriers. CRAFT has been implemented in all 17 Canadian family medicine residency programs, with each program taking advantage of the high-level nature of the CRAFT guidelines to create bespoke assessment processes and tools appropriate for their local contexts. Similarities and differences in CRAFT implementation between 5 different family medicine residency training programs, representing both English- and French-language programs from both Western and Eastern Canada, are described. Despite the intentional flexibility of the CRAFT guidelines, notable similarities in assessment processes and procedures across the 5 programs were seen. A meta-evaluation of findings from programs that have published evaluation information supports the value of CRAFT as an effective approach to programmatic assessment. While CRAFT is currently in place in family medicine residency programs in Canada, given its adaptability to different contexts as well as promising evaluation data, the CRAFT approach shows promise for application in other training environments.

Association of a Competency-Based Assessment System With Identification of and Support for Medical Residents in Difficulty.

Importance: Competency-based medical education is now established in health professions training. However, critics stress that there is a lack of published outcomes for competency-based medical education or competency-based assessment tools. Objective: To determine whether competency-based assessment is associated with better identification of and support for residents in difficulty. Design, Setting, and Participants: This cohort study of secondary data from archived files on 458 family medicine residents (2006-2008 and 2010-2016) was conducted between July 5, 2016, and March 2, 2018, using a large, urban family medicine residency program in Canada. Exposures: Introduction of the Competency-Based Achievement System (CBAS). Main Outcomes and Measures: Proportion of residents (1) with at least 1 performance or professionalism flag, (2) receiving flags on multiple distinct rotations, (3) classified as in difficulty, and (4) with flags addressed by the residency program. Results: Files from 458 residents were reviewed (pre-CBAS: n = 163; 81 [49.7%] women; 90 [55.2%] aged >30 years; 105 [64.4%] Canadian medical graduates; post-CBAS: n = 295; 144 [48.8%] women; 128 [43.4%] aged >30 years; 243 [82.4%] Canadian medical graduates). A significant reduction in the proportion of residents receiving at least 1 flag during training after CBAS implementation was observed (0.38; 95% CI, 0.377-0.383), as well as a significant decrease in the numbers of distinct rotations during which residents received flags on summative assessments (0.24; 95% CI, 0.237-0.243). There was a decrease in the number of residents in difficulty after CBAS (from 0.13 [95% CI, 0.128-0.132] to 0.17 [95% CI, 0.168-0.172]) depending on the strictness of criteria defining a resident in difficulty. Furthermore, there was a significant increase in narrative documentation that a flag was discussed with the resident between the pre-CBAS and post-CBAS conditions (0.18; 95% CI, 0.178-0.183). Conclusions and Relevance: The CBAS approach to assessment appeared to be associated with better identification of residents in difficulty, facilitating the program's ability to address learners' deficiencies in competence. After implementation of CBAS, residents experiencing challenges were better supported and their deficiencies did not recur on later rotations. A key argument for shifting to competency-based medical education is to change assessment approaches; these findings suggest that competency-based assessment may be useful.

Family physician practice visits arising from the Alberta Physician Achievement Review.

BACKGROUND: Licensed physicians in Alberta are required to participate in the Physician Achievement Review (PAR) program every 5 years, comprising multi-source feedback questionnaires with confidential feedback, and practice visits for a minority of physicians. We wished to identify and classify issues requiring change or improvement from the family practice visits, and the responses to advice. METHODS: Retrospective analysis of narrative practice visit reports data using a mixed methods design to study records of visits to 51 family physicians and general practitioners who participated in PAR during the period 2010 to 2011, and whose ratings in one or more major assessment domains were significantly lower than their peer group. RESULTS: Reports from visits to the practices of family physicians and general practitioners confirmed opportunities for change and improvement, with two main groupings - practice environment and physician performance. For 40/51 physicians (78%) suggested actions were discussed with physicians and changes were confirmed. Areas of particular concern included problems arising from practice isolation and diagnostic conclusions being reached with incomplete clinical evidence. CONCLUSION: This study provides additional evidence for the construct validity of a regulatory authority educational program in which multi-source performance feedback identifies areas for practice quality improvement, and change is encouraged by supplementary contact for selected physicians.

Defining competency-based evaluation objectives in family medicine: professionalism.

OBJECTIVE: To develop and describe observable evaluation objectives for assessing competence in professionalism, which are grounded in the experience of practising physicians. DESIGN: Modified nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using an iterative process, the expert group defined a list of observable behaviours that are indicative of professionalism, or not, in the family medicine setting. Themes relate to professional behaviour in family medicine; specific observable behaviours are those that family physicians believe are indicative of professionalism for each theme. MAIN FINDINGS: The expert group identified 12 themes and 140 specific observable behaviours to assist in the observation and discussion of professional behaviour in family medicine workplace settings. CONCLUSION: Competency-based education literature emphasizes the importance of formative evaluation and feedback. Such feedback is particularly challenging in the domain of professionalism because of its personal nature and the potential for emotional reactions. Effective dialogue between learners and teachers begins with clear expectations and reference to descriptions of relevant, specific behaviour. This research has generated a competency-based resource to assist the assessment of professional behaviour in family medicine educational programs.

Defining competency-based evaluation objectives in family medicine: procedure skills.

OBJECTIVE: To develop evaluation objectives for assessing competence in procedure skills using a key-features approach. This was part of a multiyear project to develop competency-based evaluation objectives for Certification in Family Medicine. DESIGN: Nominal group technique. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: Using a nominal group technique, the expert group developed the general key features for procedure skills. The expert group also linked the key features to already established skill dimensions in the domain of competence, to the 4 principles of family medicine, and to the CanMEDS roles. MAIN FINDINGS: The general key features were developed after 5 iterations. Ten key features were outlined and were shown to reflect all the essential skill dimensions in the domain of competence for family medicine. The key features were linked to 2 of the 4 principles of family medicine and to 4 of the CanMEDS roles. CONCLUSION: The general key features for procedure skills were developed to assess competence in procedure skills in family medicine.

Defining competency-based evaluation objectives in family medicine: communication skills.

OBJECTIVE: To provide a pragmatic approach to the evaluation of communication skills using observable behaviours, as part of a multiyear project to develop competency-based evaluation objectives for Certification in family medicine. DESIGN: A nominal group technique was used to develop themes and subthemes and to identify positive and negative observable behaviours that demonstrate competence in communication in family medicine. SETTING: The College of Family Physicians of Canada in Mississauga, Ont. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian context with respect to region, sex, language, community type, and experience. METHODS: The group used the nominal group technique to derive a list of observable behaviours that would constitute a detailed operational definition of competence in communication skills; multiple iterations were used until saturation was achieved. The group met several times a year, and membership remained unchanged during the 4 years in which the work was conducted. The iterative process was undertaken twice--once for communication with patients and once for communication with colleagues. MAIN FINDINGS: Five themes, 5 subthemes, and 106 positive and negative observable behaviours were generated. The subtheme of charting skills was defined using a key-features analysis. CONCLUSION: Communication skills were defined in terms of themes and observable behaviours. These definitions were intended to help assess family physicians' competence at the start of independent practice.

Involving users in the refinement of the competency-based achievement system: an innovative approach to competency-based assessment.

BACKGROUND: Competency-based assessment innovations are being implemented to address concerns about the effectiveness of traditional approaches to medical training and the assessment of competence. AIM: Integrating intended users' perspectives during the piloting and refinement process of an innovation is necessary to ensure the innovation meets users' needs. Failure to do so results in no opportunity for users to influence the innovation, nor for developers to assess why an innovation works or does not work in different contexts. METHODS: A qualitative participatory action research approach was used. Sixteen first-year residents participated in three focus groups and two interviews during piloting. Verbatim transcripts were analyzed individually and then across all transcripts using a constant comparison approach. RESULTS: The analysis revealed three key characteristics related to the impact on the residents' acceptance of the innovation as being a worthwhile investment of time and effort: access to frequent, timely, and specific feedback from preceptors. Findings were used to refine the innovation further. CONCLUSION: This study highlights the necessary conditions for assessing the success of implementation of educational innovations. Reciprocal communication between users and developers is vital. This reflects the approaches recommended in the Ottawa Consensus Statement on research in assessment published in Medical Teacher in March 2011.

Defining competency-based evaluation objectives in family medicine: key-feature approach.

OBJECTIVE: To develop key features for priority topics previously identified by the College of Family Physicians of Canada that, together with skill dimensions and phases of the clinical encounter, broadly describe competence in family medicine. DESIGN: Modified nominal group methodology, which was used to develop key features for each priority topic through an iterative process. SETTING: The College of Family Physicians of Canada. PARTICIPANTS: An expert group of 7 family physicians and 1 educational consultant, all of whom had experience in assessing competence in family medicine. Group members represented the Canadian family medicine context with respect to region, sex, language, community type, and experience. METHODS: The group used a modified Delphi process to derive a detailed operational definition of competence, using multiple iterations until consensus was achieved for the items under discussion. The group met 3 to 4 times a year from 2000 to 2007. MAIN FINDINGS: The group analyzed 99 topics and generated 773 key features. There were 2 to 20 (average 7.8) key features per topic; 63% of the key features focused on the diagnostic phase of the clinical encounter. CONCLUSION: This project expands previous descriptions of the process of generating key features for assessment, and removes this process from the context of written examinations. A key-features analysis of topics focuses on higher-order cognitive processes of clinical competence. The project did not define all the skill dimensions of competence to the same degree, but it clearly identified those requiring further definition. This work generates part of a discipline-specific, competency-based definition of family medicine for assessment purposes. It limits the domain for assessment purposes, which is an advantage for the teaching and assessment of learners. A validation study on the content of this work would ensure that it truly reflects competence in family medicine.

Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills.

PROBLEM ADDRESSED: Family medicine residency programs require innovative means to assess residents' competence in "soft" skills (eg, patient-centred care, communication, and professionalism) and to identify residents who are having difficulty early enough in their residency to provide remedial training. OBJECTIVE OF PROGRAM: To develop a method to assess residents' competence in various skills and to identify residents who are having difficulty. PROGRAM DESCRIPTION: The Competency-Based Achievement System (CBAS) was designed to measure competence using 3 main principles: formative feedback, guided self-assessment, and regular face-to-face meetings. The CBAS is resident driven and provides a framework for meaningful interactions between residents and advisors. Residents use the CBAS to organize and review their feedback, to guide their own assessment of their progress, and to discern their future learning needs. Advisors use the CBAS to monitor, guide, and verify residents' knowledge of and competence in important skills. CONCLUSION: By focusing on specific skills and behaviour, the CBAS enables residents and advisors to make formative assessments and to communicate their findings. Feedback indicates that the CBAS is a user-friendly and helpful system to assess competence.

Improving the Quality and Capacity of Canada's Health Services: Primary Care Physician Perspectives.

OBJECTIVE: This study set out to identify the perspectives of family physicians (FP/GPs) on the quality and capacity of the services they provide and of the system in which they work, to assess their responsiveness to potential changes and to determine their suggestions for future directions to enhance primary care services. METHODS: Thematic results from prior focus groups with FP/GPs provided direction for a questionnaire sent to practitioners in the urban study area. Seventy-four questions, most using a five-point Likert scale, were grouped into 10 sections: physician issues (based on themes from the focus groups), access to specialist services, workload, scope of practice, primary care physician networks, interdisciplinary collaborative practice, complexities and challenges of family practice, future directions, comments and demographics. RESULTS: Five hundred and eighty-three FP/GPs were surveyed, and 300 responses (52%) were analyzed for frequencies and comparisons using SPSS. In addition to informative responses to the various survey sections noted above, specific physician suggestions for future directions to improve quality and capacity were identified. These included access to specialists/consultants, teamwork/collaborative practice, access to diagnostics, electronic records/technology, time and remuneration. CONCLUSIONS: The identified suggestions by FP/GPs to enhance the quality and capacity of health services contribute to a framework for policy development at national, provincial/territorial and regional levels and can be used as a reference point for the progress of primary care reform initiatives.

The influence of residents training level on their evaluation of clinical teaching faculty.

BACKGROUND: Evaluations by learners are the most common sources of information on teaching. There is some debate about the role of these assessments, but the overall evaluation of faculty by learners was found to be valid and reliable. PURPOSE: The purpose of this study was to examine the relationship between the level of training of family medicine residents and their evaluation of emergency medicine clinical teachers over time. METHODS: A prospective cohort analysis of 6 years of faculty evaluation of 115 teachers was conducted. RESULTS: The 562 residents returned 3,046 valid individual evaluations. There was no significant association between the level of residents' training and the ratings for clinical instruction (p > .05). Resident evaluations did not vary by time of year (p > .05); however, they did significantly differ by year of evaluation, showing that ratings increased over the 6 years of the study (p < .0001). CONCLUSIONS: Neither the residents' level of training nor the timing during the academic year were significant independent predictors of perceived superior teaching performance, although ratings increased over the 6 years of the study.

Defining core procedure skills for Canadian family medicine training.

OBJECTIVE: To create a list of core and enhanced procedures suitable for family medicine training. DESIGN: Mailed or e-mailed survey using a Delphi technique. SETTING: Randomly selected family physician practices across Canada. PARTICIPANTS: Family physicians from urban, small-town, and rural practice locations and academic family physicians. All were experienced family physicians with from 3 to 36 years in practice. INTERVENTIONS: Participant physicians were asked to rate each of 158 procedures as to whether they would expect a graduate from a Canadian family practice training program to have learned and be capable of performing that procedure in their own community. In a second survey, participants were asked to verify the core and enhanced procedures lists produced from the first survey. MAIN OUTCOME MEASURES: Physicians' opinions about a comprehensive list of skills. RESULTS: Twenty-two physicians responded to the first survey (92% response rate) and 14 to the second (58% response rate). Sixty-five core procedures and 15 enhanced procedures were identified in the surveys. More procedures were ranked on the core list and were performed by rural and small-town physicians than by urban physicians. Physicians' agreement with placement of procedures on the core list ranged from 55% to 100% and of procedures on the enhanced list from 50% to 64%. Fifty-five of the procedures on the core list had agreement from more than 70% of participants. CONCLUSION: Procedure lists represent the opinions of Canadian family physicians about the importance of specific procedure skills for new family physicians in their communities. Procedure lists will be helpful for family medicine training programs to evaluate and refine their teaching of procedure skills.

Resident evaluation of clinical teachers based on teachers' certification.

OBJECTIVE: To examine the influence of emergency medicine (EM) certification of clinical teaching faculty on evaluations provided by residents. METHODS: A prospective cohort analysis was conducted of assessments between July 1994 and July 2000 on residents' evaluations of EM faculty at the University of Alberta, Edmonton, Canada. Resident- and faculty-related variables were entered anonymously using the validated evaluation tool (ER Scale). Credentialing and demographic information on EM faculty was supplemented by data obtained through a nine-question survey. Groups were compared using ANOVA. RESULTS: The 562 residents returned 705 (91%) valid evaluation sheets on 115 EM faculty members. The four domains of didactic teaching, clinical teaching, approachability, and helpfulness were assessed. The majority of ratings were in the very good or superb categories for each domain. Instructors with certification in EM had higher scores in didactic, clinical teaching compared with others, and teachers without national certification scored lower in the helpful and approachable categories (p < 0.05). The route of obtaining EM certifications either through training or practice eligibility did not affect scores. Instructors under the age of 40 years had higher scores than the older age groups in three of four categories (p < 0.05). Instructors working at the teaching sites on a half-time basis received higher scores than those working full-time, and scores varied based on site. Overall, teaching ratings improved over the study period (p < 0.05). CONCLUSIONS: Significant differences exist among instructors in the EM setting that affect their teaching rating scores. National certification in EM, academic track, rotation year, and site are all correlated with better teaching performance.