Competency-based education calls for programmatic assessment: But what does this look like in practice?

RATIONALE, AIMS, AND OBJECTIVES: Programmatic assessment has been identified as a system-oriented approach to achieving the multiple purposes for assessment within Competency-Based Medical Education (CBME, i.e., formative, summative, and program improvement). While there are well-established principles for designing and evaluating programs of assessment, few studies illustrate and critically interpret, what a system of programmatic assessment looks like in practice. This study aims to use systems thinking and the 'two communities' metaphor to interpret a model of programmatic assessment and to identify challenges and opportunities with operationalization. METHOD: An interpretive case study was used to investigate how programmatic assessment is being operationalized within one competency-based residency program at a Canadian university. Qualitative data were collected from residents, faculty, and program leadership via semi-structured group and individual interviews conducted at nine months post-CBME implementation. Data were analyzed using a combination of data-based inductive analysis and theory-derived deductive analysis. RESULTS: In this model, Academic Advisors had a central role in brokering assessment data between communities responsible for producing and using residents' performance information for decision making (i.e., formative, summative/evaluative, and program improvement). As system intermediaries, Academic Advisors were in a privileged position to see how the parts of the assessment system contributed to the functioning of the whole and could identify which system components were not functioning as intended. Challenges were identified with the documentation of residents' performance information (i.e., system inputs); use of low-stakes formative assessments to inform high-stakes evaluative judgments about the achievement of competence standards; and gaps in feedback mechanisms for closing learning loops. CONCLUSIONS: The findings of this research suggest that program stakeholders can benefit from a systems perspective regarding how their assessment practices contribute to the efficacy of the system as a whole. Academic Advisors are well positioned to support educational development efforts focused on overcoming challenges with operationalizing programmatic assessment.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation.

PURPOSE: Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change. METHOD: Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences. RESULTS: In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow. CONCLUSIONS: Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Socioeconomic status of emergency department users in Ontario, 2003 to 2009.

OBJECTIVE: Emergency department (ED) overcrowding in Canada is an ongoing problem resulting in prolonged wait times, service declines, increased patient suffering, and adverse patient outcomes. We explored the relationship between socioeconomic status (SES) and ED use in Canada's universal health care system to improve our understanding of the nature of ED users to both improve health care to the most deprived populations and reduce ED patient input. METHODS: This retrospective study took information from the National Ambulatory Care Reporting System (NACRS) database for all ED visits in Ontario between April 1, 2003, and March 31, 2010. As there is no direct measure of SES available from ED visit records, a proxy measure of SES was used, namely a deprivation index (DI) developed from material and social factors from the 2006 Canadian census using the patient's residential neighbourhood. DI scores were assigned to ED visit records using Statistics Canada's Postal Code Conversion File, which links postal and census geography. RESULTS: A total of 36,765,189 visits occurred during the study period. A cross-province trend was found wherein the most deprived population used EDs disproportionately more than the least deprived population (relative risk: 1.971 95% confidence interval 1.969-1.973, p < 0.0001). This trend was stable across the entire study period, although the divergence is attenuating. CONCLUSION: Social determinants of health clearly impact ED use patterns. People of the lowest SES use ED services disproportionately more than other socioeconomic groups. Focused health system planning and policy development directed at optimizing health services for the lowest SES populations are essential to changing ED use patterns and may be one method of decreasing ED overcrowding.

Socioeconomic composition of low-acuity emergency department users in Ontario.

OBJECTIVE: To describe the associations between the socioeconomic status of emergency department (ED) users and age, sex, and acuity of medical conditions to better understand users' common characteristics, and to better meet primary and ambulatory health care needs. DESIGN: A retrospective, observational, population-based analysis. A rigorous proxy of socioeconomic status was applied using census-based methods to calculate a relative deprivation index. SETTING: Ontario. PARTICIPANTS: All Ontario ED visits for the fiscal year April 1, 2008, to March 31, 2009, from the National Ambulatory Care Reporting System data set. MAIN OUTCOME MEASURES: Emergency department visits were ranked into deprivation quintiles, and associations between deprivation and age, sex, acuity at triage, and association with a primary care physician were investigated. RESULTS: More than 25% of ED visits in Ontario were from the most deprived population; almost half of those (12.3%) were for conditions of low acuity. Age profiles indicated that a large contribution to low-acuity ED visits was made by young adults (aged 20 to 30 years) from the most deprived population. For the highest-volume ED in Ontario, 94 of the 499 ED visits per day were for low-acuity patients from the most deprived population. Most of the highest volume EDs in Ontario (more than 200 ED visits per day) follow this trend. CONCLUSION: Overall input into EDs might be reduced by providing accessible and appropriate primary health care resources in catchment areas of EDs with high rates of low-acuity ED visits, particularly for young adults from the most deprived segment of the population.

Telehealth Ontario detection of gastrointestinal illness outbreaks.

OBJECTIVE: Prompt detection of infectious disease outbreaks and rapid introduction of mitigation strategies is a primary concern for public health, emergency and security management organizations. Traditional surveillance methods rely on astute clinical detection and reporting of disease or laboratory confirmation. Although effective, these methods are slow, dependent on physician compliance and delay timely, effective intervention. To address these issues, syndromic surveillance programs have been integrated into the health care system at the earliest points of access; in Ontario, these points are primary care providers, emergency departments (ED), and Telehealth Ontario. This study explores the role of Telehealth Ontario, a telephone helpline, as an early warning system for detection of gastrointestinal (GI) illness. METHODS: Retrospective time-series analysis of the National Ambulatory Care Reporting System (NACRS) ED discharges and Telehealth Ontario data for GI illness from June 1, 2004 to March 31, 2006. RESULTS: Telehealth Ontario recorded 184,904 calls and the NACRS registered 34,499 ED visits for GI illness. The Spearman rank correlation coefficient was calculated to be 0.90 (p < 0.0001). Time-series analysis resulted in significant correlation at lag (weekly) 0 indicating that increases in Telehealth Ontario call volume correlate with increases in NACRS data for GI illness. CONCLUSION: Telehealth Ontario call volume fluctuation reflects directly on ED GI visit data on a provincial basis. Telehealth Ontario GI call complaints are a timely, novel and representative data stream that shows promise for integration into a real-time syndromic surveillance system for detection of unexpected events.

Impact of a rapid access protocol on decreasing door-to-balloon time in acute ST elevation myocardial infarction.

OBJECTIVE: Ischemic cardiovascular disease is the leading cause of death in Canada. In ST elevation myocardial infarction (STEMI), time to reperfusion is a key determinant in reducing morbidity and mortality with percutaneous coronary intervention (PCI) being the preferred reperfusion strategy. Where PCI is available, delays to definitive care include times to electrocardiogram (ECG) diagnosis and cardiovascular laboratory access. In 2004, the Cardiac Care Network of Ontario recommended implementation of an emergency department (ED) protocol to reduce reperfusion time by transporting patients with STEMI directly to the nearest catheterization laboratory. The model was implemented in Frontenac County in April 2005. The objective of this study was to assess the effectiveness of a protocol for rapid access to PCI in reducing door-to-balloon times in STEMI. METHODS: Two 1-year periods before and after implementation of a rapid access to PCI protocol (ending March 2005 and June 2006, respectively) were studied. Administrative databases were used to identify all subjects with STEMI who were transported by regional emergency medical services (EMS) and received emergent PCI. The primary outcome measure was time from ED arrival to first balloon inflation (door-to-balloon time). Times are presented as medians and interquartile ranges (IQRs). Statistical comparisons were made using the Mann-Whitney U test and presented graphically with Kaplan-Meier curves. RESULTS: Patients transported under the rapid access protocol (n = 39) were compared with historical controls (n = 42). Median door-to-balloon time was reduced from 87 minutes (IQR 67-108) preprotocol to 62 minutes (IQR 40-80) postprotocol (p < 0.001). CONCLUSION: In our region, implementation of an EMS protocol for rapid access to PCI significantly reduced time to reperfusion for patients with STEMI.

Airbag pneumonitis: a report and discussion of a new clinical entity.

This report describes the occurrence of pneumonitis in a young male immediately after inhalation of aerosolized chemicals subsequent to motor vehicle airbag deployment. The clinical presentation was one of mild shortness of breath associated with bilateral alveolar infiltrates on chest radiology. Not previously described, this diagnosis should be considered in the differential of pulmonary infiltrates in motor vehicle crash patients.