Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study.

BACKGROUND: In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement. OBJECTIVE: We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels. METHODS: Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. RESULTS: Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested. CONCLUSIONS: There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.

Prehospital Application of the Canadian Triage and Acuity Scale by Emergency Medical Services.

OBJECTIVES: Triage is fundamental to emergency patient assessment. Effective triage systems accurately prioritize patients and help predict resource utilization. CTAS is a validated five-level triage score utilized in Emergency Departments (EDs) across Canada and internationally. Historically CTAS has been applied by triage nurses in EDs. Observational evidence suggests that the CTAS might be implemented reliably by paramedics in the prehospital setting. This is the first system-wide assessment of CTAS interrater reliability between paramedics and triage nurses during clinical practice. METHODS: Variables were extracted from hospital and EMS databases. EMS providers determined CTAS on-scene, CTAS pre-transport, and CTAS on-arrival at hospital for each patient (N=14,378). The hospital arrival EMS CTAS (CTAS arrival ) score was compared to the initial nursing CTAS score (CTAS initial ) and the final nursing CTAS score (CTAS final ) incuding nursing overrides. Interrater reliability between ED CTAS initial and EMS CTAS arrival scores was assessed. Interrater reliability between ED CTAS final and EMS CTAS arrival scores, as well as proportion of patient encounters with perfect or near-perfect agreement, were evaluated. RESULTS: Our primary outcome, interrater reliability [kappa=0.437 (p<0.001, 95% CI 0.421-0.452)], indicated moderate agreement. EMS CTAS arrival and ED CTAS initial scores had an exact or within one point match 84.3% of the time. The secondary interrater reliability outcome between hospital arrival EMS CTAS (CTAS arrival ) score and the final ED triage CTAS score (CTAS final ) showed moderate agreement with kappa =0.452 (p<0.001, 95% CI 0.437-0.466). CONCLUSIONS: Interrater reliability of CTAS scoring between triage nurses and paramedics was moderate in this system-wide implementation study.