Effect of Metrics-Based Simulation Training to Proficiency on Procedure Quality and Errors Among Novice Cardiac Device Implanters: The IMPROF Randomized Trial.

Importance: In cardiac device implant training, there is no common system to objectively assess trainees' ability to perform tasks at predetermined performance levels before in vivo practice; therefore, patients are potentially exposed to risks related to operators' early learning curve. Objective: To assess the effect on implant performance quality of novel metrics-based simulation training to proficiency (proficiency-based progression [PBP]) vs traditional simulation-based training (SBT). Design, Setting, and Participants: In this prospective randomized trial, conducted between March 8, 2022 and November 24, 2022, novice implanters were randomized (blinded) 1:1 to participate in an SBT curriculum (procedural knowledge e-learning and in-person simulation training) at an international skills training center, with proficiency demonstration requirements at each training stage for advancing (PBP approach) or without the requirements. Ultimately, trainees performed a cardiac resynchronization therapy (CRT) implant using virtual reality simulation. The procedure was video-recorded and subsequently scored using previously validated metrics by 2 independent assessors blinded to group. Physicians who had already implanted more than 20 pacemakers or defibrillators and fewer than 200 CRT systems as the first operator were eligible. Thirty-two implanters from 10 countries voluntarily enrolled in the training program and were randomized; 30 (15 per group) started and completed training. Data analysis was performed from November 27 to December 22, 2022. Intervention: Training with PBP vs SBT. Main Outcome and Measures: The primary outcome comprised 4 objectively assessed performance metrics derived from the video-recordings: number of procedural steps completed, errors, critical errors, and all errors combined. Results: Baseline experience of the 30 participants (19 [63%] male; mean [SD] number of years in implant practice, 2.0 [1.8]; median [IQR] number of implanted pacemakers or defibrillators, 47.5 [30.0-115.0]; median [IQR] number of implanted CRT systems, 3.0 [1.25-10.0]) was similar between study groups. Compared with the SBT group, the PBP group completed 27% more procedural steps (median [IQR], 31 [30-32] vs 24 [22-27]; P < .001) and made 73% fewer errors (median [IQR], 2 [1-3] vs 7 [5-8]; P < .001), 84% fewer critical errors (median [IQR], 1 [0-1] vs 3 [3-5]; P < .001), and 77% fewer all errors combined (errors plus critical errors) (median [IQR], 3 [1-3] vs 11 [8-12]; P < .001); 14 of the 15 PBP trainees (93%) demonstrated the predefined target performance level vs 0 of the 15 SBT trainees. Conclusions and Relevance: In this randomized trial, the PBP approach to novice implanter training generated superior objectively assessed performance vs SBT. If implemented broadly and systematically, PBP training may ensure safe and effective performance standards before trainees proceed to (supervised) in vivo practice. Future studies are needed to verify implications on procedure-related patient complications. Trial Registration: ClinicalTrials.gov Identifier: NCT05952908.

A validation study of intraoperative performance metrics for training novice cardiac resynchronization therapy implanters.

AIMS: Pacing/cardiac resynchronization therapy (CRT) implant training currently lacks a common system to objectively assess trainee ability to perform required tasks at predetermined performance levels. The purpose of this study was to primarily examine construct validity and reliability, secondarily discriminative validity of novel intraoperative performance metrics, developed for a reference approach to training novice CRT implanters. METHODS: Fifteen novice and eleven experienced CRT implanters performed a 3-lead implant procedure on a virtual reality simulator. Performances were video-recorded, then independently scored using predefined metrics endorsed by an international panel of experts. First, Novice and Experienced group scores were compared for steps completed and errors made. Secondly, each group was split in two around the median score of the group and subgroup scores were compared. RESULTS: The mean number of scored metrics per performance was 108 and the inter-rater reliability for scoring was 0.947. Compared with novices, experienced implanters completed more procedural Steps correctly (mean 87% vs. 73%, p = 0.001), made fewer procedural Errors (6.3 vs. 11.2, p = 0.005), Critical Errors (1.8 vs. 4.4, p = 0.004), and total errors (8.1 vs. 15.6, p = 0.002). Furthermore, the differences between the two Novice subgroups were 25% for steps completed correctly and 94% for total errors made (p < 0.001); the differences between the two Experienced subgroups were respectively 16% and 191% (p < 0.001). CONCLUSIONS: The procedure metrics used in this study reliably distinguish novice and experienced CRT implanters' performances. The metrics further differentiated performance levels within a group with similar experience. These performance metrics will underpin quality-assured novice implanter training.

International expert consensus on a scientific approach to training novice cardiac resynchronization therapy implanters using performance quality metrics.

AIMS: Pacing and Cardiac Resynchronization Therapy (CRT) procedural training for novice operators usually takes place in-vivo and methods vary across countries/institutions. No common system exists to objectively assess trainee ability to perform required tasks at predetermined performance levels prior to in-vivo practice. We sought to characterize and validate with experts a reference approach to pacing/CRT implants based on objective and explicit performance quality metrics, for the development of a reproducible, simulation-based, training curriculum aiming to operator proficiency. METHODS: Three experienced CRT implanters, a behavioural scientist and two engineers performed a detailed task deconstruction of the pacing/CRT procedure and identified the performance metrics (phases, steps, errors, critical errors) that constitute an optimal CRT implant for training purposes. The metrics were stress tested to determine reliability and score-ability and then subjected to detailed systematic review by an international panel of 15 expert implanters in a modified Delphi process. RESULTS: Thirteen procedure phases were identified, consisting of 196 steps, 122 errors, 50 critical errors. The expert panel deliberation added 16 metrics, deleted 12, and modified 43. Unanimous panel consensus on the resulting CRT procedure metrics was obtained, which verified face and content validity. CONCLUSION: A reference pacing/CRT procedure and metrics created by a core group of experts accurately characterize the essential components of performance and were endorsed by an international panel of experienced peers. The metrics will underpin quality-assured novice implanter training.