Investigating faculty assessment of anesthesia trainees and the failing-to-fail phenomenon: a randomized controlled trial.

PURPOSE: A growing body of evidence has shown that supervisors may "fail to fail" trainees even when they have judged their performance unsatisfactory. This has significant implications for the implementation of a nationwide competency-based education model of residency training. The objective of this study was to determine the incidence of "failing to fail" clearly underperforming residents. METHODS: Study participants were recruited via an email invitation sent to all departments of anesthesia at each of the hospitals affiliated with the University of Toronto. They were randomized into a high-stakes (assessment would affect the resident's progress) or low-stakes (assessment would not affect the resident's progress) group and asked to assess the performance (fail or pass grade) of a struggling resident. Participants assessed a video depicting an actor managing a scripted simulation scenario. It contained several critical clinical mistakes constituting a clear fail performance. The purpose of the study was only disclosed following the assessment. RESULTS: Of the 288 email invitations sent (144 in each group), 158 (54%) participants completed the study, with 93 in the high-stakes group and 65 in the low-stakes group. Twenty-eight participants (17.7%) failed to issue a failing grade, including 23.1% (15/65) in the high-stakes group and 13.9% (13/93) in the low-stakes group (P = 0.14). CONCLUSIONS: Though often discussed, this is the first study to quantitatively show that the "failing-to-fail" phenomenon likely occurs during residency training performance evaluations. Passing underperforming learners can potentially affect patient safety and result in severe personal consequences to the learner. The results indicate the need for better performance assessment training for faculty members.

RéSUMé: OBJECTIF: Des données probantes de plus en plus nombreuses ont montré que les superviseurs pourraient « échouer à échouer ¼ des résidents même s'ils ont jugé leur performance insatisfaisante. Cela a d'importantes répercussions sur la mise en œuvre d'un modèle national de formation en résidence axé sur les compétences. L'objectif de cette étude était de déterminer l'incidence d' « échouer à échouer ¼ les résidents dont les résultats sont clairement inadéquats. MéTHODE: Les participants à l'étude ont été recrutés au moyen d'une invitation par courriel envoyée à tous les départements d'anesthésie de chacun des hôpitaux affiliés à l'Université de Toronto. Ils ont été randomisés en un groupe à enjeu élevé (l'évaluation aurait une incidence sur la progression académique du résident) et un groupe à faible enjeu (l'évaluation n'affecterait pas la progression académique du résident) et on leur a demandé d'évaluer la performance (échec ou passage) d'un résident en difficulté. Les participants ont évalué une vidéo montrant un acteur prenant en charge un cas de simulation scénarisé. La vidéo comportait plusieurs erreurs cliniques critiques constituant clairement un échec. L'objectif de l'étude n'a été divulgué qu'après l'évaluation. RéSULTATS: Sur les 288 invitations par courriel envoyées (144 dans chaque groupe), 158 (54 %) participants ont terminé l'étude, dont 93 dans le groupe à enjeu élevé et 65 dans le groupe à faible enjeu. Vingt-huit participants (17,7 %) n'ont pas donné de note d'échec, dont 23,1 % (15/65) dans le groupe à enjeu élevé et 13,9 % (13/93) dans le groupe à enjeu faible (P = 0,14). CONCLUSION: Bien que cette question soit souvent discutée, il s'agit de la première étude à montrer quantitativement que le phénomène d' « échouer à échouer ¼ survient probablement lors des évaluations pendant la formation en résidence. Le fait de laisser passer des résidents n'ayant pas acquis les compétences peut potentiellement affecter la sécurité des patients et entraîner de graves conséquences personnelles pour le résident. Les résultats indiquent la nécessité d'une meilleure formation à l'évaluation des performances pour les membres du corps professoral.

The effect of simulator fidelity on acquiring non-technical skills: a randomized non-inferiority trial.

INTRODUCTION: As simulator fidelity (i.e., realism) increases from low to high, the simulator more closely resembles the real environment, but it also becomes more expensive. It is generally assumed that the use of high-fidelity simulators results in better learning; however, the effect of fidelity on learning non-technical skills (NTS) is unknown. This was a non-inferiority trial comparing the efficacy of high- vs low-fidelity simulators on learning NTS. METHODS: Thirty-six postgraduate medical trainees were recruited for the trial. During the pre-test phase, the trainees were randomly assigned to manage a scenario using either a high-fidelity simulator (HFS) or a low-fidelity simulator (LFS), followed by expert debriefing. All trainees then underwent a video recorded post-test scenario on a HFS, and the NTS were assessed between the two groups. The primary outcome was the overall post-test Ottawa Global Rating Scale (OGRS), while controlling for overall pre-test OGRS scores. Non-inferiority between the LFS and HFS was based on a non-inferiority margin of greater than 1. RESULTS: For our primary outcome, the mean (SD) post-test overall OGRS score was not significantly different between the HFS and LFS groups after controlling for pre-test overall OGRS scores [3.8 (0.9) vs 4.0 (0.9), respectively; mean difference, 0.2; 95% confidence interval, -0.4 to 0.8; P = 0.48]. For our secondary outcomes, the post-test total OGRS score was not significantly different between the HFS and LFS groups after controlling for pre-test total OGRS scores (P = 0.33). There were significant improvements in mean overall (P = 0.01) and total (P = 0.003) OGRS scores from pre-test to post-test. There were no significant associations between postgraduate year (P = 0.82) and specialty (P = 0.67) on overall OGRS performance. CONCLUSION: This study suggests that low-fidelity simulators are non-inferior to the more costly high-fidelity simulators for teaching NTS to postgraduate medical trainees.

Learning crisis resource management: Practicing versus an observational role in simulation training - a randomized controlled trial.

AIM: Simulation training has been shown to be an effective way to teach crisis resource management (CRM) skills. Deliberate practice theory states that learners need to actively practice so that learning is effective. However, many residency programs have limited opportunities for learners to be "active" participants in simulation exercises. This study compares the effectiveness of learning CRM skills when being an active participant versus being an observer participant in simulation followed by a debriefing. METHODS: Participants were randomized to two groups: active or observer. Active participants managed a simulated crisis scenario (pre-test) while paired observer participants viewed the scenario via video transmission. Then, a trained instructor debriefed participants on CRM principles. On the same day, each participant individually managed another simulated crisis scenario (post-test) and completed a post-test questionnaire. Two independent, blinded raters evaluated all videos using the Ottawa Global Rating Scale (GRS). RESULTS: Thirty-nine residents were included in the analysis. Normally distributed data were analyzed using paired and unpaired t-tests. Inter-rater reliability was 0.64. Active participants significantly improved from pre-test to post-test (P=0.015). There was no significant difference between the post-test performance of active participants compared to observer participants (P=0.12). CONCLUSION: We found that learning CRM principles was not superior when learners were active participants compared to being observers followed by debriefing. These findings challenge the deliberate practice theory claiming that learning requires active practice. Assigning residents as observers in simulation training and involving them in debriefing is still beneficial.

Sudden hypoxia during anesthesia in a patient with Kartagener's syndrome.

We report a case of anesthesia in an adolescent with recurrent left pneumothorax, Kartagener's syndrome, and severe learning disability with behavioral difficulties. After induction of anesthesia, he rapidly developed severe desaturation as measured on pulse oximetry. Placement of an intercostal chest drain did not remedy the situation and he was found to have blocked the left main bronchus with viscous secretions. Subsequent suctioning relieved the obstruction. Despite successful postoperative thoracic epidural analgesia and minitracheostomy for bronchial toilet, he developed bronchopneumonia that resolved with antibiotics. We discuss anesthesia for patients with Kartagener's syndrome and for patients with pneumothorax.