Examining the accuracy of residents' self-assessments and faculty assessment behaviours in anesthesiology.

BACKGROUND: Residents' accurate self-assessment and clinical judgment are essential for optimizing their clinical skills development. Evidence from the medical literature suggests that residents generally do poorly at self-assessing their performance, often due to factors relating to learners' personal backgrounds, cultures, the specific contexts of the learning environment and rater bias or inaccuracies. We evaluated the accuracy of anesthesiology residents' self-assessed Global Entrustment scores and determined whether differences between faculty and resident scores varied by resident seniority, faculty leniency, and/or year of assessment. METHODS: We employed variance components modeling techniques and analyzed 329 pairs of faculty and self-assessed entrustment scores among 43 faculty assessors and 15 residents. Using faculty scores as the gold standard, we compared faculty scores with residents' scores (xi(faculty)-xi(resident)), and determined residents' accuracy, including over- and under-confidence. RESULTS: The results indicate that residents were respectively over- and under-confident in 10.9% and 54.4% of the assessments but more consistent in their individual self-assessments (rho = 0.70) than faculty assessors. Faculty scores were significantly higher (α = 0.396; z = 4.39; p < 0.001) than residents' self-assessed scores. Being a lenient/dovish (ß = 0.121, z = 3.16, p < 0.01) and a neutral (ß = 0.137, z = 3.57, p < 0.001) faculty assessor predicted a higher likelihood of resident under-confidence. Senior residents were significantly less likely to be under-confident compared to junior residents (ß = -0.182, z =-2.45, p < 0.05). The accuracy of self-assessments did not significantly vary during the two years of the study period. CONCLUSIONS: The majority of residents' self-assessments were inaccurate. Our findings may help identify the sources of such inaccuracies.

CONTEXTE: L'autoévaluation adéquate et le jugement clinique des résidents sont essentiels pour optimiser le développement de leurs compétences cliniques. Les données probantes de la littérature médicale indiquent que les résidents ont généralement du mal à s'auto-évaluer, souvent en raison de facteurs liés à leur passé personnel, à la culture, aux contextes spécifiques de l'environnement d'apprentissage et aux préjugés ou inexactitudes des évaluateurs. Nous avons évalué l'exactitude des scores d'autoévaluation par échelles de confiance globalepar des résidents en anesthésiologie et déterminé si les différences entre les scores des enseignants et des résidents variaient en fonction du niveau de formation des résidents, de l'indulgence des enseignants ou de l'année d'évaluation. MÉTHODES: Nous avons utilisé des techniques de modélisation des composantes de la variance et analysé 329 paires de scores de confiance des enseignants et d'autoévaluation avec la participation de 43 évaluateurs du corps professoral et 15 résidents. Prenant les scores des enseignants comme référence, nous avons comparé les leurs avec ceux des résidents (xi(enseignant) -xi(résident)), et déterminé l'exactitude chez les résidents, y compris l'excès et le manque de confiance. RÉSULTATS: Les résultats indiquent que les résidents étaient trop confiants dans 10,9 % des évaluations, et pas assez confiants dans 54,4 % des cas, mais qu'ils étaient plus cohérents dans leurs autoévaluations (rho = 0,70) que ne l'étaient les enseignants. Les scores du corps professoral étaient significativement plus élevés (α = 0,396 ; z = 4,39 ; p < 0,001) que les scores d'autoévaluation des résidents. Le fait d'être un évaluateur indulgent (ß = 0,121, z = 3,16, p < 0,01) et neutre (ß = 0,137, z = 3,57, p < 0,001) prédisait une probabilité plus élevée de sous-confiance des résidents. Les résidents seniors étaient significativement moins susceptibles de manquer de confiance en eux que les résidents juniors (ß = -0,182, z = -2,45, p < 0,05). L'exactitude des autoévaluations n'a pas varié de manière significative au cours des deux années de la période d'étude. CONCLUSIONS: La majorité des autoévaluations des résidents étaient inexactes. Nos résultats peuvent aider à identifier les sources de ces inexactitudes.

Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment framework.

Introduction: Simulation training in anaesthesiology bridges the gap between theory and practice by allowing trainees to engage in high-stakes clinical training without jeopardising patient safety. However, implementing simulation-based assessments within an academic programme is highly resource intensive, and the optimal number of scenarios and faculty required for accurate competency-based assessment remains to be determined. Using a generalisability study methodology, we examine the structure of simulation-based assessment in regard to the minimal number of scenarios and faculty assessors required for optimal competency-based assessments. Methods: Seventeen anaesthesiology residents each performed four simulations which were assessed by two expert raters. Generalisability analysis (G-analysis) was used to estimate the extent of variance attributable to (1) the scenarios, (2) the assessors and (3) the participants. The D-coefficient and the G-coefficient were used to determine accuracy targets and to predict the impact of adjusting the number of scenarios or faculty assessors. Results: We showed that multivariate G-analysis can be used to estimate the number of simulations and raters required to optimise assessment. In this study, the optimal balance was obtained when four scenarios were assessed by two simulation experts. Conclusion: Simulation-based assessment is becoming an increasingly important tool for assessing the competency of medical residents in conjunction with other assessment methods. G-analysis can be used to assist in planning for optimal resource use and cost-efficacy.

Simulation-based assessment in anesthesia: an international multicentre validation study.

PURPOSE: Simulated clinical events provide a means to evaluate a practitioner's performance in a standardized manner for all candidates that are tested. We sought to provide evidence for the validity of simulation-based assessment tools in simulated pediatric anesthesia emergencies. METHODS: Nine centres in two countries recruited subjects to participate in simulated operating room events. Participants ranged in anesthesia experience from junior residents to staff anesthesiologists. Performances were video recorded for review and scored by specially trained, blinded, expert raters. The rating tools consisted of scenario-specific checklists and a global rating scale that allowed the rater to make a judgement about the subject's performance, and by extension, preparedness for independent practice. The reliability of the tools was classified as "substantial" (intraclass correlation coefficients ranged from 0.84 to 0.96 for the checklists and from 0.85 to 0.94 for the global rating scale). RESULTS: Three-hundred and ninety-one simulation encounters were analysed. Senior trainees and staff significantly out-performed junior trainees (P = 0.04 and P < 0.001 respectively). The effect size of grade (junior vs senior trainee vs staff) on performance was classified as "medium" (partial η2 = 0.06). Performance deficits were observed across all grades of anesthesiologist, particularly in two of the scenarios. CONCLUSIONS: This study supports the validity of our simulation-based anesthesiologist assessment tools in several domains of validity. We also describe some residual challenges regarding the validity of our tools, some notes of caution in terms of the intended consequences of their use, and identify opportunities for further research.

Selective left mainstem bronchial intubation in the neonatal intensive care unit / Intubação seletiva do brônquio principal esquerdo em unidade de terapia intensiva neonatal

Abstract Background: Selective neonatal left mainstem bronchial intubation to treat right lung disease is typically achieved with elaborate maneuvers, instrumentation and devices. This is often attributed to bronchial geometry which favors right mainstem entry of an endotracheal tube deliberately advanced beyond the carina. Case summary: A neonate with severe bullous emphysema affecting the right lung required urgent non-ventilation of that lung. We achieved left mainstem bronchial intubation by turning the endotracheal tube 180° such that the Murphy's eye faced the left instead of the right, and simulated a left-handed intubation by slightly orientating the endotracheal tube such that its concavity faced the left instead of the right as in a conventional right-handed intubation. Conclusion: Urgent intubation of the left mainstem bronchus with an endotracheal tube can be easily achieved by recognizing that it is the position of the endotracheal tube tip and the direction of its concavity that are the chief determinants of which bronchus an endotracheal tube goes when advanced. This is important in critically ill neonates as the margin of safety and time window are small, and the absence of double-lumen tubes. Use of fiberoptic bronchoscope and blockers should be reserved as backup plans.

Resumo Justificativa: A intubação seletiva neonatal do brônquio principal esquerdo para tratar a doença pulmonar direita é tipicamente feita com elaboradas manobras, instrumentação e dispositivos. Isso é frequentemente atribuído à geometria brônquica que favorece a entrada principal direita de um tubo endotraqueal (TET) deliberadamente avançado para além da carina. Resumo do caso: Recém-nascido com enfisema bolhoso grave que afetava o pulmão direito e precisou com urgência da não ventilação desse pulmão. Para conseguir a intubação brônquica esquerda fizemos uma rotação de 180° do TET, de forma que o olho de Murphy ficasse voltado para a esquerda, e não para a direita, e para simular uma intubação à esquerda orientamos ligeiramente o TET, de modo que sua concavidade virasse para a esquerda em vez de para a direita, como em uma intubação convencional à direita. Conclusão: A intubação urgente do brônquio principal esquerdo com um TET pode ser facilmente obtida se reconhecermos que é a posição da ponta do TET e a direção de sua concavidade que determinam para qual brônquio o TET irá quando avançado. Isso é importante em neonatos criticamente doentes diante da margem de segurança e janela de tempo pequenas e na ausência de tubos de duplo lúmen. O uso de broncofibroscópio e bloqueadores deve ser considerado como planos de segurança.

[Selective left mainstem bronchial intubation in the neonatal intensive care unit]. / Intubação seletiva do brônquio principal esquerdo em unidade de terapia intensiva neonatal.

BACKGROUND: Selective neonatal left mainstem bronchial intubation to treat right lung disease is typically achieved with elaborate maneuvers, instrumentation and devices. This is often attributed to bronchial geometry which favors right mainstem entry of an endotracheal tube deliberately advanced beyond the carina. CASE SUMMARY: A neonate with severe bullous emphysema affecting the right lung required urgent non-ventilation of that lung. We achieved left mainstem bronchial intubation by turning the endotracheal tube 180° such that the Murphy's eye faced the left instead of the right, and simulated a left-handed intubation by slightly orientating the endotracheal tube such that its concavity faced the left instead of the right as in a conventional right-handed intubation. CONCLUSION: Urgent intubation of the left mainstem bronchus with an endotracheal tube can be easily achieved by recognizing that it is the position of the endotracheal tube tip and the direction of its concavity that are the chief determinants of which bronchus an endotracheal tube goes when advanced. This is important in critically ill neonates as the margin of safety and time window are small, and the absence of double-lumen tubes. Use of fiberoptic bronchoscope and blockers should be reserved as backup plans.

"Debriefing-on-Demand": A Pilot Assessment of Using a "Pause Button" in Medical Simulation.

INTRODUCTION: Simulation is an effective tool in medical education with debriefing as the cardinal educational component. Alternate debriefing strategies might further enhance the educational value of simulation. Here, we pilot a novel strategy that allows trainees to initiate debriefing at any point during the scenario, when they consider it necessary. METHODS: With ethics approval, 8 postgraduate year 1 anesthesia residents (with no previous exposure to high-fidelity simulation) were randomly assigned to lead 2 of 8 scenarios with 2 debriefing strategies. With "debriefing-on-demand," residents had the option to initiate debriefing at any point in the scenario by activation of a "pause button"-in addition to undergoing conventional debriefing at the end of the scenario. Those randomized to "conventional debriefing" were debriefed only at the end of the scenario. All were allocated as team leader with both debriefing strategies and as a participant in remaining scenarios. Residents provided feedback regarding each method using Likert scales and completion of open-ended statements. RESULTS: Debriefing-on-demand was easily integrated into all scenarios, and most learners (88%) supported its use in future simulation sessions. The following 4 themes emerged from qualitative analyses: (1) improvements in the clarification and integration of knowledge, (2) reductions in stress/anxiety, (3) facilitated reflection on action, and (4) maintained realism comparable with conventional debriefing. CONCLUSIONS: Debriefing-on-demand was easily integrated into all scenarios and well received by these trainees new to simulation. Larger trials that use validated tools are needed to determine the absolute impact of debriefing-on-demand on stress levels and the overall learning value of simulation for trainees at different levels of training.