Simulation-based summative assessment in healthcare: an overview of key principles for practice.

BACKGROUND: Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start. METHODS: First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee. RESULTS: Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted. CONCLUSION: Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Change in Sleep Quality of Residents the Night Before High-Fidelity Simulation: Results From a Prospective 1-Year National Survey.

OBJECTIVE: The stress level of participants in high-fidelity simulation stems from various factors but may result in anticipatory anxiety causing sleep disturbances during the night prior to simulation. The objective of this survey was to determine the change in sleep quality of residents during the night prior to the simulation. METHODS: The survey was proposed for 1 year to all residents at the beginning of the simulation, in 10 simulation centres. The questionnaire combined demographics and the Leeds Sleep Evaluation Questionnaire using visual analogue scales divided into 4 sleep qualitative domains. The primary outcome was the prevalence of sleep disturbance (>10 mm on 1 domain). Secondary outcomes were the prevalence of severe sleep disturbance (>25 mm), as well as qualitatively and quantitatively reported explanatory sleep parameters. RESULTS: Among respondents, 66% [95% CI: 63 to 69] of residents had more than 10 mm and 27% [95% CI: 24 to 30] had more than 25 mm of sleep disturbance. Residents with a sleep disturbance of more than 10 mm had fewer hours of sleep (6.4 [standard deviation=1.8] vs 7.3 [standard deviation=1.3], difference: -0.9 [95% CI: -1.1 to -0.7]; P < .0001), with a higher number of night-time awakenings (1.3 [standard deviation=1.5] vs 0.7 [standard deviation=0.9], difference: 0.6 [95% CI: 0.4 to 0.8]; P < .0001). CONCLUSION: Among residents participating in the simulation, a high prevalence of change in sleep quality during the night before the simulation was noted. Strategies to help residents achieve better sleep prior to simulation should be explored.

Efecto de un debriefing (reunión informativa) combinado en los participantes en cursos de simulación interprofesional para la gestión de recursos de crisis: un estudio aleatorio, controlado y multicéntrico / Effect of combined individual–collective debriefing of participants in interprofessional simulation courses on crisis resource management: a randomized controlled multicenter trial

Objetivo: La simulación interprofesional (SIP) es eficaz para aprender gestión de recursos de crisis. La modalidad de debriefing utilizada en la SIP puede influir en el rendimiento de los participantes y en su integridad psicológica. Se evalúa y compara el rendimiento de un debriefing estándar (DE) –colectivo– con un debriefing combinado (DC) –individual y colectivo– en cursos de SIP en escenarios que simulan pacientes con patología aguda y grave. Método: Ensayo controlado, aleatorizado y multicéntrico. Se aleatorizó el tipo de debriefing realizado (DE o DC) en las sesiones de SIP. El rendimiento del debriefing se evaluó con la escala TEAM (Team Emergency Assessment Measure). La calidad de la SIP fue valorada por los participantes con la escala DASH (Debriefing Assessment for Simulation in Healthcare©). Resultados. Se aleatorizaron 40 cursos de SIP de los que se analizaron 30. Quince realizaron DE y 15 DC. Ambos grupos mejoraron entre la pre y la posprueba (p < 0,01), pero no hubo diferencias en el rendimiento global entre ambas modalidades de debriefing (p = 0,64). El DC obtuvo mejores resultados que el DE en la capacidad de liderazgo (p < 0,05), en la percepción de seguridad psicológica y en la experiencia de aprendizaje eficaz (p < 0,05). Conclusiones: Durante la SIP en situaciones de crisis, el debriefing mejora el rendimiento de los participantes, sin diferencias entre un DE y un DC. El DC podría ser más efectivo para mejorar la capacidad de liderazgo, la seguridad psicológica y la experiencia del aprendizaje

Objective: Interprofessional simulation (IPS) training is an effective way to learn crisis resource management. The type of debriefing used in IPS training may affect participants' performance and their level of psychological safety. We aimed to assess and compare performance after standard collective debriefing versus a combination of individual and collective debriefing ("combined" approach). Methods: Randomized, controlled multicenter trial. IPS sessions were randomized to have either standard or combined debriefing. Each team's performance in the IPS session was assessed with the Team Emergency Assessment Measure. The participants assessed the debriefing quality with the Debriefing Assessment for Simulation in Healthcare. Results: Forty IPS sessions were randomized, and 30 were analyzed, 15 using standard collective debriefing and 15 the combined individual–collective method. Teams' performance improved with both types of debriefing, based on pre-post testing (P<.01), and there were no significant differences in overall performance scores between the 2 types of debriefing (P=.64). However, the combined approach was associated with higher scores for leadership skills (P<.05) and psychological safety, and the participants' learning experience was better (P<.05). Conclusions: During IPS courses on crisis resource management, debriefing improves participants' performance, but similar overall results can be obtained with both debriefing methods. Combined debriefing might be more effective for improving participants' leadership skills and psychological safety and also provide a better learning experience

Effect of combined individual-collective debriefing of participants in interprofessional simulation courses on crisis resource management: a randomized controlled multicenter trial. / Efecto de un debriefing (reunión informativa) combinado en los participantes en cursos de simulación interprofesional para la gestión de recursos de crisis: un estudio aleatorio, controlado y multicéntrico.

OBJECTIVES: Interprofessional simulation (IPS) training is an effective way to learn crisis resource management. The type of debriefing used in IPS training may affect participants' performance and their level of psychological safety. We aimed to assess and compare performance after standard collective debriefing versus a combination of individual and collective debriefing ("combined" approach). MATERIAL AND METHODS: Randomized, controlled multicenter trial. IPS sessions were randomized to have either standard or combined debriefing. Each team's performance in the IPS session was assessed with the Team Emergency Assessment Measure. The participants assessed the debriefing quality with the Debriefing Assessment for Simulation in Healthcare. RESULTS: Forty IPS sessions were randomized, and 30 were analyzed, 15 using standard collective debriefing and 15 the combined individual-collective method. Teams' performance improved with both types of debriefing, based on pre-post testing (P<.01), and there were no significant differences in overall performance scores between the 2 types of debriefing (P=.64). However, the combined approach was associated with higher scores for leadership skills (P<.05) and psychological safety, and the participants' learning experience was better (P<.05). CONCLUSION: During IPS courses on crisis resource management, debriefing improves participants' performance, but similar overall results can be obtained with both debriefing methods. Combined debriefing might be more effective for improving participants' leadership skills and psychological safety and also provide a better learning experience.

OBJETIVO: La simulación interprofesional (SIP) es eficaz para aprender gestión de recursos de crisis. La modalidad de debriefing utilizada en la SIP puede influir en el rendimiento de los participantes y en su integridad psicológica. Se evalúa y compara el rendimiento de un debriefing estándar (DE) ­colectivo­ con un debriefing combinado (DC) ­individual y colectivo­ en cursos de SIP en escenarios que simulan pacientes con patología aguda y grave. METODO: Ensayo controlado, aleatorizado y multicéntrico. Se aleatorizó el tipo de debriefing realizado (DE o DC) en las sesiones de SIP. El rendimiento del debriefing se evaluó con la escala TEAM (Team Emergency Assessment Measure). La calidad de la SIP fue valorada por los participantes con la escala DASH (Debriefing Assessment for Simulation in Healthcare©). RESULTADOS: Se aleatorizaron 40 cursos de SIP de los que se analizaron 30. Quince realizaron DE y 15 DC. Ambos grupos mejoraron entre la pre y la posprueba (p < 0,01), pero no hubo diferencias en el rendimiento global entre ambas modalidades de debriefing (p = 0,64). El DC obtuvo mejores resultados que el DE en la capacidad de liderazgo (p < 0,05), en la percepción de seguridad psicológica y en la experiencia de aprendizaje eficaz (p < 0,05). CONCLUSIONES: Durante la SIP en situaciones de crisis, el debriefing mejora el rendimiento de los participantes, sin diferencias entre un DE y un DC. El DC podría ser más efectivo para mejorar la capacidad de liderazgo, la seguridad psicológica y la experiencia del aprendizaje.

[Simulation of difficult airway management for residents: prospective comparative study]. / Simulação de manejo de via aérea difícil para residentes: estudo comparativo prospectivo.

BACKGROUND AND OBJECTIVES: Procedural simulation training for difficult airway management offers acquisition opportunities. The hypothesis was that 3 hours of procedural simulation training for difficult airway management improves: acquisition, behavior, and patient outcomes as reported 6 months later. METHODS: This prospective comparative study took place in two medical universities. Second-year residents of anesthesiology and intensive care from one region participated in 3h procedural simulation (intervention group). No intervention was scheduled for their peers from the other region (control). Prior to simulation and 6 months later, residents filled-out the same self-assessment form collecting experience with different devices. The control group filled-out the same forms simultaneously. The primary endpoint was the frequency of use of each difficult airway management device within groups at 6 months. Secondary endpoints included modifications of knowledge, skills, and patient outcomes with each device at 6 months. Intervention cost assessment was provided. RESULTS: 44 residents were included in the intervention group and 16 in the control group. No significant difference was observed for the primary endpoint. In the intervention group, improvement of knowledge and skills was observed at 6 months for each device, and improvement of patient outcomes was observed with the use of malleable intubation stylet and Eschmann introducer. No such improvement was observed in the control group. Estimated intervention cost was 406€ per resident. CONCLUSIONS: A 3h procedural simulation training for difficult airway management did not improve the frequency of use of devices at 6 months by residents. However, other positive effects suggest exploring the best ratio of time/acquisition efficiency with difficult airway management simulation. CLINICALTRIALS. GOV IDENTIFIER: NCT02470195.

Simulation of difficult airway management for residents: prospective comparative study / Simulação de manejo de via aérea difícil para residentes: estudo comparativo prospectivo

Abstract Background and objectives Procedural simulation training for difficult airway management offers acquisition opportunities. The hypothesis was that 3 hours of procedural simulation training for difficult airway management improves: acquisition, behavior, and patient outcomes as reported 6 months later. Methods This prospective comparative study took place in two medical universities. Second-year residents of anesthesiology and intensive care from one region participated in 3 h procedural simulation (intervention group). No intervention was scheduled for their peers from the other region (control). Prior to simulation and 6 months later, residents filled-out the same self-assessment form collecting experience with different devices. The control group filled-out the same forms simultaneously. The primary endpoint was the frequency of use of each difficult airway management device within groups at 6 months. Secondary endpoints included modifications of knowledge, skills, and patient outcomes with each device at 6 months. Intervention cost assessment was provided. Results 44 residents were included in the intervention group and 16 in the control group. No significant difference was observed for the primary endpoint. In the intervention group, improvement of knowledge and skills was observed at 6 months for each device, and improvement of patient outcomes was observed with the use of malleable intubation stylet and Eschmann introducer. No such improvement was observed in the control group. Estimated intervention cost was 406€ per resident. Conclusions A 3 h procedural simulation training for difficult airway management did not improve the frequency of use of devices at 6 months by residents. However, other positive effects suggest exploring the best ratio of time/acquisition efficiency with difficult airway management simulation. ClinicalTrials.gov Identifier NCT02470195.

Resumo Justificativa e objetivos O treinamento em simulação para o manejo de via aérea difícil oferece oportunidades de aprendizagem. A hipótese foi que um treinamento em simulação de procedimentos de três horas, para o manejo de via aérea difícil, melhoraria o aprendizado, o comportamento e os resultados dos pacientes, conforme relatado seis meses após o treinamento. Métodos Este estudo comparativo prospectivo foi realizado em duas universidades médicas. Residentes do segundo ano de anestesiologia e terapia intensiva de uma região participaram de um curso de três horas em simulação de procedimentos (grupo intervenção). Nenhuma intervenção foi programada para seus pares da outra região (grupo controle). Antes da simulação e seis meses após, os residentes preencheram a mesma ficha de autoavaliação sobre sua experiência com diferentes dispositivos. O grupo controle preencheu os mesmos formulários simultaneamente. O desfecho primário foi a frequência de uso de cada dispositivo para o manejo de via aérea difícil dentro dos grupos aos seis meses. Os pontos de corte secundários incluíram modificações em relação ao conhecimento, às habilidades e aos resultados dos pacientes com cada dispositivo aos seis meses. A avaliação do custo da intervenção foi registrada. Resultados Foram incluídos no grupo intervenção 44 residentes e 16 no grupo controle. Nenhuma diferença significativa foi observada para o ponto de corte primário. No grupo intervenção, a melhoria do conhecimento e das habilidades foi observada aos seis meses para cada dispositivo e a melhoria dos desfechos dos pacientes foi analisada com o uso de estilete maleável e do introdutor de Eschmann para intubação. Nenhuma melhoria foi observada no grupo controle. O custo da intervenção estimado foi de 406€ por residente. Conclusões Um treinamento simulado de três horas para o manejo de via aérea difícil não melhorou a frequência do uso de dispositivos pelos residentes aos seis meses. No entanto, outros efeitos positivos sugerem a exploração da melhor relação tempo/eficiência de aquisição de conhecimento com a simulação do manejo de via aérea difícil. ClinicalTrials.gov Identifier NCT02470195.

Residual anxiety after high fidelity simulation in anaesthesiology: An observational, prospective, pilot study.

BACKGROUND: High fidelity simulation (HFS) in anaesthesiology intentionally provides stress on students, but anxiety may be detrimental if it goes on through debriefing. The primary goal of this study was to estimate the proportion of students with significant anxiety remaining after debriefing (residual anxiety [RA]). Secondary goals were to evaluate the instructors' ability to estimate students' RA and to identify potential risk factors for high RA. SUBJECTS AND METHODS: Following IRB approval and informed consent, data from a cohort of subjects were prospectively collected by an independent expert. State-anxiety after debriefing (RA) was prospectively measured using the State-Trait Anxiety Inventory (a score varying from 20 to 80/80). RA was considered significant when≥36/80. Instructors simultaneously estimated the levels of subjects' RA via a visual analogue scale. Data about subjects, stress during scenarios (including continuous heart rate monitoring), and debriefings (including DASH© quality scores) were also collected. RESULTS: Seventy study subjects (30 residents, 26 nurses and 14 anaesthetists) were enrolled during 52 HFS sessions. As concerns the primary endpoint, RA was≥36/80 in 15 subjects (21%; 95% CI: 13-32). The median RA was 30/80 [25-35]. For secondary endpoints, the instructors' estimations poorly correlated with measurements: rho=0.36 (P<0.01); limits of agreement: -16 and 22. Subjects with RA≥36/80 had significantly higher trait-anxiety (P<0.01). An easy scenario (P=0.04) and low quality debriefing (P=0.04) were associated with higher RAs. CONCLUSION: Most students experienced low anxiety after debriefing. Instructors seem to be unable to reliably estimate students' RA. Students with an anxious personality are more likely to be anxious after debriefing.

Disagreement between pulse contour analysis and transpulmonary thermodilution for cardiac output monitoring after routine therapeutic interventions in ICU patients with acute circulatory failure.

BACKGROUND AND OBJECTIVE: The present prospective study was aimed at assessing the reliability of the pulse contour method for measuring cardiac output (CO) after different routinely used therapeutic interventions that can influence vascular compliance and systemic vascular resistances in ICU patients (fluid challenges, changes in norepinephrine or dobutamine infusion rates and changes in ventilatory settings). METHODS: In ICU patients requiring CO monitoring, transpulmonary thermodilution CO (COTD) and pulse contour CO (COPC) were measured with a PiCCO device after therapeutic manoeuvre-free periods (≤ and >1 h) and after therapeutic interventions without recalibration. RESULTS: Three hundred fifty-two sets of CO measurement pairs in 63 ICU patients were performed. The biases (and percentage errors) between COPC and COTD for the overall paired measurement, therapeutic manoeuvre-free periods and therapeutic manoeuvres were 0.20 ± 1.09 (33%), -0.01 ± 0.93 (29%) and 0.37 ± 1.18 (34%), respectively. The percentage errors were 36 and 39% for fluid challenges and changes in norepinephrine infusion rate, respectively. CONCLUSION: In ICU patients requiring therapeutic interventions, COPC is frequently in disagreement with COTD.