High-stakes assessment of the non-technical skills of critical care trainees using simulation: feasibility, acceptability and reliability.

OBJECTIVE: To evaluate the use of high-fidelity simulation for summative high-stakes assessment of intensive care trainees, focusing on non-technical skills (NTS), testing feasibility and acceptability of simulation assessment, and the reliability of two NTS rating scales. DESIGN, SETTING AND PARTICIPANTS: Prospective observational study of senior intensive care trainees in a simulated specialist examination. METHODS: Participants undertook a simulated patient management scenario and were assessed using two rating scales: the Anaesthesia Non-technical Skills (ANTS) scale and the Ottawa Global Rating Scale (GRS). Assessors were trained, currently active, high-stakes examiners. Participants also completed a survey on simulation-based summative assessment. OUTCOME MEASURES: The inter-rater reliability of two rating scales for NTS assessment. We evaluated the feasibility of simulation-based assessment, and used survey results to assess acceptability to participants. RESULTS: Simulation assessment was feasible. Participants considered simulation-based high-stakes assessment to be acceptable and felt their scenario performance was reflective of real-world performance. Participants identified a need for debriefing following scenario-based assessment. Inter-rater reliability was fair for the ANTS and Ottawa GRS scores (intra-class correlation coefficient, 0.39 and 0.42, respectively). There was only fair agreement between raters for an NTS pass or fail (weighted kappa, 0.32) and for a technical skills pass or fail (weighted kappa, 0.36). CONCLUSIONS: Summative high-stakes assessment using a single simulated scenario was feasible and acceptable to senior intensive care trainees. The low inter-rater reliability for the ANTS and Ottawa GRS rating scales and for pass or fail discrimination may limit its incorporation into an existing examination format.

Handover--Enabling Learning in Communication for Safety (HELiCS): a report on achievements at two hospital sites.

Clinical handover is an area of critical concern, because deficiencies in handover pose a patient safety risk. Redesign of handover must allow for input from frontline staff to ensure that designs fit into existing practices and settings. The HELiCS (Handover--Enabling Learning in Communication for Safety) tool uses a "video-reflexive" technique: handover encounters are videotaped and played back to the practitioners involved for analysis and discussion. Using the video-reflexive process, staff of an emergency department and an intensive care unit at two different tertiary hospitals redesigned their handover processes. The HELiCS study gave staff greater insight into previously unrecognised clinical and operational problems, enhanced coordination and efficiency of care, and strengthened junior-senior communication and teaching. Our study showed that reflexive and "bottom-up" handover redesign can produce outcomes that harbour local fit, practitioner ownership and (to date) sustainability.