Safety culture and the 5 steps to safer surgery: an intervention study.

BACKGROUND: Improvements in safety culture have been postulated as one of the mechanisms underlying the association between the introduction of the World Health Organisation (WHO) Surgical Safety Checklist with perioperative briefings and debriefings, and enhanced patient outcomes. The 5 Steps to Safer Surgery (5SSS) incorporates pre-list briefings, the three steps of the WHO Surgical Safety Checklist (SSC) and post-list debriefings in one framework. We aimed to identify any changes in safety culture associated with the introduction of the 5SSS in orthopaedic operating theatres. METHODS: We assessed the safety culture in the elective orthopaedic theatres of a large UK teaching hospital before and after introduction of the 5SSS using a modified version of the Safety Attitude Questionnaire - Operating Room (SAQ-OR). Primary outcome measures were pre-post intervention changes in the six safety culture domains of the SAQ-OR. We also analysed changes in responses to two items regarding perioperative briefings. RESULTS: The SAQ-OR survey response rate was 80% (60/75) at baseline and 74% (53/72) one yr later. There were significant improvements in both the reported frequency (P<0.001) and perceived importance (P=0.018) of briefings, and in five of the six safety culture domain scores (Working Conditions, Perceptions of Management, Job Satisfaction, Safety Climate and Teamwork Climate) of the SAQ-OR (P<0.001 in all cases). Scores in the sixth domain (Stress Recognition) decreased significantly (P=0.028). CONCLUSIONS: Implementation of the 5SSS was associated with a significant improvement in the safety culture of elective orthopaedic operating theatres.

The relative reliability of actively participating and passively observing raters in a simulation-based assessment for selection to specialty training in anaesthesia.

Selection to specialty training is a high-stakes assessment demanding valuable consultant time. In one initial entry level and two higher level anaesthesia selection centres, we investigated the feasibility of using staff participating in simulation scenarios, rather than observing consultants, to rate candidate performance. We compared participant and observer scores using four different outcomes: inter-rater reliability; score distributions; correlation of candidate rankings; and percentage of candidates whose selection might be affected by substituting participants' for observers' ratings. Inter-rater reliability between observers was good (correlation coefficient 0.73-0.96) but lower between participants (correlation coefficient 0.39-0.92), particularly at higher level where participants also rated candidates more favourably than did observers. Station rank orderings were strongly correlated between the rater groups at entry level (rho 0.81, p < 0.001) but weaker at the two higher level centres (rho 0.52, p = 0.018; rho 0.58, p = 0.001). Substituting participants' for observers' ratings had less effect once scores were combined with those from other selection centre stations. Selection decisions for 0-20% of candidates could have changed, depending on the numbers of training posts available. We conclude that using participating raters is feasible at initial entry level only.

Development of workplace-based assessments of non-technical skills in anaesthesia.

Non-technical skills are recognised as crucial to good anaesthetic practice. We designed and evaluated a specialty-specific tool to assess non-technical aspects of trainee performance in theatre, based on a system previously found reliable in a recruitment setting. We compared inter-rater agreement (multir-ater kappa) for live assessments in theatre with that in a selection centre and a video-based rater training exercise. Twenty-seven trainees participated in the first in-theatre assessment round and 40 in the second. Round- 1 scores had poor inter-rater agreement (mean kappa = 0.20) and low reliability (generalisability coefficient G = 0.50). A subsequent assessor training exercise showed good inter-rater agreement, (mean kappa = 0.79) but did not improve performance of the assessment tool when used in round 2 (mean kappa = 0.14, G = 0.42). Inter-rater agreement in two selection centres (mean kappa = 0.61 and 0.69) exceeded that found in theatre. Assessment tools that perform reliably in controlled settings may not do so in the workplace.

Predictive validity of a selection centre testing non-technical skills for recruitment to training in anaesthesia.

BACKGROUND: Assessment centres are an accepted method of recruitment in industry and are gaining popularity within medicine. We describe the development and validation of a selection centre for recruitment to speciality training in anaesthesia based on an assessment centre model incorporating the rating of candidate's non-technical skills. METHODS: Expert consensus identified non-technical skills suitable for assessment at the point of selection. Four stations-structured interview, portfolio review, presentation, and simulation-were developed, the latter two being realistic scenarios of work-related tasks. Evaluation of the selection centre focused on applicant and assessor feedback ratings, inter-rater agreement, and internal consistency reliability coefficients. Predictive validity was sought via correlations of selection centre scores with subsequent workplace-based ratings of appointed trainees. RESULTS: Two hundred and twenty-four candidates were assessed over two consecutive annual recruitment rounds; 68 were appointed and followed up during training. Candidates and assessors demonstrated strong approval of the selection centre with more than 70% of ratings 'good' or 'excellent'. Mean inter-rater agreement coefficients ranged from 0.62 to 0.77 and internal consistency reliability of the selection centre score was high (Cronbach's α=0.88-0.91). The overall selection centre score was a good predictor of workplace performance during the first year of appointment. CONCLUSIONS: An assessment centre model based on the rating of non-technical skills can produce a reliable and valid selection tool for recruitment to speciality training in anaesthesia. Early results on predictive validity are encouraging and justify further development and evaluation.

Anaesthetic machine checking guidelines: have we improved our practice?

BACKGROUND AND OBJECTIVES: This study follows up an initial audit in 1992 indicating that anaesthetic machine checking practices were often incomplete. The aims were to ascertain if there has been any improvement during this period with special reference to the latest guidelines. METHODS: Following the Association of Anaesthetists of Great Britain and Ireland machine checking guidelines, a structured questionnaire, was used to interview 41 anaesthetists in the South West region on one particular day. RESULTS: Despite 80% of subjects stating that they had read the latest guidelines recently, only one undertook a complete check and 39/41 (95%) performed partial checks (omitting one or more steps in the guidelines). Steps most commonly omitted were additional monitoring, ventilator function, availability of an alternative means of ventilation and function of ancillary equipment such as laryngoscopes. Only 5/41 subjects performed any check between cases. Several of these checks have been introduced in the 2004 guidelines. CONCLUSIONS: Although there has been a significant increase in the proportion of anaesthetists undertaking machine checks since 1992 (P = 0.0007), we conclude that machine checking guidelines are still poorly followed, with checks specific to the latest guidelines most likely to be omitted.