Non-technical skills of Norwegian medical students at different training sites: a comparative, observational cohort study.

PURPOSE: Mastering non-technical skills (NTS) is a fundamental part of the training of new physicians to perform effectively and safely in the medical practice environment. Ideally, they learn these skills during medical school. Decentralized medical education is being implemented increasingly worldwide. Two of the three training sites studied, Bodø (a regional hospital) and Finnmark (a rural local hospital), implemented decentralized medical education. The third training site was the main campus in Tromsø, located at an urban university hospital. The training in Finnmark emphasised training in non-technical skills using simulation to a larger extent than the two other university campuses. This study aimed to compare the NTS performance of medical students in their last year of education at three different training sites of the same university. METHODS: This blinded cohort study included students from the three training sites who participated in identical multi-professional simulations over a six-year period. Eight raters evaluated the video recordings of eight students from each training site using the Norwegian Medical Students Non-Technical Skills (NorMS-NTS) tool. The NorMS-NTS tool, which comprises four categories and 13 elements, assesses the NTS of Norwegian medical students and assigns an overall global score. Pairwise significant differences in the NTS performance levels between the training sites studied were assessed using Tukey's test. RESULTS: The overall NTS performance levels of the medical students from Finnmark (mean 4.5) were significantly higher than those of the students from Tromsø (mean 3.8) and Bodø (mean 3.5). Similarly, the NTS performance levels at category-level of the students in Finnmark were significantly higher than those of the students from Bodø and Tromsø. Except for one category, no significant differences were observed between the students from Bodø and Tromsø in terms of the overall or category-level NTS performance. CONCLUSION: The NTS performance levels of the medical students from Finnmark, which implements rural, decentralized medical education, were significantly higher than those of the students from Tromsø and Bodø.

Collecting evidence of validity for an assessment tool for Norwegian medical students' non-technical skills (NorMS-NTS): usability and reliability when used by novice raters.

BACKGROUND: The NorMS-NTS tool is an assessment tool for assessing Norwegian medical students' non-technical skills (NTS). The NorMS-NTS was designed to provide student feedback, training evaluations, and skill-level comparisons among students at different study sites. Rather than requiring extensive rater training, the tool should capably suit the needs of busy doctors as near-peer educators. The aim of this study was to examine the usability and preliminary assess validity of the NorMS-NTS tool when used by novice raters. METHODS: This study focused on the usability of the assessment tool and its internal structure. Three raters used the NorMS-NTS tool to individually rate the team leader, a medical student, in 20 video-recorded multi-professional simulation-based team trainings. Based on these ratings, we examined the tools' internal structure by calculating the intraclass correlation coefficient (ICC) (version 3.1) interrater reliability, internal consistency, and observability. After the rating process was completed, the raters answered a questionnaire about the tool's usability. RESULTS: The ICC agreement and the sum of the overall global scores for all raters were fair: ICC (3,1) = 0.53. The correlation coefficients for the pooled raters were in the range of 0.77-0.91. Cronbach's alpha for elements, categories and global score were mostly above 0.90. The observability was high (95%-100%). All the raters found the tool easy to use, none of the elements were redundant, and the written instructions were helpful. The raters also found the tool easier to use once they had acclimated to it. All the raters stated that they could use the tool for both training and teaching. CONCLUSIONS: The observed ICC agreement was 0.08 below the suggested ICC level for formative assessment (above 0.60). However, we know that the suggestion is based on the average ICC, which is always higher than a single-measure ICC. There are currently no suggested levels for single-measure ICC, but other validated NTS tools have single-measure ICC in the same range. We consider NorMS-NTS as a usable tool for formative assessment of Norwegian medical students' non-technical skills during multi-professional team training by raters who are new to the tool. It is necessary to further examine validity and the consequences of the tool to fully validate it for formative assessments.

The development of a tool to assess medical students' non-technical skills - The Norwegian medical students' non-technical skills (NorMS-NTS).

PURPOSE: New physicians need to master non-technical skills (NTS), as high levels of NTS have been shown to increase patient safety. It has also been shown that NTS can be improved through training. This study aimed to establish the necessary NTS for Norwegian medical students to create a tool for formative and summative assessments. METHODS: Focus group interviews were conducted with colleagues and patients of newly graduated physicians. Interviews were then analyzed using card sort methods, and the identified NTS were used to establish a framework. Focus groups commented on a prototype of an NTS assessment tool. Finally, we conducted a search of existing tools and literature. The final tool was developed based on the combined inputs. RESULTS: We created Norwegian medical students' non-technical skills (NorMS-NTS) assessment tool containing four main categories; together comprising 13 elements and a rating scale for the NTS of the person observed. CONCLUSIONS: The NorMS-NTS represents a purpose-made tool for assessing newly graduated physicians' NTS. It is similar to existing assessment tools but based on domain-specific user perspectives obtained through focus group interviews and feedback, integrated with results from a literature search, and with consideration of existing NTS tools.