Construct validity of multiple mini interviews - Investigating the role of stations, skills, and raters using Bayesian G-theory.

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures.Methods: We used data from three MMI samples (N = 376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills.Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures.Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.

[Implementation and evaluation of a revised curriculum for psychiatry and psychotherapy]. / Einführung und Evaluation eines neuen Kurrikulums Psychiatrie und Psychotherapie.

BACKGROUND: Medical education in the discipline of psychiatry and psychotherapy at the University of Münster was traditionally focused on the transfer of knowledge via lectures. According to the current guidelines, the medical curriculum was modified as from the winter semester 2016/2017 to be more competency-based and the changes were evaluated. OBJECTIVE: Lectures and seminars were reduced to achieve a better linkage between theoretical and practical knowledge. Moreover, learning goals were formulated based on the German National Competence-based Catalogue of Learning Objectives in Medicine (NKLM) and entrustable professional activities (EPAs). MATERIAL AND METHODS: Almost all previous lectures are now replaced by an inverted classroom concept with e­learning. Theoretical knowledge is deepened by immediate multiple choice (MC) examinations and a seminar, which now focusses on specific practical EPAs. At the end of the semester, the students now undergo a practical, formative examination with simulated patients (actors) in addition to the former MC test. For evaluation, a representative sample of a semester cohort which took part in the previous curriculum and a similar cohort which attended the revised curriculum were investigated. Moreover, variables which might have an impact on the results were assessed, e. g. pre-existing psychiatric knowledge and motivation. RESULTS: Students taught by the modified curriculum showed a significantly better practical performance and no reduction of theoretical knowledge. Relevant influencing factors were not identified. CONCLUSION: The results show that a competency-based modification of the curriculum in the discipline of psychiatry and psychotherapy leads to more practical abilities and thus helps future physicians to be more self-determined.

Medical doctors' job specification analysis: A qualitative inquiry.

Purpose: A qualitative inquiry was conducted to investigate the qualification requirements of medical doctors in different professional fields and from different perspectives. The inquiry was part of an empirical workplace analysis. Methods: Seventy-four structured interviews were conducted and analyzed to examine critical incidents and behaviors of medical doctors working in different professional fields (clinical theory, clinical practice, practitioner) and disciplines, and from three different perspectives (medical doctors, non-medical staff, and patients). In addition, the National Competency-based Catalogue of Learning Objectives for Medical Education (Nationaler Kompetenzbasierter Lernzielkatalog Medizin / NKLM) was used. Results: The results revealed eleven relevant competencies, which could be categorized into three superordinate competence clusters: interpersonal, work-related, and self-related. The perspectives of medical doctors and non-medical staff included all eleven competencies. However, the perspective of patients did not include one interpersonal and two self-related competencies. Nearly all of the critical behaviors mentioned are included in the NKLM. However, the NKLM also includes behaviors that were not mentioned in the interviews. Conclusions: The behavior-oriented interviews resulted in a requirement profile that is very similar in structure to other competency models in occupational contexts. Comparisons of the different perspectives predominantly revealed similarities. However, the patient perspective also revealed interesting differences compared to the perspectives of medical doctors and non-medical staff. The behavior-related results of the interviews can be directly used for the development of exercises in selection and personnel development contexts and for potential appraisals specific to different medical disciplines. In future steps, the results of this initial qualitative step are to be replicated and extended using quantitative studies and a representative sample. The main overall objective is the definition of relevant competencies both for the selection and development of medical students and for the design of potential appraisals as part of personnel development programs in different medical disciplines.

Mastery learning improves students skills in inserting intravenous access: a pre-post-study.

OBJECTIVE: Inserting peripheral venous catheters (PVCs) has been identified as a core competency for medical students. Because the performance - even of hygienic standards - of both students and novice physicians is frequently inadequate, medical faculties must focus on competence-based learning objectives and deliberate practice, features that are combined in mastery learning. Our aim was to determine the competency of students in inserting PVCs before and after an educational intervention. DESIGN: This study comprised a skills assessment with pre- and post-tests of a group of third-year students who received a simulation-based intervention. A newly established curriculum involved one hour of practice at inserting PVCs on simulators. Students were required to pass a test (total 21 points, pass mark 20 points) developed on the concept of mastery learning. An unannounced follow-up test was performed one week (8 days) after the intervention. SETTING: The simulation center of the medical faculty in Muenster. PARTICIPANTS: Third-year students who received the intervention. RESULTS: One hundred and nine complete data sets were obtained from 133 students (82.5%). Most students (97.2%) passed the test after the intervention (mean score increase from 15.56 to 20.50, P<0.001). There was a significant decrease in students' performance after one week (8 days): only 74.5% of participants passed this retest (mean score reduction from 20.50 to 20.06, P<0.001). CONCLUSION: Mastery learning is an effective form of teaching practical skills to medical students, allowing a thorough preparation for the challenges of daily clinical practice.

Measuring the ambiguity tolerance of medical students: a cross-sectional study from the first to sixth academic years.

BACKGROUND: Tolerance of ambiguity, or the extent to which ambiguous situations are perceived as desirable, is an important component of the attitudes and behaviors of medical students. However, few studies have compared this trait across the years of medical school. General practitioners are considered to have a higher ambiguity tolerance than specialists. We compared ambiguity tolerance between general practitioners and medical students. METHODS: We designed a cross-sectional study to evaluate the ambiguity tolerance of 622 medical students in the first to sixth academic years. We compared this with the ambiguity tolerance of 30 general practitioners. We used the inventory for measuring ambiguity tolerance (IMA) developed by Reis (1997), which includes three measures of ambiguity tolerance: openness to new experiences, social conflicts, and perception of insoluble problems. RESULTS: We obtained a total of 564 complete data sets (return rate 90.1%) from medical students and 29 questionnaires (return rate 96.7%) from general practitioners. In relation to the reference groups defined by Reis (1997), medical students had poor ambiguity tolerance on all three scales. No differences were found between those in the first and the sixth academic years, although we did observe gender-specific differences in ambiguity tolerance. We found no differences in ambiguity tolerance between general practitioners and medical students. CONCLUSIONS: The ambiguity tolerance of the students that we assessed was below average, and appeared to be stable throughout the course of their studies. In contrast to our expectations, the general practitioners did not have a higher level of ambiguity tolerance than the students did.