Construct validity of multiple mini interviews - Investigating the role of stations, skills, and raters using Bayesian G-theory.

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures.Methods: We used data from three MMI samples (N = 376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills.Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures.Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.

Medical doctors' job specification analysis: A qualitative inquiry.

Purpose: A qualitative inquiry was conducted to investigate the qualification requirements of medical doctors in different professional fields and from different perspectives. The inquiry was part of an empirical workplace analysis. Methods: Seventy-four structured interviews were conducted and analyzed to examine critical incidents and behaviors of medical doctors working in different professional fields (clinical theory, clinical practice, practitioner) and disciplines, and from three different perspectives (medical doctors, non-medical staff, and patients). In addition, the National Competency-based Catalogue of Learning Objectives for Medical Education (Nationaler Kompetenzbasierter Lernzielkatalog Medizin / NKLM) was used. Results: The results revealed eleven relevant competencies, which could be categorized into three superordinate competence clusters: interpersonal, work-related, and self-related. The perspectives of medical doctors and non-medical staff included all eleven competencies. However, the perspective of patients did not include one interpersonal and two self-related competencies. Nearly all of the critical behaviors mentioned are included in the NKLM. However, the NKLM also includes behaviors that were not mentioned in the interviews. Conclusions: The behavior-oriented interviews resulted in a requirement profile that is very similar in structure to other competency models in occupational contexts. Comparisons of the different perspectives predominantly revealed similarities. However, the patient perspective also revealed interesting differences compared to the perspectives of medical doctors and non-medical staff. The behavior-related results of the interviews can be directly used for the development of exercises in selection and personnel development contexts and for potential appraisals specific to different medical disciplines. In future steps, the results of this initial qualitative step are to be replicated and extended using quantitative studies and a representative sample. The main overall objective is the definition of relevant competencies both for the selection and development of medical students and for the design of potential appraisals as part of personnel development programs in different medical disciplines.