The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study.

PURPOSE: The new Medical College Admission Test (MCAT) was introduced in April 2015. This report presents findings from the first study of the validity of scores from the new MCAT exam in predicting student performance in the first year of medical school (M1). METHOD: The authors analyzed data from the national population of 2016 matriculants with scores from the new MCAT exam (N = 7,970) and the sample of 2016 matriculants (N = 955) from 16 medical schools who volunteered to participate in the validity research. They examined correlations of students' MCAT total scores and total undergraduate grade point averages (UGPAs), alone and together, with their summative performance in M1, and the success rate of students with different MCAT scores in their on-time progression to the second year of medical school (M2). They assessed whether MCAT scores provided comparable prediction of performance in M1 by students' race/ethnicity, socioeconomic background, and gender. RESULTS: Correlations of MCAT scores with summative performance in M1 ranged from medium to large. Although MCAT scores and UGPAs provided similar prediction of performance in M1, using both metrics provided better prediction than either alone. Additionally, students with a wide range of MCAT scores progressed to M2 on time. Finally, MCAT scores provided comparable prediction of performance in M1 for students from different sociodemographic backgrounds. CONCLUSIONS: This study provides early evidence that scores from the new MCAT exam predict student performance in M1. Future research will examine the validity of MCAT scores in predicting performance in later years.

Qualitative Analysis of Multiple Mini Interview Interviewer Comments.

OBJECTIVE: Qualitative studies of the Multiple Mini Interview (MMI) have investigated the attitudes and thoughts of prospective students and interviewers (i.e., raters) on the MMI interview, but none have examined rater's written assessments. Concerns regarding what the MMI measures, especially across and within each interview, have sparked investigations to determine how and what raters are measuring. Raters communicate their student evaluation(s) through numerical ratings and written comments that provide score context. This study explores rater's written comments to better understand the specific information gathered during the MMI process that contributes to interviewee evaluations. METHODS: Randomized data from two US medical schools were examined with no numerical scores or other information about the interviewee provided to reviewers. In reviewing the rater comments, common words and phrases were identified to help construct themes that characterized the content (domains). Authors reviewed each other's notes and comments regarding themes and worked together to verify themes for accuracy. RESULTS: Using a directed content approach to content analysis and reviewing the rater's comments, the results indicate that raters are focused on seven different domains: perspective taking, presentation, qualities, communication, coherence, comprehension, and non-verbal. Many of the rater comments contained multiple themes. CONCLUSION: Raters' MMI comments provide the context for numerical scores allowing admissions committees to more fully understand a candidate's strengths or weaknesses. Identifying the themes in rater comments can ultimately assist the admissions committee to more comprehensively understand assessment elements that raters are using and consider important during the MMI evaluation.

Hidden item variance in multiple mini-interview scores.

The extant literature has largely ignored a potentially significant source of variance in multiple mini-interview (MMI) scores by "hiding" the variance attributable to the sample of attributes used on an evaluation form. This potential source of hidden variance can be defined as rating items, which typically comprise an MMI evaluation form. Due to its multi-faceted, repeated measures format, reliability for the MMI has been primarily evaluated using generalizability (G) theory. A key assumption of G theory is that G studies model the most important sources of variance to which a researcher plans to generalize. Because G studies can only attribute variance to the facets that are modeled in a G study, failure to model potentially substantial sources of variation in MMI scores can result in biased estimates of variance components. This study demonstrates the implications of hiding the item facet in MMI studies when true item-level effects exist. An extensive Monte Carlo simulation study was conducted to examine whether a commonly used hidden item, person-by-station (p × s|i) G study design results in biased estimated variance components. Estimates from this hidden item model were compared with estimates from a more complete person-by-station-by-item (p × s × i) model. Results suggest that when true item-level effects exist, the hidden item model (p × s|i) will result in biased variance components which can bias reliability estimates; therefore, researchers should consider using the more complete person-by-station-by-item model (p × s × i) when evaluating generalizability of MMI scores.

Variance in attributes assessed by the multiple mini-interview.

INTRODUCTION: While the extant literature has explored the impact of stations on multiple mini- interview (MMI) scores, the influence of station scenarios has been largely overlooked. METHOD: A subset of MMI scores was purposively sampled from admissions data at one US medical school. Generalizability (G) theory was used to estimate variance components attributable to applicants and two facets of generalization - scenarios, the content of the station, and items, the attributes assessed. RESULTS: G study suggests that the greatest amount of variance is attributable to the main effect of the scenario (s) facet and the interaction between applicant and scenario (ps), which account for 77% of the total variance. The item facet (i) accounts for only 0.6% of total variance; likewise, the scenario-item interaction (si) accounts for only 1.4% of the total variance. DISCUSSION: While the researchers expected to find a large variance component associated with the scenario-item interaction, this analysis does not support this assumption. The researchers interpret the small scenario-item interaction as a result of variance attributable to the item facet being subsumed by the variance attributable to the content of the scenarios. CONCLUSIONS: The results of this study reinforce the need to examine psychometric properties of the MMI.

Medical students with low self-efficacy bolstered by calling to medical speciality.

This study was performed to understand the degree to which medical students' self-efficacy (SE) moderates the influence of calling on students' speciality commitment, emphasizing the need to understand variables that predict primary care specialization. The researchers hypothesized that students who perceived their career as a calling would be more committed to their speciality, especially when students had high SE. Medical students (Years 1-4; N = 152) completed an online survey to rate their calling, speciality commitment, and SE. Calling was measured by the Brief Calling scale (Dik et al., J Career Assess 20:242-263, 2012), while speciality choice was measured by Hollenbeck et al. (J Appl Psychol 74:18-23, 1989) measure of commitment. SE was measured by the Jerusalem and Schwarzer's general SE scale (see Scholz et al., Eur J Psychol Assess 18:242-51, 2002). Calling (r = 0.24, p < 0.01) and SE (r = 0.20, p < 0.05) were found to moderately correlate with speciality commitment, thus emphasizing the possibility that they may have an interaction. The interaction of calling and SE significantly predicted speciality commitment (ß = -0.20, t(148) = -2.55, p < 0.05) and explained a significant proportion of variance in speciality commitment (R (2) = 0.12, F(3, 148) = 6.875, p < 0.001). Students with a high presence of calling may have high speciality commitment, despite low SE.

Speciality interests and career calling to medicine among first-year medical students.

The construct of calling has recently been applied to the vocation of medicine. We explored whether medical students endorse the presence of a calling or a search for a calling and how calling related to initial speciality interest. 574 first-year medical students (84 % response rate) were administered the Brief Calling Survey and indicated their speciality interest. For presence of a calling, the median response was mostly true for: 'I have a calling to a particular kind of work' and moderately true for: 'I have a good understanding of my calling as it applies to my career'. For search for a calling, median response was mildly true: 'I am trying to figure out my calling in my career' and 'I am searching for my calling as it applies to my career'. Mann-Whitney U (p < 0.05) results indicate that students interested in primary care (n = 185) versus non-primary care (n = 389) are more likely to endorse the presence of a calling. Students were more likely to endorse the presence of a calling rather than a search for a calling, with those interested in primary care expressing stronger presence of a calling to medicine.

Differences in motives between Millennial and Generation X medical students.

OBJECTIVES Three domains comprise the field of human assessment: ability, motive and personality. Differences in personality and cognitive abilities between generations have been documented, but differences in motive between generations have not been explored. This study explored generational differences in medical students regarding motives using the Thematic Apperception Test (TAT). METHODS Four hundred and twenty six students (97% response rate) at one medical school (Generation X = 229, Millennials = 197) who matriculated in 1995 & 1996 (Generation X) or in 2003 & 2004 (Millennials) wrote a story after being shown two TAT picture cards. Student stories for each TAT card were scored for different aspects of motives: Achievement, Affiliation, and Power. RESULTS A multiple analysis of variance (p < 0.05) showed significant differences between Millennials' and Generation X-ers' needs for Power on both TAT cards and needs for Achievement and Affiliation on one TAT card. The main effect for gender was significant for both TAT cards regarding Achievement. No main effect for ethnicity was noted. CONCLUSIONS Differences in needs for Achievement, Affiliation and Power exist between Millennial and Generation X medical students. Generation X-ers scored higher on the motive of Power, whereas Millennials scored higher on the motives of Achievement and Affiliation.

Influences on specialty choice for students entering person-oriented and technique-oriented specialties.

BACKGROUND: Given changing trends in specialty choices among medical students coupled with continued challenges associated with medical specialty decision-making, it is important for medical educators to understand how students make decisions about their medical career. Medical educators should be aware of how medical school-based experiences and interactions such as faculty, courses, and services impact students' specialty choices and decisions. AIM: The study determined if differences in influences on specialty decision-making exist for students entering person-oriented specialties versus technique-oriented specialties. METHODS: Three hundred and fifty-six fourth-year medical students completed a questionnaire about how the faculty, curriculum, student services, mentoring and professional development programs, lifestyle considerations, family/friends, and other factors influenced their choice of specialty. RESULTS: Chi-square analyses indicated that students pursuing person-oriented specialties compared to technique-oriented specialties were moderately more likely to be influenced by their personal physician, by school faculty, and by medical school activities; slightly more likely to be influenced by medical school offices and services; and slightly less likely to be influenced by income expectations. CONCLUSIONS: Students interested in person-oriented specialties versus technique-oriented specialties indicate differences in what influences their specialty choice. This study may be helpful to medical educators and advisors who work with students on specialty decision-making.

Person-oriented versus technique-oriented specialties: early preferences and eventual choice.

BACKGROUND: Students' selection of a specialty is an important decision in their career as a physician. While distinguishing primary care physicians from non-primary care specialists has served a purpose for how medicine is practiced and managed, considering alternative ways of grouping specialties is appropriate when exploring specialty decisions. PURPOSE: This study explored how early specialty preferences correspond to eventual specialty choice using the person-oriented versus technique-oriented taxonomy. METHOD: Participants were 349 students who completed a career plan survey during the first semester of medical school and later graduated. RESULTS: Chi-square analysis showed a statistically significant difference between students' early preference for a person-oriented or technique-oriented specialty and the specialty they chose for their residency. CONCLUSION: Students with an early preference for person-oriented specialties were more likely to choose a person-oriented specialty, whereas students with an early preference for technique-oriented specialties were less likely to enter a technique-oriented specialty.

Comparing millennial and generation X medical students at one medical school.

PURPOSE: Two main generational cohorts comprising students enrolled in medical schools today are Generation Xers (born 1965-1980) and Millennial students (born 1981-1999). A subset is Cuspars (born 1975-1980), who share traits with both generations. Population theorists ascribe different personal characteristics, attitudes, and preferences to each group. The authors examined whether selected characteristics describing Generation X and Millennial students were quantifiable using a personality measure. Differences among Generation X, Millennial, and Cuspar medical students were investigated. METHOD: Eight hundred and nine medical students (399 females and 410 males) who matriculated between 1989-94 and 2001-04 at the Northeastern Ohio Universities College of Medicine completed the 16 Personality Factor Questionnaire (16PF). Differences in responses to the 16PF among the three generations were analyzed using multivariate analysis of variance (MANOVA). RESULTS: Analyses showed significant differences for Generation X versus Millennial students on 10 of the 16 personality factors. Millennial students scored significantly higher than Generation X students on factors including Rule-Consciousness, Emotional Stability, and Perfectionism; Generation X students scored higher than Millennials on Self-Reliance. Millennials also were significantly different from Generation Xers on several other factors. Significant differences were noted among Cuspars, Generation Xers, and Millennials. CONCLUSIONS: The 16PF is a useful tool to examine differences among these groups and to help understand the factors that constitute their personalities. Given differences among the generational groups, the authors forecast possible educational implications for medical school academic affairs and student services, and suggest areas for future research.

Personality and clinical skills: any correlation?

BACKGROUND: To determine correlations between personality factors and clinical skills of second-year medical school students. METHOD: Participants were 206 medical students who had completed the Sixteen Personality Factor Questionnaire (16PF) and the Clinical Skills Assessment II (CSA II). RESULTS: of the 16PF and CSA II were analyzed using Pearson R. Results Overall CSA II score correlated positively with Warmth and negatively with Abstractedness and Privateness. Communication skills correlated positively with Warmth, Emotional Stability, and Perfectionism and negatively with Privateness. Data gathering correlated positively with Warmth and negatively with Abstractedness. Physical exam and Case presentation subtests had no correlates. CONCLUSION: The findings of this study suggest that a relationship may exist between personality and clinical skills.