The change from UMAT to UCAT for undergraduate medical school applicants: impact on selection outcomes.

OBJECTIVES: To assess whether the change from the Undergraduate Medical and Health Sciences Admissions Test (UMAT; 1991-2019) to the University Clinical Aptitude Test (UCAT) for the 2020 New South Wales undergraduate medical degree intake was associated with changes in the impact of sex, socio-economic status and remoteness of residence, and professional coaching upon selection for interview. DESIGN, SETTING, PARTICIPANTS: Cross-sectional study of applicants for the three NSW undergraduate medical programs for entry in 2019 (4114 applicants) or 2020 (4270); 703 people applied for both intakes. Applicants selected for interview were surveyed about whether they had received professional coaching for the selection test. MAIN OUTCOME MEASURES: Scores on the three sections of the UMAT (2019 entry cohort) and the five subtests of the UCAT (2020 entry); total UMAT and UCAT scores. RESULTS: Mean scores for UMAT 1 and 3 and for all four UCAT cognitive subtests were higher for men than women; the differences were statistically significant after adjusting for age, socio-economic status, and remoteness. The effect size for sex was 0.24 (95% CI, 0.18-0.30) for UMAT total score, 0.38 (95% CI, 0.32-0.44) for UCAT total score. For the 2020 intake, 2303 of 4270 applicants (53.9%) and 476 of 1074 interviewees (44.3%) were women. The effect size for socio-economic status was 0.47 (95% CI, 0.39-0.54) for UMAT, 0.43 (95% CI, 0.35-0.50) for UCAT total score; the effect size for remoteness was 0.54 (95% CI, 0.45-0.63) for UMAT, 0.48 (95% CI, 0.39-0.58) for UCAT total score. The impact of professional coaching on UCAT performance was not statistically significant among those accepted for interview. CONCLUSIONS: Women and people from areas outside major cities or of lower socio-economic status perform less well on the UCAT than other applicants. Reviewing the test and applicant quotas may be needed to achieve selection equity.

Validación de checklist para evaluación de la capacitación con simulación clínica de la atención al paciente séptico / Checklist validation for evaluation of training with clinical simulation of septic patient care / Validação de checklist para avaliação da capacitação com simulação clínica do atendimento ao paciente séptico

Objetivo: Construir y validar el contenido de un checklist para evaluación de la capacitación de profesionales del área de la salud en la atención al paciente séptico con simulación clínica. Método: Estudio de validación metodológica de instrumento, con delineamiento estructural en dos etapas: construcción del instrumento y validación de contenido del checklist utilizando la técnica Delphi en dos rondas. Resultados: La validación de contenido fue compuesta por diez ítems y cuarenta y tres subítems analizados por los evaluadores. A través del Índice de Validez de Contenido, se identificaron cuatro ítems con fuerte evidencia de validación, Índice de Validez de Contenido ≥ 0,8. En la segunda ronda Delphi presentó un porcentaje de concordancia superior al 80% para todas las variables pertinentes al instrumento. Se reestructuró el checklist según recomendaciones de los evaluadores, manteniendo los diez ítems, pero con reducción para veintiséis subítems, que en la segunda ronda Delphi presentó un porcentaje de concordancia superior al 80% para todas las variables pertinentes al instrumento. Conclusión: Método fue eficaz para validar el contenido de un checklist que evaluará la capacitación de profesionales de la salud en la atención al paciente séptico, por medio de simulación clínica

Aims: Development and validation of the contents of a checklist to evaluate the qualification of health professionals in septic patient care with clinical simulation. Method: instrument validation study, with two-stage structural design: instrument construction and validation of the checklist contents using the Delphi technique in two rounds. Results: The content validation was composed of ten items and forty-three sub-items analyzed by the evaluators. Through the Content Validity Index, four items with strong validation evidence were identified, Content Validity Index ≥ 0.8. We restructured the checklist according to the evaluators recommendations, maintaining the ten items, but reducing them to twenty-six sub-items, which in the second round Delphi presented a percentage of agreement above 80% for all variables relevant to the instrument. Conclusion: This method was effective to validate the checklist contents that will evaluate the qualification of health professionals in septic patient care, through clinical simulation

Objetivos: Construir e validar o conteúdo de um checklist para avaliação da capacitação de profissionais da área da saúde no atendimento ao paciente séptico com simulação clínica. Método: Estudo de validação metodológica de instrumento, com delineamento estrutural em duas etapas: construção do instrumento e validação de conteúdo do checklist utilizando a técnica Delphi em duas rodadas. Resultados: A validação de conteúdo foi composta por dez itens e quarenta e três subitens analisados pelos avaliadores. Por meio do Índice de Validade de Conteúdo, identificaram-se quatro itens com forte evidência de validação, Índice de Validade de Conteúdo ≥ 0,8. Reestruturou o checklist conforme recomendações dos avaliadores, mantendo os dez itens, porém com redução para vinte e seis subitens, que na 2ª rodada Delphi apresentou percentual de concordância acima de 80% para todas as variáveis pertinentes ao instrumento. Conclusão: Método foi eficaz para validar o conteúdo de um checklist que avaliará a capacitação de profissionais da saúde no atendimento ao paciente séptico, por meio de simulação clínica

Does 'online confidence' predict application success and later academic performance in medical school? A UK-based national cohort study.

OBJECTIVES: The UK Clinical Aptitude Test (UKCAT) previously piloted an assessment of 'online confidence', where candidates were asked to indicate how confident they were with their answers. This study examines the relationship between these ratings, the odds of receiving an offer to study medicine and subsequent undergraduate academic performance. DESIGN: National cohort study. SETTING: UK undergraduate medical selection. PARTICIPANTS: 56 785 UKCAT candidates who sat the test between 2013 and 2016 and provided valid responses to the online confidence pilot study. PRIMARY OUTCOME MEASURES: Two measures of 'online confidence' were derived: the well-established 'confidence bias', and; a novel 'confidence judgement' measure, developed using Item Response Theory in order to derive a more sophisticated metric of the ability to evaluate one's own performance on a task. Regression models investigated the relationships between these confidence measures, application success and academic performance. RESULTS: Online confidence was inversely related to cognitive performance. Relative underconfidence was associated with increased odds of receiving an offer to study medicine. For 'confidence bias' this effect was independent of potential confounders (OR 1.48, 1.15 to 1.91, p=0.002). While 'confidence judgement' was also a univariable predictor of application success (OR 1.22, 1.01 to 1.47, p=0.04), it was not an independent predictor. 'Confidence bias', but not 'confidence judgement', predicted the odds of passing the first year of university at the first attempt, independently of cognitive performance, with relative underconfidence positively related to academic success (OR 3.24, 1.08 to 9.72, p=0.04). No non-linear effects were observed, suggesting no 'sweet spot' exists in relation to online confidence and the outcomes studied. CONCLUSIONS: Applicants who either appear underconfident, or are better at judging their own performance on a task, are more likely to receive an offer to study medicine. However, online confidence estimates had limited ability to predict subsequent academic achievement. Moreover, there are practical challenges to evaluating online confidence in high-stakes selection.

Implantación del Examen Clínico Objetivo y Estructurado en la Facultad de Medicina de la Universidad Autónoma de Madrid / Implementation of the Objective Structured Clinical Examination (OSCE) in the Madrid Autonomous University Medical School

El examen clínico objetivo y estructurado (ECOE) evalúa las competencias adquiridas por los estudiantes al terminar el grado de medicina. La competencia es el conjunto de conocimientos y habilidades para resolver un problema profesional, con una actitud adecuada. La Conferencia de decanos de las 40 facultades de medicina de España ha establecido que todos los estudiantes del grado de medicina que finalizasen sus estudios en el curso 2015/2016 deberían superar una prueba (ECOE) que evaluase 8 competencias clínicas: anamnesis; exploración física; comunicación; habilidades técnicas; juicio clínico, manejo diagnóstico y tratamiento; prevención y promoción de la salud; relaciones interprofesionales; y aspectos éticos y legales. La Facultad de Medicina de la Universidad Autónoma de Madrid se ha incorporado a este proyecto y ha organizado su primer ECOE para los 197 alumnos de 6.° curso, con 25 estaciones o pruebas. Este trabajo describe las 7 etapas o procesos esenciales que se han seguido para la instauración del ECOE y que puede resultar de utilidad para la comprensión de este complejo examen y su implantación

The Objective Structured Clinical Examination (OSCE) evaluates clinical skills of medical students at the end of medical school training. Professional competence is the set of knowledge and clinical skills that helps to solve professional problems with the correct attitude. The Conference of Deans from all 40 Schools of Medicine in Spain agreed that all students completing their medical training in the 2015/16 academic year should take an exam to assess 8 competency areas: medical history, physical examination, communication skills, clinical judgment, diagnosis, management and treatment, prevention and health promotion, inter-professional relationships, and ethical and legal aspects. The Madrid Autonomous University Medical School took part in this national project, and conducted its first OSCE, with 25 tests, for the 197 students that concluded their training in 2016. This paper describes the 7 stages, or essential processes followed in order to establish the OSCE. This experience may be useful for the understanding and implementation of this complex competence evaluation process

Effectiveness of "fill in the blanks" over multiple choice questions in assessing final year dental undergraduates / La eficacia de «rellenar los espacios en blanco» sobre las preguntas de opción múltiple en la evaluación de estudiantes de último año de odontología

BACKGROUND: Possibility of guessing in Multiple Choice questions (MCQ) when assessing undergraduates is considered a weakness. There are limited studies on the use of "Fill in the Blanks" (FIB) to overcome this issue. OBJECTIVE: To assess the effectiveness of FIB in MCQ for assessing final year dental undergraduates. METHODS AND MATERIALS: A total of 134 final year dental undergraduates were randomly assigned to Group A and B. Group A was given a questionnaire with fifteen single best answer MCQ questions, and then the FIB questionnaire (which included the same questions in FIB form). At the same time Group B was given the FIB questionnaire initially, and then the MCQ questionnaire in the given period of time. The mean scores of the two groups were then compared. RESULTS: Group A obtained a mean score of 10.94 (SD±3.203) for MCQ, and 10.48 (SD±2.993) for FIB, whereas Group B obtained a mean score of 6.8 (SD±2.949) for FIB and 10.05 (SD±2.619) for MCQ. There was a statistically significant difference in the mean scores obtained for the two types of tests between Group A (P=.04) and Group B (P=.0001). The difference in the mean scores obtained for the FIB were statistically significant (P=.0001) between the groups, whereas the results were not statistically significant for MCQ (P=.127). CONCLUSION: MCQ results revealed that the knowledge of the two groups was similar. The differences in the scores obtained for the two types of assessment tools suggest further research is needed to investigate the factors that led to the above observation

ANTECEDENTES: Posibilidad de valorar si las preguntas de opción múltiple (MOM) son el punto débil cuando se evalúa a los estudiantes de último año. Hay estudios limitados sobre el uso de «rellenar los espacios en blanco» (REB) para superar este problema. OBJETIVO: Evaluar la eficacia de REB sobre MOM en la evaluación de los estudiantes de último año de odontología. MÉTODOS Y MATERIALES: Se asignó aleatoriamente a un total de 134 estudiantes universitarios de último año de odontología a los grupos A y B. Al grupo A se le entregó un cuestionario con 15 MOM para seleccionar la mejor opción y luego un cuestionario REB (que incluía las mismas preguntas con planteamiento REB). Al mismo tiempo, al grupo B se le entregó el cuestionario REB inicialmente y luego el cuestionario MOM en el período de tiempo determinado. A continuación se compararon las notas medias de los 2 grupos. RESULTADOS: Un grupo obtuvo una puntuación media de 10,94 (DE±3,203) en MOM y de 10,48 (DE±2,993) en el REB, mientras que el grupo B obtuvo una puntuación media de 6,8 (DE±2,949) en el REB y de 10,05 (DE±2,619) en el MOM. Hubo una diferencia estadísticamente significativa entre las notas medias obtenidas de los 2 tipos de pruebas entre el grupo A (p = 0,04) y el grupo B (p = 0,0001). La diferencia en las notas medias obtenidas del REB fue estadísticamente significativa (p = 0,0001) entre los grupos, mientras que los resultados no fueron estadísticamente significativos en el MOM (p = 0,127). CONCLUSIÓN: Los resultados del MOM revelaron que el conocimiento de los 2 grupos era similar. Las diferencias en las notas obtenidas de los 2 tipos de instrumentos de evaluación sugieren que se necesita más investigación para analizar los factores que llevaron a la reflexión anterior

Correcting the predictive validity of a selection test for the effect of indirect range restriction.

BACKGROUND: The validity of selection tests is underestimated if it is determined by simply calculating the predictor-outcome correlation found in the admitted group. This correlation is usually attenuated by two factors: (1) the combination of selection variables which can compensate for each other and (2) range restriction in predictor and outcome due to the absence of outcome measures for rejected applicants. METHODS: Here we demonstrate the logic of these artifacts in a situation typical for student selection tests and compare four different methods for their correction: two formulas for the correction of direct and indirect range restriction, expectation maximization algorithm (EM) and multiple imputation by chained equations (MICE). First we show with simulated data how a realistic estimation of predictive validity could be achieved; second we apply the same methods to empirical data from one medical school. RESULTS: The results of the four methods are very similar except for the direct range restriction formula which underestimated validity. CONCLUSION: For practical purposes Thorndike's case C formula is a relatively straightforward solution to the range restriction problem, provided distributional assumptions are met. With EM and MICE more precision is obtained when distributional requirements are not met, but access to a sophisticated statistical package such as R is needed. The use of true score correlation has its own problems and does not seem to provide a better correction than other methods.

Is perfect good? - Dimensions of perfectionism in newly admitted medical students.

BACKGROUND: Society expects physicians to perform perfectly but high levels of perfectionism are associated with symptoms of distress in medical students. This study investigated whether medical students admitted to medical school by different selection criteria differ in the occurrence of perfectionism. METHODS: Newly enrolled undergraduate medical students (n = 358) filled out the following instruments: Multidimensional Perfectionism Scale (MPS-H), Multidimensional Perfectionism Scale (MPS-F), Big Five Inventory (BFI-10), General Self-Efficacy Scale (GSE), Patient Health Questionnaire 9 (PHQ-9), and Generalized Anxiety Disorder 7 (GAD-7). Sociodemographic data such as age, gender, high school degrees, and the way of admission to medical school were also included in the questionnaire. RESULTS: The 298 participating students had significantly lower scores in Socially-Prescribed Perfectionism than the general population independently of their way of admission to medical school. Students who were selected for medical school by their high school degree showed the highest score for Adaptive Perfectionism. Maladaptive Perfectionism was the strongest predictor for the occurrence symptoms of depression and anxiety regardless of the way of admission. CONCLUSIONS: Students from all admission groups should be observed longitudinally for performance and to assess whether perfectionism questionnaires might be an additional useful instrument for medical school admission processes.

General Ability or Distinct Scholastic Aptitudes? A Multidimensional Validity Analysis of a Psychometric Higher-Education Entrance Test.

The present study explored the construct validity of the Psychometric Entrance Test (PET) for higher education in Israel, as represented by the factorial structure of the scholastic aptitudes it measures, and focused on whether the test presents a single measure of overall ability or a measure of the fields of knowledge that are being tested. In Study 1, we used Exploratory Factor Analysis to generate hypotheses regarding the factorial structure of the test. In Study 2, Confirmatory Factor Analysis was carried out to compare competing models that were constructed based on theoretical considerations and the results of Study 1. The findings indicated that a two-layered hierarchical model, encompassing both a General Ability factor and three scholastic domain-specific factors (Verbal Reasoning, Quantitative Reasoning, and English), showed the best fit. Within the framework of the CFA, several statistical procedures were applied to assess reliability (indicator and complexity) and validity (convergent and divergent.).

Elaboración y validación de una escala de autoeficacia en reanimación cardiopulmonar para estudiantes en Ciencias de la Salud / Preparation and validation of a self-efficacy scale in cardiopulmonary resuscitation for Health Sciences students

Objetivo: elaborar un instrumento fiable capaz de medir la autoeficacia del alumno en reanimación cardiopulmonar (RCP) tras un aprendizaje previo. Método: estudio de elaboración y validación de un instrumento: desarrollo del instrumento, validez de aspecto y contenido, validez de constructo. La primera fuente de extracción de los ítems que compondrían la escala de autoeficacia consistió en un cuestionario que contenía 14 preguntas semiestructuradas y cinco casos clínicos sobre la RCP y la parada cardiaca, que tuvieron que cumplimentar profesionales formados en RCP. Los ítems estaban integrados en tres escalas y fueron validados en aspecto y contenido por parte de expertos. Para la validez de constructo se contó con dos muestras de sujetos distintas (85 alumnos en la primera fase y 101 en la segunda). Se llevaron a cabo análisis descriptivos, análisis de fiabilidad mediante coeficiente alfa de Cronbach, y se realizó análisis factorial exploratorio. Se usó el programa estadístico SPSS versión 19.0. Resultados: en la primera fase, la escala A de autoeficacia obtuvo un coeficiente alfa de Cronbach de 0,91. En la segunda fase se obtuvo un alfa de Cronbach para la escala A de 0,93. El valor obtenido con la población total (n= 186), valor muy similar al encontrado con la muestra total (n= 186), fue de 0,92. La escala de autoeficacia en RCP básica tuvo una estructura unidimensional y contenía 20 ítems. Se encontraron relaciones estadísticamente significativas de los constructos cognitivo-emocionales y el valor de autoeficacia. Conclusión: se obtuvo una escala unidimensional de autoeficacia en RCP básica. Existen variables emocionales y cognitivas que intervienen como mediadoras en la conducta de reanimación (AU)

Objective: to prepare a reliable tool in order to measure the self-efficacy of Cardiopulmonary Resuscitation (CPR) students after previous training. Method: a study for preparation and validation of a tool: development, aspect and contents validity, construct validity. The first source for collecting the items which would form the self-efficacy scale consisted in a questionnaire including 14 semi-structured questions and five clinical cases on CPR and cardiac arrest, which had to be completed by professionals with CPR training. These items were classified into three scales, and were validated by experts in terms of aspect and contents. Two different samples were used for construct validity (85 students in the first stage and 101 in the second stage). Descriptive analysis was conducted, as well as reliability analysis through Cronbach’s alpha coefficient; an exploratory factor analysis was also conducted. The SPSS statistical program, version 19.0, was used for analysis. Results: in the first stage, the Self-efficacy Scale A obtained a 0.91 Cronbach’s alpha coefficient; in the second stage, Scale A achieved a 0.93 Cronbach’s alpha. The value obtained with the total population (n= 186) was 0.92, very similar to the one obtained with the total sample. The Self-efficacy Scale for basic CPR had one-dimensional structure and included 20 items. Statistically significant associations were found between cognitive-emotional constructs and the self-efficacy value. Conclusion: a one-dimensional scale was obtained for self-efficacy in basic CPR. There are emotional and cognitive variables involved as mediators in the resuscitation behavior (AU)

Association Between Medication Use and Performance on Higher Education Entrance Tests in Individuals With Attention-Deficit/Hyperactivity Disorder.

Importance: Individuals with attention-deficit/hyperactivity disorder (ADHD) are at greater risk for academic problems. Pharmacologic treatment is effective in reducing the core symptoms of ADHD, but it is unclear whether it helps to improve academic outcomes. Objective: To investigate the association between the use of ADHD medication and performance on higher education entrance tests in individuals with ADHD. Design, Setting, and Participants: This cohort study observed 61 640 individuals with a diagnosis of ADHD from January 1, 2006, to December 31, 2013. Records of their pharmacologic treatment were extracted from Swedish national registers along with data from the Swedish Scholastic Aptitude Test. Using a within-patient design, test scores when patients were taking medication for ADHD were compared with scores when they were not taking such medication. Data analysis was performed from November 24, 2015, to November 4, 2016. Exposures: Periods with and without ADHD medication use. Main Outcomes and Measures: Scores from the higher education entrance examination (score range, 1-200 points). Results: Among 930 individuals (493 males and 437 females; mean [SD] age, 22.2 [3.2] years) who had taken multiple entrance tests (n = 2524) and used ADHD medications intermittently, the test scores were a mean of 4.80 points higher (95% CI, 2.26-7.34; P < .001) during periods they were taking medication vs nonmedicated periods, after adjusting for age and practice effects. Similar associations between ADHD medication use and test scores were detected in sensitivity analyses. Conclusions and Relevance: Individuals with ADHD had higher scores on the higher education entrance tests during periods they were taking ADHD medication vs nonmedicated periods. These findings suggest that ADHD medications may help ameliorate educationally relevant outcomes in individuals with ADHD.

Encuesta de percepción de la evaluación clínica objetiva y estructurada (ECOE) por los estudiantes de sexto grado en medicina de la Universidad Miguel Hernández / Perception of objective and structured clinical evaluation (OSCE) by 6th grade students of Medicine at Miguel Hernández University

Introducción. La evaluación clínica objetiva y estructurada (ECOE) es una prueba que evalúa las competencias clínicas. Sujetos y métodos. Se realizó un cuestionario online. Resultados. La valoración organizativa de la prueba fue de 8,96 (sobre 10), y la experiencia global de los estudiantes, de 7,91 (sobre 10). Las estaciones de mayor dificultad fueron las de redacción de informes. Conclusión. La ECOE ofrece una valiosa información de la calidad percibida por el alumno, lo que permite plantear futuros planes de mejora (AU)

Introduction. The objective and structured clinical evaluation (OSCE) is a test that evaluates the clinical competences. Subjects and methods. An online questionnaire was conducted. Results. The organizational assessment of the test was 8.96 and the overall experience of the students was 7.91. The stations of greater difficulty were those of writing of reports. Conclusion. The ECOE questionnaire offers valuable information on perceived quality perceived by the student, which allows for future improvement plans (AU)

Assessment centers versus cognitive ability tests: Challenging the conventional wisdom on criterion-related validity.

Separate meta-analyses of the cognitive ability and assessment center (AC) literatures report higher criterion-related validity for cognitive ability tests in predicting job performance. We instead focus on 17 samples in which both AC and ability scores are obtained for the same examinees and used to predict the same criterion. Thus, we control for differences in job type and in criteria that may have affected prior conclusions. In contrast to Schmidt and Hunter's (1998) meta-analysis, reporting mean validity of .51 for ability and .37 for ACs, we found using random-effects models mean validity of .22 for ability and .44 for ACs using comparable corrections for range restriction and measurement error in the criterion. We posit that 2 factors contribute to the differences in findings: (a) ACs being used on populations already restricted on cognitive ability and (b) the use of less cognitively loaded criteria in AC validation research. (PsycINFO Database Record

The impacts of computer adaptive testing from a variety of perspectives.

Computer adaptive testing (CAT) is a kind of tailored testing, in that it is a form of computer-based testing that is adaptive to each test-taker's ability level. In this review, the impacts of CAT are discussed from different perspectives in order to illustrate crucial points to keep in mind during the development and implementation of CAT. Test developers and psychometricians often emphasize the efficiency and accuracy of CAT in comparison to traditional linear tests. However, many test-takers report feeling discouraged after taking CATs, and this feeling can reduce learning self-efficacy and motivation. A trade-off must be made between the psychological experiences of test-takers and measurement efficiency. From the perspective of educators and subject matter experts, nonstatistical specifications, such as content coverage, content balance, and form length are major concerns. Thus, accreditation bodies may be faced with a discrepancy between the perspectives of psychometricians and those of subject matter experts. In order to improve test-takers' impressions of CAT, the author proposes increasing the target probability of answering correctly in the item selection algorithm even if doing so consequently decreases measurement efficiency. Two different methods, CAT with a shadow test approach and computerized multistage testing, have been developed in order to ensure the satisfaction of subject matter experts. In the shadow test approach, a full-length test is assembled that meets the constraints and provides maximum information at the current ability estimate, while computerized multistage testing gives subject matter experts an opportunity to review all test forms prior to administration.

El examen MIR 2015 desde el punto de vista de la teoría de respuesta al ítem / MIR 2015 exam from the point of view of the item response theory

Introducción: En España, el acceso a la formación médica especializada, imprescindible para ejercer como médico especialista, se realiza a través de la prueba MIR. Superada esta prueba, los aspirantes pueden acceder a la formación en distintas especialidades ofertadas por numerosos hospitales a lo largo de todo el país. Sujetos y métodos: Para este trabajo se han utilizado las respuestas al examen MIR 2015 de un conjunto de 3.712 aspirantes. Resultados: Se calcularon los índices de dificultad y discriminación de todas las preguntas del examen. Las preguntas se analizaron según los valores de dichos índices y se agruparon por asignaturas, bloques y tipos de pregunta. Las preguntas con una mayor dificultad media fueron las pertenecientes a las asignaturas de fisiología, farmacología, geriatría, traumatología, neurología y cuidados paliativos. Las asignaturas cuyas preguntas mostraron valores menores de dificultad media fueron anatomía patológica, anestesiología, cirugía plástica, habilidades comunicativas, genética y enfermedades infecciosas. Conclusiones: En general, los valores de dificultad y discriminación de las preguntas de la prueba MIR resultan adecuados. La prueba discrimina mejor a los alumnos que demuestran conocimientos más bajos, y el valor óptimo de discriminación se encuentra en torno al percentil 25 de la muestra analizada (con una puntuación equivalente al percentil 41 de todos los médicos presentados al examen MIR 2015). Finalmente, se propone el uso de las metodologías propias de la teoría de respuesta al ítem con el fin de evaluar las preguntas de la prueba candidatas a ser anuladas

Introduction: In Spain, in order to gain access to specialised medical training it is mandatory to take the MIR exam. After passing said exam, the candidates can access training in different hospitals all around the country. Subjects and methods: This research was made using a database of the answers of 3,712 candidates who took the 2015 MIR exam. Results: The difficulty and discrimination index of all the questions in the exam were calculated. All the questions were analysed, taking into account the values of those parameters and classified by subject, block and kind of question. On average, those questions that were found to be most difficult correspond to the following subjects: physiology, pharmacology, geriatrics, traumatology, neurology and palliative care. The subjects with the least average difficulty were anatomical pathology, anaesthesiology, plastic surgery, communication skills, genetics and infectious diseases. Conclusions: Overall, the discrimination and difficulty values of the questions in the MIR exam are sufficient. The exam is more discriminatory for those students with the lowest discrimination levels, with percentile 25 having the highest levels. Finally, we propose that item response theory be employed as a support tool in order to decide which exam questions would be nullified

Validez concurrente del examen clínico objetivo estructurado con el portafolio electrónico, examen teórico y promedio ponderado en estudiantes de cirugía de la Universidad Privada Antenor Orrego / Concurrent validity of the objective structured clinical examination with electronic portfolio, theoretical exam and weighted average in students of surgery at the Antenor Orrego Private University

Sujetos y métodos: Este estudio correlacional descriptivo tuvo como objetivo demostrar la validez concurrente del examen clínico objetivo estructurado (ECOE) con el promedio ponderado, la nota teórica y el portafolio electrónico en 123 estudiantes de medicina del curso de Cirugía I del IX ciclo, durante el semestre académico 2014-I, de la Facultad de Medicina de la Universidad Privada Antenor Orrego (Trujillo, Perú). Resultados: Hubo correlación bivariada aceptable (r = 0,65) entre la nota teórica y el ECOE; correlación moderada (r = 0,52) entre el promedio ponderado y el ECOE; y correlación alta (r = 0,77) entre la nota del portafolio electrónico y el ECOE. Hubo correlación lineal múltiple entre el portafolio, el ECOE y el examen teórico (coeficiente de determinación múltiple R2 = 0,55). Conclusión: Este estudio demuestra que el ECOE, el portafolio electrónico y la nota teórica utilizados para evaluar a los estudiantes de un curso de cirugía de pregrado tuvieron validez concurrente en el rango de aceptable a alta, y esta información constituye la base para mejorar los estándares de evaluación

Subjects and methods: This descriptive correlational study aimed to demonstrate the concurrent validity of the objective structured clinical examination (OSCE) with the weighted average, theoretical note and the electronic portfolio of 123 medical students of the I Course of Surgery of the ninth cycle during the academic semester 2014-I, at the Faculty of Medicine of the Antenor Orrego Private University (Trujillo, Peru). Results: There was an acceptable bivariate correlation (r = 0.65) between the theoretical note and OSCE; moderate correlation (r = 0.52) between the weighted average and OSCE; and high correlation (r = 0.77) between the notes of the electronic portfolio and OSCE. There was a multiple linear correlation among the portfolio, OSCE and theoretical note (multiple determination coefficient R2 = 0.55). Conclusion: This study showed that the OSCE, electronic portfolio and theoretical note used to evaluate students in an undergraduate course of surgery had concurrent validity in the range of acceptable and high and this information forms the basis in order to improve the standards of evaluation

Factores predictores de resultados en la prueba MIR en las universidades públicas. Análisis de la cohorte 2008-2014 / Factors that may predict the success in the MIR exam in public Spanish universities. An analysis of the 2008-2014 cohort

Objetivo: Analizar la influencia de algunos factores que podrían asociarse con el rendimiento y éxito de los médicos en la prueba MIR. Sujetos y métodos: Se analizaron los resultados obtenidos por los aspirantes médicos en la prueba MIR de la convocatoria 2014-2015 originarios de 29 facultades de medicina. Las variables de análisis incluidas fueron la nota de acceso a la universidad para el curso 2008-2009, el tamaño de las promociones de entrada, el número total de los presentados a la convocatoria MIR, la diferencia entre los presentados a la convocatoria y el tamaño de la promoción de entrada, y las variables de éxito, como el porcentaje de aspirantes que superaron la prueba y su distribución en los percentiles 23 y 73. Resultados: Ni la nota de acceso a la universidad ni el tamaño de la promoción de entrada se asociaron con las variables de éxito en los resultados. La diferencia entre presentados a la prueba y estudiantes de la promoción fue un indicador significativamente negativo del éxito en la prueba. Conclusiones: El principal y único factor determinante del éxito de los resultados en la prueba MIR es la diferencia entre el número de presentados a la prueba y el de los estudiantes de la cohorte inicial. Esta diferencia representa en su mayor parte a aquellos estudiantes que difieren su candidatura a la prueba por no obtener plaza en la convocatoria natural o que prefieren cambiar la obtenida en años anteriores

Aim: To analyze the influence of some factors on the performance and success of physicians in the MIR test. Subjects and methods: The results obtained by medical aspirants in the 2014-015 MIR test call from 29 medical schools were analyzed. The analysis variables were the access mark to university for 2008-2009, the size of the input promotions, the total number of the presented to the MIR call, the difference between presented to the call and the size of the input promotion, and success variables such as the percentage of applicants who passed the test and their distribution in the 23 and 73 percentiles. Results: Neither the access cut-off to the university nor the size of the entry cohort was associated with successful results. The difference between those presented to the test and those from the input promotion was a significant negative predictor of success in the test. Conclusions: The single biggest determinant of successful outcomes in the MIR test is the difference between the number of submitted for testing and the students of the initial cohort. This difference largely represents students who delay their application to the test by not getting square or prefer to change it

Widening access in selection using situational judgement tests: evidence from the UKCAT.

CONTEXT: Widening access promotes student diversity and the appropriate representation of all demographic groups. This study aims to examine diversity-related benefits of the use of situational judgement tests (SJTs) in the UK Clinical Aptitude Test (UKCAT) in terms of three demographic variables: (i) socio-economic status (SES); (ii) ethnicity, and (iii) gender. METHODS: Outcomes in medical and dental school applicant cohorts for the years 2012 (n = 15 581) and 2013 (n = 15 454) were studied. Applicants' scores on cognitive tests and an SJT were linked to SES (parents' occupational status), ethnicity (White versus Black and other minority ethnic candidates), and gender. RESULTS: Firstly, the effect size for SES was lower for the SJT (d = 0.13-0.20 in favour of the higher SES group) than it was for the cognitive tests (d = 0.38-0.35). Secondly, effect sizes for ethnicity of the SJT and cognitive tests were similar (d = ~ 0.50 in favour of White candidates). Thirdly, males outperformed females on cognitive tests, whereas the reverse was true for SJTs. When equal weight was given to the SJT and the cognitive tests in the admission decision and when the selection ratio was stringent, simulated scenarios showed that using an SJT in addition to cognitive tests might enable admissions boards to select more students from lower SES backgrounds and more female students. CONCLUSIONS: The SJT has the potential to appropriately complement cognitive tests in the selection of doctors and dentists. It may also put candidates of lower SES backgrounds at less of a disadvantage and may potentially diversify the student intake. However, use of the SJT applied in this study did not diminish the role of ethnicity. Future research should examine these findings with other SJTs and other tests internationally and scrutinise the causes underlying the role of ethnicity.

Do later wake times and increased sleep duration of 12th graders result in more studying, higher grades, and improved SAT/ACT test scores?

BACKGROUND: The aim of this study was to investigate the relationship between sleep duration, wake time, and hours studying on high school grades and performance on the Scholastic Aptitude Test (SAT)/ American College Testing (ACT) college entrance exams. METHOD: Data were collected from 13,071 recently graduated high school seniors who were entering college in the fall of 2014. A column proportions z test with a Bonferroni adjustment was used to analyze proportional differences. Analysis of covariance (ANCOVA) was used to examine mean group differences. RESULTS: Students who woke up prior to 6 a.m. and got less than 8 h of sleep (27 %) were significantly more likely to report studying 11 or more hours per week (30 %), almost double the rate compared to students who got more than 8 h of sleep and woke up the latest (16 %). Post hoc results revealed students who woke up at 7 a.m. or later reported significantly higher high school grades than all other groups (p < 0.001), with the exception of those students who woke up between 6:01 a.m. and 7:00 a.m. and got eight or more hours of sleep. The highest reported SAT/ACT scores were from the group that woke up after 7 a.m. but got less than 8 h sleep (M = 1099.5). Their scores were significantly higher than all other groups. CONCLUSION: This study provides additional evidence that increased sleep and later wake time are associated with increased high school grades. However, this study also found that students who sleep the longest also reported less studying and lower SAT/ACT scores.

Rúbrica como evaluación de historia clínica simulada / Rubric for evaluation of simulated clinical record

Introducción: Puede demostrarse la evaluación de habilidades al confeccionar una historia clínica simulada sin pacientes. El objetivo es describir la rúbrica y compararla con un examen convencional. Sujetos y métodos: Estudio observacional descriptivo transversal. Se trata de alumnos de quinto año de medicina de la facultad estatal durante la asignatura de cirugía en un hospital universitario, curso oficial, evaluados en 2015 con elaboración de una historia clínica simulada sin pacientes sobre temas de cirugía torácica no cardiovascular e instrucciones previas. La duración fue de 60 minutos. Se realizó un análisis comparativo de todas las historias clínicas simuladas mediante evaluación convencional y rúbrica y verificación de calidad. Se usaron pruebas estadísticas. Resultados: De los 37 casos, el promedio de evaluación convencional fue ligeramente superior al de la rúbrica; su influjo fue mayor en el contexto de la totalidad del examen parcial convencional ratificado por su valor dentro del examen parcial. La nota de la rúbrica frente a la nota final calificadora del examen parcial también fue significativa. La rúbrica frente a la nota final del examen parcial convencional tuvo una correlación positiva aceptable y significativa. La evaluación con la rúbrica tuvo un número de aprobados menor que con la convencional. La fiabilidad como instrumento de valor fue aceptable (0,73). Conclusión: La rúbrica como instrumento de evaluación de una historia clínica simulada sin pacientes tiene un elevado grado de validez y fiabilidad

Introduction: Evaluation of skills can be shown to create a simulated without patients medical record. Aims to describe the rubric and to compare with conventional test. Subjects and methods: Descriptive cross-sectional observational study; students of the fifth year of medicine, State College during subject surgery at a university hospital in official course evaluated in 2015 with the elaboration of a simulated medical record without patients on issues of non-cardiovascular thoracic surgery with advance directives. Duration of 60 minutes. Comparative analysis of all the case records simulated by the conventional evaluation and method of rubric. Statistical tests were used. Results: Of 37 cases, with conventional evaluation average slightly above the rubric - but its influence is greater in the context of the totality of the conventional partial examination ratified by its value within the partial exam. Rubric note versus final note also significant partial examination ratings. Rubric versus conventional partial exam final note had an acceptable correlation. Evaluation with rubric has one number of approved less than with the conventional; reliability as an instrument of acceptable value (0.73). Conclusion: The rubric as a tool for evaluation of a simulated clinical record without patients has a high degree of validity and reliability

Item exposure control for multidimensional computer adaptive testing under maximum likelihood and expected a posteriori estimation.

Item bank stratification has been shown to be an effective method for combating item overexposure in both uni- and multidimensional computer adaptive testing. However, item bank stratification cannot guarantee that items will not be overexposed-that is, exposed at a rate exceeding some prespecified threshold. In this article, we propose enhancing stratification for multidimensional computer adaptive tests by combining it with the item eligibility method, a technique for controlling the maximum exposure rate in computerized tests. The performance of the method was examined via a simulation study and compared to existing methods of item selection and exposure control. Also, for the first time, maximum likelihood (MLE) and expected a posteriori (EAP) estimation of examinee ability were compared side by side in a multidimensional computer adaptive test. The simulation suggested that the proposed method is effective in suppressing the maximum item exposure rate with very little loss of measurement accuracy and precision. As compared to MLE, EAP generates smaller mean squared errors of the ability estimates in all simulation conditions.