Efecto del orden de las preguntas de selección múltiple en los resultados de exámenes de cardiología / Effect of order of multiple-choice questions on the results of cardiology exams

RESUMEN Introducción: Algunos autores han señalado que el armado de un examen con preguntas aleatorias puede perjudicar el rendimiento de los estudiantes. A partir de la pandemia por COVID -19 y del aislamiento social obligatorio, las actividades de educación médica de posgrado pasaron a la modalidad virtual y los exámenes se implementaron online. Objetivo: Estudiar si el ordenamiento al azar de las preguntas tiene algún efecto en los resultados de los exámenes. Material y métodos: Se redactaron 2 exámenes: uno sobre Prevención de la enfermedad cardiovascular y otro sobre Valvulopatías. Para cada uno de los temas se confeccionaron dos cuestionarios: una versión con las preguntas en el orden lógico y según dificultad estimada, y otra con las mismas preguntas distribuidas al azar. Cada examen tenía 50 preguntas de selección múltiple con 3 opciones. Puntaje máximo posible: 50 Los exámenes fueron administrados en plataforma Moodle, modalidad sincrónica, tiempo disponible 75 minutos. Los resultados se expresaron en puntajes obtenidos (rango y valores de tendencia central) y según el índice de dificultad de las preguntas. Resultados: Respondieron 284 residentes, alumnos del Curso Bianual de Cardiología. Dos grupos de 1° año (tema Prevención) y dos grupos de 2° año (Valvulopatías). No hubo diferencia entre los resultados de las dos versiones del mismo examen. Conclusiones: Este estudio careció de la potencia suficiente para fundamentar una u otra forma de ordenar las preguntas de los exámenes de ciencias clínicas. Surgieron nuevas preguntas que deberán ser respondidas en futuros estudios.

ABSTRACT Background: Some authors have pointed out that setting up an exam with random questions can impair student performance. Since the COVID-19 pandemic and compulsory social isolation, postgraduate medical education activities became virtual and exams were implemented online. Objective: The aim of the study was to analyse whether the random order of questions has any effect on test results. Method: Two exams were written: one on the Prevention of Cardiovascular Disease and another on Heart Valve Diseases. For each topic, two questionnaires were designed: a version with the questions in logical order and according to the estimated difficulty and another with the same questions randomly distributed. Each exam had 50 multiple-choice questions with 3 options, and the maximum possible score was 50 The exams were taken on the Moodle platform, with synchronous modality, and the time available was 75 minutes. The results were expressed as scores obtained (range and central tendency) and according to the index of difficulty of the questions. Results: The number of respondents was 284 residents, students of the Biannual Cardiology Course: two 1st year groups (Prevention topic) and two 2nd year groups (Valve diseases). There was no difference between the results of the two versions of the same exam. Conclusions: This study lacks sufficient power to support either way of question order in clinical sciences exams, giving rise to new queries.

ANALYSIS OF STUDENT'S FEEDBACK ON DIFFERENT FORMATS OF WRITTEN ASSESSMENT IN UNDERGRADUATE MEDICAL EDUCATION

OBJECTIVE: To assess the undergraduate medical student's preference among different formats of written assessment. MATERIAL AND METHOD: Cross sectional study was conducted on Phase I and phase II undergraduate medical students with prior information and consent. Lecture was delivered on a topic for both phases followed by a test with same question in different formats and a questionnaire on student's feedback on these formats. RESULTS: Majority of students of both phases preferred Multiple choice questions over other three formats as it was easier to answer, time saving, enjoyable, can fetch more marks and free of evaluator bias. Other formats also had their advantages and drawbacks. CONCLUSION:Assessment of any format of written assessments can usually assess any level of cognitive domain with properly framed questions and model answers although students may prefer a particular one

Perceptions and Suggestions of II year M.B.B.S. Students Regarding Updated Pathology Teaching Curriculum: An Analytic Study

Introduction: In most of the medical colleges of India, the knowledge regarding basic science subjects is taught mainly through didactic lectures, tutorials and practicals. And hence the system is more of teacher centered with less interaction or involvement from the students. There is a need to explore innovations, so as to sustain interest. Objectives: To introduce interactive teaching and assessment methods during lectures and examinations respectively and to elicit student’s perception about them. Materials and Method: A number of interactive methods were introduced in the pathology curriculum in a batch of 153 medical students of 2nd year M.B.B.S. Later the perceptions of the students were elicited regarding these sessions using a questionnaire based on Likert scale. Results: Validation of the scale indicated good internal consistency with Crohnbach’s alpha coefficient of 0.9. Majority (83%) of the students agreed or strongly agreed to like the interactive sessions. Similarly, 86% agreed or strongly agreed to like teamwork in group discussions and quiz sessions. Most popular interactive mode was MCQ’s followed by Cinemeducation. Most students were also of the view that interactivity during lectures keeps them active, more attentive in the class, makes the atmosphere more lively, also improved their communication skills, helps in retention of the topic, clearing doubts, improves attention span and hence results in better understanding of the subject. Conclusion: This project clearly proves that updated pathology curriculum is preferred by students. So, teacher should take a new role of facilitating the process of interactive learning rather than delivering elaborative lectures.

Item analysis of multiple choice questions of undergraduate pharmacology examinations in a medical college in Belagavi, Karnataka, India

Background: Multiple choice questions (MCQs) are a common method of assessment of medical students. The quality of MCQs is determined by three parameters such as difficulty index (DIF I), discrimination index (DI), and Distractor efficiency (DE). Item analysis is a valuable yet relatively simple procedure, performed after the examination that provides information regarding the reliability and validity of a test item. The objective of this study was to perform an item analysis of MCQs for testing their validity parameters.Methods: 50 items consisting of 150 distractors were selected from the formative exams. A correct response to an item was awarded one mark with no negative marking for incorrect response. Each item was analysed for three parameters such as DIF I, DI, and DE.Results: A total of 50 items consisting of 150 Distractor s were analysed. DIF I of 31 (62%) items were in the acceptable range (DIF I= 30-70%) and 30 had ‘good to excellent’ (DI >0.25). 10 (20%) items were too easy and 9 (18%) items were too difficult (DIF I <30%). There were 4 items with 6 non-functional Distractor s (NFDs), while the rest 46 items did not have any NFDs.Conclusions: Item analysis is a valuable tool as it helps us to retain the valuable MCQs and discard or modify the items which are not useful. It also helps in increasing our skills in test construction and identifies the specific areas of course content which need greater emphasis or clarity.

“Quiz – An Innovative Method Of Teaching Microbiology To Medical Undergraduate Students”.

Background & Objective: Many undergraduate medical students prefer to concentrate on clinical subjects and tend to neglect paraclinical subjects like Microbiology. A time has come when motivation is mandatory for the current generation of students. The present study attempts to introduce quiz as an innovative learning method. A quiz is a quick way of gathering information on how well our students are meeting their learning objectives. The present study is aimed to assess the students’ knowledge and interest in Microbiology subject. Methodology: The present study was conducted in the department of Microbiology, D.Y. Patil medical college, Kolhapur, on second MBBS students. Five teams were enrolled from the neighboring medical colleges. We assessed the students’ knowledge by giving pre and post MCQ tests followed by various rounds of quiz competition. Results: There was a significant increase in students’ knowledge in post MCQ test than that of pre MCQ test (P< 0.0001). It was also observed that all the students had developed interest in Microbiology subject. Conclusion: We observed from the present study that even a simple intervention like quiz can make a significant improvement in the knowledge of medical students and help them to develop interest in microbiology subject.

Evaluación de conocimientos con exámenes de selección múltiple: ¿tres o cuatro opciones de respuesta? Experiencia con el examen de admisión a posgrados médico-quirúrgicos en la Universidad de Antioquia / Evaluation of knowledge with multiple-choice tests: three of four options? Experience with admission examinations to medical and surgical postgraduate studies at University of Antioquia (Medellín, Colombia) / Avaliação de conhecimentos com exames de seleção múltipla: três ou quatro opções de resposta? Experiência com o exame de admissão a pósgraduações médico-cirúrgicos na Universidade de Antioquia

Introducción: el objetivo de este estudio fue evaluar el efecto de la reducción del número de opciones de respuesta por pregunta sobre los indicadores sicométricos de un examen de ingreso a estudios médicos de posgrado. Metodología: aplicación de índices de evaluación sicométrica desde la perspectiva de dos teorías: la clásica de la medición y la de respuesta al ítem, a una prueba de 70 preguntas hecha a 2.539 aspirantes a ingresar a los posgrados médico-quirúrgicos de la Universidad de Antioquia en el año 2014. Se eliminó la opción de respuesta elegida con menor frecuencia y se la reemplazó por azar de entre las tres restantes. Resultados: solo 52,9% de las preguntas tuvieron tres opciones funcionales de respuesta. No se encontró diferencia en la dificultad, la discriminación, el error estándar de la medición, el alfa de Cronbach ni el coeficiente de correlación biserial (teoría clásica de la medición); tampoco en la medida de dificultad de los ítems o de habilidad de las personas (teoría de respuesta al ítem) entre las pruebas con tres y cuatro opciones de respuesta. La prueba con tres opciones conservó un buen ajuste. Conclusión: una prueba con tres opciones de respuesta se comportó tan bien como su contraparte de cuatro opciones.

Introduction: The aim of this study was to evaluate the effect of reducing the number of response options per question on the psychometric indicators of an exam for admission to postgraduate medical studies, at University of Antioquia, in Medellín, Colombia. Methodology: Application of psychometric assessment indexes from the perspective or two theories: the classical of measurement and the item response, to a test of 70 questions, applied in 2014 to 2.539 candidates. The least frequently chosen distractor was eliminated and randomly replaced by one of the three remaining ones. Results: Only 52.9% of the questions had three functional distractors. No difference was found in the difficulty, discrimination, standard error of measurement, Cronbach's alpha and the coefficient of biserial correlation (classical measurement theory). Also, there was no difference in the extent of item difficulty or ability of people (item response theory). The test with three options retained a good fit. Conclusion: Multiple choice tests with three response options performed as well as their four options counterparts.

Introdução: o objetivo deste estudo foi avaliar o efeito da redução do número de opções de resposta por pergunta sobre os indicadores psicométricos de um exame de rendimento a estudos médicos de pós-graduação. Metodologia: aplicação de índices de avaliação psicométrica desde a perspectiva de duas teorias: a clássica da medição e a de resposta ao item, a uma prova de 70 perguntas feita a 2.539 aspirantes a ingressar às pós-graduações médico-cirúrgicos da Universidade de Antioquia no ano de 2014. Eliminou-se a opção de resposta eleita com menor frequência e se a substituiu por casualidade de entre as três restantes. Resultados: só 52,9% das perguntas tiveram três opções funcionais de resposta. Não se encontrou diferença na dificuldade, a discriminação, o erro regular da medição, o alfa de Cronbach nem o coeficiente de correlação bisserial (teoria clássica da medição); também não na medida de dificuldade dos itens ou de habilidade das pessoas (teoria de resposta ao item) entre as provas com três e quatro opções de resposta. A prova com três opções conservou um bom ajuste. Conclusão: uma prova com três opções de resposta se comportou tão bem como sua contraparte de quatro opções.

MCQ Supplementation in a Physiology Didactic Class: A Learning Tool.

Background and Objectives: Medical educators are facing the challenge to develop innovative creative material so as to engage the students in active learning, more so for basic sciences. In addition the innovations should help student learning, keeping time and manpower and economy constraints in mind. Overall objective of this study was to study and assess the role of MCQ supplementation in a didactic class in improving student learning. Methods: 136 First year MBBS students of a government medical college attending physiology classes were recruited for the study. Based on the university recognized syllabus the Specific learning objectives (SLOs) were randomly grouped into two categories, that is, SLOs for which didactic classes were to be supplemented with multiple choice questions (MCQs) and those SLOs for which didactic classes were to be taught without MCQ supplementation. Results: On subjecting individual student’s scores obtained in the two categories of MCQs to unpaired ‘t’ test the difference was found to be statistically significant, p = 0.025, t = 2.259. 95% CI. Conclusion: Reinforcement of Didactic class with MCQ supplementation is an effective learning tool which was well received by the participants.

Evaluation of novel methods of assessments in pharmacology to develop new attitudes and skills in Caribbean Medical School.

Background: Assessment is said to drive student learning and define the curriculum. The problem-solving type of multiple choice questions (MCQs), which can be used to probe and assess medical students in pharmacology should have a clinical vignette containing presenting complaints, abstract history, physical examination and laboratory data, followed by a single or series of questions based on it. National Board of Medical Examination (NBME), USA has an extensive bank of problem-solving MCQs, and these questions are often regarded as similar in format and focus to MCQs of United States Medical Licensing Examination (USMLE). The objective assessment of teaching and curriculum in this study is done by comparison of students’ performance in pharmacology comprehensive exam of NBME, USA before and after curriculum changes. This study was designed to obtain an objective assessment of teaching and curriculum by comparison of students’ performance in terms of student mean grades, percentage of students passed, percentage of students failed, percentage of students with honors, and individual highest scores of five semesters before and five semesters after curriculum changes in pharmacology comprehensive exam of NBME, USA among the 5th semester students of American University of the Caribbean, School of Medicine, St. Maarten. Methods: We have compared the students’ performance of pharmacology comprehensive exam of NBME using five parameters like student mean grades, percentage of students passed, percentage of students failed, percentage of students with honor, and individual highest score of five semesters May 2009, September 2009, January 2010, May 2010 and September 2010 semester batches before the introduction of curriculum changes with subsequent semesters January 2011, May 2011, September 2011, January 2012 and May 2012 semesters after the introduction of curriculum changes. Results: The pre-curriculum student performances were compared with post-curriculum changes using the Student’s t-test. The students mean scores improved significantly from 50.76 before curriculum changes to 56.54, students passed (%) increased from 94.57% before curriculum changes to 96.93% after curriculum changes and students with honors (%) increased significantly from 64.72% before curriculum change to 75.51% after curriculum changes and also seem to have remained consistently better. The students failed (%) decreased dramatically from 5.43% to 3.07% after curriculum changes. The highest individual mean score also improved significantly from 72.4 to 80.8 after curriculum changes and have remained consistent in the following semesters. Conclusions: There seems to be obvious improvements in student performance as reflected by a significant increase in mean scores, students pass (%), and students with honors (%) probably due to inclusion of problem-solving MCQs in formative and summative assessments in new curriculum compared with declarative MCQs in old curriculum. The student failed (%) decreased dramatically, which could be attributed to the changes in teaching content and format brought by curriculum changes in pharmacology. The teaching of pharmacology principles as pathophysiology of drug therapy also seemed to have prepared students better for NBME comprehensive exam and also USMLE Step 1. The clinical pharmacology exercises in small groups as role playing sessions seem to have really improved students’ comprehension and retention of the basic sciences knowledge for clinical application based on students’ feedback.

Correlation between the learning approaches of first year medical students and their performance in multiple choice questions in physiology.

Background: Students generally choose either a surface learning approach that focuses on rote learning or a deep approach that focuses on understanding. While selection for postgraduate medical courses is based on Multiple Choice Questions (MCQs), only some medical colleges in Tamil Nadu, India, use MCQs for assessing undergraduate students. Aims and Objectives: The aim of this study was to determine the relationship between the learning approaches of first year medical students using the Revised Two Factor Study Process Questionnaire (R-SPQ-2F) and their performance in Multiple Choice Questions in Physiology. Materials and Methods: The Revised Two Factor Study Process Questionnaire (R-SPQ-2F) was administered to 142 first year medical students of a private medical college in Tamil Nadu after the internal assessment examination in Physiology that included 20 MCQs. Pearson product-moment correlation coefficient was determined using SPSS 17 software, to find the relationship between the deep and surface learning approach scores with the performance in the MCQ examination. Results: The majority of our students (73%) preferred a deep approach while learning Physiology. There was a positive correlation between deep approach scores and MCQ marks (r=0.226, p=0.007, n=142) and a negative correlation between surface approach scores and MCQ marks (r=-0.258, p=0.002, n=142). Conclusion: The majority of our first year medical students exposed to a didactic-non Problem Based Learning curriculum in which assessment included MCQs in Physiology preferred a deep learning approach which was positively correlated with MCQ marks.

Multiple choice questions as a teaching learning tool in addition to assessment method.

Background and Objectives: Learning of a subject becomes effective when the student knows the purpose of learning. The guidance on the relevance of content of basic sciences to clinical sciences is important. Reinforcement of knowledge is essential to indicate the key areas of the subject for in depth study. The aim of the present study was to focus the students on relevant content and educational objectives through multiple choice questions. Methods: The study consisted of an intervention and a control group of seventy five students each. The intervention group was exposed to the new concept of weekly assessment with concept based MCQs. Results: The methodology has been effective in the study group. This has been demonstrated by the mean scores of the study group (17.67 ± 1.72) being higher than the control group (13.6 ± 2.34) (P < 0.01). Program evaluation by the students has revealed that they developed interest to study the subject in depth. It has helped them in comprehending and reinforcement of the conceptual knowledge and guided them to apply it clinical sciences. Conclusion: Well structured MCQs in the form of assessment with an immediate feedback with explanation of the applied anatomy improve the critical thinking and reasoning skills of the student.

Evaluación de la calidad de las preguntas de selección múltiple utilizadas en los exámenes de Certificación y Recertificación en Cardiología en el año 2009 / Evaluation of the Quality of Multiple-Choice Questions Used in Cardiology Certification and Recertification Exams

La Sociedad Argentina de Cardiología, en los procesos de Certificación y Recertificación de Especialistas, implementa exámenes escritos de selección múltiple. Dos observadores independientes revisaron las 200 preguntas utilizadas en dos exámenes (A y B) realizados durante 2009. Se usó el Índice de Calidad de Galofré; este instrumento toma en cuenta 10 criterios que se deben considerar en la redacción de las preguntas de selección múltiple y establece una escala de 1 a 5 puntos según la cantidad de defectos de construcción que tenga la pregunta. El valor máximo de calidad posible es 5. La media aritmética de los valores de calidad de las preguntas expresa el Índice de Calidad del examen en su totalidad. Se encontró que el 30% de las preguntas tenían muy buena calidad técnica (valores de calidad 4 y 5); cerca del 40% eran preguntas aceptables (valor de calidad 3) que se deberían mejorar y el 30% eran inaceptables (valor de calidad 1 y 2). El examen A tiene un Índice de Calidad de 2,15 y el examen B, de 3,21. En ambos exámenes se encontró que los defectos más frecuentes en la redacción de las preguntas eran la falta de viñeta (caso clínico o problema) y la exploración de conocimientos sólo a nivel de memoria o recordación de datos aislados. Se concluyó que sería conveniente constituir en la Sociedad Argentina de Cardiología un grupo de trabajo permanente para revisar y mejorar las preguntas y armar un banco de ítems.

The Argentine Society of Cardiology uses multiple-choice questions for Cardiology Certification and Recertification exams. Two independent observers revised 200 questions used in two examination tests (A and B) during 2009. We used the Galofré Quality Index, which considers 10 criteria for writing multiple-choice questions and establishes a scale ranging from 1 to 5 points according to the number of construction defects the question has. The maximum value of quality is 5. Mean value of the quality of questions represents the Quality Index of the whole test. We found that 30% of the questions had a very good technical quality (quality value 4 and 5); about 40% were acceptable questions (quality value 3) that should be improved, and 30% were unacceptable (quality value 1 and 2). The Quality Index of test A was 2.15 and of test B was 3.21. The most frequent writing defects in both exams were the lack of bullet points (case report or problem) and exploring knowledge only in terms of memory or recalling isolated information. In conclusion, it would be convenient to constitute a permanent work group in the Argentine Society of Cardiology to revise and improve the questions and create a bank of items.

Effects of an additional small group discussion to cognitive achievement and retention in basic principles of bioethics teaching methods.

Aim The place of ethics in undergraduate medical curricula is essential but the methods of teaching medical ethics did not show substantial changes. “Basic principles of bioethics” is the best knowledge to develop student’s reasoning analysis in medical ethics In this study, we investigate the effects of an additional small group discussion in basic principles of bioethics conventional lecture methods to cognitive achievement and retention. This study was a randomized controlled trial with parallel design. Cognitive scores of the basic principles of bioethics as a parameter was measured using basic principles of bioethics (Kaidah Dasar Bioetika, KDB) test. Both groups were attending conventional lectures, then the intervention group got an additional small group discussion. Result Conventional lectures with or without small group discussion significantly increased cognitive achievement of basic principles of bioethics (P= 0.001 and P= 0.000, respectively), and there were significant differences in cognitive achievement and retention between the 2 groups (P= 0.000 and P= 0.000, respectively). Conclusion Additional small group discussion method improved cognitive achievement and retention of basic principles of bioethics.

Pooling System for Multiple-Choice Questions for the National Examination for Medical Practitioners. Results of a Field Study in Japan / 医学教育

The Ministry of Health and Welfare of Japan is planning a pooling system for multiple-choice questions (MCQs) for the national examination for medical practitioners. To clarify possible problems of such a system, a field study was performed by 10 medical schools in Japan using 90 MCQs from previous examinations. Nine hundred twenty-four 6th-year students participated in the field test. For each MCQ, the correct-response rates at the originating school and those obtained in the field test were significantly correlated. Thus, the correct-response rates to questions on the field test could be predicted from the rates at the originating schools. However, for each question the correct-response rate was significantly higher for students of the originating school than for students of other schools. In the national examination, care should be taken to prevent differences in scores on the basis of question sources.

Estimation of Knowledge Level with A-, K- and X-Type Multiple-Choice Questions / 医学教育

With multiple-choice questions presenting one, two, or three of five choices (types A, K or X), the expected probability of a correct answer (P) can be obtained with a five-dimensional equation for the knowledge level (q), which is the probability of discriminating correct and incorrect items. From equations for each question type, we inversely estimated the q value (the estimated knowledge level), replacing P with the raw score rate in the actual examinations for promotion. The distribution and mean value of the raw score rate of type X differed from those for types A and K. However, distributions of estimated q values for the three question types were similar. This method can be used to estimate the actual knowledge level of students without the question type affecting the raw score but cannot be used to estimate incorrect knowledge.

Relation of Ikebukuro's Index and "Standard Score Rate" / 医学教育

Ikebukuro's index is identical to the index described by Nukada in 1978. Nukada's index has not been used because it is not realistic or useful. Because a “ standard score rate” can be calculated from the weighted average of Nukada's index, Ikebukuro's index is also closely related to the “ standard score rate.”

A new multiple-choice question format more parallel to the knowledge quantity. Relationship between expected correct answer ratio and knowledge quantity / 医学教育

In addition to the A-and K-type questions, the X<SUB>2</SUB> type question has been introduced into, and the K'-type question has been eliminated from the Japanese Medical Licensure Examination since 1997. The expected correct answer ratio (ECAR) without any knowledge is 10% for the X<SUB>2</SUB>-type in contrast to 20% for the K2-type. However it still becomes 100% with an 80% knowledge. We designed a formula to study the relation between the knowledge quantity (KQ) and ECAR. The ECAR is 7%, 13%, 25%, 43%, 70%, and 100% for the X<SUB>1, 2</SUB>-type in which the number of answer (NA) is indicated as one or two, while it is 5%, 10%, 20%, 40%, 70%, and 100% for the X<SUB>2, 3</SUB>-type in which NA is indicated as two or three. The KQ difference between 80% and 100% can be reflected to ECAR in these types and considered to be the better question type.

Analysis of Results After Introduction of X Type Questions / 医学教育

X type questions have been used for the national medical licensing examination since 1997. At Tsukuba University, X type questions have been used since 1996. We compared X typeand K type questions on the basis of the percentage of correct answers and discrimination power. The average percentage of correct answers was 68.2% for K type questions and 53.1% for X type questions. However, the average discrimination power was +0.227 for K type questions and +0.257 for X type questions. These results indicate that X type questions are more difficult and are suitable for achievement tests. The estimated knowledge quantity was 2.04 for K type questions and 2.32 for X type questions. This suggests that the person writing the questions decreased the essential difficulty of X type questions.

Experience Using Computer Quizes as an Alternative to Paper Tests / 医学教育

Some comments of the author are presented on his experience concerning the use of computer quizes in teaching fundamental microbiology to medical students. The simple true-false type question was one of the best suited forms for computer display because of its simplicity. Multiple choice questions consisting of pleural (usually five) simple true-false sentences were too cumbersome. Repeatability is one of the advantages of computer quizes. Furthermore the shuffling of questions is simple to perform on the computer. The computer quiz is one form of preparation and review, and differs from examinations in that students have mutiple chances and time to take the quiz. With the computer-quiz teachers also have the chance to make better contact with students, and through their care of students, teachers remember more faces and names of students.