Análisis psicométrico de las preguntas repetidas en los simulacros de la prueba MIR / Psychometric analysis of the questions repeated in the MIR test mockups

Introducción: El presente trabajo analiza 13 preguntas que se repitieron dos veces en distintas pruebas que simulan el examen MIR, separadas en el tiempo entre dos y 15 semanas. A partir de los resultados obtenidos, se ha analizado el proceso de aprendizaje de los estudiantes que se preparan para la prueba MIR.Material y métodos.Para el presente estudio se hizo uso de una base de datos formada por un total de 2.600 preguntas respondidas en promedio por 1.585 alumnos de características relativamente homogéneas.Resultados.En general, resulta posible afirmar que entre la primera y la segunda exposición de los alumnos a la misma pregunta se produjo una mejora de los índices analizados. De los resultados obtenidos, se observa que, en promedio, a lo largo de la preparación, la dificultad de las preguntas para el grupo de estudiantes se reduce a la mitad. En lo referente al índice de discriminación, éste se incrementa cerca del 60%, lo que supone que, según transcurre la preparación, aumenta la proporción de personas que aciertan la pregunta en el grupo fuerte con respecto a las que la aciertan en el grupo débil. Finalmente, también se produce una mejora en el valor del coeficiente de correlación biserial puntual, lo que supone una mejora de la capacidad discriminativa de las preguntas.Conclusiones.Según transcurre la preparación, las preguntas analizadas se vuelven más fáciles y más discriminativas. Además, se observa cómo no todos los alumnos consiguen el mismo aprovechamiento del proceso de aprendizaje. (AU)

Introduction: The present work analyses 13 questions that were repeated twice in different tests that simulate the MIR exam, separated in time between 2 and 15 weeks. From the results obtained, the learning process of the students preparing for the MIR test has been analysed.Material and methods.For the present study, a database made up of a total of 2,600 questions which were answered on average by 1,585 students with relatively homogeneous characteristics was employed.Results.In general, it is possible to affirm that between the first and second exposure of the students to the same question there was an improvement in the indexes analysed. From the results obtained, it is observed that, on average, throughout the preparation, the difficulty of the questions for the group of students is reduced by half. Regarding the discrimination index, this increases by about 60%, which means that as the preparation progresses, the proportion of people who get the question right in the strong group increases with respect to those who get it right in the weak group. Finally, there is also an improvement in the value of the biserial point correlation coefficient, which implies an improvement in the discriminative capacity of the questions.Conclusions.As the preparation progresses, the analysed questions become easier and discriminating. In addition, it is observed how not all students obtained the same performance. (AU)

Construction of clinical medical examination question bank based on virtual reality technology / 中华医学教育探索杂志

Using novel virtual reality (VR) technology to carry out the construction of clinical medical examination question bank, while deepening the reform of clinical medical course examination, it continues to innovate the medical professional evaluation system and improve the flexibility, diversity and scientificity of clinical medicine in teaching and assessment. It is of great and far-reaching significance to improve the teaching level and the quality of medical education in medical colleges and universities. This paper analyzes and discusses the necessity and feasibility of building a clinical medical examination question bank, and the advantages and prospects of integrating VR technology to carry out the construction of clinical medical examination question bank. At the same time, the exploration and practice of the examination question bank construction based on VR technology disscussed in detail would provide innovative thinking and reference for the clinical medical teaching and evaluation, medical personnel training and other aspects in China.

A comparison of different standard-setting methods for professional qualifying dental examination.

BACKGROUND: The outcome of assessments is determined by the standard-setting method used. Standard setting is the process of deciding what is good enough. A cutoff score of 50% was commonly used in dental schools in Malaysia. This study aims to compare the conventional, norm-referenced, and modified-Angoff standard-setting methods. METHODS: The norm-referenced method of standard setting was applied to the real scores of 40 final-year dental students on a multiple-choice question (MCQ), a short answer question (SAQ), and an objective structured clinical examination (OSCE). A panel of 10 judges set the standard using the modified-Angoff method for the same paper in one sitting. One judge set the passing score of 10 OSCE questions after 2 weeks. A comparison of the grades and pass/fail rates derived from the absolute standard, norm-referenced, and modified-Angoff methods was made. The intra-rater and inter-rater reliabilities of the modified-Angoff method were assessed. RESULTS: The passing rate for the absolute standard was 100% (40/40), for the norm-referenced method it was 62.5% (25/40), and for the modified-Angoff method it was 80% (32/40). The modified-Angoff method had good inter-rater reliability of 0.876 and excellent test-retest reliability of 0.941. CONCLUSION: There were significant differences in the outcomes of these three standard-setting methods, as shown by the difference in the proportion of candidates who passed and failed the assessment. The modified-Angoff method was found to have good reliability for use with a professional qualifying dental examination.

The feasibility of simulation-based high-stakes assessment in emergency medicine settings: A scoping review.

Using simulation in high-stakes assessments has been evolving as a method to improve the assessment process. There is a concurrent need to address challenges and establish best practices to ensure the best quality when implementing high-stakes evaluations. The aim of this study is to provide an insight for stakeholders about using multiple modalities of simulation in high-stakes evaluations by presenting challenges, best practices, and future directions described in the relevant literature. A scoping review of original studies (from the year 1994-2021) including the use of common modalities (standardized patients, high-fidelity mannequins, part-task trainers, virtual simulation, and hybrid simulation) was conducted. The search covered the common databases: PubMed, Education Resource Information Center, Cumulative Index to Nursing and Allied Health Literature, and the Cochrane library. Initial screening in the databases resulted in 111,253 articles. After the application of a refining search strategy, 47 articles were included for comprehensive evaluation. Most articles were about credentialing/licensure purposes of assessment. Many articles were specialty-specific, especially focusing on anesthesia. Most challenges described were related to the validity of the assessment that should be considered in the planning phase. Best practices were mostly related to planning for measures to assure the validity of the assessment tools and process. The use of multimodality simulation for high-stakes assessment is growing despite challenges; this growth is associated with the accumulation of experience shared in literature. This growth will help to improve planning, practices, and goals achievement from such an application.

Improvement of Faculty's Skills on the Creation of Items for Progress Testing Through Feedback to Item Writers: a Successful Experience / Melhora na Habilidade de Elaboração de Itens para o Teste de Progresso por meio de Feedback aos Autores: uma Experiência Exitosa

Abstract: The evaluation of knowledge is an important step in the teachers' work, because it is fundamental that the evaluation results be valid and reliable. Several faculty development programs have been undertaken to improve teachers' skills on item writing. Few studies, however, have explored the benefits of providing feedback to multiple-choice item writers. Progress Testing is a longitudinal assessment of students' knowledge gain. Their results are useful for generating feedback to students, teachers and medical school managers. The aim of this report is to present a successful experience of faculty development on high-quality item writing for Progress Testing through feedback about the fate of the items written by faculty members. Annually, faculties from medical schools participating in a consortium for the application of the Progress Test are asked to formulate new items to compose the test version. However, the fate of the items is not known by their authors, that is, they do not know if the items are incorporated into the test, if the items have good quality, what is the students' performance in each of the items, and what is the psychometric function of the items. In 2017, one of the schools participating in the consortium offered its item authors a feedback regarding the flaws in item writing, modification of items by the review committee and students' performance, as well as psychometric function in each item. The percentage of flawed item (items without enough quality to be selected for the test) was more than 30%. There was an inverse relation between the number of flawed item and the number of items included in the final exam, when analyzing the different content areas of the test (that is, Internal Medicine, Surgery, Pediatrics, Obstetrics and Gynecology, Public Health, and Basic Sciences). In the following year, we observed a decrease in the number of flawed item (less than 10%) and an increase in the number of eligible items to be selected for the Progress Testing. Therefore, giving feedback to item writers seems to be a good strategy for developing the faculty's ability on writing good quality items.

Resumo: A avaliação de conhecimentos é uma etapa importante do trabalho docente, pois é preciso que os resultados das avaliações sejam válidos e confiáveis. Diversos programas de desenvolvimento docente têm sido realizados para melhorar a habilidade em elaborar questões objetivas pelos professores. Poucos estudos, no entanto, exploraram os benefícios de fornecer feedback aos autores de itens de múltipla escolha. O Teste de Progresso é uma avaliação longitudinal do ganho de conhecimentos dos estudantes cujos resultados são úteis para gerar feedback aos próprios estudantes, aos professores e aos gestores das escolas médicas. O objetivo deste relato é apresentar uma experiência exitosa em gerar desenvolvimento docente na habilidade de elaborar itens de qualidade para o Teste de Progresso por meio do feedback a respeito da qualidade e do destino destes. Anualmente, os professores das escolas médicas participantes de um consórcio para aplicação do Teste de Progresso são solicitados a formular novos itens para compor a versão do teste. Porém, o destino dos itens não é conhecido por seus autores, ou seja, eles não sabem se as questões são incorporadas à prova, se elas têm boa qualidade, qual o desempenho dos estudantes em cada questão e qual o funcionamento psicométrico delas. Em 2017, uma das escolas participantes do consórcio ofereceu aos seus autores de questões uma devolutiva sobre as falhas na redação, a modificação dos itens pelo comitê revisor, o desempenho dos estudantes nas questões e o funcionamento psicométrico em cada item. A porcentagem de itens falhos, ou seja, de qualidade não satisfatória para a inclusão no teste, era superior a 30%. Houve uma relação inversa entre a quantidade de itens falhos e o número de itens incluídos na prova final, numa análise por área de conhecimento da prova (ou seja, clínica, cirurgia, pediatria, ginecologia e obstetrícia, saúde pública e ciências básicas). No ano seguinte, observamos uma diminuição no número de falhas de itens (menor que 10%) e um aumento no número de questões elegíveis a serem selecionadas para o exame de Teste de Progresso. Portanto, oferecer feedback para os redatores de questões parece ser uma boa estratégia para desenvolver a habilidade docente em elaborar itens objetivos de boa qualidade.

Profile of Public Dentistry Contests in the Southeast Region of Brazil / Perfil de Concursos Públicos de Odontologia para a Estratégia Saúde da Família da Região Sudeste do Brasil

Public contest is one of the best ways for health professionals to enter the Family Health Strategy (FHS) and other public health services. The objective of this study was to verify the profile of Dentistry public examinations in the southeast region, as well as to analyze the coherence between the proposal of FHS and the content on the professionals selection for this position. The examinations were retrieved by means of research of the sites: PCI Concursos and Folha Dirigida and sites of companies of greater representativeness in the competitive bidding in the states of the southeast region. From the entire southeastern region, 266 competitive examinations were obtained for the position of dental surgeon, from 2001 to 2017. For the data analysis,, three categories were created: QG, for the questions that included the general questions present in the tests; QT, individual technical questions that balance the requirements related to specific knowledge; and QC, corresponding to the questions related to collective character. In all the tests analyzed, the specific knowledge (QC and QT) had a higher prevalence among the contents charged. Among the areas that involved the individual technical questions, the subjects of Surgery, Dentistry, Stomatology and Pathology were the ones with the highest prevalence. Through the results, it can be seen that the public examinations for dental surgeons who want to work in the FHS in the states of the southeast region, most of them deal with knowledge of clinical dentistry specialties. (AU)

O concurso público é uma das melhores formas de inserção dos profissionais de saúde na Estratégia Saúde da Família (ESF) e em outros serviços de saúde pública. O objetivo deste estudo foi verificar o perfil de provas de concursos públicos de Odontologia da região sudeste, bem como analisar a coerência entre a proposta da ESF e o conteúdo da seleção de profissionais para este cargo. Os exames foram recuperados por meio de pesquisa em dois sites bastante acessados que disponibilizam provas de concurso público no Brasil: PCI Concursos e Folha Dirigida e em sites de empresas de maior representatividade nas licitações de concurso nos estados da região sudeste. De toda a região foram obtidas 266 provas de concursos para o cargo de cirurgião dentista, no período de 2001 a 2017. Para análise dos dados foram criadas três categorias: QG, para os quesitos que englobaram as questões gerais presentes nas provas; QT, questões técnicas individuais que equivalem aos quesitos relacionados aos conhecimentos específicos; e QC, correpondendo aos quesitos relacionados ao caráter coletivo. Em todas as provas analisadas, os conhecimentos específicos (QC e QT) tiveram maior prevalência entre os conteúdos cobrados. Dentre as áreas que envolveram os quesitos técnicos individuais, as disciplinas de Cirurgia, Dentística, Estomatologia e Patologia foram as de maior prevalência. Através dos resultados, percebe-se que os concursos públicos para os cirurgiões dentistas que querem atuar na ESF nos estados da região sudeste, abordam em sua maioria conhecimentos de especialidades clínicas da Odontologia. (AU)

Evaluating the Quality of Multiple Choice Question in Paediatric Dentistry Postgraduate Examinations.

OBJECTIVES: This study aimed to evaluate the quality of multiple choice question (MCQ) items in two postgraduate paediatric dentistry (PD) examinations by determining item writing flaws (IWFs), difficulty index (DI) and cognitive level. METHODS: This study was conducted at Mohamed Bin Rashid University of Medicine and Health Sciences, Dubai, UAE. Virtual platform-based summative versions of the general paediatric medicine (GPM) and prevention of oral diseases (POD) examinations administered during the second semester of the 2017-2018 academic year were used. Two PD faculty members independently reviewed each question to assess IWFs, DI and cognitive level. RESULTS: A total of 185 single best answer MCQs with 4-5 options were analysed. Most of the questions (81%) required information recall, with the remainder (19%) requiring higher levels of thinking and data explanation. The most common errors among IWFs were the use of "except" or "not" in the lead-in, tricky or unfocussed stems and opportunities for students to use convergence strategies. There were more IWFs in the GPM than the POD examination, but this was not statistically significant (P = 0.105). The MCQs in the GPM and POD examination were considered easy since the mean DIs (89.1% ± 8.9% and 76.5% ± 7.9%, respectively) were more than 70%. CONCLUSION: Training is an essential element of adequate MCQ writing. A general comprehensive review of all programme's MCQs is needed to emphasise the importance of avoiding IWFs. A faculty development programme is recommended to improve question-writing skills in order to align examinations with programme learning outcomes and enhance the ability to measure student competency through questions requiring higher level thinking.

Construction of Clinical Laboratory Instrument Question Database Management System / 中华医学教育探索杂志

The design principle and technologies,which were applied for examination question database management system for Clinical Laboratory Instrument,were discussed in this paper.The program was desighed with Delphi7.0,Word 2003 and Access 2003.This software function is very strong and can be appropriate for the needs of richtext questions edition,test paper output and question layout automation and database security protect.This software also can be applied for other subjects.

The Construction of Examination Question Bank in the Gynecology and Obstetrics / 中华医学教育探索杂志

The separation of teaching and test is not only important in improving teaching quality but also effective to the prevention of cheating in the exams,the improvement of the quality of exams and the correct checkout of the teaching effect.

The Establishment and Application of Examination Question Database on Nutrition and Food Hygiene / 中华医学教育探索杂志

The examination question database on Nutrition and Food Hygiene was set up and applied to reduce the effects of subjective factors on test efficacy.The test items of noun explanation,brief answer and discussion were compiled in the form of Microsoft Excel.Then the items were classified by Excel filtration function and a test paper was created by random sampling from the database.It reduces teachers' burden of making test questions and increases objectivity of test questions,which has achieved good results in teaching.