An assessment of the Chilean National Examination of Orthopaedic Surgery.

BACKGROUND: The Chilean National Examination of Orthopaedic Surgery (EMNOT) has been administered since 2009. It was developed to determine whether individual residents are meeting minimal knowledge standards at the end of their training programs. METHODS: We performed a retrospective evaluation of the EMNOT for all years it has been administered (2009-2015). The test was analyzed for content, taxonomy of questions asked (1: direct recall; 2: diagnosis; 3: evaluation/decision-making), residents' performance, difficulty index and discrimination index. RESULTS: During the years of EMNOT administration, the most frequently tested areas have been pediatric orthopaedics (22.9 %), spine (13.8 %), general orthopaedics (13.8 %) and musculoskeletal trauma (9.9 %). A significant increase in questions with images was observed, as well as a significant decrease in the percentage of Type 1 and an increase in Type 3 questions. The Difficulty Index showed a medium level of difficulty for all years the examination has been administered. The Discrimination Index showed good discrimination in 2009, fair discrimination from 2010 through 2012, and excellent discrimination from 2013 through 2015. CONCLUSION: The EMNOT has evolved over several years to include better quality questions, better discrimination, and a more representative distribution of questions covering the different orthopaedic sub-specialties. This examination represents an effective instrument for quality assurance of orthopaedic residency programs in Chile.

[An assessment of the Dundee Ready Education Environment Measure (DREEM) in Chilean university students]. / Estructura factorial de la escala DREEM en estudiantes de medicina chilenos.

BACKGROUND: The entry to a University requires an adaptation process that not all students solve with the same kind of success. Even though students social adaptation and emotional skills are essential, the educational environmental that they perceive has a significant influence in their academic life. AIM: To describe the changes in the perception about academic environment that medical students experience during the first three years of undergraduate career. MATERIAL AND METHODS: The Dundee Ready Education Environment Measure (DREEM) scale was applied to 525 first to third year medical students and an exploratory factorial analysis was made. RESULTS: Four factors were identified: Academic Perception: academic quality that students attribute to the process in which they take part, as well as to the assessment that they do of their learning outcomes (coefficient ± = 0.85); Academic Experience: refers to positive emotions that students experience during the career such as confidence, pleasure and energy (coefficient ± = 0.76); Atmosphere Perception, comfort and calm that students experiment during their academic activities (coefficient ± = 0.79); Teachers Perception: the perception that students have of teachers about their interest and disposition towards students (coefficient ± = 0.50). CONCLUSIONS: The assessment of academic environment quality is inversely associated with the lapse that the students have spent in their undergraduate careers.

[Assessment for learning: experience in an undergraduate medical theoretical course]. / Evaluación para el aprendizaje: experiencia en un curso teórico de pregrado en medicina.

BACKGROUND: Assessment for learning is a paradigm that is taking shape in the field of medical education. This approach aims to embed the assessment process within the educational and learning process. AIM: To evaluate the impact of curricular changes, from a focus of assessment of learning to one of assessment for learning, in the perception of undergraduate students of medicine and their final grades obtained in a theoretical course (TCG). MATERIAL AND METHODS: In the year 2011 lectures were reduced and intermediate assessments followed by a feedback session were introduced. The activities of each program course, surveys about student perceptions of the course and the final grades of students (assessments with multiple choice questions) were compared between the periods prior and after curricular changes (2005-2010 and 2011-2013). RESULTS: As a consequence of curricular changes, time for lectures was reduced by 19.5%, time for summative assessments was increased by 8.5%, and feedback activity, occupying 7.3% of the course time was added. There were significant improvements in student is perceptions in all areas assessed by surveys, emphasizing feedback and assessments. The overall grade assigned to the course dictated after implementing the changes increased from 6.18 to 6.59 (p < 0.001, 1-7 scale). The grades of students also improved from an average of 5.78 to 6.43 (p < 0.001, 1-7 scale). CONCLUSIONS: Assessment for learning achieved the desired educational impact without increasing the assigned curricular time. Programmatic assessment is favorably perceived by students.

Estructura factorial de la escala DREEM en estudiantes de medicina chilenos

Background: The entry to a University requires an adaptation process that not all students solve with the same kind of success. Even though students social adaptation and emotional skills are essential, the educational environmental that they perceive has a significant influence in their academic life. Aim: To describe the changes in the perception about academic environment that medical students experience during the first three years of undergraduate career. Material and Methods: The Dundee Ready Education Environment Measure (DREEM) scale was applied to 525 first to third year medical students and an exploratory factorial analysis was made. Results: Four factors were identified: Academic Perception: academic quality that students attribute to the process in which they take part, as well as to the assessment that they do of their learning outcomes (coefficient ± = 0.85); Academic Experience: refers to positive emotions that students experience during the career such as confidence, pleasure and energy (coefficient ± = 0.76); Atmosphere Perception, comfort and calm that students experiment during their academic activities (coefficient ± = 0.79); Teachers Perception: the perception that students have of teachers about their interest and disposition towards students (coefficient ± = 0.50). Conclusions: The assessment of academic environment quality is inversely associated with the lapse that the students have spent in their undergraduate careers.

Evaluación para el aprendizaje: experiencia en un curso teórico de pregrado en medicina / Assessment for learning: experience in an undergraduate medical theoretical course

Background: Assessment for learning is a paradigm that is taking shape in the field of medical education. This approach aims to embed the assessment process within the educational and learning process. Aim: To evaluate the impact of curricular changes, from a focus of assessment of learning to one of assessment for learning, in the perception of undergraduate students of medicine and their final grades obtained in a theoretical course (TCG). Material and Methods: In the year 2011 lectures were reduced and intermediate assessments followed by a feedback session were introduced. The activities of each program course, surveys about student perceptions of the course and the final grades of students (assessments with multiple choice questions) were compared between the periods prior and after curricular changes (2005-2010 and 2011-2013). Results: As a consequence of curricular changes, time for lectures was reduced by 19.5%, time for summative assessments was increased by 8.5%, and feedback activity, occupying 7.3% of the course time was added. There were significant improvements in student is perceptions in all areas assessed by surveys, emphasizing feedback and assessments. The overall grade assigned to the course dictated after implementing the changes increased from 6.18 to 6.59 (p < 0.001, 1-7 scale). The grades of students also improved from an average of 5.78 to 6.43 (p < 0.001, 1-7 scale). Conclusions: Assessment for learning achieved the desired educational impact without increasing the assigned curricular time. Programmatic assessment is favorably perceived by students.

[Perception of medical students about formative assessments during clinical courses]. / Utilidad de la evaluación formativa en cursos clínicos integrativos en estudiantes de pregrado de medicina.

BACKGROUND: Formative evaluation is a range of formal assessment employed by professors during the teaching process in order to modify curriculum activities, to improve student attainment. For students, it is helpful to evaluate their learning process. Although recommended, it is seldom used. AIM: To evaluate the perception and performance of medical students subjected to formative assessments during an integrative clinical course. MATERIAL AND METHODS: Fourth year medical students that participated in a multiple choice formative assessment, similar to the final exam during 2007 and 2008, responded a survey about the usefulness and quality of such assessment. Student achievement was expressed as the percentage of correct answers of the tests. RESULTS: The formative assessment was answered by 99% of students. In 2007 97% of students considered the experience as excellent or very good and 92% evaluated it as useful or very useful. During 2008 the figures were 89% and 79%. The students outlined that this assessment oriented their study, allowed them to discover their weaknesses and have a perception of the degree of difficulty of the final exam. Over 90% of students that took the formative evaluation, improved their academic achievement. CONCLUSIONS: Formative assessments are well evaluated by medical students and improve their academic achievement.

Utilidad de la evaluación formativa en cursos clínicos integrativos en estudiantes de pregrado de medicina / Perception of medical students about formative assessments during clinical courses

Background: Formative evaluation is a range of formal assessment employed by professors during the teaching process in order to modify curriculum activities, to improve student attainment. For students, it is helpful to evaluate their learning process. Although recommended, it is seldom used. Aim: To evaluate the perception and performance of medical students subjected to formative assessments during an integrative clinical course. Material and Methods: Fourth year medical students that participated in a multiple choice formative assessment, similar to the final exam during 2007 and 2008, responded a survey about the usefulness and quality of such assessment. Student achievement was expressed as the percentage of correct answers of the tests. Results: The formative assessment was answered by 99% of students. In 2007 97% of students considered the experience as excellent or very good and 92% evaluated it as useful or very useful. During 2008 the figures were 89% and 79%. The students outlined that this assessment oriented their study, allowed them to discover their weaknesses and have a perception of the degree of difficulty of the final exam. Over 90% of students that took the formative evaluation, improved their academic achievement. Conclusions: Formative assessments are well evaluated by medical students and improve their academic achievement.

[Are residents and interns prepared for teaching? Perceptions of undergraduate and graduate medical students]. / ¿Están preparados los residentes e internos para enseñar? Percepción de estudiantes de medicina de pre y postgrado.

BACKGROUND: Every doctor is expected to be competent in teaching. There are few initiatives to prepare medical students for this role. AIM: To explore residents (graduate students) and interns (final year undergraduate students) perceptions of the importance of acquiring teaching skills and how prepared they feel to meet this role. To determine the importance that undergraduate students give to such teaching. MATERIAL AND METHODS: Residents and interns participated in focus groups, and completed the Medical Education Readiness Questionnaire (METRQ), 5th year medical students were also invited to complete it. RESULTS: Three hundred and seventy seven subjects answered the questionnaire. The perceived importance of having teaching skills was 6.1 +/-1.2 among residents and 5.7 +/- 1.6 among interns, in a scale 1 to 7. Their perception of their own preparation for teaching was 4.3 +/- 1.6 for both groups in the same scale. Students evaluated the preparation of the residents for teaching as 5.2 +/- 1.6 and that of the interns as 4.4 +/-1.7. Seventy-eight percent of 5th year medical students reported to learn more than two. 5 hours a week from residents. Fifty-nine percent of residents and 66% of interns reported to teach up to 2.5 hours per week to the same students. Focus groups participants agreed that teaching is an important role for a physician, and that to do it properly requires personal characteristics, along with teaching skills. They also found that the best opportunities to learn how to teach are during practical training. CONCLUSIONS. Our study contributes to the recognition of the teaching role of physicians and the need for teaching training among medical students.

Análise de um sistema de avaliação de aprendizagem para internato em pediatria baseado em exame clínico objetivo estruturado, observação de prática clínica e exame escrito / Analysis of a learning assessment system for pediatric internship based upon objective structured clinical examination, clinical practice observation and written examination

OBJETIVO: Descrever e analisar três ferramentas usadas no sistema de avaliação aplicado ao internato em pediatria por um período de 7 anos na Faculdade de Medicina, Pontificia Universidad Católica de Chile, Santiago, Chile. MÉTODOS: Pesquisa observacional retrospectiva projetada para as modalidades de avaliação usadas de 2001 a 2007 que foram implementadas no internato em pediatria. As ferramentas foram as seguintes: exame clínico objetivo estruturado (objective structured clinical examination, OSCE), exame escrito e diretrizes de observação de prática clínica diária (DOPCD). Os métodos de avaliação foram aplicados ao internato em pediatria do sexto ano, com um total de 697 alunos. A análise estatística incluiu uma avaliação descritiva, com correlação e regressões linear simples e múltipla (ANOVA), teste de Bonferroni e coeficiente alfa de Cronbach. O nível de significância foi de 5 por cento. RESULTADOS: Escores de sucesso com a OSCE foram atingidos em 75,7±8 por cento, com melhor média entre o sexo feminino (p < 0,001). Os escores de OSCE melhoraram após o terceiro ano de implementação. O coeficiente alfa de Cronbach foi de 0,11-0,78. O exame escrito teve escore médio de 79,8±10 por cento, e não houve diferenças quanto a gênero. O escore médio de DOPCD 97,1±3 por cento, com melhores resultados entre as mulheres (p < 0,005). A correlação entre os três métodos de avaliação mostrou uma relação positiva moderada, exceto no ano de 2007, quando a correlação foi maior (p < 0,001). CONCLUSÕES: Análise do sistema de avaliação da aprendizagem foi realizada usando OSCE, exame escrito e DOPCD, que são complementares entre si, gerando bons resultados.

OBJECTIVE: To describe and analyze three tools used in the assessment system applied to the pediatric internship over a 7-year period at the School of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile. METHODS: Retrospective observational research design for the assessment modalities implemented in the pediatric internship from 2001 through 2007. The tools were as follows: objective structured clinical examination (OSCE), written examination and daily clinical practice observation guidelines (DCPOG). The assessment methods were applied to the sixth-year pediatric internship with a total of 697 students. Statistical analysis included a descriptive assessment, with correlation and simple linear and multiple regressions (ANOVA), Bonferroni test and Cronbach's alpha coefficient. Significance level was set at p < 0.05. RESULTS: OSCE success scores were reached in 75.7±8 percent, with a better mean among females (p < 0.001). OSCE scores improved after the third year of implementation. Cronbach's alpha coefficient was 0.11-0.78. Written examination had a mean score of 79.8±10 percent and there were no sex differences. Mean DCPOG score was 97.1±3 percent and the results were better among females (p < 0.005). Correlation between the three assessment methods showed a moderate positive relationship except in the year of 2007, where the correlation was higher (p < 0.001). CONCLUSIONS: Analysis of the learning assessment system was performed using OSCE, written examination and DCPOG, which are complementary to each other, and yielded good results.

Analysis of a learning assessment system for pediatric internship based upon objective structured clinical examination, clinical practice observation and written examination.

OBJECTIVE: To describe and analyze three tools used in the assessment system applied to the pediatric internship over a 7-year period at the School of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile. METHODS: Retrospective observational research design for the assessment modalities implemented in the pediatric internship from 2001 through 2007. The tools were as follows: objective structured clinical examination (OSCE), written examination and daily clinical practice observation guidelines (DCPOG). The assessment methods were applied to the sixth-year pediatric internship with a total of 697 students. Statistical analysis included a descriptive assessment, with correlation and simple linear and multiple regressions (ANOVA), Bonferroni test and Cronbach's alpha coefficient. Significance level was set at p < 0.05. RESULTS: OSCE success scores were reached in 75.7+/-8%, with a better mean among females (p < 0.001). OSCE scores improved after the third year of implementation. Cronbach's alpha coefficient was 0.11-0.78. Written examination had a mean score of 79.8+/-10% and there were no sex differences. Mean DCPOG score was 97.1+/-3% and the results were better among females (p < 0.005). Correlation between the three assessment methods showed a moderate positive relationship except in the year of 2007, where the correlation was higher (p < 0.001). CONCLUSIONS: Analysis of the learning assessment system was performed using OSCE, written examination and DCPOG, which are complementary to each other, and yielded good results.

¿Están preparados los residentes e internos para enseñar?: Percepción de estudiantes de medicina de pre y postgrado / Are residents and interns prepared for teaching?: Perceptions of undergraduate and graduate medical students

Background: Every doctor is expected to be competent in teaching. There are few initiatives to prepare medical students for this role. Aim: To explore residents (graduate students) and interns (final year undergraduate students) perceptions of the importance of acquiring teaching skills and how prepared they feel to meet this role. To determine the importance that undergraduate students give to such teaching. Material and Methods: Residents and interns participated in focus groups, and completed the Medical Education Readiness Questionnaire (METRQ), 5th year medical students were also invited to complete it. Results: Three hundred and seventy seven subjects answered the questionnaire. The perceived importance of having teaching skills was 6.1 ±1.2 among residents and 5.7 ± 1.6 among interns, in a scale 1 to 7. Their perception of their own preparation for teaching was 4.3 ± 1.6 for both groups in the same scale. Students evaluated the preparation of the residents for teaching as 5.2 ± 1.6 and that of the interns as 4.4 ±1.7. Seventy-eight percent of 5th year medical students reported to learn more than two. 5 hours a week from residents. Fifty-nine percent of residents and 66 percent of interns reported to teach up to 2.5 hours per week to the same students. Focus groups participants agreed that teaching is an important role for a physician, and that to do it properly requires personal characteristics, along with teaching skills. They also found that the best opportunities to learn how to teach are during practical training. Conclusions. Our study contributes to the recognition of the teaching role of physicians and the need for teaching training among medical students.

[Application of Objective Structured Clinical Examination (OSCE) for pediatric internship assessment in two schools of medicine]. / Aplicación del Examen Clínico Objetivo Estructurado (OSCE) en la evaluación final del internado de pediatría en dos escuelas de medicina.

BACKGROUND: The Objective Structured Clinical Examination (OSCE) has become a respected and widely used tool for the assessment of clinical competence in medical education. AIM: To describe the first experience of an OSCE as a summative assessment in undergraduate Pediatric Internship, in two universities. MATERIAL AND METHODS: The OSCE was structured by a committee of faculty members of the 5 campi of University of Chile and I campus of the Catholic University. A 21 station OSCE was administered simultaneously to 124 Pediatric Interns (University of Chile = 104, Catholic University = 20), in 3 centers. A total of 50 faculty members participated in the examination. The OSCE consisted of 20 clinical problems, including videotape recordings, photographs, x-rays and laboratory exams, phantoms and 7 simulated standardized parents. RESULTS: The average total OSCE score was 67.3% (range: 84.5%-43.5%). The maximum theoretic score was achieved in 19 stations. A significant correlation between station and total score, was found for 18 of the 20 clinical problems. CONCLUSIONS: The experience of using OSCE has been a success. The OSCE was an adequate procedure to assess a large number of interns simultaneously and it allowed us to measure the main objectives in all domains and a wide range of clinical competence of Pediatric Internship Programs.

[Improvement of medical student's academic performances in times of curricular reform]. / Mejoría en el desempeño académico de estudiantes de medicina en tiempos de reforma curricular.

BACKGROUND: During the last decade, academic life at the medical school of the Pontificia Universidad Católica de Chile has been thoroughly affected by a curricular reform process. Changes started in 1993 and have continued up until now. This reform did not have an experimental design to allow for a scientific evaluation of its effects. However, it seems interesting to study the evolution of indices of academic performance of our students during this period. AIM: To evaluate the academic performance of medical students between 1989 and 1999. SUBJECTS AND METHODS: All undergraduate students enrolled between 1989 and 1999. Academic performance was evaluated by 1) failure to pass one or more courses, 2) delay to complete the third year of studies, 3) withdrawal from school and 4) medical school grades. RESULTS: All indices of academic performance changed during the period of study. Failure to pass, delay and withdrawal from school significantly decreased, whereas medical school grades improved, particularly in basic and pre-clinical subjects. CONCLUSIONS: Academic performance of medical students improved consistently between 1989 and 1999. While specific causal relationships cannot be established, we believe that this improvement is likely related to the curricular reform. This reform included horizontal and vertical integration of academic contents, greater emphasis in problem-based learning and additional instances of evaluation such as the repetition exam.