The impact of repeated item development training on the prediction of medical faculty members' item difficulty index.

BACKGROUND: Item difficulty plays a crucial role in assessing students' understanding of the concept being tested. The difficulty of each item needs to be carefully adjusted to ensure the achievement of the evaluation's objectives. Therefore, this study aimed to investigate whether repeated item development training for medical school faculty improves the accuracy of predicting item difficulty in multiple-choice questions. METHODS: A faculty development program was implemented to enhance the prediction of each item's difficulty index, ensure the absence of item defects, and maintain the general principles of item development. The interrater reliability between the predicted, actual, and corrected item difficulty was assessed before and after the training, using either the kappa index or the correlation coefficient, depending on the characteristics of the data. A total of 62 faculty members participated in the training. Their predictions of item difficulty were compared with the analysis results of 260 items taken by 119 fourth-year medical students in 2016 and 316 items taken by 125 fourth-year medical students in 2018. RESULTS: Before the training, significant agreement between the predicted and actual item difficulty indices was observed for only one medical subject, Cardiology (K = 0.106, P = 0.021). However, after the training, significant agreement was noted for four subjects: Internal Medicine (K = 0.092, P = 0.015), Cardiology (K = 0.318, P = 0.021), Neurology (K = 0.400, P = 0.043), and Preventive Medicine (r = 0.577, P = 0.039). Furthermore, a significant agreement was observed between the predicted and actual difficulty indices across all subjects when analyzing the average difficulty of all items (r = 0.144, P = 0.043). Regarding the actual difficulty index by subject, neurology exceeded the desired difficulty range of 0.45-0.75 in 2016. By 2018, however, all subjects fell within this range. CONCLUSION: Repeated item development training, which includes predicting each item's difficulty index, can enhance faculty members' ability to predict and adjust item difficulty accurately. To ensure that the difficulty of the examination aligns with its intended purpose, item development training can be beneficial. Further studies on faculty development are necessary to explore these benefits more comprehensively.

Changes in self-reflective thinking level in writing and educational needs of medical students: A longitudinal study.

INTRODUCTION: The purpose of this study was to longitudinally examine the change in understanding of the self-reflection method, reflective thinking, and writing attitude and perception. Moreover, we investigated students' educational needs and methods regarding self-reflection. METHODS: The subjects were 117, who were in the pre-medical course in 2017 and were promoted to the medical course in 2019. Questions concerning students' understanding of self-reflection methods, their attitude and perception of reflective writing, and educational needs and methods regarding self-reflection were self-developed. For students' reflective thinking level in writing, we used the approach developed by Galvez-Martin, Bowman, and Morrison and adapted by Kwon. For dada analysis, χ2 test, t-test, frequency analysis was used. RESULTS: We found that students' level of understanding regarding self-reflection methods increases slightly, but not significant (χ2 = 2.238, p>0.05). There was no significant change in the level of reflective thinking in writing (χ2 = 8.003, p>0.05). The students' attitude toward reflective writing decreased in the medical course than in the pre-medical course (t = 3.475, p<0.001). The perception that reflective writing was helpful for individual improvement decreased during the medical course (t = 4.931, p<0.001). The need for self-reflection education increased in the medical course (t = -2.659, p<0.001). They preferred implementation in the first year of the medical course as an instructor-led special lecture. CONCLUSION: Self-reflective ability is not naturally developed as students' progress through grade levels. Educational intervention is needed to help students understand approaches to self-reflection and its importance in enabling them to develop their abilities as well as to participate actively in reflective writing.

Heart Rate Variability-Measured Stress and Academic Achievement in Medical Students.

OBJECTIVES: Stress can affect learning and memory in students. Previous studies on stress in students were conducted mainly through surveys. We investigated how heart rate variability (HRV)-measured stress is related to academic achievement in medical students during clerkship. SUBJECTS AND METHODS: A cross-sectional study was conducted. HRV measurements were performed in 97 third-year medical students during their family medicine clerkship course. Data on written and skilled examination scores of the end of the school year were also collected. We assessed the association between HRV-measured stress and written/skilled examination scores. RESULTS: Written examination scores showed a positive correlation with standard deviation of the normal-to-normal intervals (SDNNs) (r = 0.245, p = -0.016), sympathetic nervous system/parasympathetic nervous system (SNS/PNS) balance (r = 0.218, p = 0.033), and stress index (r = 0.381, p = 0.004). Students with an unhealthy SDNN, a dominant SNS, and a high stress index had higher scores in written examinations than students with a healthy SDNN, a balanced SNS/PNS, and a normal stress index, respectively (p = 0.004, 0.018, and 0.012, respectively). Moreover, skilled examination scores were negatively correlated with BMI (r = -0.249, p = 0.014) and were higher in female students (r = 0.240, p = 0.018). Students with an abnormal autonomic balance diagram (ABD) had a higher score than students with a normal ABD (p = 0.03). CONCLUSIONS: This study shows that medical students with higher stress measured by HRV have higher academic achievement, especially in written examinations. Further studies are needed to confirm the results of this study and to assess the long-term effects of HRV-measured stress on medical students.

Impact of video feedback system on medical students' perception of their clinical performance assessment.

BACKGROUND: Providing feedback on student performance in the clinical performance assessment (CPA) is meaningful in that it helps students understand their strengths and weaknesses. This study compared students' perception of their CPA scores before and after providing personalized video feedback. METHODS: Two identical online surveys of Year 1 medical students (N = 103) that had undergone CPA were conducted to evaluate students' perceptions about their CPA scores before and after video feedback. Students were given their test scores with assessment analysis reports immediately after completing the CPA. Top-scored students from each station agreed to provide their video-recorded performance to the rest of the students. RESULTS: After comparing their performance video and top-scored video at each station, medical students were more aware of their CPA total score, clinical performance examination (CPX) total score, score of each CPX station, section score for the CPX station, history taking section score, physical examination section score, and doctor-patient relationship section score. Moreover, students became more convinced of their own weaknesses from their history taking and patient education section after viewing video feedback than before. CONCLUSION: The use of the video feedback system might help students recognize their CPA results and identify their strengths and weaknesses.

Holistic rubric vs. analytic rubric for measuring clinical performance levels in medical students.

BACKGROUND: Task-specific checklists, holistic rubrics, and analytic rubrics are often used for performance assessments. We examined what factors evaluators consider important in holistic scoring of clinical performance assessment, and compared the usefulness of applying holistic and analytic rubrics respectively, and analytic rubrics in addition to task-specific checklists based on traditional standards. METHODS: We compared the usefulness of a holistic rubric versus an analytic rubric in effectively measuring the clinical skill performances of 126 third-year medical students who participated in a clinical performance assessment conducted by Pusan National University School of Medicine. We conducted a questionnaire survey of 37 evaluators who used all three evaluation methods-holistic rubric, analytic rubric, and task-specific checklist-for each student. The relationship between the scores on the three evaluation methods was analyzed using Pearson's correlation. Inter-rater agreement was analyzed by Kappa index. The effect of holistic and analytic rubric scores on the task-specific checklist score was analyzed using multiple regression analysis. RESULTS: Evaluators perceived accuracy and proficiency to be major factors in objective structured clinical examinations evaluation, and history taking and physical examination to be major factors in clinical performance examinations evaluation. Holistic rubric scores were highly related to the scores of the task-specific checklist and analytic rubric. Relatively low agreement was found in clinical performance examinations compared to objective structured clinical examinations. Meanwhile, the holistic and analytic rubric scores explained 59.1% of the task-specific checklist score in objective structured clinical examinations and 51.6% in clinical performance examinations. CONCLUSION: The results show the usefulness of holistic and analytic rubrics in clinical performance assessment, which can be used in conjunction with task-specific checklists for more efficient evaluation.