Usefulness of ubiquitous-based testing for evaluations in medical education / 한국의학교육

PURPOSE: The purpose of this study was to evaluate the suitability (convenience, objectiveness, and satisfaction) of ubiquitous-based testing (UBT) as a medical education evaluation tool. METHODS: UBT was administered using a smart pad in our medical school in May 2012. A questionnaire was given twice. The pre-UBT questionnaire examined possession of a tablet computer, skillfulness of smart devices, the convenience of UBT, and the usefulness of a medical educational assessment tool. The post-UBT questionnaire evaluated the satisfaction, convenience, and preference of UBT and the usefulness of a medical educational assessment tool, as in the pre-UBT test. The survey was measured on a 4-point scale: 1 is "strongly disagree" and 4 is "strongly agree." RESULTS: One hundred three students (male, 55.3%) participated in the UBT. The mean age was 29.2+/-2.4 years. In the pre-UBT questionnaire analysis, students responded affirmatively to the items about the skillfulness of smart devices, clinical skill assessment, and achievement of educational objectives. The responses to the items on the convenience and satisfaction with the UBT were positive in the post-UBT. The factors that affected the post-UBT questionnaire were as follows: knowledge assessment (p=0.041) and achievement of educational objectives (p=0.015) were significant, based on gender, and satisfaction with the UBT (p=0.002) was significant, based on possession of a tablet computer. The relationship between the ranks of this UBT and the average ranks of the three previous semesters was statistically significant (p<0.001). CONCLUSION: Convenience, objectiveness, knowledge assessment, and composition and completion were useful items in the UBT.

Can computerized tests be introduced to the Korean Medical Licensing Examination?

In November 2011, the standing Committee of the Korean Medical Licensing Examination (KMLE) recommended that the National Health Personnel Licensing Examination Board introduce computerized testing to the KMLE. Therefore this article contextualizes and explores the possibility of applying computerized testing to the KMLE. Computerized testing comprises computer-based testing (CBT), ubiquitous-based testing (UBT), internet-based testing (IBT), and computerized adaptive testing (CAT). CBT refers to testing administered via a computer as the user interface, while testing with a smart phone or smart pad as the user interface is known as UBT. IBT is testing done online, and CAT is testing tailored so that each item provided fits the examinee's ability level. The benefits and drawbacks of each computerized testing option were surveyed. Among them, I propose CAT as the final goal for KMLE. In order to implement the computerized testing more effectively, it is recommended that items contain multimedia data and should involve interpretation or problem-solving. More evidence is needed to support the positive impact of computerized testing for undergraduate medical education and primary health care. Since the rapid progress of information technology such as internet bandwidth and human-computer interface methods, the introduction of computerized testing to KMLE will soon be plausible. It is possible to increase the quality of the KMLE with the introduction of computerized testing. Medical schools should prepare for the new testing environment of the KMLE by recruiting or training specialists in this field.

Introduction to an Open Source Internet-Based Testing Program for Medical Student Examinations / 보건의료교육평가

The author developed a freely available open source internet-based testing program for medical examination. PHP and Java script were used as the programming language and postgreSQL as the database management system on an Apache web server and Linux operating system. The system approach was that a super user inputs the items, each school administrator inputs the examinees' information, and examinees access the system. The examinee's score is displayed immediately after examination with item analysis. The set-up of the system beginning with installation is described. This may help medical professors to easily adopt an internet-based testing system for medical education.

Comparative Analysis of Faculty Development in Japanese Medical Schools from 2003 through 2005 / 医学教育

1) The faculty development at each medical school from 2003 through 2005 was analysed.<BR>2) The major themes in faculty development were problem based learning, tutorial, computer based testing, and clinical training.<BR>3) Faculty development is considered an effective way to enhance the contributions of faculty members to medical education.

Analysis of the Questionarre for Nationwide Common Achievement Test System for Entering Clinical Clerkship in Japan / 医学教育

Nationwide common achievement test system for entering clinical clerkship will formally start from December 2005. Before the start, four times trial examination were carried out. Based on these trials, some problems for the test sytem were pointed out. We have analysed the problems proposed to induce more suitable examination system.

Fatigue among Medical Students in a Trial of Computer-based Testing for Common Achievement Tests / 医学教育

We investigated the degree of fatigue among medical students during the second trial of the Common Achievement Tests, which use a computer-based testing (CBT) method. A revised questionnaire for subjective fatigue symptoms proposed by the Industrial Fatigue Research meeting of the Japan Society for Occupational Health was used to examine the degree of fatigue. The CBT examinee group (n=41) sat for the examination for 6 hours using video display terminals. Significant changes were seen in 19 of the 25 items for subjective symptoms. At the end of the test period, significant differences between the CBT examinee group and the lecture participant group (n=50) were found for 15 of the items for subjective symptoms. Subjective symptoms were classified into 5 categories: sleepiness, instability, displeasure, feeling languid, and blurred vision. In the CBT examinee group, rates of symptoms increased significantly in all 5 categories. The increase in the rate of “blurred vision” was especially marked.

Statistical Analysis with the Item-Response Theory of the First Trial of the Computer-Based Nationwide Common Achievement Test in Medicine / 医学教育

Data from the first trial of the computer-based nationwide common achievement test in medicine, carried out from February through July in 2002, were analyzed to evaluate the applicability of the item-response theory. The trial test was designed to cover 6 areas of the core curriculum and included a total of 2791 items. For each area, 3 to 40 items were chosen randomly and administered to 5693 students in the fourth to sixth years; the responses of 5676 of these students were analyzed with specifically designed computer systems. Each student was presented with 100 items. The itemresponse patterns were analyzed with a 3-parameter logistic model (item discrimination, item difficulty, and guessing parameter). The main findings were: 1) Item difficulty and the percentage of correct answers were strongly correlated (r=-0.969to-0.982). 2) Item discrimination and the point-biserial correlation were moderately strongly correlated (r=0.304 to 0.511). 3) The estimated abilities and the percentage of correct answers were strongly correlated (r=0.810 to 0.945). 4) The mean ability increased with school year. 5) The correlation coefficients among the 6 curriculum area ability scores were less than 0.6. Because the nationwide common achievement test was designed to randomly present items to each student, the item-response theory can be used to adjust the differences among test sets. The first trial test was designed without considering the item-response theory, but the second trial test was administered with a design better suited for comparison. Results of an analysis of the second trial will be reported soon.

Effect of Workshops for Preparing Multiple-Choice Questions for Computer-Based Testing Used in the Nationwide Common Examination Before Clinical Clerkships / 医学教育

Computer-based testing (CBT) has been used in Japan since 2002 to assess medical students' basic and clinical medical knowledge, based on the model core-curriculum, before they start clinical clerkships. For effective CBT, multiplechoice questions must accurately assess the knowledge of students. Questions for CBT are submitted by all medical schools in Japan. However, only 40% of questions are chosen for CBT and used at random; the other 60% of questions are rejected because of poor quality. Toimprove the ability of medical staff to devise questions, workshops were held at 30 medical schools. The acceptance rate of questions from schools where workshops were held was significantly increased. The workshops were extremely effbctive for improving the quality of questions.

Comparison of Fourth-year Comprehensive Examinations and Computer-based Testing in the Nationwide Medical and Dental Student Evaluation System / 医学教育

To demonstrate the quality assurance of comprehensive examinations for fourth-year students at Nippon Medical School, scores on comprehensive examinations were compared with those on a trial of computer-based testing (CBT) of a nationwide medical and dental student evaluation system in 2003. Pearson's correlation coefficients between scores of two comprehensive examinations and the CBT score were 0.45 and 0.67, and the correlation coefficient between the average score of the two comprehensive examinations and the CBT score was 0.55. Fourth-year comprehensive examinations are useful tools for summative evaluation and prediction of CBT performance.

Report of the First Trial of Nationwide Common Achievement Test-Computer-Based Testing in Medicine / 医学教育

The first trial of common achievement test-computer-based testing was held from January through August 2002. The number of examinees was 5, 693, of whom 5, 676 were analyzed. Single-best-answer, five-choice questions were used. The highest score was 92 points, the lowest score was 19 points, and the average score was 55.9±10.2 points (standard deviation). Scores were distributed normally. The test sets did not differ significantly in difficulty, although test-set items differed for each student. The percentage of correct answers, the ∅-coefficient, and the point-biserial correlation coefficient were calculated for each category of the model core curriculum. The percentage of correct answers was highest in category A of the model core curriculum, and percentages of correct answers were similar in categories B, C, D, E, and F. The ∅-coefficient and the correlation coefficient were low in categories A and F and were highest in category C. Although the percentage of correct answers in this trial was lower than expected, many test items had discriminatory power. The Test Items Evaluation Subcommittee is now evaluating test items, determining pool items, and revising new test items for the second trial and expect to compile a useful item bank.

A Preliminary Validation of Computer-Based Testing by the Common Achievement Tests Organization in Japan: An Early Report / 医学教育

The Common Achievement Tests Organization performed the first nationwide trial of computer-based testing (CBT) used to assess students entering the clinical phase of medical education. Seventy-seven medical schools participated in the trial. We compared performance on the national CBT with performance on preclinical tests administered at Gifu University School of Medicine. Despite some methodologic differences between the national CBT and our system, the overall results correlated well. Students who did poorly on the national CBT also did poorly on Gifu University's preclinical test. Correlation of these two performance scales suggests that nationwide CBT could be used to accurately assess preclinical skills.

Problems in Preparing Multiple Choice Questions for Computer-based Testing Used in the Nationwide Common Examination before Clinical Clerkships / 医学教育

In 2002, Japanese medical students began computer-based testing (CBT) to assess their basic and clinical medical knowledge, based on the model core-curriculum, before starting clinical clerkships. Of 9, 919 multiple choice questions submitted by 80 medical schools, 2, 791 were used for CBT and 7, 128 were rejected. To improve the quality of future CBT, we analyzed why questions were rejected. The most commons reasons were difficulty, length, and inappropriate choice of answers. A training course may be needed to improve the ability of medical school staff to devise questions.