RESUMO
CONTEXT: Predicting residents' academic performance is critical for medical educational institutions to plan strategies for improving their achievement. AIMS: This study aimed to predict the performance of residents on preboard examinations based on the results of in-training examinations (ITE) using various educational data mining (DM) techniques. SETTINGS AND DESIGN: This research was a descriptive cross-sectional pilot study conducted at Iran University of Medical Sciences, Iran. PARTICIPANTS AND METHODS: A sample of 841 residents in six specialties participating in the ITEs between 2004 and 2014 was selected through convenience sampling. Data were collected from the residency training database using a researcher-made checklist. STATISTICAL ANALYSIS: The analysis of variance was performed to compare mean scores between specialties, and multiple-regression was conducted to examine the relationship between the independent variables (ITEs scores in postgraduate 1st year [PGY1] to PG 3rd year [PGY3], sex, and type of specialty training) and the dependent variable (scores of postgraduate 4th year called preboard). Next, three DM algorithms, including multi-layer perceptron artificial neural network (MLP-ANN), support vector machine, and linear regression were utilized to build the prediction models of preboard examination scores. The performance of models was analyzed based on the root mean square error (RMSE) and mean absolute error (MAE). In the final step, the MLP-ANN was employed to find the association rules. Data analysis was performed in SPSS 22 and RapidMiner 7.1.001. RESULTS: The ITE scores on the PGY-2 and PGY-3 and the type of specialty training were the predictors of scores on the preboard examination (R 2 = 0.129, P < 0.01). The algorithm with the overall best results in terms of measuring error values was MLP-ANN with the condition of ten-fold cross-validation (RMSE = 0.325, MAE = 0.212). Finally, MLP-ANN was utilized to find the efficient rules. CONCLUSIONS: According to the results of the study, MLP-ANN was recognized to be useful in the evaluation of student performance on the ITEs. It is suggested that medical, educational databases be enhanced to benefit from the potential of DM approach in the identification of residents at risk, allowing instructors to offer constructive advice in a timely manner.
RESUMO
Professionalism is a core competency in the medical profession. In this paper, we aimed to confirm the validity, reliability and acceptability of the Professionalism Mini-Evaluation Exercise (P-MEX) instrument for the emergency medicine (EM) residency program. Twenty-two EM attending physicians completed 383 P-MEX forms (the Persian version) for 90 EM residents. Construct validity was assessed via structural equation modeling (SEM). The reliability coefficient was estimated by the generalizability theory, and acceptability was assessed using two researcher-made questionnaires to evaluate the perspectives of residents and assessors. There was a consensus among the participants regarding the content of P-MEX. According to the results of SEM, the first implementation of the original model was associated with a moderate fit and high item loadings. The model modified with correlated error variances for two pairs of items showed an appropriate fit. The reliability of P-MEX was 0.81 for 14 occasions. The perception survey indicated high acceptability for P-MEX from the viewpoint of the residents and increasing satisfaction with P-MEX among the assessors over time. According to the results of the research, P-MEX is a reliable, valid, and acceptable instrument for assessing professionalism in EM residents.
