RESUMO
This paper reports the findings of a Canada based multi-institutional study designed to investigate the relationships between admissions criteria, in-program assessments, and performance on licensing exams. The study's objective is to provide valuable insights for improving educational practices across different institutions. Data were gathered from six medical schools: McMaster University, the Northern Ontario School of Medicine University, Queen's University, University of Ottawa, University of Toronto, and Western University. The dataset includes graduates who undertook the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1) between 2015 and 2017. The data were categorized into five distinct sections: demographic information as well as four matrices: admissions, course performance, objective structured clinical examination (OSCE), and clerkship performance. Common and unique variables were identified through an extensive consensus-building process. Hierarchical linear regression and a manual stepwise variable selection approach were used for analysis. Analyses were performed on data set encompassing graduates of all six medical schools as well as on individual data sets from each school. For the combined data set the final model estimated 32% of the variance in performance on licensing exams, highlighting variables such as Age at Admission, Sex, Biomedical Knowledge, the first post-clerkship OSCE, and a clerkship theta score. Individual school analysis explained 41-60% of the variance in MCCQE1 outcomes, with comparable variables to the analysis from of the combined data set identified as significant independent variables. Therefore, strongly emphasising the need for variety of high-quality assessment on the educational continuum. This study underscores the importance of sharing data to enable educational insights. This study also had its challenges when it came to the access and aggregation of data. As such we advocate for the establishment of a common framework for multi-institutional educational research, facilitating studies and evaluations across diverse institutions. This study demonstrates the scientific potential of collaborative data analysis in enhancing educational outcomes. It offers a deeper understanding of the factors influencing performance on licensure exams and emphasizes the need for addressing data gaps to advance multi-institutional research for educational improvements.
RESUMO
BACKGROUND: As the demand increases for demonstration of competence in surgical skill, the need for validated assessment tools also increases. HYPOTHESIS/PURPOSE: The purpose of this study was to validate a dry knee model for the assessment of performance of anterior cruciate ligament reconstruction (ACLR). The hypothesis was that the combination of a checklist and a previously validated global rating scale would be a valid and reliable means of assessing ACLR when performed by residents in a dry model. STUDY DESIGN: Controlled laboratory study. METHODS: All residents, sports medicine staff, and fellows were invited to perform a hamstring ACLR using anteromedial drilling and Endobutton fixation on a dry model of an anterior cruciate ligament. Previous exposure to knee arthroscopy and ACLR was recorded. A detailed surgical manuscript and technique video were sent to all participants before the study. Residents were evaluated by staff surgeons with task-specific checklists created by use of a modified Delphi procedure and the Arthroscopic Surgical Skill Evaluation Tool (ASSET). Each procedure (hand movements and arthroscopic video) was recorded and scored by a fellow blinded to the year of training of each participant. RESULTS: A total of 29 residents, 5 fellows, and 6 staff surgeons (40 participants total) performed an ACLR on the dry model. The internal reliability (Cronbach alpha) of the test when using the total ASSET score was very high (>0.9). One-way analysis of variance for the total ASSET score and the total checklist score demonstrated a difference between participants based on year of training (P < .001). Post hoc analysis demonstrated a significant difference in global ratings and checklist scores between junior residents (postgraduate year [PGY] 1-3) and senior residents (PGY 4 and 5) and senior residents and fellows (P < .05). A significant difference was seen between fellows and staff on the global rating (P < .05) but not on the checklist scores (P > .05). A good correlation was seen between the total ASSET score and prior exposure to knee arthroscopy (0.73) and ACLR (0.65). The interrater reliability (intraclass correlation coefficient) between the examiner ratings and the blinded assessor ratings for the total ASSET score was very high (>0.8). CONCLUSION: The results of this study provide evidence that the performance of an ACLR in a dry model is a reliable method of assessing a resident's knowledge of the steps and instrumentation required, and the method shows evidence of validity. CLINICAL RELEVANCE: These models may be used to ensure a minimal level of competence before residents and fellows perform ACLR in the operating room.
