The Effects of a Pass-Fail Curriculum on a Physician Assistant Program.

PURPOSE: The grading scale for students in a physician assistant program of study is not standardized. Students may be evaluated on a traditional 5-tiered A to F scale or a pass-fail system. The decision to change from ordered grading to pass-fail at an established program in the southeast was done following a change in the affiliated School of Medicine. The purpose of this study was to review effects on student scores following such a change. METHODS: The Physician Assistant National Certifying Exam (PANCE) and PACKRAT 2 exam scores for the last 2 cohorts of students scored in the 5-tiered system (2016, N = 60 and 2017, N = 59) were compared against the same for the first 2 cohorts (2018, N = 59 and 2019, N = 58) of the pass-fail system. Nonrandom sampling of all students in each cohort year was evaluated using 2-tailed t-testing. RESULTS: A total of 236 student scores were evaluated using a 95% confidence interval. The traditionally scored classes outperformed all pass-fail cohorts (means 460.67/491.86 versus 503.34/493.92). P values were found to be significant at all values between the 5-tier scored classes and the pass-fail cohorts in PANCE scoring, resulting in failure to reject the null hypothesis. This was also true for the PACKRAT 2 with the exception of the 2019 cohort, which was significant only for outperformance of the other pass-fail cohort. For the purpose of this study, the only analysis performed was scoring. CONCLUSION: For cohorts undergoing curricular change, unforeseen impacts on initial standardized exam scores may occur. In this study, PANCE scores for the first year of the 2 pass-fail cohorts decreased while the overall program scores remained at or above the national average. The pass-fail cohort did show an upward trend in the second year of the curriculum, suggesting that as programs become more familiar with the pass-fail system, steady improvements occur. This suggests that while an anticipated drop in initial scores may be expected, further studies are needed to evaluate the impact on stress reduction, long retention, and intraclass competition.

Using Data Mining for the Early Identification of Struggling Learners in Physician Assistant Education.

PURPOSE: Despite the importance of early intervention and remediation, the relatively short duration of physician assistant education programs necessitates the importance of early identification of at-risk learners. This study sought to ascertain whether machine learning was more effective than logistic regression in predicting remediation status among students, using the limited set of data available before or immediately following the first semester of study as predictor variables and academic remediation as an outcome variable. METHODS: The analysis included one institution and student data from 177 graduates between 2017 and 2019. We employed one data mining model, random forest trees, and compared it to a traditional predictive analysis method, logistic regression. Due to the small sample size, we employed leave-one-out cross-validation and bootstrap aggregation. RESULTS: Data provided evidence that the random forest algorithm correctly identified individuals who would later experience academic intervention with a 63.3% positive predictive value, whereas logistic regression exhibited a positive predictive value of 16.6%. CONCLUSIONS: This single-institution study indicates that predictive modeling, employing machine learning, may be a more effective means than traditional statistical methods of identifying and providing assistance to learners who may experience academic challenges.

PAEA End of Rotation™ Exams Accurately Identify Students at Risk of Physician Assistant National Certifying Exam Failure.

PURPOSE: Physician Assistant Education Association (PAEA) End of Rotation™ exams are used by programs across the country. However, little information exists on the predictive ability of the exams' scale scores and Physician Assistant National Certifying Exam (PANCE) performance. The purpose of this study was to evaluate End of Rotation exam scores and their relationship with poor PANCE performance (PPP). METHODS: In an IRB-approved, multi-center, multi-year study, associations between PAEA End of Rotation exam scale scores and PANCE scores were explored. A taxonomy of nested linear regression models with random intercepts was fit at the program level. Fully adjusted models controlled for year, timing of the exam, student age, and gender. RESULTS: Fully adjusted linear models found that 10-point increases in End of Rotation exam scores were associated with a 16.8-point (95% confidence interval [CI]: 14.1-19.6) to 23.5-point (95% CI: 20.6-26.5) increase in PANCE score for Women's Health and Emergency Medicine, respectively. Associations between exams did not significantly vary (P = .768). Logistic models found End of Rotation exam scores were strongly and consistently associated with lower odds of PPP, with higher exam scores (10-point increase) associated with decrements in odds of PPP, ranging between 37% and 48% across exams. The effect estimate for the Emergency Medicine exam was consistently stronger in all models. CONCLUSIONS: PAEA End of Rotation exam scores were consistently predictive of PPP. While each End of Rotation exam measures a specialty content area, the association with the overall PANCE score varied only by a change in odds of low performance or failure by a small percentage. Low End of Rotation exam scores appear to be consistent predictors of PPP in our multi-center cohort of physician assistant students.

Considerations Around Predicting Physician Assistant National Certifying Exam Scores From PACKRAT®: A Multiprogram Study.

PURPOSE: The Physician Assistant Clinical Knowledge Rating and Assessment Tool (PACKRAT®) is a known predictor of performance on the Physician Assistant National Certifying Exam (PANCE). It is unknown, however, whether these associations (1) vary across programs; (2) differ by PACKRAT metrics (first-year [PACKRAT 1], second-year [PACKRAT 2], and composite score [arithmetic mean of PACKRAT 1 and PACKRAT 2]); or (3) are modified by demographic or socioeconomic variables. METHODS: Linear and logistic hierarchical regression models (HRMs) were used to evaluate associations between PACKRAT metrics and (1) continuous PANCE scores and (2) odds of low PANCE performance (LPP), respectively. Likelihood ratio tests were used to evaluate differences in associations between programs and effect modification by demographic and socioeconomic variables. Receiver operating characteristic (ROC) curves were used to examine the sensitivity, specificity, positive predictive values, and negative predictive values for various PACKRAT metrics/cut points. Models were adjusted for demographic and socioeconomic variables. The PACKRAT scores were standardized for each year to the national mean and SD. RESULTS: Adjusted HRMs across 5 programs (n = 1014) found the composite score to have the strongest association, with a 10-percentile-point increase associated with a 22-point (95% confidence interval [CI]: 19-26) increase in PANCE score. The composite score also strongly predicted decrements in odds of LPP (odds ratio: 0.46; 95% CI: 0.38-0.55). Hierarchical regression models and ROC curves identified significant variability in associations among programs. Effect modification was not observed by any investigated variable. CONCLUSIONS: The composite score had the largest magnitudes of association with PANCE scores and odds of LPP. The significant difference in association identified between programs suggests that the predictive ability of the exam is not uniform. The lack of effect modification by demographic and socioeconomic variables suggests that associations do not significantly differ by these metrics.