Performance on the Nephrology In-Training Examination and ABIM Nephrology Certification Examination Outcomes.

BACKGROUND AND OBJECTIVES: Medical specialty and subspecialty fellowship programs administer subject-specific in-training examinations to provide feedback about level of medical knowledge to fellows preparing for subsequent board certification. This study evaluated the association between the American Society of Nephrology In-Training Examination and the American Board of Internal Medicine Nephrology Certification Examination in terms of scores and passing status. DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS: The study included 1684 nephrology fellows who completed the American Society of Nephrology In-Training Examination in their second year of fellowship training between 2009 and 2014. Regression analysis examined the association between In-Training Examination and first-time Nephrology Certification Examination scores as well as passing status relative to other standardized assessments. RESULTS: This cohort included primarily men (62%) and international medical school graduates (62%), and fellows had an average age of 32 years old at the time of first completing the Nephrology Certification Examination. An overwhelming majority (89%) passed the Nephrology Certification on their first attempt. In-Training Examination scores showed the strongest association with first-time Nephrology Certification Examination scores, accounting for approximately 50% of the total explained variance in the model. Each SD increase in In-Training Examination scores was associated with a difference of 30 U (95% confidence interval, 27 to 33) in certification performance. In-Training Examination scores also were significantly associated with passing status on the Nephrology Certification Examination on the first attempt (odds ratio, 3.46 per SD difference in the In-Training Examination; 95% confidence interval, 2.68 to 4.54). An In-Training Examination threshold of 375, approximately 1 SD below the mean, yielded a positive predictive value of 0.92 and a negative predictive value of 0.50. CONCLUSIONS: American Society of Nephrology In-Training Examination performance is significantly associated with American Board of Internal Medicine Nephrology Certification Examination score and passing status.

Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance.

The Accreditation Council for Graduate Medical Education's Next Accreditation System requires training programs to demonstrate that fellows are achieving competence in medical knowledge (MK), as part of a global assessment of clinical competency. Passing American Board of Internal Medicine (ABIM) certification examinations is recognized as a metric of MK competency. This study examines several in-training MK assessment approaches and their ability to predict performance on the ABIM Hematology or Medical Oncology Certification Examinations. Results of a Hematology In-Service Examination (ISE) and an Oncology In-Training Examination (ITE), program director (PD) ratings, demographic variables, United States Medical Licensing Examination (USMLE), and ABIM Internal Medicine (IM) Certification Examination were compared. Stepwise multiple regression and logistic regression analyses evaluated these assessment approaches as predictors of performance on the Hematology or Medical Oncology Certification Examinations. Hematology ISE scores were the strongest predictor of Hematology Certification Examination scores (ß = 0.41) (passing odds ratio [OR], 1.012; 95 % confidence interval [CI], 1.008-1.015), and the Oncology ITE scores were the strongest predictor of Medical Oncology Certification Examination scores (ß = 0.45) (passing OR, 1.013; 95 % CI, 1.011-1.016). PD rating of MK was the weakest predictor of Medical Oncology Certification Examination scores (ß = 0.07) and was not significantly predictive of Hematology Certification Examination scores. Hematology and Oncology ITEs are better predictors of certification examination performance than PD ratings of MK, reinforcing the effectiveness of ITEs for competency-based assessment of MK.

Performance on the adult rheumatology in-training examination and relationship to outcomes on the rheumatology certification examination.

OBJECTIVE: The American College of Rheumatology (ACR) Adult Rheumatology In-Training Examination (ITE) is a feedback tool designed to identify strengths and weaknesses in the content knowledge of individual fellows-in-training and the training program curricula. We determined whether scores on the ACR ITE, as well as scores on other major standardized medical examinations and competency-based ratings, could be used to predict performance on the American Board of Internal Medicine (ABIM) Rheumatology Certification Examination. METHODS: Between 2008 and 2012, 629 second-year fellows took the ACR ITE. Bivariate correlation analyses of assessment scores and multiple linear regression analyses were used to determine whether ABIM Rheumatology Certification Examination scores could be predicted on the basis of ACR ITE scores, United States Medical Licensing Examination scores, ABIM Internal Medicine Certification Examination scores, fellowship directors' ratings of overall clinical competency, and demographic variables. Logistic regression was used to evaluate whether these assessments were predictive of a passing outcome on the Rheumatology Certification Examination. RESULTS: In the initial linear model, the strongest predictors of the Rheumatology Certification Examination score were the second-year fellows' ACR ITE scores (ß = 0.438) and ABIM Internal Medicine Certification Examination scores (ß = 0.273). Using a stepwise model, the strongest predictors of higher scores on the Rheumatology Certification Examination were second-year fellows' ACR ITE scores (ß = 0.449) and ABIM Internal Medicine Certification Examination scores (ß = 0.276). Based on the findings of logistic regression analysis, ACR ITE performance was predictive of a pass/fail outcome on the Rheumatology Certification Examination (odds ratio 1.016 [95% confidence interval 1.011-1.021]). CONCLUSION: The predictive value of the ACR ITE score with regard to predicting performance on the Rheumatology Certification Examination supports use of the Adult Rheumatology ITE as a valid feedback tool during fellowship training.

The relationship between performance on the Infectious Diseases In-Training and Certification Examinations.

BACKGROUND: The Infectious Diseases Society of America In-Training Examination (IDSA ITE) is a feedback tool used to help fellows track their knowledge acquisition during fellowship training. We determined whether the scores on the IDSA ITE and from other major medical knowledge assessments predict performance on the American Board of Internal Medicine (ABIM) Infectious Disease Certification Examination. METHODS: The sample was 1021 second-year fellows who took the IDSA ITE and ABIM Infectious Disease Certification Examination from 2008 to 2012. Multiple regression analysis was used to determine if ABIM Infectious Disease Certification Examination scores were predicted by IDSA ITE scores, prior United States Medical Licensing Examination (USMLE) scores, ABIM Internal Medicine Certification Examination scores, fellowship director ratings of medical knowledge, and demographic variables. Logistic regression was used to evaluate if these same assessments predicted a passing outcome on the certification examination. RESULTS: IDSA ITE scores were the strongest predictor of ABIM Infectious Disease Certification Examination scores (ß = .319), followed by prior ABIM Internal Medicine Certification Examination scores (ß = .258), USMLE Step 1 scores (ß = .202), USMLE Step 3 scores (ß = .130), and fellowship directors' medical knowledge ratings (ß = .063). IDSA ITE scores were also a significant predictor of passing the Infectious Disease Certification Examination (odds ratio, 1.017 [95% confidence interval, 1.013-1.021]). CONCLUSIONS: The significant relationship between the IDSA ITE score and performance on the ABIM Infectious Disease Certification Examination supports the use of the ITE as a valid feedback tool in fellowship training.

Development, implementation, and results of the ASN in-training examination for fellows.

The American Society of Nephrology and the fellowship training program directors in conjunction with the National Board of Medical Examiners developed a comprehensive assessment of medical knowledge for nephrology fellows in-training. This in-training examination (ITE) consisted of 150 multiple-choice items covering 11 broad content areas in a blueprint similar to the American Board of Internal Medicine certifying examination for nephrology. Questions consisted of case vignettes to simulate real-life clinical experience. The first examination was given in April 2009 to 682 fellows and six training program directors. Examinees felt that the examination was well structured and relevant to their training experience Longitudinal performance on the examination will be helpful in allowing training programs to utilize results from content areas in identifying deficits in medical knowledge as well as assessing the results of any curriculum changes.

Developing an in-training examination for fellows: the experience of the American Society of Clinical Oncology.

The American Society of Clinical Oncology (ASCO) developed its own test -- the Medical Oncology In-Training Examination (MedOnc ITE) -- as a tool to assess trainees' knowledge of the clinical oncology subspecialty, establish consistency in educational standards across training programs, identify areas of strength and weakness in individual programs, and stimulate intraprogrammatic reading and discussion. The Accreditation Council for Graduate Medical Education Outcome Project provided additional incentive for ASCO to develop an ITE. The examination was developed in 4 years. The concept of the examination and the budget were approved by the ASCO governing board. The National Board of Medical Examiners was selected to work with ASCO. Fellowship programs were contacted to determine if they had the information technology support to hold the examination. A blueprint for the examination was developed. The test format, including the number of questions and the selection of case-based single best answers, was determined. Physician volunteers to write the questions were solicited from among program directors, various ASCO committees, and disease experts. A workshop was held to teach volunteers how to write proper case-based questions. From this pool, a smaller group of physicians was selected to develop the test and review all test questions. The final examination was developed and administered in February 2008, with scores provided to fellows and program directors in April 2008. Feedback received after the examination will be helpful for developing future MedOnc ITEs. The process ASCO went through to develop the MedOnc ITE serves as a model for other subspecialties interested in developing their own ITEs.