Associations of Internal Medicine Residency Milestone Ratings and Certification Examination Scores With Patient Outcomes.

Importance: Despite its importance to medical education and competency assessment for internal medicine trainees, evidence about the relationship between physicians' milestone residency ratings or the American Board of Internal Medicine's initial certification examination and their hospitalized patients' outcomes is sparse. Objective: To examine the association between physicians' milestone ratings and certification examination scores and hospital outcomes for their patients. Design, Setting, and Participants: Retrospective cohort analyses of 6898 hospitalists completing training in 2016 to 2018 and caring for Medicare fee-for-service beneficiaries during hospitalizations in 2017 to 2019 at US hospitals. Main Outcomes and Measures: Primary outcome measures included 7-day mortality and readmission rates. Thirty-day mortality and readmission rates, length of stay, and subspecialist consultation frequency were also assessed. Analyses accounted for hospital fixed effects and adjusted for patient characteristics, physician years of experience, and year. Exposures: Certification examination score quartile and milestone ratings, including an overall core competency rating measure equaling the mean of the end of residency milestone subcompetency ratings categorized as low, medium, or high, and a knowledge core competency measure categorized similarly. Results: Among 455 120 hospitalizations, median patient age was 79 years (IQR, 73-86 years), 56.5% of patients were female, 1.9% were Asian, 9.8% were Black, 4.6% were Hispanic, and 81.9% were White. The 7-day mortality and readmission rates were 3.5% (95% CI, 3.4%-3.6%) and 5.6% (95% CI, 5.5%-5.6%), respectively, and were 8.8% (95% CI, 8.7%-8.9%) and 16.6% (95% CI, 16.5%-16.7%) for mortality and readmission at 30 days. Mean length of stay and number of specialty consultations were 3.6 days (95% CI, 3.6-3.6 days) and 1.01 (95% CI, 1.00-1.03), respectively. A high vs low overall or knowledge milestone core competency rating was associated with none of the outcome measures assessed. For example, a high vs low overall core competency rating was associated with a nonsignificant 2.7% increase in 7-day mortality rates (95% CI, -5.2% to 10.6%; P = .51). In contrast, top vs bottom examination score quartile was associated with a significant 8.0% reduction in 7-day mortality rates (95% CI, -13.0% to -3.1%; P = .002) and a 9.3% reduction in 7-day readmission rates (95% CI, -13.0% to -5.7%; P < .001). For 30-day mortality, this association was -3.5% (95% CI, -6.7% to -0.4%; P = .03). Top vs bottom examination score quartile was associated with 2.4% more consultations (95% CI, 0.8%-3.9%; P < .003) but was not associated with length of stay or 30-day readmission rates. Conclusions and Relevance: Among newly trained hospitalists, certification examination score, but not residency milestone ratings, was associated with improved outcomes among hospitalized Medicare beneficiaries.

American Board of Internal Medicine Nephrology Procedure Requirements for Initial Certification: Time for a Change and Pursuing Consensus in the Nephrology Community.

In 1988, the American Board of Internal Medicine (ABIM) defined essential procedural skills in nephrology, and candidates for ABIM certification were required to present evidence of possessing the skills necessary for placement of temporary dialysis vascular access, hemodialysis, peritoneal dialysis, and percutaneous renal biopsy. In 1996, continuous renal replacement therapy was added to the list of nephrology requirements. These procedure requirements have not been modified since 1996 while the practice of nephrology has changed dramatically. In March 2021, the ABIM Nephrology Board embarked on a policy journey to revise the procedure requirements for nephrology certification. With the guidance of nephrology diplomates, training program directors, professional and patient organizations, and other stakeholders, the ABIM Nephrology Board revised the procedure requirements to reflect current practice and national priorities. The approved changes include the Opportunity to Train standard for placement of temporary dialysis catheters, percutaneous kidney biopsies, and home hemodialysis which better reflects the current state of training in most training programs, and the new requirements for home dialysis therapies training will align with the national priority to address the underuse of home dialysis therapies. This perspective details the ABIM process for considering changes to the certification procedure requirements and how ABIM collaborated with the larger nephrology community in considering revisions and additions to these requirements.

Analysis of Performance Trends from 2010-2019 on the American Board of Internal Medicine Nephrology Certifying Exam.

BACKGROUND: The pass rate on the American Board of Internal Medicine (ABIM) nephrology certifying exam has declined and is among the lowest of all internal medicine (IM) subspecialties. In recent years, there have also been fewer applicants for the nephrology fellowship match. METHODS: This retrospective observational study assessed how changes between 2010 and 2019 in characteristics of 4094 graduates of US ACGME-accredited nephrology fellowship programs taking the ABIM nephrology certifying exam for the first time, and how characteristics of their fellowship programs were associated with exam performance. The primary outcome measure was performance on the nephrology certifying exam. Fellowship program pass rates over the decade were also studied. RESULTS: Lower IM certifying exam score, older age, female sex, international medical graduate (IMG) status, and having trained at a smaller nephrology fellowship program were associated with poorer nephrology certifying exam performance. The mean IM certifying exam percentile score among those who subsequently took the nephrology certifying exam decreased from 56.7 (SD, 27.9) to 46.1 (SD, 28.7) from 2010 to 2019. When examining individuals with comparable IM certifying exam performance, IMGs performed less well than United States medical graduates (USMGs) on the nephrology certifying exam. In 2019, only 57% of nephrology fellowship programs had aggregate 3-year certifying exam pass rates ≥80% among their graduates. CONCLUSIONS: Changes in IM certifying exam performance, certain trainee demographics, and poorer performance among those from smaller fellowship programs explain much of the decline in nephrology certifying exam performance. IM certifying exam performance was the dominant determinant.

Gender Differences in Milestone Ratings and Medical Knowledge Examination Scores Among Internal Medicine Residents.

PURPOSE: To examine whether there are group differences in milestone ratings submitted by program directors working with clinical competency committees (CCCs) based on gender for internal medicine (IM) residents and whether women and men rated similarly on milestones perform comparably on subsequent in-training and certification examinations. METHOD: This national retrospective study examined end-of-year medical knowledge (MK) and patient care (PC) milestone ratings and IM In-Training Examination (IM-ITE) and IM Certification Examination (IM-CE) scores for 2 cohorts (2014-2017, 2015-2018) of U.S. IM residents at ACGME-accredited programs. It included 20,098/21,440 (94%) residents, with 9,424 women (47%) and 10,674 men (53%). Descriptive statistics and differential prediction techniques using hierarchical linear models were performed. RESULTS: For MK milestone ratings in PGY-1, men and women showed no statistical difference at a significance level of .01 (P = .02). In PGY-2 and PGY-3, men received statistically higher average MK ratings than women (P = .002 and P < .001, respectively). In contrast, men and women received equivalent average PC ratings in each PGY (P = .47, P = .72, and P = .80, for PGY-1, PGY-2, and PGY-3, respectively). Men slightly outperformed women with similar MK or PC ratings in PGY-1 and PGY-2 on the IM-ITE by about 1.7 and 1.5 percentage points, respectively, after adjusting for covariates. For PGY-3 ratings, women and men with similar milestone ratings performed equivalently on the IM-CE. CONCLUSIONS: Milestone ratings were largely similar for women and men. Generally, women and men with similar MK or PC milestone ratings performed similarly on future examinations. Although there were small differences favoring men on earlier examinations, these differences disappeared by the final training year. It is questionable whether these small differences are educationally or clinically meaningful. The findings suggest fair, unbiased milestone ratings generated by program directors and CCCs assessing residents.

Where Do International Medical Graduates Matriculate for Internal Medicine Training? A National Longitudinal Study.

INTRODUCTION: In 2020, roughly 25% of applicants who matched into internal medicine (IM) residencies were international medical graduates (IMGs). We examine 12-year trends in distribution of IMGs among IM training programs and explore differences in program perceptions towards IMG recruitment. METHODS: Since 2007, Association of Program Directors in Internal Medicine Annual Surveys have collected data about trainees by medical school graduate type. Sixteen additional questions regarding perceptions of IMGs were included in the 2017 spring survey. RESULTS: The 2017 survey response rate was 63.3% (236/373) and ranged from 61.9 to 70.2% for the 2007-2019 Annual Surveys. During that 12-year period, 55-70% of community programs' and 22-30% of university programs' PGY1 positions were filled by IMGs. In 2017, 45% of community programs' and 15% of university programs' interview and ranking positions were allocated to IMGs. Departmental pressure (university 45.6% [95% CI 43.7-47.5]; community 28.2% [95% CI 26.6-29.7]; p = 0.007), institutional priority (university 64.0% [95% CI 62.1-66.0]; community 41% [95% CI 36.9-44.6]; p = 0.001), and reputational concerns (university 52.8% [95% CI 50.0-55.6]; community 38.5% [95% CI 36.0-40.9]; p = 0.045) were cited as factors influencing recruitment of IMGs. CONCLUSION: Our study was limited to exploring program factors in residency recruitment and did not assess applicant preferences. There is a large, longstanding difference in the recruitment of IMGs to US community-based and university residencies, beginning during the interview and ranking process. Further research in disparities in IMG recruitment is needed, including exploring pressures, preferences, and potential biases associated with the recruitment of IMGs.

Correlations Between the USMLE Step Examinations, American College of Physicians In-Training Examination, and ABIM Internal Medicine Certification Examination.

PURPOSE: To assess the correlations between United States Medical Licensing Examination (USMLE) performance, American College of Physicians Internal Medicine In-Training Examination (IM-ITE) performance, American Board of Internal Medicine Internal Medicine Certification Exam (IM-CE) performance, and other medical knowledge and demographic variables. METHOD: The study included 9,676 postgraduate year (PGY)-1, 11,424 PGY-2, and 10,239 PGY-3 internal medicine (IM) residents from any Accreditation Council for Graduate Medical Education-accredited IM residency program who took the IM-ITE (2014 or 2015) and the IM-CE (2015-2018). USMLE scores, IM-ITE percent correct scores, and IM-CE scores were analyzed using multiple linear regression, and IM-CE pass/fail status was analyzed using multiple logistic regression, controlling for USMLE Step 1, Step 2 Clinical Knowledge, and Step 3 scores; averaged medical knowledge milestones; age at IM-ITE; gender; and medical school location (United States or Canada vs international). RESULTS: All variables were significant predictors of passing the IM-CE with IM-ITE scores having the strongest association and USMLE Step scores being the next strongest predictors. Prediction curves for the probability of passing the IM-CE based solely on IM-ITE score for each PGY show that residents must score higher on the IM-ITE with each subsequent administration to maintain the same estimated probability of passing the IM-CE. CONCLUSIONS: The findings from this study should support residents and program directors in their efforts to more precisely identify and evaluate knowledge gaps for both personal learning and program improvement. While no individual USMLE Step score was as strongly predictive of IM-CE score as IM-ITE score, the combined relative contribution of all 3 USMLE Step scores was of a magnitude similar to that of IM-ITE score.

National Internal Medicine Milestone Ratings: Validity Evidence From Longitudinal Three-Year Follow-up.

PURPOSE: To evaluate validity evidence for internal medicine milestone ratings across programs for three resident cohorts by quantifying "not assessable" ratings; reporting mean longitudinal milestone ratings for individual residents; and correlating medical knowledge ratings across training years with certification examination scores to determine predictive validity of milestone ratings for certification outcomes. METHOD: This retrospective study examined milestone ratings for postgraduate year (PGY) 1-3 residents in U.S. internal medicine residency programs. Data sources included milestone ratings, program characteristics, and certification examination scores. RESULTS: Among 35,217 participants, there was a decreased percentage with "not assessable" ratings across years: 1,566 (22.5%) PGY1s in 2013-2014 versus 1,219 (16.6%) in 2015-2016 (P = .01), and 342 (5.1%) PGY3s in 2013-2014 versus 177 (2.6%) in 2015-2016 (P = .04). For individual residents with three years of ratings, mean milestone ratings increased from around 3 (behaviors of an early learner or advancing resident) in PGY1 (ranging from a mean of 2.73 to 3.19 across subcompetencies) to around 4 (ready for unsupervised practice) in PGY3 (mean of 4.00 to 4.22 across subcompetencies, P < .001 for all subcompetencies). For each increase of 0.5 units in two medical knowledge (MK1, MK2) subcompetency ratings, the difference in examination scores for PGY3s was 19.5 points for MK1 (P < .001) and 19.0 for MK2 (P < .001). CONCLUSIONS: These findings provide evidence of validity of the milestones by showing how training programs have applied them over time and how milestones predict other training outcomes.

The American Board of Internal Medicine Maintenance of Certification Examination and State Medical Board Disciplinary Actions: a Population Cohort Study.

BACKGROUND: Some have questioned whether successful performance in the American Board of Internal Medicine (ABIM) Maintenance of Certification (MOC) program is meaningful. The association of the ABIM Internal Medicine (IM) MOC examination with state medical board disciplinary actions is unknown. OBJECTIVE: To assess risk of disciplinary actions among general internists who did and did not pass the MOC examination within 10 years of initial certification. DESIGN: Historical population cohort study. PARTICIPANTS: The population of internists certified in internal medicine, but not a subspecialty, from 1990 through 2003 (n = 47,971). INTERVENTION: ABIM IM MOC examination. SETTING: General internal medicine in the USA. MAIN MEASURES: The primary outcome measure was time to disciplinary action assessed in association with whether the physician passed the ABIM IM MOC examination within 10 years of initial certification, adjusted for training, certification, demographic, and regulatory variables including state medical board Continuing Medical Education (CME) requirements. KEY RESULTS: The risk for discipline among physicians who did not pass the IM MOC examination within the 10 year requirement window was more than double than that of those who did pass the examination (adjusted HR 2.09; 95% CI, 1.83 to 2.39). Disciplinary actions did not vary by state CME requirements (adjusted HR 1.02; 95% CI, 0.94 to 1.16), but declined with increasing MOC examination scores (Kendall's tau-b coefficient = - 0.98 for trend, p < 0.001). Among disciplined physicians, actions were less severe among those passing the IM MOC examination within the 10-year requirement window than among those who did not pass the examination. CONCLUSIONS: Passing a periodic assessment of medical knowledge is associated with decreased state medical board disciplinary actions, an important quality outcome of relevance to patients and the profession.

Relationships Between the ACGME Resident and Faculty Surveys and Program Pass Rates on the ABIM Internal Medicine Certification Examination.

PURPOSE: The Accreditation Council for Graduate Medical Education (ACGME) has surveyed residents since 2003, and faculty since 2012. Surveys are designed to assess program functioning and specify areas for improvement. The purpose of this study was to assess the association of the ACGME's resident and faculty surveys with residency-program-specific performance on the American Board of Internal Medicine (ABIM) certification exam. METHOD: Data were available from residents and faculty in 375 U.S. ACGME-accredited internal medicine programs from the 2012-2013, 2013-2014, and 2014-2015 academic years. Analysis of variance and correlations were used to examine the relationship between noncompliance with ACGME program requirements as assessed by the resident and faculty surveys, and ABIM program pass rates. RESULTS: Noncompliance reported on the resident and faculty surveys was highest for programs not meeting the ACGME program requirement of an 80% pass rate on the ABIM certification examination. This relationship was significant for overall noncompliance, both within the resident (P < .001) and faculty (P < .05) surveys, for many areas within the two surveys (correlations ranged between -.07 and -.25, and P values ranged between .20 and < .001), and for the highest levels of noncompliance across areas of the resident (P < .001) and faculty (P < .04) surveys. CONCLUSIONS: ACGME resident and faculty surveys were significantly associated with ABIM program pass rates, supporting the importance of these surveys within the ACGME's Next Accreditation System.

Networking Matters: A Social Network Analysis of the Association of Program Directors of Internal Medicine.

Theory: Networking has positive effects on career development; however, personal characteristics of group members such as gender or diversity may foster or hinder member connectedness. Social network analysis explores interrelationships between people in groups by measuring the strength of connection between all possible pairs in a given network. Social network analysis has rarely been used to examine network connections among members in an academic medical society. This study seeks to ascertain the strength of connection between program directors in the Association of Program Directors in Internal Medicine (APDIM) and its Education Innovations Project subgroup and to examine possible associations between connectedness and characteristics of program directors and programs. Hypotheses: We hypothesize that connectedness will be measurable within a large academic medical society and will vary significantly for program directors with certain measurable characteristics (e.g., age, gender, rank, location, burnout levels, desire to resign). Method: APDIM program directors described levels of connectedness to one another on the 2012 APDIM survey. Using social network analysis, we ascertained program director connectedness by measuring out-degree centrality, in-degree centrality, and eigenvector centrality, all common measures of connectedness. Results: Higher centrality was associated with completion of the APDIM survey, being in a university-based program, Educational Innovations Project participation, and higher academic rank. Centrality did not vary by gender; international medical graduate status; previous chief resident status; program region; or levels of reported program director burnout, callousness, or desire to resign. Conclusions: In this social network analysis of program directors within a large academic medical society, we found that connectedness was related to higher academic rank and certain program characteristics but not to other program director characteristics like gender or international medical graduate status. Further research is needed to optimize our understanding of connection in organizations such as these and to determine which strategies promote valuable connections.

When Old Habits Train a New Generation: Findings From a National Survey of Internal Medicine Program Directors on Procedural Training.

Resident physicians routinely perform bedside procedures that pose substantial risk to patients. However, no standard programmatic approach to supervision and procedural competency assessment among residents currently exists. The authors performed a national survey of internal medicine (IM) program directors to examine procedural assessment and supervision practices of IM residency programs. Procedures chosen were those commonly performed by medicine residents at the bedside. Of the 368 IM programs, 226 (61%) completed the survey. Programs reported the predominant method of training as 171 (74%) apprenticeship and 106 (46%) as module based. The majority of programs used direct observation to attest to competence, with 55% to 62% relying on credentialed residents. Most programs also relied on a minimum number of procedures to determine competence (64%-88%), 72% of which reported 5 procedures (a lapsed historical standard). This national survey demonstrates that procedural assessment practices for IM residents are insufficiently robust and may put patients at undue risk.

Co-creation by the ABIM Geriatric Medicine Board and the AGS - Helping Move Geriatrics Forward.

The American board of internal medicine (ABIM) establishes standards for physicians. The American geriatrics society (AGS) is a not-for-profit membership organization of nearly 6,000 health professionals devoted to improving the health, independence, and quality of life of all older people. Beginning in 2013, ABIM redesigned its governance structure, including the role of the specialty boards. Specialty boards are charged with responsibilities for oversight in four main areas: (1) the assessments used in initial certification and maintenance of certification (MOC); (2) medical knowledge self-assessment and practice assessment in the specialty; (3) building relationships with relevant professional societies and other organizational stakeholders; and (4) issues related to training requirements for initial certification eligibility within the specialty. The aim of this paper is to inform the geriatrics community regarding the function of geriatric medicine board (GMB) of the ABIM, and to invite the geriatrics community to fully engage with and leverage the GMB as a partner to: (1) develop better certification examinations and processes, identifying better knowledge and practice assessments, and in establishing appropriate training and MOC requirements for geriatric medicine; (2) leverage ABIM assets to conduct applied research to guide the field in the areas of training and certification and workforce development in geriatric medicine; (3) make MOC relevant for practicing geriatricians. Active engagement of the geriatrics community with ABIM and the GMB will ensure that certification in geriatric medicine provides the greatest possible value and meaning to physicians, patients, and the public.

Performance on the Cardiovascular In-Training Examination in Relation to the ABIM Cardiovascular Disease Certification Examination.

The American College of Cardiology In-Training Exam (ACC-ITE) is incorporated into most U.S. training programs, but its relationship to performance on the American Board of Internal Medicine Cardiovascular Disease (ABIM CVD) Certification Examination is unknown. ACC-ITE scores from third-year fellows from 2011 to 2014 (n = 1,918) were examined. Covariates for regression analyses included sex, age, medical school country, U.S. Medical Licensing Examination Step, and ABIM Internal Medicine Certification Examination scores. A secondary analysis examined fellows (n = 511) who took the ACC-ITE in the first and third years. ACC-ITE scores were the strongest predictor of ABIM CVD scores (p < 0.0001), and the most significant predictor of passing (p < 0.0001). The change in ACC-ITE scores from first to third year was a strong predictor of the ABIM CVD score (p < 0.001). The ACC-ITE is strongly associated with performance on the ABIM CVD Certification Examination.