The Relationship Between ACGME Duty Hour Requirements and Performance on the American Board of Emergency Medicine Qualifying Examination.

BACKGROUND: In 2003, the Accreditation Council for Graduate Medical Education (ACGME) instituted requirements that limited the number of hours residents could spend on duty, and in 2011, it revised these requirements. OBJECTIVE: This study explored whether the implementation of the 2003 and 2011 duty hour limits was associated with a change in emergency medicine residents' performance on the American Board of Emergency Medicine (ABEM) Qualifying Examination (QE). METHODS: Beginning with the 1999 QE and ending with the 2014 QE, candidates for whom all training occurred without duty hour requirements (Group A), candidates under the first set of duty hour requirements (Group C), and candidates under the second set of duty hour requirements (Group E) were compared. Comparisons included mean scores and pass rates. RESULTS: In Group A, 5690 candidates completed the examination, with a mean score of 82.8 and a 90.2% pass rate. In Group C, 8333 candidates had a mean score of 82.4 and a 90.5% pass rate. In Group E, there were 1269 candidates, with a mean score of 82.5 and an 89.4% pass rate. There was a small but statistically significant decrease in the mean scores (0.04, P < .001) after implementation of the first duty hour requirements, but this difference did not occur after implementation of the 2011 standards. There was no difference among pass rates for any of the study groups (χ2 = 1.68, P = .43). CONCLUSIONS: We did not identify an association between the 2003 and 2011 ACGME duty hour requirements and performance of test takers on the ABEM QE.

Initial Validity Analysis of the Emergency Medicine Milestones.

OBJECTIVES: The Accreditation Council for Graduate Medical Education (ACGME) Milestones describe behavioral markers for the progressive acquisition of competencies during residency. As a key component of the Next Accreditation System, all residents are evaluated for the acquisition of specialty-specific Milestones. The objective was to determine the validity and reliability of the emergency medicine (EM) Milestones. METHODS: The ACGME and the American Board of Emergency Medicine performed this single-event observational study. The data included the initial EM Milestones performance ratings of all categorical EM residents submitted to the ACGME from October 31, 2013, to January 6, 2014. Mean performance ratings were determined for all 23 subcompetencies for every year of residency training. The internal consistency (reliability) of the Milestones was determined using a standardized Cronbach's alpha coefficient. Exploratory factor analysis was conducted to determine how the subcompetencies were interrelated. RESULTS: EM Milestone performance ratings were obtained on 100% of EM residents (n = 5,805) from 162 residency programs. The mean performance ratings of the aggregate and individual subcompetency scores showed discrimination between residency years, and the factor structure further supported the validity of the EM Milestones. The reliability was α = 0.96 within each year of training. CONCLUSIONS: The EM Milestones demonstrated validity and reliability as an assessment instrument for competency acquisition. EM residents can be assured that this evaluation process has demonstrated validity and reliability; faculty can be confident that the Milestones are psychometrically sound; and stakeholders can know that the Milestones are a nationally standardized, objective measure of specialty-specific competency acquisition.

American Board of Emergency Medicine report on residency training information (2014-2015).

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine residency programs and the residents in those programs. We present the 2015 annual report on the status of US emergency medicine training programs.

Emergency department quality improvement activity: an inventory from the American Board of Emergency Medicine Maintenance of Certification program.

OBJECTIVES: The American Board of Emergency Medicine (ABEM) Maintenance of Certification (MOC) program requires every ABEM-certified physician to attest to participating in a quality improvement (QI) activity every 5 years. Understanding the type and frequency of these QI activities could inform the emergency medicine community about the variety of QI activities in which emergency physicians (EPs) are involved. These QI activities could provide ideas for the development of additional quality measures. METHODS: This was a retrospective descriptive study of self-reported QI activity attestations from the ABEM MOC program during 2013. Attestations were provided by ABEM-certified EPs using the ABEM MOC website. The type, number, and cumulative frequency of activities are reported. RESULTS: ABEM received 9,380 attestations for QI activities in 91 different categories. The three most commonly reported activities were acute myocardial infarction-percutaneous coronary intervention within 90 minutes of arrival (includes door-to-balloon time), door-to-doctor times, and throughput time measures. These three activities comprised 36.4% of attestations. More than half (54.4%) of the attestations were captured by the five most frequently attested activities, 67.1% by the top seven categories, and 89.9% by the top 21 categories. Of these 21 categories, 10 involved clinical protocols, nine were time-centered measures, and two were patient-centered activities. CONCLUSIONS: This report demonstrates that diverse QI activities occur in emergency departments (EDs) across the United States. The majority of reported projects are nested in a few categories, following recognized areas of emphasis in emergency care, particularly in areas using time-sensitive metrics.

Delaying the American Board of Emergency Medicine qualifying examination is associated with poorer performance.

OBJECTIVES: The initial step in certification by the American Board of Emergency Medicine (ABEM) requires passing a multiple-choice-question qualifying examination. The qualifying examination is typically taken in the first year after residency training. This study was undertaken to determine if a delay in taking the qualifying examination is associated with poorer performance. The authors also examined the relationship between in-training examination scores and qualifying examination scores. METHODS: This was a pooled time-series cross-section study. Primary measurements were initial qualifying examination scores, the timing of the qualifying examination, and in-training examination scores. The three groups, based on qualifying examination timing, were immediate, 1-year delay, and ≥2-year delay. In-training examination scores were analyzed to determine the relationship between intrinsic ability, examination timing, and qualifying examination scores. For analysis, a generic pooled ordinary least-squares dummy variable model with robust standard errors was used. A pre hoc level of significance was determined to be α < 0.01. RESULTS: There were 16,353 qualifying examination test administrations between 2000 and 2012. In-training examination scores were positively correlated with qualifying examination scores (p < 0.001). The group pass rates were 98.9% immediate, 95.6% 1-year delay, and 86.6% ≥2-year delay. After controlling for in-training examination scores, delay taking the qualifying examination of 1 year was associated with a decrease in score of -0.6 (p = 0.003). A delay in taking the qualifying examination ≥2 years was associated with a decrease in score of -2.5 points (p < 0.001). CONCLUSIONS: After accounting for innate ability using in-training examination scores, delay taking the qualifying examination was associated with poorer performance. This effect was more pronounced if the delay was ≥2 years.

Emergency physicians maintain performance on the American Board of Emergency Medicine Continuous Certification (ConCert) examination.

OBJECTIVES: The American Board of Emergency Medicine (ABEM) Maintenance of Certification (MOC) program is a four-step process that includes the Continuous Certification (ConCert) examination. The ConCert examination is a validated, summative examination that assesses medical knowledge and clinical reasoning. ABEM began administering the ConCert examination in 1989. The ConCert examination must be passed at least every 10 years to maintain certification. This study was undertaken to determine longitudinal physician performance on the ConCert examination. METHODS: In this longitudinal review, ConCert examination performance was compared among residency-trained emergency physicians (EPs) over multiple examination cycles. Longitudinal analysis was performed using a growth curve model for unbalanced data to determine the growth trajectories of EP performance over time to see if medical knowledge changed. Using initial certification qualifying examination scores, the longitudinal analysis corrected for intrinsic variances in physician ability. RESULTS: There were 15,085 first-time testing episodes from 1989 to 2012 involving three examination cycles. The mean adjusted examination scores for all physicians taking the ConCert examination for a first cycle was 85.9 (95% confidence interval [CI] = 85.8 to 85.9), the second cycle mean score was 86.2 (95% CI = 86.0 to 86.3), and the third cycle was 85.4 (95% CI = 85.0 to 85.8). Using the first examination cycle as a reference score, the growth curve model analysis resulted in a coefficient of +0.3 for the second cycle (p < 0.001) and -0.5 for the third cycle (p = 0.02). Initial qualifying (written) examination scores were significant predictors for ConCert examination scores. CONCLUSIONS: Over time, EP performance on the ConCert examination was maintained. These results suggest that EPs maintain medical knowledge over the course of their careers as measured by a validated, summative medical knowledge assessment.

American Board of Emergency Medicine report on residency training information (2013-2014), American Board of Emergency Medicine.

The American Board of Emergency Medicine gathers extensive background information on emergency medicine residency programs and the residents in them. We present the 2014 annual report on the status of US emergency medicine training programs.

Lifelong learning and self-assessment is relevant to emergency physicians.

BACKGROUND: The Lifelong Learning and Self-assessment (LLSA) component of the American Board of Emergency Medicine (ABEM) Maintenance of Certification (MOC) program is a self-assessment exercise for physicians. Beginning in 2011, an optional continuing medical education (CME) activity was added. OBJECTIVES: As a part of a CME activity option for the LLSA, a survey was used to determine the relevancy of the LLSA readings and the degree to which medical knowledge garnered by the LLSA activity would modify clinical care. METHODS: Survey results from the 2011 LLSA CME activity were reviewed. This survey was composed of seven items, including questions about the relevancy of the readings and the impact on the physician's clinical practice. The questions used a 5-point Likert scale and data underwent descriptive analyses. RESULTS: There were 2841 physicians who took the LLSA test during the study period, of whom 1354 (47.7%) opted to participate in the 2011 LLSA CME activity. All participants completed surveys. The LLSA readings were reported to be relevant to the overall clinical practice of Emergency Medicine (69.6% strongly relevant, 28.1% some relevance, and 2.3% little or no relevance), and provided information that would likely help them change their clinical practices (high likelihood 38.8%, some likelihood 53.0%, little or no change 8.2%). CONCLUSIONS: The LLSA component of the ABEM MOC program is relevant to the clinical practice of Emergency Medicine. Through this program, physicians gain new knowledge about the practice of Emergency Medicine, some of which is reported to change physicians' clinical practices.

The emergency medicine milestones: a validation study.

OBJECTIVES: The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties sought to define milestones for skill and knowledge acquisition during residency training. Milestones are significant objective observable events. The milestones are listed within a structure that is derived from the ACGME general competencies. Major groups of milestones are called "subcompetencies." The original 24 subcompetencies containing 255 milestones for emergency medicine (EM) were developed through a multiorganizational group representing most EM stakeholder groups. To assure that the milestones reflected EM resident progress throughout training, the EM Milestones Working Group (EM MWG) sought to validate the individual milestones. METHODS: A computer-based survey was sent to all EM residency programs. The survey period began on April 30, 2012, and concluded on May 15, 2012. Respondents were asked to assign each milestone to a specific level of skill or knowledge acquisition. These levels ranged from a beginning resident to an accomplished clinician. There were two different forms that divided the milestones into two groups of 12 subcompetencies each. Surveys were randomly assigned to programs. RESULTS: There were five respondents (the program director and four key faculty) requested from each of the 159 residences. There were responses from 96 programs (60.4%). Of the 795 survey recipients, 28 were excluded due to prior exposure to the EM milestones. Of the remaining 767 potential respondents, 281 completed the survey (36.6%) within a 16-day period. Based on the survey results, the working group adjusted the milestones in the following ways: one entire subcompetency (teaching) was eliminated, six new milestones were created, 34 milestones were eliminated, 26 milestones were reassigned to a lower level score, and 20 were reassigned to a higher level. Nineteen milestones were edited to provide greater clarity. The final result was 227 discrete milestones among 23 subcompetencies. CONCLUSIONS: The EM milestones were validated through a milestone assignment process using a computer-based survey completed by program directors and key faculty. Milestones were revised in accordance with the results to better align assignment within each performance level.

American Board of Emergency Medicine report on residency training information (2012-2013), American Board of Emergency Medicine.

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine residency programs and the residents in those programs. We present the 2013 annual report on the status of US emergency medicine training programs.

Components of the model of the clinical practice of emergency medicine: a survey of American Board of Emergency Medicine diplomates.

BACKGROUND: The Model of the Clinical Practice of Emergency Medicine is the basis for the content specifications of all American Board of Emergency Medicine (ABEM) examinations. This study describes the frequency with which ABEM diplomates diagnose and manage the conditions and components listed in the Model of the Clinical Practice of Emergency Medicine. OBJECTIVES: The objectives of this study were to determine the frequency with which ABEM diplomates diagnose and manage the conditions and components described in the Model of the Clinical Practice of Emergency Medicine. METHODS: The listing of conditions and components of the Model of the Clinical Practice of Emergency Medicine were sent to 16,230 randomly selected ABEM diplomates. One of five surveys was sent to each diplomate. Each condition and component was assessed by participants for the frequency that emergency physicians diagnose (D) and manage (M) that condition, as seen in their practice of Emergency Medicine. RESULTS: Of the 16,230 surveys sent, 5006 were returned (30.8% response rate). The genders of the respondents were 75% male and 24% female. The ages of the respondents were primarily in the age 40-49 years, and 30-39 years age groups. All categories of the listing of conditions and components of the Model of the Clinical Practice of Emergency Medicine were encountered frequently in the practice of Emergency Medicine, as indicated by study participants. CONCLUSIONS: A survey of practicing ABEM diplomates was useful in defining the frequency with which specific conditions and components are diagnosed and managed in the practice of Emergency Medicine.

Report on residency training information (2011-2012), American Board of Emergency Medicine.

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine residency training programs and the residents in those programs. We present the 2012 annual report on the status of US emergency medicine training programs.

2007 survey of medical toxicology practice.

To date, there appear to be no studies investigating the practice settings of all Medical Toxicology (MT) diplomates. The MT Assessment of Practice Performance Taskforce queried all MT diplomates about their current practice settings relative to the number of patients seen, the most common diagnoses, and the percent of time spent in their roles as medical toxicologists (MTs) and in their primary specialty. One hundred twenty-seven surveys were completed (44% response rate). Seventy-nine percent of respondents were affiliated with poison centers. Eighty-eight percent of participants were clinically active and reported seeing or consulting on behalf of at least ten patients over a 2-year period. Acetaminophen toxicity was the most common diagnosis encountered by respondents. Other common diagnoses included antidepressant toxicity, antipsychotic toxicity, mental status alteration, metal/environmental toxicity, envenomation, and pesticide toxicity. While respondents were likely to spend more time in direct patient care in their primary specialty, compared to consulting on behalf of patients, they were more likely to consult on behalf of patients in their role as MTs. Respondents spent more time in research, education, and population health in their role as an MT than in their primary specialty. Administrative activities were more commonly reported in association with the respondents' primary specialty than in their role as MTs. Most MTs encounter certain diagnoses with significant frequency and see a substantial number of patients within these categories. The majority spends more time on direct patient care in their primary specialty but is actively engaged in MT education, research, population health, and administration. A longitudinal assessment of MT practice patterns could inform MT curricular development and practice performance evaluation.

2007 survey of medical toxicology practice improvement activities.

To date, there appear to be no studies that assess Medical Toxicologists' (MTs) practice improvement (PI) activities in their Medical Toxicology practice settings. The MT Assessment of Practice Performance (APP) Taskforce queried all MT diplomates about (1) activities currently available in their practice settings that potentially would meet the requirements of APP, (2) potential APP activities that best fit with current MT practice, and (3) the relationship between MT practice patterns and APP requirements. One hundred twenty-seven surveys were completed. Participation in MT practice improvement activities is not universal, with approximately a third of the survey participants reporting that they are not involved in any practice improvement activity. Few respondents reported that they collected performance improvement-related data. Most who did so participated in CME, case, or chart reviews. Peer reviews, self-improvement plans based on chart reviews, and population research were considered the most valid measures of MT practice improvement. Communication skills were considered important topics for patient surveys. Suggested outcomes for peer assessment included accuracy of information provided, understanding medical staff concerns, timeliness of feedback, and helpfulness. Most respondents rated all of the APP options as being somewhat very intrusive. Access to those with sufficient knowledge of the diplomate's practice improvement program to verify APP could pose a challenge to a successful completion of APP requirements. Optimal settings for the APP program administration are hospitals and poison centers. While barriers to MT APP activities exist, studying current MT diplomates' opinions and practices could inform the future development and administration of such programs.

Career satisfaction in emergency medicine: the ABEM Longitudinal Study of Emergency Physicians.

STUDY OBJECTIVE: The primary objective of this study is to measure career satisfaction among emergency physicians participating in the 1994, 1999, and 2004 American Board of Emergency Medicine Longitudinal Study of Emergency Physicians. The secondary objectives are to determine factors associated with high and low career satisfaction and burnout. METHODS: This was a secondary analysis of a cohort database created with stratified, random sampling of 1,008 emergency physicians collected in 1994, 1999, and 2004. The survey consisted of 25 questions on professional interests, attitudes, and goals; 17 questions on training, certification, and licensing; 36 questions on professional experience; 4 questions on well-being and leisure activities; and 8 questions about demographics. Data were analyzed with a descriptive statistics and panel series regression modeling (Stata/SE 9.2 for Windows). Questions relating to satisfaction were scored with a 5-point Likert-like scale, with 1=not satisfied and 5=very satisfied. Questions relating to stress and burnout were scored with a 5-point Likert-like scale, with 1=not a problem and 5=serious problem. During analysis, answers to the questions "Overall, how satisfied are you with your career in emergency medicine?" "How much of a problem is stress in your day-to-day work for pay?" "How much of a problem is burnout in your day-to-day work for pay?" were further dichotomized to high levels (4, 5) and low levels (1, 2). RESULTS: Response rates from the original cohort were 94% (945) in 1994, 82% (823) in 1999, and 76% (771) in 2004. In 2004, 65.2% of emergency physicians reported high career satisfaction (4, 5), whereas 12.7% of emergency physicians reported low career satisfaction (1, 2). The majority of respondents (77.4% in 1994, 80.6% in 1999, 77.4% in 2004) stated that emergency medicine has met or exceeded their career expectations. Despite overall high levels of career satisfaction, one-third of respondents (33.4% in 1994, 31.3% in 1999, 31% in 2004) reported that burnout was a significant problem. CONCLUSION: Overall, more than half of emergency physicians reported high levels of career satisfaction. Although career satisfaction has remained high among emergency physicians, concern about burnout is substantial.

Report of the Task Force on Residency Training Information (2005-2006), American Board of Emergency Medicine.

The American Board of Emergency Medicine gathers extensive background information on emergency medicine residency training programs and the residents training in those programs. We present the ninth annual report on the status of US emergency medicine residency programs.

Report of the Task Force on Residency Training Information (2004-2005), American Board of Emergency Medicine.

The American Board of Emergency Medicine gathers extensive background information on emergency medicine residency training programs and the residents training in those programs. We present the eighth annual report on the status of US emergency medicine residency programs.

Report of the Task Force on Residency Training Information (2003-2004), American Board of Emergency Medicine.

The American Board of Emergency Medicine gathers extensive background information on emergency medicine residents and the programs in which they train. We present the seventh annual report on the status of US emergency medicine residency programs.

Interexaminer agreement on the American Board of Emergency Medicine oral certification examination.

STUDY OBJECTIVE: One of the principal criticisms of performance assessments, particularly those using an oral format to evaluate physician performance, is the lack of interrater reliability. The objective of this study was to assess interexaminer agreement in the scoring of candidates taking the American Board of Emergency Medicine (ABEM) oral certification examination. METHODS: Examiner-observer pairs independently scored individual candidates on each of the 6 simulated cases selected for the October 1999 examination. Simple proportionate agreement for "acceptable" and "unacceptable" scores on critical actions and performance ratings were used to measure interexaminer (examiner-observer) agreement. Further analyses were performed to assess interexaminer agreement at the level of the raw scores of all performance ratings. The candidate's actual score was determined entirely by the examiner administering the case. The study examiner was present only to observe and independently score the candidate. Examiners were blinded to each other's scoring. All examiner-observer, examiner-candidate, and observer-candidate pairings were uniquely randomized and occurred only once during the examination. RESULTS: Of 564 candidates, 186 were randomly selected for scoring by examiner-observer pairs drawn from a pool of 107 trained oral examiners. Interexaminer agreement for 47 critical actions was 97% (95% confidence interval [CI] 96% to 98%), and interexaminer agreement for 68 performance ratings was 95% (95% CI 94% to 96%). Further analyses of raw scores of performance ratings indicated that 94% (95% CI 93% to 95%) of 2,648 pairs of ratings differed by 1 point or less. CONCLUSION: High interexaminer agreement on the scoring of the ABEM oral certification examination supports the reliability of this final step in the certification process. On the basis of these findings, ABEM has incorporated this methodology into the administration of each oral examination as an ongoing quality control measure.