Electronic Health Record Use in Internal Medicine Clerkships and Sub-internships for Medical Students Graduating from 2012 to 2016.

BACKGROUND: As electronic health records (EHRs) became broadly available in medical practice, effective use of EHRs by medical students emerged as an essential aspect of medical education. While new federal clinical documentation guidelines have the potential to encourage greater medical student EHR use and enhance student learning experiences with respect to EHRs, little is known nationally about how students have engaged with EHRs in the past. OBJECTIVE: This study examines medical student accounts of EHR use during their internal medicine (IM) clerkships and sub-internships during a 5-year time period prior to the new clinical documentation guidelines. DESIGN: An online survey about EHR use was administered to medical students immediately after they completed USMLE Step 2 CK. PARTICIPANTS: The sample included 16,602 medical students planning to graduate from US medical schools from 2012 to 2016. MAIN MEASURES: Descriptive statistics were computed to determine the average percentage of students engaged in various health record activities during their IM educational experiences by graduation year. KEY RESULTS: The vast majority (99%) of medical students used EHRs during IM clerkships or sub-internships. Most students reported that they entered information into EHRs during the inpatient component of the IM clerkship (84%), outpatient component of the IM clerkship (70%), and the IM sub-internship (92%). Yet, 43% of the students who graduated in 2016 never entered admission orders and 35% of them never entered post-admission orders. CONCLUSIONS: Medical school graduates ought to be able to effectively document clinical encounters and enter orders into EHR systems. Although most students used and entered information into EHRs during their IM clinical training, many students appear to have received inadequate opportunities to enter notes or orders, in particular. Implications for graduate medical education preparedness are considered. Future research should address similar questions using comparable national data collected after the recent guideline changes.

Medical Student Use of Electronic and Paper Health Records During Inpatient Clinical Clerkships: Results of a National Longitudinal Study.

PURPOSE: An important goal of medical education is to teach students to use an electronic health record (EHR) safely and effectively. The purpose of this study is to examine medical student accounts of EHR use during their core inpatient clinical clerkships using a national sample. Paper health records (PHRs) are similarly examined. METHOD: An online survey about health record use within the inpatient component of six core clerkships was administered to medical students after they completed Step 2 Clinical Knowledge of the United States Medical Licensing Examination. The sample included 17,202 U.S. medical students graduating between 2012 and 2016. Mean percentages of clerkships in which students engaged in various health record activities were computed, and analysis of variance was used to examine differences. RESULTS: The mean percentages of clerkships in which a student accessed or entered information into an EHR increased from 78% to 93% and 59% to 72%, respectively. For students who used an EHR, the mean percentage of clerkships in which they entered information remained constant at 76%. Students entered notes during the majority of their clerkships, with increases over time. However, students entered orders in less than a quarter of their clerkships, with decreases over time. The percentage of clerkships in which students used PHRs was lower and declining. CONCLUSIONS: Although students used an EHR in the majority of their inpatient core clerkships, they received limited educational experiences related to order and note writing, which could translate into a lack of preparedness for future training and practice.

International variation in performance by clinical discipline and task on the United States medical licensing examination step 2 clinical knowledge component.

PURPOSE: To investigate country-to-country variation in performance across clinical science disciplines and tasks for examinees taking the Step 2 Clinical Knowledge (CK) component of the United States Medical Licensing Examination. METHOD: In 2012 the authors analyzed demographic characteristics, total scores, and percent-correct clinical science discipline and task scores for more than 88,500 examinees taking Step 2 CK for the first time during the 2008-2010 academic years. For each examinee and score, differences between the score and the mean performance of examinees at U.S. MD-granting medical schools were calculated, and mean differences by country of medical school were tabulated for analysis of country-to-country variation in performance by clinical discipline and task. RESULTS: Controlling for overall performance relative to U.S. examinees, results showed that international medical graduates (IMGs) performed best in Surgery and worst in Psychiatry for clinical discipline scores; for clinical tasks, IMGs performed best in Understanding Mechanisms of Disease and worst in Promoting Preventive Medicine and Health Maintenance. The pattern of results was strongest for IMGs attending schools in the Middle East and Australasia, present to a lesser degree for IMGs attending schools in Europe, and absent for IMGs attending Caribbean medical schools. CONCLUSIONS: Country-to-country differences in relative performance were present for both clinical discipline and task scores. Possible explanations include differences in learning outcomes, curriculum emphasis and clinical experience, standards of care, and culture, as well as the effects of English as a second language and relative emphasis on preparing students to take the Step 2 CK exam.

Utility of AAOS OITE scores in predicting ABOS Part I outcomes: AAOS exhibit selection.

BACKGROUND: Residency programs commonly use performance on the Orthopaedic In-Training Examination (OITE) developed by the American Academy of Orthopaedic Surgeons (AAOS) to identify residents who are lagging behind their peers and at risk for failing Part I of the American Board of Orthopaedic Surgery (ABOS) Certifying Examination. This study was designed to investigate the utility of the OITE score as a predictor of ABOS Part I performance. METHOD: Results for 3132 examinees who took Part I of the ABOS examination for the first time from 2002 to 2006 were matched with records from the 1997 to 2006 OITE tests; at least one OITE score was located for 2852 (91%) of the ABOS Part I examinees. After OITE performance was rescaled to place scores from different test years on comparable scales, descriptive statistics and correlations between ABOS and OITE scores were computed, and regression analyses were conducted to predict ABOS results from OITE performance. RESULTS: Substantial increases in the mean OITE score were observed as residents progressed through training. Stronger correlations were observed between OITE and ABOS performance during later years in training, reaching a maximum of 0.53 in years 3 and 4. Logistic regression results indicated that residents with an OITE score below the 10th percentile were much more likely to fail Part I compared with those with an OITE score above the 50th percentile. CONCLUSIONS: OITE performance was a good predictor of the ABOS score and pass-fail outcome; the OITE can be used effectively for early identification of residents at risk for failing the ABOS Part I examination.

Progress testing in clinical science education: results of a pilot project between the National Board of Medical Examiners and a US Medical School.

BACKGROUND: Though progress tests have been used for several decades in various medical education settings, a few studies have offered analytic frameworks that could be used by practitioners to model growth of knowledge as a function of curricular and other variables of interest. AIM: To explore the use of one form of progress testing in clinical education by modeling growth of knowledge in various disciplines as well as by assessing the impact of recent training (core rotation order) on performance using hierarchical linear modeling (HLM) and analysis of variance (ANOVA) frameworks. METHODS: This study included performances across four test administrations occurring between July 2006 and July 2007 for 130 students from a US medical school who graduated in 2008. Measures-nested-in-examinees HLM growth curve analyses were run to estimate clinical science knowledge growth over time and repeated measures ANOVAs were run to assess the effect of recent training on performance. RESULTS: Core rotation order was related to growth rates for total and pediatrics scores only. Additionally, scores were higher in a given discipline if training had occurred immediately prior to the test administration. CONCLUSIONS: This study provides a useful progress testing framework for assessing medical students' growth of knowledge across their clinical science education and the related impact of training.

Gathering evidence of external validity for the Foundations of Medicine examination: a collaboration between the National Board of Medical Examiners and the University of Minho.

BACKGROUND: To gather evidence of external validity for the Foundations of Medicine (FOM) examination by assessing the relationship between its subscores and local grades for a sample of Portuguese medical students. METHOD: Correlations were computed between six FOM subscores and nine Minho University grades for a sample of 90 medical students. A canonical correlation analysis was run between FOM and Minho measures. RESULTS: Moderate correlations were noted between FOM subscores and Minho grades, ranging from -0.02 to 0.53. One canonical correlation was statistically significant. The FOM variate accounted for 44% of variance in FOM subscores and 22% of variance in Minho end-of-year grades. The Minho canonical variate accounted for 34% of variance in Minho grades and 17% of the FOM subscore variances. CONCLUSIONS: The FOM examination seems to supplement local assessments by targeting constructs not currently measured. Therefore, it may contribute to a more comprehensive assessment of basic and clinical sciences knowledge.

Relationship between performance on part I of the American Board of Orthopaedic Surgery Certifying Examination and Scores on USMLE Steps 1 and 2.

BACKGROUND: This study investigated the strength of the relationship between performance on Part I of the American Board of Orthopaedic Surgery (ABOS) Certifying Examination and scores on United States Medical Licensing Examination (USMLE) Steps 1 and 2. METHOD: USMLE Step 1 and Step 2 scores on first attempt were matched with ABOS Part I results for U.S./Canadian graduates taking Part I for the first time between 2002 and 2006. Linear and logistic regression analyses investigated the relationship between ABOS Part I performance and scores on USMLE Step 1 and 2. RESULTS: Step 1 and Step 2 individually each explained 29% of the variation in Part I scores; using both scores increased this percentage to 34%. Results of logistic regression analyses showed a similar, moderately strong relationship with Part I pass/fail outcomes: Examinees with low scores on Steps 1 and 2 were at substantially greater risk for failing Part I. CONCLUSIONS: There is continuing empirical support for use of Step 1 and Step 2 scores in selection of residents to interview for orthopedics residency positions.

Use of multimedia on the step 1 and step 2 clinical knowledge components of USMLE: a controlled trial of the impact on item characteristics.

BACKGROUND: During 2007, multimedia-based presentations of selected clinical findings were introduced into the United States Medical Licensing Examination. This study investigated the impact of presenting cardiac auscultation findings in multimedia versus text format on item characteristics. METHOD: Content-matched versions of 43 Step 1 and 51 Step 2 Clinical Knowledge (CK) multiple-choice questions describing common pediatric and adult clinical presentations were administered in unscored sections of Step 1 and Step 2 CK. For multimedia versions, examinees used headphones to listen to the heart on a simulated chest while watching video showing associated chest and neck vein movements. Text versions described auscultation findings using standard medical terminology. RESULTS: Analyses of item responses for first-time examinees from U.S./Canadian and international medical schools indicated that multimedia items were significantly more difficult than matched text versions, were less discriminating, and required more testing time. CONCLUSIONS: Examinees can more readily interpret auscultation findings described in text using standard terminology than those same findings presented in a more authentic multimedia format. The impact on examinee performance and item characteristics is substantial.

Measurement characteristics of content-parallel single-best-answer and extended-matching questions in relation to number and source of options.

BACKGROUND: Previous research showed that extended-matching questions (EMQs) with eight options per set resulted in better score precision than EMQs with larger numbers of options or independent single-best-answer items (A-type) with five options. This study extends previous work using smaller numbers of options. METHOD: Ninety-six questions were presented in two formats on United States Medical Licensing Examination Step 2: as two-item EMQ sets and as independent A-types. Four versions of EMQs were used: five- and eight-option versions with options selected using statistics, and five- and eight-option versions with options selected by physicians. Seven A-type versions were used: three-, four-, five-, and eight-option versions with options selected using statistics, and three-, four-, and five-option versions with options selected by physicians. RESULTS: Items with more options were harder, required more time to complete, and had similar item discrimination. Option sets selected by physicians were easier, slightly more discriminating, and required less testing time. CONCLUSIONS: A-types with four or five options and EMQs with eight options make more efficient use of testing time. Provision of response statistics to content experts does not seem necessary to guide option selection.

Psychometric characteristics and response times for content-parallel extended-matching and one-best-answer items in relation to number of options.

BACKGROUND: This study investigated the impact of item format and number of options on the psychometric characteristics (p values and biserials) and response times for multiple-choice questions (MCQs) appearing on Step 2 of the United States Medical Licensing Examination. METHOD: In all, 192 MCQ items were used in the study. Each item was presented in two formats: in a two-item extended-matching set and as an independent item. For the extended matching format, there were two versions: a base version that included all options (10 to 26) and an 8-option version. For the independent-item format, there were three versions: a base version that included all options, and 8-option and 5-option versions created by a group of physicians that selected options without information about examinee performance. All items were embedded in unscored sections of the 2005-06 Step 2 test forms. RESULTS: Versions of items with more options were harder and required more testing time; no differences in item discrimination were observed. Mean response times for items presented in the extended-matching format were lower than for those presented as independent items, primarily because of shorter response times for the second item presented in a set. CONCLUSION: Use of the extended-matching format and smaller numbers of options per item (and more items) should result in more efficient use of testing time and greater score precision per unit of testing time.

The essential elements of undergraduate medical education in obstetrics and gynecology: a comparison of the Association of Professors of Gynecology and Obstetrics Medical Student Educational Objectives and the National Board of Medical Examiners Subject Examination.

OBJECTIVE: The objective of this study was to investigate whether the essential elements of the Association of Professors of Gynecology and Obstetrics (APGO) Medical Student Educational Objectives were adequately represented on the National Board of Medical Examiners (NBME) obstetrics and gynecology subject examination, and that the topics questioned on that examination were covered by the APGO objectives. STUDY DESIGN: The Undergraduate Medical Education Committee of APGO and the NBME staff separately reviewed the same 2 NBME obstetrics and gynecology subject examinations. The questions were mapped to the 15 essential elements of the APGO educational objectives and comparisons were made to check how well they matched. RESULTS: All the essential elements of the educational objectives were covered by the NBME subject examination. Of the questions on the examination, 99% were deemed appropriate for medical students with 70% of the questions mapping to "Priority 1" objectives. CONCLUSION: The NBME examination provides an appropriate assessment of mastery of what a medical student should learn, as represented by the APGO Medical Student Educational Objectives.

Psychometric characteristics and response times for one-best-answer questions in relation to number and source of options.

BACKGROUND: The research reported here investigated the impact of number and source of response options on the psychometric characteristics and response times for one-best-answer MCQs. METHOD: Ninety sets of MCQs were used in two studies; numbers of options in base versions of items ranged from 11 to 25. For each set, a United States Medical Licensing Examination Step 2 item-writing committee selected the five options viewed as most appropriate. For 40 used sets, two NBME staff constructed five- and eight-option versions to maximize item discrimination. All versions of items were embedded unscored on 2003-04 Step 2 test forms. RESULTS: Versions of items with more options were harder and required more testing time; no differences in item discrimination were observed in either study, but previous versions of the items in extended matching format were more discriminating than those used in the study. CONCLUSION: Use of smaller numbers of options (and more items) results in more efficient use of testing time.

Assessing the validity of the USMLE step 2 clinical knowledge examination through an evaluation of its clinical relevance.

PURPOSE: To assess the validity of the USMLE Step 2 Clinical Knowledge (CK) examination by addressing the degree to which experts view item content as clinically relevant and appropriate for Step 2 CK. METHOD: Twenty-seven experts were asked to complete three survey questions related to the clinical relevance and appropriateness of 150 Step 2 CK multiple-choice questions. Percentages, reliability estimates, and correlation coefficients were calculated and ordinary least squares regression was used. RESULTS: Results showed that 92% of expert judgments indicated the item content was clinically relevant, 90% indicated the content was appropriate for Step 2 CK, and 85% indicated the content was used in clinical practice. The regression indicated that more difficult items and more frequently used items are considered more appropriate for Step 2 CK. CONCLUSIONS: Results suggest that the majority of item content is clinically relevant and appropriate, thus providing validation support for Step 2 CK.

Using the NBME self-assessments to project performance on USMLE Step 1 and Step 2: impact of test administration conditions.

PROBLEM STATEMENT AND BACKGROUND: This study examined the extent to which performance on the NBME(R) Comprehensive Basic Science Self-Assessment (CBSSA) and NBME Comprehensive Clinical Science Self-Assessment (CCSSA) can be used to project performance on USMLE Step 1 and Step 2 examinations, respectively. METHOD: Subjects were 1,156 U.S./Canadian medical students who took either (1) the CBSSA and Step 1, or (2) the CCSSA and Step 2, between April 2003 and January 2004. Regression analyses examined the relationship between each self-assessment and corresponding USMLE Step as a function of test administration conditions. RESULTS: The CBSSA explained 62% of the variation in Step 1 scores, while the CCSSA explained 56% of Step 2 score variation. In both samples, Standard-Paced conditions produced better estimates of future Step performance than Self-Paced ones. CONCLUSIONS: Results indicate that self-assessment examinations provide an accurate basis for predicting performance on the associated Step with some variation in predictive accuracy across test administration conditions.