Using educational prescriptions to teach medical students evidence-based medicine.

PURPOSE: To evaluate feasibility and impact of evidence-based medicine (EBM) educational prescriptions (EPs) in medical student clerkships. METHODS: Students answered clinical questions during clerkships using EPs, which guide learners through the "four As" of EBM. Epidemiology fellows graded EPs using a rubric. Feasibility was assessed using descriptive statistics and student and fellow end-of-study questionnaires, which also measured impact. In addition, for each EP, students reported patient impact. Impact on EBM skills was assessed by change in EP scores over time and scores on an EBM objective structured clinical exam (OSCE) that were compared to controls from the prior year. RESULTS: 117 students completed 402 EPs evaluated by 24 fellows. Average score was 7.34/9.00 (SD 1.58). 69 students (59%) and 21 fellows (88%) completed questionnaires. Most students thought EPs improved "Acquiring" and "Appraising". Almost half thought EPs improved "Asking" and "Applying". Fellows did not value grading EPs. For 18% of EPs, students reported a "change" or "potential change" in treatment. 56% "confirmed" treatment. EP scores increased by 1.27 (95% CI: 0.81-1.72). There were no differences in OSCE scores between cohorts. CONCLUSIONS: Integrating EPs into clerkships is feasible and has impact, yet OSCEs were unchanged, and research fellows had limitations as evaluators.

Single and Multiple Ability Estimation in the SEM Framework: A Non-Informative Bayesian Estimation Approach.

Latent variable models with many categorical items and multiple latent constructs result in many dimensions of numerical integration, and the traditional frequentist estimation approach, such as maximum likelihood (ML), tends to fail due to model complexity. In such cases, Bayesian estimation with diffuse priors can be used as a viable alternative to ML estimation. The present study compares the performance of Bayesian estimation to ML estimation in estimating single or multiple ability factors across two types of measurement models in the structural equation modeling framework: a multidimensional item response theory (MIRT) model and a multiple-indicator multiple-cause (MIMIC) model. A Monte Carlo simulation study demonstrates that Bayesian estimation with diffuse priors, under various conditions, produces quite comparable results to ML estimation in the single- and multi-level MIRT and MIMIC models. Additionally, an empirical example utilizing the Multistate Bar Examination is provided to compare the practical utility of the MIRT and MIMIC models. Structural relationships among the ability factors, covariates, and a binary outcome variable are investigated through the single- and multi-level measurement models. The paper concludes with a summary of the relative advantages of Bayesian estimation over ML estimation in MIRT and MIMIC models and suggests strategies for implementing these methods.

Commentary: measurement and interpretation challenges in comparing student performance outcomes from different medical schools.

This commentary points to several measurement issues that arise in assessing medical student performance outcomes and then discusses the challenge of interpreting between-school differences. A problem often encountered in assessing student learning is creating an instrument that is at the right "pay grade." If it is too easy, ceiling effects compress scores. If it is too difficult, examinee performance can compress about chance values, and morale problems can occur. These issues are discussed in the context of a report by Williams and colleagues that measures medical student performance across five institutions on instruments assessing diagnostic pattern recognition and clinical data interpretation. The author of this commentary observes that, when interpreting between-school differences in assessing student learning, what can seem like small differences can have important consequences.

Building a competency-based curriculum: the agony and the ecstasy.

Physician competencies have increasingly been a focus of medical education at all levels. Although competencies are not a new concept, when the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) jointly agreed on six competencies for certification and maintenance of certification of physicians in 1999, it brought about renewed interest. This article gives a brief overview of how a competency-based curriculum differs from other approaches and then describes the issues that need to be considered in the design and implementation of such a curriculum. In order to achieve success, a competency-based curriculum requires careful planning, preparation and a long-term commitment from everyone involved in the educational process. Building a competency-based curriculum is really about maintaining quality control and relinquishing control to those who care the most about medical education, our students. In the face of the many challenges that are facing undergraduate medical education (UME), including declining availability of teaching patients and over-burdened faculty, instituting quality control and relinquishing control will be necessary to maintain high quality.

Impact of physician assistant care on office visit resource use in the United States.

OBJECTIVE: To investigate whether the use of physician assistants (PAs) as providers for a substantive portion of a patient's office-based visits affects office visit resource use. DATA SOURCE: Medical Expenditure Panel Survey (MEPS) Household Component data from 1996 to 2004. STUDY DESIGN: This retrospective cohort study compares the number of office-based visits per year between adults for whom PAs provided >or=30 percent of visits and adults cared for by physicians only. DATA COLLECTION/EXTRACTION METHODS: The Agency for Healthcare Research and Quality collects MEPS data using methods designed to produce data representative of the U.S. noninstitutionalized civilian population. Negative binomial regression was used to compare the number of visits per year between persons with and without PA care, adjusted for demographic, geographic, and socioeconomic factors; insurance status; health status; and medical conditions. PRINCIPAL FINDINGS: After case-mix adjustment, patients for whom PAs provided a substantive portion of care used about 16 percent fewer office-based visits per year than patients cared for by physicians only. This difference in the use of office-based visits was not offset by increased office visit resource use in other settings. CONCLUSIONS: Results indicate that the inclusion of PAs in the U.S. provider mix does not affect overall office visit resource use.

Diagnostic radiology resident compliance with recommended health guidelines: effect of resident work environment.

RATIONALE AND OBJECTIVES: To determine diagnostic radiology resident compliance with recommended health guidelines for physical activity, body weight, diet, related health indicators, and the effects of the resident work environment on compliance. MATERIALS AND METHODS: A request was electronically mailed to members of the Association of Program Directors in Radiology and the Association of Program Coordinators in Radiology in May 2007 and again in June 2007, asking members to forward to their radiology residents an invitation to complete an online health survey. Frequency counts and Fisher's exact test, respectively, were used to summarize results and to determine statistically significant relationships between survey variables. RESULTS: A total of 811 radiology residents completed the survey, representing 18% of 4,412 diagnostic radiology residents. Five hundred forty-five (67.2%) of 811 were male and 264 (32.6%) female. Two hundred ten (25.9%) were first-year, 239 (29.5%) second-year, 201 (24.8%) third-year, and 161 (19.9%) fourth-year residents. Three hundred two (37.2%) engaged in recommended guidelines for physical activity and < or =465 (57.3%) complied with each of multiple federal dietary guidelines (excluding alcohol intake). Up to 329 (40.6%) residents did not know whether they were in compliance with various dietary guidelines. A total of 426 (52.5%) residents reported working > or =60 hours/week, which significantly correlated with less physical activity (P = .013). CONCLUSION: A substantial number of residents are out of compliance with federal guidelines for physical activity and diet and are not knowledgeable about their personal dietary intake. Long work hours are related to a lack of physical activity. Radiology programs may be able to influence resident health practices by modifying work hours and the working environment, encouraging healthy dietary intake and physical activity, and instituting campaigns to inform residents and faculty about health guidelines and available wellness programs.

Defining characteristics of educational competencies.

CONTEXT: Doctor competencies have become an increasing focus of medical education at all levels. However, confusion exists regarding what constitutes a competency versus a goal, objective or outcome. OBJECTIVES: This article attempts to identify the characteristics that define a competency and proposes criteria that can be applied to distinguish between competencies, goals, objectives and outcomes. METHODS: We provide a brief overview of the history of competencies and compare competencies identified by international medical education organisations (CanMEDS 2005, Institute for International Medical Education, Dundee Outcome Model, Accreditation Council for Graduate Medical Education/American Board of Medical Specialties). Based upon this review and comparisons, as well as on definitions of competencies from the literature and theoretical and conceptual analyses of the underpinnings of competencies, the authors develop criteria that can serve to distinguish competencies from goals, objectives and outcomes. RESULTS: We propose 5 criteria which can be used to define a competency: it focuses on the performance of the end-product or goal-state of instruction; it reflects expectations that are external to the immediate instructional programme; it is expressible in terms of measurable behaviour; it uses a standard for judging competence that is not dependent upon the performance of other learners, and it informs learners, as well as other stakeholders, about what is expected of them. CONCLUSIONS: Competency-based medical education is likely to be here for the foreseeable future. Whether or not these 5 criteria, or some variation of them, become the ultimate defining criteria for what constitutes a competency, they represent an essential step towards clearing the confusion that reigns.

Missing in action: care by physician assistants and nurse practitioners in national health surveys.

OBJECTIVE: To assess applicability of national health survey data for generalizable research on outpatient care by physician assistants (PAs) and nurse practitioners (NPs). DATA SOURCES: Methodology descriptions and 2003 data files from the National Ambulatory Medical Care Survey, the National Hospital Ambulatory Medical Care Survey, the Medical Expenditure Panel Survey, and the Community Tracking Study. STUDY DESIGN: Surveys were assessed for utility for research on PA and NP patient care, with respect to survey coverage, structure, content, generalizability to the U.S. population, and validity. National estimates of patient encounters, statistically adjusted for survey design and nonresponse, were compared across surveys. DATA COLLECTION/EXTRACTION METHODS: Surveys were identified through literature review, selected according to inclusion criteria, and analyzed based on methodology descriptions. Quantitative analyses used publicly available data downloaded from survey websites. PRINCIPAL FINDINGS: Surveys varied with respect to applicability to PA and NP care. Features limiting applicability included (1) sampling schemes that inconsistently capture nonphysician practice, (2) inaccurate identification of provider type, and (3) data structure that does not support analysis of team practice. CONCLUSIONS: Researchers using national health care surveys to analyze PA and NP patient interactions should account for design features that may differentially affect nonphysician data. Workforce research that includes NPs and PAs is needed for national planning efforts, and this research will require improved survey methodologies.

Statistical criteria for setting thresholds in medical school admissions.

In 2001, Dr Jordan Cohen, President of the AAMC, called for medical schools to consider using an Medical College Admission Test (MCAT) threshold to eliminate high-risk applicants from consideration and then to use non-academic qualifications for further consideration. This approach would seem to be consistent with the recent Supreme Court ruling on the University of Michigan admissions cases. Research to support a threshold approach has been reported in many different ways, making comparability problematic. This study examines an assortment of statistical indices that have been used to determine thresholds in undergraduate science grade point average (USGPA) and MCAT sub-scores and total sum and compares them in terms of their sensitivity and specificity for determining Medical Licensing Exam (USMLE Step 1) first time failure. Data for medical school entering classes of 1992-1998 (N=752) from a large Midwestern medical school are used to determine the set of statistics that provide the most useful information for this purpose. The results support plots of risk differences, odds-ratios, sensitivity and specificity for setting cut-scores.

A comparison of statistical criteria for setting optimally discriminating MCAT and GPA thresholds in medical school admissions.

BACKGROUND: Using Medical College Admission Test-grade point average (MCAT-GPA) scores as a threshold has the potential to address issues raised in recent Supreme Court cases, but it introduces complicated methodological issues for medical school admissions. PURPOSE: To assess various statistical indexes to determine optimally discriminating thresholds for MCAT-GPA scores. METHODS: Entering classes from 1992 through 1998 (N = 752) are used to develop guidelines for cut scores that optimize discrimination between students who pass and do not pass the United States Medical Licensing Examination (USMLE) Step 1 on the first attempt. RESULTS: Risk differences, odds ratios, sensitivity, and specificity discriminated best for setting thresholds. Compensatory versus noncompensatory procedures both accounted for 54% of Step 1 failures, but demanded different performance requirements (noncompensatory MCAT-biological sciences = 8, physical sciences = 7, verbal reasoning = 7--sum of scores = 22; compensatory MCAT total = 24). CONCLUSIONS: Rational and defensible intellectual achievement thresholds that are likely to comply with recent Supreme Court decisions can be set from MCAT scores and GPAs.

Patient, faculty, and self-assessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skills.

RATIONALE AND OBJECTIVES: To develop and test the reliability, validity, and feasibility of a 360-degree evaluation to measure radiology resident competence in professionalism and interpersonal/communication skills. MATERIALS AND METHODS: An evaluation form with 10 Likert-type items related to professionalism and interpersonal/communication skills was completed by a resident, supervising radiologist and patient after resident-patient interactions related to breast biopsy procedures. Residents were also evaluated by faculty, using an end-of-rotation global rating form. Residents, faculty, and technologists were queried regarding their reaction to the assessments after a 7-month period. RESULTS: Fifty-six complete 360-degree data sets (range, 2-14 per resident) and seven rotational evaluations for seven residents were analyzed and compared. Internal consistency reliability estimates were 0.85, 0.86, and 0.87 for resident, patient, and faculty 360-degree evaluations, respectively. Correlations between resident-versus-patient, resident-versus-faculty, and patient-versus-faculty ratings for the 56 interactions were -0.06 (P =.64), 0.31 (P <.02), and 0.45 (P <.0006), respectively. Pearson correlation coefficients approached significant correlation (0.70) between the faculty global rating and patient 360-degree scores (P =.08) but not with faculty 360-degree scores. Residents and faculty felt that completing the 360-degree forms was easy, but the requirement for faculty presence during the consent process was burdensome. CONCLUSION: Results from this pilot study suggest that self, faculty, and patient evaluations of resident performance constitutes a valid and reliable assessment of resident competence. Additional data are needed to determine whether the 360-degree assessment should be incorporated into residency programs and how frequently the assessment should be performed. Requiring only a specified number of assessments per rotation would make the process less burdensome for residents and faculty.

Matriculating student perceptions of changes to the admissions interview process at the University of Wisconsin Medical School: a prospective, controlled comparison.

PURPOSE: The purpose of this study was to assess whether changes in the admissions interview process improved matriculating students' perceptions of the quality of the admissions interview process. METHODS: We surveyed matriculating medical students for a 3-year period. Over this period, the admissions process and procedures went through a review and then a re-engineering. The survey provided 1 year of baseline data (while the review was undertaken), 1 year of data as recommendations from the review were progressively implemented, and 1 year of data on full implementation of recommendations. RESULTS: From baseline to full implementation, there were statistically significant increases (p < .05) in the percentage of matriculating students who found the University of Wisconsin (UW) Medical School interview process useful (31% increase), thorough (50% increase) and better than that of other medical schools (28% increase). There was also a statistically significant decrease in the percentage of matriculating students who found the UW Medical School interview process to be less impressive than other schools (29% decrease) and in need of improvement (45% decrease). EDUCATIONAL SIGNIFICANCE/CONCLUSIONS: Changes made in the UW Medical School's interview process yielded significant increases in perceptions of the quality of the experience by matriculating students. Since interviewing is expensive for both the institution and the applicant, it should have a clear purpose. The manner in which interviews are conducted should be critically reviewed periodically to ensure that the interview continues to meet its intended needs.

Assessing personal qualities in medical school admissions.

The authors analyze the challenges to using academic measures (MCAT scores and GPAs) as thresholds for admissions and, for applicants exceeding the threshold, using personal qualities for admission decisions; review the literature on using the medical school interview and other admission data to assess personal qualities of applicants; identify challenges of developing better methods of assessing personal qualities; and propose a unified system for assessment. The authors discuss three challenges to using the threshold approach: institutional self-interest, inertia, and philosophical and historical factors. Institutional self-interest arises from the potential for admitting students with lower academic credentials, which could negatively influence indicators used to rank medical schools. Inertia can make introducing a new system complex. Philosophical and historical factors are those that tend to value maximizing academic measures. The literature identifies up to 87 different personal qualities relevant to the practice of medicine, and selecting the most salient of these that can be practically measured is a challenging task. The challenges to developing better personal quality measures include selecting and operationally defining the most important qualities, measuring the qualities in a cost-effective manner, and overcoming "cunning" adversaries who, with the incentive and resourcefulness, can potentially invalidate such measures. The authors discuss potential methods of measuring personal qualities and propose a unified system of assessment that would pool resources from certification and recertification efforts to develop competencies across the continuum with a dynamic, integrated approach to assessment.

Teaching radiology to medical students: an integrated approach.

RATIONALE AND OBJECTIVES: This study assessed medical student satisfaction with radiology lectures integrated into the 3rd-year student internal medicine clerkship, compared with faculty lectures in an independent radiology course, and investigated the effects of integrated instruction on departmental compensation from the medical school. MATERIALS AND METHODS: Students' evaluations were compared, with two-way analysis of variance, for faculty who presented lectures in an integrated radiology course and faculty who presented lectures in an independent radiology course from July 1998 to June 2001. Radiology department compensation from the medical school for each course was computed per contact hour. RESULTS: For the integrated course (663 evaluations), the mean overall faculty rating was 1.44 (1 = excellent, 5 = poor), lower (more positive) than the mean overall rating of 1.53 for the independent course (518 evaluations) (P = .037). The interaction between type of instruction and topic was significant only for chest and musculoskeletal imaging, which were rated more positive and less positive in the integrated course, respectively (P < .001). For the integrated course the radiology department received $762 per lecture hour, and for the independent course it received $296 per contact hour (all types) and $1,183 per lecture hour. CONCLUSION: Student ratings of faculty lectures in an integrated course were excellent and comparable to those in an independent radiology course. The medical school differentiated the efforts of radiology faculty in the two courses through its budgeting process, awarding greater compensation per contact hour for participation in the integrated course. If only lecture hours are considered, compensation was greater for the independent course.