Challenges to assessing professional identity in medical students: a tale of two measures.

Background: Professional identity formation (PIF), a foundational process in becoming a physician, includes establishment of values, moral principles, and self-awareness. The purpose of this report is to examine challenges in establishing the validity of measures of identity fusion as one facet of PIF. Method: Utilizing the modern approach of validity as a unitary concept, the authors generated six hypotheses to examine the evidence for the construct validity of the scores of Physician Professional Identity (PPI) and Identity Integration (IdIn), considering relationships of these measures with each other, year of training and data from a larger survey. Results: Responses from 3473 students at 8 medical schools revealed a weak association between the measures with distributions varying by cohort. PPI had a stronger relationship to cohort and IdIn was moderately associated with students' attitudes relevant to social media use. Responses were independent of response format and evidence supported the interpretation of scores for IdIn as indications of integration of identity. Discussion : Sufficient evidence was found to suggest that these measures assess aspects of PIF. Use of these measures as part of a multidimensional, longitudinal approach to refining understanding of the construct of PIF and developing effective assessment strategies.

Program evaluation models and related theories: AMEE guide no. 67.

This Guide reviews theories of science that have influenced the development of common educational evaluation models. Educators can be more confident when choosing an appropriate evaluation model if they first consider the model's theoretical basis against their program's complexity and their own evaluation needs. Reductionism, system theory, and (most recently) complexity theory have inspired the development of models commonly applied in evaluation studies today. This Guide describes experimental and quasi-experimental models, Kirkpatrick's four-level model, the Logic Model, and the CIPP (Context/Input/Process/Product) model in the context of the theories that influenced their development and that limit or support their ability to do what educators need. The goal of this Guide is for educators to become more competent and confident in being able to design educational program evaluations that support intentional program improvement while adequately documenting or describing the changes and outcomes-intended and unintended-associated with their programs.

Effects of comprehensive educational reforms on academic success in a diverse student body.

CONTEXT: Calls for medical curriculum reform and increased student diversity in the USA have seen mixed success: performance outcomes following curriculum revisions have been inconsistent and national matriculation of under-represented minority (URM) students has not met aspirations. Published innovations in curricula, academic support and pipeline programmes usually describe isolated interventions that fail to affect curriculum-level outcomes. METHODS: United States Medical Licensing Examination (USMLE) Step 1 performance and graduation rates were analysed for three classes of medical students before (matriculated 1995-1997, n=517) and after (matriculated 2003-2005, n=597) implementing broad-based reforms in our education system. The changes in pipeline recruitment and preparation programmes, instructional methods, assessment systems, academic support and board preparation were based on sound educational principles and best practices. RESULTS: Post-reform classes were diverse with respect to ethnicity (25.8% URM students), gender (51.8% female), and Medical College Admissions Test (MCAT) score (range 20-40; 24.1% scored ≤ 25). Mean±standard deviation MCAT scores were minimally changed (from 27.2±4.7 to 27.8±3.6). The Step 1 failure rate decreased by 69.3% and mean score increased by 14.0 points (effect size: d=0.67) overall. Improvements were greater among women (failure rate decreased by 78.9%, mean score increased by 15.6 points; d=0.76) and URM students (failure rate decreased by 76.5%, mean score increased by 14.6 points; d=0.74), especially African-American students (failure rate decreased by 93.6%, mean score increased by 20.8 points; d=1.12). Step 1 scores increased across the entire MCAT range. Four- and 5-year graduation rates increased by 7.1% and 5.8%, respectively. CONCLUSIONS: The effect sizes in these performance improvements surpassed those previously reported for isolated interventions in curriculum and student support. This success is likely to have resulted from the broad-based, mutually reinforcing nature of reforms in multiple components of the education system. The results suggest that a narrow reductionist view of educational programme reform is less likely to result in improved educational outcomes than a system perspective that addresses the coordinated functioning of multiple aspects of the academic enterprise.

Comprehensive changes in the learning environment: subsequent step 1 scores of academically at-risk students.

BACKGROUND: During the past 10 years at our institution, a number of changes have been instituted in the learning environment, including instructional techniques, assessment methods, academic support, and explicit board preparation. METHOD: The authors studied the Step 1 performance of students with MCAT scores of 20 to 25 in our former and current curricula. Effect sizes were calculated for score improvement using adjusted means from ANCOVA with covariates of MCAT and age. RESULTS: The overall effect size was 0.48, with larger effects seen for underrepresented minority students overall (d = 0.64) and African American students especially (d = 0.77), representing medium to large effects. Overall failure rates decreased by two thirds. CONCLUSIONS: Comprehensive changes in the learning environment were followed by substantial improvement in Step 1 performance among academically at-risk students.

An evaluation of depressed mood in two classes of medical students.

OBJECTIVE: To assess depression rates in contemporary medical students. METHOD: The Beck Depression Inventory (BDI) was administered anonymously to two medical school classes at matriculation, the end of first year, and the end of second year. RESULTS: Median scores for both classes were low at all points. The proportion of students scoring in the moderate or severely depressed range increased from 5.8% at matriculation to 10.5% by end of Year 2 for the Class of 2004, and from 5.1% to 11.9% over the same time period for the Class of 2005. Overall, 75-83% of students at every administration scored in the lowest BDI score range. CONCLUSIONS: The percentage of medical students who experience depressed mood increased over time, but to a lesser degree than in previous studies.

Measuring what medical students think about complementary and alternative medicine (CAM): a pilot study of the complementary and alternative medicine survey.

With increasing national and international support for the development of Complementary and Alternative Medicine (CAM) curricula in American medical schools, it is essential to measure what learners know and believe about CAM in order to assess outcomes of new teaching efforts. This paper describes the development and initial results of a survey designed for those purposes. The survey is constructed so that earlier single-institution studies of students' attitudes toward CAM topics, preferred ways of learning about CAM, and students' use of CAM therapies for self-care might be replicated and extended. A pilot test of the Complementary and Alternative Medicine ( CAM) Survey was conducted with third-year medical students at the University of Texas Medical Branch. Validity and reliability studies of the survey were conducted. Findings were compared to those of previously published studies. Interpretable subscales were constructed from survey questions. Students' attitudes toward CAM-related topics were generally favorable. More students were familiar with biological-based therapies than with other CAM therapies. The majority of respondents listed lectures as their preferred way to learn about CAM. Respondents' own CAM use varied; minority and economically-disadvantaged students were more likely to use CAM therapies than other students. The survey is a promising assessment of student attitudes toward and knowledge of CAM therapies.

Investigating the use of negatively phrased survey items in medical education settings: common wisdom or common mistake?

BACKGROUND: Attitude surveys in medical education often combine negative items with positive items, a "common wisdom" strategy to counteract response sets. A body of research in other fields has demonstrated that negatively phrased items affect reliability and validity by introducing measurement artifact into scores. The authors investigated the effect of negatively phrased items in the Medical School Learning Environment Survey (MSLES) with data from six medical student cohorts at the University of Texas Medical Branch, Galveston, Texas. METHOD: This study describes the impact of negatively phrased items in the MSLES through analysis of item and scale means and comparisons of coefficient alpha values. RESULTS: Findings indicate that negatively phrased items performed differently than the positively phrased items. Negatively phrased items were associated with lower scale reliability. CONCLUSION: The authors conclude, as did earlier studies, that negatively phrased items introduce an artifact into attitude measurement. The "common wisdom" practice of routinely including negative items should be employed with care.

Introduction of patient video clips into computer-based testing: effects on item statistics and reliability estimates.

PURPOSE: Using patient video clips to evaluate examinees' skills in interpreting physical examination findings is possible with computer-based testing, but the psychometric properties of video-based questions are unknown. METHOD: We developed parallel test questions incorporating video clips or text descriptions of abnormal neurologic findings and administered them to 106 fourth-year medical students finishing their Neurology Clerkship. RESULTS: Overall, video-based questions had comparable difficulty and discrimination compared to analogous text-based questions. Preliminary studies indicated similar reliability with text- and video-based questions. CONCLUSIONS: The inclusion of patient video clips in computer-based testing is feasible from technical, practical, and psychometric perspectives. Further study is needed to gather validity evidence for this novel question format.

Teaching and testing physical examination skills without the use of patients.

OBJECTIVE: To design a cardiopulmonary physical exam curriculum that does not involve the use of patients. Bedside teaching is becoming a lost art, and the use of alternative methods of instruction such as simulation has become increasingly important. Simulators have been shown to enhance physical examination skills of students and physicians in training.(1) DESCRIPTION: In 1995, a program was started to improve cardiopulmonary physical diagnosis and the teaching of auscultation at the University of Texas Medical Branch at Galveston (UTMB). The teaching manikin "Harvey" played a vital role in the development of the new curriculum. In 1997, UTMB adopted an organ-based approach to the basic science curriculum. The cardiopulmonary module in the basic science curriculum was a ten-week course taught in the second year of medical school. The physical diagnosis section of that course involved six instructional hours; four of the six hours were dedicated to cardiac auscultation and two hours to pulmonary auscultation. Only simulators and CD-ROMs were used for instruction. The 184 second-year medical students at UTMB were formed into small groups for instruction and practice. Although "Harvey" was an effective teaching tool, other simulators had to be developed for testing students' skills after instruction. It would be very difficult to administer a skills OSCE for 184 students without the development of several smaller transportable simulators. A commercially available blood pressure simulator from the Medical Plastics Laboratory, Inc., Gatesville, TX, was used to test the accuracy of students' blood pressure readings. Small auscultation transducers combined with a palpable pulse simulator, developed by one of the authors (WT) in collaboration with Andries Acoustics, Spicewood, TX, were used to efficiently test students' proficiency in cardiopulmonary auscultation. Digital simulated cardiopulmonary sounds were recorded onto a standard CD-ROM mini-disc and transmitted to the small transducers. Students used their own stethoscopes for auscultation. The targeted skills were efficiently tested in one hour of testing time per student. DISCUSSION: This cardiopulmonary instructional module was well received by the second-year medical students. In the skills OSCE, 80% of the students accurately measured systolic and diastolic blood pressure to within 5 mm Hg. Cardiopulmonary auscultation proficiency results showed average recognition of 60% for cardiac abnormalities and 88% for pulmonary sounds. Developing auscultation transducers with pulse simulation capability ensured that students could identify systole. Therefore, heart murmurs and sounds could be timed with the cardiac cycle. We found the results from the skills OSCE encouraging. Most students demonstrated reasonable competency in the skills taught, and the new transportable simulators performed well. The six-hour instructional module was meant to prepare students for their bedside teaching during the third year of medical school. The significant cost of the "Harvey" simulator may be a barrier to its widespread use for teaching. Therefore, continued development of smaller transportable simulators for teaching and testing purposes is important.

Incorporating simulators in a standardized patient exam.

OBJECTIVE: Using simulated patients during a clinical skills exam that involves many students has the advantage of standardizing the delivery of historical data. One major disadvantage is the inability to standardize the physical exam findings. We designed a simulated patient exam that incorporates simulated abnormal physical exam findings. DESCRIPTION: The simulated patient exam case was divided into three separate stations: (1) the simulated patient's history, (2) the simulated physical exam, and (3) the presentation station. Dyspnea was chosen as the chief complaint because of the broad differential of possible cardiac and pulmonary auscultatory findings. In the first station, students obtained historical data from the standardized simulated patient. Students were graded on their ability to ask appropriate historical questions. Trained observers were used to verify the numbers of historical cues obtained by the students. The second station consisted of simulated physical exam findings. Students first measured the blood pressure on a commercially available blood pressure simulator arm from the Medical Plastics Laboratory, Inc., Gatesville, TX. Students then auscultated an abnormal digital heart sound and pulmonary sound from a small auscultation transducer developed by Andries Acoustics, Spicewood, TX. Students also palpated a simulated pulse from a newly developed pulse transducer. Digital cardiopulmonary sounds and pulse data were recorded onto a CD-ROM disc and transmitted to the small transducers via a CD-ROM disc player. Students used their own stethoscopes to auscultate cardiopulmonary sounds from the small transducers. The students were graded in the second station on their ability to accurately measure a blood pressure, identify abnormal cardiopulmonary digital sounds, and finally describe a peripheral pulse. In the third station, students presented the historical data and physical exam findings to a faculty member, and then provided a differential diagnosis list based on their key findings from the other two stations. A total of 171 students (n = 171) completed the simulated patient exam. Each student completed the exam in 45 minutes. DISCUSSION: In our simulated patient exam, students were evaluated not only on their data-gathering skills for key historical findings but also on the ability to correctly identify key physical exam findings such as abnormal cardiopulmonary sounds. Key physical exam findings were then integrated into the clinical decision-making process, which was presented in the faculty presentation station. Simulated patients with abnormal cardiopulmonary findings can be used for testing purposes. However, cardiac auscultatory abnormalities such as the ventricular S3 gallop are difficult to find and usually occur in a decompensated state such as heart failure. Other physical exam findings such as pulmonary crackles and wheezes also occur in decompensated conditions. Therefore, the use of simulators during a simulated patient exam offers the possibility of introducing several abnormal physical exam findings without having an unstable patient present in an exam setting. Further, the use of simulated physical exam findings allows for complete standardization of a clinical-simulated patient exam.

Clinical performance assessment and interactive video teleconferencing: an iterative exploration.

BACKGROUND: The direct observation of students in authentic settings by faculty provides valuable feedback on performance and helps ensure mastery of clinical skills. DESCRIPTION: We explored the use of interactive video technology (IVT) as a way of involving community preceptors as raters on a clinical performance exam for 3rd-year students after their family medicine clerkship. Family medicine preceptors, from locations in their communities, observed students on campus conduct interviews and physical exams of standardized patients and then interacted with them during their case presentations. EVALUATION: We chose an action research approach to this project and conducted four independent trials. Interviews and observations were structured around three areas of concern: human, technical, and institutional. CONCLUSIONS: We feel confident in recommending IVT as a viable option for involving community preceptors in high-stakes testing and with other campus-based activities. We also report on the value of IVT in faculty development activities.