Identifying emerging research collaborations and networks: method development.

Clinical and translational research is a multidisciplinary, collaborative team process. To evaluate this process, we developed a method to document emerging research networks and collaborations in our medical center to describe their productivity and viability over time. Using an e-mail survey, sent to 1,620 clinical and basic science full- and part-time faculty members, respondents identified their research collaborators. Initial analyses, using Pajek software, assessed the feasibility of using social network analysis (SNA) methods with these data. Nearly 400 respondents identified 1,594 collaborators across 28 medical center departments resulting in 309 networks with 5 or more collaborators. This low-burden approach yielded a rich data set useful for evaluation using SNA to: (a) assess networks at several levels of the organization, including intrapersonal (individuals), interpersonal (social), organizational/institutional leadership (tenure and promotion), and physical/environmental (spatial proximity) and (b) link with other data to assess the evolution of these networks.

Engaging students and faculty: implications of self-determination theory for teachers and leaders in academic medicine.

BACKGROUND: Much of the work of teachers and leaders at academic health centers involves engaging learners and faculty members in shared goals. Strategies to do so, however, are seldom informed by empirically-supported theories of human motivation. DISCUSSION: This article summarizes a substantial body of motivational research that yields insights and approaches of importance to academic faculty leaders. After identification of key limitations of traditional rewards-based (i.e., incentives, or 'carrots and sticks') approaches, key findings are summarized from the science of self-determination theory. These findings demonstrate the importance of fostering autonomous motivation by supporting the fundamental human needs for autonomy, competence, and relatedness. In turn, these considerations lead to specific recommendations about approaches to engaging autonomous motivation, using examples in academic health centers. SUMMARY: Since supporting autonomous motivation maximizes both functioning and well-being (i.e., people are both happier and more productive), the approaches recommended will help academic health centers recruit, retain, and foster the success of learners and faculty members. Such goals are particularly important to address the multiple challenges confronting these institutions.

Assessing a method to limit influence of standardized tests on clerkship grades.

BACKGROUND: Grading committees give excessive weight to standardized-examination scores. PURPOSE: Understanding that biases are often ingrained in grading processes, we sought to assess the influence of a structured grading policy in limiting this effect. METHODS: All 7 clerkship grading committees derived students' clinical scores while blinded to examination scores. Scores were combined to yield a final rank order, which was used to derive grade cutoffs. Logit regression was performed to assess the contribution of clinical and examination scores to final grades. Results were compared to a similar analysis where committees were not blinded to examination scores. RESULTS: In contrast to prior findings, grading committees consistently assigned greater weight to clinical-performance scores in assigning final grades when blinded to examination scores. CONCLUSIONS: Grading committees may be unaware of the extent to which they discount clinical assessments when they are at odds with the results of standardized examinations. This can be addressed with a procedure that blinds grading committees to examination scores.

Medical student attitudes toward patients in diverse care settings: the impact of a patient evaluation course.

BACKGROUND: First-year medical students typically have limited exposure to patients in diverse care settings, such as rehabilitation facilities and nursing homes. PURPOSE: It is unknown whether students bring predetermined attitudes toward these patients, or whether attitudes are influenced by early exposure. We studied this in a new course that provides opportunities for students to interact with patients of various ages and disabilities. METHODS: We conducted surveys of 1st-year medical students at the University of Rochester in the year prior to the new course and during its initial year. We used factor analysis to derive underlying dimensions of students' responses. We also investigated the impact that the course had on their perceptions. RESULTS: In both years, we found that students conceptualize patient care along 2 affective dimensions (comfort and pleasure) and 2 attitudinal dimensions (bias and pessimism), rather than by type of disability. CONCLUSIONS: This 10-week course improved their affect toward these patient groups but had little effect on their general attitudes toward the value of caring for them.

History and practice of competency-based assessment.

CONTEXT: In response to historical trends in expectations of doctors, the goals of medical education are increasingly framed in terms of global competencies. The language of these competencies has tended to adopt a prescriptive, rather than descriptive, approach. However, despite widespread agreement on the importance of competency-based education and more than two decades of study, this effort has not generated a dependable set of assessment tools. DISCUSSION: Because models of competency are legislated, rather than shaped by scholarly consideration of empirical data, it is unlikely that such models directly reflect actual human behaviour. Efforts to measure clinical behaviours could benefit from increased clarity in three related conceptual areas. Firstly, the language of educational constructs should be framed in terms of data-based hypotheses, rather than in terms of intuitively plausible abilities. Secondly, these constructs should be specified in terms of the situations to which they are relevant, rather than as global personal characteristics. Finally, the resources required to measure these constructs should be rigorously established because a common resource-based metric would allow for rational selection of assessment methods. Specific methods to establish each of these objectives are discussed. CONCLUSIONS: The political process of negotiating educational objectives should not be confused with the scientific work of establishing coherent and interpretable patterns of behaviour. Although the two activities can complement one another, each has its own distinct methods and style of discourse. It is thus critical to maintain boundaries between these two approaches to defining professional performance.

Assessing teamwork: a reliable five-question survey.

BACKGROUND AND OBJECTIVES: Currently available tools to measure teamwork, an essential component of primary care, are generally very resource intensive and thus cannot be administered frequently. To explore the possibility of developing a brief teamwork-assessment instrument, we first administered 29 questions about teamwork from the Practice Environment Checklist (PEC) to all members of six clinical teams in a residency outpatient practice (n=56). We found that the scale assessed a single dimension of teamwork and that a five item survey has acceptable reliability (Cronbach alpha=0.89). In a subsequent validation study among an expanded sample of clinic staff (n=89), we found that the five-item questionnaire could be completed in less than 3 minutes. It continued to have an acceptable internal consistency (Cronbach alpha=0.82) and that all five items had sizeable item-total correlations. The resulting short form of the PEC may be useful for frequent assessment of team function.

Commentary: pitfalls in assessment of competency-based educational objectives.

Requirements for accreditation of medical professionals are increasingly cast in the language of general competencies. Because the language of these competencies is generally shaped by negotiations among stakeholders, however, it has proven difficult to attain consensus on precise definitions. This lack of clarity is amplified when attempting to measure these essentially political constructs in individual learners. The authors of this commentary frame these difficulties within modern views of test validity. The most significant obstacle to valid measurement is not necessarily a lack of useful tools but, rather, a general unwillingness to question whether the competencies themselves represent valid measurement constructs. Although competencies may prove useful in defining an overall social mission for organizations, such competencies should not be mistaken for measurable and distinct attributes that people can demonstrate in the context of their actual work.

Development of an audit method to assess the prevalence of the ACGME's general competencies in an undergraduate medical education curriculum.

BACKGROUND: Despite the use of competency-based frameworks to evaluate physicians, the role of competency-based objectives in undergraduate medical education remains uncertain. PURPOSE: By use of an audit methodology, we sought to determine how the six Accreditation Council for Graduate Medical Education (ACGME) competencies, conceptualized as educational domains, would map onto an undergraduate medical curriculum. METHODS: Standardized audit forms listing required activities were provided to course directors, who were then asked to indicate which of the domains were represented in each activity. Descriptive statistics were calculated. RESULTS: Of 1,500 activities, there was a mean of 2.13 domains per activity. Medical Knowledge was the most prevalent (44%), followed by Patient Care (20%), Interpersonal and Communication Skills (12%), Professionalism (9%), Systems-Based Practice (8%), and Practice-Based Learning and Improvement (7%). There was considerable variation by year and course. CONCLUSIONS: The domains provide a useful framework for organizing didactic components. Faculty can also consider activities in light of the domains, providing a vocabulary for instituting curricular change and innovation.

Standardizing and personalizing science in medical education.

In the century since the initial publication of the Flexner Report, medical education has emphasized a broad knowledge of science and a fundamental understanding of the scientific method, which medical educators believe are essential to the practice of medicine. The enormous growth of scientific knowledge that underlies clinical practice has challenged medical schools to accommodate this new information within the curricula. Although innovative educational modalities and new curricula have partly addressed this growth, the authors argue for a systematic restructuring of the content and structure of science education from the premedical setting through clinical practice. The overarching goal of science education is to provide students with a broad, solid foundation applicable to medicine, a deep understanding of the scientific method, and the attitudes and skills needed to apply new knowledge to patient care throughout their careers. The authors believe that to accomplish this successfully, the following changes must occur across the three major stages of medical education: (1) a reshaping of the scientific preparation that all students complete before medical school, (2) an increase in individualized science education during medical school, and (3) an emphasis on knowledge acquisition skills throughout graduate medical education and beyond to assure lifelong scientific learning. As students progress through the educational continuum, the balance of standardized and personalized scientific knowledge will shift toward personalization. Greater personalization demands that physicians possess well-refined skills in information acquisition, interpretation, and application for optimal lifelong learning and effective clinical practice.

Relationship between clinical assessment and examination scores in determining clerkship grade.

CONTEXT: Grades in clinical clerkships are typically based on a combination of clinical assessments from teachers, as well as results of more reliable (but perhaps less valid) scores on standardised tests of knowledge. It is not clear how these scores are combined in practice to yield a final summative grade. METHODS: Our subjects were 83 students who rotated through five clinical clerkships during a single year. After computing univariate correlations between clinical assessment scores and standardised examination scores, we performed logistic regression analyses for each clerkship to predict the final grade from these two variables. We compared actual grade with predicted grade under various hypothetical policies for combining these two variables. Finally, we assessed whether some students would systematically benefit from these policies. RESULTS: Clerkships varied in their univariate correlations between scores on clinical assessments and scores on standardised examinations. Clerkships with the lowest correlations tended to give more weight to standardised examination scores. Grading committees adjusted a substantial minority of grades to account for factors that were not reflected in either score. There did not appear to be a systematic bias in grading committee effect across the five clerkships. CONCLUSIONS: These results suggest a number of testable hypotheses about the cognitive processes by which evaluators combine various pieces of information to yield a summative performance score.

Patterns of communication during the preanesthesia visit.

BACKGROUND: Effective communication in the preanesthesia clinic is important in patient-centered care. Although patient-physician communication has been studied by recordings in other contexts, there have been no observational studies of the communication patterns of anesthesiologists and patients during the preanesthesia interview. METHODS: Two experienced standardized patients were trained to portray the same clinical situation by using different coping styles (maximizing information or "monitoring" vs. minimizing information or "blunting"). Interviews of standardized patients by anesthesiologists took place in the preanesthesia clinic and recorded with the knowledge of the subjects. Audio recordings were analyzed, and the visit was separated into nine components. Discussion of the risks/informed consent process was examined, looking for discussion of common morbidities. The standardized patients completed a survey on the patient-centeredness of the interview. RESULTS: Twenty-seven subjects participated in this study. Interviews with the monitor required more time: 17.4 min (confidence interval [CI] 15.2-19.6, n = 24) versus 14.5 min (CI 13.1-16.0, n = 25), P < 0.05. Most interview time was spent in obtaining the history; 2.4 min (CI 1.8-3.1) was spent discussing risks with the monitor, and only 1.6 min (CI 1.2-2.0) was spent with the blunter (P < 0.05). Neither the monitor nor the blunter scored the interview highly for involving the patient in determining the goals of the anesthetic and recovery. CONCLUSIONS: Direct recording of interactions with standardized patients is a feasible method of studying the communication skills of anesthesiologists. For this study, the anesthesia providers were able to modify their approach depending on patient type, but the monitor received more information.

Association between hand-off patients and subject exam performance in medicine clerkship students.

BACKGROUND: Teaching hospitals increasingly rely on transfers of patient care to another physician (hand-offs) to comply with duty hour restrictions. Little is known about the impact of hand-offs on medical students. OBJECTIVE: To evaluate the impact of hand-offs on the types of patients students see and the association with their subsequent Medicine Subject Exam performance. DESIGN: Observational study over 1 year. PARTICIPANTS: Third-year medical students in an Inpatient Medicine Clerkship at five hospitals with night float systems. PRIMARY OUTCOME: Medicine Subject Exam at the end of the clerkship; explanatory variables: number of fresh (without prior evaluation) and hand-off patients, diagnoses, subspecialty patients, and full evaluations performed during the clerkship, and United Stated Medical Licensing Examination (USMLE) Step I scores. MAIN RESULTS: Of the 2,288 patients followed by 89 students, 990 (43.3%) were hand-offs. In a linear regression model, the only variables significantly associated with students' Subject Exam percentile rankings were USMLE Step I scores (B = 0.26, P < 0.001) and the number of full evaluations completed on fresh patients (B =0.20, P = 0.048; model r (2) = 0.58). In other words, for each additional fresh patient evaluated, Subject Exam percentile rankings increased 0.2 points. For students in the highest quartile of Subject Exam percentile rankings, only Step I scores showed a significant association (B = 0.22, P = 0.002; r (2) = 0.5). For students in the lowest quartile, only fresh patient evaluations demonstrated a significant association (B = 0.27, P = 0.03; r (2) = 0.34). CONCLUSIONS: Hand-offs constitute a substantial portion of students' patients and may have less educational value than "fresh" patients, especially for lower performing students.

Social network analysis as a method of assessing institutional culture: three case studies.

PURPOSE: To describe the basic concepts of social network analysis (SNA), which assesses the unique structure of interrelationships among individuals and programs, and introduce some applications of this technique in assessing aspects of institutional culture at a medical center. METHOD: The authors applied SNA to three settings at their institution: team function in the intensive care unit, interdisciplinary composition of advisory committees for 53 federal career development awardees, and relationships between key function directors at an institution-wide Clinical Translational Sciences Institute (CTSI). (Key functions are the major administrative units of the CTSI.) RESULTS: In the ICU setting, SNA provides interpretable summaries of aspects of clinical team functioning. When applied to membership on mentorship committees, it allows for summary descriptions of the degree of interdisciplinarity of various clinical departments. Finally, when applied to relationships among leaders of an institution-wide research enterprise, it highlights potential problem areas in relationships among academic departments. In all cases, data collection is relatively rapid and simple, thereby allowing for the possibility of frequent repeated analyses over time. CONCLUSIONS: SNA provides a useful and standardized set of tools for measuring important aspects of team function, interdisciplinarity, and organizational culture that may otherwise be difficult to measure in an objective way.

Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review.

PURPOSE: To evaluate published evidence that the Accreditation Council for Graduate Medical Education's six general competencies can each be measured in a valid and reliable way. METHOD: In March 2008, the authors conducted searches of Medline and ERIC using combinations of search terms "ACGME," "Accreditation Council for Graduate Medical Education," "core competencies," "general competencies," and the specific competencies "systems-based practice" (SBP) and "practice based learning and improvement (PBLI)." Included were all publications presenting new qualitative or quantitative data about specific assessment modalities related to the general competencies since 1999; opinion pieces, review articles, and reports of consensus conferences were excluded. The search yielded 127 articles, of which 56 met inclusion criteria. Articles were subdivided into four categories: (1) quantitative/psychometric evaluations, (2) preliminary studies, (3) studies of SBP and PBLI, and (4) surveys. RESULTS: Quantitative/psychometric studies of evaluation tools failed to develop measures reflecting the six competencies in a reliable or valid way. Few preliminary studies led to published quantitative data regarding reliability or validity. Only two published surveys met quality criteria. Studies of SBP and PBLI generally operationalized these competencies as properties of systems, not of individual trainees. CONCLUSIONS: The peer-reviewed literature provides no evidence that current measurement tools can assess the competencies independently of one another. Because further efforts are unlikely to be successful, the authors recommend using the competencies to guide and coordinate specific evaluation efforts, rather than attempting to develop instruments to measure the competencies directly.

The effect of nonteaching services on the distribution of inpatient cases for internal medicine residents.

PURPOSE: Nonteaching services (NTSs) are becoming increasingly prevalent in academic hospitals. This study was designed to determine whether the presence of an NTS is associated with higher acuity and altered case mix on the teaching service. METHOD: The authors carried out a retrospective, cross-sectional analysis of data about all general medical admissions between January 1, 2005 and June 30, 2005 to either of two teaching hospitals in Rochester, New York. A total of 6,907 inpatients were studied, of whom 1,976 (29%) were admitted to medicine resident services and 4,931 (71%) were admitted to NTSs. Hospital billing databases were used to determine patient demographics, ICD-9 diagnoses, Charlson Comorbidity Index scores, and patient disposition. RESULTS: Compared with NTS patients, patients on resident services had higher median Charlson Comorbidity Index scores (3.0 versus 2.0, P < .001) and numbers of comorbidities (9.0 versus 8.0, P < .001) and were more likely to require intensive care (15.5% versus 7.6%, P < .001) and to die in the hospital (8.2% versus 4.5%, P < .001). Patients on the resident services were more likely to have acute renal failure, respiratory failure, septicemia, and HIV. Residents were less likely to care for patients with primary diagnoses of chest pain, cellulitis, alcohol withdrawal, and sickle cell crisis. The differences in patients' conditions between resident services and NTSs were similar in the two hospitals and also among patients who had not received intensive care. CONCLUSIONS: Patients on resident services may be more medically complex and more likely to have high-acuity diagnoses than patients on NTSs. How these differences affect residents' education, residents' career decisions, and practice styles deserves further study.

Further challenges in measuring communication skills: accounting for actor effects in standardised patient assessments.

CONTEXT: Subjective rating scales for communication skills may yield more personally meaningful responses than more standardised rating schemes. It is unclear, however, whether such evaluations may be overly biased by respondents' rating styles, which may lead to unreliable measurement of examinees' communication skills. METHODS: Our study involved 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students were rated by actors depicting standardised patients (SPs) on the same seven cases using the 19-item Rochester Communication Rating Scale (RCRS). Different students were assigned to different actors playing the same SP. We assessed the extent to which actors' personal rating styles influenced the scores they assigned to students. Main outcome measures were: between-actor variability in responses; the degree to which actors' response styles contribute to overall scores, and improvements in reliability achieved by standardising actors' ratings. RESULTS: There were statistically significant differences between actors in their mean assigned scores. Scores aggregated over 18 separate SP cases have an expected generalisability coefficient of 0.79. If raw RCRS scores are used, a total of 27 replications of the RCRS are required to achieve a Cronbach's alpha of 0.8; standardisation reduces this number to 18. CONCLUSIONS: Although actors are variable in their use of a standardised subjective scale of communication, such differences contribute to an acceptably small proportion of the total variance if scores are combined across a large number of cases. Reliability can be markedly improved by standardising scores across raters.

Changes in self-perceived abilities among male and female medical students after the first year of clinical training.

BACKGROUND: It is known that male and female medical students have different experiences in their clinical training. AIMS: To assess whether male and female medical students change in their self-rated work habits and interpersonal habits during the first year of clinical training. METHOD: Longitudinal study of self- and peer-assessment among 224 medical students in 3 consecutive classes at a private US medical school. Students rated themselves on global work habits (WH) and interpersonal attributes (IA). Students also rated and were rated by 6-12 peers on the same scale. RESULTS: In the second year of medical school, there were no differences between men and women in quartiles of self-assessed WH or IA. At the end of the third year, however, women were more likely to be in the lower quartiles of self-assessed WH (X(2) = 6.77; p = 0.03), as well as the highest quartiles of self-assessed IA (X(2) = 11.36; p = 0.003). In both years, women rated their own WH skills significantly lower than they rated their peers, while men rated themselves similarly to peers. There were no sex differences in self-assessed IA. CONCLUSIONS: Although second-year male and female medical students appear similar to one another in terms of self-assessed WH and IA, by the end of the third year women rate themselves relatively lower in WH, while men rate themselves relatively lower in IA.

Relationship between dean's letter rankings and later evaluations by residency program directors.

BACKGROUND: It is not known how well dean's letter rankings predict later performance in residency. PURPOSE: To assess the accuracy of dean's letter rankings to predict clinical performance in internship. METHOD: Participants were medical students who graduated from the University of Rochester School of Medicine and Dentistry in the classes of 2003 and 2004. In their Dean's Letter, each student was ranked as either "Outstanding" (upper quartile), "Excellent" (second quartile), "Very good" (lower 2 quartiles), or "Good" (lowest few percentile). We compared these dean's letter rankings against results of questionnaires sent to program directors 9 months after graduation. RESULTS: Response rate to the questionnaire was 58.9% (109 of 185 eligible graduates). There were no differences in response rate across the four dean's letter ranking categories. Program directors rated students in the top two categories of dean's letter rankings significantly higher than those in the very good group. Students in all three groups were rated significantly higher than those in the good group, F (3, 105) = 13.37, p < .001. Students in the very good group were most variable in their ratings by program directors, with many receiving similarly high ratings as students in the upper 2 groups. There were no differences by gender or specialty. CONCLUSION: Dean's letter rankings are a significant predictor of later performance in internship among graduates of our medical school. Students in the bottom half of the class are most likely either to underperform or overperform in internship.

Relationship between peer assessment during medical school, dean's letter rankings, and ratings by internship directors.

BACKGROUND: It is not known to what extent the dean's letter (medical student performance evaluation [MSPE]) reflects peer-assessed work habits (WH) skills and/or interpersonal attributes (IA) of students. OBJECTIVE: To compare peer ratings of WH and IA of second- and third-year medical students with later MSPE rankings and ratings by internship program directors. DESIGN AND PARTICIPANTS: Participants were 281 medical students from the classes of 2004, 2005, and 2006 at a private medical school in the northeastern United States, who had participated in peer assessment exercises in the second and third years of medical school. For students from the class of 2004, we also compared peer assessment data against later evaluations obtained from internship program directors. RESULTS: Peer-assessed WH were predictive of later MSPE groups in both the second (F = 44.90, P < .001) and third years (F = 29.54, P < .001) of medical school. Interpersonal attributes were not related to MSPE rankings in either year. MSPE rankings for a majority of students were predictable from peer-assessed WH scores. Internship directors' ratings were significantly related to second- and third-year peer-assessed WH scores (r = .32 [P = .15] and r = .43 [P = .004]), respectively, but not to peer-assessed IA. CONCLUSIONS: Peer assessment of WH, as early as the second year of medical school, can predict later MSPE rankings and internship performance. Although peer-assessed IA can be measured reliably, they are unrelated to either outcome.

Seasonal affective disorder.

Patients with seasonal affective disorder have episodes of major depression that tend to recur during specific times of the year, usually in winter. Like major depression, seasonal affective disorder probably is underdiagnosed in primary care settings. Although several screening instruments are available, such screening is unlikely to lead to improved outcomes without personalized and detailed attention to individual symptoms. Physicians should be aware of comorbid factors that could signal a need for further assessment. Specifically, some emerging evidence suggests that seasonal affective disorder may be associated with alcoholism and attention-deficit/hyperactivity disorder. Seasonal affective disorder often can be treated with light therapy, which appears to have a low risk of adverse effects. Light therapy is more effective if administered in the morning. It remains unclear whether light is equivalent to drug therapy, whether drug therapy can augment the effects of light therapy, or whether cognitive behavior therapy is a better treatment choice.