Measuring the effects of an international health professions faculty development fellowship: the FAIMER Institute.

While there are many examples of evaluations of faculty development programs in resource rich countries, evaluation of transnational programs for faculty from developing countries is limited. We describe evaluation of the effects of the FAIMER Institute, an international health professions education fellowship that incorporates not only education content, but also leadership and management topics and, in addition, strives to develop a sustained community of educators. Data were obtained via retrospective pre/post surveys, as well as interviews. Results indicate that participating health professions faculty from developing countries are augmenting their knowledge and skills in education leadership, management, and methodology, and applying that knowledge at their home institutions. Fellows' perceptions of importance of, and their own competence in, all curriculum theme areas increased. Interviews confirmed a nearly universal gain of at least one leadership skill. Findings suggest that the high-engagement experience of the FAIMER model offering integration of education and leadership/management tools necessary to implement change, provides knowledge and skills which are useful across cultural and national contexts and results in the development of a supportive, global, professional network.

Capacity building in medical education and health outcomes in developing countries: the missing link.

BACKGROUND: Finding evidence for the link between capacity building in medical education and improved health outcomes in developing countries is an important challenge. We describe the Foundation for Advancement of International Medical Education and Research (FAIMER) Institute, a two year, part-time fellowship in medical education methodology and leadership and its evaluation as a model to bridge this gap by collecting quantitative and qualitative data on intermediary outcomes. METHODS: FAIMER has used the following framework of human capacity building programs: 1) identify young and talented individuals with potential to become agents for change; 2) organize and deliver an effective learning intervention that is relevant for the environment; 3) facilitate the opportunity for real-life application of acquired knowledge and skills with support; and 4) promote development of a sustainable career path with opportunities for growth and advancement. RESULTS: Twenty-three percent of curriculum innovation projects were directly related to community health. Of the 35 fellows in the first three classes of the Institute, there have been 11 promotions, 9 peer-reviewed publications and 14 international poster presentations, indicating development of the medical education field. Other qualitative and quantitative program evaluation data are presented. DISCUSSION: The link between capacity building in medical education and improved health can be demonstrated in several ways: align curriculum with local health needs, place learners in community clinical settings, teach basic healthcare workers, become involved in national policy development and develop the field of medical education. CONCLUSION: While experimental models may not be possible to evaluate the effect of capacity building, methods described may help support the connection between improved medical education and health.

The relationship between competence and performance: implications for assessing practice performance.

OBJECTIVE: This paper aims to describe current views of the relationship between competence and performance and to delineate some of the implications of the distinctions between the two areas for the purpose of assessing doctors in practice. METHODS: During a 2-day closed session, the authors, using their wide experiences in this domain, defined the problem and the context, discussed the content and set up a new model. This was developed further by e-mail correspondence over a 6-month period. RESULTS: Competency-based assessments were defined as measures of what doctors do in testing situations, while performance-based assessments were defined as measures of what doctors do in practice. The distinction between competency-based and performance-based methods leads to a three-stage model for assessing doctors in practice. The first component of the model proposed is a screening test that would identify doctors at risk. Practitioners who 'pass' the screen would move on to a continuous quality improvement process aimed at raising the general level of performance. Practitioners deemed to be at risk would undergo a more detailed assessment process focused on rigorous testing, with poor performers targeted for remediation or removal from practice. CONCLUSION: We propose a new model, designated the Cambridge Model, which extends and refines Miller's pyramid. It inverts his pyramid, focuses exclusively on the top two tiers, and identifies performance as a product of competence, the influences of the individual (e.g. health, relationships), and the influences of the system (e.g. facilities, practice time). The model provides a basis for understanding and designing assessments of practice performance.

Conceptual challenges in tailoring physician performance assessment to individual practice.

Practice inevitably narrows over time. Therefore, testing of established doctors requires that their assessment be tailored to a far narrower practice than is appropriate for testing of new doctors who have not yet differentiated. In this paper, we address the conceptual challenges of tailoring physician assessment to individual practice. Testing of established doctors needs to reflect that physicians specialise, often in idiosyncratic ways; otherwise, the testing will not be credible among established doctors and will not reflect the realities of their practice. Despite the importance of these goals, the conceptual and methodological challenges of creating tailored assessments remain daunting.

Facing the challenges of competency-based assessment of postgraduate dental training: Longitudinal Evaluation of Performance (LEP).

CONTEXT: The evaluation of competence in the health professions is of great importance to the public and professionals alike. Recent efforts to design dependable and accurate systems of assessment for demanding clinical roles are increasingly attempting to focus on all-round competence of practitioners. Many challenges are faced in this field as a balance between robust assessment methodology and feasibility in practice is crucial to implementation and adoption. OBJECTIVES: The authors discuss some of the challenges faced by educators and clinicians involved in the development of systems of assessment for the health professions, and describe a method which aims to address these issues in the assessment of postgraduate dental training in Scotland. DISCUSSION: Three of the major challenges facing educators and clinicians involved in the design of competency-based systems of assessment are considered: the requirement for evaluation in different areas of competence; the importance of association of assessment with the training objectives, and the types and focus of the assessment introduced. Issues around the use of formative and summative assessment, and the perception that these must always remain completely separate, are discussed in detail. SOLUTIONS: A proposal is made for the introduction of a method of assessment which has been designed keeping these challenges in mind. The rationale behind this assessment method is described.

Setting performance standards for medical practice: a theoretical framework.

BACKGROUND: The assessment of performance in the real world of medical practice is now widely accepted as the goal of assessment at the postgraduate level. This is largely a validity issue, as it is recognised that tests of knowledge and in clinical simulations cannot on their own really measure how medical practitioners function in the broader health care system. However, the development of standards for performance-based assessment is not as well understood as in competency assessment, where simulations can more readily reflect narrower issues of knowledge and skills. This paper proposes a theoretical framework for the development of standards that reflect the more complex world in which experienced medical practitioners work. METHODS: The paper reflects the combined experiences of a group of education researchers and the results of literature searches that included identifying current health system data sources that might contribute information to the measurement of standards. CONCLUSION: Standards that reflect the complexity of medical practice may best be developed through an "expert systems" analysis of clinical conditions for which desired health care outcomes reflect the contribution of several health professionals within a complex, three-dimensional, contextual model. Examples of the model are provided, but further work is needed to test validity and measurability.

Certification and specialization: do they matter in the outcome of acute myocardial infarction?

PURPOSE: To learn whether there are differences among certified and self-designated cardiologists, internists, and family practitioners in terms of the mortality of their patients with acute myocardial infarction (AMI). METHOD: Data on all patients admitted with AMI were collected for calendar year 1993 by the Pennsylvania Health Care Cost Containment Council and analyzed. Certified and self-designated family practitioners, internists, and cardiologists (n = 4,546) were compared with respect to the characteristics of their patients' illnesses. In addition, a regression model was fitted in which mortality was the dependent measure and the independent variables were the probability of death, hospital characteristics (location and the availability of advanced cardiac care), and physician characteristics (patient volume, years since graduation from medical school, specialty, and certification status). RESULTS: On average, cardiologists treated more patients than did generalists, and their patients were less severely ill. In the regression analysis, all variables were statistically significant except the availability of advanced cardiac care. Holding all other variables constant, treatment by a certified physician was associated with a 15% reduction in mortality among patients with AMI. CONCLUSIONS: Less patient mortality was associated with treatment by physicians who were cardiologists, cared for larger numbers of AMI patients, were closer to their graduation from medical school, and were certified.

Clinical teaching: maintaining an educational role for doctors in the new health care environment.

CONTEXT AND OBJECTIVES: Good clinical teaching is central to medical education but there is concern about maintaining this in contemporary, pressured health care environments. This paper aims to demonstrate that good clinical practice is at the heart of good clinical teaching. METHODS: Seven roles are used as a framework for analysing good clinical teaching. The roles are medical expert, communicator, collaborator, manager, advocate, scholar and professional. RESULTS: The analysis of clinical teaching and clinical practice demonstrates that they are closely linked. As experts, clinical teachers are involved in research, information retrieval and sharing of knowledge or teaching. Good communication with trainees, patients and colleagues defines teaching excellence. Clinicians can 'teach' collaboration by acting as role models and by encouraging learners to understand the responsibilities of other health professionals. As managers, clinicians can apply their skills to the effective management of learning resources. Similarly skills as advocates at the individual, community and population level can be passed on in educational encounters. The clinicians' responsibilities as scholars are most readily applied to teaching activities. Clinicians have clear roles in taking scholarly approaches to their practice and demonstrating them to others. CONCLUSION: Good clinical teaching is concerned with providing role models for good practice, making good practice visible and explaining it to trainees. This is the very basis of clinicians as professionals, the seventh role, and should be the foundation for the further development of clinicians as excellent clinical teachers.

A review of the evaluation of clinical teaching: new perspectives and challenges.

PURPOSE: This article discusses the importance of the process of evaluation of clinical teaching for the individual teacher and for the programme. Measurement principles, including validity, reliability, efficiency and feasibility, and methods to evaluate clinical teaching are reviewed. CONTEXT: Evaluation is usually carried out from the perspective of the learner. This article broadens the evaluation to include the perspectives of the teacher, the patient and the institutional administrators and payers in the health care system and recommends evaluation strategies. RESULTS: Each perspective provides specific feedback on factors or attributes of the clinical teacher's performance in the domains of medical expert, professional, scholar, communicator, collaborator, patient advocate and manager. Teachers should be evaluated in all domains relevant to their teaching objectives; these include knowledge, clinical competence, teaching effectiveness and professional attributes. CONCLUSIONS AND IMPLICATIONS: Using this model of evaluation, a connection can be made between teaching and learning about all the expected roles of a physician. This can form the basis for systematic investigation into the relationship between the quality of teaching and the desired outcomes, the improvement of student learning and the achievement of better health care practice. It is suggested that the extent of effort and resources devoted to evaluation should be commensurate with the value assigned to the evaluation process and its outcomes.

The first decade of the American Board of Internal Medicine certification in critical care medicine: an overview of examinees and certificate holders from 1987 through 1996.

OVERVIEW: This study reviews the first decade of critical care medicine (CCM) certification by the American Board of Internal Medicine (1987-1996). Included are the characteristics of examinee and certificate-holder groups; examination performances from different underlying disciplines of internal medicine, with or without formal CCM training; and the influence of background and a training program as correlates of examination performance. DATA SOURCES: The CCM certification examination has been offered biennially since November 1987. Performance data on the American Board of Internal Medicine examinations in internal medicine and its subspecialties and added qualifications were available for candidates taking the CCM examinations. For examinees with formal CCM training, residency program director ratings, and information regarding the program characteristics of size and percentage of United States and Canadian medical graduates were also available. STUDY SELECTION: All examinees who ever attempted certification were included in this study. The study cohort for each of the five examination administrations consists of all first-time takers. CONCLUSIONS: Cohort sizes have decreased since formal training became an admission requirement in 1993. Percentages of International Medical Graduates and women attempting and achieving certification have increased steadily. Examination performance was positively associated with formal training, internal medicine examination performance, recent medical training, and pulmonary disease certification. For those with formal training, performance was also positively associated with training program director ratings of overall clinical competence and completion of a training program with a higher proportion of United States and Canadian medical graduates.

Standards and reliability in evaluation: when rules of thumb don't apply.

The purpose of this paper is to identify situations in which two rules of thumb in evaluation do not apply. The first rule is that all standards should be absolute. When selection decisions are being made or when classroom tests are given, however, relative standards may be better. The second rule of thumb is that every test should have a reliability of .80 or better. Depending on the circumstances, though, the standard error of measurement, the consistency of pass/fail classifications, and the domain-referenced reliability coefficients may be better indicators of reproducibility.

Association between licensing examination scores and resource use and quality of care in primary care practice.

CONTEXT: Clinical competence is a determinant of the quality of care delivered, and may be associated with use of health care resources by primary care physicians. Clinical competence is assumed to be assessed by licensing examinations, yet there is a paucity of information on whether scores achieved predict subsequent practice. OBJECTIVE: To determine if licensing examination scores were associated with selected aspects of quality of care and resource use in initial primary care practice. DESIGN: Prospective cohort study of recently licensed family physicians, followed up for the first 18 months of practice. SETTING: The Quebec health care system. PARTICIPANTS: A total of 614 family physicians who passed the licensing examination between 1991 and 1993 and entered fee-for-service practice in Quebec. MAIN OUTCOME MEASURES: All patients seen by physicians were identified by the universal health insurance board and all health services provided to these patients were retrieved for the 18 months prior to (baseline) and after (follow-up) the physicians' entry into practice. Medical service and prescription claims files were used to measure rates of resource use (specialty consultation, symptom-relief prescribing compared with disease-specific prescribing) and quality of care (inappropriate prescribing, mammography screening). Baseline data were used to adjust for differences in practice population. RESULTS: Study physicians saw a total of 1116389 patients, of whom 113535 (10.2%) were elderly and 83391 (7.5%) were women aged 50 to 69 years. Physicians with higher licensing examination scores referred more of their patients for consultation (3.8/1000 patients per SD increase in score; 95% confidence interval [CI], 1.2-7.0; P = .005), prescribed to elderly patients fewer inappropriate medications (-2.7/1000 patients per SD increase in score; 95% CI, -4.8 to -0.7; P=.009) and more disease-specific medications relative to symptom-relief medications (3.9/1000 patients per SD increase in score; 95% CI, 0.3 to 7.4; P= .03), and referred more women aged 50 to 69 years (6.6/1000 patients per SD increase in score; 95% CI, 1.2-11.9; P = .02) for mammography screening. If patients of physicians with the lowest scores had experienced the same rates of consultation, prescribing, and screening as patients of physicians with the highest scores, an additional 3027 patients would have been referred, 179 fewer elderly patients would have been prescribed symptom-relief medication, 912 more elderly patients would have been prescribed disease-specific medication, 189 fewer patients would have received inappropriate medication, and 121 more women would have received mammography screening. CONCLUSIONS: Licensing examination scores are significant predictors of consultation, prescribing, and mammography screening rates in initial primary care practice.

Examiner differences in the mini-CEX.

OBJECTIVE: The objective of this study was to analyze whether faculty ratings of residents, using the mini-CEX oral exam format, differed in stringency or were influenced by the clinical setting. It also sought to learn whether the examiners were satisfied with the format. METHOD: A mini-CEX encounter consisted of a single faculty member observing a resident conduct a focused history and physical examination in an inpatient, outpatient, or emergency room setting. After asking the resident for a diagnosis and treatment plan, the faculty member rated the resident and provided educational feedback. The encounters were intended to be short and occur as a routine part of the training, so each resident would be evaluated on many occasions by different faculty. SAMPLE: Sixty-four attending physicians evaluated residents from five internal medicine training programs; data were analyzed for 355 mini-CEX encounters involving 88 residents. RESULTS: There were not large differences among the examiners in their ratings. Moreover, there were not great differences among the ratings in terms of the training program with which the examiner was associated, the setting of the mini-CEX, or the nature of the patient. The examiners were generally satisfied with the format and their level of satisfaction was correlated with the residents' perceptions of the format. CONCLUSION: The mini-CEX adapts itself to a broad range of clinical situations, and these results show that it should produce roughly comparable scores over examiners and settings. This makes it a worthwhile device for evaluation at the local level.