Content Analysis of Family Medicine Resident Peer Observations.

BACKGROUND AND OBJECTIVES: Direct observation is a critical part of assessing learners' achievement of the Accreditation Council for Graduate Medical Education (ACGME) Milestones and subcompetencies. Little research exists identifying the content of peer feedback among residents; this study explored the content of residents' peer assessments as they relate to ACGME Milestone subcompetencies in a family medicine residency program. METHODS: Using content from a mobile app-based observation tool (M3App), we examined resident peer observations recorded between June 2014 and November 2017, tabulating frequency of observation for each ACGME subcompetency and calculating the proportion of observations categorized under each subcompetency, as well as for each postgraduate year (PGY) class. We also coded each observation on three separate dimensions: "positive," "constructive," and "actionable." We used the χ2 test for independence, and estimated odds ratios and 95% confidence intervals for two-by-two comparisons to compare numbers of observations within each category. RESULTS: Our data include 886 peer observations made by 54 individual residents. The most frequently observed competencies were in patient care, communication, and professionalism (56%, 47%, and 38% of observations, respectively). Practice-based learning and improvement was observed least frequently (16% of observations). On average, 97.25% of the observations were positive, 85% were actionable, and 6% were constructive. CONCLUSIONS: When asked to review their peers, residents provide comments that are primarily positive and actionable. In addition, residents tend to provide more feedback on certain subcompetencies compared to others, suggesting that programs may rely on peer feedback for specific subcompetencies. Peers can provide perspective on the behaviors and skills of fellow residents.

Residency Faculty Teaching Evaluation: What Do Faculty, Residents, and Program Directors Want?

BACKGROUND AND OBJECTIVES: The Accreditation Council for Graduate Medical Education Common Residency Program Requirements stipulate that each faculty member's performance be evaluated annually. Feedback is essential to this process, yet the culture of medicine poses challenges to developing effective feedback systems. The current study explores existing and ideal characteristics of faculty teaching evaluation systems from the perspectives of key stakeholders: faculty, residents, and residency program directors (PDs). METHODS: We utilized two qualitative approaches: (1) confidential semistructured telephone interviews with PDs from a convenience sample of eight family medicine residency programs, (2) qualitative responses from an anonymous online survey of faculty and residents in the same eight programs. We used inductive thematic analysis to analyze the interviews and survey responses. Data collection occurred in the fall of 2017. RESULTS: All eight (100%) of the PDs completed interviews. Survey response rates for faculty and residents were 79% (99/126) and 70% (152/216), respectively. Both PD and faculty responses identified a desire for actionable, real-time, frequent feedback used to foster continued professional development. Themes unique to faculty included easy accessibility and feedback from peers. Residents expressed an interest in in-person feedback and a process minimizing potential retribution. Residents indicated that feedback should be based on shared understanding of what skill(s) the faculty member is trying to address. CONCLUSIONS: PDs, faculty, and residents share a desire to provide faculty with meaningful, specific, and real-time feedback. Programs should strive to provide a culture in which feedback is an integral part of the learning process for both residents and faculty.

The Impact of a Community-Based Serious Illness Care Program on Healthcare Utilization and Patient Care Experience.

OBJECTIVE: Healthcare organizations are expanding community-based serious illness care programs to deliver care for homebound patients. Programs typically focus on home-based primary care or home-based palliative care, yet this population may require both services. We developed and evaluated a primary and palliative care program serving seriously ill older adults, called the Reaching Out to Enhance the Health of Adults in Their Communities and Homes (REACH) program. Our objective was to determine the impact of the REACH program on healthcare utilization and the patient care experience. DESIGN: Case study using medical record review and telephone interviews. SETTING: Home-based serious illness care program in central North Carolina. PARTICIPANTS: Patients enrolled in the REACH program from August 2014 to March 2016 (n = 159). INTERVENTION: Home-based primary and palliative care delivered by an interdisciplinary team for seriously ill older adults. MEASUREMENTS: Structured medical record review including demographics, health and functional status, and acute hospitalization and emergency department (ED) visits in the 12 months preceding and following initiation of the REACH program. A convenience sample of participants completed telephone interviews that measured the quality-of-care experience and quality of communication. RESULTS: REACH patients had a 43% reduction in hospital visits and a 25% reduction in ED visits. Participants in telephone interviews reported a high-quality care experience and very good quality of communication with REACH providers. CONCLUSION: A care model that integrates home-based primary care and palliative care has the potential to reduce health service utilization and enhance the care experience for older patients with serious illness. J Am Geriatr Soc 67:825-830, 2019.

Auditory Narrative Does Not Improve Learning when Added to Visual Computer-Based Learning Modules.

BACKGROUND: Computer-based learning (CBL) is considered by many to be an effective means of education. However, features of computer-based modules purported to contribute to learning have not been studied with medical student education. OBJECTIVE: This study was designed to evaluate the effect of auditory supplements in a computer-based instructional module on learning and knowledge retention. DESIGN: A prospective, randomized controlled trial comparing the effectiveness of two types of web-based instructional presentations used to teach key aspects of systems-based practice to fourth-year medical students. INTERVENTION: The intervention and control group each received a computer-based module comprised of the same mix of visual and written material, but the intervention group also received an auditory narration of the materials. MAIN MEASURES: The primary outcome measures were the difference in the students' scores between the two modules using an online 8-item knowledge test completed immediately after the first exposure to the module and again 1 to 7 months later. Students were also asked whether they considered themselves auditory learners. Learning efficiency (the amount of learning per unit time) was calculated for each student and the Mann-Whitney U test was used to compare scores between the two groups. RESULTS: One hundred thirty fourth-year medical students were randomized by a computer program to one of the two modules. All students completed the first knowledge test and 86 (66%) students completed the second test. Test scores did not differ significantly between the two groups in either the first or the second test. Learning efficiency was lower in the intervention group. Self-identification as auditory learners had no effect on performance. CONCLUSIONS: The addition of narration to a computer-based instructional module did not improve learning or knowledge retention even in students who self-identified as auditory learners and resulted in overall lower learning efficiency.

Variation in Triple Aim Measures: Implications of Clinical Signatures in Family Medicine Residency Programs.

BACKGROUND: Evidence from several specialties suggests that practice patterns developed in residency influence the quality and cost of care long after completion of training. Improving the quality, cost, and patient experience of care (the "Triple Aim") is foundational to future health systems change. OBJECTIVE: We measured variation in Triple Aim measures among family medicine residency programs in a regional quality improvement collaborative (I3 Population Health Collaborative). METHODS: We calculated medians and interquartile ranges for each of 11 Triple Aim measures and compared them with median splits of population and practice characteristics, including payer mix, patient race and age, electronic health record used, registry use, and National Committee for Quality Assurance patient-centered medical home recognition. RESULTS: All 22 participating family medicine residency programs provided baseline data. The number of practices reporting data on individual measures ranged from 9 to 17 (41%-77%). We found variation averaging 51% across all measures, from a low of 12% for readmission rates to 94% for emergency department visit rates. Variations were stable over time. We found no significant relationships between practice or population characteristics and measures, nor between practice characteristics and outcomes variation. CONCLUSIONS: The 22 family medicine residency programs in our study showed substantial variation in quality, cost, and patient experience of care. These variations did not appear to result from differences in practice characteristics, payer mix, or patient demographics.

Implementing Lean in Academic Primary Care.

BACKGROUND: Lean is emerging as a quality improvement (QI) strategy in health care, but there has been minimal adoption in primary care teaching practices. This study describes a strategy for implementing Lean in an academic family medicine center and provides a formative assessment of this approach. METHODS: A case study of the University of North Carolina Family Medicine Center that used the Consolidated Framework for Implementation Research to guide a formative evaluation. The implementation strategy included partnering with Lean content experts and creating a leadership team; planning and completing QI events and Lean training modules; and evaluating and reporting activities related to QI and training. RESULTS: During the initial period of Lean implementation, there was (1) minimal to no change in the quality of care as determined by the Preventive Care Index (46-48); (2) a decrease patient appointment cycle time from 89 minutes to 65 minutes; (3) an increase in overall practice productivity from $8144 to $9160; (4) a decrease in patient satisfaction from 94% to 91%; and (5) an increase in monthly visit volume from 4112 to 5076. CONCLUSION: Lean had an uneven effect on QI in an academic primary care practice during the first year of implementation.

Implementation and evaluation of a pharmacist-led electronic visit program for diabetes and anticoagulation care in a patient-centered medical home.

PURPOSE: Results of a study evaluating quality-of-care, financial, and patient satisfaction outcomes of pharmacist-conducted telehealth visits for diabetes management and warfarin monitoring are reported. METHODS: A retrospective pre-post study was conducted to determine the impact of an electronic visit (e-visit) program targeting 2 groups of outpatients: adults with uncontrolled diabetes and warfarin-treated adults performing patient self-testing (PST) for monitoring of International Normalized Ratio (INR) values. RESULTS: A total of 36 patients participated in the e-visit program during the 2-year study period. Among warfarin-treated patients, the percentage of INR values in the desired range increased relative to preenrollment values (from 62.5% to 72.7%, p = 0.07), and the frequency of extreme INR values (values of <1.5 or >5.0) decreased (from 4.8% to 0.01%, p = 0.01); the margin per patient was $300 during the first year and $191 annually thereafter. In the diabetes group, a decrease from baseline in glycosylated hemoglobin values of 3.4 percentage points was observed at 5.7 months after enrollment (p < 0.001), with significant improvements in frequencies of statin use, aspirin use, and blood pressure control; the margin was $100 per patient. The overall median patient satisfaction survey score was 39 of 40. CONCLUSION: An online e-visit model for warfarin monitoring was an efficient, safe, and cost-effective method for implementing PST. Pharmacist-led management of diabetes through e-visits, often in combination with in-person visits, generated revenue while significantly improving clinical outcomes.

Triple Aim Is Triply Tough: Can You Focus on Three Things at Once?

BACKGROUND AND OBJECTIVES: The I3 POP Collaborative sought to improve health of patients attending North Carolina, South Carolina, and Virginia primary care teaching practices using the triple aim framework of better quality, appropriate utilization, and enhanced patient experience. We examined change in triple aim measures over 3 years, and identified correlates of improvement. METHODS: Twenty-nine teaching practices representing 23 residency programs participated. The Institute for Health Care Improvement Breakthrough Series Collaborative model was tailored to focus on at least one triple aim goal and programs submitted data annually on all collaborative measures. Outcome measures included quality (chronic illness, prevention); utilization (hospitalization, emergency department visits, referrals) and patient experience (access, continuity). Participant interviews explored supports and barriers to improvement. RESULTS: Six of 29 practices (21%) were unable to extract measures from their electronic health records (EHR). All of the remaining 23 practices reported improvement in at least one measure, with 11 seeing at least 10% improvement; seven (24%) improved measures in all three triple aim areas, with two experiencing at least 10% improvement. Practices with a greater number of patient visits were more likely to show improved measures (odds ratio [OR] 10.8, 95% confidence interval [CI]: .68-172.2, P=0.03). Practice interviews revealed that engaged leadership and systems supports were more common in higher performing practices. CONCLUSIONS: Simultaneous attainment of improvement in all three triple aim goals by teaching practices is difficult. I3 POP practices that were able to pull and report data improved on at least one measure. Future work needs to focus on cultivating leadership and systems supporting large scale improvement.

Electronic Health Record Effects on Work-Life Balance and Burnout Within the I3 Population Collaborative.

BACKGROUND: Physician burnout is a problem that often is attributed to the use of the electronic health record (EHR). OBJECTIVE: To estimate the prevalence of burnout and work-life balance satisfaction in primary care residents and teaching physicians, and to examine the relationship between these outcomes, EHR use, and other practice and individual factors. METHODS: Residents and faculty in 19 primary care programs were anonymously surveyed about burnout, work-life balance satisfaction, and EHR use. Additional items included practice size, specialty, EHR characteristics, and demographics. A logistic regression model identified independent factors associated with burnout and work-life balance satisfaction. RESULTS: In total, 585 of 866 surveys (68%) were completed, and 216 (37%) respondents indicated 1 or more symptoms of burnout, with 162 (75%) attributing burnout to the EHR. A total of 310 of 585 (53%) reported dissatisfaction with work-life balance, and 497 (85%) indicated that use of the EHR affected their work-life balance. Respondents who spent more than 6 hours weekly after hours in EHR work were 2.9 times (95% confidence interval [CI] 1.9-4.4) more likely to report burnout and 3.9 times (95% CI 1.9-8.2) more likely to attribute burnout to the EHR. They were 0.33 times (95% CI 0.22-0.49) as likely to report work-life balance satisfaction, and 3.7 times (95% CI 2.1-6.7) more likely to attribute their work-life balance satisfaction to the EHR. CONCLUSIONS: More after-hours time spent on the EHR was associated with burnout and less work-life satisfaction in primary care residents and faculty.

What's the Right Referral Rate? Specialty Referral Patterns and Curricula Across I3 Collaborative Primary Care Residencies.

BACKGROUND AND OBJECTIVES: Specialty physician visits account for a significant portion of ambulatory visits nationally, contribute significantly to cost of care, and are increasing over the past decade. Marked variability in referral rates exists among primary care practices without obvious causality. We present data describing the referral process and specialty referral curriculum within the I3 collaborative. METHODS: Residency directors were surveyed about residency characteristics related to referrals. Specialty physician referral rates were obtained from each program and then correlated to program characteristics referral rates in four domains: presence and type of referral curriculum, process of referral review, faculty preceptor characteristics, and use of referral data for administrative processes. RESULTS: The survey response rate was 87%; 10 programs submitted complete referral data. Three programs (23%) reported a formal curriculum addressing the process of making a referral, and four programs (31%) reported a curriculum on appropriateness of subspecialty referrals. Specialty referral rates varied from 7%-31% of active residency patients, with no relationship to age, payor status, or race. DISCUSSION: Marked variability in referral rates and patterns exist within primary care residency training programs. Specialty referral practices are a key driver of total cost of care yet few curricula exist that address appropriateness, quantity, or process of specialty referrals. Practice patterns often develop during residency training, therefore an opportunity exists to improve training and practice around referrals.

Piloting the Mobile Medical Milestones Application (M3App©): A Multi-Institution Evaluation.

BACKGROUND AND OBJECTIVES: Competency-based evaluation of the Accreditation Council for Graduate Medical Education (ACGME) Milestones requires the development of new evaluation tools that can better capture learners' behavior. This study describes the implementation and initial assessment of an innovative point-of-care mobile application, the M3App,© linked to the Family Medicine Milestones. METHODS: Seven family medicine residency programs in North Carolina implemented the M3App.© Program faculty and residents were surveyed prior to implementation regarding current evaluation methods and their quality and use and acceptability of electronic evaluation tools. Surveys were repeated after implementation for comparison. RESULTS: All seven programs successfully implemented the M3App. Most faculty members found the tool well designed, easy to use, beneficial to the quality and efficiency of feedback they provide, and to their knowledge of Milestones. Residents reported significant increases in the volume and quality of written feedback they receive. CONCLUSIONS: The M3App provides an efficient, convenient tool for assessing Milestones that can improve the quantity and quality of feedback residents receive from faculty. Improved faculty perception of knowledge of Milestones after M3App implementation suggests that the tool is also effective for faculty development.

Learnings From the Pilot Implementation of Mobile Medical Milestones Application.

BACKGROUND: Implementation of the educational milestones benefits from mobile technology that facilitates ready assessments in the clinical environment. We developed a point-of-care resident evaluation tool, the Mobile Medical Milestones Application (M3App), and piloted it in 8 North Carolina family medicine residency programs. OBJECTIVE: We sought to examine variations we found in the use of the tool across programs and explored the experiences of program directors, faculty, and residents to better understand the perceived benefits and challenges of implementing the new tool. METHODS: Residents and faculty completed presurveys and postsurveys about the tool and the evaluation process in their program. Program directors were interviewed individually. Interviews and open-ended survey responses were analyzed and coded using the constant comparative method, and responses were tabulated under themes. RESULTS: Common perceptions included increased data collection, enhanced efficiency, and increased perceived quality of the information gathered with the M3App. Residents appreciated the timely, high-quality feedback they received. Faculty reported becoming more comfortable with the tool over time, and a more favorable evaluation of the tool was associated with higher utilization. Program directors reported improvements in faculty knowledge of the milestones and resident satisfaction with feedback. CONCLUSIONS: Faculty and residents credited the M3App with improving the quality and efficiency of resident feedback. Residents appreciated the frequency, proximity, and specificity of feedback, and faculty reported the app improved their familiarity with the milestones. Implementation challenges included lack of a physician champion and competing demands on faculty time.

Tackling the triple aim in primary care residencies: the I3 POP Collaborative.

BACKGROUND AND OBJECTIVES: The I3 POP Collaborative's goal is to improve care of populations served by primary care residencies in North Carolina, South Carolina, and Virginia by dramatically improving patients' experience, quality of care, and cost-effectiveness. We examine residency baseline triple aim measures, compare with national benchmarks, and identify practice characteristics associated with data reporting. METHODS: We used a cross-sectional design, with 27 primary care residency programs caring for over 300,000 patients. Outcome measures were obtained via data pulls from electronic health records and practice management system submitted by residencies; they include quality measure sets for chronic illness and prevention, patient experience (usual provder continuity and time to third available), and utilization (emergency visits, hospitalizations, referrals, high-end radiology). RESULTS: Thirteen practices (48%) reported all required baseline measures. We found associations between data reporting ability with registry use (59% versus 0%) and having a faculty member involved in data management (69% versus 29%). Reported measures varied widely; examples include colorectal cancer screening (median: 61%, range: 28%--80%), provider continuity (median: 52%, range: 1%--68%), subspecialty referral rate (median: 24%, range: 10%--51%). Seventy percent of patient-centered medical homes (PCMH) recognized practices had usual provider continuity (UPC) > or = collaborative median versus 0% of non-PCMH recognized practices. Median data were similar to national comparisons for chronic disease measures, lower for prevention and better for utilization. CONCLUSIONS: Baseline triple aim data are highly variable among residencies, but residency care is comparable to available national standards. Registry use and faculty leadership in data management are critical success factors for assessing practice performance.

Evaluation of prenatal and pediatric group visits in a residency training program.

BACKGROUND: It is well established that group visits offer an appropriate alternative to individual care with respect to efficiency, clinical effectiveness, and patient and provider satisfaction and are feasible in the training setting. The purpose of this paper is to describe resident educational outcomes from participation in prenatal and well-child group visits over the last 6 years. METHODS: We surveyed the 48 physicians who graduated from the University of North Carolina Family Medicine Residency from 2006 through 2011 regarding their current scope of practice, the number of group visits they experienced, and the educational value of group visits. RESULTS: Thirty-four (71%) of graduates responded. Twelve respondents (35%) include prenatal care in their current practice, 29 (85%) include pediatric care, and five (15%) include group visits. As residents, all respondents participated in group visits. Respondents most valued what they learned in group visits from patient questions, from the exposure to a bolus of patients at the same stage of development, and from faculty role modeling. CONCLUSIONS: Group visits are a potentially valuable adjunct to the standard training experience, with benefits for learning efficiency, scope of practice, and the promotion of patient-centered care that can be carried forward into practice.

Colorado family physicians' attitudes toward medical marijuana.

INTRODUCTION: Over the last decade, the use of medical marijuana has expanded dramatically; it is now permitted in 16 states and the District of Columbia. Our study of family physicians in Colorado is the first to gather information about physician attitudes toward this evolving practice. METHODS: We distributed an anonymous web-based electronic survey to the 1727 members of the Colorado Academy of Family Physicians' listserv. Items included individual and practice characteristics as well as experience with and attitudes toward medical marijuana. RESULTS: Five hundred twenty family physicians responded (30% response rate). Of these, 46% did not support physicians recommending medical marijuana; only 19% thought that physicians should recommend it. A minority thought that marijuana conferred significant benefits to physical (27%) and mental (15%) health. Most agreed that marijuana poses serious mental (64%) and physical (61%) health risks. Eighty-one percent agreed that physicians should have formal training before recommending medical marijuana, and 92% agreed that continuing medical education about medical marijuana should be available to family physicians. CONCLUSIONS: Despite a high prevalence of use in Colorado, most family physicians are not convinced of marijuana's health benefits and believe its use carries risks. Nearly all agreed on the need for further medical education about medical marijuana.

A randomized controlled trial of two different types of web-based instructional methods: one with case-based scenarios and one without.

BACKGROUND: Computer-based learning (CBL) is an effective form of medical education. Educators have developed recommendations for instructional design but there is only minimal research that evaluates these recommendations. AIM: To evaluate the effect of case-based questions contained in computer modules on learning efficacy. METHODS: The authors conducted a randomized controlled trial in 124 medical students of two CBL PowerPoint modules-based on Medicare. The modules were identical except one contained 11 case-based scenarios followed by multiple choice questions. The primary outcome measurement was a previously validated, 11-item knowledge test taken at the end of the module and at the end of the academic year to test retention. RESULTS: The students who studied the module with case-based questions answered one more item correctly in the first test (8.9 vs. 10.00 correct answers, p = 0.001). This difference had disappeared by the time of the second test (4.2 vs. 4.7, p = 0.095). CONCLUSIONS: This study shows that computer modules with case-based questions enhance learning in the short-term but at the expense of increased time and so decreased learning efficiency. This learning benefit was not maintained.

Practice transformation in teaching settings: lessons from the I³ PCMH collaborative.

BACKGROUND: The Patient-centered Medical Home (PCMH) model provides a roadmap for practices engaged in practice transformation to improve quality, accessibility, and satisfaction. Primary care residencies can use these principles to transform their practices, but it is unclear how best to facilitate this transformation. This paper describes the design, implementation, and initial outcomes of an academic PCMH collaborative. METHODS: The I³ PCMH Collaborative adapted the Institute for Healthcare Improvement's Breakthrough Collaborative model to facilitate practice transformation in 25 primary care teaching sites across North Carolina, South Carolina, and Virginia. The National Committee for Quality Assurance (NCQA) PCMH Recognition Program provided the goal and outcome measures. Surveys at baseline, midpoint, and end of the 20-month collaborative period, as well as activity assessments, described practice characteristics, tracked progress, and identified key lessons. RESULTS: Twelve programs (48%) achieved NCQA PCMH recognition or submitted applications during the collaborative, and nine programs (36%) planned to submit applications by July 2011. A majority of programs characterized improvements toward becoming a PCMH as "significant" (56%) or "sustainable" (12%). Sixteen (64%) programs credited the collaborative with helping to maintain focus on practice transformation in the face of competing priorities. Twenty-one (84%) programs indicated willingness to participate in a future practice improvement collaborative. CONCLUSIONS: A heterogeneous group of primary care residency programs working together can achieve substantial, measurable improvement toward becoming PCMHs, with a modest investment in collaborative infrastructure.

Improving chronic illness care in teaching practices: learnings from the I³ collaborative.

BACKGROUND: Improving the quality of care in residencies is critical for the profession and for our discipline, but how to do this on a large scale is unclear. The purpose of the I³ collaborative was to assess the feasibility of a regional quality improvement collaborative limited to residencies and to improve significantly dramatically the quality of care for diabetes and congestive heart failure. METHODS: Ten residencies in North and South Carolina with more than 345,000 patient visits/year, 252 residents and 92 faculty participated in an Institute for Healthcare Improvement breakthrough series type collaborative, enriched with additional support for academic settings, over 3 years. RESULTS: We improved measured quality of care for diabetes modestly and congestive heart failure (CHF) significantly/substantially, including a 380% reduction of hospitalizations for CHF. Success factors include funding from regional foundations, the use of regional approach for recruitment of residencies and active management of the collaborative, regular data submission, and a blended curriculum with a combination of biannual face to face meetings and monthly telephone conferences. CONCLUSIONS: A regional strategy is feasible and can strongly support quality improvement; investment in residency redesign can reduce total cost of care.

WellBabies: mothers' perspectives on an innovative model of group well-child care.

BACKGROUND AND OBJECTIVES: Typical well-child visits are often unsatisfactory both to providers and patients. Group visits have been shown effective in some settings but have not been recently assessed for well-child care. Here we describe a model for infant well-child visits, WellBabies, along with participating mothers' perspectives and comparisons of quality outcomes. METHODS: We invited mothers receiving prenatal care in our practice to participate in WellBabies after delivery. Thirteen women participated, and 11 were available for individual interviews. We identified and summarized common themes in interview transcripts and compared quality outcomes with babies who had standard individual well-child visits. RESULTS: Participants expressed largely positive reactions to WellBabies, identifying mutual support, information sharing, parental involvement in care, and time spent in the visit as particularly valuable. One participant expressed concern about lack of individual time with a provider. Quality outcomes were similar to those in the comparison group. CONCLUSIONS: Group well-child visits can effectively address important issues in children's health care and are well-received by participants.

Teaching evidence-based medicine skills: an exploratory study of residency graduates' practice habits.

BACKGROUND AND OBJECTIVES: The long-term effect of teaching critical appraisal (CA) and evidence-based medicine (EBM) skills is unknown. This study explores long-term behaviors and learner satisfaction after a 3-year longitudinal CA/EBM curriculum. METHODS: Telephone interviews were conducted with 1996-1998 graduates of an academic family medicine residency program with an established CA/EBM curriculum. The graduates were all in clinical practice. RESULTS: Ten of 17 graduates met inclusion criteria and consented to be interviewed. Their age range was 31-58, and all had been in practice 3 to 5 years. Six were female. Most participants did not regularly practice CA or use EBM skills. Instead, colleagues were the most commonly used information source. Time constraints and clinical production pressure were the primary barriers to practicing EBM. Despite not practicing CA and EBM, participants generally were satisfied with their training in these skills. Respondents said they used continuing education meetings and reading journals to keep current. CONCLUSIONS: In this study, residents instructed in CA and EBM skills did not regularly practice these skills. Time and workload pressures appear to be major barriers to these behaviors. Those training residents to integrate EBM into clinical practice should evaluate short- and long-term clinically oriented behaviors to assure educational effectiveness.