Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education.

Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.

Clinical problem solving and social determinants of health: a descriptive study using unannounced standardized patients to directly observe how resident physicians respond to social determinants of health.

Objectives While the need to address patients' social determinants of health (SDoH) is widely recognized, less is known about physicians' actual clinical problem-solving when it comes to SDoH. Do physicians include SDoH in their assessment strategy? Are SDoH incorporated into their diagnostic thinking and if so, do they document as part of their clinical reasoning? And do physicians directly address SDoH in their "solution" (treatment plan)? Methods We used Unannounced Standardized Patients (USPs) to assess internal medicine residents' clinical problem solving in response to a patient with asthma exacerbation and concern that her moldy apartment is contributing to symptoms - a case designed to represent a clear and direct link between a social determinant and patient health. Residents' clinical practices were assessed through a post-visit checklist and systematic chart review. Patterns of clinical problem solving were identified and then explored, in depth, through review of USP comments and history of present illness (HPI) and treatment plan documentation. Results Residents fell into three groups when it came to clinical problem-solving around a housing trigger for asthma: those who failed to ask about housing and therefore did not uncover mold as a potential trigger (neglectors ­ 21%; 14/68); those who asked about housing in negative ways that prevented disclosure and response (negative elicitors ­ 23%, 16/68); and those who elicited and explored the mold issue (full elicitors ­ 56%; 38/68) [corrected]. Of the full elicitors 53% took no further action, 26% only documented the mold; and 21% provided resources/referral. In-depth review of USP comments/explanations and residents' notes (HPI, treatment plan) revealed possible influences on clinical problem solving. Failure to ask about housing was associated with both contextual factors (rushed visit) and interpersonal skills (not fully engaging with patient) and with possible differences in attention ("known" vs. unknown/new triggers, usual symptoms vs. changes, not attending to relocation, etc.,). Use of close-ended questions often made it difficult for the patient to share mold concerns. Negative responses to sharing of housing information led to missing mold entirely or to the patient not realizing that the physician agreed with her concerns about mold. Residents who fully elicited the mold situation but did not take action seemed to either lack knowledge or feel that action on SDoH was outside their realm of responsibility. Those that took direct action to help the patient address mold appeared to be motivated by an enhanced sense of urgency. Conclusions Findings provide unique insight into residents' problem solving processes including external influences (e.g., time, distractions), the role of core communication and interpersonal skills (eliciting information, creating opportunities for patients to voice concerns, sharing clinical thinking with patients), how traditional cognitive biases operate in practice (premature closure, tunneling, and ascertainment bias), and the ways in which beliefs about expectancies and scope of practice may color clinical problem-solving strategies for addressing SDoH.

Igniting activation: Using unannounced standardized patients to measure patient activation in smoking cessation.

INTRODUCTION: Despite a decline, smoking rates have remained high, especially in communities with lower income, education, and limited insurance options. Evidence shows that physician-initiated counseling on smoking cessation is effective and saves lives, and that specific skills are needed to appropriately lead this type of patient-physician communication. Residency is a critical moment for future physicians and may be the optimal time to learn, practice, and refine this skillset. Unannounced Standardized Patients (USPs) have been found to be effective, incognito evaluators of resident practices. METHODS: This study introduced rigorously trained actors (USPs) into two urban, safety-net clinics to assess resident ability to engage, activate, and counsel a pre-contemplative smoker. A complementary chart review assessed appropriate documentation in the patient's electronic health record (EHR) and its relationship to counseling style and prescribing practices. RESULTS: Resident scores (% well done) on patient education and engagement were low (33% and 23%, respectively). Residents who coupled cessation advice with an open discussion style activated their patients more than those who solely advised cessation across all comparable measures. On EHR documentation, residents who accurately documented smoking history were more likely to directly advise their patient to quit smoking when compared to residents who did not document (t(97) = 2.828, p = .006, Cohen's D = 0.56). CONCLUSIONS: Results highlight the need to reinforce training in patient-centered approaches including motivational interviewing, counseling, and shared decision-making. Future research should focus on the effects of smokers in pre-contemplation on physician counseling style and examine the relationship between medical training and provider communication to guide interventions.

Telephone Smoking-Cessation Counseling for Smokers in Mental Health Clinics: A Patient-Randomized Controlled Trial.

INTRODUCTION: People with a mental health diagnosis have high rates of tobacco use and encounter limited availability of tobacco treatment targeted to their needs. This study compared the effectiveness of a specialized telephone smoking-cessation intervention developed for mental health patients with standard state quit-line counseling. DESIGN: RCT. SETTING/PARTICIPANTS: The study was conducted at six Veterans Health Administration facilities in the Northeast U.S. Participants were 577 mental health clinic patients referred by their providers for smoking-cessation treatment. INTERVENTION: From 2010 to 2012, the study implemented a telephone program that included patient referral from a mental health provider, mailed cessation medications, and telephone counseling. Participants were randomized to receive a specialized multisession telephone counseling protocol (n=270) or transfer to their state's quit-line for counseling (n=307). MAIN OUTCOME MEASURES: Participants completed telephone surveys at baseline, 2 months, and 6 months. The study's primary outcome was self-reported 30-day abstinence at 6 months. Secondary outcomes were self-reported 30-day abstinence, counseling satisfaction and counseling content at 2 months, and self-reported use of cessation treatment and quit attempts at 6 months. Logistic regression was used to compare treatment groups on outcomes, controlling for baseline cigarettes per day and site. Inverse probability weighting and multiple imputation were used to handle missing abstinence outcomes. Data were analyzed in 2014-2015. RESULTS: At 6 months, participants in the specialized counseling arm were more likely to report 30-day abstinence (26% vs 18%, OR=1.62, 95% CI=1.24, 2.11). There was no significant group difference in abstinence at 2 months (18% vs 14%, OR=1.31, 95% CI=0.49, 3.49). Participants in the specialized arm were more likely to be assisted with developing a quit plan; receive follow-up calls after quitting; and receive counseling on several domains, including motivation, confidence, smoking triggers, coping with urges, and mental health symptoms (all p<0.05). Specialized counseling participants were more satisfied with treatment and more likely to find the counseling useful (p<0.05). CONCLUSIONS: The specialized counseling intervention was more effective at helping patients quit than transfer to a state quit-line. Patients were more satisfied with the specialized counseling program. TRIAL REGISTRATION: This study is registered at www.clinicaltrials.gov NCT00724308.

Cutting too deep? Assessing the impact of a shorter surgery clerkship on students' clinical skills and knowledge.

BACKGROUND: The aim of this study was to compare the performance of students completing an 8-week versus a 6-week surgery clerkship on an objective structured clinical examination (OSCE) and the National Board of Medical Examiners (NBME) clinical science surgery examination. METHODS: One hundred fifteen students from the 8-week clerkship and 99 from the 6-week clerkship were included. Performance on a summative OSCE was assessed using behaviorally anchored checklists. NBME exams were graded using the NBME's standard scaled scores. Results were compared using 2-tailed, independent-samples, unequal-variance t tests. RESULTS: Mean OSCE scores for the 8-week and 6-week curricula were not statistically different. Mean NBME scores also did not statistically differ. Six-week students performed significantly better in the specific OSCE subdomains of blood pressure, orthostatic blood pressure, rectal exam, and fecal occult blood test. CONCLUSIONS: Overall OSCE and NBME exam performance did not differ between 8-week and 6-week surgery clerkship students.

The impact of primary care resident physician training on patient weight loss at 12 months.

OBJECTIVE: It is unclear whether training physicians to counsel obese patients leads to weight loss. This study assessed whether a 5-h multimodal longitudinal obesity curriculum for residents on the basis of the 5As (assess, advise, agree, assist, and arrange) was associated with weight loss in their obese patients. DESIGN AND METHODS: Twenty-three primary care internal medicine residents were assigned by rotation schedule to intervention (curriculum) or control groups. We then conducted follow-up chart reviews to determine weight change at up to 12 months following the index visit. 158 obese patients (76 in the intervention group and 82 in the control group) completed exit interviews; 22 patients who presented for acute care at the index visit were excluded. Chart reviews were conducted on the 46 patients in the intervention group and 41 patients in the control group who were seen again within 12 months of the index visit and had follow-up weight measurements. RESULTS: The main outcome of interest was mean change in weight at 12 months compared between the intervention and control groups. Patients of residents in the intervention group had a mean weight loss of -1.53 kg (s.d. = 3.72) although the patients of those in the control group had a mean weight gain of 0.30 kg (s.d. = 3.60), P = 0.03. Six (15.8%) patients in the intervention group and 2 (5.4%) patients in the control group lost >5% body weight (P = 0.14). CONCLUSIONS: Although the magnitude of weight loss was small, this study shows that training physicians to counsel patients can produce measurable patient outcomes.

Creating an objective structured teaching examination to evaluate a dental faculty development program.

This article describes the creation of an objective structured teaching examination (OSTE) to assess a dental faculty development program. An OSTE is a performance-based measure that utilizes standardized students and is designed to measure observable teaching skills. In spring 2010, the authors developed, implemented, and evaluated an OSTE to assess a New York University College of Dentistry (NYUCD) faculty development program, Class ACTS (Advanced Clinical Teaching Scholars). They created a three-station OSTE to measure changes in teaching skills resulting from participation in Class ACTS. Dental student teaching assistants were trained to act as standardized students and patients and to use the rating forms. The faculty members' OSTE scores and ratings increased after participating in the Class ACTS program in all fifteen teaching domains tested, and statistically significant improvements occurred in nine of the fifteen domains. A search of the relevant literature suggests that this is the first time that an OSTE has been used to assess a faculty development program in dental education. This study's results appear to support NYUCD's faculty development efforts to improve the teaching skills of its faculty members.

Assessment of medical student clinical reasoning by "lay" vs physician raters: inter-rater reliability using a scoring guide in a multidisciplinary objective structured clinical examination.

BACKGROUND: To determine whether a "lay" rater could assess clinical reasoning, interrater reliability was measured between physician and lay raters of patient notes written by medical students as part of an 8-station objective structured clinical examination. METHODS: Seventy-five notes were rated on core elements of clinical reasoning by physician and lay raters independently, using a scoring guide developed by physician consensus. Twenty-five notes were rerated by a 2nd physician rater as an expert control. Kappa statistics and simple percentage agreement were calculated in 3 areas: evidence for and against each diagnosis and diagnostic workup. RESULTS: Agreement between physician and lay raters for the top diagnosis was as follows: supporting evidence, 89% (κ = .72); evidence against, 89% (κ = .81); and diagnostic workup, 79% (κ = .58). Physician rater agreement was 83% (κ = .59), 92% (κ = .87), and 96% (κ = .87), respectively. CONCLUSIONS: Using a comprehensive scoring guide, interrater reliability for physician and lay raters was comparable with reliability between 2 expert physician raters.

Vulnerable patients' perceptions of health care quality and quality data.

BACKGROUND: Little is known about how patients served by safety-net hospitals utilize and respond to hospital quality data. OBJECTIVE: To understand how vulnerable, lower income patients make health care decisions and define quality of care and whether hospital quality data factor into such decisions and definitions. METHODS: Mixed quantitative and qualitative methods were used to gather primary data from patients at an urban, tertiary-care safety-net hospital. The study hospital is a member of the first public hospital system to voluntarily post hospital quality data online for public access. Patients were recruited from outpatient and inpatient clinics. Surveys were used to collect data on participants' sociodemographic characteristics, health literacy, health care experiences, and satisfaction variables. Focus groups were used to explore a representative sample of 24 patients' health care decision making and views of quality. Data from focus group transcripts were iteratively coded and analyzed by the authors. RESULTS: Focus group participants were similar to the broader diverse, low-income clinic. Participants reported exercising choice in making decisions about where to seek health care. Multiple sources influenced decision-making processes including participants' own beliefs and values, social influences, and prior experiences. Hospital quality data were notably absent as a source of influence in health care decision making for this population largely because participants were unaware of its existence. Participants' views of hospital quality were influenced by the quality and efficiency of services provided (with an emphasis on the doctor-patient relationship) and patient centeredness. When presented with it, patients appreciated the hospital quality data and, with guidance, were interested in incorporating it into health care decision making. CONCLUSIONS: Results suggest directions for optimizing the presentation, content, and availability of hospital quality data. Future research will explore how similar populations form and make choices based on presentation of hospital quality data.

An intervention to improve care and reduce costs for high-risk patients with frequent hospital admissions: a pilot study.

BACKGROUND: A small percentage of high-risk patients accounts for a large proportion of Medicaid spending in the United States, which has become an urgent policy issue. Our objective was to pilot a novel patient-centered intervention for high-risk patients with frequent hospital admissions to determine its potential to improve care and reduce costs. METHODS: Community and hospital-based care management and coordination intervention with pre-post analysis of health care utilization. We enrolled Medicaid fee-for-service patients aged 18-64 who were admitted to an urban public hospital and identified as being at high risk for hospital readmission by a validated predictive algorithm. Enrolled patients were evaluated using qualitative and quantitative interview techniques to identify needs such as transportation to/advocacy during medical appointments, mental health/substance use treatment, and home visits. A community housing partner initiated housing applications in-hospital for homeless patients. Care managers facilitated appropriate discharge plans then worked closely with patients in the community using a harm reduction approach. RESULTS: Nineteen patients were enrolled; all were male, 18/19 were substance users, and 17/19 were homeless. Patients had a total of 64 inpatient admissions in the 12 months before the intervention, versus 40 in the following 12 months, a 37.5% reduction. Most patients (73.3%) had fewer inpatient admissions in the year after the intervention compared to the prior year. Overall ED visits also decreased after study enrollment, while outpatient clinic visits increased. Yearly study hospital Medicaid reimbursements fell an average of $16,383 per patient. CONCLUSIONS: A pilot intervention for high-cost patients shows promising results for health services usage. We are currently expanding our model to serve more patients at additional hospitals to see if the pilot's success can be replicated. TRIAL REGISTRATION: Clinicaltrials.gov Identifier: NCT01292096.

New measures to establish the evidence base for medical education: identifying educationally sensitive patient outcomes.

Researchers lack the rich evidence base and benchmark patient outcomes needed to evaluate the effectiveness of medical education practice and guide policy. The authors offer a framework for medical education research that focuses on physician-influenced patient outcomes that are potentially sensitive to medical education. Adapting the concept of ambulatory care sensitive conditions, which provided traction to health services research by defining benchmark patient outcomes to measure health system performance, the authors introduce the concept and propose the adoption of educationally sensitive patient outcomes and suggest two measures: patient activation and clinical microsystem activation. They assert that the ultimate goal of medical education is to ensure that measurement of future physicians' competence and skills is based not only on biomedical knowledge and critical clinical skills but also on the ability to translate these competencies into effective patient- and systems-level outcomes. The authors consider methodological approaches and challenges to measuring such outcomes and argue for large, multiinstitutional, prospective cohort studies and the development of a national Database for Research in Education in Academic Medicine to provide the needed infrastructure. They advocate taking the next steps to establish an educational evidence base to guide the academic medical centers of the 21st century in aligning medical education practice with health care delivery that meets the needs of individuals and populations.

Two decades of Title VII support of a primary care residency: process and outcomes.

PURPOSE: To assess 23 years of Health Resources and Services Administration (HRSA) Title VII Training in Primary Care Medicine and Dentistry funding to the New York University School of Medicine/Bellevue Primary Care Internal Medicine Residency Program. The program, begun in 1983 within a traditional, inner-city, subspecialty-oriented internal medicine program, evolved into a crucible of systematic innovation, catalyzed and made feasible by initiatives funded by the HRSA. The curriculum stressed three pillars of generalism: psychosocial medicine, clinical epidemiology, and health policy. It developed tight, objectives-driven, effective, nonmedical specialty blocks and five weekly primary care activities that created a paradigm-driven, community-based, role-modeling matrix. Innovation was built in. Every block and activity was evaluated immediately and in an annual, program-wide retreat. Evaluation evolved from behavioral checklists of taped interviews to performance-based, systematic, annual objective structured clinical examinations. METHOD: The authors reviewed eight grant proposals, project reports, and curriculum and program evaluations. They also quantitatively and qualitatively surveyed the 122 reachable graduates from the first 20 graduating classes of the program. RESULTS: Analysis of program documents revealed recurring emphases on the use of proven educational models, strategic innovation, and assessment and evaluation to design and refine the program. There were 104 respondents (85%) to the survey. A total of 87% of the graduates practice as primary care physicians, 83% teach, and 90% work with the underserved; 54% do research, 36% actively advocate on health issues for their patients, programs, and other constituencies, and 30% publish. Graduates cited work in the community and faculty excitement and energy as essential elements of the program's impact; overall, graduates reported high personal and career satisfaction and low burnout. CONCLUSIONS: With HRSA support, a focused, innovative program evolved which has already met each of the six recommendations for future innovation of the Alliance for Academic Internal Medicine Education Redesign Task Force. This article is part of a theme issue of Academic Medicine on the Title VII health professions training programs.

A survey of psychiatric residency directors on current priorities and preparation for public-sector care.

OBJECTIVE: This study assessed how resident psychiatrists are being prepared to deliver effective public-sector care. METHODS: Ten leaders in psychiatric education and practice were interviewed about which tasks they consider to be essential for effective public-sector care. The leaders identified 16 tasks. Directors of all general psychiatry residency programs in the United States were then surveyed to determine how they rate the importance of these tasks for delivery of care and how their training program prepares residents to perform each task. RESULTS: A total of 114 of 150 residency directors (76 percent) responded to the survey. Factor analysis divided 14 of the tasks into three categories characterized by the extent to which their performance requires integration of services: within the mental health system (for example, lead a multidisciplinary team), across social service systems (for example, interact with staff of supportive housing programs), and across institutions with different missions (for example, distinguish behavioral problems from underlying psychiatric disorders among prisoners). Preparation for tasks that involved integration of services across institutions was rated as least important, was least likely to be required, and was covered by less intensive teaching modalities. Tasks entailing integration within the mental health system were rated as most important, preparation was most likely to be required, and they were covered most intensively. Midway between these two categories, but significantly different from each, were tasks relying on integration across social service systems. CONCLUSIONS: Tasks that involved integrating services across institutions with different missions were consistently downplayed in training. Yet the importance of such tasks is underscored by the assessments of the psychiatric leaders who were interviewed, the high valuation placed on this type of integration by a substantial subset of training directors, and the extent of mental illness among populations who are institutionalized in nonpsychiatric settings.

Effect of communications training on medical student performance.

CONTEXT: Although physicians' communication skills have been found to be related to clinical outcomes and patient satisfaction, teaching of communication skills has not been fully integrated into many medical school curricula or adequately evaluated with large-scale controlled trials. OBJECTIVE: To determine whether communications training for medical students improves specific competencies known to affect outcomes of care. DESIGN AND SETTING: A communications curriculum instituted in 2000-2001 at 3 US medical schools was evaluated with objective structured clinical examinations (OSCEs). The same OSCEs were administered to a comparison cohort of students in the year before the intervention. PARTICIPANTS: One hundred thirty-eight randomly selected medical students (38% of eligible students) in the comparison cohort, tested at the beginning and end of their third year (1999-2000), and 155 students in the intervention cohort (42% of eligible students), tested at the beginning and end of their third year (2000-2001). INTERVENTION: Comprehensive communications curricula were developed at each school using an established educational model for teaching and practicing core communication skills and engaging students in self-reflection on their performance. Communications teaching was integrated with clinical material during the third year, required clerkships, and was supported by formal faculty development. MAIN OUTCOME MEASURES: Standardized patients assessed student performance in OSCEs on 21 skills related to 5 key patient care tasks: relationship development and maintenance, patient assessment, education and counseling, negotiation and shared decision making, and organization and time management. Scores were calculated as percentage of maximum possible performance. RESULTS: Adjusting for baseline differences, students exposed to the intervention significantly outperformed those in the comparison cohort on the overall OSCE (65.4% vs 60.4%; 5.1% difference; 95% confidence interval [CI], 3.9%-6.3%; P<.001), relationship development and maintenance (5.3% difference; 95% CI, 3.8%-6.7%; P<.001), organization and time management (1.8% difference; 95% CI, 1.0%-2.7%; P<.001), and subsets of cases addressing patient assessment (6.7% difference; 95% CI, 5.9%-7.8%; P<.001) and negotiation and shared decision making (5.7% difference; 95% CI, 4.5%-6.9%; P<.001). Similar effects were found at each of the 3 schools, though they differed in magnitude. CONCLUSIONS: Communications curricula using an established educational model significantly improved third-year students' overall communications competence as well as their skills in relationship building, organization and time management, patient assessment, and negotiation and shared decision making-tasks that are important to positive patient outcomes. Improvements were observed at each of the 3 schools despite adaptation of the intervention to the local curriculum and culture.