Racial, Ethnic, and Socioeconomic Differences in Primary Care No-Show Risk with Telemedicine During the COVID-19 Pandemic.

BACKGROUND: The coronavirus 2019 (COVID-19) pandemic resulted in rapid implementation of telemedicine. Little is known about the impact of telemedicine on both no-show rates and healthcare disparities on the general primary care population during the pandemic. OBJECTIVE: To compare no-show rates between telemedicine and office visits in the primary care setting, while controlling for the burden of COVID-19 cases, with focus on underserved populations. DESIGN: Retrospective cohort study. SETTING: Multi-center urban network of primary care clinics between April 2021 and December 2021. PARTICIPANTS: A total of 311,517 completed primary care physician visits across 164,647 patients. MAIN MEASURES: The primary outcome was risk ratio of no-show incidences (i.e., no-show rates) between telemedicine and office visits across demographic sub-groups including age, ethnicity, race, and payor type. RESULTS: Compared to in-office visits, the overall risk of no-showing favored telemedicine, adjusted risk ratio of 0.68 (95% CI 0.65 to 0.71), absolute risk reduction (ARR) 4.0%. This favorability was most profound in several cohorts with racial/ethnic and socioeconomic differences with risk ratios in Black/African American 0.47 (95% CI 0.41 to 0.53), ARR 9.0%; Hispanic/Latino 0.63 (95% CI 0.58 to 0.68), ARR 4.6%; Medicaid 0.58 (95% CI 0.54 to 0.62) ARR 7.3%; Self-Pay 0.64 (95% CI 0.58 to 0.70) ARR 11.3%. LIMITATION: The analysis was limited to physician-only visits in a single setting and did not examine the reasons for visits. CONCLUSION: As compared to office visits, patients using telemedicine have a lower risk of no-showing to primary care appointments. This is one step towards improved access to care.

Recognizing Non-vertebral Manifestation of Mycobacterium avium Complex Osteomyelitis in a Patient With HIV.

Disseminated Mycobacterium avium complex (MAC) infection is predominantly seen in immunocompromised individuals, such as those with HIV infection and CD4 counts <50 cells/mm3. It commonly manifests with nonspecific signs and symptoms, such as weight loss, fevers, night sweats, diarrhea, lymphadenopathy, hepatosplenomegaly, and cytopenias. This is a case of disseminated MAC osteomyelitis in an HIV patient. The lack of constitutional symptoms, in this case, presented a diagnostic challenge. In addition, nonvertebral osteomyelitis is an uncommon manifestation, making this case of disseminated MAC osteomyelitis a unique presentation.

Exploring the role of telehealth in providing equitable healthcare to the vulnerable patient population during COVID-19.

The COVID 19 pandemic resulted in widespread telehealth implementation. Existent health disparities were widened, with under-represented minorities (URM) disproportionately affected by COVID. In this study, we assess the role of telehealth in improving access to care in the URMs and vulnerable populations. We noted a statistically significant increase in the number of visits in Hispanic or Latino patients (15.2% increase, p < 0.01) and Black patients (19% increase, p < 0.01). Based on payer type, there was a statistically significant increase in the number of visits in the Medicare (10.2%, p = 0.0001) and Medicaid (16.2%, p < 0.01) groups. We also noted increased access to care with telehealth in patients who were 65 and older (10.6%, p = 0.004). This highlights the importance of telehealth in increasing access to care and promoting health equity in the URM and vulnerable patient populations.

Improving trainee clinical documentation through a novel curriculum in internal medicine.

BACKGROUND: Clinical documentation is a key component of practice. Trainees rarely receive formal training in documentation or assessment of their documentation. Effective methods of improving documentation remain unknown. OBJECTIVE: The objective of this study was to determine if the implementation of a documentation curriculum led to improvement in admission note quality. DESIGNS: Admission notes written prior to implementation of the curriculum and after the curriculum intervention were assessed. Notes were assessed from two-time frames for both years to account for improvement with time not associated with the intervention. SETTINGS AND PARTICIPANTS: Admission notes written by University of Cincinnati interns were assessed. INTERVENTIONS: The documentation curriculum consisted of educational sessions and routine admission note assessments with feedback. MAIN OUTCOMES AND MEASURES: Admission notes were assessed via the 16 checklist items and two global assessment items of the Admission Note Assessment Tool (ANAT). RESULTS: Six ANAT items showed statistically significant differences. The review of systems item improved with the intervention only (odds ratio: 3.61, p < .001) while the assessment and plan item 1 and global assessment item 2 improved with time only (ß = .08, p = .03 and ß = .25, p = .02, respectively) in univariate models. In univariate models the physical exam item, diagnostic data item 2, and global assessment item 1 showed improvement with both intervention and time, respectively, with additive effects seen in models with both intervention and time. CONCLUSION: Several aspects of documentation can improve with a formal documentation curriculum which includes a routine assessment with feedback, and some aspects of documentation improve with time.

Coproducing the Learning Environment: Lessons Learned from a Year of Near-Peer Teaching.

PURPOSE: Few medical schools offer electives with the goal of teaching medical students to be effective teachers prior to residency. We developed a novel year-long, longitudinal course, the Clinical Teaching Elective (CTE), that develops fourth-year medical students as student teachers within Clinical Skills (CS). APPROACH/METHODS: The elective was designed by Clinical Skills (CS) Course Directors and two fourth-year medical students (M4) as a longitudinal elective. The elective involves teaching in the Simulation Center where M4 student instructors teach first and second-year medical students. Each session, in addition to simulated patient case topics, emphasizes application of a key topic within medical education (ie clinical reasoning, reflective practice, dual process reasoning). DISCUSSION: Six "teaching takeaways" were crafted to summarize common themes experienced by near-peer medical student educators. Teaching is not about the destination, but rather the diagnostic journey.Students thrive when learning is co-produced.A little bit of praise goes a long way.You can't please every learner.When students struggle, there is more to teach than just the answer.Facilitating learner independent thinking promotes future autonomy. SIGNIFICANCE: A novel CTE for fourth-year medical students that emphasizes medical education pedagogy prepares students to serve as educators in residency. The CTE provides an opportunity for medical students to develop into effective clinical educators prior to residency. The focus of our elective on medical education pedagogy furthers medical student understanding of adult learning theory and fosters professional development in teaching clinical reasoning.

Developing the Expected Entrustment Score: Accounting for Variation in Resident Assessment.

BACKGROUND: Clinical competency committees (CCCs) and residency program leaders may find it difficult to interpret workplace-based assessment (WBA) ratings knowing that contextual factors and bias play a large role. OBJECTIVE: We describe the development of an expected entrustment score for resident performance within the context of our well-developed Observable Practice Activity (OPA) WBA system. DESIGN: Observational study PARTICIPANTS: Internal medicine residents MAIN MEASURE: Entrustment KEY RESULTS: Each individual resident had observed entrustment scores with a unique relationship to the expected entrustment scores. Many residents' observed scores oscillated closely around the expected scores. However, distinct performance patterns did emerge. CONCLUSIONS: We used regression modeling and leveraged large numbers of historical WBA data points to produce an expected entrustment score that served as a guidepost for performance interpretation.

Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program.

INTRODUCTION: Narrative assessment data are valuable in understanding struggles in resident performance. However, it remains unknown which themes in narrative data that occur early in training may indicate a higher likelihood of struggles later in training, allowing programs to intervene sooner. METHODS: Using learning analytics, we identified 26 internal medicine residents in three cohorts that were below expected entrustment during training. We compiled all narrative data in the first 6 months of training for these residents as well as 13 typically performing residents for comparison. Narrative data were blinded for all 39 residents during initial phases of an inductive thematic analysis for initial coding. RESULTS: Many similarities were identified between the two cohorts. Codes that differed between typical and lower entrusted residents were grouped into two types of themes: three explicit/manifest and three implicit/latent with six total themes. The explicit/manifest themes focused on specific aspects of resident performance with assessors describing 1) Gaps in attention to detail, 2) Communication deficits with patients, and 3) Difficulty recognizing the "big picture" in patient care. Three implicit/latent themes, focused on how narrative data were written, were also identified: 1) Feedback described as a deficiency rather than an opportunity to improve, 2) Normative comparisons to identify a resident as being behind their peers, and 3) Warning of possible risk to patient care. DISCUSSION: Clinical competency committees (CCCs) usually rely on accumulated data and trends. Using the themes in this paper while reviewing narrative comments may help CCCs with earlier recognition and better allocation of resources to support residents' development.

Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane.

PROBLEM: Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system. APPROACH: To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time. OUTCOMES: The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps. NEXT STEPS: Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments.

A Dynamic Risk Management Approach for Reducing Harm From Invasive Bedside Procedures Performed During Residency.

Internal medicine (IM) residents frequently perform invasive bedside procedures during residency training. Bedside procedure training in IM programs may compromise patient safety. Current evidence suggests that IM training programs rely heavily on the number of procedures completed during training as a proxy for resident competence instead of using objective postprocedure patient outcomes. The authors posit that the results of procedural training effectiveness should be reframed with outcome metrics rather than process measures alone. This article introduces the as low as reasonably achievable (ALARA) approach, which originated in the nuclear industry to increase safety margins, to help assess and reduce bedside procedural risks. Training program directors are encouraged to use ALARA calculations to define the risk trade-offs inherent in current procedural training and assess how best to reliably improve patient outcomes. The authors describe 5 options to consider: training all residents in bedside procedures, training only select residents in bedside procedures, training no residents in bedside procedures, deploying 24-hour procedure teams supervised by IM faculty, and deploying 24-hour procedure teams supervised by non-IM faculty. The authors explore how quality improvement approaches using process maps, fishbone diagrams, failure mode effects and analyses, and risk matrices can be effectively implemented to assess training resources, choices, and aims. Future research should address the drivers behind developing optimal training programs that support independent practice, correlations with patient outcomes, and methods that enable faculty to justify their supervisory decisions while adhering to ALARA risk management standards.

See One, Do One, Forget One: Early Skill Decay After Paracentesis Training.

INTRODUCTION: Internal medicine residents perform paracentesis, but programs lack standard methods for assessing competence or maintenance of competence and instead rely on number of procedures completed. This study describes differences in resident competence in paracentesis over time. METHODS: From 2016 to 2017, internal medicine residents (n = 118) underwent paracentesis simulation training. Competence was assessed using the Paracentesis Competency Assessment Tool (PCAT), which combines a checklist, global scale, and entrustment score. The PCAT also delineates two categorical cut-point scores: the Minimum Passing Standard (MPS) and the Unsupervised Practice Standard (UPS). Residents were randomized to return to the simulation lab at 3 and 6 months (group A, n = 60) or only 6 months (group B, n = 58). At each session, faculty raters assessed resident performance. Data were analyzed to compare resident performance at each session compared with initial training scores, and performance between groups at 6 months. RESULTS: After initial training, all residents met the MPS. The number achieving UPS did not differ between groups: group A = 24 (40%), group B = 20 (34.5%), p = 0.67. When group A was retested at 3 months, performance on each PCAT component significantly declined, as did the proportion of residents meeting the MPS and UPS. At the 6-month test, residents in group A performed significantly better than residents in group B, with 52 (89.7%) and 20 (34.5%) achieving the MPS and UPS, respectively, in group A compared with 25 (46.3%) and 2 (3.70%) in group B (p < .001 for both comparison). DISCUSSION: Skill in paracentesis declines as early as 3 months after training. However, retraining may help interrupt skill decay. Only a small proportion of residents met the UPS 6 months after training. This suggests using the PCAT to objectively measure competence would reclassify residents from being permitted to perform paracentesis independently to needing further supervision.

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine.

BACKGROUND: Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM). OBJECTIVE: To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods. DESIGN: The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward. PARTICIPANTS: NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members. MAIN MEASURES: The number and description of RSQMs generated during this process. KEY RESULTS: Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM general care-21, diabetes mellitus-16, hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission. CONCLUSIONS: Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

Development and Establishment of Initial Validity Evidence for a Novel Tool for Assessing Trainee Admission Notes.

BACKGROUND: Documentation is a key component of practice, yet few curricula have been published to teach trainees proper note construction. Additionally, a gold standard for assessing note quality does not exist, and no documentation assessment tools integrate with established competency-based frameworks. OBJECTIVE: To develop and establish initial validity evidence for a novel tool that assesses key components of trainee admission notes and maps to the Accreditation Council for Graduate Medical Education (ACGME) milestone framework. DESIGN: Using an iterative, consensus building process we developed the Admission Note Assessment Tool (ANAT). Pilot testing was performed with both the supervising attending and study team raters not involved in care of the patients. The finalized tool was piloted with attendings from other institutions. PARTICIPANTS: Local experts participated in tool development and pilot testing. Additional attending physicians participated in pilot testing. MAIN MEASURES: Content, response process, and internal structure validity evidence was gathered using Messick's framework. Inter-rater reliability was assessed using percent agreement. KEY RESULTS: The final tool consists of 16 checklist items and two global assessment items. Pilot testing demonstrated rater agreement of 72% to 100% for checklist items and 63% to 70% for global assessment items. Note assessment required an average of 12.3 min (SD 3.7). The study generated validity evidence in the domains of content, response process, and internal structure for use of the tool in rating admission notes. CONCLUSIONS: The ANAT assesses individual components of a note, incorporates billing criteria, targets note "bloat," allows for narrative feedback, and provides global assessments mapped to the ACGME milestone framework. The ANAT can be used to assess admission notes by any attending and at any time after note completion with minimal rater training. The ANAT allows programs to implement routine note assessment for multiple functions with the use of a single tool.

A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments.

PURPOSE: To examine the reliability and attributable facets of variance within an entrustment-derived workplace-based assessment system. METHOD: Faculty at the University of Cincinnati Medical Center internal medicine residency program (a 3-year program) assessed residents using discrete workplace-based skills called observable practice activities (OPAs) rated on an entrustment scale. Ratings from July 2012 to December 2016 were analyzed using applications of generalizability theory (G-theory) and decision study framework. Given the limitations of G-theory applications with entrustment ratings (the assumption that mean ratings are stable over time), a series of time-specific G-theory analyses and an overall longitudinal G-theory analysis were conducted to detail the reliability of ratings and sources of variance. RESULTS: During the study period, 166,686 OPA entrustment ratings were given by 395 faculty members to 253 different residents. Raters were the largest identified source of variance in both the time-specific and overall longitudinal G-theory analyses (37% and 23%, respectively). Residents were the second largest identified source of variation in the time-specific G-theory analyses (19%). Reliability was approximately 0.40 for a typical month of assessment (27 different OPAs, 2 raters, and 1-2 rotations) and 0.63 for the full sequence of ratings over 36 months. A decision study showed doubling the number of raters and assessments each month could improve the reliability over 36 months to 0.76. CONCLUSIONS: Ratings from the full 36 months of the examined program of assessment showed fair reliability. Increasing the number of raters and assessments per month could improve reliability, highlighting the need for multiple observations by multiple faculty raters.

The Power of Contribution and Attribution in Assessing Educational Outcomes for Individuals, Teams, and Programs.

Recent discussions have brought attention to the utility of contribution analysis for evaluating the effectiveness and outcomes of medical education programs, especially for complex initiatives such as competency-based medical education. Contribution analysis focuses on the extent to which different entities contribute to an outcome. Given that health care is provided by teams, contribution analysis is well suited to evaluating the outcomes of care delivery. Furthermore, contribution analysis plays an important role in analyzing program- and system-level outcomes that inform program evaluation and program-level improvements for the future. Equally important in health care, however, is the role of the individual. In the overall contribution of a team to an outcome, some aspects of this outcome can be attributed to individual team members. For example, a recently discharged patient with an unplanned return to the emergency department to seek care may not have understood the discharge instructions given by the nurse or may not have received any discharge guidance from the resident physician. In this example, if it is the nurse's responsibility to provide discharge instructions, that activity is attributed to him or her. This and other activities attributed to different individuals (e.g., nurse, resident) combine to contribute to the outcome for the patient. Determining how to tease out such attributions is important for several reasons. First, it is physicians, not teams, that graduate and are granted certification and credentials for medical practice. Second, incentive-based payment models focus on the quality of care provided by an individual. Third, an individual can use data about his or her performance on the team to help drive personal improvement. In this article, the authors explored how attribution and contribution analyses can be used in a complimentary fashion to discern which outcomes can and should be attributed to individuals, which to teams, and which to programs.

Variation in Entrustment When Sharing a Single Assessment System Between University- and Community-Based Residency Programs: A Comparison.

PURPOSE: Given resource constraints, many residency programs would consider adopting an entrustment-based assessment system from another program if given the opportunity. However, it is unclear if a system developed in one context would have similar or different results in another. This study sought to determine if entrustment varied between programs (community based and university based) when a single assessment system was deployed in different contexts. METHOD: The Good Samaritan Hospital (GSH) internal medicine residency program adopted the observable practice activity (OPA) workplace-based assessment system from the University of Cincinnati (UC). Comparisons for OPA-mapped subcompetency entrustment progression for programs and residents were made at specific timepoints over the course of 36 months of residency. Data collection occurred from August 2012 to June 2017 for UC and from September 2013 to June 2017 for GSH. RESULTS: GSH entrustment ratings were higher than UC for all but the 11th, 15th, and 36th months of residency (P < .0001) and were also higher for the majority of subcompetencies and competencies (P < .0001). The rate of change for average monthly entrustment was similar, with GSH having an increase of 0.041 each month versus 0.042 for UC (P = .73). Most residents progressed from lower to higher entrustment, but there was significant variation between residents in each program. CONCLUSIONS: Despite the deployment of a single entrustment-based assessment system, important outcomes may vary by context. Further research is needed to understand the contributions of tool, context, and other factors on the data these systems produce.

A Feasibility Study to Attribute Patients to Primary Interns on Inpatient Ward Teams Using Electronic Health Record Data.

PURPOSE: To inform graduate medical education (GME) outcomes at the individual resident level, this study sought a method for attributing care for individual patients to individual interns based on "footprints" in the electronic health record (EHR). METHOD: Primary interns caring for patients on an internal medicine inpatient service were recorded daily by five attending physicians of record at University of Cincinnati Medical Center in August 2017 and January 2018. These records were considered gold standard identification of primary interns. The following EHR variables were explored to determine representation of primary intern involvement in care: postgraduate year, progress note author, discharge summary author, physician order placement, and logging clicks in the patient record. These variables were turned into quantitative attributes (e.g., progress note author: yes/no), and informative attributes were selected and modeled using a decision tree algorithm. RESULTS: A total of 1,511 access records were generated; 116 were marked as having a primary intern assigned. All variables except discharge summary author displayed at least some level of importance in the models. The best model achieved 78.95% sensitivity, 97.61% specificity, and an area under the receiver-operator curve of approximately 91%. CONCLUSIONS: This study successfully predicted primary interns caring for patients on inpatient teams using EHR data with excellent model performance. This provides a foundation for attributing patients to primary interns for the purposes of determining patient diagnoses and complexity the interns see as well as supporting continuous quality improvement efforts in GME.