Equity in prenatal healthcare services globally: an umbrella review.

BACKGROUND: Timely, appropriate, and equitable access to quality healthcare during pregnancy is proven to contribute to better health outcomes of birthing individuals and infants following birth. Equity is conceptualized as the absence of differences in healthcare access and quality among population groups. Healthcare policies are guides for front-line practices, and despite merits of contemporary policies striving to foster equitable healthcare, inequities persist. The purpose of this umbrella review is to identify prenatal healthcare practices, summarize how equities/inequities are reported in relation to patient experiences or health outcomes when accessing or using services, and collate equity reporting characteristics. METHODS: For this umbrella review, six electronic databases were searched (Medline, EMBASE, APA PsychInfo, CINAHL, International Bibliography of the Social Sciences, and Cochrane Library). Included studies were extracted for publication and study characteristics, equity reporting, primary outcomes (prenatal care influenced by equity/inequity) and secondary outcomes (infant health influenced by equity/inequity during pregnancy). Data was analyzed deductively using the PROGRESS-Plus equity framework and by summative content analysis for equity reporting characteristics. The included articles were assessed for quality using the Risk of Bias Assessment Tool for Systematic Reviews. RESULTS: The search identified 8065 articles and 236 underwent full-text screening. Of the 236, 68 systematic reviews were included with first authors representing 20 different countries. The population focus of included studies ranged across prenatal only (n = 14), perinatal (n = 25), maternal (n = 2), maternal and child (n = 19), and a general population (n = 8). Barriers to equity in prenatal care included travel and financial burden, culturally insensitive practices that deterred care engagement and continuity, and discriminatory behaviour that reduced care access and satisfaction. Facilitators to achieve equity included innovations such as community health workers, home visitation programs, conditional cash transfer programs, virtual care, and cross-cultural training, to enhance patient experiences and increase their access to, and use of health services. There was overlap across PROGRESS-Plus factors. CONCLUSIONS: This umbrella review collated inequities present in prenatal healthcare services, globally. Further, this synthesis contributes to future solution and action-oriented research and practice by assembling evidence-informed opportunities, innovations, and approaches that may foster equitable prenatal health services to all members of diverse communities.

Exploring the Quality of Feedback in Entrustable Professional Activity Narratives Across 24 Residency Training Programs.

Background Competency-based medical education (CBME) has been implemented in many residency training programs across Canada. A key component of CBME is documentation of frequent low-stakes workplace-based assessments to track trainee progression over time. Critically, the quality of narrative feedback is imperative for trainees to accumulate a body of evidence of their progress. Suboptimal narrative feedback will challenge accurate decision-making, such as promotion to the next stage of training. Objective To explore the quality of documented feedback provided on workplace-based assessments by examining and scoring narrative comments using a published quality scoring framework. Methods We employed a retrospective cohort secondary analysis of existing data using a sample of 25% of entrustable professional activity (EPA) observations from trainee portfolios from 24 programs in one institution in Canada from July 2019 to June 2020. Statistical analyses explore the variance of scores between programs (Kruskal-Wallis rank sum test) and potential associations between program size, CBME launch year, and medical versus surgical specialties (Spearman's rho). Results Mean quality scores of 5681 narrative comments ranged from 2.0±1.2 to 3.4±1.4 out of 5 across programs. A significant and moderate difference in the quality of feedback across programs was identified (χ2=321.38, P<.001, ε2=0.06). Smaller programs and those with an earlier launch year performed better (P<.001). No significant difference was found in quality score when comparing surgical/procedural and medical programs that transitioned to CBME in this institution (P=.65). Conclusions This study illustrates the complexity of examining the quality of narrative comments provided to trainees through EPA assessments.

Making sense of competency-based medical education (CBME) literary conversations: A BEME scoping review: BEME Guide No. 78.

BACKGROUND: Competency-based medical education (CBME) received increased attention in the early 2000s by educators, clinicians, and policy makers as a way to address concerns about physician preparedness and patient safety in a rapidly changing healthcare environment. Opinions and perspectives around this shift in medical education vary and, to date, a systematic search and synthesis of the literature has yet to be undertaken. The aim of this scoping review is to present a comprehensive map of the literary conversations surrounding CBME. METHODS: Twelve different databases were searched from database inception up until 29 April 2020. Literary conversations were extracted into the following categories: perceived advantages, perceived disadvantages, challenges/uncertainties/skepticism, and recommendations related to CBME. RESULTS: Of the 5757 identified records, 387 were included in this review. Through thematic analysis, eight themes were identified in the literary conversations about CBME: credibility, application, community influence, learner impact, assessment, educational developments, organizational structures, and societal impacts of CBME. Content analysis supported the development of a heat map that provides a visual illustration of the frequency of these literary conversations over time. CONCLUSIONS: This review serves two purposes for the medical education research community. First, this review acts as a comprehensive historical record of the shifting perceptions of CBME as the construct was introduced and adopted by many groups in the medical education global community over time. Second, this review consolidates the many literary conversations about CBME that followed the initial proposal for this approach. These findings can facilitate understanding of CBME for multiple audiences both within and outside of the medical education research community.

Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool.

BACKGROUND: Narrative feedback, like verbal feedback, is essential to learning. Regardless of form, all feedback should be of high quality. This is becoming even more important as programs incorporate narrative feedback into the constellation of evidence used for summative decision-making. Continuously improving the quality of narrative feedback requires tools for evaluating it, and time to score. A tool is needed that does not require clinical educator expertise so scoring can be delegated to others. OBJECTIVE: To develop an evidence-based tool to evaluate the quality of documented feedback that could be reliably used by clinical educators and non-experts. METHODS: Following a literature review to identify elements of high-quality feedback, an expert consensus panel developed the scoring tool. Messick's unified concept of construct validity guided the collection of validity evidence throughout development and piloting (2013-2020). RESULTS: The Evaluation of Feedback Captured Tool (EFeCT) contains 5 categories considered to be essential elements of high-quality feedback. Preliminary validity evidence supports content, substantive, and consequential validity facets. Generalizability evidence supports that EFeCT scores assigned to feedback samples show consistent interrater reliability scores between raters across 5 sessions, regardless of level of medical education or clinical expertise (Session 1: n=3, ICC=0.94; Session 2: n=6, ICC=0.90; Session 3: n=5, ICC=0.91; Session 4: n=6, ICC=0.89; Session 5: n=6, ICC=0.92). CONCLUSIONS: There is preliminary validity evidence for the EFeCT as a useful tool for scoring the quality of documented feedback captured on assessment forms. Generalizability evidence indicated comparable EFeCT scores by raters regardless of level of expertise.

Vitalizing the evaluation of curricular implementation: a framework for attending to the "how and whys" of curriculum evolution.

As curricular reforms are implemented, there is often urgency among scholars to swiftly evaluate curricular outcomes and establish whether desired impacts have been realized. Consequently, many evaluative studies focus on summative program outcomes without accompanying evaluations of implementation. This runs the risk of Type III errors, whereby outcome evaluations rest on unverified assumptions about the appropriate implementation of prescribed curricular activities. Such errors challenge the usefulness of the evaluative studies, casting doubt on accumulated knowledge about curricular innovations, and posing problems for educational systems working to mobilize scarce resources. Unfortunately, however, there is long-standing inattention to the evaluation of implementation in health professions education (HPE). To address this, we propose an accessible framework that provides substantive guidance for evaluative research on implementation of curricular innovations. The Prescribed-Intended-Enacted-Sustainable (PIES) framework that is articulated in this paper, introduces new concepts to HPE-with a view to facilitating more nuanced examination of the evolution of curricula as they are implemented. Critically, the framework is theoretically grounded, integrating evaluation and implementation science as well as education theory. It outlines when, how, and why evaluators need to direct attention to curricular implementation, providing guidance on how programs can map out meaningful evaluative research agendas. Ultimately, this work is intended to support evaluators and educators, seeking to design evaluation studies that provide more faithful, useful representations of the intricacies of curricular change implementation.

Monitoring the integrity and usability of policy evaluation tools within an evolving sociocultural context: A demonstration of reflexivity using the CFPC Family Medicine Longitudinal Survey.

RATIONALE, AIMS, AND OBJECTIVES: Over the last decade, policy changes have prompted Canadian medical education to emphasize a transformation to competency-based education, and subsequent development of evaluation tools. The pandemic provides a unique opportunity to emphasize the value of reflexive monitoring, a cyclical and iterative process of appraisal and adaptation, since tools are influenced by social and cultural factors relevant at the time of their development. METHODS: Deductive content analysis of documents and resources about the advancement of primary care. Reflexive monitoring of the Family Medicine Longitudinal Survey (FMLS), an evaluation tool for physician training. RESULTS: The FMLS tool does not explore all training experiences that are currently relevant; including, incorporating technology, infection control and safety, public health services referrals, patient preferences for care modality, and trauma-informed culturally safe care. CONCLUSION: The results illustrate that reflection promotes the validity and usefulness of the data collected to inform policy performance and other initiatives.

Perceptions of family medicine in Canada through the eyes of learners.

OBJECTIVE: To examine the perceptions of family medicine (FM) residents about their chosen specialty and how they perceive that patients, other specialists, and the government value FM. DESIGN: Self-report data from the Family Medicine Longitudinal Survey collected from 2014 (time 1 [T1]) to 2016 (time 2 [T2]). SETTING: Canada. PARTICIPANTS: Family medicine residents from 16 out of the 17 FM residency programs. MAIN OUTCOME MEASURES: Responses to statements in the survey were evaluated using a 5-point Likert scale (from strongly disagree to strongly agree). Data were analyzed in 2 ways: cross sectionally (participation in either T1 or T2), and longitudinally (participation in both T1 and T2). RESULTS: For both the cross-sectional cohorts (T1, n = 916; T2, n = 785) and the repeated-measures cohort (n = 420), most residents responded positively to feeling proud of becoming a family physician, with little change from entrance to exit. For both cohorts, a higher proportion of residents at the end of training reported that other medical specialists value the contributions of family physicians (P < .001); however, fewer believed that the government perceived FM as essential to the health care system (P < .001). CONCLUSION: Most participating Canadian FM residents feel proud to become family physicians. This feeling may come from the perceptions of others who are believed to value FM, including other specialists. Measuring attitudinal perceptions offers a window to discover how FM is viewed and can offer a way to measure the effect of strategies implemented to advance the discipline of FM.

Eco-Normalization: Evaluating the Longevity of an Innovation in Context.

PURPOSE: When initiating an educational innovation, successful implementation and meaningful, lasting change can be elusive. This elusiveness stems from the difficulty of introducing changes into complex ecosystems. Program evaluation models that focus on implementation fidelity examine the inner workings of an innovation in the real-world context. However, the methods by which fidelity is typically examined may inadvertently limit thinking about the trajectory of an innovation over time. Thus, a new approach is needed, one that focuses on whether the conditions observed during the implementation phase of an educational innovation represent a foundation for meaningful, long-lasting change. METHOD: Through a critical review, authors examined relevant models from implementation science and developed a comprehensive framework that shifts the focus of program evaluation from exploring snapshots in time to assessing the trajectory of an innovation beyond the implementation phase. RESULTS: Durable and meaningful "normalization" of an innovation is rooted in how the local aspirations and practices of the institutional system and the people doing the work interact with the grand aspirations and features of the innovation. Borrowing from Normalization Process Theory, the Consolidated Framework for Implementation Research, and Reflexive Monitoring in Action, the authors developed a framework, called Eco-Normalization, that highlights 6 critical questions to be considered when evaluating the potential longevity of an innovation. CONCLUSIONS: When evaluating an educational innovation, the Eco-Normalization model focuses our attention on the ecosystem of change and the features of the ecosystem that may contribute to (or hinder) the longevity of innovations in context.

Five ways to get a grip on the shortcomings of logic models in program evaluation.

Logic models are perhaps the most widely used tools in program evaluation work. They provide reasonably straightforward, visual illustrations of plausible links between program activities and outcomes. Consequently, they are employed frequently in stakeholder engagement, communication, and evaluation project planning. However, their relative simplicity comes with multiple drawbacks that can compromise the integrity of evaluation studies. In this Black Ice article, we outline key considerations and provide practical strategies that can help those engaged in evaluation work to identify and mitigate some limitations of logic models.

Les modèles logiques sont vraisemblablement les outils d'évaluation de programme les plus utilisés. Ils illustrent visuellement de façon assez simple les liens plausibles entre les activités du programme et les résultats obtenus. Par conséquent, ils sont fréquemment utilisés pour la mobilisation des parties prenantes, la communication et la planification de tels projets. Toutefois, leur relative simplicité s'accompagne de multiples inconvénients qui peuvent compromettre l'intégrité des études d'évaluation. Dans cet article de (la rubrique) Terrain glissant, nous proposons des éléments essentiels et des stratégies pratiques à prendre en considération lorsqu'on entreprend une évaluation pour être en mesure de cibler et de remédier à certaines limites des modèles logiques.

Process and outcome evaluation of a CBME intervention guided by program theory.

RATIONALE: Competency-based medical education (CBME) has gained momentum as an improved training model, but literature on outcomes of CBME, including evaluation of implementation processes, is minimal. We present a case for the following: (a) the development of a program theory is essential prior to or in the initial stages of implementation of CBME; (b) the program theory should guide the strategies and methods for evaluation that will answer questions about anticipated and unintended outcomes; and (c) the iterative process of testing assumptions and hypotheses will lead to modifications to the program theory to inform best practices of implementing CBME. METHODS: We use the Triple C Competency-based Curriculum as a worked example to illustrate how process and outcome evaluation, guided by a program theory, can lead to meaningful enhancement of CBME curriculum, assessment, and implementation strategies. Using a mixed methods design, the processes and outcomes of Triple C were explored through surveys, interviews, and historical document review, which captured the experiences of various stakeholders. FINDINGS: The theory-led program evaluation process was able to identify areas that supported CBME implementation: the value of a strong nondirective national vertical core supporting the transformation in education, program autonomy, and adaptability to pre-existing local context. Areas in need of improvement included the need for ongoing support from College of Family Physicians of Canada (CFPC) and better planning for shifts in program leadership over time. CONCLUSIONS: Deliberately pairing evaluation alongside change is an important activity and, when accomplished, yields valuable information from the experiences of those implementing and experiencing a program. Evaluation and the development of an updated program theory facilitate the introduction of new changes and theories that build on these findings, which also supports the desired goal of contributing toward cumulative science rather than "reinventing the wheel."

Exploring Change After Implementation of Family Medicine Residency Curriculum Reform.

BACKGROUND AND OBJECTIVES: In 2010, the College of Family Physicians of Canada (CFPC) launched its competency-based medical education (CBME) approach to residency curriculum and assessment. Named Triple C, this innovation was developed to ensure graduates of family medicine training programs are competent to begin unsupervised practice. Further, Triple C was intended to promote interest in practicing comprehensive family medicine. A program evaluation plan was launched by the CFPC alongside the implementation of Triple C to explore if intended outcomes were achieved. METHODS: We conducted retrospective secondary data analysis of survey findings from graduating family medicine residents from two sources: National Physician Survey (NPS 2007 and 2010); and the Family Medicine Longitudinal Survey (FMLS 2015). Demographics and practice intentions reported by residents in the NPS 2007, NPS 2010, and FMLS 2015 were included in the analyses and a comparison between years was undertaken using a series of Pearson χ2 test. RESULTS: Findings indicate that in comparison to pre-Triple C (NPS 2007 and NPS 2010), significantly more residents reported the intention to include palliative care, intrapartum care, in-patient hospital care, care in the home, and practicing in rural settings after the implementation of Triple C (FMLS 2015; P<0.01). CONCLUSIONS: Family medicine graduates report an increase in intention to include a broader range of clinical domains after implementation of Triple C. While a causal relationship cannot be determined, using a historical control in the form of survey data that predates Triple C implementation could support future approaches to evaluation of education reform.

Association of a Competency-Based Assessment System With Identification of and Support for Medical Residents in Difficulty.

Importance: Competency-based medical education is now established in health professions training. However, critics stress that there is a lack of published outcomes for competency-based medical education or competency-based assessment tools. Objective: To determine whether competency-based assessment is associated with better identification of and support for residents in difficulty. Design, Setting, and Participants: This cohort study of secondary data from archived files on 458 family medicine residents (2006-2008 and 2010-2016) was conducted between July 5, 2016, and March 2, 2018, using a large, urban family medicine residency program in Canada. Exposures: Introduction of the Competency-Based Achievement System (CBAS). Main Outcomes and Measures: Proportion of residents (1) with at least 1 performance or professionalism flag, (2) receiving flags on multiple distinct rotations, (3) classified as in difficulty, and (4) with flags addressed by the residency program. Results: Files from 458 residents were reviewed (pre-CBAS: n = 163; 81 [49.7%] women; 90 [55.2%] aged >30 years; 105 [64.4%] Canadian medical graduates; post-CBAS: n = 295; 144 [48.8%] women; 128 [43.4%] aged >30 years; 243 [82.4%] Canadian medical graduates). A significant reduction in the proportion of residents receiving at least 1 flag during training after CBAS implementation was observed (0.38; 95% CI, 0.377-0.383), as well as a significant decrease in the numbers of distinct rotations during which residents received flags on summative assessments (0.24; 95% CI, 0.237-0.243). There was a decrease in the number of residents in difficulty after CBAS (from 0.13 [95% CI, 0.128-0.132] to 0.17 [95% CI, 0.168-0.172]) depending on the strictness of criteria defining a resident in difficulty. Furthermore, there was a significant increase in narrative documentation that a flag was discussed with the resident between the pre-CBAS and post-CBAS conditions (0.18; 95% CI, 0.178-0.183). Conclusions and Relevance: The CBAS approach to assessment appeared to be associated with better identification of residents in difficulty, facilitating the program's ability to address learners' deficiencies in competence. After implementation of CBAS, residents experiencing challenges were better supported and their deficiencies did not recur on later rotations. A key argument for shifting to competency-based medical education is to change assessment approaches; these findings suggest that competency-based assessment may be useful.

Depression Outcomes in Adults Attending Family Practice Were Not Improved by Screening, Stepped-Care, or Online CBT during a 12-Week Study when Compared to Controls in a Randomized Trial.

There is uncertainty regarding possible benefits of screening for depression in family practice, as well as the most effective treatment approach when depression is identified. Here, we examined whether screening patients for depression in primary care, and then treating them with different modalities, was better than treatment-as-usual (TAU) alone. Screening was carried out for depression using the 9-item Patient Health Questionnaire (PHQ-9), with a score of ≥10 indicating significant depressive symptoms. PHQ-9 scores were given to family physicians prior to patients being seen (except for the Control group). Patients (n = 1,489) were randomized to one of four groups. Group #1 were controls (n = 432) in which PHQ-9 was administered, but results were not shared. Group #2 was screening followed by TAU (n = 426). Group #3 was screening followed by both TAU and the opportunity to use an online cognitive behavioral therapy (CBT) treatment program (n = 440). Group #4 utilized an evidence-based Stepped-care pathway for depression (n = 191, note that this was not available at all clinics). Of the study sample 889 (60%) completed a second PHQ-9 rating at 12 weeks. There were no statistically significant differences in baseline PHQ-9 scores between these groups. Compared to baseline, mean PHQ-9 scores decreased significantly in the depressed patients over 12 weeks, but there were no statistically significant differences between any groups at 12 weeks. Thus, for those who were depressed at baseline Control group (Group #1) scores decreased from 15.3 ± 4.2 to 4.0 ± 2.6 (p < 0.001), Screening group (Group #2) scores decreased from 15.5 ± 3.9 to 4.6 ± 3.0 (p < 0.001), Online CBT group (Group #3) scores decreased from 15.4 ± 3.8 to 3.4 ± 2.7 (p < 0.01), and the Stepped-care pathway group (Group #4) scores decreased from 15.3 ± 3.6 to 5.4 ± 2.8 (p < 0.05). In conclusion, these findings from this controlled randomized study do not suggest that using depression screening tools in family practice improves outcomes. They also suggest that much of the depression seen in primary care spontaneously resolves and do not support suggestions that more complex treatment programs or pathways improve depression outcomes in primary care. Replication studies are required due to study limitations.

An integrated web-based mental health intervention of assessment-referral-care to reduce stress, anxiety, and depression in hospitalized pregnant women with medically high-risk pregnancies: a feasibility study protocol of hospital-based implementation.

BACKGROUND: At prevalence rates of up to 40%, rates of depression and anxiety among women with medically complex pregnancies are 3 times greater than those in community-based samples of pregnant women. However, mental health care is not a component of routine hospital-based antenatal care for medically high-risk pregnant women. OBJECTIVE: The purpose of this study is to evaluate the effectiveness and feasibility of the hospital-based implementation of a Web-based integrated mental health intervention comprising psychosocial assessment, referral, and cognitive behavioral therapy (CBT) for antenatal inpatients. METHODS: This study is a quasi-experimental design. Pregnant women are eligible to participate if they are (1) <37 weeks gestation, (2) admitted to the antenatal inpatient unit for >72 hours, (3) able to speak and read English or be willing to use a translation service to assist with completion of the questionnaires and intervention, (4) able to complete follow-up email questionnaires, (5) >16 years of age, and (6) not actively suicidal. Women admitted to the unit for induction (eg, <72-hour length of stay) are excluded. A minimum sample of 54 women will be recruited from the antenatal high-risk unit of a large, urban tertiary care hospital. All women will complete a Web-based psychosocial assessment and 6 Web-based CBT modules. Results of the psychosocial assessment will be used by a Web-based clinical decision support system to generate a clinical risk score and clinician prompts to provide recommendations for the best treatment and referral options. The primary outcome is self-reported prenatal depression, anxiety, and stress symptoms at 6-8 weeks postrecruitment. Secondary outcomes are postpartum depression, anxiety, and stress symptoms; self-efficacy; mastery; self-esteem; sleep; relationship quality; coping; resilience; Apgar score; gestational age; birth weight; maternal-infant attachment; infant behavior and development; parenting stress/competence at 3-months postpartum; and intervention cost-effectiveness, efficiency, feasibility, and acceptability. All women will complete email questionnaires at 6-8 weeks postrecruitment and 3-months postpartum. Qualitative interviews with 10-15 health care providers and 15-30 women will provide data on feasibility and acceptability of the intervention. RESULTS: The study was funded in September, 2014 and ethics was approved in November, 2014. Subject recruitment will begin January, 2015 and results are expected in December, 2015. Results of this study will determine (1) the effectiveness of an integrated Web-based prenatal mental health intervention on maternal and infant outcomes and (2) the feasibility of implementation of the intervention on a high-risk antenatal unit. CONCLUSIONS: This study will provide evidence and guidance regarding the implementation of a Web-based mental health program into routine hospital-based care for women with medically high-risk pregnancies.