Exploring the merits of research performance measures that comply with the San Francisco Declaration on Research Assessment and strategies to overcome barriers of adoption: qualitative interviews with administrators and researchers.

BACKGROUND: In prior research, we identified and prioritized ten measures to assess research performance that comply with the San Francisco Declaration on Research Assessment, a principle adopted worldwide that discourages metrics-based assessment. Given the shift away from assessment based on Journal Impact Factor, we explored potential barriers to implementing and adopting the prioritized measures. METHODS: We identified administrators and researchers across six research institutes, conducted telephone interviews with consenting participants, and used qualitative description and inductive content analysis to derive themes. RESULTS: We interviewed 18 participants: 6 administrators (research institute business managers and directors) and 12 researchers (7 on appointment committees) who varied by career stage (2 early, 5 mid, 5 late). Participants appreciated that the measures were similar to those currently in use, comprehensive, relevant across disciplines, and generated using a rigorous process. They also said the reporting template was easy to understand and use. In contrast, a few administrators thought the measures were not relevant across disciplines. A few participants said it would be time-consuming and difficult to prepare narratives when reporting the measures, and several thought that it would be difficult to objectively evaluate researchers from a different discipline without considerable effort to read their work. Strategies viewed as necessary to overcome barriers and support implementation of the measures included high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting for researchers, guidance for evaluators, and sharing of approaches across research institutes. CONCLUSIONS: While participants identified many strengths of the measures, they also identified a few limitations and offered corresponding strategies to address the barriers that we will apply at our organization. Ongoing work is needed to develop a framework to help evaluators translate the measures into an overall assessment. Given little prior research that identified research assessment measures and strategies to support adoption of those measures, this research may be of interest to other organizations that assess the quality and impact of research.

DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey.

OBJECTIVE: The San Francisco Declaration on Research Assessment (DORA) advocates for assessing biomedical research quality and impact, yet academic organizations continue to employ traditional measures such as Journal Impact Factor. We aimed to identify and prioritize measures for assessing research quality and impact. METHODS: We conducted a review of published and grey literature to identify measures of research quality and impact, which we included in an online survey. We assembled a panel of researchers and research leaders, and conducted a two-round Delphi survey to prioritize measures rated as high (rated 6 or 7 by ≥ 80% of respondents) or moderate (rated 6 or 7 by ≥ 50% of respondents) importance. RESULTS: We identified 50 measures organized in 8 domains: relevance of the research program, challenges to research program, or productivity, team/open science, funding, innovations, publications, other dissemination, and impact. Rating of measures by 44 panelists (60%) in Round One and 24 (55%) in Round Two of a Delphi survey resulted in consensus on the high importance of 5 measures: research advances existing knowledge, research plan is innovative, an independent body of research (or fundamental role) supported by peer-reviewed research funding, research outputs relevant to discipline, and quality of the content of publications. Five measures achieved consensus on moderate importance: challenges to research productivity, potential to improve health or healthcare, team science, collaboration, and recognition by professional societies or academic bodies. There was high congruence between researchers and research leaders across disciplines. CONCLUSIONS: Our work contributes to the field by identifying 10 DORA-compliant measures of research quality and impact, a more comprehensive and explicit set of measures than prior efforts. Research is needed to identify strategies to overcome barriers of use of DORA-compliant measures, and to "de-implement" traditional measures that do not uphold DORA principles yet are still in use.

Barriers to cross-disciplinary knowledge flow: The case of medical education research.

INTRODUCTION: The medical education research field operates at the crossroads of two distinct academic worlds: higher education and medicine. As such, this field provides a unique opportunity to explore new forms of cross-disciplinary knowledge exchange. METHODS: Cross-disciplinary knowledge flow in medical education research was examined by looking at citation patterns in the five journals with the highest impact factor in 2017. To grasp the specificities of the knowledge flow in medical education, the field of higher education was used as a comparator. In total, 2031 citations from 64 medical education and 41 higher education articles published in 2017 were examined. RESULTS: Medical education researchers draw on a narrower range of knowledge communities than their peers in higher education. Medical education researchers predominantly cite articles published in health and medical education journals (80% of all citations), and to a lesser extent, articles published in education and social science journals. In higher education, while the largest share of the cited literature is internal to the domain (36%), researchers cite literature from across the social science spectrum. Findings suggest that higher education scholars engage in conversations with academics from a broader range of communities and perspectives than their medical education colleagues. DISCUSSION: Using Pierre Bourdieu's concepts of doxa and field, it is argued that the variety of epistemic cultures entering the higher education research space contributes to its interdisciplinary nature. Conversely, the existence of a relatively homogeneous epistemic culture in medicine potentially impedes cross-disciplinary knowledge exchange.

When all else fails: The (mis)use of qualitative research in the evaluation of complex interventions.

RATIONAL, AIMS, AND OBJECTIVES: Qualitative research has been promoted as an important component of the evaluation of complex interventions to support the scale up and spread of health service interventions, but is currently not being maximized in practice. We aim to identify and explore the sociocultural and structural factors that impact the uses (and misuses) of qualitative research in the evaluation of complex health services interventions. METHODS: We conducted a qualitative analysis of data collected in a multiple case study of the evaluation and scale up and spread of three health service intervention. RESULTS: Our findings demonstrate the challenges of meaningfully integrating qualitative research in evaluation programmes lead by clinicians with limited qualitative expertise and operating within an environment dominated by biomedical research, even with methodological support. CONCLUSIONS: Based on these findings we encourage ongoing engagement of qualitative researchers in evaluation programmes to begin to refine our methodological understanding, while also suggesting changes to medical education and evaluation funding models to create fertile environments for interdisciplinary collaborations.

Understanding decisions to scale up: a qualitative case study of three health service intervention evaluations.

OBJECTIVE: Efforts to scale up evidence-based health care interventions are seen as a key strategy to address complex health system challenges. However, scale-up efforts have shown significant variability. We address the gap between scale-up theory and practice by exploring the socio-cultural factors at play in the evaluation and scale-up of three interventions within the clinical field. METHODS: A qualitative multiple case study was conducted to characterize the evaluation and scale-up efforts of three interventions. We interviewed 18 participants, including clinicians and researchers across the three cases. Using Pierre Bourdieu's concepts of field and capital as a theoretical lens, we conducted a thematic analysis of the data. RESULTS: Despite the espoused goals of ensuring that health service interventions are always based on high-quality evidence within the clinical field, this study demonstrates that the outcomes of the evaluations are not the only factor in the decision to engage in scale-up efforts. Important socio-cultural factors also come into play. Bourdieu uses the term capital to refer to the resources that agents compete for and with their acquisition, accumulate power and/or social standing. The type of evidence valued in the clinical field and the ability to leverage capital in demonstrating that value are also important factors. CONCLUSIONS: Determining if an intervention is effective and should be scaled up is more complex in practice than described in the literature. Efforts are needed to explicitly include the role of social processes in the current frameworks guiding scaling-up efforts.

Interdisciplinarity in medical education research: myth and reality.

The medical education (Med Ed) research community characterises itself as drawing on the insights, methods, and knowledge from multiple disciplines and research domains (e.g. Sociology, Anthropology, Education, Humanities, Psychology). This common view of Med Ed research is echoed and reinforced by the narrative used by leading Med Ed departments and research centres to describe their activities as "interdisciplinary." Bibliometrics offers an effective method of investigating scholarly communication to determine what knowledge is valued, recognized, and utilized. By empirically examining whether knowledge production in Med Ed research draws from multiple disciplines and research areas, or whether it primarily draws on the knowledge generated internally within the field of Med Ed, this article explores whether the characterisation of Med Ed research as interdisciplinary is substantiated. A citation analysis of 1412 references from research articles published in 2017 in the top five Med Ed journals was undertaken. A typology of six knowledge clusters was inductively developed. Findings show that the field of Med Ed research draws predominantly from two knowledge clusters: the Applied Health Research cluster (made of clinical and health services research), which represents 41% of the references, and the Med Ed research cluster, which represents 40% of the references. These two clusters cover 81% of all references in our sample, leaving 19% distributed among the other knowledge clusters (i.e., Education, disciplinary, interdisciplinary and topic centered research). The quasi-hegemonic position held by the Applied Health and Med Ed research clusters confines the other sources of knowledge to a peripheral role within the Med Ed research field. Our findings suggest that the assumption that Med Ed research is an interdisciplinary field is not convincingly supported by empirical data and that the knowledge entering Med Ed comes mostly from the health research domain.

Problematizing assumptions about interdisciplinary research: implications for health professions education research.

This article critically examines three assumptions underlying recent efforts to advance interdisciplinary research-defined in this article as communication and collaboration between researchers across academic disciplines (e.g. Sociology, Psychology, Biology)-and examines these assumptions' implications for health professions education research (HPER). These assumptions are: (1) disciplines are silos that inhibit the free flowing of knowledge across fields and stifle innovative thinking; (2) interdisciplinary research generates a better understanding of the world as it brings together researchers from various fields of expertise capable of tackling complex problems; and (3) interdisciplinary research reduces fragmentation across groups of researchers by eliminating boundaries. These assumptions are among the new beliefs shaping the contemporary academic arena; they orient academics' and university administrators' decisions toward expanding interdisciplinary research and training, but without solid empirical evidence. This article argues that the field of HPER has largely adopted the premises of interdisciplinary research but has not yet debated the potential effects of organizing around these premises. The authors hope to inspire members of the HPER community to critically examine the ubiquitous discourse promoting interdisciplinarity, and engage in reflection about the future of the field informed by evidence rather than by unsubstantiated assumptions. For example: Should research centres and graduate programs in HPER encourage the development of interdisciplinary or disciplinary-trained researchers? Should training predominantly focus on methods and methodologies or draw more on disciplinary-based knowledge? What is the best route toward increasing the field's profile within academia and attracting the best students and researchers to engage in HPER? These are questions that merit attention at the current juncture as the future of the HPER field relies on decisions made in the present time.

Examining grounded theory through the lens of rationalist epistemology.

The objective of scientific, or more broadly, academic knowledge is to provide an understanding of the social and natural world that lies beyond common sense and everyday thinking. Academics use an array of techniques, methods and conceptual apparatuses to achieve this goal. The question we explore in this essay is the following: Does the grounded theory approach, in the constructivist version developed by Kathy Charmaz, provide the necessary methodological tools for the creation of knowledge and theories beyond everyday thinking? To conduct our analysis, we have drawn on the rationalist epistemology originally developed by Gaston Bachelard and taken up a few decades later by Pierre Bourdieu and colleagues to look at the epistemological foundation of the CGT methods as defined by Charmaz. We focussed on two distinctive epistemological features characterising constructivist grounded theory (CGT): the use of inductive reasoning to generate interpretative theory; and the primacy of subjectivity over objectivity as the preferred path to knowledge making. While the usefulness of CGT for conducting qualitative research and understanding the perspective of social actors has been acknowledged by scholars in health professions education research and other research areas, the inductivist logic on which it draws raises questions concerning the nature of the knowledge yielded by this approach. As we argue in this article, it is still unclear in what way the interpretative theory generated by CGT is not a duplication of everyday thinking expressed through meta-narratives. It is also unclear how the understanding of social phenomena can be refined if the use of inductive procedures logically implies the creation of a new theory each time a study is conducted. We engage with these questions to broaden the epistemological conversation within the health professions education research community. It is our hope that scholars in the field will engage in this epistemological conversation and advance it in new directions.

Confined to a tokenistic status: Social scientists in leadership roles in a national health research funding agency.

The idea of interdisciplinarity has been taken up by academic and governmental organisations around the world and enacted through science policies, funding programs and higher education institutions. In Canada, interdisciplinarity led to a major transformation in health research funding. In 2000, the federal government closed the Medical Research Council (MRC) and created the Canadian Institutes of Health Research (CIHR). From the outset, CIHR's vision and goals were innovative, as it sought to include the social sciences within its purview alongside more traditional health research sectors. The extent to which it has been successful in this endeavour, however, remains unknown. The aim of our study was to examine how CIHR's intentions to foster inclusiveness and cooperation across disciplines were implemented in the agency's own organisational structure. We focused on social scientists' representation on committees and among decision-makers between 2000 and 2015, one of the key mandates of CIHR being to include the social sciences within its remit and support research in this area. We examined the composition of the Governing Council, the Institute Scientific Directors, the Chairs of the College of Reviewers, and two International Review Panels invited by CIHR. We targeted these committees and decision-makers since they hold the power to influence the field of Canadian health research through the decisions they make. Our findings show that, while CIHR was created with the mandate to support the entire spectrum of health-related research-including the social sciences-this call for inclusiveness has not yet been materialized in the agency's organisational structure. Social scientists, as well as researchers from neighbouring disciplines such as social epidemiology, health promotion and the humanities, are still confined to low levels of representation within CIHR's highest echelons. This imbalance limits social scientists' input into health research in Canada and undermines CIHR's interdisciplinary ambition.

Interdisciplinary promises versus practices in medicine: the decoupled experiences of social sciences and humanities scholars.

This paper explores social scientists' and humanities (SSH) scholars' integration within the academic medical research environment. Three questions guided our investigation: Do SSH scholars adapt to the medical research environment? How do they navigate their career within a culture that may be inconsistent with their own? What strategies do they use to gain legitimacy? The study builds on three concepts: decoupling, doxa, and epistemic habitus. Twenty-nine semi-structured interviews were conducted with SSH scholars working in 11 faculties of medicine across Canada. Participants were selected through purposeful and snowball sampling. The data were analyzed by thematic content analysis. For most of our participants, moving into medicine has been a challenging experience, as their research practices and views of academic excellence collided with those of medicine. In order to achieve some level of legitimacy more than half of our participants altered their research practices. This resulted in a dissonance between their internalized appreciation of academic excellence and their new, altered, research practices. Only six participants experienced no form of challenge or dissonance after moving into medicine, while three decided to break with their social science and humanities past and make the medical research community their new home. We conclude that the work environment for SSH scholars in faculties of medicine does not deliver on the promise of inclusiveness made by calls for interdisciplinarity in Canadian health research.

Meanings and perceptions of patient-centeredness in social work, nursing and medicine: a comparative study.

Answering patients' evolving, more complex needs has been recognized as a main incentive for the development of interprofessional care. Thus, it is not surprising that patient-centered practice (PCP) has been adopted as a major outcome for interprofessional education. Nevertheless, little research has focused on how PCP is perceived across the professions. This study aimed to address this issue by adopting a phenomenological approach and interviewing three groups of professionals: social workers (n = 10), nurses (n = 10) and physicians (n = 8). All the participants worked in the same department (the General Internal Medicine department of a university affiliated hospital). Although the participants agreed on a core meaning of PCP as identifying, understanding and answering patients' needs, they used many dimensions to define PCP. Overall, the participants expressed value for PCP as a philosophy of care, but there was the sense of a hierarchy of patient-centeredness across the professions, in which both social work and nursing regarded themselves as more patient-centered than others. On their side, physicians seemed inclined to accept their lower position in this hierarchy. Gieryn's concept of boundary work is employed to help illuminate the nature of PCP within an interprofessional context.

What do we do? Practices and learning strategies of medical education leaders.

BACKGROUND: Continuous changes in undergraduate and postgraduate medical education require faculty to assume a variety of new leadership roles. While numerous faculty development programmes have been developed, there is little evidence about the specific practices of medical education leaders or their learning strategies to help inform their design. AIM: This study aimed to explore what medical education leaders' actually do, their learning strategies and recommendations for faculty development. METHOD: A total of 16 medical education leaders from a variety of contexts within the faculty of medicine of a large North American medical school participated in semi-structured interviews to explore the nature of their work and the learning strategies they employ. Using thematic analysis, interview transcripts were coded inductively and then clustered into emergent themes. RESULTS: Findings clustered into four key themes of practice: (1) intrapersonal (e.g., self-awareness), (2) interpersonal (e.g., fostering informal networks), (3) organizational (e.g., creating a shared vision) and (4) systemic (e.g. strategic navigation). Learning strategies employed included learning from experience and example, reflective practice, strategic mentoring or advanced training. CONCLUSIONS: Our findings illuminate a four-domain framework for understanding medical education leader practices and their learning preferences. While some of these findings are not unknown in the general leadership literature, our understanding of their application in medical education is unique. These practices and preferences have a potential utility for conceptualizing a coherent and relevant approach to the design of faculty development strategies for medical education leadership.

The CanMEDS role of Collaborator: How is it taught and assessed according to faculty and residents?

OBJECTIVE: To explore the perspectives of paediatric residents and faculty regarding how the Collaborator role is taught and assessed. METHODS: Using a constructivist grounded theory approach, focus groups at four Canadian universities were conducted. Data were analyzed iteratively for emergent themes. RESULTS: Residents reported learning about collaboration through faculty role modelling but did not perceive that it was part of the formal curriculum. Faculty reported that they were not trained in how to effectively model this role. Both groups reported a need for training in conflict management, particularly as it applies to intraprofessional (physician-to-physician) relationships. Finally, the participants asserted that current methods to assess residents on their performance as collaborators are suboptimal. CONCLUSIONS: The Collaborator role should be a formal part of the residency curriculum. Residents need to be better educated with regard to managing conflict and handling intraprofessional relationships. Finally, innovative methods of assessing residents on this non-medical expert role need to be created.

OBJECTIF: Explorer les points de vue des résidents et des professeurs de pédiatrie sur l'enseignement et l'évaluation du rôle de collaborateur. MÉTHODOLOGIE: Au moyen d'une démarche axée sur la théorie constructiviste, les chercheurs ont formé des groupes de travail dans quatre universités canadiennes. Ils ont analysé les données de manière itérative afin d'en faire ressortir les thèmes émergents. RÉSULTATS: Les résidents ont déclaré apprendre la collaboration par l'exemple des professeurs, mais n'avaient pas l'impression qu'elle faisait partie du programme officiel. Les professeurs ont déclaré ne pas avoir de formation sur la manière efficace de démontrer ce rôle. Les deux groupes ont rendu compte d'un besoin de formation en gestion de conflits, notamment dans le cadre des relations intraprofessionnelles (entre médecins). Enfin, les participants ont fait valoir que les méthodes actuelles d'évaluation du rendement des résidents à titre de collaborateurs sont sous-optimales. CONCLUSIONS: Le rôle de collaborateur devrait faire partie intégrante du programme de résidence. Les résidents doivent être mieux formés en matière de gestion de conflits et de relations intraprofessionnelles. Enfin, il faut créer des méthodes novatrices d'évaluation des résidents à l'égard de ce rôle d'expert non médical.

The CanMEDS portfolio: a tool for reflection in a fellowship programme.

BACKGROUND: We developed a portfolio framed by the CanMEDS roles for use in a paediatric fellowship programme. The portfolio includes four components: a record of participation and achievement; career goals and professional development; physical evidence; and reflective writing. METHODS: Once the portfolio was in use for 6 months, we studied how fellows and faculty members were using the portfolio, and what they found to be its advantages and disadvantages. RESULTS: Fellows reported that it kept them organised and assisted them in setting their goals. They appreciated having a central place to record their accomplishments, as this allowed them to keep a thorough curriculum vitae. The portfolio was helpful in giving them the opportunity to honestly reflect on their achievements and setbacks, and, after reviewing this in their own minds, they were in a strong position to set an agenda for their meetings with supervisors. Both the fellows and supervisors were in agreement that the portfolio led to improved discussions at their meetings. Both groups also reported that this new tool was useful in furthering the career development of trainees, which was one of the main goals in its inception. Faculty supervisors also had a stronger sense of the fellows' work, and also of any gaps in training, as a result of using this tool. DISCUSSION: We hope that this reflection tool will be adapted for use in other training programmes. If it is introduced elsewhere, we would recommend that learners and staff receive ample training in its use so that it can be maximally effective.

The future of medical education: a Canadian environmental scan.

OBJECTIVES: One hundred years after the Flexner report remade medical education in North America, many countries are reviewing the purpose and organisation of medical education. In Canada, a national study is being undertaken to define important issues and challenges for the future of medical education. The objectives of this paper are to describe the process of conducting an empirical environmental scan at a national level, and to present the research findings of this scan. METHODS: Thirty national key informant interviews were conducted, transcribed and coded to identify key themes. Interview data were triangulated with data sourced from 34 commissioned literature reviews and a series of national focus groups. RESULTS: Ten key issues or priorities were identified and used to generate detailed review papers used by the Association of Faculties of Medicine of Canada to create a blueprint for the evolution of medical education. The new priorities have major implications for areas ranging from admissions, curriculum content, educational process and the need to articulate the purpose and responsibilities of medical schools in society. DISCUSSION: This research provides a case study of how an empirical research approach can be used to identify and validate priorities for changes in medical education at a national level. This approach may be of interest in other countries.

The origins of the field of medical education research.

PURPOSE: This paper answers two related questions: When did the education of doctors become a subject for scientific inquiry? And what were the political and economic contexts, the worldviews, structures and events, that enabled the emergence and development of medical education research at that time and in that manner? METHOD: A detailed, concurrent, chronological textual analysis, with triangulation between textual form and contents, was conducted between 2006 and 2008. The primary texts were the complete volumes of the Journal of Medical Education (then the only journal in its field) from 1955 through 1959. Texts were analyzed within a theoretical framework grounded in the literature on interdisciplinary fields. RESULTS: In 1955, the academic medical community was just awakening to the possibility of medical education research; by 1959, it was institutionalized. Major factors contributing to its emergence included the increasing importance of scientific research generally, money for medical education research, the explosive growth of scientific knowledge, and growing calls for public accountability of medical education. CONCLUSIONS: Many factors led to the emergence of medical education research in the late 1950s within a particular sociohistorical context. The nature of research in this field, which is currently the subject of debate, is also of necessity historically situated and contingent, drawing on its roots in this era. A historical understanding will inform further analysis of the events, structures, and worldviews that underpin the definition(s) of legitimate knowledge production within the field of medical education research.

The mindsets of medical education leaders: how do they conceive of their work?

PURPOSE: How a leader perceives his or her organization affects that individual's decision making and beliefs about the best way to influence it. The goal of this study was to understand how medical education leaders conceive of their work. METHOD: The first author interviewed 16 medical education leaders in the Faculty of Medicine at the University of Toronto from June 2005 until February 2006. The sample represented different practice contexts to ensure a diverse overview of experiences. Using the theoretical framework of Bolman and Deal, the authors examined and described the perceptual frames through which these leaders perceive their endeavors. Transcripts were analyzed and then mapped onto Bolman and Deal's four cognitive lenses (i.e., frames). RESULTS: Fourteen of the 16 leaders used all cognitive frames. The human resource perspective was favored by all participants, followed closely by the symbolic (14/16) and political (14/16). Although most attended to the structural frame (14/16), only three placed any significant emphasis on it. In addition to identifying and describing the elements of this typology for medical education leadership, a new frame emerged of assessing interpersonal and work style in order to determine how to socially situate individuals. CONCLUSIONS: This study uniquely contributes by supporting the utility of the Bolman and Deal typology in the medical education context and supports the value for leaders to reflect on their organizational work from a variety of perspectives (including the frames). Medical education leadership development programs need to attend to enhancing the awareness of these perspectives.