Monitoring Open Science as transformative change: Towards a systemic framework.

Following a flurry of policies for Open Science (OS), there is now a wave of initiatives to monitor its adoption. However, the great diversity of understandings and activities related to Open Science makes monitoring very challenging. There is a danger that by focusing on what can be readily observed (e.g. publications) many other OS activities are overlooked (e.g. participation), with a potential narrowing of OS scope, streetlight effects, and deviation from the values of OS. Since Open Science can be understood as a systemic transformation of the research system, we have borrowed concepts from Transformative Innovation Policies frameworks which aim at evaluating socio-technical transitions. In accordance with this view of OS as a systemic transformation, we propose that the new monitoring efforts should shift towards: (i) systemic perspectives which considers the various actions related to OS, including policies and outputs (e.g. datasets) but also processes (e.g. participatory events), outcomes (e.g. citizen interest in science) and expected impacts (e.g. better scientific contributions to addressing societal problems); (ii) implementation of monitoring as reflexive learning (rather than accountability or benchmarking); (iii) mapping the directionality of the activities and the values associated with the choices in directions. In summary, a monitoring framework for OS requires a profound change in conventional monitoring practices. The scope should broaden from current focus on outputs (such as publications) towards the processes of connection that make science 'open' (usage, co-creation and dialogue), as well as towards outcomes (changes in practices) and the longer-term impacts that reflect the values and normative commitments of OS.

The Societal Readiness Thinking Tool: A Practical Resource for Maturing the Societal Readiness of Research Projects.

In this paper, we introduce the Societal Readiness (SR) Thinking Tool to aid researchers and innovators in developing research projects with greater responsiveness to societal values, needs, and expectations. The need for societally-focused approaches to research and innovation-complementary to Technology Readiness (TR) frameworks-is presented. Insights from responsible research and innovation (RRI) concepts and practice, organized across critical stages of project-life cycles are discussed with reference to the development of the SR Thinking Tool. The tool is designed to complement not only shortfalls in TR approaches, but also improve upon other efforts to integrate RRI, sustainability, and design thinking in research and innovation cycles. Operationalization and early-stage user tests of the Tool are reported, along with discussion of potential future iterations and applications.

"Be an ambassador for change that you would like to see": a call to action to all stakeholders for co-creation in healthcare and medical research to improve quality of life of people with a neuromuscular disease.

BACKGROUND: Patient and public involvement for co-creation is increasingly recognized as a valuable strategy to develop healthcare research targeting patients' real needs. However, its practical implementation is not as advanced and unanimously accepted as it could be, due to cultural differences and complexities of managing healthcare programs and clinical studies, especially in the rare disease field. MAIN BODY: The European Neuromuscular Centre, a European foundation of patient organizations, involved its key stakeholders in a special workshop to investigate the position of the neuromuscular patient community with respect to healthcare and medical research to identify and address gaps and bottlenecks. The workshop took place in Milan (Italy) on January 19-20, 2018, involving 45 participants who were mainly representatives of the patient community, but also included experts from clinical centers, industry and regulatory bodies. In order to provide practical examples and constructive suggestions, specific topics were identified upfront. The first set of issues concerned the quality of life at specific phases of a patient's life, such as at the time of diagnosis or during pediatric to adult transition, and patient involvement in medical research on activities in daily living including patient reported outcome measures. The second set of issues concerned the involvement of patients in the management of clinical research tools, such as registries and biobanks, and their participation in study design or marketing authorization processes. Introductory presentations were followed by parallel working group sessions, to gain constructive contributions from all participants. The concept of shared decision making was used to ensure, in discussions, a partnership-based identification of the wishes and needs of all stakeholders involved, and the "ladder of participation" tool served as a model to evaluate the actual and the desired level of patients' involvement in all topics addressed. A general consensus on the outcome of the meeting was collected during the final plenary session. This paper reports the outcome of the workshop and the specific suggestions derived from the analysis of the first set of topics, related to quality of life. The outcomes of the second set of topics are reported elsewhere and are only briefly summarized herein for the sake of completeness. CONCLUSIONS: The neuromuscular community proved to be very active and engaged at different levels in the healthcare initiatives of interest. The workshop participants critically discussed several topics, providing practical examples where different stakeholders could play a role in making a change and bridging gaps. Overall, they indicated the need for education of all stakeholders for better communication, where everyone should become an ambassador to promote real change. Support should also come from institutions and healthcare bodies both at structural and economic level.

"The impact of European Neuromuscular Centre (ENMC) workshops on the neuromuscular field; 25 years on ".

Since 1992, the European Neuromuscular Centre facilitated workshops to bring experts in the field of neuromuscular disorders together. After organising more than 235 workshops, it is time to evaluate what impact these 25 years of ENMC workshops have had on the neuromuscular research field and on people affected by a neuromuscular condition. To measure this, workshop topics were retrospectively evaluated and bibliometric analyses on the citation scores of ENMC-derived publications were performed. In addition, a personalized survey was used to investigate the actual achievement and implementation of workshop deliverables. The evaluation of 25 years' workshop topics revealed a strong representation of muscular dystrophies, congenital and mitochondrial myopathies. The publications derived from ENMC workshops scored "high impact" as illustrated by the Mean Normalized Citation Score of 1.24. Also 16% of the ENMC papers belong to the top 10% best cited articles in the neuromuscular field. The main outcome of the personalised survey was that 90% of all workshop deliverables were started and either ongoing or completed. Of these deliverables, 78% were implemented in the field; bringing state-of-the-art knowledge and new collaborations to researchers and clinicians, improving designs of clinical trials and innovating tools to make accurate diagnoses.

The Position of Neuromuscular Patients in Shared Decision Making. Report from the 235th ENMC Workshop: Milan, Italy, January 19-20, 2018.

In the era of patient-centered medicine, shared decision-making (SDM) - in which healthcare professionals and patients exchange information and preferences and jointly reach a decision - has emerged as the gold standard model for the provision of formal healthcare. Indeed, in many geographical settings, patients are frequently invited to participate in choices concerning the design and delivery of their medical management. From a clinical perspective, benefits of this type of patient involvement encompass, for example, enhanced treatment satisfaction, improved medical compliance, better health outcomes, and maintained or promoted quality of life. Yet, although the theory and enactment of SDM in healthcare are well-described in the literature [1-3], comparatively less attention has been devoted to contextualizing questions relating to if, when, and how to include patients in decisions within medical research. In this context, patient involvement would be expected to be potentially relevant for and applicable to a wide range of activities and processes, from the identification of research priorities and development of grant applications, to the design of patient information and consent procedures, formulation of interventions, identification and recruitment of study sample populations, feasibility of a clinical trial, identification, selection, and specification of endpoints and outcomes in clinical trials and observational studies, data collection and analysis, and dissemination of results. To this end, 45 clinicians, healthcare professionals, researchers, patients, caregivers, and representatives from regulatory authorities and pharmaceutical companies from 15 different countries met to discuss the level of involvement of patients with neuromuscular diseases, specifically in the following settings of medical research for neuromuscular diseases: i) registries and biobanks; ii) clinical trials; and iii) regulatory processes. In this report, we present summaries of the talks that were given during the workshop, as well as discussion outcomes from the three topic areas listed above.

Improving the evaluation of worldwide biomedical research output: classification method and standardised bibliometric indicators by disease.

OBJECTIVE: Since most biomedical research focuses on a specific disease, evaluation of research output requires disease-specific bibliometric indicators. Currently used methods are insufficient. The aim of this study is to develop a method that enables detailed analysis of worldwide biomedical research output by disease. DESIGN: We applied text mining techniques and analysis of author keywords to link publications to disease groups. Fractional counting was used to quantify disease-specific biomedical research output of an institution or country. We calculated global market shares of research output as a relative measure of publication volume. We defined 'top publications' as the top 10% most cited publications per disease group worldwide. We used the percentage of publications from an institution or country that were top publications as an indicator of research quality. RESULTS: We were able to classify 54% of all 6.5 million biomedical publications in our database (based on Web of Science) to a disease group. We could classify 78% of these publications to a specific institution. We show that between 2000 and 2012,'other infectious diseases' were the largest disease group with 337 485 publications. Lifestyle diseases, cancers and mental disorders have grown most in research output. The USA was responsible for the largest number of top 10% most cited publications per disease group, with a global share of 45%. Iran (+3500%) and China (+700%) have grown most in research volume. CONCLUSIONS: The proposed method provides a tool to assess biomedical research output in new ways. It can be used for evaluation of historical research performance, to support decision-making in management of research portfolios, and to allocate research funding. Furthermore, using this method to link disease-specific research output to burden of disease can contribute to a better understanding of the societal impact of biomedical research.

Societal output and use of research performed by health research groups.

The last decade has seen the evaluation of health research pay more and more attention to societal use and benefits of research in addition to scientific quality, both in qualitative and quantitative ways. This paper elaborates primarily on a quantitative approach to assess societal output and use of research performed by health research groups (societal quality of research). For this reason, one of the Dutch university medical centres (i.e. the Leiden University Medical Center (LUMC)) was chosen as the subject of a pilot study, because of its mission to integrate top patient care with medical, biomedical and healthcare research and education. All research departments were used as units of evaluation within this university medical centre.The method consisted of a four-step process to reach a societal quality score per department, based on its (research) outreach to relevant societal stakeholders (the general public, healthcare professionals and the private sector). For each of these three types of stakeholders, indicators within four modes of communication were defined (knowledge production, knowledge exchange, knowledge use and earning capacity). These indicators were measured by a bottom-up approach in a qualitative way (i.e. all departments of the LUMC were asked to list all activities they would consider to be of societal relevance), after which they were converted into quantitative scores. These quantitative scores could then be compared to standardised scientific quality scores that are based on scientific publications and citations of peer-reviewed articles.Based on the LUMC pilot study, only a weak correlation was found between societal and scientific quality. This suggests that societal quality needs additional activities to be performed by health research groups and is not simply the consequence of high scientific quality. Therefore we conclude that scientific and societal evaluation should be considered to be synergistic in terms of learning for the future, accountability and advocacy.This quantitative approach to assess societal quality in a quantitative sense is based on indicators that function as proxies for society quality on different levels, based on the communication of researchers with their societal stakeholders (i.e. knowledge production, knowledge exchange and knowledge use). The methodology presented is just a first attempt to compare scientific quality scores (publication and citation scores) with societal quality scores in a quantitative way. This comparison can be used by organisations (e.g. university medical centres) in their planning and control cycle.