Exploring virtual funding committee practices in the allocation of National Institute for Health and Care Research funding: A netnographic study.

Background: Funding committees, comprising members with a range of knowledge, skills, and experience, are considered integral to the decision-making process of funding organisations for recommending or allocating research funding. However, there is limited research investigating the decision-making processes, the role of members and their social interactions during funding committee meetings conducted both virtually and face-to-face. Methods: Using a mixed-methods design and following netnography principles, the study observed nine National Institute for Health and Care Research programmes funding committee meetings conducted virtually during October 2020 to December 2021; complemented by interviews with committee chairs and members (18 interviews) and NIHR staff (12 interviews); an online survey (50 responses); and documentary analysis. Personal reflections through immersive journals also formed part of the analysis. Results: Three main themes were identified from the observations, interviews, and online survey: efficiency of virtual committee meetings (importance of preparation, and the role of formality, process, and structure); understanding the effect of virtual committee meetings on well-being (effects of fatigue and apprehension, and the importance of work life balance); understanding social interactions and engagement (levels of engagement, contribution and inclusivity, awareness of unconscious bias and the value of social networking). Conclusions: Examining the decision-making practices of one funding organisation across several research programmes, across multiple committee meetings over one year has generated new insights around funding committee practices that previous studies have not been able to explore or investigate. Overall, it was observed that fair and transparent funding recommendations and outcomes can be achieved through virtual funding committees. However, whilst virtual funding committees have many benefits and opportunities, such as the potential to increase membership diversity and inclusivity, and be more environmentally sustainable, more evidence is needed to evaluate their effectiveness, with particular focus on issues of fatigue, engagement, and committee cohesion, especially when new committee members join.

A scoping review on what constitutes a good research culture.

Background: The crisis in research culture is well documented, covering issues such as a tendency for quantity over quality, unhealthy competitive environments, and assessment based on publications, journal prestige and funding. In response, research institutions need to assess their own practices to promote and advocate for change in the current research ecosystem. The purpose of the scoping review was to explore ' What does the evidence say about the 'problem' with 'poor' research culture, what are the benefits of 'good' research culture, and what does 'good' look like?' Aims: To examine the peer-reviewed and grey literature to explore the interplay between research culture, open research, career paths, recognition and rewards, and equality, diversity, and inclusion, as part of a larger programme of activity for a research institution. Methods: A scoping review was undertaken. Six databases were searched along with grey literature. Eligible literature had relevance to academic research institutions, addressed research culture, and were published between January 2017 to May 2022. Evidence was mapped and themed to specific categories. The search strategy, screening and analysis took place between April-May 2022. Results: 1666 titles and abstracts, and 924 full text articles were assessed for eligibility. Of these, 253 articles met the eligibility criteria for inclusion. A purposive sampling of relevant websites was drawn from to complement the review, resulting in 102 records included in the review. Key areas for consideration were identified across the four themes of job security, wellbeing and equality of opportunity, teamwork and interdisciplinary, and research quality and accountability. Conclusions: There are opportunities for research institutions to improve their own practice, however institutional solutions cannot act in isolation. Research institutions and research funders need to work together to build a more sustainable and inclusive research culture that is diverse in nature and supports individuals' well-being, career progression and performance.

Online survey exploring researcher experiences of research funding processes in the UK: the effort and burden of applying for funding and fulfilling reporting requirements.

OBJECTIVE: To explore researchers' experiences of funding processes, the effort and burden involved in applying for funding, obtaining funding and/or fulfilling reporting requirements with a UK health and social care research funder. DESIGN/SETTING: A cross-sectional online survey study with open (free-text) and closed questions (August to November 2021). PARTICIPANTS: Researchers with experience of applying for/obtaining funding and/or experience of fulfilling reporting requirements for UK health and social care research funded between January 2018 and June 2021. RESULTS: The survey was completed by 182 researchers, of which 176 had experience with applying for/obtaining funding, and 143 had experience with fulfilling reporting requirements during the timeframe. The majority of the 176 respondents (58%) completed between 7 and 13 key processes in order to submit an application and 69% felt that it was critically important to undertake these key processes. Respondents (n=143) reported submitting an average of 17 reports as part of research monitoring to a range of organisations (eg, funders, Higher Education Institutions). However, only 33% of respondents felt it was critically important to provide the requested reporting information to the different organisations. Thematic analysis of free-text questions on application and reporting identified themes relating to process inefficiencies including streamlining and alignment of systems, lack of understanding of processes including a need for improved communication and feedback from organisations with clear explanations about what information is needed, when and why, the support required by respondents and the time, effort and impact on workload and well-being. CONCLUSIONS: Through this study, we were able to identify funding processes that are considered by some to be effortful, but necessary, as well as those that were perceived as unnecessary, complex and repetitive, and may waste some researchers time and effort and impact on well-being. Possible solutions to increase efficiency and enhance value in these processes were identified.

The post-award effort of managing and reporting on funded research: a scoping review.

INTRODUCTION: Reporting is a mechanism for funding organisations to monitor and manage the progress, outputs, outcomes, and impacts of the research they fund. Inconsistent approaches to reporting and post-award management, and a growing demand for research information, can lead to perception of unnecessary administrative effort that impacts on decision-making and research activity. Identifying this effort, and what stakeholders see as unmet need for improvement, is crucial if funders and Higher Education Institutions (HEIs) are to streamline their practices and provide better support with reporting activities. In this review, we summarise the processes in post-award management, compare current practices, and explore the purpose of collecting information on funded research. We also identify areas where unnecessary effort is perceived and improvement is needed, using previously reported solutions to inform recommendations for funders and HEIs. METHODS: We conducted a scoping review of the relevant research and grey literature. Electronic searches of databases, and manual searches of journals and funder websites, resulted in inclusion of 52 records and 11 websites. Information on HEI and funder post-award management processes was extracted, catalogued, and summarised to inform discussion. RESULTS: Post-award management is a complex process that serves many purposes but requires considerable effort, particularly in the set up and reporting of research. Perceptions of unnecessary effort stem from inefficiencies in compliance, data management and reporting approaches, and there is evidence of needed improvement in mechanisms of administrative support, research impact assessment, monitoring, and evaluation. Solutions should focus on integrating digital systems to reduce duplication, streamlining reporting methods, and improving administrative resources in HEIs. CONCLUSIONS: Funders and HEIs should work together to support a more efficient post-award management process. The value of research information, and how it is collected and used, can be improved by aligning practices and addressing the specific issues highlighted in this review.

The use and acceptability of preprints in health and social care settings: A scoping review.

BACKGROUND: Preprints are open and accessible scientific manuscript or report that is shared publicly, through a preprint server, before being submitted to a journal. The value and importance of preprints has grown since its contribution during the public health emergency of the COVID-19 pandemic. Funders and publishers are establishing their position on the use of preprints, in grant applications and publishing models. However, the evidence supporting the use and acceptability of preprints varies across funders, publishers, and researchers. The scoping review explored the current evidence on the use and acceptability of preprints in health and social care settings by publishers, funders, and the research community throughout the research lifecycle. METHODS: A scoping review was undertaken with no study or language limits. The search strategy was limited to the last five years (2017-2022) to capture changes influenced by COVID-19 (e.g., accelerated use and role of preprints in research). The review included international literature, including grey literature, and two databases were searched: Scopus and Web of Science (24 August 2022). RESULTS: 379 titles and abstracts and 193 full text articles were assessed for eligibility. Ninety-eight articles met eligibility criteria and were included for full extraction. For barriers and challenges, 26 statements were grouped under four main themes (e.g., volume/growth of publications, quality assurance/trustworthiness, risks associated to credibility, and validation). For benefits and value, 34 statements were grouped under six themes (e.g., openness/transparency, increased visibility/credibility, open review process, open research, democratic process/systems, increased productivity/opportunities). CONCLUSIONS: Preprints provide opportunities for rapid dissemination but there is a need for clear policies and guidance from journals, publishers, and funders. Cautionary measures are needed to maintain the quality and value of preprints, paying particular attention to how findings are translated to the public. More research is needed to address some of the uncertainties addressed in this review.

An in-depth exploration of researcher experiences of time and effort involved in health and social care research funding in the UK: The need for changes.

The need to reform the way in which research is undertaken is clear, with reducing research bureaucracy and waste at the forefront of this issue for the UK government, funding organisations, higher education institutions and wider research community. The aim of this study was to describe researchers' experiences of the time, effort and burden involved in funding processes-namely applying for research funding and fulfilling reporting requirements. This was an in-depth qualitative study using semi-structured interviews with researchers who had experience applying for funding and/or completing reporting requirements for a UK health and social care research funder between January 2018 and June 2021. Following thematic analysis, five key themes were identified describing researcher experiences of key issues around time, efforts and burden associated with funding processes. These themes encompassed (1) issues with the current funding model for health and social care research, (2) time and effort involved in funding processes, (3) the need for a streamlined end-to-end process, (4) implications for work-life balance, and (5) addressing the need for better support and communication. The findings from this study describe researcher experiences of tasks in the research pathway that currently take considerable time and effort. It was clear that whilst some of this time and effort is considered necessary, some is exacerbated by inefficient and ineffective processes, such as perceived under-funding of research or lack of clarity with regards to funder expectations. This in turn contributes to unnecessary researcher burden, research waste and negative research culture. Better investment in health and social care research and in the researchers themselves who design and deliver the research, alongside improvements in transparency, streamlining and research support could ensure a more positive research culture, and improve the quality of funded research.

What works for peer review and decision-making in research funding: a realist synthesis.

INTRODUCTION: Allocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom. METHODS: Realist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding. RESULTS: We analysed 96 publications and 36 website sources. Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations (CMOCs) for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to write and review applications, promoted increased investment into innovation, and lowered cost of panels. CONCLUSIONS: Reports of 50 interventions in different areas of peer review provide useful guidance on ways of solving common issues with the peer review process. Evidence of the broader impact of these interventions on the research ecosystem is still needed, and future research should aim to identify processes that consistently work to improve peer review across funders and research contexts.

Identification and comparison of key criteria of feedback of funding decisions: mixed-methods analysis of funder and applicant perspectives.

OBJECTIVE: This study investigated the content, quality and value of feedback given to applicants who applied to one of four research programmes in the UK funded (or jointly funded) by the National Institute for Health Research (NIHR). METHODS: A mixed-method phased approach was conducted using document analysis and an online survey. Phase 1 examined 114 NIHR applicant feedback documents comprised written feedback from funding committees and external peer-reviewers and a conceptual framework of the key components of feedback was developed using content analysis. Phase 2 was an online survey completed by 113 NIHR applicants. Frequencies of responses to closed questions were calculated. Perceptions of quality and value of feedback were identified using content analysis of open-text responses. RESULTS: In phase 1, a conceptual framework was developed with seven overarching categories: 'Study structure and quality'; 'Team and infrastructure'; 'Acceptability to patients and professionals'; 'Study justification and design'; 'Risks and contingencies'; 'Outputs'; 'Value for money'. A higher frequency of feedback was provided at stage 2 and for successful applications across the majority of components. In phase 2, frequency data showed that opinion on feedback was dependent on funding outcome. Content analysis revealed four main themes: 'Committee transparency'; 'Content validity and reliability'; 'Additional support'; Recognition of effort and constraints'. CONCLUSIONS: This study provides key insights and understanding into the quality, content and value of feedback provided to NIHR applicants. The study identified key areas for improvement that can arise in NIHR funding applications, as well as in the feedback given to applicants that are applicable to other funding organisations. These findings could be used to inform funding application guidance documents to help researchers strengthen their applications and used more widely by other funders to inform their feedback processes.

Decision-making approaches used by UK and international health funding organisations for allocating research funds: A survey of current practice.

Innovations in decision-making practice for allocation of funds in health research are emerging; however, it is not clear to what extent these are used. This study aims to better understand current decision-making practices for the allocation of research funding from the perspective of UK and international health funders. An online survey (active March-April 2019) was distributed by email to UK and international health and health-related funding organisations (e.g., biomedical and social), and was publicised on social media. The survey collected information about decision-making approaches for research funding allocation, and covered assessment criteria, current and past practices, and considerations for improvements or future practice. A mixed methods analysis provided descriptive statistics (frequencies and percentages of responses) and an inductive thematic framework of key experiences. Thirty-one responses were analysed, representing government-funded organisations and charities in the health sector from the UK, Europe and Australia. Four themes were extracted and provided a narrative framework. 1. The most reported decision-making approaches were external peer review, triage, and face-to-face committee meetings; 2. Key values underpinned decision-making processes. These included transparency and gaining perspectives from reviewers with different expertise (e.g., scientific, patient and public); 3. Cross-cutting challenges of the decision-making processes faced by funders included bias, burden and external limitations; 4. Evidence of variations and innovations from the most reported decision-making approaches, including proportionate peer review, number of decision-points, virtual committee meetings and sandpits (interactive workshop). Broadly similar decision-making processes were used by all funders in this survey. Findings indicated a preference for funders to adapt current decision-making processes rather than using more innovative approaches: however, there is a need for more flexibility in decision-making and support to applicants. Funders indicated the need for information and empirical evidence on innovations which would help to inform decision-making in research fund allocation.

Using digital tools in the recruitment and retention in randomised controlled trials: survey of UK Clinical Trial Units and a qualitative study.

BACKGROUND: Recruitment and retention of participants in randomised controlled trials (RCTs) is a key determinant of success but is challenging. Trialists and UK Clinical Research Collaboration (UKCRC) Clinical Trials Units (CTUs) are increasingly exploring the use of digital tools to identify, recruit and retain participants. The aim of this UK National Institute for Health Research (NIHR) study was to identify what digital tools are currently used by CTUs and understand the performance characteristics required to be judged useful. METHODS: A scoping of searches (and a survey with NIHR funding staff), a survey with all 52 UKCRC CTUs and 16 qualitative interviews were conducted with five stakeholder groups including trialists within CTUs, funders and research participants. A purposive sampling approach was used to conduct the qualitative interviews during March-June 2018. Qualitative data were analysed using a content analysis and inductive approach. RESULTS: Responses from 24 (46%) CTUs identified that database-screening tools were the most widely used digital tool for recruitment, with the majority being considered effective. The reason (and to whom) these tools were considered effective was in identifying potential participants (for both Site staff and CTU staff) and reaching recruitment target (for CTU staff/CI). Fewer retention tools were used, with short message service (SMS) or email reminders to participants being the most reported. The qualitative interviews revealed five themes across all groups: 'security and transparency'; 'inclusivity and engagement'; 'human interaction'; 'obstacles and risks'; and 'potential benefits'. There was a high level of stakeholder acceptance of the use of digital tools to support trials, despite the lack of evidence to support them over more traditional techniques. Certain differences and similarities between stakeholder groups demonstrated the complexity and challenges of using digital tools for recruiting and retaining research participants. CONCLUSIONS: Our studies identified a range of digital tools in use in recruitment and retention of RCTs, despite the lack of high-quality evidence to support their use. Understanding the type of digital tools in use to support recruitment and retention will help to inform funders and the wider research community about their value and relevance for future RCTs. Consideration of further focused digital tool reviews and primary research will help to reduce gaps in the evidence base.

Role of feasibility and pilot studies in randomised controlled trials: a cross-sectional study.

OBJECTIVES: To assess the value of pilot and feasibility studies to randomised controlled trials (RCTs) funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme. To explore the methodological components of pilot/feasibility studies and how they inform full RCTs. STUDY DESIGN: Cross-sectional study. SETTING: Both groups included NIHR HTA programme funded studies in the period 1 January 2010-31 December 2014 (decision date). Group 1: stand-alone pilot/feasibility studies published in the HTA Journal or accepted for publication. Group 2: all funded RCT applications funded by the HTA programme, including reference to an internal and/or external pilot/feasibility study. The methodological components were assessed using an adapted framework from a previous study. MAIN OUTCOME MEASURES: The proportion of stand-alone pilot and feasibility studies which recommended proceeding to full trial and what study elements were assessed. The proportion of 'HTA funded' trials which used internal and external pilot and feasibility studies to inform the design of the trial. RESULTS: Group 1 identified 15 stand-alone pilot/feasibility studies. Study elements most commonly assessed were testing recruitment (100% in both groups), feasibility (83%, 100%) and suggestions for further study/investigation (83%, 100%). Group 2 identified 161 'HTA funded' applications: 59 cited an external pilot/feasibility study where testing recruitment (50%, 73%) and feasibility (42%, 73%) were the most commonly reported study elements: 92 reported an internal pilot/feasibility study where testing recruitment (93%, 100%) and feasibility (44%, 92%) were the most common study elements reported. CONCLUSIONS: 'HTA funded' research which includes pilot and feasibility studies assesses a variety of study elements. Pilot and feasibility studies serve an important role when determining the most appropriate trial design. However, how they are reported and in what context requires caution when interpreting the findings and delivering a definitive trial.

Identifying trial recruitment uncertainties using a James Lind Alliance Priority Setting Partnership - the PRioRiTy (Prioritising Recruitment in Randomised Trials) study.

BACKGROUND: Despite the problem of inadequate recruitment to randomised trials, there is little evidence to guide researchers on decisions about how people are effectively recruited to take part in trials. The PRioRiTy study aimed to identify and prioritise important unanswered trial recruitment questions for research. The PRioRiTy study - Priority Setting Partnership (PSP) included members of the public approached to take part in a randomised trial or who have represented participants on randomised trial steering committees, health professionals and research staff with experience of recruiting to randomised trials, people who have designed, conducted, analysed or reported on randomised trials and people with experience of randomised trials methodology. METHODS: This partnership was aided by the James Lind Alliance and involved eight stages: (i) identifying a unique, relevant prioritisation area within trial methodology; (ii) establishing a steering group (iii) identifying and engaging with partners and stakeholders; (iv) formulating an initial list of uncertainties; (v) collating the uncertainties into research questions; (vi) confirming that the questions for research are a current recruitment challenge; (vii) shortlisting questions and (viii) final prioritisation through a face-to-face workshop. RESULTS: A total of 790 survey respondents yielded 1693 open-text answers to 6 questions, from which 1880 potential questions for research were identified. After merging duplicates, the number of questions was reduced to 496. Questions were combined further, and those that were submitted by fewer than 15 people and/or fewer than 6 of the 7 stakeholder groups were excluded from the next round of prioritisation resulting in 31 unique questions for research. All 31 questions were confirmed as being unanswered after checking relevant, up-to-date research evidence. The 10 highest priority questions were ranked at a face-to-face workshop. The number 1 ranked question was "How can randomised trials become part of routine care and best utilise current clinical care pathways?" The top 10 research questions can be viewed at www.priorityresearch.ie . CONCLUSION: The prioritised questions call for a collective focus on normalising trials as part of clinical care, enhancing communication, addressing barriers, enablers and motivators around participation and exploring greater public involvement in the research process.

National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme research funding and UK burden of disease.

BACKGROUND: HTA Programme funding is governed by the need for evidence and scientific quality, reflecting funding of the National Institute for Health Research (NIHR) by the NHS. The need criterion incorporates covering the spectrum of diseases, but also taking account of research supported by other funders. This study compared the NIHR HTA Programme portfolio of research with the UK burden of disease as measured by Disability-adjusted Life Years (DALYs). METHODS: A retrospective cross-sectional study using a cohort of all funded primary research and evidence syntheses projects received by the HTA Programme from April 2011 to March 2016 (n = 363); to determine the proportion of spend by disease compared with burden of disease in the UK calculated using 2015 UK DALY data. RESULTS: The programme costing just under £44 million broadly reflected UK DALY burden by disease. Spend was lower than disease burden for cancer, cardiovascular and musculoskeletal diseases, which may reflect the importance of other funders, notably medical charities, which concentrate on these diseases. CONCLUSION: The HTA Programme spend, adjusted for other relevant funders, broadly matches disease burden in the UK; no diseases are being neglected.

The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews.

BACKGROUND: We sought to analyse the impacts found, and the methods used, in a series of assessments of programmes and portfolios of health research consisting of multiple projects. METHODS: We analysed a sample of 36 impact studies of multi-project research programmes, selected from a wider sample of impact studies included in two narrative systematic reviews published in 2007 and 2016. We included impact studies in which the individual projects in a programme had been assessed for wider impact, especially on policy or practice, and where findings had been described in such a way that allowed them to be collated and compared. RESULTS: Included programmes were highly diverse in terms of location (11 different countries plus two multi-country ones), number of component projects (8 to 178), nature of the programme, research field, mode of funding, time between completion and impact assessment, methods used to assess impact, and level of impact identified. Thirty-one studies reported on policy impact, 17 on clinician behaviour or informing clinical practice, three on a combined category such as policy and clinician impact, and 12 on wider elements of impact (health gain, patient benefit, improved care or other benefits to the healthcare system). In those multi-programme projects that assessed the respective categories, the percentage of projects that reported some impact was policy 35% (range 5-100%), practice 32% (10-69%), combined category 64% (60-67%), and health gain/health services 27% (6-48%). Variations in levels of impact achieved partly reflected differences in the types of programme, levels of collaboration with users, and methods and timing of impact assessment. Most commonly, principal investigators were surveyed; some studies involved desk research and some interviews with investigators and/or stakeholders. Most studies used a conceptual framework such as the Payback Framework. One study attempted to assess the monetary value of a research programme's health gain. CONCLUSION: The widespread impact reported for some multi-project programmes, including needs-led and collaborative ones, could potentially be used to promote further research funding. Moves towards greater standardisation of assessment methods could address existing inconsistencies and better inform strategic decisions about research investment; however, unresolved issues about such moves remain.

Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme.

BACKGROUND: This report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review. OBJECTIVES: (1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme. DATA SOURCES: We searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014. REVIEW METHODS: This narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015. RESULTS: The literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers. DISCUSSION: The findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence's remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities' research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish® (researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established. LIMITATIONS: There were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme. CONCLUSIONS: Research funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines. FUNDING: The National Institute for Health Research HTA programme.