An assessment of the impact of the NHS Health Technology Assessment Programme.

OBJECTIVES: To consider how the impact of the NHS Health Technology Assessment (HTA) Programme should be measured. To determine what models are available and their strengths and weaknesses. To assess the impact of the first 10 years of the NHS HTA programme from its inception in 1993 to June 2003 and to identify the factors associated with HTA research that are making an impact. DATA SOURCES: Main electronic databases from 1990 to June 2005. The documentation of the National Coordinating Centre for Health Technology Assessment (NCCHTA). Questionnaires to eligible researchers. Interviews with lead investigators. Case study documentation. REVIEW METHODS: A literature review of research programmes was carried out, the work of the NCCHTA was reviewed, lead researchers were surveyed and 16 detailed case studies were undertaken. Each case study was written up using the payback framework. A cross-case analysis informed the analysis of factors associated with achieving payback. Each case study was scored for impact before and after the interview to assess the gain in information due to the interview. The draft write-up of each study was checked with each respondent for accuracy and changed if necessary. RESULTS: The literature review identified a highly diverse literature but confirmed that the 'payback' framework pioneered by Buxton and Hanney was the most widely used and most appropriate model available. The review also confirmed that impact on knowledge generation was more easily quantified than that on policy, behaviour or especially health gain. The review of the included studies indicated a higher level of impact on policy than is often assumed to occur. The survey showed that data pertinent to payback exist and can be collected. The completed questionnaires showed that the HTA Programme had considerable impact in terms of publications, dissemination, policy and behaviour. It also showed, as expected, that different parts of the Programme had different impacts. The Technology Assessment Reports (TARs) for the National Institute for Health and Clinical Excellence (NICE) had the clearest impact on policy in the form of NICE guidance. Mean publications per project were 2.93 (1.98 excluding the monographs), above the level reported for other programmes. The case studies revealed the large diversity in the levels and forms of impacts and the ways in which they arise. All the NICE TARs and more than half of the other case studies had some impact on policy making at the national level whether through NICE, the National Screening Committee, the National Service Frameworks, professional bodies or the Department of Health. This underlines the importance of having a customer or 'receptor' body. A few case studies had very considerable impact in terms of knowledge production and in informing national and international policies. In some of these the principal investigator had prior expertise and/or a research record in the topic. The case studies confirmed the questionnaire responses but also showed how some projects led to further research. CONCLUSIONS: This study concluded that the HTA Programme has had considerable impact in terms of knowledge generation and perceived impact on policy and to some extent on practice. This high impact may have resulted partly from the HTA Programme's objectives, in that topics tend to be of relevance to the NHS and have policy customers. The required use of scientific methods, notably systematic reviews and trials, coupled with strict peer reviewing, may have helped projects publish in high-quality peer-reviewed journals. Further research should cover more detailed, comprehensive case studies, as well as enhancement of the 'payback framework'. A project that collated health research impact studies in an ongoing manner and analysed them in a consistent fashion would also be valuable.

Identifying the impact of diabetes research.

AIMS: Assessment of the impact of health research is a growing but problematic field. We examined how a combination of approaches might together inform assessment of the impact of a body of diabetes research published in 1981 and help identify factors behind success. METHODS: Three broad approaches were applied to the work of one team leader of acknowledged influence. Standard bibliographic analysis was complemented by a second approach which categorized the importance of the primary publications to the papers citing them, in four domains. In parallel, a third approach involved qualitative assessment using surveys, critical pathway analysis by, and interviews of, co-authors and external experts. Extending the approach incorporated key additional publications from other years. RESULTS: In 1981, the team leader published 29 papers. Citations to these 29 first generation papers varied from 1 to 76 and resulted in 799 second generation papers. Citations to these produced 12 891 third generation papers. Analysis of second generation papers suggested the cited first generation paper was thought to be of considerable or essential significance in only 9% of cases. While much research made little impact, qualitative analysis included a wealth of information, sometimes missed by standard bibliographic techniques, on where the identified research influenced important streams of clinical development. Analysis covered major research studies (such as the Diabetes Control and Complications Trial), insulin pump therapy, and career development of co-authors. CONCLUSIONS: Understanding the impact of research requires multiple approaches. With refinement, these techniques could be employed more widely and potentially could inform research policy.

Payback arising from research funding: evaluation of the Arthritis Research Campaign.

OBJECTIVES: Using a structured evaluation framework to systematically review and document the outputs and outcomes of research funded by the Arthritis Research Campaign in the early 1990s. To illustrate the strengths and weaknesses of different modes of research funding. METHODS: The payback framework was applied to 16 case studies of research grants funded in the early 1990s. Case study methodology included bibliometric analysis, literature and archival document review and key informant interviews. RESULTS: A range of research paybacks was identified from the 16 research grants. The payback included 302 peer-reviewed papers, postgraduate training and career development, including 28 PhD/MDs, research informing recommendations in clinical guidelines, improved quality of life for people with RA and the reduction of the likelihood of recurrent miscarriage for women with antiphospholipid syndrome. The payback arising from project grants appeared to be similar to that arising from other modes of funding that were better resourced. CONCLUSIONS: There is a wide diversity of research payback. Short focused project grants seem to provide value for money.

Routine monitoring of performance: what makes health research and development different?

Increasing attention is being directed to measuring and monitoring the use of health-related R&D funding, partly to justify this expenditure and partly to ensure that R&D effort is directed to achieving the paybacks desired by funders. These paybacks include contributing to knowledge, contributing to R&D capacity, political benefits, benefits to the health service and to patients, and more general economic benefits. This paper addresses the issues that must be considered when designing a routine performance management system for health R&D. Conventional methods of routine performance management are often rendered inappropriate in this context by the intangible and unpredictable outcomes of research, which are heterogeneous across projects and programmes and which can be hard to attribute to particular R&D support. Instead, to be effective in this context, a routine system must combine quantitative and qualitative indicators, utilising information from a number of different sources. The system must achieve acceptable levels (defined by the funder) on each of the following criteria: it must measure those dimensions of payback that are valued by the funder; it must be decision-relevant; it must be consistent with truthful compliance; it must minimise perverse incentives; and it must have acceptable net costs. It is vitally important that the system itself generates a positive payback. We illustrate these issues by outlining a system that might be used to monitor the payback from government-funded R&D.

How can payback from health services research be assessed?

Throughout the world there is a growing recognition that health care should be research-led. This strengthens the requirement for expenditure on health services research to be justified by demonstrating the benefits it produces. However, payback from health research and development is a complex concept and little used term. Five main categories of payback can be identified: knowledge; research benefits; political and administrative benefits; health sector benefits; and broader economic benefits. Various models of research utilization together with previous assessments of payback from research helped in the development of a new conceptual model of how and where payback may occur. The model combines an input-output perspective with an examination of the permeable interfaces between research and its environment. The model characterizes research projects in terms of Inputs, Processes, and Primary Outputs. The last consist of knowledge and research benefits. There are two interfaces between the project and its environment. The first (Project Specification, Selection and Commissioning) is the link with Research Needs Assessment. The second (Dissemination) should lead to Secondary Outputs (which are policy or administrative decisions), and usually Applications (which take the form of behavioural changes), from which Impacts or Final Outcomes result. It is at this final stage that health and wider economic benefits can be measured. A series of case studies were used to assess the feasibility both of applying the model and the payback categorization. The paper draws various conclusions from the case studies and identifies a range of issues for further work.