The Use of Evidence to Design an Essential Package of Health Services in Pakistan: A Review and Analysis of Prioritisation Decisions at Different Stages of the Appraisal Process.

BACKGROUND: Pakistan embarked on a process of designing an essential package of health services (EPHS) as a pathway towards universal health coverage (UHC). The EPHS design followed an evidence-informed deliberative process; evidence on 170 interventions was introduced along multiple stages of appraisal engaging different stakeholders tasked with prioritising interventions for inclusion. We report on the composition of the package at different stages, analyse trends of prioritised and deprioritised interventions and reflect on the trade-offs made. METHODS: Quantitative evidence on cost-effectiveness, budget impact, and avoidable burden of disease was presented to stakeholders in stages. We recorded which interventions were prioritised and deprioritised at each stage and carried out three analyses: (1) a review of total number of interventions prioritised at each stage, along with associated costs per capita and disability-adjusted life years (DALYs) averted, to understand changes in affordability and efficiency in the package, (2) an analysis of interventions broken down by decision criteria and intervention characteristics to analyse prioritisation trends across different stages, and (3) a description of the trajectory of interventions broken down by current coverage and cost-effectiveness. RESULTS: Value for money generally increased throughout the process, although not uniformly. Stakeholders largely prioritised interventions with low budget impact and those preventing a high burden of disease. Highly cost-effective interventions were also prioritised, but less consistently throughout the stages of the process. Interventions with high current coverage were overwhelmingly prioritised for inclusion. CONCLUSION: Evidence-informed deliberative processes can produce actionable and affordable health benefit packages. While cost-effective interventions are generally preferred, other factors play a role and limit efficiency.

The use of cost-effectiveness analysis for health benefit package design - should countries follow a sectoral, incremental or hybrid approach?

BACKGROUND: Countries around the world are increasingly rethinking the design of their health benefit package to achieve universal health coverage. Countries can periodically revise their packages on the basis of sectoral cost-effectiveness analyses, i.e. by evaluating a broad set of services against a 'doing nothing' scenario using a budget constraint. Alternatively, they can use incremental cost-effectiveness analyses, i.e. to evaluate specific services against current practice using a threshold. In addition, countries may employ hybrid approaches which combines elements of sectoral and incremental cost-effectiveness analysis - a country may e.g. not evaluate the comprehensive set of all services but rather relatively small sets of services targeting a certain condition. However, there is little practical guidance for countries as to which kind of approach they should follow. METHODS: The present study was based on expert consultation. We refined the typology of approaches of cost-effectiveness analysis for benefit package design, identified factors that should be considered in the choice of approach, and developed recommendations. We reached consensus among experts over the course of several review rounds. RESULTS: Sectoral cost-effectiveness analysis is especially suited in contexts with large allocative inefficiencies in current service provision and can, in theory, realize large efficiency gains. However, it may be challenging to implement a comprehensive redesign of the package in practice. Incremental cost-effectiveness analysis is especially relevant in contexts where specific new services may impact the sustainability of the health system. It may potentially support efficiency improvement, but its focus has typically been on new services while existing inefficiencies remain unchallenged. The use of hybrid approach may be a way forward to address the strengths and weaknesses of sectoral and incremental analysis areas. Such analysis may be especially useful to target disease areas with suspected high inefficiencies in service provision, and would then make good use of the available research capacity and be politically rewarding. However, disease-specific analyses bear the risk of not addressing resource allocation inefficiencies across disease areas. CONCLUSIONS: Countries should carefully select their approach of cost-effectiveness analyses for benefit package design, based on their decision-making context.

Building implementable packages for universal health coverage.

Since no country or health system can provide every possible health service to everyone who might benefit, the prioritisation of a defined subset of services for universal availability is intrinsic to universal health coverage (UHC). Creating a package of priority services for UHC, however, does not in itself benefit a population-packages have impact only through implementation. There are inherent tensions between the way services are formulated to facilitate criteria-driven prioritisation and the formulations that facilitate implementation, and service delivery considerations are rarely well incorporated into package development. Countries face substantial challenges bridging from a list of services in a package to the elements needed to get services to people. The failure to incorporate delivery considerations already at the prioritisation and design stage can result in packages that undermine the goals that countries have for service delivery. Based on a range of country experiences, we discuss specific choices about package structure and content and summarise some ideas on how to build more implementable packages of services for UHC, arguing that well-designed packages can support countries to bridge effectively from intent to implementation.

Decision-making processes for essential packages of health services: experience from six countries.

Many countries around the world strive for universal health coverage, and an essential packages of health services (EPHS) is a central policy instrument for countries to achieve this. It defines the coverage of services that are made available, as well as the proportion of the costs that are covered from different financial schemes and who can receive these services. This paper reports on the development of an analytical framework on the decision-making process of EPHS revision, and the review of practices of six countries (Afghanistan, Ethiopia, Pakistan, Somalia, Sudan and Zanzibar-Tanzania).The analytical framework distinguishes the practical organisation, fairness and institutionalisation of decision-making processes. The review shows that countries: (1) largely follow a similar practical stepwise process but differ in their implementation of some steps, such as the choice of decision criteria; (2) promote fairness in their EPHS process by involving a range of stakeholders, which in the case of Zanzibar included patients and community members; (3) are transparent in terms of at least some of the steps of their decision-making process and (4) in terms of institutionalisation, express a high degree of political will for ongoing EPHS revision with almost all countries having a designated governing institute for EPHS revision.We advise countries to organise meaningful stakeholder involvement and foster the transparency of the decision-making process, as these are key to fairness in decision-making. We also recommend countries to take steps towards the institutionalisation of their EPHS revision process.

The Use of Evidence-Informed Deliberative Processes for Health Benefit Package Design in Kazakhstan.

Kazakhstan strives to obtain Universal Health Coverage (UHC) by using health technology assessment (HTA) for determining their health benefit package. This paper reports on employing evidence-informed deliberative processes (EDPs), a practical and stepwise approach to enhance legitimate health benefit package design in Kazakhstan. METHODS: The Ministry of Health of Kazakhstan approved the operationalization and application of EDPs during March 2019 and December 2020. We used a combination of desk research, conducting HTA, online surveys as well as a face-to-face workshop in Nur-Sultan, Kazakhstan, and two online workshops to prioritize 25 selected health technologies. During the latter, we tested two alternative approaches to prioritization: quantitative multicriteria decision analysis (MCDA) and the use of decision rules. RESULTS: For each of the HTA reports, evidence summaries were developed according to the decision criteria (safety, social priority disease, severity of disease, effectiveness, cost-effectiveness, level of evidence, financial risk protection and budget impact). When appraising the evidence, the advisory committee preferred using quantitative MCDA, and only when this would result in any controversy could decision rules be applied. CONCLUSIONS: Despite several challenges, including a partial disruption because of the COVID-19 pandemic, implementation of the process will likely play a key role in determining an evidence-informed and transparent health benefit package.

Exploring the economic impact of inappropriate antibiotic use: the case of upper respiratory tract infections in Ghana.

BACKGROUND: Antibiotic consumption is increasing worldwide, particularly in low and middle-income countries (LMICs). Access to lifesaving antibiotics in LMICs is crucial while minimising inappropriate use. Studies assessing the economic impact of inappropriate antibiotic use in LMICs are lacking. We explored the economic impact of inappropriate antibiotic use using the example of upper respiratory tract infections (URIs) in Ghana, as part of the ABACUS (AntiBiotic ACcess and USe) project. METHODS: A top-down, retrospective economic impact analysis of inappropriate antibiotic use for URIs was conducted. Two inappropriate antibiotic use situations were considered: (1) URIs treated with antibiotics, against recommendations from clinical guidelines; and (2) URIs that should have been treated with antibiotics according to clinical guidelines, but were not. The analysis included data collected in Ghana during the ABACUS project (household surveys and exit-interviews among consumers buying antibiotics), scientific literature and stakeholder consultations. Included cost types related to health care seeking behaviour for URIs. Additionally, cost saving projections were computed based on potential effects of future interventions that improve antibiotic use. RESULTS: Health care costs related to inappropriate antibiotic use for URIs were estimated to be around 20 million (M) USD annually, including 18 M USD for situation 1 and 2 M USD for situation 2. Travel costs and lost income due to travel, together, were estimated to be around 44 M USD for situation 1 and 18 M USD for situation 2. Possible health care cost savings range from 2 to 12 M USD for situation 1 and from 0.2 to 1 M USD for situation 2. CONCLUSIONS: This study indicates that inappropriate antibiotic use leads to substantial economic costs in a LMIC setting that could have been prevented. We recommend investment in novel strategies to counter these unnecessary expenditures. As the projections indicate, this may result in considerable cost reductions. By tackling inappropriate use, progress can be made in combatting antibiotic resistance.

The Use of Evidence-Informed Deliberative Processes for Health Insurance Benefit Package Revision in Iran.

BACKGROUND: Iran considers the revision of its health insurance benefit package (HIBP) as a means to achieve universal health coverage (UHC). Yet, its decision-making process has been criticised for being weak in terms of accountability and transparency. This paper reports on the development and implementation of the HIBP revision in Iran in the period 2019-2021, employing evidence-informed deliberative processes (EDPs), a framework for benefit package design with the explicit aim of optimising the legitimacy of decision-making. METHODS: The High Council for Health Insurance (HCHI) is coordinating the HIBP revision: it planned the six steps of the EDP framework with support from World Health Organization (WHO) and Radboudumc in 2019, and conducted a pilot project on multiple sclerosis (MS) diagnosis and treatment in 2020. RESULTS: Implementation of the MS pilot project concerned the installation of advisory committees (involving some 60 stakeholders in supportive task forces, a technical working group [TWG] and a national advisory committee [NAC]), the selection of decision criteria (relating to quality of care, necessity, and sustainability), the inclusion of services for evaluation (nine in total), and the assessment and appraisal of these services. CONCLUSION: Implementation of the priority setting process for MS diagnosis and treatment services has likely improved the legitimacy of decision-making by involving stakeholders who engaged in deliberation based on available evidence in a stepwise, transparent process. It is expected to improve the quality of care for MS patients as well as its financial accessibility, at a zero net budget impact. The pilot project has served to help Iran's health system move faster toward UHC for a broader range of essential health services.

The Dutch Citizen Forum on Public Reimbursement of Healthcare: A Qualitative Analysis of Opinion Change.

BACKGROUND: A deliberative Citizen Forum 'Choices in healthcare' was held in the Netherlands to obtain insight into the criteria informed citizens would propose for the public reimbursement of healthcare. During 3 weekends, 24 citizens participated in evidence-informed deliberation on the basis of 8 case studies. The aim of this study was to assess how the opinions of 8 participants in the deliberative Citizens Forum changed and if so, why participants themselves believe their opinions have changed, whether participation influenced their perceived reasonableness of other participants in the forum and whether it influenced their opinions about involvement of citizens in decision-making. METHODS: Semi-structured interviews were held with 8 participants before and after their participation in the Citizen Forum. Using the method of reconstructing interpretive frames opinions about the public reimbursement of healthcare were reconstructed. RESULTS: Participants' opinions changed over time; they became more aware of the complexity of decision-making and came to accept that there are limits to the available resources and accept cost as a criterion for reimbursement decisionmaking. Participants report that exchanging arguments and personal experiences with other participants made them change their initial opinions. Participants ascribed increases in the perceived reasonableness of other participants' opinions to feelings of group-bonding and becoming more familiar with each other's personal circumstances. Participants further believe that citizens represent an additional opinion to that of other stakeholders and believe their opinions should be considered in relation to those of other stakeholders, given they are provided with opportunities for critical discussion. CONCLUSION: Organized deliberation should allow for the exchange of arguments and the sharing of personal experiences which is linked to learning. On the one hand this is reflected in the uptake of new arguments and on the other hand in the revision, specification or expansion of personal argumentation. Providing opportunities for critical deliberation is key to prevent citizens from adhering to initial emotional reactions that remain unchallenged and which may no longer be supported after deliberation.

Evidence-Informed Deliberative Processes for Legitimate Health Benefit Package Design - Part I: Conceptual Framework.

BACKGROUND: Countries around the world are increasingly rethinking the design of their health benefit packages to achieve universal health coverage (UHC). Health technology assessment (HTA) bodies support governments in these decisions, but employ value frameworks that do not sufficiently account for the intrinsically complex and value-laden political reality of benefit package design. METHODS: Several years ago, evidence-informed deliberative processes (EDPs) were developed to address this issue. An EDP is a practical and stepwise approach for HTA bodies to enhance legitimate health benefit package design based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of values, and to interpret available evidence on these values. We further developed the conceptual framework and initial 2019 guidance based on academic knowledge exchange, analysing practices of HTA bodies, surveying HTA bodies and experts around the globe, and implementation of EDPs in several countries around the world. RESULTS: EDPs stem from the general concept of legitimacy, which is translated into four elements - stakeholder involvement ideally operationalised through stakeholder participation with deliberation; evidence-informed evaluation; transparency; and appeal. The 2021 practical guidance distinguishes six practical steps of a HTA process and provides recommendations on how these elements can be implemented in each of these steps. CONCLUSION: There is an increased attention for legitimacy, deliberative processes for HTA and health benefit package design, but the development of theories and methods for such processes remain behind. The added value of EDPs lies in the operationalisation of the general concept of legitimacy into practical guidance for HTA bodies.

Evidence-Informed Deliberative Processes for Health Benefit Package Design - Part II: A Practical Guide.

BACKGROUND: Countries around the world are using health technology assessment (HTA) for health benefit package design. Evidence-informed deliberative processes (EDPs) are a practical and stepwise approach to enhance legitimate health benefit package design based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. This paper reports on the development of practical guidance on EDPs, while the conceptual framework of EDPs is described in a companion paper. METHODS: The first guide on EDPs (2019) is further developed based on academic knowledge exchange, surveying 27 HTA bodies and 66 experts around the globe, and the implementation of EDPs in several countries. We present the revised steps of EDPs and how selected HTA bodies (in Australia, Brazil, Canada, France, Germany, Scotland, Thailand and the United Kingdom) organize key issues of legitimacy in their processes. This is based on a review of literature via PubMed and HTA bodies' websites. RESULTS: HTA bodies around the globe vary considerable in how they address legitimacy (stakeholder involvement ideally through participation with deliberation; evidence-informed evaluation; transparency; and appeal) in their processes. While there is increased attention for improving legitimacy in decision-making processes, we found that the selected HTA bodies are still lacking or just starting to develop activities in this area. We provide recommendations on how HTA bodies can improve on this. CONCLUSION: The design and implementation of EDPs is in its infancy. We call for a systematic analysis of experiences of a variety of countries, from which general principles on EDPs might subsequently be inferred.

Financing of surgery and anaesthesia in sub-Saharan Africa: a scoping review.

OBJECTIVE: This study aimed to provide an overview of current knowledge and situational analysis of financing of surgery and anaesthesia across sub-Saharan Africa (SSA). SETTING: Surgical and anaesthesia services across all levels of care-primary, secondary and tertiary. DESIGN: We performed a scoping review of scientific databases (PubMed, EMBASE, Global Health and African Index Medicus), grey literature and websites of development organisations. Screening and data extraction were conducted by two independent reviewers and abstracted data were summarised using thematic narrative synthesis per the financing domains: mobilisation, pooling and purchasing. RESULTS: The search resulted in 5533 unique articles among which 149 met the inclusion criteria: 132 were related to mobilisation, 17 to pooling and 5 to purchasing. Neglect of surgery in national health priorities is widespread in SSA, and no report was found on national level surgical expenditures or budgetary allocations. Financial protection mechanisms are weak or non-existent; poor patients often forego care or face financial catastrophes in seeking care, even in the context of universal public financing (free care) initiatives. CONCLUSION: Financing of surgical and anaesthesia care in SSA is as poor as it is underinvestigated, calling for increased national prioritisation and tracking of surgical funding. Improving availability, accessibility and affordability of surgical and anaesthesia care require comprehensive and inclusive policy formulations.

The CAPACITI Decision-Support Tool for National Immunization Programs.

OBJECTIVES: Immunization programs in low-income and middle-income countries (LMICs) are faced with an ever-growing number of vaccines of public health importance recommended by the World Health Organization, while also financing a greater proportion of the program through domestic resources. More than ever, national immunization programs must be equipped to contextualize global guidance and make choices that are best suited to their setting. The CAPACITI decision-support tool has been developed in collaboration with national immunization program decision makers in LMICs to structure and document an evidence-based, context-specific process for prioritizing or selecting among multiple vaccination products, services, or strategies. METHODS: The CAPACITI decision-support tool is based on multi-criteria decision analysis, as a structured way to incorporate multiple sources of evidence and stakeholder perspectives. The tool has been developed iteratively in consultation with 12 countries across Africa, Asia, and the Americas. RESULTS: The tool is flexible to existing country processes and can follow any type of multi-criteria decision analysis or a hybrid approach. It is structured into 5 sections: decision question, criteria for decision making, evidence assessment, appraisal, and recommendation. The Excel-based tool guides the user through the steps and document discussions in a transparent manner, with an emphasis on stakeholder engagement and country ownership. CONCLUSIONS: Pilot countries valued the CAPACITI decision-support tool as a means to consider multiple criteria and stakeholder perspectives and to evaluate trade-offs and the impact of data quality. With use, it is expected that LMICs will tailor steps to their context and streamline the tool for decision making.

Use of Evidence-Informed Deliberative Processes by Health Technology Assessment Agencies Around the Globe.

BACKGROUND: Evidence-informed deliberative processes (EDPs) were recently introduced to guide health technology assessment (HTA) agencies to improve their processes towards more legitimate decision-making. The EDP framework provides guidance that covers the HTA process, ie, contextual factors, installation of an appraisal committee, selecting health technologies and criteria, assessment, appraisal, and communication and appeal. The aims of this study were to identify the level of use of EDPs by HTA agencies, identify their needs for guidance, and to learn about best practices. METHODS: A questionnaire for an online survey was developed based on the EDP framework, consisting of elements that reflect each part of the framework. The survey was sent to members of the International Network of Agencies for Health Technology Assessment (INAHTA). Two weeks following the invitation, a reminder was sent. The data collection took place between September-December 2018. RESULTS: Contact persons from 27 member agencies filled out the survey (response rate: 54%), of which 25 completed all questions. We found that contextual factors to support HTA development and the critical elements regarding conducting and reporting on HTA are overall in place. Respondents indicated that guidance was needed for specific elements related to selecting technologies and criteria, appraisal, and communication and appeal. With regard to best practices, the Canadian Agency for Drugs and Technologies and the National Institute for Health and Care Excellence (NICE, UK) were most often mentioned. CONCLUSION: This is the first survey among HTA agencies regarding the use of EDPs and provides useful information for further developing a practical guide for HTA agencies around the globe. The results could support HTA agencies in improving their processes towards more legitimate decision-making, as they could serve as a baseline measurement for future monitoring and evaluation.

Implementing evidence-informed deliberative processes in health technology assessment: a low income country perspective.

The purpose of this paper is to discuss the potential feasibility and utility of evidence-informed deliberative processes (EPDs) in low income country (LIC) contexts. EDPs are implemented in high and middle income countries and thought to improve the quality, consistency, and transparency of decisions informed by health technology assessment (HTA). Together these would ultimately improve the legitimacy of any decision making process. We argue-based on our previous work and in light of the priority setting literature-that EDPs are relevant and feasible within LICs. The extreme lack of resources necessitates making tough decisions which may mean depriving populations of potentially valuable health technologies. It is critical that the decisions and the decision making bodies are perceived as fair and legitimate by the people that are most affected by the decisions. EDPs are well aligned with the political infrastructure in some LICs, which encourages public participation in decision making. Furthermore, many countries are committed to evidence-informed decision making. However, the application of EDPs may be hampered by the limited availability of evidence of good quality, lack of interest in transparency and accountability (in some LICs), limited capacity to conduct HTA, as well as limited time and financial resources to invest in a deliberative process. While EDPs would potentially benefit many LICs, mitigating the identified potential barriers would strengthen their applicability. We believe that implementation studies in LICs, documenting the contextualized enablers and barriers will facilitate the development of context specific improvement strategies for EDPs.

Increasing the Legitimacy of Tough Choices in Healthcare Reimbursement: Approach and Results of a Citizen Forum in The Netherlands.

BACKGROUND: Some studies in the Netherlands have gauged public views on principles for healthcare priority setting, but they fall short of comprehensively explaining the public disapproval of several recent reimbursement decisions. OBJECTIVE: To obtain insight into citizens' preferences and identify the criteria they would propose for decisions pertaining to the benefits package of basic health insurance. METHODS: Twenty-four Dutch citizens were selected for participation in a Citizen Forum, which involved 3 weekends. Deliberations took place in small groups and in plenary, guided by 2 moderators, on the basis of 8 preselected case studies, which participants later compared and prioritized under the premise that not all treatments can or need to be reimbursed. Participants received opportunities to inform themselves through written brochures and live interactions with 3 experts. RESULTS: The Citizen Forum identified 16 criteria for inclusion or exclusion of treatments in the benefits package; they relate to the condition (2 criteria), treatment (11 criteria), and individual characteristics of those affected by the condition (3 criteria). In most case studies, it was a combination of criteria that determined whether or not participants favored inclusion of the treatment under consideration in the benefits package. Participants differed in their opinion about the relative importance of criteria, and they had difficulty in operationalizing and trading off criteria to provide a recommendation. CONCLUSIONS: Informed citizens are prepared to make and, to a certain extent, capable of making reasoned choices about the reimbursement of health services. They realize that choices are both necessary and possible. Broad public support and understanding for making tough choices regarding the benefits package of basic health insurance is not automatic: it requires an investment.

Integrating economic and health evidence to inform Covid-19 policy in low- and middle- income countries.

Covid-19 requires policy makers to consider evidence on both population health and economic welfare. Over the last two decades, the field of health economics has developed a range of analytical approaches and contributed to the institutionalisation of processes to employ economic evidence in health policy. We present a discussion outlining how these approaches and processes need to be applied more widely to inform Covid-19 policy; highlighting where they may need to be adapted conceptually and methodologically, and providing examples of work to date. We focus on the evidential and policy needs of low- and middle-income countries; where there is an urgent need for evidence to navigate the policy trade-offs between health and economic well-being posed by the Covid-19 pandemic.

How does participating in a deliberative citizens panel on healthcare priority setting influence the views of participants?

A deliberative citizens panel was held to obtain insight into criteria considered relevant for healthcare priority setting in the Netherlands. Our aim was to examine whether and how panel participation influenced participants' views on this topic. Participants (n = 24) deliberated on eight reimbursement cases in September and October, 2017. Using Q methodology, we identified three distinct viewpoints before (T0) and after (T1) panel participation. At T0, viewpoint 1 emphasised that access to healthcare is a right and that prioritisation should be based solely on patients' needs. Viewpoint 2 acknowledged scarcity of resources and emphasised the importance of treatment-related health gains. Viewpoint 3 focused on helping those in need, favouring younger patients, patients with a family, and treating diseases that heavily burden the families of patients. At T1, viewpoint 1 had become less opposed to prioritisation and more considerate of costs. Viewpoint 2 supported out-of-pocket payments more strongly. A new viewpoint 3 emerged that emphasised the importance of cost-effectiveness and that prioritisation should consider patient characteristics, such as their age. Participants' views partly remained stable, specifically regarding equal access and prioritisation based on need and health gains. Notable changes concerned increased support for prioritisation, consideration of costs, and cost-effectiveness. Further research into the effects of deliberative methods is required to better understand how they may contribute to the legitimacy of and public support for allocation decisions in healthcare.

Multicriteria Decision Analysis to Support Health Technology Assessment Agencies: Benefits, Limitations, and the Way Forward.

OBJECTIVE: Recent years have witnessed an increased interest in the use of multicriteria decision analysis (MCDA) to support health technology assessment (HTA) agencies for setting healthcare priorities. However, its implementation to date has been criticized for being "entirely mechanistic," ignoring opportunity costs, and not following best practice guidelines. This article provides guidance on the use of MCDA in this context. METHODS: The present study was based on a systematic review and consensus development. We developed a typology of MCDA studies and good implementation practice. We reviewed 36 studies over the period 1990 to 2018 on their compliance with good practice and developed recommendations. We reached consensus among authors over the course of several review rounds. RESULTS: We identified 3 MCDA study types: qualitative MCDA, quantitative MCDA, and MCDA with decision rules. The types perform differently in terms of quality, consistency, and transparency of recommendations on healthcare priorities. We advise HTA agencies to always include a deliberative component. Agencies should, at a minimum, undertake qualitative MCDA. The use of quantitative MCDA has additional benefits but also poses design challenges. MCDA with decision rules, used by HTA agencies in The Netherlands and the United Kingdom and typically referred to as structured deliberation, has the potential to further improve the formulation of recommendations but has not yet been subjected to broad experimentation and evaluation. CONCLUSION: MCDA holds large potential to support HTA agencies in setting healthcare priorities, but its implementation needs to be improved.