The Price of Practice Change: Assessing the Cost of Integrating Research Findings Into Clinical Practice.

BACKGROUND: Clinicians, health care administrators, and implementation scientists know that it takes intentional effort, resources, and implementation strategies to integrate research findings into routine clinical practice. An oft-cited concern for those considering whether and how to implement an evidence-based program is how much it will cost to implement the change. Yet information about the cost of implementation is not often available to health care decision-makers. Teams that received Implementation Award funding from PCORI are conducting implementation projects to promote the uptake of evidence-based practices in health care settings. As part of their implementation efforts, a number of teams have examined the costs of implementation. In this Topical Collection, 5 teams will report their findings on implementation costs and discuss their methods for data collection and analysis. DISCUSSION: The teams' costing efforts provide specific information about the costs sites can expect to incur in promoting the uptake of specific evidence-based programs. In addition, the papers illuminate 3 key features of the teams' approaches to measuring the cost of implementation: (1) the use of specific micro-costing methods with time-driven activity-based costing serving as the most popular method; (2) different ways to categorize and organize costs, including a site-based and non-site-based framework; and (3) cost collection challenges experienced by the teams. CONCLUSION: The cost of implementation is a critical consideration for organizations seeking to improve practice in accordance with research findings. This Topical Collection describes detailed approaches to providing this type of cost information and highlights insights to be gained from a rigorous focus on implementation cost.

Return of aggregate results to study participants: Facilitators, barriers, and recommendations.

Background: Most researchers and study participants believe that the summary, or aggregate, results of health research should be returned to study participants. However, researchers often do not return aggregate results. A better understanding of the impediments to results return could support improvements in this practice. Methods: This qualitative study convened eight virtual focus groups, four with investigators and four with patient partners from research studies funded by the Patient-Centered Outcomes Research Institute (PCORI). In total, 23 investigators and 20 partners participated. We explored perspectives, experiences, influences, and recommendations related to aggregate results return. Results: Focus group participants described the ethical importance of returning aggregate results, as well as the benefits to study participants. They also noted important impediments to results return, emphasizing IRB and logistical challenges and describing a lack of support for the practice both on the part of institutions and the field at large. Participants highlighted the value of patients and caregivers' perspectives and contributions to results return, which focused on returning the most relevant findings through effective channels and formats. They further emphasized the importance of planning and identified resources that could support results return. Conclusion: Researchers, funders, and the field can better facilitate results return by promoting standardized processes in research, such as the earmarking of funds for results return and inclusion of results returns milestones in research plans. More intentional policies, infrastructures, and resources that support results return may lead to more widespread return of study results to those who make these studies possible.

Health funders' dissemination and implementation practices: results from a survey of the Ensuring Value in Research (EViR) Funders' Forum.

BACKGROUND: A significant gap persists between evidence from research and its use in practice. Research funders, important actors in the health research system, can help reduce this gap by initiating dissemination and implementation (D&I) activities. The specific types of D&I activities funders currently lead have not been explored thoroughly. The Ensuring Value in Research (EViR) Funders' Forum-an international collaboration of health-related research funders-was established in 2017 to address research waste issues and increase the value of research. The Forum surveyed funders to learn about their D&I practices and challenges. METHODS: We distributed a five-item exploratory survey to participating funders in August 2018. The results informed the development of a survey instrument, distributed in June 2019. The survey instrument contained 15 items prompting respondents to categorize and describe their level of effort in six practice areas: release of findings, dissemination, knowledge exchange/partnering, implementation, building capacity, and implementation research. In addition, funders were asked to describe examples of their practices in detail. Thirty-one funders completed the survey instrument, a 58% response rate. RESULTS: Most funders regard D&I as a high priority, but funders vary in levels of activity per practice area. Over half of respondents reported that they have at least some activity in all D&I practice areas surveyed, with the exception of implementation research. The vast majority indicated some or significant activity in release of findings (97%) and dissemination (87%). Nearly one-fifth of funders (19%) indicated that implementation is outside their remit, and 26% indicated that implementation research is outside their remit. Survey respondents shared a broad range of examples of activities in each practice area. Lack of evidence for successful approaches and measuring impact were named frequently as challenges and as potential areas for collaboration. CONCLUSIONS: Although models of dissemination and implementation vary across organizations, the majority of funders indicated that D&I of research findings is a priority. Funders indicated a need for evidence on effectiveness of various approaches to D&I. Increased collaboration between funders, including sharing good practices, will increase our collective learning and knowledge development.

Lessons Learned from Developing Plain Language Summaries of Research Studies.

BACKGROUND: Plain language translation may facilitate the public's ability to understand and use results of scientific research. Brief description of activity: This article describes the Patient-Centered Outcomes Research Institute's (PCORI) approach to and lessons learned from developing plain language summaries of PCORI-funded research for the lay public. IMPLEMENTATION: We developed and tested a standard template for the summaries, incorporating feedback in the template design from focus groups with members of the public. Between February 2017 and March 2020, we completed translation of 272 plain language summaries of PCORI-funded studies, covering topics including cardiovascular disease, obesity, cancer, mental health, asthma, HIV/AIDS, and comparative effectiveness research methods. RESULTS: Templates use a question-and-answer format, with sections on the rationale, methods, results, limitations, and how the research will help inform decisions. In addition to feedback on template heading wording and order, focus group participants stressed the importance of establishing relevance and conveying credibility and limitations. LESSONS LEARNED: Lessons learned relate to supporting consistency across individual summaries, carefully prioritizing content to include in the summaries, and balancing plain language and reading level with precision. These lessons learned from template development and implementation may be useful to other organizations or publishers contemplating similar efforts to make scientific research results more accessible. [HLRP: Health Literacy Research and Practice. 2021;5(2):e155-e161.] Plain Language Summary: The Patient-Centered Outcomes Research Institute (PCORI) funds comparative effectiveness research. This research compares the benefits and harms of two or more health care choices. In this article, we describe lessons learned from PCORI's efforts to develop plain language summaries of results from the research it funds. These lessons may help other organizations that want to share research results in plain language.

Future Directions for Cost-effectiveness Analyses in Health and Medicine.

OBJECTIVES: In 2016, the Second Panel on Cost-effectiveness in Health and Medicine updated the seminal work of the original panel from 2 decades earlier. The Second Panel had an opportunity to reflect on the evolution of cost-effectiveness analysis (CEA) and to provide guidance for the next generation of practitioners and consumers. In this article, we present key topics for future research and policy. METHODS: During the course of its deliberations, the Second Panel discussed numerous topics for advancing methods and for improving the use of CEA in decision making. We identify and consider 7 areas for which the panel believes that future research would be particularly fruitful. In each of these areas, we highlight outstanding research needs. The list is not intended as an exhaustive inventory but rather a set of key items that surfaced repeatedly in the panel's discussions. In the online Appendix , we also list and expound briefly on 8 other important topics. RESULTS: We highlight 7 key areas: CEA and perspectives (determining, valuing, and summarizing elements for the analysis), modeling (comparative modeling and model transparency), health outcomes (valuing temporary health and path states, as well as health effects on caregivers), costing (a cost catalogue, valuing household production, and productivity effects), evidence synthesis (developing theory on learning across studies and combining data from clinical trials and observational studies), estimating and using cost-effectiveness thresholds (empirically representing 2 broad concepts: opportunity costs and public willingness to pay), and reporting and communicating CEAs (written protocols and a quality scoring system). CONCLUSIONS: Cost-effectiveness analysis remains a flourishing and evolving field with many opportunities for research. More work is needed on many fronts to understand how best to incorporate CEA into policy and practice.

Setting Boundaries: Public Views on Limiting Patient and Physician Autonomy in Health Care Decisions.

We obtained and qualitatively analyzed input from more than nine hundred citizens during seventy-six public deliberation sessions about patient and physician autonomy in decision making, setting health care boundaries, and the tensions among competing social values. Generally, participants resisted interference with the patient-physician relationship and believed strongly in the freedom of patient and physician to control individual medical decisions. However, during deliberation participants identified two situations where boundaries and regulations in health care were more acceptable: protecting people from harm and allocating limited resources. The core value of individual freedom was tempered in varying degrees by the values of concern for the greater good and fairness in allocating resources. Where tensions between values emerged, participants used different concepts-including accountability, transparency, trust, personal responsibility, and moral obligation-to navigate trade-offs. Fairly balancing the public's desire to protect individual freedom with their sense of responsibility for protecting the common good may be the key to developing acceptable, workable policies that promote evidence-based medical practice.

Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine.

IMPORTANCE: Since publication of the report by the Panel on Cost-Effectiveness in Health and Medicine in 1996, researchers have advanced the methods of cost-effectiveness analysis, and policy makers have experimented with its application. The need to deliver health care efficiently and the importance of using analytic techniques to understand the clinical and economic consequences of strategies to improve health have increased in recent years. OBJECTIVE: To review the state of the field and provide recommendations to improve the quality of cost-effectiveness analyses. The intended audiences include researchers, government policy makers, public health officials, health care administrators, payers, businesses, clinicians, patients, and consumers. DESIGN: In 2012, the Second Panel on Cost-Effectiveness in Health and Medicine was formed and included 2 co-chairs, 13 members, and 3 additional members of a leadership group. These members were selected on the basis of their experience in the field to provide broad expertise in the design, conduct, and use of cost-effectiveness analyses. Over the next 3.5 years, the panel developed recommendations by consensus. These recommendations were then reviewed by invited external reviewers and through a public posting process. FINDINGS: The concept of a "reference case" and a set of standard methodological practices that all cost-effectiveness analyses should follow to improve quality and comparability are recommended. All cost-effectiveness analyses should report 2 reference case analyses: one based on a health care sector perspective and another based on a societal perspective. The use of an "impact inventory," which is a structured table that contains consequences (both inside and outside the formal health care sector), intended to clarify the scope and boundaries of the 2 reference case analyses is also recommended. This special communication reviews these recommendations and others concerning the estimation of the consequences of interventions, the valuation of health outcomes, and the reporting of cost-effectiveness analyses. CONCLUSIONS AND RELEVANCE: The Second Panel reviewed the current status of the field of cost-effectiveness analysis and developed a new set of recommendations. Major changes include the recommendation to perform analyses from 2 reference case perspectives and to provide an impact inventory to clarify included consequences.

The Role of Decision Models in Health Care Policy: A Case Study.

BACKGROUND: In 2009, the Centers for Medicare and Medicaid Services (CMS) underwent a National Coverage Determination on computed tomography colonography (CTC) to screen for colorectal cancer. The Cancer Intervention & Surveillance Network developed decision models to inform this decision. The purpose of our study was to investigate the role of models in this decision. METHODS: We performed a descriptive case study. We conducted semistructured telephone interviews with members of the CMS coverage and analysis group (CAG) and Medicare Coverage and Analysis Advisory Committee (MEDCAC) panelists. Informed by previously published literature, we developed a coding scheme to analyze interview transcripts, MEDCAC meeting transcripts, and the final CMS decision memo. RESULTS: Four members of the CAG and 8 MEDCAC panelists were interviewed. The total number of codes across all study documents was 772. We found evidence that decision makers believed in the adequacy of models to inform decision making. In interview transcripts, the code Models Are Adequate to Inform was more frequent than the code Models Are Inadequate to Inform (47 times v. 5). Discussion of model conceptualization dominated the MEDCAC meeting (Model Conceptualization assigned 113 times) and was frequently discussed during interviews (Model Conceptualization assigned 84 times). We also found evidence that the models helped to focus the policy discussion. Across study documents, the codes Focus on Cost, Focus on Clinical-Health Impact, and Focus on Inadequacy of Evidence Base were assigned 99, 98, and 97 times, respectively. CONCLUSIONS: Decision makers involved in the CTC decision believed in the adequacy of models to inform coverage decisions. The model played a role in focusing the CTC coverage policy discussion.

Understanding An Informed Public's Views On The Role Of Evidence In Making Health Care Decisions.

Policy makers and practitioners increasingly believe that medical evidence plays a critical role in improving care and health outcomes and lowering costs. However, public understanding of the role of evidence-based care may be different. Public deliberation is a process that convenes diverse citizens and has them learn about and consider ethical or values-based dilemmas and weigh alternative views. The Community Forum Deliberative Methods Demonstration project, sponsored by the Agency for Healthcare Research and Quality, obtained informed public views on the role of evidence in health care decisions through seventy-six deliberative groups involving 907 people overall, in the period August-November 2012. Although participants perceived evidence as being essential to high-quality care, they also believed that personal choice or clinical judgment could trump evidence. They viewed doctors as central figures in discussing evidence with patients and key arbiters of whether to follow evidence in individual cases. They found evidence of harm to individuals or the community to be more compelling than evidence of effectiveness. These findings indicate that increased public understanding of evidence can play an important role in advancing evidence-based care by helping create policies that better reflect the needs and values of the public.

Effectiveness of public deliberation methods for gathering input on issues in healthcare: Results from a randomized trial.

UNLABELLED: Public deliberation elicits informed perspectives on complex issues that are values-laden and lack technical solutions. This Deliberative Methods Demonstration examined the effectiveness of public deliberation for obtaining informed public input regarding the role of medical evidence in U.S. healthcare. We conducted a 5-arm randomized controlled trial, assigning participants to one of four deliberative methods or to a reading materials only (RMO) control group. The four deliberative methods reflected important differences in implementation, including length of the deliberative process and mode of interaction. The project convened 76 groups between August and November 2012 in four U.S. LOCATIONS: Chicago, IL; Sacramento, CA; Silver Spring, MD; and Durham, NC, capturing a sociodemographically diverse sample with specific attention to ensuring inclusion of Hispanic, African-American, and elderly participants. Of 1774 people recruited, 75% participated: 961 took part in a deliberative method and 377 participants comprised the RMO control group. To assess effectiveness of the deliberative methods overall and of individual methods, we evaluated whether mean pre-post changes on a knowledge and attitude survey were statistically different from the RMO control using ANCOVA. In addition, we calculated mean scores capturing participant views of the impact and value of deliberation. Participating in deliberation increased participants' knowledge of evidence and comparative effectiveness research and shifted participants' attitudes regarding the role of evidence in decision-making. When comparing each deliberative method to the RMO control group, all four deliberative methods resulted in statistically significant change on at least one knowledge or attitude measure. These findings were underscored by self-reports that the experience affected participants' opinions. Public deliberation offers unique potential for those seeking informed input on complex, values-laden topics affecting broad public constituencies.

Deliberation: obtaining informed input from a diverse public.

OBJECTIVES: Health care decision makers require public input to incorporate diverse values into programs and policies. Deliberation, one method for obtaining input, seeks to apply inclusive principles wherein diverse groups provide perspectives to inform decisions. We evaluate whether participants of different racial, ethnic, and educational backgrounds show differences in the effect of deliberation and the value placed on deliberation participation. METHODS: We surveyed 907 participants before and after deliberation. Regression models examined associations between demographics and change in knowledge and attitudes, and perceived impact. RESULTS: Changes in knowledge about using medical evidence in decision-making were not associated with race, ethnicity, or education. Changes in attitudes were not associated with these characteristics with one exception. African American, Hispanic, and participants with lower educational attainment reported more perceived impact. CONCLUSION: Similar results across demographic groups suggest deliberation's promise for obtaining input from a diverse public to inform health programs and policies.

Can we better prioritize resources for cost-utility research?

PURPOSE: We examined 512 published cost-utility analyses (CUAs) in the U.S. and other developed countries from 1976 through 2001 to determine: 1) the types of interventions studied; 2) whether they cover diseases and conditions with the highest burden; and, 3) to what extent they have covered leading health concerns defined by the Healthy People 2010 report. DATA AND METHODS: We compared rankings of the most common diseases covered by the CUAs to rankings of U.S. disease burden. We also examined the extent to which CUAs covered key Healthy People 2010 priorities. RESULTS: CUAs have focused mostly on pharmaceuticals (40%) and surgical procedures (16%). When compared to leading causes of DALYs, the data show overrepresentation of CUAs in cerebrovascular disease, diabetes, breast cancer, and HIV/AIDS, and underrepresentation in depression and bipolar disorder, injuries, and substance abuse disorders. Few CUAs have targeted Healthy People 2010 areas, such as physical activity. CONCLUSIONS: Published CUAs are associated with burden measures, but have not covered certain important health problems. These discrepancies do not alone indicate that society has been targeting resources for research inefficiently, but they do suggest the need to formalize the question of where each CUA research dollar might do the most good.

Federal sponsorship of cost-effectiveness and related research in health care: 1997-2001.

OBJECTIVES: To describe recent federal sponsorship of cost-effectiveness and related health economics research to provide insight into the functioning of existing research support systems and assess the roles of federal health agencies. METHODS: Using the PubMed database, we identified cost-effectiveness and related publications citing support from a US government entity and published during the period of 1997 through 2001, and audited them for information on funding sources, study type, and content focus. RESULTS: Five Department of Health and Human Services agencies and centers and the Veterans Administration are cited as funders in 74% of 520 federally supported health economics publications we identified. Three-fourths of federally supported publications address five areas of high disease burden: infections, cancer, HIV/AIDS, cardiovascular disease, and substance abuse. Other high burden diseases, including mental health, diabetes, and injuries, receive less attention. Federal support of health economics studies of health education and care delivery-intervention types underexamined in the field-is relatively strong but most often focuses on substance abuse or mental health services. Each of the top federal funders has a distinct funding pattern, but there are substantial areas of overlap within which we could not identify content domains specific to one funder or another. CONCLUSIONS: Federal support of health economics research has paralleled growth in the field. Federal funders support projects consistent with their mission and focus on high-burden disease areas. However, overlapping funding areas, ambiguity concerning agency interests within overlapping content areas, and gaps in some disease and intervention areas suggest that the coordination of health economics research funding could be improved.

Making policy when the evidence is in dispute.

Policymakers often struggle with medical issues that are the subject of fierce scientific debate. On closer examination, many of these debates are manifestations of conflicting perspectives and values as much as disagreements over the evidence. We summarize common factors underlying recent debates and outline a series of questions that can help disentangle questions of evidence from those of values. These questions focus on identifying the most important outcomes, evaluating the quality of evidence, and assessing the trade-offs involved. We then use four recent policy debates-involving prostate-specific antigen (PSA) screening, high-dose chemotherapy for breast cancer, antibiotic therapy for otitis media, and newborn hearing screening-to illustrate how this approach can help clarify areas of agreement and disagreement of the opposing sides.

Treatment with ursodeoxycholic acid is associated with weight gain in patients with primary biliary cirrhosis.

BACKGROUND: Ursodeoxycholic acid (UDCA) is the established treatment of primary biliary cirrhosis (PBC) and is a safe and well-tolerated medication. Nevertheless, patients often anecdotally complain of weight gain while on this drug. GOALS: We compared weight changes in patients treated with UDCA and those on placebo to characterize this potential side effect. STUDY: One-hundred eighty patients with PBC who were enrolled into a randomized, controlled trial received either UDCA (13-15 mg/kg/d) or an identical placebo. Changes from baseline weight were calculated at 12, 24, 36, and 48 months. Other markers of disease activity, including liver biochemistries, serum lipids, histologic stage, and Mayo Risk Score were evaluated in both groups. RESULTS: The proportion of patients who gained weight during the first 12 months of therapy was significantly greater in the UDCA than placebo group (67/86 [78%] versus 43/73 [57%] respectively, P = 0.005). Patients in the UDCA group gained an average of 3.6 +/- 6.5% kg (2.2 +/- 5.1 kg) which was significantly greater than the average of 0.6 +/- 6.9% kg (0.6 +/- 4.9 kg) gained in the placebo group (P = 0.04). The biggest change in weight occurred in the first 12 months of treatment (P < 0.001); after this, weight was maintained for the 4-year duration of treatment. There was no significant correlation between initial body mass index (BMI) and weight change or changes in disease activity and weight. CONCLUSION: UDCA treatment in patients with PBC is associated with a significant weight gain that occurs in the first 12 months of treatment, persists for the duration of treatment, and occurs independent of baseline BMI. Discussions with PBC patients beginning UDCA treatment should include the beneficial effects this medication has on disease outcome, but should also mention weight gain as a possible side effect.