Development and Evaluation of the Algorithm CErtaInty Tool (ACE-IT) to Assess Electronic Medical Record and Claims-based Algorithms' Fit for Purpose for Safety Outcomes.

INTRODUCTION: Electronic health record (EHR) or medical claims-based algorithms (i.e., operational definitions) can be used to define safety outcomes using real-world data. However, existing tools do not allow researchers and decision-makers to adequately appraise whether a particular algorithm is fit for purpose (FFP) to support regulatory decisions on drug safety surveillance. Our objective was to develop a tool to enable regulatory decision-makers and other stakeholders to appraise whether a given algorithm is FFP for a specific decision context. METHODS: We drafted a set of 77 generic items informed by regulatory guidance documents, existing instruments, and publications. The outcome of ischemic stroke served as an exemplar to inform the development of draft items. The items were designed to be outcome independent. We conducted a three-round online Delphi panel to develop and refine the tool and achieve consensus on items (> 70% agreement) among panel participants composed of regulators, researchers from pharmaceutical organizations, academic clinicians, methodologists, pharmacoepidemiologists, and cardiologists. We conducted a qualitative analysis of panel responses. Five pairs of reviewers independently evaluated two ischemic stroke algorithm validation studies to test its application. We developed a user guide, with explanation and elaboration for each item, guidance on essential and additional elements for user responses, and an illustrative example of a complete assessment. Furthermore, we conducted a 2-h online stakeholder panel of 16 participants from regulatory agencies, academic institutions, and industry. We solicited input on key factors for an FFP assessment, their general reaction to the Algorithm CErtaInty Tool (ACE-IT), limitations of the tool, and its potential use. RESULTS: The expert panel reviewed and made changes to the initial list of 77 items. The panel achieved consensus on 38 items, and the final version of the ACE-IT includes 34 items after removal of duplicate items. Applying the tool to two ischemic stroke algorithms demonstrated challenges in its application and identified shared concepts addressed by more than one item. The ACE-IT was viewed positively by the majority of stakeholders. They identified that the tool could serve as an educational resource as well as an information-sharing platform. The time required to complete the assessment was identified as an important limitation. We consolidated items with shared concepts and added a preliminary screen section and a summary assessment box based on their input. The final version of the ACE-IT is a 34-item tool for assessing whether algorithm validation studies on safety outcomes are FFP. It comprises the domains of internal validity (24 items), external validity (seven items), and ethical conduct and reporting of the validation study (three items). The internal validity domain includes sections on objectives, data sources, population, outcomes, design and setting, statistical methods, reference standard, accuracy, and strengths and limitations. The external validity domain includes items that assess the generalizability to a proposed target study. The domain on ethics and transparency includes items on ethical conduct and reporting of the validation study. CONCLUSION: The ACE-IT supports a structured, transparent, and flexible approach for decision-makers to appraise whether electronic health record or medical claims-based algorithms for safety outcomes are FFP for a specific decision context. Reliability and validity testing using a larger sample of participants in other therapeutic areas and further modifications to reduce the time needed to complete the assessment are needed to fully evaluate its utility for regulatory decision-making.

Advancing clinical trials in nursing homes: A proposed roadmap to success.

An effective clinical research effort in nursing homes to address prevention and treatment of COVID-19 faced overwhelming challenges. Under the Health Care Systems Research Network-Older Americans Independence Centers AGING Initiative, a multidisciplinary Stakeholder Advisory Panel was convened to develop recommendations to improve the capability of the clinical research enterprise in US nursing homes. The Panel considered the nursing home as a setting for clinical trials, reviewed the current state of clinical trials in nursing homes, and ultimately developed recommendations for the establishment of a nursing home clinical trials research network that would be centrally supported and administered. This report summarizes the Panel's recommendations, which were developed in alignment with the following core principles: build on available research infrastructure where appropriate; leverage existing productive partnerships of researchers with groups of nursing homes and nursing home corporations; encompass both efficacy and effectiveness clinical trials; be responsive to a broad range of stakeholders including nursing home residents and their care partners; be relevant to an expansive range of clinical and health care delivery research questions; be able to pivot as necessary to changing research priorities and circumstances; create a pathway for industry-sponsored research as appropriate; invest in strategies to increase diversity in study populations and the research workforce; and foster the development of the next generation of nursing home researchers.

Advancing Clinical Trials in Nursing Homes: A Proposed Roadmap to Success.

An effective clinical research effort in nursing homes to address prevention and treatment of COVID-19 faced overwhelming challenges. Under the Health Care Systems Research Network-Older Americans Independence Centers AGING Initiative, a multidisciplinary Stakeholder Advisory Panel was convened to develop recommendations to improve the capability of the clinical research enterprise in US nursing homes. The Panel considered the nursing home as a setting for clinical trials, reviewed the current state of clinical trials in nursing homes, and ultimately developed recommendations for the establishment of a nursing home clinical trials research network that would be centrally supported and administered. This report summarizes the Panel's recommendations, which were developed in alignment with the following core principles: build on available research infrastructure where appropriate; leverage existing productive partnerships of researchers with groups of nursing homes and nursing home corporations; encompass both efficacy and effectiveness clinical trials; be responsive to a broad range of stakeholders including nursing home residents and their care partners; be relevant to an expansive range of clinical and health care delivery research questions; be able to pivot as necessary to changing research priorities and circumstances; create a pathway for industry-sponsored research as appropriate; invest in strategies to increase diversity in study populations and the research workforce; and foster the development of the next generation of nursing home researchers.

Stakeholders' views on priorities essential for establishing a supportive environment for clinical trials in nursing homes.

BACKGROUND: The U.S. clinical research enterprise in nursing homes was unprepared to mount clinical trials in nursing homes to address urgent questions relevant to prevention and treatment during the COVID-19 pandemic. We identify priorities essential for establishing a supportive environment for future clinical trials in nursing homes. METHODS: Two cross-sectional online questionnaires were administered between January and February 2021. One was administered to nursing home providers, researchers, and policymakers; respondents rated the importance of attributes of researchers, facilities, leaders and staff for conducting clinical trials in nursing homes. Because importance may depend on trial type, respondents rated each attribute for efficacy trials (testing an intervention in ideal circumstances) and effectiveness trials (testing an intervention in "real world" circumstances). We calculated the attribute rating means and standard deviations, and used content analysis to characterize open-ended responses. The second questionnaire for resident family members and advocates included open-ended questions about nursing home research, and factors influencing willingness to participate. RESULTS: The attributes rated as most essential for conducting efficacy and effectiveness trials in nursing homes are research team attributes, that is, that researchers recognize regulatory constraints; understand and adapt to nursing home workflow; and work collaboratively with nursing home leaders to identify priorities. Resident and facility diversity emerged as essential for effectiveness trials; important dimensions included resident race, ethnicity and income, as well as nursing home urban/rural location, quality ratings, geography, staffing ratios, size, and profit status. Caregivers and resident advocates stressed the importance of communication among participants, researchers, and nursing home leadership and staff at all stages of a trial. CONCLUSION: Developing a robust U.S. clinical research enterprise capable of efficiently mounting future clinical trials in nursing homes will require a reimagining of the relationships that exist between researchers, facilities, nursing home leaders, and residents, with a research infrastructure specifically focused on supporting and fostering these connections.

Advancing Clinical Trials in Nursing Homes: A Proposed Roadmap to Success.

An effective clinical research effort in nursing homes to address prevention and treatment of COVID-19 faced overwhelming challenges. Under the Health Care Systems Research Network-Older Americans Independence Centers AGING Initiative, a multidisciplinary Stakeholder Advisory Panel was convened to develop recommendations to improve the capability of the clinical research enterprise in US nursing homes. The Panel considered the nursing home as a setting for clinical trials, reviewed the current state of clinical trials in nursing homes, and ultimately developed recommendations for the establishment of a nursing home clinical trials research network that would be centrally supported and administered. This report summarizes the Panel's recommendations, which were developed in alignment with the following core principles: build on available research infrastructure where appropriate; leverage existing productive partnerships of researchers with groups of nursing homes and nursing home corporations; encompass both efficacy and effectiveness clinical trials; be responsive to a broad range of stakeholders including nursing home residents and their care partners; be relevant to an expansive range of clinical and health care delivery research questions; be able to pivot as necessary to changing research priorities and circumstances; create a pathway for industry-sponsored research as appropriate; invest in strategies to increase diversity in study populations and the research workforce; and foster the development of the next generation of nursing home researchers.