Calibrating a Transformer-Based Model's Confidence on Community-Engaged Research Studies: Decision Support Evaluation Study.

BACKGROUND: Deep learning offers great benefits in classification tasks such as medical imaging diagnostics or stock trading, especially when compared with human-level performances, and can be a viable option for classifying distinct levels within community-engaged research (CEnR). CEnR is a collaborative approach between academics and community partners with the aim of conducting research that is relevant to community needs while incorporating diverse forms of expertise. In the field of deep learning and artificial intelligence (AI), training multiple models to obtain the highest validation accuracy is common practice; however, it can overfit toward that specific data set and not generalize well to a real-world population, which creates issues of bias and potentially dangerous algorithmic decisions. Consequently, if we plan on automating human decision-making, there is a need for creating techniques and exhaustive evaluative processes for these powerful unexplainable models to ensure that we do not incorporate and blindly trust poor AI models to make real-world decisions. OBJECTIVE: We aimed to conduct an evaluation study to see whether our most accurate transformer-based models derived from previous studies could emulate our own classification spectrum for tracking CEnR studies as well as whether the use of calibrated confidence scores was meaningful. METHODS: We compared the results from 3 domain experts, who classified a sample of 45 studies derived from our university's institutional review board database, with those from 3 previously trained transformer-based models, as well as investigated whether calibrated confidence scores can be a viable technique for using AI in a support role for complex decision-making systems. RESULTS: Our findings reveal that certain models exhibit an overestimation of their performance through high confidence scores, despite not achieving the highest validation accuracy. CONCLUSIONS: Future studies should be conducted with larger sample sizes to generalize the results more effectively. Although our study addresses the concerns of bias and overfitting in deep learning models, there is a need to further explore methods that allow domain experts to trust our models more. The use of a calibrated confidence score can be a misleading metric when determining our AI model's level of competency.

The Concept of Resistance to Substance Use and a Research Approach: The Resist! Project.

Illicit substance use is dangerous in both acute and chronic forms, frequently resulting in lethal poisoning, addiction, and other negative consequences. Similar to research in other psychiatric conditions, whose ultimate goal is to enable effective prevention and treatment, studies in substance use are focused on factors elevating the risk for the disorder. The rapid growth of the substance use problem despite the effort invested in fighting it, however, suggests the need in changing the research approach. Instead of attempting to identify risk factors, whose neutralization is often infeasible if not impossible, it may be more promising to systematically reverse the perspective to the factors enhancing the aspect of liability to disorder that shares the same dimension but is opposite to risk, that is, resistance to substance use. Resistance factors, which enable the majority of the population to remain unaffected despite the ubiquity of psychoactive substances, may be more amenable to translation. While the resistance aspect of liability is symmetric to risk, the resistance approach requires substantial changes in sampling (high-resistance rather than high-risk) and using quantitative indices of liability. This article provides an overview and a practical approach to research in resistance to substance use/addiction, currently implemented in a NIH-funded project. The project benefits from unique opportunities afforded by the data originating from two longitudinal twin studies, the Virginia Twin Study of Adolescent and Behavioral Development and the Minnesota Twin Family Study. The methodology described is also applicable to other psychiatric disorders.

Urban-Rural Disparities in Deaths of Despair: A County-Level Analysis 2004-2016 in the U.S.

Introduction: The purpose of this study is to examine nationwide disparities in drug, alcohol, and suicide mortality; evaluate the association between county-level characteristics and these mortality rates; and illustrate spatial patterns of mortality risk to identify areas with elevated risk. Methods: The authors applied a Bayesian spatial regression technique to investigate the association between U.S. county-level characteristics and drug, alcohol, and suicide mortality rates for 2004-2016, accounting for spatial correlation that occurs among counties. Results: Mortality risks from drug, alcohol, and suicide were positively associated with the degree of rurality, the proportion of vacant housing units, the population with a disability, the unemployed population, the population with low access to grocery stores, and the population with no health insurance. Conversely, risks were negatively associated with Hispanic population, non-Hispanic Black population, and population with a bachelor's degree or higher. Conclusions: Spatial disparities in drug, alcohol, and suicide mortality exist at the county level across the U.S. social determinants of health; educational attainment, degree of rurality, ethnicity, disability, unemployment, and health insurance status are important factors associated with these mortality rates. A comprehensive strategy that includes downstream interventions providing equitable access to healthcare services and upstream efforts in addressing socioeconomic conditions is warranted to effectively reduce these mortality burdens.

Attention-Based Models for Classifying Small Data Sets Using Community-Engaged Research Protocols: Classification System Development and Validation Pilot Study.

BACKGROUND: Community-engaged research (CEnR) is a research approach in which scholars partner with community organizations or individuals with whom they share an interest in the study topic, typically with the goal of supporting that community's well-being. CEnR is well-established in numerous disciplines including the clinical and social sciences. However, universities experience challenges reporting comprehensive CEnR metrics, limiting the development of appropriate CEnR infrastructure and the advancement of relationships with communities, funders, and stakeholders. OBJECTIVE: We propose a novel approach to identifying and categorizing community-engaged studies by applying attention-based deep learning models to human participants protocols that have been submitted to the university's institutional review board (IRB). METHODS: We manually classified a sample of 280 protocols submitted to the IRB using a 3- and 6-level CEnR heuristic. We then trained an attention-based bidirectional long short-term memory unit (Bi-LSTM) on the classified protocols and compared it to transformer models such as Bidirectional Encoder Representations From Transformers (BERT), Bio + Clinical BERT, and Cross-lingual Language Model-Robustly Optimized BERT Pre-training Approach (XLM-RoBERTa). We applied the best-performing models to the full sample of unlabeled IRB protocols submitted in the years 2013-2019 (n>6000). RESULTS: Although transfer learning is superior, receiving a 0.9952 evaluation F1 score for all transformer models implemented compared to the attention-based Bi-LSTM (between 48%-80%), there were key issues with overfitting. This finding is consistent across several methodological adjustments: an augmented data set with and without cross-validation, an unaugmented data set with and without cross-validation, a 6-class CEnR spectrum, and a 3-class one. CONCLUSIONS: Transfer learning is a more viable method than the attention-based bidirectional-LSTM for differentiating small data sets characterized by the idiosyncrasies and variability of CEnR descriptions used by principal investigators in research protocols. Despite these issues involving overfitting, BERT and the other transformer models remarkably showed an understanding of our data unlike the attention-based Bi-LSTM model, promising a more realistic path toward solving this real-world application.

A Guide to Selecting Participatory Research Methods Based on Project and Partnership Goals.

Participatory research engages community stakeholders in the research process, from problem identification and developing the research question, to dissemination of results. There is increasing recognition in the field of health research that community-engaged methods can be used throughout the research process. The volume of guidance for engaging communities and conducting participatory research has grown steadily in the past 40+ years, in many countries and contexts. Further, some institutions now require stakeholder engagement in research as a condition of funding. Interest in collaborating in the research process is also growing among patients and the public. This article provides an overview for selecting participatory research methods based on project and partnerships goals.

Developing a classification system and algorithm to track community-engaged research using IRB protocols at a large research university.

Community-engaged research (CEnR) is now an established research approach. The current research seeks to pilot the systematic and automated identification and categorization of CEnR to facilitate longitudinal tracking using administrative data. We inductively analyzed and manually coded a sample of Institutional Review Board (IRB) protocols. Comparing the variety of partnered relationships in practice with established conceptual classification systems, we developed five categories of partnership: Non-CEnR, Instrumental, Academic-led, Cooperative, and Reciprocal. The coded protocols were used to train a deep-learning algorithm using natural language processing to categorize research. We compared the results to data from three questions added to the IRB application to identify whether studies had a community partner and the type of engagement planned. The preliminary results show that the algorithm is potentially more likely to categorize studies as CEnR compared to investigator-recorded data and to categorize studies at a higher level of engagement. With this approach, universities could use administrative data to inform strategic planning, address progress in meeting community needs, and coordinate efforts across programs and departments. As scholars and technical experts improve the algorithm's accuracy, universities and research institutions could implement standardized reporting features to track broader trends and accomplishments.

Participatory Action Planning to Address the Opioid Crisis in a Rural Virginia Community Using the SEED Method.

The SEED Method is a multi-stakeholder approach that was created to involve diverse stakeholders in the development and prioritization of research questions using community-based participatory research (CBPR) principles. Here we describe an adaptation of the SEED Method that focuses on developing and prioritizing strategies for addressing a health problem and bringing stakeholders together to develop and implement community action plans based on those strategies. We describe steps for implementing the SEED Method for community action planning and the results of a case study in a rural Virginia community with high opioid prescription and mortality rates. A participatory research team worked with three groups of Topic stakeholders to gather data, develop conceptual models, and create and prioritize strategies for reducing prescription and non-prescription opioid misuse and overdoses. Each group came up with 19 to 25 strategies and prioritized their top five, which included actions, services or programs, strategies, policies, and system changes. Attendees at community action planning meetings reviewed the 15 prioritized strategies, proposed three additional strategies, and prioritized their top choices. Community stakeholders started four work groups to implement the selected strategies in collaboration with the research team.

A lung cancer research agenda that reflects the diverse perspectives of community stakeholders: process and outcomes of the SEED method.

PLAIN ENGLISH SUMMARY: There is a need for methods that engage lay people and other stakeholders, such as patients and healthcare providers, in developing research questions about health issues important to them and their communities. Involving stakeholders helps ensure that funding goes to research that addresses their concerns. The SEED Method engages stakeholders in a systematic process to explore health issues and develop research questions. Diverse groups of stakeholders participate at three levels: as collaborators that lead the process throughout, as participants who use their expertise to develop the questions, and as consultants who provide additional perspectives about the health topic. We used the SEED Method to engage 61 stakeholders from different socioeconomic and professional backgrounds to create research questions on lung cancer outcomes. Participants included cancer patients and caregivers, healthcare providers and administrators, and policymakers from a rural Virginia community. They developed causal models that diagrammed factors that influence lung cancer outcomes and the relationships between them. They used these models to develop priority research questions. The questions reflect the participants' diverse perspectives and address different areas of inquiry related to lung cancer outcomes, including access to care, support systems, social determinants of health, and quality of care. Participants felt well prepared to perform the project tasks because they had the opportunity to review lung cancer information, receive causal model and research question development training, and participate in facilitated group activities. The SEED Method can be used in a variety of settings and applied to any health topic of interest to stakeholders. ABSTRACT: Background Engagement of stakeholders in prioritization of health research can help ensure that funding is directed to research that reflects their concerns and needs. The Stakeholder Engagement in quEstion Development and Prioritization (SEED) Method is a multi-stakeholder methodology that uses principles of community engagement and causal modeling to develop health research questions that reflect the priorities of patients, clinicians, and other community stakeholders. We conducted a demonstration of the SEED Method to generate research questions on lung cancer outcomes, and to evaluate the process, outcomes, and effectiveness of the method for generating a research agenda that reflects diverse stakeholder perspectives. Methods The SEED Method engages community members at three levels: collaboration, participation, and consultation. We conducted a demonstration project from November, 2015 to July, 2016, in a rural Virginia community that was experiencing a significant disparity in lung cancer outcomes. A community research team led the project and selected three distinct stakeholder groups (Topic groups, TG) for participatory engagement in analysis of the health issue, causal modeling, and research question development. We evaluated the quality of stakeholder engagement and compared TG causal models and research questions to evaluate the diversity of stakeholder perspectives resulting from the methodology. Results The resulting research agenda poses questions on how a broad range of topics including access to care, support systems and coping mechanisms, social determinants of health, and quality of care impacts lung cancer outcomes. Participants felt well prepared for the tasks they were asked to perform due to the technical trainings and facilitated modeling and question development activities that are part of the SEED Method. The causal models and research questions developed by the Topic Groups reflected the diverse perspectives of the stakeholders. Conclusions The SEED Method has the potential to generate relevant stakeholder-centered research agendas on a variety of health-related topics, and to create community capacity for sustained research engagement.

Assessing the Impacts and Ripple Effects of a Community-University Partnership: A Retrospective Roadmap.

Each community-based participatory research (CBPR) partnership may incur "ripple effects" - impacts that happen outside the scope of planned projects. We used brainstorming and interviewing to create a roadmap that incorporated input from nine CBPR participants and five community/academic partners to retrospectively assess the ripple effects observed after five years of participatory research in one urban community. The resulting roadmap reflected a range of community impacts which we then divided into four key areas: impacts in the community (i.e., strategies, programs, and policies implemented by community partners), impacts on the CBPR team, impacts on individuals (participants and community members), and contributions to the field and the university. Our approach focused on observing what happened in the community that was directly or indirectly related to our partnership, process, products, and relationships. Much of the impact we observed reflected the synergy of sharing our research and community voice with responsive partners and stakeholders.

Changes in midlife death rates across racial and ethnic groups in the United States: systematic analysis of vital statistics.

OBJECTIVE: To systematically compare midlife mortality patterns in the United States across racial and ethnic groups during 1999-2016, documenting causes of death and their relative contribution to excess deaths. DESIGN: Trend analysis of US vital statistics among racial and ethnic groups. SETTING: United States, 1999-2016. POPULATION: US adults aged 25-64 years (midlife). MAIN OUTCOME MEASURES: Absolute changes in mortality measured as average year-to-year change during 1999-2016 and 2012-16; excess deaths attributable to increasing mortality; and relative changes in mortality measured as relative difference between mortality in 1999 versus 2016 and the nadir year versus 2016, and the slope of modeled mortality trends for 1999-2016 and for intervals between joinpoints. RESULTS: During 1999-2016, all cause mortality in midlife increased not only among non-Hispanic (NH) whites but also among NH American Indians and Alaskan Natives. Although all cause mortality initially decreased among NH blacks, Hispanics, and NH Asians and Pacific Islanders, this trend ended in 2009-11. Drug overdoses were the leading cause of increased mortality in midlife in each population, but mortality also increased for alcohol related conditions, suicides, and organ diseases involving multiple body systems. Although midlife mortality among NH whites increased across a multitude of conditions, a similar trend affected non-white populations. Absolute (year-to-year) increases in midlife mortality among non-white populationsoften matched or exceeded those of NH whites, especially in 2012-16, when the rate of increase intensified for many causes of death. During 1999-2016, NH American Indians and Alaskan Natives experienced large increases in midlife mortality from 12 causes, not only drug overdoses (411.4%) but also hypertensive diseases (269.3%), liver cancer (115.1%), viral hepatitis (112.1%), and diseases of the nervous system (99.8%). NH blacks experienced increased midlife mortality from 17 causes, including drug overdoses (149.6%), homicides (21.4%), hypertensive diseases (15.5%), obesity (120.7%), and liver cancer (49.5%). NH blacks also experienced retrogression: after a period of stable or declining midlife mortality early in 1999-2016, death rates increased for alcohol related liver disease, chronic lower respiratory tract disease, suicides, diabetes, and pancreatic cancer. Among Hispanics, midlife mortality increased across 12 causes, including drug overdoses (80.0%), hypertensive diseases (40.6%), liver cancer (41.8%), suicides (21.9%), obesity (106.6%), and metabolic disorders (60.0%). Retrogression also occurred in this population; after a period of declining mortality, death rates increased for alcohol related liver disease, mental and behavioral disorders involving psychoactive substances, and homicides. NH Asians and Pacific Islanders were least affected by this trend but also experienced increases in midlife mortality from drug overdoses (300.6%), alcohol related liver disease (62.9%), hypertensive diseases (28.3%), and brain cancer (56.6%). The suicide rate in this group increased by 29.7% after 2001. The relative increase in US midlife mortality differed by sex and geography. For example, the relative increase in fatal drug overdoses was greater among women than among men. Although the relative increase in midlife mortality was generally greater in non-metropolitan (ie, rural) areas, the relative increase in drug overdoses among NH whites and Hispanics was greatest in suburban fringe areas of large cities, and among NH blacks was greatest in small cities. CONCLUSIONS: Mortality in midlife in the US has increased across racial-ethnic populations for a variety of conditions, especially in recent years, offsetting years of progress in lowering mortality rates. This reversal carries added consequences for racial groups with high baseline mortality rates, such as for NH blacks and NH American Indians and Alaskan Natives. That death rates are increasing throughout the US population for dozens of conditions signals a systemic cause and warrants prompt action by policy makers to tackle the factors responsible for declining health in the US.

A Patient and Provider Research Agenda on Diabetes and Hypertension Management.

INTRODUCTION: A demonstration project in Richmond, Virginia involved patients and other stakeholders in the creation of a research agenda on dietary and behavioral management of diabetes and hypertension. Given the impact of these diseases on morbidity and mortality, considerable research has been directed at the challenges patients face in chronic disease management. The continuing need to understand disparities and find evidence-based interventions to improve outcomes has been fruitful, but disparities and unmet needs persist. METHODS: The Stakeholder Engagement in Question Development (SEED) method is a stakeholder engagement methodology that combines engagement with a review of available evidence to generate research questions that address current research gaps and are important to patients and other stakeholders. Using the SEED method, patients and other stakeholders participated in research question development through a combination of collaborative, participatory, and consultative engagement. Steps in the process included: (1) identifying the topic and recruiting participants; (2) conducting focus groups and interviews; (3) developing conceptual models; (4) developing research questions; and (5) prioritizing research questions. RESULTS: Stakeholders were involved in the SEED process from February to August 2015. Eighteen questions were prioritized for inclusion in the research agenda, covering diverse domains, from healthcare provision to social and environmental factors. Data analysis took place September to May 2016. During this time, researchers conducted a literature review to target research gaps. CONCLUSIONS: The stakeholder-prioritized, novel research questions developed through the SEED process can directly inform future research and guide the development of evidence that translates more directly to clinical practice.

Translating Research into Action: A Framework for Research That Supports Advances In Population Health.

The research community faces a growing need to deliver useful data and actionable evidence to support health systems and policymakers on ways to optimize the health of populations. Translating science into policy has not been the traditional strong suit of investigators, who typically view a journal publication as the endpoint of their work. They are less accustomed to seeing their data as an input to the work of communities and policymakers to improve population health. This article offers four suggestions as potential solutions: (1) shaping a research portfolio around user needs, (2) understanding the decision-making environment, (3) engaging stakeholders, and (4) strategic communication.

Translating evidence into population health improvement: strategies and barriers.

Among the challenges facing research translation-the effort to move evidence into policy and practice-is that key questions chosen by investigators and funders may not always align with the information priorities of decision makers, nor are the findings always presented in a form that is useful for or relevant to the decisions at hand. This disconnect is a problem particularly for population health, where the change agents who can make the biggest difference in improving health behaviors and social and environmental conditions are generally nonscientists outside of the health professions. To persuade an audience that does not read scientific journals, strong science may not be enough to elicit change. Achieving influence in population health often requires four ingredients for success: research that is responsive to user needs, an understanding of the decision-making environment, effective stakeholder engagement, and strategic communication. This article reviews the principles and provides examples from a national and local initiative.