REFORMS: Consensus-based Recommendations for Machine-learning-based Science.

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

Representations of an Ebola 'outbreak' through Story Technologies.

BACKGROUND: Attempts to understand biosocial phenomena using scientific methods are often presented as value-neutral and objective; however, when used to reduce the complexity of open systems such as epidemics, these forms of inquiry necessarily entail normative considerations and are therefore fashioned by political worldviews (ideologies). From the standpoint of poststructural theory, the character of these representations is at most limited and partial. In addition, these modes of representation (as stories) do work (as technologies) in the service of, or in resistance to, power. METHODS: We focus on a single Ebola case cluster from the 2013-2016 outbreak in West Africa and examine how different disciplinary forms of knowledge production (including outbreak forecasting, active epidemiological surveillance, post-outbreak serosurveys, political economic analyses, and ethnography) function as Story Technologies. We then explore how these technologies are used to curate 'data,' analysing the erasures, values, and imperatives evoked by each. RESULTS: We call attention to the instrumental-in addition to the descriptive-role Story Technologies play in ordering contingencies and establishing relationships in the wake of health crises. DISCUSSION: By connecting each type of knowledge production with the systems of power it reinforces or disrupts, we illustrate how Story Technologies do ideological work. These findings encourage research from pluriversal perspectives and advocacy for measures that promote more inclusive modes of knowledge production.

The Role of Artificial Intelligence Model Documentation in Translational Science: Scoping Review.

BACKGROUND: Despite the touted potential of artificial intelligence (AI) and machine learning (ML) to revolutionize health care, clinical decision support tools, herein referred to as medical modeling software (MMS), have yet to realize the anticipated benefits. One proposed obstacle is the acknowledged gaps in AI translation. These gaps stem partly from the fragmentation of processes and resources to support MMS transparent documentation. Consequently, the absence of transparent reporting hinders the provision of evidence to support the implementation of MMS in clinical practice, thereby serving as a substantial barrier to the successful translation of software from research settings to clinical practice. OBJECTIVE: This study aimed to scope the current landscape of AI- and ML-based MMS documentation practices and elucidate the function of documentation in facilitating the translation of ethical and explainable MMS into clinical workflows. METHODS: A scoping review was conducted in accordance with PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. PubMed was searched using Medical Subject Headings key concepts of AI, ML, ethical considerations, and explainability to identify publications detailing AI- and ML-based MMS documentation, in addition to snowball sampling of selected reference lists. To include the possibility of implicit documentation practices not explicitly labeled as such, we did not use documentation as a key concept but as an inclusion criterion. A 2-stage screening process (title and abstract screening and full-text review) was conducted by 1 author. A data extraction template was used to record publication-related information; barriers to developing ethical and explainable MMS; available standards, regulations, frameworks, or governance strategies related to documentation; and recommendations for documentation for papers that met the inclusion criteria. RESULTS: Of the 115 papers retrieved, 21 (18.3%) papers met the requirements for inclusion. Ethics and explainability were investigated in the context of AI- and ML-based MMS documentation and translation. Data detailing the current state and challenges and recommendations for future studies were synthesized. Notable themes defining the current state and challenges that required thorough review included bias, accountability, governance, and explainability. Recommendations identified in the literature to address present barriers call for a proactive evaluation of MMS, multidisciplinary collaboration, adherence to investigation and validation protocols, transparency and traceability requirements, and guiding standards and frameworks that enhance documentation efforts and support the translation of AI- and ML-based MMS. CONCLUSIONS: Resolving barriers to translation is critical for MMS to deliver on expectations, including those barriers identified in this scoping review related to bias, accountability, governance, and explainability. Our findings suggest that transparent strategic documentation, aligning translational science and regulatory science, will support the translation of MMS by coordinating communication and reporting and reducing translational barriers, thereby furthering the adoption of MMS.

Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis.

To those involved in discussions about rigor, reproducibility, and replication in science, conversation about the "reproducibility crisis" appear ill-structured. Seemingly very different issues concerning the purity of reagents, accessibility of computational code, or misaligned incentives in academic research writ large are all collected up under this label. Prior work has attempted to address this problem by creating analytical definitions of reproducibility. We take a novel empirical, mixed methods approach to understanding variation in reproducibility discussions, using a combination of grounded theory and correspondence analysis to examine how a variety of authors narrate the story of the reproducibility crisis. Contrary to expectations, this analysis demonstrates that there is a clear thematic core to reproducibility discussions, centered on the incentive structure of science, the transparency of methods and data, and the need to reform academic publishing. However, we also identify three clusters of discussion that are distinct from the main body of articles: one focused on reagents, another on statistical methods, and a final cluster focused on the heterogeneity of the natural world. Although there are discursive differences between scientific and popular articles, we find no strong differences in how scientists and journalists write about the reproducibility crisis. Our findings demonstrate the value of using qualitative methods to identify the bounds and features of reproducibility discourse, and identify distinct vocabularies and constituencies that reformers should engage with to promote change.

Reparations for Black American descendants of persons enslaved in the U.S. and their potential impact on SARS-CoV-2 transmission.

BACKGROUND: In the United States, Black Americans are suffering from a significantly disproportionate incidence of COVID-19. Going beyond mere epidemiological tallying, the potential for racial-justice interventions, including reparations payments, to ameliorate these disparities has not been adequately explored. METHODS: We compared the COVID-19 time-varying Rt curves of relatively disparate polities in terms of social equity (South Korea vs. Louisiana). Next, we considered a range of reproductive ratios to back-calculate the transmission rates ßi→j for 4 cells of the simplified next-generation matrix (from which R0 is calculated for structured models) for the outbreak in Louisiana. Lastly, we considered the potential structural effects monetary payments as reparations for Black American descendants of persons enslaved in the U.S. would have had on pre-intervention ßi→j and consequently R0. RESULTS: Once their respective epidemics begin to propagate, Louisiana displays Rt values with an absolute difference of 1.3-2.5 compared to South Korea. It also takes Louisiana more than twice as long to bring Rt below 1. Reasoning through the consequences of increased equity via matrix transmission models, we demonstrate how the benefits of a successful reparations program (reflected in the ratio ßb→b/ßw→w) could reduce R0 by 31-68%. DISCUSSION: While there are compelling moral and historical arguments for racial-injustice interventions such as reparations, our study considers potential health benefits in the form of reduced SARS-CoV-2 transmission risk. A restitutive program targeted towards Black individuals would not only decrease COVID-19 risk for recipients of the wealth redistribution; the mitigating effects would also be distributed across racial groups, benefiting the population at large.

Reparations for Black American Descendants of Persons Enslaved in the U.S. and Their Estimated Impact on SARS-CoV-2 Transmission.

Background In the United States, Black Americans are suffering from significantly disproportionate incidence and mortality rates of COVID-19. The potential for racial-justice interventions, including reparations payments, to ameliorate these disparities has not been adequately explored. Methods We compared the COVID-19 time-varying R t curves of relatively disparate polities in terms of social equity (South Korea vs. Louisiana). Next, we considered a range of reproductive ratios to back-calculate the transmission rates ß i→j for 4 cells of the simplified next-generation matrix (from which R 0 is calculated for structured models) for the outbreak in Louisiana. Lastly, we modeled the effect that monetary payments as reparations for Black American descendants of persons enslaved in the U.S. would have had on pre-intervention ß i→j . Results Once their respective epidemics begin to propagate, Louisiana displays R t values with an absolute difference of 1.3 to 2.5 compared to South Korea. It also takes Louisiana more than twice as long to bring R t below 1. We estimate that increased equity in transmission consistent with the benefits of a successful reparations program (reflected in the ratio ß b→b / ß w→w ) could reduce R 0 by 31 to 68%. Discussion While there are compelling moral and historical arguments for racial injustice interventions such as reparations, our study describes potential health benefits in the form of reduced SARS-CoV-2 transmission risk. As we demonstrate, a restitutive program targeted towards Black individuals would not only decrease COVID-19 risk for recipients of the wealth redistribution; the mitigating effects would be distributed across racial groups, benefitting the population at large.