Accuracy and Efficiency of Machine Learning-Assisted Risk-of-Bias Assessments in "Real-World" Systematic Reviews : A Noninferiority Randomized Controlled Trial.

BACKGROUND: Automation is a proposed solution for the increasing difficulty of maintaining up-to-date, high-quality health evidence. Evidence assessing the effectiveness of semiautomated data synthesis, such as risk-of-bias (RoB) assessments, is lacking. OBJECTIVE: To determine whether RobotReviewer-assisted RoB assessments are noninferior in accuracy and efficiency to assessments conducted with human effort only. DESIGN: Two-group, parallel, noninferiority, randomized trial. (Monash Research Office Project 11256). SETTING: Health-focused systematic reviews using Covidence. PARTICIPANTS: Systematic reviewers, who had not previously used RobotReviewer, completing Cochrane RoB assessments between February 2018 and May 2020. INTERVENTION: In the intervention group, reviewers received an RoB form prepopulated by RobotReviewer; in the comparison group, reviewers received a blank form. Studies were assigned in a 1:1 ratio via simple randomization to receive RobotReviewer assistance for either Reviewer 1 or Reviewer 2. Participants were blinded to study allocation before starting work on each RoB form. MEASUREMENTS: Co-primary outcomes were the accuracy of individual reviewer RoB assessments and the person-time required to complete individual assessments. Domain-level RoB accuracy was a secondary outcome. RESULTS: Of the 15 recruited review teams, 7 completed the trial (145 included studies). Integration of RobotReviewer resulted in noninferior overall RoB assessment accuracy (risk difference, -0.014 [95% CI, -0.093 to 0.065]; intervention group: 88.8% accurate assessments; control group: 90.2% accurate assessments). Data were inconclusive for the person-time outcome (RobotReviewer saved 1.40 minutes [CI, -5.20 to 2.41 minutes]). LIMITATION: Variability in user behavior and a limited number of assessable reviews led to an imprecise estimate of the time outcome. CONCLUSION: In health-related systematic reviews, RoB assessments conducted with RobotReviewer assistance are noninferior in accuracy to those conducted without RobotReviewer assistance. PRIMARY FUNDING SOURCE: University College London and Monash University.

Interventions to minimize blood loss in very preterm infants-A systematic review and meta-analysis.

Blood loss in the first days of life has been associated with increased morbidity and mortality in very preterm infants. In this systematic review we included randomized controlled trials comparing the effects of interventions to preserve blood volume in the infant from birth, reduce the need for sampling, or limit the blood sampled. Mortality and major neurodevelopmental disabilities were the primary outcomes. Included studies underwent risk of bias-assessment and data extraction by two review authors independently. We used risk ratio or mean difference to evaluate the treatment effect and meta-analysis for pooled results. The certainty of evidence was assessed using GRADE. We included 31 trials enrolling 3,759 infants. Twenty-five trials were pooled in the comparison delayed cord clamping or cord milking vs. immediate cord clamping or no milking. Increasing placental transfusion resulted in lower mortality during the neonatal period (RR 0.51, 95% CI 0.26 to 1.00; participants = 595; trials = 5; I2 = 0%, moderate certainty of evidence) and during first hospitalization (RR 0.70, 95% CI 0.51, 0.96; 10 RCTs, participants = 2,476, low certainty of evidence). The certainty of evidence was very low for the other primary outcomes of this review. The six remaining trials compared devices to monitor glucose levels (three trials), blood sampling from the umbilical cord or from the placenta vs. blood sampling from the infant (2 trials), and devices to reintroduce the blood after analysis vs. conventional blood sampling (1 trial); the certainty of evidence was rated as very low for all outcomes in these comparisons. Increasing placental transfusion at birth may reduce mortality in very preterm infants; However, extremely limited evidence is available to assess the effects of other interventions to reduce blood loss after birth. In future trials, infants could be randomized following placental transfusion to different blood saving approaches. Trial registration: PROSPERO CRD42020159882.

The views of health guideline developers on the use of automation in health evidence synthesis.

BACKGROUND: The increasingly rapid rate of evidence publication has made it difficult for evidence synthesis-systematic reviews and health guidelines-to be continually kept up to date. One proposed solution for this is the use of automation in health evidence synthesis. Guideline developers are key gatekeepers in the acceptance and use of evidence, and therefore, their opinions on the potential use of automation are crucial. METHODS: The objective of this study was to analyze the attitudes of guideline developers towards the use of automation in health evidence synthesis. The Diffusion of Innovations framework was chosen as an initial analytical framework because it encapsulates some of the core issues which are thought to affect the adoption of new innovations in practice. This well-established theory posits five dimensions which affect the adoption of novel technologies: Relative Advantage, Compatibility, Complexity, Trialability, and Observability. Eighteen interviews were conducted with individuals who were currently working, or had previously worked, in guideline development. After transcription, a multiphase mixed deductive and grounded approach was used to analyze the data. First, transcripts were coded with a deductive approach using Rogers' Diffusion of Innovation as the top-level themes. Second, sub-themes within the framework were identified using a grounded approach. RESULTS: Participants were consistently most concerned with the extent to which an innovation is in line with current values and practices (i.e., Compatibility in the Diffusion of Innovations framework). Participants were also concerned with Relative Advantage and Observability, which were discussed in approximately equal amounts. For the latter, participants expressed a desire for transparency in the methodology of automation software. Participants were noticeably less interested in Complexity and Trialability, which were discussed infrequently. These results were reasonably consistent across all participants. CONCLUSIONS: If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.

Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research.

Background: Identifying new, eligible studies for integration into living systematic reviews and maps usually relies on conventional Boolean updating searches of multiple databases and manual processing of the updated results. Automated searches of one, comprehensive, continuously updated source, with adjunctive machine learning, could enable more efficient searching, selection and prioritisation workflows for updating (living) reviews and maps, though research is needed to establish this. Microsoft Academic Graph (MAG) is a potentially comprehensive single source which also contains metadata that can be used in machine learning to help efficiently identify eligible studies. This study sought to establish whether: (a) MAG was a sufficiently sensitive single source to maintain our living map of COVID-19 research; and (b) eligible records could be identified with an acceptably high level of specificity. Methods: We conducted an eight-arm cost-effectiveness analysis to assess the costs, recall and precision of semi-automated workflows, incorporating MAG with adjunctive machine learning, for continually updating our living map. Resource use data (time use) were collected from information specialists and other researchers involved in map production. Our systematic review software, EPPI-Reviewer, was adapted to incorporate MAG and associated machine learning workflows, and also used to collect data on recall, precision, and manual screening workload. Results: The semi-automated MAG-enabled workflow dominated conventional workflows in both the base case and sensitivity analyses. At one month our MAG-enabled workflow with machine learning, active learning and fixed screening targets identified 469 additional, eligible articles for inclusion in our living map, and cost £3,179 GBP per week less, compared with conventional methods relying on Boolean searches of Medline and Embase. Conclusions: We were able to increase recall and coverage of a large living map, whilst reducing its production costs. This finding is likely to be transferrable to OpenAlex, MAG's successor database platform.

The efficacy of nudge theory strategies in influencing adult dietary behaviour: a systematic review and meta-analysis.

BACKGROUND: Obesity has become a world-wide epidemic and is spreading to countries with emerging economies. Previously tested interventions are often too costly to maintain in the long term. This leaves a need for improved strategies for management of the epidemic. Nudge Theory presents a new collection of methods, deemed "nudges", which have the potential for low-cost and broad application to guide healthier lifestyle choices without the need for restrictive regulation. There has not yet been a large-scale examination of the effectiveness of nudges, despite several policy making bodies now considering their use. METHODS: To address this gap in knowledge, an adapted systematic review methodology was used to collect and consolidate results from current Nudge papers and to determine whether Nudge strategies are successful in changing adults' dietary choices for healthier ones. RESULTS: It was found that nudges resulted in an average 15.3 % increase in healthier dietary or nutritional choices, as measured by a change in frequency of healthy choices or a change in overall caloric consumption. All of the included studies were from wealthy nations, with a particular emphasis on the United States with 31 of 42 included experiments. CONCLUSIONS: This analysis demonstrates Nudge holds promise as a public health strategy to combat obesity. More research is needed in varied settings, however, and future studies should aim to replicate previous results in more geographically and socioeconomically diverse countries.

A new approach for the study of lung smooth muscle phenotypes and its application in a murine model of allergic airway inflammation.

Phenotypes of lung smooth muscle cells in health and disease are poorly characterized. This is due, in part, to a lack of methodologies that allow for the independent and direct isolation of bronchial smooth muscle cells (BSMCs) and vascular smooth muscle cells (VSMCs) from the lung. In this paper, we describe the development of a bi-fluorescent mouse that permits purification of these two cell populations by cell sorting. By subjecting this mouse to an acute allergen based-model of airway inflammation that exhibits many features of asthma, we utilized this tool to characterize the phenotype of so-called asthmatic BSMCs. First, we examined the biophysical properties of single BSMCs from allergen sensitized mice and found increases in basal tone and cell size that were sustained ex vivo. We then generated for the first time, a comprehensive characterization of the global gene expression changes in BSMCs isolated from the bi-fluorescent mice with allergic airway inflammation. Using statistical methods and pathway analysis, we identified a number of differentially expressed mRNAs in BSMCs from allergen sensitized mice that code for key candidate proteins underlying changes in matrix formation, contractility, and immune responses. Ultimately, this tool will provide direction and guidance for the logical development of new markers and approaches for studying human lung smooth muscle.