Evaluation of a semi-automated data extraction tool for public health literature-based reviews: Dextr.

INTRODUCTION: There has been limited development and uptake of machine-learning methods to automate data extraction for literature-based assessments. Although advanced extraction approaches have been applied to some clinical research reviews, existing methods are not well suited for addressing toxicology or environmental health questions due to unique data needs to support reviews in these fields. OBJECTIVES: To develop and evaluate a flexible, web-based tool for semi-automated data extraction that: 1) makes data extraction predictions with user verification, 2) integrates token-level annotations, and 3) connects extracted entities to support hierarchical data extraction. METHODS: Dextr was developed with Agile software methodology using a two-team approach. The development team outlined proposed features and coded the software. The advisory team guided developers and evaluated Dextr's performance on precision, recall, and extraction time by comparing a manual extraction workflow to a semi-automated extraction workflow using a dataset of 51 environmental health animal studies. RESULTS: The semi-automated workflow did not appear to affect precision rate (96.0% vs. 95.4% manual, p = 0.38), resulted in a small reduction in recall rate (91.8% vs. 97.0% manual, p < 0.01), and substantially reduced the median extraction time (436 s vs. 933 s per study manual, p < 0.01) compared to a manual workflow. DISCUSSION: Dextr provides similar performance to manual extraction in terms of recall and precision and greatly reduces data extraction time. Unlike other tools, Dextr provides the ability to extract complex concepts (e.g., multiple experiments with various exposures and doses within a single study), properly connect the extracted elements within a study, and effectively limit the work required by researchers to generate machine-readable, annotated exports. The Dextr tool addresses data-extraction challenges associated with environmental health sciences literature with a simple user interface, incorporates the key capabilities of user verification and entity connecting, provides a platform for further automation developments, and has the potential to improve data extraction for literature reviews in this and other fields.

Recommendation mapping of the World Health Organization's guidelines on tuberculosis: A new approach to digitizing and presenting recommendations.

OBJECTIVE: Having up-to-date health policy recommendations accessible in one location is in high demand by guideline users. We developed an easy to navigate interactive approach to organize recommendations and applied it to tuberculosis (TB) guidelines of the World Health Organization (WHO). STUDY DESIGN: We used a mixed-methods study design to develop a framework for recommendation mapping with seven key methodological considerations. We define a recommendation map as an online repository of recommendations from several guidelines on a condition, providing links to the underlying evidence and expert judgments that inform them, allowing users to filter and cross-tabulate the search results. We engaged guideline developers, users, and health software engineers in an iterative process to elaborate the WHO eTB recommendation map. RESULTS: Applying the seven-step framework, we included 228 recommendations, linked to 103 guideline questions and organized the recommendation map according to key components of the health question, including the original recommendations and rationale (https://who.tuberculosis.recmap.org/). CONCLUSION: The recommendation mapping framework provides the entire continuum of evidence mapping by framing recommendations within a guideline questions' population, interventions, and comparators domains. Recommendation maps should allow guideline developers to organize their work meaningfully, standardize the automated publication of guidelines through links to the GRADEpro guideline development tool, and increase their accessibility and usability.