Remote Tool-Based Adjudication for Grading Diabetic Retinopathy.

PURPOSE: To present and evaluate a remote, tool-based system and structured grading rubric for adjudicating image-based diabetic retinopathy (DR) grades. METHODS: We compared three different procedures for adjudicating DR severity assessments among retina specialist panels, including (1) in-person adjudication based on a previously described procedure (Baseline), (2) remote, tool-based adjudication for assessing DR severity alone (TA), and (3) remote, tool-based adjudication using a feature-based rubric (TA-F). We developed a system allowing graders to review images remotely and asynchronously. For both TA and TA-F approaches, images with disagreement were reviewed by all graders in a round-robin fashion until disagreements were resolved. Five panels of three retina specialists each adjudicated a set of 499 retinal fundus images (1 panel using Baseline, 2 using TA, and 2 using TA-F adjudication). Reliability was measured as grade agreement among the panels using Cohen's quadratically weighted kappa. Efficiency was measured as the number of rounds needed to reach a consensus for tool-based adjudication. RESULTS: The grades from remote, tool-based adjudication showed high agreement with the Baseline procedure, with Cohen's kappa scores of 0.948 and 0.943 for the two TA panels, and 0.921 and 0.963 for the two TA-F panels. Cases adjudicated using TA-F were resolved in fewer rounds compared with TA (P < 0.001; standard permutation test). CONCLUSIONS: Remote, tool-based adjudication presents a flexible and reliable alternative to in-person adjudication for DR diagnosis. Feature-based rubrics can help accelerate consensus for tool-based adjudication of DR without compromising label quality. TRANSLATIONAL RELEVANCE: This approach can generate reference standards to validate automated methods, and resolve ambiguous diagnoses by integrating into existing telemedical workflows.

Smartphone EEG and remote online interpretation for children with epilepsy in the Republic of Guinea: Quality, characteristics, and practice implications.

PURPOSE: Children with epilepsy in low-income countries often go undiagnosed and untreated. We examine a portable, low-cost smartphone-based EEG technology in a heterogeneous pediatric epilepsy cohort in the West African Republic of Guinea. METHODS: Children with epilepsy were recruited at the Ignace Deen Hospital in Conakry, 2017. Participants underwent sequential EEG recordings with an app-based EEG, the Smartphone Brain Scanner-2 (SBS2) and a standard Xltek EEG. Raw EEG data were transmitted via Bluetooth™ connection to an Android™ tablet and uploaded for remote EEG specialist review and reporting via a new, secure web-based reading platform, crowdEEG. The results were compared to same-visit Xltek 10-20 EEG recordings for identification of epileptiform and non-epileptiform abnormalities. RESULTS: 97 children meeting the International League Against Epilepsy's definition of epilepsy (49 male; mean age 10.3 years, 29 untreated with an antiepileptic drug; 0 with a prior EEG) were enrolled. Epileptiform discharges were detected on 21 (25.3%) SBS2 and 31 (37.3%) standard EEG recordings. The SBS2 had a sensitivity of 51.6% (95%CI 32.4%, 70.8%) and a specificity of 90.4% (95%CI 81.4%, 94.4%) for all types of epileptiform discharges, with positive and negative predictive values of 76.2% and 75.8% respectively. For generalized discharges, the SBS2 had a sensitivity of 43.5% with a specificity of 96.2%. CONCLUSIONS: The SBS2 has a moderate sensitivity and high specificity for the detection of epileptiform abnormalities in children with epilepsy in this low-income setting. Use of the SBS2+crowdEEG platform permits specialist input for patients with previously poor access to clinical neurophysiology expertise.

Herbarium specimens reveal substantial and unexpected variation in phenological sensitivity across the eastern United States.

Phenology is a key biological trait that can determine an organism's survival and provides one of the clearest indicators of the effects of recent climatic change. Long time-series observations of plant phenology collected at continental scales could clarify latitudinal and regional patterns of plant responses and illuminate drivers of that variation, but few such datasets exist. Here, we use the web tool CrowdCurio to crowdsource phenological data from over 7000 herbarium specimens representing 30 diverse flowering plant species distributed across the eastern United States. Our results, spanning 120 years and generated from over 2000 crowdsourcers, illustrate numerous aspects of continental-scale plant reproductive phenology. First, they support prior studies that found plant reproductive phenology significantly advances in response to warming, especially for early-flowering species. Second, they reveal that fruiting in populations from warmer, lower latitudes is significantly more phenologically sensitive to temperature than that for populations from colder, higher-latitude regions. Last, we found that variation in phenological sensitivities to climate within species between regions was of similar magnitude to variation between species. Overall, our results suggest that phenological responses to anthropogenic climate change will be heterogeneous within communities and across regions, with large amounts of regional variability driven by local adaptation, phenotypic plasticity and differences in species assemblages. As millions of imaged herbarium specimens become available online, they will play an increasingly critical role in revealing large-scale patterns within assemblages and across continents that ultimately can improve forecasts of the impacts of climatic change on the structure and function of ecosystems.This article is part of the theme issue 'Biological collections for understanding biodiversity in the Anthropocene'.

Crowds Replicate Performance of Scientific Experts Scoring Phylogenetic Matrices of Phenotypes.

Scientists building the Tree of Life face an overwhelming challenge to categorize phenotypes (e.g., anatomy, physiology) from millions of living and fossil species. This biodiversity challenge far outstrips the capacities of trained scientific experts. Here we explore whether crowdsourcing can be used to collect matrix data on a large scale with the participation of nonexpert students, or "citizen scientists." Crowdsourcing, or data collection by nonexperts, frequently via the internet, has enabled scientists to tackle some large-scale data collection challenges too massive for individuals or scientific teams alone. The quality of work by nonexpert crowds is, however, often questioned and little data have been collected on how such crowds perform on complex tasks such as phylogenetic character coding. We studied a crowd of over 600 nonexperts and found that they could use images to identify anatomical similarity (hypotheses of homology) with an average accuracy of 82% compared with scores provided by experts in the field. This performance pattern held across the Tree of Life, from protists to vertebrates. We introduce a procedure that predicts the difficulty of each character and that can be used to assign harder characters to experts and easier characters to a nonexpert crowd for scoring. We test this procedure in a controlled experiment comparing crowd scores to those of experts and show that crowds can produce matrices with over 90% of cells scored correctly while reducing the number of cells to be scored by experts by 50%. Preparation time, including image collection and processing, for a crowdsourcing experiment is significant, and does not currently save time of scientific experts overall. However, if innovations in automation or robotics can reduce such effort, then large-scale implementation of our method could greatly increase the collective scientific knowledge of species phenotypes for phylogenetic tree building. For the field of crowdsourcing, we provide a rare study with ground truth, or an experimental control that many studies lack, and contribute new methods on how to coordinate the work of experts and nonexperts. We show that there are important instances in which crowd consensus is not a good proxy for correctness.

CrowdCurio: an online crowdsourcing platform to facilitate climate change studies using herbarium specimens.

Phenology is a key aspect of plant success. Recent research has demonstrated that herbarium specimens can provide important information on plant phenology. Massive digitization efforts have the potential to greatly expand herbarium-based phenological research, but also pose a serious challenge regarding efficient data collection. Here, we introduce CrowdCurio, a crowdsourcing tool for the collection of phenological data from herbarium specimens. We test its utility by having workers collect phenological data (number of flower buds, open flowers and fruits) from specimens of two common New England (USA) species: Chelidonium majus and Vaccinium angustifolium. We assess the reliability of using nonexpert workers (i.e. Amazon Mechanical Turk) against expert workers. We also use these data to estimate the phenological sensitivity to temperature for both species across multiple phenophases. We found no difference in the data quality of nonexperts and experts. Nonexperts, however, were a more efficient way of collecting more data at lower cost. We also found that phenological sensitivity varied across both species and phenophases. Our study demonstrates the utility of CrowdCurio as a crowdsourcing tool for the collection of phenological data from herbarium specimens. Furthermore, our results highlight the insight gained from collecting large amounts of phenological data to estimate multiple phenophases.

Next-generation phenomics for the Tree of Life.

The phenotype represents a critical interface between the genome and the environment in which organisms live and evolve. Phenotypic characters also are a rich source of biodiversity data for tree building, and they enable scientists to reconstruct the evolutionary history of organisms, including most fossil taxa, for which genetic data are unavailable. Therefore, phenotypic data are necessary for building a comprehensive Tree of Life. In contrast to recent advances in molecular sequencing, which has become faster and cheaper through recent technological advances, phenotypic data collection remains often prohibitively slow and expensive. The next-generation phenomics project is a collaborative, multidisciplinary effort to leverage advances in image analysis, crowdsourcing, and natural language processing to develop and implement novel approaches for discovering and scoring the phenome, the collection of phentotypic characters for a species. This research represents a new approach to data collection that has the potential to transform phylogenetics research and to enable rapid advances in constructing the Tree of Life. Our goal is to assemble large phenomic datasets built using new methods and to provide the public and scientific community with tools for phenomic data assembly that will enable rapid and automated study of phenotypes across the Tree of Life.