Your browser doesn't support javascript.
Show: 20 | 50 | 100
Results 1 - 10 de 10
Filter
1.
EuropePMC; 2021.
Preprint in English | EuropePMC | ID: ppcovidwho-313630

ABSTRACT

Automatic scoring engines have been used for scoring approximately fifteen million test-takers in just the last three years. This number is increasing further due to COVID-19 and the associated automation of education and testing. Despite such wide usage, the AI-based testing literature of these "intelligent" models is highly lacking. Most of the papers proposing new models rely only on quadratic weighted kappa (QWK) based agreement with human raters for showing model efficacy. However, this effectively ignores the highly multi-feature nature of essay scoring. Essay scoring depends on features like coherence, grammar, relevance, sufficiency and, vocabulary. To date, there has been no study testing Automated Essay Scoring: AES systems holistically on all these features. With this motivation, we propose a model agnostic adversarial evaluation scheme and associated metrics for AES systems to test their natural language understanding capabilities and overall robustness. We evaluate the current state-of-the-art AES models using the proposed scheme and report the results on five recent models. These models range from feature-engineering-based approaches to the latest deep learning algorithms. We find that AES models are highly overstable. Even heavy modifications(as much as 25%) with content unrelated to the topic of the questions do not decrease the score produced by the models. On the other hand, irrelevant content, on average, increases the scores, thus showing that the model evaluation strategy and rubrics should be reconsidered. We also ask 200 human raters to score both an original and adversarial response to seeing if humans can detect differences between the two and whether they agree with the scores assigned by auto scores.

2.
EuropePMC; 2021.
Preprint in English | EuropePMC | ID: ppcovidwho-295083

ABSTRACT

Automatic scoring engines have been used for scoring approximately fifteen million test-takers in just the last three years. This number is increasing further due to COVID-19 and the associated automation of education and testing. Despite such wide usage, the AI-based testing literature of these "intelligent" models is highly lacking. Most of the papers proposing new models rely only on quadratic weighted kappa (QWK) based agreement with human raters for showing model efficacy. However, this effectively ignores the highly multi-feature nature of essay scoring. Essay scoring depends on features like coherence, grammar, relevance, sufficiency and, vocabulary. To date, there has been no study testing Automated Essay Scoring: AES systems holistically on all these features. With this motivation, we propose a model agnostic adversarial evaluation scheme and associated metrics for AES systems to test their natural language understanding capabilities and overall robustness. We evaluate the current state-of-the-art AES models using the proposed scheme and report the results on five recent models. These models range from feature-engineering-based approaches to the latest deep learning algorithms. We find that AES models are highly overstable. Even heavy modifications(as much as 25%) with content unrelated to the topic of the questions do not decrease the score produced by the models. On the other hand, irrelevant content, on average, increases the scores, thus showing that the model evaluation strategy and rubrics should be reconsidered. We also ask 200 human raters to score both an original and adversarial response to seeing if humans can detect differences between the two and whether they agree with the scores assigned by auto scores.

5.
J Infect ; 83(1): 96-103, 2021 07.
Article in English | MEDLINE | ID: covidwho-1198895

ABSTRACT

OBJECTIVES: Patients requiring haemodialysis are at increased risk of serious illness with SARS-CoV-2 infection. To improve the understanding of transmission risks in six Scottish renal dialysis units, we utilised the rapid whole-genome sequencing data generated by the COG-UK consortium. METHODS: We combined geographical, temporal and genomic sequence data from the community and hospital to estimate the probability of infection originating from within the dialysis unit, the hospital or the community using Bayesian statistical modelling and compared these results to the details of epidemiological investigations. RESULTS: Of 671 patients, 60 (8.9%) became infected with SARS-CoV-2, of whom 16 (27%) died. Within-unit and community transmission were both evident and an instance of transmission from the wider hospital setting was also demonstrated. CONCLUSIONS: Near-real-time SARS-CoV-2 sequencing data can facilitate tailored infection prevention and control measures, which can be targeted at reducing risk in these settings.


Subject(s)
COVID-19 , SARS-CoV-2 , Bayes Theorem , Hospitals , Humans , Molecular Epidemiology , Renal Dialysis/adverse effects
6.
Cell ; 184(1): 64-75.e11, 2021 01 07.
Article in English | MEDLINE | ID: covidwho-1064909

ABSTRACT

Global dispersal and increasing frequency of the SARS-CoV-2 spike protein variant D614G are suggestive of a selective advantage but may also be due to a random founder effect. We investigate the hypothesis for positive selection of spike D614G in the United Kingdom using more than 25,000 whole genome SARS-CoV-2 sequences. Despite the availability of a large dataset, well represented by both spike 614 variants, not all approaches showed a conclusive signal of positive selection. Population genetic analysis indicates that 614G increases in frequency relative to 614D in a manner consistent with a selective advantage. We do not find any indication that patients infected with the spike 614G variant have higher COVID-19 mortality or clinical severity, but 614G is associated with higher viral load and younger age of patients. Significant differences in growth and size of 614G phylogenetic clusters indicate a need for continued study of this variant.


Subject(s)
Amino Acid Substitution , COVID-19/transmission , COVID-19/virology , SARS-CoV-2/genetics , SARS-CoV-2/pathogenicity , Spike Glycoprotein, Coronavirus/genetics , Aspartic Acid/analysis , Aspartic Acid/genetics , COVID-19/epidemiology , Genome, Viral , Glycine/analysis , Glycine/genetics , Humans , Mutation , SARS-CoV-2/growth & development , United Kingdom/epidemiology , Virulence , Whole Genome Sequencing
8.
Cell ; 184(5): 1171-1187.e20, 2021 03 04.
Article in English | MEDLINE | ID: covidwho-1051523

ABSTRACT

SARS-CoV-2 can mutate and evade immunity, with consequences for efficacy of emerging vaccines and antibody therapeutics. Here, we demonstrate that the immunodominant SARS-CoV-2 spike (S) receptor binding motif (RBM) is a highly variable region of S and provide epidemiological, clinical, and molecular characterization of a prevalent, sentinel RBM mutation, N439K. We demonstrate N439K S protein has enhanced binding affinity to the hACE2 receptor, and N439K viruses have similar in vitro replication fitness and cause infections with similar clinical outcomes as compared to wild type. We show the N439K mutation confers resistance against several neutralizing monoclonal antibodies, including one authorized for emergency use by the US Food and Drug Administration (FDA), and reduces the activity of some polyclonal sera from persons recovered from infection. Immune evasion mutations that maintain virulence and fitness such as N439K can emerge within SARS-CoV-2 S, highlighting the need for ongoing molecular surveillance to guide development and usage of vaccines and therapeutics.


Subject(s)
COVID-19/immunology , Genetic Fitness , Immune Evasion , SARS-CoV-2/genetics , Spike Glycoprotein, Coronavirus/genetics , Angiotensin-Converting Enzyme 2/chemistry , Antibodies, Neutralizing/genetics , Antibodies, Neutralizing/immunology , Antibodies, Viral/immunology , COVID-19/virology , Humans , Mutation , Phylogeny , SARS-CoV-2/chemistry , SARS-CoV-2/pathogenicity , Spike Glycoprotein, Coronavirus/chemistry , Virulence
10.
Nat Microbiol ; 6(1): 112-122, 2021 01.
Article in English | MEDLINE | ID: covidwho-989837

ABSTRACT

Coronavirus disease 2019 (COVID-19) was first diagnosed in Scotland on 1 March 2020. During the first month of the outbreak, 2,641 cases of COVID-19 led to 1,832 hospital admissions, 207 intensive care admissions and 126 deaths. We aimed to identify the source and number of introductions of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) into Scotland using a combined phylogenetic and epidemiological approach. Sequencing of 1,314 SARS-CoV-2 viral genomes from available patient samples enabled us to estimate that SARS-CoV-2 was introduced to Scotland on at least 283 occasions during February and March 2020. Epidemiological analysis confirmed that early introductions of SARS-CoV-2 originated from mainland Europe (the majority from Italy and Spain). We identified subsequent early outbreaks in the community, within healthcare facilities and at an international conference. Community transmission occurred after 2 March, 3 weeks before control measures were introduced. Earlier travel restrictions or quarantine measures, both locally and internationally, would have reduced the number of COVID-19 cases in Scotland. The risk of multiple reintroduction events in future waves of infection remains high in the absence of population immunity.


Subject(s)
COVID-19/epidemiology , COVID-19/virology , SARS-CoV-2/genetics , Adult , Aged , Europe/epidemiology , Genome, Viral , High-Throughput Nucleotide Sequencing , Humans , Male , Middle Aged , Molecular Epidemiology , Phylogeny , SARS-CoV-2/isolation & purification , Spain/epidemiology , Travel/statistics & numerical data
SELECTION OF CITATIONS
SEARCH DETAIL