An evaluation of two commercial deep learning-based information retrieval systems for COVID-19 literature.
J Am Med Inform Assoc
; 28(1): 132-137, 2021 01 15.
Article
in English
| MEDLINE | ID: covidwho-1066363
ABSTRACT
The COVID-19 pandemic has resulted in a tremendous need for access to the latest scientific information, leading to both corpora for COVID-19 literature and search engines to query such data. While most search engine research is performed in academia with rigorous evaluation, major commercial companies dominate the web search market. Thus, it is expected that commercial pandemic-specific search engines will gain much higher traction than academic alternatives, leading to questions about the empirical performance of these tools. This paper seeks to empirically evaluate two commercial search engines for COVID-19 (Google and Amazon) in comparison with academic prototypes evaluated in the TREC-COVID task. We performed several steps to reduce bias in the manual judgments to ensure a fair comparison of all systems. We find the commercial search engines sizably underperformed those evaluated under TREC-COVID. This has implications for trust in popular health search engines and developing biomedical search engines for future health crises.
Keywords
Full text:
Available
Collection:
International databases
Database:
MEDLINE
Main subject:
Information Systems
/
Information Storage and Retrieval
/
Search Engine
/
Deep Learning
/
COVID-19
Type of study:
Experimental Studies
Limits:
Humans
Language:
English
Journal:
J Am Med Inform Assoc
Journal subject:
Medical Informatics
Year:
2021
Document Type:
Article
Affiliation country:
Jamia
Similar
MEDLINE
...
LILACS
LIS