Research on Data acquisition technology of COVID-19 information Based on SCRAPY-REDIS

ABSTRACT

The novel coronavirus pneumonia has been spreading in China and worldwide. People have been paying close attention to the epidemic situation through the Internet. It is of great practical significance to obtain public opinion information and supervise the epidemic situation from the Internet. In this context, the improved SCRAPY-Redis framework and cluster technology are used to crawl epidemic data, analyze and filter epidemic sensitive words, and realize visualization through numpy, panda, Matplotlib and wordcloud. After experimental analysis and running test, the improved framework is proved to be more effective than the traditional scrapy framework in speed and performance, with stable performance and good operation. © 2021 IEEE.