Applying machine-learning to rapidly analyse large qualitative text datasets to inform the COVID-19 pandemic response: Comparing human and machine-assisted topic analysis techniques

BackgroundMachine-assisted topic analysis (MATA) uses artificial intelligence methods to assist qualitative researchers to analyse large amounts of textual data. This could allow qualitative researchers to inform and update public health interventions in real-time, to ensure they remain acceptable and effective during rapidly changing contexts (such as a pandemic). In this novel study we aimed to understand the potential for such approaches to support intervention implementation, by directly comparing MATA and human-only thematic analysis techniques when applied to the same dataset (1472 free-text responses from users of the COVID-19 infection control intervention Germ Defence). MethodsIn MATA, the analysis process included an unsupervised topic modelling approach to identify latent topics in the text. The human research team then described the topics and identified broad themes. In human-only codebook analysis, an initial codebook was developed by an experienced qualitative researcher and applied to the dataset by a well-trained research team, who met regularly to critique and refine the codes. To understand similarities and difference, formal triangulation using a convergence coding matrix compared the findings from both methods, categorising them as agreement, complementary, dissonant, or silent. ResultsHuman analysis took much longer (147.5 hours) than MATA (40 hours). Both human-only and MATA identified key themes about what users found helpful and unhelpful (e.g. Boosting confidence in how to perform the behaviours vs Lack of personally relevant content). Formal triangulation of the codes created showed high similarity between the findings. All codes developed from the MATA were classified as in agreement or complementary to the human themes. Where the findings were classified as complementary, this was typically due to slightly differing interpretations or nuance present in the human-only analysis. ConclusionsOverall, the quality of MATA was as high as the human-only thematic analysis, with substantial time savings. For simple analyses that do not require an in-depth or subtle understanding of the data, MATA is a useful tool that can support qualitative researchers to interpret and analyse large datasets quickly. These findings have practical implications for intervention development and implementation, such as enabling rapid optimisation during public health emergencies. Contributions to the literatureO_LINatural language processing (NLP) techniques have been applied within health research due to the need to rapidly analyse large samples of qualitative data. However, the extent to which these techniques lead to results comparable to human coding requires further assessment. C_LIO_LIWe demonstrate that combining NLP with human analysis to analyse free-text data can be a trustworthy and efficient method to use on large quantities of qualitative data. C_LIO_LIThis method has the potential to play an important role in contexts where rapid descriptive or exploratory analysis of very large datasets is required, such as during a public health emergency. C_LI

Effectiveness of digital interventions to improve household and community infection prevention and control behaviours and to reduce incidence of respiratory and/or gastro-intestinal infections: A rapid systematic review

BackgroundDigital interventions have potential to efficiently support improved hygiene practices to reduce transmission of COVID-19. ObjectiveTo evaluate the evidence for digital interventions to improve hygiene practices within the community. MethodsWe reviewed articles published between 01 January 2000 and 26 May 2019 that presented a controlled trial of a digital intervention to improve hygiene behaviours in the community. We searched MEDLINE, Embase, PsycINFO, Cochrane Controlled Register of Trials (CENTRAL), China National Knowledge Infrastructure and grey literature. Trials in hospitals were excluded, as were trials aiming at prevention of sexually transmitted infections; only target diseases with transmission mechanisms similar to COVID-19 (e.g. respiratory and gastrointestinal infections) were included. Trials had to evaluate a uniquely digital component of an intervention. Study designs were limited to randomised controlled trials, controlled before-and-after trials, and interrupted time series analyses. Outcomes could be either incidence of infections or change in hygiene behaviours. The Risk of Bias 2 tool was used to assess study quality. ResultsWe found seven studies that met the inclusion criteria. Six studies reported successfully improving self-reported hygiene behaviour or health outcomes, but only one of these six trials confirmed improvements using objective measures (reduced consultations and antibiotic prescriptions), Germ Defence. Settings included kindergartens, workplaces, and service station restrooms. Modes of delivery were diverse: WeChat, website, text messages, audio messages to mobiles, electronic billboards, and electronic personal care records. Four interventions targeted parents of young children with educational materials. Two targeted the general population; these also used behaviour change techniques or theory to inform the intervention. Only one trial had low risk of bias, Germ Defence; the most common concerns were lack of information about the randomisation, possible bias in reporting of behavioural outcomes, and lack of an analysis plan and possible selective reporting of results. ConclusionThere was only one intervention that was judged to be at low risk of bias, Germ Defence, which reduced incidence and severity of illness, as confirmed by objective measures. Further evaluation is required to determine the effectiveness of the other interventions reviewed.

Current infection control behaviour patterns in the UK, and how they can be improved by 'Germ Defence', an online behavioural intervention to reduce the spread of COVID-19 in the home.

BackgroundGerm Defence (https://germdefence.org/) is a freely available website providing behavioural advice for infection control within households, using behaviour change techniques. This observational study reports current infection control behaviours in the home in UK and international users of the website, and examine how they might be improved to reduce the spread of COVID-19. Method28,285 users sought advice from four website pathways (to protect themselves generally, to protect others if the user was showing symptoms, to protect themselves if household members were showing symptoms, and to protect a household member who is at high risk) and completed outcome measures of current infection control behaviours within the home (self-isolation, social distancing, putting shopping/packages aside, wearing face-covering, cleaning and disinfecting, handwashing), and intentions to change these behaviours. ResultsCurrent user behaviours mean scores varied across all infection control measures but were between sometimes and quite often, except handwashing ( very often). Behaviours were similar regardless of the website pathway used. After using Germ Defence, users recorded intentions to improve infection control behaviour across all website pathways and for all behaviours. ConclusionsSelf-reported infection control behaviours other than handwashing are lower than is optimal for infection prevention, although reported handwashing is much higher. The advice using behaviour change techniques in Germ Defence led to intentions to improve these behaviours. This has been shown previously to reduce the incidence, severity and transmission of infections. These findings suggest that promoting Germ Defence within national and local public health guidance could reduce COVID-19 transmission. O_TEXTBOXSection 1: What is already known on this topicO_LIUntil a vaccine can prevent COVID-19, protective behaviours (such as social distancing, handwashing, cleaning/disinfecting) must be used to limit the spread. C_LIO_LIA digital behaviour change intervention to improve protective behaviours (handwashing) within the home succeeded in reducing infection transmission, healthcare utilisation and infection severity during the H1N1 pandemic (the PRIMIT trial). C_LIO_LIWe need to understand current levels of protective behaviour in the UK, and how to improve them, to prevent a second wave. C_LI Section 2: What this study addsO_LIOur study suggests that few people are undertaking sufficient protective infection control behaviours in the home to reduce transmission C_LIO_LIProviding targeted digital interventions such as Germ Defence (for example through public health and primary care networks) offers a feasible method of increasing intentions to undertake these behaviours. C_LI C_TEXTBOX