Do online interviews lead to different results compared to personal interviews? Estimates of mode effects using a mixed mode survey

ABSTRACT

The increasing use of online surveys (CAWI) is a trend that became more pronounced in the wake of the COVID-19 pandemic. The switch from face-to-face interviews to online surveys raises the question of mode effects. Do the results of online surveys differ from those based on telephone interviews? This study examines this question using data from the AKCOVID Panel Survey, which has a mixed-mode design and was conducted in the first year of the COVID-19 pandemic. The results of regression analyses for 46 different variables controlling for the sample characteristics of CAWI and CATI show that mode effects are topic-specific. For some topics, significant and substantial mode effects are estimated (social trust, worries about the future, financial problems and health). When people are interviewed by telephone, they answer more socially desirable, express worries less strongly, assess their financial situation, social status and health better, report more trust in other people, and present themselves as more prosocial than in CAWI interviews. Not or less affected by mode effects are 'factual' questions about crisis-related changes in income without reference to one's financial situation, questions about working conditions, family relations, or a range of political attitudes. The paper concludes that results of interviewer-led surveys from pre-pandemic times can often not be directly compared with current results based on online surveys and points to the importance of heterogeneous mode effects that are hitherto under-researched. © 2023, The Author(s).