Challenges in Regression Test Selection for End-to-End Testing of Microservice-based Software Systems

ABSTRACT

Dynamic regression test selection (RTS) techniques aim to minimize testing efforts by selecting tests using per-test execution traces. However, most existing RTS techniques are not applicable to microservice-based, or, more generally, distributed systems, as the dynamic program analysis is typically limited to a single system. In this paper, we describe our distributed RTS approach, microRTS, which targets automated and manual end-to-end testing in microservice-based software systems. We employ microRTS in a case study on a set of 20 manual end-to-end test cases across 12 versions of the German COVID-19 contact tracing application, a modern microservice-based software system. The results indicate that initially microRTS selects all manual test cases for each version. Yet, through semi-automated filtering of test traces, we are able to effectively reduce the testing effort by 10-50%. In contrast with prior results on automated unit tests, we find method-level granularity of per-test execution traces to be more suitable than class-level for manual end-to-end testing. CCS CONCEPTS • Software and its engineering ${\rightarrow}$ Software testing and debugging. © 2022 ACM.