Automated engineering of domain-specific metamorphic testing environments

Abstract

Context: Testing is essential to improve the correctness of software systems. Metamorphic testing (MT) is an approach especially suited when the system under test lacks oracles, or they are expensive to compute. However, building an MT environment for a particular domain (e.g., cloud simulation, model transformation, machine learning) requires substantial effort. Objective: Our goal is to facilitate the construction of MT environments for specific domains. Method: We propose a model-driven engineering approach to automate the construction of MT environments. Starting from a meta-model capturing the domain concepts, and a description of the domain execution environment, our approach produces an MT environment featuring comprehensive support for the MT process. This includes the definition of domain-specific metamorphic relations, their evaluation, detailed reporting of the testing results, and the automated search-based generation of follow-up test cases. Results: Our method is supported by an extensible platform for Eclipse, called GOTTEN. We demonstrate its effectiveness by creating an MT environment for simulation-based testing of data centres and comparing with existing tools; its suitability to conduct MT processes by replicating previous experiments; and its generality by building another MT environment for video streaming APIs. Conclusion: GOTTEN is the first platform targeted at reducing the development effort of domain-specific MT environments. The environments created with GOTTEN facilitate the specification of metamorphic relations, their evaluation, and the generation of new test cases