| Research |
|
|
My research interests include (automatic) software testing, especially in testing configurable software systems, applying and prioritizing combinatorial interaction testing, and the application of machine learning techniques in software engineering research. I am also interested in converting research ideas to real tools that are applicable for practitioners. CONFIGURATION AWARE PRIORITIZATION FOR REGRESSION TESTING Configurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Configurability requires extra effort for testing because there is evidence that running the same test case on different configurations may detect different faults. Differentiating test cases and configurations as two independent factors for testing, we must consider not just which test case to utilize, but also which configurations. Ideally, an exhaustive testing approach would combine every test case with every possible configuration. But since the full configuration space of most software systems is huge, it is infeasible to test all possible configurations with all test cases. Instead, sampling techniques are applied. Despite successful sampling techniques, sometimes it is still costly to run only a configuration sample. In particular, the cost is magnified when new features and functionality are added as a system evolves, and the new version is regression tested. Regression testing is an important but expensive way to build confidence that software changes introduce no new faults as software evolves, and many efforts have been made to improve its performance given limited resources. For example, regression test selection and test case prioritization have been extensively researched, but they have rarely been considered for configurations. In this dissertation, we provide cost-effective prioritization techniques for regression testing configurable systems -- configuration aware regression testing. Specifically, we first generalize the problem of configuration sampling, and systematically compare different sampling techniques for configurations in non-trivial software systems, across multiple consecutive versions. We then investigate different prioritization techniques for ordering configurations sampled. We also investigate the relative cost-benefits of prioritizing test cases and configurations, as two independent factors, by presenting a comprehensive method for prioritizing both test cases and configurations. These techniques and methods are evaluated through empirical studies. Finally, we extend or modify basic prioritization techniques for different practical environments that involve additional factors (e.g., cost) or constraints (e.g., without testing records of prior versions). Cost Effective Prioritization For Configurable Software Systems:The overall goal of this research is to provide a framework for configuration aware prioritization techniques (as shown in the figure below), and to investigate relative importance of different factors that have impact on the effectiveness of prioritization. To achieve this goal, we are performing the following activities, each is mapped to a box in the figure labeled with the same name: (1) sample full configuration space (sample): due to the large configuration space in most systems, it is infeasible to test all possible configurations so sampling of the configurations should occur instead. Many methods can be used to sample configurations: use a single default configuration, randomly select configurations, generate configurations by CIT or other techniques. The method of sampling should neither be too arbitrary nor too random. (2) develop prioritization techniques and execute a combined test in different orders (prioritize). (3) evaluate the framework and analyze the impacts of configurations and other factors (evaluate).
Configuration Aware Regression Testing:Configurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Researchers have been creating techniques for testing configurable software, but to date, only a little research has addressed the problems of regression testing configurable systems as they evolve. Whereas problems such as selective retesting and test prioritization at the test case level have been extensively researched, these problems have rarely been considered at the configuration level. In this research we address the problem of providing configuration-aware regression testing for evolving software systems. We use combinatorial interaction testing techniques to model and generate configuration samples for use in regression testing. We conduct an empirical study on a non-trivial evolving software system to measure the impact of changes in configurations on testing effectiveness, and to compare the effectiveness of different configuration prioritization techniques on early fault detection. Our results show that configurations can have a large impact on fault detection and that prioritization of configurations can be exective.Combinatorial Interaction Regression Testinng:Regression testing is an expensive part of the software maintenance process. Effective regression testing techniques select and order (or prioritize) test cases between successive releases of a program. However, selection and prioritization are dependent on the quality of the initial test suite. An effective and cost efficient test generation technique is combinatorial interaction testing, CIT, which systematically samples all t-way combinations of input parameters. Research on CIT, to date, has focused on single version software systems. There has been little work that empirically assesses the use of CIT test generation as the basis for selection or prioritization. In this research we examine the effectiveness of CIT across multiple versions of two software subjects. Our results show that CIT performs well in finding seeded faults when compared with an exhaustive test set. We examine several CIT prioritization techniques and compare them with a re-generation/prioritization technique. We find that prioritized and re-generated/prioritized CIT test suites may find faults earlier than unordered CIT test suites, although the re-generated/prioritized test suites sometimes exhibit decreased fault detection.Ant Colony System with Interaction Testing:In this project, we applied Ant Colony System to test suite generation for interaction testing.Combinatorial Design In Behaviroal Model Based Testing:In this project, we present a novel approach for generating interaction combinations based on constraint resolution. Our approach can generate maximal interaction coverage in the presence of unrestricted constraints as supported by the underlying solver. It supports seeding with general predicates, and can deal with projections on compound values as the combination factors. Our approach is motivated by the application to behavioral model-based testing in the Spec Explorer tool, where parameter combinations must be generated such that all path conditions of a model action have at least one combination which enables the path, which we ensure by seeding. It is applied in a large-scale project for model-based quality assurance of interoperability documentation at Microsoft. (I participated in this project when I was doing my internship at Microsoft Research with the Protocol Tools and Test Team.)Conference and Workshop Proceedings:
|
| Last updated by Xiao on Oct. 2011. |