COMBINATORIAL TESTING
-Test note of “Essential Software Test Design”
2015-09-06
Content
16.1 Coverage
16.2 Combination Strategies
16.3 Input Parameter Modeling
16.3.1 Structural Input Parameter Modeling
16.3.2 Functional Input Parameter Modeling
16.4 Conflicts in the Input Parameter Model
16.5 Tools
The aim of combinatorial testing is to provide a solution to combinatorial explosion.
Example:
1. To illustrate combinatorial explosion during test case selection, consider testing the «classical» function triangle(side1, side2, side3).
2. Combinatorial explosion during configuration identification occurs in a similar manner. Consider testing (a part of) a Web based system which should support several:
- Different operating systems (Windows, MacOS, Unix, Linux),
- Different browsers (2 versions of Internet Explorer),
- Different types of data connections (modem, three ADSL speeds, and fiber optical connection).
Figure 16:1: Input parameter model of triangle example.
16.1 Coverage
The ability to vary coverage is a central concept in combinatorial testing. Higher coverage will give you better test quality but also more test cases/configurations. Figure 16:2 shows a subsumption hierarchy of the most commonly used coverage criteria used in combinatorial testing. A coverage criterion X subsumes another coverage criterion Y if and only if 100% coverage with respect to X automatically results in 100 % coverage with respect to Y. Subsumption is indicted in the figure by arrows.
Figure 16:2: Subsumption hierarchy for combination strategy coverage criteria
The simplest form of coverage used in combinatorial testing is 1-wise coverage. It requires each parameter value of every parameter to be included in at least one combination. Figure 16:3 shows a set of combinations that satisfies 1-wise coverage with respect to the model of the triangle problem in Figure 16:1.
Figure 16:3: 1-wise coverage of the triangle example.
A natural extension of 1-wise coverage is 2-wise (pair-wise) coverage. It requires every subcombination of values of any two (pair) of parameters to be included in the set of combinations.
Figure 16:4: 2-wise coverage of the triangle example.
Usually test cases containing more than one invalid value are less effective in detecting faults. The main reason is that code for error handling generally checks the input values one at a time and as soon as the first invalid input value is found the error handler is invoked and the execution is terminated.
Figure 16:6: 2-wise valid coverage of the triangle example.
Figure 16:7: Single error coverage of the triangle example.
The final coverage criterion included in the subsumption hierarchy in Figure 16:2 is called base choice coverage.
- The first step to satisfy base choice coverage is to identify a base combination. The base combination consists of the base values of each parameter.
- The tester is free to pick any value as the base value but it is recommended to pick the valid value that is most often used.
- From the base combination new combinations are derived by altering the values of one parameter at a time such that each value of that parameter is included in at least one combination while keeping the rest of the parameters fixed in their base values.
Figure 16:8 shows a set of combinations that satisfy base choice coverage under the assumption that the base combination is <A2, B2, C2>.
Figure 16:8: Base choice coverage of the triangle example.
16.2 Combination Strategies
Combination strategies are algorithms that select a set of combinations from an input parameter model, such that the selected set of combinations satisfy some coverage criterion. A goal with most combination strategies is to keep the set of combinations as small as possible. This goal is relatively easy to fulfill for simple coverage criteria, such as:
- 1-wise,
- base choice,
- single error,
- 1-wise valid.
In fact, for all of these coverage criteria it is possible to find minimal sets of combinations. In contrast, for the more complex coverage criteria, such as 2-wise, 2-wise valid and higher, the problem of finding a minimal set of combinations that satisfy the coverage criterion is more difficult.
More than 20 combination strategies exist. Some of the more known are Each Choice (EC) that satisfies 1-wise coverage, Base Choice (BC) that satisfies base choice coverage, Automatic Efficient Test Generator (AETG) ESSENTIAL SOFTWARE TEST DESIGN that satisfies t-wise coverage for t ≥ 2, and Orthogonal Arrays (OA) and In Parameter Order (IPO), both satisfying 2-wise coverage.
16.3 Input Parameter Modeling
One of the key success factors of combinatorial testing is input parameter modeling. In most cases it is possible to define several different models for the same test problem. The contents of the model have a direct impact on the quality of the test cases.
Figure 16:9: Alternative input parameter model for the triangle problem.
16.3.1 Structural Input Parameter Modeling
In structural input parameter modeling, the parameters in the interface of the test object are mapped one-to-one onto parameters in the input parameter model.
The main advantage with the structured approach is that it is a simple task to create the input parameter model.
The drawback with the structural approach is that there are no guarantees that the resulting test cases will take all aspects of the functionality of the test object into account.
16.3.2 Functional Input Parameter Modeling
In functional input parameter modeling it is not necessary to consider each input parameter in isolation and we are not restricted to only include properties of the input parameters in the model. Instead we can model aspects of the functionality and possibly the state of the test object.
The major advantage with the functional approach is that it generally results in test cases with higher quality than test cases from a structural model.
The two drawbacks are that it may be difficult to identify the input parameter model parameters and it may be difficult to translate a combination into actual inputs of the test case since the values of more than one parameter may affect the same input parameter.
16.4 Conflicts in the Input Parameter Model
Input parameter modeling sometimes result in models in which some of the sub-combinations are strange or even meaningless. Any impossible or semantically meaningless subcombination is called a conflict. As an example consider the model in Figure 16:9. The meaning of the sub-combination <A3, B1> is unclear.
Figure 16:10: The steps to replace conflicting combinations while preserving coverage. Conflicts marked in bold.
16.5 Tools
When deciding on tools there are several properties to look for：
- Which coverage criteria are supported by the tool.
- To what extent can the tool handle confl icts in the input parameter model?
- Does the tool support predefi ned combinations?
- Can the tool export its results in any known format?
- What infrastructure does the tool require?
- What does the tool cost?
combinatorial testing, which contains, among other things links to more than 20 different tool suppliers.
PICT is a free Perl-based tool from Microsoft supporting t-wise coverage as well as 2-wise valid coverage, single error coverage, and variable strength coverage. It supports conflicts and pre-selection of test cases. Despite the rather crude user interface this is probably the most powerful tool at the moment.
Telcordia offers a Web commercial, based service in which the user enters an input parameter model in a Web interface and a resulting set of combinations is returned to the user. This service is based on the AETG combination strategy, which means that there is support for t-wise testing for t≥2. There is also support for handling conflicts in the input parameter model.