Testing Part-4
1
Choosing Test Cases by Partitioning
Creating a good test suite is a challenging and interesting design problem. We want to pick a
set of test cases that is small enough to run quickly, yet large enough to validate the program.
To do this, we divide the input space into subdomains, each consisting of a set of inputs.
Taken together the subdomains completely cover the input space, so that every input lies in
at least one subdomain. Then we choose one test case from each subdomain, and that’s our
test suite.
The idea behind subdomains is to partition the input space into sets of similar inputs on
which the program has similar behavior. Then we use one representative of each set. This
approach makes the best use of limited testing resources by choosing dissimilar test cases,
and forcing the testing to explore parts of the input space that random testing might not
reach.
We can also partition the output space into subdomains (similar outputs on which the
program has similar behavior) if we need to ensure our tests will explore different parts of the
output space. Most of the time, partitioning the input space is sufficient
2
Partitioning: Example: BigInteger.multiply()
BigInteger has a method that multiplies two BigIntegers together:
public BigInteger multiply(BigInteger val)
For example, here’s how it might be used:
BigInteger a = ...;
BigInteger b = ...;
BigInteger ab = a.multiply(b);
We have a two-dimensional input space, consisting of all the pairs of integers (a,b). Now let’s
partition it. Thinking about how multiplication works, we might start with these partitions:
▪ a and b are both positive
▪ a and b are both negative
▪ a is positive, b is negative
▪ a is negative, b is positive
There are also some special cases for multiplication that we should check: 0, 1, and -1.
▪ a or b is 0, 1, or -1
A good tester, would suspect that the implementor of BigInteger might try to make it faster by
using int or long internally when possible, and only use the expensive general representation
when the value is too big. So we should try integers that are bigger than the biggest long.
▪ a or b is small
▪ the absolute value of a or b is bigger than Long.MAX_VALUE
3
Partitioning: Example: BigInteger.multiply()
We partition the input space (a, b) according to these
observations and choose a and b independently from:
▪ 0
▪ 1
▪ -1
▪ small positive integer
▪ small negative integer
▪ huge positive integer
▪ huge negative integer
So this will produce 7 × 7 = 49 partitions/combinations that completely cover the space of pairs
of integers.
To produce the test suite, we would pick an arbitrary pair (a,b) from each square of the grid, for
example:
• (a,b) = (-3, 25) to cover (small negative, small positive)
• (a,b) = (0, 30) to cover (0, small positive)
• (a,b) = (2^100, 1) to cover (large positive, 1)
• etc.
The figure shows how the two-dimensional (a,b) space is divided by this partition, and the
points are test cases that we might choose to completely cover the partition. 4
Partitioning: Example: max()
Math.max() function:
/**
* @param a an argument
* @param b another argument
* @return the larger of a and b.
*/
public static int max(int a, int b)
Mathematically, this method is a function of the following type:
max : int × int → int
From the specification, it makes sense to partition this function as:
▪ a<b
▪ a=b
▪ a>b
Our test suite might then be:
▪ (a, b) = (1, 2) to cover a < b
▪ (a, b) = (9, 9) to cover a = b
▪ (a, b) = (-5, -6) to cover a > b
5
Include Boundaries in the Partition
Bugs often occur at boundaries between subdomains. Some examples:
▪ 0 is a boundary between positive numbers and negative numbers
▪ the maximum and minimum values of numeric types, like int and double
▪ emptiness (the empty string, empty list, empty array) for collection types
▪ the first and last element of a collection
Why do bugs often happen at boundaries?
▪ Programmers often make off-by-one mistakes (like writing <= instead of < , or initializing a
counter to 0 instead of 1).
▪ Some boundaries may need to be handled as special cases in the code.
It’s important to include boundaries as subdomains in your partition, so that boundaries are
covered in your testing.
Let’s redo max : int × int → int .
6
Include Boundaries in the Partition
▪ relationship between a and b
▪ a<b
▪ a=b
▪ a>b
▪ value of a
▪ a=0
▪ a<0
▪ a>0
▪ a = minimum integer
▪ a = maximum integer
▪ value of b
▪ b=0
▪ b<0
▪ b>0
▪ b = minimum integer
▪ b = maximum integer
We have 3 x 5 x 5 = 75 combinations. If we cannot afford this, only make sure to cover each part.
So let’s pick test values that cover all these parts:
▪ (1, 2) covers a < b, a > 0, b > 0
▪ (-1, -3) covers a > b, a < 0, b < 0
▪ (0, 0) covers a = b, a = 0, b = 0
▪ (Integer.MIN_VALUE, Integer.MAX_VALUE) covers a < b, a = minint, b = maxint 7
▪ (Integer.MAX_VALUE, Integer.MIN_VALUE) covers a > b, a = maxint, b = minint
Two Extremes for Covering the Partition
After partitioning the input space, we can choose how exhaustive we want the
test suite to be:
▪ Full Cartesian product (all combinations).
Every legal combination of the partition dimensions is covered by one test
case. This is what we did for the BigIneger multiply example, and it gave us 7 ×
7 = 49 test cases. For the max example that included boundaries, which has
three dimensions with 3 parts, 5 parts, and 5 parts respectively, it would mean
up to 3 × 5 × 5 = 75 test cases. In practice not all of these combinations are
possible, however. For example, there’s no way to cover the combination a < b,
a=0, b=0, because a can’t be simultaneously less than zero and equal to zero.
▪ Cover each part.
Every part of each dimension is covered by at least one test case, but not
necessarily every combination. With this approach, the test suite for max
might be as small as 5 test cases if carefully chosen. That’s the approach we
took above, which allowed us to choose 5 test cases.
8
Automated Unit Testing with JUnit
A test that tests an individual module in isolation is called a unit test. Unit testing helps
debugging since you can be more confident that the bug is found in that module, rather than
anywhere in the program.
A JUnit unit test is written as a method preceded by the annotation @Test. Below are 4 Junit
tests for Math.max():
@Test
public void testALessThanB() {
assertEquals(2, Math.max(1, 2));
}
@Test
public void testBothEqual() {
assertEquals(9, Math.max(9, 9));
}
@Test
public void testAGreaterThanB() {
assertEquals(-5, Math.max(-5, -6));
}
If an assertion in a test method fails, then that test method returns immediately, and JUnit
records a failure for that test. A test class can contain any number of @Test methods, which are
run independently when you run the test class with JUnit. Even if one test method fails, the
others will still be run 9
Black Box and Glass Box Testing
Black box testing means choosing test cases only from the
documentation (requirements or specification), not the implementation
of the function. That’s what we’ve been doing in our examples so far.
We partitioned and looked for boundaries in multiply and max without
looking at the actual code for these functions.
Glass box testing means choosing test cases with knowledge of how the
function is actually implemented. For example, if the implementation
selects different algorithms depending on the input, then you should
partition according to those domains. If the implementation keeps an
internal cache that remembers the answers to previous inputs, then you
should test repeated inputs.
10
Coverage
One way to judge a test suite is to ask how thoroughly it exercises the
program. This notion is called coverage. Here are three common kinds of
coverage:
▪ Statement coverage: is every statement run by some test case?
▪ Branch coverage: for every if or while statement in the program, are both
the true and the false directions taken by some test case?
▪ Path coverage: is every possible combination of branches — every path
through the program — taken by some test case?
Branch coverage is stronger (requires more tests to achieve) than statement
coverage, and path coverage is stronger than branch coverage. In industry,
100% statement coverage is a common goal, but even that is rarely achieved
due to unreachable defensive code (like “should never get here” assertions).
100% branch coverage is highly desirable. Unfortunately 100% path coverage is
infeasible, requiring exponential-size test suites to achieve.
11
Unit Testing vs. Integration Testing, and Stubs
In contrast to a unit test, an integration test tests a combination of modules. If all you
have are integration tests, then when a test fails, you have to hunt for the bug. It
might be anywhere in the program.
Integration tests are still important, because a program can fail at the connections
between modules.
Suppose you’re building a web search engine. Two of your modules might be
getWebPage(), which downloads web pages, and extractWords(), which splits a page
into its component words:
/** @return the contents of the web page downloaded from url */
public static String getWebPage(URL url) {...}
/** @return the words in string s, in the order they appear,
* where a word is a contiguous sequence of
* non-whitespace and non-punctuation characters
*/
public static List<String> extractWords(String s) { ... }
12
Unit Testing vs. Integration Testing, and Stubs
These methods might be used by another module makeIndex() as part of the
web crawler that makes the search engine’s index:
/** @return an index mapping a word to the set of URLs
* containing that word, for all webpages in the input set
*/
public static Map<String, Set<URL>> makeIndex(Set<URL> urls) {
...
for (URL url : urls) {
String page = getWebPage(url);
List<String> words = extractWords(page);
...
}
...
}
In our test suite, we would want:
▪ unit tests just for getWebPage() that test it on various URLs
▪ unit tests just for extractWords() that test it on various strings
▪ unit tests for makeIndex() that test it on various sets of URLs 13
Unit Testing vs. Integration Testing, and Stubs
• Don’t make your test cases for for extractWords() depend on getWebPage() to be
correct. Because getWebPage() may be buggy! Instead, store web page content as a
literal string, and pass it directly to extractWords(). So that if extractWords() fails,
you can be more confident that the bug is in extractWords().
• However, the unit tests for makeIndex() can’t easily be isolated in this way. When a
test case calls makeIndex(), it is testing the correctness of the code inside
makeIndex(), but also all the methods called by makeIndex(). If the test fails, the bug
might be in any of those methods.
• So better to use stub versions of getWebPage() and extractWords(). In case of
failure, we would know that the bug is not in them.
• A stub for getWebPage() wouldn’t access the internet at all, but instead would
return mock web page content no matter what URL was passed to it. A stub for a
class is often called a mock object.
14
Automated Testing and Regression Testing
• Automated testing means running the tests and checking their results
automatically.
• The code that runs tests on a module is a test driver (also known as a test
harness).
• A test driver should invoke the module itself on fixed test cases and
automatically check that the results are correct. The result of the test driver
should be either “all tests OK” or “these tests failed: …” A framework like
JUnit allows you to build and run this kind of test driver, with a suite of
automated tests.
• JUnit helps, but you still have to come up with good test cases yourself.
• Automatic test generation is a hard problem, with weak solutions.
15
Automated Testing and Regression Testing
Ideally, any change, anywhere in the code, must be followed by a full run of all
of your tests.
This prevents your program from regressing — introducing other bugs when
you fix new bugs or add new features. Running all your tests after every
change is called regression testing.
Whenever a bug is fixed, add its corresponding input to the regression test
suite. This helps to populate your test suite with good test cases.
Saving regression tests also protects against bug resurrection, which happens
quite often.
16
