Pairwise testing
(a powerful technique)
Key points
Understanding what the challenges are with test
design
Master the 2 pairwise design techniques; Orthogonal
Arrays and All-pairs algorithm
Learn about the powers and dangers of this technique
Understanding Challenges
A simple mobile phone
3 call types;
free, chargeable (local) and chargeable (international)
3 message types;
text, mms and video
2 account types;
pay as you go and contract
Understanding Challenges
Car Insurance
3 policy types (TP, TPFT, FC)
3 storage modes (garaged, drive, road)
4 NCD (0, 1, 2 and 3+ years)
2 licence types (full and provisional)
5 age categories (17-21, 22-30, 31-40, 41-50 and 51+)
5 engine sizes (<1000, 1000-1600, 1601-2000, 2001-
2999 and 3000+)
Understanding Challenges
So how do we select a subset of test cases?
The good news ☺
4 parameters each have 3 values
= 3 x 3 x 3 x 3 = 81 test cases
pairwise = 9 test cases
13 parameters each have 3 values
= 3 x 3 x 3 x 3…= 1,594,323 test cases
pairwise = 15 test cases
20 parameters each have 10 values
= 10 x 10 x 10 x 10….= a lot of test cases !!!
pairwise = 180 test cases
Why does pairwise testing work?
The majority of faults are either single or double
mode
There are 2 recognised methods for generating all the
pairs
Orthogonal arrays
All pairs algorithm
Orthogonal Arrays
Orthogonal Arrays
Example 1
A simple mobile phone
3 call types;
free, chargeable (local) and chargeable (international)
3 message types;
text, mms and video
2 account types;
pay as you go and contract
Using Orthogonal Arrays
Step 1: identify the variables
call type, messaging and account type = 3
Step 2: determine the number of choices for
each variable:-
call type: free, local, international = 3
messaging: text, mms, video = 3
account type: pay as you go, contract = 2
Contd..
Step 3: locate the orthogonal array which has a
column for each variable and values within the
columns that correspond to the choices for each
variable
Contd..
Contd..
Step 4: map the problem onto the array
var 1 = call type 1=free, 2=local, 3=international
var 2 = messaging 1=text, 2=mms, 3=video
var 3 = account type 1=PAYG, 2=contract, 3=don’t care
Contd..
Contd..
Step 5: construct the test cases
Example 2
Car Insurance
3 policy types (TP, TPFT, FC)
3 storage modes (garaged, drive, road)
4 NCD (0, 1, 2 and 3+ years)
2 licence types (full and provisional)
5 age categories (17-21, 22-30, 31-40, 41-50 and 51+)
5 engine sizes (<1000, 1000-1600, 1601-2000, 2001-2999
and 3000+)
example 2
ALL PAIR ALGORITHIM
Simplified way of generating all pairs combinations
Does not need external devices such as orthogonal
arrays
Usually produced using a program
http://www.satisfice.com
http://www.pairwise.org/about.asp
Mobile phone example using allpairs
So what’s the difference?
In some instances they produce the same number of
test cases
In some instances the all-pairs algorithm
produces fewer test cases
In some instances orthogonal arrays produce
fewer test cases
What experts say
Some Statistics
Running 1% to 20% of all possible tests you will
find 70% to 85% of the total bugs
Cohen reported that test cases created by the “allpairs”
algorithm provided better code coverage than random
tests
300 random tests: 67% statement coverage and
58% decision coverage
200 “all-pairs” tests: 92% statement coverage and
85% decision coverage
A word of caution
Do not test all combinations
May not produce the most common combinations
No priority is given to the tests
Some combinations produced may be infeasible
Q&A
THANK YOU!