Software testing
““ Testing
Testing isis the
the process
process of
of
executing
executing aa program
program withwith the
the
intent
intent of
of finding
finding errors.”
errors.”
Glen
GlenMyers
Myers
12/08/21 CMPE 310 Winter 2001 1
Limits of software testing
Testing can only prove the presence of
bugs - never their absence
WHY???
12/08/21 CMPE 310 Winter 2001 2
Exhaustive testing
Assume that there are
520 possible different
execution flows. If
we execute one test
per millisecond, it
would take 3.170
years to test this
program!!
loop < 20x
12/08/21 CMPE 310 Winter 2001 3
The Peculiarity of Testing
Testing is the process of executing a program
with the intent of finding an error
A good test case has a high probability of
finding an as-yet undiscovered error
A successful test is one that uncovers
an as-yet undiscovered error
12/08/21 CMPE 310 Winter 2001 4
Who tests the software better??
developer independent tester
Understands the system Must learn about the system,
but, will test “gently” but, will attempt to break it
and, is driven by “delivery” and, is driven by quality
12/08/21 CMPE 310 Winter 2001 5
Testing Principles
All testing should be traceable to customer
requirements
Tests should be planned long before testing begins
The Pareto principle applies to software testing as
well
Testing should begin in the small and progress
toward testing in the large
To be most effective testing should be performed by
a third party
12/08/21 CMPE 310 Winter 2001 6
Testability = developing a
program be tested easily
Operability - “The better it works, the more efficiently it can be
tested”
Observability - the results are easy to see, distinct output is
generated for each input, incorrect output is easily identified
Controllability - processing can be controlled, tests can be
automated & reproduced
Decomposability - software modules can be tested
independently
Simplicity - no complex architecture and logic
Stability - few changes are requested during testing
Understandability - program is easy to understand
12/08/21 CMPE 310 Winter 2001 7
Test case design
“Bugs lurk in corners
and congregate at
boundaries…”
Boris Beizer
OBJECTIVE to uncover errors
CRITERIA in a complete manner
CONSTRAINT with a minimum of effort and time
12/08/21 CMPE 310 Winter 2001 8
Approaches to tests
requirements
output
input
events
Black Box Testing White Box Testing
12/08/21 CMPE 310 Winter 2001 9
White box testing
Also called glass
box testing
CANNOT BE
EXHAUSTIVE!!!
12/08/21 CMPE 310 Winter 2001 10
Approaches to White Box
Testing
Exercise all independent paths in a
program at least once
Exercise all logical decisions on their true
and false sides
Execute all loops at their boundaries and
within their operational bounds
Exercise internal data structures to assure
their validity
12/08/21 CMPE 310 Winter 2001 11
Why bother with white box
testing?
Black box testing: (1)Requirements fulfilled and (2)
Interfaces available and working
Question: Why white box testing?
• Logical errors and incorrect assumptions are
inversely proportional to the probability that a
program path will be executed
• We often believe that a logical path is not likely to
be executed when, in fact, it may be executed on
a regular basis
• Typographical errors are random
12/08/21 CMPE 310 Winter 2001 12
Exhaustive testing
If-then-else There are 520=1014 (approx.)
possible paths!
loop < 20x
Selective Testing
12/08/21 CMPE 310 Winter 2001 13
Selective testing
a selected path Selective:
Basis Path testing
Condition testing
Dataflow testing
Loop testing
12/08/21 CMPE 310 Winter 2001 14
Basis set
Basis set = a set of paths that will execute all
statements and all conditions in a program at
least once (can have redundancies)
The basis set is not unique
Cyclomatic complexity defines the number of
independent paths in the basis set
Goal: Define test cases for basis set
12/08/21 CMPE 310 Winter 2001 15
Basis path testing
Rationale:
Rationale I want to test all
statements of a program
at least once.
Compute the cyclomatic complexity of the
program P - V(P)
Use it to determine how many test you have to
do:
V(P) provides an upper bound of tests that
must be executed to guarantee coverage of all
program statements
12/08/21 CMPE 310 Winter 2001 16
Basis path testing
1. We start with computing V(P):
1 Since V(P) = 4, there are four
paths
2
Path 1: 1,2,3,6,7,8
4 3
Path 2: 1,2,3,5,7,8
5 6
Path 3: 1,2,4,7,8
Path 4: 1,2,4,7,2,4…7,8
7
2. We derive test cases to exercise
8 these paths.
12/08/21 CMPE 310 Winter 2001 17
Documenting test cases
Name
Number
Values for the inputs
Expected outputs
Short description (if needed)
12/08/21 CMPE 310 Winter 2001 18
Basis path testing notes
you don’t need a flow chart,
but the picture will help when
you trace program paths
count each simple logical test,
compound tests count as 2 or
more
basis path testing should be
applied to critical modules
12/08/21 CMPE 310 Winter 2001 19
Example
Program
void
void f()
f() {{
if(count
if(count >> 10){
10){
fixed_count
fixed_count == fixed_count
fixed_count ++ count;
count;
done
done == 1;
1;
}} else
else if(count
if(count >5){
>5){
fixed_count
fixed_count --;
--;
}} else
else {{
fixed_count
fixed_count == count
count ** 4;
4;
}}
}}
12/08/21 CMPE 310 Winter 2001 20
Flow Graph
void f()
11
if(count > 10)
22
if(count >5)
44 33
fixed_count = fixed_count + count;
fixed_count = count * 4; 55 66 done = 1;
77
88 fixed_count --;
return;
12/08/21 CMPE 310 Winter 2001 21
Derivation of the test cases
V(P) = 3 (2 conditions + 1)
11
Independent paths:
(a) 1, 2, 3, 8 22
(b) 1, 2, 4, 5, 7, 8 44 33
(c) 1, 2, 4, 6, 7, 8 55 66
Test cases: 77
(a) count 13 88
(b) count 8
(c) count 2
12/08/21 CMPE 310 Winter 2001 22
Comment
We have not consider all possible cases
The number of possible cases also in
this simple example is infinite …
Why?
Suppose you embed your program in a
HW with a documented bug each time
44 is incremented to 45 ...
12/08/21 CMPE 310 Winter 2001 23
Not required for exams
Condition Testing
Condition testing is a test case design method
that is based on testing all possible conditions
in a program
A condition in a program may contain:
Boolean operators and variables
Parenthesis
All of them can fail!!!
Relational operators
Arithmetic expressions
Condition testing focus on testing
…. each condition in the program
12/08/21 CMPE 310 Winter 2001 24
Kinds of condition testing
Branch testing: for each condition C, the true and the
false branches resulting from it and every simple condition in C
need to be executed at least once
Domain testing: it uses information on the domain;
given a relational condition in the program such as a<b I have to
test it with with values of a and b a<b, a=b, a>b and a and be
VERY close
given a boolean condition with n variables, I have to test it for each
combination of the values (that is 2n cases)
There are other approaches, such as Branch and Relational
operator testing
12/08/21 CMPE 310 Winter 2001 25
Not required for exams
Data Flow Testing
Data flow testing identifies paths in the
program that go from the assignment of
a value to a variable to the use of such
variable, to make sure that the var is
properly used.
The DU (Definition - Usage) testing
strategy requires that each DU chain is
covered at least once
12/08/21 CMPE 310 Winter 2001 26
Data Flow Testing
Considering x, there are two DU
[1]
[1] void
void k()
k() {{ paths:
[2]
[2] xx == 11;
11; (a) [2]-[4]
[3]
[3] if
if (p(cond1))
(p(cond1)) {{
[4] yy == xx ++ 1;
(b) [2]-[6]
[4] 1;
[5]
[5] }} else
else if if (q(cond2))
(q(cond2)) {{ We need therefore to derive test
[6]
[6] ww == xx ++ 3;
3; cases to match the following
[7]
[7] }} else
else {{
[8] ww == yy ++ 1;
conditions:
[8] 1;
[9]
[9] }} (a) k() is executed and
[10]}
[10]} p(cond1) is true
(b) k() is executed and
p(cond1) is false and
q(cond2) is true
12/08/21 CMPE 310 Winter 2001 27
Loops
Cornerstone of every program
Loops can lead to non-terminating programs
Error in indexes of loops are very easy to
occur
Loop Testing is a white-box testing technique
that focuses only on the validity of loop constructs
12/08/21 CMPE 310 Winter 2001 28
Loop testing
simple loop
nested concatenated unstructured
loops loops loops
12/08/21 CMPE 310 Winter 2001 29
Loop testing: simple loops
Minimum conditions - simple loops
1. skip the loop entirely
2. only one pass through the loop
3. two passes through the loop
4. m passes through the loop m < n
5. (n-1), n, and (n+1) passes through the loop
where n is the maximum number of allowable
passes
12/08/21 CMPE 310 Winter 2001 30
Nested loops
Just extending simple loop testing: number
of tests increases geometrically
Reduce the number of tests:
start at the innermost loop; set all other loops to minimum
values
conduct simple loop test; add out of range or excluded
values
work outwards while keeping inner nested loops to
typical values
continue until all loops have been tested
12/08/21 CMPE 310 Winter 2001 31
Concatenated loops
Same strategies as simple loops if the loops
counters are independent
Same strategies as nested loops if the loops
counters depend on each other:
int k;
for (k=0;k<10;k++)
k=0;k<10;k++
{
w();
if p(m) break;
}
for (;k<10;k++)
;k<10;k++ {
r();
}
12/08/21 CMPE 310 Winter 2001 32
How do we select the
strategy?
The requirements of reliability of the
system
The resources available
The kind of programs
functional languages do not have loops…
The tools available to support the
testing process
12/08/21 CMPE 310 Winter 2001 33
Black box testing
requirements
output
input
events
12/08/21 CMPE 310 Winter 2001 34
Categories of errors in
black box testing
incorrect of missing functions
interface errors
errors in data structures or external
database access
performance errors
initialization and termination errors
12/08/21 CMPE 310 Winter 2001 35
Example
abs(integer x)
Return x if x >= 0
Return -x if x<0
The concrete value of x does not
matter!
12/08/21 CMPE 310 Winter 2001 36
Issues on black-box testing
How is functional validity tested?
What classes of input will make good test cases?
Is the system particularly sensitive to certain input
values?
How are the boundaries of a data class isolated?
What data rates and data volume can the system
tolerate?
What effect will specific combinations of data have
on system operations?
12/08/21 CMPE 310 Winter 2001 37
Techniques for black box testing
Graph-Based Testing Methods
Equivalence Partitioning
Boundary Value Analysis
Comparison Testing
12/08/21 CMPE 310 Winter 2001 38
Not required for exams
Graph-Based Testing
Complex Systems are hard to understand
They can be modeled as interacting objects
Test can be performed to ensure that all the
required relations are in place
Pressman, p. 472
12/08/21 CMPE 310 Winter 2001 39
Equivalence partitioning
Partitioning is based mouse picks on menu
on input conditions
output
user format
queries requests
responses
to prompts
numerical
data
command key input
12/08/21 CMPE 310 Winter 2001 40
Equivalence classes (1)
Kinds of Valid data
user supplied commands
responses to system prompts
file names
computational data
physical parameters
bounding values
initiation values
output data formatting commands
responses to error messages
graphical data (e.g., mouse picks)
12/08/21 CMPE 310 Winter 2001 41
Equivalence classes (2)
Kinds of Invalid data
data outside bounds of the program
physically impossible data
proper value supplied in wrong place
12/08/21 CMPE 310 Winter 2001 42
Defining equivalence classes
Input condition is a range: one valid
and two invalid classes are defined
Input condition requires specific value:
one valid and two invalid classes are
defined
Input condition is boolean: one valid
and one invalid class are defined
12/08/21 CMPE 310 Winter 2001 43
Boundary value analysis
mouse picks on menu
output
user format
queries requests
output responses
domain to prompts
numerical
data
command key input
12/08/21 CMPE 310 Winter 2001 44
Boundary value testing
Range a..b
test cases: a, b, just above a, just below b
Number of values:
test cases: max, min, just below min, just
above max
Output bounds should be checked
Boundaries of externally visible data
structures shall be checked (e.g. arrays)
12/08/21 CMPE 310 Winter 2001 45
Comparison testing
Safety critical systems
Develop several implementations for a specification
Run tests with all and compare results
if not identical: investigate in
detail and/or go on a majority
criteria
Problems:
expensive
errors in specification
all implementations return the same but incorrect result
12/08/21 CMPE 310 Winter 2001 46
Automating software testing
Manual software testing is time
consuming
Software testing has to be repeated
after every change (regression testing)
Write test drivers that can run
automatically and produce a test report
12/08/21 CMPE 310 Winter 2001 47
