Software Testing
Testing 1
Background
Main objectives of a project: High Quality &
High Productivity (Q&P)
Quality has many dimensions
reliability, maintainability, interoperability etc.
Reliability is perhaps the most important
Reliability: The chances of software failing
More defects => more chances of failure =>
lesser reliability
Hence Q goal: Have as few defects as
possible in the delivered software
Testing 2
Faults & Failure
Failure: A software failure occurs if the
behavior of the s/w is different from
expected/specified.
Fault: cause of software failure
Fault = bug = defect
A defect has the potential to cause failure.
Definition of a defect is environment,
project specific
Testing 3
Role of Testing
Reviews are human processes - can not catch all
defects
Hence there will be requirement defects, design
defects and coding defects in code
These defects have to be identified by testing
Therefore testing plays a critical role in ensuring
quality.
All defects remaining from before as well as new
ones introduced have to be identified by testing.
Testing 4
Detecting defects in Testing
During testing, a program is executed
with a set of test cases
Failure during testing => defects are
present
No failure => confidence grows, but can
not say “defects are absent”
Defects detected through failures
To detect defects, must cause failures
during testing Testing 5
Test Oracle
A test oracle is a mechanism, different from the program
itself, that can be used to check the correctness of the
output of the program for the test cases.
Conceptually, we can consider testing a process in which
the test cases are given to the test oracle and the program
under testing. The output of the two is then compared to
determine if the program behaved correctly for the test
cases
Testing 6
Role of Test cases
Ideally would like the following for test cases
No failure implies “no defects” or “high quality”
If defects present, then some test case causes a failure
Psychology of testing is important
should be to ‘reveal’ defects(not to show that it works!)
test cases must be “destructive
Role of test cases is clearly very critical
Only if test cases are “good”, the confidence
increases after testing
Testing 7
Test case design
During test planning, have to design a set of
test cases that will detect defects present
These are the two fundamental goals of a
practical testing activity—maximize the
number of errors detected and minimize the
number of test cases (i.e., minimize the cost)
Two approaches to design test cases
functional or black box
structural or white box
Both are complimentary;
Testing 8
Black Box testing
Software tested to be treated as a block box
Specification for the black box is given
The expected behavior of the system is used to
design test cases i.e test cases are determined
solely from specification.
Internal structure of code not used for test case
design
The tester only knows the inputs that can be
given to the system and what output the system
should give.
Testing 9
Black box Testing…
Premise: Expected behavior is specified.
Hence just test for specified expected
behavior
How it is implemented is not an issue.
For modules,specification produced in
design specify expected behavior
For system testing, SRS specifies
expected behavior
Testing 10
Equivalence Class partitioning
Divide the input space into equivalent classes
If the software works for a test case from a
class the it is likely to work for all
Can reduce the set of test cases if such
equivalent classes can be identified
Getting ideal equivalent classes is impossible
Approximate it by identifying classes for
which different behavior is specified
Testing 11
Equivalence class partitioning…
E.g. if a function was not designed for
negative numbers then it will fail for all
the negative numbers
For robustness, should form equivalent
classes for invalid inputs also
Each group of inputs for which the
behavior is expected to be different
from others is considered a separate
equivalence class
Testing 12
Equivalent class partitioning..
Every condition specified as input is an
equivalent class
Define invalid equivalent classes also
E.g. range 0< value<Max specified
one range is the valid class
input < 0 is an invalid class
input > max is an invalid class
Whenever that entire range may not be
treated uniformly - split
Testing into classes 13
Equivalent class partitioning..
Another approach for forming equivalence classes is
to consider any special value for which the behavior
could be different as an equivalence class.
For example, the value 0 could be a special value for
an integer input
Should consider eq. classes in outputs also and then
give test cases for different classes
For an output equivalence class, the goal is to have
inputs such that the output for that test case lies in
the output equivalence class.
Testing 14
E.g.: Compute rate of interest given loan amount,
monthly installment, and number of months
Equivalent classes in output: + rate, rate = 0 ,-ve rate
Have test cases to get these outputs
During testing, it is important to test for each of
these, that is, give inputs such that each of these
three outputs are generated.
Determining test cases for output classes may be
more difficult, but output classes have been found to
reveal errors that are not revealed by just
considering the input classes.
Testing 15
Equivalence class…
Once eq classes selected for each of the
inputs, test cases have to be selected
A somewhat good strategy which
requires more test cases is to have a
test case cover at most one valid
equivalence class for each input, and
have one separate test case for each
invalid equivalence class
Testing 16
Example
Consider a program that takes 2 inputs
– a string s and an integer n
Program determines n most frequent
characters
Tester believes that programmer may
deal with diff types of chars separately
A set of valid and invalid equivalence
classes is given
Testing 17
Example..
Input Valid Eq Class Invalid Eq class
S 1: Contains numbers 1: non-ascii char
2: Lower case letters 2: str len > N
3: upper case letters
4: special chars
5: str len between 0-N(max)
N Int in valid range 3: Int out of range
Testing 18
Example…
Test cases (i.e. s , n) with first method
s : str of len < N with lower case, upper case,
numbers, and special chars, and n=5
Plus test cases for each of the invalid eq classes
Total test cases: 1+3= 4
With the second approach
A separate str for each type of char (i.e. a str of
numbers, one of lower case, …) + invalid cases
Total test cases will be 4 + 3 = 7
Testing 19
Boundary value analysis
Programs often fail on special values
These values often lie on boundary of
equivalence classes
Test cases that have boundary values have
high yield
These are also called extreme cases
A BV test case is a set of input data that lies
on the edge of a eq class of input/output
Testing 20
BVA...
For each equivalence class
choose values on the edges of the class
choose values just outside the edges
E.g. if 0 <= x <= 1.0
0.0 , 1.0 are edges inside
-0.1,1.1 are just outside
E.g. a bounded list - have a null list , a
maximum value list
Consider outputs also and have test cases
generate outputs on the boundary
Testing 21
BVA…
In BVA we determine the value of vars that
should be used
If input is a defined range, then there are 6
boundary values plus 1 normal value (tot: 7)
If multiple inputs, how to combine them into
test cases; two strategies possible
Try all possible combination of BV of diff variables,
with n vars this will have 7n test cases!
Select BV for one var; have other vars at normal
values + 1 of all normal values i.e. 6n+1
Testing 22
BVA.. (test cases for two vars – x and y)
Testing 23
Cause Effect graphing
Equivalence classes and boundary value
analysis consider each input separately
To handle multiple inputs, different
combinations of equivalent classes of inputs
can be tried
Number of combinations can be large – if n
diff input conditions such that each condition
is valid/invalid, total: 2n
Cause effect graphing helps in selecting
combinations as input conditions
Testing 24
CE-graphing
Identify causes and effects in the system
Cause: distinct input condition which can be true
or false
Effect: distinct output condition (T/F)
Identify which causes can produce which
effects; can combine causes
Decision table can be used for forming the
test cases
Testing 25
CE graphing: Example
A bank database which allows two commands
Credit acc# amt
Debit acc# amt
Requirements
If credit and acc# valid, then credit
If debit and acc# valid and amt less than balance,
then debit
Invalid command - message
Testing 26
Example…
Causes
C1: command is credit
C2: command is debit
C3: acc# is valid
C4: amt is valid
Effects
Print “Invalid command”
Print “Invalid acct#”
Print “Debit amt not valid”
Debit account
Credit account
Testing 27
Example…
# 1 2 3 4 5
C1 0 x x x x
C2 0 x 1 1 x
C3 x 0 1 1 1
C4 x x 0 1 1
E1 1
E2 1
E3 1
E4 1
E5 1
Testing 28
Special cases
Programs often fail on special cases
These depend on nature of inputs, types of
data structures, etc.
No good rules to identify them
One way is to guess when the software
might fail and create those test cases
Also called error guessing
Testing 29
Error Guessing
Use experience and judgement to guess situations
where a programmer might make mistakes
Special cases can arise due to assumptions about
inputs, user, operating environment, business, etc.
E.g. A program to count frequency of words
file empty, file non existent, file only has blanks, contains
only one word, all words are same, multiple consecutive
blank lines, multiple blanks between words, blanks at the
start, words in sorted order, blanks at end of file, etc.
Perhaps the most widely used in practice
Testing 30
White box testing
Black box testing focuses only on functionality
What the program does; not how it is implemented
White box testing focuses on implementation
Aim is to exercise different program structures with
the intent of uncovering errors
Is also called structural testing
Various criteria exist for test case design
Test cases have to be selected to satisfy
coverage criteria
Testing 31
Control flow based criteria
Considers the program as control flow graph
Nodes represent code blocks – i.e. set of
statements always executed together
An edge (i,j) represents a possible transfer of
control from i to j
Assume a start node and an end node
A path is a sequence of nodes from start to
end
Testing 32
Statement Coverage Criterion
Criterion: Each statement is executed at least once
during testing
I.e. set of paths executed during testing should
include all nodes
Limitation: does not require a decision to evaluate to
false if no else clause
E.g. if ( x>=0) x = -x; return(x)
The set of test cases {x = 0} achieves 100% statement
coverage, but error not detected
Guaranteeing 100% coverage not always possible
due to possibility of unreachable nodes
Testing 33
Branch coverage
Criterion: Each edge should be traversed at
least once during testing
i.e. each decision must evaluate to both true
and false during testing
Branch coverage implies stmt coverage
If multiple conditions in a decision, then all
conditions need not be evaluated to T and F
Testing 34
In a Project
Both functional and structural should be used
Test plans are usually determined using functional
methods; during testing, for further rounds, based on
the coverage, more test cases can be added
Structural testing is useful at lower levels only; at
higher levels ensuring coverage is difficult
Hence, a combination of functional and structural at
unit testing
Functional testing (but monitoring of coverage) at
higher levels
Testing 35
Integration and Testing
Incremental testing requires incremental
‘building’ I.e. incrementally integrate parts to
form system
Integration & testing are related
During coding, different modules are coded
separately
Integration - the order in which they should
be tested and combined
Integration is driven mostly by testing needs
Testing 36
Levels of Testing
The code contains requirement defects,
design defects, and coding defects
Nature of defects is different for
different injection stages
One type of testing will be unable to
detect the different types of defects
Different levels of testing are used to
uncover these defects
Testing 37
User needs Acceptance testing
Requirement System testing
specification
Design Integration testing
code Unit testing
Testing 38
Unit Testing
Different modules tested separately
Focus: defects injected during coding
Essentially a code verification technique,
UT is closely associated with coding
Frequently the programmer does UT; coding
phase sometimes called “coding and unit
testing”
Testing 39
Integration Testing
Focuses on interaction of modules in a
subsystem
Unit tested modules combined to form
subsystems
Test cases to “exercise” the interaction
of modules in different ways
May be skipped if the system is not too
large
Testing 40
System Testing
Entire software system is tested
Focus: does the software implement the
requirements?
Validation exercise for the system with
respect to the requirements
Generally the final testing stage before the
software is delivered
May be done by independent people
Most time consuming test phase
Testing 41
Acceptance Testing
Focus: Does the software satisfy user needs?
Generally done by end users/customer in
customer environment, with real data
Only after successful AT software is deployed
Any defects found,are removed by developers
Acceptance test plan is based on the
acceptance test criteria in the SRS
Testing 42
Other forms of testing
Performance testing
tools needed to “measure” performance
Stress testing
load the system to peak, load generation tools
needed
Testing 43
Test Plan
Testing usually starts with test plan and ends
with acceptance testing
Test plan is a general document that defines
the scope and approach for testing for the
whole project
Inputs are SRS, project plan, design
Test plan identifies what levels of testing will
be done, what units will be tested, etc in the
project
Testing 44
Test Plan…
Test plan usually contains
Test unit specs: what units need to be
tested separately
Features to be tested: these may include
functionality, performance, usability,…
Approach: criteria to be used, when to
stop, how to evaluate, etc
Test deliverables
Schedule and task allocation
Testing 45
Test case specifications
Test plan focuses on approach; does not deal
with details of testing a unit
Test case specification has to be done
separately for each unit
Based on the plan (approach, features,..) test
cases are determined for a unit
Expected outcome also needs to be specified
for each test case
Testing 46
Test case specifications…
Seq.No Condition Test Data
Expected successful
to be tested result
Testing 47
Test case execution and analysis
Test summary report is often an output – gives a
summary of test cases executed, effort, defects
found, etc
Monitoring of testing effort is important to ensure
that sufficient time is spent
Testing 48
Defect logging and tracking
A large software may have thousands of
defects, found by many different people
Often person who fixes (usually the coder) is
different from who finds
Due to large scope, reporting and fixing of
defects cannot be done informally
Defects found are usually logged in a defect
tracking system and then tracked to closure
Defect logging and tracking is one of the best
practices in industry
Testing 49
Defect logging…
A defect in a software project has a life
cycle of its own, like
Found by someone, sometime and logged
along with info about it (submitted)
Job of fixing is assigned; person debugs
and then fixes (fixed)
The manager or the submitter verifies that
the defect is indeed fixed (closed)
More elaborate life cycles possible
Testing 50
Defect logging…
Testing 51
Defect logging…
During the life cycle, info about defect
is logged at diff stages to help debug as
well as analysis
Defects generally categorized into a few
types, and type of defects is recorded
ODC is one classification
Some std categories: Logic, standards, UI,
interface, performance, documentation,..
Testing 52
Defect logging…
Severity of defects in terms of its
impact on sw is also recorded
Severity useful for prioritization of fixing
One categorization
Critical: Show stopper
Major: Has a large impact
Minor: An isolated defect
Cosmetic: No impact on functionality
Testing 53
Defect logging and tracking…
Ideally, all defects should be closed
Sometimes, organizations release software
with known defects (hopefully of lower
severity only)
Organizations have standards for when a
product may be released
Defect log may be used to track the trend of
how defect arrival and fixing is happening
Testing 54
Defect analysis for prevention
Quality control focuses on removing defects
Goal of defect prevention is to reduce the
defect injection rate in future
DP done by analyzing defect log, identifying
causes and then remove them
Is an advanced practice, done only in mature
organizations
Finally results in actions to be undertaken by
individuals to reduce defects in future
Testing 55
Metrics - Defect removal efficiency
Basic objective of testing is to identify
defects present in the programs
Testing is good only if it succeeds in this goal
Defect removal efficiency of a QC activity =
% of present defects detected by that QC
activity
High DRE of a quality control activity means
most defects present at the time will be
removed
Testing 56
Defect removal efficiency …
DRE for a project can be evaluated only when all
defects are know, including delivered defects
Delivered defects are approximated as the number of
defects found in some duration after delivery
The injection stage of a defect is the stage in which it
was introduced in the software, and detection stage
is when it was detected
These stages are typically logged for defects
With injection and detection stages of all defects,
DRE for a QC activity can be computed
Testing 57
Defect Removal Efficiency …
DREs of different QC activities are a
process property - determined from
past data
Past DRE can be used as expected
value for this project
Process followed by the project must be
improved for better DRE
Testing 58
Metrics – Reliability Estimation
High reliability is an important goal being
achieved by testing
Reliability is usually quantified as a probability
or a failure rate
For a system it can be measured by counting
failures over a period of time
Measurement often not possible for software
as due to fixes reliability changes, and with
one-off, not possible to measure
Testing 59
Reliability Estimation…
Sw reliability estimation models are used to
model the failure followed by fix model of
software
Data about failures and their times during the
last stages of testing is used by these model
These models then use this data and some
statistical techniques to predict the reliability
of the software
A simple reliability model is given in the book
Testing 60
Summary
Testing plays a critical role in removing
defects, and in generating confidence
Testing should be such that it catches
most defects present, i.e. a high DRE
Multiple levels of testing needed for this
Incremental testing also helps
At each testing, test cases should be
specified, reviewed, and then executed
Testing 61
Summary …
Deciding test cases during planning is the
most important aspect of testing
Two approaches – black box and white box
Black box testing - test cases derived from
specifications.
Equivalence class partitioning, boundary value,
cause effect graphing, error guessing
White box - aim is to cover code structures
statement coverage, branch coverage
Testing 62
Summary…
In a project both used at lower levels
Test cases initially driven by functional
Coverage measured, test cases enhanced using
coverage data
At higher levels, mostly functional testing
done; coverage monitored to evaluate the
quality of testing
Defect data is logged, and defects are tracked
to closure
The defect data can be used to estimate
reliability, DRE
Testing 63
