Software Testing

(continued)
Lecture 11b

Dr. R. Mall

1
Organization of this
Lecture:
 Review of last lecture.
 Data flow testing
 Mutation testing
 Cause effect graphing
 Performance testing.
 Test summary report
 Summary

2
Review of last lecture

 White box testing:

 requires knowledge about
internals of the software.
 design and code is required.
 also called structural testing.

3
Review of last lecture

 We discussed a few white-box test

strategies.
 statement coverage
 branch coverage
 condition coverage
 path coverage

4
Data Flow-Based Testing

 Selects test paths of a

program:
 according to the locations of
 definitions and uses of different
variables in a program.

5
Data Flow-Based Testing

 For a statement numbered S,

 DEF(S) = {X/statement S contains a
definition of X}
 USES(S)= {X/statement S contains
a use of X}
 Example: 1: a=b; DEF(1)={a},
USES(1)={b}.
 Example: 2: a=a+b; DEF(1)={a},
USES(1)={a,b}.

6
Data Flow-Based Testing

 A variable X is said to be live at

statement S1, if
 X is defined at a statement S:
 there exists a path from S to S1
not containing any definition of
X.
7
DU Chain Example
1 X(){
2 a=5; /* Defines variable a */
3 While(C1) {
4 if (C2)
5 b=a*a; /*Uses variable a */
6 a=a-1; /* Defines variable a */
7 }
8 print(a); } /*Uses variable a */

8
Definition-use chain (DU
chain)

 [X,S,S1],
 S and S1 are statement numbers,
 X in DEF(S)
 X in USES(S1), and
 the definition of X in the statement S is
live at statement S1.

9
Data Flow-Based Testing

 One simple data flow testing

strategy:
 every DU chain in a program be
covered at least once.

10
Data Flow-Based Testing

 Data flow testing strategies:

 useful for selecting test paths of a
program containing nested if and
loop statements

11
Data Flow-Based Testing
 1 X(){
 2 B1; /* Defines variable a */
 3 While(C1) {
 4 if (C2)
 5 if(C4) B4; /*Uses variable a */

 6 else B5;
 7 else if (C3) B2;
 8 else B3; }
 9 B6 }

12
Data Flow-Based Testing

 [a,1,5]: a DU chain.
 Assume:
 DEF(X) = {B1, B2, B3, B4, B5}
 USED(X) = {B2, B3, B4, B5, B6}
 There are 25 DU chains.
 However only 5 paths are needed to cover
these chains.

13
Mutation Testing

 The software is first tested:

 using an initial testing method based on white-
box strategies we already discussed.
 After the initial testing is complete,
 mutation testing is taken up.
 The idea behind mutation testing:
 make a few arbitrary small changes to a
program at a time.
14
Mutation Testing

 Each time the program is

changed,
 it is called a mutated program
 the change is called a mutant.

15
Mutation Testing

 A mutated program:
 tested against the full test suite of the
program.
 If there exists at least one test case in the
test suite for which:
 a mutant gives an incorrect result,
 then the mutant is said to be dead.

16
Mutation Testing
 If a mutant remains alive:
 even after all test cases have been exhausted,
 the test suite is enhanced to kill the mutant.
 The process of generation and killing of
mutants:
 can be automated by predefining a set of
primitive changes that can be applied to the
program.

17
Mutation Testing

 The primitive changes can be:

 altering an arithmetic operator,
 changing the value of a constant,
 changing a data type, etc.

18
Mutation Testing

 A major disadvantage of mutation

testing:
 computationally very expensive,
 a large number of possible mutants
can be generated.

19
Cause and Effect Graphs

 Testing would be a lot easier:

 if we could automatically generate test cases
from requirements.
 Work done at IBM:
 Can requirements specifications be
systematically used to design functional test
cases?

20
Cause and Effect Graphs

 Examine the requirements:

 restate them as logical relation between
inputs and outputs.
 The result is a Boolean graph
representing the relationships
 called a cause-effect graph.

21
Cause and Effect Graphs

 Convert the graph to a decision

table:
 each column of the decision table
corresponds to a test case for
functional testing.

22
Steps to create cause-
effect graph
 Study the functional requirements.
 Mark and number all causes and
effects.
 Numbered causes and effects:
 become nodes of the graph.

23
Steps to create cause-
effect graph
 Draw causes on the LHS
 Draw effects on the RHS
 Draw logical relationship between
causes and effects
 as edges in the graph.
 Extra nodes can be added
 to simplify the graph

24
Drawing Cause-Effect
Graphs
A B
If A then B

A
B
C
If (A and B)then C

25
Drawing Cause-Effect
Graphs
A
B
C
If (A or B)then C
A
B
C
If (not(A and B))then C

26
Drawing Cause-Effect
Graphs
A
B
C
If (not (A or B))then C

A B
If (not A) then B

27
Cause effect graph-
Example
 A water level monitoring
system
 used by an agency involved in
flood control.
 Input: level(a,b)
 a is the height of water in dam in
meters
 b is the rainfall in the last 24
hours in cms

28
Cause effect graph-
Example
 Processing
 The function calculates whether the level
is safe, too high, or too low.
 Output
 message on screen
 level=safe
 level=high
 invalid syntax

29
Cause effect graph-
Example

 We can separate the requirements into

5 clauses:
 first five letters of the command is “level”
 command contains exactly two
parameters
 separated by comma and enclosed in
parentheses

30
Cause effect graph-
Example
 Parameters A and B are real numbers:
 such that the water level is calculated to be
low
 or safe.
 The parameters A and B are real numbers:
 such that the water level is calculated to be
high.

31
Cause effect graph-
Example

 Command is syntactically valid

 Operands are syntactically valid.

32
Cause effect graph-
Example

 Three effects
 level = safe
 level = high
 invalid syntax

33
Cause effect graph-
Example
1 10 E
3
2 11
3 E
1
4
E
5 2

34
Cause effect graph-
Decision table
Test 1 Test 2 Test 3 Test 4 Test 5
Cause 1 I I I S I
Cause 2 I I I X S
Cause 3 I S S X X
Cause 4 S I S X X
Cause 5 S S I X X
Effect 1 P P A A A
Effect 2 A A P A A
Effect 3 A A A P P

35
Cause effect graph-
Example

 Put a row in the decision table for

each cause or effect:
 in the example, there are five rows
for causes and three for effects.

36
Cause effect graph-
Example

 The columns of the decision table

correspond to test cases.
 Define the columns by examining
each effect:
 list each combination of causes that can
lead to that effect.

37
Cause effect graph-
Example

 We can determine the number of

columns of the decision table
 by examining the lines flowing into the
effect nodes of the graph.

38
Cause effect graph-
Example

 Theoretically we could have generated

25=32 test cases.
 Using cause effect graphing technique
reduces that number to 5.

39
Cause effect graph

 Not practical for systems which:

 include timing aspects
 feedback from processes is used for
some other processes.

40
Testing
 Unit testing:
 test the functionalities of a single module or
function.
 Integration testing:
 test the interfaces among the modules.
 System testing:
 test the fully integrated system against its
functional and non-functional requirements.
41
Integration testing

 After different modules of a system

have been coded and unit tested:
 modules are integrated in steps
according to an integration plan
 partially integrated system is tested at
each integration step.

42
System Testing

 System testing:
 validate a fully developed
system against its
requirements.

43
Integration Testing

 Develop the integration plan by

examining the structure chart :
 big bang approach
 top-down approach
 bottom-up approach
 mixed approach

44
Example Structured
Design
root
Valid-numbers rms
rms
Valid-numbers

Get-good-data Compute-solution Display-solution

Validate-
Get-data data

45
Big bang Integration
Testing

 Big bang approach is the simplest

integration testing approach:
 all the modules are simply put together
and tested.
 this technique is used only for very small
systems.

46
Big bang Integration
Testing
 Main problems with this approach:
 if an error is found:
 it is very difficult to localize the error
 the error may potentially belong to any of
the modules being integrated.
 debugging errors found during big
bang integration testing are very
expensive to fix.

47
Bottom-up Integration
Testing
 Integrate and test the bottom level
modules first.
 A disadvantage of bottom-up testing:
 when the system is made up of a large
number of small subsystems.
 This extreme case corresponds to the big
bang approach.

48
Top-down integration
testing
 Top-down integration testing starts with
the main routine:
 and one or two subordinate routines in the
system.
 After the top-level 'skeleton’ has been
tested:
 immediate subordinate modules of the
'skeleton’ are combined with it and tested.

49
Mixed integration testing

 Mixed (or sandwiched) integration

testing:
 uses both top-down and bottom-up
testing approaches.
 Most common approach

50
Integration Testing

 In top-down approach:
 testing waits till all top-level modules are
coded and unit tested.
 In bottom-up approach:
 testing can start only after bottom level
modules are ready.

51
Phased versus Incremental
Integration Testing

 Integration can be incremental or

phased.
 In incremental integration testing,
 only one new module is added to
the partial system each time.

52
Phased versus Incremental
Integration Testing

 In phased integration,
 a group of related modules are
added to the partially integrated
system each time.
 Big-bang testing:
 a degenerate case of the phased
integration testing.

53
Phased versus Incremental
Integration Testing
 Phased integration requires less
number of integration steps:
 compared to the incremental integration
approach.
 However, when failures are detected,
 it is easier to debug if using incremental
testing
 since errors are very likely to be in the
newly integrated module.

54
System Testing

 System tests are designed to

validate a fully developed
system:
 to assure that it meets its
requirements.

55
System Testing

 There are essentially three main

kinds of system testing:
 Alpha Testing
 Beta Testing
 Acceptance Testing

56
Alpha testing

 System testing is carried out

 by the test team within the
developing organization.

57
Beta Testing

 Beta testing is the system testing:

 performed by a select group of
friendly customers.

58
Acceptance Testing

 Acceptance testing is the system

testing performed by the customer
 to determine whether he should
accept the delivery of the system.

59
System Testing

 During system testing, in addition

to functional tests:
 performance tests are performed.

60
Performance Testing

 Addresses non-functional
requirements.
 May sometimes involve testing hardware
and software together.
 There are several categories of
performance testing.

61
Stress testing

 Evaluates system performance

 when stressed for short periods of
time.
 Stress testing
 also known as endurance testing.

62
Stress testing
 Stress tests are black box tests:
 designed to impose a range of
abnormal and even illegal input
conditions
 so as to stress the capabilities of the
software.

63
Stress Testing
 If the requirements is to handle a
specified number of users, or
devices:
 stress testing evaluates system
performance when all users or
devices are busy simultaneously.

64
Stress Testing
 If an operating system is supposed to
support 15 multiprogrammed jobs,
 the system is stressed by attempting to run 15
or more jobs simultaneously.
 A real-time system might be tested
 to determine the effect of simultaneous arrival
of several high-priority interrupts.

65
Stress Testing

 Stress testing usually involves an element

of time or size,
 such as the number of records transferred per
unit time,
 the maximum number of users active at any
time, input data size, etc.
 Therefore stress testing may not be
applicable to many types of systems.
66
Volume Testing
 Addresses handling large amounts of
data in the system:
 whether data structures (e.g. queues,
stacks, arrays, etc.) are large enough to
handle all possible situations
 Fields, records, and files are stressed to
check if their size can accommodate all
possible data volumes.

67
Configuration Testing
 Analyze system behavior:
 in various hardware and software
configurations specified in the requirements
 sometimes systems are built in various
configurations for different users
 for instance, a minimal system may serve a
single user,
 other configurations for additional users.

68
Compatibility Testing

 These tests are needed when the

system interfaces with other
systems:
 check whether the interface
functions as required.

69
Compatibility testing
Example
 If a system is to communicate
with a large database system to
retrieve information:
 a compatibility test examines speed
and accuracy of retrieval.

70
Recovery Testing

 These tests check response to:

 presence of faults or to the loss of
data, power, devices, or services
 subject system to loss of resources
 check if the system recovers properly.

71
Maintenance Testing

 Diagnostic tools and procedures:

 help find source of problems.
 It may be required to supply
 memory maps
 diagnostic programs
 traces of transactions,
 circuit diagrams, etc.

72
Maintenance Testing

 Verify that:
 all required artifacts for
maintenance exist
 they function properly

73
Documentation tests

 Check that required documents

exist and are consistent:
 user guides,
 maintenance guides,
 technical documents

74
Documentation tests

 Sometimes requirements specify:

 format and audience of specific
documents
 documents are evaluated for
compliance

75
Usability tests

 All aspects of user interfaces are

tested:
 Display screens
 messages
 report formats
 navigation and selection problems

76
Environmental test
 These tests check the system’s ability to perform
at the installation site.
 Requirements might include tolerance for
 heat
 humidity
 chemical presence
 portability
 electrical or magnetic fields
 disruption of power, etc.

77
Test Summary Report

 Generated towards the end of testing

phase.
 Covers each subsystem:
 a summary of tests which have been
applied to the subsystem.

78
Test Summary Report

 Specifies:
 how many tests have been applied to a
subsystem,
 how many tests have been successful,
 how many have been unsuccessful, and the
degree to which they have been unsuccessful,
 e.g. whether a test was an outright failure
 or whether some expected results of the test were
actually observed.

79
Regression Testing

 Does not belong to either unit

test, integration test, or system
test.
 In stead, it is a separate dimension
to these three forms of testing.

80
Regression testing

 Regression testing is the running of

test suite:
 after each change to the system or after
each bug fix
 ensures that no new bug has been
introduced due to the change or the bug
fix.

81
Regression testing

 Regression tests assure:

 the new system’s performance is at
least as good as the old system
 always used during phased system
development.

82
Summary

 We discussed two additional

white box testing
methodologies:
 data flow testing
 mutation testing

83
Summary

 Data flow testing:

 derive test cases based on definition and use
of data
 Mutation testing:
 make arbitrary small changes
 see if the existing test suite detect these
 if not, augment test suite

84
Summary

 Cause-effect graphing:
 can be used to automatically derive test
cases from the SRS document.
 Decision table derived from cause-effect
graph
 each column of the decision table forms
a test case

85
Summary

 Integration testing:
 Develop integration plan by
examining the structure chart:
 big bang approach
 top-down approach
 bottom-up approach
 mixed approach

86
Summary: System testing

 Functional test
 Performance test
 stress
 volume
 configuration
 compatibility
 maintenance
87