Functional System Testing

Written by
Adam Carmi
 Outline
• Goal of testing
• Test cases, test suites and test data
• What is functional system testing?
• Coverage
• Functional testing techniques:
– Functional analysis
– Equivalence partitioning
– Boundary value analysis
 The goal of software testing
• The process of uncovering evidence of defects in
software systems
– Does not include efforts associated with tracking down
bugs and fixing them
• No amount of testing will improve the quality of a
computer program
– The more testing we do of a system, the more
convinced we might be of its correctness
– Testing cannot in general prove a system works 100%
correctly
 Test cases
• The basic component of testing is a Test Case
• In its most general form: (inputs, expected-result)
– inputs include system state, user commands and data
values to be processed
– expected result includes visible/audible interface
changes or changes in the system state
• Test cases are organized into Test Suites
– functionality, security, performance, …
 Test case execution

• A running of the software (under test) that

provides the inputs specified in the test case
and observing the results and comparing
them to those specified by the test case
– If the actual result varies from the expected
result, then a failure has been detected
 Test data
• An effective test strategy requires careful acquisition and
preparation of test data prior to testing
– Testing can suffer if test data is poor
• Test data concerns:
– Depth: quantity and size of data
– Breadth: variance of data values and data types
– Scope: completeness, relevance and accuracy of data
• Result of a query should be valid for the specific purpose of the
query, and not due to a missing or inappropriate value
– Conditions: data should reflect specific “conditions” in the domain
• Data that would otherwise arrive after performing specific operations
over time
• Test data and test results are expensive to construct
 Example: Test data for TVRS
Name: test1.db

Description: Violation records designed for validating violation lookup

Issuing policeman ID Offender’s last name Offender’s first name Violation ID

8700342 Josef Rachel 243567

6386541 Levi Dan 237812

1346329 Porat Dan 264683

8245731 Josef Dina 255245

8700342 longLastN longFirstN 000345

… … … …
 Specification vs. implementation
• The basic approaches to testing software are based on its
specification and implementation
• White box testing – test cases and data are constructed
based on the code that implements the software
– quality and correctness of computations is validated
– will not be further discussed in this tutorial
• Black box testing – test cases and data are constructed
based solely on the software’s specification
• Gray box testing – test cases are constructed to directly
target various modules and layers of the system (requires
architectural insight)
 Functional System Testing
• Testing of a completed application to determine
that it provides all of the behaviors required of it
– Testing of completed increments that provide some
degree of end-user functionality
• Search for defects that are variances between the
actual operation of the system and the
requirements for the system
• System is treated as a black/gray box
 How much testing is adequate?

• Completely validating IEEE 754 floating-

point division requires 264 test-cases!
float divide(float x, float y)

• From practical and economic perspectives,

exhaustive testing is usually not possible
– Which software pieces should we test?
– Which test cases should we choose?
 Coverage
• Coverage is a measure of how completely a test
suite exercises the capabilities of a piece of
software
– “Each line of code should be executed at least once”
– “One test case should be constructed from each
specified requirement”
• It is necessary to use testing techniques that
narrow down the number of test cases allowing the
broadest testing coverage with the least effort
 Technique: Functional Analysis

• Analyze the expected behavior of the system

according to its functional specification
• Generate a test procedure for each of the possible
usage scenarios
– Corresponds to use case scenarios
– Analyze how a change in one part of the system affects
other parts
– “Grand tour” test cases: the result of one test case
produces the data that is the input to the next test case
Example: Functional Analysis I

Use
UseCase:
Case:Remove
RemoveTraffic
TrafficViolation
Violation
1.1. Supervisor
Supervisorcalls
callsfor
fordeletion
deletionofofthe
thechosen
chosenTraffic
TrafficViolation
Violation
2.2. TVRS
TVRSprompts
promptsSupervisor
Supervisorforforconfirmation
confirmation
3.3. Supervisor
Supervisorconfirms
confirms
4.4. TVRS
TVRSrequests
requestsOffendersDB
OffendersDBtotodelete
deletethe
theTraffic
TrafficViolation
Violation
from
fromthe
theoffender’s
offender’srecord
record
5.5. OffendersDB
OffendersDBapproves
approvesthat
thatthe
theTraffic
TrafficViolation
Violationhas
hasbeen
been
deleted
deleted
6.6. TVRS
TVRSallows
allowsSupervisor
Supervisortotolook
lookupupaanew
newTraffic
TrafficViolation
Violationasas
described
describedininthe
the“Lookup
“LookupTraffic
TrafficViolation”
Violation”UCUC
 Example: Functional Analysis II
Test case ID: 134543
Pre-conditions: 1. TVRS initialized with test1.db database
2. Violation 243567 displayed in the “Lookup Violation” dialog
Related use cases: “Lookup Traffic Violation”, “Remove Traffic Violation”
Expected result Action
Confirmation dialog is displayed Press “Delete” button
“Lookup Violation” dialog is displayed Press the “Yes” button
A message dialog stating that violation Enter “243567” at “Violation ID” text field
“243567” is not stored in TVRS and press the “Search” button
Test results
Actual results: Passed
Failed
Defect diagnosis:
Example: Functional Analysis III
How
Howdo dowe weknow
knowthat
that
violation
violation243567
243567isis
??stored
storedininthe
thesystem
system

In
Inaddition,
addition,aaquery
query
could
couldbe
berun
runon
onthe
the
Offenders database
Offenders database

Verify effects
of change

Filled when
Can
Canaatester
tester the test case
diagnose
diagnosethethecause
cause is executed
of
ofaadefect?
defect?
 Technique: Equivalence
Partitioning
• Identifies ranges of input and initial conditions
that are expected to produce the same result
• A group of test cases form an equivalence class if:
– They test the same feature/scenario
– If one test reveals a fault, the other ones (probably) will
too
– If a test does not reveal a fault, the other ones
(probably) will not either
• It is adequate to use only a single representative of
the equivalence class
Example: Equivalence Partitioning I

Input
Inputvalue
valuespecification
specificationfor
for“Lookup
“LookupViolation”
Violation”form:
form:

Valid values Field name

[0-9]{0, 9} Violation ID

[a-zA-Z]{0, 10} Offender’s first name

[a-zA-Z]{0, 10} Offender’s last name

 Example: Equivalence Partitioning II
Invalid Invalid Valid Valid Field
representative equivalent representative equivalent
values values classes
classes
-1, 1234567890 ID < 0 or ID > 00243567 Known violation Violation ID
999999999

23ab@ Non numeric ID 32456720 Unknown violation

“” Empty
Hasalongname Character# > 10 David Unknown violation Offender’s
first name
ad0@am Invalid character Rachel Single known
violation
Dan Many known
violations
“” Empty
… … … …
 Example: Equivalence Partitioning III

• The number of test cases to choose from is reduced to

(3 + 3) × (4 + 2) × (4 + 2) = 216
• The actual number can be further limited
– Single invalid field per test case (3 × 4 × 4 + 7 = 55)
– Importance of use case
– Resources available
– Most frequent input
– Life-critical software
– Infeasible test cases
– Randomly
– ...
 Technique: Boundary Value
Analysis
• Based on experience / heuristics
– Testing boundary conditions of equivalence classes is
more effective
• Choose input boundary values as equivalence
classes representatives
• Choose inputs that invoke output boundary values
• Examples:
– (0, 10] ⇒ validate using 0, 1, 2, 9, 10, 11
– Read up to 5 elements ⇒ validate reading 0, 1, 4, 5, 6 elements
 BVA as an equivalence
partitioning extension
• Choose one (or more) arbitrary value(s) in
each equivalence class
• Choose valid values exactly on lower and
upper boundaries of equivalence class
• Choose invalid values immediately below
and above each boundary (if applicable)
 General purpose test-suite
construction technique
• May be used to obtain reasonable coverage with little
effort
– Use cautiously!
– Unsuitable when values of different fields are related

1.1. While
Whiletest
testcases
casescan
canbebeadded
added
•• Each
Each new test caseshould
new test case shouldinclude
includeasasmany
manyun-included
un-includedvalid
validnon-boundary
non-boundary
equivalence class representatives as possible
equivalence class representatives as possible
2.2. While
Whiletest
testcases
casescan
canbebeadded
added
•• Each
Each new test caseshould
new test case shouldinclude
includeasasmany
manyun-included
un-includedvalid
validboundary
boundary
equivalence class representatives as possible
equivalence class representatives as possible
3.3. While
Whiletest
testcases
casescan
canbebeadded
added
•• Each
Eachnew
newtest
testcase
caseshould
shouldinclude
includeaasingle
singleinvalid
invalidequivalence
equivalenceclass
class
representative
representativethat
thathas
hasnot
notbeen
beenincluded
includedbefore
before
4.4. Manually
Manually replace/remove redundant or infeasibletest-cases
replace/remove redundant or infeasible test-cases
 Example: Country Club I
Specification
Specification
Friday - Saturday Sunday - Thursday Day

Student Member Visitor Student Member Visitor Guest

status
Admission fee Age
(years)
30 10 35 20 10 25 [0, 16)

65 25 70 45 25 50 [16, 60)

45 15 50 30 15 35 [60, 120]
 Example: Country Club II
Invalid Invalid Valid Valid Field
representative equivalent representative equivalent
values values classes
classes
Mon, Sun, Thu Sun - Thu Day

Fri, Sat Fri - Sat

AAcombo
combobox
boxisisused
used
Visitor Visitor
for Guest
forchoosing
choosingthe day
thestatus
day
Member and
andguest
Member gueststatus
status
Student Student

14@a Non-numeric 2, 0, 15 [0, 16) Age

value
-1, 121 Age < 0 or 34, 16, 59 [16, 60)
Age > 120
100, 60, 120 [60, 120]
 Example: Country Club III
Result Age Guest status Day Test case ID
25 2 Visitor Mon 1
25 34 Member Fri 2 valid
30 100 Student Mon 3
25 0 Visitor Sun 4
10 16 Member Sat 5
30 60 Student Thu 6 valid
10 15 Member Sun 7 )boundary(
45 120 Student Sat 8
50 59 Visitor Thu 9
Invalid age 14@a Member Mon 10
Invalid age -1 Student Fri 11 invalid
Invalid age 121 Visitor Fri 12