SoftwareTesting Introduction
Dr Atul Gupta
DrAtulGupta
Errors defects faults & Failures
Errors,defects,faults&Failures
Error:Amistake madebyaprogrammer
Example:Misunderstoodtherequirements.
Defect/fault/bug:Manifestationofanerrorin
a program
aprogram.
Example:
Incorrectcode:
Correct code
Correctcode:
if(a<b){foo(a,b);}
if(a>b){foo(a,b);}
if
(a>b) {foo(a b) }
Failure:
Failure:Manifestationofoneormorefaults
Manifestation of one or more faults in
in
theobservedprogrambehavior
Failure
Incorrect
Incorrectprogrambehaviorduetoafaultin
program behavior due to a fault in
theprogram.
Failurecanbedeterminedonlywithrespectto
Failure can be determined only with respect to
asetofrequirementspecifications.
Anecessary
A necessary conditionforafailuretooccuris
condition for a failure to occur is
thatexecutionoftheprogramforcethe
erroneous portion of the program to be
erroneousportionoftheprogramtobe
executed.
Sufficient condition?
T ti P bl
TestingProblem
Error-revealing
inputs cause
failure
I
Inputs
t
Program
Outputs
Erroneous
outputs indicate
failure
Testing Activities in the SDLC
TestingActivitiesintheSDLC
Requirement
Specification
Acceptance
Testing
SRS
Tested Software
System Design
System Testing
System Design
Subsystem
Detailed Design
g
Integration
g
Testing
g
Detailed Design
Tested Units
Coding
Unit Testing
Code
Levels of Testing
LevelsofTesting
Type of Testing
Performed By
Lowleveltesting
Unit(module) testing
Integration
g
testingg
Programmer/Developer
Development team
Developmentteam
Highleveltesting
System testing
Function testing
Acceptance testing
IndependentTestGroup
Independent
Test Group
IndependentTestGroup
Customer
Testing Terms
TestingTerms
Test
Testcase
case Asetof
A set of Inputs,executionpreconditions,and
Inputs, execution preconditions, and
expectedoutcomesfortestinganspecificaspectofCUT
TestSuite
Test Suite Acollection
A collection oftestcasesfortheCUT
of test cases for the CUT
TestCriterion Asetoftestrequirements
Failure
Failure Anobservation
An observation ofthebehaviorofCUTdifferentfrom
of the behavior of CUT different from
expectedone
Fault
Fault Acondition
A condition inCUTthatmaycausefailure
in CUT that may cause failure
Effectiveness Faultdetectioncapability
Efficiency Theaveragetestingcost(i.e.effort)toidentifya
fault intheprogram
A Testing Process
ATestingProcess
The Code Under Test (and
its Specifications)
Procedural
Systems
Object-Oriented
Systems
Test Case Selection
(Test Input Selection)
Test Case Selection
(Test Sequence
Selection + Test Input
Selection)
Test Case Execution
Comparison with
Expected Outcomes
(Test Oracle)
Test Execution Results
(Pass/Fail Information)
NO
Stop?
YES
Stop
ATestingProcess
g
The Code Under Test (and
its Specifications)
Procedural
Systems
Object-Oriented
Systems
Test Case Selection
(Test Input Selection)
Test Case Selection
(Test Sequence
Selection + Test Input
Selection)
Test Case Execution
Comparison with
Expected Outcomes
(Test Oracle)
YES
Fail
?
Debugging/
Bug Fixing
NO
Test Execution Results
(Pass/Fail Information)
Stop?
YES
Stop
A Testing Process
ATestingProcess
The Code Under Test (and
its Specification)
Procedural
Systems
Object-Oriented
Systems
Test Case Selection
(Test Input Selection)
Test Case Selection
(Test Sequence
Selection + Test Input
Selection)
Test Case Execution
Comparison with
Expected Outcomes
(Test Oracle)
YES
Debugging/
Bug Fixing
Test Execution Results
(Pass/Fail Information)
Fail
?
NO
Stop
NO
Stop?
YES
Stop
Unit Testing
UnitTesting
Doneonindividualunits(Class
Done on individual units (Class orsmallcluster
or small cluster
classesinOOSystemandafunction ormodules
in Procedural Systems)
inProceduralSystems)
Testunitw.r.tunitspecification
Mostlydonebydevelopers/programmers
M l d
b d l
/
requiresstubs anddrivers
FurtherPopularizedbytheavailabilityof Unit
TestingFrameworks
g
StubsandDrivers
Stub
Stub
dummyunitswhich
simulates the
simulatesthe
functionofother
units required to
unitsrequiredto
testunitundertest
Driver
e.g.
g unit
relationships
eg.
g to unit test B
in isolation
Stub for A
A
Driver
Codethatexecutes
unit under test
unitundertest
C
Stub for C
Integration Testing
IntegrationTesting
Testsagroupofunits,modules,orasubsystem
Testsubsystemstructurew.r.tdesign,subsystem
y
g ,
y
functions
Focusesonunit
Focuses on unit interfaces
Donebyone/groupofdevelopers
Undertakenbasedondependency considerations
IntegrationTestApproaches
g
pp
Nonincremental(BigBangintegration)
Non incremental (Big Bang integration )
unit test each module independently
combine
bi allll the
th units
it to
t form
f
the
th subsystem
b t
in
i
one step, and test the combination
Incremental
instead of testing each unit in isolation, the
next unit to be tested is first combined with
the set of units that have already been tested
integration approaches:approaches:
Functional System: Top-down, Bottom-up
OO System: Based on dependency considerations
Comparison
Non Incremental
NonIncremental
Incremental
requiresmorestubs,drivers
requireslessstubs,drivers
moduleinterfacingerrors
detectedlate
moduleinterfacingerrors
detectedearly
debuggingerrorsisdifficult
debuggingerrorsiseasier
resultsinmorethoroughtesting
g
g
ofmodules
E
Example:IntegrationTestinginFunctionalSystems
l I
i T i i F
i
lS
Example:IntegrationTestinginOOSystems
A
Topdown Integration
TopdownIntegration
Begin with the top module in the module call
hierarchy
Stub modules are produced
Stubs are often complicated
The next module to be tested is any module
with at least one previously tested
p
((calling)
g) module
superordinate
After a module has been tested, one of its
stubs is replaced by an actual module (the
nextt one to
t be
b tested)
t t d) and
d its
it required
i d stubs
t b
TopdownIntegrationTesting
Example:
A
Stub B
Stub C
Stub D
TopdownIntegrationTesting
Example:
Stub E
Stub F
Stub C
Stub D
BottomUpIntegration
Beginwiththeterminalmodules(thosethat
do not call other modules) of the modules call
donotcallothermodules)ofthemodulescall
hierarchy
Adrivermoduleisproducedforeverymodule
p
y
Thenextmoduletobetestedisanymodule
whosesubordinatemodules(themodulesit
calls)haveallbeentested
ll ) h
ll b
t t d
Afteramodulehasbeentested,itsdriveris
replaced by an actual module (the next one to
replacedbyanactualmodule(thenextoneto
betested)anditsdriver
Example:ModuleHierarchy
A
BottomUpIntegrationTesting
Example:
Driver E
Driver F
BottomUpIntegrationTesting
Example:
Driver A
Comparison
TopdownIntegration
Advantage
askeletalversionofthe
programcanexistearly
Disadvantage
BottomupIntegration
Disadvantage
theprogramasawholedoes
t i t til th l t
d l
notexistuntilthelastmodule
isadded
requiredstubscouldbe
q
expensive
No clear winner
System Testing
SystemTesting
Process of attempting to demonstrate that
the program or system does not meet its
original requirements
req irements and objectives
objecti es as
stated in the requirements specification
Test cases derived from
requirements specification
system objectives, user documentation
System Testing: An Example
SystemTesting:AnExample
Types of System Tests
TypesofSystemTests
Functional Testing
g
Testing Functionality to be delivered by the system
Specialized Testing (Testing NFRs)
Volume testing
to determine whether the program can handle the required volumes of data,
requests, etc.
Load/Stress testing
to identify peak load conditions at which the program will fail to handle required
processing loads within required time spans
Usability (human factors) testing
to identify discrepancies between the user interfaces of a product and the
human engineering requirements of its potential users.
Security Testing
to show that the programs security requirements can be subverted
Types of System Tests
TypesofSystemTests
Specialized Testing
Performancetesting
todeterminewhethertheprogrammeetsitsperformance
requirements (eg response times throughput rates etc )
requirements(eg.responsetimes,throughputrates,etc.)
Recoverytesting
ttodeterminewhetherthesystemorprogrammeetsits
d t
i
h th th
t
t it
requirementsforrecoveryafterafailure
Installabilitytesting
Installability testing
toidentifywaysinwhichtheinstallationproceduresleadto
incorrectresults
ConfigurationTesting
todeterminewhethertheprogramoperatesproperlywhenthe
software or hardware is configured in a required manner
softwareorhardwareisconfiguredinarequiredmanner
Types of System Tests
TypesofSystemTests
Specialized
p
Testing
g
Compatibility/conversiontesting
todeterminewhetherthecompatibilityobjectivesoftheprogram
p
y j
p g
havebeenmetandwhethertheconversionprocedureswork
Reliability/availabilitytesting
todeterminewhetherthesystemmeetsitsreliabilityand
availabilityrequirements
Resourceusagetesting
Resource usage testing
todeterminewhethertheprogramusesresources(memory,disk
space,etc.)atlevelswhichexceedrequirements
Acceptance Testing
AcceptanceTesting
Performed by the customer or end user
Compare the software to its initial
requirements and needs of its end users
Alpha and Beta Testing
AlphaandBetaTesting
Tests performed on a SW Product before
its released to a wide user community.
Alpha testing
conducted at the developers site by a User
tests conducted in a controlled environment
Beta testing
conducted
d t d att one or more User
U
sites
it by
b th
the
end user of the SW
it is a live
live use of the SW in an environment
over which the developer has no control
Regression Testing
RegressionTesting
Testing changes made to the code
To ensure that no new bugs are introduced in the
y
system
Can be performed at any level unit, integration, and
system level
In practice, mostly performed at unit and system level in
GUI testing
Challenges include
Reducing Regression Test Suite size
Identifyy Invalid test cases
Write additional test cases
Automation
When to Stop Testing ?
WhentoStopTesting?
Stop
Stopwhenthescheduledtimefortesting
when the scheduled time for testing
expires
Stopwhenallthetestcasesexecute
g
withoutdetectingerrors
bothcriteriaarenotgood
Better Test Completion Criteria
BetterTestCompletionCriteria
Make use of Coverage based Testing
MakeuseofCoveragebasedTesting
Example:Testcasesderivedfrom
1)satisfyingmulticonditioncoverageand
)
y g
g
2)boundaryvalueanalysisand
3) cause effect graphing
3)causeeffectgraphing
Better Test Completion Criteria
BetterTestCompletionCriteria
Statethecompletioncriteriaintermsof
State
the completion criteria in terms of
numberoferrorstobefound.Thisrequires:
Anestimateoftotalnumber
A
ti t f t t l
b oferrorsinthe
f
i th
program
Anestimateofthe%oferrors thatcanbe
foundthroughtesting
Anestimatesofwhatfractionoferrors
originate inparticulardesignprocesses,and
in particular design processes, and
duringwhatphasesoftesting theyget
detected.
detected.
Better Test Completion Criteria
BetterTestCompletionCriteria
Plotthenumberoferrorsfoundperunittimeduringatest
phase.
Therateoferrordetectionfallsbelowaspecifiedthreshold
#E
Errors ffound
#E
Errors found
40
403020101 2 3 4 5
W k
Week
40
403020101 2 3 4 5
W k
Week
Summary (PartII)
Summary(Part
II)
Testingisperformedatdifferentlevels
g p
UnitTestingisthefirstformaltestactivity
UnitTestingcanbehighlyefficientandcost
effective
ff i
IntegrationTestingisincremental
Regressiontestingisabouttestingchanges
R
i t ti i b t t ti
h
madetothecode
Toolsupportishighlydesirableinsoftware
Tool support is highly desirable in software
testing
y
Howcantheeffectivenessandefficiencyof
softwaretestingcanbeenhancedfurther?
R f
References
Beizer,Boris(1990).SoftwareTesting
Techniques (SecondEditioned.)
(Second Edition ed )
Myers,Glenford J.(1979).TheArtofSoftware
Testing.JohnWileyandSons
Wikipedia
Wikipedia http://en.wikipedia.org/wiki/Software_testing
http://en wikipedia org/wiki/Software testing
