Introduction to Software
Testing
Chapter 9.1
Syntax-based Testing
Paul Ammann & Jeff Offutt
Ch. 9 : Syntax Coverage
Four Structures for
Modeling Software
Input Graph Logic Synta
Space Applie s Applied x
d to to
Applie
Sourc FSMs
d to
e
Specs DNF
Source Specs Sourc Model
e s
Design Use Integ Input
Introduction to Software Testing, Edition 2 (Ch 07)
cases © Ammann & Offutt 2
Using the Syntax to Generate
Tests
• Lots of software artifacts follow strict syntax
rules
• The syntax is often expressed as a grammar in
a language such as BNF (Backus-Naur Form)
• Syntactic descriptions can come from many
sources
– Programs
– Integration elements
– Design documents
– Input descriptions
• Tests are created with two general goals
– Cover the syntax in some way
– Violate the syntax (invalid tests)
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 3
Grammar Coverage Criteria
• Software engineering makes practical use of
automata theory in several ways
– Programming languages defined in BNF
– Program behavior described as finite state
machines
– Allowable inputs defined by grammars
• A simple regular expression:
‘*’ is closure
(G s n | B t operator, zero or
n)* more occurrences
‘|’ is choice,
either one can be
used
• Any sequence of “G s n” and “B t n”
• ‘G’ and ‘B’ could represent commands, methods, or
events
• ‘s’, ‘t’, and ‘n’ can represent arguments, parameters,
or values
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 4
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 5
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 6
Test Cases from Grammar
• A string that satisfies the derivation rules is
said to be “in the grammar”
• A test case is a sequence of strings that satisfy
the regular expression
• Suppose ‘s’, ‘t’ and ‘n’ are numbers
G 26 08 01
90
Could be one test with four
B 22 06 27 parts or four separate
tests, etc.
94
G 22 11 21
94
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 7
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 8
BNF Grammars
Stream ::= action*
Start
action ::= actG | actB symbol
Non-
actG ::= “G” s n terminals
actB ::= “B” t n Production
s ::= digit1-3 rule
t ::= digit1-3 Termina
ls
n ::= digit “.” digit “.” digit
2 2 2
digit ::= “0” | “1” | “2” | “3” | “4” | “5” | “6” |
“7” | “8” | “9”
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 9
Wikipedia
In computer science, terminal and nonterminal
symbols are the lexical elements used in
specifying the production rules constituting a
formal grammar.
Terminal symbols are the elementary symbols
of the language defined by a formal grammar.
Nonterminal symbols (or syntactic variables)
are replaced by groups of terminal symbols
according to the production rules.
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 10
Using Grammars
Stream ::= action action *
::= actG action*
::= G s n action*
::= G digit1-3 digit2 . digit2 . digit2 action*
::= G digitdigit digitdigit.digitdigit.digitdigit action*
::= G 25 08.01.90 action*
…
• Recognizer : Is a string (or test) in the
grammar ?
– This is called parsing
– Tools exist to support parsing
– Programs can use them for input validation
• Generator : Given a grammar, derive strings in
the grammar
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 11
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 12
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 13
Mutation as Grammar-Based
Testing
Grammar-based
Testing
UnMutated Derivations Mutated Derivations
(valid strings) (invalid strings)
Grammar Mutation Ground String
Now we can Mutation
define generic
(invalid strings)
coverage criteria
Invalid Strings Valid Strings
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 14
Grammar-based Coverage Criteria
(9.1.1)
• The most common and straightforward criteria
use every terminal and every production at
least once
Terminal Symbol Coverage (TSC) : TR contains
each terminal symbol t in the grammar G.
Production Coverage (PDC) : TR contains each
production p in the grammar G.
• PDC subsumes TSC
• Grammars and graphs are interchangeable
– PDC is equivalent to EC, TSC is equivalent to NC
• Other graph-based coverage criteria could be defined
on grammar
– But have not
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 15
Grammar-based Coverage Criteria
• A related criterion is the impractical one of
deriving all possible strings
Derivation Coverage (DC) : TR contains every
possible string that can be derived from the
grammar G.
• The number of TSC tests is bound by the number of
terminal symbols
– 13 in the stream grammar
• The number of PDC tests is bound by the number of
productions
– 18 in the stream grammar
• The number of DC tests depends on the details of the
grammar
– 2,000,000,000 in the stream grammar !
• All TSC, PDC and DC tests are in the grammar … how
about tests that are NOT in the grammar ?
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 16
Mutation Testing
(9.1.2)
• Grammars describe both valid and invalid
strings
• Both types can be produced as mutants
• A mutant is a variation of a valid string
– Mutants may be valid or invalid strings
• Mutation is based on “mutation operators”
and “ground strings”
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 17
What is Mutation ?
mutatio
General View n
We are performing mutation analysis
operato
whenever we grammar
rs
s
• use well defined rules Applied universally
or according to
• defined on syntactic descriptions
empirically verified
• to make systematic changes distributions
• to the syntax or to objects developed from
grammar ground strings
the syntax
(tests or
programs)
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 18
Mutation Testing
• Ground string: A string in the grammar
– The term “ground” is used as an analogy to
algebraic ground terms
• Mutation Operator : A rule that specifies
syntactic variations of strings generated from
a grammar
• Mutant : The result of one application of a
mutation operator
– A mutant is a string either in the grammar or very
close to being in the grammar
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 19
Mutants and Ground Strings
• The key to mutation testing is the design of
the mutation operators
– Well designed operators lead to powerful testing
• Sometimes mutant strings are based on
ground strings
• Sometimes they are derived directly from the
grammar
– Ground strings are used for valid tests
– InvalidValid
tests Mutants
do not need ground strings
Invalid
Mutants
Ground Strings
Mutants 7 26
08.01.90
G 26 08.01.90 B 26
08.01.90 B 22 06.27.1
B 22 06.27.94 B 45
06.27.94
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 20
Questions About Mutation
• Should more than one operator be applied at the
same time ?
– Should a mutated string contain more than one mutated
element?
– Usually not – multiple mutations can interfere with each other
– Experience with program-based mutation indicates not
– Recent research is finding exceptions
• Should every possible application of a mutation
operator be considered ?
– Necessary with program-based mutation
• Mutation operators have been defined for many
languages
– Programming languages (Fortran, Lisp, Ada, C, C++, Java)
– Specification languages (SMV, Z, Object-Z, algebraic specs)
Introduction– Modeling
to Software Testing, edition languages
2 (Ch 9) (Statecharts,© Ammann
activity
& Offutt diagrams) 21
Killing Mutants
• When ground strings are mutated to create
valid strings, the hope is to exhibit different
behavior from the ground string
• This is normally used when the grammars are
programming languages, the strings are
programs, and the ground strings are pre-
existing programs
• Killing Mutants : Given a mutant m M for a
derivation D and a test t, t is said to kill m if and
only if the output of t on D is different from the
output of t on m
• The derivation D may be represented by the list
of productions or by the final string
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 22
Syntax-based Coverage Criteria
• Coverage is defined in terms of killing mutants
Mutation Coverage (MC) : For each m M, TR
contains exactly one requirement, to kill m.
• Coverage in mutation equates to number of mutants
killed
• The amount of mutants killed is called the mutation
score
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 23
Syntax-based Coverage Criteria
• When creating invalid strings, we just apply
the operators
• This results in two simple criteria
• It makes sense to either use every operator
once or every production once
Mutation Operator Coverage (MOC) : For each
mutation operator, TR contains exactly one
requirement, to create a mutated string m
that is derived using the mutation operator.
Mutation Production Coverage (MPC) : For
each mutation operator, TR contains several
requirements, to create one mutated string m
that includes every production that can be
mutated by that operator.
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 24
Example
Stream ::= action*
action ::= actG | actB Grammar
actG ::= “G” s n
actB ::= “B” t n
s ::= digit1-3
t ::= digit1-3
n ::= digit2 “.” digit2 “.” digit2
digit ::= “0” | “1” | “2” | “3” | “4” | “5” | “6” | “7” | “8” | “9”
Ground Mutation Operators
String • Exchange actG and actB
G 25 • Replace digits with all
08.01.90 other digits
B 21 Mutants using MPC
06.27.94 B 25 08.01.90 G 21
Mutants using
06.27.94
MOC
G 15 08.01.90 B 22
B 25 08.01.90
06.27.94
B 23 06.27.94
G 35 08.01.90 B 23
06.27.94
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 25
Mutation Testing
• The number of test requirements for mutation
depends on two things
– The syntax of the artifact being mutated
– The mutation operators
• Mutation testing is very difficult to apply by
hand
• Mutation testing is very effective – considered
the “gold standard” of testing
• Mutation testing is often used to evaluate other
criteria
Introduction to Software Testing, edition 2 (Ch 9) © Ammann & Offutt 26
Instantiating Grammar-Based
Testing
Grammar-Based Testing
Program- Integratio Model- Input-
basedString n String Based
String Based
mutation mutation String
mutation
• Program mutation
• FSMs
mutation • Input
Grammar • Model
• Valid strings validation
• Mutants are not checking
• Valid strings testing
tests • XML and
• • Traces are
• Compiler testing Must kill mutants tests
others
• Test how classes
• Valid and invalid Grammar • Invalid strings
strings interact • No ground
• Valid strings strings
• Mutants are not • Input validation • Mutants are
tests testing tests
• Must kill mutants • XML and others
Introduction to Software Testing, edition 2 (Ch 9) • Includes OO • & Offutt
© Ammann Valid strings 27
Structure of Chapter
Program-based Integration Model-based Input space
Grammar 9.2.1 9.3.1 9.4.1 9.5.1
Gramm Programming No known Algebraic Input
ar languages applications specifications languages,
including XML
Summar Compiler Input space
y testing testing
Valid? Valid & invalid Valid
Mutatio 9.2.2 9.3.2 9.4.2 9.5.2
n
Gramm Programming Programming FSMs Input
ar languages languages languages,
including XML
Summar Mutates Tests Model Error checking
y programs integration checking
Ground? Yes Yes Yes No
Valid? Yes, must Yes, must Yes No
compile compile
Tests? Mutants not Mutants not Traces are Mutants are
tests
Introduction to Software Testing, edition 2 (Ch 9)
tests tests
© Ammann & Offutt
tests 28
