Introduction to Software

Testing

Software Testing and Quality

Assurance
Part 7
Syntax Based Testing
Using the Syntax to Generate
Tests
Lots of software artifacts follow strict syntax rules
The syntax is often expressed as some sort of
grammar such as BNF
Syntactic descriptions can come from many
sources
Programs
Integration elements
Design documents
Input descriptions
Tests are created with two general goals
Cover the syntax in some way
Violate the syntax (invalid tests – mutation testing)
2
 Grammar Coverage Criteria –
Example Grammar
• Simple grammar for a toy language of arithmetic expressions in
BNF notation

3
 Example Derivations

4
 Using Grammars
Recognizer : Given a string (or test), is the string in the
grammar ?
This is called parsing
Tools exist to support parsing
Programs can use them for input validation
Generator : Given a grammar, derive strings in the
grammar

5
 Syntax-based Coverage Criteria
The most common and straightforward use every
terminal and every production at least once

Terminal Symbol Coverage (TSC) : TR contains

each terminal symbol t in the grammar G.

Production Coverage (PC) : TR contains each

production p in the grammar G.
• PC subsumes TSC
• Grammars and graphs are interchangeable
• Think of PC as Edge-Coverage while TSC is node
coverage
 Syntax-based Coverage Criteria
A related criterion is the impractical one of
deriving all possible strings
Derivation Coverage (DC) : TR contains every
possible string that can be derived from the
grammar G.

© Ammann & Offutt Introduction to Software Testing (Ch 5),

7
www.introsoftwaretesting.com
Coverage Criteria
Terminal Symbol Coverage (TSC)
TR contains each terminal in the grammar.
One test case per terminal.
Toy language example: 40 test requirements
Production Coverage (PDC)
TR contains each production rule in the grammar.
One test case per production (hence PC subsumes TSC).
Toy language example: 47 test requirements
Derivation Coverage (DC)
TR contains every possible derivation of the grammar.
One test case per derivation required.
Not practical - TR usually infinite as in our toy language.
When applicable, DC subsumes PDC.
8
Exercise

9
Solution

10
 Grammar vs. Ground String
 Grammar in terms of software is the actual syntax rules of a
program. Therefore, we would be interested in mutating the
grammar itself if the software under test is the actual compiler
and we want to check if the compiler would world correctly
incase of an erroneous grammar. For example we are checking
if a Java compiler is working correctly.
 Ground String in terms of software is an instance of a
program that was created based on syntax rules defined in a
grammar. As such, any Java program IS-A ground string of the
Java grammar. Mutating a ground string means we are testing
the Java program that we have created. If we mutate it to
create invalid ground strings then we creating syntactically
incorrect programs which is the job of the compiler to check. If
we are mutate a Java program that results in valid string
(syntactically correct Java program) but one that would have a
11 different behavior from the original program.
Grammar-Based Mutation
What is mutation?
A set of rules (mutation operators)
To be applied on a syntactical descriptions (grammars)
Or ground strings (instances of grammar derivations)
By applying systematic changes to the syntax
(grammar) or to objects (ground strings) developed
from the syntax
In simple terms, we are systematically defining and
implementing ways to mess up what is defined as
proper with the purpose of generating invalid
derivations.
Those invalid derivations should be detected as
errors by the software under test. In other words, we
are making sure the software is not doing what
12

it is not supposed to do.

 Mutation as Grammar-Based
Testing
Grammar-based
Testing

UnMutated Derivations Mutated Derivations

(valid strings) (invalid strings)

Grammar Mutation Ground String

Generic (invalid strings) Mutation
coverage
criteria can
13
now be Invalid Strings Valid Strings
defined
 Mutation Testing
Mutation Operator : A rule that specifies
syntactic variations of strings generated from a
grammar

Mutant : The result of one application of a

mutation operator
A mutant is a string

© Ammann & Offutt Introduction to Software Testing (Ch 5),

14
www.introsoftwaretesting.com
 Mutation Testing
Mutation
Operator

Grammar Mutant
Grammar

Mutation
Operator

Ground String
Mutant Ground
String
15
 Mutant Grammar
Mutation Mutation Mutation
Operator 1 Operator 2 Operator 3

Grammar

Mutant Mutant Mutant

Grammar 1 Grammar 2 Grammar 3

16
 Mutant Ground Strings
Mutation Mutation Mutation
Operator 1 Operator 2 Operator 3

Ground String

Mutant Mutant Mutant

Ground Ground Ground
String 1 String 2 String 3
17
 Questions About Mutation
Should more than one operator be applied at the
same time ?
Should a mutated string contain one mutated
element or several?
Almost certainly not – multiple mutations can
interfere with each other

18
 Grammar-Based Mutation
Mutations over a grammar can be considered
with the purpose of generating invalid derivations.
Invalid derivations must be recognized as errors
by the software under test.
Example mutation operators:
Terminal and nonterminal deletion: remove a
terminal or nonterminal symbol from a production.
Terminal and Non-terminal duplication:
duplicate a terminal or nonterminal symbol in a
production.
Terminal replacement: replace a terminal by
another terminal.
19
Non-terminal replacement
 Grammar Mutations -
Example

20
 Exercise
Consider nonterminal duplication over expr.
Describe the mutated grammar and provide one
mutant derivation per each mutated production rule.

21
Exercise

22
Program-Based Mutation
Program-Based Mutation
 Killing Mutants
When ground strings are mutated to create valid
strings, the hope is to exhibit different
behavior from the ground string

Killing Mutants : Given a mutant m  M for a

derivation D and a test t, t is said to kill m if and
only if the output of t on D is different from the
output of t on m

The derivation D may be represented by the list

of productions or by the final string
25
Killing Mutants
Mutation
Operator

Mutant
Grammar
Grammar

Test

If Output 1 does not equal Output 2 then mutant is killed!

Killing Mutants
Mutation
Operator

Mutant Ground
Ground String
String

Test

If Output 1 does not equal Output 2 then mutant is killed!

Program-Based Mutation –
Example 1
Program-Based Mutation –
Example 2

none
m1
m1 and m3

Observe that m2 can not be

killed, Why not?
Program-Based Mutation –
Example 2
 Syntax-based Coverage Criteria
Coverage is defined in terms of killing mutants.

Mutation Coverage (MC) : For each m  M, TR

contains exactly one requirement, to kill m.

• Coverage in mutation equates to number of mutants killed

• The amount of mutants killed is called the

mutation score. For example if we have 5 mutants
and only 4 were killed then the mutation score is
80%.

31
 Syntax-based Coverage Criteria
When creating invalid strings, we just apply the operators
This results in two simple criteria
It makes sense to either use every operator once or every
production once

Mutation Operator Coverage (MOC) : For each

mutation operator, TR contains exactly one
requirement, to create a mutated string m
that is derived using the mutation operator.
Mutation Production Coverage (MPC) : For
each mutation operator, TR contains several
requirements, to create one mutated string m
that includes every production that can be
mutated by that operator.
32
Example
Stream ::= action*
action ::= actG | actB Grammar
actG ::= “G” s n
actB ::= “B” t n
s ::= digit1-3
t ::= digit1-3
n ::= digit2 “.” digit2 “.” digit2
digit ::= “0” | “1” | “2” | “3” | “4” | “5” | “6” | “7” | “8” |
“9”
Ground
String
G 18
08.01.90
B 14
06.27.94
Mutants using Mutation Operators
MOC • Exchange actG and actB
B 18 08.01.90 • Replace digits with
B 19 06.27.94 another digit
33
Example
Stream ::= action*
action ::= actG | actB Grammar
actG ::= “G” s n
actB ::= “B” t n
s ::= digit1-3
t ::= digit1-3
n ::= digit2 “.” digit2 “.” digit2
digit ::= “0” | “1” | “2” | “3” | “4” | “5” | “6” | “7” | “8” |
“9”
Ground
String
G 18
08.01.90
Mutants using MPC
B 14 Operators
Mutation B 18 08.01.90 G 14
06.27.94
• Exchange actG and actB 06.27.94
• Replace digits with G 28 08.01.90 B 11
another digit 06.27.94
G 38 08.01.90 B 13
06.27.94
34
G 48 08.01.90 B 15
 Mutation Testing
The number of test requirements for
mutation depends on two things
The syntax of the artifact being mutated
The mutation operators

Mutation testing is very effective –

considered the “gold standard” of testing

35
 Syntax-Based Coverage Criteria
Mutation Coverage (MC) : For each m  M, TR
contains exactly one requirement, to kill m.

• The RIP model:

• Reachability : The test causes the faulty statement
to be reached (in mutation – the mutated
statement)
• Infection : The test causes the faulty statement to
result in an incorrect state
• Propagation : The incorrect state propagates to
incorrect output

• The RIP model leads to two variants of mutation

36
coverage …
© Ammann & Offutt Introduction to Software Testing (Ch 5),
www.introsoftwaretesting.com
 Syntax-Based Coverage Criteria

• 1) Strongly Killing Mutants:

Given a mutant m  M for a program P and a test t, t
is said to strongly kill m if and only if the output of t
on P is different from the output of t on m

• 2) Weakly Killing Mutants:

Given a mutant m  M that modifies a location l in a
program P, and a test t, t is said to weakly kill m if
and only if the state of the execution of P on t is
different from the state of the execution of m
immediately on t after l

• Weakly killing satisfies reachability and infection,

but not propagation
© Ammann & Offutt Introduction to Software Testing (Ch 5),
37
www.introsoftwaretesting.com
 Weak Mutation
Weak Mutation Coverage (WMC) : For each m  M,
TR contains exactly one requirement, to weakly kill
m.
• “Weak mutation” is so named because it is easier to
kill mutants under this assumption

• Weak mutation also requires less analysis

• Some mutants can be killed under weak mutation but

not under strong mutation (no propagation)

38
 Trivial Example
Assume a program that uses the larger value resulting
from an addition function and a square function

//…code……
r1 = add (a , b);
r2 = square (x) ;
//uses the larger of r1 and r2
//……rest of the code…

39
 Trivial Example
int square (int x){ int square (int x){
if (x == 0 ) if (x == 0 )
return 0; return 0;
else else
return x*x; // fault return x*2; //m1
} }

int add (int a, int b){ int add (int a, int b){
return a+b; return a+b;
} }

No Reachability: make x equals 0

Reachability (but no infection): make x equals 2
Infection (but no propagation): make x equals anything
other than 0 or 2 but that x*2 and x*x are both less than a+b
Propagation: make x equals anything other than 0 or 2, but
40 that x*2 is less than a+b while x*x is more than a+b