Recursive Descent Parsing
! Goal
• Determine if we can produce the string to be parsed from the
grammar's start symbol
! Approach
• Construct a parse tree: starting with start symbol at root,
recursively replace nonterminal with RHS of production
! Key question: which production to use?
• There can be many productions for each nonterminal
• We could try them all, backtracking if we are unsuccessful
• But this is slow!
! Answer: lookahead
• Keep track of next token on the input string to be processed
• Use this to guide selection of production
CMSC 330 - Spring 2011 1
Recursive Descent Example
E
E → id = n | { L }
{ L }
L→E;L|ε
E ; L
{x=3;{y=4;};}
x = 3 E ; L
lookahead { L } ε
E ; L
y = 4 ε
CMSC 330 - Spring 2011 2
Recursive Descent: Basic Strategy
! Initially, “current node” is start node
! When processing the current node, 4 possibilities
• Node is the empty string
Ø Move to next node in DFS order that has not yet been
processed
• Node is a terminal that matches lookahead
Ø Advance lookahead by one symbol and move to next node in
DFS order to be processed
• Node is a terminal that does not match lookahead
Ø Fail! String cannot be parsed
• Node is a nonterminal
Ø Pick a production based on lookahead, generate children, then
process children recursively
CMSC 330 - Spring 2011 3
Recursive Descent Parsing (cont.)
! Key step
• Choosing which production should be selected
! Two approaches
• Backtracking
Ø Choose some production
Ø If fails, try different production
Ø Parse fails if all choices fail
• Predictive parsing
Ø Analyze grammar to find “First sets” for productions
Ø Compare with lookahead to decide which production to select
Ø Parse fails if lookahead does not match First
CMSC 330 - Spring 2011 4
First Sets
! Example
• Suppose the lookahead is x
• For grammar S → xyz | abc
Ø Select S → xyz since 1st terminal in RHS matches x
• For grammar S → A | B A→x |y B→z
Ø Select S → A, since A can derive string beginning with x
! In general
• We want to choose a production that can derive a
sentential form beginning with the lookahead
• Need to know what terminals may be first in any
sentential form derived from a nonterminal / production
CMSC 330 - Spring 2011 5
First Sets
! Definition
• First(γ), for any sentential form γ, is the set of initial
terminals of all strings that γ may expand to
• We’ll use this to decide what production to apply
! Examples
• Given grammar S → xyz | abc
Ø First(xyz) = { x }, First(abc) = { a }
Ø First(S) = First(xyz) U First(abc) = { x, a }
• Given grammar S → A | B A→x |y B→z
Ø First(x) = { x }, First(y) = { y }, First(A) = { x, y }
Ø First(z) = { z }, First(B) = { z }
Ø First(S) = { x, y, z }
CMSC 330 - Spring 2011 6
Calculating First(γ)
! For a terminal a
• First(a) = { a }
! For a nonterminal N
• If N → ε, then add ε to First(N)
• If N → α1 α2 ... αn, then (note the αi are all the
symbols on the right side of one single production):
Ø Add First(α1α2 ... αn) to First(N), where First(α1 α2 ... αn) is
defined as
• First(α1) if ε ∉ First(α1)
• Otherwise (First(α1) – ε) ∪ First(α2 ... αn)
Ø If ε ∈ First(αi) for all i, 1 ≤ i ≤ k, then add ε to First(N)
CMSC 330 - Spring 2011 7
First( ) Examples
E → id = n | { L } E → id = n | { L } | ε
L→E;L|ε L→E;L
First(id) = { id } First(id) = { id }
First("=") = { "=" } First("=") = { "=" }
First(n) = { n } First(n) = { n }
First("{")= { "{" } First("{")= { "{" }
First("}")= { "}" } First("}")= { "}" }
First(";")= { ";" } First(";")= { ";" }
First(E) = { id, "{" } First(E) = { id, "{", ε }
First(L) = { id, "{", ε } First(L) = { id, "{", ";" }
CMSC 330 - Spring 2011 8
Recursive Descent Parser Implementation
! For terminals, create function parse_a
• If lookahead is a then parse_a consumes the lookahead
by advancing to the next token and then returns
• Otherwise fails with a parse error if lookahead is not a
! For each nonterminal N, create a function parse_N
• Called when we’re trying to parse a part of the input
which corresponds to (or can be derived from) N
• parse_S for the start symbol S begins the parse
CMSC 330 - Spring 2011 9
Parser Implementation (cont.)
! The body of parse_N for a nonterminal N does
the following
• Let N → β1 | ... | βk be the productions of N
Ø Here βi is the entire right side of a production- a sequence of
terminals and nonterminals
• Pick the production N → βi such that the lookahead is
in First(βi)
Ø It must be that First(βi) ∩ First(βj) = ∅ for i ≠ j
Ø If there is no such production, but N → ε then return
Ø Otherwise fail with a parse error
• Suppose βi = α1 α2 ... αn. Then call parse_α1(); ... ;
parse_αn() to match the expected right-hand side,
and return
CMSC 330 - Spring 2011 10
Recursive Descent Parser
! Given grammar S → xyz | abc
• First(xyz) = { x }, First(abc) = { a }
! Parser
parse_S( ) {
if (lookahead == “x”) {
parse_x; parse_y; parse_z); // S → xyz
}
else if (lookahead == “a”) {
parse_a; parse_b; parse_c; // S → abc
}
else error( );
}
CMSC 330 - Spring 2011 11
Recursive Descent Parser
! Given grammar S → A | B A→x |y B
→z
• First(A) = { x, y }, First(B) = { z }
parse_A( ) {
! Parser if (lookahead == “x”)
parse_S( ) { parse_x(); // A → x
if ((lookahead == “x”) || else if (lookahead == “y”)
(lookahead == “y”)) parse_y(); // A → y
parse_A( ); // S → A else error( );
}
else if (lookahead == “z”) parse_B( ) {
parse_B( ); // S → B if (lookahead == “z”)
else error( ); parse_z(); // B → z
} else error( );
}
CMSC 330 - Spring 2011 12
Example
E → id = n | { L } First(E) = { id, "{" }
L→E;L|ε
parse_E( ) { parse_L( ) {
if (lookahead == “id”) { if ((lookahead == “id”) ||
parse_id(); (lookahead == “{”)) {
parse_=(); // E → id = n parse_E( );
parse_n(); parse_; (); // L → E ; L
} parse_L( );
else if (lookahead == “{“) { }
parse_{ (); else ; // L → ε
parse_L( ); // E → { L } }
parse_} ();
}
else error( );
}
CMSC 330 - Spring 2011 13
Things to Notice
! If you draw the execution trace of the parser
• You get the parse tree
! Examples
• Grammar • Grammar
S → xyz S→A|B
S → abc A→x |y
• String “xyz” B→z
parse_S( ) • String “x” S
S
parse_x() parse_S( ) |
/|\
parse_y()
x y z
parse_A( ) A
parse_z() parse_x |
x
CMSC 330 - Spring 2011 14
Things to Notice (cont.)
! This is a predictive parser
• Because the lookahead determines exactly which
production to use
! This parsing strategy may fail on some grammars
• Possible infinite recursion
• Production First sets overlap
• Production First sets contain ε
! Does not mean grammar is not usable
• Just means this parsing method not powerful enough
• May be able to change grammar
CMSC 330 - Spring 2011 15
Left Factoring
! Consider parsing the grammar E → ab | ac
• First(ab) = a
• First(ac) = a
• Parser cannot choose between RHS based on
lookahead!
! Parser fails whenever A → α1 | α2 and
• First(α1) ∩ First(α2) != ε or ∅
! Solution
• Rewrite grammar using left factoring
CMSC 330 - Spring 2011 16
Left Factoring Algorithm
! Given grammar
• A → xα1 | xα2 | … | xαn | β
! Rewrite grammar as
• A → xL | β
• L → α1 | α2 | … | αn
! Repeat as necessary
! Examples
• S → ab | ac ⇨ S → aL L→b|c
• S → abcA | abB | a ⇨ S → aL L → bcA | bB | ε
• L → bcA | bB | ε ⇨ L → bL’ | ε L’ → cA |
B
CMSC 330 - Spring 2011 17
Left Recursion
! Consider grammar S → Sa | ε
• First(Sa) = a, so we’re ok as far as which production
• Try writing parser parse_S( ) {
if (lookahead == “a”) {
parse_S( );
parse_a (); // S → Sa
}
else { }
}
• Body of parse_S( ) has an infinite loop
Ø if (lookahead = "a") then parse_S( )
• Infinite loop occurs in grammar with left recursion
CMSC 330 - Spring 2011 18
Right Recursion
! Consider grammar S → aS | ε
• Again, First(aS) = a
• Try writing parser parse_S( ) {
if (lookahead == “a”) {
parse_a();
parse_S( ); // S → aS
}
else { }
}
• Will parse_S( ) infinite loop?
Ø Invoking parse_tok( ) will advance lookahead, eventually stop
• Top down parsers handles grammar w/ right recursion
CMSC 330 - Spring 2011 19
Expression Grammar for Top-Down Parsing
E → T E'
E' → ε | + E
T → P T'
T' → ε | * T
P→n | (E)
• Notice we can always decide what production to
choose with only one symbol of lookahead
CMSC 330 - Spring 2011 20
Tradeoffs with Other Approaches
! Recursive descent parsers are easy to write
• The formal definition is a little clunky, but if you follow
the code then it’s almost what you might have done
if you weren't told about grammars formally
• They're unable to handle certain kinds of grammars
! Recursive descent is good for a simple parser
• Though tools can be fast if you’re familiar with them
! Can implement top-down predictive parsing as a
table-driven parser
• By maintaining an explicit stack to track progress
CMSC 330 - Spring 2011 21
Tradeoffs with Other Approaches
! More powerful techniques need tool support
• Can take time to learn tools
! Main alternative is bottom-up, shift-reduce parser
• Replaces RHS of production with LHS (nonterminal)
• Example grammar
Ø S → aA, A → Bc, B → b
• Example parse
Ø abc ⇒ aBc ⇒ aA ⇒ S
Ø Derivation happens in reverse
• Something to look forward to in CMSC 430
CMSC 330 - Spring 2011 22
What’s Wrong With Parse Trees?
! Parse trees contain too much information
• Example
Ø Parentheses
Ø Extra nonterminals for precedence
• This extra stuff is needed for parsing
! But when we want to reason about languages
• Extra information gets in the way (too much detail)
CMSC 330 - Spring 2011 23
Abstract Syntax Trees (ASTs)
! An abstract syntax tree is a more compact,
abstract representation of a parse tree, with only
the essential parts
parse
AST
tree
CMSC 330 - Spring 2011 24
Abstract Syntax Trees (cont.)
! Intuitively, ASTs correspond to the data structure
you’d use to represent strings in the language
• Note that grammars describe trees
• So do OCaml datatypes (which we’ll see later)
• E → a | b | c | E+E | E-E | E*E | (E)
CMSC 330 - Spring 2011 25
