DAYANANDA SAGAR

UNIVERSITY

DEPARTMENT OF CSE

Syllabus:
Introduction: Language Processors, Structure of compiler, The science of
building a compiler, Applications of compiler technology. LEXICAL AND
SYNTAXANALYSIS: Role of lexical Analyzer, Specification of Tokens, Lexical
Analyzer generator Lex. SYNTAX ANALYSIS I: Role of Parser, Syntax error
handling, Error recovery strategies, Writing a grammar: Lexical vs Syntactic
Analysis, Eliminating ambiguity, Left recursion, Left factoring.
 Language Processors:Compilers

• Compiler is a program that can read a programming

in one language –source language and translate it
into an equivalent program in another language –
target language
• •Compilers is to report any errors in the source
program that it detects during the translation process.
 Language Processors: Compilers
• Target program is executable Machine language
program
• It will be loaded and executed with the help of loader
to process inputs and produce outputs.
Language Processors: Interpreter

An interpreter is another common kind of language

processor.
An interpreter directly executes the operations specified
in the source program on inputs supplied by the user.
It executes the source program statement by statement.
Difference b/w compiler & interpreter
Compiler is a program which translate the –source program into an
target program.Target program is executable Machine language
program, it will be loaded and executed with the help of loader to
process inputs and produce outputs. The machine language target
program produced by a compiler is usually much faster than an
interpreter at mapping inputs to outputs.
An interpreter directly executes the operations specified in the source
program on inputs supplied by the user. It executes the source program
statement by statement. An interpreter can usually give better error
diagnostics than a compiler because it executes the source program
statement by statement.
 Ex : Java Language
processor
• Java language processor combines both compilation as
well as interpretation.
• •A Java source program may first be compiled into
intermediate form called bytecodes.
• The bytecodes are then interpreted by the virtual machine
• bytecodes compiled on one machine can be interpreted on
another machine.(across a network)
• To achieve faster processing of inputs to outputs some
Java compilers called “Just in time compilers” translate
the bytecodes into machine language immediately before
they run the intermediate program to process the input.
Ex : Java Language
processor
 Language Processing System
• A source program may be divided into modules
stored in separate les. The task of collecting the
source program is sometimes entrusted to a separate
program, called a preprocessor.
• The preprocessor may also expand short hands,
called macros, into source language statements. The
modified source program is then fed to a compiler.
• The compiler may produce an assembly-language
program as its output, because assembly language is
easier to produce as output and is easier to debug .
 Language Processing System
• The assembly language is then processed by a
program called an assembler that produces
relocatable machine code as its output.
• Large programs are often compiled in pieces, so the
relocatable machine code may have to be linked
together with other relocatable object files and library
files into the code that actually runs on the machine.
The linker resolves external memory addresses,
where the code in one le may refer to a location in
another file.
• The loader then puts together all of the executable
object files into memory for execution.
Language Processing System
Phases of Compiler
 Phases of Compiler

1. Lexical Analyzer
• It is also called a scanner.
• It takes the output of the preprocessor (which performs file
inclusion and macro expansion) as the input which is in a pure
high-level language.
• It reads the characters from the source program and groups them
into lexemes (sequence of characters that “go together”).
• Each lexeme corresponds to a token.
• Tokens are defined by regular expressions which are understood by
the lexical analyzer.
• It also removes lexical errors (e.g., erroneous characters),
comments, and white space .
 Phases of Compiler
2. Syntax Analyzer:
• It is sometimes called a parser.
• It constructs the parse tree.
• It takes all the tokens one by one and uses Context-Free Grammar to
construct the parse tree.
• Why Grammar?
The rules of programming can be entirely represented in a few
productions. Using these productions we can represent what the
program actually is. The input has to be checked whether it is in the
desired format or not.
• In this phase, token arrangements are checked against the source
code grammar, i.e. the parser checks if the expression made by the
tokens is syntactically correct.
 Phases of Compiler

3. Semantic Analyzer
• Semantic analysis checks whether the parse tree constructed follows the rules
of language.
• For example, assignment of values is between compatible data types, and
adding string to an integer.
• Also, the semantic analyzer keeps track of identifiers, their types and
expressions; whether identifiers are declared before use or not etc.
• The semantic analyzer produces an annotated syntax tree as an output.

4. Intermediate Code Generator

• After semantic analysis the compiler generates an intermediate code of the
source code for the target machine.
• It represents a program for some abstract machine.
• It is in between the high-level language and the machine language.
• This intermediate code should be generated in such a way that it makes it
easier to be translated into the target machine code.
 Phases of Compiler
5.Code Optimization
• The next phase does code optimization of the intermediate code.
• Optimization can be assumed as something that removes
unnecessary code lines, and arranges the sequence of statements
in order to speed up the program execution without wasting
resources (CPU, memory).

6. Code Generation
• In this phase, the code generator takes the optimized
representation of the intermediate code and maps it to the target
machine language.
• The code generator translates the intermediate code into a
sequence of (generally) re-locatable machine code.
• Sequence of instructions of machine code performs the task as the
intermediate code would do.
 Phases of Compiler
Symbol Table
• It is a data-structure maintained throughout all the
phases of a compiler.
• All the identifier's names along with their types are
stored here.
• The symbol table makes it easier for the compiler to
quickly search the identifier record and retrieve it.
• The symbol table is also used for scope management.
The Science of building a compiler
1. Modelling in complier design and
implementation
2. The science of code optimization
 1. Modelling in complier design and
implementation
• The study of compilers is mainly a study of how we design the right
mathematical models and choose the right algorithms, while
balancing the need for generality and power against simplicity and
efficiency.

• Some of most fundamental models are finite-state machines and

regular expressions. These models are useful for describing the lexical
units of programs (keywords, identifiers, and such) and for describing
the algorithms used by the compiler to recognize those units.
• Also among the most fundamental models are context-free grammars,
used to describe the syntactic structure of programming languages such
as the nesting of parentheses or control constructs.
• Similarly, trees are an important model for representing the structure of
programs and their translation into object code.
Applications of Compiler Technology
Applications of Compiler Technology
Applications of Compiler Technology
Applications of Compiler Technology
Applications of Compiler Technology
Applications of Compiler Technology
Applications of Compiler Technology
Applications of Compiler Technology
Module 3- Chapter 2

Lexical Analysis
 Chapter 2: Outline
• Role of Lexical analyzer
• Specifications of tokens
• Lexical analyzer generator Lex
 Lexical analysis vs Parsing
Lexical analysis Parsing
A Scanner simply turns an input String A parser converts this list of tokens
(say a file) into a list of tokens. into a Tree-like object to represent how
These tokens represent things like the tokens fit together to form a
identifiers, parentheses, operators etc. cohesive whole
(sometimes referred to as a sentence).

The lexical analyzer (the "lexer") A parser does not give the nodes
parses individual symbols from the any meaning beyond structural
source code file into tokens. From cohesion. The next thing to do is
there, the "parser" proper turns those extract meaning from this structure
whole tokens into sentences of your (sometimes called contextual
grammar analysis).
 TOKEN LEXEME PATTERN:
Token: Token is a sequence of characters that can be treated as
a
single logical entity. Typical tokens are,
1) Identifiers 2) keywords 3) operators 4) special symbols
5)constants
Pattern: A set of strings in the input for which the same token is
produced as output. This set of strings is described by a rule called a
pattern associated with the token.

Lexeme: A lexeme is a sequence of characters in the source

program that is matched by the pattern for a token.
 Token Lexeme
int MAX(int a, int b)
{
if(a>b)
return a;
else
return b; Lexeme Token
} int KEYWORD

MAX IDENTIFIER
( OPERATOR
a IDENTIFIER
> OPERATOR
. .
. .
. .
 LEXICAL ERRORS
• It is hard for a lexical analyzer to tell, without the aid of other
components, that there is a source-code error. For instance, if the
string fi is encountered for the first time in a C program in the
context:
• fi ( a == f(x)) ...
• a lexical analyzer cannot tell whether fi is a misspelling of the
keyword if or an undeclared function identifier. Since fi is a valid
lexeme for the token id, the lexical analyzer must return the token id
to the parser and let some other phase of the compiler | probably the
parser in this case | handle an error due to transposition of the letters.
• Suppose a situation arises in which the lexical analyzer is unable to
proceed because none of the patterns for tokens matches any prefix
of the remaining input.
 Error recovery strategies
The simplest recovery strategy is panic mode" recovery.
We delete successive characters from the remaining input, until
the lexical analyzer can find a well-formed token at the
beginning of what input is left. This recovery technique may
confuse the parser, but in an interactive computing environment
it may be quite adequate.
Other possible error-recovery actions are:
1. Delete one character from the remaining input.
2. Insert a missing character into the remaining input.
3. Replace a character by another character.
4. Transpose two adjacent characters
Creating Lexical analyzer with lex
 Structure of Lex Program
• A Lex program (the .l file ) consists of three parts:

%{
Declarations
%}

%%
translation rules
%%
auxiliary procedures
 Structure of Lex Program
1. The declarations section includes declarations of variables, manifest
constants(A manifest constant is an identifier that is declared to
represent a constant e.g. # define PIE 3.14), and regular definitions.
2. The translation rules of a Lex program are statements of the form :
• p1 {action 1}
• p2 {action 2}
• p3 {action 3
• where each p is a regular expression and each action is a program
fragment describing what action the lexical analyzer should take
when a pattern p matches a lexeme. In Lex the actions are written in
C.
3. The third section holds whatever auxiliary procedures are
needed by the actions. Alternatively these procedures can be
compiled separately and loaded with the lexical analyzer.
 Lex program to count vowels and consonants
%{
int vow=0;
int cons=0;
%}
%%
[aeiouAEIOU] {vow++;}
[a-zA-Z] {cons++;}
%%
int main()
{
printf("Enter the string \n");
yylex();
printf("THe number of vowel are %d\n",vow);
printf("The number of consonant is %d",cons);
return 0;
}
 Built in Variables
yyin Of the type File*. This point to the current file being parsed by
the lexer. It is standard input file that stores input source
program
yyout Of the type File*. This point to location where the output od the
lexer will be written.

yytext The text of the matched pattern is stored in the variable (char*).
i,e when lexer matches or recognizes the token from input token
the lexeme stored in null terminated string called yytext.

yyleng Gives the length of the matched pattern. The value in the yyleng
is same as strlen () functions

yylineno Provides the current line number information

yylval This is a global variable used to store the value of any token
 Built in Functions
yylex() This is a starting point of lex program from which scanning of
source program starts

yywrap() This function is called when end of file is encountered. If yywrap

returns 0 the scanner continues scanning, if it returns 1 the
scanner does not return tokens.
yyless(int n) This function can be used to push back all first n characters of
the token being read

yymore() This function tells lexer to attach the next token to current token

yyerror() For displaying error messages this function is used.

 Conflict resolution in Lex
Two rules that Lex uses to decide on the proper
lexeme to select, when several prefixes of the
input match one or more patterns:
1. Always prefer a longer prefix to a shorter
prefix.
2. If the longest possible prefix matches two or
more patterns, prefer the pattern listed first in
the Lex program.
 Lookahead Operator
• Lookahead operator is the additional operator that is read by lex in
order to distinguish additional pattern for a token.
• Lexical analyzer is used to read one character ahead of valid lexeme
and then retracts to produce token.
• At times, it is needed to have certain characters at the end of input to
match with a pattern. In such cases, slash (/) is used to indicate end of
part of pattern that matches the lexeme.
• (eg.) In some languages keywords are not reserved. So the statements
IF (I, J) = 5 and IF(condition) THEN
• results in conflict whether to produce IF as an array name or a
keyword. To resolve this the lex rule for keyword IF can be written as,
• IF/\ (.* \) {
• letter }
 Functions of Lexical Analyzer
• 1. It produces stream of tokens.
• 2. It eliminates blank and comments
• 3. It generates symbol table which stores info about
identifiers, constants encountered in the input
• 4. It reports the error encountered while generating
the tokens
It works in two phases.
In first phase it performs scan in the second phase it
generates series of tokens.
 YACC
• YACC stands for Yet Another Compiler
Compiler.
• YACC provides a tool to produce a parser for a
given grammar.
• YACC is a program designed to compile a LALR
(1) grammar.
• It is used to produce the source code of the
syntactic analyzer of the language produced
by LALR (1) grammar.
• The input of YACC is the rule or grammar and
the output is a C program
Construction of translation using
YACC
 YACC
• The input of YACC is the rule or grammar, and the
output is a C program. If we have a file translate.y that
consists of YACC specification, then the UNIX system
command is:
• YACC translate.y
• This command converts the file translate.y into a C
file y.tab.c. It represents an LALR parser prepared in C
with some other user’s prepared C routines. By
compiling y.tab.c along with the lfl library, we will get
the desired object program a.out that performs the
operation defined by the original YACC program.
 DECLARATIONS SECTION
•Declarations of tokens.
• Yacc requires token names to be declared as such
using the keyword %token.
•Declaration of the start symbol using the
keyword %start
•C declarations: included files, global variables,
types.
DEFINITIONS SECTION
Module 3- Chapter 3

Syntax Analysis
 Syntax Analysis
• Second phase in compilation.
• It basically checks for the syntax of the language.
• It takes the tokens from the lexical analyzer and
groups them in such a way that some
programming syntax can be recognized.
• The syntactic error will be generated , if syntax
cannot be recognized.
• This overall process is called syntax checking of
language.
Parser
Need and Role of Parser
 Need and Role of Parser
• In the process of compilation the parser and
lexical analyzer work together.
• That mean when parser require string of
tokens it involves lexical analyzer.
• In turn the lexical analyzer supplies tokens to
syntax analyzer.
• The parser collects sufficient number of
tokens and builds a parse tree.
 Need and Role of Parser
• By building parse tree , parser smartly finds
the syntactical errors if any.
• Parser should recover from the commonly
occurring errors.
 Why Lexical and Syntax Analyzer are
separated out?
• The lexical analyzer scans the input program and
collects the tokens from it.
• Parser then builds a parse tree using tokens.
• Two independent activities are carried out by two
phases.
• Separating out of these two phases has two
advantages.
a. It accelerates the process of compilation.
b. The errors in the source input can be identified
precisely.
 Basic Issues in Parsing
• Three important issues
a.Specification of Syntax
b.Representation of input after parsing
c. Parsing algorithms
 Specification of syntax

• Specification of syntax means how to write

any programming statement.
• There are certain characteristics of
specification of syntax.
 Representation of input after parsing
• This is important because all the subsequent
phases of compiler takes the information from
the parse tree being generated.
• The information suggested by any input
programming statement should not be
differed after building the syntax tree for it.
 Parsing algorithms

• How these algorithms works?

• Are they efficient in nature?
• Kind of input they require.
• Merits and limitations of algorithms.
 Programming errors in phases
1. Lexical errors include misspellings of identifiers,
keywords, or operators |e.g., the use of an identifier
elipseSize instead of ellipseSize | and missing quotes
around text intended as a string.
2. Syntactic errors include misplaced semicolons or extra
or missing braces; that is, \{" or \}." As another example,
in C or Java, the appearance of a case statement without
an enclosing switch is a syntactic error (however, this
situation is usually allowed by the parser and caught
later in the processing, as the compiler attempts to
generate code).
 Programming errors in phases
3.Semantic errors include type mismatches between
operators and operands, e.g., the return of a value in a
Java method with result type void.
4. Logical errors can be anything from incorrect
reasoning on the part of the programmer to the use in a
C program of the assignment operator =instead of the
comparison operator ==. The program containing =
maybe well formed; however, it may not re ect the
programmer's intent.
 Error Handler in Parser
The error handler in a parser has goals that are
simple to state but challenging to realize:
1. Report the presence of errors clearly and
accurately.
2. Recover from each error quickly enough to
detect subsequent errors.
3.Add minimal overhead to the processing of
correct programs
 Error recovery strategies in
Parser(syntactic phase recovery)

1.Panic Mode Recovery

2.Phrase level Recovery
3.Global correction
4.Error productions
 1. Panic mode recovery
• With this method, on discovering an error, the parser
discards input symbols one at a time until one of a
designated set of synchronizing tokens is found.
• The synchronizing tokens are usually delimiters, such as
semicolon or }, whose role in the source program is clear
and unambiguous. The compiler designer must select the
synchronizing tokens appropriate for the source language.
While panic-mode correction often skips a considerable
amount of input without checking it for additional errors, it
has the advantage of simplicity, and, unlike some methods to
be considered later, is guaranteed not to go into an infinite
loop.
 2. Phrase level recovery
• On discovering an error, a parser may perform local correction on the
remaining input; that is, it may replace a prefix of the remaining input by
some string that allows the parser to continue.
• A typical local correction is to replace a comma by a semicolon, delete
an extraneous semicolon, or insert a missing semicolon.
• The choice of the local correction is left to the compiler designer. Of course,
we must be careful to choose replacements that do not lead to infinite loops,
as would be the case, for example, if we always inserted something on the
input ahead of the current input symbol.
• Phrase-level replacement has been used in several error-repairing
compilers, as it can correct any input string. Its major drawback is the
difficulty it has incoping with situations in which the actual error has
occurred before the point of detection.
 3.Error productions
• By anticipating common errors that might be
encountered, we can augment the grammar for the
language at hand with productions that generate the
erroneous constructs.
• A parser constructed from a grammar augmented by
these error productions detects the anticipated errors
when an error production is used during parsing.
• The parser can then generate appropriate error
diagnostics about the erroneous construct that has
been recognized in the input.
 4. Global correction
• The parser examines the whole
program and tries to find out the
closest match for it which is error-
free.
• The closest match program has less
number of insertions, deletions, and
changes of tokens to recover from
erroneous input.
• Due to high time and space
complexity, this method is not
Context Free Grammar
 Ambiguous Grammar
A grammar is said to be ambiguous if it generates more than one parse tree
for sentence of language L(G)

1. Consider the grammar

EE+E |E *E|id
Check for ambiguous or not
 Ambiguous Grammar
A grammar is said to be ambiguous if it generates more than one parse tree
for sentence of language L(G)
• 1.Show that the following grammar is
ambiguous.
S-aSbS
S bSaS
S ε
Types of Parser
 Problems with Top Down Parsing
• 1.Backtracking
• 2.Left Recursion
• 3.Left Factoring
 Backtracking
• Backtracking is a technique in which for
expansion of non-terminal symbol, we choose
one alternative and if some mismatch occurs
then we try another alternative if any,
 Backtracking
• if for a non-terminal there are multiple production
rules beginning with the same input symbol then to
get the correct derivation we need to try all these
alternatives.
• Secondly, in backtracking we need to move some
levels upward in order to check the possibilities.
This increases lot of overhead in implementation of
parsing. And hence it becomes necessary to
eliminate the backtracking by modifying the
grammar.
1. Construct parse tree for the input string w=cad using top down parser.
ScAd
Aab|a
 Left Recursive Grammar
• A grammar becomes left-recursive if it has any non-
terminal ‘A’ whose derivation contains ‘A’ itself as the left-
most symbol.
• Left-recursive grammar is considered to be a problematic
situation for top-down parsers.
• Top-down parsers start parsing from the Start symbol,
which in itself is non-terminal.
• So, when the parser encounters the same non-terminal in
its derivation, it becomes hard for it to judge when to
stop parsing the left non-terminal and it goes into an
infinite loop.
Left Recursive Grammar
 2.Consider the following grammar
AABd l Aa l a
BBe l b
Remove Left Recursion

Solution:
PR1: AABd
PR2: AAa
PR3: Aa
PR4: BBe
PR5: Bb
 Removal of Left Recursion
• Combining PR1 and PR3
A ABd l a
It is of the form AAα l β
where
A A
α Bd
βa
 Removal of Left Recursion
Rewrite the production rule as
AβA’
A’ αA’
A’ ε

• AaA’
• A’BdA’
• A’ε
 Removal of Left Recursion
• Combining PR2 and PR3
A Aa l a
It is of the form AAα l β
where
A A
α a
βa
 Removal of Left Recursion
Rewrite the production rule as
AβA’
A’ αA’
A’ ε

• AaA’
• A’aA’
• A’ε
 Removal of Left Recursion
• Combining PR4 and PR5
B Be l b
It is of the form AAα l β
where
A B
α e
βb
 Removal of Left Recursion
Rewrite the production rule as
AβA’
A’ αA’
A’ ε

• BbB’
• B’eB’
• B’ε
• The grammar without left recursion is as
follows:
• AaA’
• A’BdA’ l aA’
• A’ε
• BbB’
• B’eB’
• B’ε
 2.Consider the following grammar
AAc l Aad l bd l c
Remove Left Recursion

Solution:
PR1: AAc
PR2: AAad
PR3: Abd
PR4: Ac
 Removal of Left Recursion
• Combining PR1 and PR3
A Ac l bd
It is of the form AAα l β
where
A A
α c
βbd
 Removal of Left Recursion
Rewrite the production rule as
AβA’
A’ αA’
A’ ε

• AbdA’
• A’cA’
• A’ε
 Removal of Left Recursion
• Combining PR2 and PR4
AAad l c
It is of the form Aα l β
where
A A
α ad
βc
 Removal of Left Recursion
Rewrite the production rule as
AβA’
A’ αA’
A’ ε

• AcA’
• A’adA’
• A’ε
 Removal of Left Recursion
Grammar after eliminating left recursion

• AbdA’ l cA’
• A’cA’ l adA’
• A’ ε
Left Factoring
Left Factoring
Left Factoring
Do left factoring for the following grammar

1) SiEtS |iEtSeS | a
Eb
Solution:
PR1:SiEtS
PR2:SiEtSeS
PR3:Sa
PR4:E->b
 Left factoring
Combining production rule SiEtS | iEtSeS
is of the form Aαβ1 | αβ2| αβ3….

Applying left factoring

AαA’
A’β1 |β2|β3
For the above grammar
• A S
• αiEtS
• β1 ε
• β 2eS

Hence
SiEtSS’ (AαA’)

S’ ε |eS (A’β1 |β2)

Eb

Left Factored Grammar is

SiEtSS’ | a
S’ ε |eS
Eb
Do left factoring for the following grammar

2) AaAB |aA | a
BbB | b
Solution:
PR1:AaAB
PR2:AaA
PR3:Aa
PR4:BbB
PR5:Bb
 Left factoring
Combining production rule PR1,PR2,PR3 AaAB
| aA |a
is of the form Aαβ1 | αβ2|β3

Applying left factoring

AαA’
A’β1 |β2|β3
For the above grammar
• A A
• αa
• β1 AB
• β 2 A
• β 3 ε

Hence
AaA’ (AαA’)

A’ AB | A | ε (A’β1 |β2 |β3)

• A’ AB | A | ε
For the above grammar
• A A’
• αA
• β1 B
• β 2 ε

• A’AA”
• A’’B| ε
Where A’=S
– We have
– SAS’
– S’B|ε
 Left factoring
Combining production rule BbB | b
is of the form Aαβ1 | αβ2

Applying left factoring

AαA’
A’β1 |β2
For the above grammar
• A B
• αb
• β1 B
• β 2 ε

Hence
BbB’ (AαA’)

B’ B | ε (A’β1 |β2 )

 Left Factoring
• AaA’
• SAS’ |ε
• S’B|ε
• BbB’
• B’ B | ε
 Left Factoring
3)Do left factoring in the following grammar-
S → a | ab | abc | abcd
Solution:
PR1: S → a
PR2:S ab
PR3:S abc
PR4:Sabcd
 Left factoring
Combining production rule S → a | ab | abc | abcd
is of the form Aαβ1 | αβ2| αβ3| αβ4

Applying left factoring

AαA’
A’β1 |β2| β3| β4
For the above grammar
• A S
• αa
• β1 ε
• β 2 b
• β 3 bc
• β 3 bcd

Hence
SaS’ (AαA’)

S’ ε | b | bc | bcd (A’β1 |β2 |β3 |β4 )

 Left factoring
Combining production rule
S’ → b | bc | bcd
is of the form Aαβ1 | αβ2| αβ3

Applying left factoring

AαA’
A’β1 |β2| β3
For the above grammar
• A S’
• αb
• β1 ε
• β 2c
• β 3cd

Hence
S’bA (AαA’) here S’’=A

A ε | c | cd (A’β1 |β2 |β3 )

 Left Factoring
• Hence Left factored grammar is
• SaS’
• S’bA
• A ε | c | cd