Lexical Analyzer in Perspective: Parser Source Program Token

Lexical Analyzer in Perspective
token
source lexical
analyzer parser
program
get next
token
symbol
table
Important Issue:
What are Responsibilities of each Box ?
Focus on Lexical Analyzer and Parser
1
Why to separate Lexical analysis and parsing
o Simplicity of design
o Improving compiler efficiency
o Enhancing compiler portability
2
Tokens, Patterns, and Lexemes
o A token is a pair a token name and an optional token
attribute
o A pattern is a description of the form that the

lexemes of a token may take
o A lexeme is a sequence of characters in the source

program that matches the pattern for a token
3
Example
Token Informal description Sample lexemes

if Characters i, f if
else Characters e, l, s, e else
relation < or > or <= or >= or == or != <=, !=
id Letter followed by letter and digits pi, score, D2

number Any numeric constant 3.14159, 0, 6.02e23
literal Anything but sorrounded by core dumped
4
Using Buffer to Enhance Efficiency
Current token
E = M * C * * 2 eof
lexeme beginning forward (scans

ahead to find
pattern match)
if forward at end of first half then begin
reload second half ; Block I/O
forward : = forward + 1
end
else if forward at end of second half then begin
reload first half ; Block I/O
move forward to biginning of first half
end
else forward : = forward + 1 ;
5
Algorithm: Buffered I/O with Sentinels
Current token
E = M * eof C * * 2 eof eof
lexeme beginning forward (scans

forward : = forward + 1 ;
ahead to find
pattern match)
if forward is at eof then begin
if forward at end of first half then begin
reload second half ; Block I/O
forward : = forward + 1
end
else if forward at end of second half then begin
reload first half ; Block I/O
move forward to biginning of first half
end
else / * eof within buffer signifying end of input * /
terminate lexical analysis
end 2nd eof no more input ! 6
Chomsky Hierarchy
0 Unrestricted A
1 Context-Sensitive | LHS | | RHS |
2 Context-Free |LHS | = 1
3 Regular |RHS| = 1 or 2 ,
A a | aB, or
A a | Ba
7
Formal Language Operations
OPERATION DEFINITION
union of L and M L M = {s | s is in L or s is in M}
written L M
concatenation of L LM = {st | s is in L and t is in M}
and M written LM

Kleene closure of L L*= Li
written L* i 0
L* denotes zero or more concatenations of L

positive closure of L
written L+ L+= L
i 1
i
L+ denotes one or more concatenations of L
8
Formal Language Operations
Examples
L = {A, B, C, D } D = {1, 2, 3}
L D = {A, B, C, D, 1, 2, 3 }
LD = {A1, A2, A3, B1, B2, B3, C1, C2, C3, D1, D2, D3 }
L2 = { AA, AB, AC, AD, BA, BB, BC, BD, CA, DD}
L4 = L2 L2 = ??
L* = { All possible strings of L plus }
L+ = L* -
L (L D ) = ??
L (L D )* = ??
9
Language & Regular Expressions
A Regular Expression is a Set of Rules /
Techniques for Constructing Sequences of Symbols
(Strings) From an Alphabet.
Let Be an Alphabet, r a Regular Expression

Then L(r) is the Language That is Characterized
by the Rules of r
10
Rules for Specifying Regular Expressions:
fix alphabet
is a regular expression denoting {}
If a is in , a is a regular expression that denotes {a}
Let r and s be regular expressions with languages L(r)

and L(s). Then
p
(a) (r) | (s) is a regular expression L(r) L(s)
r
e
c
(b) (r)(s) is a regular expression L(r) L(s)
e
d (c) (r)* is a regular expression (L(r))*
e
n
c (d) (r) is a regular expression L(r)
e
All are Left-Associative. Parentheses are dropped as
allowed by precedence rules. 11
EXAMPLES of Regular Expressions
L = {A, B, C, D } D = {1, 2, 3}
A|B|C|D =L
(A | B | C | D ) (A | B | C | D ) = L2
(A | B | C | D )* = L*
(A | B | C | D ) ((A | B | C | D ) | ( 1 | 2 | 3 )) = L (L D)
12
Algebraic Properties of
Regular Expressions
AXIOM DESCRIPTION
r|s=s|r | is commutative
r | (s | t) = (r | s) | t | is associative
(r s) t = r (s t) concatenation is associative
r(s|t)=rs|rt
(s|t)r=sr|tr concatenation distributes over |
r = r
r = r Is the identity element for concatenation
r* = ( r | )* relation between * and

r** = r* * is idempotent
13
Token Recognition
How can we use concepts developed so far to assist in
recognizing tokens of a source language ?
Assume Following Tokens:
if, then, else, relop, id, num
Given Tokens, What are Patterns ?

Grammar:
if if stmt |if expr then stmt
then then |if expr then stmt else stmt
else else |
expr term relop term | term
relop < | <= | > | >= | = | <> term id | num
id letter ( letter | digit )*
num digit + (. digit + ) ? ( E(+ | -) ? digit + ) ?
14
Overall
Regular Token Attribute-Value
Expression
ws - -
if if -
then then -
else else -
id id pointer to table entry
num num pointer to table entry
< relop LT
<= relop LE
= relop EQ
<> relop NE
> relop GT
>= relop GE
Note: Each token has a unique token identifier to define category of lexemes
15
Transition diagrams
Transition diagram for relop
16
Transition diagrams (cont.)
Transition diagram for reserved words and identifiers
17
Transition diagram for unsigned numbers
18
Transition diagram for whitespace
19
Lexical Analyzer Generator - Lex
Lex Source program Lexical

lex.l lex.yy.c
Compiler
lex.yy.c
C a.out
compiler
Input stream Sequence

a.out
of tokens
20
Lexical errors
Some errors are out of power of lexical
analyzer to recognize:
fi (a == f(x))
However, it may be able to recognize errors
like:
d = 2r
Such errors are recognized when no pattern
for tokens matches a character sequence
21
Error recovery
Panic mode: successive characters are ignored

until we reach to a well formed token
Delete one character from the remaining input
Insert a missing character into the remaining
input
Replace a character by another character
Transpose two adjacent characters
Minimal Distance
22

Lexical Analyzer in Perspective: Parser Source Program Token

Uploaded by

Copyright:

Available Formats

Lexical Analyzer in Perspective: Parser Source Program Token

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lexical Analyzer in Perspective: Parser Source Program Token

Uploaded by

Copyright:

Available Formats

Lexical Analyzer in Perspective

o Improving compiler efficiency

o Enhancing compiler portability

o A pattern is a description of the form that the

o A lexeme is a sequence of characters in the source

Token Informal description Sample lexemes

id Letter followed by letter and digits pi, score, D2

lexeme beginning forward (scans

E = M * eof C * * 2 eof eof

lexeme beginning forward (scans

1 Context-Sensitive | LHS | | RHS |

L* denotes zero or more concatenations of L

L+ denotes one or more concatenations of L

Let Be an Alphabet, r a Regular Expression

Let r and s be regular expressions with languages L(r)

r* = ( r | )* relation between * and

Given Tokens, What are Patterns ?

Lex Source program Lexical

Input stream Sequence

Panic mode: successive characters are ignored

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.