Baker CS341 Packet PDF

CS 341
Dr. Baker
Fall 200?
For late-breaking information and news related to the class, see
http://www.cs.utexas.edu/users/dbaker/cs341/
Acknowledgements:
This material was assembled by Elaine Rich, who allowed me to use it wholesale. I am
greatly indebted to her for her efforts.
Table of Contents
I. Lecture Notes
A. Overview and Introduction
1.
2.
The Three Hour Tour Through Automata Theory

What is a Language?
B. Regular Languages and Finite State Machines

3.
4.
5.
6.
7.
8.
9.
10.
11.
Regular Languages
Finite State Machines
Nondeterministic Finite State Machines
Interpreters for Finite State Machines
Equivalence of Regular Languages and FSMs
Languages that Are and Are Not Regular
A Review of Equivalence Relations
State Minimization
Summary of Regular Languages and Finite State Machines
C. Context-Free Languages and Pushdown Automata

12.
13.
14.
15.
16.
17.
18.
19.
Context Free Grammars

Parse Trees
Pushdown Automata
Pushdown Automata and Context-Free Languages
Grammars and Normal Forms
Top Down Parsing
Bottom Up Parsing
Languages that Are and Are Not Context Free
D. Recursively Enumerable Languages, Turing Machines, and

Decidability
20.
21.
22.
23.
24.
Turing Machines
Computing with Turing Machines
Recursively Enumerable and Recursive Languages
Turing Machine Extensions
Problem Encoding, Turing Machine Encoding, and the Universal Turing
Machine
25. Grammars and Turing Machines
26. Undecidability
27. Introduction to Complexity Theory
II. Homework
A. Review
1.
Basic Techniques
B. Regular Languages and Finite State Machines

2.
3.
4.
5.
6.
7.
8.
9.
10.
Strings and Languages

Languages and Regular Expressions
Deterministic Finite Automata
Regular Expressions in UNIX
Nondeterministic Finite Automata
Review of Equivalence Relations
Finite Automata, Regular Expressions, and Regular Grammars
Languages that Are and Are Not Regular
State Minimization
C. Context-Free Languages and Pushdown Automata

11.
12.
13.
14.
15.
16.

Parse Trees
Pushdown Automata
Pushdown Automata and Context-Free Grammars
Parsing
Languages that Are and Are Not Context-Free
D. Recursively Enumerable Languages, Turing Machines, and

Decidability
17.
18.
19.
20.
21.
Turing Machines
Unrestricted Grammars
Undecidability
E. Review
22. Review
III.
Supplementary Materials
The Three Hour Tour through Automata Theory
Review of Mathematical Concepts
Regular Languages and Finite State Machines
Context-Free Languages and Pushdown Automata
Recursively Enumerable Languages, Turing Machines, and Decidability
I. Lecture Notes

Read Supplementary Materials: The Three Hour Tour Through Automata Theory
Read Supplementary Materials: Review of Mathematical Concepts
Read K & S Chapter 1
Do Homework 1.
Let's Look at Some Problems
int alpha, beta;
alpha = 3;
beta = (2 + 5) / 10;
(1) Lexical analysis: Scan the program and break it up into variable names, numbers, etc.
(2) Parsing: Create a tree that corresponds to the sequence of operations that should be executed, e.g.,
/
+
10
2
5
(3) Optimization: Realize that we can skip the first assignment since the value is never used and that we can precompute the
arithmetic expression, since it contains only constants.
(4) Termination: Decide whether the program is guaranteed to halt.
(5) Interpretation: Figure out what (if anything) it does.
A Framework for Analyzing Problems

We need a single framework in which we can analyze a very diverse set of problems.
The framework we will use is Language Recognition
A language is a (possibly infinite) set of finite length strings over a finite alphabet.
Languages
(1) = {0,1,2,3,4,5,6,7,8,9}
L
= {w *: w represents an odd integer}
= {w *: the last character of w is 1,3,5,7, or 9}
= (0123456789)*
(13579)
(2) = {(,)}
L
= {w *: w has matched parentheses}
= the set of strings accepted by the grammar:
S(S)
S SS
S
(3) L = {w: w is a sentence in English}
Examples:
Mary hit the ball.
Colorless green ideas sleep furiously.
The window needs fixed.
(4) L = {w: w is a C program that halts on all inputs}
Lecture Notes 1
The Three Hour Tour
Encoding Output in the Input String

(5) Encoding multiplication as a single input string
L = {w of the form: <integer>x<integer>=<integer>, where <integer> is any well formed integer, and the third integer is
the product of the first two}
12x9=108
12=12
12x8=108
(6) Encoding prime decomposition
L = {w of the form: <integer1>/<integer2>,<integer3> , where integers 2 - n represent the prime decomposition of
integer 1.
15/3,5
2/2
More Languages
(7) Sorting as a language recognition task:
L = {w1 # w2: n 1,
w1 is of the form int1, int2, intn,
w2 is of the form int1, int2, intn, and
w2 contains the same objects as w1 and w2 is sorted}
Examples:
1,5,3,9,6#1,3,5,6,9 L
1,5,3,9,6#1,2,3,4,5,6,7 L
(8) Database querying as a language recognition task:
L = {d # q # a:
d is an encoding of a database,
q is a string representing a query, and
a is the correct result of applying q to d}
Example:
(name, age, phone), (John, 23, 567-1234) (Mary, 24, 234-9876 )# (select name age=23) # (John) L
The Traditional Problems and their Language Formulations are Equivalent
By equivalent we mean:
If we have a machine to solve one, we can use it to build a machine to do the other using just the starting machine and other
functions that can be built using a machine of equal or lesser power.
Consider the multiplication example:
L = {w of the form:
<integer>x<integer>=<integer>, where
<integer> is any well formed integer, and
the third integer is the product of the first two}
Given a multiplication machine, we can build the language recognition machine:
Given the language recognition machine, we can build a multiplication machine:
Lecture Notes 1
The Three Hour Tour
A Framework for Describing Languages

Clearly, if we are going to work with languages, each one must have a finite description.
Finite Languages: Easy. Just list the elements of the language.
L = {June, July, August}
Infinite Languages: Need a finite description.
Grammars let us use recursion to do this.
Grammars 1
Grammars 2
(1) The Language of Matched Parentheses

S(S)
S SS
S
SO
SAO
A A D
AD
DO
DE
O1
O3
O5
O7
O9
E 0
E 2
E 4
E 6
E 8
(2) The Language of Odd Integers

S1
S3
S5
S7
S9
S0S
S1S
S2S
S3S
S4S
S5S
S6S
S7S
S8S
S9S
Grammars 3
(3) The Language of Simple Arithmetic Expressions
S <exp>
<exp> <number>
<exp> (<exp>)
<exp> - <exp>
<exp> <exp> <op> <exp>
<op> + | - | * | /
<number> <digit>
<number> <digit> <number>
<digit > 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
Lecture Notes 1
The Three Hour Tour
Grammars as Generators and Acceptors

Top Down Parsing
Bottom Up Parsing
The Language Hierarchy
Recursively Enumerable
Languages
Recursive
Languages
Context-Free
Languages
Regular
Languages
Lecture Notes 1
The Three Hour Tour
Regular Grammars
In a regular grammar, all rules must be of the form:
<one nonterminal> <one terminal> or
or
<one nonterminal>
<one terminal><one nonterminal>
So, the following rules are okay:

S
Sa
S aS
But these are not:
S ab
S SS
aS b
Regular Expressions and Languages
Regular expressions are formed from and the characters in the target alphabet, plus the operations of:
Concatenation: means followed by
Or (Set Union): means Or (Union)
Kleene *: * means 0 or more occurrences of concatenated together.
At Least 1: + means 1 or more occurrences of concatenated together.
(): used to group the other operators
Examples:
(1) Odd integers:
(0123456789)*(13579)
(2) Identifiers:
(A-Z)+((A-Z) (0-9))*
(3) Matched Parentheses
(1) The Language of Matched Parentheses
S(S)
S SS
S
(2) The Language of Simple Arithmetic Expressions
S <exp>
<exp> <number>
<exp> (<exp>)
<exp> - <exp>
<exp> <exp> <op> <exp>
<op> + | - | * | /
<number> <digit>
<number> <digit> <number>
<digit > 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
Lecture Notes 1
The Three Hour Tour
English:
S NP VP
NP the NP1 | NP1
NP1 ADJ NP1 | N
N boy | boys
VP V | V NP
V run | runs
What about boys runs
A much simpler example:
Not All Languages are Context-Free
anbncn, n 1
Example: A grammar to generate all strings of the form anbncn, n 1

S aBSc
S aBc
Ba aB
Bc bc
Bb bb
Languages
Recursive
Languages
Context-Free
Languages
Regular
Languages
Lecture Notes 1
The Three Hour Tour
A Machine Hierarchy
Finite State Machines 1

An FSM to accept odd integers:
1,3,5,7,9
1,3,5,7,9
0,2,4,6,8
0,2,4,6,8
An FSM to accept identifiers:
letter
letter or digit
blank, delimiter
or digit
delimiter or blank
anything
Pushdown Automata
A PDA to accept strings with balanced parentheses:

(//(
s
)/(/
Example: (())()
Stack:
Pushdown Automaton 2
A PDA to accept strings of the form w#wR:
a//a
a/a/
#//
s
b//b
Lecture Notes 1
f
b/b/
The Three Hour Tour
A Nondeterministic PDA
A PDA to accept strings of the form wwR
a//a
a/a/
//
s
b//b
b/b/
PDA 3
A PDA to accept strings of the form anbncn
Turing Machines
A Turing Machine to accept strings of the form anbncn
S
d//R
//R
a,e//R
b,f//R
a,b,e,f//L
a/d/R
b,c
b/e/R
c,d,f,
c/f/L
a,d,e,
,e,f//R
a,b,c,d
e,f//R
Lecture Notes 1
The Three Hour Tour
A Two Tape Turing Machine

A Turing Machine to accept {w#wR}
A Two Tape Turing Machine to do the same thing
Simulating k Tapes with One

A multitrack tape:
0
a
1
a
1
b
0
b
0
b
0
a
0
a
0
0
b
0
0
a
0
Can be encoded on a single tape with an alphabet consisting of symbols corresponding to :

{{,a,b,#,} x {0,1} x
{,a,b,#,} x {0,1}}
Example:
2nd square: (,0,a,1))
Simulating a Turing Machine with a PDA with Two Stacks
a
a
b
a
Lecture Notes 1
#
a
a
b
a
The Three Hour Tour
The Universal Turing Machine

Encoding States, Symbols, and Transitions
Suppose the input machine M has 5 states, 4 tape symbols, and a transition of the form:
(s,a,q,b), which can be read as:
in state s, reading an a, go to state q, and write b.
We encode this transition as:
q000,a00,q010,a01
A series of transitions that describe an entire machine will look like
q000,a00,q010,a01#q010,a00,q000,a00
a
a00a00a01
#
q000
Church's Thesis
(Church-Turing Thesis)
An algorithm is a formal procedure that halts.
The Thesis: Anything that can be computed by any algorithm can be computed by a Turing machine.
Another way to state it: All "reasonable" formal models of computation are equivalent to the Turing machine. This isn't a formal
statement, so we can't prove it. But many different computational models have been proposed and they all turn out to be
equivalent.
Example: unrestricted grammars
A Machine Hierarchy
FSMs
PDAs
Turing Machines
Lecture Notes 1
The Three Hour Tour
10
Languages and Machines
Languages
Recursive
Languages
Context-Free
Languages
Regular
Languages
FSMs
PDAs
Turing Machines
Where Does a Particular Problem Go?

Showing what it is -- generally by construction of:
A grammar, or a machine
Showing what it isn't -- generally by contradiction, using:
Counting
Example: anbn
Closure properties
Diagonalization
Reduction
Closure Properties
Regular Lanugages are Closed Under:
Union
Concatenation
Kleene closure
Complementation
Reversal
Intersection
Context Free Languages are Closed Under:
Union
Concatenation
Kleene Closure
Reversal
Intersection with regular languages
Etc.
Lecture Notes 1
The Three Hour Tour
11
Using Closure Properties

Example:
L = {anbmcp: nm or m p} is not deterministic context-free.
Two theorems we'll prove later:
Theorem 3.7.1: The class of deterministic context-free languages is closed under complement.
Theorem 3.5.2: The intersection of a context-free language with a regular language is a context-free language.
If L were a deterministic CFL, then the complement of L (L') would be a deterministic CFL.
But L' a*b*c* = {anbncn}, which we know is not context-free, much less deterministic context-free. Thus a contradiction.
Diagonalization
The power set of the integers is not countable.
Imagine that there were some enumeration:
Set 1
Set 2
Set 3
Set 4
Set 5
1
1
1
1
1
1
1
1
1
But then we could create a new set

New Set
But this new set must necessarily be different from all the other sets in the supposedly complete enumeration. Yet it should be
included. Thus a contradiction.
More on Cantor
Of course, if we're going to enumerate, we probably want to do it very systematically, e.g.,
Set 1
Set 2
Set 3
Set 4
Set 5
Set 6
Set 7
1
1
1
1
1
1
1
1
1
1
1
1
1
Read the rows as bit vectors, but read them backwards. So Set 4 is 100. Notice that this is the binary encoding of 4.
This enumeration will generate all finite sets of integers, and in fact the set of all finite sets of integers is countable.
But when will it generate the set that contains all the integers except 1?
Lecture Notes 1
The Three Hour Tour
12
The Unsolvability of the Halting Problem

Suppose we could implement
HALTS(M,x)
M: string representing a Turing Machine
x: string representing the input for M
If M(x) halts then True
else False
Then we could define
TROUBLE(x)
x: string
If HALTS(x,x) then loop forever
else halt
So now what happens if we invoke TROUBLE(TROUBLE), which invokes
HALTS(TROUBLE,TROUBLE)
If HALTS says that TROUBLE halts on itself then TROUBLE loops. IF HALTS says that TROUBLE loops, then TROUBLE
halts.
Viewing the Halting Problem as Diagonalization
First we need an enumeration of the set of all Turing Machines. We'll just use lexicographic order of the encodings we used as
inputs to the Universal Turing Machine. So now, what we claim is that HALTS can compute the following table, where 1 means
the machine halts on the input:
Machine 1
Machine 2
Machine 3
TROUBLE
Machine 5
I1
1
I2
I3
TROUBLE
I5
1
1
1
But we've defined TROUBLE so that it will actually behave as:

TROUBLE
Or maybe HALT said that TROUBLE(TROUBLE) would halt. But then TROUBLE would loop.
Lecture Notes 1
The Three Hour Tour
13
Decidability
Languages
Recursive
Languages
Context-Free
Languages
Regular
Languages
Can always say yes or no

Can enumerate from the grammar.
Can say yes by enumerating and checking
Let's Revisit Some Problems

int alpha, beta;
alpha = 3;
beta = (2 + 5) / 10;
(1) Lexical analysis: Scan the program and break it up into variable names, numbers, etc.
(2) Parsing: Create a tree that corresponds to the sequence of operations that should be executed, e.g.,
/
+
2
10
5
(3) Optimization: Realize that we can skip the first assignment since the value is never used and that we can precompute the
arithmetic expression, since it contains only constants.
(4) Termination: Decide whether the program is guaranteed to halt.
(5) Interpretation: Figure out what (if anything) useful it does.
Lecture Notes 1
The Three Hour Tour
14
So What's Left?
Formalize and Prove Things

FSMs
Nondeterminism
State minimization
Implementation
Equivalence of regular expressions and FSMs
Properties of Regular Languages
Context-Free Languages and PDAs
Equivalence of CFGs and nondeterministic PDAs
Properties of context-free languages
Parsing and determinism
Turing Machines and Computability
Recursive and recursively enumerable languages
Extensions of Turing Machines
Undecidable problems for Turing Machines and unrestricted grammars
Lecture Notes 1
The Three Hour Tour
15
What Is a Language?
Do Homework 2.
Grammars, Languages, and Machines
Language
L
Grammar
Accepts
Machine
Strings: the Building Blocks of Languages

An alphabet is a finite set of symbols:
English alphabet: {A, B, C, , Z}

Binary alphabet: {0, 1}
A string over an alphabet is a finite sequence of symbols drawn from the alphabet.
English string: happynewyear
binary string: 1001101
We will generally omit from strings unless doing so would lead to confusion.
The set of all possible strings over an alphabet is written *.
binary string: 1001101 {0,1}*
The shortest string contains no characters. It is called the empty string and is written or (epsilon).
The set of all possible strings over an alphabet is written *.
More on Strings
The length of a string is the number of symbols in it.
|| = 0
|1001101| = 7
A string a is a substring of a string b if a occurs contiguously as part of b.
aaa
is a substring of
aaabbbaaa
aaaaaa is not a substring of
aaabbbaaa
Every string is a substring (although not a proper substring) of itself.
is a substring of every string. Alternatively, we can match anywhere.
Notice the analogy with sets here.
Lecture Notes 2
What is a Language?
Operations on Strings
Concatenation: The concatenation of two strings x and y is written x || y, xy, or xy and is the string formed by appending the
string y to the string x.
|xy| = |x| + |y|
If x = and y = food, then xy =
If x = good and y = bye, then |xy| =
Note: x = x = x for all strings x.
Replication: For each string w and each natural number i, the string wi is defined recursively as
w0 =
for each i 1
wi = wi-1 w
Like exponentiation, the replication operator has a high precedence.
Examples:
a3 =
(bye)2 =
a0b3 =
String Reversal
An inductive definition:
(1) If |w| = 0 then wR = w =
(2) If |w| 1 then a : w = ua
(a is the last character of w)
and
wR = auR
Example:
(abc)R =
More on String Reversal
Theorem: If w, x are strings, then (wx)R = xRwR
Example: (dogcat)R
= (cat)R(dog)R = tacgod
Proof (by induction on |x|):

Basis: |x| = 0. Then x = , and (wx)R = (w)R = (w)R = wR = RwR = xRwR
Induction Hypothesis: If |x| n, then (wx)R = xRwR
Induction Step: Let |x| = n + 1. Then x = u a for some character a and |u| = n
(wx)R = (w (ua))R
= ((wu)a)R
= a(wu)R
= auRwR
= (ua)RwR
= xRwR
associativity
definition of reversal
induction hypothesis
definition of reversal
dogcat
w
x
u a
Lecture Notes 2
What is a Language?
Defining a Language
A language is a (finite or infinite) set of finite length strings over a finite alphabet .
Example: Let = {a, b}
Some languages over : , {}, {a, b}, {, a, aa, aaa, aaaa, aaaaa}
The language * contains an infinite number of strings, including:
L = {x {a, b}* : all a's precede all b's}
, a, b, ab, ababaaa
Example Language Definitions
L = {x : y {a, b}* : x = ya}

L = {an, n 0 }
L = an (If we say nothing about the range of n, we will assume that it is drawn from N, i.e., n 0.)
L = {x#y: x,y {0-9}* and square(x) = y}
L = {} = (the empty languagenot to be confused with {}, the language of the empty string)
Techniques for Defining Languages
Languages are sets. Recall that, for sets, it makes sense to talk about enumerations and decision procedures. So, if we want
to provide a computationally effective definition of a language we could specify either a
Language generator, which enumerates (lists) the elements of the language, or a

Language recognizer, which decides whether or not a candidate string is in the language and returns True if it is and
False if it isn't.
Example: The logical definition:

language recognizer.
L = {x : y {a, b}* : x = ya} can be turned into either a language generator or a
How Large are Languages?
The smallest language over any alphabet is .

|| = 0
The largest language over any alphabet is *.
|*| = ?
- If = then * = {} and |*| = 1
- If then |*| is countably infinite because its elements can be enumerated in 1 to 1 correspondence with the
integers as follows:
1. Enumerate all strings of length 0, then length 1, then length 2, and so forth.
2. Within the strings of a given length, enumerate them lexicographically.
E.g., aa, ab, ba, bb
So all languages are either finite or countably infinite. Alternatively, all languages are countable.
Operations on Languages 1
Normal set operations: union, intersection, difference, complement
Examples: = {a, b}
L1 = strings with an even number of a's
L2 = strings with no b's
L1 L2 =
L1 L2 =
L2 - L1 =
( L2 - L1) =
Lecture Notes 2
What is a Language?
Concatenation: (based on the definition of concatenation of strings)
If L1 and L2 are languages over , their concatenation L = L1 L2, sometimes L1L2, is
{w *: w = x y for some x L1 and y L2}
Examples:
L1 = {cat, dog}
L1 = {an: n 1}
L2 = {apple, pear}
L2 = {an: n 3}
L1 L2 = {catapple, catpear, dogapple, dogpear}

L1 L2 =
Identities:
L = L = L (analogous to multiplication by 0)
L{}= {}L = L L (analogous to multiplication by 1)
Replicated concatenation:
Ln = LLL L (n times)
L1 = L
L0 = {}
Example:
L = {dog, cat, fish}
L0 = {}
L1 = {dog, cat, fish}
L2 = {dogdog, dogcat, dogfish, catdog, catcat, catfish, fishdog, fishcat, fishfish}
Concatenating Languages Defined Using Variables

L1 = an = {an : n 0}
L1 L2 = {an : n 0}{bn : n 0} = { an bm : n,m 0}
L2 = bn = {bn : n 0}
(common mistake: ) anbn = { an bn : n 0}
Note: The scope of any variable used in an expression that invokes replication will be taken to be the entire expression.
L = 1n2m
L = anbman
Kleene Star (or Kleene closure): L* = {w * : w = w1 w2 wk for some k 0 and some w1, w2, wk L}
Alternative definition: L* = L0 L1 L2 L3
Note: L, L*
Example:
L = {dog, cat, fish}
L* = {, dog, cat, fish, dogdog, dogcat, fishcatfish, fishdogdogfishcat, }
Another useful definition: L+ = L L*
(L+ is the closure of L under concatenation)
Alternatively, L+ = L1 L2 L3
L+ = L*-{}
L+ = L*
Lecture Notes 2
if L
if L
What is a Language?
Regular Languages
Read Supplementary Materials: Regular Languages and Finite State Machines: Regular Languages
Do Homework 3.
Regular Grammars, Languages, and Machines
Regular Expression
or
Regular Grammar
Regular
Language
Accepts
Finite
State
Machine
Pure Regular Expressions

The regular expressions over an alphabet are all strings over the alphabet {(, ), , , *} that can be obtained as
follows:
1. and each member of is a regular expression.
2. If , are regular expressions, then so is
3. If , are regular expressions, then so is .
4. If is a regular expression, then so is *.
5. If is a regular expression, then so is ().
6. Nothing else is a regular expression.
If = {a,b} then these are regular expressions:
, a, bab, ab , (ab)*a*b*
So far, regular expressions are just (finite) strings over some alphabet, {(, ), , , *}.
Regular Expressions Define Languages
Regular expressions define languages via a semantic interpretation function we'll call L:
1. L() = and L(a) = {a} for each a
2. If , are regular expressions, then
L() = L()L()
= all strings that can be formed by concatenating to some string from L() some string from L().
Note that if either or is , then its language is , so there is nothing to concatenate and the result is .
3. If , are regular expressions, then L() = L() L()
4. If is a regular expression, then L(*) = L()*
5. L( () ) = L()
A language is regular if and only if it can be described by a regular expression.
A regular expression is always finite, but it may describe a (countably) infinite language.
Lecture Notes 3
Regular Languages
Regular Languages
An equivalent definition of the class of regular languages over an alphabet :
The closure of the languages
{a} a and
[1]
with respect to the functions:
concatenation,
[2]
union, and
[3]
Kleene star.
[4]
In other words, the class of regular languages is the smallest set that includes all elements of [1] and that is closed under [2],
[3], and [4].
Closure and Closed
Informally, a set can be defined in terms of a (usually small) starting set and a group of functions over elements from the set.
The functions are applied to members of the set, and if anything new arises, its added to the set. The resulting set is called
the closure over the initial set and the functions. Note that the functions(s) may only be applied a finite number of times.
Examples:
The set of natural numbers N can be defined as the closure over {0} and the successor (succ(n) = n+1) function.
Regular languages can be defined as the closure of {a} a and and the functions of concatenation, union, and
Kleene star.
We say a set is closed over a function if applying the function to arbitrary elements in the set does not yield any new elements.
Examples:
The set of natural numbers N is closed under multiplication.
Regular languages are closed under intersection.
See Supplementary MaterialsReview of Mathematical Concepts for more formal definitions of these terms.
Examples of Regular Languages
L( a*b* ) =
L( (a b) ) =
L( (a b)* ) =
L( (ab)*a*b*) =
L = {w {a,b}* : |w| is even}
L = {w {a,b}* : w contains an odd number of a's}
Augmenting Our Notation
It would be really useful to be able to write in a regular expression.
Example: (a ) b (Optional a followed by b)
But we'd also like a minimal definition of what constitutes a regular expression. Why?
Observe that
0 = {} (since 0 occurrences of the elements of any set generates the empty string), so
* = {}
So, without changing the set of languages that can be defined, we can add to our notation for regular expressions if we
specify that
L() = {}
We're essentially treating the same way that we treat the characters in the alphabet.
Having done this, you'll probably find that you rarely need in any regular expression.
Lecture Notes 3
Regular Languages
More Regular Expression Examples

L( (aa*) ) =
L( (a )* ) =
L = { w {a,b}* : there is no more than one b}
L = { w {a,b}* : no two consecutive letters are the same}
Further Notational Extensions of Regular Expressions
A fixed number of concatenations: n means (n times).

At Least 1: + means 1 or more occurrences of concatenated together.
Shorthands for denoting sets, such as ranges, e.g., (A-Z) or (letter-letter)
Example: L = (A-Z)+((A-Z)(0-9))*
A replicated regular expression n, where n is a constant.

Example: L = (0 1)20
Intersection: (well prove later that regular languages are closed under intersection)
Example: L = (a3)* (a5)*
Operator Precedence in Regular Expressions
Regular expressions are strings in the language of regular expressions. Thus to interpret them we need to:
1. Parse the string
2. Assign a meaning to the parse tree
Parsing regular expressions is a lot like parsing arithmetic expressions. To do it, we must assign precedence to the operators:
Regular
Arithmetic
Expressions
Expressions
Highest
Kleene star
exponentiation
concatenation
intersection
Lowest
multiplication
union
addition
a b* c d*
x y2 + i j2
Regular Expressions and Grammars

Recall that grammars are language generators. A grammar is a recipe for creating strings in a language.
Regular expressions are analogous to grammars, but with two special properties:
1.
2.
The have limited power. They can be used to define only regular languages.
They don't look much like other kinds of grammars, which generally are composed of sets of production rules.
But we can write more "standard" grammars to define exactly the same languages that regular expressions can define.
Specifically, any such grammar must be composed of rules that:
have a left hand side that is a single nonterminal

have a right hand side that is , or a single terminal, or a single terminal followed by a single nonterminal.
Lecture Notes 3
Regular Languages
L={w {a, b}* : |w| is even}
Regular Grammar Example
((aa) (ab) (ba) (bb))*
Notice how these rules correspond naturally to a FSM:
S
S aT
S bT
Ta
Tb
T aS
T bS
a, b
S
T
a, b
Generators and Recognizers

Generator
Recognizer
Language
Regular Languages
Regular Expressions
Regular Grammars
Lecture Notes 3
Regular Languages

Read K & S 2.1
Do Homeworks 4 & 5.
A DFSM to accept odd integers:
Definition of a Deterministic Finite State Machine (DFSM)

M = (K, , , s, F), where
K is a finite set of states

is an alphabet
s K is the initial state
F K is the set of final states, and
is the transition function. It is function from (K ) to K
i.e., each element of maps from: a state, input symbol pair to a new state.
Informally, M accepts a string w if M winds up in some state that is an element of F when it has finished reading w (if not, it
rejects w).
The language accepted by M, denoted L(M), is the set of all strings accepted by M.
Deterministic finite state machines (DFSMs) are also called deterministic finite state automata (DFSAs or DFAs).
Computations Using FSMs
A computation of A FSM is a sequence of configurations, where a configuration is any element of K *.
The yields relation |-M:
(q, w) |-M (q', w') iff
w = a w' for some symbol a , and
(q, a) = q'
(The yields relation effectively runs M one step.)
|-M * is the reflexive, transitive closure of |-M.
(The |-M* relation runs M any number of steps.)
Formally, a FSM M accepts a string w iff
(s, w) |-M * (q, ), for some q F.
An Example Computation
A DFSM to accept odd integers:
On input 235, the configurations are:
(q0, 235)
|-M
(q0, 35)
|-M
|-M
Thus (q0, 235) |-M* (q1, ). (What does this mean?)
Lecture Notes 4

A DFSM to accept $.50 in change:
More Examples
((aa) (ab) (ba) (bb))*
(b )(ab)*(a )
More Examples
L1 = {w {a, b}* : every a is immediately followed a b}
A regular expression for L1:
A DFSM for L1:
L2 = {w {a, b}* : every a has a matching b somewhere before it}

A regular expression for L2:
A DFSM for L2:
Lecture Notes 4
Another Example: Socket-based Network Communication

Client
open socket
send request
Server
= {Open, Req, Reply, Close}
send reply
L = Open (Req Reply)* (Req ) Close
send request
send reply
close socket
M=
Definition of a Deterministic Finite State Transducer (DFST)

M = (K, , O, , s, F), where
is an input alphabet
O is an output alphabet
is the transition function. It is function from
(K ) to (K O*)
i.e., each element of maps from: a state, input symbol pair
to :
a new state and zero or more output symbols (an output string)
M computes a function M(w) if, when it reads w, it outputs M(w).
Theorem: The output language of a deterministic finite state transducer (on final state) is regular.
A Simple Finite State Transducer
Convert 1's to 0's and 0's to 1's (this isn't just a finite state task -- it's a one state task)
1/0
q0
0/1
An Odd Parity Generator
After every three bits, output a fourth bit such that each group of four bits has odd parity.
Lecture Notes 4

Read K & S 2.2, 2.3
Read Supplementary Materials: Regular Languages and Finite State Machines: Proof of the Equivalence of Nondeterministic
and Deterministic FSAs.
Do Homework 6.
Definition of a Nondeterministic Finite State Machine (NDFSM/NFA)
M = (K, , , s, F), where
is an alphabet
is the transition relation. It is a finite subset of
(K ( {})) K
i.e., each element of contains:
a configuration (state, input symbol or ), and a new state.
M accepts a string w if there exists some path along which w drives M to some element of F.
The language accepted by M, denoted L(M), is the set of all strings accepted by M, where computation is defined analogously to
DFSMs.
L= {w : there is a symbol ai not appearing in w}
A Nondeterministic FSA
The idea is to guess (nondeterministically) which character will be the one that doesn't appear.
Another Nondeterministic FSA

L1= {w : aa occurs in w}
L2= {x : bb occurs in x}
L3= {y : L1 or L2 }
M1 =
10
11
a, b
12
b
b
M2=
b
20
b
21
a, b
22
a
a
M3=
Lecture Notes 5
Analyzing Nondeterministic FSAs

a
a
b
b
a
Does this FSA accept:
baaba
Remember: we just have to find one accepting path.
Nondeterministic and Deterministic FSAs
Clearly, {Languages accepted by a DFSA} {Languages accepted by a NDFSA}
(Just treat as )
More interestingly,
Theorem: For each NDFSA, there is an equivalent DFSA.
Proof: By construction
b,c
a
q1
a,c
q0
b
q2
a,b
c
q3
Another Nondeterministic Example
b* (b(a c)c b(a b) (c ))* b
a, c
2
b
c,
a,b
5
c,
b
8
Lecture Notes 5
A Real Example
See enemy
Found by enemy
Hide
Run
See sword
Coast clear
See sword
Found by enemy
Brother
kills enemy
See laser
Reach for Sword
Pick up
Laser
Sword picked up
Swing Sword
Get stabbed
Kill enemy
Die
Kill enemy
Become King
Dealing with Transitions
E(q) = {p K : (q,w) |-*M (p, w}. E(q) is the closure of {q} under the relation
An algorithm to compute E(q):
{(p,r) : there is a transition (p, , r) }
Defining the Deterministic FSA

Given a NDFSA M = (K, , , s, F),
we construct M' = (K', , ', s', F'), where
K' = 2K
s' = E(s)
F' = {Q K : Q F }
' (Q, a) = {E(p) : p K and (q, a, p)
for some q Q}
Example: computing ' for the missing letter machine
s' =
{q0, q1, q2, q3}
' = { ({q0, q1, q2, q3}, a, {q2, q3}),
({q0, q1, q2, q3}, b, {q1, q3}),
({q0, q1, q2, q3}, c, {q1, q2}),
({q1, q2}, a, {q2}), ({q1, q2}, b, {q1}), ({q1, q2}, c, {q1, q2})
({q1, q3}, a, {q3}), ({q1, q3}, b, {q1, q3}), ({q1, q3}, c, {q1})
({q2, q3}, a, {q2, q3}), ({q2, q3}, b, {q3}), ({q2, q3}, c, {q2})
({q1}, b, {q1}), ({q1}, c, {q1})
({q2}, a, {q2}), ({q2}, c, {q2})
({q3}, a, {q3}), ({q3}, b, {q3}) }
Lecture Notes 5
b,c
a
q1
a,c
q0
b
q2
a,b
c
q3
1.
2.
3.
4.
5.
An Algorithm for Constructing the Deterministic FSA

Compute the E(q)s:
Compute s' = E(s)
Compute ':
' (Q, a) = {E(p) : p K and (q, a, p) for some q Q}
Compute K' = a subset of 2K
Compute F' = {Q K' : Q F }
An Example - The Or Machine
L1= {w : aa occurs in w}
L2= {x : bb occurs in x}
L3= {y : L1 or L2 }
a
a, b
b
10
11
12
00
a, b
a
20
21
22
a
Another Example
b* (b(a c)c b(a b) (c ))* b
a, c
2
b
c,
a, b
5
c,
b
8
E(q) =
' =
Lecture Notes 5
Sometimes the Number of States Grows Exponentially
b,c
Example:
The missing letter machine, with || = n
No. of states after 0 chars: 1
No. of new states after 1 char:
=n
n 1
a,c
n
No. of new states after 2 chars:
= n(n-1)/2
n 2
No. of new states after 3 chars:
a
q1
q0
= n(n-1)(n-2)/6
n 3
b
q2
c
q3
a,b
Total number of states after n chars: 2n
What If The Original FSA is Deterministic?

M=
1,3,5,7,9
1,3,5,7,9
q0
1.
2.
3.
q1
0,2,4,6,8
4.
5.
0,2,4,6,8
Compute the E(q)s:

s' = E(q0) =
Compute '
({q0}, odd, {q1})
({q0}, even, {q0})
({q1}, odd, {q1})
({q1}, even, {q0})
K' = {{q0}, {q1}}
F' = {{q1}}
M' = M
The real meaning of determinism

A FSA is deterministic if, for each input and state, there is at most one possible transition.
DFSAs are always deterministic. Why?
NFSAs can be deterministic (even with -transitions and implicit dead states), but the formalism allows nondeterminism,
in general.
Determinism implies uniquely defined machine behavior.
Lecture Notes 5

Deterministic FSAs as Algorithms
S:
Example: No more than one b
a,b
b
S
T:
b
T
Length of Program: |K| (|| + 2)

Time required to analyze string w: O(|w| ||)
s := get-next-symbol;
if s = end-of-file then accept;
else if s = a then go to S;
else if s = b then go to T;
s:= get-next-symbol;
if s = end-of-file then accept;
else if s = a then go to T;
else if s = b then go to U;
etc.
We have to write new code for every new FSM.

Until accept or reject do:
A Deterministic FSA Interpreter
To simulate M = (K, , , s, F):
Simulate the no more than one b machine on input: aabaa
ST := s;
Repeat
i := get-next-symbol;
if i end-of-string then
ST := (ST, i)
Until i = end-of-string;
If ST F then accept else reject
Nondeterministic FSAs as Algorithms
Real computers are deterministic, so we have three choices if we want to execute a nondeterministic FSA:
1.
Convert the NDFSA to a deterministic one:

Conversion can take time and space 2K.
Time to analyze string w: O(|w|)
2.
Simulate the behavior of the nondeterministic one by constructing sets of states "on the fly" during execution
No conversion cost
Time to analyze string w: O(|w| K2)
3.
Do a depth-first search of all paths through the nondeterministic machine.
Lecture Notes 6
A Nondeterministic FSA Interpreter

To simulate M = (K, , , s, F):
SET ST;
ST := E(s);
Repeat
ST1 :=
For all q ST do
For all r (q, i) do
ST1 := ST1 E(r);
ST := ST1;
A Deterministic Finite State Transducer Interpreter

To simulate M = (K, , O, , s, F), given that:
1(state, symbol) returns a single new state
(i.e., M is deterministic), and
2(state, symbol) returns an element of O*, the
string to be output.
ST := s;
Repeat:
write(2(ST, i));
ST := 1(ST, i)
Lecture Notes 6

Read K & S 2.4
Read Supplementary Materials: Regular Languages and Finite State Machines: Generating Regular Expressions from Finite
State Machines.
Do Homework 8.
Theorem: The set of languages expressible using regular expressions (the regular languages) equals the class of languages
recognizable by finite state machines. Alternatively, a language is regular if and only if it is accepted by a finite state machine.
Proof Strategies
Possible Proof Strategies for showing that two sets, a and b are equal (also for iff):
1.
Start with a and apply valid transformation operators until b is produced.
Example:
Prove:
A (B C) = (A B) (A C)
A (B C)
= (B C) A
= (B A) (C A)
= (A B) (A C)
2.
commutativity
distributivity
commutativity
Do two separate proofs: (1) a b, and (2) b a, possibly using totally different techniques. In this case, we show first (by
construction) that for every regular expression there is a corresponding FSM. Then we show, by induction on the number of
states, that for every FSM, there is a corresponding regular expression.
For Every Regular Expression There is a Corresponding FSM
We'll show this by construction.

Example:
a*(b )a*
Review - Regular Expressions

The regular expressions over an alphabet * are all strings over the alphabet {(, ), , , *} that can be obtained as follows:
We also allow and +, etc. but these are just shorthands for * and *, etc. so they do not need to be considered for
completeness.
Lecture Notes 7
For Every Regular Expression There is a Corresponding FSM

Formalizing the Construction: The class of regular languages is the smallest class of languages that contains and each of the
singleton strings drawn from , and that is closed under
Union
Concatenation, and
Kleene star
Clearly we can construct an FSM for any finite language, and thus for and all the singleton strings. If we could show that the
class of languages accepted by FSMs is also closed under the operations of union, concatenation, and Kleene star, then we could
recursively construct, for any regular expression, the corresponding FSM, starting with the singleton strings and building up the
machine as required by the operations used to express the regular expression.
FSMs for Primitive Regular Expressions
An FSM for :
An FSM for (*):
An FSM for a single element of :
Closure of FSMs Under Union

To create a FSM that accepts the union of the languages accepted by machines M1 and M2:
1. Create a new start state, and, from it, add -transitions to the start states of M1 and M2.
Closure of FSMs Under Concatenation

To create a FSM that accepts the concatenation of the languages accepted by machines M1 and M2:
1. Start with M1.
2. From every final state of M1, create an -transition to the start state of M2.
3. The final states are the final states of M2.
Lecture Notes 7
Closure of FSMs Under Kleene Star

To create an FSM that accepts the Kleene star of the language accepted by machine M1:
1. Start with M1.
2. Create a new start state S0 and make it a final state (so that we can accept ).
3. Create an -transition from S0 to the start state of M1.
4. Create -transitions from all of M1's final states back to its start state.
5. Make all of M1's final states final.
Note: we need a new start state, S0, because the start state of the new machine must be a final state, and this may not be true of
M1's start state.
Closure of FSMs Under Complementation

To create an FSM that accepts the complement of the language accepted by machine M1:
1. Make M1 deterministic.
2. Reverse final and nonfinal states.
A Complementation Example
a
b
q1
q2
Closure of FSMs Under Intersection

L1 L2 =
L1
L2
Write this in terms of operations we have already proved closure for:
Union
Concatenation
Kleene star
Complementation
An Example
(b ab*a)*ab*
Lecture Notes 7
For Every FSM There is a Corresponding Regular Expression

Proof:
(1) There is a trivial regular expression that describes the strings that can be recognized in going from one state to itself ({} plus
any other single characters for which there are loops) or from one state to another directly (i.e., without passing through any other
states), namely all the single characters for which there are transitions.
(2) Using (1) as the base case, we can build up a regular expression for an entire FSM by induction on the number assigned to
possible intermediate states we can pass through. By adding them in only one at a time, we always get simple regular
expressions, which can then be combined using union, concatenation, and Kleene star.
Key Ideas in the Proof
Idea 1: Number the states and, at each induction step, increase by one the states that can serve as intermediate states.
b
a
Idea 2: To get from state I to state J without passing through any intermediate state numbered greater than K, a machine may
either:
1. Go from I to J without passing through any state numbered greater than K-1 (which we'll take as the induction hypothesis), or
2. Go from I to K, then from K to K any number of times, then from K to J, in each case without passing through any
intermediate states numbered greater than K-1 (the induction hypothesis, again).
So we'll start with no intermediate states allowed, then add them in one at a time, each time building up the regular expression
with operations under which regular languages are closed.
The Formula
Adding in state k as an intermediate state we can use to go from i to j, described using paths that don't use k:
i
R(i, j, k) = R(i, j, k - 1)
/* what you could do without k
R(i, k, k-1)
/* go from i to the new intermediate state without using k or higher
R(k, k, k-1)*
/* then go from the new intermediate state back to itself as many times as you want
R(k, j, k-1)
Solution:
R(s, q, N)
Lecture Notes 7
/* then go from the new intermediate state to j without using k or higher

q F
An Example of the Induction

b
a
Going through no intermediate states:

(1,1,0) =
(1,2,0) = a
(1, 3, 0) =
Allow 1 as an intermediate state:
(2,3,0) = a
(3,3,0) = b
(3,4,0) = a

(1, 3, 2) = (1, 3, 1) (1, 2, 1)(2, 2, 1)*(2, 3, 1)
=
a
*
a
= aa
(1, 3, 3) = (1, 3, 2) (1, 3, 2)(3, 3, 2)*(3, 3, 2)
= aa
aa ( b)* ( b)
= aab*
(1, 4, 3) = (1, 4, 2) (1, 3, 2)(3, 3, 2)*(3, 4, 2)
=
aa ( b)* a
= aab*a
An Easier Way - See Packet
a
1
2
b
b
3
a
(1) Create a new initial state and a new, unique final state, neither of which is part of a loop.
2
b
b
3
a
Lecture Notes 7
(2) Remove states and arcs and replace with arcs labelled with larger and larger regular expressions. States can be removed in
any order, but dont remove either the start or final state.
2
b
ba*b
aa*b
(Notice that the removal of state 3 resulted in two new paths because there were two incoming paths to 3 from another state and 1
outgoing path to another state, so 21 = 2.) The two paths from 2 to 1 should be coalesced by unioning their regular expressions
(not shown).
ab aaa*b ba*b
a
5
(ab aaa*b ba*b)*(a )

4
Thus, the equivalent regular expression is:

(ab aaa*b ba*b)*(a )
Using Regular Expressions in the Real World (PERL)

Matching floating point numbers:
-? ([0-9]+(\.[0-9]*)? | \.[0-9]+)
Matching IP addresses:
([0-9]+ (\ . [0-9]+) {3})
Finding doubled words:

\< ([A-Za-z]+) \s+ \1 \>
From Friedl, J., Mastering Regular Expressions, OReilly,1997.

Note that some of these constructs are more powerful than regular expressions.
Lecture Notes 7
Regular Grammars and Nondeterministic FSAs

Any regular language can be defined by a regular grammar, in which all rules
have a left hand side that is a single nonterminal
have a right hand side that is , a single terminal, a single nonterminal, or a single terminal followed by a single nonterminal.
Example: L={w {a, b}* : |w| is even}
((aa) (ab) (ba) (bb))*
S
S aT
S bT
Ta
Tb
T aS
T bS
a, b
S
T
a, b
An Algorithm to Generate the NDFSM from a Regular Grammar
1.
2.
3.
4.
5.
6.
7.
Create a nonterminal for each state in the NDFSM.

s is the start state.
If there are any rules of the form X w, for some w, then create an additional state labeled #.
For each rule of the form X w Y, add a transition from X to Y labeled w (w ).
For each rule of the form X w, add a transition from X to # labeled w (w ).
For each rule of the form X , mark state X final.
Mark state # final.
Example 1 - Even Length Strings
S
S aT
S bT
Ta
Tb
T aS
T bS
Example 2 - One Character Missing

S
S aB
S aC
S bA
S bC
S cA
S cB
Lecture Notes 7
A bA
A cA
A
B aB
B cB
B
C aC
C bC
C
An Algorithm to Generate a Regular Grammar from an NDFSM

1.
2.
3.
4.
Create a nonterminal for each state in the NDFSM.

The start state becomes the starting nonterminal
For each transition (T, a) = U, make a rule of the form T aU.
For each final state T, make a rule of the form T .
Example:
a
b
X
Y
a
Conversion Algorithms between Regular Language Formalisms
Regular
Grammar
NFSM
(NFA)
Regular
Expression
DFSM
(DFA)
Lecture Notes 7
Languages That Are and Are Not Regular

Read L & S 2.5, 2.6
Read Supplementary Materials: Regular Languages and Finite State Machines: The Pumping Lemma for Regular Languages.
Do Homework 9.
Deciding Whether a Language is Regular
Theorem: There exist languages that are not regular.
Lemma: There are an uncountable number of languages.
Proof of Lemma:
Let: be a finite, nonempty alphabet, e.g., {a, b, c}.
Then * contains all finite strings over .
e.g., {, a, b, c, aa, ab, bc, abc, bba, bbaa, bbbaac}
* is countably infinite, because its elements can be enumerated one at a time, shortest first.
Any language L over is a subset of *,
e.g., L1 = {a, aa, aaa, aaaa, aaaaa, }

L2 = {ab, abb, abbb, abbbb, abbbbb, }
The set of all possible languages is thus the power set of *.
The power set of any countably infinite set is not countable. So there are an uncountable number of languages over *.
Some Languages Are Not Regular
Theorem: There exist languages that are not regular.
Proof:
(1) There are a countably infinite number of regular languages. This true because every description of a regular language is of
finite length, so there is a countably infinite number of such descriptions.
(2) There are an uncountable number of languages.
Thus there are more languages than there are regular languages. So there must exist some language that is not regular.
Showing That a Language is Regular
Techniques for showing that a language L is regular:
1. Show that L has a finite number of elements.
2. Exhibit a regular expression for L.
3. Exhibit a FSA for L.
4. Exhibit a regular grammar for L.
5. Describe L as a function of one or more other regular languages and the operators , , , *, -, . We use here the fact that
the regular languages are closed under all these operations.
6. Define additional operators and prove that the regular languages are closed under them. Then use these operators as in 5.
Example
Let = {0, 1, 2, 9}
Let L * be the set of decimal representations for nonnegative integers (with no leading 0's) divisible by 2 or 3.
L1 = decimal representations of nonnegative integers without leading 0's.
L1 = 0 {1, 2, 9}{0 - 9}*
So L1 is regular.
L2 = decimal representations of nonnegative integers without leading 0's divisible by 2
L2 = L1 *{0, 2, 4, 6, 8}
So L2 is regular.
Lecture Notes 8
Example, Continued
L3 = L1 and divisible by 3
Recall that a number is divisible by 3 if and only if the sum of its digits is divisible by 3. We can build a FSM to determine that
and accept the language L3a, which is composed of strings of digits that sum to a multiple of 3.
L3 = L1 L3a
Finally, L = L2 L3
Another Example
= {0 - 9}
L = {w : w is the social security number of a living US resident}
Finiteness - Theoretical vs. Practical

Any finite language is regular. The size of the language doesn't matter.
Parity
Soc. Sec. #
Checking
Checking
But, from an implementation point of view, it very well may.

When is an FSA a good way to encode the facts about a language?
What are our alternatives?
FSA's are good at looking for repeating patterns. They don't bring much to the table when the language is just a set of unrelated
strings.
Showing that a Language is Not Regular
The argument, I can't find a regular expression or a FSM, won't fly. (But a proof that there cannot exist a FSM is ok.)
Instead, we need to use two fundamental properties shared by regular languages:
1.
We can only use a finite amount of memory to record essential properties.

Example:
anbn is not regular
2.
The only way to generate/accept an infinite language with a finite description is to use Kleene star (in regular expressions) or
cycles (in automata). This forces some kind of simple repetitive cycle within the strings.
Example:
ab*a generates aba, abba, abbba, abbbba, etc.
Example:
{an : n 1 is a prime number} is not regular.
Lecture Notes 8
Exploiting the Repetitive Property

b
b
If a FSM of n states accepts any string of length n, how many strings does it accept?
n
________
babbbbab
x y
z
L = bab*ab
xy*z must be in L.
So L includes: baab, babab, babbab, babbbbbbbbbbab
The Pumping Lemma for Regular Languages
If L is regular, then
N 1, such that
strings w L, where |w| N,
x, y, z, such that
w = xyz
and
|xy| N,
and
y ,
and
q 0, xyqz is in L.
Example: L = anbn
aaaaaaaaaabbbbbbbbbb
x
y
z
N1
long strings w
x, y, z
Call it N
We pick one
We show no x, y, z
Example: anbn is not Regular

N is the number from the pumping lemma (or one more, if N is odd).
Choose w = aN/2bN/2. (Since this is what it takes to be long enough: |w| N)
1
2
aaaaaaaaaa bbbbbbbbbb
x
y
z
We show that there is no x, y, z with the required properties:
|xy| N,
y ,
q 0, xyqz is in L.
Three cases to consider:
y falls in region 1:
y falls across regions 1 and 2:
y falls in region 3:
Lecture Notes 8
Example: anbn is not Regular

Second try:
Choose w to be be aNbN. (Since we get to choose any w in L.)
1
2
aaaaaaaaaa bbbbbbbbbb
x
y
z
We show that there is no x, y, z with the required properties:
|xy| N,
y ,
q 0, xyqz is in L.
Since |xy| N, y must be in region 1. So y = ag for some g 1. Pumping in or out (any q but 1) will violate the constraint that the
number of as has to equal the number of bs.
A Complete Proof Using the Pumping Lemma

Proof that L = {anbn} is not regular:
Suppose L is regular. Since L is regular, we can apply the pumping lemma to L. Let N be the number from the pumping lemma
for L. Choose w = aNbN. Note that w L and |w| N. From the pumping lemma, there exists some x, y, z where xyz = w and
|xy| N, y , and q 0, xyqz L. Because |xy| N, y = a|y| (y is all as). We choose q = 2 and xyqz = aN+|y|bN. Because |y| >
0, then xy2z L (the string has more as than bs). Thus for all possible x, y, z: xyz = w, q, xyqz L. Contradiction. L is
not regular.
Note: the underlined parts of the above proof is boilerplate that can be reused. A complete proof should have this text or
something equivalent.
You get to choose w. Make it a single string that depends only on N. Choose w so that it makes your proof easier.
You may end up with various cases with different q values that reach a contradiction. You have to show that all possible cases
lead to a contradiction.
Proof of the Pumping Lemma
Since L is regular it is accepted by some DFSA, M. Let N be the number of states in M. Let w be a string in L of length N or
more.
N
x
y
x
y
Then, in the first N steps of the computation of M on w, M must visit N+1 states. But there are only N different states, so it must
have visited the same state more than once. Thus it must have looped at least once. We'll call the portion of w that corresponds
to the loop y. But if it can loop once, it can loop an infinite number of times. Thus:
M can recognize xyqz for all values of q 0.
y (since there was a loop of length at least one)
|xy| N (since we found y within the first N steps of the computation)
Lecture Notes 8
Another Pumping Example

L = {w=aJbK : K > J} (more b's than a's)
Choose w = aNbN+1
N
aaaaaaaaaabbbbbbbbbbb
x
y
z
We are guaranteed to pump only a's, since |xy| N. So there exists a number of copies of y that will cause there to be more a's
than b's, thus violating the claim that the pumped string is in L.
A Slightly Different Example of Pumping

L = {w=aJbK : J > K} (more a's than b's)
Choose w = aN+1bN
N
x
y
z
We are guaranteed that y is a string of at least one a, since |xy| N. But if we pump in a's we get even more a's than b's, resulting
in strings that are in L.
What can we do?
Another Slightly Different Example of Pumping

L = {w=aJbK : J K}
Choose w = aN+1bN
N
x
y
z
We are guaranteed that y is a string of at least one a, since |xy| N. But if we pump in a's we get even more a's than b's, resulting
in strings that are in L.
If we pump out, then if y is just a then we still have a string in L.
What can we do?
Lecture Notes 8
Another Pumping Example

L = abanbn
Choose w = abaNbN
N
abaaaaaaaaaabbbbbbbbbbb
x y
z
What are the choices for (x, y):
(, a)
(, ab)
(, aba+)
(a, b)
(a, ba+)
(aba*, a+)
What if L is Regular?
Given a language L that is regular, pumping will work:
L = (ab)*
Choose w = (ab)N
There must exist an x, y, and z where y is pumpable.

abababab ababab abababababab
x
y
z
Suppose y = ababab
Then,
for all q 0,
x yqz L
Note that this does not prove that L is regular. It just fails to prove that it is not.
Using Closure Properties
Once we have some languages that we can prove are not regular, such as anbn, we can use the closure properties of regular
languages to show that other languages are also not regular.
= {a, b}
L = {w : w contains an equal number of a's and b's }
a*b* is regular. So, if L is regular, then L1 = L a*b* is regular.
Example:
But L1 is precisely anbn. So L is not regular.
Dont Try to Use Closure Backwards

One Closure Theorem:
If L1 and L2 are regular, then so is L3 = L1 L2.
But what if L3 and L1 are regular? What can we say about L2?
L3 = L1 L2.
Example:
Lecture Notes 8
ab = ab anbn
A Harder Example of Pumping

= {a}
L = {w = aK : K is a prime number}
|x| + |z| is prime.

|x| + |y| + |z| is prime.
|x| + 2|y| + |z| is prime.
|x| + 3|y| + |z| is prime, and so forth.
N
aaaaaaaaaaaaa
x
y z
Distribution of primes:
||| | |
|
| |
||| | |
|
| |
|
|
Distribution of |x| + q|y| + |z|:

|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
But the Prime Number Theorem tells us that the primes "spread out", i.e., that the number of primes not exceeding x is
asymptotic to x/ln x.
Note that when q = |x| + |z|, |xyqz| = (|y| + 1)(|x| + |z|), which is composite (non-prime) if both factors are > 1. If youre careful
about how you choose N in a pumping lemma proof, you can make this true for both factors.
Automata Theory is Just the Scaffolding

Our results so far give us tools to:
Show a language is regular by:
Showing that it has a finite number of elements,
Providing a regular expression that defines it,
Constructing a FSA that accepts it, or
Exploiting closure properties
Show a language is not regular by:
Using the pumping lemma, or
Exploiting closure properties.
But to use these tools effectively, we may also need domain knowledge (e.g., the Prime Number Theorem).
More Examples
= {0, 1, 2, 3, 4, 5, 6, 7}
L = {w = the octal representation of a number that is divisible by 7}
Example elements of L:
7, 16 (14), 43 (35), 61 (49), 223 (147)
= {W, H, Q, E, S, T, B (measure bar)}

L = {w = w represents a song written in 4/4 time}
More Examples
Example element of L:
WBWBHHBHQQBHHBQEEQEEB
Lecture Notes 8
More Examples
= {0 - 9}
L = {w = is a prime Fermat number}
The Fermat numbers are defined by
n
Fn =
22
+ 1, n = 1, 2, 3,
Example elements of L:
F1 = 5, F2 = 17, F3 = 257, F4 = 65,537
Another Example
= {0 - 9, *, =}
L = {w = a*b=c: a, b, c {0-9}+ and
int(a) * int(b) = int(c)}
The Bottom Line

A language is regular if:
OR
The Bottom Line (Examples)
The set of decimal representations for nonnegative

integers divisible by 2 or 3
The social security numbers of living US residents.
Parity checking
anbn
ajbk where k>j
ak where k is prime
The set of strings over {a, b} that contain an equal

number of a's and b's.
The octal representations of numbers that are divisible
by 7
The songs in 4/4 time
The set of prime Fermat numbers
Decision Procedures
A decision procedure is an algorithm that answers a question (usually yes or no) and terminates. The whole idea of a
decision procedure itself raises a new class of questions. In particular, we can now ask,
1.
2.
3.
Is there a decision procedure for question X?

What is that procedure?
How efficient is the best such procedure?
Clearly, if we jump immediately to an answer to question 2, we have our answer to question 1. But sometimes it makes sense to
answer question 1 first. For one thing, it tells us whether to bother looking for answers to questions 2 and 3.
Examples of Question 1:
Is there a decision procedure, given a regular expression E and a string S, for determining whether S is in L(E)?
Is there a decision procedure, given a Turing machine T and an input string S, for determining whether T halts on S?
Lecture Notes 8
Decision Procedures for Regular Languages

Let M be a deterministic FSA. There is a decision procedure to determine whether:
w L(M) for some fixed w

L(M) is empty
L(M) is finite
L(M) is infinite
Let M1 and M2 be two deterministic FSAs. There is a decision procedure to determine whether M1 and M2 are equivalent. Let L1
and L2 be the languages accepted by M1 and M2. Then the language
L
= (L1 L2) (L1 L2)

= (L1 - L2) (L2 - L1)
must be regular. L is empty iff L1 = L2. There is a decision procedure to determine whether L is empty and thus whether L1 = L2
and thus whether M1 and M2 are equivalent.
L1
Lecture Notes 8
L2
L1
L2
L1,2

Do Homework 7.
A relation R is an equivalence relation if it is: reflexive, symmetric, and transitive.
Example: R = the reflexive, symmetric, transitive closure of:
(Bob, Bill), (Bob, Butch), (Butch, Bud),
(Jim, Joe), (Joe, John), (Joe, Jared),
(Tim, Tom), (Tom, Tad)
An equivalence relation on a nonempty set A creates a partition of A. We write the elements of the partition as [a1], [a2],
Example:
Another Equivalence Relation

Example: R = the reflexive, symmetric, transitive closure of:
(apple, pear), (pear, banana), (pear, peach),
(peas, mushrooms), (peas, onions), (peas, zucchini)
(bread, rice), (rice, potatoes), (rice, pasta)
Partition:
Lecture Notes 9
State Minimization for DFAs

Read K & S 2.7
Do Homework 10.
State Minimization
Consider:
a
a
5
a
a
6
Is this a minimal machine?

State Minimization
Step (1): Get rid of unreachable states.
a
1
2
b
a, b
3
State 3 is unreachable.
Step (2): Get rid of redundant states.
b
3
a
States 2 and 3 are redundant.
Getting Rid of Unreachable States
We can't easily find the unreachable states directly. But we can find the reachable ones and determine the unreachable ones from
there. An algorithm for finding the reachable states:
a
1
2
b
a, b
3
Lecture Notes 10
State Minimization
Getting Rid of Redundant States

Intuitively, two states are equivalent to each other (and thus one is redundant) if all strings in * have the same fate, regardless of
which of the two states the machine is in. But how can we tell this?
The simple case:
a, b
1
b
2
a
a, b
Two states have identical sets of transitions out.
Getting Rid of Redundant States
The harder case:

b
3
a
The outcomes are the same, even though the states aren't.
Finding an Algorithm for Minimization
Capture the notion of equivalence classes of strings with respect to a language.
Capture the (weaker) notion of equivalence classes of strings with respect to a language and a particular FSA.
Prove that we can always find a deterministic FSA with a number of states equal to the number of equivalence classes of strings.
Describe an algorithm for finding that deterministic FSA.
Defining Equivalence for Strings
We want to capture the notion that two strings are equivalent with respect to a language L if, no matter what is tacked on to them
on the right, either they will both be in L or neither will. Why is this the right notion? Because it corresponds naturally to what
the states of a recognizing FSM have to remember.
Example:
(1)
(2)
Suppose L = {w {a,b}* : |w| is even}. Are (1) and (2) equivalent?

Suppose L = {w {a,b}* : every a is immediately followed by b}. Are (1) and (2) equivalent?
Lecture Notes 10
State Minimization
Defining Equivalence for Strings

If two strings are equivalent with respect to L, we write x L y. Formally, x L y if, z *,
xz L iff yz L.
Notice that L is an equivalence relation.
Example:
= {a, b}
L = {w * : every a is immediately followed by b }
a
b
bbb
baa
aa
bb
aba
aab
The equivalence classes of L:
|L | is the number of equivalence classes of L.

Another Example of L
= {a, b}
L = {w * : |w| is even}
a
b
aa
aabb
bbaa
aabaa
bb
aba
aab
bbb
baa
Yet Another Example of L
= {a, b}
L = aab*a
a
b
aa
ab
aabb
aabaa
aabbba
aabbaa
ba
bb
aaa
aba
aab
bab
An Example of L Where All Elements of L Are Not in the Same Equivalence Class
= {a, b}
L = {w {a, b}* : no two adjacent characters are the same}
bb
aba
a
aab
b
baa
aa
aabb
Lecture Notes 10
State Minimization
aabaa
aabbba
aabbaa
Is |
L| Always Finite?
= {a, b}
L = anbn
a
b
aaaa
aaaaa
aa
aba
aaa
Bringing FSMs into the Picture
L is an ideal relation.
What if we now consider what happens to strings when they are being processed by a real FSM?
= {a, b}
L = {w * : |w| is even}
a
1
2
a, b
b
a, b
3
Define ~M to relate pairs of strings that drive M from s to the same state.
Formally, if M is a deterministic FSM, then x ~M y if there is some state q in M such that (s, x) |-*M (q, ) and (s, y) |-*M (q, ).
Notice that M is an equivalence relation.
An Example of ~M
= {a, b}
L = {w * : |w| is even}
a
1
2
a, b
b
a, b
3
a
b
aa
aabb
bbaa
aabaa
bb
aba
aab
bbb
baa
The equivalence classes of ~M:
|~M| =
Lecture Notes 10
State Minimization
Another Example of ~M
= {a, b}
L = {w * : |w| is even}
a,b
1
2
a, b
a
b
aa
aabb
bbaa
aabaa
bb
aba
aab
bbb
baa
The equivalence classes of ~M:
|~M| =
The Relationship Between L and ~M

L:
[, aa, bb, aabb, bbaa]

|w| is even
[a, b, aba, aab, bbb, baa, aabaa] |w| is odd
~M, 3 state machine:

q1: [, aa, bb, aabb, bbaa]
|w| is even
q2: [a, aba, baa, aabaa] (ab ba aa bb)*a
q3: [b, aab, bbb]
(ab ba aa bb)*b
~M, 2 state machine:
q1: [, aa, bb, aabb, bbaa]
|w| is even
q2: [a, b, aba, aab, bbb, baa, aabaa] |w| is odd
~M is a refinement of L.
The Refinement
L
[even length]
~M
(3 state)
[even length]
[odd length]
odd ending
in a
odd ending
in b
(S)
(R)
An equivalence relation R is a refinement of another one S iff

xRy xSy
In other words, R makes all the same distinctions S does, plus possibly more.
|R| |S|
Lecture Notes 10
State Minimization
~M is a Refinement of L.
Theorem: For any deterministic finite automaton M and any strings x, y *, if x ~M y, then x L y.
Proof: If x ~M y, then x and y drive m to the same state q. From q, any continuation string w will drive M to some state r. Thus
xw and yw both drive M to r. Either r is a final state, in which case they both accept, or it is not, in which case they both reject.
But this is exactly the definition of L.
L |.
Corollary: |~M | |
Going the Other Way
When is this true?
If x L(M) y then x ~M y.
Finding the Minimal FSM for L
What's the smallest number of states we can get away with in a machine to accept L?
Example:
L = {w * : |w| is even}
Minimal number of states for M(L) =

This follows directly from the theorem that says that, for any machine M that accepts L, |~M| must be at least as large as |L |.
Can we always find a machine with this minimal number of states?
The Myhill-Nerode Theorem
Theorem: Let L be a regular language. Then there is a deterministic FSA that accepts L and that has precisely |L | states.
Proof: (by construction)
M=
K states, corresponding to the equivalence classes of L.
s = [], the equivalence class of under L.
F = {[x] : x L}
([x], a) = [xa]
For this construction to prove the theorem, we must show:
1. K is finite.
2. is well defined, i.e., ([x], a) = [xa] is independent of x.
3. L = L(M)
Lecture Notes 10
State Minimization
The Proof
(1) K is finite.
Since L is regular, there must exist a machine M, with |~M| finite. We know that
|~M| |L|
Thus |L| is finite.
(2) is well defined.
This is assured by the definition of L, which groups together precisely those strings that have the same fate with respect to L.
The Proof, Continued
(3) L = L(M)
Suppose we knew that ([x], y) |-M* ([xy], ).
Now let [x] be [] and let s be a string in *.
Then
([], s) |-M* ([s], )
M will accept s if [s] F.
By the definition of F, [s] F iff all strings in [s] are in L.
So M accepts precisely the strings in L.
The Proof, Continued
Lemma: ([x], y) |-M* ([xy], )
By induction on |y|:
Trivial if |y| = 0.
Suppose true for |y| = n.
Show true for |y| = n+1
Let y = y'a, for some character a. Then,
|y'| = n
([x], y'a) |-M* ([xy'], a)
(induction hypothesis)
(definition of )
([xy',] a) |-M* ([xy'a], )
(trans. of |-M*)
([x], y'a) |-M* ([xy'a], )
(definition of y)
([x], y) |-M* ([xy], )
Another Version of the Myhill-Nerode Theorem
Theorem: A language is regular iff |L| is finite.
Example:
Consider:
L = anbn
a, aa, aaa, aaaa, aaaaa
Equivalence classes:
Proof:
Regular |L| is finite: If L is regular, then there exists an accepting machine M with a finite number of states N. We know that
N |L|. Thus |L| is finite.
|L| is finite regular: If |L| is finite, then the standard DFSA ML accepts L. Since L is accepted by a FSA, it is regular.
Lecture Notes 10
State Minimization
Constructing the Minimal DFA from L

= {a, b}
L = {w {a, b}* : no two adjacent characters are the same}
1: []
2: [a, ba, aba, baba, ababa, ...]
3: [b, ab, bab, abab, ...]
4: [bb, aa, bba, bbb, ...]
(b)(ab)*a
(a)(ba)*b
the rest
Equivalence classes become states

Start state is []
Final states are all equivalence classes in L
([x], a) = [xa]
2
a
a b
b
a, b
b
3
Using Myhill-Nerode to Prove that L is not Regular

L = {an : n is prime}
Consider:
a
aa
aaa
aaaa
Equivalence classes:
So Where Do We Stand?
1. We know that for any regular language L there exists a minimal accepting machine ML.
2. We know that |K| of ML equals |L|.
3. We know how to construct ML from L.
But is this good enough?
Consider:
a
a
1
5
a
a
6
Lecture Notes 10
State Minimization
Constructing a Minimal FSA Without Knowing L

We want to take as input any DFSA M' that accepts L, and output a minimal, equivalent DFSA M.
What we need is a definition for "equivalent", i.e., mergeable states.
Define q p iff for all strings w *, either w drives M to an accepting state from both q and p or it drives M to a rejecting state
from both q and p.
Example:
= {a, b}
L = {w * : |w| is even}
a
1
2
a, b
b
a, b
3
Constructing as the Limit of a Sequence of Approximating Equivalence Relations n
(Where n is the length of the input strings that have been considered so far)
We'll consider input strings, starting with , and increasing in length by 1 at each iteration. We'll start by way overgrouping
states. Then we'll split them apart as it becomes apparent (with longer and longer strings) that their behavior is not identical.
Initially, 0 has only two equivalence classes: [F] and [K - F], since on input , there are only two possible outcomes, accept or
reject.
Next consider strings of length 1, i.e., each element of . Split any equivalence classes of 0 that don't behave identically on all
inputs. Note that in all cases, n is a refinement of n-1.
Continue, until no splitting occurs, computing n from n-1.
Constructing , Continued
More precisely, for any two states p and q K and any n 1, q n p iff:
1. q n-1 p, AND
2. for all a , (p, a) n-1 (q, a)
Lecture Notes 10
State Minimization
The Construction Algorithm

The equivalence classes of 0 are F and K-F.
Repeat for n = 1, 2, 3
For each equivalence class C of n-1 do
For each pair of elements p and q in C do
For each a in do
See if (p, a) n-1 (q, a)
If there are any differences in the behavior of p and q, then split them and create a new equivalence
class.
Until n = n-1. is this answer. Then use these equivalence classes to coalesce states.
An Example
= {a, b}
b
1
a
a
a,b
0 =
1 =
2 =
Another Example
(a*b*)*
a
b
b
2
a
0 =
1 =
Minimal machine:
Lecture Notes 10
State Minimization
10
Another Example
Example: L={w {a, b}* : |w| is even}

((aa) (ab) (ba) (bb))*
S
S aT
S bT
Ta
Tb
T aS
T bS
a, b
T
a, b
#
b
Convert to deterministic:
S = {s}
=
Another Example, Continued
Minimize:
a,b
S(1)
T(2)
a,b
a,b
#S(3)
0 =
1 =
Minimal machine:
Lecture Notes 10
State Minimization
11

Grammars, Languages, and Machines
Language
L
Grammar
Accepts
Machine
Regular Grammars, Languages, and Machines

Most interesting languages are infinite. So we can't write them down. But we can write down finite grammars and finite
machine specifications, and we can define algorithms for mapping between and among them.
Grammars
Regular
Expressions
Machines
Nondeterministic
FSAs
Deterministic
FSAs
Regular
Grammars
Minimal
DFSAs
What Does Finite State Really Mean?
There are two kinds of finite state problems:
Those in which:
Some history matters.
Only a finite amount of history matters. In particular, it's often the case that we don't care what order things
occurred in.
Examples:
Parity
Money in a vending machine
Seat belt buzzer
Those that are characterized by patterns.
Examples:
Switching circuits:
Telephone
Railroad
Traffic lights
Lexical analysis
grep
Lecture Notes 11
Context-Free Grammars
Read K & S 3.1
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Context-Free Grammars
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Designing Context-Free Grammars.
Do Homework 11.
Context-Free Grammars, Languages, and Pushdown Automata
Context-Free
Language
L
Context-Free
Grammar
Accepts
Pushdown
Automaton
Grammars Define Languages

Think of grammars as either generators or acceptors.
Example: L = {w {a, b}* : |w| is even}
Regular Expression
(aa ab ba bb)*
Derivation
(Generate)
choose aa
choose ab
yields
Lecture Notes 12
T
a
a a a b
Parse (Accept)
Regular Grammar
S
S aT
S bT
Ta
Tb
T aS
T bS
S
a T
b
a a b
use corresponding FSM
Derivation is Not Necessarily Unique

Example: L = {w {a, b}* : there is at least one a}
Regular Expression
Regular Grammar
(a b)*a (a b)*
Sa
S bS
S aS
S aT
Ta
Tb
T aT
T bT
choose a from (a b)
choose a from (a b)
choose a
choose a
choose a from (a b)
choose a from (a b)
S
a
S
S
a
S
a
T
a
T
a
More Powerful Grammars

Regular grammars must always produce strings one character at a time, moving left to right.
But sometimes it's more natural to describe generation more flexibly.
Example 1: L = ab*a
S aBa
B
B bB
S aB
Ba
B bB
vs.
Example 2: L = anb*an
SB
S aSa
B
B bB
Key distinction: Example 1 has no recursion on the nonregular rule.
Remove all restrictions on the form of the right hand sides.
S abDeFGab
Keep requirement for single non-terminal on left hand side.
S
but not ASB or aSb
Examples:
Lecture Notes 12
or ab
balanced parentheses
S
S SS
S (S)
anbn
SaSb
S
A context-free grammar G is a quadruple (V, , R, S), where:
V is the rule alphabet, which contains nonterminals (symbols that are used in the grammar but that do not appear in strings in
the language) and terminals,
(the set of terminals) is a subset of V,
R (the set of rules) is a finite subset of (V - ) V*,
S (the start symbol) is an element of V - .
x G y is a binary relation where x, y V* such that x = A and y = for some rule A in R.
Any sequence of the form
w0 G w1 G w2 G . . . G wn
e.g., (S) (SS) ((S)S)
is called a derivation in G. Each wi is called a sentinel form.
The language generated by G is
{w * : S G* w}
A language L is context free if L = L(G) for some context-free grammar G.

Example Derivations
G = (W, , R, S), where
W = {S} ,
= {a, b},
R=
{ S a,
S aS,
S aSb}
S
a
S
S
a
a
S
S
S
a
b
S
a S b
a
a
a
b
b
S
S
a
Another Example - Unequal a's and b's

L = {anbm : n m}
G = (W, , R, S), where
W = {a, b, S, A, B},
= {a, b},
R=
Lecture Notes 12
SA
SB
Aa
A aA
A aAb
Bb
B Bb
B aBb
/* more a's than b's

/* more b's than a's
S NP VP
NP the NP1 | NP1
NP1 ADJ NP1 | N
ADJ big | youngest | oldest
N boy | boys
VP V | V NP
V run | runs
English
the boys run
big boys run
the youngest boy runs
the youngest oldest boy runs
the boy run
Who did you say Bill saw coming out of the hotel?
Arithmetic Expressions
The Language of Simple Arithmetic Expressions
G = (V, , R, E), where
V = {+, *, id, T, F, E},
= {+, *, id},
R = { E id
EE+E
EE*E}
E
id
id
id
(id
id
id
id)
(id
id
id
id)
id
Arithmetic Expressions -- A Better Way

The Language of Simple Arithmetic Expressions
V = {+, *, (, ), id, T, F, E},
= {+, *, (, ), id},
R={ EE+T
E T
TT*F
TF
F (E)
F id
}
Lecture Notes 12
Examples:
id + id * id
id * id * id
BNF
Backus-Naur Form (BNF) is used to define the syntax of programming languages using context-free grammars.
Main idea: give descriptive names to nonterminals and put them in angle brackets.
Example: arithmetic expressions:
expression expression + term
expression term
term term * factor
term factor
factor (expression)
factor id
The Language of Boolean Logic

V = {, , , , (, ), id, E, E1, E2, E3, E4 },
= {, , , , (, ), id},
R = { E E E1
E E1
E1 E1 E2
E1 E2
E2 E2 E3
E2 E3
E3 E4
E3 E4
E4 (E)
E4 id }
Boolean Logic isn't Regular

Suppose it were regular. Then there is an N as specified in the pumping theorem.
Let w be a string of length 2N + 1 + 2|id| of the form:
w = ( ( ( ( ( ( id ) ) ) ) ) ) id
N
x y
y = (k for some k > 0 because |xy| N.
Then the string that is identical to w except that it has k additional ('s at the beginning would also be in the language. But it can't
be because the parentheses would be mismatched. So the language is not regular.
Lecture Notes 12
All Regular Languages Are Context Free

(1) Every regular language can be described by a regular grammar. We know this because we can derive a regular grammar from
any FSM (as well as vice versa). Regular grammars are special cases of context-free grammars.
a, b
S
T
a, b
(2) The context-free languages are precisely the languages accepted by NDPDAs. But every FSM is a PDA that doesn't bother
with the stack. So every regular language can be accepted by a NDPDA and is thus context-free.
(3) Context-free languages are closed under union, concatenation, and Kleene *, and and each single character in are clearly
context free.
Lecture Notes 12
Parse Trees
Read K & S 3.2
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Derivations and Parse Trees.
Do Homework 12.
Parse Trees
Regular languages:
We care about recognizing patterns and taking appropriate actions.
Example: A parity checker
Structure
Context free languages:
We care about structure.
E
id
id
id
(id
id
*
id)
Parse Trees Capture Essential Structure
E id
EE+E
EE*E
E
id
id
id
Lecture Notes 13
(id
id
id
id)
(id
Parse Trees
id
id
+
id)
id
Parse Trees are Just Trees

root
height
nodes
leaves
yield
Leaves are all labeled with terminals or .
Other nodes are labeled with nonterminals.
A path is a sequence of nodes, starting at the root, ending at a leaf, and following branches in the tree.
The length of the yield of any tree T with height H and branching factor (fanout) B is
Derivations
To capture structure, we must capture the path we took through the grammar. Derivations do that.
S
S SS
S (S)
1
2
3
4
5
6
S SS (S)S ((S))S (())S (())(S) (())()
S SS (S)S ((S))S ((S))(S) (())(S) (())()
1
2
3
5
4
6
S
S
(
S
(
Alternative Derivations
S
S SS
S (S)
S SS (S)S ((S))S (())S (())(S) (())()

S SS SSS S(S)S S((S))S S(())S S(())(S) S(())() (())()
S
S
(
S
(
)
)
S
)
S
S
(
)
( S )
Lecture Notes 13
Parse Trees
Ordering Derivations
Consider two derivations:
1
2
3
4
5
6
7
S SS (S)S ((S))S (())S (())(S) (())()
S SS (S)S ((S))S ((S))(S) (())(S) (())()
1
2
3
4
5
6
7
We can write these, or any, derivation as
D1 = x1 x2 x3 xn
D2 = x1' x2' x3' xn'
Consider three derivations:

1
2
3
4
(1) S SS (S)S ((S))S
5
(())S
We say that D1 precedes D2, written D1< D2, if:

D1 and D2 are the same length > 1, and
There is some integer k, 1 < k < n, such that:
for all i k, xi = xi'
xk-1 = x'k-1 = uAvBw : u, v, w V*,
and A, B V -
xk = uyvBw, where A y R
xk' = uAvzw where B z R
xk+1 = x'k+1 = uyvzw
Comparing Several Derivations
6
7
(())(S) (())()
(2) S SS (S)S ((S))S ((S))(S) (())(S) (())()

(3) S SS (S)S ((S))S ((S))(S) ((S))() (())()
D1 < D2
D2 < D3
But D1 does not precede D3.
All three seem similar though. We can define similarity:
D1 is similar to D2 iff the pair (D1, D2) is in the reflexive, symmetric, transitive closure of <.
Note: similar is an equivalence class.
In other words, two derivations are similar if one can be transformed into another by a sequence of switchings in the order of rule
applications.
Parse Trees Capture Similarity
1
2
3
4
5
6
7
(1) S SS (S)S ((S))S (())S (())(S) (())()
(2) S SS (S)S ((S))S ((S))(S) (())(S) (())()
(3) S SS (S)S ((S))S ((S))(S) ((S))() (())()
D1 < D2
D2 < D3
All three derivations are similar to each other. This parse tree describes this equivalence class of the similarity relation:
S
S
(
S
(
Lecture Notes 13
Parse Trees
The Maximal Element of <

S
S
(
S
(
There's one derivation in this equivalence class that precedes all others in the class.
We call this the leftmost derivation. There is a corresponding rightmost derivation.
The leftmost (rightmost) derivation can be used to construct the parse tree and the parse tree can be used to construct the leftmost
(rightmost) derivation.
Another Example
E id
EE+E
EE*E
(1) E E+E E+E*E E+E*id E+id*id id+id*id

(2) E E*E E*id E+E*id E+id*id id+id*id
id
id
id
[id
id
id]
id
[id
id
id]
id
id
Ambiguity
A grammar G for a language L is ambiguous if there exist strings in L for which G can generate more than one parse tree (note
that we don't care about the number of derivations).
The following grammar for arithmetic expressions is ambiguous:
E id
EE+E
EE*E
Often, when this happens, we can find a different, unambiguous grammar to describe L.
Lecture Notes 13
Parse Trees
Resolving Ambiguity in the Grammar

Parse :
id + id * id

V = {+, *, (, ), id, T, F, E},
= {+, *, (, ), id},
R={ EE+T
E T
TT*F
TF
F (E)
F id }
Another Example
The following grammar for the language of matched parentheses is ambiguous:
S
S SS
S (S)
S
S
(
S
(
)
)
S
)
S
(
S
(
( S )
Resolving the Ambiguity with a Different Grammar
One problem is the production.

A different grammar for the language of balanced parentheses:
S
S S1
S1 S1 S1
S1 (S1)
S1 ()
S
S1
S1
(
Lecture Notes 13
Parse Trees
S1
( )
S1
)
A General Technique for Eliminating

If G is any context-free grammar for a language L and L then we can construct an alternative grammar G' for L by:
1.
2.
3.
Find the set N of nullable variables:

A variable V is nullable if either:
there is a rule
(1) V
or there is a rule
(2) V PQRsuch that P, Q, R, are all nullable
So begin with N containing all the variables that satisfy (1). Evaluate all other variables with respect to (2). Continue until
no new variables can be added to N.
For every rule of the form
P Q for some Q in N, add a rule
P
Delete all rules of the form
V
Sometimes Eliminating Ambiguity Isn't Possible
The boys hit the ball with the bat.
S NP VP
NP the NP1 | NP1 | NP2
NP1 ADJ NP1 | N
NP2 NP1 PP
ADJ big | youngest | oldest
The boys hit the ball with the autograph.
N boy | boys | ball | bat | autograph
VP V | V NP
VP VP PP
V hit| hits
PP with NP
Why It's Not Possible
We could write an unambiguous grammar to describe L but it wouldn't always get the parses we want. Any grammar that is
capable of getting all the parses will be ambiguous because the facts required to choose a derivation cannot be captured in
the context-free framework.
Example: Our simple English grammar
[[The boys] [hit [the ball] [with [the bat]]]]
[[The boys] [hit [the ball] [with [the autograph]]]]
There is no grammar that describes L that is not ambiguous.
Example: L = {anbncm} {anbmcm}
S S1 | S2
S1 S1c | A
A aAb |
S2 aS2|B
B bBc |
Now consider the strings anbncn

They have two distinct derivations
Inherent Ambiguity of CFLs

A context free language with the property that all grammars that generate it are ambiguous is inherently ambiguous.
L = {anbncm} {anbmcm} is inherently ambiguous.
Other languages that appear ambiguous given one grammar, turn out not to be inherently ambiguous because we can find an
unambiguous grammar.
Examples:
Balanced Parentheses
Whenever we design practical languages, it is important that they not be inherently ambiguous.
Lecture Notes 13
Parse Trees
Pushdown Automata
Read K & S 3.3.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Designing Pushdown Automata.
Do Homework 13.
Recognizing Context-Free Languages
Two notions of recognition:
(1) Say yes or no, just like with FSMs
(2) Say yes or no, AND
if yes, describe the structure
c
Just Recognizing
We need a device similar to an FSM except that it needs more power.

The insight: Precisely what it needs is a stack, which gives it an unlimited amount of memory with a restricted structure.
(
(
(
(
(
(
Finite
State
Controller
(
Definition of a Pushdown Automaton
M = (K, , , , s, F), where:

is the input alphabet
is the stack alphabet
is the transition relation. It is a finite subset of
(K
state
( {})
input or
string of symbols to pop

from top of stack
M accepts a string w iff

(s, w, ) |-M* (p, , )
Lecture Notes 14
K
state
string of symbols to
push on top of stack
for some state p F
Pushdown Automata
A PDA for Balanced Brackets

[//[
s
]/[/
M = (K, , , , s, F), where:
K = {s}
= {[, ]}
= {[}
F = {s}
contains:
the states
the input alphabet
the stack alphabet
((s, [, ), (s, [ ))
((s, ], [ ), (s, ))
Important:
This does not mean that the stack is empty.
An Example of Accepting
[//[
s
]/[/
contains:
[1]
((s, [, ), (s, [ ))
[2]
((s, ], [ ), (s, ))
input = [ [ [ ] [ ] ] ]
trans
1
1
1
2
1
2
2
2
state
s
s
s
s
s
s
s
s
s
unread input
[[[][]]]
[[][]]]
[][]]]
][]]]
[]]]
]]]
]]
]
stack
[
[[
[[[
[[
[[[
[[
[
An Example of Rejecting
[//[
s
]/[/
contains:
[1]
((s, [, ), (s, [ ))
[2]
((s, ], [ ), (s, ))
input = [ [ ] ] ]
trans
state
unread input
stack
s
[[]]]
1
s
[]]]
[
1
s
]]]
[[
2
s
]]
[
2
s
]
none!
s
]
We're in s, a final state, but we cannot accept because the input string is not empty. So we reject.
Lecture Notes 14
Pushdown Automata
A PDA for anbn

First we notice:
We'll use the stack to count the a's.
This time, all strings in L have two regions. So we need two states so that a's can't follow b's. Note the similarity to the
regular language a*b*.
A PDA for wcwR

A PDA to accept strings of the form wcwR:
a//a
a/a/
c//
s
b//b
b/b/
M = (K, , , , s, F), where:

K = {s, f}
= {a, b, c}
= {a, b}
F = {f}
contains:
((s, a, ), (s, a))
((s, b, ), (s, b))
((s, c, ), (f, ))
((f, a, a), (f, ))
((f, b, b), (f, ))
the states
the input alphabet
the stack alphabet
the final states
a//a
a/a/
c//
s
b//b
f
b/b/
contains:
[1]
((s, a, ), (s, a))
[2]
((s, b, ), (s, b))
[3]
((s, c, ), (f, ))
[4]
((f, a, a), (f, ))
[5]
((f, b, b), (f, ))
input = b a c a b
trans
2
1
3
5
6
Lecture Notes 14
state
s
s
s
f
f
f
unread input
bacab
acab
cab
ab
b
stack
b
ab
ab
b
Pushdown Automata
A Nondeterministic PDA
L = wwR
S
S aSa
S bSb
A PDA to accept strings of the form wwR:
a//a
a/a/
//
s
b//b
b/b/
M = (K, , , , s, F), where:

K = {s, f}
= {a, b, c}
= {a, b}
F = {f}
contains:
((s, a, ), (s, a))
((s, b, ), (s, b))
((s, , ), (f, ))
((f, a, a), (f, ))
((f, b, b), (f, ))
the states
the input alphabet
the stack alphabet
the final states
a//a
a/a/
//
s
b//b
b/b/
[1]
[2]
[3]
((s, a, ), (s, a))

((s, b, ), (s, b))
((s, , ), (f, ))
input: a a b b a a
trans
state
s
s
f
f
unread input
aabbaa
abbaa
abbaa
bbaa
stack
a
a
state
s
s
s
s
f
f
f
f
unread input
aabbaa
abbaa
bbaa
baa
baa
aa
a
stack
a
aa
baa
baa
aa
a
1
3
4
none
trans
1
1
2
3
5
4
4
Lecture Notes 14
[4]
[5]
Pushdown Automata
((f, a, a), (f, ))

((f, b, b), (f, ))
L = {ambn : m n}
A context-free grammar for L:
S
S Sb
S aSb
A PDA to accept L:
a//a
/* more b's
b/a/
b/a/
b//
2
b//
Accepting Mismatches
L = {ambn m n; m, n >0}
a//a
b/a/
b/a/
If stack and input are empty, halt and reject.
If input is empty but stack is not (m > n) (accept):
a//a
/a/
/a/
b/a/
b/a/
2
If stack is empty but input is not (m < n) (accept):
a//a
1
/a/
b/a/
/a/
b/a/
2
b//
4
Lecture Notes 14
b//
Pushdown Automata
Eliminating Nondeterminism
A PDA is deterministic if, for each input and state, there is at most one possible transition. Determinism implies uniquely
defined machine behavior.
a//a
/a/
b/a/
/a/
b/a/
1
b//
4
b//
Jumping to the input clearing state 4:

Need to detect bottom of stack, so push Z onto the stack before we start.
a//a
/a/
b/a/
/a/
b/a/
//Z
1
/Z/
b/Z/
4
b//
Jumping to the stack clearing state 3:

Need to detect end of input. To do that, we actually need to modify the definition of L to add a termination character
(e.g., $)
L = {anbmcp : n,m,p 0 and (n m or m p)}
S NC
S QP
NA
NB
Aa
A aA
A aAb
Bb
B Bb
B aBb
/* n m, then arbitrary c's

/* arbitrary a's, then p m
/* more a's than b's
/* more b's than a's
C | cC
P B'
P C'
B' b
B' bB'
B' bB'c
C' c | C'c
C' C'c
C' bC'c
Q | aQ
/* add any number of c's

/* more b's than c's
/* more c's than b's
/* prefix with any number of a's
L = {anbmcp : n,m,p 0 and (n m or m p)}

//Z
S
a//a
S'
machine for N
a//
b,c
clear and accept
machine for P
Lecture Notes 14
Pushdown Automata
Another Deterministic CFL

L = {anbn} {bn an}
A CFG for L:
A NDPDA for L:
SA
SB
A
A aAb
B
B bBa
A DPDA for L:
More on PDAs
What about a PDA to accept strings of the form ww?
Every FSM is (Trivially) a PDA
Given an FSM M = (K, , , s, F)
and elements of of the form
(
p,
old state,
i,
input,
q
)
new state
We construct a PDA M' = (K, , , , s, F)

where =
/* stack alphabet
and
each transition (p, i, q)
becomes
( (
p,
i,
),
old state,
input,
don't look at stack
q,
new state
)
don't push on stack
In other words, we just don't use the stack.

Alternative (but Equivalent) Definitions of a NDPDA
Example: Accept by final state at end of string (i.e., we don't care about the stack being empty)
We can easily convert from one of our machines to one of these:
1. Add a new state at the beginning that pushes # onto the stack.
2. Add a new final state and a transition to it that can be taken if the input string is empty and the top of the stack is #.
Converting the balanced parentheses machine:
//#
(//(
S
)/(/
(//(
S'
)/(/
/#/
F
The new machine is nondeterministic:
( ) ( )
The stack will be:
Lecture Notes 14
Pushdown Automata
What About PDA's for Interesting Languages?

EE+T
ET
TT*F
TF
F (E)
F id
(1) (2, , E), (2, E+T)
(2) (2, , E), (2, T)
(3) (2, , T), (2, T*F)
(4) (2, , T), (2, F)
(5) (2, , F), (2, (E) )
(6) (2, , F), (2, id)
(7) (2, id, id), (2, )
(8) (2, (, ( ), (2, )
(9) (2, ), ) ), (2, )
(10) (2, +, +), (2, )
(11) (2, *, *), (2, )
//E
1
Example:
a+b*c
But what we really want to do with languages like this is to extract structure.
Comparing Regular and Context-Free Languages
Regular Languages
Context-Free Languages
context-free grammars
parse
= NDPDAs
regular expressions
- or regular grammars
recognize
= DFSAs
Lecture Notes 14
Pushdown Automata

Read K & S 3.4.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Context-Free Languages and PDAs.
Do Homework 14.
PDAs and Context-Free Grammars
Theorem: The class of languages accepted by PDAs is exactly the class of context-free languages.
Recall: context-free languages are languages that can be defined with context-free grammars.
Restate theorem:
Can describe with context-free grammar Can accept by PDA

Going One Way
Lemma: Each context-free language is accepted by some PDA.

Proof (by construction by top-down parse conversion algorithm):
The idea: Let the stack do the work.
Example: Arithmetic expressions
EE+T
ET
TT*F
TF
F (E)
F id
(1)
(2)
(3)
(4)
(5)
(6)
//E
1
(2, , E), (2, E+T)

(2, , E), (2, T)
(2, , T), (2, T*F)
(2, , T), (2, F)
(2, , F), (2, (E) )
(2, , F), (2, id)
(7) (2, id, id), (2, )

(8) (2, (, ( ), (2, )
(9) (2, ), ) ), (2, )
(10) (2, +, +), (2, )
(11) (2, *, *), (2, )
The Top-down Parse Conversion Algorithm
Given G = (V, , R, S)
Construct M such that L(M) = L(G)
M = ({p, q}, , V, , p, {q}), where contains:
(1) ((p, , ), (q, S))
push the start symbol on the stack
(2) ((q, , A), (q, x)) for each rule A x in R
replace left hand side with right hand side
(3) ((q, a, a), (q, )) for each a
read an input character and pop it from the stack
The resulting machine can execute a leftmost derivation of an input string in a top-down fashion.
Lecture Notes 15
Example of the Algorithm

L = {anb*an}
(1)
(2)
(3)
(4)
(5)
input = a a b b a a
trans
state
p
q
q
q
q
q
q
q
q
q
q
q
q
q
0
3
6
3
6
2
5
7
5
7
4
6
6
(p, , ), (q, S)
(q, , S), (q, )
(q, , S), (q, B)
(q, , S), (q, aSa)
(q, , B), (q, )
(q, , B), (q, bB)
(q, a, a), (q, )
(q, b, b), (q, )
0
1
2
3
4
5
6
7
unread input
aabbaa
aabbaa
aabbaa
abbaa
abbaa
bbaa
bbaa
bbaa
baa
baa
aa
aa
a
S
SB
S aSa
B
B bB
stack
S
aSa
Sa
aSaa
Saa
Baa
bBaa
Baa
bBaa
Baa
aa
a
Another Example
L = {anbmcpdq : m + n = p + q}
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(p, , ), (q, S)
(q, , S), (q, aSd)
(q, , S), (q,T)
(q, , S), (q,U)
(q, , T), (q, aTc)
(q, , T), (q, V)
(q, , U), (q, bUd)
(q, , U), (q, V)
(q, , V), (q, bVc
(q, , V), (q, )
(q, a, a), (q, )
(q, b, b), (q, )
(q, c, c), (q, )
(q, d, d), (q, )
0
1
2
3
4
5
6
7
8
9
10
11
12
13
S aSd
ST
SU
T aTc
TV
U bUd
UV
V bVc
V
input = a a b c d d
The Other WayBuild a PDA Directly

L = {anbmcpdq : m + n = p + q}
(1)
(2)
(3)
(4)
(5)
S aSd
ST
SU
T aTc
TV
U bUd
UV
V bVc
V
(6)
(7)
(8)
(9)
a//a
b//a
c/a/
b//a
1
2
//
d/a/
c/a/
d/a/
3
//
4
//
input = a a b c d d
Lecture Notes 15
Notice Nondeterminism
Machines constructed with the algorithm are often nondeterministic, even when they needn't be. This happens even with trivial
languages.
Example: L = anbn
A grammar for L is:
A machine M for L is:

(0) ((p, , ), (q, S))
(1) ((q, , S), (q, aSb))
(2) ((q, , S), (q, ))
(3) ((q, a, a), (q, ))
(4) ((q, b, b), (q, ))
[1] S aSb
[2] S
But transitions 1 and 2 make M nondeterministic.

A nondeterministic transition group is a set of two or more transitions out of the same state that can fire on the same
configuration. A PDA is nondeterministic if it has any nondeterministic transition groups.
A directly constructed machine for L:
Going The Other Way

Lemma: If a language is accepted by a pushdown automaton, it is a context-free language (i.e., it can be described by a contextfree grammar).
Proof (by construction)
Example: L = {wcwR : w {a, b}*}
a//a
contains:
((s, a, ), (s, a))
((s, b, ), (s, b))
((s, c, ), (f, ))
((f, a, a), (f, ))
((f, b, b), (f, ))
a/a/
c//
s
b//b
b/b/
M = ({s, f}, {a, b, c}, {a, b}, , s,{f}), where:

First Step: Make M Simple
A PDA M is simple iff:
1. there are no transitions into the start state, and
2. whenever ((q, x, ), (p, ) is a transition of M and q is not the start state, then , and || 2.
Step 1: Add s' and f':
a//a
//Z
s'
a/a/
/Z/
c//
s
b//b
f'
b/b/
Step 2:
(1)
Assure that || 1.
(2)
Assure that || 2.
(3)
Assure that || = 1.
Lecture Notes 15
Making M Simple
a//a
c//
//Z
s'
a/a/
/Z/
b//b
f'
b/b/
M = ({s, f, s', f'}, {a, b, c}, {a, b, Z}, , s',{f'}), =

((s', , ), (s, Z))
((s, a, ), (s, a))
((s, a, Z), (s, aZ))
((s, a, a), (s, aa))
((s, a, b), (s, ab))
((s, b, ), (s, b))
((s, b, Z), (s, bZ))
((s, b, a), (s, ba))
((s, b, b), (s, bb))
((s, c, ), (f, ))
((s, c, Z), (f, Z))
((s, c, a), (f, a))
((s, c, b), (f, b))
((f, a, a), (f, ))
((f, a, a), (f, ))
((f, b, b), (f, ))
((f, b, b), (f, ))
((f, , Z), (f', ))
Second Step - Creating the Productions
The basic idea -- simulate a leftmost derivation of M on any input string.
Example:
abcba
S [1]
<s, Z, f'> [2]
a
<s, a, f> [4]

b
<s, b, f> [5]

c
<f, b, f> [7]

b
<f, Z, f'> [8]

<f, a, f> [6]
a
<f, , f> [9]
<f, , f> [9]
<f', , f'> [10]
If the nonterminal <s1, X, s2> * w, then the PDA starts in state s1 with (at least) X on the stack and after consuming w and
popping the X off the stack, it ends up in state s2.
Start with the rule:
S <s, Z, f> where s is the start state, f is the (introduced) final state and Z is the stack bottom symbol.
Transitions ((s1, a, X), (s2, YX)) become a set of rules:
<s1, X, q> a <s2, Y, r> <r, X, q> for a {}, q,r K
Transitions ((s1, a, X), (s2, Y)) becomes a set of rules:
<s1, X, q> a <s2, Y, q> for a {}, q K
Transitions ((s1, a, X), (s2, )) become a rule:
for a {}
<s1, X, s2> a
Lecture Notes 15
Creating Productions from Transitions

((s', , ), (s, Z))
((s, a, Z), (s, aZ))
((s, a, a), (s, aa))

((s, a, b), (s, ab))
((s, b, Z), (s, bZ))
((s, b, a), (s, ba))
((s, b, b), (s, bb))
((s, c, Z), (f, Z))
((s, c, a), (f, a))
((s, c, b), (f, b))
((f, a, a), (f, ))
((f, b, b), (f, ))
((f, , Z), (f', ))
S <s, Z, f'>
[1]
<s, Z, f'> a <s, a, f> <f, Z, f'>

<s, Z, s> a <s, a, f> <f, Z, s>
<s, Z, f> a <s, a, s> <s, Z, f>
<s, Z, s> a <s, a, s> <s, Z, f>
<s, Z, s'> a <s, a, f> <f, Z, s'>
<s, a, f> a <s, a, f> <f, a, f>
<s, a, f> b <s, b, f> <f, a, f>
<s, a, f> c <f, a, f>

<s, b, f> c <f, b, f>
<f, a, f> a <f, , f>
<f, b, f> b <f, , f>
<f, Z, f'> <f', , f'>
<f, , f>
<f' , f'>
[2]
[x]
[x]
[x]
[x]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]

Regular Languages
parse
= NDPDAs
regular exprs.
or
regular grammars
recognize
= DFSAs
Lecture Notes 15

Read K & S 3.7.
Recognizing Context-Free Languages
Two notions of recognition:
(1) Say yes or no, just like with FSMs
(2) Say yes or no, AND

if yes, describe the structure
Now it's time to worry about extracting structure (and doing so efficiently).
Optimizing Context-Free Languages
For regular languages:
Computation = operation of FSMs. So,
Optimization = Operations on FSMs:
Conversion to deterministic FSMs
Minimization of FSMs
For context-free languages:
Computation = operation of parsers. So,
Optimization = Operations on languages
Operations on grammars
Parser design
Before We Start: Operations on Grammars
There are lots of ways to transform grammars so that they are more useful for a particular purpose.
the basic idea:
1. Apply transformation 1 to G to get of undesirable property 1. Show that the language generated by G is unchanged.
2. Apply transformation 2 to G to get rid of undesirable property 2. Show that the language generated by G is unchanged AND
that undesirable property 1 has not been reintroduced.
3. Continue until the grammar is in the desired form.
Examples:
Getting rid of rules (nullable rules)
Getting rid of sets of rules with a common initial terminal, e.g.,
A aB, A aC become A aD, D B | C
Conversion to normal forms
Lecture Notes 16
Normal Forms
If you want to design algorithms, it is often useful to have a limited number of input forms that you have to deal with.
Normal forms are designed to do just that. Various ones have been developed for various purposes.
Examples:
Clause form for logical expressions to be used in resolution theorem proving

Disjunctive normal form for database queries so that they can be entered in a query by example grid.
Various normal forms for grammars to support specific parsing techniques.
Clause Form for Logical Expressions
x : [Roman(x) know(x, Marcus)] [hate(x, Caesar) (y : z : hate(y, z) thinkcrazy(x, y))]

becomes
Roman(x) know(x, Marcus) hate(x, Caesar) hate(y, z) thinkcrazy(x, z)
Disjunctive Normal Form for Queries

(category = fruit or category = vegetable)
and
(supplier = A or supplier = B)
becomes
(category = fruit and supplier = A)
(category = fruit and supplier = B)
(category = vegetable and supplier = A)
(category = vegetable and supplier = B)
Category
fruit
fruit
vegetable
vegetable
or
or
or
Supplier
Price
A
B
A
B
Normal Forms for Grammars
Two of the most common are:
Chomsky Normal Form, in which all rules are of one of the following two forms:
X a, where a , or
X BC, where B and C are nonterminals in G
Greibach Normal Form, in which all rules are of the following form:
X a , where a and is a (possibly empty) string of nonterminals
If L is a context-free language that does not contain , then if G is a grammar for L, G can be rewritten into both of these normal
forms.
Lecture Notes 16
What Are Normal Forms Good For?

Examples:
Chomsky Normal Form:
X a, where a , or
X BC, where B and C are nonterminals in G
The branching factor is precisely 2. Tree building algorithms can take advantage of that.
Greibach Normal Form

X a , where a and is a (possibly empty) string of nonterminals
Precisely one nonterminal is generated for each rule application. This means that we can put a bound on the number of rule
applications in any successful derivation.
Conversion to Chomsky Normal Form
Let G be a grammar for the context-free language L where L.
We construct G', an equivalent grammar in Chomsky Normal Form by:
0.
Initially, let G' = G.
1.
Remove from G' all productions:
1.1.
If there is a rule A B and B is nullable, add the rule A and delete the rule B .
Example:
S aA
A B | CD
B
Ba
C BD
Db
D
Remove from G' all unit productions (rules of the form A B, where B is a nonterminal):
2.1.
Remove from G' all unit productions of the form A A.
2.2.
For all nonterminals A, find all nonterminals B such that A * B, A B.
2.3.
Create G'' and add to it all rules in G' that are not unit productions.
2.4.
For all A and B satisfying 3.2, add to G''
A y1 | y2 | where B y1 | y2 | is in G".
2.5.
Set G' to G''.
Example:
Aa
AB
A EF
BA
B CD
BC
C ab
At this point, all rules whose right hand sides have length 1 are in Chomsky Normal Form.
2.
Lecture Notes 16

3.
Remove from G' all productions P whose right hand sides have length greater than 1 and include a terminal (e.g., A
aB or A BaC):
3.1.
Create a new nonterminal Ta for each terminal a in .
3.2.
Modify each production P by substituting Ta for each terminal a.
3.3.
Add to G', for each Ta, the rule Ta a
Example:
A aB
A BaC
A BbC
Ta a
Tb b
4.
Remove from G' all productions P whose right hand sides have length greater than 2 (e.g., A BCDE)
4.1.
For each P of the form A N1N2N3N4Nn, n > 2, create new nonterminals M2, M3, Mn-1.
4.2.
Replace P with the rule A N1M2.
4.3.
Add the rules M2 N2M3, M3 N3M4, Mn-1 Nn-1Nn
Example:
A BCDE
(n = 4)
A BM2
M2 C M3
M3 DE
Lecture Notes 16
Top Down Parsing

Read K & S 3.8.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Parsing, Sections 1 and 2.
Do Homework 15.
Parsing
Two basic approaches:
Top Down
E
E
E
E
.
.
.
id
Bottom Up
E
E
F
id + id
id + id
id
id
A Simple Parsing Example

A simple top-down parser for arithmetic expressions, given the grammar
EE+T
ET
TT*F
TF
F (E)
F id
F id(E)
[1]
[2]
[3]
[4]
[5]
[6]
[7]
A PDA that does a top down parse:

(0)
(1)
(2)
(3)
(4)
(5)
(6)
(1, , ), (2, E)
(2, , E), (2, E+T)
(2, , E), (2, T)
(2, , T), (2, T*F)
(2, , T), (2, F)
(2, , F), (2, (E) )
(2, , F), (2, id)
Lecture Notes 17
(7) (2, , F), (2, id(E))

(8) (2, id, id), (2, )
(9) (2, (, ( ), (2, )
(10) (2, ), ) ), (2, )
(11) (2, +, +), (2, )
(12) (2, *, *), (2, )
Top Down Parsing
How Does It Work?

Example:
id + id * id(id)
Stack:
E
What Does It Produce?
The leftmost derivation of the string. Why?
E E + T T + T F + T id + T
id + T * F id + F * F id + id * F
id + id * id(E) id + id * id(T)
id + id * id(F) id + id * id(id)
E
E
id
id
F
id
( E )
T
F
id
But the Process Isn't Deterministic

(0) (1, , ), (2, E)
(1) (2, , E), (2, E+T)
(2) (2, , E), (2, T)
(3) (2, , T), (2, T*F)
(4) (2, , T), (2, F)
(5) (2, , F), (2, (E) )
(6) (2, , F), (2, id)
(7) (2, , F), (2, id(E))
(8) (2, id, id), (2, )
(9) (2, (, ( ), (2, )
(10) (2, ), ) ), (2, )
(11) (2, +, +), (2, )
(12) (2, *, *), (2, )
Lecture Notes 17
nondeterministic
nondeterministic
nondeterministic
Top Down Parsing
Is Nondeterminism A Problem?
Yes.
In the case of regular languages, we could cope with nondeterminism in either of two ways:
Create an equivalent deterministic recognizer (FSM)
Simulate the nondeterministic FSM in a number of steps that was still linear in the length of the input string.
For context-free languages, however,
The best straightforward general algorithm for recognizing a string is O(n3) and the best (very complicated) algorithm is
based on a reduction to matrix multiplication, which may get close to O(n2).
We'd really like to find a deterministic parsing algorithm that could run in time proportional to the length of the input string.
Is It Possible to Eliminate Nondeterminism?
In this case: Yes
In general: No
Some definitions:
A PDA M is deterministic if it has no two transitions such that for some (state, input, stack sequence) the two transitions
could both be taken.
A language L is deterministic context-free if L$ = L(M) for some deterministic PDA M.
Theorem: The class of deterministic context-free languages is a proper subset of the class of context-free languages.
Proof: Later.
Adding a Terminator to the Language
We define the class of deterministic context-free languages with respect to a terminator ($) because we want that class to be as
large as possible.
Theorem: Every deterministic CFL (as just defined) is a context-free language.
Proof:
Without the terminator ($), many seemingly deterministic cfls aren't. Example:
a* {anbn : n> 0}
Possible Solutions to the Nondeterminism Problem
1)
2)
3)
Modify the language
Add a terminator $
Change the parsing algorithm
Modify the grammar
Lecture Notes 17
Top Down Parsing
Modifying the Parsing Algorithm

What if we add the ability to look one character ahead in the input string?
Example:
id + id * id(id)
E E + T T + T F + T id + T
id + T * F id + F * F id + id * F
Considering transitions:
(5) (2, , F), (2, (E) )
(6) (2, , F), (2, id)
(7) (2, , F), (2, id(E))
If we add to the state an indication of what character is next, we have:
(5) (2, (, , F), (2, (E) )
(6) (2, id, , F), (2, id)
(7) (2, id, , F), (2, id(E))
Modifying the Language
So we've solved part of the problem. But what do we do when we come to the end of the input? What will be the state indicator
then?
The solution is to modify the language. Instead of building a machine to accept L, we will build a machine to accept L$.
Using Lookahead
EE+T
ET
TT*F
TF
F (E)
F id
F id(E)
[1]
[2]
[3]
[4]
[5]
[6]
[7]
(0) (1, , ), (2, E))

(1) (2, , E), (2, E+T)
(2) (2, , E), (2, T)
(3) (2, , T), (2, T*F)
(4) (2, , T), (2, F)
(5) (2, (, , F), (2, (E) )
(6) (2, id, , F), (2, id)
(7) (2, id, , F),(2, id(E))
(8) (2, id, id), (2, )
(9) (2, (, ( ), (2, )
(10) (2, ), ) ), (2, )
(11) (2, +, +), (2, )
(12) (2, *, *), (2, )
For now, we'll ignore the issue of when we read the lookahead character and the fact that we only care about it if the top symbol
on the stack is F.
1)
2)
3)
Modify the language
Add a terminator $
Add one character look ahead

Modify the grammar
Lecture Notes 17
Top Down Parsing
Modifying the Grammar

Getting rid of identical first symbols:
[6]
[7]
F id
F id(E)
(6) (2, id, , F),(2, id)

(7) (2, id, , F),(2, id(E))
F id A
A
A (E)
(6') (2, id, , F), (2, id A)

(7') (2, (, , A), (2, )
(8') (2, (, , A), (2, (E))
Replace with:
[6']
[7']
[8']
The general rule for left factoring:

A 1
A 2
A n
are rules with and n 2, then replace them by the rules:
A A'
A' 1
A' 2
A' n
Whenever
Modifying the Grammar

Getting rid of left recursion:
[1]
[2]
EE+T
ET
(1) (2, , E), (2, E+T)

(2) (2, , E), (2, T)
The problem:
E
E
E
Replace with:
[1]
[2]
[3]
Lecture Notes 17
E T E'
E' + T E'
E'
(1) (2, , E), (2, T E')

(2) (2, , E'), (2, + T E')
(3) (2, , E'), (2, )
Top Down Parsing
Getting Rid of Left Recursion

The general rule for eliminating left recursion:
If G contains the following rules:
A A1
A A2
A A3
A An
A 1 (where 's do not start with A)
A 2
A m
Replace them with:

A' 1A'
A' 2A'
A' 3A'
A' nA'
A'
A 1A'
A 2A'
A mA'
and n > 0, then

I.
II.
III.
Modify the language

A.
Add a terminator $
A.
Modify the grammar
A.
Left factor
B.
Get rid of left recursion
LL(k) Languages
We have just offered heuristic rules for getting rid of some nondeterminism.
We know that not all context-free languages are deterministic, so there are some languages for which these rules won't work.
We define a grammar to be LL(k) if it is possible to decide what production to apply by looking ahead at most k symbols in the
input string.
Specifically, a grammar G is LL(1) iff, whenever
A | are two rules in G:
1. For no terminal a do and derive strings beginning with a.
2. At most one of | can derive .
3. If * , then does not derive any strings beginning with a terminal in FOLLOW(A), defined to be the set of terminals
that can immediately follow A in some sentential form.
We define a language to be LL(k) if there exists an LL(k) grammar for it.
Lecture Notes 17
Top Down Parsing
Implementing an LL(1) Parser

If a language L has an LL(1) grammar, then we can build a deterministic LL(1) parser for it. Such a parser scans the input Left to
right and builds a Leftmost derivation.
The heart of an LL(1) parser is the parsing table, which tells it which production to apply at each step.
For example, here is the parsing table for our revised grammar of arithmetic expressions without function calls:
id
+
*
(
)
V\
E
ETE'
ETE'
E'
E'+TE'
E'
T
TFT'
TFT'
T'
T'
T'*FT'
T'
F
Fid
F(E)
Given input id + id * id, the first few moves of this parser will be:
E
ETE'
TE'
TFT'
FT'E'
Fid
idT'E'
T'E'
T'
E'
$
E'
T'
id + id * id$
id + id * id$
id + id * id$
id + id * id$
+ id * id$
+ id * id$
But What If We Need a Language That Isn't LL(1)?

Example:
ST if C then ST else ST
ST if C then ST
We can apply left factoring to yield:
ST if C then ST S'
S' else ST |
Now we've procrastinated the decision. But the language is still ambiguous. What if the input is
if C1 then if C2 then ST1 else ST2
Which bracketing (rule) should we choose?
A common practice is to choose
S' else ST
We can force this if we create the parsing table by hand.

I.
II.
III.
Modify the language

A.
Add a terminator $
A.
B.
Use a parsing table
C.
Tailor parsing table entries by hand
Modify the grammar
A.
Left factor
B.
Lecture Notes 17
Top Down Parsing
The Price We Pay

Old Grammar
[1]
EE+T
[2]
ET
[3]
[4]
TT*F
TF
[5]
[6]
[7]
F (E)
F id
F id(E)
New Grammar
E TE'
E' +TE'
E'
T FT'
T' *FT'
T'
F (E)
F idA
A
A (E)
input = id + id + id
E
T
F
id
E'
T'
T
F
E'
T'
id
F
id
E'
T'

Regular Languages
= NDPDAs
parse
find deterministic grammars
find efficient parsers
regular exprs.
or
regular grammars
= DFSAs
recognize
minimize FSAs
Lecture Notes 17
Top Down Parsing
Bottom Up Parsing
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Parsing, Section 3.
Bottom Up Parsing
An Example:
EE+T
ET
TT*F
TF
F (E)
F id
[1]
[2]
[3]
[4]
[5]
[6]
id
id
id
Creating a Bottom Up PDA

There are two basic actions:
1. Shift an input symbol onto the stack
2. Reduce a string of stack symbols to a nonterminal
M will be:
$/S/
p
So, to construct M from a grammar G, we need the following transitions:

(1) The shift transitions:
((p, a, ), (p, a)), for each a
(2) The reduce transitions:
((p, , R), (p, A)), for each rule A in G.
(3) The finish up transition (accept):
((p, $, S), (q, ))
(This is the bottom-up CFG to PDA conversion algorithm.)
Lecture Notes 18
Bottom Up Parsing
(p, a, ), (p, a) for each a

(p, , T + E), (p, E)
(p, , T), (p, E)
(p, , F * T), (p, T)
(p, , F), (p, T)
(p, , )E( ), (p, F)
(p, , id), (p, F)
(p, $, E), (q, )
0
1
2
3
4
5
6
7
trans (action)
state
p
0 (shift)
p
6 (reduce F id)
p
4 (reduce T F)
p
2 (reduce E T)
p
0 (shift)
p
0 (shift)
p
6 (reduce F id)
p
4 (reduce T F)
p
0 (shift)
p
0 (shift)
p
6 (reduce F id)
p
3 (reduce T T * F) p
1 (reduce E E + T) p
7 (accept)
q
M for Expressions
unread input
id + id * id$
+ id * id$
+ id * id$
+ id * id$
+ id * id$
id * id$
* id$
* id$
* id$
id$
$
$
$
$
$
stack
id
F
T
E
+E
id+E
F+E
T+E (could also reduce)
*T+E
id*T+E
F*T+E (could also reduce T F)
T+E
E
The Parse Tree

E
id
id
id $
Producing the Rightmost Derivation

We can reconstruct the derivation that we found by reading the results of the parse bottom to top, producing:
E
E+ T
E+ T* F
E+ T* id
E+ F* id
E+ id* id
T+ id*id
F+ id*id
id+ id*id
This is exactly the rightmost derivation of the input string.
Lecture Notes 18
Bottom Up Parsing

1)
Modify the language
Add a terminator $
2)
Use a parsing table
Tailor parsing table entries by hand
Switch to a bottom-up parser
3)
Modify the grammar
Left factor

Solving the Shift vs. Reduce Ambiguity With a Precedence Relation
Let's return to the problem of deciding when to shift and when to reduce (as in our example).
We chose, correctly, to shift * onto the stack, instead of reducing
T+E
to E.
This corresponds to knowing that + has low precedence, so if there are any other operations, we need to do them first.
Solution:
1. Add a one character lookahead capability.
2. Define the precedence relation
P
( V
top
stack
symbol
{ $} )
next
input
symbol
If (a,b) is in P, we reduce (without consuming the input) . Otherwise we shift (consuming the input).
How Does It Work?
We're reconstructing rightmost derivations backwards. So suppose a rightmost derivation contains
abx
corresponding to a rule A a and not some rule X ab

Abx
*
S
We want to undo rule A. So if the top of the stack is
a
and the next input character is b, we reduce

now, before we put the b on the stack.
To make this happen, we put (a, b) in P. That means we'll try to reduce if a is on top of the stack and b is the next character. We
will actually succeed if the next part of the stack is .
Lecture Notes 18
Bottom Up Parsing
Example
T*F
T
*
E
corresponding to a rule TT*F

Input: id * id * id
We want to undo rule T. So if the top of the stack is

F
*
and the next input character is anything legal, we reduce.
T
The precedence relation for expressions:
V\
(
)
id
+
*
E
T
F
id
A Different Example
E+T
*
E
corresponding to a rule EE+T
We want to undo rule E if the input is

or
but not
E+T$
E + T + id
E + T * id
The top of the stack is

T
+
E
The precedence relation for expressions:
V\
(
)
id
+
*
E
T
F
Lecture Notes 18
id
Bottom Up Parsing
What About If Then Else?

ST if C then ST
What if the input is
if
C1
then
if
C2
then
ST1
1
else
ST2
2

We don't put (ST, else) in the precedence relation, so we will not reduce at 1. At 2, we reduce:
ST2
else
ST1
then
C2
if
then
C1
if
2
1
Resolving Reduce vs. Reduce Ambiguities

(p, a, ), (p, a) for each a
(p, , T + E), (p, E)
(p, , T), (p, E)
(p, , F * T), (p, T)
(p, , F), (p, T)
(p, , ) E ( ), (p, F)
(p, , id), (p, F)
(p, $, E), (q, )
0
1
2
3
4
5
6
7
trans (action)
state
p
0 (shift)
p
6 (reduce F id)
p
4 (reduce T F)
p
2 (reduce E T)
p
0 (shift)
p
0 (shift)
p
6 (reduce F id)
p
4 (reduce T F)
p
0 (shift)
p
0 (shift)
p
6 (reduce F id)
p
3 (reduce T T * F) p
1 (reduce E E + T) p
7 (accept)
q
Lecture Notes 18
unread input
id + id * id$
+ id * id$
+ id * id$
+ id * id$
+ id * id$
id * id$
* id$
* id$
* id$
id$
$
$
$
$
$
Bottom Up Parsing
stack
id
F
T
E
+E
id+E
F+E
T+E (could also reduce)
*T+E
id*T+E
F*T+E (could also reduce T F)
T+E
E
The Longest Prefix Heuristic

A simple to implement heuristic rule, when faced with competing reductions, is:
Choose the longest possible stack string to reduce.
Example:
Suppose the stack has
F * T
+ E
We call grammars that become unambiguous with the addition of a precedence relation and the longest string reduction heuristic
weak precedence grammars.
Possible Solutions to the Nondeterminism Problem in a Bottom Up Parser
1)
Modify the language
Add a terminator $
2)
Add one character lookahead
Use a precedence table
Add the longest first heuristic for reduction
Use an LR parser
3)
Modify the grammar

LR Parsers
LR parsers scan each input Left to right and build a Rightmost derivation. They operate bottom up and deterministically using a
parsing table derived from a grammar for the language to be recognized.
A grammar that can be parsed by an LR parser examining up to k input symbols on each move is an LR(k) grammar. Practical
LR parsers set k to 1.
An LALR ( or Look Ahead LR) parser is a specific kind of LR parser that has two desirable properties:
The parsing table is not huge.
Most useful languages can be parsed.
Another big reason to use an LALR parser:
There are automatic tools that will construct the required parsing table from a grammar and some optional additional
information.
We will be using such a tool:
Lecture Notes 18
yacc
Bottom Up Parsing
How an LR Parser Works

Input String
state
state
state
state
Lexical Analyzer
Output Token
Stack
Parsing Table
In simple cases, think of the "states" on the stack as corresponding to either terminal or nonterminal characters.
In more complicated cases, the states contain more information: they encode both the top stack symbol and some facts about
lower objects in the stack. This information is used to determine which action to take in situations that would otherwise be
ambiguous.
The Actions the Parser Can Take
At each step of its operation, an LR parser does the following two things:
1)
2)
Based on its current state, it decides whether it needs a lookahead token. If it does, it gets one.
Based on its current state and the lookahead token if there is one, it chooses one of four possible actions:
Shift the lookahead token onto the stack and clear the lookahead token.
Reduce the top elements of the stack according to some rule of the grammar.
Detect the end of the input and accept the input string.
Detect an error in the input.
Lecture Notes 18
Bottom Up Parsing
A Simple Example
0: S rhyme $end ;
1: rhyme sound place ;
2: sound DING DONG ;
3: place DELL
state 0 (empty)
$accept : _rhyme $end
DING shift 3
. error
rhyme goto 1
sound goto 2
state 1 (rhyme)
$accept : rhyme_$end
$end accept
. error
state 2 (sound)
rhyme : sound_place
DELL shift 5
. error
place goto 4
state 3 (DING)
sound : DING_DONG
DONG shift 6
. error
state 4 (place)
rhyme : sound place_ (1)
. reduce 1
the rule this came from

state 3
current position of input
if none of the others match
push state 2
if we see EOF, accept
by rule 1
state 5 (DELL)
place : DELL_ (3)
. reduce 3
state 6 (DONG)
sound : DING DONG_ (2)
. reduce 2
When the States Are More than Just Stack Symbols

[1] <stmt> procname ( <paramlist>)
[2] <stmt> <exp> := <exp>
[3] <paramlist> <paramlist>, <param> | <param>
[4 ] <param> id
[5] <exp> arrayname (<subscriptlist>)
[6] <subscriptlist> <subscriptlist>, | 
[7] id
Example:
id
(
procname
procname ( id)
Should we reduce id by rule 4 or rule 7?

procid
proc(
procname
The parsing table can get complicated as we incorporate more stack history into the states.
Lecture Notes 18
Bottom Up Parsing
The Language Interpretation Problem:

Input: -(17 * 83.56) + 72 / 12
Output: -1414.52
Input: -(17 * 83.56) + 72 / 12
Compute the answer
Output: -1414.52
Input: -(17 * 83.56) + 72 / 12
Parse the input
A tree of actions, whose structure corresponds to the structure of the input.
Compute the answer
Output: -1414.52
Lecture Notes 18
Bottom Up Parsing

Input: -(17 * 83.56) + 72 / 12
1
Lexical analysis of the input
A string of input tokens, corresponding to the primitive objects of which the input is composed:
-(id * id) + id / id
2
Parse the input
A tree of actions, whose structure corresponds to the structure of the input.
Compute the answer

Output: -1414.52
yacc and lex
Lexical analysis of the input
1
2
Parse the input
Where do the procedures to do these things come from?

regular expressions that describe patterns
lex
1
lexical analyzer
grammar rules and other facts about the language
yacc
parser
Lecture Notes 18
Bottom Up Parsing
10
lex
The input to lex:
definitions
%%
rules
%%
user routines
All strings that are not matched by any rule are simply copied to the output.
Rules:
[ \t]+;
get rid of blanks and tabs
[A-Za-z][A-Za-z0-9]*
return(ID);
find identifiers
[0-9]+
sscanf(yytext, "%d", &yylval);

return (INTEGER); }
return INTEGER and put the value in yylval
How Does lex Deal with Ambiguity in Rules?

lex invokes two disambiguating rules:
1.
2.
The longest match is prefered.

Among rules that matched the same number of characters, the rule given first is preferred.
Example:
integer
[a-z]+
input:
action 1
action 2
integers
integer
take action 2
take action 1
yacc
(Yet Another Compiler Compiler)
The input to yacc:

declarations
%%
rules
%%
#include "lex.yy.c"
any other programs
This structure means that lex.yy.c will be compiled as part of y.tab.c, so it will have access to the same token names.
Declarations:
%token name1 name2
Rules:
V
V
V
Lecture Notes 18
:a b c
:a b c
:a b c
{action}
{$$ = $2}
returns the value of b
Bottom Up Parsing
11
Example
Input to yacc:
%token DING DONG DELL
%%
rhyme :
sound place ;
sound :
DING DONG ;
place :
DELL
%%
#include "lex.yy.c"
state 3 (DING)
sound : DING_DONG
DONG shift 6
. error
state 4 (place)
rhyme : sound place_ (1)
. reduce 1
state 5 (DELL)
place : DELL_ (3)
. reduce 3
state 6 (DONG)
sound : DING DONG_ (2)
. reduce 2
state 0 (empty)
$accept : _rhyme $end
DING shift 3
. error
rhyme goto 1
sound goto 2
state 1 (rhyme)
$accept : rhyme_$end
$end accept
. error
state 2 (sound)
rhyme : sound_place
DELL shift 5
. error
place goto 4
How Does yacc Deal with Ambiguity in Grammars?

The parser table that yacc creates represents some decision about what to do if there is ambiguity in the input grammar rules.
How does yacc make those decisions? By default, yacc invokes two disambiguating rules:
1. In the case of a shift/reduce conflict, shift.
2. In the case of a reduce/reduce conflict, reduce by the earlier grammar rule.
yacc tells you when it has had to invoke these rules.
Shift/Reduce Conflicts - If Then Else
ST if C then ST
What if the input is
if
C1
then
if
C2
then
ST1
1
else
ST2
2

yacc will choose to shift rather than reduce.
ST2
else
ST1
then
C2
if
then
C1
if
Lecture Notes 18
2
1
Bottom Up Parsing
12
Shift/Reduce Conflicts - Left Associativity

We know that we can force left associativity by writing it into our grammars.
Example:
EE+T
ET
T id
E
E
E
T
T
T
id
id
id
What does the shift rather than reduce heuristic if we instead write:
EE+E
E id
id
id
id
Shift/Reduce Conflicts - Operator Precedence

Recall the problem:
input:
id + id * id
T
+
E
Should we reduce or shift on * ?
The "always shift" rule solves this problem.

But what about:
id * id + id
T
*
E
Should we reduce or shift on + ?

This time, if we shift, we'll fail.
One solution was the precedence table, derived from an unambiguous grammar, which can be encoded into the parsing table of an
LR parser, since it tells us what to do for each top-of-stack, input character combination.
Operator Precedence
We know that we can write an unambiguous grammar for arithmetic expressions that gets the precedence right. But it turns out
that we can build a faster parser if we instead write:
E E + E | E * E | (E) | id
And, in addition, we specify operator precedence. In yacc, we specify associativity (since we might not always want left) and
precedence using statements in the declaration section of our grammar:
%left '+' '-'
%left '*' '/'
Operators on the first line have lower precedence than operators on the second line, and so forth.
Lecture Notes 18
Bottom Up Parsing
13
Reduce/Reduce Conflicts
Recall:
2.
In the case of a reduce/reduce conflict, reduce by the earlier grammar rule.
This can easily be used to simulate the longest prefix heuristic, "Choose the longest possible stack string to reduce."
EE+T
ET
TT*F
TF
F (E)
F id
[1]
[2]
[3]
[4]
[5]
[6]
Generating an Executable System

Step 1: Create the input to lex and the input to yacc.
Step 2:
$ lex ourlex.l
$ yacc ouryacc.y
$ cc -o ourprog y.tab.c -ly -ll
creates lex.yy.c
creates y.tab.c
actually compiles y.tab.c and lex.yy.c, which is included.
-ly links the yacc library, which includes main and yyerror.
-ll links the lex library
Step 3: Run the program

$ ourprog
Runtime Communication Between lex and yacc-Generated Modules
Parser
ask
for a
token
read the value of the token
return
a
token
yylval
Lexical Analyer
set the value of the token
Summary
Efficient parsers for languages with the complexity of a typical programming language or command line interface:
Make use of special purpose constructs, like precedence, that are very important in the target languages.
May need complex transition functions to capture all the relevant history in the stack.
Use heuristic rules, like shift instead of reduce, that have been shown to work most of the time.
Would be very difficult to construct by hand (as a result of all of the above).
Can easily be built using a tool like yacc.
Lecture Notes 18
Bottom Up Parsing
14
Languages That Are and Are Not Context-Free

Read K & S 3.5, 3.6, 3.7.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Closure Properties of Context-Free
Languages
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: The Context-Free Pumping Lemma.
Do Homework 16.
Deciding Whether a Language is Context-Free
Theorem: There exist languages that are not context-free.
Proof:
(1) There are a countably infinite number of context-free languages. This true because every description of a context-free
language is of finite length, so there are a countably infinite number of such descriptions.
(2) There are an uncountable number of languages.
Thus there are more languages than there are context-free languages.
So there must exist some languages that are not context-free.
Example: {anbncn}
Showing that a Language is Context-Free
Techniques for showing that a language L is context-free:
1.
2.
3.
Exhibit a context-free grammar for L.

Exhibit a PDA for L.
Use the closure properties of context-free languages.
Unfortunately, these are weaker than they are for regular languages.
The Context-Free Languages are Closed Under Union
Let G1 = (V1, 1, R1, S1) and
G2 = (V2, 2, R2, S2)
Assume that G1 and G2 have disjoint sets of nonterminals, not including S.
Let L = L(G1) L(G2)
We can show that L is context-free by exhibiting a CFG for it:
The Context-Free Languages are Closed Under Concatenation

Let G1 = (V1, 1, R1, S1) and
G2 = (V2, 2, R2, S2)
Assume that G1 and G2 have disjoint sets of nonterminals, not including S.
Let L = L(G1) L(G2)
Lecture Notes 19
Languages That Are and Are Not Context Free
The Context-Free Languages are Closed Under Kleene Star

Let G1 = (V1, 1, R1, S1)
Assume that G1 does not have the nonterminal S.
Let L = L(G1)*
What About Intersection and Complement?

We know that they share a fate, since
L1 L2 = L1 L2
But what fate?
We proved closure for regular languages two different ways. Can we use either of them here:
1. Given a deterministic automaton for L, construct an automaton for its complement. Argue that, if closed under complement
and union, must be closed under intersection.
2. Given automata for L1 and L2, construct a new automaton for L1 L2 by simulating the parallel operation of the two original
machines, using states that are the Cartesian product of the sets of states of the two original machines.
More on this later.
The Intersection of a Context-Free Language and a Regular Language is Context-Free
L = L(M1), a PDA = (K1, , 1, 1, s1, F1)
R = L(M2), a deterministic FSA = (K2, , , s2, F2)
We construct a new PDA, M3, that accepts L R by simulating the parallel execution of M1 and M2.
M = (K1 K2, , 1, , (s1, s2), F1 F2)
Insert into :
a, ), (p1,
)) in 1,
For each rule ((q1,
a,
p2)
in ,
and each rule (q2,
(((q1, q2), a, ), ((p1, p2), ))
, ), (p1,
) in 1,
For each rule ((q1,
and each state q2
in K2,
(((q1, q2), , ), ((p1, q2), ))
This works because: we can get away with only one stack.
Lecture Notes 19
Example
L=
a nb n
(aa)*(bb)*
b/a/
a//a
b/a/
a
b
b
3
4
b
((A, a, ), (A, a))

((A, b, a), (B, ))
((B, b, a), (B, ))
(1, a, 2)
(1, b, 3)
(2, a, 1)
(3, b, 4)
(4, b, 3)
A PDA for L:
Dont Try to Use Closure Backwards
One Closure Theorem:
If L1 and L2 are context free, then so is
L3 = L1 L2.
But what if L3 and L1 are context free? What can we say about L2?
L3 = L1 L2.
Example:
anbnc* = anbnc* anbncn
The Context-Free Pumping Lemma

This time we use parse trees, not automata as the basis for our argument.
S
If L is a context-free language, and if w is a string in L where |w| > K, for some value of K, then w can be rewritten as uvxyz,
where |vy| > 0 and |vxy| M, for some value of M.
uxz, uvxyz, uvvxyyz, uvvvxyyyz, etc. (i.e., uvnxynz, for n 0) are all in L.
Lecture Notes 19
Some Tree Basics

root
height
nodes
leaves
yield
Theorem: The length of the yield of any tree T with height H and branching factor (fanout) B is BH.
Proof: By induction on H. If H is 1, then just a single rule applies. By definition of fanout, the longest yield is B.
Assume true for H = n.
Consider a tree with H = n + 1. It consists of a root, and some number of subtrees, each of which is of height n (so induction
hypothesis holds) and yield Bn. The number of subtrees B. So the yield must be B(Bn) or Bn+1.
What Is K?
S
Let T be the number of nonterminals in G.

If there is a tree of height > T, then some nonterminal occurs more than once on some path. If it does, we can pump its yield.
Since a tree of height = T can produce only strings of length BT, any string of length > BT must have a repeated nonterminal and
thus be pumpable.
So K = BT, where T is the number of nonterminals in G and B is the branching factor (fanout).
What is M?
S
Assume that we are considering the bottom most two occurrences of some nonterminal. Then the yield of the upper one is at
most BT+1 (since only one nonterminal repeats).
So M = BT+1.
Lecture Notes 19
The Context-Free Pumping Lemma

Theorem: Let G = (V, , R, S) be a context-free grammar with T nonterminal symbols and fanout B. Then any string w L(G)
where |w| > K (BT) can be rewritten as w = uvxyz in such a way that:
|vy| > 0,
|vxy| M (BT+1), (making this the "strong" form),
for every n 0, uvnxynz is in L(G).
Proof:
Let w be such a string and let T be the parse tree with root labeled S and with yield w that has the smallest number of leaves
among all parse trees with the same root and yield. T has a path of length at least T+1, with a bottommost repeated nonterminal,
which we'll call A. Clearly v and y can be repeated any number of times (including 0). If |vy| = 0, then there would be a tree with
root S and yield w with fewer leaves than T. Finally, |vxy| BT+1.
An Example of Pumping
L = {anbncn : n 0}
Choose w = aibici where i > K/3 (making |w| > K)
S
Unfortunately, we don't know where v and y fall. But there are two possibilities:
1. If vy contains all three symbols, then at least one of v or y must contain two of them. But then uvvxyyz contains at least one
out of order symbol.
2. If vy contains only one or two of the symbols, then uvvxyyz must contain unequal numbers of the symbols.
Using the Strong Pumping Lemma for Context Free Languages
If L is context free, then
There exist K and M (with M K) such that
For all strings w, where |w| > K,
(Since true for all such w, it must be true for any paricular one, so you pick w)
(Hint: describe w in terms of K or M)
there exist u, v, x, y, z such that w = uvxyz and
|vy| > 0, and

|vxy| M, and
for all n 0, uvnxynz is in L.
We need to pick w, then show that there are no values for uvxyz that satisfy all the above criteria. To do that, we just need to
focus on possible values for v and y, the pumpable parts. So we show that all possible picks for v and y violate at least one of
the criteria.
Write out a single string, w (in terms of K or M) Divide w into regions.
For each possibility for v and y (described in terms of the regions defined above), find some value n such that uvnxynz is not in L.
Almost always, the easiest values are 0 (pumping out) or 2 (pumping in). Your value for n may differ for different cases.
Lecture Notes 19
why the resulting string is not in L
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
Convince the reader that there are no other cases.
Q. E. D.
A Pumping Lemma Proof in Full Detail
Proof that L = {anbncn : n 0} is not context free.
Suppose L is context free. The context free pumping lemma applies to L. Let M be the number from the pumping lemma.
Choose w = aMbMcM. Now w L and |w| > M K. From the pumping lemma, for all strings w, where |w| > K, there exist u, v, x,
y, z such that w = uvxyz and |vy| > 0, and |vxy| M, and for all n 0, uvnxynz is in L. There are two main cases:
1. Either v or y contains two or more different types of symbols (a, b or c). In this case, uv2xy2z is not of the form
a*b*c* and hence uv2xy2z L.
2. Neither v nor y contains two or more different types of symbols. In this case, vy may contain at most two types of
symbols. The string uv0xy0z will decrease the count of one or two types of symbols, but not the third, so uv0xy0z L
Cases 1 and 2 cover all the possibilities. Therefore, regardless of how w is partitioned, there is some uvnxynz that is not in L.
Contradiction. Therefore L is not context free.
Note: the underlined parts of the above proof is boilerplate that can be reused. A complete proof should have this text or
something equivalent.
Context-Free Languages Over a Single-Letter Alphabet
Theorem: Any context-free language over a single-letter alphabet is regular.
Examples:
L
L
= {anbn}
= {anan}
= {a2n}
= {w {a}* : |w| is even}
L
L
= {wwR : w {a, b}*}

= {wwR : w {a}*}
= {ww: w {a}*}
= {w {a}* : |w| is even}
L
L
= {anbm : n, m 0 and n m}
= {anam : n, m 0 and n m}
=
Proof: See Parikh's Theorem
Lecture Notes 19
Another Language That Is Not Context Free
L = {an : n 1 is prime}
Two ways to prove that L is not context free:

1. Use the pumping lemma:
Choose a string w = an such that n is prime and n > K.
w = aaaaaaaaaaaaaaaaaaaaaaa
u v x
y z
Let vy = ap and uxz = ar. Then r + kp must be prime for all values of k. This can't be true, as we argued to show that L was not
regular.
2.
|L| = 1. So if L were context free, it would also be regular. But we know that it is not. So it is not context free either.
Using Pumping and Closure

L = {w {a, b, c}* : w has an equal number of a's, b's, and c's}
L is not context free.
Try pumping: Let w = aKbKcK
Now what?
Using Intersection with a Regular Language to Make Pumping Tractable

L = {tt : t {a, b}* }
Let's try pumping:
|w| > K
t
v
What if u is
v is
x is
y is
z is
,
w,
,
w, and
Then all pumping tells us is that
Lecture Notes 19
t
y
tntn is in L.
L = {tt : t {a, b}* }

aMbaMb:
What if we let |w| > M, i.e. choose to pump the string

Now v and y can't be t, since |vxy| M:
t
v
t
y
Suppose |v| = |y|. Now we have to show that repeating them makes the two copies of t different. But we cant.
L = {tt : t {a, b}* }
But let's consider L' = L a*b*a*b*
This time, we let |w| > 2M, and the number of both a's and b's in w >M:
1
2
3
4
aaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbb
t
t
u
v x y
z
Now we use pumping to show that L' is not context free.
First, notice that if either v or y contains both a's and b's, then we immediately violate the rules for L' when we pump.
So now we know that v and y must each fall completely in one of the four marked regions.
L' = {tt : t {a, b}* } a*b*a*b*
|w| > 2M, and the number of both a's and b's in w >M:
1
2
3
4
aaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbb
t
t
u
v x y
z
Consider the combinations of (v, y):
(1,1)
(2,2)
(3,3)
(4,4)
(1,2)
(2,3)
(3,4)
(1,3)
(2,4)
(1,4)
Lecture Notes 19
The Context-Free Languages Are Not Closed Under Intersection

Proof: (by counterexample)
Consider L = {anbncn: n 0}
L is not context-free.
L1 = {anbncm: n, m 0} /* equal a's and b's
L2 = {ambncn: n, m 0} /* equal b's and c's
Let
Both L1 and L2 are context-free.

But L = L1 L2.
So, if the context-free languages were closed under intersection, L would have to be context-free. But it isn't.
The Context-Free Languages Are Not Closed Under Complementation
Proof: (by contradiction)
By definition:
L1 L2 = L1 L2
Since the context-free languages are closed under union, if they were also closed under complementation, they would necessarily
be closed under intersection. But we just showed that they are not. Thus they are not closed under complementation.
The Deterministic Context-Free Languages Are Closed Under Complement
Proof:
Let L be a language such that L$ is accepted by the deterministic PDA M. We construct a deterministic PDA M' to accept (the
complement of L)$, just as we did for FSMs:
1.
2.
3.
4.
5.
6.
Initially, let M' = M.

M' is already deterministic.
Make M' simple. Why?
Complete M' by adding a dead state, if necessary, and adding all required transitions into it, including:
Transitions that are required to assure that for all input, stack combinations some transition can be followed.
If some state q has a transition on (, ) and if it does not later lead to a state that does consume something then
make a transiton on (, ) to the dead state.
Swap final and nonfinal states.
Notice that M is still deterministic.
Lecture Notes 19
An Example of the Construction

L = anbn
M accepts L$ (and is deterministic):

a//a
b/a/
b/a/
$//
$//
Set M = M'. Make M simple.
a/a/aa
a/Z/aZ
//Z
b/a/
0
1
b/a/
$/Z/
2
$/Z/
The Construction, Continued
Add dead state(s) and swap final and nonfinal states:
a/a/aa
a/Z/aZ
b/a/
//Z
b/a/
$/Z/
0
1
2
$/Z/
b/Z/, $/a/
a//, $/a/, b/Z/
4
a//, b//, $//, /a/, /Z/
Issues: 1) Never having the machine die
2) (L$) (L)$
3) Keeping the machine deterministic
Deterministic vs. Nondeterministic Context-Free Languages
Theorem: The class of deterministic context-free languages is a proper subset of the class of context-free languages.
Proof: Consider L = {anbmcp : m n or m p}
L is context free (we have shown a grammar for it).
But L is not deterministic. If it were, then its complement L1 would be deterministic context free, and thus certainly context free.
But then
L2 = L1 a*b*c* (a regular language)
would be context free. But
L2 = {anbncn : n 0}, which we know is not context free.
Thus there exists at least one context-free language that is not deterministic context free.
Note that deterministic context-free languages are not closed under union, intersection, or difference.
Lecture Notes 19
10
Decision Procedures for CFLs & PDAs

Decision Procedures for CFLs
There are decision procedures for the following (G is a CFG):
Deciding whether w L(G).
Deciding whether L(G) = .
Deciding whether L(G) is finite/infinite.
Such decision procedures usually involve conversions to Chomsky Normal Form or Greibach Normal Form. Why?
Theorem: For any context free grammar G, there exists a number n such that:
1. If L(G) , then there exists a w L(G) such that |w| < n.
2. If L(G) is infinite, then there exists w L(G) such that n |w| < 2n.
There are not decision procedures for the following:
Deciding whether L(G) = *.
Deciding whether L(G1) = L(G2).
If we could decide these problems, we could decide the halting problem. (More later.)
Decision Procedures for PDAs
There are decision procedures for the following (M is a PDA):
Deciding whether w L(M).
Deciding whether L(M) = .
Deciding whether L(M) is finite/infinite.
Convert M to its equivalent PDA and use the corresponding CFG decision procedure. Why avoid using PDAs directly?
There are not decision procedures for the following:
Deciding whether L(M) = *.
Deciding whether L(M1) = L(M2).
If we could decide these problems, we could decide the halting problem. (More later.)
Lecture Notes 19
11

Regular Languages
parse
= NDPDAs
parse
find deterministic grammars
find efficient parsers
closed under:
concatenation
union
Kleene star
intersection w/ reg. langs

pumping lemma
deterministic nondeterministic
regular exprs.
or
regular grammars
recognize
= DFSAs
recognize
minimize FSAs
closed under:
concatenation
union
Kleene star
complement
intersection
pumping lemma
deterministic = nondeterministic
Languages
Recursive
Languages
Context-Free
Languages
Regular
Languages
FSMs
D
ND
PDAs
Lecture Notes 19
12
Turing Machines
Read K & S 4.1.
Do Homework 17.
Grammars, Recursively Enumerable Languages, and Turing Machines
Recursively
Enumerable
Language
Unrestricted
Grammar
Accepts
Turing
Machine
Turing Machines
Can we come up with a new kind of automaton that has two properties:
powerful enough to describe all computable things
unlike FSMs and PDAs
simple enough that we can reason formally about it
like FSMs and PDAs
unlike real computers
Turing Machines
At each step, the machine may:

go to a new state, and
either
write on the current square, or
move left or right
Finite State Control

s1, s2, h1, h2
A Formal Definition
A Turing machine is a quintuple (K, , , s, H):
K is a finite set of states;
is an alphabet, containing at least and , but not or ;
s K is the initial state;
H K is the set of halting states;
is a function from:
(K - H)
to
K
non-halting state input symbol

state
such that
(a) if the input symbol is , the action is , and
(b) can never be written .
Lecture Notes 20
Turing Machines
( {, })
action (write or move)
Notes on the Definition

1.
The input tape is infinite to the right (and full of ), but has a wall to the left. Some definitions allow infinite tape in both
directions, but it doesn't matter.
2.
is a function, not a relation. So this is a definition for deterministic Turing machines.
3.
must be defined for all state, input pairs unless the state is a halt state.
4.
Turing machines do not necessarily halt (unlike FSM's). Why? To halt, they must enter a halt state. Otherwise they loop.
5.
Turing machines generate output so they can actually compute functions.

A Simple Example
A Turing Machine Odd Parity Machine:
= 0, 1, ,
s=
H=
=
Formalizing the Operation
(1)
(2)
A configuration of a Turing machine

M = (K, , , s, H) is a member of
K
state
input up
to scanned
square
(*( - {}))
input after
scanned square
The input after the scanned square may be empty, but it may not end with a blank. We assume the entire tape to the right of the
input is filled with blanks.
(1)
(2)
(q, aab, b)
(h, aabb, )
Lecture Notes 20
=
=
(q, aabb)
(h, aabb)
a halting configuration
Turing Machines
Yields
(q1, w1a1u1) |-M (q2, w2a2u2),
a1 and a2 ,
(1) b , w1 = w2, u1 = u2, and a2 = b

|
(rewrite without moving the head)
w1
b {, }, (q1, a1) = (q2, b) and either:
iff
| a1
b
| u1
b
w2
a2
a
aabb
aaab
aaab
aaab
| a1
|u1|
aaab
aaab
u2
Yields, Continued
(2) b = , w1 = w2a2, and either
(a) u2 = a1u1, if a1 or u1 ,
|
w1
w2
or
|
a
|
a
(b) u2 = , if a1 = and u1 =
|
w1
a
|
a
a2
a
a1
a
u2
a
|
w1
u1
b
a1
b
|u1|
If we scan left off the first square of the blank region, then drop that square from the configuration.
Yields, Continued
(3) b = , w2 = w1a1, and either
(a) u1 = a2u2
|
w1
a
a1
a
w2
a
(b) u1 = u2 = and a2 =
|
w1
or
w2
a
u1
b
a2
a1
u2
|u1|
a2
aaab
aaab
aaab
|u2|
aaab
If we scan right onto the first square of the blank region, then a new blank appears in the configuration.
Lecture Notes 20
Turing Machines
Yields, Continued
For any Turing machine M, let |-M* be the reflexive, transitive closure of |-M.
Configuration C1 yields configuration C2 if
C1 |-M* C2.
A computation by M is a sequence of configurations C0, C1, , Cn for some n 0 such that
C0 |-M C1 |-M C2 |-M |-M Cn.
We say that the computation is of length n or that it has n steps, and we write
C0 |-Mn Cn
A Context-Free Example
M takes a tape of a's then b's, possibly with more a's, and adds b's as required to make the number of b's equal the number of a's.
K = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
= a, b, , , 1, 2
s= 0
H = {9}
a/1
/
a,1,2/
a/1
1,2/
1/
b/2
2/
/2
5
2/
/
1/a;2/b
7
8
/
9
An Example Computation
(0, aaab) |-M

(1, aaab) |-M
(2, 1aab) |-M
(3, 1aab) |-M
(3, 1aab) |-M
(3, 1aab) |-M
(4, 1aa2) |-M
...
Lecture Notes 20
Turing Machines
Notes on Programming
The machine has a strong procedural feel.
It's very common to have state pairs, in which the first writes on the tape and the second move. Some definitions allow both
actions at once, and those machines will have fewer states.
There are common idioms, like scan left until you find a blank.
Even a very simple machine is a nuisance to write.
A Notation for Turing Machines
(1) Define some basic machines
Symbol writing machines

For each a - {}, define Ma, written just a, = ({s, h}, , , s, {h}),
for each b - {}, (s, b) = (h, a)
(s, ) = (s, )
Example:
a writes an a
Head moving machines

For each a {, }, define Ma, written R() and L():
for each b - {}, (s, b) = (h, a)
(s, ) = (s, )
Examples:
R moves one square to the right
aR writes an a and then moves one square to the right.
A Notation for Turing Machines, Cont'd

(2) The rules for combining machines: as with FSMs
>M1
b
M2
M3
Start in the start state of M1.

Compute until M1 reaches a halt state.
Examine the tape and take the appropriate transition.
Start in the start state of the next machine, etc.
Halt if any component reaches a halt state and has no place to go.
If any component fails to halt, then the entire machine may fail to halt.
Lecture Notes 20
Turing Machines
Shorthands
a
M1
M2
becomes
M1
M2
becomes
M1
a, b
M2
M1
all elems of
M2
or
M1M2
MM
all elems of
except a
M1
M2
becomes
M2
becomes
M1
xa
M2
and x takes on the value of the current square
M1
a, b
M2
becomes
M1
x = a, b
M2
and x takes on the value of the current square

M
x?y
M2
if x = y then take the transition
e.g.,
>
Rx
if the current square is not blank, go right and copy it.
Some Useful Machines

>R
find the first blank square to the right of the current square
R
>L
find the first blank square to the left of the current square
L
>R
find the first nonblank square to the right of the current square
R
>L
find the first nonblank square to the left of the current square
L
Lecture Notes 20
Turing Machines
More Useful Machines

La
find the first occurrence of a to the left of the current square
Ra,b
find the first occurrence of a or b to the right of the current square
La,b
b
M1
find the first occurrence of a or b to the left of the current square, then go to M1 if the detected
character is a; go to M2 if the detected character is b
M2
Lx=a,b
find the first occurrence of a or b to the left of the current square and set x to the value found
Lx=a,bRx
find the first occurrence of a or b to the left of the current square, set x to the value found, move one
square to the right, and write x (a or b)
An Example
w w {1}*
w3
Input:
Output:
111
Example:
>R1,
L
#R#R#L
1
H
A Shifting Machine S
Input:
Output:
w
w
Example:
abba
> L
LxR
x=
L
Lecture Notes 20
Turing Machines

Read K & S 4.2.
Do Homework 18.
Turing Machines as Language Recognizers
Convention: We will write the input on the tape as:
w , w contains no s
The initial configuration of M will then be:
(s, w)
A recognizing Turing machine M must have two halting states: y and n
Any configuration of M whose state is:
y is an accepting configuration
n is a rejecting configuration
Let 0, the input alphabet, be a subset of M-{,}
Then M decides a language L 0* iff for any string
w 0*it is true that:
if w L then M accepts w, and
if w L then M rejects w.
A language L is recursive if there is a Turing machine M that decides it.
A Recognition Example
L = {anbncn : n 0}
Example: aabbcc
Example: aaccb
a
> R
, b, c
b,c
a, b
a
a R
c, a, c,
b,c
a, b, c, a
n
b, c
b R
c L
, a, b, a
Lecture Notes 21
Another Recognition Example
L = {wcw : w {a, b}*}

Example: abbcabb
Example: acabb
>R
Rc,
x=a,b
R#
c
n
(y ? x )
Ry=#
y?x
#L
Do Turing Machines Stop?
FSMs
Always halt after n steps, where n is the length of the input. At that point, they either accept or reject.
PDAs
Don't always halt, but there is an algorithm to convert any PDA into one that does halt.
Turing machines Can do one of three things:

(1) Halt and accept
(2) Halt and reject
(3) Not halt
And now there is no algorithm to determine whether a given machine always halts.
Computing Functions
Let 0 - {, } and let w 0*
Convention: We will write the input on the tape as:
The initial configuration of M will then be:
(s, w)
Define M(w) = y iff:

M halts if started in the input configuration,
the tape of M when it halts is y, and
y 0*
Let f be any function from 0* to 0*.
We say that M computes f if, for all w 0*, M(w) = f(w)
A function f is recursive if there is a Turing machine M that computes it.
Lecture Notes 21
Example of Computing a Function

f(w) = ww
Input: w
Output: ww
Define the copy machine C:
Remember the S machine:

ww
> L R
ww
ww
LxR
x=
L
Then the machine to compute f is just
>C S L
Computing Numeric Functions
We say that a Turing machine M computes a function f from Nk to N provided that

num(M(n1;n2;nk)) = f(num(n1), num(nk))
Example: Succ(n) = n + 1
We will represent n in binary. So n 0 1{0,1}*
Input:
n
1111
Output: n+1
Output: 10000
Why Are We Working with Our Hands Tied Behind Our Backs?
Turing machines are more powerful than any of the other formalisms we have studied so far.
Turing machines are a lot harder to work with than all the real computers we have available.
Why bother?
The very simplicity that makes it hard to program Turing machines makes it possible to reason formally about what they can do.
If we can, once, show that anything a real computer can do can be done (albeit clumsily) on a Turing machine, then we have a
way to reason about what real computers can do.
Lecture Notes 21

Read K & S 4.5.
Recursively Enumerable Languages
Let 0, the input alphabet to a Turing machine M, be a subset of M - {, }
Let L 0*.
M semidecides L iff
for any string w 0*,
wL
wL
M halts on input w
M does not halt on input w
M(w) =
L is recursively enumerable iff there is a Turing machine that semidecides it.

Examples of Recursively Enumerable Languages
L = {w {a, b}* : w contains at least one a}
a
>R
b b b b b b
L = {w {a, b, (, ) }* : w contains at least one set of balanced parentheses}
> R),
L(,
L
b b b b b b)
Recursively Enumerable Languages that Aren't Also Recursive

A Real Life Example:
L = {w {friends} : w will answer the message you've just sent out}
Theoretical Examples
L = {Turing machines that halt on a blank input tape}
Theorems with valid proofs.
Lecture Notes 22
Why Are They Called Recursively Enumerable Languages?

Enumerate means list.
We say that Turing machine M enumerates the language L iff, for some fixed state q of M,
L = {w : (s, ) |-M* (q, w)}
q
A language is Turing-enumerable iff there is a Turing machine that enumerates it.

Note that q is not a halting state. It merely signals that the current contents of the tape should be viewed as a member of L.
Recursively Enumerable and Turing Enumerable
Theorem: A language is recursively enumerable iff it is Turing-enumerable.
Proof that Turing-enumerable implies RE: Let M be the Turing machine that enumerates L. We convert M to a machine M' that
semidecides L:
1. Save input w.
2. Begin enumerating L. Each time an element of L is enumerated, compare it to w. If they match, accept.
w
=w?
halt
w3, w2, w1
M
M'
The Other Way

Proof that RE implies Turing-enumerable:
If L * is a recursively enumerable language, then there is a Turing machine M that semidecides L.
A procedure to enumerate all elements of L:
Enumerate all w * lexicographically.
e.g., , a, b, aa, ab, ba, bb,
As each string wi is enumerated:
1. Start up a copy of M with wi as its input.
2. Execute one step of each Mi initiated so far, excluding only those that have previously halted.
3. Whenever an Mi halts, output wi.
[1]
[2]
[3]
[4]
[5]
[6]
Lecture Notes 22
a
a
a
a
a
[1]
[2]
[3]
[4]
[5]
b [1]
b [2]
b [3]
aa [1]
aa [2]
aa [3]
ab [1]
ab [2]
ba [1]
Every Recursive Language is Recursively Enumerable

If L is recursive, then there is a Turing machine that decides it.
From M, we can build a new Turing machine M' that semidecides L:
1. Let n be the reject (and halt) state of M.
2. Then add to '
((n, a), (n, a)) for all a
a/a
y
What about the other way around?

Not true. There are recursively enumerable languages that are not recursive.
The Recursive Languages Are Closed Under Complement
Proof: (by construction) If L is recursive, then there is a Turing machine M that decides L.
We construct a machine M' to decide L by taking M and swapping the roles of the two halting states y and n.
M:
M':
This works because, by definition, M is

deterministic
complete
Are the Recursively Enumerable Languages Closed Under Complement?
M:
M':
Lemma: There exists at least one language L that is recursively enumerable but not recursive.
Proof that M' doesn't exist: Suppose that the RE languages were closed under complement. Then if L is RE, L would be RE. If
that were true, then L would also be recursive because we could construct M to decide it:
1. Let T1 be the Turing machine that semidecides L.
2. Let T2 be the Turing machine that semidecides L.
3. Given a string w, fire up both T1 and T2 on w. Since any string in * must be in either L or L, one of the two machines will
eventually halt. If it's T1, accept; if it's T2, reject.
But we know that there is at least one RE language that is not recursive. Contradiction.
Lecture Notes 22
Recursive and RE Languages

Theorem: A language is recursive iff both it and its complement are recursively enumerable.
Proof:
L recursive implies L and L are RE: Clearly L is RE. And, since the recursive languages are closed under complement,
L is recursive and thus also RE.
L and L are RE implies L recursive: Suppose L is semidecided by M1 and L is semidecided by M2. We construct M to
decide L by using two tapes and simultaneously executing M1 and M2. One (but not both) must eventually halt. If it's M1,
we accept; if it's M2 we reject.
Lexicographic Enumeration
We say that M lexicographically enumerates L if M enumerates the elements of L in lexicographic order. A language L is
lexicographically Turing-enumerable iff there is a Turing machine that lexicographically enumerates it.
Example: L = {anbncn}
Lexicographic enumeration:
Proof
Theorem: A language is recursive iff it is lexicographically Turing-enumerable.
Proof that recursive implies lexicographically Turing enumerable: Let M be a Turing machine that decides L. Then M'
lexicographically generates the strings in * and tests each using M. It outputs those that are accepted by M. Thus M'
lexicographically enumerates L.
*3, *2, *1
L?
yes
no
*k
M
M'
Proof, Continued
Proof that lexicographically Turing enumerable implies recursive: Let M be a Turing machine that lexicographically enumerates
L. Then, on input w, M' starts up M and waits until either M generates w (so M' accepts), M generates a string that comes after w
(so M' rejects), or M halts (so M' rejects). Thus M' decides L.
w
= w?
yes
> w?
no
no more Lis?
no
L3, L2, L1
M
M'
Lecture Notes 22
Partially Recursive Functions

Languages
recursive
recursively
enumerable
Tm always halts
Tm halts if yes
domain
Functions
recursive
?
range
Suppose we have a function that is not defined for all elements of its domain.
Example: f: N N, f(n) = n/2
Partially Recursive Functions
domain
range
One solution: Redefine the domain to be exactly those elements for which f is defined:
domain
range
But what if we don't know? What if the domain is not a recursive set (but it is recursively enumerable)? Then we want to define
the domain as some larger, recursive set and say that the function is partially recursive. There exists a Turing machine that halts
if given an element of the domain but does not halt otherwise.
Lecture Notes 22
Language
Summary
IN
OUT
Semidecidable
Enumerable
Unrestricted grammar
Recursively
Enumerable
Decision procedure
Lexicicographically enumerable
Complement is recursively enumer.
Recursive
CF grammar
PDA
Closure
Context Free
Regular expression
FSM
Closure
Lecture Notes 22
Regular
Diagonalization
Reduction
Pumping
Closure
Pumping
Closure

Read K & S 4.3.1, 4.4.
Do Homework 19.
Turing Machine Definitions
An alternative definition of a Turing machine:
(K, , , , s, H):
is a finite set of allowable tape symbols. One of these is .
is a subset of not including , the input symbols.
is a function from:
K
to
K
state,
( - {}) {, }
tape symbol, L or R
Example transition: ((s, a), (s, b, ))

Do these Differences Matter?
Remember the goal:
Define a device that is:
powerful enough to describe all computable things,
simple enough that we can reason formally about it
Both definitions are simple enough to work with, although details may make specific arguments easier or harder.
But, do they differ in their power?
Answer: No.
Consider the differences:
One way or two way infinite tape: we're about to show that we can simulate two way infinite with ours.
Rewrite and move at the same time: just affects (linearly) the number of moves it takes to solve a problem.
In fact, there are lots of extensions we can make to our basic Turing machine model. They may make it easier to write Turing
machine programs, but none of them increase the power of the Turing machine because:
We can show that every extended machine has an equivalent basic machine.
We can also place a bound on any change in the complexity of a solution when we go from an extended machine to a basic
machine.
Some possible extensions:
Multiple tapes
Two-way infinite tape
Multiple read heads
Two dimensional sheet instead of a tape
Random access machine
Nondeterministic machine
Lecture Notes 23
Multiple Tapes
The transition function for a k-tape Turing machine:

((K-H) ,
,
,
,
,
1
2
.
.
k)
(K, 1' {, }
, 2' {, }
, .
, .
, k' {, })
to
Input: input as before on tape 1, others blank

Output: output as before on tape 1, others ignored
An Example of a Two Tape Machine
Copying a string
Lecture Notes 23
Another Two Tape Example - Addition
Adding Tapes Adds No Power

Theorem: Let M be a k-tape Turing machine for some k 1. Then there is a standard Turing machine M' where ', and such
that:
For any input string x, M on input x halts with output y on the first tape iff M' on input x halts at the same halting state and
with the same output on its tape.
If, on input x, M halts after t steps, then M' halts after a number of steps which is O(t (|x| + t)).
Proof: By construction
0
a
1
a
0
a
0
0
b
0
a
1
b
0
b
0
b
0
a
1
b
0
The Operation of M'

b
a
0
0
0
b
a
b
0
0
0
0
a
0
Alphabet (') of M' = ( {0, 1})k

e.g., , (, 0, , 0), (, 0, a, 1)
0
a
1
0
a
0
1.
Set up the multitrack tape:

1)
Shift input one square to right, then set up each square appropriately.
2.
Simulate the computation of M until (if) M would halt: (start each step to the right of the divided tape)
1)
Scan left and store in the state the k-tuple of characters under the read heads. Move back right.
2)
Scan left and update each track as required by the transitions of M. Move back right.
i)
If necessary, subdivide a new square into tracks.
3.
When M would halt, reformat the tape to throw away all but track 1, position the head correctly, then go to M's halt
state.
How Many Steps Does M' Take?
Let:
x be the input string, and
t be the number of steps it takes M to execute.
Step 1 (initialization)
O(|x|)
Step 2 ( computation)
Number of passes = t
Work at each pass:
2.1 = 2 (length of tape)
= 2 (|x| + 2 + t)
2.2 = 2 (|x| + 2 + t)
Total = O(t (|x| + t))
Step 3 (clean up)
O(length of tape)
Total = O(t (|x| + t))
Lecture Notes 23
Two-Way Infinite Tape

Our current definition:
Track 1
Track 2
Proposed definition:
Simulation:
Simulating a PDA
The components of a PDA:
Finite state controller
Input tape
Stack
The simulation:
Finite state controller:
Input tape:
Stack:
Track 1
(Input)
Track 2
Corresponding to
a
a
Simulating a Turing Machine with a PDA with Two Stacks
a
a
b
a
Lecture Notes 23
#
a
a
b
a
Random Access Turing Machines

A random access Turing machine has:
a fixed number of registers
a finite length program, composed of instructions with operators such as read, write, load, store, add, sub, jump
a tape
a program counter
Theorem: Standard Turing machines and random access Turing machines compute the same things. Furthermore, the number of
steps it takes a standard machine is bounded by a polynomial in the number of steps it takes a random access machine.
Nondeterministic Turing Machines
A nondeterministic Turing machine is a quintuple
(K, , , s, H)
where K, , s, and H are as for standard Turing machines, and is a subset of
((K - H) ) (K ( {, }))
abab
abab
abab
abab
bbab
What does it mean for a nondeterministic Turing machine to compute something?

Semidecides - at least one halts.
Decides - ?
Computes - ?
Nondeterministic Semideciding
Let M = (K, , , s, H) be a nondeterministic Turing machine. We say that M accepts an input
w ( - {, })* iff
(s, w) yields a least one accepting configuration.
We say that M semidecides a language
L ( - {, })* iff
for all w ( - {, })*:
w L iff
(s, w) yields a least one halting configuration.
An Example
L = {w {a, b, c, d}* : there are two of at least one letter}
a/
2
/
0
b/
a/
b/
b
c/
c/
d/
4
d/
Lecture Notes 23
Nondeterministic Deciding and Computing

M decides a language L if, for all w ( - {, })* :
1. all of M's computations on w halt, and
2. w L iff at least one of M's computations accepts.
M computes a function f if, for all w ( - {, })* :
1. all of M's computations halt, and
2. all of M's computations result in f(w)
Note that all of M's computations halt iff:
There is a natural number N, depending on M and w, such that there is no configuration C satisfying
(s, w) |-MN C.
An Example of Nondeterministic Deciding
L = {w {0, 1}* : w is the binary encoding of a composite number}
M decides L by doing the following on input w:
1.
Nondeterministically choose two binary numbers 1 < p, q, where |p| and |q| |w|, and write them on the tape, after w,
separated by ;.
110011;111;1111
2.
Multiply p and q and put the answer, A, on the tape, in place of p and q.
110011;1011111
3.
Compare A and w. If equal, go to y. Else go to n.

Equivalence of Deterministic and Nondeterministic Turing Machines
Theorem: If a nondeterministic Turing machine M semidecides or decides a language, or computes a function, then there is a
standard Turing machine M' semideciding or deciding the same language or computing the same function.
Note that while nondeterminism doesnt change the computational power of a Turing Machine, it can exponentially increase its
speed!
Proof: (by construction)
For semideciding: We build M', which runs through all possible computations of M. If one of them halts, M' halts
Recall the way we did this for FSMs: simulate being in a combination of states.
Will this work here?
What about:
Lecture Notes 23
Try path 1. If it accepts, accept. Else

Try path 2. If it accepts, accept. Else
The Construction
At any point in the operation of a nondeterministic machine M, the maximum number of branches is
r=
|K|
(|| + 2)
states
actions
So imagine a table:
1
(p-,-)
(p-,-)
(q1,1)
(q1,2)
(q1,n)
(q2,1)
2
(p-,-)
(p-,-)
3
(p-,-)
(p-,-)
r
(p-,-)
(p-,-)
(p-,-)
(p-,-)
(q|K|,n)
Note that if, in some configuration, there are not r different legal things to do, then some of the entries on that row will repeat.
Md:
(suppose r = 6)
Tape 1:
Input
Tape 2:
1 3 2 6 5 4 3 6
Md chooses its 1st move from column 1

Md chooses its 2nd move from column 3
Md chooses its 3rd move from column 2
until there are no more numbers on Tape 2

Md either:
discovers that M would accept, or
comes to the end of Tape 2.
In either case, it halts.
M' (the machine that simulates M):
Tape 1:
Input
Tape 2:
Copy of Input
Md
Tape 3:
1 3 2 6 5 4 3 6
Steps of M':
write on Tape 3
until Md accepts do
(1) copy Input from Tape 1 to Tape 2
(2) run Md
(3) if Md accepts, exit
(4) otherwise, generate lexicographically next string on Tape 3.
Pass
Tape3
Lecture Notes 23
2
1
3
2
7
6
8
11
9
12
2635
Nondeterministic Algorithms
Other Turing Machine Extensions

Multiple heads (on one tape)
Emulation strategy: Use tracks to keep track of tape heads. (See book)
Multiple tapes, multiple heads
Emulation strategy: Use tracks to keep track of tapes and tape heads.
Two-dimensional semi-infinite tape
Emulation strategy: Use diagonal enumeration of two-dimensional grid. Use second tape to help you keep track of
where the tape head is. (See book)
Two-dimensional infinite tape (really a sheet)
Emulation strategy: Use modified diagonal enumeration as with the semi-infinite case.
What About Turing Machine Restrictions?

Can we make Turing machines even more limited and still get all the power?
Example:
We allow a tape alphabet of arbitrary size. What happens if we limit it to:
One character?
Two characters?
Three characters?
Lecture Notes 23
Problem Encoding, TM Encoding, and the Universal TM

Read K & S 5.1 & 5.2.
Encoding a Problem as a Language
A Turing Machines deciding a language is analogous to the TM solving a decision problem.
Problem: Is the number n prime?
Instance of the problem: Is the number 9 prime?
Encoding of the problem, n: n as a binary number. Example: 1001
Problem: Is an undirected graph G connected?
Instance of the problem: Is the following graph connected?
1
2
4
3
5
Encoding of the problem, G:

1) |V| as a binary number
2) A list of edges represented by pairs of binary numbers being the vertex numbers that the edge connects
3) All such binary numbers are separated by /.
Example: 101/1/10/10/11/1/100/10/101
Problem View vs. Language View
Problem View: It is unsolvable whether a Turing Machine halts on a given input. This is called the Halting Problem.
Language View: Let H = {M, w : TM M halts on input string w}
H is recursively enumerable but not recursive.
Problem: All our machines so far are hardwired.
Question: Does it make sense to talk about a programmable Turing machine that accepts as input
program input string
executes the program, and outputs
output string
Yes, it's called the Universal Turing Machine.
Notice that the Universal Turing machine semidecides H = {M, w : TM M halts on input string w} = L(U).
To define the Universal Turing Machine U we need to do two things:
1. Define an encoding operation for Turing machines.
2. Describe the operation of U given an input tape containing two inputs:
encoded Turing machine M,
encoded input string to be given to M.
Lecture Notes 24
Problem Encoding, Turing Machine Encoding, and the Universal Turing Machine
Encoding a Turing Machine M

We need to describe M = (K, , , s, H) as a string. To do this we must:
1. Encode
2. Specify s.
3. Specify H (and y and n, if applicable)
1. To encode , we need to:
1. Encode the states
2. Encode the tape alphabet
3. Specify the transitions
1.1 Encode the states as
qs : s {0, 1}+ and
|s| = i and
i is the smallest integer such that 2i |K|
Example: 9 states
i=4
s = q0000,
remaining states: q0001, q0010, q0011,
q0100, q0101, q0110, q0111, q1000
Encoding a Turing Machine M, Continued
1.2 Encode the tape alphabet as
as : s {0, 1}+ and
|s| = j and
(the + 2 allows for and )
j is the smallest integer such that 2j || + 2
Example: = {, , a, b} j = 3
=
a000
=
a001
=
a010
=
a011
a=
a100
b=
a101
Encoding a Turing Machine M, Continued
1.3 Specify transitions as (state, input, state, output)
Example: (q00,a000,q11,a000)
2. Specify s as q0i
3. Specify H:
States with no transitions out are in H.
If M decides a language, then H = {y, n}, and we will adopt the convention that y is the lexicographically smaller of
the two states.
y = q010
n = q011
Encoding Input Strings
We encode input strings to a machine M using the same character encoding we use for M.
For example, suppose that we are using the following encoding for symbols in M:
symbol
a
Then we would represent the string s = aaa as
Lecture Notes 24
representation
a000
a001
a010
a011
a100
"s" = s = a001a100a100a000a100
An Encoding Example
Consider M = ({s, q, h}, {, ,a}, , s, {h}), where =
state
s
s
s
q
q
q
symbol
a
(q, )
(h, )
(s, )
(s, a)
(s, )
(q, )
state/symbol
s
q
h
representation
q00
q01
q11
a000
a001
a010
a011
a100
The representation of M, denoted, "M", M, or sometimes (M) =

(q00,a100,q01,a000), (q00,a000,q11,a000), (q00,a001,q00,a011),
(q01,a100,q00,a100), (q01,a000,q00,a011), (q01,a001,q01,a011)
Another Win of Encoding
One big win of defining a way to encode any Turing machine M:
It will make sense to talk about operations on programs (Turing machines). In other words, we can talk about some
Turing machine T that takes another Turing machine (say M1) as input and transforms it into a different machine
(say M2) that performs some different, but possibly related task.
Example of a transforming TM T:
Input: a machine M1 that reads its input tape and performs some operation P on it.
Output: a machine M2 that performs P on an empty input tape:
>R
L R M1
The specification for U:
U("M" "w") = "M(w)"
"M ------------------------------ M"

1
0
0
0
"w------------------------w"
0
0
0
"
"
"w--------------------w"
1
0
0
0
0
"M ---------------------------- M"
1
0
0
0
0
q
0
0
0
Initialization of U:
1. Copy "M" onto tape 2
2. Insert "" at the left edge of tape 1, then shift w over.
3. Look at "M", figure out what i is, and write the encoding of state s on tape 3.
Lecture Notes 24
The Operation of U
a
0
0
1
1
0
0
0
"M ---------------------------- M"
1
0
0
0
q
0
0
0
1
a
0
0
0
0
0
Simulate the steps of M:

1. Start with the heads:
tape 1: the a of the character being scanned,
tape 2: far left
tape 3: far left
2. Simulate one step:
1. Scan tape 2 for a quadruple that matches current state, input pair.
2. Perform the associated action, by changing tapes 1 and 3. If necessary, extend the tape.
3. If no quadruple found, halt. Else go back to 2.
An Example
Tape 1: a001a000a100a100a000a100
a
a
a
Tape 2: (q00,a000,q11,a000), (q00,a001,q00,a011),
(q00,a100,q01,a000), (q01,a000,q00,a011),
(q01,a001,q01,a011), (q01,a100,q00,a100)
Tape 3: q01
Result of simulating the next step:

Tape 1: a001a000a100a100a000a100
a
a
a
Tape 3: q00
If A Universal Machine is Such a Good Idea

Could we define a Universal Finite State Machine?
Such a FSM would accept the language
L = {"F" "w" : F is a finite state machine, and w L(F) }
Lecture Notes 24
Grammars and Turing Machines

Do Homework 20.
Grammars, Recursively Enumerable Languages, and Turing Machines

L
Recursively
Enumerable
Language
Unrestricted
Grammar
Accepts
Turing
Machine
An unrestricted, or Type 0, or phrase structure grammar G is a quadruple
(V, , R, S), where
V is an alphabet,
(the set of terminals) is a subset of V,
R (the set of rules) is a finite subset of
(V*
(V-)
V*)
V*,
context
N
context
result
S (the start symbol) is an element of V - .
We define derivations just as we did for context-free grammars.
The language generated by G is
{w * : S G* w}
There is no notion of a derivation tree or rightmost/leftmost derivation for unrestricted grammars.
Example: L = anbncn, n > 0
S aBSc
S aBc
Ba aB
Bc bc
Bb bb
Another Example
L = {w {a, b, c}+ : number of a's, b's and c's is the same}
S ABCS
S ABC
AB BA
BC CB
AC CA
BA AB
Lecture Notes 25
CA AC
CB BC
Aa
Bb
Cc
A Strong Procedural Feel

Unrestricted grammars have a procedural feel that is absent from restricted grammars.
Derivations often proceed in phases. We make sure that the phases work properly by using nonterminals as flags that we're in a
particular phase.
It's very common to have two main phases:
Generate the right number of the various symbols.
Move them around to get them in the right order.
No surprise: unrestricted grammars are general computing devices.
Equivalence of Unrestricted Grammars and Turing Machines
Theorem: A language is generated by an unrestricted grammar if and only if it is recursively enumerable (i.e., it is semidecided
by some Turing machine M).
Proof:
Only if (grammar TM): by construction of a nondeterministic Turing machine.
If (TM grammar): by construction of a grammar that mimics backward computations of M.
Proof that Grammar Turing Machine
Given a grammar G, produce a Turing machine M that semidecides L(G).
M will be nondeterministic and will use two tapes:
1
a
1
a
0
S
0
b
0
T
0
a
0
a
0
0
b
0
For each nondeterministic "incarnation":

Tape 1 holds the input.
Tape 2 holds the current state of a proposed derivation.
At each step, M nondeterministically chooses a rule to try to apply and a position on tape 2 to start looking for the left hand side
of the rule. Or it chooses to check whether tape 2 equals tape 1. If any such machine succeeds, we accept. Otherwise, we keep
looking.
Lecture Notes 25
Proof that Turing Machine Grammar

Suppose that M semidecides a language L (it halts when fed strings in L and loops otherwise). Then we can build M' that halts in
the configuration (h, ).
We will define G so that it simulates M' backwards.
We will represent the configuration (q, uaw) as
>uaqw<
M'
goes from
Then, if w L, we require that our grammar produce a derivation of the form

S G >h<
(produces final state of M')
G* >abq< (some intermediate state of M')
G* >sw< (the initial state of M')
G w<
(via a special rule to clean up >s)
(via a special rule to clean up <)
G w
The Rules of G
S >h<
(the halting configuration)
>s
<
(clean-up rules to be applied at the end)
Rules that correspond to :

If (q, a) = (p, b) :
bp aq
If (q, a) = (p, ) :
abp aqb b
ap< aq<
If (q, a) = (p, ), a
pa aq
If (q, ) = (p, )
pb qb
p< q<
Lecture Notes 25
M' = (K, {a}, , s, {h}), where

={ ((s, ), (q, )),
((q, a), (q, )),
((q, ), (t, )),
((t, a), (p, )),
((t, ), (h, )),
((p, ), (t, ))
A REALLY Simple Example

1
2
3
4
5
6
L = a*
S >h<
>s
<
(1)
(2)
(3)
(4)
(5)
(6)
q s
aq sa
q< s<
aq aq
aaq aqa
aq< aq<
t q
ta qa
t< qh<
>s
<
1
2
3
q s
aq sa
q< s<
aq aq
aaq aqa
aq< aq<
4
5
6
7
8
9
>saa<
>aqa<
>aaq<
>aaq<
>aat<
>ap<
>at<
>p<
>t<
>h<
Lecture Notes 25
1
2
2
3
4
6
4
6
5
(3)
(4)
(5)
(6)
t q
ta qa
t< qh<
>t<
>p<
>at<
>ap<
>aat<
>aaq<
>aaq<
>aqa<
>saa<
aa<
aa
10
11
12
13
14
15
16
17
1
14
17
13
17
13
12
9
8
5
2
3
An Alternative Proof
An alternative is to build a grammar G that simulates the forward operation of a Turing machine M. It uses alternating symbols
to represent two interleaved tapes. One tape remembers the starting string, the other working tape simulates the run of the
machine.
The first (generate) part of G:
Creates all strings over * of the form
w = Qs a1 a1 a2 a2 a3 a3
The second (test) part of G simulates the execution of M on a particular string w. An example of a partially derived string:
a 1 b 2 c c b 4 Q3 a 3
Examples of rules:
b b Q 4 b 4 Q 4 (rewrite b as 4)
b 4 Q 3 Q 3 b 4 (move left)
The third (cleanup) part of G erases the junk if M ever reaches h.
Example rule:
#ha1a#h
(sweep # h to the right erasing the working tape)
Computing with Grammars

We say that G computes f if, for all w, v *,
SwS G* v iff v = f(w)
Example:
S1S
G* 11
f(x) = succ(x)
S11S G* 111
A function f is called grammatically computable iff there is a grammar G that computes it.
Theorem: A function f is recursive iff it is grammatically computable.
In other words, if a Turing machine can do it, so can a grammar.
Example of Computing with a Grammar
f(x) = 2x, where x is an integer represented in unary
G = ({S, 1}, {1}, R, S), where R =
S1 11S
SS
Example:
Input:
S111S
Output:
Lecture Notes 25
More on Functions: Why Have We Been Using Recursive as a Synonym for Computable?
Primitive Recursive Functions
Define a set of basic functions:
zerok (n1, n2, nk) = 0
identityk,j (n1, n2, nk) = nj
successor(n) = n + 1
Combining functions:
Composition of g with h1, h2, hk is
g(h1( ), h2( ), hk( ))
Primitive recursion of f in terms of g and h:
f(n1,n2,nk, 0) = g(n1,n2,nk)
f(n1,n2,nk,m+1) = h(n1,n2,nk, m, f(n1, n2,nk,m))
Example:
plus(n, 0) = n
plus(n, m+1) = succ(plus(n, m))
Primitive Recursive Functions and Computability
Trivially true: all primitive recursive functions are Turing computable.

What about the other way: Not all Turing computable functions are primitive recursive.
Proof:
Lexicographically enumerate the unary primitive recursive functions, f0, f1, f2, f3, .
Define g(n) = fn(n) + 1.
G is clearly computable, but it is not on the list. Suppose it were fm for some m. Then
fm(m) = fm(m) + 1, which is absurd.
0
f0
f1
f2
f3
f4
27
Suppose g is f3. Then g(3) = 27 + 1 = 28. Contradiction.

Functions that Aren't Primitive Recursive
Example:
Ackermann's function:
A(0, y) = y + 1
A(x + 1, 0) = A(x, 1)
A(x + 1, y + 1) = A(x, A(x + 1, y))
0
1
1
2
2
3
3
4
4
5
5
6
11
3
4
5
13
13
65533
29
265536-3
61
*
125
65536
22
22
265536
1017 seconds since big bang

1087 protons and neutrons
5939
10-23 light seconds = width
digits
% 1010
of proton or neutron
Thus writing digits at the speed of light on all protons and neutrons in the universe (all lined up) starting at the big bang would
have produced 10127 digits.
* 19,729 digits
# 105940 digits
Lecture Notes 25
Recursive Functions
A function is -recursive if it can be obtained from the basic functions using the operations of:
Composition,
Recursive definition, and
Minimalization of minimalizable functions:
The minimalization of g (of k + 1 arguments) is a function f of k arguments defined as:
if such an m exists,
f(n1,n2,nk) = the least m such at g(n1,n2,nk,m)=1,
0
otherwise
A function g is minimalizable iff for every n1,n2,nk, there is an m such that g(n1,n2,nk,m)=1.
Theorem: A function is -recursive iff it is recursive (i.e., computable by a Turing machine).
Partial Recursive Functions
Consider the following function f:
f(n) = 1 if TM(n) halts on a blank tape
0 otherwise
The domain of f is the natural numbers. Is f recursive?
domain
range
Theorem: There are uncountably many partially recursive functions (but only countably many Turing machines).
Functions and Machines
Partial Recursive
Functions
Recursive
Functions
Primitive Recursive
Functions
Turing Machines
Lecture Notes 25
Languages
Recursive
Languages
Context-Free
Languages
Deterministic
Context-Free
Languages
Regular
Languages
FSMs
DPDAs
NDPDAs
Turing Machines
Is There Anything In Between CFGs and Unrestricted Grammars?

Answer: yes, various things have been proposed.
Context-Sensitive Grammars and Languages:
A grammar G is context sensitive if all productions are of the form
xy
and |x| |y|
In other words, there are no length-reducing rules.
A language is context sensitive if there exists a context-sensitive grammar for it.
Examples:
L = {anbncn, n > 0}
L = {w {a, b, c}+ : number of a's, b's and c's is the same}
Lecture Notes 25
Context-Sensitive Languages are Recursive

The basic idea: To decide if a string w is in L, start generating strings systematically, shortest first. If you generate w, accept. If
you get to strings that are longer than w, reject.
Linear Bounded Automata
A linear bounded automaton is a nondeterministic Turing machine the length of whose tape is bounded by some fixed constant k
times the length of the input.
L = {anbncn : n 0}
Example:
aabbcc
a
> R
,b,c
b,c
a,b
a R
c,a,c,
b,c
a,b,c,a
b,c
b R
c L
,a,b,a
n
y
Context-Sensitive Languages and Linear Bounded Automata
Theorem: The set of context-sensitive languages is exactly the set of languages that can be accepted by linear bounded automata.
Proof: (sketch) We can construct a linear-bounded automaton B for any context-sensitive language L defined by some grammar
G. We build a machine B with a two track tape. On input w, B keeps w on the first tape. On the second tape, it
nondeterministically constructs all derivations of G. The key is that as soon as any derivation becomes longer than |w| we stop,
since we know it can never get any shorter and thus match w. There is also a proof that from any lba we can construct a contextsensitive grammar, analogous to the one we used for Turing machines and unrestricted grammars.
Theorem: There exist recursive languages that are not context sensitive.
Lecture Notes 25
Languages
Recursive
Languages
Context-Sensitive
Languages
Context-Free
Languages
Deterministic
Context-Free
Languages
Regular
Languages
FSMs
DPDAs
NDPDAs
Turing Machines
Lecture Notes 25
10
The Chomsky Hierarchy
Languages
Context-Sensitive
Languages
Context-Free
Languages
Type 0
Type 1
Type 2
Regular
(Type 3)
Languages
FSMs
PDAs
Turing Machines
Lecture Notes 25
11
Undecidabilty
Read K & S 5.1, 5.3, & 5.4.
Read Supplementary Materials: Recursively Enumerable Languages, Turing Machines, and Decidability.
Do Homeworks 21 & 22.
Church's Thesis
(Church-Turing Thesis)
An algorithm is a formal procedure that halts.
The Thesis: Anything that can be computed by any algorithm can be computed by a Turing machine.
Another way to state it: All "reasonable" formal models of computation are equivalent to the Turing machine.
This isn't a formal statement, so we can't prove it. But many different computational models have been proposed and they all turn
out to be equivalent.
Examples:
unrestricted grammars
lambda calculus
cellular automata
DNA computing
quantum computing (?)

Suppose we could implement the decision procedure
HALTS(M, x)
M: string representing a Turing Machine
x: string representing the input for M
If M(x) halts then True
else False
Then we could define
TROUBLE(x)
x: string
If HALTS(x, x) then loop forever
else halt
So now what happens if we invoke TROUBLE(TROUBLE), which invokes HALTS(TROUBLE, TROUBLE)
If HALTS says that TROUBLE halts on itself then TROUBLE loops. IF HALTS says that TROUBLE loops, then TROUBLE
halts. Either way, we reach a contradiction, so HALTS(M, x) cannot be made into a decision procedure.
Lecture Notes 26
Undecidability
Another View
The Problem View: The halting problem is undecidable.
The Language View: Let H =
{"M" "w" : TM M halts on input string w}
Why?
H is recursively enumerable because it can be semidecided by U, the Universal Turing Machine.
But H cannot be recursive. If it were, then it would be decided by some TM MH. But MH("M" "w") would have to be:
If M is not a syntactically valid TM, then False.
else HALTS("M" "w")
But we know cannot that HALTS cannot exist.
If H were Recursive
H = {"M" "w" : TM M halts on input string w}
Theorem: If H were also recursive, then every recursively enumerable language would be recursive.
Proof: Let L be any RE language. Since L is RE, there exists a TM M that semidecides it.
Suppose H is recursive and thus is decided by some TM O (oracle).
We can build a TM M' from M that decides L:
1. M' transforms its input tape from w to "M""w".
2. M' invokes O on its tape and returns whatever answer O returns.
So, if H were recursive, all RE languages would be. But it isn't.
Undecidable Problems, Languages that Are Not Recursive, and Partial Functions
The Problem View: The halting problem is undecidable.
The Language View: Let H =
{"M" "w" : TM M halts on input string w}
The Functional View: Let f (w) = M(w)
f is a partial function on *
"M""w" pairs
Lecture Notes 26
Undecidability
Other Undecidable Problems About Turing Machines
Given a Turing machine M, does M halt on the empty tape?

Given a Turing machine M, is there any string on which M halts?
Given a Turing machine M, does M halt on every input string?
Given two Turing machines M1 and M2, do they halt on the same input strings?
Given a Turing machine M, is the language that M semidecides regular? Is it context-free? Is it recursive?
Post Correspondence Problem
Consider two lists of strings over some alphabet . The lists must be finite and of equal length.
A = x1, x2, x3, , xn
B = y1, y2, y3, , yn
Question: Does there exist some finite sequence of integers that can be viewed as indexes of A and B such that, when elements of
A are selected as specified and concatenated together, we get the same string we get when elements of B are selected also as
specified?
For example, if we assert that 1, 3, 4 is such a sequence, were asserting that x1x3x4 = y1y3y4
Any problem of this form is an instance of the Post Correspondence Problem.
Is the Post Correspondence Problem decidable?
Post Correspondence Problem Examples
i
1
2
3
A
1
10111
10
B
111
10
0
i
1
2
3
A
10
011
101
B
101
11
011
Some Languages Aren't Even Recursively Enumerable

A pragmatically non RE language: L1={ (i, j) : i, j are integers where the low order five digits of i are a street address number
and j is the number of houses with that number on which it rained on November 13, 1946 }
An analytically non RE language: L2={x : x = "M" of a Turing machine M and M("M") does not halt}
Why isn't L2 RE? Suppose it were. Then there would be a TM M* that semidecides L2. Is "M*" in L2?
If it is, then M*("M*") halts (by the definition of M* as a semideciding machine for L2)
But, by the definition of L2, if "M*" L2, then M*("M*") does not halt.
Contradiction. So L2 is not RE.
Another Non RE Language
H
Why not?
Lecture Notes 26
Undecidability
Reduction
Let L1, L2 * be languages. A reduction from L1 to L2 is a recursive function : * * such that
x L1 iff (x) L2.
Example:
L1 = {a, b : a,b N : b = a + 1}
= Succ
a, b becomes
Succ(a), b
L2 = {a, b : a,b N : a = b}
If there is a Turing machine M2 to decide L2, then I can build a Turing machine M1 to decide L1:
1. Take the input and apply Succ to the first number.
2. Invoke M2 on the result.
3. Return whatever answer M2 returns.
Reductions and Recursive Languages
Theorem: If there is a reduction from L1 to L2 and L2 is recursive, then L1 is recursive.
x
x L 1?
M1
y = M2
y L2?
(x)
yes
yes
no
no
Theorem: If there is a reduction from L1 to L2 and L1 is not recursive, then L2 is not recursive.
Reductions and RE Languages

Theorem: If there is a reduction from L1 to L2 and L2 is RE, then L1 is RE.
x
x L1?
M1
y = M2
y L2?
(x)
halt
halt
Theorem: If there is a reduction from L1 to L2 and L1 is not RE, then L2 is not RE.
Lecture Notes 26
Undecidability
Can it be Decided if M Halts on the Empty Tape?

This is equivalent to, "Is the language L2 = {"M" : Turing machine M halts on the empty tape} recursive?"
L1
=H=
{s = "M" "w" : Turing machine M halts on input string w}
(?M2)
L2 =
{s = "M" : Turing machine M halts on the empty tape}
Let be the function that, from "M" and "w", constructs "M*", which operates as follows on an empty input tape:
1. Write w on the tape.
2. Operate as M would have.
If M2 exists, then M1 = M2(M(s)) decides L1.
A Formal Reduction Proof
Prove that L2 = {M: Turing machine M halts on the empty tape} is not recursive.
Proof that L2 is not recursive via a reduction from H = {M, w: Turing machine M halts on input string w}, a non-recursive
language. Suppose that there exists a TM, M2 that decides L2. Construct a machine to decide H as M1(M, w) = M2((M, w)).
The function creates from M and w a new machine M*. M* ignores its input and runs M on w, halting exactly when M halts
on w.
M, w H M halts on w M* always halts L(M*) M* L2 M2 accepts M1 accepts.
M, w H M does not halt on w L(M*) M* L2 M2 rejects M1 rejects.
Thus, if there is a machine M2 that decides L2, we could use it to build a machine that decides H. Contradiction. L2 is not
recursive.
Important Elements in a Reduction Proof
A clear declaration of the reduction from and to languages and what youre trying to prove with the reduction.
A description of how a machine is being constructed for the from language based on an assumed machine for the to
language and a recursive function.
A description of the functions inputs and outputs. If is doing anything nontrivial, it is a good idea to argue that it is
recursive.
Note that machine diagrams are not necessary or even sufficient in these proofs. Use them as thought devices, where
needed.
Run through the logic that demonstrates how the from language is being decided by your reduction. You must do both
accepting and rejecting cases.
Declare that the reduction proves that your to language is not recursive.
The Most Common Mistake: Doing the Reduction Backwards
The right way to use reduction to show that L2 is not recursive:

1. Given that L1 is not recursive,
2. Reduce L1 to L2, i.e. show how to solve L1 (the known one) in terms of L2 (the unknown one)
L1
L2
Example: If there exists a machine M2 that solves L2, the problem of deciding whether a Turing machine halts on a blank tape,
then we could do H (deciding whether M halts on w) as follows:
1. Create M* from M such that M*, given a blank tape, first writes w on its tape, then simulates the behavior of M.
2. Return M2("M*").
Doing it wrong by reducing L2 (the unknown one to L1): If there exists a machine M1 that solves H, then we could build a
machine that solves L2 as follows:
1. Return (M1("M", "")).
Lecture Notes 26
Undecidability
Why Backwards Doesn't Work

Suppose that we have proved that the following problem L1 is unsolvable: Determine the number of days that have elapsed since
the beginning of the universe.
Now consider the following problem L2: Determine the number of days that had elapsed between the beginning of the universe
and the assassination of Abraham Lincoln.
L1
Reduce L1 to L2:
L1 = L2 + (now - 4/9/1865)
L2
Reduce L2 to L1:
L2 = L1 - (now - 4/9/1865)
L2
L1
Why Backwards Doesn't Work, Continued
L1 = days since beginning of universe

L2 = elapsed days between the beginning of the universe and the assassination of Abraham Lincoln.
L3 = days between the assassination of Abraham Lincoln and now.
L1
Considering L2:
Reduce L1 to L2:
L1 = L2 + (now - 4/9/1865)
L2
Reduce L2 to L1:
L2 = L1 - (now - 4/9/1865)
L2
L1
L1
Considering L3:
Reduce L1 to L3:
L1 = oops
L3
Reduce L3 to L1:
L3 = L1 - 365 - (now - 4/9/1866)
L3
L1
Is There Any String on Which M Halts?
L1
=H=
(?M2)
L2 =
{s = "M" : there exists a string on which Turing machine M halts}
Let be the function that, from "M" and "w", constructs "M*", which operates as follows:
1. M* examines its input tape.
2. If it is equal to w, then it simulates M.
3. If not, it loops.
Clearly the only input on which M* has a chance of halting is w, which it does iff M would halt on w.
Lecture Notes 26
Undecidability
Does M Halt on All Inputs?

L1
{s = "M" : Turing machine M halts on the empty tape}
(?M2)
L2 =
{s = "M" : Turing machine M halts on all inputs}
Let be the function that, from "M", constructs "M*", which operates as follows:
1. Erase the input tape.
2. Simulate M.
Clearly M* either halts on all inputs or on none, since it ignores its input.
Rice's Theorem
Theorem: No nontrivial property of the recursively enumerable languages is decidable.
Alternate statement: Let P: 2*{true, false} be a nontrivial property of the recursively enumerable languages. The language
{M: P(L(M)) = True} is not recursive.
By "nontrivial" we mean a property that is not simply true for all languages or false for all languages.
Examples:
L contains only even length strings.
L contains an odd number of strings.
L contains all strings that start with "a".
L is infinite.
L is regular.
Note:
Rice's theorem applies to languages, not machines. So, for example, the following properties of machines are decidable:
M contains an even number of states
M has an odd number of symbols in its tape alphabet
Of course, we need a way to define a language. We'll use machines to do that, but the properties we'll deal with are properties of
L(M), not of M itself.
Proof of Rice's Theorem
Proof: Let P be any nontrivial property of the RE languages.
= H = {s = "M" "w" : Turing machine M halts on input string w}
L1
(?M2)
L2 =
{s = "M" : P(L(M)) = true}
Either P() = true or P() = false. Assume it is false (a matching proof exists if it is true). Since P is nontrivial, there is some
language LP such that P(LP) is true. Let MP be some Turing machine that semidecides LP.
Let construct "M*", which operates as follows:
1. Copy its input y to another track for later.
2. Write w on its input tape and execute M on w.
3. If M halts, put y back on the tape and execute MP.
4. If MP halts on y, accept.
Claim: If M2 exists, then M1 = M2(M(s)) decides L1.
Lecture Notes 26
Undecidability
Why?
Two cases to consider:
"M" "w" H M halts on w M* will halt on all strings that are accepted by MP L(M*) = L(MP) = LP P(L(M*)) =
P(LP) = true M2 decides P, so M2 accepts "M*" M1 accepts.
"M" "w" H M doesnt halt on w M* will halt on nothing L(M*) = P(L(M*)) = P() = false M2 decides
P, so M2 rejects "M*" M1 rejects.
Using Rices Theorem
Theorem: No nontrivial property of the recursively enumerable languages is decidable.

To use Rices Theorem to show that a language L is not recursive we must:
Specify a language property, P(L)
Show that the domain of P is the set of recursively enumerable languages.
Show that P is nontrivial:

P is true of at least one language
P is false of at least one language
Using Rices Theorem: An Example
= {s = "M" : there exists a string on which Turing machine M halts}.

= {s = "M" : L(M) }
Specify a language property, P(L):

P(L) = True iff L

The domain of P is the set of languages semidecided by some TM. This is exactly the set of RE languages.
Show that P is nontrivial:

P is true of at least one language: P({}) = True
P is false of at least one language: P() = False
Inappropriate Uses of Rices Theorem
Example 1:
L
= {s = "M" : M writes a 1 within three moves}.
Specify a language property, P(L)

P(M?) = True if M writes a 1 within three moves,
False otherwise
??? The domain of P is the set of all TMs, not their languages
Example 2:
L
= {s = "M1" "M2": L(M1) = L(M2)}.
Specify a language property. P(L)

P(M1?, M2?) = True if L(M1) = L(M2)
False otherwise
??? The domain of P is RE RE
Lecture Notes 26
Undecidability
Given a Turing Machine M, is L(M) Regular (or Context Free or Recursive)?

Is this problem decidable?
No, by Rices Theorem, since being regular (or context free or recursive) is a nontrivial property of the recursively enumerable
languages.
We can also show this directly (via the same technique we used to prove the more general claim contained in Rices Theorem):
Given a Turing Machine M, is L(M) Regular (or Context Free or Recursive)?

L1 = H = {s = "M" "w" : Turing machine M halts on input string w}
(?M2)
L2 =

{s = "M" : L(M) is regular}
Let be the function that, from "M" and "w", constructs "M*", whose own input is a string
t = "M*" "w*"
M*("M*" "w*") operates as follows:
1. Copy its input to another track for later.
2. Write w on its input tape and execute M on w.
3. If M halts, invoke U on "M*" "w*".
4. If U halts, halt and accept.
If M2 exists, then M2(M*(s)) decides L1 (H).
Why?
If M does not halt on w, then M* accepts (which is regular).
If M does halt on w, then M* accepts H (which is not regular).
Undecidable Problems About Unrestricted Grammars
Given a grammar G and a string w, is w L(G)?

Given a grammar G, is L(G)?
Given two grammars G1 and G2, is L(G1) = L(G2)?
Given a grammar G, is L(G) = ?
Given a Grammar G and a String w, Is w L(G)?
L1
=H=
(?M2)
L2 =
{s = "G" "w" : w L(G)}
Let be the construction that builds a grammar G for the language L that is semidecided by M. Thus
w L(G) iff M(w) halts.
Then
("M" "w") = "G" "w"
Lecture Notes 26
Undecidability
Undecidable Problems About Context-Free Grammars
Given a context-free grammar G, is L(G) = *?

Given two context-free grammars G1 and G2, is L(G1) = L(G2)?
Given two context-free grammars G1 and G2, is L(G1) L(G2) = ?
Is context-free grammar, G ambiguous?
Given two pushdown automata M1 and M2, do they accept precisely the same language?
Given a pushdown automaton M, find an equivalent pushdown automaton with as few states as possible.
Given Two Context-Free Grammars G1 and G2, Is L(G1) = L(G2)?
L1 =
{s = "G" a CFG G and L(G) = *}
(?M2)
L2 =
{s = "G1" "G2" : G1 and G2 are CFGs and L(G1) = L(G2)}
Let append the description of a context free grammar G* that generates *.

Then, ("G") = "G" "G*"
Non-RE Languages
There are an uncountable number of non-RE languages, but only a countably infinite number of TMs (hence RE languages).
The class of non-RE languages is much bigger than that of RE languages!
Intuition: Non-RE languages usually involve either infinite search or knowing a TM will infinite loop to accept a string.
{M: M is a TM that does not halt on the empty tape}
{M: M is a TM and L(M) = *}
{M: M is a TM and there does not exist a string on which M halts}
Proving Languages are not RE

Diagonalization
Complement RE, not recursive
Reduction from a non-RE language
Rices theorem for non-RE languages (not covered)
Diagonalization
L={M: M is a TM and M(M) does not halt} is not RE
Suppose L is RE. There is a TM M* that semidecides L. Is M* in L?
If it is, then M*(M*) halts (by the definition of M* as a semideciding machine for L)
But, by the definition of L, if M* L, then M*(M*) does not halt.
Contradiction. So L is not RE.
(This is a very bare-bones diagonalization proof.)
Diagonalization can only be easily applied to a few non-RE languages.
Lecture Notes 26
Undecidability
10
Complement of an RE, but not Recursive Language

Example: H = {M, w: M does not accept w}
Consider H = {M, w: M is a TM that accepts w}:
H is REit is semidecided by U, the Universal Turing Machine.
H is not recursiveit is equivalent to the halting problem, which is undecidable.
From the theorem, H is not RE.
Reductions and RE Languages
Theorem: If there is a reduction from L1 to L2 and L2 is RE, then L1 is RE.
x
x L1?
M1
y = M2
y L2?
(x)
halt
halt
Theorem: If there is a reduction from L1 to L2 and L1 is not RE, then L2 is not RE.
Reduction from a known non-RE Language
Using a reduction from a non-RE language:
L1 = H = {M, w: Turing machine M does not halt on input string w}

(?M2)
L2 = {M: there does not exist a string on which Turing machine M halts}
Let be the function that, from M and w, constructs M*, which operates as follows:
1. Erase the input tape (M* ignores its input).
2. Write w on the tape
3. Run M on w.
M, w
M1
M*
M*
w
M2
halt
halt
halt halt
M, w H M does not halt on w M* does not halt on any input M* halts on nothing M2 accepts (halts).
M, w H M halts on w M* halts on everything M2 loops.
If M2 exists, then M1(M, w) = M2(M(M, w)) and M1 semidecides L1. Contradiction. L1 is not RE. L2 is not RE.
Lecture Notes 26
Undecidability
11
Language
Summary
IN
OUT
Semidecidable
Enumerable
Unrestricted grammar
Recursively
Enumerable
Decision procedure
Lexicicographically enumerable
Complement is recursively enumer.
Recursive
CF grammar
PDA
Closure
Context Free
Regular expression
FSM
Closure
Lecture Notes 26
Regular
Undecidability
Diagonalization
Reduction
Pumping
Closure
Pumping
Closure
12
Introduction to Complexity Theory

Read K & S Chapter 6.
Most computational problems you will face your life are solvable (decidable). We have yet to address whether a problem is
easy or hard. Complexity theory tries to answer this question.
Recall that a computational problem can be recast as a language recognition problem.
Some easy problems:
Pattern matching
Parsing
Database operations (select, join, etc.)
Sorting
Some hard problems:
Traveling salesman problem
Boolean satisfiability
Knapsack problem
Optimal flight scheduling
Hard problems usually involve the examination of a large search space.
Big-O Notation
Gives a quick-and-dirty measure of function size

Used for time and space metrics
A function f(n) is O(g(n)) whenever there exists a constant c, such that |f(n)| c|g(n)| for all n 0.
(We are usually most interested in the smallest and simplest function, g.)
Examples:
2n3 + 3n2log(n) + 75n2 + 7n + 2000 is O(n3)
752n + 200n5 + 10000 is O(2n)
A function f(n) is polynomial if f(n) is O(p(n)) for some polynomial function p.
If a function f(n) is not polynomial, it is considered to be exponential, whether or not it is O of some exponential function
(e.g. n log n).
In the above two examples, the first is polynomial and the second is exponential.
Comparison of Time Complexities
Speed of various time complexities for different values of n, taken to be a measure of problem size. (Assumes 1 step per
microsecond.)
f(n)\n
10
20
30
40
50
60
.00001 sec.
.00002 sec.
.00003 sec.
.00004 sec.
.00005 sec.
.00006 sec.
n
.0001 sec.
.0004 sec.
.0009 sec.
.0016 sec.
.0025 sec.
.0036 sec.
n2
.001 sec.
.008 sec.
.027 sec.
.064 sec.
.125 sec.
.216 sec.
n3
.1 sec.
3.2 sec.
24.3 sec.
1.7 min.
5.2 min.
13.0 min.
n5
.001 sec.
1.0 sec.
17.9 min.
12.7 days
35.7 yr.
366 cent.
2n
.059 sec.
58 min.
6.5 yr.
3855 cent.
2x108 cent.
1.3x1013 cent.
3n
Faster computers dont really help. Even taking into account Moores Law, algorithms with exponential time complexity are
considered intractable. Polynomial time complexities are strongly desired.
Lecture Notes 27
Complexity Theory
Polynomial Land
If f1(n) and f2(n) are polynomials, then so are:
f1(n) + f2(n)
f1(n) f2(n)
f1(f2(n))
This means that we can sequence and compose polynomial-time algorithms with the resulting algorithms remaining polynomialtime.
Computational Model
For formally describing the time (and space) complexities of algorithms, we will use our old friend, the deciding TM (decision
procedure).
There are two parts:
The problem to be solved must be translated into an equivalent language recognition problem.
A TM to solve the language recognition problem takes an encoded instance of the problem (of size n symbols) as input
and decides the instance in at most TM(n) steps.
We will classify the time complexity of an algorithm (TM) to solve it by its big-O bound on TM(n).
We are most interested in polynomial time complexity algorithms for various types of problems.
Encoding a Problem
Traveling Salesman Problem: Given a set of cities and the distances between them, what is the minimum distance tour a
salesman can make that covers all cities and returns him to his starting city?
Stated as a decision question over graphs: Given a graph G = (V, E), a positive distance function for each edge d: EN+, and a
bound B, is there a circuit that covers all V where d(e) B? (Here a minimization problem was turned into a bound problem.)
A possible encoding the problem:
Give |V| as an integer.
Give B as an integer.
Enumerate all (v1, v2, d) as a list of triplets of integers (this gives both E and d).
All integers are expressed as Boolean numbers.
Separate these entries with commas.
Note that the sizes of most reasonable problem encodings are polynomially related.
What about Turing Machine Extensions?
Most TM extensions are can be simulated by a standard TM in a time polynomially related to the time of the extended machine.
k-tape TM can be simulated in O(T2(n))

Random Access Machine can be simulated in O(T3(n))
(Real programming languages can be polynomially related to the RAM.)

BUT The nondeterminism TM extension is different.
A nondeterministic TM can be simulated by a standard TM in O(2p(n)) for some polynomial p(n).
Some faster simulation method might be possible, but we dont know it.
Recall that a nondeterministic TM can use a guess and test approach, which is computationally efficient at the expense of
many parallel instances.
Lecture Notes 27
Complexity Theory
The Class P
P = { L : there is a polynomial-time deterministic TM, M that decides L }
Roughly speaking, P is the class of problems that can be solved by deterministic algorithms in a time that is polynomially related
to the size of the respective problem instance.
The way the problem is encoded or the computational abilities of the machine carrying out the algorithm are not very important.
Example: Given an integer n, is there a positive integer m, such that n = 4m?
Problems in P are considered tractable or easy.
The Class NP
NP = { L: there is a polynomial time nondeterministic TM, M that decides L }
Roughly speaking, NP is the class of problems that can be solved by nondeterministic algorithms in a time that is polynomially
related to the size of the respective problem instance.
Many problems in NP are considered intractable or hard.
Examples:
Traveling salesman problem: Given a graph G = (V, E), a positive distance function for each edge d: EN+, and a
bound B, is there a circuit that covers all V where d(e) B?
Subgraph isomorphism problem: Given two graphs G1 and G2, does G1 contain a subgraph isomorphic to G2?
The Relationship of P and NP
Recursive
NP
P
Were considering only solvable (decidable) problems.

Clearly P NP.
P is closed under complement.
NP probably isnt closed under complement. Why?
Whether P = NP is considered computer sciences greatest unsolved problem.
Lecture Notes 27
Complexity Theory
Why NP is so Interesting
To date, nearly all decidable problems with polynomial bounds on the size of the solution are in this class.
Most NP problems have simple nondeterministic solutions.
The hardest problems in NP have exponential deterministic time complexities.
Nondeterminism doesnt influence decidability, so maybe it shouldnt have a big impact on complexity.
Showing that P = NP would dramatically change the computational power of our algorithms.
Stephen Cooks Contribution (1971)
Emphasized the importance of polynomial time reducibility.
Pointed out the importance of NP.
Showed that the Boolean Satisfiability (SAT) problem has the property that every other NP problem can be
polynomially reduced to it. Thus, SAT can be considered the hardest problem in NP.
Suggested that other NP problems may also be among the hardest problems in NP.
This hardest problems in NP class is called the class of NP-complete problems.

Further, if any of these NP-complete problems can be solved in deterministic polynomial time, they all can and, by implication,
P = NP.
Nearly all of complexity theory relies on the assumption that P NP.
Polynomial Time Reducibility
A language L1 is polynomial time reducible to L2 if there is a polynomial-time recursive function such that x L1 iff (x)
L2.
If L1 is polynomial time reducible to L2, we say L1 reduces to L2 (polynomial time is assumed) and we write it as L1 L2.
Lemma: If L1 L2, then (L2 P) (L1 P). And conversely, (L1 P) (L2 P).
Lemma: If L1 L2 and L2 L3 then L1 L3.
L1 and L2 are polynomially equivalent whenever both L1 L2 and L2 L1.
Polynomially equivalent languages form an equivalence class. The partitions of this equivalence class are related by the partial
order .
P is the least element in this partial order.
What is the maximal element in the partial order?
Lecture Notes 27
Complexity Theory
The Class NP-Complete

A language L is NP-complete if L NP and for all other languages L NP, L L.
NP-Complete problems are the hardest problems in NP.
Lemma: If L1 and L2 belong to NP, L1 is NP-complete and L1 L2, then L2 is NP-complete.
Thus to prove a language L2 is NP-complete, you must do the following:
1.
Show that L2 NP.
2.
Select a known NP-complete language L1.
3.
Construct a reduction from L1 to L2.
4.
Show that is polynomial-time function.
M1
w
(w)
M2
How do we get started? Is there a language that is NP-complete?

Boolean Satisfiability (SAT)
Given a set of Boolean variables U = {u1, u2, , um} and a Boolean expression in conjunctive normal form (conjunctions of
clausesdisjunctions of variables or their negatives), is there a truth assignment to U that makes the Boolean expression true
(satisfies the expression)?
Note: All Boolean expressions can be converted to conjunctive normal form.
Example: (x1 x2 x3) (x3 x4 x2)
Cooks Theorem: SAT is NP-complete.
1. Clearly SAT NP.
2. The proof constructs a complex Boolean expression that satisfied exactly when a NDTM accepts an input string x
where |w| = n. Because the NDTM is in NP, its running time is O(p(n)). The number of variables is polynomially
related to p(n).
SAT is NP-complete because SAT NP and for all other languages L NP, L SAT.
Reduction Roadmap
SAT
3SAT
3DM
VC
PARTITION
HC
CLIQUE
The early NP-complete reductions took this structure. Each phrase represents a problem. The arrow represents a reduction from
one problem to another.
Today, thousands of diverse problems have been shown to be NP-complete.
Lets now look at these problems.
Lecture Notes 27
Complexity Theory
3SAT (3-satisfiability)
Boolean satisfiability where each clause has exactly 3 terms.
3DM (3-Dimensional Matching)
Consider a set M X Y Z of disjoint sets, X, Y, & Z, such that |X| = |Y| = |Z| = q. Does there exist a matching, a subset
M M such that |M| = q and M partitions X, Y, and Z?
This is a generalization of the marriage problem, which has two sets men & women and a relation describing acceptable
marriages. Is there a pairing that marries everyone acceptably?
The marriage problem is in P, but this 3-sex version of the problem is NP-complete.
PARTITION
Given a set A and a positive integer size, s(a) N+, for each element, a A. Is there a subset A A such that
s(a) = s(a) ?
aA
aA-A
VC (Vertex Cover)
Given a graph G = (V, E) and an integer K, such that 0 < K |V|, is there a vertex cover of size K or less for G, that is, a subset
V V such that |V| K and for each edge, (u, v) E, at least one of u and v belongs to V?
CLIQUE
Given a graph G = (V, E) and an integer J, such that
0 < J |V|, does G contain a clique of size J or more, that is a subset V V such that |V| J and every two vertices in V are
joined by an edge in E?
HC (Hamiltononian Circuit)
Given a graph G = (V, E), does there exist a Hamiltonian circuit, that is an ordering <v1, v2, , vn> of all V such that
(v|V|, v1) E and (vi, vi+1) E for all i, 1 i < |V|?
Traveling Salesman Prob. is NP-complete
Given a graph G = (V, E), a positive distance function for each edge d: EN+, and a bound B, is there a circuit that covers all V
where d(e) B?
To prove a language TSP is NP-complete, you must do the following:
1. Show that TSP NP.
2. Select a known NP-complete language L1.
3. Construct a reduction from L1 to TSP.
4. Show that is polynomial-time function.
TSP NP: Guess a set of roads. Verify that the roads form a tour that hits all cities. Answer yes if the guess is a tour and the
sum of the distances is B.
Reduction from HC: Answer the Hamiltonian circuit question on G = (V, E) by constructing a complete graph where roads
have distance 1 if the edge is in E and 2 otherwise. Pose the TSP problem, is there a tour of length |V|?
Lecture Notes 27
Complexity Theory
Notes on NP-complete Proofs

The more NP-complete problems are known, the easier it is to find a NP-complete problem to reduce from.
Most reductions are somewhat complex.
It is sufficient to show that a restricted version of the problem is NP-complete.
More Theory
NP has a rich structure that includes more than just P and NP-complete. This structure is studied in later courses on the theory of
computation.
The set of recursive problems outside of NP (and including NP-complete) are called NP-hard. There is a proof technique to
show that such problems are at least as hard as NP-complete problems.
Space complexity addresses how much tape does a TM use in deciding a language. There is a rich set of theories surrounding
space complexity.
Recursive
NP-hard
NP
NP-complete
(part of NP-hard)
P
Dealing with NP-completeness

You will likely run into NP-complete problems in your career. For example, most optimization problems are NP-complete.
Some techniques for dealing with intractable problems:
Recognize when there is a tractable special case of the general problem.
Use other techniques to limit the search space.
For optimization problems, seek a near-optimal solution.
The field of linear optimization springs out of the latter approach. Some linear optimization solutions can be proven to be near
optimal.
A branch of complexity theory deals with solving problems within some error bound or probability.
For more: Read Computers and Intractability: A Guide to the Theory of NP-Completeness by Michael R. Garey and David S.
Johnson, 1979.
Lecture Notes 27
Complexity Theory
II. Homework
CS 341 Homework 1
Basic Techniques
1. What are these sets? Write them using braces, commas, numerals, (for infinite sets), and only.
(a) ({1, 3, 5} {3, 1}) {3, 5, 7}
(b) {{3}, {3, 5}, {{5, 7}, {7, 9}}}
(c) ({1, 2, 5} - {5, 7, 9}) ({5, 7, 9} - {1, 2, 5})
(d) 2{7, 8, 9} - 2{7, 9}
(e) 2
(f) {x : y N where x = y2}
(g) {x : x is an integer and x2 = 2}
2. Prove each of the following:

(a) A (B C) = (A B) (A C)
(b) A (B C) = (A B) (A C)
(c) A (A B) = A
(d) A (A B) = A
(e) A - (B C) = (A - B) (A - C)
3. Write each of the following explicitly:
(a) {1} {1, 2} {1, 2, 3}
(b) {1, 2}
(c) 2{1,2} {1, 2}
4. Let R = {(a, b), (a, c), (c, d), (a, a), (b, a)}. What is R R, the composition of R with itself? What is R-1, the
inverse of R? Is R, R R, or R-1 a function?
5. What is the cardinality of each of the following sets? Justify your answer.
(a) S = N - {2, 3, 4}
(b) S = {, {}}
(c) S = 2{a, b, c}
(d) S = {a, b, c} {1, 2, 3, 4}
(e) S = {a, b, , z} N
6. Consider the chart of example relations in Section 3.2. For the first six, give an example that proves that the
relation is missing each of the properties that the chart claims it is missing. For example, show that Mother-of
is not reflexive, symmetric, or transitive.
7. Let A, B be two sets. If 2A = 2B, must A = B? Prove your answer.
8. For each of the following sets, state whether or not it is a partition of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}.
(a) {{0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}, {9}, {10}}
(b) {, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}, {9}, {10}}
(c) {{1, 2}, {3, 4}, {5, 6}, {7, 8}, {9, 10}}
(d) {{1, 2}, {2, 3}, {3, 4}, {4, 5}, {5, 6}, {6, 7}, {7, 8}, {8, 9}, {9, 10}}
9. For each of the following relations, state whether it is a partial order (that is not also total), a total order, or
neither. Justify your answer.
(a) DivisibleBy, defined on the natural numbers. (x, y) DivisibleBy iff x is evenly divisible by y. So, for
example, (9, 3) DivisibleBy but (9, 4) DivisibleBy.
Homework 1
Basic Techniques
(b) LessThanOrEqual defined on ordered pairs of natural numbers. (a, b) (x, y) iff a x or (a = x and
b y). For example, (1,2) (2,1) and (1,2) (1,3).
(c) The relation defined by the following boolean matrix:
1
1
1
1
1
1
1
1
10. Are the following sets closed under the following operations? If not, what are the respective closures?
(a) The odd integers under multiplication.
(b) The positive integers under division.
(c) The negative integers under subtraction.
(d) The negative integers under multiplication.
(e) The odd length strings under concatenation.
11. What is the reflexive transitive closure R* of the relation
R = {(a, b), (a, c), (a, d), (d, c), (d, e)} Draw a directed graph representing R*.
12. For each of the following relations R, over some domain D, compute the reflexive, symmetric, transitive
closure R. Try to think of a simple descriptive name for the new relation R. Since R must be an equivalence
relation, describe the partition that R induces on D.
(a) Let D be the set of 50 states in the US. xy, xRy iff x shares a boundary with y.
(b) Let D be the natural numbers. xy, xRy iff y = x+3.
(c) Let D be the set of strings containing no symbol except a. xy, xRy iff y = xa. (i.e., if y equals x
concatenated with a).
13. Consider an infinite rectangular grid (like an infinite sheet of graph paper). Let S be the set of intersection
points on the grid. Let each point in S be represented as a pair of (x,y) coordinates where adjacent points differ
in one coordinate by exactly 1 and coordinates increase (as is standard) as you move up and to the right.
(a) Let R be the following relation on S: (x1,y1)(x2,y2), (x1,y1)R(x2,y2) iff x2= x1+1 and y2=y1+1. Let R be
the reflexive, symmetric, transitive closure of R. Describe in English the partition P that R induces on S. What
is the cardinality of P?
(b) Let R be the following relation on S: (x1,y1)(x2,y2), (x1,y1)R(x2,y2) iff (x2= x1+1 and y2=y1+1) or (x2= x11 and y2=y1+1). Let R be the reflexive, symmetric, transitive closure of R. Describe in English the partition P
that R induces on S. What is the cardinality of P?
(c) Let R be the following relation on S: (x1,y1)(x2,y2), (x1,y1)R(x2,y2) iff (x2,y2) is reachable from (x1,y1) by
moving two squares in any one of the four directions and then one square in a perpendicular direction. Let R
be the reflexive, symmetric, transitive closure of R. Describe in English the partition P that R induces on S.
What is the cardinality of P?
14. Is the transitive closure of the symmetric closure of a binary relation necessarily reflexive? Prove it or give
a counterexample.
15. Give an example of a binary relation that is not reflexive but has a transitive closure that is reflexive.
16. For each of the following functions, state whether or not it is (i) one-to-one, (ii) onto, and (iii) idempotent.
Justify your answers.
(a) +: P P P, where P is the set of positive integers, and
+(a, b) = a + b (In other words, simply addition defined on the positive integers)
(b) X : B B B, where B is the set {True, False}
Homework 1
Basic Techniques
X(a, b) = the exclusive or of a and b

17. Consider the following set manipulation problems:
(a) Let S = {a, b}. Let T = {b, c}. List the elements of P, defined as
P = 2S 2T.
(b) Let Z be the set of integers. Let S = {x Z: y Z and x = 2y}. Let T = {x Z: y Z and x = 2y}.
Let W = S T. Describe W in English. List any five consecutive elements of W. Let X = T S. What is X?
Solutions
1. (a)
(b)
(c)
(d)
(e)
(f)
(g)
{3, 5}
{3, 5, 7}
{1, 2, 7, 9}
{8}, {7, 8}, {8, 9}, {7, 8, 9}
{}
{0, 1, 4, 9, 25, 36} (the perfect squares)
(since the square root of 2 is not an integer)
2. (a)
A (B C)
= (B C) A
= (B A) (C A)
= (A B) (A C)
commutativity
distributivity
commutativity
(b)
A (B C)
= (B C) A
= (B A) (C A)
= (A B) (A C)
commutativity
distributivity
commutativity
(c)
A (A B)
= (A B) A
=A
commutativity
absorption
3. (a)
(b)
(c)
{(1,1,1), (1,1,2), (1,1,3), (1,2,1), (1,2,2),, (1,2,3)}
{(,1), (,2), ({1}, 1), ({1}, 2), ({2}, 1), ({2}, 2), ({1,2}, 1), ({1,2}, 2)}
4.
R R = {(a, a), (a, d), (a, b), (b, b), (b, c), (b, a), (a, c)}
R inverse = {(b, a), (c, a), (d, c), (a, a), (a, b)}
None of R, R R or R inverse is a function.
5. (a) S = {0, 1, 5, 6, 7, }. S has the same number of elements as N. Why? Because there is a bijection
between S and N: f: S N, where f(0) = 0, f(1) = 1, x 5, f(x) = x - 3. So |S| = 0.
(b) 2.
(c) S = all subsets of {a, b, c}. So S = {, {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c}}. So |S| = 8. We
could also simply have used the fact that the cardinality of the power set of a finite set of cardinality c
is 2c.
(d) S = {(a, 1), (a, 2), (a, 3), (a, 4), (b, 1), (b, 2), (b, 3), (b, 4), (c, 1), (c, 2), (c, 3), (c, 4)}. So |S| = 12. Or
we could have used the fact that, for finite sets, |A B| = |A| * |B|.
(e) S = {(a, 0), (a, 1), , (b, 0), (b, 1),} Clearly S contains an infinite number of elements. But are there
the same number of elements in S as in N, or are there more (26 times more, to be precise)? The
answer is that there are the same number. |S| = 0. To prove this, we need a bijection from S to N. We
can define this bijection as an enumeration of the elements of S:
(a, 0), (b, 0), (c, 0),
(First enumerate all 26 elements of S that have 0 as their second element)
Homework 1
Basic Techniques
(a, 1), (b, 1), (c, 1),

and so forth.
(Next enumerate all 26 elements of S that have 1 as their second element)
6. Mother-of:
Not reflexive: Eve is not the mother of Eve (in fact, no one is her own mother).
Not symmetric: mother-of(Eve, Cain), but not Mother-of(Cain, Eve).
Not transitive: Each person has only one mother, so if Mother-of(x, y) and Mother-of(y, z),
the only way to have Mother-of(x, z) would be if x and y are the same person, but we know
that that's not possible since Mother-of(x, y) and no one can be the mother of herself).
Would-recognize-picture-of:
Not symmetric: W-r-p-o(Elaine, Bill Clinton), but not W-r-p-o (Bill Clinton, Elaine)
Not transitive: W r-p-o(Elaine, Bill Clinton) and W r-p-o(Bill Clinton, Bill's mom) but not
W-r-p-o(Elaine, Bill's mom)
Has-ever-been-married-to:
Not reflexive: No one is married to him or herself.
Not transitive: H-e-b-m-t(Dave, Sue) and H-e-b-m-t(Sue, Jeff) but not
H-e-b-m-t(Dave, Jeff)
Ancestor-of: Not reflexive: not Ancestor-of(Eve, Eve) (in fact, no one is their own ancestor).
Not symmetric: Ancestor-of(Eve, Cain) but not Ancestor-of(Cain, Eve)
Hangs-out-with: Not transitive: Hangs-out-with(Bill, Monica) and Hangs-out-with(Monica, Linda Tripp),
but not Hangs-out-with(Bill, Linda Tripp).
Less-than-or-equal-to: Not symmetric: 1 2, but not 2 1.
7. Yes, if 2A = 2B, then A must equal B. Suppose it didn't. Then there is some element x that is in one set but
not the other. Call the set x is in A. Then 2A must contain {x}, which must not be in 2B, since x B. This
would mean that 2A 2B, which contradicts our premise.
8. (a)
(b)
(c)
(d)
yes
no, since no element of a partition can be empty.
no, 0 is missing
no, since, each element of the original set S must appear in only one element of a partition of S.
9. (a) DivisibleBy is a partial order. x (x, x) DivisibleBy, so DivisibleBy is reflexive. For x to be

DivisibleBy y, x must be greater than or equal to y. So the only way for both (x, y) and (y, x) to be in
DivisibleBy is for x and y to be equal. Thus DivisibleBy is antisymmetric. And if x is DivisibleBy y and y is
DivisibleBy z, then x is DivisibleBy z. So DivisibleBy is transitive. But DivisibleBy is not a total order. For
example neither (2, 3) nor (3, 2) is in it.
(b) LessThanOrEqual defined on ordered pairs is a total order. This is easy to show by relying on the fact
that for the natural numbers is a total order.
(c) This one is not a partial order at all because, although it is reflexive and antisymmetric, it is not
transitive. For example, it includes (4, 1) and (1, 3), but not (4, 3).
10. (a) The odd integers are closed under multiplication. Every odd integer can be expressed as 2n+1 for some
value of n N. So the product of any two odd integers can be written as (2n+1)(2m+1) for some values of n
and m. Multiplying this out, we get 4(n+m) +2n + 2m +1, which we can rewrite as 2(2(n+m) + n + m) +1,
which must be odd.
(b) The positive integers are not closed under division. To show that a set is not closed under an operation, it
is sufficient to give one counterexample. 1/2 is not an integer. The closure of the positive integers under
division is the positive rationals.
(c) The negative integers are not closed under subtraction. -2 - (-4) = 2. The closure of the negative numbers
under subtraction is the integers.
(d) The negative integers are not closed under multiplication. -2 * -2 = 4. The closure of the negative
numbers under multiplication is the nonzero integers. Remember that the closure is the smallest set that
Homework 1
Basic Techniques
contains all the necessary elements. Since it is not possible to derive zero by multiplying two negative
numbers, it must not be in the closure set.
(e) The odd length strings are not closed under concatenation. "a" || "b" = "ab", which is of length 2. The
closure is the set of strings of length 2. Note that strings of length 1 are not included. Why?
11. R* = R {(x, x) : x {a, b, c, d, e}} {(a, e)}
12. (a) The easiest way to start to solve a problem like this is to start writing down the elements of R and see if
a pattern emerges. So we start with the elements of R: {(TX, LA), (LA, TX), (TX, NM), (NM, TX), (LA, Ark),
(Ark, LA), (LA Miss), (Miss, LA) }. To construct R, we first add all elements of the form (x, x), so we add
(TX,TX), and so forth. Then we add the elements required to establish transitivity:
(NM, TX), (TX, LA) (NM, TX)
(TX, LA), (LA, Ark) (TX, Ark)
(NM, TX), (TX, Ark) (NM, Ark), and so forth.
If we continue this process, we will see that the reflexive, symmetric, transitive closure R relates all states
except Alaska and Hawaii to each other and each of them only to themselves. So R can be described as
relating two states if its possible to drive from one to the other without leaving the country. The partition is:
[Alaska]
[Hawaii]
[all other 48 states]
(b) R includes, for example {(0, 3), (3, 6), (6, 9), (9, 12) }. When we compute the transitive closure, we
add, among other things {(0, 6), (0, 9), (0,12)}. Now try this starting with (1,4) and (2, 5). Its clear that x,y,
xRy iff x = y (mod 3). In other words, two numbers are related iff they have the same remainder mod 3. The
partition is:
[0, 3, 6, 9, 12 ]
[1, 4, 7, 10, 13 ]
[2, 5, 8, 11, 14 ]
(c) R relates all strings composed solely of as to each other. So the partition is
[, a, aa, aaa, aaaa, ]
13. (a) Think of two points being related via R if you can get to the second one by starting at the first and
moving up one square and right one square. When we add transitivity, we gain the ability to move diagonally
by two squares, or three, or whatever. So P is an infinite set. Each element of P consists of the set of points
that fall on an infinite diagonal line running from lower left to upper right.
(b) Now we can more upward on either diagonal. And we can move up and right followed by up and left,
and so forth. The one thing we cant do is move directly up or down or right or left exactly one square. So take
any given point. To visualize the points to which it is related under R, imagine a black and white chess board
where the squares correspond to points on our grid. Each point is related to all other points of the same color.
Thus the cardinality of P is 2.
(c) Now every point is related to every other point. The cardinality of P is 1.
14. You might think that for all relations R on some domain D, the transitive closure of the symmetric closure
of R (call it TC(SC(R))) must be reflexive because for any two elements x, y D such that (x, y) R, we'll
have (x, y), (y, z) SC(R) and therefore (x, x), (y, y) TC(SC(R)). This is all true, but does not prove that for
all z D, (z, z) TC(SC(R)). Why not? Suppose there is a z D such that there is no y D for which (y, z)
R or (z, y) R. (If you look at the graph of R, z is an isolated vertex with no edges in or out.) Then (z, z)
TC(SC(R)). So the answer is no, with R = on domain {a} as a simple counterexample: TC(SC(R)) = , yet
it should contain (a, a) if it were reflexive.
15. R = {(a, b), (b, a)} on domain {a, b} does the trick easily.
Homework 1
Basic Techniques
16. (a) (i) + is not one-to-one. For example +(1, 3) = +(2, 2) = 4.

(ii) + is not onto. There are no two positive integers that sum to 1.
(iii) + is not idempotent. +(1, 1) 1.
(b) (i) X is not one-to-one. For example True X True = False X False = False.
(ii) X is onto. Proof: True X True = False. True X False = True. In general, when the domain is a
finite set, it's easy to show that a function is onto: just show one way to derive each element.
(iii) X is not idempotent. True X True = False.
17. (a) P = {,{b}}
(b) S is the set of even numbers. T is the set powers of 2. W is the set of even numbers that are not powers
of 2. So W = { .-6, -4, -2, 0, 6, 10, 12, 14, 18, }. X is the set of numbers that are powers of 2 but are not
even. Theres only one element of X. X = {1}.
Homework 1
Basic Techniques
CS 341 Homework 2
1. Let = {a, b}. Let L1 = {x *: |x| < 4}. Let L2 = {aa, aaa, aaaa}. List the elements in each of the
following languages L:
(a) L3 = L1 L2
(b) L4 = L1 L2
(c) L5 = L1 L4
(d) L6 = L1 - L2
2. Consider the language L = anbncm. Which of the following strings are in L?
(a)
(b) ab
(c) c
(d) aabc
(e) aabbcc
(f) abbcc
3. It probably seems obvious to you that if you reverse a string, the character that was originally first becomes
last. But the definition we've given doesn't say that; it says only that the character that was originally last
becomes first. If we want to be able to use our intuition about what happens to the first character in a proof, we
need to turn it into a theorem. Prove x,a where x is a string and a is a single character, (ax)R = xRa.
4. For each of the following binary functions, state whether or not it is (i) one-to-one, (ii) onto, (iii) idempotent,
(iv) commutative, and (v) associative. Also (vi) state whether or not it has an identity, and, if so, what it is.
Justify your answers.
(a) || : S S S, where S is the set of strings of length 0
||(a, b) = a || b (In other words, simply concatenation defined on strings)
(b) || : L L L where L is a language over some alphabet
||(a, b) = {w *: w = x || y for some x a and y b} In other words, the concatenation of two
languages A and B is the set of strings that can be derived by taking a string from A and then
concatenating onto it a string from B.
5. We can define a unary function F to be self-inverse iff x Domain(F) F(F(x)) = x. The Reverse function
on strings is self-inverse, for example.
(a) Give an example of a self-inverse function on the natural numbers, on sets, and on booleans.
(b) Prove that the Reverse function on strings is self-inverse.
Solutions
1. First we observe that L1 = {, a, b, aa, ab, ba, bb, aaa, aab, aba, abb, baa, bab, bba, bbb}.
(a) L3 = {, a, b, aa, ab, ba, bb, aaa, aab, aba, abb, baa, bab, bba, bbb, aaaa}
(b) L4 = {aa, aaa}
(c) L5 = every way of selecting one element from L1 followed by one element from L4:
{aa, aaa, baa, aaaa, abaa, baaa, bbaa, aaaaa, aabaa, abaaa, abbaa, baaaa, babaa, bbaaa, bbbaa}
{aaa, aaaa, baaa, aaaaa, abaaa, baaaa, bbaaa, aaaaaa, aabaaa, abaaaa, abbaaa, baaaaa, babaaa,
bbaaaa, bbbaaa}. Note that we've written aa, just to make it clear how this string was derived. It
should actually be written as just aa. Also note that some elements are in both of these sets (i.e.,
there's
more than one way to derive them). Eliminating duplicates (since L is a set and thus does not contain
duplicates), we get:
{aa, aaa, baa, aaaa, abaa, baaa, bbaa, aaaaa, aabaa, abaaa, abbaa, baaaa, babaa, bbaaa, bbbaa, aaaaaa,
aabaaa, abaaaa, abbaaa, baaaaa, babaaa, bbaaaa, bbbaaa}
(d) L6 = every string that is in L1 but not in L2: {, a, b, ab, ba, bb, aab, aba, abb, baa, bab, bba, bbb}.
Homework 2
2. (a)
(b)
(c)
(d)
(e)
(f)
Yes. n = 0 and m = 0.
Yes. n = 1 and m = 0.
Yes. n = 0 and m = 1.
No. There must be equal numbers of a's and b's.
Yes. n = 2 and m = 2.
No. There must be equal numbers of a's and b's.
3. Prove: x,a where x is a string and a is a single character, (ax)R = xRa. We'll use induction on the length of
x. If |x| = 0 (i.e, x = ), then (a)R = a = Ra. Next we show that if this is true for all strings of length n, then it
is true for all strings of length n + 1. Consider any string x of length n + 1. Since |x| > 0, we can rewrite x as yb
for some single character b.
(ax)R = (ayb)R
Rewrite of x as yb
= b(ay)R
Definition of reversal
= b(yRa)
Induction hypothesis (since |x| = n + 1, |y| = n)
= (b yR) a
Associativity of concatenation
= xRa
Definition of reversal: If x = yb then xR = byR
4. (a)
(b)
(i) || is not one-to-one. For example, ||(ab, c) = ||(a, bc) = abc.

(ii) || is onto. Proof: s S, ||(s, ) = s, so every element of s can be generated.
(iii) || is not idempotent. ||(a, a) a.
(iv) || is not commutative. ||(ab, cd) (cd, ab)
(v) || is associative.
(vi) || has as both a left and right identity.
(i) || is not one to one. For example, Let = {a, b, c}. ||({a}, {bc}) = {abc} = ||({ab}, {c})
(ii) || is onto. Proof: L *, ||(L, {}) = L, so every element of s can be generated. Notice that this
proof is very similar to the one we used to show that concatenation of strings is onto. Both proofs rely
on
the fact that is an identity for concatenation of strings. Given the way in which we defined
concatenation of languages as the concatenation of strings drawn from the two languages, {} is an
identity for concatenation of languages and thus it enables us to prove that all languages can be derived
from the concatenation operation.
(iii) || is not idempotent. ||({a}, {a}) = {aa}
(iv) || is not commutative. ||({a}, {b}) = {ab}. But ||({b}, {a}) = {ba}.
(v) || is associative.
(vi) || has {} as both a left and right identity.
5. (a)
(b)
Integers: F(x) = -x is self-inverse. Sets: Complement is self-inverse. Booleans: Not is self-inverse.

We'll prove this by induction on the length of the string.
Base case: If |x| = 0 or 1, then xR = x. So (xR)R = xR = x.
Show that if this is true for all strings of length n, then it is true for all strings of length n + 1. Any
string s of length n + 1 can be rewritten as xa for some single character a. So now we have:
sR
= a xR
definition of string reversal
R R
(s )
= (a xR)R
substituting a xR for sR
= (xR)Ra
by the theorem we proved above in (3)
= xa
induction hypothesis
=s
since xa was just a way of rewriting s
Homework 2
CS 341 Homework 3
1. Describe in English, as briefly as possible, each of the following (in other words, describe the language
defined by each regular expression):
(a) L( ((a*a) b) b )
(b) L( (((a*b*)*ab) ((a*b*)*ba))(b a)* )
2. Rewrite each of these regular expressions as a simpler expression representing the same set.
(a) * a* b* (a b)*
(b) ((a*b*)* (b*a*)*)*
(c) (a*b)* (b*a)*
3. Let = {a, b}. Write regular expressions for the following sets:
(a) All strings in * whose number of a's is divisible by three.
(b) All strings in * with no more than three a's.
(c) All strings in * with exactly one occurrence of the substring aaa.
4. Which of the following are true? Prove your answer.
(a) baa a*b*a*b*
(b) b*a* a*b* = a* b*
(c) a*b* c*d* =
(d) abcd (a(cd)*b)*
5. Show that L((a b)*) = L(a* (ba*)*).
6. Consider the following:
(a) ((a b) (ab))*
(b) (a+ anbn)
(c) ((ab)* )
(d) (((ab) c)* (b c*))
(e) (* (bb*))
(i) Which of the above are pure regular expressions?
(ii) For each of the above that is a regular expression, give a simplified equivalent pure regular expression.
(iii) Which of the above represent regular languages?
7. True - False: For all languages L1, L2, and L3
(a) (L1L2)* = L1*L2*
(b) (L1 L2)* = L1* L2*
(c) (L1 L2) L3 = L1 L3 L2 L3
(d) (L1 L2) L3 = (L1 L3) (L2 L3)
(e) L1+)* = L1*
(f) (L1+)+ = L1+
(g) (L1*)+ = (L1+)*
(h) L1* = L1+
(i) (ab)*a = a(ba)*
(j) (a b)* b (a b)* = a* b (a b)*
(k) [(a b)* b (a b)* (a b)* a (a b)*] = (a b)*
(l) [(a b)* b (a b)* (a b)* a (a b)*] = (a b)+
(m) [(a b)* b a (a b)* a*b*] = (a b)*
Homework 3
(n) (L1L2L3)* = L1*L2*L3*

(o) (L1* L3*) = (L1* L3*)*
(p) L1* L1 = L1+
(q) (L1 L2)* = (L2 L1)*
(r) L1* (L2 L3)+ = (L1* L2+ L1* L3+)
(s) L1* =
(t) L1* = {}
(u) (L1 - L2) = (L2 - L1)
(v) ((L1 L2) (L1 L3))* = (L1 (L2 L3))*
8. Let L = {w {a, b}* : w contains bba as a substring}. Find a regular expression for {a, b}* - L.
9. Let = {a,b}. For each of the following sets of strings (i.e., languages) L, first indicate which of the
example strings are in the language and which are not. Then, if you can, write a concise description, in English,
of the strings that are in the language.
Example strings: (1) aaabbb, (2) abab, (3) abba, (4)
(a) L = {w : for some u *, w = uRu}
(b) L = {w : ww = www}
(c) L = {w : for some u *, www = uu}
10. Write a regular expression for the language consisting of all odd integers without leading zeros.
11. Let = {a, b}. Let L = {, a, b}. Let R be a relation defined on * as follows: xy, xRy iff y = xb. Let R
be the reflexive, transitive closure of L under R. Let L = {x : y L such that yRx}. Write a regular
expression for L.
Solutions
1. (a)
(b)
Any string of a's and/or b's with zero or more a's followed by a single b.
Any string of a's and/or b's with at least one occurrence of ab or ba.
* = {}, and (a b)*.

a* (a b)*.
b* (a b)*. So since the first three terms describe subsets of the last one, unioning them into the last
one doesn't add any elements. Thus we can write simply (a b)*.
(b) To solve this one, we'll use some identities for regular expressions. We don't have time for an extensive
study of such identities, but these are useful ones:
((a*b*)* (b*a*)*)*
=
Using (A*B*)* = (A B)* (Both simply describe any string that is composed of elements of
A and elements of B concatenated together in any order)
((a b)*(b a)*)*
=
Using (A B) = (B A) (Set union is commutative)
((a b)*(a b)*)*
=
Using A*A* = A*
((a b)*)*
=
Using (A*)* = A*
(a b)*
2. (a)
Homework 3
(c) (a*b)* (b*a)* = (a b)* (In other words, all strings over {a, b}.) How do we know that? (a*b)* is the
union of and all strings that end in b. (b*a)* is the union of and all strings that end in a. Clearly any string
over {a, b} must either be empty or it must end in a or b. So we've got them all.
3. (a) The a's must come in groups of three, but of course there can be arbitrary numbers of b's everywhere. So:
(b*ab*ab*a)*b*
Since the first expression has * around it, it can occur 0 or more times, to give us any number of a's
that is divisible by 3.
(b) Another way to think of this is that there are three optional a's and all the b's you want. That gives us:
b* (a ) b* (a ) b* (a ) b*
(c) Another way to think of this is that we need one instance of aaa. All other instances of aa must occur
with
either b or end of string on both sides. The aaa can occur anywhere so we'll plunk it down, then list the
options for everything else twice, once on each side of it:
(ab aab b)*
aaa
(ba baa b)*
4. (a) True. Consider the defining regular expression: a*b*a*b*. To get baa, take no a's, then one b, then two
a's then no b's.
(b) True. We can prove that two sets X and Y are equal by showing that any string in X must also be in Y
and vice versa. First we show that any string in b*a* a*b* (which we'll call X) must also be in a* b*
(which we'll call Y). Any string in X must have two properties: (from b*a*): all b's come before all a's; and
(from a*b*): all a's come before all b's. The only way to have both of these properties simultaneously is to be
composed of only a's or only b's. That's exactly what it takes to be in Y.
Next we must show that every string in Y is in X. Every string in Y is either of the form a* or b*. All strings
of the form a* are in X since we simply take b* to be b0, which gives us a* a* = a*. Similarly for all strings
of the form b*, where we take a* to be a0.
(c) False. Remember that to show that any statements is false it is sufficient to find a single counterexample:
a*b* and c*d*. Thus a*b* c*d* , which is therefore not equal to .
(d) False. There is no way to generate abcd from (a(cd)*b)*. Let's call the language generated by
(a(cd)*b)* L. Notice that every string in L has the property that every instance of (cd)* is immediately
preceded by a. abcd does not possess that property.
5. That the language on the right is included in the language on the left is immediately apparent since every
string in the right-hand language is a string of a's and b's. To show that any string of a's and b's is contained in
the language on the right, we note that any such string begins with zero or more a's. If there are no b's, then the
string is contained in a*. If there is at least one b, we strip off any initial a's as a part of a* and examine the
remainder. If there are no more b's, the remainder is in ba*. If there is at least one more b to the right, then we
strip of the initial b and any following consecutive a's (a string in ba*) and examine the remainder. Repeat the
last two steps until the end of the string is reached. Thus, every string of a's and b's is included in the language
on the right.
6. (i) a, c, e (b contains superscript n; d contains )
(ii) (a) = (a b)*
(c) =
(e) = b*
(iii) a, c, d, e (b is {ambn : m > n}, which is not regular)
7. (a) F, (b) F, (c) T, (d) F, (e) T, (f) T, (g) T, (h) F, (i) T, (j) T, (k) F, (l) T, (m) T, (n) F, (o) F, (p) T (by def. of
+), (q) T, (r) F, (s) T, (t) F, (u) F, (v) T.
Homework 3
8. (a ba)* ( b bbb*) = (a ba)*b*

9. (a) (1) no (2) no, (3) yes, (4) yes
L is composed of strings whose second half is the reverse of the first half.
(b) (1) no (2) no (3) no (4) yes
L contains only the empty string.
(c) (1) no (2) yes (3) no (4) yes
L contains strings of even length whose first half is the same as the second half. To see why this is so,
notice that |uu| is necessarily even, since it's |u| times 2. So we must assure that |www| is also even. This will
only happen if |w| is even. To discover what u is for any proposed w, we must first write out www. Then we
split it in half and call that u. Suppose that w can be described as the concatenation of two strings of equal
length, r and s. (We know we can do this, since we already determined that |w| is even.) Then w will be equal
to rs and www will be rsrsrs. So u must equal both rsr and srs. There can only be such a u if r and s are the
same.
10. ( ((1-9)(0-9)*))(13579), or, without using or the dash notation,
(13579)
((123456789) (0123456789)* (13579))
11. Whew. A lot of formalism. The key is to walk through it one step at a time. Its good practice. R relates
pairs of strings that are identical except that the second one has one extra b concatenated on the end. So it
includes, for example, {(a, ab), (ab, abb), (abb, abbb), (b, bb), (bb, bbb), }. Now we have to compute R.
Consider the element a. First, we must add (a, a) to make R reflexive. Now we must consider transitivity. R
gives us (a, ab). But it also gives us (ab, abb), so, by transitivity, R must contain (a, abb). In fact, a must be
related to all strings in the language ab*. Similarly must be related to all strings in b* or simply b*. And b
must be related to all strings in bb*. We could also notice many other things, such as the fact that ab is related
to all strings in abb*, but we dont need to bother to do that to solve this problem. What we need to do is to
figure out what L is. Its all strings that are related via R to some element of L. There are three elements of L,
{, a, b}. So L = b* ab* bb*. But every string in bb* is also in b*, so we can simplify to b* ab*.
Homework 3
CS 341 Homework 4
1. If M is a deterministic finite automaton. Under exactly what circumstances is L(M)?
2. Describe informally the languages accepted by each of the following deterministic FSMs:
(from Elements of the Theory of Computation, H. R. Lewis and C. H. Papdimitriou, Prentice-Hall, 1998.)
Homework 4
3. Construct a deterministic FSM to accept each of the following languages:

(a) {w {a, b}* : each a in w is immediately preceded and followed by a b}
(b) {w {a, b}* : w has abab as a substring}
(c) {w {a, b}* : w has neither aa nor bb as a substring}
(d) {w {a, b}* : w has an odd number of a's and an even number of b's}
(e) {w {a, b}* : w has both ab and ba as substrings}
4. Construct a deterministic finite state transducer over {a, b} for each of the following tasks:
(a) On input w produce an, where n is the number of occurrences of the substring ab in w.
(b) On input w produce an, where n is the number of occurrences of the substring aba in w.
(c) On input w produce a string of length w whose ith symbol is an a if i = 1 or if i > 1 and the ith and (i-1)st
symbols of w are different; otherwise, the ith symbol of the output is b.
5. Construct a dfa accepting L = {w {a, b}* : w contains no occurrence of the string ab}.
6. What language is accepted by the following fsa?
b
1
a
a
4
a
b
3
a
a
6
a,b
7. Give a dfa accepting {x {a, b}* : at least one a in x is not immediately followed by b}.
8. Let L = {w {a, b}* : w does not end in ba}.
(a) Construct a dfa accepting L.
(b) Give a regular expression for L.
9. Consider L = {anbn : 0 n 4}
(a) Show that L is regular by giving a dfa that accepts it.
(b) Give a regular expression for L.
10. Construct a deterministic finite state machine to accept strings that correspond to odd integers without
leading zeros.
11. Imagine a traffic light. Let = {a}. In other words, the input consists just of a string of a's. Think of
each a as the output from a timer that signals the light to change. Construct a deterministic finite state
transducer whose outputs are drawn from the set {Y, G, R} (corresponding to the colors yellow, green, and
red). The outputs of the transducer should correspond to the standard traffic light behavior.
12. Recall the finite state machine that we constructed in class to accept $1.00 in change or bills. Modify
the soda machine so that it actually does something (i.e., some soda comes out) by converting our finite state
acceptor to a finite state transducer. Let there be two buttons, one for Coke at $.50 and one for Water at
$.75 (yes, it's strange that water costs more than Coke. The world is a strange place). In any case, there will
now be two new symbols in the input alphabet, C and W. The machine should behave as follows:
Homework 4
The machine should keep track of how much money has been inserted. If it ever gets more than $1.50, it
should spit back enough to get it under $1.00 but keep it above $.75.
If the Coke or Water button is pushed and enough money has been inserted, the product and the change
should be output.
If a button is pushed and there is not enough money, the machine should remember the button push and
wait until there is enough money, at which point it should output the product and the change.
13. Consider the problem of designing an annoying buzzer that goes off whenever you try to drive your car
and you're not wearing a seat belt. (For simplicity, we'll just worry about the driver's possible death wish. If
you want to make this harder, you can worry about the other seats as well.) Design a finite state transducer
whose inputs are drawn from the alphabet {KI, KR, SO, SU, BF, BU}, representing the following events,
respectively: "key just inserted into ignition", "key just removed from ignition", "seat just became
occupied", "seat just became unoccupied", "belt has just been fastened", and "belt has just been unfastened".
The output alphabet is {ON, OFF}. The buzzer should go on when ON is output and stay off until OFF is
output.
14. Is it possible to construct a finite state transducer that can output the following sequence:
1010010001000010000010000001
If it is possible, design one. If it's not possible, why not?
Solutions
1. L(M) iff the initial state is a final state. Proof: M will halt in its initial state given as input. So: (IF)
If the initial state is a final state, then when M halts in the initial state, it will be in a final state and will
accept as an element of L(M). (ONLY IF) If the initial state is not a final state, then when M halts in the
initial state, it will reject its input, namely . So the only way to accept is for the initial state to be a final
state.
2.
Homework 4
3.
Homework 4
Homework 4
4. (a)
(b)
Homework 4
(c)
5.
a
a,b
1
6. (aa)* (bb* bb*a(aa)*) = (aa)*b+( a(aa)*) = all strings of a's and b's consisting of an even number of
a's, followed by at least one b, followed by zero or an odd number of a's.
7.
b
a
a,b
b
(b) a (a b)* (b aa)
8. (a)
a
2
a
a
4
Homework 4
9. (a)
a
b
b
a,b
b
a
b
a
a,b
(b) ( ab aabb aaabbb aaaabbbb)
Homework 4
CS 341 Homework 5
Regular expressions are all over the place in UNIX, including the programs grep, sed, and vi.
There's a regular expression pattern matcher built into the programming language perl. There's also
one built into the majordomo maillist program, to be used as a way to filter email messages. So it's
easy to see that people have found regular expressions extremely useful. Each of the programs that
uses the basic idea offers its own definition of what a regular expression is. Some of them are more
powerful than others. The definition in perl is shown on the reverse of this page.
1. Write perl regular expressions to do the following things. If you have easy access to a perl
interpreter, you might even want to run them.
(a) match occurrences of your phone number
(b) match occurrences of any phone number
(c) match occurrences of any phone number that occurs more than once in a string
(d) match occurrences of any email address that occurs more than once in a string
(e) match the Subject field of any mail message from yourself
(f) match any email messages where the address of the sender occurs in the body of
the message
2. Examine the constructs in the perl regular expression definition closely. Compare them to the
much more limited definition we are using. Some of them can easily be described in terms of the
primitive capabilities we have. In other words, they don't offer additional power, just additional
convenience. Some of them, though, are genuinely more powerful, in the sense that they enable you
to define languages that aren't regular (i.e., they cannot be recognized with Finite State Machines).
Which of the perl constructs actually add power to the system? What is it about them that makes
them more powerful?
Homework 5
Regular Expressions in perl
from Programming in Perl, Larry Wall and Randall L. Scwartz, OReilly & Associates, 1990.
Homework 5
CS 341 Homework 6
1. (a) Which of the following strings are accepted by the nondeterministic finite automaton shown on the left below?
(i)
a
(ii)
aa
(iii)
aab
(iv)
b
a
b
b
(b) Which of the following strings are accepted by the nondeterministic finite automaton on the right above?
(i)
(ii)
ab
(iii)
abab
(iv)
aba
(v)
abaa
2. Write regular expressions for the languages accepted by the nondeterministic finite automata of problem 1.
3. For any FSM F, let |F| be the number of states in F. Let R be the machine shown on the right in problem 1.
Let L = {w {0, 1}* : M such that M is an FSM, L(M) = L(R), |M| |R|, and w is the binary encoding of |M|}. Write a
regular expression for L.
4. Draw state diagrams for nondeterministic finite automata that accept these languages:
(a) (ab)*(ba)* aa*
(b) ((ab aab)*a*)*
(c) ((a*b*a*)*b)*
(d) (ba b)* (bb a)*
5. Some authors define a nondeterministic finite automaton to be a quintuple (K, , , S, F), where K, , , and F are as we
have defined them and S is a finite set of initial states, in the same way that F is a finite set of final states. The automaton may
nondeterministically begin operating in any of these initial states. Explain why this definition is not more general than ours in
any significant way.
6. (a) Find a simple nondeterministic finite automaton accepting ((a b)*aabab).
(b) Convert the nondeterministic finite automaton of Part (a) into a deterministic finite automaton by the method described
in class and in the notes.
(c) Try to understand how the machine constructed in Part (b) operates. Can you find an equivalent deterministic machine
with fewer states?
7. Construct a NDFSA that accepts the language (ba ((a bb) a*b)).
Homework 6
8. Construct a deterministic finite automaton equivalent to the following nondeterministic automaton:
0
a,b
b
b
4
a
9. L = { w {a, b}* : every a is followed by at least one b }

(a) Write a regular expression that describes L.
(b) Write a regular grammar that describes L.
(c) Construct an FSM that accepts precisely L.
10. Consider the following regular grammar, which defines a language L:
S -> bF
S -> aS
F ->
F -> bF
F -> aF
(a) Construct an FSM that accepts precisely L.
(b) Write a regular expression that describes L.
(c) Describe L in English.
Solutions
1. (a) [i.] yes [ii.] yes [iii.] no [iv.] yes
(b) [i.] yes [ii.] yes [iii.] yes [iv.] yes [v.] no
2. (a) a* Note that the second state could be eliminated, since there's no path from it to a final state.
(b) (ab aba)* Notice that we could eliminate the start state and make the remaining final state the start state and we'd
still get the same result.
3. To determine L, we need first to consider the set of machines that accept the same language as R. It turns out that we dont
actually need to know what all such machines look like because we can immediately see that theres at least one with four
states (R), one with 5, one with 6, and so forth, and all that we need to establish L is the sizes of the machines, not their
structures.. From R, we can construct an infinite number of equivalent machines by adding any number of redundant states.
For example, we could add a new, nonfinal state 1 that is reachable from the start state via an transition. Since 1 is not final
and it doesnt go anywhere, it cannot lead to an accepting path, so adding it to R has no affect on Rs behavior. Now we have
an equivalent machine with 5 states. We can do it again to yield 6, and so forth. Thus the set of numbers we need to
represent is simply 4 n. Now all we have to do is to describe the binary encodings of these numbers. If we want to
disallow leading zeros, we get 1(0 1) (0 1) (0 1)*. There must be at least three digits of which the first must be 1.
4. (a) The easiest way to do this is to make a 2 state FSA for aa* and a 4 state one for (ab)*(ba)*, then make a seventh state,
the start state, that nondeterministically guesses which class an input string will fall into.
(b) First we simplify. ((ab aab)*a*)*
/ (L1*L2*)* = (L1 L2)* /
((ab aab) a)*
/ union is associative /
(ab aab a)*, which can be rewritten as
Homework 6
(ab a)*. This is so because aab can be formed by one application a, followed by one of ab. So it
is redundant inside a Kleene star. Now we can write a two state machine:
a
b
If you put the loop on a on the start state, either in place of where we have it, or in addition to it, it's also right.
(c) First we simplify:
((a*b*a*)*b)*
/ (L1*L2*L3*)* = (L1 L2 L3)* /
((a b a)*b)*
/ union is idempotent /
((a b)*b)*
There is a simple 2 state NDFSM accepting this, which is the empty string and all strings ending with b.
(d) This is the set of strings where either:
(1) every a is preceded by a b,
or (2) all b's occur in pairs.
So we can make a five state nondeterministic machine by making separate machines (each with two states) for the two
languages and then introducing transitions from the start state to both of them.
5. To explain that any construct A is not more general or powerful than some other construct B, it suffices to show that any
instance of A can be simulated by a corresponding instance of B. So in this case, we have to show how to take a multiple start
state NDFSA, A, and convert it to a NDFSA, B, with a single start state. We do this by initially making B equal to A. Then
add to B a new state we'll call S0. Make it the only start state in B. Now add transitions from S0 to each of the states that
was a start state in A. So B has a single start state (thus it satisfies our original definition of a NDFSA), but it simulates the
behavior of A since the first thing it does is to move, nondeterministically, to all of A's start states and then it exactly mimics
the behavior of A.
6. If you take the state machine as it is given, add a new start state and make transitions from it to the given start states, you
have an equivalent machine in the form that weve been using.
7. (a)
a
q0
a
q1
b
q2
a
q3
b
q4
q5
a,b
(b)
(1) Compute the E(q)s. Since there are no transitions, E(q), for all states q is just {q}.
(2) S = {q0}
(3) =
{
({q0}, a, {q0, q1}),
({q0}, b, {{q0}),
({q0, q1}, a, {q0, q1, q2}),
({q0, q1}, b, {q0}),
({q0, q1, q2}, a, {q0, q1, q2}),
({q0, q1, q2}, b, {q0, q3}),
({q0, q3}, a, {q0, q1, q4}),
({q0, q3}, b, {q0}),
({q0, q1, q4}, a, {q0, q1, q2}),
({q0, q1, q4}, b, {q0, q5}),
({q0, q5}, a, {q0, q1}),
({q0, q5}, b, {q0}) }
Homework 6
(4) K = {{q0}, {q0, q1}, {q0, q1, q2}, {q0, q3}, {q0, q1, q4}, {q0, q5}}
(5) F = {{q0, q5}}
(c) There isnt a simpler machine since we need a minimum of six states in order to keep track of how many characters
(between 0 and 5) of the required trailing string we have seen so far.
8. We can build the following machine really easily. We make the path from 1 to 2 to 3 for the ba option. The rest is for the
second choice. We get a nondeterministic machine, as we generally do when we use the simple approach.
1
b
a
a
4
In this case, we could simplify our machine if we wanted to and get rid of state 4 by adding a transition on b from 2 to 5.
9. (1) E(q0) = {q0, q1}, E(q1) = {q1}, E(q2) = {q2}, E(q3) = {q3, q4}, E(q4) = {q4}
(2) s = {q0, q1}
(3) =
{ ({q0, q1}, a, {q0, q1}),
({q0, q1}, b, {q0, q1, q2, q4}),
({q0, q1, q2, q4}, a, {q0, q1, q3, q4}),
({q0, q1, q2, q4), b, {q0, q1, q2, q4}) }
({q0, q1, q3, q4}, a, {q0, q1, q3, q4}),
({q0, q1, q3, q4}, b, {q0, q1, q2, q4}),
(4) K = { {q0, q1}, {q0, q1, q3, q4}, {q0, q1, q2, q4} }
(5) F = { {q0, q1, q3, q4}, {q0, q1, q2, q4} }
This machine corresponds to the regular expression a*b(a b)*
10. (a)
a
a
b
S
F
b
(b) (a b)*ba* OR a*b(a b)*

(c) L = { w {a, b}* : there is at least one b }
Homework 6
CS 341 Homework 7
1. Assume a finite domain that includes just the specific cities mentioned here. Let R = the reflexive,
symmetric, transitive closure of:
(Austin, Dallas), (Dallas, Houston), (Dallas, Amarillo), (Austin, San Marcos),
(Philadelphia, Pittsburgh), (Philadelphia, Paoli), (Paoli, Scranton),
(San Francisco, Los Angeles), (Los Angeles, Long Beach), (Long Beach, Carmel)
(a) Draw R as a graph.
(b) List the elements of the partition defined by R on its domain.
2. Let R be a relation on the set of positive integers. Define R as follows:
{(a, b) : (a mod 2) = (b mod 2)} In other words, R(a, b) iff a and b have the same remainder when
divided by 2.
(a) Consider the following example integers: 1, 2, 3, 4, 5, 6. Draw the subset of R involving just these values as
a graph.
(b) How many elements are there in the partition that R defines on the positive integers?
(c) List the elements of that partition and show some example elements.
3. Consider the language L, over the alphabet = {a, b}, defined by the regular expression
a*(b ) a*
Let R be a relation on *, defined as follows:
R(x, y) iff both x and y are in L or neither x nor y is in L. In other words, R(x,y) if x and y have
identical status with respect to L.
(a) Consider the following example elements of *: , b, aa, bb, aabaaa, bab, bbaabb. Draw the subset of R
involving just these values as a graph.
(b) How many elements are there in the partition that R defines on *?
(c) List the elements of that partition and show some example elements.
Solutions
1.
(b) [cities in Texas], [cities in Pennsylvania], [cities in California]
2.
(b) Two
(c) [even integers] Examples: 2, 4, 6, 106
[odd integers] Examples: 1, 3, 5, 17, 11679
3.
(a) (Hint: L is the language of strings with no more than one b.)
(b) Two
(c) [strings in L] Examples: , aa, b, aabaaa
[strings not in L] Examples: bb, bbaabb, bab
Homework 7
CS 341 Homework 8
1. We showed that the set of finite state machines is closed under complement. To do that, we presented a
technique for converting a deterministic machine M into a machine M' such that L(M') is the complement of
L(M). Why did we insist that M be deterministic? What happens if we interchange the final and nonfinal states
of a nondeterministic finite automaton?
2. Give a direct construction for the closure under intersection of the languages accepted by finite automata.
(Hint: Consider an automaton whose set of states is the Cartesian product of the sets of states of the two
original automata.) Which of the two constructions, the one given in the text or the one suggested in this
problem, is more efficient when the two languages are given in terms of nondeterministic finite automata?
3. Using the either of the construction techniques that we discussed, construct a finite automaton that accepts
the language defined by the regular expression: a*(ab ba )b*.
4. Write a regular expression for the language recognized by the following FSM:
a
b
b
a
a
a,b
b
5. Consider the following FSM M:
a
b
1
a
(a) Write a regular expression for the language accepted by M.

(b) Give a deterministic FSM that accepts the complement of the language accepted by M.
6. Construct a deterministic FSM to accept each of the following languages:
(a) (aba aabaa)*
(b) (ab)*(aab)*
7. Consider the language L = {w (a, b)* : w has an odd number of a's}
(a) Write a regular grammar for L.
(b) Use that grammar to derive a (possibly nondeterministic) FSA to accept L.
Homework 8
8. Construct a deterministic FSM to accept the intersection of the languages accepted by the following FSMs:
a
2'
1'
a
a
3'
9. Consider the following FSM M:

a
a
b
1
a,b
(a) Give a regular expression forL(M).

(b) Describe L(M) in English.
Solutions
1. We define acceptance for a NDFSA corresponding to the language L as there existing at least one path that
gets us to a final state. There can be many other paths that don't, but we ignore them. So, for example, we
might accept a string S that gets us to three different states, one of which accepts (which is why we accept the
string) and two of which don't (but we don't care). If we simply flip accepting and nonaccepting states to get a
machine that represents the complement of L, then we still have to follow all possible paths, so that same string
S will get us to one nonaccepting state (the old accepting state), and two accepting states (the two states that
previously were nonaccepting but we ignored). Unfortunately, we could ignore the superfluous nonaccepting
paths in the machine for L, but now that those same paths have gotten us to accepting states, we can't ignore
them, and we'll have to accept S. In other words, we'll accept S as being in the complement of L, even though
we also accepted it as being in L. The key is that in a deterministic FSA, a rejecting path actually means reject.
Thus it makes sense to flip it and accept if we want the complement of L. In a NDFSA, a rejecting path doesn't
actually mean reject. So it doesn't make sense to flip it to an accepting state to accept the complement of L.
2.
Homework 8
Homework 8
4. Without using the algorithm for finding a regular expression from an FSM, we can note in this case that the
lower right state is a dead state, i.e., an absorbing, non-accepting state. We can leave and return to the initial
state, the only accepting state, by reading ab along the upper path or by reading ba along the lower path. These
can be read any number of times, in any order, so the regular expression is (ab ba)*. Note that is included,
as it should be.
5. (a) ((a ba)(ba)*b)
(b)
a
{1}
a
{2, 3}
b
{0}
a
b
{2}
a,b
6. (a)
a
a
a
b
a,b
b
b
a
a,b
b
Homework 8
(b)
a
a
b
b
b
a
a,b
7. (a) Nonterminal S is the starting symbol. We'll use it to generate an odd number of a's. We'll also use the
nonterminal E, and it will always generate an even number of a's. So, whenever we generate an a, we must
either stop then, or we must generate the nonterminal E to reflect the fact that if we generate any more a's, we
must generate an even number of them.
Sa
b
S aE
(b)
a
S bS
Eb
S
a
E
E bE
b
E aS
a
b
#
8.
b
<1, 1'>
<2, 1'>
b
<3, 1'>
a
b
a
b
<1, 2'>
<2,2'>
<3, 2'>
a
b
a
b
<1,3'>
<2, 3'>
<3, 3'>
b
b
a
9. (a) (a bb*aa)* ( bb*(a ))
(b) All strings in {a, b}* that contain no occurrence of bab.
Homework 8
CS 341 Homework 9
1. Show that the following are not regular.
(a) L = {wwR : w {a, b}*}
(b) L = {ww : w {a, b}*}
(c) L = {ww' : w {a, b}*}, where w' stands for w with each occurrence of a replaced by b, and vice versa.
2. Show that each of the following is or is not a regular language. The decimal notation for a number is the
number written in the usual way, as a string over the alphabet {-, 0, 1, , 9}. For example, the decimal
notation for 13 is a string of length 2. In unary notation, only the symbol 1 is used; thus 5 would be represented
as 11111 in unary notation.
(a) L = {w : w is the unary notation for a natural number that is a multiple of 7}
(b) L = {w : w is the decimal notation for a natural number that is a multiple of 7}
(c) L = {w : w is the unary notation for a natural number n such that there exists a pair p and q of twin primes,
both > n.} Two numbers p and q are a pair of twin primes iff q = p + 2 and both p and q are prime. For
example, (3, 5) is a pair of twin primes.
(d) L = {w : w is, for some n 1, the unary notation for 10n}
(e) L = {w : w is, for some n 1, the decimal notation for 10n}
(f) L = {w is of the form x#y, where x, y {1}+ and y = x+1 when x and y are interpreted as unary numbers}
(For example, 11#111 and 1111#11111 L, while 11#11, 1#111, and 1111 L.)
(g) L = {anbj: |n j| = 2}
(h) L = {uwwRv : u, v, w {a, b}+}
(i) L = {w {a, b}* : for each prefix x of w, #a(x) #b(x)}
3. Are the following statements true or false? Explain your answer in each case. (In each case, a fixed alphabet
is assumed.)
(a) Every subset of a regular language is regular.
(b) Let L = L1 L2. If L is regular and L2 is regular, L1 must be regular.
(c) If L is regular, then so is L = {xy : x L and y L}.
(d) {w : w = wR} is regular.
(e) If L is a regular language, then so is L = {w : w L and wR L}.
(f) If C is any set of regular languages, C (the union of all the elements of C) is a regular language.
(g) L = {xyxR : x, y *} is regular.
(h) If L = L1 L2 is a regular language and L1 is a regular language, then L2 is a regular language.
(i) Every regular language has a regular proper subset.
(j) If L1 and L2 are nonregular languages, then L1 L2 is also not regular.
4. Show that the language L = {anbm : n m} is not regular.
5. Prove or disprove the following statement:
If L1 and L2 are not regular languages, then L1 L2 is not regular.
6. Show that the language L = {x {a, b}* : x = anbambamax(m,n)} is not regular.
7. Show that the language L = {x {a, b}* : x contains exactly two more b's than a's} is not regular.
8. Show that the language L = {x {a, b}* : x contains twice as many a's as b's} is not regular.
Homework 9
9. Let L = {w : #a(w) = #b(w)}. ( #a(w) = the number of as in w.)

(a) Is L regular?
(b) Is L* regular?
Solutions
1. (a) L = {wwR : w {a, b}*}. L is the set of all strings whose first half is equal to the reverse of the second
half. All strings in L must have even length. If L is regular, then the pumping lemma tells us that N 1, such
that strings w L, where |w| N, x, y, z, such that w = xyz, |xy| N, y , and q 0, xyqz is in L. We
must pick a string w L and show that it does not meet these requirements.
First, dont get confused by the fact that we must pick a string w, yet we are looking for strings of the form
wwR. These are two independent uses of the variable name w. It just happens that the problem statement uses
the same variable name that the pumping lemma does. If it helps, restate the problem as L = {ssR : s {a, b}*}.
We need to choose a long w, i.e., one whose length is greater than N. But it may be easier if we choose one
that is even longer than that. Remember that the fact that |xy| N guarantees that y (the pumpable region) must
occur within the first N characters of w. If we dont want to have to consider a lot of different possibilities for
what y could be, it will help to choose a w with a long first region. Lets let w = aNbbaN. We know that y must
consist of one or more as in the region before the bs. Clearly if we pump in any extra as, we will no longer
have a string in L. Thus we know that L is not regular.
Notice that we could have skipped the bs altogether and chosen w = aNaN. Again, wed know that y must be a
string of one or more as. Unfortunately, if y is of even length (and it could be: remember we dont get to pick
y), then we can pump in all the copies of y we want and still have a string in L. Sure, the boundary between the
first half and the second half will move, that that doesnt matter. It is usually good to choose a string with a
long, uniform first region followed by a definitive boundary between it and succeeding regions so that when
you pump, its clearly the first region that has changed.
(b) L = {ww : w {a, b}*}. Well use the pumping lemma. Again, dont get confused by the use of the
variable w both to define L and as the name for the string we will choose to pump on. As is always the case,
the only real work we have to do is to choose an appropriate string w. We need one that is long enough (i.e., |w|
N). And we need one with firm boundaries between regions. So lets choose w = aNbaNb. Since |xy| N, we
know that y must occur in the first a region. Clearly if we pump in any additional as, the two halves of w will
no longer be equal. Q. E. D. By the way, we could have chosen other strings for w. For example, let w =
baNbaN. But then there are additional choices for what y could be (since y could include the initial b) and we
would have to work through them all.
(c) L = {ww' : w {a, b}*}, where w' stands for w with each occurrence of a replaced by b, and vice versa.
We can prove this easily using the pumping lemma. Let w = aNbN. Since |xy| N, y must be a string of all as.
So, when we pump (either in or out), we modify the first part of w but not the second part. Thus the resulting
string is not in L.
We could also solve this problem just by observing that, if L is regular, so is L = L a*b*. But L is just anbn,
which we have already shown is not regular. Thus L is not regular either.
2. (a) L = {w : w is the unary notation for a natural number that is a multiple of 7}. L is regular since it can be
described by the regular expression (1111111)*.
Homework 9
(b) L = {w : w is the decimal notation for a natural number that is a multiple of 7}. L is regular. We can
build a deterministic FSM M to accept it. M is based on the standard algorithm for long division. The states
represent the remainders we have seen so far (so there are 7 of them, corresponding to 0 6). The start state, of
course, is 0, corresponding to a remainder of 0. So is the final state. The transitions of M are as follows:
si {0 - 6} and cj {0 - 9}, (si, cj) = (10si + cj) mod 7
So, for example, on the input 962, M would first read 9. When you divide 7 into 9 you get 1 (which we dont
care about since we dont actually care about the answer we just care whether the remainder is 0) with a
remainder of 2. So M will enter state 2. Next it reads 6. Since it is in state 2, it must divide 7 into 2*10 +6
(26). It gets a remainder of 5, so it goes to state 5. Next it reads 2. Since it is in state 5, it must divide 7 into
5*10 + 5 (52), producing a remainder of 3. Since 3 is not zero, we know that 862 is not divisible by 7, so M
rejects.
(c) L = {w : w is the unary notation for a natural number such that there exists a pair p and q of twin primes,
both > n.}. L is regular. Unfortunately, this time we dont know how to build a PDA for it. We can, however,
prove that it is regular by considering the following two possibilities:
(1) There is an infinite number of twin primes. In this case, for every n, there exists a pair of twin primes
greater than n. Thus L = 1*, which is clearly regular.
(2) There is not an infinite number of twin primes. In this case, there is some largest pair. There is thus
also a largest n that has a pair greater than it. Thus the set of such ns is finite and so is L (the unary
encodings of those values of n). Since L is finite, it is clearly regular.
It is not known which of these cases is true. But interestingly, from our point of view, it doesnt matter. L is
regular in either case. It may bother you that we can assert that L is regular when we cannot draw either an
FSM or a regular expression for it. It shouldnt bother you. We have just given a nonconstructive proof that L
is regular (and thus, by the way, that some FSM M accepts it). Not all proofs need to be constructive. This
situation isnt really any different from the case of L = {w : w is the unary encoding of the number of siblings
I have}. You know that L is finite and thus regular, even though you do not know how many siblings I have
and thus cannot actually build a machine to accept L.
(d) L = {w : w is, for some n 1, the unary notation for 10n}. So L = {1111111111, 1100, 11000, }. L isnt
regular, since clearly any machine to accept L will have to count the 1s. We can prove this using the pumping
lemma: Let w = 1P, N P and P is some power of 10. y must be some number of 1s. Clearly, it can be of
length at most P. When we pump it in once, we get a string s whose maximum length is therefore 2P. But the
next power of 10 is 10P. Thus s cannot be in L.
(e) L = {w : w is, for some n 1, the decimal notation for 10n}. Often its easier to work with unary
representations, but not in this case. This L is regular, since it is just 100*.
(f) L = {w is of the form x#y, where x, y {1}+ and y = x+1 when x and y are interpreted as unary numbers}
(For example, 11#111 and 1111#11111 L, while 11#11, 1#111, and 1111 L.) L isnt regular. Intuitively, it
isnt regular because any machine to accept it must count the 1s before the # and then compare that number to
the number of 1s after the #. We can prove that this is true using the pumping lemma: Let w = 1N#1N+1. Since
|xy| N, y must occur in the region before the #. Thus when we pump (either in or out) we will change x but
not make the corresponding change to y, so y will no longer equal x +1. The resulting string is thus not in L.
(g) L = {anbj: |n j| = 2}. L isnt regular. L consists of all strings of the form a*b* where either the number
of as is two more than the number of bs or the number of bs is two more than the number of as. We can
show that L is not regular by pumping. Let w = aNbN+2. Since |xy| N, y must equal ap for some p > 0. We can
pump y out once, which will generate the string aN-pbN+2, which is not in L.
Homework 9
(h) L = {uwwRv : u, v, w {a, b}+}. L is regular. This may seem counterintuitive. But any string of length
at least four with two consecutive symbols, not including the first and the last ones, is in L. We simply make
everything up to the first of the two consecutive symbols u. The first of the two consecutive symbols is w. The
second is wR. And the rest of the string is v. And only strings with at least one pair of consecutive symbols
(not including the first and last) are in L because w must end with some symbol s. wR must start with that same
symbol s. Thus the string will contain two consecutive occurrences of s. L is regular because it can be
described the regular expression (a b)+ (aa bb) (a b)+.
(i) L = {w {a, b}* : for each prefix x of w, #a(x) #b(x)}. First we need to understand exactly what L is.
In order to do that, we need to define prefix. A string x is a prefix of a string y iff z * such that y = xz. In
other words, x is a prefix of y iff x is an initial substring of y. For example, the prefixes of abba are , a, ab,
abb, and abba. So L is all strings over {a, b}* such that, at any point in the string (reading left to right), there
have never been more bs than as. The strings , a, ab, aaabbb, and ababa are in L. The strings b, ba, abba, and
ababb are not in L. L is not regular, which we can show by pumping. Let w = aNbN. So y = ap, for some
nonzero p. If we pump out, there will be fewer as than bs in the resulting string s. So s is not in L since every
string is a prefix of itself.
3. (a) Every subset of a regular language is regular. FALSE. Often the easiest way to show that a universally
quantified statement such as this is false by showing a counterexample. So consider L = a*. L is clearly
regular, since we have just shown a regular expression for it. Now consider L = ai: i is prime. L L. But we
showed in class that L is not regular.
(b) Let L = L1 L2. If L is regular and L2 is regular, L1 must be regular. FALSE. We know that the
regular languages are closed under intersection. But it is important to keep in mind that this closure lemma (as
well as all the others we will prove) only says exactly what it says and no more. In particular, it says that:
If L1 is regular and L2 is regular
Then L is regular.
Just like any implication, we cant run this one backward and conclude anything from the fact that L is regular.
Of course, we cant use the closure lemma to say that L1 must not be regular either. So we cant apply the
closure lemma here at all. A rule of thumb: it is almost never true that you can prove the converse of a closure
lemma. So it makes sense to look first for a counterexample. We dont have to look far. Let L = . Let L2 =
. So L and L2 are regular. Now let L1 = {ai: i is prime}. L1 is not regular. Yet L = L1 L2. Notice that
we could have made L2 anything at all and its intersection with would have been . When you are looking
for counterexamples, it usually works to look for very simple ones such as or *, so its a good idea to start
there first. works well in this case because were doing intersection. * is often useful when were doing
union.
(c) If L is regular, then so is L = {xy : x L and y L}. TRUE. Proof: Saying that y L is equivalent to
saying that y L. Since the regular languages are closed under complement, we know that L is also regular. L
is thus the concatenation of two regular languages. The regular languages are closed under concatenation.
Thus L must be regular.
(d) L = {w : w = wR} is regular. FALSE. L is NOT regular. You can prove this easily by using the
pumping lemma and letting w = aNbaN.
(e) If L is a regular language, then so is L = {w : w L and wR L}. TRUE. Proof: Saying that wR L is
equivalent to saying that w LR. If w must be in both L and LR, that is equivalent to saying that L = L LR.
L is regular because the problem statement says so. LR is also regular because the regular languages are closed
Homework 9
under reversal. The regular languages are closed under intersection. So the intersection of L and LR must be
regular.
Proof that the regular languages are closed under reversal (by construction): If L is regular, then there exists
some FSM M that accepts it. From M, we can construct a new FSM M that accepts LR. M will effectively run
M backwards. Start with the states of M equal to states of M. Take the state that corresponds to the start state
of M and make it the final state of M. Next we want to take the final states of M and make them the start states
of M. But M can have only a single start state. So create a new start state in M and create an epsilon
transition from it to each of the states in M that correspond to final states of M. Now just flip the arrows on all
the transitions of M and add these new transitions to M.
(f) If C is any set of regular languages, C is a regular language. FALSE. If C is a finite set of regular
languages, this is true. It follows from the fact that the regular languages are closed under union. But suppose
that C is an infinite set of languages. Then this statement cannot be true. If it were, then every language would
be regular and we have proved that there are languages that are not regular. Why is this? Because every
language is the union of some set of regular languages. Let L be an arbitrary language whose elements are w1,
w2, w3, . Let C be the set of singleton languages {{w1}, {w2}, {w3}, } such that wi L. The number of
elements of C is equal to the cardinality of L. Each individual element of C is a language that contains a single
string, and so it is finite and thus regular. L = C. Thus, since not all languages are regular, it must not be the
case that C is guaranteed to be regular. If youre not sure you follow this argument, you should try to come
up with a specific counterexample. Choose an L such that L is not regular, and show that it can be described as
C for some set of languages C.
(g) L = {xyxR : x, y *} is regular. TRUE. Why? Weve already said that xxR isnt regular. This looks a
lot like that, but it differs in a key way. L is the set of strings that can be described as some string x, followed
by some string y (where x and y can be chosen completely independently), followed by the reverse of x. So, for
example, it is clear that abcccccba L (assuming ={a, b, c}). We let x = ab, y = ccccc, and xR = ba. Now
consider abbcccccaaa. You might think that this string is not in L. But it is. We let x = a, y = bbcccccaa, and
xR = a. What about acccb? This string too is in L. We let x = , y = acccb, and xR = . Note the following
things about our definition of L: (1) There is no restriction on the length of x. Thus we can let x = . (2)There
is no restriction on the relationship of y to x. And (3) R = . Thus L is in fact equal to * because we can take
any string w in * and rewrite it as w , which is of the form xyxR. Since * is regular, L must be regular.
(h) If L = L1 L2 is a regular language and L1 is a regular language, then L2 is a regular language.
FALSE. This is another attempt to use a closure theorem backwards. Let L1 = *. L1 is clearly regular. Since
L1 contains all strings over , the union of L1 with any language is just L1 (i.e., L = *). If the proposition
were true, then all languages L2 would necessarily be regular. But we have already shown that there are
languages that are not regular. Thus the proposition must be false.
(i) Every regular language has a regular proper subset. FALSE. is regular. And it is subset of every set.
Thus it is a subset of every regular language. However, it is not a proper subset of itself. Thus this statement is
false. However the following two similar statements are true:
(1) Every regular language has a regular subset.
(2) Every regular language except has a regular proper subset.
(j) If L1 and L2 are nonregular languages, then L1 L2 is also not regular. False. Let L1 = {anbm, n m}
and L2 = {anbm, n m}. L1 L2 = a*b*, which is regular.
Homework 9
4. If L were regular, then its complement, L1, would also be regular. L1 contains all strings over {a, b} that are
not in L. There are two ways not to be in L: have any a's that occur after any b's (in other words, not have all
the a's followed by all the b's), or have an equal number of a's and b's. So now consider
L2 = L1 a*b*
L2 contains only those elements of L1 in which the a's and b's are in the right order. In other words,
L2 = anbn
But if L were regular, then L1 would be regular. Then L2, since it is the intersection of two regular languages
would also be regular. But we have already shown that it (anbn) is not regular. Thus L cannot be regular.
5. This statement is false. To prove it, we offer a counter example. Let L1 = {anbm : n=m} and let L2 =
{anbm : n m}. We have shown that both L1 and L2 are not regular. However,
L1 L2 = a*b*, which is regular.
There are plenty of other examples as well. Let L1 = {an: n 1 and n is prime}. Let L2 = {an: n 1 and n is not
prime}. Neither L1 nor L2 is regular. But L1 L2 = a+, which is clearly regular.
6. This is easy to prove using the pumping lemma. Let w = aNbaNbaN. We know that xy must be contained
within the first block of a's. So, no matter how y is chosen (as long as it is not empty, as required by the
lemma), for any i > 2, xyiz L, since the first block of a's will be longer than the last block, which is not
allowed. Therefore L is not regular.
7. First, let L' = L a*b*, which must be regular if L is. We observe that L' = anbn+2 : n 0. Now use the
pumping lemma to show that L' is not regular in the same way we used it to show that anbn is not regular.
8. We use the pumping lemma. Let w = a2NbN. xy must be contained within the block of a's, so when we pump
either in or out, it will no longer be true that there will be twice as many a's as b's, since the number of a's
changes but not the number of b's. Thus the pumped string will not be in L. Therefore L is not regular.
9. (a) L is not regular. We can prove this using the pumping lemma. Let w = aNbN. Since y must occur within
the first N characters of w, y = ap for some p > 0. Thus when we pump y in, we will have more as than bs,
which produces strings that are not in L.
(b) L* is also not regular. To prove this, we need first to prove a lemma, which well call EQAB: s, s
L* #a(s) = #b(s). To prove the lemma, we first observe that any string s in L* must be able to be
decomposed into at least one finite sequence of strings, each element of which is in L. Some strings will have
multiple such decompositions. In other words, there may be more than one way to form s by concatenating
together strings in L. For any string s in L*, let SQ be some sequence of elements of L that, when concatenated
together, form s. It doesnt matter which one. Define the function HowMany on the elements of L*.
HowMany(x) returns the length of SQ. Think of HowMany as telling you how many times we went through the
Kleene star loop in deriving x. We will prove EQAB by induction on HowMany(s).
Base case: If HowMany(s) = 0, then s = . #a(s) = #b(s).
Induction hypothesis: If HowMany(s) N, then #a(s) = #b(s).
Show: If HowMany(s) = N+1, then #a(s) = #b(s).
If HowMany(s) = N+1, then w,y such that s = wy, w L*, HowMany(w) = N, and y L. In other words,
we can decompose s into a part that was formed by concatenating together N instances of L plus a second part
that is just one more instance of L. Thus we have:
Homework 9
(1) #a(y) = #b(y).

(2) #a(w) = #b(w).
(3) #a(s) = #a(w) + #a(y)
(4) #b(s) = #b(w) + #b(y).
(5) #b(s) = #a(w) + #b(y)
(6) #b(s) = #a(w) + #a(y)
(7) #b(s) = #a(s)
Definition of L
Induction hypothesis
s = wy
s = wy
4, 2
5, 1
6, 3
Q. E. D.
Now we can show that L* isnt regular using the pumping lemma. Let w = aNbN. Since y must occur within the
first N characters of w, y = ap for some p > 0. Thus when we pump y in, we will have a string with more as
than bs. By EQAB, that string cannot be in L*.
Homework 9
CS 341 Homework 10
State Minimization
1. (a) Give the equivalence classes under L for these languages:
(i) L = (aab ab)*
(ii) L = {x : x contains an occurrence of aababa}
(iii) L = {xxR : x {a, b}*}
(iv) L = {xx : x {a, b}*}
(v) Ln = {a, b}a{a, b}n, where n > 0 is a fixed integer
(vi) The language of balanced parentheses
(b) For those languages in (a) for which the answer is finite, give a deterministic finite automaton with the smallest number
of states that accepts the corresponding language.
2. Let L = {x {a, b}* : x contains at least one a and ends in at least two b's}.
(a) Write a regular expression for L.
(b) Construct a deterministic FSM that accepts L.
(c) Let RL be the equivalence relation of the Myhill-Nerode Theorem. What partition does RL induce on the set
{a, bb, bab, abb, bba, aab, abba, bbaa, baaba}?
(d) How many equivalence classes are there in the partition induced on * by RL?
3. Let L = {x {a, b}* : x begins with a and ends with b}.
(a) What is the nature of the partition induced on * by RL, the equivalence relation of the Myhill-Nerode Theorem? That
is, how many classes are there in the partition and give a description of the strings in each.
(b) Using these equivalence classes, construct the minimum state deterministic FSM that accepts L.
4. Suppose that we are given a language L and a deterministic FSM M that accepts L. Assume L is a subset of {a, b, c}*. Let
RL and RM be the equivalence relations defined in the proof of the Myhill-Nerode Theorem. True or False:
(a) If we know that x and y are two strings in the same equivalence class of RL, we can be sure that they are in the same
equivalence class of RM.
(b) If we know that x and y are two strings in the same equivalence class of RM, we can be sure that they are in the same
equivalence class of RL.
(c) There must be at least one equivalence class of RL that has contains an infinite number of strings.
(d) RM induces a partition on {a, b, c}* that has a finite number of classes.
(e) If L, then [] (the equivalence class containing ) of RL cannot be an infinite set.
5. Use the Myhill-Nerode Theorem to prove that {anbmcmax(m,n)bpdp : m, n, p 0} is not regular.
6. (a) In class we argued that the intersection of two regular languages was regular on the basis of closure properties of
regular languages. We did not show a construction for the FSM that recognizes the intersection of two regular languages.
Such a construction does exist, however, and it is suggested by the fact that L1 L2 = * - ((* - L1) (* - L2)).
Given two deterministic FSMs, M1 and M2, that recognize two regular languages L1 and L2, we can construct an FSM that
recognizes L = L1 L2 (in other words strings that have all the required properties of both L1 and L2), as follows:
1. Construct machines M1' and M2', as deterministic versions of M1 and M2. This step is necessary because complementation
only works on deterministic machines.
2. Construct machines M1'' and M2'', from M1' and M2', using the construction for complementation, that recognize * - L1
and * - L2, respectively.
3. Construct M3, using the construction for union and the machines M1'' and M2'', that recognizes
((* - L1) (* - L2)). This will be a nondeterministic FSM.
4. Construct M4, the deterministic equivalent of M3.
5. Construct ML, using the construction for complementation, that recognizes * - ((* - L1) (* - L2)).
Now consider:
Homework 10
= {a, b}
L1 = {w * : all a's occur in pairs}
State Minimization
e.g., aa, aaaa, aabaa, aabbaabbb L1

aaa, baaab, ab L1
1
L2 = {w * : w contains the string bbb}

Use the procedure outlined above to construct an FSM ML that recognizes L = L1 L2.
Is ML guaranteed to be deterministic?
(b) What are the equivalence classes under L for the language L = L1 L2?
(c) What are the equivalence classes under ~M for ML in (a) above?
(d) Show how ~M is a refinement of L .
(e) Use the minimization algorithm that we have discussed to construct from ML in (a) above a minimal state machine that
accepts L.
7. If you had trouble with this last one, make up another pair of L1 and L2 and try again.
Solutions
1. (a)
(i) L = (aab ab)*
1. [, aab, ab, and all other elements of L]
2. [a or wa : w L]
3. [aa or waa : w L]
4. [everything else, i.e., strings that can never become elements of L because they contain illegal
substrings such as bb or aaa]
(ii) L = {x : s contains an occurrence of aababa}
1. [(a b)*aababa(a b)*, i.e., all elements of L]
2. [ or any string not in L and ending in b but not in aab or aabab, i.e., no progress yet on
"aababa"]
3. [wa for any w [2]; alternatively, any string not in L and ending in a but not in aa, aaba, or
aababa]
4. [any string not in L and ending in aa]
5. [any string not in L and ending in aab]
6. [any string not in L and ending in aaba]
7. [any string not in L and ending in aabab]
Note that this time there is no "everything else". Strings never become hopeless in this
language. They simply fail if we get to the end without finding "aababa".
(iii)L = {xxR : x {a, b}*}
1. [a, which is the only string for which the continuations that lead to acceptance are all strings of
the form wa : where w L]
2. [b, which is the only string for which the continuations that lead to acceptance are all strings of
the form wb : where w L]
3. [ab, which is the only string for which the continuations that lead to acceptance are all strings
of the form wba : where w L]
4. [aa, which is the only string for which the continuations that lead to acceptance are all strings
of the form waa : where w L]
And so forth. Every string is in a distinct equivalence class.
(iv) L = {xx : x {a, b}*}
1. [a, which is the only string for which the continuations that lead to acceptance are all strings
that would be in L except that they are missing a leading a]
2. [b, which is the only string for which the continuations that lead to acceptance are all strings
that would be in L except that they are missing a leading b]
3. [ab, which is the only string for which the continuations that lead to acceptance are all strings
that would be in L except that they are missing a leading ab]
Homework 10
State Minimization
4. [aa, which is the only string for which the continuations that lead to acceptance are all strings
that would be in L except that they are missing a leading aa]
And so forth. Every string is in a distinct equivalence class.
(v) Ln = {a, b}a{a, b}n
0. []
1. [a, b]
2. [aa, ba]
3. [aaa, aab, baa, bab]
.
.
.
n+2. [(a b)a(a b)n]
n+3. [strings that can never become elements of Ln]
There is a finite number of strings in any specific language Ln. So there is a finite number of equivalence classes of
L. Every string in Ln must be of length n+2. So there are n+3 equivalence classes (numbered 0 to n+2, to indicate the length
of the strings in the class) of strings that may become elements of Ln, plus one for strings that are already hopeless, either
because they don't start with ab or aa, or because they are already too long.
(vi) L = The language of balanced parentheses
1. [w*(w*: w L]
/* i.e., one extra left parenthesis somewhere in the string
2. [w*((w* : w L]
/*
two
3. [w*(((w* : w L]
4. [w*((((w* : w L]
5. [w*(((((w* : w L]
and so on. There is an infinite number of equivalence classes.
Each of these classes is distinct, since ) is an acceptable continuation for 1, but none of the others; )) is acceptable for
2, but none of the others, ))) is acceptable for 3, but none of the others, and so forth.
1. (b)
(i)
b
a
b
b
a
4
(ii) There's always a very simple nondeterministic FSM to recognize all strings that contain a specific substring. It's just a
chain of states for the desired substring, with loops on all letters of on the start state and the final state. In this case, the
machine is:
a
2
a
3
b
4
a
5
b
6
a
7
a,b
1
a,b
To construct a minimal, deterministic FSM, you have two choices. You can use our algorithm to convert the NDFSM to a
deterministic one and then use the minimization algorithm to get the minimal machine. Or you can construct the minimal
FSM directly. In any case, it is:
Homework 10
State Minimization
a
a
a,b
1
b
a
(iii) There is no FSM for this language, since it is not regular.

(iv) There is no FSM for this language, since it is not regular.
(v)
a,b
0
a,b
a,b
3
a,b
..
n+2
a,b
n+3
(vi) There is no FSM for this language, since it is not regular.

2. (a) (a b)*a(a b)*bbb* or
(b)
b
(a b)*a(a b)* bb
a
a
0
1
a
b
3
b
(c) It's easiest to answer (d) first, and then to consider (c) as a special case.
(d)
[0] = all strings that contain no a
[1] = all strings that end with a
[2] = all strings that end with ab
[3] = all strings that contain at least one a and that end with bb, i.e., all strings in L.
It is clear that these classes are pairwise disjoint and that their union is {a, b}*. Thus they represent a partition of {a, b}*. It
is also easy to see that they are the equivalence classes of RL of the Myhill-Nerode Theore, since all the members of one
equivalence class will, when suffixed by any string z, form strings all of which are in L or all of which are not in L. Further,
for any x and y from different equivalence classes, it is easy to find a z such that one of xz, yz is in L and the other is not.
Letting the equivalence relation RL be restricted to the set in part (c), gives the partition
{{bb}, {a, bba, abba, bbaa, baaba}, {bab, aab}, {abb}}.
Homework 10
State Minimization
3. (a)
[1] = {}
[2] = b (a b)*
[3] = a a(a b)*a
[4] = a(a b)*b
(b)
a
1
a
b
a
2
a,b
4. (a) F, (b) T, (c) T (* is infinite and the number of equivalence classes is finite), (d) T, (e) F.
5. Choose any two distinct strings of a's: call them ai and aj (i < j). Then they must be in different equivalence classes of RL
since aibici L but ajbici L. Therefore, there is an infinite number of equivalence classes and L is not regular.
M1, which recognizes L1, is:
6. (a)
a,b
a
a
M2, which recognizes L2, is:
a,b
b
a
a
Step (1) M1' = M1 because M1 is deterministic.

M2' = M2 because M2 is deterministic.
Step (2) M1'' is M1' except that states 3 and 4 are the final states.
M2'' is M2' except that states 5, 6, and 7 are the final states.
Step (3) M3 is:
a,b
a
a
1
a,b
b
a
a
Homework 10
State Minimization
Step (4) M4 is:

{1, 2, 5}, a, {3, 5}
b, {2, 6}
{3, 5}, a, {2, 5}
b, {4, 6}
{2, 6}, a, {3, 5}
b, {2, 7}
{4, 5}, a, {4, 5}

b, {4, 6}
{4, 7}, a, {4, 5}
b, {4, 8}
{2, 8}, a, {3, 8}
b, {2, 8}
{2, 5}, a, {3, 5}

b, {2, 6}
{4, 6}, a, {4, 5}
b, {4, 7}
{2, 7}, a, {3, 5}
b, {2, 8}
{4, 8}, a, {4, 8}

b, {4, 8}
{3, 8}, a, {2, 8}
b, {4, 8}
s = {1, 2, 5}
F = K - {2, 8}, i.e., all states except {2, 8} are final states.
You may find it useful at this point to draw this out.
Step (5) ML = M4 except that now there is a single final state, {2, 8}.
ML is deterministic. M4 is deterministic by construction, and Step 5 can not introduce any nondeterminism since it doesn't
introduce any transitions.
(b)
1. [strings without bbb and with any a's in pairs, including ]

2. [strings without bbb but with a single a at the end]
3. [strings that cannot be made legal because they have a single a followed by a b]
4. [strings without bbb, with any a's in pairs, and ending in a single b]
5. [strings without bbb, with any a's in pairs, and ending with just two b's]
6. [strings with bbb and with any a's in pairs]
7. [strings with bbb but with a single a at the end]
(c)
{1, 2, 5}
{3, 5}
{2, 6}
{2, 5}
{4, 6}
{2, 7}
{4, 5}
{4, 7}
{2, 8}
{4, 8}
{3, 8}
(d)
[]
[strings without bbb but with a single a at the end]
[strings without bbb, with any a's in pairs, and ending in a single b]
[strings without bbb and with at least one pair of a's and any a's in pairs]
[strings that cannot be made legal because they have a single a followed by a b
and where every b is preceded by an a and the last character is b]
[strings without bbb, with any a's in pairs, and ending with just two b's]
[strings that cannot be made legal because they have a single a followed by b
and where there is no bbb and the last character is a]
and where there is no bbb but there is at least one bb and the last
character is b]
[all strings in L]
and where there is a bbb, but the ab violation came before the first bbb]
[strings with bbb but with a single a at the end]
{1, 2, 5} {2,5} = 1
{3, 5} = 2
{4, 6} {4, 5} {4, 7} {4, 8} = 3
{2, 6} = 4
{2, 7} = 5
{2, 8} = 6
{3, 8} = 7
Homework 10
State Minimization
(e) 0 = A: [{2, 8}],

B: [{1, 2, 5}, {2, 5}, {3, 5}, {4, 6}, {4, 5}, {4, 7}, {4, 8}, {2, 6}, {2, 7}, {3, 8}]
To compute 1: Consider B (since clearly A cannot be split). We need to look at all the single character transitions out of
each of these states. We've already done that in Step (3) of part (a) above, so we can use that table to tell us which of our
current states each state goes to. Now we just need to use that to determine which element of 0 they go to. We notice that all
transitions are to elements of B except: ({2, 7}, b, A) and ({3, 8}, a, A). So we must split these two states from B. and they
must be distinct from each other because their a and b behaviors are reversed. So we have:
1 = A: [{2, 8}],
B: [{1, 2, 5}, {2, 5}, {3, 5}, {4, 6}, {4, 5}, {4, 7}, {4, 8}, {2, 6}]
C: [{2, 7}]
D: [{3, 8}]
To compute 2: Again we consider B:
On reading an a, all elements of B go to elements of B.
But on b: ({2, 6}, b, C), so we must split off {2, 6}. This gives us:
2 = A: [{2, 8}],
B: [{1, 2, 5}, {2, 5}, {3, 5}, {4, 6}, {4, 5}, {4, 7}, {4, 8}]
C: [{2, 7}]
D: [{3, 8}]
E: [{2, 6}]
On reading an a, all elements of B go to elements of B.
But on b, {1, 2, 5} and {2, 5} go to E, while everyone else goes to B. So we have to split these two off. This gives us:
3 = A: [{2, 8}],
B: [{3, 5}, {4, 6}, {4, 5}, {4, 7}, {4, 8}]
C: [{2, 7}]
D: [{3, 8}]
E: [{2, 6}]
F: [{1, 2, 5}, {2, 5}]
On reading b, all elements of B stay in B. But on reading a, {3, 5} goes to F, so we split it off. This gives us:
4 = A: [{2, 8}],
B: [{4, 6}, {4, 5}, {4, 7}, {4, 8}]
C: [{2, 7}]
D: [{3, 8}]
E: [{2, 6}]
F: [{1, 2, 5}, {2, 5}]
G: [{3, 5}]
To compute 5: Again we consider B: On both inputs, all elements of B stay in B. So we do no further splitting, and we
assert that = 4. Notice that this is identical to what we expected from (d) above.
Homework 10
State Minimization
We can now show the minimal machine:
a
b
G
a,b
a
b
a
E
b
a
C
b
D
a
Homework 10
State Minimization
CS 341 Homework 11
1. Consider the grammar G = (V, , R, S), where
V = {a, b, S, A},
= {a, b},
R = { S AA,
A AAA,
A a,
A bA,
A Ab }.
(a) Which strings of L(G) can be produced by derivations of four or fewer steps?
(b) Give at least four distinct derivations for the string babbab.
(c) For any m, n, p > 0, describe a derivation in G of the string bmabnabp.
2. Construct context-free grammars that generate each of these languages:
(a) {wcwR : w {a, b}*}
(b) {wwR : w {a, b}*}
(c) {w {a, b}* : w = wR}
3. Consider the alphabet = {a, b, (, ), , *, }. Construct a context-free grammar that generates all strings in
* that are regular expressions over {a, b}.
4. Let G be a context-free grammar and let k > 0. We let Lk(G) L(G) be the set of all strings that have a
derivation in G with k or fewer steps.
(a) What is L5(G), where G = ({S, (, )}, {(, )}, {S , S SS, S (S) })?
(b) Show that, for all context-free grammars G and all k > 0, Lk(G) is finite.
5. Let G = (V, , R, S), where
V = {a, b, S},
= {a, b},
R = { S aSb,
S aSa,
S bSa,
S bSb,
S }.
Show that L(G) is regular.
6. A program in a procedural programming language, such as C or Java, consists of a list of statements, where
each statement is one of several types, such as:
(1) assignment statement, of the form id := E, where E is any arithmetic expression (generated by the grammar
using T and F that we presented in class).
(2) conditional statement, e.g., "if E < E then statement", or while statement , e.g. "while E < E do statement".
(3) goto statement; furthermore each statement could be preceded by a label.
(4) compound statement, i.e., many statements preceded by a begin, followed by an end, and separated by ";".
Give a context-free grammar that generates all possible statements in the simplified programming language
described above.
7. Show that the following languages are context free by exhibiting context-free grammars generating each:
(a) {ambn : m n}
Homework 11
(b) {ambncpdq : m + n = p + q}
(c) {w {a, b}* : w has twice as many b's as a's}
(d) {uawb : u, w {a, b}*, |u| = |w|}
8. Let = {a, b, c}. Let L be the language of prefix arithmetic defined as follows:
(i) any member of is a well-formed expression (wff).
(ii) if and are any wff's, then so are A, S, M, and D.
(iii) nothing else is a wff.
(One might think of A, S, M, and D as corresponding to the operators +, -, , /, respectively. Thus in L we could
write Aab instead of the usual (a + b), and MSabDbc, instead of ((a - b) (b/c)). Note that parentheses are
unnecessary to resolve ambiguities in L.)
(a) Write a context-free grammar that generates exactly the wff's of L.
(b) Show that L is not regular.
9. Consider the language L = {amb2nc3ndp : p > m, and m, n 1}.
(a) What is the shortest string in L?
(b) Write a context-free grammar to generate L.
Solutions
1. (a) We can do an exhaustive search of all derivations of length no more than 4:
S AA aA aa
S AA aA abA aba
S AA aA aAb aab
S AA bAA baA baa
S AA bAA bAa baa
S AA AbA abA aba
S AA AbA Aba aba
S AA Aa aa
S AA Aa bAa baa
S AA Aa Aba aba
S AA AbA abA aba
S AA AbA Aba aba
S AA AAb aAb aab
S AA AAb Aab aab
Many of these correspond to the same parse trees, just applying the rules in different orders. In any case, the
strings that can be generated are: aa, aab, aba, baa.
(b) Notice that A bA bAb bab, and also that A Ab bAb bab. This suggests 8 distinct
derivations:
S AA AbA AbAb Abab * babbab
S AA AAb AbAb Abab * babbab
S AA bAA bAbA babA * babbab
S AA AbA bAbA babA * babbab
Where each of these four has 2 ways to reach babbab in the last steps. And, of course, one could interleave the
productions rather than doing all of the first A, then all of the second A, or vice versa.
(c) This is a matter of formally describing a sequence of applications of the rules in terms of m, n, p that will
produce the string bmabnabp.
S
/* by rule S AA */
Homework 11
AA
* /* by m applications of rule A bA */
bmAA
/* by rule A a */
bmaA
* /* by n applications of rule A bA */
bmabnA
* by p applications of rule A Ab */
bmabnAbp
/* by rule A a */
bmabnabp
Clearly this derivation (and some variations on it) produce bmabnabp for each m, n, p.
2. (a) G = (V, , R, S) with V = {S, a, b, c}, = {a, b, c}, R = {
S aSa
S bSb
Sc
}.
(b) Same as (a) except remove c from V and and replace the last rule, S c, by S .
(c) This language very similar to the language of (b). (b) was all even length palindromes; this is all
palindromes. We can use the same grammar as (b) except that we must add two rules:
Sa
Sb
3. This is easy. Recall the inductive definition of regular expressions that was given in class :
This definition provides the basis for a grammar for regular expressions:
G = (V, , R, S) with V = {S, a, b, (, ), , *, }, = { a, b, (, ), , *, }, R= {
S
/* part of rule 1, above
Sa
/*
"
Sb
/*
"
S SS
/* rule 2
SSS
/* rule 3
S S*
/* rule 4
S (S)
/* rule 5
}
4. (a) We omit derivations that don't produce strings in L (i.e, they still contain nonterminals).
L1 : S
L2 : S (S) ()
L3 : S SS S
S (S) ((S)) (())
L4 : S SS (S)S ()S ()
S SS S(S) (S) ()
S (S) ((S)) (((S))) ((()))
L5 : S SS (S)S (S)(S) ()(S) ()()
Homework 11
S SS (S)S ((S))S (())S (())

S (S) ((S)) (((S))) ((((S)))) (((())))
So L5 = {, (), (()), ((())), (((()))), ()() }
(b) We can give a (weak) upper bound on the number of strings in LK(G). Let P be the number of rules in G
and let N be the largest number of nonterminals on the right hand side of any rule in G. For the first derivation
step, we start with S and have P choices of derivations to take. So at most P strings can be generated. (Generally
there will be many fewer, since many rules may not apply, but we're only producing an upper bound here, so
that's okay.) At the second step, we may have P strings to begin with (any one of the ones produced in the first
step), each of them may have up to N nonterminals that we could choose to expand, and each nonterminal could
potentially be expanded in P ways. So the number of strings that can be produced is PNP. Note that many of
them aren't strings in L since they may still contain nonterminals, but this number is an upper bound on the
number of strings in L that can be produced. At the third derivation step, each of those strings may again have N
nonterminals that can be expanded and P ways to expand each. In general, an upper bound on the number of
strings produced after K derivation steps is PKN(K-1), which is clearly finite. The key here is that there is a finite
number of rules and that each rule produces a string of finite length.
5. We will show that L(G) is precisely the set of all even length strings in {a, b}*. But we know that that
language is regular. QED.
First we show that only even length strings are generated by G. This is trivial. Every rule that generates any
terminal characters generates two. So only strings of even length can be generated.
Now we must show that all strings of even length are produced. This we do by induction on the length of the
strings:
Base case: LG (by application of the last rule). So we generate the only string of length 0.
Induction hypothesis: All even length strings of length N (for even N) can be generated from S.
Induction step: We need to show that any string of length N+2 can be generated. Any string w of length N + 2 (N
0) can be rewritten as xyz, where x and z are single characters and |y| = N. By the induction hypothesis, we
know that all values of y can be generated from S. We now consider all possible combinations of values for x
and z that can be used to create w. There are four, and the first four rules in the grammar generate, for any string
T derivable from S, the four strings that contain T plus a single character appended at the beginning and at the
end. Thus all possible strings w of length N+2 can be generated.
6. Since we already have a grammar for expressions (E), we'll just use E in this grammar and treat it as though it
were a terminal symbol. Of course, what we really have to do is to combine this grammar with the one for E. As
we did in our grammar for E, we'll use the terminal string id to stand for any identifier.
G = (V, , R, S), where V = {S, U, C, L, T, E, :, = <, >, ;, a-z, id}, = { :, = <, >, ;, a-z, id}, and
R= {
SLU
/* a statement can be a label followed by an unlabeled statement
SU
/* or a statement can be just an unlabeled statement. We need to
make the distinction between S and U if we want to prevent a
statement from being preceded by an arbitrary number of labels.
U id := E
/* assignment statement
U if E T E then S
/* if statement
U while E T E do S
/* while statement
U goto L
/* goto statement
U begin S; S end
/* compound statement
L id
/* a label is just an identifier
Homework 11
T < | > | =
/* we use T to stand for a test operator. We introduce the | (or) notation

here for convenience.
}
There's one problem we haven't addressed here. We have not guaranteed that every label that appears after a goto
statement actually appears in the program. In general, this cannot be done with a context-free grammar.
7. (a) L = {ambm : m n}. This one is very similar to Example 8 in Supplementary Materials: Context-Free
Languages and Pushdown Automata: Designing Context-Free Grammars. The only difference is that in that case,
m n. So you can use any of the ideas presented there to solve this problem.
(b) L = {ambncpdq : m + n = p + q}. This one is somewhat like (a): For any string ambncpdq L, we will
produce a's and d's in parallel for a while. But then one of two things will happen. Either m q, in which case
we begin producing a's and c's for a while, or m q, in which case we begin producing b's and d's for a while.
(You will see that it is fine that it's ambiguous what happens if m = q.) Eventually this process will stop and we
will begin producing the innermost b's and c's for a while. Notice that any of those four phases could produce
zero pairs. Since the four phases are distinct, we will need four nonterminals (since, for example, once we start
producing c's, we do not want ever to produce any d's again). So we have:
G = ({S, T, U, V, a, b, c, d}, {a, b, c, d}, R, S), where
R = {S aSd, S T, S U, T aTc, T V, U bUd, U V, V bVc, V }
Every derivation will use symbols S, T, V in sequence or S, U, V in sequence. As a quick check for fencepost
errors, note that the shortest string in L is , which is indeed generated by the grammar. (And we do not need any
rules S or T .)
How do we know this grammar works? Notice that any string for which m = q has two distinct derivations:
S * amSdm amTdm amVdm ambncpdq, and
S * amSdm amUdm amVdm ambncpdq
Every string ambncpdq L for which m q has a derivation:
S
/* by q application of rule S aSd */
aqSdq
/* by rule S T */
aqTdq
/* by m - q application of rule rule T aTc */
aqam-qTcm-qdq = amTcm-qdq
/* by rule T V */
amVcm-qdq
/* by n = p - (m - q) applications of rule V bVc */
ambnVcp-(m-q)cm-qdq = ambnVcpdq
/* by rule V
ambncpdq
For the other case (m q), you can show the corresponding derivation. So every string in L is generated by G.
And it can be shown that no string not in L is generated by G.
(c) L = {w {a, b}* : w has twice as many b's as a's}. This one is sort of tricky. Why? Because L doesnt
care about the order in which the a's and b's occur. But grammars do. One solution is:
G = ({S, a, b}, {a, b}, R, S), where R = {S SaSbSbS, S SbSaSbS, S SbSbSaS, S }
Try some examples to convince yourself that this grammar works. Why does it work? Notice that all the rules
for S preserve the invariant that there are twice as many b's as a's. So we're guaranteed not to generate any strings
that aren't in L. Now we just have to worry about generating all the strings that are in L. The first three rules
handle the three possible orders in which the symbols b,b, and a can occur.
Homework 11
Another approach you could take is to build a pushdown automaton for L and then derive a grammar from it.
This may be easier simply because PDA's are good at counting. But deriving the grammar isn't trivial either. If
you had a hard time with this one, don't worry.
(d) L = {uawb : u, w {a, b}*, |u| = |w|}. This one fools some people since you might think that the a and b
are correlated somehow. But consider the simpler language L' = {uaw : u, w {a, b}*, |u| = |w|}. This one
seems easier. We just need to generate u and w in parallel, keeping something in the middle that will turn into a.
Now back to L: L is just L' with b tacked on the end. So a grammar for L is:
G = ({S, T, a, b}, {a, b}, R, S), where R = {S Tb, T aTa, T aTb, T bTa, T bTb, T a}.
8. (a) G = ({S, A, M, D, F, a, b, c}. {A, M, D, S, a, b, c}, R, S), where R = {
Fa
F AFF
Fb
F SFF
Fc
F MFF
F DFF }
(b) First, we let L' = L A*a*. L' = {Anan+1 : n 0}. L' can easily be shown to be nonregular using the
Pumping Theorem, so, since the regular languages are closed under intersection, L must not be regular.
9. (a) abbcccdd
(b) G = ({S, X, Y, a, b, c, d}, {a, b, c, d}, R, S), where R is either:
(S aXdd, X Xd, X aXd, X bbYccc, Y bbYccc, Y ), or
(S aSd, S Sd, S aMdd, M bbccc, M bbMccc)
Homework 11
CS 341 Homework 12
Parse Trees
1. Consider the grammar G = ({+, *, (, ), id, T, F, E}, {+, *, (, ), id}, R, E}, where
R = {E E+T, E T, T T * F, T F, F (E), F id}.
Give two derivations of the string id * id + id, one of which is leftmost and one of which is not leftmost.
2. Draw parse trees for each of the following:
(a) The simple grammar of English we presented in class and the string "big Jim ate green cheese."
(b) The grammar of Problem 1 and the strings id + (id + id) * id and (id * id + id * id).
3. Present a context-free grammar that generates , the empty language.
4. Consider the following grammar (the start symbol is S; the alphabets are implicit in the rules):
S SS | AAA |
A aA | Aa | b
(a) Describe the language generated by this grammar.
(b) Give a left-most derivation for the terminal string abbaba.
(c) Show that the grammar is ambiguous by exhibiting two distinct derivation trees for some terminal string.
(d) If this language is regular, give a regular (right linear) grammar generating it and construct the
corresponding FSM. If the language is not regular, prove that it is not.
5. Consider the following language : L = {wRw" : w {a, b}* and w" indicates w with each occurrence of a
replaced by b, and vice versa}. Give a context-free grammar G that generates L and a parse tree that shows that
aababb L.
6. (a) Consider the CFG that you constructed in Homework 11, Problem 2 for {wcwR : w {a, b}*}. How many
derivations are there, using that grammar, for the string aabacabaa?
(b) Show parse tree(s) corresponding to your derivation(s). Is the grammar ambiguous?
7. Consider the language L = {w {a, b}* : w contains equal numbers of a's and b's}
(a) Write a context-free grammar G for L.
(b) Show two derivations (if possible) for the string aabbab using G. Show at least one leftmost derivation.
(c) Do all your derivations result in the same parse tree? If so, see if you can find other parse trees or convince
yourself there are none.
(d) If G is ambiguous (i.e., you found multiple parse trees), remove the ambiguity. (Hint: look out for two
recursive occurrences of the same nonterminal in the right side of a rule, e.g, X XX)
(e) See how many parse trees you get for aabbab using the grammar developed in (d).
Solutions
3. G = ({S} , , R, S), where R is any set of rules that can't produce any strings in *. So, for example, R =
{S S} does the trick. So does R = .
Homework 12
Parse Trees
4. (a) (a*ba*ba*ba*)*
(b) S AAA aAAA abAA abAaA abbaA abbaAa abbaba
(c)
S
S
A
A
A
a
b
(d) G = ({S, S1, B1, B2, B3, a, b}, {a, b}, R, S), where R = {
S
B1 aB1
S S1
B1 bB2
S1 aS1
B2 aB2
S1 bB1
B2 bB3
a
S
b
B3 aB3
B3
B3 S1
a
S1
B3
B1
b
B2
a
5. G = ({S, a, b}, {a, b}, R, S), R = { S aSb, S bSa, S }
S
a
S
a
S
b
b
a
6. (a) The grammar is G = (V, , R, S) with V = {S, a, b, c}, = {a, b, c}, R = {S aSa, S bSb, S c}.
There is a single derivation:
S aSA aaSaa aabSbaa aabaSabaa aabacabaa
(b) There is a single parse tree. The grammar is unambiguous.
7. (a) G = (V, , R, S) with V = {S }, = {a, b }, R = {
S aSb
S bSa
S
S SS
}
(b) (i) S SS aSbS aaSbbS aabbaSb aabbab /* This is the leftmost derivation of the most
"sensible" parse.
(ii) S SS SSS aSbSS aaSbbSS aabbSS aabbaSbS aabbabS aabbab /* This is the
leftmost derivation of a parse that introduced an unnecessary S in the first step, which was then eliminated by
rewriting it as in the final step.
Homework 12
Parse Trees
(c) No. The two derivations shown here have different parse trees. They do, however, have the same
bracketing, [a[ab]b][ab].(In other words, they have similar essential structures.) They differ only in how S is
introduced and then eliminated. But there are other derivations that correspond to additional parse trees, and
some of them correspond to a completely different bracketing, [a[ab][ba]b]. One derivation that does this is
(ii) S aSb aSSb aabSb aabbab
(d) This is tricky. Recall that we were able to eliminate ambiguity in the case of the balanced parentheses
language just by getting rid of except at the very top to allow for the empty string. If we do the same thing here,
we get R = { S
ST
T ab
T aTb
T ba
T bTa
T TT
But aabbab still has multiple parses in this grammar. This language is different from balanced parens since we
can go back and forth between being ahead on a's and being ahead on b's (whereas, in the paren language, we
must always either be even or be ahead on open paren). So the two parses correspond to the bracketings
[aabb][ab] and [a [ab] [ba] b]. The trouble is the rule T TT, which can get applied at the very top of the tree
(as in the case of the first bracketing shown here), or anywhere further down (as in the case of the second one).
We clearly need some capability for forming a string by concatenating a perfectly balanced string with another
one, since, without that, we'll get no parse for the string abba. Just nesting won't work. We have to be able to
combine nesting and concatenation, but we have to control it. It's tempting to think that maybe an unambiguous
grammar doesn't exist, but it's pretty easy to see how to build a deterministic pda (with a bottom of stack symbol)
to accept this language, so there must be an unambiguous grammar. What we need is the notion of an A region,
in which we are currently ahead on a's, and a B region, in which we are currently ahead on b's. Then at the top
level, we can allow an A region, followed by a B region, followed by an A region and so forth. Think of
switching between regions as what will happen when the stack is empty and we're completely even on the number
of a's and b's that we've seen so far. For example, [ab][ba] is one A region followed by one B region. Once we
enter an A region, we stay in it, always generating an a followed (possibly after something else embedded in the
middle) by a b. After all, the definition of an A region, is that we're always ahead on a's. Only when we are
even, can we switch to a B region. Until then, if we want to generate a b, we don't need to do a pair starting with
b. We know we're ahead on a's, so make any desired b's go with an a we already have. Once we are even, we
must either quit or move to a B region. If we allow for two A regions to be concatenated at the top, there will be
ambiguity between concatenating two A regions at the top vs. staying in a single one. We must, however, allow
two A regions to be concatenated once we're inside an A region. Consider [a[ab][ab]b] Each [ab] is a perfectly
balanced A region and they are concatenated inside the A region whose boundaries are the first a and the final b.
So we must distinguish between concatenation within a region (which only happens with regions of the same
type, e.g, two A's within an A) and concatenation at the very top level, which only happens between different
types.
Also, we must be careful of any rule of the form X XX for another reason. Suppose we have a string that
corresponds to XXX. Is that the first X being rewritten as two, or the second one being rewritten as two. We
need to force a single associativity.
Homework 12
Parse Trees
All of this leads to the following set of rules R:

S
S Ta
S Tb
/* start with an A region, then optionally a B, then an A, and so forth

/* start with a B region, then optionally an A, then a B, and so forth
Ta A /* just a single A region

Ta AB
/* two regions, an A followed by a B
Ta ABTa
/* we write this instead of Ta TaTa to allow an arbitrary number of regions,
but force associativity,
Tb B /* these next three rules are the same as the previous three but starting with b
Tb BA
Tb BATb
A A1/* this A region can be composed of a single balanced set of a's and b's
A A1A
/* or it can have arbitrarily many such balanced sets.
A1 aAb
/* a balanced set is a string of A regions inside a matching a, b pair
A1 ab
/* or it bottoms out to just the pair a, b
B B1 /* these next four rules are the same as the previous four but for B regions
B B1B
B1 bBa
B1 ba
(e) The string aabbab is a single A region, and has only one parse tree in this grammar, corresponding to
[[aabb][ab]]. You may also want to try abab, abba, and abaababb to see how G works.
Homework 12
Parse Trees
CS 341 Homework 13
Pushdown Automata
1. Consider the pushdown automaton M = (K, , , , s, F), where
K = {s, f},
F = {f},
= {a, b},
= {a},
= {((s, a, ), (s, a)), ((s, b, ), (s, a)), ((s, a, ), (f, )), ((f, a, a), (f, )), ((f, b, a), (f, ))}.
(a) Trace all possible sequences of transitions of M on input aba.
(b) Show that aba, aa, abb L(M), but baa, bab, baaaa L(M).
(c) Describe L(M) in English.
2. Construct pushdown automata that accept each of the following:
(a) L = the language generated by the grammar G = (V, , R, S), where
V = {S, (, ), [, ]},
= {(, ), [, ]},
R = { S ,
S SS,
S [S],
S (S)}.
(b) L = {ambn : m n 2m}.
(c) L = {w {a, b}* : w = wR}.
(d) L = {w {a, b}* : w has equal numbers of a's and b's}.
(e) L = {w {a, b}* : w has twice as many a's as b's}.
(f) L = {ambn : m n }
(g) L = {uawb: u and w {a, b}* and |u| = |w|}
replaced by b, and vice versa}. In Homework 12, problem 5, you wrote a context-free grammar for L. Now give
a PDA M that accepts L and trace a computation that shows that aababb L.
4. Construct a context-free grammar for the language of problem 2(b): L = ({ambn: m n 2m}).
Solutions
1. (a) There are three possible computations of M on aba:
(s, aba, ) |- (s, ba, a) |- (s, a, aa) |- (s, , aaa)
(s, aba, ) |- (s, ba, a) |- (s, a, aa) |- (f, , aa)
(s, aba, ) |- (f, ba, )
None of these is an accepting configuration.
(b) This is done by tracing the computation of M on each of the strings, as shown in (a).
(c) L(M) is the set of strings whose middle symbol is a. In other words,
L(M) = {xay {a, b}* : |x| = |y|}.
2. (a) Notice that the square brackets and the parentheses must be properly nested. So the strategy will be to push
the open brackets and parens and pop them against matching close brackets and parens as they are read in. We
only need one state, since all the counting will be done on the stack. Since L, the start state can be final.
Thus we have M = ({s}, {(, ), [, ]}, {(, [}, , s {s}), where (sorry about the confusing use of parentheses both as
part of the notation and as symbols in the language):
Homework 13
Pushdown Automata
= {((s, (, ), (s, ()),

/* push ( */
((s, [, ), (s, [)),
/* push [ */
((s, ), (),(s, )),
/* if the input character is ) and the top of the stack is (, they match */
((s, ], [), (s, ))}
/* same for matching square brackets */
If we run out of input and stack at the same time, we'll accept.
(b) Clearly we need to use the stack to count the a's and then compare that count to the b's as they're read in. The
complication here is that for every a, there may be either one or two b's. So we'll need nondeterminism. Every
string in L has two regions, the a region followed by the b region (okay, they're hard to tell apart in the case of ,
but trivially, this even true there). So we need a machine with at least two states.
There are two ways we could deal with the fact that, each time we see an a, we don't know whether it will be
matched by one b or two. The first is to push either one or two characters onto the stack. In this case, once we
get to the b's, we'll pop one character for every b we see. A nondeterministic machine that follows all paths of
combinations of one or two pushed characters will find at least one match for every string in L. The alternative is
to push a single character for every a and then to get nondeterministic when we're processing the b's: For each
stack character, we accept either one b or two. Here's a PDA that takes the second approach. You may want to
try writing one that does it the other way. This machine actually needs three states since it needs two states for
processing b's to allow for the case where two b's are read but only a single a is popped. So M = ({s, f, g}, {a, b},
{a}, , s, {f, g}), where
= { ((s, a, ), (s, a)),
/* Read an a and push one onto the stack */
((s, , ), (f, )),
/* Jump to the b reading state */
((f, b, a), (f, )),
/* Read a single b and pop an a */
((f, b, a), (g, )), /* Read a single b and pop an a but get ready to read a second one */
((g, b, ), (f, ))}.
/* Read a b without popping an a */
(c) A PDA that accepts {w : w = wR} is just a variation of the PDA that accepts {wwR} (which you'll find in
Lecture Notes 14). You can modify that PDA by adding two transitions ((s, a, ), (f, )) and ((s, b, ), (f, )),
which have the effect of making odd length palindromes accepted by skipping their middle symbol.
(d) We've got another counting problem here, but now order doesn't matter -- just numbers. Notice that with
languages of this sort, it's almost always easier to build a PDA than a grammar, since PDAs do a good job of
using their stack for counting, while grammars have to consider the order in which characters are generated. If
you don't believe this, try writing a grammar for this language.
Consider any string w in L. At any point in the processing of w, one of three conditions holds: (1) We have seen
equal numbers of a's and b's; (2) We have seen more a's than b's; or (3) We have seen more b's than a's. What we
need to do is to use the stack to count whichever character we've seen more of. Then, when we see the
corresponding instance of the other character we can "cancel" them by consuming the input and popping off the
stack. So if we've seen an excess of a's, there will be a's on the stack. If we've seen an excess of b's there will be
b's on the stack. If we're even, the stack will be empty. Then, whatever character comes next, we'll start counting
it by putting it on the stack. In fact, we can build our PDA so that the following invariant is always maintained:
(Number of a's read so far) - (Number of b's read so far)
=
(Number of a's on stack) - (Number of b's on stack)
Notice that w L if and only if, when we finish reading w,
[(Number of a's read so far) - (Number of b's read so far)] = 0.
So, if we build M so that it maintains this invariant, then we know that if M consumes w and ends with its stack
empty, it has seen a string in L. And, if its stack isn't empty, then it hasn't seen a string in L.
Homework 13
Pushdown Automata
To make this work, we need to be able to tell if the stack is empty, since that's the only case where we might
consider pushing either a or b. Recall that we can't do that just by writing as the stack character, since that
always matches, even if the stack is not empty. So we'll start by pushing a special character # onto the bottom of
the stack. We can then check to see if the stack is empty by seeing if # is on top. We can do all the real work in
our PDA in a single state. But, because we're using the bottom of stack symbol #, we need two additional states:
the start state, in which we do nothing except push # and move to the working state, and the final state, which we
get to once we've popped # and can then do nothing else. Considering all these issues, we get M = ({s, q, f}, {a,
b}, {#, a, b}, , s, {f}), where
= { ((s, , ), (q, #)),
/* push # and move to the working state q */
((q, a, #), (q, a#)),
/* the stack is empty and we've got an a, so push it */
((q, a, a), (q, aa)),
/* the stack is counting a's and we've got another one so push it */
((q, b, a), (q, )),
/* the stack is counting a's and we've got b, so cancel a and b */
((q, b, #), (q, b#)),
/* the stack is empty and we've got a b, so push it */
((q, b, b), (q, bb)),
/* the stack is counting b's and we've got another one so push it */
((q, a, b), (q, )),
/* the stack is counting b's and we've got a, so cancel b and a */
((q, , #), (f, ))}.
/* the stack is empty of a's and b's. Pop the # and quit. */
To convince yourself that M does the job, you should show that M does in fact maintain the invariant we stated
above.
The only nondeterminism in this machine involves the last transition in which we guess that we're at the end of
the input. There is an alternative way to solve this problem in which we don't bother with the bottom of stack
symbol #. Instead, we substitute a lot of nondeterminism and we sometimes push a's on top of b's, and so forth.
Most of those paths will end up in dead ends. The machine has fewer states but is harder to analyze. Try to
construct it if you like.
(e) This one is similar to (d) except that there are two a's for every b. Recall the two techniques for matching two
to one that we discussed in our solution to (b). This time, though, we do know that there are always two a's to
every b. We don't need nondeterminism to allow for either one or two. But, because we no longer know that all
the a's come first, we do need to consider what to do in the two cases: (1) We're counting b's on the stack; and (2)
We're counting a's on the stack. If we're counting b's, let's take the approach in which we push two b's every time
we see one. Then, when we go to cancel a's, we can just pop one b for each a. If we see twice as many a's as b's,
we'll end up with an empty stack. Now what if we're counting a's? We'll push one a for every one we see. When
we see b, we pop two a's. The only special case we need to consider arises in strings such as "aba", where we'll
only have seen a single a at the point at which we see the b. What we need to do is to switch from counting a's to
counting b's, since the b counts twice. Thus the invariant that we want to maintain is now
(Number of a's read so far) - 2*(Number of b's read so far)
=
(Number of a's on stack) - (Number of b's on stack)
We can do all this with M = ({s, q, f}, {a, b}, {#, a, b}, , s, {f}), where
= { ((s, , ), (q, #)),
/* push # and move to the working state q */
((q, a, #), (q, a#)),
/* the stack is empty and we've got an a, so push it */
((q, a, a), (q, aa)),
/* the stack is counting a's and we've got another one so push it */
((q, b, aa), (q, )),
/* the stack is counting a's and we've got b, so cancel aa and b */
((q, b, a#), (q, b#)),
/* the stack contains a single a and we've got b, so cancel the a and b
and start counting b's, since we have a shortage of one a */
((q, b, #), (q, bb#)),
/* the stack is empty and we've got a b, so push two b's */
((q, b, b), (q, bbb)),
/* the stack is counting b's and we've got another one so push two */
((q, a, b), (q, )),
/* the stack is counting b's and we've got a, so cancel b and a */
((q, , #), (f, ))}.
/* the stack is empty of a's and b's. Pop the # and quit. */
Homework 13
Pushdown Automata
You should show that M preserves the invariant above.

(f) The idea here is to push each a as we see it. Then, on the first b, move to a second state and pop an a for each
b. If we get to the end of the string and either the stack is empty (m = n) or there are still a's on the stack (m > n)
then we accept. If we find a b and there's no a to pop, then there will be no action and so we'll fail. This machine
is nondeterministic for two reasons. The first is that, in case there are no b's, we must be able to guess that we've
reached the end of the input string and go pop all the a's off the stack. The second is that if there were b's but
fewer than the number of a's, then we must guess that we've reached the end of the input string and pop all the a's
off the stack. If we guess wrong, that path will just fail, but it will never cause us to accept something we
shouldn't, since it only pops off extra a's, which is what we want anyway. We don't need a separate state for the
final popping phase, since we're willing to accept either m = n or m > n. This contrasts with the example we did
in class, where we required that m > n. In that case, the state where we run out of input and the stack at the same
time (i.e., m = n) had to be a rejecting state. Thus we needed an additional accepting state where we popped the
stack.
M = ({1, 2}, {a, b}, {a}, 1, {2}, =
((1, a, ), (1, a))
((1, b, a), (2, ))
((1, , ), (2, )
((2, b, a), (2, )
((2, , a), (2, )
/* push an a on the stack for every input a

/* pop an a for the first b and go to the b-popping state
/* in case there aren't any b's -- guess end of string and go pop any a's
/* for each input b, pop an a off the stack
/* if we run out of input while there are still a's on the stack,
then pop the a's and accept
(g) The idea here is to create a nondeterministic machine. In the start state (1), it reads a's and b's, and for each
character seen, it pushes an x on the stack. Thus it counts the length of u. If it sees an a, it may also guess that
this is the required separator a and go to state 2. In state 2, it reads a's and b's, and for each character seen, pops
an x off the stack. If there's nothing to pop, the machine will fail. If it sees a b, it may also guess that this is the
required final b and go to the final state, state 3. The machine will then accept if both the input and the stack are
empty.
M = ({1, 2, 3}, {a, b}, {x}, s, {2}, =
((1, a, ), (1, x))
((1, b, ), (1, x))
((1, a, ), (2, ))
((2, a, x), (2, )
((2, b, x), (2, )
((2, b, ), (3, )
/* push an x on the stack for every input a

/* push an x on the stack for every input b
/* guess that this is the separator a. No stack action
/* for each input a, pop an x off the stack
/* for each input b, pop an x off the stack
/* guess that this is the final b and go to the final state
3.
a//a
a/b/
p
b//b
4.
//
q
b/a/
S
S aSb
S aSbb
Homework 13
Pushdown Automata
CS 341 Homework 14
1. In class, we described an algorithm for constructing a PDA to accept a language L, given a context free
grammar for L. Let L be the balanced brackets language defined by the grammar G = ({S, [, ]}, {[, ]}, R, S),
where R =
S , S SS, S [S]
Apply the construction algorithm to this grammar to derive a PDA that accepts L. Trace the operation of the
PDA you have constructed on the input string [[][]].
2. Consider the following PDA M:
//
a//a
a//
b/a/
(a) What is L(M)?

(b) Give a deterministic PDA that accepts L(M) (not L(M)$).
3. Write a context-free grammar for L(M), where M is
//
a//a
b/a/
//
//
b//b
//
/b/
c/b/
4. Consider the language L = {bam1bam2bbamn : n 2, m1, , mn 0, and mi mj for some i, j}

(a) Give a nondeterministic PDA that accepts L.
(b) Write a context-free grammar that generates L.
(c) Prove that L is not regular.
Solutions
1. This is a very simple mechanical process that you should have no difficulty carrying out, and getting the
following PDA, M = ({p, q}, {[, ]}, {S, [, ]}, , p, {q}), where
=
{((p, , ), (q, S)),
((q, , S), (q, )), ((q, , S), (q, SS)), ((q, , S), (q, [S])),
((q, [, [), (q, )), ((q, ], ]), (q, ))}
2. (a) L(M) = {anbna : n 0}
(b)
a//a
a//aa
b/a/
a//
b//
a//
b/a/
Homework 14
3. Don't even try to use the grammar construction algorithm. Just observe that L = {anbnbmcp : m p and n and p
0}, or, alternatively {anbmcp : m n + p and n and p 0}. It can be generated by the following rules:
S S1S2
S1 aS1b
/* S1 generates the anbn part. */
S1
S2 bS2
/* S2 generates the bmcp part. */
S2 bS2c
S2
4. (a)
a,b//
b,//
a//a
b//
b//
b//
5
/a/
a/a/
/a/
a//
b//
a,b//
b//
7
a,b//
We use state 2 to skip over an arbitrary number of bai groups that aren't involved in the required mismatch.
We use state 3 to count the first group of a's we care about.
We use state 4 to count the second group and make sure it's not equal to the first.
We use state 5 to skip over an arbitrary number of bai groups in between the two we care about.
We use state 6 to clear the stack in the case that the second group had fewer a's than the first group did.
We use state 7 to skip over any remaining bai groups that aren't involved in the required mismatch.
(b)
S A'bLA'
S A'bRA'
L ab | aL | aLa
R ba | Ra | aRa
A' bAA' |
A aA |
(c)
Let L1 = ba*ba*, which is obviously regular.

If L is regular then
L2 = L L1 is regular.
L2 = banbam, n m
L2 L1 must also be regular.
But L2 L1 = banbam, n = m, which can easily be shown, using the pumping theorem, not to be regular.
Homework 14
/* L will take care of two groups where the first group has more a's */
/* R will take care of two groups where the second group has more a's */
CS 341 Homework 15
Parsing
1. Show that the following languages are deterministic context free.
(a) {ambn : m n}
(b) {wcwR : w {a, b}*}
(c) {cambm : m 0} {damb2m : m 0}
(d) (amcbm : m 0} {amdb2m : m 0}
2. Consider the context-free grammar: G = (V, , R, S), where V = {(, ), ., a, S, A}, = {(, ), .}, and R =
{ S (),
S a,
S (A),
A S,
A A.S}
(If you are familiar with the programming language LISP, notice that L(G) contains all
atoms and lists, where the symbol a stands for any non-null atom.)
(a) Apply left factoring and the rule for getting rid of left recursion to G. Let G' be the resulting grammar.
Argue that G' is LL(1). Construct a deterministic pushdown automaton M that accepts L(G)$ by doing a top
down parse. Study the computation of M on the string ((()).a).
(b) Repeat Part (a) for the grammar resulting from G if one replaces the first rule by A .
(c) Repeat Part (a) for the grammar resulting from G if one replaces the last rule by A S.A.
3. Answer each of the following questions True or False. If you choose false, you should be able to state a
counterexample.
(a) If a language L can be described by a regular expression, we can be sure it is a context-free language.
(b) If a language L cannot be described by a regular expression, we can be sure it is not a context-free
language.
(c) If L is generated by a context-free grammar, then L cannot be regular.
(d) If there is no pushdown automaton accepting L, then L cannot be regular.
(e) If L is accepted by a nondeterministic finite automaton, then there is some deterministic PDA accepting L.
(f) If L is accepted by a deterministic PDA, then L' (the complement of L) must be regular.
(g) If L is accepted by a deterministic PDA, then L' must be context free.
(h) If, for a given L in {a, b}*, there exist x, y, z, such that y and xynz L for all n 0, then L must be
regular.
(i) If, for a given L in {a, b}*, there do not exist u, v, x, y, z such that |vy| 1 and uvnxynz L for all n 0,
then L cannot be regular.
(j) If L is regular and L = L1 L2 for some L1 and L2, then at least one of L1 and L2 must be regular.
(k) If L is context free and L = L1L2 for some L1 and L2, then L1 and L2 must both be context free.
(l) If L is context free, then L* must be regular.
(m) If L is an infinite context-free language, then in any context-free grammar generating L there exists at least
one recursive rule.
(n) If L is an infinite context-free language, then there is some context-free grammar generating L that has no
rule of the form A B, where A and B are nonterminal symbols.
(o) Every context-free grammar can be converted into an equivalent regular grammar.
(p) Given a context-free grammar generating L, every string in L has a right-most derivation.
4. Recall problem 4 from Homework 12. It asked you to consider the following grammar for a language L (the
start symbol is S; the alphabets are implicit in the rules):
S SS | AAA |
A aA | Aa | b
(a) It is not possible to convert this grammar to an equivalent one in Chomsky Normal Form. Why not?
Homework 15
Parsing
(b) Modify the grammar as little as possible so that it generates L - . Now convert this new grammar to
Chomsky Normal Form. Is the resulting grammar still ambiguous? Why or why not?
(c) From either the original grammar for L - or the one in Chomsky Normal Form, construct a PDA that
accepts L - .
replaced by b, and vice versa}. In Homework 12, problem 5, you wrote a context-free grammar for L. Then, in
Homework 13, problem 3, you wrote a PDA M that accepts L and traced one of its computations. Now decide
whether you think L is deterministic context free. Defend your answer.
6. Convert the following grammar for arithmetic expressions to Chomsky Normal Form:
EE+T
ET
TT*F
TF
F (E)
F id
7. Again, consider the grammar for arithmetic expressions given in Problem 6. Walk through the process of
doing a top down parse of the following strings using that grammar. Point out the places where a decision has to
be made about what to do.
(a) id * id + id
(b) id * id * id
Solutions
1. (a) L = {ambn : m n}. To show that a language L is deterministic context free, we need to show a
deterministic PDA that accepts L$. We did that for L = {ambn : m n} in class. (See Lecture Notes 14).
(b) L = {wcwR : w {a, b}*}. In class (again see Lecture Notes 14), we built a deterministic PDA to accept L
= {wcwR : w {a, b}*}. Its easy to turn it into a deterministic PDA that accepts L$.
(c) L = {cambm : m 0} {damb2m : m 0}. Often its hard to build a deterministic PDA for a language that is
formed by taking the union of two other languages. For example, {ambm : m 0} {amb2m : m 0} would be
hard (in fact its impossible) because we have no way of knowing, until we run out of bs, whether were
expecting two bs for each a or just one. However, {cambm : m 0} {damb2m : m 0} is actually quite easy.
Every string starts with a c or a d. If its a c, then we know to look for one b for each a; if its a d, then we know
to look for two. So the first thing we do is to start our machine like this:
1
c
0
d
2
The machine that starts in state 1 is our classic machine for anbn, except of course that it must have a final
transition on $ to its final state.
We have two choices for the machine that starts in state 2. It can either push one a for every a it sees, and then
pop an a for every pair of bs, or it can push two as for every a it sees, and then pop one a for every b.
Homework 15
Parsing
(d) L = (amcbm : m 0} {amdb2m : m 0}. Now weve got another unioned language. But this time we dont
get a clue from the first character which part of the language were dealing with. That turns out to be okay
though, because we do find out before we have to start processing the bs whether weve got two bs for each a or
just one. Recall the two approaches we mentioned for machine 2 in the last problem. What we need here is the
first, the one that pushes a single a for each a it sees. Then, when we see a c or d, we branch and either pop an a
for each b or pop an a for every two bs.
2. (a) We need to apply left factoring to the two rules S () and S (A). We also need to eliminate the left
recursion from A A . S. Applying left factoring, we get the first column shown here. Then getting rid of left
recursion gets us the second column:
S (S
S (S
S )
S )
S A)
S A)
Sa
Sa
AS
A SA
A A.S
A .SA
A
(b) Notice that the point of the first rule, which was S (), was to get a set of parentheses with no A inside.
An alternative way to do that is to dump that rule but to add the rule A . Now we always introduce an A
when we expand S, but we can get rid of it later. If we do this, then theres no left factoring to be done. We still
have to get rid of the left recursion on A, just as we did above, however.
(c) If we change A A.S to S S.A, then theres no left recursion to get rid of and we can leave the rules
unchanged. Notice, though, that well get different parse trees this way, which may or may not be important. To
see this, consider the string (a.a.a) and parse it using both the original grammar and the one we get if we change
the last rule.
3. (a) True, since all regular languages are context-free.
(b) False, there exist languages that are context-free but not regular.
(c) False. All regular languages are also context-free and thus are generated by context-free grammars.
(d) True, since if L were regular, it would also be context free and thus would be accepted by some PDA.
(e) True, since there must also be a deterministic FSM and thus a deterministic PDA.
(f) False. Consider L = anbn. L' = {w {a, b}* : either some b comes before some a or there is an unequal
number of a's and b's.}. Clearly this language is not regular since we can't count the a's and b's.
(g) True, since the deterministic context-free languages are closed under complement.
(h) False. Suppose L = anc*bn, which is clearly not regular. Let x = aa, y = c, and z = bb. xynz L.
(i) False. L could be finite.
(j) False. L1 could be anbn and L2 could be { anbm : n m}. Neither is regular. But L1 L2 = {}, which
is regular.
(k) False. Let L1 = a* and L2 = {anbmcm : n m}. L2 is not context free. But L = L1L2 = a*bmcm, which is
context free.
(l) False. Let L = wwR.
(m) True.
(n) True, since we have a procedure for eliminating such unit productions.
(o) False, since there exist context-free languages that are not regular.
(p) True.
4. (a) No grammar in Chomsky Normal Form can generate , yet L.
Homework 15
Parsing
(b) In the original grammar, we could generate zero copies of AAA (by letting S go to ), one copy of AAA
(by letting S go to AAA), two copies (by letting S go to SS and then each of them to AAA), three copies of AAA
(by letting S go to SS, then one of the Ss goes to SS, then all three go to AAA), and so forth. We want to make
sure that we can still get one copy, two copies, three copies, etc. We just want to eliminate the zero copies
option. Note that the only role of S is to determine how many copies of AAA are produced. Once we have
generated As we can never go back and apply the S rules again. So all we have to do is to eliminate the
production S . The modified grammar to accept L - is thus:
G = ({S, A, B, C, a, b}, {a, b}, R, S), where R = {
S SS | AAA
A aA | Aa | b
If we convert this grammar to Chomsky Normal Form, we get:
G = ({S, A, B, C, a, b}, {a, b}, R, S), where R = {
S SS
A AC
S AB
Ab
B AA
Ca
A CA }
This grammar is still ambiguous.
(c) (from the grammar of part (b)): M = ({p, q}, {a, b}, {S, A, a, b,}, , p, {q})
= { ((p, , ), (q, S))
((q, , A), (q, aA))
((q, , S), (q, SS)
((q, , A), (q, Aa))
((q, , S), (q, AAA)
((q, , A), (q, b))
((q, a, a), (q, ))
((q, b, b), (q, )) }
5. L is not deterministic context free for essentially the same reason that wwR is not.
EE+T
ET
TT*F
TF
F (E)
F id
Step 2. There are no rules. We show steps 3, 4, and 5 next to each other, so it's clear where the rules in steps 4
and 5 came from. In each case, the first rule that is derived from a step 3 rule is next to its source. If more than
one rule is derived from any given step 3 rule, the second and others are shown immediately under the first.
That's why there are some blank lines in the first two columns.
6. The original grammar was:
Homework 15
Parsing
Step 5.
Step 4.
Step 3.
E * T, F
T * F
G" =
E TMF
E LER
E id
T LER
T id
P+
M*
L(
R)
E T T'
E LF'
E id
T L F'
T id
P+
M*
L(
R)
E EPT
TT*F
T TMF
F (E)
F LER
F id
Then we add:
ET*F
E (E)
E id
T (E)
T id
Homework 15
F id
E E E'
E' PT
T T T'
T' M F
F L F'
F' E R
F id
EE+T
Parsing
(since T' M F)
(since F' E R)
(since F' E R)
CS 341 Homework 16
Languages that Are and Are Not Context-Free
1. Show that the following languages are context-free. You can do this by writing a context free grammar or a
PDA, or you can use the closure theorems for context-free languages. For example, you could show that L is the
union of two simpler context-free languages.
(a) L = ancbn
(b) L = {a, b}* - {anbn : n 0}
(c) L = {ambncpdq : n = q, or m p or m + n = p + q}
(d) L = {a, b}* - L1, where L1 is the language {babaabaaabban-1banb : n n 1}.
2. Show that the following languages are not context-free.
2
(a) L = { a n : n 0}
(b) L = {www : w {a, b}*}
(c) L = {w {a, b, c}* : w has equal numbers of a's, b's, and c's}
(d) L = {anbman : n m}
(e) L = {anbmcnd(n+m) : m, n 0}
3. Give an example of a context free language ( *) that contains a subset that is not context free. Describe the
subset.
4. What is wrong with the following "proof" that anb2nan is context free?
(1) Both {anbn : n 0} and {bnan : n 0} are context free.
(2) anb2nan = {anbn}{bnan }
(3) Since the context free languages are closed under concatenation, anb2nan is context free.
5. Consider the following context free grammar: G = ({S, A, a, b}, {a, b}, R, S), where R = {
S aAS
Sa
A SbA
A SS
A ba }
(a) Answer each of the following questions True or False:
(i) From the fact that G is context free, it follows that there is no regular expression for L(G).
(ii) L(G) contains no strings of length 3.
(iii) For any string w L(G), there exists u, v, x, y, z such that w = uvxyz, |vy| 1, and uvnxynz L(G)
for all n 0.
(iv) If there exist languages L1 and L2 such that L(G) = L1 L2, then L1 and L2 must both be context
free.
(v) The language (L(G))R is context free.
(b) Give a leftmost derivation according to G of aaaabaa.
(c) Give the parse tree corresponding to the derivation in (b).
(d) Give a nondeterministic PDA that accepts L(G).
6. Show that the following language is context free:
L = {xxRyyRzzR : x, y, z {a, b}*}.
Homework 16
7. Suppose that L is context free and R is regular.

(a) Is L - R necessarily context free?
(b) Is R - L necessarily context free?
8. Let L1 = {anbm : n m}. Let R1 = {(a b)* : there is an odd number of a's and an even number of b's}. Show
a pda that accepts L1 R1.
Solutions
1. (a) L = ancbn. We can easily do this one by building a CFG for L. Our CFG is almost the same as the one we
did in class for anbn:
S aSB
Sc
(b) L = {a, b}* - {anbn : n 0}. In other words, weve got the complement of anbn. So we look at how a string
could fail to be in anbn. There are two ways: either the as and bs are out of order or there are not equal numbers
of them. So our language L is the union of two other languages:
L1 = (a b)* - a*b* (strings where the as and bs are out of order)
L2 = anbm n m (strings where the as and bs are in order but there arent matching numbers of them)
L1 is context free. We gave a context-free grammar for it in class (Lecture 12). L2 is the complement of the
regular language a*b*, so it is also regular and thus context free. Since the union of two context-free languages
is context free, L is context free.
(c) L = {ambncpdq : n = q or m p or m + n = p + q}. This one looks scary, but its just the union of three quite
simple context-free languages:
L1 = ambncpdq : n = q
L2 = ambncpdq : m p
L3 =ambncpdq : m + n = p + q
You can easily write context-free grammars for each of these languages.
(d) L = {a, b}* - L1, where L1 is the language {babaabaaabban-1banb : n n 1}. This one is interesting. L1
is not context free. But its complement L is. There are two ways to show this:
1. We could build a PDA. We cant build a PDA for L1: if we count the first group of as then well need to
pop them to match against the second. But then what do we do for the third? L is easier though. We dont
need to check that all the a groups are right. We simply have to find one that is wrong. We can do that with a
nondeterministic pda P. P goes through the string making sure that the basic b(a+b)+ structure is followed.
Also, it nondeterministically chooses one group of as to count. It then checks that the following group does
not contain one more a. If any branch succeeds (i.e., it finds a mismatched pair of adjacent a groups) then P
accepts.
2. We could use the same technique we used in (b) and (c) and decompose L into the union of two simpler
languages. Just as we did in (b), we ask how a string could fail to be in L1 and thus be in L. The answer is
that it could fail to have the correct b(a+b)+ structure or it could have regions of as that dont follow the rule
that each region must contain one more a than did its predecessor. Thus L is the union of two languages:
L2 = (a b)* - b(a+b)+
L3 = {xbambanby {a, b}* : m+1 n}.
Its easy to show that L2 is context free: Since b(a+b)+ is regular its complement is regular and thus context
free. L3 is also context free. You can build either a CFG or a PDA for it. Its very similar to the simpler
language anbm n m that we used in (b) above. So L1 = L2 L3 is context free.
2
2. (a) L = { a n : n 0}. Suppose L = { a n : n 0} were context free. Then we could pump. Let n = M2. So w
2
is the string with M 2 , or M4, a's.) Clearly |w| K, since M > K. So uvvxyyz must be in L (whatever v and y
Homework 16
are). But it can't be. Why not? Given our w, the next element of L is the string with (M2+1)2 a's. That's M4 +
2M2 + 1 (expanding it out). But we know that |vxy| M, so we can't pump in more than M a's when we pump
only once. Thus the string we just created can't have more than M4 + M a's. Clearly not enough.
(b) L = {www : w {a, b}*}. The easiest way to do this is not to prove directly that L = {www : w {a, b}*}
is not context free. Instead, let's consider L1 = L a*ba*ba*b. If L is context free, L1 must also be. L1 =
{anbanbanb : n 0}. To show that L1 is not context free, let's choose w = aMbaMbaMb. First we observe that
neither v nor y can contain b, because if either did, then, when we pump, we'd have more than three b's, which is
not allowed. So both must be in one of the three a regions. We consider the cases:
(1, 1) That group of a's will no longer match the other two, so the string is not in L1.
(2, 2)
"
(3, 3)
"
(1, 2) At least one of these two groups will have something pumped into it and will no longer match the
one that is left out.
(2, 3)
"
(1, 3) excluded since |vxy| M, so vxy can't span the middle region of a's.
(c) L = {w {a, b, c}*. Again, the easiest thing to do is first to intersect L = {w {a, b, c}* : w has equal
numbers of a's, b's, and c's} with a regular language. This time we construct L1 = L a*b*c*. L1 must be
context free if L is. But L1 = anbncn, which we've already proven is not context free. So L isn't either.
(d) L = {anbman : n m}. We'll use the pumping lemma to show that L = {anbman : n m} is not context free.
Choose w = aMbMaM. We know that neither v nor y can cross a and b regions, because if one of them did, then,
when we pumped, we'd get a's and b's out of order. So we need only consider the cases where each is in one of
the three regions of w (the first group of a's, the b's, and the second group of a's.)
(1, 1) The first group of a's will no longer match the second group.
(2, 2) If we pump in b's, then at some point there will be more b's than a's, and that's not allowed.
(3, 3) Analogous to (1, 1)
(1, 2) We must either (or both) pump a's into region 1, which means the two a regions won't match, or,
if y is not empty, we'll pump in b's but then eventually there will be more b's than a's.
(2, 3) Analogous to (1, 2)
(1, 3) Ruled out since |vxy| M, so vxy can't span the middle region of b's.
(e) L = {anbmcnd(n+m) : m, n 0}. We can show that L = {anbmcnd(n+m) : m, n 0} is not context free by
pumping. We choose w = aMbMcMd2M. Clearly neither v nor y can cross regions and include more than one letter,
since if that happened we'd get letters out of order when we pumped. So we only consider the cases where v and
y fall within a single region. We'll consider four regions, corresponding to a, b, c, and d.
(1, 1) We'll change the number of a's and they won't match the c's any more.
(1, 2) If v is not empty, we'll change the a's and them won't match the c's. If y is nonempty, we'll change the
number of b's and then we won't have the right number of d's any more.
(1, 3), (1, 4) are ruled out because |vxy| M, so vxy can't span across any whole regions.
(2, 2) We'll change the number of b's but then we won't have the right number of d's.
(2, 3) If v is not empty, we'll change the b's without changing the d's. If y is not empty, we'll change the c's and
they'll no longer match the a's.
(2, 4) is ruled out because |vxy| M, so vxy can't span across any whole regions.
(3, 3) We'll change the number of c's and they won't match the a's.
(3, 4) If v is not empty, we'll change c's and they won't match a's. If y is not empty, we'll change d's without
changing b's.
(4, 4) We'll change d's without changing a's or b's.
Homework 16
3. Let L = { anbmcp : n = m or m = p}. L is clearly context free. We can build a nondeterministic PDA M to
accept it. M has two forks, one of which compares n to m and the other of which compares m to p (skipping over
the a's). L1 = {anbmcp : n = m and m = p} is a subset of L. But L1 = anbncn, which we know is not context free.
4. (1) is fine. (2) is fine if we don't over interpret it. In particular, although both languages are defined in terms
of the variable n, the scope of that variable is a single language. So within each individual language definition,
the two occurrences of n are correctly interpreted to be occurrences of a single variable, and thus the values must
be same both times. However, when we concatenate the two languages, we still have two separate language
definitions with separate variables. So the two n's are different. This is the key. It means that we can't assume
that, given {anbn}{bnan }, we choose the same value of n for the two strings we choose. For example, we could
get a2b2b3a3, which is a2b5a3, which is clearly not in {anb2nan}.
5. (a)
(i) False, since all regular languages are also context free.
(ii) True.
(iii) False. For example a L, but is not long enough to contain pumpable substrings.
(iv) False.
(v) True, since the context-free languages are closed under reversal.
(b) S aAs aSSS aaSS aaaS aaaaAS aaaabaS aaaabaa.
(c)
S
a
A
b
S
a
(d) M = ({p, q}, {a, b}, {a, b}, p, {q}, ), where =

{((p, , ), (q, S)), ((q, a, a), (q, )), ((q, b, b), (q, )), ((q, , S), (q, aAS)), ((q, , S), (q, a)),
((q, , A), (q, SbA)), ((q, , A), (q, SS)), ((q, , A), (q, ba)) }
6. The easy way to show that L = {xxRyyRzzR : x, y, z {a, b}*} is context free is to recall that we've already
shown that {xxR: x {a, b}*} is context free, and the context-free languages are closed under concatenation.
But we can also do this directly by giving a grammar for L:
S AAA
A aAa
A bAb
A
7. (a) L - R is context free. L - R = L R' (the complement of R). R' is regular (since the regular languages are
closed under complement) and the intersection of a context-free language and a regular language is context-free,
so L - R is context free.
(b) R - L need not be context free. R - L = R L'. But L' may not be context free, since the context-free
languages are not closed under complement. (The deterministic ones are, but L may not be deterministic.) If we
let R = *, then R - L is exactly the complement of L.
8. M1, which accepts L1 = ({1, 2}, {a, b}, {a}, , 1, {2}), =
((1, a, ), (1, a))
((1, b, a), (2, ))
((1, , ), (2, )
((2, b, a), (2, )
Homework 16
((2, , a), (2, )

M2, which accepts R1 = ({1, 2, 3, 4}, {a, b}, , 1, {2}), =
(1, a, 2)
(1, b, 3)
(2, a, 1)
(2, b, 4)
(3, a, 4)
(3, b, 1)
(4, a, 3)
(4, b, 2)
M3, which accepts L1 R1 = ({(1, 1), (1, 2), (1, 3), (1, 4), (2, 1), (2, 2,), (2, 3), (2, 4)}, {a, b}, {a}, , (1,1),
{(2,2)}), =
(((1, 1), , ), ((2, 1), ))
(((2, 1), b, a), ((2, 3), ))
(((1, 1), a, ), ((1, 2), a))
(((1, 2), , ), ((2, 2), ))
(((2, 2), b, a), ((2, 4), ))
(((1, 1), b, a), ((2, 3), ))
(((1, 3), , ), ((2, 3), ))
(((2, 3), b, a), ((2, 1), ))
(((1, 2), a, ), ((1, 1), a))
(((1, 4), , ), ((2, 4), ))
(((2, 4), b, a), ((2, 2), ))
(((1, 2), b, a), ((2, 4), ))
(((2, 1), , a), ((2, 1), ))
(((1, 3), a, ), ((1, 4), a))
(((2, 2), , a), ((2, 2), ))
(((1, 3), b, a), ((2, 1), ))
(((2, 3), , a), ((2, 3), ))
(((1, 4), a, ), ((1, 3), a))
(((2, 4), , a), ((2, 4), ))
(((1, 4), b, a), ((2, 2), ))
Homework 16
CS 341 Homework 17
Turing Machines
1. Let M = (K, , , s, {h}), where
K = {q0, q1, h},
= {a, b, , },
s = q0,
and is given by the following table,
q
q0
q0
q0
q0
q1
q1
q1
q1
a
b
a
b
(q,
)
(q1, b)
(q1, a)
(h, )
(q0, )
(q0, )
(q0, )
(q0, )
(q1, )
(a) Trace the computation of M starting from the configuration (q0, aabbba).
(b) Describe informally what M does when started in q0 on any square of a tape.
2. Repeat Problem 1 for the machine M = (K, , , s, {h}), where
K = {q0, q1, q2, h},
= {a, b, , },
s = q0,
and is given by the following table (the transitions on are (q, ) = (q, ), and are omitted).
q
q0
q0
q0
q1
q1
q1
q2
q2
q2
a
b
a
b
a
b
(q,
)
(q1, )
(q0, )
(q0, )
(q1, )
(q2, )
(q1, )
(q2, )
(q2, )
(h, )
Start from the configuration (q0, abbbbaba).

3. Let M be the Turing machine M = (K, , , s, {h}), where
K = {q0, q1, q2, h},
= {a, , },
s = q0,
and is given by the following table.
Let n 0. Describe carefully what M does when started in the configuration (q0, ana).
Homework 17
Turing Machines
q
q0
q0
q0
q1
q1
q1
q2
q2
q2
(q,
)
(q1, )
(q0, )
(q0, )
(q2, )
(h, )
(q1, )
(q2, a)
(q0, )
(q2, )
4. Design and write out in full a Turing machine that scans to the right until it finds two consecutive a's and then
halts. The alphabet of the Turing machine should be {a, b, , }.
5. Give a Turing machine (in our abbreviated notation) that takes as input a string w {a, b}* and squeezes out
the a's. Assume that the input configuration is (s, w) and the output configuration is (h, w'), where w' = w
with all the a's removed.
6. Give a Turing machine (in our abbreviated notation) that shifts its input two characters to the right.
Input:
w
Output:
w
7. (L & P 5.7.2) Show that if a language is recursively enumerable, then there is a Turing machine that
enumerates it without ever repeating an element of the language.
Solutions
/
1. (a)
/
a,b,/
q0
q1
a/b; b/a
/
h
q0, aabbba
q1, babbba
q0, babbba
q1, bbbbba
q0, bbbbba
q1, bbabba
q0, bbabba
q1, bbaaba
q0, bbaaba
q1, bbaaaa
q0, bbaaaa
q1, bbaaab
q0, bbaaab
h, bbaaab
(b) Converts all a's to b's, and vice versa, starting with the current symbol and moving right.
Homework 17
Turing Machines
2. (a)
b,/
a,/
a/
q0, abbbbaba
q0, abbbbaba
q0, abbbbaba
q0, abbbbaba
q1, abbbbaba
q1, abbbbaba
q2, abbbbaba
h, abbbbaba
a,b,/
b/
q0
q1
q2
/
h
(b) M goes right until if finds an a, then left until it finds a b, then right until it finds a blank.
/
3.
a/a
a/
a/
q0
q1
q2
/
h
q0, a a a a a
q1, a a a a a
q2, a a a a
q0, a a a a
q1, a a a a
q2, a a a
q0, a a a
q1, a a a
h,a a a
/
M changes every other a, moving left from the start, to a blank. If n is odd, it loops. If n is even, it halts.
4.
a/
a/a
q0
q1
b,/
M = (K, , , s, {h}), where

K = {q0, q1, h},
= {a, b, , },
s = q0
b,/
5. The idea here is that first we'll push all b's to the left, thus squeezing all the a's to the right. Then we'll just
replace the a's with blanks. In more detail: scan the string from left to right. Every time we find an a, if there are
any b's to the right of it we need to shift them left. So scan right, skipping as many a's as there are. When we
find a b, swap it with the last a. That squeezes one a further to the right. Go back to the left end and repeat.
Eventually all the a's will come after all the b's. At that point, when we look for a b following an a, all we'll find
is a blank. At that point, we just clean up by rewriting all the a's as blanks.
>Ra,
Rb,
aLbL
L
a
Homework 17
Turing Machines
6. The idea is to start with the rightmost character of w, rewrite it as a blank, then move two squares to the right
and plunk that character back down. Then scan left for the next leftmost character, do the same thing, and so
forth.
>L
R2aLL
R3
7. Suppose that M is the Turing machine that enumerates L. Then construct M* to enumerate L with no
repetitions: M* will begin by simulating M. But whenever M outputs a string, M* will first check to see if the
string has been output before (see below). If it has, it will just continue looking for strings. If not, it will output
it, and it will also write the string, with # in front of it, at the right end of the tape. To check whether a string has
been output before, M* just scans its tape checking for the string it is about to output.
Homework 17
Turing Machines
CS 341 Homework 18
1. Present Turing machines that decide the following languages over {a, b}:
(a)
(b) {}
(c) {a}
(d) {a}*
2. Consider the simple (regular, in fact) language L = {w {a,b}* : |w| is even}
(a) Give a Turing machine that decides L.
(b) Give a Turing machine that semidecides L.
3. Give a Turing machine (in our abbreviated notation) that accepts L = {anbman : m > n}
4. Give a Turing machine (in our abbreviated notation) that accepts L = {ww : w {a, b}*}
5. Give a Turing machine (in our abbreviated notation) that computes the following function from strings in {a,
b}* to strings in {a, b}* : f(w) = wwR.
6. Give a Turing machine that computes the function f: {a,b,c}* N (the integers), where f(w) = the number of
a's (in unary) in w.
7. Let w and x be any two positive integers encoded in unary. Show a Turing machine M that computes
f(w, x) = w + x.
Represent the input to M as
w;x
8. Two's complement form provides a way to represent both positive and negative binary integers. Suppose that
the number of bits allocated to each number is k (generally the word size). Then each positive integer is
represented simply as its binary encoding, with leading zeros. Each negative integer n is represented as the result
of subtracting |n| from 2k, where k is the number of bits to be used in the representation. Given a fixed k, it is
possible to represent any integer n if -2k-1 n 2k-1 -1. The high order digit of each number indicates its sign: it
is zero for positive integers and 1 for negative integers.
Examples, for k = 4:
0 = 0000, 1 = 0001, 2 = 0010, 3 = 0011, 4 = 0100, 5 = 0101, 6 = 0110, 7 = 0111
-1 = 1111, -2 = 1110, -3 = 1101, -4 = 1100, -5 = 1011, -6 = 1010, -7 = 1001, -8 = 1000
Since Turing machines don't have fixed length words, we'd like to be able to represent any integer. We will
represent positive integers with a single leading 0. We will represent each negative integer n as the result of
subtracting n from 2i+1, where i is the smallest value such that 2i |n|. For example, -65 will be represented as
1111111, since 27 (128) 65, so we subtract 65 (01000001 in binary) from 28 (in binary, 100000000). We need
the extra digit (i.e., we subtract from 2i+1 rather than from 2i) because, in order for a positive number to be
interpreted as positive, it must have a leading 0, thus consuming an extra digit.
Let w be any integer encoded in two's complement form. Show a Turing machine that computes f(w) = -w.
Homework 18
Solutions
1 (a) We should reject everything, since no strings are in the language.
>n
(b) Other than the left boundary symbol, the tape should be blank:
>R
y
(c) Just the single string a:
>R
a
n
(d) Any number of a's:
a
>R
(a,)
2. (a)
a,b
>R
a,b
>R
Homework 18
a,b
(b)
a,b
3. The idea is to make a sequence of passes over the input. On each pass, we mark off (with d, e, and f) a
matching a, b, and a. This corresponds to the top row of the machine shown here. When there are no matching
groups left, then we accept if there is nothing left or if there are b's in the middle. If there is anything else, we
reject. It turns out that a great deal of this machine is essentially error checking. We get to the R on the second
row as soon as we find the first "extra" b. We can loop in it as long as we find b's. If we find a's we reject. If we
find a blank, then the string had just b's, which is okay, so we accept. Once we find an f, we have to go to the
separate state R on the third row to skip over the f's and make sure we get to the final blank without either any
more b's or any more a's.
d,e
a,e
R
a d
b,f
R
fL
a,b
R
4. The hard part here is that we don't know where the middle of the string is. So we don't know where the
boundary between the first occurrence of w ends and the second begins. We can break this problem into three
subroutines, which will be executed in order:
(1) Find the middle and mark it. If there's a lone character in the middle (i.e., the length of the input string isn't
even), then reject immediately.
(2) Bounce back and forth between the beginning of the first w and the beginning of the second, marking off
characters if they match and rejecting if they don't.
(3) If we get to the end of the w's and everything has matched, accept.
Let's say a little more about step (1). We need to put a marker in the middle of the string. The easiest thing to do
is to make a double marker. We'll use ##. That way, we can start at both ends (bouncing back and forth), moving
a marker one character toward the middle at each step. For example, if we start with the tape aabbaabb,
after one mark off step we'll have a#abbaab#b, then aa#bbaa#bb, and finally
aabb##aabb. So first we shift the whole input string two squares to the right on the tape to make room
for the two markers. Then we bounce back and forth, moving the two markers toward the center. If they meet,
we've got an even length string and we can continue. If they don't, we reject right away.
Homework 18
5. The idea is to work from the middle. We'll scan right to the rightmost character of w (which we find by
scanning for the first blank, then backing up (left) one square. We'll rewrite it so we know not to deal with it
again (a's will become 1's; b's will become 2's.) Then we move right and copy it. Now if we scan back left past
any 1's or 2's, we'll find the next rightmost character of w. We rewrite it to a 1 or a 2, then scan right to a blank
and copy it. We keep this up until, when we scan back to the left, past any 1's or 2's, we hit a blank. That means
we've copied everything. Now we scan to the right, replacing 1's by a's and 2's by b's. Finally, we scan back to
the left to position the read head to the left of w.
a
>R L
1Ra
L1,2 L1,2
b
2Rb
a
1
a,b
2
b
L
6. The idea here is that we need to write a 1 for every a and we can throw away b's and c's. We want the 1's to
end up at the left end of the string, so then all we have to do to clean up at the end is erase all the b's and c's to the
right of the area with the 1's. So, we'll start at the left edge of the string. We'll skip over b's and c's until we get
to an a. At that point, rewrite it as b so we don't count it again. Then (remembering it in the state) scan left until
we get to the blank (if this is the first 1) or we get to a 1. In either case, move one square to the right and write a
1. We are thus overwriting a b or a c, but we don't care. We're going to throw them away anyway. Now start
again scanning to the right looking for an a. At some point, we'll come to the end of string blank instead. At that
point, just travel leftward, rewriting all the b's and c's to blank, then cross the 1's and land on the blank at the left
of the string.
>R
bL,1R1
b,c
b,c
1
7. All we have to do is to concatenate the two strings. So shift the second one left one square, covering up the
semicolon.
>R; R 1
;L1
LL
Homework 18
8. Do a couple of examples of the conversion to see what's going on. What you'll observe is that we want to scan
from the right. Initially, we may see some zeros, and those will stay as zeros. If we ever see a 1, then we rewrite
the first one as a 1. After that, we're dealing with borrowing, so we swap all digits: every zero becomes a one and
every one becomes a zero, until we hit the blank at the end of the string and halt.
>R L
1
1L
1
0
Homework 18
CS 341 Homework 19
1. Consider the language L = {wwR}.
(a) Describe a one tape Turing machine to accept L.
(b) Describe a two tape Turing machine to accept L.
(c) How much more efficient is the two tape machine?
2. Give (in abbreviated notation) a nondeterministic Turing machine that accepts the language
L = {wwRuuR : w, u {a, b}*}
Solutions
(1) (a) The one tape machine needs to bounce back and forth between the beginning of the input string and the
end, marking off matching symbols.
(b) The two tape machine works as follows: If the input is , accept. If not, copy the input to the second tape and
record in the state that you have processed an even number of characters so far. Now, start the first tape at the
left end and the second tape at the right end . Check that the symbols on the two tapes are the same. If not,
reject. If so, move the first tape head to the right and the second tape head to the left. Also record that you have
processed an odd number and continue, each time using the state to keep track of whether youve seen an even or
odd number of characters so far. When you reach the end of the input tape, accept if youve seen an even number
of characters. Reject if youve seen an odd number. (The even/odd counter is necessary to make sure that you
reject strings such as aba.)
(c) The one tape machine takes time proportional to the square of the length of the input, since for an input of
length n it will make n passes over the input, each of which takes on average n/2 steps. The two tape machine
takes time that's linear in n. It takes n steps to copy, then another n steps to compare.
2. The idea is just to use nondeterminism to guess the location of the boundary between the w and u regions.
Each path will choose a spot, shift the u region to the right, and insert a boundary marker #. Once this is done,
the machine simply checks each region for wwR. If we get a string in L, one of the guessed paths will work.
Homework 19
CS 341 Homework 20
1. Find grammars that generate the following languages:
(a) L = {ww : w {a, b}*}
n
(b) L = { a 2 : n 0}
(c) L = { anb2nc3n : n 1}
(d) L = {wR : w is the social security number of a living American citizen}
(e) L = {wcmdn : w {a, b}* and m = the number of a's in w and n equals the number of b's in w}
2. Find a grammar that computes the function f(w) = ww, where w {a, b}*.
Solutions
1. (a) L = {ww : w {a, b}*}
There isnt any way to generate the two ws in the correct order. Suppose we try. Then we could get aSa.
Suppose we want b next. Then we need Sa to become bSab, since the new b has to come after the a thats
already there. That could work. Now we have abSab. Lets say we want a next. Now Sab has to become aSaba.
The problem is that, as the length of the string grows, so does the number of rules well need to cope with all the
patterns we could have to replace. In a finite number of rules, we cant deal with replacing S (which we need to
do to get the next character in the first occurrence of w), and adding a new character that is arbitrarily far away
from S.
The other approach we could try would be have a rule S WW, and then let W generate a string of a's and b's.
But this won't work, since we have no way to control the expansion of the two W's so that they produce the same
thing.
So what we need to do is to generate wwR and then, carefully, reverse the order of the characters in wR. What
well do is to start by erecting a wall (#) at the right end of the string. Then well generate wwR. Then, in a
second phase, well take the characters in the second w and, one at a time, starting with the leftmost, move it right
and then move it past the wall. At each step, we move each character up to the wall and then just over it, but we
dont reverse characters once they get over the wall. The first part of the grammar, which will generate wTwR,
looks like this:
S S1 #
S1 aS1a
S1 bS1b
S1 T
This inserts the wall at the right.
T will mark the left edge of the portion that needs to be reversed.
At this point, we can generate strings such as abbbTbbba#. What we need to do now is to reverse the string of
as and bs that is between T and #. To do that, we let T spin off a marker Q, which we can pass rightward
through the string. As it moves to the right, it will take the first a or b it finds with it. It does this by swapping the
character it is carrying (the one just to the right of it) with the next one to the right. It also moves itself one
square to the right. The four rules marked with * accomplish this. When Qs character gets to the # (the rules
marked **), the a or b will swap places with the # (thus hopping the fence) and the Q will go away. We can keep
doing this until all the as and bs are behind the fence and in the right order. Then the final T# will drop out.
Here are the rules for this phase:
Homework 20
T TQ
Qaa aQa
*
Qab bQa
*
Qbb bQb
*
Qba aQb
*
Qa# #a
**
Qb# #b
**
T#
So with R as given above, the grammar G = ({S, S1, #, T, Q, a, b}, {a, b}, R, S}
n
(b) L = { a 2 : n 0}
The idea here is first to generate the first string, which is just a. Then think about the next one. You can derive it
by taking the previous one, and, for every a, write two as. So we get aa. Now to get the third one, we do the
same thing. Each of the two as becomes two and we have four, and so forth. So we need a rule to get us started
and to indicate the possibility of duplication. Then we need rules to actually do the duplication. To make
duplication happen, we need a symbol that gets generated by S indicating the option to repeat. Well use P.
Since duplication can happen an arbitrary number of times, we need P to spin off as many individual duplication
commands as we want. Well use R for that. The one other thing we need is to make sure, if we start a
duplication step, that we finish it. In other words, suppose we currently have aaaa. If we start duplicating the as,
we must duplicate all of them. Otherwise, we might end up with, for example, seven as. So well introduce a
left edge marker, #. Once we fire up a duplication (by creating an R), well only stop (i.e., get rid of R) when R
has made it all the way to the other end of the string (namely the left end since it starts at the right). So we get
the following rules:
S #aP
P lets us start up duplication processes as often as we like.
P
When weve done as many as we want, we get rid of P.
P RP
R will actually do a duplication by moving leftward, duplicating every a it sees.
aR Raa
Actually duplicates one a, and moves R one square to the left so it moves on to the next a
#R #
Get rid of R once it's made it all the way to the left
#
Get of # at the end
So with R as given above, the grammar G = ({S, P, R, #, a, b}, {a, b}, R, S}
(c) L = { anb2nc3n : n 1}
This one is very similar to anbncn. The only difference is that we will churn out b's in pairs and c's in triples each
time we expand S. So we get:
S aBSccc
S aBccc
Ba aB
Bc bbc
Bb bbb
So with R as given above, the grammar G = ({S, B, a, b, c}, {a, b, c}, R, S}
(d) L = {wR : w is the social security number of a living American citizen}
This one is regular. There is a finite number of such social security numbers. So we need one rule for each
number. Each rule is of the form S <valid number>. So with that collection of rules as R, the grammar G =
({S, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, R, S}
(e) L = {wcmdn : w {a, b}* and m = the number of a's in w and n equals the number of b's in w}
The idea here is to generate a c every time we generate an a and to generate a d every time we generate a b. We'll
do this by generating the nonterminals C and D, which we will use to generate c's and d's once everything is in
the right place. Once we've finished generating all the a's and b's we want, the next thing we need to do is to get
Homework 20
all the D's to the far right of the string, all the C's next, and then have the a's and b's left alone at the left. We
guarantee that everything must line up that way by making sure that C can't become c and D can't become d
unless things are right. To do this, we require that D can only become d if it's all the way to the right (i.e., it's
followed by #) or it's got a d to its right. Similarly with C. We can do this with the following rules;
S S1#
S1 aS1C
S1 bS1D
S1
DC CD
D# d
Dd dd
C# c
Cd cd
Cc cc
#
So with R as given above, the grammar G = ({S, S1, C, D, #, a, b, c, d}, {a, b, c, d}, R, S}
2. We need to find a grammar that computes the function f(w) = ww. So we'll get inputs such as SabaS. Think of
the grammar we'll build as a procedure, which will work as described below. At any given time, the string that
has just been derived will be composed of the following regions:
<the part of w that
has already been
inserted
copied>
<the part of w that

has not yet been
T (inserted when the

<the part of the
W (also
first character moves
second w that has
copied, which may

into the copy region)
have within it a
character (preceded by #)
that is currently being
copied by being moved
through the region>
been copied so far,

which may have
within it a character
(preceded by %) that
is currently being
moved through the
region>
when T
is)
Most of the rules come in pairs, one dealing with an a, the other with b.
SS
Sa aS#a
Handles the empty string.

Move S past the first a to indicate that it has already been copied. Then start copying it by
introducing a new a, preceded by the special marker #, which we'll use to push the new a to the
right end of the string.
Sb bS#b
Same for copying b.
#aa a#a
Move the a we're copying past the next character if it's an a.
#ab b#a
Move the a we're copying past the next character if it's a b.
#ba a#b
Same two rules for pushing b.
#bb b#b
"
#aS #aTW We've gotten to the end of w. This is the first character to be copied, so the initial S is at the end
of w. We need to create a boundary between w and the copied w. T will be that boundary. We
also need to create a boundary for the end of the copied w. W will be that boundary. T and W
are adjacent at this point because we haven't copied any characters into the copy region yet.
#bS #aTW Same if we get to the end of w pushing b.
#aT T%a
Jump the a we're copying into the copy region (i.e., to the right of T). Get rid of #, since we're
done with it. Introduce %, which we'll use to push the copied a through the copy region.
#bT T%b Same if we're pushing b.
Homework 20
%aa a%a
%ab b%a
%ba a%b
%bb b%b
%aW aW
%bW bW
ST
W
Push a to the right through the copied region in exactly the same way we pushed it through w,
except we're using % rather than # as the pusher. This rule pushes a past a.
Pushes a past b.
Same two rules for pushing b.
"
We've pushed an a all the way to the right boundary, so get rid of %, the pusher.
Same for a pushed b.
All the characters from w have been copied, so they're all to the left of S, which causes S to be
adjacent to the middle marker T. We can now get rid of our special walls. Here we get rid of S
and T.
Gid rid of W. Note that if we do this before we should, there's no way to get rid of %, so any
derivation path that does this will fail to produce a string in {a, b}*.
So with R as given above, the grammar G = ({S, T, W, #, %,a, b}, {a, b}, R, S}
Homework 20
CS 341 Homework 21
Undecidability
1. Which of the following problems about Turing machines are solvable, and which are undecidable? Explain
your answers carefully.
(a) To determine, given a Turing machine M, a state q, and a string w, whether M ever reaches state q when
started with input w from its initial state.
(b) To determine, given a Turing machine M and a string w, whether M ever moves its head to the left when
started with input w.
(c) To determine, given two Turing machines, whether one semidecides the complement of the language
semidecided by the other.
(d) To determine, given a Turing machine M, whether the language semidecided by M is finite.
2. Show that it is decidable, given a pushdown automaton M with one state, whether L(M) = *. (Hint: Show
that such an automaton accepts all strings if and only if it accepts all strings of length one.)
3. Which of the following problems about context-free grammars are solvable, and which are undecidable?
Explain your answers carefully.
(a) To determine, given a context-free grammar G, is L(G)?
(b) To determine, given a context-free grammar G, is {} = L(G)?
(c) To determine, given two context-free grammars G1 and G2, is L(G1) L(G2)?
4. The nonrecursive languages L that we have discussed in class all have the property that either L or the
complement of L is recursively enumerable.
(a) Show by a counting argument that there is a language L such that neither L nor its complement is recursively
enumerable.
(b) Give an example of such a language.
Solutions
1. (a) To determine, given a Turing machine M, a state q, and a string w, whether M ever reaches state q when
started with input w from its initial state. This is not solvable. We can reduce H to it. Essentially, if we can tell
whether a machine M ever reaches some state q, then let q be M's halt state (and we can massage M so it has only
one halt state). If it ever gets to q, it must have halted. More formally:
L1 = H =
(?M2) L2 =
{s = "M" "w" : M halts on input string w}
{s : "M" "w" "q" : M reaches state q when started with input w from its initial state}
Let ' create, from M the machine M* as follows. Initially M* equals M. Next, a new halting state H is created in
M*. Then, from each state that was a halting state in M, we create transitions in M* such that for all possible
values of the current tape square, M* goes to H. We create no other transitions to H. Notice that M* will end up
in H in precisely the same situations in which M halts.
Now let ("M" "w") = '("M") "w" "H"
So, if M2 exists, then M1 exists. It invokes ' to create M*. Then it passes "M*", "w", and "H" to M2 and returns
whatever M2 returns. But M1 doesn't exist. So neither does M2.
Homework 21
Undecidability
(b) To determine, given a Turing machine M and a string w, whether M ever moves its head to the left when
started with input w. This one is solvable. We will assume that M is deterministic. We can build the deciding
machine D as follows. D starts by simulating the operation of M on w. D keeps track on another tape of each
configuration of M that it has seen so far. Eventually, one of the following things must happen:
1. M moves its head to the left. In this case, we say yes.
2. M is stuck on some square s of the tape. In other words, it is in some state p looking at some square s on the
tape and it has been in this configuration before. If this happens and M didn't go left yet, then M simply
hasn't moved off of s. And it won't from now on, since it's just going to do the same thing at this point as it
did the last time it was in this configuration. So we say no.
3. M moves off the right hand edge of the input w. So it is in some state p looking at a blank. Within k steps (if
k is the number of states in M), M must repeat some state p. If it does this without moving left, then again we
know that it never will. In other words, if the last time it was in the configuration in which it was in state p,
looking at a blank, there was nothing to the right except blanks, and it can't move left, and it is again in that
same situation, it will do exactly the same thing again. So we say no.
(c) To determine, given two Turing machines, whether one semidecides the complement of the language
semidecided by the other. This one is not solvable. We can reduce to it the problem, "Given a Turing machine
M, is there any string at all on which M halts?" (Which is equivalent to "Is L(M) = ?") In the book we show
that this problem is not solvable. What we'll do is to build a machine M* that semidecides the language *,
which is the complement of the language . If we could build a machine to tell, given two Turing machines,
whether one semidecides the complement of the language semidecided by the other, then to find out whether any
given machine M accepts anything, we'd pass M and our constructed M* to this new machine. If it says yes, then
M accepts . If it says no, then M must accept something. Formally:
L1 =
{s = "M" M halts on some string w}
(?M2) L2 =
{s = "M1" "M2" : M1 decides the complement of the language semidecided by M2}
M accepts strings over some input alphabet . Let ' construct a machine M* that semidecides the language *.
Then ("M") = "M" "'(M)".
So, if M2 exists, then M1 exists. It invokes ' to create M*. Then it passes "M" and "M*" to M2 and returns the
opposite of whatever M2 returns (since M2 says yes if L(M) = and M1 wants to say yes if L(M) ). But M1
doesn't exist. So neither does M2.
(d) To determine, given a Turing machine M, whether the language semidecided by M is finite. This one isn't
solvable. We can reduce to it the problem, "Given a Turing machine M, does M halt on ?" We'll construct,
from M, a new machine M*, which erases its input tape and then simulates M. M* halts on all inputs iff M halts
on . If M doesn't halt on , then M* halts on no inputs. So there are two situations: M* halts on all inputs (i.e.,
L(M*) is infinite) or M* halts on no inputs (i.e., L(M*) is finite). So, if we could build a Turing machine M2 to
decide whether L(M*) is finite or infinite, we could build a machine M1 to decide whether M halts on .
Formally:
{s = "M" M halts on }
L1 =
(?M2) L2 =
Homework 21
{s = "M" is finite}
Undecidability
Let construct the machine M* from "M" as described above.

So, if M2 exists, then M1 exists. It invokes to create M* which accepts a finite language precisely if M accepts
. But M1 doesn't exist. So neither does M2.
2. M only has one state S. If S is not a final state, then L(M) = , which is clearly not equal to *, so we say no.
Now suppose that S is a final state. Then M accepts . Does it also accept anything else? To accept any single
character c in , there must be a transition ((S, c, ), (S, )). In other words, we must be able to end up in S with
an empty stack if, looking at an empty stack, we see c. If there is not such a transition for every element c of ,
then we say no, since we clearly cannot get even all the one character strings in *. Now, suppose that all those
required transitions do exist. Then, we can stay in S with an empty stack (and thus accept) no matter what
character we see next and no matter what is on the stack (since these transitions don't check the stack). So, if M
accepts all strings in * of length one, then it accepts all strings in *. Note that if M is deterministic, then if it
does have all the required transitions it will have no others, since all possible configurations are accounted for
3. (a) To determine, given a context-free grammar G, is L(G) This is solvable by using either top down or
bottom up parsing on the string .
(b) To determine, given a context-free grammar G, is {} = L(G) This is solvable. By the context-free
pumping theorem, we know that, given a context-free grammar G generating a language L(G), if there is a string
of length greater than BT in L, then vy can be pumped out to create a shorter string also in L (the string must be
shorter since |vy| >0). We can, of course, repeat this process until we reduce the original string to one of length
less than BT. This means that if there any strings in L, there are some strings of length less than BT. So, to see
whether L = {}, we do the following: First see whether L(G) by parsing. If not, we say no. If is in L, then
we need to determine whether any other strings are also in L. To do this, we test all strings in * of length up to
BT+1. If we find one, we say no, L {}. If we don't find any, we can assert that L = {}. Why? If there is a
longer string in L and we haven't found it yet, then we know, by the pumping theorem, that we could pump out vy
until we got a string of length BT or less. If were not in L, we could just test up to length BT and if we didn't
find any elements of L at all, we could stop, since if there were bigger ones we could pump out and get shorter
ones but there aren't any. However, because is in L, what about the case where we pump out and get ? That's
why we go up to BT+1. If there are any long strings that pump out to , then there is a shortest such string, which
can't be longer than BT+1 since that's the longest string we can pump out (by the strong version of the pumping
theorem).
(c) To determine, given two context-free grammars G1 and G2, is L(G1) L(G2) This isn't solvable. If it were,
then we could reduce the unsolvable problem of determining whether L(G1) = L(G2) to it. Notice that L(G1) =
L(G2) iff L(G1) L(G2) and L(G2) L(G1). So, if we could solve the subset problem, then to find out whether
L(G1) = L(G2), all we do is ask whether the first language is a subset of the second and vice versa. If both
answers are yes, we say yes. Otherwise, we say no. Formally:
L1 =
{s : s = G1 G2, G1 and G2 are context-free grammars, and L(G1) = L(G2) }
(?M2) L2 = {s : s = G1 G2, G1 and G2 are context-free grammars, and L(G1) L(G2) }

If M2 exists, then M1(G1 G2) = M2(G1 G2) AND M2(G2 G1). To write this out in our usual notation so that the last
function that gets applied is M2, is sort of tricky, but it can, of course be done: Don't worry about doing it. If you
can write any function for M1 that is guaranteed to be recursive if M2 exists, then you've done the proof.
Homework 21
Undecidability
4. (a) If any language L is recursively enumerable, then there is a Turing machine that semidecides it. Every
Turing machine has a description of finite length. Therefore, the number of Turing machines, and thus the
number of recursively enumerable languages, is countably infinite (since the power set of a countable set is
countably infinite). If, for some language L, its complement is re, then it must have a semideciding Turing
machine, so there is a countably infinite number of languages whose complement is recursively enumerable. But
there is an uncountable number of languages. So there must be languages that are not recursively enumerable
and do not have recursively enumerable complements.
(b) L = {"M" : M halts on the input 0 and M doesn't halt on the input 1}.
The complement of L = {"M" : M doesn't halt on the input 0 or M halts on the input 1}. Neither of these
languages is recursively enumerable because of the doesn't halt piece.
Homework 21
Undecidability
CS 341 Homework 22
Review
1. Given the following language categories:
A:
L is finite.
B:
L is not finite but is regular.
C:
L is not regular but is deterministic context free
D:
L is not deterministic context free but is context free
E:
L is not context free but is Turing decidable
F:
L is not Turing decidable but is Turing acceptable
G:
L is not Turing acceptable
Assign the appropriate category to each of the following languages. Make sure you can justify your answer.
a. _____ {anbkn : k = 1 or k = 2, n 0}
b. _____ {anbkn : k = 0 or k = 1, n 0}
c. _____ {anbncn : n 0}
d. _____ {anbncm : n 0, m 0}
e. _____ {anbn : n 0} a*
f. _____ {anbm : n is prime and m is even}
g. _____ {anbmcm+n : n 0, m 0}
h. _____ {anbmcmn : n 0, m 0}
i. _____ {anbm : n 0, m 0}
j. _____ {xy : x a*, y b*, |x| = |y|}
k. _____ {xy : x a*, y a*, |x| = |y|}
l. _____ {x : x {a, b, c}*, and x has 5 or more a's}
m._____ {"M" : M accepts at least 1 string}
n._____ {"M" : M is a Turing machine that halts on input and |"M"| 1000}
o._____ {"M" : M is a Turing machine with 50 states}
p._____ {"M" : M is a Turing machine such that L(M) = a*}
q. _____ {x : x {A, B, C, D, E, F, G}, and x is the answer you write to this question}
Solutions
a. __D__ {anbkn : k = 1 or k = 2, n 0}
We havent discussed many techniques for proving that a context free language isnt deterministic, so we cant
prove that this one isnt. But essentially the reason this one isnt is that we dont know what to do when we see
bs. Clearly, we can build a pda M to accept this language. As M reads each a, it pushes it onto the stack. When
it starts seeing bs, it needs to start popping as. But theres no way to know, until either it runs out of bs or it
gets to the n+1st b, whether to pop an a for each b or hold back and pop an a for every other b. So M is not
deterministic.
b. __C__ {anbkn : k = 0 or k = 1, n 0}
This one is looks very similar to a, but its different in one key way. Remember that the definition of
deterministic context free is that it is possible to build a deterministic pda to accept L$. So now, we can build a
deterministic pda M as follows: Push each a onto the stack. When we run out of as, the next character will
either be $ (in the case where k = 0) or b (in the case where k = 1). So we know right away which case were
dealing with. If M sees a b, it goes to a state where it pops one b for each a and accepts if it comes out even. If it
sees $, it goes to a state where it clears the stack and accepts.
c. __E__ {anbncn : n 0}
We proved that this is recursive by showing a grammar for it in Lecture Notes 24. We used the pumping theorem
to prove that it isnt context free in Lecture Notes 19.
Homework 22
Review
d. __C__ {anbncm : n 0, m 0}
This one is context free. We need to compare the as to the bs, but the cs are independent. So a grammar to
generate this one is:
SAC
AaAb
A
CcC
C
Its deterministic because we can build a pda that always knows what to do: push as, pop an a for each b, then
simply scan the cs.
e. __C__ {anbn : n 0} a*
This one is equivalent to b, since a* = anb0n.
f. __E__ {anbm : n is prime and m is even}
This one is recursive because we can write an algorithm to determine whether a number is prime and another one
to determine whether a number is even. The proof that it is essentially the same as the one we did in class that an:
n is prime is not context free.
g. __C__ {anbmcm+n : n 0, m 0}
This one is context free. A grammar for it is:
SaSc
SbSc
S
Its deterministic because we can build a deterministic pda M for it: M pushes each a onto its stack. It also
pushes an a for each b. Then, when it starts seeing cs, it pops one a for each c. If it runs out of as and cs at the
same time, it accepts.
h. __E__ {anbmcmn : n 0, m 0}
This one is similar to g, but because the number of cs is equal to the product of n and m, rather than the sum,
there is no way to know how many cs to generate until we know both how many as there are and how many bs.
Clearly we can write an algorithm to do it, so its recursive. To prove this , we need to use the pumping theorem.
Let w = aMbMcMM. Call the as, region 1, the bs region 2, and the cs region 3. Clearly neither v nor y can span
regions since, if they did, wed get a string with letters out of order. So we need only consider the following
possibilities:
(1, 1) The number of cs will no longer be the product of n and m.
(1, 3) Ruled out by |vxy| M.
i. __B__ {anbm : n 0, m 0}
This one is regular. It is defined by the regular expression a*b*. It isnt finite, which we know from the presence
of Kleene star in the regular expression.
j. __C__ {xy : x a*, y b*, |x| = |y|}
This one is equivalent to anbn, which weve already shown is context free and not regular. W showed a
deterministic pda to accept it in Lecture Notes 14.
k. __B__ {xy : x a*, y a*, |x| = |y|}
This one is {w = a*: |w| is even}. Weve shown a simple two state FSM for this one.
l. __B__ {x : x {a, b, c}*, and x has 5 or more a's}
This one also has a simple FSM F that accepts it. F has six states. It simply counts as, up to five. If it ever gets
to 5, it accepts.
Homework 22
Review
m.__F__ {"M" : M accepts at least 1 string}

This one isnt recursive. We know from Rices Theorem that it cant be, since another way to say this is
{"M" : L(M) contains at least 1 string}
We can also show that this one isnt recursive by reduction, which is done in the Supplementary Materials.
n.__A__ {"M" : M is a Turing machine that halts on input and |"M"| 1000}
This one is finite because of the limit on the length of the strings that can be used to describe M. So its finite
(and thus regular) completely independently of the requirement that M must halt on . You may wonder whether
we can actually build a finite state machine F to accept this language. What we know for sure is that F exists. It
must for any finite language. Whether we can build it or not is a separate question. The undecidability of the
halting problem tells us that we cant build an algorithm to determine whether an arbitrary TM M halts on . But
that doesnt mean that we cant look at most Turing Machines and tell. So, here, it is likely that we could write
out all the TMs of length less than 1000 and figure out which ones accept . We could then build a deciding
FSM F. But even if we cant, that doesnt mean that no such FSM exists. It just means that we dont know what
it is. This is no different from the problem of building an FSM to accept all strings of the form mm/dd/yy, such
that mm/dd/yy is your birthday. A simple machine F to do this exists. You know how to write it. I dont
because I dont know when your birthday is. But that fact that I dont know how to build F says nothing about its
existence.
o.__E__ {"M" : M is a Turing machine with 50 states}
This one looks somewhat similar to n. But its different in a key way. This set isn't finite because there is no
limit on the number of tape symbols that M can use. So we can't do the same trick we can do in n, where we
could simply list all the machines that met the length restriction. With even a single state, I can build a TM
whose description is arbitrarily long. I simply tell it what to do in state one if its reading character 1. Then what
to do if its reading character 2. Then character 3, and so forth. Theres no limit to the number of characters, so
theres no limit to the length of the string I must write to consider all of them. Given that the language is not
finite, we need a TM to decide it. Why? What we need to do is to check to make sure that the string is a
syntactically valid encoding of a Turing Machine. Recall the syntax of an encoding. When we see the first a???
symbol that encodes a tape symbol, we know how many digits it has. All the others must have the same number
of digits. So we have to remember that number. Since there's no limit to it, we can't remember it in a finite
number of states. Since we need to keep referring to it, we can't remember it on a stack. So we need a TM. But
the TM is a straightforward program that will always halt. Thus the language is recursive.
p.__G__ {"M" : M is a Turing machine such that L(M) = a*}
This one isn't recursive. Again, we know that from Rices Theorem. And we can prove it by reduction, which
we did in the supplementary materials for the more general case of any alphabet . But this language is even
harder than many we have considered, such as H. It isn't even recursively enumerable. Why? Informally, the
TM languages that are re are the ones where we can discover positive instances by simulation (like, H, where we
ask whether M halts on a particular w?). But how can we try all strings in a*? Proving this formally is beyond
the scope of this class.
q. __A__ {x : x {A, B, C, D, E, F, G}, and x is the answer you write to this question}
This one is finite. In fact, it is a language of cardinality 1. Thus its regular and there exists an FSM F that
accepts it. You may feel that theres some sort of circularity here. There really isnt, but even if there were, we
can use the same argument here that we used in n. Even if we didnt know how to build F, we still know that it
exists.
Homework 22
Review
III. Supplementary
Materials

Analyzing Problems as Opposed to Algorithms
In CS336 you learned to analyze algorithms. This semester, we're going to analyze problems.
We're going to see that there is a hierarchy of problems: easy, hard, impossible.
For each of the first two, we'll see that for any problem, there are infinitely many programs to
solve the problem.
By the way, this is trivial. Take one program and add any number of junk steps. We'll
formalize this later once we have some formalisms to work with.
Some may be better than others. But in some cases, we can show some sort of boundary on how
good an algorithm we can attempt to find for a particular problem.
Let's Look at Some Problems

Let's look at a collection of problems, all which could arise in considering one piece of C++
code. [slide - Let's Look at Some Problems]
As we move from problem 1 to problem 5, things get harder in two key ways. The first is that it
seems we'll need more complicated, harder to write and design programs. In other words, it's
going to take more time to write the programs. The second is that the programs are going to take
a lot longer to run. As we get to problems 4 and 5, it's not even clear there is a program that will
do the job. In fact, in general for problem 4 there isn't. For problem 5, in general we can't even
get a formal statement of the problem, much less an algorithmic solution.
Languages
Characterizing Problems as Language Recognition Tasks
In order to create a formal theory of problems (as opposed to algorithms), we need a single,
relatively straightforward framework that we can use to describe any kind of possibly computable
function. The one we'll use is language recognition.
What is a Language?
A language is a set of strings over an alphabet, which can be any finite collection of symbols.
[Slide - Languages]
Defining a Problem as a Language Recognition Task
We can define any problem as a language recognition task. In other words, we can output just a
boolean, True or False. Some problems seem naturally to be described as recognition tasks. For
example, accept grammatical English sentences and reject bad ones. (Although the truth is that
English is so squishy it's nearly impossible to formalize this. So let's pick another example -accept the syntactically valid C programs and reject the others.)
Problems that you think of more naturally as functions can also be described this way. We define
the set of input strings to consist of strings that are formed by concatenating an input to an
output. Then we only accept strings that have the correct output concatenated to each input.
[Slide - Encoding Output]
Branching Out -- Allowing for Actual Output
Although it is simpler to characterize problems simply as recognition problems and it is possible
to reformulate functional problems as recognition problems, we will see that we can augment the
formalisms we'll develop to allow for output as well.
Defining Languages Using Grammars
Now what we need is a general mechanism for defining languages. Of course, if we have a finite
language, we can just enumerate all the strings in it. But most interesting languages are infinite.
What we need is a finite mechanism for specifying infinite languages. Grammars can do this.
The standard way to write a grammar is as a production system, composed of rules with a left
hand side and a right hand side. Each side contains a sequence of symbols. Some of these
symbols are terminal symbols, i.e., symbols in the language we're defining. Others are drawn
from a finite set of nonterminal symbols, which are internal symbols that we use just to help us
define the language. Of these nonterminal symbols, one is special -- we'll call it the start symbol.
[Slide - Grammars 1]
If there is a grammar that defines a language, then there is an infinite number of such grammars.
Some may be better, from various points of view than others. Consider the grammar for odd
integers. What different grammars could we write? One thing we could do would be to introduce
the idea of odd and even digits. [Slide - Grammars 2]
Sometimes we use single characters, disjoint from the characters of the target language, in our
rules. But sometimes we need more symbols. Then we often use < and > to mark multiple
character nonterminal symbols. [Slide - Grammars 3]
Notice that we've also introduced a notation for OR so that we don't have to write as many
separate rules. By the way, there are lots of ways of writing a grammar of arithmetic expressions.
This one is simple but it's not very good. It doesn't help us at all to determine the precedence of
operators. Later we'll see other grammars that do that.
Grammars as Generators and as Acceptors
So far, we've defined problems as language recognition tasks. But when you look at the
grammars we've considered, you see that there's a sense in which they seem more naturally to be
generators than recognizers. If you start with S, you can generate all the strings in the language
defined by the grammar. We'll see later that we'll use the idea of a grammar as a generator (or an
enumerator) as one way to define some interesting classes of languages.
But you can also use grammars as acceptors, as we've suggested. There are two ways to do that.
One is top-down. By that we mean that you start with S, and apply rules. [work this out for a
simple expression for the Language of Simple Arithmetic Expressions] At some point, you'll
generate a string without any nonterminals (i.e., a string in the language). Check and see if it's
the one you want. If so accept. If not, try again. If you do this systematically, then if the string is
in the language, you'll eventually generate it. If it isn't, you may or may not know when you
should give up. More on that later.
The other approach is bottom up. In this approach, we simply apply the rules sort of backwards,
i.e., we run them from right to left, matching the string to the right hand sides of the rules and
continuing until we generate S and nothing else. [work this out for a simple expression for the
Language of Simple Arithmetic Expressions] Again, there are lots of possibilities to consider
and there's no guarantee that you'll know when to stop if the string isn't in the language.
Actually, for this simple grammar there is, but we can't assure that for all kinds of grammars.
Remember that our whole goal in this exercise is to describe classes of problems, characterize
them as easy or hard, and define computational mechanisms for solving them. Since we've
decided to characterize problems as languages to be recognized, what we need to do is to create a
language hierarchy, in which we start with very simple languages and move toward more
complex ones.
Regular Languages
Regular languages are very simple languages that can be defined by a very restricted kind of
grammar. In these grammars, the left side of every rule is a single nonterminal and the right side
is a single terminal optionally followed by a single nonterminal. [slide - Regular Grammars] If
you look at what's going on with these simple grammars, you can see that as you apply rules,
starting with S, you generate a terminal symbol and (optionally) have a new nonterminal to work
with. But you can never end up with multiple nonterminals at once. (Recall our first grammar for
Odd Integers [slide - Grammars 1]).
Of course, we also had another grammar for that same language that didn't satisfy this restriction.
But that's okay. If it is possible to define the language using the restricted formalism, then it falls
into the restricted class. The fact that there are other, less restricted ways to define it doesn't
matter.
It turns out that there is an equivalent, often useful, way to describe this same class of languages,
using regular expressions. [Slide Regular Expressions and Languages] Regular expressions
don't look like grammars, in the sense that there are no production rules, but they can be used to
define exactly the same set of languages that the restricted class of regular grammars can define.
Here's a regular expression for the language that consists of odd integers, and one for the
language of identifiers. We can try to write a regular expression for the language of matched
parenthesis, but we won't succeed.
Intuitively, regular languages are ones that can be defined without keeping track of more than a
finite number of things at once. So, looking back at some of our example languages [slide
Languages], the first is regular and none of the others is.
Context Free Languages
To get more power in how we define languages, we need to return to the more general production
rule structure.
Suppose we allow rules where the left hand side is composed of a single symbol and the right
hand side can be anything. We then have the class of context-free grammars. We define the
class of context-free languages to include any language that can be generated by a context-free
grammar.
The context-free grammar formalism allows us to define many useful languages, including the
languages of matched parentheses and of equal numbers of parentheses but in any order [slide Context-Free Grammars]. We can also describe the language of simple arithmetic expressions
[slide - Grammars 3].
Although this system is a lot more powerful (and useful) than regular languages are, it is not
adequate for everything. We'll see some quite simple artificial language it won't work for in a
minute. But it's also inadequate for things like ordinary English. [slide - English Isn't ContextFree].
Recursively Enumerable Languages
Now suppose we remove all restrictions from the form of our grammars. Any combination of
symbols can appear on the left hand side and any combination of symbols can appear on the
right. The only real restriction is that there can be only a finite number of rules. For example, we
can write a grammar for the language that contains strings of the form a n b n c n . [slide Unrestricted Grammars]
Once we remove all restrictions, we clearly have the largest set of languages that can be
generated by any finite grammar. We'll call the languages that can be generated in this way the
class of recursively enumerable languages. This means that, for any recursively enumerable
language, it is possible, using the associated grammar, to generate all the strings in the language.
Of course, it may take an infinite amount of time if the language contains an infinite number of
strings. But any given string, if it is enumerated at all, will be enumerated in a finite amount of
time. So I guess we could sit and wait. Unfortunately, of course, we don't know how long to
wait, which is a problem if we're trying to decide whether a string is in the language by
generating all the strings and seeing if the one we care about shows up.
Recursive Languages
There is one remaining set of languages that it is useful to consider. What about the recursively
enumerable languages where we could guarantee that, after a finite amount of time, either a given
string would be generated or we would know that it isn't going to be. For example, if we could
generate all the strings of length 1, then all the strings of length 2, and so forth, we'd either
generate the string we want or we'd just wait until we'd gone past the length we cared about and
then report failure. From a practical point of view, this class is very useful since we like to deal
with solutions to problems that are guaranteed to halt. We'll call this class of languages the
recursive languages. This means that we can not only generate the strings in the language, we
can actually, via some algorithm, decide whether a string is in the language and halt, with an
answer, either way.
Clearly the class of recursive languages is a subset of the class of recursively enumerable ones.
But, unfortunately, this time we're not going to be able to define our new class by placing
syntactic restrictions on the form of the grammars we use. There are some useful languages,
such, as a n b n c n , that are recursive. There are some others, unfortunately, that are not.
The Whole Picture
[Slide - The Language Hierarchy]
Computational Devices
Formal Models of Computational Devices
If we want to make formal statements about the kinds of computing power required to solve
various kinds of problems, then we need simple, precise models of computation.
We're looking for models that make it easy to talk about what can be computed -- we're not
worrying about efficiency at this point.
When we described languages and grammars, we saw that we could introduce several different
structures, each with different computational power. We can do the same thing with machines.
Let's start with really simple devices and see what they can do. When we find limitations, we can
expand their power.
The only memory consists of the ability to be in one of a finite number of states. The machine
operates by reading an input symbol and moving to a new state that is determined solely by the
state it is in and the input that it reads. There is a unique start state and one or more final states.
If the input is exhausted and the machine is in a final state, then it accepts the input. Otherwise it
rejects it.
Example: An FSM to accept odd integers. [Slide - Finite State Machines 1]

Example: An FSM to accept valid identifiers. [Slide - Finite State Machines 2]
Example: How about an FSM to accept strings with balanced parentheses?
Notice one nice feature of every finite state machine -- it will always halt and it will always
provide an answer, one way or another. As we'll see later, not all computational systems offer
these guarantees.
But we've got this at a price. There is only a finite amount of memory. So, for example, we can't
count anything. We need a stronger device.
Push Down Automata
Deterministic PDAs
Add a single stack to an FSM. Now the action of the machine is a function of its state, the input
it reads, and the values at the top of the stack.
Example: A PDA to accept strings with balanced parentheses. [Slide - Push Down Automata]
Notice that this really simple machine only has one state. It's not using the states to remember
anything. All its memory is in the stack.
Example: A PDA to accept strings of the form w#wR, where w {a,b}* [slide - Pushdown
Automaton 2].
Example: How about a PDA to accept strings with some number of a's, followed by the same
number of b's, followed by the same number of c's? [slide - PDA 3] It turns out that this isn't
possible. The problem is that we could count the a's on the stack, then pop for b's. But how
could we tell if there is the right number of c's. We'll see in a bit that we shouldn't be too
surprised about this result. We can create PDA's to accept precisely those languages that we can
generate with CFGs. And remember that we had to use an unrestricted grammar to generate this
language.
Nondeterministic PDAs
Example: How about a PDA to accept strings of the form w wR? We can do this one if we
expand our notion of a PDA to allow it to be nondeterministic. The problem is that we don't
know when to imagine that the reversal starts. What we need to do is to guess. In particular, we
need to try it at every point. We can do this by adding an epsilon transition from the start state
(in which we're pushing w) to the final state in which we're popping as we read wR.[slide - A
Nondeterministic PDA] Adding this kind of nondeterminism actually adds power to the PDA
notion. And actually, it is the class of nondeterministic PDA's that is equivalent to the class of
context-free languages. No surprise, since we were able to write a context free grammar for this
language
By the way, it also makes sense to talk about nondeterministic finite state machines. But it turns
out that adding nondeterminism to finite state machines doesn't increase the class of things they
can compute. It just makes it easier to describe some machines. Intuitively, the reason that
nondeterminism doesn't buy you anything with finite state machines is that we can simulate a
nondeterministic machine with a deterministic machine. We just make states that represent sets
of states in the nondeterministic machine. So in essence, we follow all paths. If one of them
accepts, we accept.
Then why can't we do that with PDA's? For finite state machines, there must be a finite number
of states. DAH. So there is a finite number of subsets of states and we can just make them the
states of our new machine. Clunky but finite. Once we add the stack, however, there is no
longer a finite number of states of the total machine. So there is not a finite number of subsets of
states. So we can't simulate being in several states at once just using states. And we only have
one stack. Which branch would get it? That's why adding nondeterminism actually adds power
for PDAs.
Turing Machines
Clearly there are still some things we cannot do with PDAs. All we have is a single stack. We
can count one thing. If we need to count more than one thing (such as a's and b's in the case of
languages defined by a n b n c n ), we're in trouble.
So we need to define a more powerful computing device. The formalism we'll use is called the
Turing Machine, after its inventor, Alan Turing. There are many different (and equivalent) ways
to write descriptions of Turing Machines, but the basic idea is the same for all of them [slide Turing Machines]. In this new formalism, we allow our machines to write onto the input tape.
They can write on top of the input. They can also write past the input. This makes it easier to
define computation that actually outputs something besides yes or no if we want to. But, most
importantly, because we view the tape as being of infinite length, all limitations of finiteness or
limited storage have been removed, even though we continue to retain the core idea of a finite
number of states in the controller itself.
Notice, though, that Turing Machines are not guaranteed to halt. Our example one always does.
But we could certainly build one that scans right until it finds a blank (writing nothing) and then
scans left until it finds the start symbol and then scans right again and so forth. That's a legal (if
stupid) Turing Machine. Unfortunately, (see below) it's not always possible to tell, given a
Turning Machine, whether it is guaranteed to halt. This is the biggest difference between Turing
Machines and the FSMs and PDAs, both of which will always halt.
Extensions to Turing Machines
You may be thinking, wow, this Turing Machine idea sure is restrictive. For example, suppose
we want to accept all strings in the simple language {w # w R }. We saw that this was easy to do
in one pass with a pushdown automaton. But to do this with the sort of Turing Machine we've
got so far would be really clunky. [work this out on a slide] We'd have to start at the left of the
string, mark a character, move all the way to the right to find the corresponding character, mark
it, scan back left, do it again, and so forth. We've just transformed a linear process into an n 2
one.
But suppose we had a Turing Machine with 2 tapes. The first thing we'll do is to copy the input
onto the second tape. Now start the read head of the first tape at the left end of the input and the
read head of the second tape at the right end. At each step in the operation of the machine, we
check to make sure that the characters being read on the two tapes are the same. And we move
the head on tape 1 right and the head on tape 2 to the left. We run out of input on both machines
at the same time, we accept. [slide -A Two Head Turing Machine]
The big question now is, "Have we created a new notational device, one that makes it easier to
describe how a machine will operate, or have we actually created a new kind of device with more
power than the old one? The answer is the former. We can prove that by showing that we can
simulate a Turing Machine with any finite number of tapes by a machine that computes the same
thing but only has one tape. [slide - Simulating k Heads with One] The key idea here is to use
the one tape but to think of it has having some larger number of tracks. Since there is a finite
tape alphabet, we know that we can encode any finite number of symbols in a finite (but larger)
symbol alphabet. For example, to simulate our two headed machine with a tape alphabet of 3
symbols plus start and blank, we will need 2*2*5*5 or 100 tape symbols. So to do this
simulation, we must do two main things: Encode all the information from the old, multi-tape
machine on the new, single tape machine and redesign the finite state controller so that it
simulates, in several moves, each move of the old machine.
It turns out that any "reasonable" addition you can think of to our idea of a Turing Machine is
implementable with the simple machine we already have. For example, any nondeterministic
Turing Machine can be simulated by a deterministic one. This is really significant. In this, in
some ways trivial, machine, we have captured the idea of computability.
Okay, so our Turing Machines can do everything any other machine can do. It also goes the
other way. We can propose alternative structures that can do everything our Turing Machines
can do. For example, we can simulate any Turing Machine with a deterministic PDA that has
two stacks rather than one. What this machine will do is read its input tape once, copying onto
the first stack all the nonblank symbols. Then it will pop all those symbols off, one at a time, and
move them to the second stack. Now it can move along its simulated tape by transfering symbols
from one stack to the other. [slide - Simulating a Turing Machine with Two Stacks]
So now, having shown that we can simulate anything on a simple Turing Machine, it should
come as no surprise that we can design a Turing Machine that takes as its input the definition of
another Turing Machine, along with an input for that machine. What our machine does is to
simulate the behavior of the machine it is given, on the given input.
Remember that to simulate a k-tape machine by a 1 tape machine we had first to state how to
encode the multiple tapes. Then we had to state how the machine would operate on the
encoding. We have to do the same thing here. First we need to decide how to encode the states
and the tape symbols of the input machine, which we'll call M. There's no upper bound on how
many states or tape symbols there will be. So we can't encode them with single symbols. Instead
we'll encode states as strings that start with a "q" and then have a binary encoding of the state
number (with enough leading zeros so all such encodings take the same number of digits). We'll
encode tape symbols as an "a" followed by a binary encoding of the count of the symbol. And
we'll encode "move left" as 10, "move right" as 01, and stay put as 00. We'll use # as a delimiter
between transitions. [slide - Encoding States, Symbols, and Transitions]
Next, we need a way to encode the simulation of the operation of M. We'll use a three tape
machine as our Universal Turing Machine. (Remember, we can always implement it on a one
tape machine, but this is a lot easier to describe.) We'll use one tape to encode the tape of M, the
second tape contains the encoding of M, and the third tape encodes the current state of M during
the simulation. [slide - The Universal Turing Machine]
A Hierarchy of Computational Devices
These various machines that we have just defined, fall into an inclusion hierarchy, in the sense
that the simpler machines can always be simulate by the more powerful ones. [Slide - A
Machine Hierarchy]
The Equivalence of the Language Hierarchy and the Computational

Hierarchy
Okay, this probably comes as no surprise. The machine hierarchy we've just examined exactly
mirrors the language hierarchy. [Slide - Languages and Machines]
Actually, this is an amazing result. It seems to suggest that there's something quite natural about
these categories.
Church's Thesis
If we really want to talk about naturalness, can we say anything about whether we've captured
what it means to be computable? Church's Thesis (also sometimes called the Church-Turing
Thesis) asserts that the precise concept of the Turing Machine that halts on all inputs corresponds
to the intuitive notion of an algorithm. Think about it. Clearly a Turing Machine that halts
defines an algorithm. But what about the other way around? Could there be something that is
computable by some kind of algorithm that is not computable by a Turing Machine that halts?
From what we've seen so far, it may seem unlikely, since every extension we can propose to the
Turing Machine model turns out possibly to make things more convenient, but it never extends
the formal power. It turns out that people have proposed various other formalisms over the last
50 years or so, and they also turn out to be no more powerful than the Turing Machine. Of
course, something could turn up, but it seems unlikely.
Techniques for Showing that a Problem (or Language) Is Not in a

Particular Circle in the Hierarchy
Counting
a n b n is not regular.
Closure Properties
L = { a n b m c p : m n or m p } is not deterministic context-free. [slide - Using Closure
Properties] Notice that L' contains all strings that violate at least one of the requirements for L.
So they may be strings that aren't composed of a string of a's, followed by a string of b's,
followed by a string of c's. Or they may have that property but they violate the rule that m, n, and
p cannot all be the same. In other words, they are all the same. So if we intersect L' with the
regular expression a*b*c*, we throw away everything that isn't a string of a's then b's then c's,
and we're left with strings of n a's, followed, by n b's, followed by n c's.
Diagonalization
Remember the proof that the power set of the integers isn't countable. If it were, there would be
a way of enumerating the sets, thus setting them in one to one correspondence with the integers.
But suppose there is such a way [slide - Diagonalization]. Then we could represent it in a table
where element (i, j) is 1 precisely in case the number j is present in the set i. But now construct a
new set, represented as a new row of the table. In this new row, element i will be 1 if element
(i,i) of the original table was 0, and vice versa. This row represents a new set that couldn't have
been in the previous enumeration. Thus we get a contradiction and the power set of the integers
must not be countable.
We can use this technique for perhaps the most important result in the theory of computing.

There are recursively enumerable languages that are not recursive. In other words, there are sets
that can be enumerated, but there is no decision procedure for them. Any program that attempts
to decide whether a string is in the language may not halt.
One of the most interesting such sets is the following. Consider sets of ordered pairs where the
first element is a description of a Turing Machine. The second element is an input to the
machine. We want to include only those ordered pairs where the machine halts on the input.
This set is not recursively enumerable. In other words, there's no way to write an algorithm that,
given a machine and an input, determines whether or not the machine halts on the input. [slide The Unsolvability of the Halting Problem]
Suppose there were such a machine. Let's call it HALTS. HALTS(M,x) returns true if Turing
machine M halts on input x. Otherwise it returns false. Now we write a Turing Machine
program that implements the TROUBLE algorithm. Now what happens if we invoke
HALTS(TROUBLE,TROUBLE)? If HALTS says true, namely that TROUBLE will halt on
10
itself, then TROUBLE loops (i.e., it doesn't halt, thus contradicting our assumption that HALTS
could do the job). But if HALTS says FALSE, namely that TROUBLE will not halt on itself,
then TROUBLE promptly halts, thus again proving our supposed oracle HALTS wrong. Thus
HALTS cannot exist.
We've used a sort of stripped down version of diagonalization here [slide - Viewing the Halting
Problem as Diagonalization] in which we don't care about the whole row of the item that
creates the contradiction. We're only invoking HALTS with two identical inputs. It's just the
single element that we care about and that causes the problem.
Let's Revisit Some Problems

Let's look again at the collection of problems that we started this whole process with. [slide Let's Revisit Some Problems]
Problem 1 can be solved with a finite state machine.
Problem 2 can be solved with a PDA.
Problem 3 can be solved with a Turing Machine.
Problem 4 can be semi solved with a Turning Machine, but it isn't guaranteed to halt.
Problem 5 can't even be stated.
So What's Left?
11

1 Sets
1.1 What is a Set?
A set is simply a collection of objects. The objects (which we call the elements or members of the set) can be anything:
numbers, people, fruits, whatever. For example, all of the following are sets:
A = {13, 11, 8, 23}
B = {8, 23, 11, 13}
C = {8, 8, 23, 23, 11, 11, 13, 13}
D = {apple, pear, banana, grape}
E = {January, February, March, April, May, June, July, August, September, October, November, December}
F = {x : x E and x has 31 days}
G = {January, March, May, July, August, October, December}
N = the nonnegative integers (We will generally call this set N, the natural numbers.)
H = {i : x N and i = 2x}
I = {0, 2, 4, 6, 8, }
J = the even natural numbers
K = the syntactically valid C programs
L = {x : x K and x never gets into an infinite loop}
Z = the integers ( -3, -2, -1, 0, 1, 2, 3, )
In the definitions of F and H, we have used the colon notation. Read it as "such that". We've also used the standard
symbol for "element of". We will also use for "not an element of". So, for example, 17 A is true.
Remember that a set is simply a collection of elements. So if two sets contain precisely the same elements (regardless of
the way we actually defined the sets), then they are identical. Thus F and G are the same set, as are H, I, and J.
An important aside: Zero is an even number. This falls out of any reasonable definition we can give for even numbers.
For example, the one we used to define set H above. Or consider: 2 is even and any number that can be derived by
adding or subtracting 2 from an even number is also even. In order to construct a definition for even numbers that does
not include zero, we'd have to make a special case. That would make for an inelegant definition, which we hate. And, as
we'll see down the road, we'd also have to make corresponding special cases for zero in a wide variety of algorithms.
Since a set is defined only by what elements it contains, it does not matter what order we list the elements in. Thus A and
B are the same set.
Our definition of a set considers only whether or not an element is contained within the set. It does not consider how
many times the element is mentioned. In other words, duplicates don't count. So A, B, and C are all equal.
Whenever we define a set, it would be useful if we could also specify a decision procedure for it. A decision procedure
for a set S is an algorithm that, when presented with an object O, returns True if O S and False otherwise. Consider set
K above (the set of all syntactically valid C programs). We can easily decide whether or not an object is an element of K.
First, of course, it has to be a string. If you bring me an apple, I immediately say no. If it is a string, then I can feed it to a
C compiler and let it tell me whether or not the object is in K. But now consider the set L (C programs that are guaranteed
to halt on all inputs). Again, I can reject apples and anything else that isn't even in K. I can also reject some programs
that clearly do loop forever. And I can accept some C programs, for example ones that don't contain any loops at all. But
what about the general problem. Can I find a way to look at an arbitrary C program and tell whether or not it belongs in L.
It turns out, as we'll see later, that the answer to this is no. We can prove that no program to solve this problem can exist.
But that doesnt mean that the set L doesn't exist. It's a perfectly fine set. There just isn't a decision procedure for it.
The smallest set is the set that contains no elements. It is called the empty set, and is written or {}.
When you are working with sets, it is very important to keep in mind the difference between a set and the elements of a
set. Given a set that contains more than one element, this not usually tricky. It's clear that {1, 2} is distinct from either the
number 1 or the number 2. It sometimes becomes a bit trickier though with singleton sets (sets that contain only a single
element). But it is equally true here. So, for example, {1} is distinct from the number 1. As another example, consider
{}. This is a set that contains one element. That element is in turn a set that contains no elements (i.e., the empty set).
1.2 Relating Sets to Each Other

We say that A is a subset of B (which we write as A B) if every element of A is also an element of B. The symbol we
use for subset () looks somewhat like . This is no accident. If A B, then there is a sense in which the set A is "less
than or equal to" the set B, since all the elements of A must be in B, but there may be elements of B that are not in A.
Given this definition, notice that every set is a subset of itself. This fact turns out to offer us a useful way to prove that
two sets A and B are equal: First prove that A is a subset of B. Then prove that B is a subset of A. We'll have more to say
about this later in Section 6.2.
We say that A is proper subset of B (written A B) if A B and A B. The following Venn diagram illustrates the
proper subset relationship between A and B:
Notice that the empty set is a subset of every set (since, trivially, every element of , all none of them, is also an element
of every other set). And the empty set is a proper subset of every set other than itself.
It is useful to define some basic operations that can be performed on sets:
The union of two sets A and B (written A B) contains all elements that are contained in A or B (or both). We can
easily visualize union using a Venn diagram. The union of sets A and B is the entire hatched area:
The intersection of two sets A and B (written A B) contains all elements that are contained in both A and B. In the
Venn diagram shown above, the intersection of A and B is the double hatched area in the middle.
The difference of two sets A and B (written A - B) contains all elements that are contained in A but not in B. In both of
the following Venn diagrams, the hatched region represents A - B.
The complement of a set A with respect to a specific domain D (written as A or A) contains all elements of D that are
not contained in A (i.e, A = D - A). For example, if D is the set of residents of Austin and A is the set of Austin residents
who like barbeque, then A is the set of Austin residents who don't like barbeque. The complement of A is shown as the
hatched region of the following Venn diagram:
Two sets are disjoint if they have no elements in common (i.e., their intersection is empty). In the following Venn
diagram, A and B are disjoint:
So far, we've talked about operations on pairs of sets. But just as we can extend binary addition and sum up a whole set of
numbers, we can extend the binary operations on sets and perform then on sets of sets. Recall that for summation, we
have the notation
Similarly, we'll introduce
7 Ai
and
1 Ai
to indicate the union of a set of sets and the intersection of a set of sets, respectively.
Now consider a set A. For example, let A = {1, 2, 3}. Next, let's enumerate the set of all subsets of A:
{, {1}, {2}, {3}, {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}}
We call this set the power set of A, and we write it 2A. The power set of A is interesting because, if we're working with
the elements of A, we may well care about all the ways in which we can combine those elements.
Now for one final property of sets. Again consider the set A above. But this time, rather than looking for all possible
subsets, let's just look for a single way to carve A up into subsets such that each element of A is in precisely one subset.
For example, we might choose any of the following sets of subsets:
{{1}, {2, 3}}
or
{{1, 3}, 2}
or
{{1, 2, 3}}
We call any such set of subsets a partition of A. Partitions are very useful. For example, suppose we have a set S of
students in a school. We need for every student to be assigned to precisely one lunch period. Thus we must construct a
partition of S: a set of subsets, one for each lunch period, such that each student is in precisely one subset. More formally,
we say that is a partition of a set A if and only if (a) no element of is empty; (b) all members of are disjoint
(alternatively, each element of A is in only one element of ); and (c) 7 = A (alternatively, each element of A is in
some element of and no element not in A is in any element of ).
This notion of partitioning a set is fundamental to programming. Every time you analyze the set of possible inputs to your
program and consider the various cases that must be dealt with, you're forming a partition of the set of inputs: each input
must fall through precisely one path in your program. So it should come as no surprise that, as we build formal models of
computational devices, we'll rely heavily on the idea of a partition on a set of inputs as an analytical technique.
2 Relations and Functions

In the last section, we introduced some simple relations that can hold between sets (subset and proper subset) and we
defined some operations (functions) on sets (union, intersection, difference, and complement). But we haven't yet defined
formally what we mean by a relation or a function. Let's do that now. (By the way, the reason we introduced relations and
functions on sets in the last section is that we're going to use sets as the basis for our formal definitions of relations and
functions and we will need the simple operations we just described as part of our definitions.)
2.1 Relations
An ordered pair is a sequence of two objects. Given any two objects, x and y, there are two ordered pairs that can be
formed. We write them as (x, y) and (y, x). As the name implies, in an ordered pair (as opposed to in a set), order matters
(unless x and y happen to be equal).
The Cartesian product of two sets A and B (written A B) is the set of all ordered pairs (a, b) such that a A and b B.
For example, let A be a set of people {Dave, Sue, Billy} and let B be a set of desserts {cake, pie, ice cream}. Then
A B = { (Dave, cake), (Dave, pie), (Dave, ice cream),
(Sue, cake), (Sue, pie), (Sue, ice cream),
(Billy, cake), (Billy, pie), (Billy, ice cream)}
As you can see from this example, the Cartesian product of two sets contains elements that represent all the ways of
pairing someone from the first set with someone from the second. Note that A B is not the same as B A. In our
example,
B A = { (cake, Dave), (pie, Dave), (ice cream, Dave),
(cake, Sue), (pie, Sue), (ice cream, Sue),
(cake, Billy), (pie, Billy), (ice cream, Billy)}
We'll have more to say about the cardinality (size) of sets later, but for now, let's make one simple observation about the
cardinality of a Cartesian product. If A and B are finite and if there are p elements in A and q elements in B, then there
are p*q elements in AB (and in BA).
We're going to use Cartesian product a lot. It's our basic tool for constructing complex objects out of simpler ones. For
example, we 're going to define the class of Finite State Machines as the Cartesian product of five sets. Each individual
finite state machine then will be a five tuple (K, , , s, F) drawn from that Cartesian product. The sets will be:
1. The set of all possible sets of states: {{q1}, {q1, q2}, {q1, q2, q3}, }. We must draw K from this set.
2. The set of all possible input alphabets: {{a}, {a, b, c}, {, , }, {1, 2, 3, 4}, {1, w, h, j, k}, {q, a, f}, {a, , 3, j,
f}}. We must draw from this set.
3. The set of all possible transition functions, which tell us how to move from one state to the next. We must draw
from this set.
4. The set of all possible start states. We must draw s from this set.
5. The set of all possible sets of final states. (If we land in one of these when we've finished processing an input string,
then we accept the string, otherwise we reject.) We must draw F from this set.
Let's return now to the simpler problem of choosing dessert. Suppose we want to define a relation that tells us, for each
person, what desserts he or she likes. We might write the Dessert relation, for example as
{(Dave, cake), (Dave, ice cream), (Sue, pie), (Sue, ice cream)}
In other words, Dave likes cake and ice cream, Sue likes pie and ice cream, and Billy hates desserts.
We can now define formally what a relation is. A binary relation over two sets A and B is a subset of A B. Our dessert
relation clearly satisfies this definition. So do lots of other relations, including common ones defined on the integers. For
example, Less than (written <) is a binary relation on the integers. It contains an infinite number of elements drawn from
the Cartesian product of the set of integers with itself. It includes, for example:
{(1,2), (2,3), (3, 4), }
Notice several important properties of relations as we have defined them. First, a relation may be equal to the empty set.
For example, if Dave, Sue, and Billy all hate dessert, then the dessert relation would be {} or .
Second, there are no constraints on how many times a particular element of A or B may occur in the relation. In the
dessert example, Dave occurs twice, Sue occurs twice, Billy doesn't occur at all, cake occurs once, pie occurs once, and
ice cream occurs twice.
If we have two or more binary relations, we may be able combine them via an operation we'll call composition. For
example, if we knew the number of fat grams in a serving of each kind of dessert, we could ask for the number of fat
grams in a particular person's dessert choices. To compute this, we first use the Dessert relation to find all the desserts
each person likes. Next we get the bad news from the FatGrams relation, which probably looks something like this:
{(cake, 25), (pie, 15), (ice cream, 20)
Finally, we see that the composed relation that relates people to fat grams is {(Dave, 25), (Dave, 20), (Sue, 15), (Sue,
20)}. Of course, this only worked because when we applied the first relation, we got back desserts, and our second
relation has desserts as its first component. We couldn't have composed Dessert with Less than, for example.
Formally, we say that the composition of two relations R1 A B and R2 B C, written R2 R1 is {(a,c) : (a, b)
R1 and (b, c) R2}. Note that in this definition, we've said that to compute R2 R1, we first apply R1, then R2. In other
words we go right to left. Some definitions go the other way. Obviously we can define it either way, but it's important to
check carefully what definition people are using and to be consistent in what you do. Using this notation, we'd represent
the people to fat grams composition described above as FatGrams Dessert.
Now let's generalize a bit. An ordered pair is a sequence (where order counts) of two elements. We could also define an
ordered triple as a sequence of three elements, an ordered quadruple as a sequence of four elements, and so forth. More
generally, if n is any positive integer, then an ordered n-tuple is a sequence of n elements. For example, (Ann, Joe, Mark)
is a 3-tuple.
We defined binary relation using our definition of an ordered pair. Now that we've extended our definition of an ordered
pair to an ordered n-tuple, we can extend our notion of a relation to allow for an arbitrary number of elements to be
related. We define an n-ary relation over sets A1, A2, An as a subset of A1 A2 An. The n sets may be
different, or they may be the same. For example, let A be a set of people:
A = {Dave, Sue, Billy, Ann, Joe, Mark, Cathy, Pete}
Now suppose that Ann and Dave are the parents of Billy, Ann and Joe are the parents of Mark, and Mark and Sue are the
parents of Cathy. Then we could define a 3-ary (or ternary) relation Child-of as the following subset of A A A:
{(Ann, Dave, Billy), (Ann, Joe, Mark), (Mark, Sue, Cathy)}
2.2 Functions
Relations are very general. They allow an object to be related to any number of other objects at the same time (as we did
in the dessert example above). Sometimes, we want a more restricted notion, in which each object is related to a unique
other object. For example, (at least in an ideal world without criminals or incompetent bureaucrats) each American
resident is related to a unique social security number. To capture this idea we need functions. A function from a set A to
a set B is a special kind of a binary relation over A and B in which each element of A occurs precisely once. The dessert
relation we defined earlier is not a function since Dave and Sue each occur twice and Billy doesn't occur at all. We
haven't restricted each person to precisely one dessert. A simple relation that is a function is the successor function Succ
defined on the integers:
Succ(n) = n + 1.
Of course, we cannot write out all the elements of Succ (since there are an infinite number of them), but Succ includes:
{, (-3, -2), (-2, -1), (-1, 0), (0, 1), (1, 2), (2, 3), }
It's useful to define some additional terms to make it easy to talk about functions. We start by writing
f: A B,
which means that f is a function from the set A to the set B. We call A the domain of f and B the codomain or range. We
may also say that f is a function from A to B. If a A, then we write
f(a),
which we read as "f of a" to indicate the element of B to which a is related. We call this element the image of a under f or
the value of f for a. Note that, given our definition of a function, there must be exactly one such element. We'll also call a
the argument of f. For example we have that
Succ(1) = 2, Succ (2) = 3, and so forth.
Thus 2 is the image (or the value) of the argument 1 under Succ.
Succ is a unary function. It maps from a single element (a number) to another number. But there are lots of interesting
functions that map from ordered pairs of elements to a value. We call such functions binary functions. For example,
integer addition is a binary function:
+: (Z Z) Z
Thus + includes elements such as ((2, 3), 5), since 2 + 3 is 5. We could also write
+((2,3)) = 5
We have double parentheses here because we're using the outer set to indicate function application (as we did above
without confusion for Succ) and the inner set to define the ordered pair to which the function is being applied. But this is
confusing. So, generally, when the domain of a function is the Cartesian product of two or more sets, as it is here, we drop
the inner set of parentheses and simply write
+(2,3) = 5.
Alternatively, many common binary functions are written in infix notation rather than the prefix notation that is standard
for all kinds of function. This allows us to write
2+3 = 5
So far, we've had unary functions and binary functions. But just as we could define n-ary relations for arbitrary values of
n, we can define n-ary functions. For any positive integer n, an n-ary function f is a function is defined as
F: (D1 D2 Dn) R
For example, let Z be the set of integers. Then
QuadraticEquation: (Z Z Z) F
is a function whose domain is an ordered triple of integers and whose domain is a set of functions. The definition of
Quadratic Equation is:
QuadraticEquation(a, b, c) (x) = ax2 + bx + c
What we did here is typical of function definition. First we specify the domain and the range of the function. Then we
define how the function is to compute its value (an element of the range) given its arguments (an element of the domain).
QuadraticEquation may seem a bit unusual since its range is a set of functions, but both the domain and the range of a
function can be any set of objects, so sets of functions qualify.
Recall that in the last section we said that we could compose binary relations to derive new relations. Clearly, since
functions are just special kinds of binary relations, if we can compose binary relations we can certainly compose binary
functions. Because a function returns a unique value for each argument, it generally makes a lot more sense to compose
functions than it does relations, and you'll see that although we rarely compose relations that aren't functions, we compose
functions all the time. So, following our definition above for relations, we define the composition of two functions F1
A B and F2 B C, written F2 F1 is {(a,c) : b (a, b) F1 and (b, c) F2}. Notice that the composition of two
functions must necessarily also be a function. We mentioned above that there is sometimes confusion about the order in
which relations (and now functions) should be applied when they are composed. To avoid this problem, let's introduce a
new notation F(G(x)). We use the parentheses here to indicate function application, just as we did above. So this notation
is clear. Apply F to the result of first applying G to x. This notation reads right to left as does our definition of the
notation.
A function is a special kind of a relation (one in which each element of the domain occurs precisely once). There are also
special kinds of functions:
A function f : D R is total if it is defined for every element of D (i.e., every element of D is related to some element of
R). The standard mathematical definition of a function requires totality. The reason we haven't done that here is that, as
we pursue the idea of "computable functions", we'll see that there are total functions whose domains cannot be effectively
defined (for example, the set of C programs that always halt). Thus it is useful to expand the definition of the function's
domain (e.g., to the set of all C programs) and acknowledge that if the function is applied to certain elements of the
domain (e.g., programs that don't halt), its value will be undefined. We call this broader class of functions (which does
include the total functions as a subset) the set of partial functions. For the rest of our discussion in this introductory unit,
we will consider only total functions, but be prepared for the introduction of partial functions later.
A function f : D R is one to one if no element of the range occurs more than once. In other words, no two elements of
the domain map to the same element of the range. Succ is one to one. For example, the only number to which we can
apply Succ and derive 2 is 1. QuadraticEquation is also one to one. But + isn't. For example, both +(2,3) and +(4,1)
equal 5.
A function f : D R is onto if every element of R is the value of some element of D. Another way to think of this is that
a function is onto if all of the elements of the range are "covered" by the function. As we defined it above, Succ is onto.
But let's define a different function Succ' on the natural numbers (rather than the integers). So we define
Succ' : N N.
Succ' is not onto because there is no natural number i such that Succ'(i) = 0.
The easiest way to envision the differences between an arbitrary relation, a function, a one to one function and an onto
function is to make two columns (the first for the domain and the second for the range) and think about the sort of
matching problems you probably had on tests in elementary school.
Let's consider the following five matching problems and let's look at various ways of relating the elements of column 1
(the domain) to the elements of column 2 (the range):
1
The relationship in example 1 is a relation but it is not a function, since there are three values associated A. The second
example is a function since, for each object in the first column, there is a single value in the second column. But this
function is neither one to one (because x is derived from both A and B) nor onto (because z can't be derived from
anything). The third example is a function that is one to one (because no element of the second column is related to more
than one element of the first column). But it still isn't onto because z has been skipped: nothing in the first column derives
it. The fourth example is a function that is onto (since every element of column two has an arrow coming into it), but it
isn't one to one, since z is derived from both C and D. The fifth and final example is a function that is both one to one and
onto. By the way, see if you can modify either example 3 or example 4 to make them both one to one and onto. You're
not allowed to change the number of elements in either column, just the arrows. You'll notice that you can't do it. In order
for a function to be both one to one and onto, there must be equal numbers of elements in the domain and the range.
The inverse of a binary relation R is simply the set of ordered pairs in R with the elements of each pair reversed.
Formally, if R A B, then R-1 B A = {(b, a): (a, b) R}. If a relation is a way of associating with each element of
A a corresponding element of B, then think of its inverse as a way of associating with elements of B their corresponding
elements in A. Every relation has an inverse. Every function also has an inverse, but that inverse may not also be a
function. For example, look again at example two of the matching problems above. Although it is a function, its inverse
is not. Given the argument x, should we return the value A or B? Now consider example 3. Its inverse is also not a
(total) function, since there is no value to be returned for the argument z. Example four has the same problem example
two does. Now look at example five. Its inverse is a function. Whenever a function is both one to one and onto, its
inverse will also be a function and that function will be both one to one and onto.
Inverses are useful. When a function has an inverse, it means that we can move back and forth between columns one and
two without loss of information. Look again at example five. We can think of ourselves as operating in the {A, B, C}
universe or in the {x, y, z} universe interchangeably since we have a well defined way to move from one to the other. And
if we move from column one to column two and then back, we'll be exactly where we started. Functions with inverses
(alternatively, functions that are both one to one and onto) are called bijections. And they may be used to define
isomorphisms between sets, i.e., formal correspondences between the elements of two sets, often with the additional
requirement that some key structure be preserved. We'll use this idea a lot. For example, there exists an isomorphism
between the set of states of a finite state machine and a particular set of sets of input strings that could be fed to the
machine. In this isomorphism, each state is associated with precisely the set of strings that drive the machine to that state.
3 Binary Relations on a Single Set

Although it makes sense to talk about n-ary relations, for arbitrary values of n (and we have), it turns out that the most
useful relations are often binary ones -- ones where n is two. In fact, we can make a further claim about some of the most
useful relations we'll work with: they involve just a single set. So instead of being subsets of A B, for arbitrary values of
A and B, they are subsets of A A, for some particular set of A of interest. So let's spend some additional time looking at
this restricted class of relations.
3.1 Representing Binary Relations on a Single Set

If we're going to work with some binary relation R, and, in particular, if we are going to compute with it, we need some
way to represent the relation. We have several choices. We could:
1) List the elements of R.
2) Encode R as a computational procedure. There are at least two ways in which a computational procedure can define
R. It may:
a) enumerate the elements of R, or
b) return a boolean value, when given an ordered pair. True means that the pair is in R; False means it is not.
3) Encode R as a directed graph.
4) Encode R as an incidence matrix.
For example, consider the mother-of relation M in a family in which Doreen is the mother of Ann, Ann is the mother of
Catherine, and Catherine is the mother of Allison. To exploit approach 1, we just write
M = {(Doreen, Ann), (Ann, Catherine), (Catherine, Allison)}.
Clearly, this approach only works for finite relations.
The second approach simply requires code appropriate to the particular relation we're dealing with. One appeal of this
approach is that it works for both finite and infinite relations, although, of course, a program that enumerates elements of
an infinite relation will never halt.
Next we consider approach 3. Assuming that we are working with a finite relation R A A, we can build a directed
graph to represent R as follows:
1) Construct a set of nodes, one for each element of A that appears in any element of R.
2) For each ordered pair in R, draw an edge from the first element of the pair to the second.
The following directed graph represents our example relation M defined above:
Doreen
Ann
Catherine
Allison
And, finally, approach 4: Again assuming a finite relation R A A, we can build an incidence matrix to represent R as
follows:
1) Construct a square boolean matrix S whose number of rows and columns equals the number of elements of A that
appear in any element of R.
2) Label one row and one column for each such element of A.
3) For each element (p, q) of R, set S(p, q) to 1(or True). Set all other elements of S to 0 (or False).
The following boolean matrix represents our example relation M defined above:
Doreen
Ann
Catherine
Allison
Doreen
0
0
0
0
Ann
1
0
0
0
Catherine
0
1
0
0
Allison
0
0
1
0
3.2 Properties of Relations

Many useful binary relations have some kind of structure. For example, it might be the case that every element of the
underlying set is related to itself. Or it might happen that if a is related to b, then b must necessarily be related to a.
There's one special kind of relation, called an equivalence relation that is particularly useful. But before we can define it,
we need first to define each of the individual properties that equivalence relations possess.
A relation R A A is reflexive if, for each a A, (a, a) R. In other words a relation R on the set A is reflexive if
every element of A is related to itself. For example, consider the relation Address defined as "lives at same address as".
Address is a relation over a set of people. Clearly every person lives at the same address as him or herself, so Address is
reflexive. So is the Less than or equal to relation on the integers. Every integer is Less than or equal to itself. But the
Less than relation is not reflexive: in fact no number is Less than itself. Both the directed graph and the matrix
representations make it easy to tell if a relation is reflexive. In the graph representation, every node will have an edge
looping back to itself. In the graph representation, there will be ones along the major diagonal:
1
1
1
A relation R A A is symmetric if, whenever (a, b) R, so is (b, a). In other words, if a is related to b, then b is
related to a. The Address relation we described above is symmetric. If Joe lives with Ann, then Ann lives with Joe. The
Less than or equal relation is not symmetric (since, for example, 2 3, but it is not true that 3 2). The graph
representation of a symmetric relation has the property that between any two nodes, either there is an arrow going in both
directions or there is an arrow going in neither direction. So we get graphs with components that look like this:
If we choose the matrix representation, we will end up with a symmetric matrix (i.e., if you flip it on its major diagonal,
you'll get the same matrix back again). In other words, if we have a matrix with 1's wherever there is a number in the
following matrix, then there must also be 1's in all the squares marked with an *:
*
1
*
3
*
A relation R A A is antisymmetric if, whenever (a, b) R and a b, then (b, a) R The Mother-of relation we
described above is antisymmetric: if Ann is the mother of Catherine, then one thing we know for sure is that Catherine is
not also the mother of Ann. Our Address relation is clearly not antisymmetric, since it is symmetric. There are, however,
relations that are neither symmetric nor antisymmetric. For example, the Likes relation on the set of people: If Joe likes
Bob, then it is possible that Bob likes Joe, but it is also possible that he doesn't.
A relation R A A is transitive if, whenever (a, b) R and (b, c) R, (a, c) R. A simple example of a transitive
relation is Less than. Address is another one: if Joe lives with Ann and Ann lives with Mark, then Joe lives with Mark.
Mother-of is not transitive. But if we change it slightly to Ancestor-of, then we get a transitive relation. If Doreen is an
ancestor of Ann and Ann is an ancestor of Catherine, then Doreen is an ancestor of Catherine.
The three properties of reflexivity, symmetry, and transitivity are almost logically independent of each other. We can find
simple, possibly useful relationships with seven of the eight possible combinations of these properties:
None of the properties

Just reflexive
Just symmetric
Just transitive
Reflexive and symmetric
Reflexive and transitive
Symmetric and transitive
All three
Domain
people
people
people
people
people
numbers
Example
Mother-of
Would-recognize-picture-of
Has-ever-been-married-to
Ancestor-of
Hangs-out-with (assuming we can say one hangs out with oneself)
Less than or equal to
numbers
people
Equality
Address
To see why we can't find a good example of a relation that is symmetric and transitive but not reflexive, consider a simple
relation R on {1, 2, 3, 4}. As soon as R contains a single element that relates two unequal objects (e.g., (1, 2)), it must, for
symmetry, contain the matching element (2, 1). So now we have R = {(1, 2), (2, 1)}. To make R transitive, we must add
(1, 1). But that also makes R reflexive.
3.3 Equivalence Relations

Although all the combinations we just described are possible, one combination is of such great importance that we give it
a special name. A relation is an equivalence relation if it is reflexive, symmetric and transitive. Equality (for numbers,
strings, or whatever) is an equivalence relation (what a surprise, given the name). So is our Address (lives at same
address) relation.
Equality is a very special sort of equivalence relation because it relates an object only to itself. It doesn't help us much to
carve up a large set into useful subsets. But in fact, equivalence relations are an incredibly useful way to carve up a set.
Why? Let's look at a set P, with five elements, which we can draw as a set of nodes as follows:
10
1
2
3
4
5
Now let's build an equivalence relation E on P. The first thing we have to do is to relate each node to itself, in order to
make the relation reflexive. So we've now got:
1
2
3
4
5
Now let's add one additional element (1,2). As soon as we do that, we must also add (2,1), since E must be symmetric. So
now we've got:
1
2
3
4
5
Suppose we now add (2,3). We must also add (3,2) to maintain symmetry. In addition, because we have (1, 2) and (2, 3),
we must create (1,3) for transitivity. And then we need (3, 1) to restore symmetry. That gives us
1
2
3
4
5
Notice what happened here. As soon as we related 3 to 2, we were also forced to relate 3 to 1. If we hadn't, we would no
longer have had an equivalence relation. See what happens now if you add (3, 4) to E.
What we've seen in this example is that an equivalence relation R on a set S carves S up into a set of clusters, which we'll
call equivalence classes. This set of equivalence classes has the following key property:
For any s, t S, if s Classi and (s, t) R, then t Classi.
In other words, all elements of S that are related under R are in the same equivalence class. To describe equivalence
classes, we'll use the notation [a] to mean the equivalence class to which a belongs. Or we may just write [description],
where description is some clear property shared by all the members of the class. Notice that in general there may be lots
of different ways to describe the same equivalence class. In our example, for instance, [1], [2], and [3] are different names
for the same equivalence class, which includes the elements 1, 2, and 3. In this example, there are two other equivalence
classes as well: [4] and [5].
It is possible to prove that if R is an equivalence relation on a nonempty set A then the equivalence classes of R constitute
a partition of A. Recall that is a partition of a set A if and only if (a) no element of is empty; (b) all members of
are disjoint; and (c) 7 = A . In other words, if we want to take a set A and carve it up into a set of subsets, an
equivalence relation is a good way to do it.
For example, our Address relation carves up a set of people into subsets of people who live together. Let's look at some
more examples:
11
Let A be the set of all strings of letters. Let SameLength A A relate strings whose lengths are the same.
SameLength is an equivalence relation that carves up the universe of all strings into a collection of subsets, one for
each natural number (i.e., strings of length 0, strings of length 1, etc.).
Let Z be the set of integers. Let EqualMod3 Z Z relate integers that have the same remainder when divided by 3.
EqualMod3 has three equivalence classes, [0], [1], and [2]. [0] includes 0, 3, 6, etc.
Let CP be the set of C programs, each of which accepts an input of variable length. We'll call the length of any
specific input n. Let SameComplexity CP CP relate two programs if their running-time complexity is the same.
More specifically, (c1, c2) SameComplexity precisely in case:
m1, m2, k [n > k, RunningTime(c1) m1*RunningTime(c2) AND RunningTime(c2) m2*RunningTime(c1)]
Not every relation that connects "similar" things is an equivalence relation. For example, consider SimilarCost(x, y),
which holds if the price of x is within $1 of the price of y. Suppose A costs $10, B costs $10.50, and C costs $11.25.
Then SimilarCost(A, B) and SimilarCost(B, C), but not SimilarCost(A, C). So SimilarCost is not transitive, although it is
reflexive and symmetric.
3.4 Orderings
Important as equivalence relations are, they're not the only special kind of relation worth mentioning. Let's consider two
more.
A partial order is a relation that is reflexive, antisymmetric, and transitive. If we write out any partial order as a graph,
we'll see a structure like the following one for the relation SubstringOf. Notice that in order to make the graph relatively
easy to read, we'll adopt the convention that we don't write in the links that are required by reflexivity and transitivity.
But, of course, they are there in the relation itself:
ddyxabcdbce
yxab
bcd
bce
xab
ab
bc
"" (the empty string)

Read an arrow from x to y as meaning that (x, y) is an element of the relation. So, in this example, "a" is a substring of
"ab", which is a substring of "xab", and so forth. Note that in a partial order, it is often the case that there are some
elements (such as "ab" and "bc") that are not related to each other at all (since neither is a substring of the other).
Sometimes a partial order on a domain D defines a mininal and/or a maximal element. In this example, there is a minimal
element, the empty string, which is a substring of every string. There appears from this picture to be a maximal element,
but that's just because we drew only a tiny piece of the graph. Every string is a substring of some longer string, so there is
in fact no maximal element.
12
Let's consider another example based on the subsumption relation between pairs of logical expressions. A logical
expression A subsumes another expression B iff (if and only if), whenever A is true B must be true regardless of the values
assigned to the variables and functions of A and B. For example: x P(x) subsumes P(A), since, regardless of what the
predicate P is and independently of any axioms we have about it, and regardless of what object A represents, if x P(x) is
true, then P(A) must be true. Why is this a useful notion? Suppose we're building a theorem proving or reasoning
program. If we already know x P(x), and we are then told P(A), we can throw away this new fact. It doesn't add to our
knowledge (except perhaps to focus our attention on the object A) since it is subsumed by something we already knew. A
small piece of the subsumption relation on logical expressions is shown in the following graph. Notice that now there is a
maximal element, False, which subsumes everything (in other words, if we have the assertion False in our knowledge base,
we have a contradiction even if we know nothing else). There is also a minimal element, True, which tells us nothing.
False
x P(x)
P(B)
x R(x) S(x)
x R(x) T(x)
P(A)
P(A) Q(A)
x R(x)
True
A total order R A A is a partial order that has the additional property that a, b A, either (a, b) or (b, a) R. In
other words, every pair of elements must be related to each other one way or another. The classic example of a total order
is (or , if you prefer) on the integers. The relation is reflexive since every integer is equal to itself. It's antisymmetric
since if a b and a b, then for sure it is not also true that b a. It's transitive: if a b and b c, then a c. And, given
any two integers a and b, either a b or b a. If we draw any total order as a graph, we'll get something that looks like
this (again without the reflexive and transitive links shown):
6
5
4
3
This is only a tiny piece of the graph, of course. It continues infinitely in both directions. But notice that, unlike our
earlier examples of partial orders, there is no splitting in this graph. For every pair of elements, one is above and one is
below.
4 Important Properties of Binary Functions

Any relation that uniquely maps from all elements of its domain to elements of its range is a function. The two sets
involved can be anything and the mapping can be arbitrary. However, most of the functions we actually care about behave
in some sort of regular fashion. It is useful to articulate a set of properties that many of the functions that we'll study have.
When these properties are true of a function, or a set of functions, they give us techniques for proving additional
properties of the objects involved. In the following definitions, we'll consider an arbitrary binary function # defined over a
set we'll call A with elements we'll call a, b, and c. As examples, we'll consider functions whose actual domains are sets,
integers, strings, and boolean expressions.
13
A binary function # is commutative iff

Examples:
a+b=b+a
ab=ba
a AND b = b AND a
a,b a # b = b # a
integer addition
set intersection
boolean and
A binary function # is associative iff
a,b,c (a # b) # c = a # (b # c)
Examples:
(a + b) + c = a + (b + c)
(a b) c = a (b c)
(a AND b) AND c = a AND (b AND c)
(a || b) || c = a || (b || c)
a a # a = a.
A binary function # is idempotent iff

Examples:
integer addition
set intersection
boolean and
string concatenation
min(a, a) = a
aa=a
a AND a = a
integer min
set intersection
boolean and
The distributivity property relates two binary functions: A function # distributes over another function ! iff
a,b,c a # (b ! c) = (a # b) ! (a # c) and (b ! c) # a = (b # a) ! (c # a)
Examples:
a * (b + c) = (a * b) + (a * c)
integer multiplication over addition
a (b c) = (a b) (a c)
set union over intersection
a AND (b OR c)=(a AND b) OR (a AND c) boolean AND over OR
The absorption laws also relate two binary functions to each other: A function # absorbs another function ! iff
a,b a # (a ! b) = a
Examples:
a (a b) = a
a OR (a AND b) = a
set intersection absorbs union

boolean OR absorbs AND
It is often the case that when a function is defined over some set A, there are special elements of A that have particular
properties with respect to that function. In particular, it is worth defining what it means to be an identity and to be a zero:
An element a is an identity for the function # iff
Examples:
b*1=b
b+0=b
b=b
b OR False = b
b || "" = b
x A, x # a = x and a # x = x
1 is an identity for integer multiplication
0 is an identity for integer addition
is an identity for set union
False is an identity for boolean OR
"" is an identity for string concatenation
Sometimes it is useful to differentiate between a right identity (one that satisfies the first requirement above) and a left
identity (one that satisfies the second requirement above). But for all the functions we'll be concerned with, if there is a
left identity, it is also a right identity and vice versa, so we will talk simply about an identity.
x A, x # a = a and a # x = a
An element a is a zero for the function # iff

Examples:
b*0=0
b=
b AND FALSE = FALSE
0 is a zero for integer multiplication

is a zero for set intersection
FALSE is a zero for boolean AND
Just as with identities, it is sometimes useful to distinguish between left and right zeros, but we won't need to.
14
Although we're focusing here on binary functions, there's one important property that unary functions may have that is
worth mentioning here:
A unary function % is a self inverse iff x %(%x)) = x. In other words, if we compose the function with itself (apply it
twice), we get back the original argument. Note that this is not the same as saying that the function is its own inverse. In
most of the cases we'll consider (including the examples given here), it is not. A single application of the function
produces a new value, but if we apply the function a second time, we get back to where we started.
Examples:
-(-(a)) = a
1/(1/a)
a=a
( a) = a
(aR)R = a
Multiplying by -1 is a self inverse for integers

Dividing into 1 is a self inverse for integers
Complement is a self inverse for sets
Negation is a self inverse for booleans
Reversal is a self inverse for strings
5 Relations and Functions on Sets

In the last two sections, we explored various useful properties of relations and functions. With those tools in hand, let's
revisit the basic relations and functions on sets.
5.1 Relations
We have defined two relations on sets: subset and proper subset. What can we say about them? Subset is a partial order,
since it is reflexive (every set is a subset of itself), transitive (if A B and B C, then A C) and antisymmetric (if A
B and A B, then it must not be true that B A). For example, we see that the subset relation imposes the following
partial order if you read each arrow as "is a subset of":
Z (the integers)
Odd numbers
Even numbers
Prime numbers excluding 2
{3, 17, 29}
{4, 10}
What about proper subset? It is not a partial order since it is not reflexive.
5.2 Functions
All of the functional properties we defined above apply in one way or another to the functions we have defined on sets.
Further, as we saw above, there some set functions have a zero or an identity. We'll summarize here (without proof) the
most useful properties that hold for the functions we have defined on sets:
Commutativity
AB=BA
AB=BA
Associativity
(A B) C = A (B C)
(A B) C = A (B C)
15
Idempotency
AA=A
AA=A
Distributivity
(A B) C = (A C) (B C)
(A B) C = (A C) (B C)
Absorption
(A B) A = A
(A B) A = A
Identity
A=A
Zero
A=
Self Inverse
A=A
In addition, we will want to make use of the following theorems that can be proven to apply specifically to sets and their
operations (as well as to boolean expressions):
De Morgan's laws
AB=AB
AB=AB
6 Proving Properties of Sets

A great deal of what we do when we build a theory about some domain is to prove that various sets of objects in that
domain are equal. For example, in our study of automata theory, we are going to want to prove assertions such as these:
The set of strings defined by some regular expression E is identical to the set of strings defined by some second
regular expression E'.
The set of strings that will be accepted by some given finite state automaton M is the same as the set of strings that
will be accepted by some new finite state automaton M' that is smaller than M.
The set of languages that can be defined using regular expressions is the same as the set of languages that can be
accepted by a finite state automaton.
The set of problems that can be solved by a Turing Machine with a single tape is the same as the set of problems that
can be solved by a Turing Machine with any finite number of tapes.
So we become very interested in the question, "How does one prove that two sets are identical"? There are lots of ways
and many of them require special techniques that apply in specific domains. But it's worth mentioning two very general
approaches here.
6.1 Using Set Identities and the Definitions of the Functions on Sets
Sometimes we want to compare apples to apples. We may, for example, want to prove that two sets of strings are
identical, even though they may have been derived differently. In this case, one approach is to use the set identity
theorems that we enumerated in the last section. Suppose, for example, that we want to prove that
A (B (A C)) = A
We can prove this as follows:
A (B (A C)) = (A B) (A (A C))
Distributivity
= (A B) ((A C) A)
Commutativity
= (A B) A
Absorption
=A
Absorption
Sometimes, even when we're comparing apples to apples, the theorems we've listed aren't enough. In these cases, we need
to use the definitions of the operators. Suppose, for example, that we want to prove that
A-B =AB
We can prove this as follows (where U stands for the Universe with respect to which we take complement):
16
A-B
= {x : x A and x B}
= {x : x A and (x U and x B)}
= {x : x A and x U - B}
= {x : x A and x B}
=AB
6.2 Showing Two Sets are the Same by Showing that Each is a Subset of
the Other
Sometimes, though, our problem is more complex. We may need to compare apples to oranges, by which I mean that we
are comparing sets that aren't even defined in the same terms. For example, we will want to be able to prove that A: the
set of languages that can be defined using regular expressions is the same as B: the set of languages that can be accepted
by a finite state automaton. This seems very hard: Regular expressions look like
a* (b ba)*
Finite state machines are a collection of states and rules for moving from one state to another. How can we possibly prove
that these A and B are the same set? The answer is that we can show that the two sets are equal by showing that each is a
subset of the other. For example, in the case of the regular expressions and the finite state machines, we will show first
that, given a regular expression, we can construct a finite state machine that accepts exactly the strings that the regular
expression describes. That gives us A B. But there might still be some finite state machines that don't correspond to
any regular expressions. So we then show that, given a finite state machine, we can construct a regular expression that
defines exactly the same strings that the machine accepts. That gives us B A. The final step is to exploit the fact that
A B and B A A = B
7 Cardinality of Sets
It seems natural to ask, given some set A, "What is the size of A?" or "How many elements does A contain?" In fact,
we've been doing that informally. We'll now introduce formal techniques for discussing exactly what we mean by the size
of a set. We'll use the term cardinality to describe the way we answer such questions. So we'll reply that the cardinality
of A is X, for some appropriate value of X. For simple cases, determining the value of X is straightforward. In other
cases, it can get quite complicated. For our purposes, however, we can get by with three different kinds of answers: a
natural number (if A is finite), "countably infinite" (if A has the same number of elements as there are integers), and
"uncountably infinite" (if A has more elements than there are integers).
We write the cardinality of a set A as |A|.
A set A is finite and has cardinality n N (the natural numbers) if either A = or there is a bijection from A to {1, 2,
n}, for some value of n. In other words, a set is finite if either it is empty or there exists a one-to-one and onto mapping
from it to a subset of the positive integers. Or, alternatively, a set is finite if we can count its elements and finish. The
cardinality of a finite set is simply a natural number whose value is the number of elements in the set.
A set is infinite if it is not finite. The question now is, "Are all infinite sets the same size?" The answer is no. And we
don't have to venture far to find examples of infinite sets that are not the same size. So we need some way to describe the
cardinality of infinite sets. To do this, we need to define a set of numbers we'll call the cardinal numbers. We'll use these
numbers as our measure of the size of sets. Initially, we'll define all the natural numbers to be cardinal numbers. That lets
us describe the cardinality of finite sets. Now we need to add new cardinal numbers to describe the cardinality of infinite
sets.
Let's start with a simple infinite set N, the natural numbers. We need a new cardinal number to describe the (infinite)
number of natural numbers that there are. Following Cantor, we'll call this number 0. (Read this as "aleph null". Aleph
is the first symbol of the Hebrew alphabet.)
17
Next, we'll say that any other set that contains the same number of members as N does also has cardinality 0. We'll also
call a set with cardinality 0 countably infinite. And one more definition: A set is countable if it is either finite or
countably infinite.
To show that a set has cardinality 0, we need to show that there is a bijection between it and N. The existence of such a
bijection proves that the two sets have the same number of elements. For example, the set E of even natural numbers has
cardinality 0. To prove this, we offer the bijection:
Even : E N
Even(x) = x/2
So we have the following mapping from E to N:
E
0
2
4
6
N
0
1
2
3
This one was easy. The bijection was obvious. Sometimes it's less so. In harder cases, a good way to think about the
problem of finding a bijection from some set A to N is that we need to find an enumeration of A. An enumeration of A is
simply a list of the elements of A in some order. Of course, if A is infinite, the list will be infinite, but as long as we can
guarantee that every element of A will show up eventually, we have an enumeration. But what is an enumeration? It is in
fact a bijection from A to the positive integers, since there is a first element, a second one, a third one, and so forth. Of
course, what we need is a bijection to N, so we just subtract one. Thus if we can devise a technique for enumerating the
elements of A, then our bijection to N is simply
Enum : A N
Enum(x) = x's position in the enumeration - 1
Let's consider an example of this technique:
Theorem: The union of a countably infinite number of countably infinite sets is countably infinite.
To prove this theorem, we need a way to enumerate all the elements of the union. The simplest thing to do would be to
start by dumping in all the elements of the first set, then all the elements of the second, etc. But, since the first set is
infinite, we'll never get around to considering any of the elements of the other sets. So we need another technique. If we
had a finite number of sets to consider, we could take the first element from each, then the second element from each, and
so forth. But we also have an infinite number of sets, so if we try that approach, we'll never get to the second element of
any of the sets. So we follow the arrows as shown below. The numbers in the squares indicate the order in which we
select elements for the enumeration. This process goes on forever, but it is systematic and it guarantees that, if we wait
long enough, any element of any of the sets will eventually be enumerated.
Element 1
Element 2
Element 3
Set 1
1
2
6
7
Set 2
3
5
8
Set 3
4
Set 4
It turns out that a lot of sets have cardinality 0. Some of them, like the even natural numbers, appear at first to contain
fewer elements. Some of them, like the union of a countable number of countable sets, appear at first to be bigger. But in
both cases there is a bijection between the elements of the set and the natural numbers, so the cardinality is 0.
However, this isn't true for every set. There are sets with more than 0 elements. There are more than 0 real numbers,
for example. As another case, consider an arbitrary set S with cardinality 0. Now consider the power set of S (the set of
all subsets of S). This set has cardinality greater than 0. To prove this, we need to show that there exists no bijection
18
between the power set of S and the integers. To do this, we will use a technique called diagonalization. Diagonalization
is a kind of proof by contradiction. It works as follows:
Let's start with the original countably infinite set S. We can enumerate the elements of S (since it's countable), so there's a
first one, a second one, etc. Now we can represent each subset SS of S as a binary vector that contains one element for
each element of the original set S. If SS contains element 1 of S, then the first element of its vector will be 1, otherwise 0.
Similarly for all the other elements of S. Of course, since S is countably infinite, the length of each vector will also be
countably infinite. Thus we might represent a particular subset SS of S as the vector:
Elem 1 of S
1
Elem 2 of S
0
Elem 3 of S
0
Elem 4 of S
1
Elem 5 of S
1
Elem 6 of S
0
Next, we observe that if the power set P of S were countably infinite, then there would be an enumeration of it that put its
elements in one to one correspondence with the natural numbers. Suppose that enumeration were the following (where
each row represents one element of P as described above. Ignore for the moment the numbers enclosed in parentheses.):
Elem 1 of P
Elem 2 of P
Elem 3 of P
Elem 4 of P
Elem 5 of P
Elem 6 of P
Elem 1 of S
1
(1)
0
1
0
1
1
Elem 2 of S
0
1
(2)
1
0
0
1
Elem 3 of S
0
0
0
(3)
1
1
1
Elem 4 of S
0
0
0
0
(4)
0
0
Elem 5 of S
0
0
0
0
0
(5)
0
Elem 6 of S
0
0
0
0
0
0
(6)
.
..
..
..
..
..
..
If this really is an enumeration of P, then it must contain all elements of P. But it doesn't. To prove that it doesn't, we will
construct an element L P that is not on the list. To do this, consider the numbers in parentheses in the matrix above.
Using them, we can construct L:
(1)
(2)
(3)
(4)
(5)
(6)
..
What we mean by (1) is that if (1) is a 1 then 0; if (1) is a 0, then 1. So we've constructed the representation for an
element of P. It must be an element of P since it describes a possible subset of S. But we've built it so that it differs from
the first element in the list above by whether or not it includes element 1 of S. It differs from the second element in the list
above by whether or not it includes element 2 of S. And so forth. In the end, it must differ from every element in the list
above in at least one place. Yet it is clearly an element of P. Thus we have a contradiction. The list above was not an
enumeration of P. But since we made no assumptions about it, no enumeration of P can exist. In particular, if we try to
fix the problem by simply adding our new element to the list, we can just turn around and do the same thing again and
create yet another element that 's not on the list. Thus there are more than 0 elements in P. We'll say that sets with more
than 0 elements are uncountably infinite.
The real numbers are uncountably infinite. The proof that they are is very similar to the one we just did for the power set
except that it's a bit tricky because, when we write out each number as an infinite sequence of digits (like we wrote out
each set above as an infinite sequence of 0's and 1's), we have to consider the fact that several distinct sequences may
represent the same number.
Not all uncountably infinite sets have the same cardinality. There is an infinite number of cardinal numbers. But we won't
need any more. All the uncountably infinite sets we'll deal with (and probably all the ones you can even think of unless
you keep taking power sets) have the same cardinality as the reals and the power set of a countably infinite set.
Thus to describe the cardinality of all the sets we'll consider, we will use one of the following:
The natural numbers, which we'll use to count the number of elements of finite sets,
19
0, which is the cardinality of all countably infinite sets, and

uncountable, which is the cardinality of any set with more than 0 members.
8 Closures
Imagine some set A and some property P. If we care about making sure that A has property P, we are likely to do the
following:
1. Examine A for P. If it has property P, we're happy and we quit.
2. If it doesnt, then add to A the smallest number of additional elements required to satisfy P.
Let's consider some examples:
Let A be a set of friends you're planning to invite to a party. Let P be "A should include everyone who is likely to
find out about the party" (since we don't want to offend anyone). Let's assume that if you invite Bill and Bill has a
friend Bob, then Bill may tell Bob about the party. This means that if you want A to satisfy P, then you have to invite
not only your friends, but your friends' friends, and their friends, and so forth. If you move in a fairly closed circle,
you may be able to satisfy P by adding a few people to the guest list. On the other hand, it's possible that you'd have
to invite the whole city before P would be satisfied. It depends on the connectivity of the FriendsOf relation in your
social setting. The problem is that whenever you add a new person to A, you have to turn around and look at that
person's friends and consider whether there are any of them who are not already in A. If there are, they must be
added, and so forth. There's one positive feature of this problem, however. Notice that there is a unique set that does
satisfy P, given the initial set A. There aren't any choices to be made.
Let A be a set of 6 people. Let P be "A can enter a baseball tournament". This problem is different from the last in
two important ways. First, there is a clear limit to how many elements we have to add to A in order to satisfy P. We
need 9 people and when we've got them we can stop. But notice that there is not a unique way to satisfy P (assuming
that we know more than 9 people). Any way of adding 3 people to the set will work.
Let A be the Address relation (which we defined earlier as "lives at same address as"). Since relations are sets, we
should be able to treat Address just as we've treated the sets of people in our last two examples. We know that
Address is an equivalence relation. So we'll let P be the property of being an equivalence relation (i.e., reflexive,
symmetric, and transitive). But suppose we are only able to collect facts about living arrangements in a piecemeal
fashion. For example, we may learn that Address contains {(Dave, Mary), (Sue, Pete), (John, Bill)}. Immediately we
know, because Address must be reflexive, that it must also contain {(Dave, Dave), (Mary, Mary), (Sue, Sue), (Pete,
Pete), (John, John), (Bill, Bill)}. And, since Address must also be symmetric it must contain {(Mary, Dave), (Pete,
Sue), (Bill, John)}. Now suppose that we discover that Mary lives with Sue. We add {(Mary, Sue)}. To make
Address symmetric again, we must add {(Sue, Mary)}. But now we also have to make it transitive by adding {(Dave,
Sue), (Sue, Dave)}.
Let A be the set of natural numbers. Let P be "the sum of any two elements of A is also in A." Now we've got a
property that is already satisfied. The sum of any two natural numbers is a natural number. This time, we don't have
to add anything to A to establish P.
Let A be the set of natural numbers. Let P be "the quotient of any two elements of A is also in A." This time we have
a problem. 3/5 is not a natural number. We can add elements to A to satisfy P. If we do, we end up with exactly the
rational numbers.
In all of these cases, we're going to want to say that A is closed with respect to P if it possesses P. And, if we have to add
elements to A in order to satisfy P, we'll call a smallest such expanded A that does satisfy P a closure of A with respect to
P. What we need to do next is to define both of these terms more precisely .
8.1 Defining Closure

The first set of definitions of closure that we'll present is very general, although it does require that we can describe
property P as an n-ary relation (for some value of n). We can use it to describe what we did in all but one of the examples
above, although in a few cases it will be quite cumbersome to do so.
Let n be an integer greater than or equal to 1. Let R be an n-ary relation on a set D. Thus elements of R are of the form
(d1, d2, , dn). We say that a subset S of D is closed under R if, whenever:
20
1. d1, d2, dn-1 S, (all of the first n-1 elements are already in the set S) and
2. (d1, d2, dn-1, dn) R (the last element is related to the n-1 other elements via R)
it is also true that dn S.
A set S' is a closure of S with respect to R (defined on D) iff:
1. S S',
2. S' is closed under R, and
3. T (T D and T is closed under R) |S'| |T|.
In other words, S' is a closure of S with respect to R if it is an extension of S that is closed under R and if there is no
smaller set that also meets both of those requirements. Note that we can't say that S' must be the smallest set that will do
the job, since we do not yet have any guarantee that there is a unique such smaller set (recall the softball example above).
These definitions of closure are a very natural way to describe our first example above. Drawing from a set A of people,
you start with S equal to your friends. Then, to compute your invitee list, you simply take the closure of S with respect to
the relation FriendOf, which will force you to add to A your friends' friends, their friends, and so forth.
Now consider our second example, the case of the baseball team. Here there is no relation R that specifies, if one or more
people are already on the team, then some specific other person must also be on. The property we care about is a property
of the team (set) as a whole and not a property of patterns of individuals (elements). Thus this example, although similar,
is not formally an instance of closure as we have just defined it. This turns out to be significant and leads us to the
following definition:
Any property that asserts that a set S is closed under some relation R is a closure property of S. It is possible to prove that
if P is a closure property, as just defined, on a set A and S is a subset of A, then the closure of S with respect to R exists
and is unique. In other words, there exists a unique minimal set S' that contains S and is closed under R. Of all of our
examples above, the baseball example is the only one that cannot be described in the terms of our definition of a closure
property. The theorem that we have just stated (without proof) guarantees, therefore, that it will be the only one that does
not have a unique minimal solution.
The definitions that we have just provided also work to describe our third example, in which we want to compute the
closure of a relation (since, after all, a relation is a set). All we have to do is to come up with relations that describe the
properties of being reflexive, symmetric, and transitive. To help us see what those relations need to be, let's recall our
definitions of symmetry, reflexivity, and transitivity:
A relation R A A is reflexive if, for each a A, (a, a) R.
A relation R A A is symmetric if, whenever (a, b) R, so is (b, a).
A relation R A A is transitive if, whenever (a, b) R and (b, c) R, (a, c) R.
Looking at these definitions, we can come up with three relations, Reflexivity, Symmetry, and Transitivity. All three are
relations on relations, and they will enable us to define these three properties using the closure definitions we've given so
far. All three definitions assume a base set A on which the relation we are interested is defined:
a A, ((a, a)) Reflexivity. Notice the double parentheses here. Reflexivity is a unary relation, where each
element is itself an ordered pair. It doesn't really "relate" two elements. It is simply a list of ordered pairs. To see
how it works to define reflexive closure, imagine a set A = {x, y}. Now suppose we start with a relation R on A =
{(x, y)}. Clearly R isn't reflexive. And the Reflexivity relation tells us that it isn't because the reflexivity relation on
A contains {((x, x)), ((y, y))}. This is a unary relation. So n, in the definition of closure, is 1. Consider the first
element ((x, x)). We consider all the components before the nth (i.e., first) and see if they're in A. This means we
consider the first zero components. Trivially, all zero of them are in A. So the nth (the first) must also be. This
means that (x, x) must be in R. But it isn't. So to compute the closure of R under Reflexivity, we add it. Similarly for
(y, y).
a,b A, a b [((a, b), (b, a)) Symmetry]. This one is a lot easier. Again, suppose we start with a set A = {x,
y} and a relation R on A = {(x, y)}. Clearly R isn't symmetric. And Symmetry tells us that. Symmetry on A = {((x,
y), (y, x)), ((y, x), (x, y))}. But look at the first element of Symmetry. It tells us that for R to be closed, whenever (x,
y) is in R, (y, x) must also be. But it isn't. To compute the closure of R under Symmetry, we must add it.
21
a,b,c A, [a b b c] [((a, b), (b, c), (a, c)) Transitivity]. Now we will exploit a ternary relation.
Whenever the first two elements of it are present in some relation R, then the third must also be if R is transitive. This
time, let's start with a set A = {x, y, z} and a relation R on A = {(x, y), (y, z)}. Clearly R is not transitive. The
Transitivity relation on A is {((x, y), (y, z), (x, z)), ((x, z), (z, y), (x, y)), ((y, x), (x, z), (y, z)), ((y, z), (z, x), (y, x)),
((z, x), (x, y), (z, y)), ((z, y), (y, x), (z, x))}. Look at the first element of it. Both of the first two components of it are
in R. But the third isn't. To make R transitive, we must add it.
These definitions also work to enable us to describe the closure of the integers under division as the rationals. Following
the definition, A is the set of rationals. S (a subset of A) is the integers and R is QuotientClosure, defined as:
a,b,c A, [a/b = c] [(a, b, c) QuotientClosure].
So we've got a quite general definition of closure. And it makes it possible to prove the existence of a unique closure for
any set and any relation R. The only constraint is that this definition works only if we can define the property we care
about as an n-ary relation for some finite n. There are cases of closure where this isn't possible, as we saw above, but we
won't need to worry about them in this class.
So we don't really need any new definitions. We've offered a general definition of closure of a set (any set) under some
relation (which is the way we use to define a property). But most of the cases of closure that we'll care about involve the
special case of the closure of a binary relation given some property that may or may not be naturally describable as a
relation. For example, one could argue that the relations we just described to define the properties of being reflexive,
symmetric, and transitive are far from natural. Thus it will be useful to offer the following alternative definitions. Don't
get confused though by the presence of two definitions. Except when we cannot specify our property P as a relation (and
we won't need to deal with any such cases), these new definitions are simply special cases of the one we already have.
We say that a binary relation B on a set T is closed under property P if B possesses P. For example, LessThanOrEqual is
closed under transitivity since it is transitive. Simple enough. Next:
Let B be a binary relation on a set T. A relation B' is a closure of B with respect to some property P iff:
1. B B',
2. B' is closed under P, and
3. There is no smaller relation B'' that contains B and is closed under P.
So, for example, the transitive closure of B = {(1, 2), (2, 3)} is the smallest new relation B' that contains B but is
transitive. So B' = {(1, 2), (2, 3), (1, 3)}.
You'll generally find it easier to use these definitions than our earlier ones. But keep in mind that, with the earlier
definitions it is possible to prove the existence of a unique closure. Since we went through the process of defining
reflexivity, symmetry, and transitivity using those definitions, we know that there always exists a unique reflexive,
symmetric, and transitive closure for any binary relation. We can exploit that fact that the same time that we use the
simpler definitions to help us find algorithms for computing those closures.
8.2 Computing Closures

Suppose we are given a set and a property and we want to compute the closure of the set with respect to the property.
Let's consider two examples:
Compute the symmetric closure of a binary relation B on a set A. This is trivial. Simply make sure that, for all
elements x of A, (x, x) B.
Compute the transitive closure of a binary relation B on a set A. This is harder. We can't just add a fixed number of
elements to B and then quit. Every time we add a new element, such as (x, y), we have to look to see whether there's
some element (y, z) so now we also have to add (x, z). And, similarly, we must check for any element (w, x) that
would force us to add (w, y). If A is infinite, there is no guarantee that this process will ever terminate. Our theorem
that guaranteed that a unique closure exists did not guarantee that it would contain a finite number of elements and
thus be computable in a finite amount of time.
22
We can, however, guarantee that the transitive closure of any binary relation on a finite set is computable. How? A very
simple approach is the following algorithm for computing the transitive closure of a binary relation B with N elements on
a set A:
Set Trans = B;
/* Initially Trans is just the original relation.
/* We need to find all cases where (x, y) and (y, z) are in Trans. Then we must insert (x, z) into Trans if
/* it isn't already there.
Boolean AddedSomething = True;
/* We'll keep going until we make one whole pass through
without adding any new elements to Trans.
while AddedSomething = True do
AddedSomething = False;
Xcounter := 0;
Foreach element of Trans do
xcounter := xcounter + 1;
x = Trans[xcounter][1]
/* Pull out the first element of the current element of Trans
y = Trans[xcounter][2]
/* Pull out the second element of the i'th element of Trans
/* So if the first element of Trans is (p, q), then
/* x = p and y = q the first time through.
zcounter := 0;
foreach element of Trans do
zcounter := zcounter + 1;
if Trans[zcounter][1] = y then do /* We've found another element (y, ?) and we may need to
z = Trans[zcounter][2];
/* add (x, ?) to Trans.
if (x, z) Trans then do /* we have to add it
Insert(Trans, (x, z))
AddedSomething = True;
end;
end;
end;
end;
end;
This algorithm works. Try it on some simple examples. But it's very inefficient. There are much more efficient
algorithms. In particular, if we represent a relation as an incidence matrix, we can do a lot better. Using Warshall's
algorithm, for example, we can find the transitive closure of a relation of n elements using 2n3 bit operations. For a
description of that algorithm, see, for example, Kenneth Rosen, Discrete Mathematics and its Applications, McGraw-Hill.
9 Proof by Mathematical Induction

In the last section but one, as a sideline to our main discussion of cardinality, we presented diagonalization as a proof
technique. In this section, we'll present one other very useful proof technique.
Mathematical induction is a technique for proving assertions about the set of positive integers. A proof by induction of
assertion A about some set of positive integers greater than or equal to some specific value has two steps:
1. Prove that A holds for the smallest value we're concerned with. We'll call this value v. Generally v = 0 or 1, but
sometimes A may hold only once we get past some initial unusual cases.
2. Prove that n v, A(n) A(n+1)
We'll call A(n) the induction hypothesis. Since we're trying to prove that A(n) A(n+1), we can assume the induction
hypothesis as an axiom in our proof.
Let's do a simple example and use induction to prove that the sum of the first n odd positive integers is n2. Notice that this
appears to be true:
(n = 1) 1
= 1 = 12
(n = 2) 1 + 3
= 4 = 22
(n = 3) 1 + 3 + 5
= 9 = 32
(n = 4) 1 + 3 + 5 + 7 = 16 = 42, and so forth.
To prove it, we need to follow the two steps that we just described:
23
1.
2.
Let v = 1: 1 = 12
Prove that, n 0,
n +1
( Odd i = n ) ( Odd i = ( n + 1) 2 )
2
i =1
i =1
To do this, we observe that the sum of the first n+1 odd integers is the sum of the first n of them plus the n+1'st, i.e.,
n +1
Odd
i =1
Odd
i =1
2
=n
+ Odd n +1
+ Odd n +1
(Using the induction hypothesis)
= n + 2n + 1
( Odd n+1 is 2n + 1)
2
= (n + 1)
Thus we have shown that the sum of the first n+1 odd integers must be equivalent to (n+1)2 if it is known that the sum of
the first n of them is equivalent to n2.
Mathematical induction lets us prove properties of positive integers. But it also lets us prove properties of other things if
the properties are described in terms of integers. For example, we could talk about the size of finite sets, or the length of
finite strings. Let's do one with sets: For any finite set A, |2A| = 2|A|. In other words, the cardinality of the power set of A
is 2 raised to the power of the cardinality of A. We'll prove this by induction on the number of elements of A (|A|). We
follow the same two steps:
1. Let v = 0. So A is , |A| = 0, and A's power set is {}, whose cardinality is 1 = 20 = 2|A|.
2. Prove that, n 0, if |2A| = 2|A| is true for all sets A of cardinality n, then it is also true for all sets S of cardinality n+1.
We do this as follows. Since n 0, and any such S has n + 1 elements, S must have at least one element. Pick one
and call it a. Now consider the set T that we get by removing a from S. |T| must be n. So, by the induction
hypothesis (namely that |2A| = 2|A| if |A| = n), our claim is true for T and we have |2T| = 2|T|. Now let's return to the
power set of the original set S. It has two parts: those subsets that include a and those that don't. The second part is
exactly 2T, so we know that it has 2|T| = 2n elements. The first part (all the subsets that include a) is exactly all the
subsets that don't include a with a added in). Since there are 2n subsets that don't include a and there are the same
number of them once we add a to each, we have that the total number of subsets of our original set S is 2n (for the
ones that don't include a) plus 2n (for the ones that do include a), for a total of 2(2n) = 2n+1, which is exactly 2|S|.
Why does mathematical induction work? It relies on the well-ordering property of the integers, which states that every
nonempty set of nonnegative integers has a least element. Let's see how that property assures us that mathematical
induction is valid as a proof technique. We'll use the technique of proof by contradiction to show that, given the wellordering property, mathematical induction must be valid. Once we have done an induction proof, we know that A(v)
(where v is 0 or 1 or some other starting value) is true and we know that n 0, A(n) A(n+1). What we're using the
technique to enable us to claim is that, therefore, n v, A(n). Suppose the technique were not valid and there was a set
S of nonnegative integers v for which A(n) is False. Then, by the well-ordering property, there is a smallest element in
this set. Call it x. By definition, x must be equal to or greater than v. But it cannot actually be v because we proved A(v).
So it must be greater than v. But now consider x - 1. Since x - 1 is less than x, it cannot be in S (since we chose x to be
the smallest value in S). If it's not in S, then we know A(x - 1). But we proved that n 0, A(n) A(n+1), so A(x - 1)
A(x). But we assumed A(x). So that assumption led us to a contradiction; thus it must be false.
Sometimes the principle of mathematical induction is stated in a slightly different but formally equivalent way:
1. Prove that A holds for the smallest value v with which we're concerned.
2. State the induction hypothesis H, which must be of the form, "There is some integer n v such that A is true for all
integers k where v k n."
3. Prove that (n v, A(n)) A(n + 1). In other words prove that whenever A holds for all nonnegative integers
starting with v, up to an including n, it must also hold for n + 1.
You can use whichever form of the technique is easiest for a particular problem.
24

1 Regular Languages
The first class of languages that we will consider is the regular languages. As we'll see later, there are several quite
different ways in which regular languages can be defined, but we'll start with one simple technique, regular expressions.
A regular expression is an expression (string) whose "value" is a language. Just as 3 + 4 and 14/2 are two arithmetic
expressions whose values are equal, so are (a b)* and a* (a b)*b(a b)* two regular expressions whose values are
equal. We will use regular expressions to denote languages just as we use arithmetic expressions to denote numbers. Just
as there are some numbers, like , that cannot be expressed by arithmetic expressions of integers, so too there are some
languages, like anbn, that cannot be expressed by regular expressions. In fact, we will define the class of regular
languages to be precisely those that can be described with regular expressions.
Let's continue with the analogy between arithmetic expressions and regular expressions. The syntax of arithmetic
expressions is defined recursively:
1. Any numeral in {0, 1,2, ...} is an arithmetic expression.
2. If and are expressions, so is ( + ).
3. If and are expressions, so is ( * ).
4. If and are expressions, so is ( - ).
5. If and are expressions, so is (/).
6. If is an expression, so is -.
These operators that we've just defined have associated with them a set of precedence rules, so we can write -3 + 4*5
instead of (-3 + (4*5)).
Now let's return to regular expressions. The syntax of regular expressions is also defined recursively:
Similarly there are precedence rules for regular expressions, so we can write a* bc instead of (a* (bc)). Note that *
binds more tightly than does concatenation, which binds more tightly than .
In both cases (arithmetic expressions and regular expressions) there is a distinction between the expression and its value.
5 + 7 is not the same expression as 3 * 4, even though they both have the same value. (You might imagine the expressions
being in quotation marks: "5 + 7" is clearly not the same as "3 * 4". Similarly, "John's a good guy" is a different sentence
from "John is nice", even though they both have the same meaning, more or less.) The rules that determine the value of an
arithmetic expression are quite familiar to us, so much so that we are usually not consciously aware of them. But regular
expressions are less familiar, so we will explicitly specify the rules that define the values of regular expressions. We do
this by recursion on the structure of the expression. Just remember that the regular expression itself is a syntactic object
made of parentheses and other symbols, whereas its value is a language. We define L() to be the function that maps
regular expressions to their values. We might analogously define L() for arithmetic expressions and say that L(5 + 7) = 12
= L(3 * 4). L() is an example of a semantic interpretation function, i. e., it is a function that maps a string in some
language to its meaning. In our case, of course, the language from which the arguments to L() will be drawn is the
language of regular expressions as defined above.
L() for regular expressions is defined as follows:
1. L() = and L(a) = {a} for each a
2. If , are regular expressions, then
3.
4.
5.
L(()) = L() L()

= all strings that can be formed by concatenating to some string from some string from .
Note that if either or is , then its language is , so there is nothing to concatenate and the result is .
If , are regular expressions, then L(()) = L() L()
If is a regular expression, then L(*) = L()*
L( () ) = L()
So, for example, let's find the meaning of the regular expression (a b)*b:
L((a b)*b)
= L((a b)*) L(b)
= L(a b)* L(b)
= (L(a) L(b))* L(b)
= ({a} {b})* {b}
= {a, b}* {b}
which is just the set of all strings ending in b. Another example is L(((a b)(a b))a(a b)*) = {xay: x and y are strings
of a's and b's and |x| = 2}. The distinction between an expression and its meaning is somewhat pedantic, but you should
try to understand it. We will usually not actually write L() because it is generally clear from context whether we mean the
regular expression or the language denoted by it. For example, a (a b)* is technically meaningless since (a b)* is a
regular expression, not a set. Nonetheless, we use it as a reasonable abbreviation for a L((a b)*), just as we write 3 +
4 = 4 + 3 to mean that the values of "3 + 4" and "4 + 3", not the expressions themselves, are identical.
Here are some useful facts about regular expressions and the languages they describe:
(a b)* = (a*b*)* = (b*a*)* = set of all strings composed exclusively of a's and b's (including the empty string)
(a b)c = (ac bc) Concatenation distributes over union
c(a b) = (ca cb)
"
a* b* (a b)* The right-hand expression denotes a set containing strings of mixed a's and b's, while the lefthand expression does not.
(ab)* a*b*
In the right-hand expression, all a's must precede all b's. That's not true for the left-hand
expression.
a* * = a* = a*
There is an algebra of regular expressions, but it is rather complex and not worth the effort to learn it. Therefore, we will
rely primarily on our knowledge of what the expressions mean to determine the equivalence (or non-equivalence) or
regular expressions.
We are now in a position to state formally our definition of the class of regular languages: Given an alphabet , the set of
regular languages over is precisely the set of languages that can be defined using regular expressions with respect to .
Another equivalent definition (given our definition of regular expressions and L()), is that the set of regular languages over
an alphabet is the smallest set that contains and each of the elements of , and that is closed under the operations of
concatenation, union, and Kleene star (rules 2, 3, and 4 above).
2 Proof of the Equivalence of Nondeterministic and Deterministic FSAs

In the lecture notes, we stated the following:
Theorem: For each NDFSA, there is an equivalent DFSA.
This is an extremely significant theorem. It says that, for finite state automata, nondeterminism doesn't add power. It adds
convenience, but not power. As we'll see later, this is not true for all other classes of automata.
In the notes, we provided the first step of a proof of this theorem, namely an algorithm to construct, from any NDFSA, an
equivalent DFSA. Recall that the algorithm we presented was the following: Given a nondeterministic FSA M (K, , , s,
F), we derive an equivalent deterministic FSA M' = (K', , ', s', F') as follows:
1.
2.
3.
4.
5.
Compute E(q) for each q in K. q K, E(q) = {p K : (q, ) |-*M (p, )}. In other words, E(q) is the set of states
reachable from q without consuming any input.
Compute s' = E(s).
Compute ', which is defined as follows: Q2K and a, '(Q, a) = {E(p) : p K and (q, a, p) for some q
Q}. Recall that the elements of 2K are sets of states from the original machine M. So what we've just said is that
to compute the transition out of one of these "set" states, find all the transitions out of the component states in the
original machine, then find all the states reachable from them via epsilon transitions. The new state is the set of all
states reachable in this way. We'll actually compute ' by first computing it for the new start state s' and each of the
elements of . Each state thus created becomes an element of K', the set of states of M'. Then we compute ' for
any new states that were just created. We continue until there are no additional reachable states. (so although ' is
defined for all possible subsets of K, we'll only bother to compute it for the reachable such subsets and in fact we'll
define K' to include just the reachable configurations.)
Compute K' = that subset of 2K that is reachable, via ', as defined in step 3, from s'.
Compute F' = {Q K' : Q F }. In other words, each constructed "set" state that contains at least one final
state from the original machine M becomes a final state in M'.
However, to complete the proof of the theorem that asserts that there is an equivalent DFSA for every NDFSA, we need
next to prove that the algorithm we have defined does in fact produce a machine that is (1) deterministic, and (2)
equivalent to the original machine.
Proving (1) is trivial. By the definition in step 3 of ', we are guaranteed that ' is defined for all reachable elements of K'
and that it is single valued.
Next we must prove (2). In other words, we must prove that M' accepts a string w if and only if M accepts w. We
constructed the transition function ' of M' so that each step of the operation of M' mimics an "all paths" simulation of M.
So we believe that the two machines are identical, but how can we prove that they are? Suppose we could prove the
following:
Lemma: For any string w * and any states p, q K, (q, w) |-*M (p, ) iff (E(q), w) |-*M' (P, ) for some P
K' that contains p. In other words, if the original machine M starts in state q and, after reading the string w, can
land in state p, then the new machine M' must behave as follows: when started in the state that corresponds to the
set of states the original machine M could get to from q without consuming any input, M' reads the string w and
lands in one of its new "set" states that contains p. Furthermore, because of the only-if part of the lemma, M'
must end up in a "set" state that contains only states that M could get to from q after reading w and following any
available epsilon transitions.
If we assume that this lemma is true, then the proof that M' is equivalent to M is straightforward: Consider any string w
*. If w L(M) (i.e., the original machine M accepts w) then the following two statements must be true:
1. The original machine M, when started in its start state, can consume w and end up in a final state. This must be true
given the definition of what it means for a machine to accept a string.
2.
(E(s), w) |-*M' (Q, ) for some Q containing some f F. In other words, the new machine, when started in its start
state, can consume w and end up in one of its final states. This follows from the lemma, which is more general and
describes a computation from any state to any other. But if we use the lemma and let q equal s (i.e., M begins in its
start state) and p = f for some f F (i.e., M ends in a final state), then we have that the new machine M', when started
in its start state, E(s), will consume w and end in a state that contains f. But if M' does that, then it has ended up in
one of its final states (by the definition of K' in step 5 of the algorithm). So M' accepts w (by the definition of what it
means for a machine to accept a string). Thus M' accepts precisely the same set of strings that M does.
Now all we have to do is to prove the lemma. What the lemma is saying is that we've built M' from M in such a way that
the computations of M' mirror those of M and guarantee that the two machines accept the same strings. But of course we
didn't build M' to perform an entire computation. All we did was to describe how to construct '. In other words, we
defined how individual steps of the computation work. What we need to do now is to show that the individual steps, when
taken together, do the right thing for strings of any length. The obvious way to do that, since we know what happens one
step at a time, is to prove the lemma by induction on |w|.
We must first prove that the lemma is true for the base case, where |w| = 0 (i.e., w = ). To do this, we actually have to do
two proofs, one to establish the if part of the lemma, and the other to establish the only if part:
Basis step, if part: Prove (q, w) |-*M (p, ) if (E(q), w) |-*M' (P, ) for some P K' that contains p. Or, turning it around to
make it a little clearer,
[ (E(q), w) |-*M' (P, ) for some P K' that contains p ] (q, w) |-*M (p, )
If |w| = 0, then M' makes no moves. So it must end in the same state it started in, namely E(q). If we're told that it ends in
some state that contains p, then p E(q). But, given our definition of E(x), that means exactly that, in the original
machine M, p is reachable from q just be following transitions, which is exactly what we need to show.
Basis step, only if part: Recall that only if is equivalent to implies. So now we need to show:
[ (q, w) |-*M (p, ) ] (E(q), w) |-*M' (P, ) for some P K' that contains p
If |w| = 0, and the original machine M goes from q to p with only w as input, it must go from q to p following just
transitions. In other words p E(q). Now consider the new machine M'. It starts in E(q), the set state that includes all
the states that are reachable from q via transitions. Since the new machine is deterministic, it will make no moves at all
if its input is . So it will halt in exactly the same state it started in, namely E(q). Since we know that p E(q), we know
that M' has halted in a set state that includes p, which is exactly what we needed to show.
Next we'll prove that if the lemma is true for all strings w of length k, k 0, then it is true for all strings of length k + 1.
Considering strings of length k + 1, we know that we are dealing with strings of a least one character. So we can rewrite
any such string as zx, where x is a single character and z is a string of length k. The way that M and M' process z will thus
be covered by the induction hypothesis. We'll use our definition of ', which specifies how each individual step of M'
operates, to show that, assuming that the machines behave correctly for the first k characters, they behave correctly for the
last character also and thus for the entire string of length k + 1. Recall our definition of ':
'(Q, a) = {E(p) : p K and (q, a, p) for some q Q}.
To prove the lemma, we must show a relationship between the behavior of:
M:
(q, w) |-*M (p, ), and
M':
(E(q), w) |-*M' (P, ) for some P K' that contains p
Rewriting w as zx, we have
M:
(q, zx) |-*M (p, )
M':
(E(q), zx) |-*M' (P, ) for some P K' that contains p
Breaking each of these computations into two pieces, the processing of z followed by the processing of x, we have:
M:
(q, zx) |-*M (si, x) |-*M (p, )
M':
(E(q), zx) |-*M' (S, x) |-*M' (P, ) for some P K' that contains p
In other words, after processing z, M will be in some set of states si, and M' will be in some state, which we'll call S.
Again, we'll split the proof into two parts:
Induction step, if part:
[ (E(q), zx) |-*M' (S, x) |-*M' (P, ) for some P K' that contains p ] (q, zx) |-*M (si, x) |-*M (p, )
If, after reading z, M' is in state S, we know, from the induction hypothesis, that the original machine M, after reading z,
must be in some set of states si and that S is precisely that set. Now we just have to describe what happens at the last step
when the two machines read x. If we have that M', starting in S and reading x lands in P, then we know, from the
definition of ' above, that P contains precisely the states that M could land in after starting in any si and reading x. Thus
if p P, p must be a state that M could land in.
Induction step, only if part:
(q, zx) |-*M (si, x) |-*M (p, ) (E(q), zx) |-*M' (S, x) |-*M' (P, ) for some P K' that contains p
By the induction hypothesis, we know that if M, after processing z, can reach some set of states si, then S (the state M' is in
after processing z) must contain precisely all the si's. Knowing that, and our definition of ', we know that from S, reading
x, M' must be in some set state P that contains precisely the states that M can reach starting in any of the si's, reading x,
and then following all transitions. So, after consuming w (zx), M', when started in E(q) must end up in a state P that
contains all and only the states p that M, when started in q, could end up in.
This theorem is a very useful result when we're dealing with FSAs. It's true that, by and large, we want to build
deterministic machines. But, because we have provided a constructive proof of this theorem, we know that we can design
a deterministic FSA by first describing a nondeterministic one (which is sometimes a much simpler task), and then
applying our algorithm to construct a corresponding deterministic one.
3 Generating Regular Expressions from Finite State Machines
4 The Pumping Lemma for Regular Languages

4.1 What Does the Pumping Lemma Tell Us?
The pumping lemma is a powerful technique for showing that a language is not regular. The lemma describes a property
that must be true of any language if it is regular. Thus, if we can show that some language L does not possess this
property, then we know that L is not regular.
The key idea behind the pumping lemma derives from the fact that every regular language is recognizable by some FSM
M (actually an infinite number of finite state machines, but we can just pick any one of them for our purposes here). So
we know that, if a language L is regular, then every string s in L must drive M from the start state to some final state.
There are only two ways this can happen:
1. s could be short and thus drive M from the start state to a final state without traversing any loops. By short we mean
that if M has N states, then |s| < N. If s were any longer, M would have to visit some state more than once while
processing s; in other words it would loop.
2. s could be long and M could traverse at least one loop. If it does that, then observe that another way to take M from
the start state to the same final state would be to skip the loop. Some shorter string w in L would do exactly that. Still
further ways to take M from the start state to the final state would be to traverse the loop two times, or three times, or
any number of times. An infinite set of longer strings in L would do this.
Given some regular language L, we know that there exists an infinite number of FSMs that accept L, and the pumping
lemma relies on the existence of at least one such machine M. In fact, it needs to know the number of states in M. But
what if we don't have M handy? No problem. All we need is to know that M exists and that it has some number of states,
which we'll call N. As long as we make no assumptions about what N is, other than that it is at least 1, we can forge ahead
without figuring out what M is or what N is.
The pumping lemma tells us that if a language L is regular, then any sufficiently long (as defined above) string s in L must
be "pumpable". In other words, there is some substring t of s that corresponds to a loop in the recognizing machine M.
Any string that can be derived from s either by pumping t out once (to get a shorter string that will go through the loop one
fewer times) or by pumping t in any number of times (to get longer strings that will go through the loop additional times)
must also be in L since they will drive M from its start state to its final state. If we can show that there is even one
sufficiently long string s that isn't pumpable, then we know that L must not be regular.
You may be wondering why we can only guarantee that we can pump out once, yet we must be able to pump in an
arbitrary number of times. Clearly we must be able to pump in an arbitrary number of times. If there's a loop in our
machine M, there is no limit to the number of times we can traverse it and still get to a final state. But why only once for
pumping out? Sometimes it may in fact be possible to pump out more. But the lemma doesn't require that we be able to.
Why? When we pick a string s that is "sufficiently long", all we know is that it is long enough that M must visit at least
one state more than once. In other words, it must traverse at least one loop of length one at least once. It may do more,
but we can't be sure of it. So the only thing we're guaranteed is that we can pump out the one pass through the loop that
we're sure must exist.
Pumping Lemma:
N 1, such that
strings w, where |w| N,
x, y, z, such that
w = xyz, and
|xy| N, and
y , and
q 0, xyqz is in L.
10
The lemma we've just stated is sometimes referred to as the Strong Pumping Lemma. That's because there is a weaker
version that is much less easy to use, yet no easier to prove. We won't say anything more about it, but at least now you
know what it means if someone refers to the Strong Pumping Lemma.
4.2 Using the Pumping Lemma

The key to using the pumping lemma correctly to prove that a language L is not regular is to understand the nested
quantifiers in the lemma. Remember, our goal is to show that our language L fails to satisfy the requirements of the
lemma (and thus is not regular). In other words, we're looking for a counterexample. But when do we get to pick any
example we want and when do we have to show that there is no example? The lemma contains both universal and
existential quantifiers, so we'd expect some of each. But the key is that we want to show that the lemma does not apply to
our language L. So we're essentially taking the not of it. What happens when we do that? Remember two key results
from logic:
x P(x) = x P(x)
x P(x) = x P(x)
So if the lemma says something must be true for all strings, we show that there exists at least one string for which it's false.
If the lemma says that there exists some object with some properties, we show that all possible objects fail to have those
properties. More specifically:
At the top level, the pumping lemma states
L regular N 1, P(L, N), where P is the rest of the lemma (think of P as the Pumpable property).
To show that the lemma does not correctly describe our language L, we must show
(N N 1, P(L, N)), or, equivalently,
N P(L, N)
The lemma asserts first that there exists some magic number N, which defines what me mean by "sufficiently long." The
lemma doesnt tell us what N is (although we know it derives from the number of states in some machine M that accepts
L). We need to show that no such N exists. So we don't get to pick a specific N to work with. Instead:
We must carry N through the rest of what we do as a variable and make no assumptions about it.
Next, we must look inside the pumpable property. The lemma states that every string that is longer than N must be
pumpable. To prove that that isn't true of L, all we have to do is to find a single long string in L that isn't pumpable. So:
We get to pick a string w.
Next, the lemma asserts that there is a way to carve up our string into substrings x, y, and z such that pumping works. So
we must show that there is no such x, y, z triple. This is the tricky part. To show this, we must enumerate all logically
possible ways of carving up w into x, y, and z. For each such possibility, we must show that at least one of the pumping
requirements is not satisfied. So:
We don't get to pick x, y, and z. We must show that pumping fails for all possible x, y, z triples.
Sometimes, we can show this easily without actually enumerating ways of carving up w into x, y, and z. But in other
cases, it may be necessary to enumerate two or more possibilities.
Let's look at an example. The classic one is L = axbx is not regular. Intuitively, we knew it couldn't be. To decide
whether a string is in L, we have to count the a's, and then compare that number to the number of b's. Clearly no machine
with a finite number of states can do that. Now we're actually in a position to prove this. We show that for any value of N
we'll get a contradiction. We get to choose any w we want as long as it's length is greater than or equal to N. Let's choose
w to be be aNbN. Next, we must show that there is no x, y, z triple with the required properties:
|xy| N,
y ,
q 0, xyqz is in L.
Suppose there were. Then it might look something like this (this is just for illustration):
1
|
2
11
Don't take this picture to be making any claim about what x, y, and z are. But what the picture does show is that w is
composed of two regions:
1. The initial segment, which is all a's.
2. The final segment, which is all b's.
Typically, as we attempt to show that there is no x, y, z triple that satisfies all the conditions of the pumping lemma, what
we'll do is to consider the ways that y can be spread within the regions of w. In this example, we observe immediately that
since |xy| N, y must be ag for some g 1. (In other words, y must lie completely within the first region.) Now there's just
one case to consider. Clearly we'll add just a's as we pump, so there will be more a's than b's, so we'll generate strings that
are not in L. Thus w is not pumpable, we've found a contradiction, and L is not regular.
The most common mistake people make in applying the pumping lemma is to show a particular x, y, z triple that isn't
pumpable. Remember, you must show that all such triples fail to be pumpable.
Suppose you try to apply the pumping lemma to a language L and you fail to find a counterexample. In other words, every
string w that you examine is pumpable. What does that mean? Does it mean that L is regular? No. It's true that if L is
regular, you will be unable to find a counterexample. But if L isn't regular, you may fail if you just don't find the right w.
In other words, even in a non regular language, there may be plenty of strings that are pumpable. For example, consider
L = axbx ay. In other words, if there are any b's there must be the same number of them as there are a's, but it's also okay
just to have a's. We can prove that this language is not pumpable by choosing w = aNbN, just as we did for our previous
example, and the proof will work just as it did above. But suppose that were less clever. Let's choose w = aN. Now again
we know that y must be ag for some g 1. But now, if we pump y either in or out, we still get strings in L, since all strings
that just contain a's are in L. We haven't proved anything.
Remember that, when you go to apply the pumping lemma, the one thing that is in your control is the choice of w. As you
get experience with this, you'll notice a few useful heuristics that will help you find a w that is easy to work with:
1. Choose w so that there are distinct regions, each of which contains only one element of . When we considered L =
axbx, we had no choice about this, since every element of L (except ) must have a region of a's followed by a region
of b's. But suppose we were interested in L' = {w {a, b}*: w contains an equal number of a's and b's}. We might
consider choosing w = (ab)N. But now there are not clear cut regions. We won't be able to use pumping successfully
because if y = ab, then we can pump to our hearts delight and we'll keep getting strings in L. What we need to do is to
choose aNbN, just as we did when we were working on L. Sure, L' doesn't require that all the a's come first. But
strings in which all the a's do come first are fine elements of L', and they produce clear cut regions that make the
pumping lemma useful.
2. Choose w so that the regions are big enough that there is a minimal number of configurations for y across the regions.
In particular, you must pick w so that it has length at least N. But there's no reason to be parsimonious. For example,
when we were working on axbx, we could have chosen w = aN/2bN/2. That would have been long enough. But then we
couldn't have known that y would be a string of a's. We would have had to consider several different possibilities for
y. (You might want to work this one out to see what happens.) It will generally help to choose w so that each region
is of length at least N.
3. Whenever possible, choose w so that there are at least two regions with a clear boundary between them. In particular,
you want to choose w so that there are at least two regions that must be related in some way (e.g., the a region must be
the same length as the b region). If you follow this rule and rule 2 at the same time, then you'll be assured that as you
pump y, you'll change one of the regions without changing the other, thus producing strings that aren't in your
language.
The pumping lemma is a very powerful tool for showing that a language isn't regular. But it does take practice to use it
right. As you're doing the homework problems for this section, you may find it helpful to use the worksheet that appears
on the next page.
12
Using the Pumping Lemma for Regular Languages

There exists an N 1, (Just call it N) such that
for all strings w, where |w| N,
(Since true for all w, it must be true for any particular one, so you pick w)
(Hint: describe w in terms of N)
there exist x, y, z, such that w = xyz
and |xy| N,
and y ,
and for all q 0, xyqz is in L.
(Since must hold for all y, we show that it cant hold for any y that
meets
the requirements: |xy| N, and y . To do this:
Write out w:
List all the possibilities for y:

[1]
[2]
[3]
[4]
For each possibility for y, xyqz must be in L, for all q. So:
For each possibility for y, find some value of q such that xyqz is not in L. Generally q
will be either 0 or 2.
y
[1]
[2]
[3]
[4]
Q.E.D.
13

1 Context-Free Grammars
Suppose we want to generate a set of strings (a language) L over an alphabet . How shall we specify our language? One
very useful way is to write a grammar for L. A grammar is composed of a set of rules. Each rule may make use of the
elements of (which we'll call the terminal alphabet or terminal vocabulary), as well as an additional alphabet, the nonterminal alphabet or vocabulary. To distinguish between the terminal alphabet and the non-terminal alphabet, we will use
lower-case letters: a, b, c, etc. for the terminal alphabet and upper-case letters: A, B, C, S, etc. for the non-terminal alphabet.
(But this is just a convention. Any character can be in either alphabet. The only requirement is that the two alphabets be
disjoint.)
A grammar generates strings in a language using rules, which are instructions, or better, licenses, to replace some nonterminal symbol by some string. Typical rules look like this:
S ASa, B aB, A SaSSbB.
In context-free grammars, rules have a single non-terminal symbol (upper-case letter) on the left, and any string of terminal
and/or non-terminal symbols on the right. So even things like A A and B are perfectly good context-free grammar
rules. What's not allowed is something with more than one symbol to the left of the arrow: AB a, or a single terminal
symbol: a Ba, or no symbols at all on the left: Aab. The idea is that each rule allows the replacement of the symbol
on its left by the string on its right. We call these grammars context free because every rule has just a single nonterminal on
its left. We cant add any contextual restrictions (such as aAa). So each replacement is done independently of all the others.
To generate strings we start with a designated start symbol often S (for "sentence"), and apply the rules as many times as we
please whenever any one is applicable. To get this process going, there will clearly have to be at least one rule in the
grammar with the start symbol on the left-hand side. (If there isn't, then the grammar won't generate any strings and will
therefore generate , the empty language.) Suppose, however, that the start symbol is S and the grammar contains both the
rules S AB and S aBaa. We may apply either one, producing AB as the "working string" in the first case and aBaa in
the second.
Next we need to look for rules that allow further rewriting of our working string. In the first case (where the working string is
AB), we want rules with either A or B on the left (any non-terminal symbol of the working string may be rewritten by rule at
any time); in the latter case, we will need a rule rewriting B. If, for example, there is a rule B aBb, then our first working
string could be rewritten as AaBb (the A stays, of course, awaiting its chance to be replaced), and the second would become
aaBbaa.
How long does this process continue? It will necessarily stop when the working string has no symbols that can be replaced.
This would happen if either:
(1) the working string consists entirely of terminal symbols (including, as a special case, when the working string is , the
empty string), or
(2) there are non-terminal symbols in the working string but none appears on the left-hand side of any rule in the grammar
(e.g., if the working string were AaBb, but no rule had A or B on the left).
In the first case, but not the second, we say that the working string is generated by the grammar. Thus, a grammar generates,
in the technical sense, only strings over the terminal alphabet, i.e., strings in *. In the second case, we have a blocked or
non-terminated derivation but no generated string.
It is also possible that in a particular case neither (1) nor (2) is achieved. Suppose, for example, the grammar contained only
the rules S Ba and B bB, with S the start symbol. Then using the symbol to connect the steps in the rewriting
process, all derivations proceed in the following way:
S Ba bBa bbBa bbbBa bbbbBa ...
The working string is always rewriteable (in only one way, as it happens), and so this grammar would not produce any
terminated derivations, let alone any terminated derivations consisting entirely of terminal symbols (i.e., generated strings).
Thus this grammar generates the language .
Now let us look at our definition of a context-free grammar in a somewhat more formal way. A context-free grammar (CFG)
G consists of four things:
(1) V, a finite set (the total alphabet or vocabulary), which contains two subsets, (the terminal symbols, i.e., the ones that
will occur in strings of the language) and V - (the nonterminal symbols, which are just working symbols within the
grammar).
(2) , a finite set (the terminal alphabet or terminal vocabulary).
(3) R, a finite subset of (V - ) x V*, the set of rules. Although each rule is an ordered pair (nonterminal, string), well
generally use the notion nonterminal string to describe our rules.
(4) S, the start symbol or initial symbol, which can be any member of V - .
For example, suppose G = (V, , R, S), where
V = {S, A, B, a, b}, = {a, b}, and R = {S AB, A aAa, A a, B Bb, B b}
Then G generates the string aaabb by the following derivation:
(1)
S AB aAaB aAaBb aaaBb aaabb
Formally, given a grammar G, the two-place relation on strings called "derives in one step" and denoted by (or by G if
we want to remind ourselves that the relation is relative to G) is defined as follows:
(u, v) iff strings w, x, y V* and symbol A (V - ) such that u = xAy, v = xwy, and (A w) R.
In words, two strings stand in the "derives in one step" relation for a given grammar just in case the second can be produced
from the first by rewriting a single non-terminal symbol in a way allowed by the rules of the grammar.
(u, v) is commonly written in infix notation, thus: u v.
This bears an obvious relation to the "yields in one step" relation defined on configurations of a finite automaton. Recall that
there we defined the "yields in zero or more steps" relation by taking the reflexive transitive closure of the yields in one step
relation. Well do that again here, giving us "yields in zero or more steps" denoted by * (or G*, to be explicit), which
holds of two strings iff the second can be derived from the first by finitely many successive applications of rules of the
grammar. In the example grammar above:
S AB, and therefore also S * AB.

S * aAaB, but not S aAaB (since aAaB cannot be derived from S in one step).
A aAa and A :* aAa (This is true even though A itself is not derivable from S. If this is not clear, read the
definitions of and * again carefully.)
S * S (taking zero rule applications), but not S S (although the second would be true if the grammar happened to
contain the rule S S, a perfectly legitimate although rather useless rule). Note carefully the difference between , the
connective used in grammar rules, versus and *, indicators that one string can be derived from another by means of
the rules.
Formally, given a grammar G, we define a derivation to be any sequence of strings

w0 w1 wn
In other words, a derivation is a finite sequence of strings such that each string, except the first, is derivable in one step from
the immediately preceding string by the rules of the grammar. We can also refer to it as a derivation of wn from w0. Such a
derivation is said to be of length n, or to be a derivation of n steps. (1) above is a 5-step derivation of aaabb from S according
to the given grammar G.
Similarly, A aAa is a one-step derivation of aAa from A by the grammar G. (Note that derivations do not have to begin
with S, nor indeed do they have to begin with a working string derivable from S. Thus, AA aAaA aAaa is also a wellformed derivation according to G, and so we are entitled to write AA * aAaa).
The strings generated by a grammar G are then just those that are (i) derivable from the start symbol, and (ii) composed
entirely of terminal symbols. That is, G = (V, , R, S) generates w iff w * and S * w. Thus, derivation (l) above shows
that the string aaabb is generated by G. The string aAa, however, is not generated by G, even though it is derivable from S,
because it contains a non-terminal symbol. It may be a little harder to see that the string bba is not generated by G. One
would have to convince oneself that there exists no derivation beginning with S and ending in bba according to the rules of G.
(Question: Is this always determinable in general, given any arbitrary context-free grammar G and string w? In other words,
can one always tell whether or not a given w is "grammatical" according to G? We'll find out the answer to this later.)
The language generated by a grammar G is exactly the set of all strings generated--no more and no less. The same remarks
apply here as in the case of regular languages: a grammar generates a language iff every string in the language is generated by
the grammar and no strings outside the language are generated.
And now our final definition (for this section). A language L is context free if and only if there exists a context-free grammar
that generates it.
Our example grammar happens to generate the language a(aa)*bb*. To prove this formally would require a somewhat
involved argument about the nature of derivations allowed by the rules of G, and such a proof would not necessarily be easily
extended to other grammars. In other words, if you want to prove that a given grammar generates a particular language, you
will in general have to make an argument which is rather specific to the rules of the grammar and show that it generates all the
strings of the particular language and only those. To prove that a grammar generates a particular string, on the other hand, it
suffices to exhibit a derivation from the start symbol terminating in that string. (Question: if such a derivation exists, are we
guaranteed that we will be able to find it?) To prove that a grammar does not generate a particular string, we must show that
there exists no derivation that begins with the start symbol and terminates in that string. The analogous question arises here:
when can we be sure that our search for such a derivation is fruitless and be called off? (We will return to these questions
later.)
2 Designing Context-Free Grammars

To design a CFG for a language, a helpful heuristic is to imagine generating the strings from the outside in to the middle. The
nonterminal that is currently "at work" should be thought of as being at the middle of the string when you are building a string
where two parts are interdependent. Eventually the "balancing" regions are done being generated, and the nonterminal that's
been doing the work will give way to a different nonterminal (if there's more stuff to be done between the regions just
produced) or to some terminal string (often ) otherwise. If parts of a string have nothing to do with each other, do not try to
produce them both with one rule. Try to identify the regions of the string that must be generated in parallel due to a
correlation between them: they must be generated by the same nonterminal(s). Regions that have no relation between each
other can be generated by different nonterminals (and usually should be.)
Here is a series of examples building in complexity. For each one you should generate a few sample strings and build parse
trees to get an intuition about what is going on. One notational convention that well use to simplify writing language
descriptions: If a description makes use of a variable (e.g., an), theres an implied statement that the description holds for all
integer values 0.
Example 1: The canonical example of a context-free language is L = anbn, which is generated by the grammar
G = ({S, a, b}, {a, b}, R, S) where R = {S aSb, S }.
Each time an a is generated, a corresponding b is generated. They are created in parallel. The first a, b pair created is the
outermost one. The nonterminal S is always between the two regions of a's and b's. Clearly any string anbn L is produced
by this grammar, since
n
n n
S
aSb L"""
a nSb
"""
! a b ,
n steps
Therefore L L(G).
We must also check that no other strings not in anbn are produced by the grammar, i.e., we must confirm that L(G) L.
Usually this is easy to see intuitively, though you can prove it by induction, typically on the length of a derivation. For
illustration, we'll prove L(G) L for this example, though in general you won't need to do this in this class.
Claim: x, x L(G) x L. Proof by induction on the length of the derivation of G producing x.
Base case: Derivation has length 1. Then the derivation must be S , and L.
Induction step: Assume all derivations of length k produce a string in L, and show the claim holds for derivations of length k
+ 1. A derivation of length k + 1 looks like:
S aSb
axb
"
"L ""
!
k steps
for some terminal string x such that S * x. By the induction hypothesis, we know that x L (since x is produced by a
derivation of length k), and so x = anbn for some n (by definition of L). Therefore, the string axb produced by the length k + 1
derivation is axb = aanbnb = an+1bn+1 L. Therefore by induction, we have proved L(G) L.
Example 2: L = {xy: |x| = |y| and x {a, b}* and y {c, d}*}. (E.g., , ac, ad, bc, bd, abaccc L. ) Here again we will
want to match a's and b's against c's and d's in parallel. We could use two strategies. In the first,
G = ({S, a , b, c, d}, {a , b , c, d}, R , S) where R = {S aSc, S aSd, S bSc, S bSd, S }.
This explicitly enumerates all possible pairings of a, b symbols with c, d symbols. Clearly if the number of symbols allowed
in the first and second halves of the strings is n, the number of rules with this method is n2 + 1, which would be inefficient for
larger alphabets. Another approach is:
G = ({S, L, R, a, b, c, d}, {a, b, c, d}, R, S) where R = {S LSR, S , L a, L b, R c, R d}.
(Note that L and R are nonterminals here.) Now the number of rules is 2n+2.
Example 3: L = {wwR : w {a, b}*}. Any string in L will have matching pairs of symbols. So it is clear that the CFG G =
({S, a, b}, {a, b}, R, S}, where R = {S aSa, S bSb, S } generates L, because it produces matching symbols in
parallel. How can we prove L(G) = L? To do half of this and prove that L L(G) (i.e, every element of L is generated by
G), we note that any string x L must either be (which is generated by G (since S )), or it must be of the form awa or
bwb for some w L. This suggests an induction proof on strings:
Claim: x, x L x L(G). Proof by induction on the length of x.
Base case: L and L(G).
Induction step: We must show that if the claim holds for all strings of length k, it holds for all strings of length k+2 (We
use k+2 here rather than the more usual k+1 because, in this case, all strings in L have even length. Thus if a string in L has
length k, there are no strings in L of length k +1.). If |x| = k+2 and x L, then x = awa or x = bwb for some w L. |w| = k,
so, by the induction hypothesis, w L(G). Therefore S * w. So either S aSa * awa, and x L(G), or S bSb *
bwb, and x L(G).
Conversely, to prove that L(G) L, i.e., that G doesn't generate any bad strings, we would use an induction on the length of a
derivation.
Claim: x, x L(G) x L. Proof by induction on length of derivation of x.
Base case: length 1. S and L.
Induction step: Assume the claim is true for derivations of length k, and show the claim holds for derivations of length k+1. A
derivation of length k + 1 looks like:
S aSa
awa
"
"L
""
!
k steps
or like
S bSb
L
bwb
""
""
!
k steps
for some terminal string w such that S * w. By the induction hypothesis, we know that w L (since w is produced by a
derivation of length k), and so x = awa is also in L, by the definition of L. (Similarly for the second class of derivations that
begin with the rule S bSb.)
As our example languages get more complex, it becomes harder and harder to write detailed proofs of the correctness of our
grammars and we will typically not try to do so.
Example 4: L = {anb2n}. You should recognize that b2n = (bb)n, and so this is just like the first example except that instead of
matching a and b, we will match a and bb. So we want
G = ({S, a, b}, {a, b}, R, S} where R = {S aSbb, S }.
If you wanted, you could use an auxiliary nonterminal, e.g.,

G = ({S, B, a, b}, {a, b}, R, S} where R = {S aSB, S , B bb}, but that is just cluttering things up.
Example 5: L = {anbncm}. Here, the cm portion of any string in L is completely independent of the anbn portion, so we should
generate the two portions separately and concatenate them together. A solution is
G = ({S, N, C, a, b, c}, {a, b, c}, R, S} where R = {S NC, N aNb, N , C cC, C }.
This independence buys us freedom: producing the c's to the right is completely independent of making the matching anbn, and
so could be done in any manner, e.g., alternate rules like
C CC, C c, C
would also work fine. Thinking modularly and breaking the problem into more manageable subproblems is very helpful for
designing CFG's.
Example 6: L = {anbmcn}. Here, the bm is independent of the matching ancn. But it cannot be generated "off to the side." It
must be done in the middle, when we are done producing a and c pairs. Once we start producing the b's, there should be no
more a, c pairs made, so a second nonterminal is needed. Thus we have
G = ({S, B, a, b, c}, {a, b, c}, R, S} where R = {S , S aSc, S B, B bB, B }.
We need the rule S . We dont need it to end the recursion on S. We do that with S B. And we have B . But if n
= 0, then we need S so we dont generate any ac pairs.
Example 7: L = a*b*. The numbers of a's and b's are independent, so there is no reason to use any rules like S aSb which
create an artificial correspondence. We can independently produce a's and b's, using
G = ({S, A, B, a, b}, {a, b}, R, S} where R = {S AB, A aA, A , B bB, B }
But notice that this language is not just context free. It is also regular. So we expect to be able to write a regular grammar
(recall the additional restrictions that apply to rules in a regular grammar) for it. Such a grammar will produce a's, and then
produce b's. Thus we could write
G = ({S, B, a, b}, {a, b}, R, S} where R = {S , S aS, S bB, B bB, B }.
Example 8: L = {ambn: m n}. There are several ways to approach this one. One thing we could do is to generate a's and b's
in parallel, and also freely put in extra b's. This intuition yields
G = ({S, a, b}, {a, b}, R, S} where R = {S aSb, S Sb, S }.
Intuitively, this CFG lets us put in any excess b's at any time in the derivation of a string in L. Notice that to keep the S
between the two regions of a's and b's, we must use the rule S Sb; replacing that rule with SbS would be incorrect,
producing bad strings (allowing extra b's to be intermixed with the a's).
Another way to approach this problem is to realize that {ambn : m n}= {ambm+k : k 0} = {ambkbm: k 0}. Therefore, we
can produce a's and b's in parallel, then when we're done, produce some more b's. So a solution is
G = ({S, B, a, b}, {a, b}, R, S} where R = {S , S aSb, S B, B bB, B }.

Intuitively, this CFG produces the matching a, b pairs, then any extra b's are generated in the middle. Note that this strategy
requires two nonterminals since there are two phases in the derivation using this strategy.
Since {ambn : m n}= {ambkbm: k 0}= {ambmbk: k 0}, there is a third strategy: generate the extra b's to the right of the
balanced ambm string. Again the generation of the extra b's is now separated from the generation of the matching portion, so
two distinct nonterminals will be needed. In addition, since the two parts are concatenated rather than imbedded, we'll need
another nonterminal to produce that concatenation. So we've got
G = ({S, M, B, a, b}, {a, b}, R, S} where R = {S MB, M aMb, M , B bB, B }.
Example 9: L = {a 1 b 1 L a k b k : k 0} . E.g., , abab, aabbaaabbbabab L. Note that L = {anbn}* which gives a clue
how to do this. We know how to produce matching strings anbn, and we know how to do concatenation of strings. So a
solution is
G = ({S, M, a, b}, {a, b}, R, S} where R = {S MS, S , M aMb, M }.
n
Any string x = a b L a b
n1
n1
nk
nk
L can be generated by the canonical derivation

S
/* k applications of rule S MS */
*
M kS
/* one application of rule S */
Mk
* /* n1 applications of rule M aMb */

a n Mb n M k 1
1
/* one application of rule M */
a n b n M k 1
1
* /* repeating on k-1 remaining M */
n n
n n
a 1b L a k b k
Of course the rules could be applied in many different orders.
3 Derivations and Parse Trees

Let's again look at the very simple grammar G = (V, , R, S), where
V = {S, A, B, a, b}, = {a, b}, and R = {S AB, A aAa, A a, B Bb, B b}
As we saw in an earlier section, G can generate the string aaabb by the following derivation:
(1)
S AB aAaB aAaBb aaaBb aaabb
Now let's consider the fact that there are other derivations of the string aaabb using our example grammar:
(2)
S AB ABb Abb aAabb aaabb
(3)
S AB ABb aAaBb aAabb aaabb
(4)
S AB ABb aAaBb aaaBb aaabb
(5)
S AB aAaB aaaB aaaBb aaabb
(6)
S AB aAaB aAaBb aAabb aaabb
If you examine all these derivations carefully, you will see that in each case the same rules have been used to rewrite the same
symbols; they differ only in the order in which those rules were applied. For example, in (2) we chose to rewrite the B in
ABb as b (producing Abb) before rewriting the A as aAa, whereas in (3) the same.processes occur in the opposite order. Even
though these derivations are technically different (they consist of distinct sequences of strings connected by ) it seems that
in some sense they should all count as equivalent. This equivalence is expressed by the familiar representations known as
derivation trees or parse trees.
The basic idea is that the start symbol of the grammar becomes the root of the tree. When this symbol is rewritten by a
grammar rule S x1x2...xn, we let the tree "grow" downward with branches to each of the new nodes x1, x2, , xn; thus:
S
x1
x2
...
xn
When one of these xi symbols is rewritten, it in turn becomes the "mother" node with branches extending to each of its
"daughter" nodes in a similar fashion. Each of the derivations in (1) through (6) would then give rise to the following parse
tree:
S
A
a
B
a
A note about tree terminology: for us, a tree always has a single root node, and the left-to-right order of nodes is significant;
i.e., X is not the same tree as X.
Y
The lines connecting nodes are called branches, and their top-to-bottom orientation is also significant. A mother node is
connected by a single branch to each of the daughter nodes beneath it. Nodes with the same mother are called sisters, e.g.,
the topmost A and B in the tree above are sisters, having S as mother.
Nodes without daughters are called leaves; e.g., each of the nodes labelled with a lower-case letter in the tree above. The
string formed by the left-to-right sequence of leaves is called the yield (aaabb in the tree above).
It sometimes happens that a grammar allows the derivations of some string by nonequivalent derivations, i.e., derivations that
do not reduce to the same parse tree. Suppose, for example, the grammar contained the rules S A, S B, A b and B
b Then the two following derivations of the string b correspond to the two distinct parse trees shown below.
SAb
SBb
A grammar with this property is said to be ambiguous. Such ambiguity is highly undesirable in grammars of programming
languages such as C, LISP, and the like, since the parse tree (the syntactic structure) assigned to a string determines its
translation into machine language and therefore the sequence of commands to be executed. Designers of programming
languages, therefore, take great pains to assure that their grammars (the rules that specify the well-formed strings of the
language) are unambiguous. Natural languages, on the other hand, are typically rife with ambiguities (cf. "They are flying
planes," "Visiting relatives can be annoying," "We saw her duck," etc.), a fact that makes computer applications such as
machine translation, question-answering systems, and so on, maddeningly difficult.
More Examples of Context-Free Grammars and Parse Trees:

(1) G = (V, , R, S), where
V = {S, NP, VP, D, N, V, chased, the, dog, cat},
= {chased, the, dog, cat }
R= {S NP VP
NP D N
VP V NP
V chased
N dog
N cat
D the }
(2)
G = (V, , R, S), where

V = {S, a, b}, = {a, b},
R = {S aSb
S
L(G) = {anbn : n 0}
S
a
S
a
S
a
b
b
G = (V, , R, S),
where
V = {S, A, a, b},
= {a, b},
R = {S aS
S bA
A aA
A bS
S
L(G) = {w {a, b}* :
w contains an even
number of b's}
(3)
A
b
S
b
A
b
4 Designing Pushdown Automata

In a little bit, we will prove that for every grammar G, there is push down automaton that accepts L(G). That proof is
constructive. In other words, it describes an algorithm that takes any context-free grammar and constructs the corresponding
PDA. Thus, in some sense, we don't need any other techniques for building PDAs (assuming that we already know how to
build grammars). But the PDAs that result from this algorithm are often highly nondeterministic. Furthermore, simply using
the algorithm gives us no insight into the actual structure of the language we're dealing with. Thus, it is useful to consider
how to design PDA's directly; the strategies and insights are different from those we use to design a CFG for a language, and
any process that increases our understanding can't be all bad ....
In designing a PDA for a language L, one of the most useful strategies is to identify the different regions that occur in the
strings in L. As a rule of thumb, each of these regions will correspond to at least one distinct state in the PDA. In this
respect, PDAs are very similar to finite state machines. So, for example, just as a finite state machine that accepts a*b* needs
two states (one for reading a's and one for reading b's after we're done reading a's), so will a PDA that acceps {anbn) need two
states. Recognizing the distinct regions is part of the art, although it is usually obvious.
Example 1: In really simple cases, there may not be more than one region. For example, consider {anan}. What we realize
here is that we've really got is just the set of all even length strings of a's, i.e., (aa)*. In this case, the "border" between the
first half of the a's and the second half is spurious.
Example 2: L = {anbn}. A good thing to try first here is to make a finite state machine for a*b* to get the basic idea of the use
of regions. (This principle applies to designing PDA's in general: use the design of a finite state machine to generate the
states that are needed just to recognize the basic string structure. This creates the skeleton of your PDA. Then you can add
the appropriate stack operations to do the necessary counting.) Clearly you'll need two states since you want to read a's, then
read b's. To accept L, we also need to count the a's in state one in order to match them against the b's in state two. This gives
the PDA M = ({s, f}, {a, b}, {I}, , s, {f}), where =
{ ((s, a, ), (s, I)),

((s, , ), (f, )),
((f, b, I), (f, e))}.
/* read a's and count them */

/* guess that we're done with a's and ready to start on b's */
/* read b's and compare them to the a's in the stack */
(Notice that the stack alphabet need not be in any way similar to the input alphabet. We could equally well have pushed a's,
but we don't need to.) This PDA nondeterministically decides when it is done reading a's. Thus one valid computation is
(a, aabb, ) |- (s, abb, I) |- (f, abb, I),
which is then stuck and so M rejects along this path. Since a different accepting computation of aabb exists, this is no
problem, but you might want to elimimate the nondeterminism if you are bothered by it. Note that the nondeterminism arises
from the transition; we only want to take it if we are done reading a's. The only way to know that there are no more a's is to
read the next symbol and see that it's ab. (This is analogous to unfolding a loop in a program.) One other wrinkle: L, so
now state s must be final in order to accept . The resulting deterministic PDA is M = ({s, f}, {a, b}, {I}, , s, {s, f}), where
=
{ ((s, a, ), (s, I)),
((s, b, I), (f, )),
/* only go to second phase if there's a b */
((f, b, I), (f, ))}.
/* read b's and compare them to the a's in the stack */
Notice that this DPDA can still get stuck and thus fail, e.g., on input b or aaba (i.e., strings that aren't in L). Determinism for
PDA's simply means that there is at most one applicable transition, not necessarily exactly one.
Example 3: L = {ambmcndn}. Here we have two independent concerns, matching the a's and b's, and then matching the c's and
d's. Again, start by designing a finite state machine for the language L' that is just like L in structure but where we don't care
how many of each letter there are. In other words a*b*c*d*. It's obvious that this machine needs four states. So our PDA
must also have four states. The twist is that we must be careful that there is no unexpected interaction between the two
independent parts ambm and cndn. Consider the PDA M = ({1,2,3,4}, {a,b,c,d}, {I}, , 1, {4}), where =
{ ((1, a, ), (1, I)),
((1, , ), (2, )),

((2, b, I), (2, )),
((2, , ), (3, )),
((3, c, ), (3, I))
((3, , ), (4, ))}.
((4, d, I), (4, ))}.
/* guess that we're ready to quit reading a's and start reading b's */
/* read b's and compare to as */
/* guess that we're ready to quit reading bs and start reading cs */
/* read cs and count them */
/* guess that were ready to quit reading cs and start reading ds */
/* read ds and compare them to cs */
It is clear that every string in L is accepted by this PDA. Unfortunately, some other strings are also, e.g., ad. Why is this?
Because it's possible to go from state 2 to 3 without clearing off all the I marks we pushed for the a's That means that the
leftover Is are available to match ds. So this PDA is accepting the language {ambncpdq : m n and m + p = n + q}, a
superset of L. E.g., the string aabcdd is accepted.
One way to fix this problem is to ensure that the stack is really cleared before we leave phase 2 and go to phase 3; this must be
done using a bottom of stack marker, say B. This gives M = ({s, l, 2, 3, 4}, {a, b, c, d}, {B, I}, , s, {4}), where =
{ ((s, , ), (1, B)),
/* push the bottom marker onto the stack */
((1, a, ), (1, I)),
((1, , ), (2, )),
((2, b, I), (2, )),
((2, , B), (3, )),
/* confirm stack is empty, then get readty to start reading cs */
((3, c, ), (3, I))
((3, , ), (4, ))}.
((4, d, I), (4, ))}.
A different, probably cleaner, fix is to simply use two different symbols for the counting of the a's and the c's. This gives us
M = ({1,2, 3,4}, {a,b,c,d}, {A,C}, , 1, {4}), where =
{ ((1, a, ), (1, A)),
((1, , ), (2, )),
((2, b, A), (2, )),
((2, , ), (3, )),
/* guess that we're ready to quit reading b's and start reading c's */
((3, c, ), (3, C)),
((3, , ), (4, )),
((4, d, C), (4, ))}.
Now if an input has more a's than b's, there will be leftover A's on the stack and no way for them to be removed later, so that
there is no way such a bad string would be accepted.
As an exercise, you want to try making a deterministic PDA for this one.
Example 4: L = {anbn} {bnan). Just as with nondeterministic finite state automata, whenever the language were concerned
with can be broken into cases, a reasonable thing to do is build separate PDAs for the each of the sublanguages. Then we
build the overall machine so that it, each time it sees a string, it nondeterministically guesses which case the string falls into.
(For example, compare the current problem to the simpler one of making a finite state machine for the regular language a*b*
b*a*.) Taking this approach here, we get M = ({s, 1, 2, 3, 4), {a, b}, {I}, , s, {2, 4}), where =
{ ((s, , ), (1, )),
/* guess that this is an instance of anbn */
((s, , ), (3, )),
/* guess that this is an instance of bnan */
((1, a, ), (1, I)),
/* as come first so read and count them */
((1, , ), (2, )),
/* begin the b region following the as */
((2, b, I), (2, )),
/* read bs and compare them to the as */
((3, b, ), (3, I)),
/* bs come first so read and count them */
((3, , ), (4, )),
/* begin the a region following the bs */
((4, a, I), (4, ))}.
/* read as and compare them to the bs */
10
Notice that although L, the start state s is not a final state, but there is a path (in fact two) from s to a final state.
Now suppose that we want a deterministic machine. We can no longer use this strategy. The -moves must be eliminated by
looking ahead. Once we do that, since L, the start state must be final. This gives us M = ({s, 1, 2, 3, 4}, {a, b}, {l}, , s,
{s, 2, 4}), where =
{ ((s, a, ), (1, )),
((s, b, ), (3, )),
((1, a, ), (1, I)),
((1, b, I), (2, )),
((2, b, I), (2, )),
((3, b, ), (3, I)),
((3, a, I), (4, )),
((4, a, I), (4, ))}.
/* if the first character is a, then this is an instance of anbn */

/* if the first character is b, then this is an instance of bnan */
/* as come first so read and count them */
/* begin the b region following the as */
/* read bs and compare them to the as */
/* bs come first so read and count them */
/* begin the a region following the bs */
/* read as and compare them to the bs */
Example 5: L = {wwR : w {a, b}*}. Here we have two phases, the first half and the second half of the string. Within each
half, the symbols may be mixed in any particular order. So we expect that a two state PDA should do the trick. See the
lecture notes for how it works.
Example 6: L = {wwR : w a*b*}. Here the two halves of each element of L are themselves split into two phases, reading
a's, and reading b's. So the straightforward approach would be to design a four-state machine to represent these four phases.
This gives us M = ({1, 2, 3, 4), {a, b}, {a, b), , 1, {4}), where =
{ ((1, a, ), (1, a))
/*push a's*/
((1, , ), (2, )),
((2, b, ), (2, b)),
/* push b's */
((2, , ), (3, )),
/* guess that we're ready to quit reading the first w and start reading wR */
((3, b, b), (3, )),
/* compare 2nd b's to 1st b's */
((3, , ), (4, )),
/* guess that we're ready to quit reading b's and move to the last region of a's */
((4, a, a), (4, ))}
/* compare 2nd a's to 1st a's */
You might want to compare this to the straightforward nondeterministic finite state machine that you might design to accept
a*b*b*a*.
There are various simplifications that could be made to this machine. First of all, notice that L = {ambnbnam}. Next, observe
that bnbn = (bb)n, so that, in effect, the only requirement on the b's is that there be an even number of them. And of course a
stack is not even needed to check that. So an alternate solution only needs three states, giving M = ({1, 2, 3}, {a, b}, {a},
{a}, , 1, {3}), where =
{ ((1, a, ), (1, a))
/*push a's*/
((1, , ), (2, )),
((2, bb, ), (2, )),
/* read bb's */
((2, , ), (3, )),
/* guess that we're ready to quit reading b's and move on the final group of a's */
((3, a, a), (3, ))}.
/* compare 2nd a's to 1st a's */
This change has the fringe benefit of making the PDA more deterministic since there is no need to guess where the middle of
the b's occurs. However, it is still nondeterministic.
So let's consider another modification. This time, we go ahead and push the a's and the b's that make up w. But now we
notice that we can match wR against w in a single phase: the required ordering b*a* in wR will automatically be enforced if
we simply match the input with the stack! So now we have the PDA M= ({1, 2, 3}, {a, b}, {a, b}, , 1, {3}), where =
{ ((1, a, ), (1, a))
/*push a's*/
((1, , ), (2, )),
((2, b, ), (2, b)),
/* push b's */
11
((2, , ), (3, )),

((3, a, a), (3, ))
((3, b, b), (3, ))}.
/* guess that we're ready to quit reading the first w and start reading wR */
/* compare wR to w*/
"
Notice that this machine is still nondeterministic. As an exercise, you might try to build a deterministic machine to accept this
language. You'll find that it's impossible; you've got to be able to tell when the end of the strings is reached, since it's possible
that there aren't any b's in between the a regions. This suggests that there might be a deterministic PDA that accepts L$, and
in fact there is. Interestingly, even that is not possible for the less restrictive language L = {wwR : w {a, b}*} (because
there's no way to tell without guessing where w ends and wR starts). Putting a strong restriction on string format often makes a
language more tractable. Also note that {wwR : w a*b+} is accepted by a determinstic PDA; find such adeterminstic PDA
as an exercise.
Example 7: Consider L = {w {a, b}* : #(a, w) = #(b, w)}. In other words every string in L has the same number of a's as
b's (although the a's and b's can occur in any order). Notice that this language imposes no particular structure on its strings,
since the symbols may be mixed in any order. Thus the rule of thumb that we've been using doesn't really apply here. We
don't need multiple states for multiple string regions. Instead, we'll find that, other than possible bookkeeping states, one
"working" state will be enough.
Sometimes there may be a tradeoff between the degree of nondeterminism in a pda and its simplicity. We can see that in this
example. One approach to designing a PDA to solve this problem is to keep a balance on the stack of the excess a's or b's.
For example, if there is an a on the stack and we read b, then we cancel them. If, on the other hand, there is an a on the stack
and we read another a, we push the new a on the stack. Whenever the stack is empty, we know that we've seen matching
number of a's and b's so far. Let's try to design a machine that does this as deterministically as possible. One approach is M =
({s, q, f}, {a, b}, {a, b, c}, , s, {f}), where =
1
((s, , ), (q, c))
/* Before we do anything else, push a marker, c, on the stack so we'll be able to
tell when the stack is empty. Then leave state s so we don't ever do this again.
2
((q, a, c), (q ,ac))
/* If the stack is empty (we find the bottom c) and we read an a, push c back
and then the a (to start counting a's).
3
((q, a, a), (q, aa))
/* If the stack already has a's and we read an a, push the new one.
4
((q, a, b), (q, ))
/* If the stack has b's and we read an a, then throw away the top b and the new a.
5
((q, b, c), (q, bc))
/* If the stack is empty (we find the bottom c) and we read a b, then
start counting b's.
6
((q, b, b), (q, bb))
/* If the stack already has b's and we read b, push the new one.
7
((q, b, a), (q, ))
/* If the stack has a's and we read a b, then throw away the top a and the new b.
8
((q, , c), (f, ))
/* If the stack is empty then, without reading any input, move to f, the final state.
Clearly we only want to take this transition when we're at the end of the input.
This PDA attempts to solve our problem deterministically, only pushing an a if there is not a b on the stack. In order to tell
that there is not a b, this PDA has to pop whatever is on the stack and examine it. In order to make sure that there is always
something to pop and look at, we start the process by pushing the special marker c onto the stack. (Recall that there is no way
to check directly for an empty stack. If we write just for the value of the current top of stack, we'll get a match no what the
stack looks like.) Notice, though, that despite our best efforts, we still have a nondeterministic PDA because, at any point in
reading an input string, if the number of a's and b's read so far are equal, then the stack consists only of c, and so transition 8
((q, , c), (f, )) may be taken, even if there is remaining input. But if there is still input, then either transition 1 or 5 also
applies. The solution to this problem is to add a terminator to L.
Another thing we could do is to consider a simpler PDA that doesn't even bother trying to be deterministic. Consider M
=({s}, {a, b}, {a, b}, , s, {s}), where =
1
((s, a, ), (s, a))
/* If we read an a, push a.
2
((s, a, b), (s, ))
/* Cancel an input a and a stack b.
3
((s, b, ), (s, b))
/* If we read b, push b.
4
((s, b, a), (s, ))
/* Cancel and input b and a stack a.
Now, whenever we're reading a and b is on the stack, there are two applicable transitions: 1, which ignores the b and pushes
the a on the stack, and 2, which pops the b and throws away the a (in other words, it cancels the a and b against each other).
12
Transitions 3 and 4 do the same two things if we're reading b. It is clear that if we always perform the cancelling transition
when we can, we will accept every string in L. What you might worry about is whether, due to this larger degree of freedom,
we might not also be able to wrongly accept some string not in L. In fact this will not happen because you can prove that M
has the property that, if x is the string read in so far, and y is the current stack contents,
#(a, x) - #(b, x) = #(a, y) - #(b, y).
This formula is an invariant of M. We can prove it by induction on the length of the string read so far: It is clearly true
initially, before M reads any input, since 0 - 0 - 0 - 0. And, if it holds before taking a transition, it continues to hold afterward.
We can prove this as follows:
Let x' be the string read so far and let y' be the contents of the stack at some arbitrary point in the computation. Then let us
see what effect each of the four possible transitions has. We first consider:
((s, a, ), (s, a)): After taking this transition we have that x' = xa and y' = ay. Thus we have
#(a, x') - #(b, x')
= /* x' = xa */
#(a, xa) - #(b, xa)
= /* #(b, xa) = #(b, x)
#(a, xa) - #(b, x)
=
#(a, x) + 1 - #(b, x)
= /* induction hypothesis */
#(a, y) + 1 - #(b, y)
=
#(a, ay) - #(b, ay)
= /* y' = ay */
#(a, y') - #(b, y')
So the invariant continues to be true for x' and y' after the transition is taken. Intuitively, the argument is simply that when this
transition is taken, it increments #(a, x) and #(a, y), preserving the invariant equation. The three other transitions also preserve
the invariant as can be seen similarly:
((s, a, b), (s, )) increments #(a, x) and decrements #(b, y), preserving equality.
((s, b, ), (s, b)) increments #(b, x) and #(b, y), preserving equality.
((s, b, a), (s, )) increments #(b, x) and decrements #(a, y), preserving equality.
Therefore, the invariant holds initially, and taking any transitions continues to preserve it, so it is always true, no matter what
string is read and no matter what transitions are taken. Why is this a good thing to know? Because suppose a string x L is
read by M. Since x L, we know that #(a, x) - #(b, x) 0, and therefore, by the invariant equation, when the whole string x
has been read in, the stack contents y will satisfy #(a, y) - #(b, y) 0 Thus the stack cannot be empty, and x cannot be
accepted, no matter what sequence of transitions is taken. Thus no bad strings are accepted by M.
5 Context-Free Languages and PDA's

Theorem: The context-free languages are exactly the languages accepted by nondeterministic PDA's.
In other words, if you can describe a language with a context-free grammar, you can build a nondeterministic PDA for it, and
vice versa. Note here that the class of context-free languages is equivalent to the class of languages accepted by
nondeterministic PDAs. This is different from what we observed when we were considering regular languages. There we
showed that nondeterminism doesnt buy us any power and that we could build a deterministic finite state machine for every
regular language. Now, as we consider context-free languages, we find that determinism does buy us power: there are
languages that are accepted by nondeterministic PDAs for which no deterministic PDA exists. And those languages are
context free (i.e., they can be described with context-free grammars). So this theorem differs from the similar theorem that we
proved for regular languages and claims equivalence for nondeterministic PDAs rather than deterministic ones.
13
Well prove this theorem by construction in two steps: first well show that, given a context-free grammar G, we can construct
a PDA for L(G). Then well show (actually, well just sketch this second proof) that we can go the other way and construct,
from a PDA that accepts some language L, a grammar for L.
Lemma: Every context-free language is accepted by some nondeterministic PDA.
To prove this lemma, we give the following construction. Given some CFG G = (V, , R, S), we construct an equivalent
PDA M in the following way. M = (K, , , , s, F), where
K = {p, q }
(the PDA always has just 2 states)
s=p
(p is the initial state)
F = {q}
(q is the only final state)
=
(the input alphabet is the terminal alphabet of G)
=V
(the stack alphabet is the total alphabet of G)
contains
(1) the transition ((p, , ), (q, S))
(2) a transition ((q, , A), (q, )) for each rule A in G
(3) a transition ((q, a, a), (q, )) for each a
Notice how closely the machine M mirrors the structure of the original grammar G. M works by using its stack to simulate a
derivation by G. Using the transition created in (1), M begins by pushing S onto its stack and moving to its second state, q,
where it will stay for the rest of its operation. Think of the contents of the stack as Ms expectation for what it must find in
order to have seen a legal string in L(G). So if it finds S, it will have found such a string. But if S could be rewritten as some
other sequence , then if M found it would also have found a string in L(G). All the transitions generated by (2) take care
of these options by allowing M to replace a stack symbol A by a string whenever G contains the rule A . Of course, at
some point we actually have to look at the input. Thats what M does in the transitions generated in (3). If the stack contains
an expectation of some terminal symbol and if the input string actually contains that symbol, M consumes the input symbol
and pops the expected symbol off the stack (effectively canceling out the expectation with the observed symbol). These steps
continue, and if M succeeds in emptying its stack and reading the entire input string, then the input is accepted.
Lets consider an example. Let G = (V = {S, a, b, c}, = {a, b, c}, R = {S aSa, S bSb, S c}, S). This grammar
generates {xcxR : x {a, b}*}. Carrying out the construction we just described for this example CFG gives the following
PDA:
M = ({p, q}, {a, b, c}, {S, a, b, c}, , p, {q}), where
= { ((p, , ), (q, S))
((q, , S), (q, aSa))
((q, , S), (q, bSb))
((q, , S), (q, c))
((q, a, a), (q, ))
((q, b, b), (q, ))
((q, c, c), (q, ))}
Here is a derivation of the string abacaba by G:
(1) S aSa abSba : abaSaba abacaba
And here is a computation by M accepting that same string:
(2)
(p, abacaba, ) |- (q, abacaba, S) |- (q, abacaba, aSa) |- (q, bacaba, Sa) |- (q, bacaba, bSba) |- (q, acaba, Sba) |(q, acaba, aSaba) |- (q, caba, Saba) |- (q, caba, caba) |- (q, aba, aba) |- (q, ba, ba) |- (q, a, a) |- (q, , )
14
If you look at the successive stack contents in computation (2) above, you will see that they are, in effect, tracing out a
derivation tree for the string abacaba:
S
a,
S
b
S
a
b
b
c
M is alternately extending the tree and checking to see if leaves of the tree match the input string. M is thus acting as a topdown parser. A parser is something that determines whether a presented string is generated by a given grammar (i.e., whether
the string is grammatical or well-formed), and, if it is, calculates a syntactic structure (in this case, a parse tree) assigned to
that string by the grammar. Of course, the machine M that we have just described does not in fact produce a parse tree,
although it could be made to do so by adding some suitable output devices. M is thus not a parser but a recognizer. Well
have more to say about parsers later, but we can note here that parsers play an important role in many kinds of computer
applications including compilers for programming languages (where we need to know the structure of each command), query
interpreters for database systems (where we need to know the structure of each user query), and so forth.
Note that M is properly non-deterministic. From the second configuration in (2), we could have gone to (q, abacaba, bSb) or
to (q, abacaba, c), for example, but if wed done either of those things, M would have reached a dead end. M in effect has to
guess which one of a group of applicable rules of G, if any, is the right one to derive the given string. Such guessing is highly
undesirable in the case of most practical applications, such as compilers, because their operation can be slowed down to the
point of uselessness. Therefore, programming languages and query languages (which are almost always context-free, or
nearly so) are designed so that they can be parsed deterministically and therefore compiled or interpreted in the shortest
possible time. A lot of attention has been given to this problem in Computer Science, as you will learn if you take a course in
compilers. On the other hand, natural languages, such as English, Japanese, etc., were not "designed" for this kind of parsing
efficiency. So, if we want to deal with them by computer, as for example, in machine translation or information retrieval
systems, we have to abandon any hope of deterministic parsing and strive for maximum non-deterministic efficiency. A lot of
effort has been devoted to these problems as well, as you will learn if you take a course in computational linguistics.
To complete the proof of our lemma, we need to prove that L(M) = L(G). The proof is by induction and is reasonably
straightforward. Well omit it here, and turn instead to the other half of the theorem:
Lemma: If M is a non-deterministic PDA, there is a context-free grammar G such that L(G) = L(M).
Again, the proof is by construction. Unfortunately, this time the construction is anything by natural. Wed never want
actually to do it. We just care that the construction exists because it allows us to prove this crucial result. The basic idea
behind the construction is to build a grammar that has the property that if we use it to create a leftmost derivation of some
string s then we will have simulated the behavior of M while reading s. The nonterminals of the grammar are things like <s,
Z, f> (recall that we can use any names we want for our nonterminals). The reason we use such strange looking nonterminals
is to make it clear what each one corresponds to. For example, <s, Z, f> will generate all strings that M could consume in the
process of moving from state s with Z on the stack to state f having popped Z off the stack.
To construct G from M, we proceed in two steps: First we take our original machine M and construct a new simple
machine M (see below). We do this so that there will be fewer cases to consider when we actually do the construction of a
grammar from a machine. Then we build a grammar from M.
A pda M is simple iff:
(1) There are no transitions into the start state, and
(2) Whenever ((q, a, ), (p, )) is a transition in M and q is not the start state, then and || 2.
15
In other words, M is simple if it always consults its topmost stack symbol (and no others) and replaces that symbol either with
0, 1, or 2 new symbols. We need to treat the start state separately since of course when M starts, its stack is empty and there
is nothing to consult. But we do need to guarantee that the start state cant bypass the restriction of (2) if it also functions as
something other than the start state i.e., it is part of a loop. Thus constraint (1).
Although not all machines are simple, there is an algorithm to construct an equivalent simple machine from any machine M.
Thus the fact that our grammar construction algorithm will work only on simple machines in no way limits the applicability of
the lemma that says that for any machine there is an equivalent grammar.
Given any PDA M, we construct an equivalent simple PDA M as follows:
(1) Let M = M.
(2) Add to M a new start state s and a new final state f. Add a transition from s to Ms original start state that consumes no
input and pushes a special stack bottom symbol Z onto the stack. Add transitions from of all of Ms original final states to
f. These transitions should consume no input but they should pop the bottom of stack symbol Z from the stack. For example,
if we start with a straightforward two-state PDA that accepts wcwR, then this step produces:
a//a
c//
//Z
s'
a/a/
/Z/
s
b//b
f'
b/b/
(3) (a) Assure that || 1. In other words, make sure that no transition looks at more than one symbol on the stack. It is easy
to do this. If there are any transitions in M that look at two or more symbols, break them down into multiple transitions that
examine one symbol apiece.
(b) Assure that || 1. In other words, make sure that each transition pushes no more than one symbol onto the stack.
(The rule for simple allows us to push 2, but youll see why we restrict to 1 at this point in a minute.) Again, if M has any
transitions that push more than one symbol, break them apart into multiple steps.
(c) Assure that || = 1. We already know that || isnt greater than 1. But it could be zero. If there are any transitions that
dont examine the stack at all, then change them so that they pop off the top symbol, ignore it, and push it right back on.
When we do this, we will increase by one the length of the string that gets pushed onto the stack. Now you can see why we
did step (b) as we did. If, after completing (b) we never pushed more than one symbol, we can go ahead and do (c) and still
be assured that we never push more than two symbols (which is what we require for M to be simple).
Well omit the proof that this procedure does in fact produce a new machine M that is simple and equivalent to M.
Once we have a simple machine M (K, , , , s, f) derived from our original machine M (K, , , , s, F), we are
ready to construct a grammar G for L(M) (and thus, equivalently, for L(M). We let G = (V, , R, S), where V contains a
start symbol S, all the elements of , and a new nonterminal symbol <q, A, p> for every q and p in K and every A = or any
symbol in the stack alphabet of M (which is the stack alphabet of M plus the special bottom of stack marker). The tricky part
is the construction of R, the rules of G. R contains all the following rules (although in fact most will be useless in the sense
that the nonterminal symbol on the left hand side will never be generated in any derivation that starts with S):
(1) The special rule S <s, Z, f>, where s is the start state of the original machine M, Z is the special bottom of stack
symbol that M pushes when it moves from s to s, and f is the new final state of M. This rule says that to be a string in
L(M) you must be a string that M can consume if it is started in state s with Z on the top of the stack and it makes it to state f
having popped Z off the stack. All the rest of the rules will correspond to the various paths by which M might do that.
(2) Consider each transition ((q, a, B), (r, C)) of M where a is either or a single input symbol and C is either a single symbol
or . In other words, each transition of M that pushes zero or one symbol onto the stack. For each such transition and each
state p of M, we add the rule
<q, B, p> a<r, C, p>.
Read these rule as saying that one way in which M can go from q to p and pop B off the stack is by consuming an a, going to
16
state r, pushing a C on the stack (all of which are specified by the transition were dealing with), then getting eventually to p
and popping off the stack the C that the transition specifies must be pushed. Think of these rules this way. The transition that
motivates them tells us how to make a single move from q to r while consuming the input symbol a and popping the stack
symbol B. So think about the strings that could drive M from q to some arbitrary state p (via this transition) and pop B from
the stack in the process. They include all the strings that start with a and are followed by the strings that can drive M from r
on to p provided that they also cause the C that got pushed to be dealt with and popped. Note that of course we must also pop
anything else we push along the way, but we dont have to say that explicitly since if we havent done that we cant get to C to
pop it.
(3) Next consider each transition ((q, a, B), (r, CD)) of M, where C and D are stack symbols. In other words, consider every
transition that pushes two symbols onto the stack. (Recall that since M is simple, we only have to consider the cases of 0, 1,
or 2 symbols being pushed.) Now consider all pairs of states v and w in K (where v and w are not necessarily distinct). For
all such transitions and pairs of states, construct the rule
<q, B, v> a<r, C, w><w, D, v>
These rules are a bit more complicated than the ones that were generated in (2) just because they describe computations that
involve two intermediate states rather than one, but they work the same way.
(4) For every state q in M, we add the rule
<q, , q>
These rules let us get rid of spurious nonterminals so that we can actually produce strings composed solely of terminal
symbols. They correspond to the fact that M can (trivially) get from a state back to itself while popping nothing simply by
doing nothing (i.e., reading the empty string).
See the lecture notes for an example of this process in action. As youll notice, the grammars that this procedure generates are
very complicated, even for very simple machines. From larger machines, one would get truly enormous grammars (most of
whose rules turn out to be useless, as a matter of fact). So, if one is presented with a PDA, the best bet for finding an
equivalent CFG is to figure out the language accepted by the PDA and then proceed intuitively to construct a CFG that
generates that language.
Well omit here the proof that this process does indeed produce a grammar G such that L(G) = L(M).
6 Parsing
Almost always, the reason we care about context-free languages is that we want to build programs that "interpret" or
"understand" them. For example, programming languages are context free. So are most data base query languages.
Command languages that need capabilities (such as matching delimiters) that can't exist in simpler, regular languages are also
context free.
The interpretation process for context free languages generally involves three parts (although these logical parts may be
interleaved in various ways in the interpretation program):
1. Lexical analysis, in which individual characters are combined, generally using finite state machine techniques, to form the
building blocks of the language.
2. Parsing, in which a tree structure is assigned to the string.
3. Semantic interpretation, in which "meaning", often in the form of executable code, is attached to the nodes of the tree and
thus to the entire string itself.
For example, consider the input string "orders := orders + 1;", which might be a legal string in any of a number of
programming languages. Lexical analysis first divides this string of characters into a sequence of six tokens, each of which
corresponds to a basic unit of meaning in the language. The tokens generally contain two parts, an indication of what kind of
thing they are and the actual value of the string that they matched. The six tokens are (with the kind of token shown, followed
by its value in parentheses):
<id> (orders)
:=
<id> orders
<op> (+)
<id> (1)
;
Assume that we have a grammar for our language that includes the following rules:
17
<statement> <assignment statement> ;

<statement> <loop statement> ;
<assignment statement> <id> := <expr>
<expr> <expr> <op> <expr>
<expr> <id>
Using this grammar and the string of tokens produced above, parsing assigns to the string a tree structure like
<statement>
<assignment statement>
<id> orders
:=
<expr>
<expr>
<id> orders
<op> +
<expr>
<id> 1
Finally, we need to assign a meaning to this string. If we attach appropriate code to each node of this tree, then we can
execute this statement by doing a postorder traversal of the tree. We start at the top node, <statement> and traverse its left
branch, which takes us to <assignment statement>. We go down its left branch, and, in this case, we find the address of the
variable orders. We come back up to <assignment statement>, and then go down its middle branch, which doesnt tell us
anything that we didnt already know from the fact that were in an assignment statement. But we still need to go down the
right branch to compute the value that is to be stored. To do that, we start at <expr>. To get its value, we must examine its
subtrees. So we traverse its left branch to get the current value for orders. We then traverse the middle branch to find out
what operation to perform, and then the right branch and get 1. We hand those three things back up to <expr>, which applies
the + operator and computes a new value, which we then pass back up to <assignment statement> and then to <statement>.
Lexical analysis is a straightforward process that is generally done using a finite state machine. Semantic interpretation can
be arbitrarily complex, depending on the language, as well as other factors, such as the degree of optimization that is desired.
Parsing, though, is in the middle. It's not completely straightforward, particularly if we are concerned with efficiency. But it
doesn't need to be completely tailored to the individual application. There are some general techniques that can be applied to
a wide variety of context-free languages. It is those techniques that we will discuss briefly here.
6.1
Parsing as Search
Recall that a parse tree for a string in a context-free language describes the set of grammar rules that were applied in the
derivation of the string (and thus the syntactic structure of the string). So to parse a string we have to find that set of rules.
How shall we do it? There are two main approaches:
1. Top down, in which we start with the start symbol of the grammar and work forward, applying grammar rules and
keeping track of what we're doing, until we succeed in deriving the string we're interested in.
2. Bottom up, in which we start with the string we're interested in. In this approach, we apply grammar rules "backwards".
So we look for a rule whose right hand side matches a piece of our string. We "apply" it and build a small subtree that
will eventually be at the bottom of the parse tree. For example, given the assignment statement we looked at above, me
might start by building the tree whose root is <expr> and whose (only) leaf is <id> orders. That gives us a new "string"
to work with, which in this case would be orders := <expr> <op> <id>(1). Now we look for a grammar rule that matches
part of this "string" and apply it. We continue until we apply a rule whose left hand side is the start symbol. At that
point, we've got a complete tree.
Whichever of these approaches we choose, wed like to be as efficient as possible. Unfortunately, in many cases, were
18
forced to conduct a search, since at any given point it may not be possible to decide which rule to apply. There are two
reasons why this might happen:
Our grammar may be ambiguous and there may actually be more than one legal parse tree for our string. We will
generally try to design languages, and grammars for them, so that this doesnt happen. If a string has more than one parse
tree, it is likely to have more than one meaning, and we rarely want to use languages where users cant predict the
meaning of what they write.
There may be only a single parse tree but it may not be possible to know, without trying various alternatives and seeing
which ones work, what that tree should be. This is the problem well try to solve with the introduction of various specific
parsing techniques.
6.2
Top Down Parsing
To get a better feeling for why a straightforward parsing algorithm may require search, lets consider again the following
grammar for arithmetic expressions:
(1)
EE+T
(2)
ET
(3)
TT*F
(4)
TF
(5)
F (E)
(6)
F id
Lets try to do a top down parse, using this grammar, of the string id + id * id. We will begin with a tree whose only node is
E, the start symbol of the grammar. At each step, we will attempt to expand the leftmost leaf nonterminal in the tree.
Whenever we rewrite a nonterminal as a terminal (for example, when we rewrite F as id), well climb back up the tree and
down another branch, each time expanding the leftmost leaf nonterminal). We could do it some other way. For example, we
could always expand the rightmost nonterminal. But since we generally read the input string left to right, it makes sense to
process the parse tree left to right also.
No sooner do we get started on our example parse but were faced with a choice. Should we expand E by applying rule (1) or
rule (2)? If we choose rule (1), what were doing is choosing the interpretation in which + is done after * (since + will be at
the top of the tree). If we choose rule (2), were choosing the interpretation in which * is done after + (since * will be nearest
the top of the tree, which well detect at the next step when we have to find a way to rewrite T). We know (because weve
done this before and because we know that we carefully crafted this grammar to force * to have higher precedence than +) that
if we choose rule (2), well hit a dead end and have to back up, since there will be no way to deal with + inside T.
Lets just assume for the moment that our parser also knows the right thing to do. It then produces
E
E
Since E is again the leftmost leaf nonterminal, we must again choose how to expand it. This time, the right thing to do is to
choose rule (2), which will rewrite E as T. After that, the next thing to do is to decide how to rewrite T. The right thing to do
is to choose rule (4) and rewrite T as F. Then the next thing to do is to apply rule (6) and rewrite F as id. At this point, weve
generated a terminal symbol. So we read an input symbol and compare it to the one weve generated. In this case, it matches,
so we can continue. If it didnt match, wed know wed hit a dead end and wed have to back up and try another way of
expanding one of the nodes higher up in the tree. But since we found a match, we can continue. At this point, the tree looks
like
E
E
19
F
id
Since we matched a terminal symbol (id), the next thing to do is to back up until we find a branch that we havent yet
explored. We back all the way up to the top E, then down its center branch to +. Since this is a terminal symbol, we read the
next input symbol and check for a match. Weve got one, so we continue by backing up again to E and taking the third
branch, down to T. Now we face another choice. Should we apply rule (3) or rule (4). Again, being smart, well choose to
apply rule (3), producing
E
E
T
T
F
id
The rest of the parse is now easy. Well expand T to F and then match the second id. Then well match F to the last id.
But how can we make our parser know what we knew?
In this case, one simple heuristic we might try is to consider the rules in the order in which they appear in the grammar. That
will work for this example. But suppose the input had been id * id * id. Now we need to choose rule (2) initially. And were
now in big trouble if we always try rule (1) first. Why? Because well never realize were on the wrong path and back up and
try rule (2). If we choose rule (1), then we will produce the partial parse tree
E
E
But now we again have an E to deal with. If we choose rule (1) again, we have
E
And then we have another E, and so forth. The problem is that rule (1) contains left recursion. In other words, a symbol, in
this case E, is rewritten as a sequence whose first symbol is identical to the symbol that is being rewritten.
We can solve this problem by rewriting our grammar to get rid of left recursion. Theres an algorithm to do this that always
works. We do the following for each nonterminal A that has any left recursive rules. We look at all the rules that have A on
their left hand side, and we divide them into two groups, the left recursive ones and the other ones. Then we replace each rule
with another related rule as shown in the following table:
Left recursive rules:
Original rules
A A1
A A2
A A3
A An
New rules
A 1A
A 2A
A 3A
A nA
A
20
A 1
A 2
A n
Non-left recursive rules:
A 1A
A 2A
A nA
The basic idea is that, using the original grammar, in any successful parse, A may be expanded some arbitrary number of
times using the left recursive rules, but if were going to get rid of A (which we must do to derive a terminal string), then we
must eventually apply one of the nonrecursive rules. So, using the original grammar, we might have something like
A
2
A
3
A
1
Notice that, whatever 1, 3, and 2 are, 1, which came from one of the nonrecursive rules, comes first. Now look at the new
set of rules in the right hand column above. They say that A must be rewritten as a string that starts with the right hand side of
one of the nonrecursive rules (i.e., some i). But, if any of the recursive rules had been applied first, then there would be
further substrings, after the i, derived from those recursive rules. We introduce the new nonterminal A to describe what
those things could look like, and we write rules, based on the original recursive rules, that tell us how to rewrite A. Using
this new grammar, wed get a parse tree for 1 3 2 that would look like
A
1
A
3
A
2
If we apply this transformation algorithm to our grammar for arithmetic expressions, we get
(1)
E + T E
(1) E
(2)
E T E
(3)
T * F T
(3) T
(4)
T F T
(5)
F (E)
(6)
F id
Now lets return to the problem of parsing id + id * id. This time, there is only a single way to expand the start symbol, E, so
we produce, using rule (2),
E
T
Now we need to expand T, and again, there is only a single choice. If you continue with this example, youll see that if you
have the ability to peek one character ahead in the input (well call this character the lookahead character), then its possible
to know, at each step in the parsing process, which rule to apply.
21
Youll notice that this parse tree assigns a quite different structure to the original string. This could be a serious problem
when we get ready to assign meaning to the string. In particular, if we get rid of left recursion in our grammar for arithmetic
expressions, well get parse trees that associate right instead of left. For example, well interpret a + b + c as
(a + b) + c using the original grammar, but
a + (b + c) using the new grammar.
For this and various other reasons, it often makes sense to change our approach and parse bottom up, rather than top down.
6.3
Bottom Up Parsing
Now lets go back to our original grammar for arithmetic expressions:

(1)
EE+T
(2)
ET
(3)
TT*F
(4)
TF
(5)
F (E)
(6)
F id
Lets try again to parse the string id + id * id, this time working bottom up. Well scan the input string from left to right, just
as weve always done with all the automata weve built. A bottom up parser can perform two basic operations:
1. Shift an input symbol onto the parsers stack.
2. Reduce a string of symbols from the top of the stack to a nonterminal symbol, using one of the rules of the grammar.
Each time we do this, we also build the corresponding piece of the parse tree.
When we start, the stack is empty, so our only choice is to get the first input symbol and shift it onto the stack. The first input
symbol is id, so it goes onto the stack. Next, we have a choice. We can either use rule (6) to reduce id to F, or we can get the
next input symbol and push it onto the stack. Its clear that we need to apply rule (6) now. Why? There are no other rules
that can consume an id directly. So we have to do this reduction before we can do anything else with id. But could we wait
and do it later? No, because reduction always applies to the symbols at the top of the stack. If we push anything on before we
reduce id, well never again get id at the top of the stack. So it will just sit there, unable to participate in any rules. So the
next thing we need to do is to reduce id to F, giving us a stack containing just F, and the parse tree (remember were building
up from the bottom):
F
id
Before we continue, lets observe that the reasoning we just did is going to be the basis for the design of a smart
deterministic bottom up parser. Without that reasoning, a dumb, brute force parser would have to consider both paths at this
first choice point: the one we took, as well as the one that fails to reduce and instead pushes + onto the stack. That second
path will dead end eventually, so even a brute force parser will eventually get the right answer. But for efficiency, wed like
to build a deterministic parser if we can. Well return to the question of how we do that after we finish with this example so
we can see all the places were going to have to make our parser smart.
At this point, the parsers stack contains F and the remaining input is + id * id. Again we must choose between reducing the
top of the stack or pushing on the next input symbol. Again, by looking ahead and analyzing the grammar, we can see that
eventually we will need to apply rule (1). To do so, the first id will have to have been promoted to a T and then to an E. So
lets next reduce by rule (4) and then again by rule (2), giving the parse tree and stack:
22
E
T
F
id
At this point, there are no further reductions to consider, since there are no rules whose right hand side is just E. So we must
consume the next input symbol + and push it onto the stack. Now, again, there are no available reductions. So we read the
next symbol, and the stack then contains id + E (well write the stack so that we push onto the left). Again, we need to
promote id before we can do anything else, so we promote it to F and then to T. Now weve got:
E
T
id
id
Notice that weve now got two parse tree fragments. Since were working up from the bottom, we dont know yet how theyll
get put together. The next thing we have to do is to choose between reducing the top three symbols on the stack (T + E) to E
using rule (1) or shifting on the next input symbol. By the way, dont be confused about the order of the symbols here. Well
always be matching the right hand sides of the rules reversed because the last symbol we read (and thus the right most one
well match) is at the top of the stack.
Okay, so what should we choose to do, reduce or shift? This is the first choice weve had to make where there isnt one
correct answer for all input strings. When there was just one universally correct answer, we could compute it simply by
examining the grammar. Now we cant do that. In the example were working with, we dont want to do the reduction, since
the next operator is *. We want the parse tree to correspond to the interpretation in which * is applied before +. That means
that + must be at the top of the tree. If we reduce now, it will be at the bottom. So we need to shift * on and do a reduction
that will build the multiplication piece of the parse tree before we do a reduction involving +. But if the input string had been
id + id + id, wed want to reduce now in order to cause the first + to be done first, thus producing left associativity. So we
appear to have reached a point where well have to branch. Since our grammar wont let us create the interpretation in which
we do the + first, if we choose that path first, well eventually hit a dead end and have to back up. Wed like not to waste time
exploring dead end paths, however. Well come back to how we can make a parser smart enough to do that later. For now,
lets just forge ahead and do the right thing.
As we said, what we want to do here is not to reduce but instead to shift * onto the stack. So the stack now contains * T + E.
At this point, there are no available reductions (since there are no rules whose right hand side contains * as the last symbol),
so we shift the next symbol, resulting in the stack id * T + E. Clearly we have to promote id to F (following the same
argument that we used above), so weve got
F
E
id
id
id
23
Next, we need to reduce (since there arent any more input symbols to shift), but now we have another decision to make:
should we reduce the top F to T, or should we reduce the top three symbols, using rule (3) to T? The right answer is to use
rule (3), producing:
E
id
id
id
Finally, we need to apply rule (1), to produce the single symbol E on the top of the stack, and the parse tree:
E
E
id
id
F
*
id
In a bottom up parse, were done when we consume all the input and produce a stack that contains a single symbol, the start
symbol. So were done (although see the class notes for an extension of this technique in which we add to the input and endof-input symbol $ and consume it as well).
Now lets return to the question of how we can build a parser that makes the right choices at each step of the parsing process.
As we did the example parse above, there were two kinds of decisions that we had to make:
Whether to shift or reduce (well call these shift/reduce conflicts), and
Which of several available reductions to perform (well call these reduce/reduce conflicts).
Lets focus first on shift/reduce conflicts. At least in this example, it was always possible to make the right decision on these
conflicts if we had two kinds of information:
A good understanding of what is going on in the grammar. For example, we noted that theres nothing to be done with a
raw id that hasnt been promoted to an F.
A peek at the next input symbol (the one that were considering shifting), which we call the lookahead symbol. For
example, when we were trying to decide whether to reduce T + E or shift on the next symbol, we looked ahead and saw
that the next symbol was *. Since we know that * has higher precedence than +, we knew not to reduce +, but rather to
wait and deal with * first.
So we as people can be smart and do the right thing. The important question is, Can we build a parser that is smart and does
the right thing? The answer is yes. For simple grammars, like the one were using, its fairly straightforward to do so. For
more complex grammars, the algorithms that are needed to produce a correct deterministic parser are way beyond the scope of
this class. In fact, theyre not something most people ever want to deal with. And thats okay because there are powerful
tools for building parsers. The input to the tools is a grammar. The tool then applies a variety of algorithms to produce a
parser that does the right thing. One of the most widely used such tools is yacc, which well discuss further in class. See the
yacc documentation for some more information about how it works.
Although we dont have time to look at all the techniques that systems like yacc use to build deterministic bottom up parsers,
we will look at one of the structures that they can build. A precedence table tells us whether to shift or reduce. It uses just
two sources of information, the current top of stack symbol and the lookahead symbol. We wont describe how this table is
24
constructed, but lets look at an example of one and see how it works. For our expression grammar, we can build the
following precedence table (where $ is a special symbol concatenated to the end of each input string that signals the end of the
input):
V\
(
)
id
+
*
E
T
F
id
Heres how to read the table. Compare the left most column to the top of the stack and find the row that matches. Now
compare the symbols along the top of the chart to the lookahead symbol and find the column that matches. If theres a dot in
the correponding square of the table, then reduce. Otherwise, shift. So lets go back to our example input string id + id * id.
Remember that we had a shift/reduce conflict when the stacks contents were T + E and the next input symbol was *. So we
look at the next to the last row of the table, the one that has T as the top of stack symbol. Then we look at the column headed
*. Theres no dot, so we dont reduce. But notice that if the lookahead symbol had been +, wed have found a dot, telling us
to reduce, which is exactly what wed want to do. Thus this table captures the precedence relationships between the operators
* and +, plus the fact that we want to associate left when faced with operators of equal precedence.
Deterministic, bottom up parsers of the sort that yacc builds are driven by an even more powerful table called a parse table.
Think of the parse table as an extension of the precedence table that contains additional information that has been extracted
from an examination of the grammar.
7 Closure Properties of Context-Free Languages

Union: The CFL's are closed under union. Proof: If L1 and L2 are CFL's, then there are, by definition, CFG's Gl = (V1, 1,
R1, S1) and G2 = (V2, 2, R2, S2) generating L1 and L2, respectively. (Assume that the non-terminal vocabularies of the two
grammars are disjoint. We can always rename symbols to achieve this, so there are no accidental interactions between the
two rule sets.) Now form CFG G = (V1 V2, 1 2, R1 R2 {S S1, S S2}, S). G generates L1 L2 since every
derivation from the start symbol of G must begin either S S1 or S S2 and thereafter to derive a string generated by G1 or
by G2, respectively. Thus all strings in L(G1) L(G2) are generated, and no others.
Concatenation: The CFL's are closed under concatenation. The proof is similar. Given Gl and G2 as above, form G = (V1
V2, 1 2, R1 R2 {S S1S2}, S). G generates L1L2 since every derivation from S must begin S S1S2 and proceed
thereafter to derive a string of L1 concatenated to a string of L2. No other strings are produced by G.
Kleene star: The CFL's are closed under Kleene star. Proof: If L is a CFL, it is generated by some CFG G = (V, , R, S).
Using one new nonterminal symbol S', we can form a new CFG G' = (V S', , R (S' , S' S'S }, S'). G' generates
L* since there is a derivation of (S' ), and there are other derivations of the form S' S'S S'SS ... S'S...SS
S...SS, which produce finite concatenations of strings of L. G generates no other strings.
Intersection: The CFL's are not closed under intersection. To prove this, it suffices to show one example of two CFL's
whose intersection is not context free. Let L1 = {aibicj: i, j 0}. L1 is context-free since it is generated by a CFG with the
rules S AC, A aAb, A , C cC, C . Similarly, let L2 = {akbmcm : k, m 0}. L2 is context-free, since it is
generated by a CFG similar to the one for L1. Now consider L3 = L1 L2 = {anbncn : n 0}. L3 is not context free. We'll
prove that in the next section using the context-free pumping lemma. Intuitively, L3 isn't context free because we can't count
a's, b's, and c's all with a single stack.
25
Intersection with regular languages: The CFL's are, however, closed under intersection with the regular languages. Given a
CFL, L and a regular language R, the intersection L R is a CFL. Proof: Since L is context-free, there is some nondeterministic PDA accepting it, and since R is regular, there is some deterministic finite state automaton that accepts it. The
two automata can now be merged into a single PDA by a straightforward technique described in class.
Complementation: The CFL's are not closed under complement. Proof: Since the CFL's are closed under union, if they
were also closed under complement, this would imply that they are closed under intersection. This is so because of the settheoretic equality (L1 L2) = (L1 L2).
8 The Context-Free Pumping Lemma

There is a pumping lemma for context-free languages, just as there is one for regular languages. It's a bit more complicated,
but we can use it in much the same way to show that some language L is not in the class of context-free languages. In order to
see why the pumping lemma for context-free languages is true, we need to make some observations about parse trees:
1.
2.
A path in a parse tree is a continuously descending sequence of connected nodes.

The length of a path in a parse tree is the number of connections (branches) in it.
A
A A
3. The height of a parse tree is the length of the longest path in it. For example, the parse tree above is of height 2.
4. The width of a parse tree is the length of its yield (the string consisting of its leaves). For example, the parse tree above is
of width 5.
We observe that in order for a parse tree to achieve a certain width it must attain a certain minimum height. How are height
and width connected? The relationship depends on the rules of the grammar generating the parse tree.
Suppose, for example, that a certain CFG contains the rule A AAAA. Focusing just on derivations involving this rule, we
see that a tree of height 1 would have a width of 4. A tree of height 2 would have a maximum width of 16 (although there are
narrower trees of height 2 of course).
A
A
A
A
A
A
A
A
A
With height 3, the maximum width is 64 (i.e., 43), and in general a tree of height n has maximum width of 4n. Or putting it
another way, if a tree is wider than 4n then it must be of height greater than n.
Where does the 4 come from? Obviously from the length of the right-hand side of the rule A AAAA. If we had started
with the rule A AAAAAA, we would find that a tree of height n has maximum width 6n.
What about other rules in the grammar? If it contained both the rules A AAAA and A AAAAAA, for example, then the
maximum width would be determined by the longer right-hand side. And if there were no other rules whose right-hand sides
were longer than 6, then we could confidently say that any parse tree of height n could be no wider than 6n.
Let p = the maximum length of the right-hand side of all the rules in G. Then any parse tree generated by G of height m can
26
be no wider than pm. Equivalently, any parse tree that is generated by G and that is wider than pm must have height greater
than m.
Now suppose we have a CFG G = (V, , R, S). Let n = |(V - )|, the size of the non-terminal alphabet. If G generates a parse
tree of width greater than pn, then, by the above reasoning, the tree must be of height greater than n, i.e., it contains a path of
length greater than n. Thus there are more than n + 1 nodes on this path (the number of nodes being one greater than the
length of the path), and all of them are non-terminal symbols except possibly the last. Since there are only n distinct nonterminal symbols in G, some such symbol must occur more than once along this path (by the Pigeonhole Principle). What this
says is that if a CFG generates a long enough string, its parse tree is going to be sufficiently high that it is guaranteed to have a
path with some repeated non-terminal symbol along it. Let us represent this situation by the following diagram:
S
w
Call the generated string w. The parse tree has S as its root, and let A be a non-terminal symbol (it could be S itself, of
course) that occurs at least twice on some path (indicated by the dotted lines).
Another observation about parse trees: If the leaves are all terminal symbols, then every non-terminal symbol in the tree is the
root of a subtree having terminal symbols as its leaves. Thus, the lower instance of A in the tree above must be the root of a
tree with some substring of w as its leaves. Call this substring x. The upper instance of A likewise roots a tree with a string of
terminal symbols as its leaves, and furthermore, from the geometry of the tree, we see that this string must include x as a
substring. Call the larger string, therefore, vxy. This string vxy is also a substring of the generated string w, which is to say
that for some strings u and z, w = uvxyz. Attaching these names to the appropriate substrings we have the following diagram:
S
w=
Now, assuming that such a tree is generated by G (which will be true on the assumption that G generates some sufficiently
long string), we can conclude that G must generate some other parse trees as well and therefore their associated terminal
strings. For example, the following tree must also be generated:
S
w=
27
since whatever sequence of rules produced the lower subtree

A
x
could have been applied when the upper A was being rewritten.
Similarly, the sequence of rules that expanded the upper A originally to yield the string vAy could have been applied to the
lower A as well, and if the resulting third A were now rewritten to produce x, we would have:
S
A
A
w=
Clearly this process could be repeated any number of times to give an infinite number of strings of the form
u
v1 v2 v3vn
x
y1 y2 y3yn
z, for all values of n 0.
We need one further observation before we are ready to state the Pumping Lemma. Consider again any string w that is
sufficiently long that its derivation contains at least one repeating nonterminal (A in our example above). Of course, there
may be any number of occurrences of A, but let's consider the bottom two. Consider the subtree whose root is the second A
up from the bottom (shown in bold):
S
A
A
w=
Notice that the leaves of this subtree correspond to the sequence vxy. How long can this sequence be? The answer relies on
the fact that this subtree contains exactly one repeated nonterminal (since we chose it that way). So the maximum height of
this subtree is pn+1. (Recall that p is the length of the longest rule in the grammar and n is the number of nonterminals in the
grammar.) Why n+1? Because we have n+1 nonterminals available (all n of them plus the one repeated one). So we know
that |vxy| must be M, where M is some constant that depends on the grammar and that is in fact pn+1. We are now ready to
state the Pumping Lemma for context-free languages.
Pumping Lemma for Context-Free Languages: Let G be a context-free grammar. Then there are some constants K and M
28
depending on G such that, for every string w L(G) where |w| > K, there are strings u, v, x, y, z such that
(1) w = uvxyz,
(2) |vy| > 0,
(3) |vxy| M, and
(4) for all n 0, uvnxynz L(G).
Remarks: The constant K in the lemma is just pn referred to above - the length of the longest right-hand side of a rule of G
raised to the power of the number of non-terminal symbols. In applying the lemma we won't care what the value of K actually
is, only that some such constant exists. If G generates an infinite language, then clearly there will be strings in L(G) longer
than K, no matter what K is. If L(G) is finite, on the other hand, then the lemma still holds (trivially), because K will have a
value greater than the longest strings in L(G). So all strings in L(G) longer than K are guaranteed to be "pumpable," but no
such strings exist, so the lemma is trivially true because the antecedent of the conditional is false. Similarly for M, which is
actually bigger than K; it is pn+1. But, again, all we care about is that if L(G) is infinite then M exists. Without knowing what
it is, we can describe strings in terms of it and know that we must have pumpable strings.
This lemma, like the pumping lemma for regular languages, addresses the question of how strings grow longer and longer
without limit so as to produce infinite languages. In the case of regular languages, we saw that strings grow by repeating
some substring any number of times: xynz L for all n 0. When does this happen? Any string in the language of sufficient
length is guaranteed to contain such a "pumpable" substring. What length is sufficient? The number of states in the
minimum-state deterministic finite state machine that accepts the language. This sets the lower bound for guaranteed
pumpability.
For context-free languages, strings grow by repeating two substrings simultaneously: uvnxynz L for all n 0. This, too,
happens when a string in the language is of sufficient length. What is sufficient? Long enough to guarantee that its parse tree
contains a repeated non-terminal along some path. Strings this long exceed the lower bound for guaranteed context-free
pumpability.
What about the condition that |vy| > 0, i.e., v and y cannot both be the empty string? This could happen if by the rules of G
we could get from some non-terminal A back to A again without producing any terminal symbols in the process, and that's
possible with rules like A B, B C, C A, all perfectly good context-free rules. But given that we have a string w
whose length is greater than or equal to K, its derivation must have included some rules that make the string grow longer;
otherwise w couldn't have gotten as long as it did. Therefore, there must be some path in the derivation tree with a repeated
non-terminal that involves branching rules, and along this path, at least one of v or y is non-empty.
Recall that the corresponding condition for regular languages was y . We justified this by pointing out that if a sufficiently
long string w was accepted by the finite state machine, then there had to be a loop in the machine and that loop must read
something besides the empty string; otherwise w couldn't be as long as it is and still be accepted.
And, finally, what about condition (3), |vxy| M? How does this compare to the finite state pumping lemma? The
corresponding condition there was that |xy| K. Since |y| |xy|, this certainly tells us that the pumpable substring y is
(relatively) short. |xy| K also tells us that y occurs close to the beginning of the string w = xyz. The context-free version, on
the other hand, tells us that |vxy| M, where v and y are the pumpable substrings. Since |v| |vxy| M and |y| |vxy| M, we
know that the pumpable substrings v and y are short. Furthermore, from |vxy| M, we know that v and y must occur close to
each other (or at least not arbitrarily far away from each other). Unlike in the regular pumping lemma, though, they do not
necessarily occur close to the beginning of the string w = uvxyz. This is the reason that context-free pumping lemma proofs
tend to have more cases: the v and y pumpable substrings can occur anywhere within the string w.
Note that this Pumping Lemma, like the one for regular languages, is an if-then statement not an iff statement. Therefore, it
cannot be used to show that a language is context-free, only that it is not.
Example 1: Show that L = {anbncn : n 0} is not context-free.
If L were context-free (i.e., if there were a context-free grammar generating L), then the Pumping Lemma would apply. Then
29
there would be a constant K such that every string in L of length greater than K would be "pumpable." We show that this is
not so by exhibiting a string w in L that is of length greater than K and that is not pumpable. Since we want to rely on clause
3 of the pumping lemma, and it relies on M > K, we'll actually choose w in terms of M.
Let w = aMbMcM. (Note that this is a particular string, not a language or a variable expression for a string. M is some
number whose exact value we don't happen to know; it might be 23, for example. If so, w would be the unique string
a23b23c23.) This string is of length greater than K (of length 3M, where M is greater than K, in fact), and it is a string in the
language {anbncn : n 0}. Therefore, it satisfies the criteria for a pumpable string according to the Pumping Lemma-provided, of course, that L is context-free.
What does it mean to say that aMbMcM is pumpable? It means that there is some way to factor this string into five parts u,v,x,y,z - meeting the following conditions:
1. v and y are not both the empty string (although any of u, x, or z could be empty),
2. |vxy| M,
3. uxz L, uvxyz L, uvvxyyz L, uvvvxyyyz L, etc.; i.e., for all n 0, uvnxynz L.
We now show that there is no way to factor aMbMcM into 5 parts meeting these conditions; thus, aMbMcM is not a pumpable
string, contrary to the stipulation of the Pumping Lemma, and thus L is not a context-free language.
How do we do this? We show that no matter how we try to divide aMbMcM in ways that meet the first two conditions, the third
condition always falls. In other words, every "legal" division of aMbMcM falls to be pumpable-that is, there is some value of n
for which uvnxynz L.
There are clearly a lot of ways to divide this string into 5 parts, but we can simplify the task by grouping the divisions into
cases just as we did with the regular language Pumping Lemma:
Case 1: Either v or y consists of more than different letter (e.g., aab). No such division is pumpable, since for any n 2,
uvnxynz will contain some letters not in the correct order to be in L. Now that weve eliminated this possibility, all the
remaining cases can assume that both v and y contain only as, only bs, or only cs (although one could also be ).
Case 2: Both v and y are located within aM. No such division is pumpable, since we will pump in only as. So, for n 2,
uvnxynz will contain more a's than b's or c's and therefore won't be in L. (Note that n = 0 also works.)
Cases 3, 4: Both v and y are located within bM or cM. No such division is pumpable, by the same logic as in Case 2.
Case 5: v is located within aM and y is located within bM. No such division is pumpable, since for n 2, uvnxynz will contain
more a's than c's or more b's than c's (or both) and therefore won't be in L. (n = 0 also works here.)
Cases 6, 7: v is located within aM and y is located within cM, or v is located within bM and y is located within cM. No such
division is pumpable, by the same logic as in Case 5.
Since every way of dividing aMbMcM into 5 parts (such that the 2nd and 4th are not both empty) is covered by (at least one of
the above 7 cases, and in each case we find that the resulting division is not pumpable, we conclude that there is no division of
aMbMcM that is pumpable. Since all this was predicated on the assumption that L was a context-free language, we conclude
that L, is not context-free after all.
Notice that we didn't need to use condition the fact that |vxy| must be less than M in this proof, although we could have used it
as an alternative way to handle case 6, since it prevents v and y from being separated by a region of size M, which is exactly
the size of the region of b's that occurs between the a's and the c's.
Example 2: Show that L = {w {a, b c}* | #(a, w) = #(b, w) = #(c, w)} is not context free. (We use the notation #(a, w) to
mean the number of a's in the string w.)
Let's first try to use the pumping lemma. We could again choose w = aMbMcM. But now we can't immediately brush off case 1
as we did in Example 1, since L allows for strings that have the a's, b's, and c's interleaved. In fact, this time there are ways
to divide aMbMcM into 5 parts (v, y not both empty), such that the result is pumpable. For example, if v were ab and y were c,
then uvnxynz would be in L for all n 0, since it would still contain equal numbers of a's, b's, and cs.
30
So we need some additional help, which we'll find in the closure theorems for context-free languages. Our problem is that the
definition of L is too loose, so it's too easy for the strings that result from pumping to meet the requirements for being in L.
We need to make L more restrictive. Intersection with another language would be one way we could do that. Of course, since
the context-free languages are not closed under intersection, we can't simply consider some new language L' = L L2, where
L2 is some arbitrary context-free language. Even if we could use pumping to show that L' isn't context free, we'd know
nothing about L. But the context-free languages are closed under intersection with regular languages. So if we construct a
new language L' = L L2, where L2 is some arbitrary regular language, and then show that L' is not context free, we know
that L isn't either (since, if it were, its intersection with a regular language would also have to be context free). Generally in
problems of this sort, the thing to do is to use intersection with a regular language to constrain L so that all the strings in it
must have identifiable regions of symbols. So what we want to do here is to let L2 = a*b*c*. Then L' = L L2 = anbncn. If
we hadn't just proved that anbncn isn't context free, we could easily do so. In either case, we know immediately that L isn't
context free.
31
Recursively Enumerable Languages,

Turing Machines, and Decidability
1 Problem Reduction: Basic Concepts and Analogies
The concept of problem reduction is simple at a high level. You simply take an algorithm that solves one problem and use it
as a subroutine to solve another problem. For example, suppose we have two TM's: C, which turns w into ww and
SL, which turns w into w (i.e., it shifts w one square to the left). Then we can build a TM M' that computes f(w) =
ww by simply letting M' =CSL. We have reduced the problem of computing f to the problem of copying a string and then
shifting it.
Let's consider another example. Suppose we had a function sqr(m: integer): integer, which accepts an integer input m and
returns m2. Then we could write the following function to compute g(m) = m2 + 2m + 1:
function g(m: integer): integer;
begin
return sqr(m + 1);
end;
We have reduced the problem of computing m2 + 2m + 1 to the problem of computing m + 1 and squaring it.
We are going to be using reduction specifically for decision problems. In other words, we're interested in a particular class of
boolean functions whose job is to look at strings and tell us, for each string, whether or not it is in the language we are
concerned with. For example, suppose we had a TM M that decides the language a*b+. Then we could make a new TM M' to
decide a*b*: M' would find the first blank to the right of its input and write a single b there. It would then move its read head
back to the beginning of the string and invoke M on the tape. Why does this work? Because x a*b* iff xb a*b+.
Looking at this in the more familiar procedural programming style, we are saying that if we have a function:
f1(x: string): boolean, which tells us whether x a*b+, then we could write the following function that will correctly tell us
whether x a*b*.
function f2(x: string): boolean;
begin
return f1(x || 'b');
end;
If, for some reason, you believed that a*b* were an undecidable language, then this reduction would force you to believe that
a*b+ is also undecidable. Why? Because we have shown that we can decide a*b* provided only that f1 does in fact decide
a*b+. If we know that we can't decide a*b*, there must be something standing in the way. Unless we're prepared to believe
that subroutine invocation is not computable or that concatenation of a single character to the end of a string is not
computable, we must assign blame to f1 and conclude that we didn't actually have a correct f1 program to begin with.
These examples should all make sense. The underlying concept is simple. Things will become complicated in a bit because
we will begin considering TM descriptions as the input stings about which we want to ask questions, rather than simple strings
of a's and b's.
Sometimes, we'll use a slightly different but equivalent way of asking our question. Rather than asking whether a language is
decidable, we may ask whether a problem is solvable. When we say that a problem is unsolvable, what we mean is that the
corresponding language is undecidable. For example, consider the problem of determining, given a TM M and string w,
whether or not M accepts w. We can ask whether this problem is solvable. But notice that this same problem can be phrased
a language recognition problem because it is equivalent to being able to decide the language:
H = {"M" "w" : w L(M)}.
Read this as: H is the language that consists of all strings that can be formed from two parts: the encoding of a Turing
Machine M, followed by the encoding of a string w (which we can think of as the input to M), with the additional constraint
that TM M halts on input w (which is equivalent to saying that w is in the language accepted by M).
Solving a problem is the higher level notion, which is commonly used in the programming/algorithm context. In our
theoretical framework, we use the term deciding a language because we are talking about Turing Machines, which operate
on strings, and we have a carefully constructed theory that lets us talk about Turing Machines as language recognizers.
In the following section, well go through several examples in which we use the technique of problem reduction to show that
some new problem is unsolvable (or, alternatively, that the corresponding language is undecidable). All of these proofs
depend ultimately on our proof, using diagonalization, of the undecidability of the halting problem (H above).
2 Some Reductions Presented in Gory Detail

Example 1: Given a TM M, does M halt on input ? (i.e., given M, is L(M)?) This problem is undecidable because we
can reduce the Halting Problem H to it. What this means is that an algorithm to answer this question could be used as a
subroutine in an algorithm (which is otherwise clearly effective) to solve the Halting problem. But we know the Halting
problem is unsolvable; therefore this question is unsolvable. So how do we prove this?
First we'll prove this using the TM/language framework. In other words, were going to show that the following language LE
is not decidable:
LE = {"M": L(M)}
We will show that if LE is decidable, so is H = {"M" "w" : w L(M)}.
Suppose LE is decidable; then some TM MLE decides it. We can now show how to construct a new Turing Machine MH,
which will invoke MLE as a subroutine, and which will decide H. In the process of doing so, well use only clearly
computable functions (other than MLE, of course). So when we finish and realize that we have a contradiction (since we know
that MH cant exist), we know that the blame must rest on MLE and thus we know that MLE cannot exist.
MH is the Turing Machine that operates as follows on the inputs "M", "w":
1.
2.
Construct a new TM M*, which behaves as follows:

1.1. Copy "w" onto its tape.
1.2. Execute M on the resulting tape.
Invoke MLE(M*).
If MLE returns True, then we know that M (the original input to MH) halts on w. If MLE returns False, then we know that it
doesnt. Thus we have built a supposedly unbuildable MH. How did we do it? We claimed when we asserted the existence of
MLE that we could answer what appears to be a more limited question, does M halt on the empty tape? But we can find out
whether M halts on any other specific input (w) by constructing a machine (M*) that starts by writing w on top of whatever
was originally on its tape (thus it ignores its actual input) and then proceeds to do whatever M would have done. Thus M*
behaves the same on all inputs. Thus if we knew what it does on any one input, wed know what it does for all inputs. So we
ask MLE what it does on the empty string. And that tells us what it does all the time, which must be, by the way we
constructed it, whatever the original machine M does on w.
The only slightly tricky thing here is the procedure for constructing M*. Are we sure that it is in fact computable? Maybe
weve reached the contradiction of claiming to have a machine to solve H not by erroneously claiming to have MLE but rather
by erroneously claiming to have a procedure for constructing M*. But thats not the case. Why not? Its easy to see how to
write a procedure that takes a string w and builds M*. For example, if "w" is "ab", then M* must be:
ERaRbLM, where E is a TM that erases its tape and then moves the read head back to the first square.
In other words, we erase the tape, move back to the left, then move right one square (leaving one blank ahead of the new tape
contents), write a, move right, write b, move left until we get back to the blank thats just to the left of the input, and then
execute M.
The Turing Machine to construct M* is a bit too complicated to write here, but we can see how it works by describing it in a
more standard procedural way: It first writes ER. Then, for each character in w, it writes that character and R. Finally it
writes LM.
To make this whole process even clearer, lets look at this problem not from the point of view of the decidability of the
language H but rather by asking whether we can solve the Halting problem. To do this, lets describe in standard code how
we could solve the Halting problem if we had a subroutine MLE(M: TM) that could tell us whether a particular Turing
Machine halts on the empty string. Well assume a datatype TM. If you want to, you can think of objects of this type as
essentially strings that correspond to valid Turing Machines. Its like thinking of a type Cprogram, which is all the strings that
are valid C programs.
We can solve the Halting program with following function Halts:
Function Halts(M: TM, w: string): Boolean;
M* := Construct(M, w);
Return MLE(M*);
end;
Function Construct(M: TM, w: string): TM;
/* Construct builds a machine that first erases its tape. Then it copies w onto its tape and moves its
/* read head back to the left ready to begin reading w. Finally, it executes M.
Construct := Erase;
/* Erase is a string that corresponds to the TM that erases its input tape.
For each character c in w do
Construct := Construct || "R" || c;
end;
Construct := Construct || LM;
Return(Construct);
Function MLE(M: TM): Boolean;
The function we claim tells us whether M halts on the empty string.
The most common mistake that people make when theyre trying to use reduction to show that a problem isnt solvable (or
that a language isnt decidable) is to do the reduction backwards. In this case, that would mean we would put forward the
following argument: Suppose we had a program Halts to solve the Halting problem (the general problem of determining
whether a TM M halts on some arbitrary input w). Then we could use it as a subroutine to solve the specific problem of
determining whether a TM M halts on the empty string. Wed simply invoke Halts and pass it M and . If Halts returns True,
then we say yes. If Halts returns False, we say no. But since we know that Halts cant exist, no solution to our problem can
exist either. The flaw in this argument is the last sentence. Clearly, since Halts cant exist, this particular approach to
solving our problem wont work. But this says nothing about whether there might be some other way to solve our problem.
To see this flaw even more clearly, lets consider applying it in a clearly ridiculous way: Suppose we had a program Halts to
solve the Halting problem. Then we could use it as a subroutine to solve the problem of adding two numbers a and b. Wed
simply invoke Halts and pass it the trivial TM that simply halts immediately and the input . If Halts returns True, then we
return a+b. If Halts returns False, we also return a+b. But since we know that Halts cant exist, no solution to our problem
can exist either. Just as before, we have certainly written a procedure for adding two numbers that wont work, since Halts
cant exist. But there are clearly other ways to solve our problem. We dont need Halts. Thats totally obvious here. Its less
so in the case of attempting to build MLE. But the logic is the same in both cases: flawed.
Example 2: Given a TM M, is L(M) ? (In other words, does M halt on anything at all?). Lets do this one first using the
solvability of the problem perspective. Then well do it from the decidability of the language point of view.
This time, we claim that there exists:
Function MLA(M: TM): Boolean;
Returns T if M halts on any inputs at all and False otherwise.
We show that if this claim is true and MLA does in fact exist, then we can build a function MLE that solves the problem of
determining whether a TM accepts the empty string. We already know, from our proof in Example 1, that this problem isnt
solvable. So if we can do it using MLA (and some set of clearly computable functions), we know that MLA cannot in fact
exist.
The reduction we need for this example is simple. We claim we have a machine MLA that tells us whether some machine M
accepts anything at all. If we care about some particular input to M (for example, we care about ), then we will build a new
machine M* that erases whatever was originally on its tape. Then it copies onto its tape the input we care about (i.e., ) and
runs M. Clearly this new machine M* is oblivious to its actual input. It either always accepts (if M accepts ) or always
rejects (if M rejects ). It accepts everything or nothing. So what happens if we pass M* to MLA? If M* always accepts,
then its language is not the empty set and MLA will return True. This will happen precisely in case M halts on . If M*
always rejects, then its language is the empty set and MLA will return False. This will happen precisely in case M doesnt
halt on . Thus, if MLA really does exist, we have a way to find out whether any machine M halts on :
Function MLE(M: TM): Boolean;
M* := Construct(M);
Return MLA(M*);
end;
Function Construct(M: TM): TM; /* This time, we build an M* that simply erases its input and then runs M
/*(thus running M on ).
Construct := Erase;
/* Erase is a string that corresponds to the TM that erases its input tape.
Construct := Construct || M;
Return(Construct);
But we know that MLE cant exist. Since everything in its code, with the exception of MLA, is trivially computable, the only
way it cant exist is if MLA doesnt actually exist. Thus weve shown that the problem determining whether or not a TM M
halts on any inputs at all isnt solvable.
Notice that this argument only works because everything else that is done, both in MLE and in Construct is clearly
computable. We could write it all out in the Turing Machine notation, but we dont need to, since its possible to prove that
anything that can be done in any standard programming language can also be done with a Turing Machine. So the fact that
we can write code for it is good enough.
Whenever we want to try to use this approach to decide whether or not some new problem is solvable, we can choose to
reduce to the new problem any problem that we already know to be unsolvable. Initially, the only problem we knew wasnt
solvable was the Halting problem, which we showed to be unsolvable using diagonalization. But once we have used
reduction to show that other problems arent solvable either, we can use any of them for our next problem. The choice is up
to you. Whatever is easiest is the thing to use.
When we choose to use the problem solvability perspective, there is always a risk that we may make a mistake because we
havent been completely rigorous in our definition of the mechanisms that we can use to solve problems. One big reason for
even defining the Turing Machine formalism is that it is both simple and rigorously defined. Thus, although the problem
solvability perspective may seem more natural to us, the language decidability perspective gives us a better way to construct
rigorous proofs.
So lets show that the language
LA = {"M": L(M) } is undecidable.
We will show that if LA were decidable, then LE = {"M": L(M)} would also be decidable. But of course, we know that it
isnt.
Suppose LA is decidable; then some TM MLA decides it. We can now show how to construct a new Turing Machine MLE,
which will invoke MLA as a subroutine, and which will decide LE:
MLE(M): /* A decision procedure for LE = {"M": L(M)}
1. Construct a new TM M*, which behaves as follows:
1.1. Erase its tape.
1.2. Execute M on the resulting empty tape.
2. Invoke MLA(M*).
Its clear that MLE effectively decides LE (if MLA really exists). Why? MLE returns True iff MLA returns True. That happens,
by its definition, if it is passed a TM that accepts at least one string. We pass it M*. M* accepts at least one string (in fact, it
accepts all strings) precisely in case M accepts the empty string. If M does not accept the empty string, then M* accepts
nothing and MLE returns False.
Example 3: Given a TM M, is L(M) = *? (In other words, does M accept everything?). We can show that this problem is
unsolvable by using almost exactly the same technique we just used. In example 2, we wanted to know whether a TM
accepted anything at all. Now we want to know whether it accepts everything. We will answer this question by showing that
the language
L = {"M": L(M) = *} is undecidable.
Recall the machine M* that we constructed for Example 2. It erases its tape and then runs M on the empty tape. Clearly M*
either accepts nothing or it accepts everything, since its behavior is independent of its input. M* is exactly what we need for
this proof too. Again well choose to reduce the language LE = {"M": L(M)} to our new problem L:
If L is decidable, then theres a TM ML that decides it. In other words, theres a TM that tells us whether or not some other
machine M accepts everything. If ML exists, then we can define the following TM to decide LE:
2. Invoke ML(M*).
Step 2 will return True if M* halts on all strings in * and False otherwise. So it will return True if and only M halts on .
This would seem to be a correct decision procedure for LE. But we know that such a procedure cannot exist and the only
possible flaw in the procedure weve given is ML. So ML doesnt exist either.
Example 4: Given a TM M, is L(M) infinite? Again, we can use M*. Remember that M* either halts on nothing or it halts
on all elements of *. Assuming that , that means that M* either halts on nothing or it halts on an infinite number of
strings. It halts on everything if its input machine M halts on . Otherwise it halts on nothing. So we can show that the
language
LI = {"M": L(M) is infinite}
is undecidable by reducing the language LE = {"M": L(M)} to it.
If LI is decidable, then there is a Turing Machine MLI that decides it. Given MLI, we decide LE as follows:
2. Invoke MLI(M*).
Step 2 will return True if M* halts on an infinite number of strings and False otherwise. So it will return True if and only M
halts on .
This idea that a single construction may be the basis for several reduction proofs is important. It derives from the fact that
several quite different looking problems may in fact be distinguishing between the same two cases.
Example 5: Given two TMs, M1 and M2, is L(M1) = L(M2)? In other words, is the language
LEQ = {"M1" "M2": L(M1) = L(M2)} decidable?
Now, for the first time, we want to answer a question about the relationship of two Turing Machines to each other. How can
we solve this problem by reducing to it any of the problems we already know to be undecidable? They all involve only a
single machine. The trick is to use a constant, a machine whose behavior we are certain of. So we define M#, which halts on
all inputs. M# is trivial. It ignores its input and goes immediately to a halt state.
If LEQ is decidable, then there is a TM MLEQ that decides it. Using MLEQ and M#, we can decide the language
L = {"M": L(M) = *} (which we showed in example 3 isnt decidable) as follows:
ML(M): /* A decision procedure for L
1. Invoke MLEQ(M, M#).
Clearly M accepts everything if it is equivalent to M#, which is exactly what MLEQ tells us.
This reduction is an example of an easy one. To solve the unsolvable problem, we simply pass the input directly into the
subroutine that we are assuming exists, along with some simple constant. We don't need to do any clever constructions. The
reason this was so simple is that our current problem LEQ, is really just a generalization of a more specific problem weve
already shown to be unsolvable. Clearly if we cant solve the special case (determining whether a machine is equivalent to
M#), we cant solve the more general problem (determining whether two arbitrary machines are equivalent).
Example 6: Given a TM M, is L(M) regular? Alternatively, is
LR = {"M": L(M) is regular} decidable?
.
To answer this one, well again need to use an interesting construction. To do this, well make use of the language
H = {"M" "w" : w L(M)}
Recall that H is just the set of strings that correspond to a (TM, input) pair, where the TM halts on the input. H is not
decidable (thats what we proved by diagonalization). But it is semidecidable. We can easily built a TM Hsemi that halts
whenever the TM M halts on input w and that fails to halt whenever M doesnt halt on w. All Hsemi has to do is to simulate the
execution of M on w. Note also that H isnt regular (which we can show using the pumping theorem).
Suppose that, from M and Hsemi, we construct a machine M$ that behaves as follows: given an input string w, it first runs Hsemi
on w. Clearly, if Hsemi fails to halt on w, M$ will also fail to halt. But if Hsemi halts, then we move to the next step, which is to
run M on . If we make it here, then M$ will halt precisely in case M would halt on . So our new machine M$ will either:
1. Accept H, which it will do if L(M), or
2. Accept , which it will do if L(M).
Thus we see that M$ will accept either
1. A non regular language, H, which it will do if L(M), or
2. A regular language , which it will do if L(M).
So, if we could tell whether M$ accepts a regular language or not, wed know whether or not M accepts .
Were now ready to show that LR isnt decidable. If it were, then there would be some TM MLR that decided it. But MLR
cannot exist, because, if it did, we could reduce LE = {"M": L(M)} to it as follows:
1. Construct a new TM M$(w), which behaves as follows:
1.1. Execute Hsemi on w.
2.
3.
1.2. Execute M on .
Invoke MLR(M$).
If the result of step 2 is True, return False; if the result of step 2 is False, return True.
MLE, as just defined, effectively decides LE. Why? If L(M), then L(M$) is H, which isnt regular, so MLR will say False
and we will, correctly, say True. If L, then L(M$) is , which is regular, so MLR will say True and we will, correctly, say
False.
By the way, we can use exactly this same argument to show that LC = {"M": L(M) is context free} and LD = {"M": L(M) is
recursive} are undecidable. All we have to do is to show that H is not context free (by pumping) and that it is not recursive
(which we did with diagonalization).
Example 7: Given a TM M and state q, is there any configuration (p, uav), with p q, that yields a configuration whose state
is q? In other words, is there any state p that could possibly lead M to q? Unlike many (most) of the questions we ask about
Turing Machines, this one is not about future behavior. (e.g., Will the Turing Machine do such and such when started from
here?) So were probably not even tempted to try simulation (which rarely works anyway).
But there is a way to solve this problem. In essence, we dont need to consider the infinite number of possible configurations
of M. All we need to do is to examine the (finite) transition table of M to see whether there is any transition from some state
other than q (call it p) to q. If there is such a transition (i..e., if p, , such that (p, ) = (q, )), then the answer is yes.
Otherwise, the answer is no.
3 Rices Theorem
Rices Theorem makes a very general statement about an entire class of languages that are not recursive. Thus, for some
languages, it is an alternative to problem reduction as a technique for proving that a language is not recursive.
There are several different forms in which Rices Theorem can be stated. Well present two of them here:
(Form 1) Any nontrivial property of the recursively enumerable languages is not decidable.
(Form 2) Suppose that C is a proper, nonempty subset of the class of all recursively enumerable languages. Then the
following language is undecidable: LC = {<M>: L(M) is an element of C}.
These two statements look quite different but are in fact nearly equivalent. (Although Form 1 is stronger since it makes a
claim about the language, no matter how you choose to define the language. So it applies given a grammar, for example.
Form 2 only applies directly if the way you define the language is by a Turing Machine that semidecides it. But even this
difference doesn't really matter since we have algorithms for constructing a Turing Machine from a grammar and vice versa.
So it would just take one more step if we started with a grammar and wanted to use Form 2). But, if we want to prove that a
language of Turing machine descriptions is not recursive using Rice's Theorem, you must do the same things, whichever
description of it you prefer.
Well consider Form 1 first. To use it, we first, we have to guarantee that the property we are concerned with is a property
(predicate) whose domain is the set of recursively enumerable languages. Here are some properties P whose domains are the
RE languages:
1) P is true of any RE language that contains an even number of strings and false of any RE language that contains an odd
number of strings.
2) P is true of any RE language that contains all stings over its alphabet and false for all RE languages that are missing any
strings over their alphabet.
3) P is true of any RE language that is empty (i.e., contains no strings) and false of any RE language that contains any strings.
4) P is true of any RE language
5) P is true of any RE language that can be semidecided by a TM with an even number of states and false for any RE language
that cannot be semidecided by such a TM.
6) P is true of any RE language that contains at least one string of infinite length and false of any RE language that contains no
infinite strings.
Here are some properties whose domains are not the RE languages:
1 ) P is true of Turing machines whose first move is to write "a" and false of other Turing machines.
2) P is true of two tape Turing machines and false of all other Turing machines.
3) P is true of the negative numbers and false of zero and the positive numbers.
4) P is true of even length strings and false of odd length strings.
We can attempt to use Rices Theorem to tell us that properties in the first list are undecidable. It wont help us at all for
properties in the second list.
But now we need to do one more thing: We must show that P is a nontrivial property. Any property P is nontrivial if it is not
equivalent to True or False. In other words, it must be true of at least one language and false of at least one language.
Lets look at properties 1-6 above:
1) P is true of {a, b} and false of {a}, so it is nontrivial.
2) Lets just consider the case in which is {a, b}. P is true of * and false of {a}.
3) P is true of and P is false of every other RE language.
4) P is true of any RE language and false of nothing, so P is trivial.
5) P is true of any RE language and false of nothing, so P is trivial. Why? Because, for any RE language L there exists a
semideciding TM M. If M has an even number of states, then P is clearly true. If M has an odd number of states, then create
a new machine M identical to M except it has one more state. This state has no effect on Ms behavior because there are no
transitions in to it. But it guarantees that M has an even number of states. Since M accepts L (because M does), P is true of
L. So P is true of all RE languages.
6) P is false for all RE languages. Why? Because the definition of a language is a set of strings, each of finite length. So no
RE language contains a string of infinite length.
So we can use Rices theorem to prove that the set of RE languages possessing any one of properties 1, 2, or 3 is not
recursive. But it does not tell us anything about the set of RE languages possessing property 3, 4, or 5.
In summary, to apply this version of Rices theorem, it is necessary to do three things:
0) Specify a property P.
1) Show that the domain of P is the set of recursively enumerable languages.
2) Show that P is nontrivial by showing:
a) That P is true of at least one language, and
b) That P is false of at least on language.
Now lets try to use Form 2. We must find a C that is a proper, nonempty subset of the class of all recursively enumerable
language.
First we notice that this version is stated in terms of C, a subset of the RE languages, rather than P, a property (predicate) that
is true of the RE languages. But this is an insignificant difference. Given a universe U, then one way to define a subset S of
U is by a characteristic function that, for any candidate element x, returns true if x S and false otherwise. So the P that
corresponds to S is simply whatever property the characteristic function tests for. Alternatively, for any subset S there must
exist a characteristic function for S (although that function need not be computable thats a different issue.) So given a set
S, we can define the property P as is a member of S. or possesses whatever property it takes to be determined to be n S.
So Form 2 is stated in terms of the set of languages that satisfy some property P instead of being stated in terms of P directly,
but as there is only one such set for any property P and there is only one such property P (viewed simply as a truth table,
ignoring how you say it in English) for any set, it doesnt matter which specification we use.
Next we notice that this version requires that C be a proper, nonempty subset of the class of RE languages. But this is exactly
the same as requiring that P be nontrivial. Why? For P to be nontrivial, then there must exist at least one language of which it
is true and one of which it is false. Since there must exist one language of which it is true, the set of languages that satisfy it
isnt empty. Since there must exist one language of which it is false, the set of languages that satisfy it is not exactly the set of
RE languages and so we have a proper subset.
So, to use Form 2 of Rices Theorem requires that we:
0) Specify some set C (by specifying a membership predicate P or some other way).
1) Show that C is a subset of the set of RE languages (which is equivalent to saying that the domain of its membership
predicate is the set of RE languages)
2) Show that C is a proper nonempty subset of the recursive languages by showing that
a) C (i.e., its characteristic function P is not trivially false), and
b) C RE (i.e., its characteristic function P is not trivially true).

Baker CS341 Packet PDF

Uploaded by

Copyright:

Available Formats

Baker CS341 Packet PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Baker CS341 Packet PDF

Uploaded by

Copyright:

Available Formats

CS 341

The Three Hour Tour Through Automata Theory

B. Regular Languages and Finite State Machines

C. Context-Free Languages and Pushdown Automata

Context Free Grammars

D. Recursively Enumerable Languages, Turing Machines, and

B. Regular Languages and Finite State Machines

Strings and Languages

C. Context-Free Languages and Pushdown Automata

Context Free Grammars

D. Recursively Enumerable Languages, Turing Machines, and

The Three Hour Tour Through Automata Theory

A Framework for Analyzing Problems

The Three Hour Tour

Encoding Output in the Input String

Given the language recognition machine, we can build a multiplication machine:

The Three Hour Tour

A Framework for Describing Languages

(1) The Language of Matched Parentheses

(2) The Language of Odd Integers

The Three Hour Tour

Grammars as Generators and Acceptors

The Language Hierarchy

The Three Hour Tour

<one terminal><one nonterminal>

So, the following rules are okay:

The Three Hour Tour

A much simpler example:

Not All Languages are Context-Free

Example: A grammar to generate all strings of the form anbncn, n 1

The Three Hour Tour

Finite State Machines 1

A PDA to accept strings with balanced parentheses:

The Three Hour Tour

The Three Hour Tour

A Two Tape Turing Machine

A Two Tape Turing Machine to do the same thing

Simulating k Tapes with One

Can be encoded on a single tape with an alphabet consisting of symbols corresponding to :

2nd square: (,0,a,1))

Simulating a Turing Machine with a PDA with Two Stacks

The Three Hour Tour

The Universal Turing Machine

The Three Hour Tour

Languages and Machines

Where Does a Particular Problem Go?

The Three Hour Tour

Using Closure Properties

But then we could create a new set

The Three Hour Tour

The Unsolvability of the Halting Problem

But we've defined TROUBLE so that it will actually behave as:

The Three Hour Tour

Can always say yes or no

Let's Revisit Some Problems

The Three Hour Tour

Formalize and Prove Things

Regular Languages and Finite State Machines

The Three Hour Tour