I.

Lecture Notes
 The Three Hour Tour Through Automata Theory
Read Supplementary Materials: The Three Hour Tour Through Automata Theory
Read Supplementary Materials: Review of Mathematical Concepts
Read K & S Chapter 1
Do Homework 1.

Let's Look at Some Problems

int alpha, beta;
alpha = 3;
beta = (2 + 5) / 10;
(1) Lexical analysis: Scan the program and break it up into variable names, numbers, etc.
(2) Parsing: Create a tree that corresponds to the sequence of operations that should be executed, e.g.,
/

+ 10

2 5
(3) Optimization: Realize that we can skip the first assignment since the value is never used and that we can precompute the
arithmetic expression, since it contains only constants.
(4) Termination: Decide whether the program is guaranteed to halt.
(5) Interpretation: Figure out what (if anything) it does.

A Framework for Analyzing Problems

We need a single framework in which we can analyze a very diverse set of problems.
The framework we will use is Language Recognition

A language is a (possibly infinite) set of finite length strings over a finite alphabet.

Languages
(1) Σ = {0,1,2,3,4,5,6,7,8,9}
L = {w ∈ Σ*: w represents an odd integer}
= {w ∈ Σ*: the last character of w is 1,3,5,7, or 9}
= (0∪1∪2∪3∪4∪5∪6∪7∪8∪9)*
(1∪3∪5∪7∪9)
(2) Σ = {(,)}
L = {w ∈ Σ*: w has matched parentheses}
= the set of strings accepted by the grammar:
S→(S)
S → SS
S→ε
(3) L = {w: w is a sentence in English}
Examples: Mary hit the ball.
Colorless green ideas sleep furiously.
The window needs fixed.
(4) L = {w: w is a C program that halts on all inputs}

Lecture Notes 1 The Three Hour Tour 1

 Encoding Output in the Input String

(5) Encoding multiplication as a single input string

L = {w of the form: <integer>x<integer>=<integer>, where <integer> is any well formed integer, and the third integer is
the product of the first two}
12x9=108 12=12 12x8=108
(6) Encoding prime decomposition
L = {w of the form: <integer1>/<integer2>,<integer3> …, where integers 2 - n represent the prime decomposition of
integer 1.
15/3,5 2/2
More Languages

(7) Sorting as a language recognition task:

L = {w1 # w2: ∃n ≥1,
w1 is of the form int1, int2, … intn,
w2 is of the form int1, int2, … intn, and
w2 contains the same objects as w1 and w2 is sorted}

Examples:
1,5,3,9,6#1,3,5,6,9 ∈ L
1,5,3,9,6#1,2,3,4,5,6,7 ∉ L

(8) Database querying as a language recognition task:

L = {d # q # a:
d is an encoding of a database,
q is a string representing a query, and
a is the correct result of applying q to d}
Example:
(name, age, phone), (John, 23, 567-1234) (Mary, 24, 234-9876 )# (select name age=23) # (John) ∈ L

The Traditional Problems and their Language Formulations are Equivalent

By equivalent we mean:

If we have a machine to solve one, we can use it to build a machine to do the other using just the starting machine and other
functions that can be built using a machine of equal or lesser power.

Consider the multiplication example:

L = {w of the form:
<integer>x<integer>=<integer>, where
<integer> is any well formed integer, and
the third integer is the product of the first two}

Given a multiplication machine, we can build the language recognition machine:

Given the language recognition machine, we can build a multiplication machine:

Lecture Notes 1 The Three Hour Tour 2

 A Framework for Describing Languages

Clearly, if we are going to work with languages, each one must have a finite description.

Finite Languages: Easy. Just list the elements of the language.

L = {June, July, August}

Infinite Languages: Need a finite description.

Grammars let us use recursion to do this.

Grammars 1 Grammars 2

(1) The Language of Matched Parentheses

S→(S)
S → SS
S→ε

(2) The Language of Odd Integers S→O

S→1 S→AO
S→3 A →A D
S→5 A→D
S→7 D→O
S→9 D→E
S→0S O→1
S→1S O→3
S→2S O→5
S→3S O→7
S→4S O→9
S→5S E→ 0
S→6S E→ 2
S→7S E→ 4
S→8S E→ 6
S→9S E→ 8

Grammars 3
(3) The Language of Simple Arithmetic Expressions
S → <exp>
<exp> → <number>
<exp> → (<exp>)
<exp> → - <exp>
<exp> → <exp> <op> <exp>
<op> → + | - | * | /
<number> → <digit>
<number> → <digit> <number>
<digit > → 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Lecture Notes 1 The Three Hour Tour 3

 Grammars as Generators and Acceptors

Top Down Parsing

4 + 3

Bottom Up Parsing

4 + 3

The Language Hierarchy

Recursively Enumerable
Languages

Recursive
Languages

Context-Free
Languages

Regular
Languages

Lecture Notes 1 The Three Hour Tour 4

 Regular Grammars

In a regular grammar, all rules must be of the form:

<one nonterminal> → <one terminal> or ε

or

<one nonterminal> → <one terminal><one nonterminal>

So, the following rules are okay:

S→ε
S→a
S → aS

But these are not:

S → ab
S → SS
aS → b
Regular Expressions and Languages

Regular expressions are formed from ∅ and the characters in the target alphabet, plus the operations of:
• Concatenation: αβ means α followed by β
• Or (Set Union): α∪β means α Or (Union) β
• Kleene *: α* means 0 or more occurrences of α concatenated together.
• At Least 1: α+ means 1 or more occurrences of α concatenated together.
• (): used to group the other operators

Examples:

(1) Odd integers:

(0∪1∪2∪3∪4∪5∪6∪7∪8∪9)*(1∪3∪5∪7∪9)

(2) Identifiers:
(A-Z)+((A-Z) ∪(0-9))*

(3) Matched Parentheses

Context Free Grammars

(1) The Language of Matched Parentheses

S→(S)
S → SS
S→ε

(2) The Language of Simple Arithmetic Expressions

S → <exp>
<exp> → <number>
<exp> → (<exp>)
<exp> → - <exp>
<exp> → <exp> <op> <exp>
<op> → + | - | * | /
<number> → <digit>
<number> → <digit> <number>
<digit > → 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Lecture Notes 1 The Three Hour Tour 5

 Not All Languages are Context-Free
English: S → NP VP
NP → the NP1 | NP1
NP1 → ADJ NP1 | N
N → boy | boys
VP →V | V NP
V → run | runs
What about “boys runs”

A much simpler example: anbncn, n ≥ 1

Unrestricted Grammars

Example: A grammar to generate all strings of the form anbncn, n ≥ 1

S → aBSc
S → aBc
Ba → aB
Bc → bc
Bb → bb

The Language Hierarchy

Recursively Enumerable
Languages

Recursive
Languages

Context-Free
Languages

Regular
Languages

Lecture Notes 1 The Three Hour Tour 6

 A Machine Hierarchy

Finite State Machines 1

An FSM to accept odd integers:

1,3,5,7,9
1,3,5,7,9

0,2,4,6,8
0,2,4,6,8

Finite State Machines 2

An FSM to accept identifiers:

letter
letter or digit

blank, delimiter
delimiter or blank
or digit
anything
Pushdown Automata

A PDA to accept strings with balanced parentheses:

(//(
s
)/(/

Example: (())()

Stack:

Pushdown Automaton 2

A PDA to accept strings of the form w#wR:

a//a a/a/
#//
s f

b//b b/b/

Lecture Notes 1 The Three Hour Tour 7

 A Nondeterministic PDA
A PDA to accept strings of the form wwR

a//a a/a/
ε//
s f

b//b b/b/

PDA 3
A PDA to accept strings of the form anbncn

Turing Machines

A Turing Machine to accept strings of the form anbncn

S
d//R
❑//R
a,e//R b,f//R
a,b,e,f//L
a a/d/R b b/e/R c c/f/L ←

b,c c,d,f,❑ a,d,e,❑

❑,e,f//R

f a,b,c,d n
e,f//R
❑

à ❑ a a b b c c a ❑

Lecture Notes 1 The Three Hour Tour 8

 A Two Tape Turing Machine
A Turing Machine to accept {w#wR}

à ❑ a b a a # a a b a ❑ ❑

A Two Tape Turing Machine to do the same thing

à ❑ a b a a # a a b a ❑ ❑

à ❑ a b a a # a a b a ❑ ❑

Simulating k Tapes with One

A multitrack tape:

à ❑ a b a ❑ ❑
à 0 0 1 0 0 0 0 ❑ ❑
à a b b a b a
0 1 0 0 0 0 0

Can be encoded on a single tape with an alphabet consisting of symbols corresponding to :

{{à,a,b,#,❑} x {0,1} x
{à,a,b,#,❑} x {0,1}}

Example: 2nd square: (❑,0,a,1))

Simulating a Turing Machine with a PDA with Two Stacks

à a b a a # a a b a
Ý

a #
a a
b a
a b
à a

Lecture Notes 1 The Three Hour Tour 9

 The Universal Turing Machine
Encoding States, Symbols, and Transitions

Suppose the input machine M has 5 states, 4 tape symbols, and a transition of the form:

(s,a,q,b), which can be read as:

in state s, reading an a, go to state q, and write b.

We encode this transition as:

q000,a00,q010,a01

A series of transitions that describe an entire machine will look like

q000,a00,q010,a01#q010,a00,q000,a00

The Universal Turing Machine

a a b

a00a00a01

# # #

q000

Church's Thesis
(Church-Turing Thesis)

An algorithm is a formal procedure that halts.

The Thesis: Anything that can be computed by any algorithm can be computed by a Turing machine.

Another way to state it: All "reasonable" formal models of computation are equivalent to the Turing machine. This isn't a formal
statement, so we can't prove it. But many different computational models have been proposed and they all turn out to be
equivalent.
Example: unrestricted grammars
A Machine Hierarchy

FSMs

PDAs

Turing Machines

Lecture Notes 1 The Three Hour Tour 10

 Languages and Machines

Recursively Enumerable
Languages

Recursive
Languages

Context-Free
Languages

Regular
Languages

FSMs

PDAs

Turing Machines

Where Does a Particular Problem Go?

Showing what it is -- generally by construction of:

• A grammar, or a machine
Showing what it isn't -- generally by contradiction, using:
• Counting
Example: anbn
• Closure properties
• Diagonalization
• Reduction

Closure Properties

Regular Lanugages are Closed Under:

§ Union
§ Concatenation
§ Kleene closure
§ Complementation
§ Reversal
§ Intersection

Context Free Languages are Closed Under:

§ Union
§ Concatenation
§ Kleene Closure
§ Reversal
§ Intersection with regular languages
Etc.

Lecture Notes 1 The Three Hour Tour 11

 Using Closure Properties

Example:
L = {anbmcp: n≠m or m ≠ p} is not deterministic context-free.

Two theorems we'll prove later:

Theorem 3.7.1: The class of deterministic context-free languages is closed under complement.

Theorem 3.5.2: The intersection of a context-free language with a regular language is a context-free language.

If L were a deterministic CFL, then the complement of L (L') would be a deterministic CFL.

But L' ∩ a*b*c* = {anbncn}, which we know is not context-free, much less deterministic context-free. Thus a contradiction.

Diagonalization

The power set of the integers is not countable.

Imagine that there were some enumeration:

1 2 3 4 5
Set 1 1
Set 2 1 1
Set 3 1 1
Set 4 1
Set 5 1 1 1 1 1

But then we could create a new set

New Set 1

But this new set must necessarily be different from all the other sets in the supposedly complete enumeration. Yet it should be
included. Thus a contradiction.

More on Cantor

Of course, if we're going to enumerate, we probably want to do it very systematically, e.g.,

1 2 3 4 5 6 7
Set 1 1
Set 2 1
Set 3 1 1
Set 4 1
Set 5 1 1
Set 6 1 1
Set 7 1 1 1

Read the rows as bit vectors, but read them backwards. So Set 4 is 100. Notice that this is the binary encoding of 4.
This enumeration will generate all finite sets of integers, and in fact the set of all finite sets of integers is countable.
But when will it generate the set that contains all the integers except 1?

Lecture Notes 1 The Three Hour Tour 12

 The Unsolvability of the Halting Problem

Suppose we could implement

HALTS(M,x)
M: string representing a Turing Machine
x: string representing the input for M
If M(x) halts then True
else False
Then we could define
TROUBLE(x)
x: string
If HALTS(x,x) then loop forever
else halt

So now what happens if we invoke TROUBLE(TROUBLE), which invokes

HALTS(TROUBLE,TROUBLE)

If HALTS says that TROUBLE halts on itself then TROUBLE loops. IF HALTS says that TROUBLE loops, then TROUBLE
halts.
Viewing the Halting Problem as Diagonalization

First we need an enumeration of the set of all Turing Machines. We'll just use lexicographic order of the encodings we used as
inputs to the Universal Turing Machine. So now, what we claim is that HALTS can compute the following table, where 1 means
the machine halts on the input:

I1 I2 I3 TROUBLE I5
Machine 1 1
Machine 2 1 1
Machine 3
TROUBLE 1 1
Machine 5 1 1 1 1

But we've defined TROUBLE so that it will actually behave as:

TROUBLE 1 1 1

Or maybe HALT said that TROUBLE(TROUBLE) would halt. But then TROUBLE would loop.

Lecture Notes 1 The Three Hour Tour 13

 Decidability

Recursively Enumerable
Languages

Recursive
Languages

Context-Free
Languages

Regular
Languages

Can always say yes or no

Can enumerate from the grammar.

Can say yes by enumerating and checking

Let's Revisit Some Problems

int alpha, beta;
alpha = 3;
beta = (2 + 5) / 10;

(1) Lexical analysis: Scan the program and break it up into variable names, numbers, etc.
(2) Parsing: Create a tree that corresponds to the sequence of operations that should be executed, e.g.,
/

+ 10

2 5

(3) Optimization: Realize that we can skip the first assignment since the value is never used and that we can precompute the
arithmetic expression, since it contains only constants.
(4) Termination: Decide whether the program is guaranteed to halt.
(5) Interpretation: Figure out what (if anything) useful it does.

Lecture Notes 1 The Three Hour Tour 14

 So What's Left?

• Formalize and Prove Things

• Regular Languages and Finite State Machines

• FSMs
• Nondeterminism
• State minimization
• Implementation
• Equivalence of regular expressions and FSMs
• Properties of Regular Languages
• Context-Free Languages and PDAs
• Equivalence of CFGs and nondeterministic PDAs
• Properties of context-free languages
• Parsing and determinism
• Turing Machines and Computability
• Recursive and recursively enumerable languages
• Extensions of Turing Machines
• Undecidable problems for Turing Machines and unrestricted grammars

Lecture Notes 1 The Three Hour Tour 15

 What Is a Language?
Do Homework 2.

Grammars, Languages, and Machines

Language
L

Grammar

Accepts

Machine

Strings: the Building Blocks of Languages

An alphabet is a finite set of symbols: English alphabet: {A, B, C, …, Z}

Binary alphabet: {0, 1}

A string over an alphabet is a finite sequence of symbols drawn from the alphabet.

English string: happynewyear

binary string: 1001101

We will generally omit “ ” from strings unless doing so would lead to confusion.

The set of all possible strings over an alphabet Σ is written Σ*.

binary string: 1001101 ∈ {0,1}*

The shortest string contains no characters. It is called the empty string and is written “ ” or ε (epsilon).

The set of all possible strings over an alphabet Σ is written Σ*.

More on Strings

The length of a string is the number of symbols in it.

|ε| = 0
|1001101| = 7

A string a is a substring of a string b if a occurs contiguously as part of b.

aaa is a substring of aaabbbaaa
aaaaaa is not a substring of aaabbbaaa

Every string is a substring (although not a proper substring) of itself.

ε is a substring of every string. Alternatively, we can match ε anywhere.

Notice the analogy with sets here.

Lecture Notes 2 What is a Language? 1

 Operations on Strings

Concatenation: The concatenation of two strings x and y is written x || y, x⋅y, or xy and is the string formed by appending the
string y to the string x.
|xy| = |x| + |y|

If x = ε and y = “food”, then xy =

If x = “good” and y = “bye”, then |xy| =

Note: x⋅ε = ε⋅x = x for all strings x.

Replication: For each string w and each natural number i, the string wi is defined recursively as
w0 = ε
wi = wi-1 w for each i ≥ 1

Like exponentiation, the replication operator has a high precedence.

Examples:
a3 =
(bye)2 =
a0b3 =

String Reversal

An inductive definition:

(1) If |w| = 0 then wR = w = ε

(2) If |w| ≥ 1 then ∃ a ∈ Σ: w = u⋅a
(a is the last character of w)
and
wR = a⋅uR
Example:
(abc)R =
More on String Reversal

Theorem: If w, x are strings, then (w⋅x)R = xR⋅wR

Example: (dogcat)R = (cat)R⋅(dog)R = tacgod

Proof (by induction on |x|):

Basis: |x| = 0. Then x = ε, and (w⋅x)R = (w⋅ε)R = (w)R = ε⋅wR = εR⋅wR = xR⋅wR

Induction Hypothesis: If |x| ≤ n, then (w⋅x)R = xR⋅wR

Induction Step: Let |x| = n + 1. Then x = u a for some character a and |u| = n

(w⋅x)R = (w⋅ (u⋅a))R

= ((w⋅u)⋅a)R associativity
= a⋅(w⋅u)R definition of reversal
= a⋅uR⋅wR induction hypothesis
= (u⋅a)R⋅wR definition of reversal
= xR⋅wR
dogcat
w x
u a

Lecture Notes 2 What is a Language? 2

 Defining a Language

A language is a (finite or infinite) set of finite length strings over a finite alphabet Σ.

Example: Let Σ = {a, b}

Some languages over Σ: ∅, {ε}, {a, b}, {ε, a, aa, aaa, aaaa, aaaaa}

The language Σ* contains an infinite number of strings, including: ε, a, b, ab, ababaaa

Example Language Definitions

L = {x ∈ {a, b}* : all a's precede all b's}

L = {x : ∃y ∈ {a, b}* : x = ya}

L = {an, n ≥ 0 }

L = an (If we say nothing about the range of n, we will assume that it is drawn from N, i.e., n ≥ 0.)

L = {x#y: x,y ∈ {0-9}* and square(x) = y}

L = {} = ∅ (the empty language—not to be confused with {ε}, the language of the empty string)

Techniques for Defining Languages

Languages are sets. Recall that, for sets, it makes sense to talk about enumerations and decision procedures. So, if we want
to provide a computationally effective definition of a language we could specify either a

• Language generator, which enumerates (lists) the elements of the language, or a

• Language recognizer, which decides whether or not a candidate string is in the language and returns True if it is and
False if it isn't.

Example: The logical definition: L = {x : ∃y ∈ {a, b}* : x = ya} can be turned into either a language generator or a
language recognizer.

How Large are Languages?

• The smallest language over any alphabet is ∅. |∅| = 0

• The largest language over any alphabet is Σ*. |Σ*| = ?
- If Σ = ∅ then Σ* = {ε} and |Σ*| = 1
- If Σ ≠ ∅ then |Σ*| is countably infinite because its elements can be enumerated in 1 to 1 correspondence with the
integers as follows:
1. Enumerate all strings of length 0, then length 1, then length 2, and so forth.
2. Within the strings of a given length, enumerate them lexicographically. E.g., aa, ab, ba, bb

• So all languages are either finite or countably infinite. Alternatively, all languages are countable.

Operations on Languages 1
Normal set operations: union, intersection, difference, complement…
Examples: Σ = {a, b} L1 = strings with an even number of a's
L2 = strings with no b's
L1 ∪ L2 =
L1 ∩ L2 =
L2 - L1 =
¬( L2 - L1) =

Lecture Notes 2 What is a Language? 3

 Operations on Languages 2

Concatenation: (based on the definition of concatenation of strings)

If L1 and L2 are languages over Σ, their concatenation L = L1 L2, sometimes L1⋅L2, is

{w ∈ Σ*: w = x y for some x ∈ L1 and y ∈ L2}

Examples:
L1 = {cat, dog} L2 = {apple, pear} L1 L2 = {catapple, catpear, dogapple, dogpear}
L1 = {an: n ≥ 1} L2 = {an: n ≤ 3} L1 L2 =

Identities:
L∅ = ∅L = ∅ ∀L (analogous to multiplication by 0)
L{ε}= {ε}L = L ∀L (analogous to multiplication by 1)

Replicated concatenation:
Ln = L⋅L⋅L⋅ … ⋅L (n times)
L1 = L
L0 = {ε}

Example:
L = {dog, cat, fish}
L0 = {ε}
L1 = {dog, cat, fish}
L2 = {dogdog, dogcat, dogfish, catdog, catcat, catfish, fishdog, fishcat, fishfish}

Concatenating Languages Defined Using Variables

L1 = an = {an : n ≥ 0} L2 = bn = {bn : n ≥ 0}
L1 L2 = {an : n ≥ 0}{bn : n ≥ 0} = { an bm : n,m ≥ 0} (common mistake: ) ≠ anbn = { an bn : n ≥ 0}

Note: The scope of any variable used in an expression that invokes replication will be taken to be the entire expression.

L = 1n2m
L = anbman

Operations on Languages 3

Kleene Star (or Kleene closure): L* = {w ∈ Σ* : w = w1 w2 … wk for some k ≥ 0 and some w1, w2, … wk ∈ L}

Alternative definition: L* = L0 ∪ L1 ∪ L2 ∪ L3 ∪ …

Note: ∀L, ε ∈ L*

Example:
L = {dog, cat, fish}
L* = {ε, dog, cat, fish, dogdog, dogcat, fishcatfish, fishdogdogfishcat, …}

Another useful definition: L+ = L L* (L+ is the closure of L under concatenation)

Alternatively, L+ = L1 ∪ L2 ∪ L3 ∪ …

L+ = L*-{ε} if ε ∉ L
L+ = L* if ε ∈ L

Lecture Notes 2 What is a Language? 4

 Regular Languages
Read Supplementary Materials: Regular Languages and Finite State Machines: Regular Languages
Do Homework 3.

Regular Grammars, Languages, and Machines

L Regular
Language

Regular Expression
or
Regular Grammar Accepts

Finite
State
Machine

“Pure” Regular Expressions

The regular expressions over an alphabet Σ are all strings over the alphabet Σ ∪ {“(“, “)”, ∅, ∪, *} that can be obtained as
follows:

1. ∅ and each member of Σ is a regular expression.

2. If α , β are regular expressions, then so is αβ
3. If α , β are regular expressions, then so is α∪β.
4. If α is a regular expression, then so is α*.
5. If α is a regular expression, then so is (α).
6. Nothing else is a regular expression.

If Σ = {a,b} then these are regular expressions: ∅, a, bab, a∪b , (a∪b)*a*b*

So far, regular expressions are just (finite) strings over some alphabet, Σ ∪ {“(“, “)”, ∅, ∪, *}.

Regular Expressions Define Languages

Regular expressions define languages via a semantic interpretation function we'll call L:
1. L(∅) = ∅ and L(a) = {a} for each a ∈ Σ
2. If α , β are regular expressions, then
L(αβ) = L(α)⋅L(β)
= all strings that can be formed by concatenating to some string from L(α) some string from L(β).
Note that if either α or β is ∅, then its language is ∅, so there is nothing to concatenate and the result is ∅.
3. If α , β are regular expressions, then L(α∪β) = L(α) ∪ L(β)
4. If α is a regular expression, then L(α*) = L(α)*
5. L( (α) ) = L(α)

A language is regular if and only if it can be described by a regular expression.

A regular expression is always finite, but it may describe a (countably) infinite language.

Lecture Notes 3 Regular Languages 1

 Regular Languages
An equivalent definition of the class of regular languages over an alphabet Σ:
The closure of the languages
{a} ∀a ∈ Σ and ∅ [1]
with respect to the functions:
• concatenation, [2]
• union, and [3]
• Kleene star. [4]

In other words, the class of regular languages is the smallest set that includes all elements of [1] and that is closed under [2],
[3], and [4].

“Closure” and “Closed”

Informally, a set can be defined in terms of a (usually small) starting set and a group of functions over elements from the set.
The functions are applied to members of the set, and if anything new arises, it’s added to the set. The resulting set is called
the closure over the initial set and the functions. Note that the functions(s) may only be applied a finite number of times.

Examples:
The set of natural numbers N can be defined as the closure over {0} and the successor (succ(n) = n+1) function.
Regular languages can be defined as the closure of {a} ∀a∈Σ and ∅ and the functions of concatenation, union, and
Kleene star.

We say a set is closed over a function if applying the function to arbitrary elements in the set does not yield any new elements.

Examples:
The set of natural numbers N is closed under multiplication.
Regular languages are closed under intersection.

See Supplementary Materials—Review of Mathematical Concepts for more formal definitions of these terms.

Examples of Regular Languages

L( a*b* ) =
L( (a ∪ b) ) =
L( (a ∪ b)* ) =
L( (a∪b)*a*b*) =
L = {w ∈ {a,b}* : |w| is even}
L = {w ∈ {a,b}* : w contains an odd number of a's}

Augmenting Our Notation

It would be really useful to be able to write ε in a regular expression.
Example: (a ∪ ε) b (Optional a followed by b)

But we'd also like a minimal definition of what constitutes a regular expression. Why?

Observe that
∅0 = {ε} (since 0 occurrences of the elements of any set generates the empty string), so
∅* = {ε}

So, without changing the set of languages that can be defined, we can add ε to our notation for regular expressions if we
specify that
L(ε) = {ε}
We're essentially treating ε the same way that we treat the characters in the alphabet.
Having done this, you'll probably find that you rarely need ∅ in any regular expression.

Lecture Notes 3 Regular Languages 2

 More Regular Expression Examples

L( (aa*) ∪ ε ) =
L( (a ∪ ε)* ) =
L = { w ∈ {a,b}* : there is no more than one b}
L = { w ∈ {a,b}* : no two consecutive letters are the same}

Further Notational Extensions of Regular Expressions

• A fixed number of concatenations: αn means αααα…α (n times).

• At Least 1: α+ means 1 or more occurrences of α concatenated together.
• Shorthands for denoting sets, such as ranges, e.g., (A-Z) or (letter-letter)
Example: L = (A-Z)+((A-Z)∪(0-9))*

• A replicated regular expression αn, where n is a constant.

Example: L = (0 ∪ 1)20

• Intersection: α∩β (we’ll prove later that regular languages are closed under intersection)
Example: L = (a3)* ∩ (a5)*

Operator Precedence in Regular Expressions

Regular expressions are strings in the language of regular expressions. Thus to interpret them we need to:
1. Parse the string
2. Assign a meaning to the parse tree
Parsing regular expressions is a lot like parsing arithmetic expressions. To do it, we must assign precedence to the operators:
Regular Arithmetic
Expressions Expressions

Highest Kleene star exponentiation

concatenation
multiplication
intersection

Lowest union addition

a b* ∪ c d* x y2 + i j2

Regular Expressions and Grammars

Recall that grammars are language generators. A grammar is a recipe for creating strings in a language.

Regular expressions are analogous to grammars, but with two special properties:

1. The have limited power. They can be used to define only regular languages.
2. They don't look much like other kinds of grammars, which generally are composed of sets of production rules.

But we can write more "standard" grammars to define exactly the same languages that regular expressions can define.
Specifically, any such grammar must be composed of rules that:

• have a left hand side that is a single nonterminal

• have a right hand side that is ε, or a single terminal, or a single terminal followed by a single nonterminal.

Lecture Notes 3 Regular Languages 3

 Regular Grammar Example
L={w ∈ {a, b}* : |w| is even}

((aa) ∪ (ab) ∪ (ba) ∪ (bb))* Notice how these rules correspond naturally to a FSM:

S→ε a, b
S → aT
S → bT S T
T→a
T→b a, b
T → aS
T → bS

Generators and Recognizers

Generator Recognizer

Language

Regular Languages

Regular Expressions
Regular Grammars ?

Lecture Notes 3 Regular Languages 4

 Finite State Machines
Read K & S 2.1
Do Homeworks 4 & 5.
Finite State Machines 1

A DFSM to accept odd integers:

Definition of a Deterministic Finite State Machine (DFSM)

M = (K, Σ, δ, s, F), where K is a finite set of states

Σ is an alphabet
s ∈ K is the initial state
F ⊆ K is the set of final states, and
δ is the transition function. It is function from (K × Σ) to K
i.e., each element of δ maps from: a state, input symbol pair to a new state.

Informally, M accepts a string w if M winds up in some state that is an element of F when it has finished reading w (if not, it
rejects w).

The language accepted by M, denoted L(M), is the set of all strings accepted by M.

Deterministic finite state machines (DFSMs) are also called deterministic finite state automata (DFSAs or DFAs).

Computations Using FSMs

A computation of A FSM is a sequence of configurations, where a configuration is any element of K ×Σ*.

The yields relation |-M:
(q, w) |-M (q', w') iff
• w = a w' for some symbol a ∈ Σ, and
• δ (q, a) = q'
(The yields relation effectively runs M one step.)

|-M * is the reflexive, transitive closure of |-M.

(The |-M* relation runs M any number of steps.)

Formally, a FSM M accepts a string w iff

(s, w) |-M * (q, ε), for some q ∈ F.

An Example Computation

A DFSM to accept odd integers:

On input 235, the configurations are:
(q0, 235) |-M (q0, 35)
|-M
|-M

Thus (q0, 235) |-M* (q1, ε). (What does this mean?)

Lecture Notes 4 Finite State Machines 1

 Finite State Machines 2

A DFSM to accept $.50 in change:

More Examples

((aa) ∪ (ab) ∪ (ba) ∪ (bb))*

(b ∪ ε)(ab)*(a ∪ ε)

More Examples

L1 = {w ∈ {a, b}* : every a is immediately followed a b}

A regular expression for L1:

A DFSM for L1:

L2 = {w ∈ {a, b}* : every a has a matching b somewhere before it}

A regular expression for L2:

A DFSM for L2:

Lecture Notes 4 Finite State Machines 2

 Another Example: Socket-based Network Communication

Client Server Σ = {Open, Req, Reply, Close}

open socket
send request
send reply L = Open (Req Reply)* (Req ∪ ε) Close
send request
send reply
… M=
close socket

Definition of a Deterministic Finite State Transducer (DFST)

M = (K, Σ, O, δ, s, F), where

K is a finite set of states

Σ is an input alphabet
O is an output alphabet
s ∈ K is the initial state
F ⊆ K is the set of final states, and
δ is the transition function. It is function from
(K × Σ) to (K × O*)
i.e., each element of δ maps from: a state, input symbol pair
to : a new state and zero or more output symbols (an output string)

M computes a function M(w) if, when it reads w, it outputs M(w).

Theorem: The output language of a deterministic finite state transducer (on final state) is regular.

A Simple Finite State Transducer

Convert 1's to 0's and 0's to 1's (this isn't just a finite state task -- it's a one state task)

1/0

q0

0/1

An Odd Parity Generator

After every three bits, output a fourth bit such that each group of four bits has odd parity.

Lecture Notes 4 Finite State Machines 3

 Nondeterministic Finite State Machines
Read K & S 2.2, 2.3
Read Supplementary Materials: Regular Languages and Finite State Machines: Proof of the Equivalence of Nondeterministic
and Deterministic FSAs.
Do Homework 6.

Definition of a Nondeterministic Finite State Machine (NDFSM/NFA)

M = (K, Σ, ∆, s, F), where

K is a finite set of states

Σ is an alphabet
s ∈ K is the initial state
F ⊆ K is the set of final states, and
∆ is the transition relation. It is a finite subset of
(K × (Σ ∪ {ε})) × K
i.e., each element of ∆ contains:
a configuration (state, input symbol or ε), and a new state.

M accepts a string w if there exists some path along which w drives M to some element of F.

The language accepted by M, denoted L(M), is the set of all strings accepted by M, where computation is defined analogously to
DFSMs.

A Nondeterministic FSA
L= {w : there is a symbol ai∈Σ not appearing in w}

The idea is to guess (nondeterministically) which character will be the one that doesn't appear.

Another Nondeterministic FSA

L1= {w : aa occurs in w}
L2= {x : bb occurs in x}
L3= {y : ∈ L1 or L2 }

M1 = a a a, b

10 11 12
b
b

M2= b b a, b

20 21 22
a
a
M3=

Lecture Notes 5 Nondeterministic Finite State Machines 1

 Analyzing Nondeterministic FSAs
a
a
b

b b a

Does this FSA accept: baaba

Remember: we just have to find one accepting path.

Nondeterministic and Deterministic FSAs

Clearly, {Languages accepted by a DFSA} ⊆ {Languages accepted by a NDFSA}

(Just treat δ as ∆)
More interestingly, Theorem: For each NDFSA, there is an equivalent DFSA.
Proof: By construction

b,c
ε ¬a
q1

a,c
q0 ε ¬b
q2

a,b
ε ¬c
q3

Another Nondeterministic Example

b* (b(a ∪ c)c ∪ b(a ∪ b) (c ∪ ε))* b

c
ε a, c
b 3 4
ε b c
1 2
b a,b c, ε
5 6 7

c, ε b

Lecture Notes 5 Nondeterministic Finite State Machines 2

 A “Real” Example

See enemy
Found by enemy
Hide Run See sword
Coast clear
See sword
Found by enemy See laser
Brother
kills enemy Reach for Sword Pick up
Laser
Sword picked up

Swing Sword

Get stabbed Kill enemy

Die
Kill enemy

Become King

Dealing with ε Transitions

E(q) = {p ∈ K : (q,w) |-*M (p, w}. E(q) is the closure of {q} under the relation {(p,r) : there is a transition (p, ε, r) ∈ ∆}
An algorithm to compute E(q):

Defining the Deterministic FSA

Given a NDFSA M = (K, Σ, ∆, s, F),

we construct M' = (K', Σ, δ', s', F'), where
K' = 2K
s' = E(s)
F' = {Q ⊆ K : Q ∩ F ≠ ∅} b,c
δ' (Q, a) = ∪{E(p) : p ∈ K and (q, a, p) ∈ ∆ ε ¬a
q1
for some q ∈ Q}
Example: computing δ' for the missing letter machine
s' = {q0, q1, q2, q3}
a,c
δ' = { ({q0, q1, q2, q3}, a, {q2, q3}),
q0 ε ¬b
({q0, q1, q2, q3}, b, {q1, q3}),
q2
({q0, q1, q2, q3}, c, {q1, q2}),
({q1, q2}, a, {q2}), ({q1, q2}, b, {q1}), ({q1, q2}, c, {q1, q2})
({q1, q3}, a, {q3}), ({q1, q3}, b, {q1, q3}), ({q1, q3}, c, {q1})
a,b
({q2, q3}, a, {q2, q3}), ({q2, q3}, b, {q3}), ({q2, q3}, c, {q2})
ε ¬c
({q1}, b, {q1}), ({q1}, c, {q1})
q3
({q2}, a, {q2}), ({q2}, c, {q2})
({q3}, a, {q3}), ({q3}, b, {q3}) }

Lecture Notes 5 Nondeterministic Finite State Machines 3

 An Algorithm for Constructing the Deterministic FSA
1. Compute the E(q)s:
2. Compute s' = E(s)
3. Compute δ':
δ' (Q, a) = ∪{E(p) : p ∈ K and (q, a, p) ∈ ∆ for some q ∈ Q}
4. Compute K' = a subset of 2K
5. Compute F' = {Q ∈ K' : Q ∩ F ≠ ∅ }

An Example - The Or Machine

L1= {w : aa occurs in w}
L2= {x : bb occurs in x}
L3= {y : ∈ L1 or L2 }

a a a, b
b
10 11 12

ε b

00

ε
b b a, b
a
20 21 22

a
Another Example
b* (b(a ∪ c)c ∪ b(a ∪ b) (c ∪ ε))* b

c
ε a, c
b 3 4
ε b c
1 2
b a, b c, ε
5 6 7

c, ε b

8
E(q) =

δ' =

Lecture Notes 5 Nondeterministic Finite State Machines 4

 Sometimes the Number of States Grows Exponentially

Example: The missing letter machine, with |Σ| = n b,c

No. of states after 0 chars: 1 ε ¬a
q1
æ n ö
No. of new states after 1 char: ç ÷ =n
è n − 1ø
a,c
æ n ö q0 ε ¬b
No. of new states after 2 chars: ç ÷ = n(n-1)/2
è n − 2ø q2

æ n ö
No. of new states after 3 chars: ç ÷ = n(n-1)(n-2)/6
è n − 3ø a,b
ε ¬c
Total number of states after n chars: 2n
q3

What If The Original FSA is Deterministic?

M=
1. Compute the E(q)s:
1,3,5,7,9 2. s' = E(q0) =
1,3,5,7,9
3. Compute δ'
({q0}, odd, {q1})
q0 q1 ({q0}, even, {q0})
({q1}, odd, {q1})
0,2,4,6,8 ({q1}, even, {q0})
4. K' = {{q0}, {q1}}
0,2,4,6,8
5. F' = {{q1}}
M' = M

The real meaning of “determinism”

A FSA is deterministic if, for each input and state, there is at most one possible transition.

DFSAs are always deterministic. Why?

NFSAs can be deterministic (even with ε-transitions and implicit dead states), but the formalism allows nondeterminism,
in general.

Determinism implies uniquely defined machine behavior.

Lecture Notes 5 Nondeterministic Finite State Machines 5

 Interpreters for Finite State Machines

Deterministic FSAs as Algorithms

Example: No more than one b S: s := get-next-symbol;

if s = end-of-file then accept;
else if s = a then go to S;
a a a,b else if s = b then go to T;
b b T: s:= get-next-symbol;
S T U if s = end-of-file then accept;
else if s = a then go to T;
else if s = b then go to U;
Length of Program: |K| × (|Σ| + 2) etc.
Time required to analyze string w: O(|w| × |Σ|)

We have to write new code for every new FSM.

Until accept or reject do:

A Deterministic FSA Interpreter

To simulate M = (K, Σ, δ, s, F): Simulate the no more than one b machine on input: aabaa

ST := s;
Repeat
i := get-next-symbol;
if i ≠ end-of-string then
ST := δ(ST, i)
Until i = end-of-string;
If ST ∈ F then accept else reject

Nondeterministic FSAs as Algorithms

Real computers are deterministic, so we have three choices if we want to execute a nondeterministic FSA:

1. Convert the NDFSA to a deterministic one:

• Conversion can take time and space 2K.
• Time to analyze string w: O(|w|)

2. Simulate the behavior of the nondeterministic one by constructing sets of states "on the fly" during execution
• No conversion cost
• Time to analyze string w: O(|w| × K2)

3. Do a depth-first search of all paths through the nondeterministic machine.

Lecture Notes 6 Interpreters for Finite State Machines 1

 A Nondeterministic FSA Interpreter

To simulate M = (K, Σ, ∆, s, F):

SET ST;
ST := E(s);
Repeat
i := get-next-symbol;
if i ≠ end-of-string then
ST1 := ∅
For all q ∈ ST do
For all r ∈ ∆(q, i) do
ST1 := ST1 ∪ E(r);
ST := ST1;

Until i = end-of-string;
If ST ∩ F ≠ ∅ then accept else reject

A Deterministic Finite State Transducer Interpreter

To simulate M = (K, Σ, O, δ, s, F), given that:

δ1(state, symbol) returns a single new state
(i.e., M is deterministic), and
δ2(state, symbol) returns an element of O*, the
string to be output.

ST := s;
Repeat:
i := get-next-symbol;
if i ≠ end-of-string then
write(δ2(ST, i));
ST := δ1(ST, i)
Until i = end-of-string;
If ST ∈ F then accept else reject

Lecture Notes 6 Interpreters for Finite State Machines 2

 Equivalence of Regular Languages and FSMs
Read K & S 2.4
Read Supplementary Materials: Regular Languages and Finite State Machines: Generating Regular Expressions from Finite
State Machines.
Do Homework 8.

Equivalence of Regular Languages and FSMs

Theorem: The set of languages expressible using regular expressions (the regular languages) equals the class of languages
recognizable by finite state machines. Alternatively, a language is regular if and only if it is accepted by a finite state machine.

Proof Strategies

Possible Proof Strategies for showing that two sets, a and b are equal (also for iff):

1. Start with a and apply valid transformation operators until b is produced.

Example:
Prove:
A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C)
A ∩ (B ∪ C) = (B ∪ C) ∩ A commutativity
= (B ∩ A) ∪ (C ∩ A) distributivity
= (A ∩ B) ∪ (A ∩ C) commutativity

2. Do two separate proofs: (1) a Þ b, and (2) b Þa, possibly using totally different techniques. In this case, we show first (by
construction) that for every regular expression there is a corresponding FSM. Then we show, by induction on the number of
states, that for every FSM, there is a corresponding regular expression.

For Every Regular Expression There is a Corresponding FSM

We'll show this by construction.

Example:

a*(b ∪ ε)a*

Review - Regular Expressions

The regular expressions over an alphabet Σ* are all strings over the alphabet Σ ∪ {(, ), ∅, ∪, *} that can be obtained as follows:
1. ∅ and each member of Σ is a regular expression.
2. If α , β are regular expressions, then so is αβ.
3. If α , β are regular expressions, then so is α∪β.
4. If α is a regular expression, then so is α*.
5. If α is a regular expression, then so is (α).
6. Nothing else is a regular expression.

We also allow ε and α+, etc. but these are just shorthands for ∅* and αα*, etc. so they do not need to be considered for
completeness.

Lecture Notes 7 Equivalence of Regular Languages and FSMs 1

 For Every Regular Expression There is a Corresponding FSM

Formalizing the Construction: The class of regular languages is the smallest class of languages that contains ∅ and each of the
singleton strings drawn from Σ, and that is closed under
• Union
• Concatenation, and
• Kleene star
Clearly we can construct an FSM for any finite language, and thus for ∅ and all the singleton strings. If we could show that the
class of languages accepted by FSMs is also closed under the operations of union, concatenation, and Kleene star, then we could
recursively construct, for any regular expression, the corresponding FSM, starting with the singleton strings and building up the
machine as required by the operations used to express the regular expression.

FSMs for Primitive Regular Expressions

An FSM for ∅: An FSM for ε (∅*):

An FSM for a single element of Σ:

Closure of FSMs Under Union

To create a FSM that accepts the union of the languages accepted by machines M1 and M2:
1. Create a new start state, and, from it, add ε-transitions to the start states of M1 and M2.

Closure of FSMs Under Concatenation

To create a FSM that accepts the concatenation of the languages accepted by machines M1 and M2:
1. Start with M1.
2. From every final state of M1, create an ε-transition to the start state of M2.
3. The final states are the final states of M2.

Lecture Notes 7 Equivalence of Regular Languages and FSMs 2

 Closure of FSMs Under Kleene Star

To create an FSM that accepts the Kleene star of the language accepted by machine M1:
1. Start with M1.
2. Create a new start state S0 and make it a final state (so that we can accept ε).
3. Create an ε-transition from S0 to the start state of M1.
4. Create ε-transitions from all of M1's final states back to its start state.
5. Make all of M1's final states final.

Note: we need a new start state, S0, because the start state of the new machine must be a final state, and this may not be true of
M1's start state.

Closure of FSMs Under Complementation

To create an FSM that accepts the complement of the language accepted by machine M1:
1. Make M1 deterministic.
2. Reverse final and nonfinal states.

A Complementation Example
a
b
q1 q2

Closure of FSMs Under Intersection

L1 ∩ L2 =
L1 L2

Write this in terms of operations we have already proved closure for:

• Union
• Concatenation
• Kleene star
• Complementation
An Example

(b ∪ ab*a)*ab*

Lecture Notes 7 Equivalence of Regular Languages and FSMs 3

 For Every FSM There is a Corresponding Regular Expression

Proof:
(1) There is a trivial regular expression that describes the strings that can be recognized in going from one state to itself ({ε} plus
any other single characters for which there are loops) or from one state to another directly (i.e., without passing through any other
states), namely all the single characters for which there are transitions.

(2) Using (1) as the base case, we can build up a regular expression for an entire FSM by induction on the number assigned to
possible intermediate states we can pass through. By adding them in only one at a time, we always get simple regular
expressions, which can then be combined using union, concatenation, and Kleene star.

Key Ideas in the Proof

Idea 1: Number the states and, at each induction step, increase by one the states that can serve as intermediate states.

b
a a
1 2 3

I K J

Idea 2: To get from state I to state J without passing through any intermediate state numbered greater than K, a machine may
either:
1. Go from I to J without passing through any state numbered greater than K-1 (which we'll take as the induction hypothesis), or
2. Go from I to K, then from K to K any number of times, then from K to J, in each case without passing through any
intermediate states numbered greater than K-1 (the induction hypothesis, again).
So we'll start with no intermediate states allowed, then add them in one at a time, each time building up the regular expression
with operations under which regular languages are closed.

The Formula

Adding in state k as an intermediate state we can use to go from i to j, described using paths that don't use k:

i k j

R(i, j, k) = R(i, j, k - 1) /* what you could do without k

∪
R(i, k, k-1) /* go from i to the new intermediate state without using k or higher
°
R(k, k, k-1)* /* then go from the new intermediate state back to itself as many times as you want
°
R(k, j, k-1) /* then go from the new intermediate state to j without using k or higher

Solution: ∪ R(s, q, N) ∀q ∈ F

Lecture Notes 7 Equivalence of Regular Languages and FSMs 4

 An Example of the Induction

b
a a a
1 2 3 4

Going through no intermediate states:

(1,1,0) = ε (1,2,0) = a (1, 3, 0) = ∅ (2,3,0) = a (3,3,0) = ε ∪ b (3,4,0) = a
Allow 1 as an intermediate state:

Allow 2 as an intermediate state:

(1, 3, 2) = (1, 3, 1) ∪ (1, 2, 1)(2, 2, 1)*(2, 3, 1)
= ∅ ∪ a ε* a
= aa
Allow 3 as an intermediate state:
(1, 3, 3) = (1, 3, 2) ∪ (1, 3, 2)(3, 3, 2)*(3, 3, 2)
= aa ∪ aa (ε ∪ b)* (ε ∪ b)
= aab*
(1, 4, 3) = (1, 4, 2) ∪ (1, 3, 2)(3, 3, 2)*(3, 4, 2)
= ∅ ∪ aa (ε ∪ b)* a
= aab*a

An Easier Way - See Packet

a
1 2
b
b a
b 3

(1) Create a new initial state and a new, unique final state, neither of which is part of a loop.

ε a
4 1 2
b
b a
b 3

a
ε ε

Lecture Notes 7 Equivalence of Regular Languages and FSMs 5

(2) Remove states and arcs and replace with arcs labelled with larger and larger regular expressions. States can be removed in
any order, but don’t remove either the start or final state.

ε a
4 1 2
b
ba*b
aa*b

ε ε

(Notice that the removal of state 3 resulted in two new paths because there were two incoming paths to 3 from another state and 1
outgoing path to another state, so 2×1 = 2.) The two paths from 2 to 1 should be coalesced by unioning their regular expressions
(not shown).

ε ab ∪ aaa*b ∪ ba*b
4 1

ε a

(ab ∪ aaa*b ∪ ba*b)*(a ∪ ε)

4 5

Thus, the equivalent regular expression is:

(ab ∪ aaa*b ∪ ba*b)*(a ∪ ε)

Using Regular Expressions in the Real World (PERL)

Matching floating point numbers:

-? ([0-9]+(\.[0-9]*)? | \.[0-9]+)

Matching IP addresses:

([0-9]+ (\ . [0-9]+) {3})

Finding doubled words:

\< ([A-Za-z]+) \s+ \1 \>

From Friedl, J., Mastering Regular Expressions, O’Reilly,1997.

Note that some of these constructs are more powerful than regular expressions.
Lecture Notes 7 Equivalence of Regular Languages and FSMs 6
 Regular Grammars and Nondeterministic FSAs

Any regular language can be defined by a regular grammar, in which all rules
• have a left hand side that is a single nonterminal
• have a right hand side that is ε, a single terminal, a single nonterminal, or a single terminal followed by a single nonterminal.

Example: L={w ∈ {a, b}* : |w| is even}

((aa) ∪ (ab) ∪ (ba) ∪ (bb))*

S→ε T→a
S → aT T→b
S → bT T → aS
T → bS

a, b

S T

a, b

An Algorithm to Generate the NDFSM from a Regular Grammar

1. Create a nonterminal for each state in the NDFSM.

2. s is the start state.
3. If there are any rules of the form X → w, for some w∈Σ, then create an additional state labeled #.
4. For each rule of the form X → w Y, add a transition from X to Y labeled w (w ∈ Σ ∪ ε).
5. For each rule of the form X → w, add a transition from X to # labeled w (w ∈ Σ).
6. For each rule of the form X → ε, mark state X final.
7. Mark state # final.

Example 1 - Even Length Strings

S→ε T→a
S → aT T→b
S → bT T → aS
T → bS

Example 2 - One Character Missing

S→ε A → bA C → aC
S → aB A → cA C → bC
S → aC A→ε C→ε
S → bA B → aB
S → bC B → cB
S → cA B→ε
S → cB

Lecture Notes 7 Equivalence of Regular Languages and FSMs 7

 An Algorithm to Generate a Regular Grammar from an NDFSM

1. Create a nonterminal for each state in the NDFSM.

2. The start state becomes the starting nonterminal
3. For each transition δ(T, a) = U, make a rule of the form T → aU.
4. For each final state T, make a rule of the form T → ε.

Example:
a
b
X Y

a
b

Conversion Algorithms between Regular Language Formalisms

Regular
Grammar

NFSM
(NFA)
Regular
Expression

DFSM
(DFA)

Lecture Notes 7 Equivalence of Regular Languages and FSMs 8

 Languages That Are and Are Not Regular
Read L & S 2.5, 2.6
Read Supplementary Materials: Regular Languages and Finite State Machines: The Pumping Lemma for Regular Languages.
Do Homework 9.
Deciding Whether a Language is Regular

Theorem: There exist languages that are not regular.

Lemma: There are an uncountable number of languages.
Proof of Lemma:
Let: Σ be a finite, nonempty alphabet, e.g., {a, b, c}.

Then Σ* contains all finite strings over Σ.

e.g., {ε, a, b, c, aa, ab, bc, abc, bba, bbaa, bbbaac}

Σ* is countably infinite, because its elements can be enumerated one at a time, shortest first.

Any language L over Σ is a subset of Σ*, e.g., L1 = {a, aa, aaa, aaaa, aaaaa, …}
L2 = {ab, abb, abbb, abbbb, abbbbb, …}
The set of all possible languages is thus the power set of Σ*.

The power set of any countably infinite set is not countable. So there are an uncountable number of languages over Σ*.

Some Languages Are Not Regular

Theorem: There exist languages that are not regular.
Proof:
(1) There are a countably infinite number of regular languages. This true because every description of a regular language is of
finite length, so there is a countably infinite number of such descriptions.
(2) There are an uncountable number of languages.

Thus there are more languages than there are regular languages. So there must exist some language that is not regular.

Showing That a Language is Regular

Techniques for showing that a language L is regular:

1. Show that L has a finite number of elements.
2. Exhibit a regular expression for L.
3. Exhibit a FSA for L.
4. Exhibit a regular grammar for L.
5. Describe L as a function of one or more other regular languages and the operators ⋅, ∪, ∩, *, -, ¬. We use here the fact that
the regular languages are closed under all these operations.
6. Define additional operators and prove that the regular languages are closed under them. Then use these operators as in 5.

Example
Let Σ = {0, 1, 2, … 9}
Let L ⊆ Σ* be the set of decimal representations for nonnegative integers (with no leading 0's) divisible by 2 or 3.

L1 = decimal representations of nonnegative integers without leading 0's.

L1 = 0 ∪ {1, 2, … 9}{0 - 9}*
So L1 is regular.

L2 = decimal representations of nonnegative integers without leading 0's divisible by 2

L2 = L1 ∩ Σ*{0, 2, 4, 6, 8}
So L2 is regular.

Lecture Notes 8 Languages That Are and Are Not Regular 1

 Example, Continued
L3 = L1 and divisible by 3

Recall that a number is divisible by 3 if and only if the sum of its digits is divisible by 3. We can build a FSM to determine that
and accept the language L3a, which is composed of strings of digits that sum to a multiple of 3.

L3 = L1 ∩ L3a

Finally, L = L2 ∪ L3
Another Example

Σ = {0 - 9}
L = {w : w is the social security number of a living US resident}

Finiteness - Theoretical vs. Practical

Any finite language is regular. The size of the language doesn't matter.

Parity Soc. Sec. #

Checking Checking

But, from an implementation point of view, it very well may.

When is an FSA a good way to encode the facts about a language?

What are our alternatives?

FSA's are good at looking for repeating patterns. They don't bring much to the table when the language is just a set of unrelated
strings.

Showing that a Language is Not Regular

The argument, “I can't find a regular expression or a FSM”, won't fly. (But a proof that there cannot exist a FSM is ok.)

Instead, we need to use two fundamental properties shared by regular languages:

1. We can only use a finite amount of memory to record essential properties.

Example:
anbn is not regular

2. The only way to generate/accept an infinite language with a finite description is to use Kleene star (in regular expressions) or
cycles (in automata). This forces some kind of simple repetitive cycle within the strings.
Example:
ab*a generates aba, abba, abbba, abbbba, etc.
Example:
{an : n ≥ 1 is a prime number} is not regular.

Lecture Notes 8 Languages That Are and Are Not Regular 2

 Exploiting the Repetitive Property

b
b a a b

If a FSM of n states accepts any string of length ≥ n, how many strings does it accept?

L = bab*ab n
________
babbbbab
x y z

xy*z must be in L.

So L includes: baab, babab, babbab, babbbbbbbbbbab

The Pumping Lemma for Regular Languages

If L is regular, then
∃ N ≥ 1, such that
∀ strings w ∈ L, where |w| ≥ N,
∃ x, y, z, such that w = xyz
and |xy| ≤ N,
and y ≠ ε,
and ∀ q ≥ 0, xyqz is in L.

Example: L = anbn

aaaaaaaaaabbbbbbbbbb
x y z

∃N≥1 Call it N
∀ long strings w We pick one
∃ x, y, z We show no x, y, z

Example: anbn is not Regular

N is the number from the pumping lemma (or one more, if N is odd).

Choose w = aN/2bN/2. (Since this is what it takes to be “long enough”: |w| ≥ N)

1 2
aaaaaaaaaa bbbbbbbbbb
x y z

We show that there is no x, y, z with the required properties:

|xy| ≤ N,
y ≠ ε,
∀ q ≥ 0, xyqz is in L.

Three cases to consider:

• y falls in region 1:

• y falls across regions 1 and 2:

• y falls in region 3:

Lecture Notes 8 Languages That Are and Are Not Regular 3

 Example: anbn is not Regular
Second try:

Choose w to be be aNbN. (Since we get to choose any w in L.)

1 2
aaaaaaaaaa bbbbbbbbbb
x y z

We show that there is no x, y, z with the required properties:

|xy| ≤ N,
y ≠ ε,
∀ q ≥ 0, xyqz is in L.

Since |xy| ≤ N, y must be in region 1. So y = ag for some g ≥ 1. Pumping in or out (any q but 1) will violate the constraint that the
number of a’s has to equal the number of b’s.

A Complete Proof Using the Pumping Lemma

Proof that L = {anbn} is not regular:

Suppose L is regular. Since L is regular, we can apply the pumping lemma to L. Let N be the number from the pumping lemma
for L. Choose w = aNbN. Note that w ∈ L and |w| ≥ N. From the pumping lemma, there exists some x, y, z where xyz = w and
|xy| ≤ N, y ≠ ε, and ∀ q ≥ 0, xyqz ∈L. Because |xy| ≤ N, y = a|y| (y is all a’s). We choose q = 2 and xyqz = aN+|y|bN. Because |y| >
0, then xy2z ∉ L (the string has more a’s than b’s). Thus for all possible x, y, z: xyz = w, ∃q, xyqz ∉ L. Contradiction. ∴ L is
not regular.

Note: the underlined parts of the above proof is “boilerplate” that can be reused. A complete proof should have this text or
something equivalent.

You get to choose w. Make it a single string that depends only on N. Choose w so that it makes your proof easier.
You may end up with various cases with different q values that reach a contradiction. You have to show that all possible cases
lead to a contradiction.

Proof of the Pumping Lemma

Since L is regular it is accepted by some DFSA, M. Let N be the number of states in M. Let w be a string in L of length N or
more.

N
aaaaaaaaaabbbbbbbbbb
x y
x y

Then, in the first N steps of the computation of M on w, M must visit N+1 states. But there are only N different states, so it must
have visited the same state more than once. Thus it must have looped at least once. We'll call the portion of w that corresponds
to the loop y. But if it can loop once, it can loop an infinite number of times. Thus:
• M can recognize xyqz for all values of q ≥ 0.
• y ≠ ε (since there was a loop of length at least one)
• |xy| ≤ N (since we found y within the first N steps of the computation)

Lecture Notes 8 Languages That Are and Are Not Regular 4

 Another Pumping Example

L = {w=aJbK : K > J} (more b's than a's)

Choose w = aNbN+1

N
aaaaaaaaaabbbbbbbbbbb
x y z

We are guaranteed to pump only a's, since |xy| ≤ N. So there exists a number of copies of y that will cause there to be more a's
than b's, thus violating the claim that the pumped string is in L.

A Slightly Different Example of Pumping

L = {w=aJbK : J > K} (more a's than b's)

Choose w = aN+1bN
N
aaaaaaaaaabbbbbbbbbbb
x y z

We are guaranteed that y is a string of at least one a, since |xy| ≤ N. But if we pump in a's we get even more a's than b's, resulting
in strings that are in L.

What can we do?

Another Slightly Different Example of Pumping

L = {w=aJbK : J ≥ K}

Choose w = aN+1bN

N
aaaaaaaaaabbbbbbbbbbb
x y z

We are guaranteed that y is a string of at least one a, since |xy| ≤ N. But if we pump in a's we get even more a's than b's, resulting
in strings that are in L.

If we pump out, then if y is just a then we still have a string in L.

What can we do?

Lecture Notes 8 Languages That Are and Are Not Regular 5

 Another Pumping Example
L = abanbn

Choose w = abaNbN
N
abaaaaaaaaaabbbbbbbbbbb
x y z

What are the choices for (x, y):

(ε, a)
(ε, ab)
(ε, aba+)
(a, b)
(a, ba+)
(aba*, a+)

What if L is Regular?

Given a language L that is regular, pumping will work: L = (ab)* Choose w = (ab)N

There must exist an x, y, and z where y is pumpable.

abababab ababab abababababab

x y z

Suppose y = ababab Then, for all q ≥ 0, x yqz ∈ L

Note that this does not prove that L is regular. It just fails to prove that it is not.

Using Closure Properties

Once we have some languages that we can prove are not regular, such as anbn, we can use the closure properties of regular
languages to show that other languages are also not regular.

Example: Σ = {a, b}
L = {w : w contains an equal number of a's and b's }
a*b* is regular. So, if L is regular, then L1 = L ∩ a*b* is regular.

But L1 is precisely anbn. So L is not regular.

Don’t Try to Use Closure Backwards

One Closure Theorem:
If L1 and L2 are regular, then so is L3 = L1 ∩ L2.

But what if L3 and L1 are regular? What can we say about L2?

L3 = L1 ∩ L2.

Example: ab = ab ∩ anbn

Lecture Notes 8 Languages That Are and Are Not Regular 6

 A Harder Example of Pumping

Σ = {a}
L = {w = aK : K is a prime number} |x| + |z| is prime.
|x| + |y| + |z| is prime.
N |x| + 2|y| + |z| is prime.
aaaaaaaaaaaaa |x| + 3|y| + |z| is prime, and so forth.
x y z

Distribution of primes:
||| | | | | | | | | | |
||| | | | | | | | | | |

Distribution of |x| + q|y| + |z|:

| | | | | | | | | | |
| | | | | | | | | | |

But the Prime Number Theorem tells us that the primes "spread out", i.e., that the number of primes not exceeding x is
asymptotic to x/ln x.

Note that when q = |x| + |z|, |xyqz| = (|y| + 1)×(|x| + |z|), which is composite (non-prime) if both factors are > 1. If you’re careful
about how you choose N in a pumping lemma proof, you can make this true for both factors.

Automata Theory is Just the Scaffolding

Our results so far give us tools to:

• Show a language is regular by:
• Showing that it has a finite number of elements,
• Providing a regular expression that defines it,
• Constructing a FSA that accepts it, or
• Exploiting closure properties
• Show a language is not regular by:
• Using the pumping lemma, or
• Exploiting closure properties.

But to use these tools effectively, we may also need domain knowledge (e.g., the Prime Number Theorem).

More Examples
Σ = {0, 1, 2, 3, 4, 5, 6, 7}
L = {w = the octal representation of a number that is divisible by 7}

Example elements of L:
7, 16 (14), 43 (35), 61 (49), 223 (147)

More Examples
Σ = {W, H, Q, E, S, T, B (measure bar)}
L = {w = w represents a song written in 4/4 time}

Example element of L:
WBWBHHBHQQBHHBQEEQEEB

Lecture Notes 8 Languages That Are and Are Not Regular 7

 More Examples
Σ = {0 - 9}
L = {w = is a prime Fermat number}

The Fermat numbers are defined by

n
Fn = 22 + 1, n = 1, 2, 3, …

Example elements of L:
F1 = 5, F2 = 17, F3 = 257, F4 = 65,537

Another Example

Σ = {0 - 9, *, =}
L = {w = a*b=c: a, b, c ∈ {0-9}+ and int(a) * int(b) = int(c)}

The Bottom Line

A language is regular if:

OR

The Bottom Line (Examples)

• The set of decimal representations for nonnegative • The set of strings over {a, b} that contain an equal
integers divisible by 2 or 3 number of a's and b's.
• The social security numbers of living US residents. • The octal representations of numbers that are divisible
• Parity checking by 7
• anbn • The songs in 4/4 time
• ajbk where k>j • The set of prime Fermat numbers
• ak where k is prime

Decision Procedures
A decision procedure is an algorithm that answers a question (usually “yes” or “no”) and terminates. The whole idea of a
decision procedure itself raises a new class of questions. In particular, we can now ask,
1. Is there a decision procedure for question X?
2. What is that procedure?
3. How efficient is the best such procedure?

Clearly, if we jump immediately to an answer to question 2, we have our answer to question 1. But sometimes it makes sense to
answer question 1 first. For one thing, it tells us whether to bother looking for answers to questions 2 and 3.
Examples of Question 1:
Is there a decision procedure, given a regular expression E and a string S, for determining whether S is in L(E)?
Is there a decision procedure, given a Turing machine T and an input string S, for determining whether T halts on S?

Lecture Notes 8 Languages That Are and Are Not Regular 8

 Decision Procedures for Regular Languages
Let M be a deterministic FSA. There is a decision procedure to determine whether:
• w ∈ L(M) for some fixed w
• L(M) is empty
• L(M) is finite
• L(M) is infinite

Let M1 and M2 be two deterministic FSAs. There is a decision procedure to determine whether M1 and M2 are equivalent. Let L1
and L2 be the languages accepted by M1 and M2. Then the language
L = (L1 ∩ ¬L2) ∪ (¬L1 ∩ L2)
= (L1 - L2) ∪ (L2 - L1)
must be regular. L is empty iff L1 = L2. There is a decision procedure to determine whether L is empty and thus whether L1 = L2
and thus whether M1 and M2 are equivalent.

L1 L2 L1 L2 L1,2

Lecture Notes 8 Languages That Are and Are Not Regular 9

 A Review of Equivalence Relations
Do Homework 7.

A Review of Equivalence Relations

A relation R is an equivalence relation if it is: reflexive, symmetric, and transitive.

Example: R = the reflexive, symmetric, transitive closure of:

(Bob, Bill), (Bob, Butch), (Butch, Bud),
(Jim, Joe), (Joe, John), (Joe, Jared),
(Tim, Tom), (Tom, Tad)

An equivalence relation on a nonempty set A creates a partition of A. We write the elements of the partition as [a1], [a2], …
Example:

Another Equivalence Relation

Example: R = the reflexive, symmetric, transitive closure of:

(apple, pear), (pear, banana), (pear, peach),
(peas, mushrooms), (peas, onions), (peas, zucchini)
(bread, rice), (rice, potatoes), (rice, pasta)

Partition:

Lecture Notes 9 A Review of Equivalence Relations 1

 State Minimization for DFAs
Read K & S 2.7
Do Homework 10.
State Minimization
Consider:
a
a b b
1 2 3 4
a a
b b

b 5

a
b a
6
Is this a minimal machine?
State Minimization
Step (1): Get rid of unreachable states.

a
1 2
b
a, b

State 3 is unreachable.

Step (2): Get rid of redundant states.

b a
1 2

a b b

States 2 and 3 are redundant.

Getting Rid of Unreachable States

We can't easily find the unreachable states directly. But we can find the reachable ones and determine the unreachable ones from
there. An algorithm for finding the reachable states:

a
1 2
b
a, b

Lecture Notes 10 State Minimization 1

 Getting Rid of Redundant States

Intuitively, two states are equivalent to each other (and thus one is redundant) if all strings in Σ* have the same fate, regardless of
which of the two states the machine is in. But how can we tell this?

The simple case:

a, b
1 2
b a

3 a, b

Two states have identical sets of transitions out.

Getting Rid of Redundant States

The harder case:

b a
1 2

a b b

The outcomes are the same, even though the states aren't.

Finding an Algorithm for Minimization

Capture the notion of equivalence classes of strings with respect to a language.

Capture the (weaker) notion of equivalence classes of strings with respect to a language and a particular FSA.

Prove that we can always find a deterministic FSA with a number of states equal to the number of equivalence classes of strings.

Describe an algorithm for finding that deterministic FSA.

Defining Equivalence for Strings

We want to capture the notion that two strings are equivalent with respect to a language L if, no matter what is tacked on to them
on the right, either they will both be in L or neither will. Why is this the right notion? Because it corresponds naturally to what
the states of a recognizing FSM have to remember.

Example:

(1) a b b a b

(2) b a b a b

Suppose L = {w ∈ {a,b}* : |w| is even}. Are (1) and (2) equivalent?

Suppose L = {w ∈ {a,b}* : every a is immediately followed by b}. Are (1) and (2) equivalent?

Lecture Notes 10 State Minimization 2

 Defining Equivalence for Strings

If two strings are equivalent with respect to L, we write x ≈L y. Formally, x ≈L y if, ∀z ∈ Σ*,
xz ∈ L iff yz ∈ L.
Notice that ≈L is an equivalence relation.
Example:
Σ = {a, b}
L = {w ∈ Σ* : every a is immediately followed by b }

ε aa bbb
a bb baa
b aba
aab
The equivalence classes of ≈L:

|≈L | is the number of equivalence classes of ≈L.

Another Example of ≈L
Σ = {a, b}
L = {w ∈ Σ* : |w| is even}

ε bb aabb
a aba bbaa
b aab aabaa
aa bbb
baa
The equivalence classes of ≈L:

Yet Another Example of ≈L

Σ = {a, b}
L = aab*a

ε ba aabb
a bb aabaa
b aaa aabbba
aa aba aabbaa
ab aab
bab
The equivalence classes of ≈L:

An Example of ≈L Where All Elements of L Are Not in the Same Equivalence Class

Σ = {a, b}
L = {w ∈ {a, b}* : no two adjacent characters are the same}
ε bb aabaa
a aba aabbba
b aab aabbaa
aa baa
aabb
The equivalence classes of ≈L:

Lecture Notes 10 State Minimization 3

 ≈L| Always Finite?
Is |≈
Σ = {a, b}
L = anbn
ε aa aaaa
a aba aaaaa
b aaa
The equivalence classes of ≈L:

Bringing FSMs into the Picture

≈L is an ideal relation.

What if we now consider what happens to strings when they are being processed by a real FSM?

Σ = {a, b} L = {w ∈ Σ* : |w| is even}

a
1 2
a, b
b
a, b

Define ~M to relate pairs of strings that drive M from s to the same state.

Formally, if M is a deterministic FSM, then x ~M y if there is some state q in M such that (s, x) |-*M (q, ε) and (s, y) |-*M (q, ε).

Notice that M is an equivalence relation.

An Example of ~M

Σ = {a, b} L = {w ∈ Σ* : |w| is even}

a
1 2
a, b
b
a, b

ε bb aabb
a aba bbaa
b aab aabaa
aa bbb
baa

The equivalence classes of ~M: |~M| =

Lecture Notes 10 State Minimization 4

 Another Example of ~M

Σ = {a, b} L = {w ∈ Σ* : |w| is even}

a,b
1 2
a, b

ε bb aabb
a aba bbaa
b aab aabaa
aa bbb
baa
The equivalence classes of ~M: |~M| =

The Relationship Between ≈L and ~M

≈L: [ε, aa, bb, aabb, bbaa] |w| is even

[a, b, aba, aab, bbb, baa, aabaa] |w| is odd

~M, 3 state machine:

q1: [ε, aa, bb, aabb, bbaa] |w| is even
q2: [a, aba, baa, aabaa] (ab ∪ ba ∪ aa ∪ bb)*a
q3: [b, aab, bbb] (ab ∪ ba ∪ aa ∪ bb)*b

~M, 2 state machine:

q1: [ε, aa, bb, aabb, bbaa] |w| is even
q2: [a, b, aba, aab, bbb, baa, aabaa] |w| is odd

~M is a refinement of ≈L.

The Refinement

≈L [even length] [odd length] (S)

~M [even length] odd ending odd ending (R)

(3 state) in a in b

An equivalence relation R is a refinement of another one S iff

xRy → xSy
In other words, R makes all the same distinctions S does, plus possibly more.

|R| ≥ |S|

Lecture Notes 10 State Minimization 5

 ~M is a Refinement of ≈L.

Theorem: For any deterministic finite automaton M and any strings x, y ∈ Σ*, if x ~M y, then x ≈L y.

Proof: If x ~M y, then x and y drive m to the same state q. From q, any continuation string w will drive M to some state r. Thus
xw and yw both drive M to r. Either r is a final state, in which case they both accept, or it is not, in which case they both reject.
But this is exactly the definition of ≈L.

Corollary: |~M | ≥ |≈
≈L |.

Going the Other Way

When is this true?

If x ≈L(M) y then x ~M y.

Finding the Minimal FSM for L

What's the smallest number of states we can get away with in a machine to accept L?

Example: L = {w ∈ Σ* : |w| is even}

The equivalence classes of ≈L:

Minimal number of states for M(L) =

This follows directly from the theorem that says that, for any machine M that accepts L, |~M| must be at least as large as |≈L |.

Can we always find a machine with this minimal number of states?

The Myhill-Nerode Theorem

Theorem: Let L be a regular language. Then there is a deterministic FSA that accepts L and that has precisely |≈L | states.
Proof: (by construction)
M= K states, corresponding to the equivalence classes of ≈L.
s = [ε], the equivalence class of ε under ≈L.
F = {[x] : x ∈ L}
δ([x], a) = [xa]
For this construction to prove the theorem, we must show:
1. K is finite.
2. δ is well defined, i.e., δ([x], a) = [xa] is independent of x.
3. L = L(M)

Lecture Notes 10 State Minimization 6

 The Proof
(1) K is finite.
Since L is regular, there must exist a machine M, with |~M| finite. We know that
|~M| ≥ |≈L|
Thus |≈L| is finite.
(2) δ is well defined.
This is assured by the definition of ≈L, which groups together precisely those strings that have the same fate with respect to L.

The Proof, Continued

(3) L = L(M)
Suppose we knew that ([x], y) |-M* ([xy], ε).
Now let [x] be [ε] and let s be a string in Σ*.
Then
([ε], s) |-M* ([s], ε)
M will accept s if [s] ∈ F.
By the definition of F, [s] ∈ F iff all strings in [s] are in L.
So M accepts precisely the strings in L.
The Proof, Continued

Lemma: ([x], y) |-M* ([xy], ε)

By induction on |y|:
Trivial if |y| = 0.
Suppose true for |y| = n.
Show true for |y| = n+1
Let y = y'a, for some character a. Then,
|y'| = n
([x], y'a) |-M* ([xy'], a) (induction hypothesis)
([xy',] a) |-M* ([xy'a], ε) (definition of δ)
([x], y'a) |-M* ([xy'a], ε) (trans. of |-M*)
([x], y) |-M* ([xy], ε) (definition of y)

Another Version of the Myhill-Nerode Theorem

Theorem: A language is regular iff |≈L| is finite.

Example:
Consider: L = anbn
a, aa, aaa, aaaa, aaaaa …
Equivalence classes:
Proof:
Regular → |≈L| is finite: If L is regular, then there exists an accepting machine M with a finite number of states N. We know that
N ≥ |≈L|. Thus |≈L| is finite.

|≈L| is finite → regular: If |≈L| is finite, then the standard DFSA ML accepts L. Since L is accepted by a FSA, it is regular.

Lecture Notes 10 State Minimization 7

 Constructing the Minimal DFA from ≈L

Σ = {a, b}
L = {w ∈ {a, b}* : no two adjacent characters are the same}

The equivalence classes of ≈L:

1: [ε] ε
2: [a, ba, aba, baba, ababa, ...] (b∪ε)(ab)*a
3: [b, ab, bab, abab, ...] (a∪ε)(ba)*b
4: [bb, aa, bba, bbb, ...] the rest

• Equivalence classes become states

• Start state is [ε]
• Final states are all equivalence classes in L
• δ([x], a) = [xa]

2
a a

a, b
1 a b 4

b b
3

Using Myhill-Nerode to Prove that L is not Regular

L = {an : n is prime}

Consider: ε
a
aa
aaa
aaaa

Equivalence classes:

So Where Do We Stand?
1. We know that for any regular language L there exists a minimal accepting machine ML.
2. We know that |K| of ML equals |≈L|.
3. We know how to construct ML from ≈L.
But is this good enough?

Consider:
a
a b b
1 2 3 4
a a
b b

b 5

a
b a
6

Lecture Notes 10 State Minimization 8

 Constructing a Minimal FSA Without Knowing ≈L

We want to take as input any DFSA M' that accepts L, and output a minimal, equivalent DFSA M.

What we need is a definition for "equivalent", i.e., mergeable states.

Define q ≡ p iff for all strings w ∈ Σ*, either w drives M to an accepting state from both q and p or it drives M to a rejecting state
from both q and p.

Example:
Σ = {a, b} L = {w ∈ Σ* : |w| is even}

a
1 2
a, b
b
a, b

Constructing ≡ as the Limit of a Sequence of Approximating Equivalence Relations ≡n

(Where n is the length of the input strings that have been considered so far)

We'll consider input strings, starting with ε, and increasing in length by 1 at each iteration. We'll start by way overgrouping
states. Then we'll split them apart as it becomes apparent (with longer and longer strings) that their behavior is not identical.

Initially, ≡0 has only two equivalence classes: [F] and [K - F], since on input ε, there are only two possible outcomes, accept or
reject.

Next consider strings of length 1, i.e., each element of Σ. Split any equivalence classes of ≡0 that don't behave identically on all
inputs. Note that in all cases, ≡n is a refinement of ≡n-1.

Continue, until no splitting occurs, computing ≡n from ≡n-1.

Constructing ≡, Continued

More precisely, for any two states p and q ∈ K and any n ≥ 1, q ≡n p iff:
1. q ≡n-1 p, AND
2. for all a ∈ Σ, δ(p, a) ≡n-1 δ(q, a)

Lecture Notes 10 State Minimization 9

 The Construction Algorithm
The equivalence classes of ≡0 are F and K-F.
Repeat for n = 1, 2, 3 …
For each equivalence class C of ≡n-1 do
For each pair of elements p and q in C do
For each a in Σ do
See if δ(p, a) ≡n-1 δ(q, a)
If there are any differences in the behavior of p and q, then split them and create a new equivalence
class.
Until ≡n = ≡n-1. ≡ is this answer. Then use these equivalence classes to coalesce states.

An Example
Σ = {a, b}
b
1 a 2 3
b a
a b
a a

4 b 5 b 6

a,b
≡0 =

≡1 =

≡2 =

Another Example
(a*b*)*
a b
b
1 2
a
≡0 =

≡1 =

Minimal machine:

Lecture Notes 10 State Minimization 10

 Another Example
Example: L={w ∈ {a, b}* : |w| is even}

((aa) ∪ (ab) ∪ (ba) ∪ (bb))*

S→ε T→a
S → aT T→b
S → bT T → aS
T → bS
a, b a

S T #
a, b b

Convert to deterministic:
S = {s}
δ=

Another Example, Continued

Minimize:

a,b
S(1) T(2)

a,b a,b

#S(3)

≡0 =

≡1 =

Minimal machine:

Lecture Notes 10 State Minimization 11

 Summary of Regular Languages and Finite State Machines
Grammars, Languages, and Machines

Language
L

Grammar

Accepts

Machine

Regular Grammars, Languages, and Machines

Most interesting languages are infinite. So we can't write them down. But we can write down finite grammars and finite
machine specifications, and we can define algorithms for mapping between and among them.

Grammars Machines

Regular Nondeterministic
Expressions FSAs

Deterministic
FSAs

Regular
Grammars
Minimal
DFSAs

What Does “Finite State” Really Mean?

There are two kinds of finite state problems:
• Those in which:
• Some history matters.
• Only a finite amount of history matters. In particular, it's often the case that we don't care what order things
occurred in.
Examples:
• Parity
• Money in a vending machine
• Seat belt buzzer
• Those that are characterized by patterns.
Examples:
• Switching circuits:
• Telephone
• Railroad
• Traffic lights
• Lexical analysis
• grep

Lecture Notes 11 Summary of Regular Languages and Finite State Machines 1

 Context-Free Grammars
Read K & S 3.1
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Context-Free Grammars
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Designing Context-Free Grammars.
Do Homework 11.

Context-Free Grammars, Languages, and Pushdown Automata

Context-Free
Language
L

Context-Free
Grammar

Accepts

Pushdown
Automaton

Grammars Define Languages

Think of grammars as either generators or acceptors.

Example: L = {w ∈ {a, b}* : |w| is even}

Regular Expression Regular Grammar

S→ε
(aa ∪ ab ∪ ba ∪ bb)* S → aT
S → bT
T→a
T→b
T → aS
T → bS

Derivation choose aa S
(Generate) choose ab a T
yields a S
a T
b
a a a b a a a b
Parse (Accept) use corresponding FSM

Lecture Notes 12 Context-Free Grammars 1

 Derivation is Not Necessarily Unique

Example: L = {w ∈ {a, b}* : there is at least one a}

Regular Expression Regular Grammar

(a ∪ b)*a (a ∪ b)* S→a

S → bS
choose a from (a ∪ b) S → aS
choose a from (a ∪ b) S → aT
choose a T→a
T→b
choose a T → aT
choose a from (a ∪ b) T → bT
choose a from (a ∪ b)
S S
a S a T
a S a T
a a

More Powerful Grammars

Regular grammars must always produce strings one character at a time, moving left to right.

But sometimes it's more natural to describe generation more flexibly.

Example 1: L = ab*a

S → aBa S → aB
B→ε vs. B→a
B → bB B → bB

Example 2: L = anb*an

S→B
S → aSa
B→ε
B → bB

Key distinction: Example 1 has no recursion on the nonregular rule.

Context-Free Grammars

Remove all restrictions on the form of the right hand sides.

S → abDeFGab

Keep requirement for single non-terminal on left hand side.

S→

but not ASB → or aSb → or ab →

Examples: balanced parentheses anbn

S→ε S→aSb
S → SS S→ε
S → (S)

Lecture Notes 12 Context-Free Grammars 2

 Context-Free Grammars

A context-free grammar G is a quadruple (V, Σ, R, S), where:

• V is the rule alphabet, which contains nonterminals (symbols that are used in the grammar but that do not appear in strings in
the language) and terminals,
• Σ (the set of terminals) is a subset of V,
• R (the set of rules) is a finite subset of (V - Σ) × V*,
• S (the start symbol) is an element of V - Σ.

x ÞG y is a binary relation where x, y ∈ V* such that x = αAβ and y = αχβ for some rule A→χ in R.

Any sequence of the form

w0 ÞG w1 ÞG w2 ÞG . . . ÞG wn
e.g., (S) Þ (SS) Þ ((S)S)
is called a derivation in G. Each wi is called a sentinel form.

The language generated by G is {w ∈ Σ* : S ÞG* w}

A language L is context free if L = L(G) for some context-free grammar G.

Example Derivations

G = (W, Σ, R, S), where

W = {S} ∪ Σ,
Σ = {a, b},
R= { S → a,
S → aS,
S → aSb}

S S
a S a S b
a S b a S b
a S a S
a S b a S
a a

Another Example - Unequal a's and b's

L = {anbm : n ≠ m} S→A /* more a's than b's

S→B /* more b's than a's
G = (W, Σ, R, S), where A→a
W = {a, b, S, A, B}, A → aA
Σ = {a, b}, A → aAb
R= B→b
B → Bb
B → aBb

Lecture Notes 12 Context-Free Grammars 3

 English
S → NP VP the boys run
NP → the NP1 | NP1 big boys run
NP1 → ADJ NP1 | N the youngest boy runs
ADJ → big | youngest | oldest
N → boy | boys the youngest oldest boy runs
VP →V | V NP the boy run
V → run | runs
Who did you say Bill saw coming out of the hotel?

Arithmetic Expressions

The Language of Simple Arithmetic Expressions

G = (V, Σ, R, E), where

V = {+, *, id, T, F, E},
Σ = {+, *, id},
R = { E → id
E→E+E
E→E*E}

E E

E + E E * E

id E * E E + E id

id id id id

id + (id * id) (id + id) * id

Arithmetic Expressions -- A Better Way

The Language of Simple Arithmetic Expressions

G = (V, Σ, R, E), where Examples:

V = {+, *, (, ), id, T, F, E},
Σ = {+, *, (, ), id}, id + id * id
R={ E→E+T
E→ T
T→T*F
T→F id * id * id
F → (E)
F → id }

Lecture Notes 12 Context-Free Grammars 4

 BNF

Backus-Naur Form (BNF) is used to define the syntax of programming languages using context-free grammars.

Main idea: give descriptive names to nonterminals and put them in angle brackets.

Example: arithmetic expressions:

áexpressionñ → áexpressionñ + átermñ
áexpressionñ → átermñ
átermñ → átermñ * áfactorñ
átermñ → áfactorñ
áfactorñ → (áexpressionñ)
áfactorñ → áidñ

The Language of Boolean Logic

G = (V, Σ, R, E), where
V = {∧, ∨, ¬,Þ , (, ), id, E, E1, E2, E3, E4 },
Σ = {∧, ∨, ¬, Þ, (, ), id},
R = { E → E Þ E1
E → E1
E1 → E1 ∨ E2
E1 →E2
E2 → E2 ∧ E3
E2 → E3
E3 → ¬ E4
E3 → E4
E4 →(E)
E4 → id }

Boolean Logic isn't Regular

Suppose it were regular. Then there is an N as specified in the pumping theorem.

Let w be a string of length 2N + 1 + 2|id| of the form:
w = ( ( ( ( ( ( id ) ) ) ) ) ) Þ id
N
x y
y = (k for some k > 0 because |xy| ≤ N.

Then the string that is identical to w except that it has k additional ('s at the beginning would also be in the language. But it can't
be because the parentheses would be mismatched. So the language is not regular.

Lecture Notes 12 Context-Free Grammars 5

 All Regular Languages Are Context Free

(1) Every regular language can be described by a regular grammar. We know this because we can derive a regular grammar from
any FSM (as well as vice versa). Regular grammars are special cases of context-free grammars.

a, b

S T

a, b

(2) The context-free languages are precisely the languages accepted by NDPDAs. But every FSM is a PDA that doesn't bother
with the stack. So every regular language can be accepted by a NDPDA and is thus context-free.

(3) Context-free languages are closed under union, concatenation, and Kleene *, and ε and each single character in Σ are clearly
context free.

Lecture Notes 12 Context-Free Grammars 6

 Parse Trees
Read K & S 3.2
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Derivations and Parse Trees.
Do Homework 12.
Parse Trees

Regular languages:

We care about recognizing patterns and taking appropriate actions.

Example: A parity checker

Structure
Context free languages:

We care about structure.

E

E + E

id E * E

id id

id + (id * id)

Parse Trees Capture Essential Structure

E → id
E→E+E
E→E*E

E E

E + E E * E

id E * E E + E id

id id id id

id + (id * id) (id + id) * id

Lecture Notes 13 Parse Trees 1

 Parse Trees are Just Trees

root

height
nodes

leaves

yield

Leaves are all labeled with terminals or ε.

Other nodes are labeled with nonterminals.
A path is a sequence of nodes, starting at the root, ending at a leaf, and following branches in the tree.
The length of the yield of any tree T with height H and branching factor (fanout) B is ≤

Derivations
To capture structure, we must capture the path we took through the grammar. Derivations do that.
S→ε
S → SS
S → (S)
1 2 3 4 5 6
S Þ SS Þ (S)S Þ ((S))S Þ (())S Þ (())(S) Þ (())()
S Þ SS Þ (S)S Þ ((S))S Þ ((S))(S) Þ (())(S) Þ (())()
1 2 3 5 4 6
S

S S

( S ) ( S )

( S ) ε

ε
Alternative Derivations
S→ε
S → SS
S → (S)

S Þ SS Þ (S)S Þ ((S))S Þ (())S Þ (())(S) Þ (())()

S Þ SS Þ SSS Þ S(S)S Þ S((S))S Þ S(())S Þ S(())(S) Þ S(())()Þ (())()

S S

S S S S

( S ) ( S ) S S ( S )

( S ) ε ε ( S ) ε

ε ( S )

Lecture Notes 13 Parse Trees 2

 Ordering Derivations
Consider two derivations:

1 2 3 4 5 6 7
S Þ SS Þ (S)S Þ ((S))S Þ (())S Þ (())(S) Þ (())()

S Þ SS Þ (S)S Þ ((S))S Þ ((S))(S) Þ (())(S) Þ (())()

1 2 3 4 5 6 7

We can write these, or any, derivation as We say that D1 precedes D2, written D1< D2, if:
D1 = x1 Þ x2 Þ x3 Þ … Þ xn • D1 and D2 are the same length > 1, and
D2 = x1' Þ x2' Þ x3' Þ … Þ xn' • There is some integer k, 1 < k < n, such that:
• for all i ≠ k, xi = xi'
• xk-1 = x'k-1 = uAvBw : u, v, w ∈ V*,
and A, B ∈ V - Σ
• xk = uyvBw, where A → y ∈ R
• xk' = uAvzw where B → z ∈ R
• xk+1 = x'k+1 = uyvzw
Comparing Several Derivations
Consider three derivations:
1 2 3 4 5 6 7
(1) S Þ SS Þ (S)S Þ ((S))S Þ (())S Þ(())(S) Þ(())()

(2) S Þ SS Þ (S)S Þ ((S))S Þ ((S))(S) Þ (())(S) Þ(())()

(3) S Þ SS Þ (S)S Þ ((S))S Þ ((S))(S) Þ ((S))() Þ(())()

D1 < D2
D2 < D3
But D1 does not precede D3.
All three seem similar though. We can define similarity:
D1 is similar to D2 iff the pair (D1, D2) is in the reflexive, symmetric, transitive closure of <.
Note: similar is an equivalence class.
In other words, two derivations are similar if one can be transformed into another by a sequence of switchings in the order of rule
applications.
Parse Trees Capture Similarity
1 2 3 4 5 6 7
(1) S Þ SS Þ (S)S Þ ((S))S Þ (())S Þ(())(S) Þ(())()

(2) S Þ SS Þ (S)S Þ ((S))S Þ ((S))(S) Þ(())(S) Þ(())()

(3) S Þ SS Þ (S)S Þ ((S))S Þ ((S))(S) Þ((S))() Þ(())()

D1 < D2
D2 < D3

All three derivations are similar to each other. This parse tree describes this equivalence class of the similarity relation:
S

S S

( S ) ( S )

( S ) ε

Lecture Notes 13 Parse Trees 3

 The Maximal Element of <

S S

( S ) ( S )

( S ) ε

There's one derivation in this equivalence class that precedes all others in the class.

We call this the leftmost derivation. There is a corresponding rightmost derivation.

The leftmost (rightmost) derivation can be used to construct the parse tree and the parse tree can be used to construct the leftmost
(rightmost) derivation.

Another Example
E → id
E→E+E
E→E*E

(1) E Þ E+E Þ E+E*E Þ E+E*id Þ E+id*id Þ id+id*id

(2) E Þ E*E Þ E*id Þ E+E*id Þ E+id*id Þ id+id*id

E E

E + E E * E

id E * E E + E id

id id id id
id + [id * id] [id + id] * id

Ambiguity

A grammar G for a language L is ambiguous if there exist strings in L for which G can generate more than one parse tree (note
that we don't care about the number of derivations).

The following grammar for arithmetic expressions is ambiguous:

E → id
E→E+E
E→E*E

Often, when this happens, we can find a different, unambiguous grammar to describe L.

Lecture Notes 13 Parse Trees 4

 Resolving Ambiguity in the Grammar
G = (V, Σ, R, E), where Parse : id + id * id
V = {+, *, (, ), id, T, F, E},
Σ = {+, *, (, ), id},
R={ E→E+T
E→ T
T→T*F
T→F
F → (E)
F → id }

Another Example
The following grammar for the language of matched parentheses is ambiguous:

S→ε
S → SS
S → (S)

S S

S S S S

( S ) ( S ) S S ( S )

( S ) ε ε ( S ) ε

ε ( S )

Resolving the Ambiguity with a Different Grammar

One problem is the ε production.

A different grammar for the language of balanced parentheses:

S→ε S
S → S1
S1 → S1 S1 S1
S1 → (S1)
S1 → () S1 S1

( S1 ) ( )
( )

Lecture Notes 13 Parse Trees 5

 A General Technique for Eliminating ε
If G is any context-free grammar for a language L and ε ∉ L then we can construct an alternative grammar G' for L by:

1. Find the set N of nullable variables:

A variable V is nullable if either:
there is a rule
(1) V → ε
or there is a rule
(2) V → PQR…such that P, Q, R, … are all nullable
So begin with N containing all the variables that satisfy (1). Evaluate all other variables with respect to (2). Continue until
no new variables can be added to N.
2. For every rule of the form
P → αQβ for some Q in N, add a rule
P → αβ
3. Delete all rules of the form
V→ε
Sometimes Eliminating Ambiguity Isn't Possible

S → NP VP The boys hit the ball with the bat.

NP → the NP1 | NP1 | NP2
NP1 → ADJ NP1 | N
NP2 → NP1 PP
ADJ → big | youngest | oldest
N → boy | boys | ball | bat | autograph The boys hit the ball with the autograph.
VP →V | V NP
VP → VP PP
V → hit| hits
PP → with NP
Why It's Not Possible
• We could write an unambiguous grammar to describe L but it wouldn't always get the parses we want. Any grammar that is
capable of getting all the parses will be ambiguous because the facts required to choose a derivation cannot be captured in
the context-free framework.
Example: Our simple English grammar
[[The boys] [hit [the ball] [with [the bat]]]]
[[The boys] [hit [the ball] [with [the autograph]]]]
• There is no grammar that describes L that is not ambiguous.
Example: L = {anbncm} ∪ {anbmcm}

S → S1 | S2
S1 → S1c | A Now consider the strings anbncn
A → aAb | ε
S2 → aS2|B They have two distinct derivations
B → bBc | ε
Inherent Ambiguity of CFLs
A context free language with the property that all grammars that generate it are ambiguous is inherently ambiguous.

L = {anbncm} ∪ {anbmcm} is inherently ambiguous.

Other languages that appear ambiguous given one grammar, turn out not to be inherently ambiguous because we can find an
unambiguous grammar.

Examples: Arithmetic Expressions

Balanced Parentheses

Whenever we design practical languages, it is important that they not be inherently ambiguous.

Lecture Notes 13 Parse Trees 6

 Pushdown Automata
Read K & S 3.3.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Designing Pushdown Automata.
Do Homework 13.

Recognizing Context-Free Languages

Two notions of recognition:

(1) Say yes or no, just like with FSMs
(2) Say yes or no, AND
if yes, describe the structure

a + b * c
Just Recognizing

We need a device similar to an FSM except that it needs more power.

The insight: Precisely what it needs is a stack, which gives it an unlimited amount of memory with a restricted structure.

( ( ( ( ( ) ) ) ) ( ) ( ( ) )

( Finite
( State
( Controller
(
( (

Definition of a Pushdown Automaton

M = (K, Σ, Γ, ∆, s, F), where:

K is a finite set of states
Σ is the input alphabet
Γ is the stack alphabet
s ∈ K is the initial state
F ⊆ K is the set of final states, and
∆ is the transition relation. It is a finite subset of

(K × (Σ ∪ {ε}) × Γ* ) × ( K × Γ* )

state input or ε string of symbols to pop state string of symbols to

from top of stack push on top of stack

M accepts a string w iff

(s, w, ε) |-M* (p, ε, ε) for some state p ∈ F

Lecture Notes 14 Pushdown Automata 1

 A PDA for Balanced Brackets

[//[
s
]/[/

M = (K, Σ, Γ, ∆, s, F), where:

K = {s} the states
Σ = {[, ]} the input alphabet
Γ = {[} the stack alphabet
F = {s}
∆ contains:

((s, [, ε), (s, [ ))

((s, ], [ ), (s, ε))

Important:
This does not mean that the stack is empty.
An Example of Accepting

[//[
s
]/[/

∆ contains:
[1] ((s, [, ε), (s, [ ))
[2] ((s, ], [ ), (s, ε))
input = [ [ [ ] [ ] ] ]

trans state unread input stack

s [[[][]]] ε
1 s [[][]]] [
1 s [][]]] [[
1 s ][]]] [[[
2 s []]] [[
1 s ]]] [[[
2 s ]] [[
2 s ] [
2 s ε ε

An Example of Rejecting

[//[
s
]/[/
∆ contains:
[1] ((s, [, ε), (s, [ ))
[2] ((s, ], [ ), (s, ε))
input = [ [ ] ] ]

trans state unread input stack

s [[]]] ε
1 s []]] [
1 s ]]] [[
2 s ]] [
2 s ] ε
none! s ] ε
We're in s, a final state, but we cannot accept because the input string is not empty. So we reject.

Lecture Notes 14 Pushdown Automata 2

 A PDA for anbn

First we notice:
• We'll use the stack to count the a's.
• This time, all strings in L have two regions. So we need two states so that a's can't follow b's. Note the similarity to the
regular language a*b*.

A PDA for wcwR

A PDA to accept strings of the form wcwR:

a//a a/a/
c//
s f

b//b b/b/

M = (K, Σ, Γ, ∆, s, F), where:

K = {s, f} the states
Σ = {a, b, c} the input alphabet
Γ = {a, b} the stack alphabet
F = {f} the final states
∆ contains:
((s, a, ε), (s, a))
((s, b, ε), (s, b))
((s, c, ε), (f, ε))
((f, a, a), (f, ε))
((f, b, b), (f, ε))
An Example of Accepting

a//a a/a/
c//
s f

b//b b/b/

∆ contains:
[1] ((s, a, ε), (s, a))
[2] ((s, b, ε), (s, b))
[3] ((s, c, ε), (f, ε))
[4] ((f, a, a), (f, ε))
[5] ((f, b, b), (f, ε))

input = b a c a b
trans state unread input stack
s bacab ε
2 s acab b
1 s cab ab
3 f ab ab
5 f b b
6 f ε ε

Lecture Notes 14 Pushdown Automata 3

 A Nondeterministic PDA
L = wwR
S→ε
S → aSa
S → bSb
A PDA to accept strings of the form wwR:

a//a a/a/
ε//
s f

b//b b/b/

M = (K, Σ, Γ, ∆, s, F), where:

K = {s, f} the states
Σ = {a, b, c} the input alphabet
Γ = {a, b} the stack alphabet
F = {f} the final states
∆ contains:
((s, a, ε), (s, a))
((s, b, ε), (s, b))
((s, ε, ε), (f, ε))
((f, a, a), (f, ε))
((f, b, b), (f, ε))
An Example of Accepting

a//a a/a/
ε//
s f

b//b b/b/

[1] ((s, a, ε), (s, a)) [4] ((f, a, a), (f, ε))
[2] ((s, b, ε), (s, b)) [5] ((f, b, b), (f, ε))
[3] ((s, ε, ε), (f, ε))
input: a a b b a a

trans state unread input stack

s aabbaa ε
1 s abbaa a
3 f abbaa a
4 f bbaa ε
none

trans state unread input stack

s aabbaa ε
1 s abbaa a
1 s bbaa aa
2 s baa baa
3 f baa baa
5 f aa aa
4 f a a
4 f ε ε

Lecture Notes 14 Pushdown Automata 4

 L = {ambn : m ≤ n}
A context-free grammar for L:
S→ε
S → Sb /* more b's
S → aSb
A PDA to accept L:

a//a b/a/
b/a/ b/ε/
1 2
b/ε/

Accepting Mismatches

L = {ambn m ≠ n; m, n >0}

a//a b/a/
b/a/
1 2

• If stack and input are empty, halt and reject.

• If input is empty but stack is not (m > n) (accept):

a//a b/a/ ε/a/

b/a/ ε/a/
1 2 3

• If stack is empty but input is not (m < n) (accept):

a//a b/a/ ε/a/

b/a/ ε/a/
1 2 3

b//
4 b//

Lecture Notes 14 Pushdown Automata 5

 Eliminating Nondeterminism

A PDA is deterministic if, for each input and state, there is at most one possible transition. Determinism implies uniquely
defined machine behavior.

a//a b/a/ ε/a/

b/a/ ε/a/
1 2 3

b//
4 b//

• Jumping to the input clearing state 4:

Need to detect bottom of stack, so push Z onto the stack before we start.

a//a b/a/ ε/a/

ε//Z b/a/ ε/a/
0 1 2 3 ε/Z/

b/Z/
4 b//

• Jumping to the stack clearing state 3:

Need to detect end of input. To do that, we actually need to modify the definition of L to add a termination character
(e.g., $)

L = {anbmcp : n,m,p ≥ 0 and (n ≠ m or m ≠ p)}

S → NC /* n ≠ m, then arbitrary c's C → ε | cC /* add any number of c's

S → QP /* arbitrary a's, then p ≠ m P → B' /* more b's than c's
N→A /* more a's than b's P → C' /* more c's than b's
N→B /* more b's than a's B' → b
A→a B' → bB'
A → aA B' → bB'c
A → aAb C' → c | C'c
B→b C' → C'c
B → Bb C' → bC'c
B → aBb Q → ε | aQ /* prefix with any number of a's

L = {anbmcp : n,m,p ≥ 0 and (n ≠ m or m ≠ p)}

ε//Z a//a
S S' machine for N

a// b,c

clear and accept

machine for P

Lecture Notes 14 Pushdown Automata 6

 Another Deterministic CFL

L = {anbn} ∪ {bn an}

A CFG for L: A NDPDA for L:

S→A
S→B
A→ε
A → aAb
B→ε
B → bBa

A DPDA for L:

More on PDAs

What about a PDA to accept strings of the form ww?

Every FSM is (Trivially) a PDA

Given an FSM M = (K, Σ, ∆, s, F)

and elements of ∆ of the form
( p, i, q )
old state, input, new state

We construct a PDA M' = (K, Σ, Γ, ∆, s, F)

where Γ = ∅ /* stack alphabet
and
each transition (p, i, q) becomes
( ( p, i, ε ), ( q, ε ) )
old state, input, don't look at stack new state don't push on stack

In other words, we just don't use the stack.

Alternative (but Equivalent) Definitions of a NDPDA

Example: Accept by final state at end of string (i.e., we don't care about the stack being empty)
We can easily convert from one of our machines to one of these:
1. Add a new state at the beginning that pushes # onto the stack.
2. Add a new final state and a transition to it that can be taken if the input string is empty and the top of the stack is #.
Converting the balanced parentheses machine:

(//( ε//# (//(

S S S'
)/(/ )/(/
ε/#/

The new machine is nondeterministic:

( ) ( )
Ý
The stack will be: #

Lecture Notes 14 Pushdown Automata 7

 What About PDA's for Interesting Languages?

E→E+T Arithmetic Expressions

E→T
T→T*F ε/ε/E
T→F 1 2
F → (E)
F → id

(1) (2, ε, E), (2, E+T) Example:

(2) (2, ε, E), (2, T) a+b*c
(3) (2, ε, T), (2, T*F)
(4) (2, ε, T), (2, F)
(5) (2, ε, F), (2, (E) )
(6) (2, ε, F), (2, id)
(7) (2, id, id), (2, ε)
(8) (2, (, ( ), (2, ε)
(9) (2, ), ) ), (2, ε)
(10) (2, +, +), (2, ε)
(11) (2, *, *), (2, ε)

But what we really want to do with languages like this is to extract structure.

Comparing Regular and Context-Free Languages

Regular Languages Context-Free Languages

• regular expressions • context-free grammars

- or -
• regular grammars
• recognize • parse
• = DFSAs • = NDPDAs

Lecture Notes 14 Pushdown Automata 8

 Pushdown Automata and Context-Free Grammars
Read K & S 3.4.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Context-Free Languages and PDAs.
Do Homework 14.

PDAs and Context-Free Grammars

Theorem: The class of languages accepted by PDAs is exactly the class of context-free languages.

Recall: context-free languages are languages that can be defined with context-free grammars.

Restate theorem: Can describe with context-free grammar ⇔ Can accept by PDA

Going One Way

Lemma: Each context-free language is accepted by some PDA.

Proof (by construction by “top-down parse” conversion algorithm):

The idea: Let the stack do the work.

Example: Arithmetic expressions

E→E+T
E→T
T→T*F ε/ε/E
T→F 1 2
F → (E)
F → id

(1) (2, ε, E), (2, E+T) (7) (2, id, id), (2, ε)
(2) (2, ε, E), (2, T) (8) (2, (, ( ), (2, ε)
(3) (2, ε, T), (2, T*F) (9) (2, ), ) ), (2, ε)
(4) (2, ε, T), (2, F) (10) (2, +, +), (2, ε)
(5) (2, ε, F), (2, (E) ) (11) (2, *, *), (2, ε)
(6) (2, ε, F), (2, id)
The Top-down Parse Conversion Algorithm

Given G = (V, Σ, R, S)
Construct M such that L(M) = L(G)

M = ({p, q}, Σ, V, ∆, p, {q}), where ∆ contains:

(1) ((p, ε, ε), (q, S))

push the start symbol on the stack

(2) ((q, ε, A), (q, x)) for each rule A → x in R

replace left hand side with right hand side

(3) ((q, a, a), (q, ε)) for each a ∈ Σ

read an input character and pop it from the stack

The resulting machine can execute a leftmost derivation of an input string in a top-down fashion.

Lecture Notes 15 Pushdown Automata and Context-Free Languages 1

 Example of the Algorithm
L = {anb*an}
0 (p, ε, ε), (q, S)
(1) S→ε 1 (q, ε, S), (q, ε)
(2) S→B 2 (q, ε, S), (q, B)
(3) S → aSa 3 (q, ε, S), (q, aSa)
(4) B→ε 4 (q, ε, B), (q, ε)
(5) B → bB 5 (q, ε, B), (q, bB)
6 (q, a, a), (q, ε)
input = a a b b a a 7 (q, b, b), (q, ε)
trans state unread input stack
p aabbaa ε
0 q aabbaa S
3 q aabbaa aSa
6 q abbaa Sa
3 q abbaa aSaa
6 q bbaa Saa
2 q bbaa Baa
5 q bbaa bBaa
7 q baa Baa
5 q baa bBaa
7 q aa Baa
4 q aa aa
6 q a a
6 q ε ε

Another Example
L = {anbmcpdq : m + n = p + q}
0 (p, ε, ε), (q, S)
(1) S → aSd 1 (q, ε, S), (q, aSd)
(2) S→T 2 (q, ε, S), (q,T)
(3) S→U 3 (q, ε, S), (q,U)
(4) T → aTc 4 (q, ε, T), (q, aTc)
(5) T→V 5 (q, ε, T), (q, V)
(6) U → bUd 6 (q, ε, U), (q, bUd)
(7) U→V 7 (q, ε, U), (q, V)
(8) V → bVc 8 (q, ε, V), (q, bVc
(9) V→ε 9 (q, ε, V), (q, ε)
10 (q, a, a), (q, ε)
11 (q, b, b), (q, ε)
input = a a b c d d 12 (q, c, c), (q, ε)
13 (q, d, d), (q, ε)

The Other Way—Build a PDA Directly

L = {anbmcpdq : m + n = p + q}

(1) S → aSd (6) U → bUd

(2) S→T (7) U→V
(3) S→U (8) V → bVc
(4) T → aTc (9) V→ε
(5) T→V

a//a b//a c/a/ d/a/

b//a c/a/ d/a/
1 2 3 4
ε/ε/ ε/ε/ ε/ε/
input = a a b c d d

Lecture Notes 15 Pushdown Automata and Context-Free Languages 2

 Notice Nondeterminism

Machines constructed with the algorithm are often nondeterministic, even when they needn't be. This happens even with trivial
languages.
Example: L = anbn

A grammar for L is: A machine M for L is:

(0) ((p, ε, ε), (q, S))
[1] S → aSb (1) ((q, ε, S), (q, aSb))
[2] S → ε (2) ((q, ε, S), (q, ε))
(3) ((q, a, a), (q, ε))
(4) ((q, b, b), (q, ε))
But transitions 1 and 2 make M nondeterministic.

A nondeterministic transition group is a set of two or more transitions out of the same state that can fire on the same
configuration. A PDA is nondeterministic if it has any nondeterministic transition groups.

A directly constructed machine for L:

Going The Other Way

Lemma: If a language is accepted by a pushdown automaton, it is a context-free language (i.e., it can be described by a context-
free grammar).
Proof (by construction)

Example: L = {wcwR : w ∈ {a, b}*}

∆ contains:
a//a a/a/ ((s, a, ε), (s, a))
c// ((s, b, ε), (s, b))
s f ((s, c, ε), (f, ε))
((f, a, a), (f, ε))
b//b b/b/ ((f, b, b), (f, ε))

M = ({s, f}, {a, b, c}, {a, b}, ∆, s,{f}), where:

First Step: Make M Simple

A PDA M is simple iff:
1. there are no transitions into the start state, and
2. whenever ((q, x, β), (p, γ) is a transition of M and q is not the start state, then β ∈ Γ, and |γ| ≤ 2.

Step 1: Add s' and f':

a/ε/a a/a/
ε/ε/Z c// ε/Z/
s' s f f'

b/ε/b b/b/

Step 2:
(1) Assure that |β| ≤ 1.

(2) Assure that |γ| ≤ 2.

(3) Assure that |β| = 1.

Lecture Notes 15 Pushdown Automata and Context-Free Languages 3

 Making M Simple

a/ε/a a/a/
ε/ε/Z c// ε/Z/
s' s f f'

b/ε/b b/b/

M = ({s, f, s', f'}, {a, b, c}, {a, b, Z}, ∆, s',{f'}), ∆=

((s', ε, ε), (s, Z))
((s, a, ε), (s, a)) ((s, a, Z), (s, aZ))
((s, a, a), (s, aa))
((s, a, b), (s, ab))
((s, b, ε), (s, b)) ((s, b, Z), (s, bZ))
((s, b, a), (s, ba))
((s, b, b), (s, bb))
((s, c, ε), (f, ε)) ((s, c, Z), (f, Z))
((s, c, a), (f, a))
((s, c, b), (f, b))
((f, a, a), (f, ε)) ((f, a, a), (f, ε))
((f, b, b), (f, ε)) ((f, b, b), (f, ε))
((f, ε, Z), (f', ε))

Second Step - Creating the Productions

The basic idea -- simulate a leftmost derivation of M on any input string.

Example: abcba
S [1]

<s, Z, f'> [2]

a <s, a, f> [4] <f, Z, f'> [8]

b <s, b, f> [5] <f, a, f> [6] ε <f', ε, f'> [10]

c <f, b, f> [7] a <f, ε, f> [9] ε

b <f, ε, f> [9] ε

If the nonterminal <s1, X, s2> Þ* w, then the PDA starts in state s1 with (at least) X on the stack and after consuming w and
popping the X off the stack, it ends up in state s2.

Start with the rule:

S → <s, Z, f’> where s is the start state, f’ is the (introduced) final state and Z is the stack bottom symbol.

Transitions ((s1, a, X), (s2, YX)) become a set of rules:

<s1, X, q> → a <s2, Y, r> <r, X, q> for a ∈ Σ ∪ {ε}, ∀q,r ∈ K

Transitions ((s1, a, X), (s2, Y)) becomes a set of rules:

<s1, X, q> → a <s2, Y, q> for a ∈ Σ ∪ {ε}, ∀q ∈ K

Transitions ((s1, a, X), (s2, ε)) become a rule:

<s1, X, s2> → a for a ∈ Σ ∪ {ε}

Lecture Notes 15 Pushdown Automata and Context-Free Languages 4

 Creating Productions from Transitions

S → <s, Z, f'> [1]

((s', ε, ε), (s, Z))
((s, a, Z), (s, aZ)) <s, Z, f'> → a <s, a, f> <f, Z, f'> [2]
<s, Z, s> → a <s, a, f> <f, Z, s> [x]
<s, Z, f> → a <s, a, s> <s, Z, f> [x]
<s, Z, s> → a <s, a, s> <s, Z, f> [x]
<s, Z, s'> → a <s, a, f> <f, Z, s'> [x]
((s, a, a), (s, aa)) <s, a, f> → a <s, a, f> <f, a, f> [3]
((s, a, b), (s, ab)) …
((s, b, Z), (s, bZ)) …
((s, b, a), (s, ba)) <s, a, f> → b <s, b, f> <f, a, f> [4]
((s, b, b), (s, bb)) …
((s, c, Z), (f, Z)) …
((s, c, a), (f, a)) <s, a, f> → c <f, a, f>
((s, c, b), (f, b)) <s, b, f> → c <f, b, f> [5]
((f, a, a), (f, ε)) <f, a, f> → a <f, ε, f> [6]
((f, b, b), (f, ε)) <f, b, f> → b <f, ε, f> [7]
((f, ε, Z), (f', ε)) <f, Z, f'> → ε <f', ε, f'> [8]
<f, ε, f> → ε [9]
<f' ε, f'> → ε [10]

Comparing Regular and Context-Free Languages

Regular Languages Context-Free Languages

• regular exprs. • context-free grammars

• or
• regular grammars
• recognize • parse
• = DFSAs • = NDPDAs

Lecture Notes 15 Pushdown Automata and Context-Free Languages 5

 Grammars and Normal Forms
Read K & S 3.7.
Recognizing Context-Free Languages

Two notions of recognition:

(1) Say yes or no, just like with FSMs

(2) Say yes or no, AND

if yes, describe the structure

a + b * c

Now it's time to worry about extracting structure (and doing so efficiently).

Optimizing Context-Free Languages

For regular languages:

Computation = operation of FSMs. So,
Optimization = Operations on FSMs:
Conversion to deterministic FSMs
Minimization of FSMs
For context-free languages:
Computation = operation of parsers. So,
Optimization = Operations on languages
Operations on grammars
Parser design

Before We Start: Operations on Grammars

There are lots of ways to transform grammars so that they are more useful for a particular purpose.
the basic idea:
1. Apply transformation 1 to G to get of undesirable property 1. Show that the language generated by G is unchanged.
2. Apply transformation 2 to G to get rid of undesirable property 2. Show that the language generated by G is unchanged AND
that undesirable property 1 has not been reintroduced.
3. Continue until the grammar is in the desired form.

Examples:
• Getting rid of ε rules (nullable rules)
• Getting rid of sets of rules with a common initial terminal, e.g.,
• A → aB, A → aC become A → aD, D → B | C
• Conversion to normal forms

Lecture Notes 16 Grammars and Normal Forms 1

 Normal Forms

If you want to design algorithms, it is often useful to have a limited number of input forms that you have to deal with.

Normal forms are designed to do just that. Various ones have been developed for various purposes.

Examples:

• Clause form for logical expressions to be used in resolution theorem proving

• Disjunctive normal form for database queries so that they can be entered in a query by example grid.
• Various normal forms for grammars to support specific parsing techniques.

Clause Form for Logical Expressions

∀x : [Roman(x) ∧ know(x, Marcus)] → [hate(x, Caesar) ∨ (∀y : ∃z : hate(y, z) → thinkcrazy(x, y))]

becomes

¬Roman(x) ∨ ¬know(x, Marcus) ∨ hate(x, Caesar) ∨ ¬hate(y, z) ∨ thinkcrazy(x, z)

Disjunctive Normal Form for Queries

(category = fruit or category = vegetable)

and
(supplier = A or supplier = B)

becomes

(category = fruit and supplier = A) or

(category = fruit and supplier = B) or
(category = vegetable and supplier = A) or
(category = vegetable and supplier = B)

Category Supplier Price

fruit A
fruit B
vegetable A
vegetable B

Normal Forms for Grammars

Two of the most common are:

• Chomsky Normal Form, in which all rules are of one of the following two forms:
• X → a, where a ∈ Σ, or
• X → BC, where B and C are nonterminals in G

• Greibach Normal Form, in which all rules are of the following form:
• X → a β, where a ∈ Σ and β is a (possibly empty) string of nonterminals

If L is a context-free language that does not contain ε, then if G is a grammar for L, G can be rewritten into both of these normal
forms.

Lecture Notes 16 Grammars and Normal Forms 2

 What Are Normal Forms Good For?
Examples:
• Chomsky Normal Form:
• X → a, where a ∈ Σ, or
• X → BC, where B and C are nonterminals in G
♦ The branching factor is precisely 2. Tree building algorithms can take advantage of that.

• Greibach Normal Form

• X → a β, where a ∈ Σ and β is a (possibly empty) string of nonterminals
♦Precisely one nonterminal is generated for each rule application. This means that we can put a bound on the number of rule
applications in any successful derivation.
Conversion to Chomsky Normal Form

Let G be a grammar for the context-free language L where ε ∉ L.

We construct G', an equivalent grammar in Chomsky Normal Form by:
0. Initially, let G' = G.
1. Remove from G' all ε productions:
1.1. If there is a rule A → αBβ and B is nullable, add the rule A → αβ and delete the rule B → ε.
Example:
S → aA
A → B | CD
B→ε
B→a
C → BD
D→b
D→ε
Conversion to Chomsky Normal Form

2. Remove from G' all unit productions (rules of the form A → B, where B is a nonterminal):
2.1. Remove from G' all unit productions of the form A → A.
2.2. For all nonterminals A, find all nonterminals B such that A Þ* B, A ≠ B.
2.3. Create G'' and add to it all rules in G' that are not unit productions.
2.4. For all A and B satisfying 3.2, add to G''
A → y1 | y2 | … where B → y1 | y2 | is in G".
2.5. Set G' to G''.
Example: A→a
A→B
A → EF
B→A
B → CD
B→C
C → ab
At this point, all rules whose right hand sides have length 1 are in Chomsky Normal Form.

Lecture Notes 16 Grammars and Normal Forms 3

 Conversion to Chomsky Normal Form

3. Remove from G' all productions P whose right hand sides have length greater than 1 and include a terminal (e.g., A →
aB or A → BaC):
3.1. Create a new nonterminal Ta for each terminal a in Σ.
3.2. Modify each production P by substituting Ta for each terminal a.
3.3. Add to G', for each Ta, the rule Ta → a

Example:
A → aB
A → BaC
A → BbC

Ta → a
Tb → b
Conversion to Chomsky Normal Form

4. Remove from G' all productions P whose right hand sides have length greater than 2 (e.g., A → BCDE)
4.1. For each P of the form A → N1N2N3N4…Nn, n > 2, create new nonterminals M2, M3, … Mn-1.
4.2. Replace P with the rule A → N1M2.
4.3. Add the rules M2 → N2M3, M3 → N3M4, … Mn-1 → Nn-1Nn

Example:
A → BCDE (n = 4)

A → BM2
M2 → C M3
M3 → DE

Lecture Notes 16 Grammars and Normal Forms 4

 Top Down Parsing
Read K & S 3.8.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Parsing, Sections 1 and 2.
Do Homework 15.
Parsing

Two basic approaches:

Top Down

E Þ E Þ E

E + T . + T
.
.
id

Bottom Up
E

T T

F F F

id + id Þ id + id Þ id + id

A Simple Parsing Example

A simple top-down parser for arithmetic expressions, given the grammar

[1] E→E+T
[2] E→T
[3] T→T*F
[4] T→F
[5] F → (E)
[6] F → id
[7] F → id(E)

A PDA that does a top down parse:

(0) (1, ε, ε), (2, E) (7) (2, ε, F), (2, id(E))

(1) (2, ε, E), (2, E+T) (8) (2, id, id), (2, ε)
(2) (2, ε, E), (2, T) (9) (2, (, ( ), (2, ε)
(3) (2, ε, T), (2, T*F) (10) (2, ), ) ), (2, ε)
(4) (2, ε, T), (2, F) (11) (2, +, +), (2, ε)
(5) (2, ε, F), (2, (E) ) (12) (2, *, *), (2, ε)
(6) (2, ε, F), (2, id)

Lecture Notes 17 Top Down Parsing 1

 How Does It Work?

Example: id + id * id(id)

Stack:

What Does It Produce?

The leftmost derivation of the string. Why?

E Þ E + T Þ T + T Þ F + T Þ id + T Þ
id + T * F Þ id + F * F Þ id + id * F Þ
id + id * id(E) Þ id + id * id(T) Þ
id + id * id(F) Þ id + id * id(id)

E + T

T T * F

F F id ( E )

id id T

id

But the Process Isn't Deterministic

(0) (1, ε, ε), (2, E)

(1) (2, ε, E), (2, E+T) nondeterministic
(2) (2, ε, E), (2, T)
(3) (2, ε, T), (2, T*F) nondeterministic
(4) (2, ε, T), (2, F)
(5) (2, ε, F), (2, (E) )
(6) (2, ε, F), (2, id) nondeterministic
(7) (2, ε, F), (2, id(E))
(8) (2, id, id), (2, ε)
(9) (2, (, ( ), (2, ε)
(10) (2, ), ) ), (2, ε)
(11) (2, +, +), (2, ε)
(12) (2, *, *), (2, ε)

Lecture Notes 17 Top Down Parsing 2

 Is Nondeterminism A Problem?
Yes.

In the case of regular languages, we could cope with nondeterminism in either of two ways:
• Create an equivalent deterministic recognizer (FSM)
• Simulate the nondeterministic FSM in a number of steps that was still linear in the length of the input string.

For context-free languages, however,

• The best straightforward general algorithm for recognizing a string is O(n3) and the best (very complicated) algorithm is
based on a reduction to matrix multiplication, which may get close to O(n2).

We'd really like to find a deterministic parsing algorithm that could run in time proportional to the length of the input string.

Is It Possible to Eliminate Nondeterminism?

In this case: Yes

In general: No

Some definitions:

• A PDA M is deterministic if it has no two transitions such that for some (state, input, stack sequence) the two transitions
could both be taken.

• A language L is deterministic context-free if L$ = L(M) for some deterministic PDA M.

Theorem: The class of deterministic context-free languages is a proper subset of the class of context-free languages.

Proof: Later.
Adding a Terminator to the Language

We define the class of deterministic context-free languages with respect to a terminator ($) because we want that class to be as
large as possible.

Theorem: Every deterministic CFL (as just defined) is a context-free language.

Proof:

Without the terminator ($), many seemingly deterministic cfls aren't. Example:
a* ∪ {anbn : n> 0}

Possible Solutions to the Nondeterminism Problem

1) Modify the language

• Add a terminator $
2) Change the parsing algorithm
3) Modify the grammar

Lecture Notes 17 Top Down Parsing 3

 Modifying the Parsing Algorithm

What if we add the ability to look one character ahead in the input string?

Example: id + id * id(id)
Ý
E Þ E + T Þ T + T Þ F + T Þ id + T Þ
id + T * F Þ id + F * F Þ id + id * F

Considering transitions:
(5) (2, ε, F), (2, (E) )
(6) (2, ε, F), (2, id)
(7) (2, ε, F), (2, id(E))

If we add to the state an indication of what character is next, we have:

(5) (2, (, ε, F), (2, (E) )
(6) (2, id, ε, F), (2, id)
(7) (2, id, ε, F), (2, id(E))
Modifying the Language

So we've solved part of the problem. But what do we do when we come to the end of the input? What will be the state indicator
then?

The solution is to modify the language. Instead of building a machine to accept L, we will build a machine to accept L$.

Using Lookahead

(0) (1, ε, ε), (2, E))

[1] E→E+T (1) (2, ε, E), (2, E+T)
[2] E→T (2) (2, ε, E), (2, T)
[3] T→T*F (3) (2, ε, T), (2, T*F)
[4] T→F (4) (2, ε, T), (2, F)
[5] F → (E) (5) (2, (, ε, F), (2, (E) )
[6] F → id (6) (2, id, ε, F), (2, id)
[7] F → id(E) (7) (2, id, ε, F),(2, id(E))
(8) (2, id, id), (2, ε)
(9) (2, (, ( ), (2, ε)
(10) (2, ), ) ), (2, ε)
(11) (2, +, +), (2, ε)
(12) (2, *, *), (2, ε)

For now, we'll ignore the issue of when we read the lookahead character and the fact that we only care about it if the top symbol
on the stack is F.
Possible Solutions to the Nondeterminism Problem

1) Modify the language

• Add a terminator $
2) Change the parsing algorithm
• Add one character look ahead
3) Modify the grammar

Lecture Notes 17 Top Down Parsing 4

 Modifying the Grammar

Getting rid of identical first symbols:

[6] F → id (6) (2, id, ε, F),(2, id)

[7] F → id(E) (7) (2, id, ε, F),(2, id(E))

Replace with:

[6'] F → id A (6') (2, id, ε, F), (2, id A)

[7'] A→ε (7') (2, ¬(, ε, A), (2, ε)
[8'] A → (E) (8') (2, (, ε, A), (2, (E))

The general rule for left factoring:

Whenever A → αβ1
A → αβ2 …
A → αβn
are rules with α ≠ ε and n ≥ 2, then replace them by the rules:
A → αA'
A' → β1
A' → β2 …
A' → βn

Modifying the Grammar

Getting rid of left recursion:

[1] E→E+T (1) (2, ε, E), (2, E+T)

[2] E→T (2) (2, ε, E), (2, T)

The problem:
E

E + T

E + T
Replace with:

[1] E → T E' (1) (2, ε, E), (2, T E')

[2] E' → + T E' (2) (2, ε, E'), (2, + T E')
[3] E' → ε (3) (2, ε, E'), (2, ε)

Lecture Notes 17 Top Down Parsing 5

 Getting Rid of Left Recursion

The general rule for eliminating left recursion:

If G contains the following rules: Replace them with:

A → Aα1 A' → α1A'

A → Aα2 … A' → α2A' …
A → Aα3 A' → α3A'
A → Aαn A' → αnA'
A' → ε
A → β1 (where β's do not start with Aα) A → β1A'
A → β2 A → β2A'
… …
A → βm A → βmA'

and n > 0, then

Possible Solutions to the Nondeterminism Problem

I. Modify the language

A. Add a terminator $
II. Change the parsing algorithm
A. Add one character look ahead
III. Modify the grammar
A. Left factor
B. Get rid of left recursion

LL(k) Languages

We have just offered heuristic rules for getting rid of some nondeterminism.

We know that not all context-free languages are deterministic, so there are some languages for which these rules won't work.

We define a grammar to be LL(k) if it is possible to decide what production to apply by looking ahead at most k symbols in the
input string.

Specifically, a grammar G is LL(1) iff, whenever

A → α | β are two rules in G:
1. For no terminal a do α and β derive strings beginning with a.
2. At most one of α | β can derive ε.
3. If β Þ* ε, then α does not derive any strings beginning with a terminal in FOLLOW(A), defined to be the set of terminals
that can immediately follow A in some sentential form.

We define a language to be LL(k) if there exists an LL(k) grammar for it.

Lecture Notes 17 Top Down Parsing 6

 Implementing an LL(1) Parser

If a language L has an LL(1) grammar, then we can build a deterministic LL(1) parser for it. Such a parser scans the input Left to
right and builds a Leftmost derivation.

The heart of an LL(1) parser is the parsing table, which tells it which production to apply at each step.
For example, here is the parsing table for our revised grammar of arithmetic expressions without function calls:
V\ΣΣ id + * ( ) $
E E→TE' E→TE'
E' E'→+TE' E'→ε E'→ε
T T→FT' T→FT'
T' T'→ε T'→*FT' T'→ε T'→ε
F F→id F→(E)

Given input id + id * id, the first few moves of this parser will be:
E id + id * id$
E→TE' TE' id + id * id$
T→FT' FT'E' id + id * id$
F→id idT'E' id + id * id$
T'E' + id * id$
T'→ε E' + id * id$

But What If We Need a Language That Isn't LL(1)?

Example:

ST → if C then ST else ST
ST → if C then ST

We can apply left factoring to yield:

ST → if C then ST S'
S' → else ST | ε

Now we've procrastinated the decision. But the language is still ambiguous. What if the input is

if C1 then if C2 then ST1 else ST2

Which bracketing (rule) should we choose?

A common practice is to choose S' → else ST

We can force this if we create the parsing table by hand.

Possible Solutions to the Nondeterminism Problem

I. Modify the language

A. Add a terminator $
II. Change the parsing algorithm
A. Add one character look ahead
B. Use a parsing table
C. Tailor parsing table entries by hand
III. Modify the grammar
A. Left factor
B. Get rid of left recursion

Lecture Notes 17 Top Down Parsing 7

 The Price We Pay

Old Grammar New Grammar

[1] E→E+T E → TE'
[2] E→T E' → +TE'
E' → ε
[3] T→T*F T → FT'
[4] T→F T' → *FT'
T' → ε
[5] F → (E) F → (E)
[6] F → id F → idA
[7] F → id(E) A→ε
A → (E)
input = id + id + id

T E'

F T' + T E'

id A ε F T' + T E'

ε id A ε F T' ε

ε id A ε

Comparing Regular and Context-Free Languages

Regular Languages Context-Free Languages

• regular exprs. • context-free grammars

or
• regular grammars
• = DFSAs • = NDPDAs
• recognize • parse
• minimize FSAs • find deterministic grammars
• find efficient parsers

Lecture Notes 17 Top Down Parsing 8

 Bottom Up Parsing
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Parsing, Section 3.

Bottom Up Parsing
An Example:

[1] E→E+T
[2] E→T
[3] T→T*F
[4] T→F
[5] F → (E)
[6] F → id

id + id * id $

Creating a Bottom Up PDA

There are two basic actions:
1. Shift an input symbol onto the stack
2. Reduce a string of stack symbols to a nonterminal

M will be:
$/S/
p q

So, to construct M from a grammar G, we need the following transitions:

(1) The shift transitions:

((p, a, ε), (p, a)), for each a ∈ Σ

(2) The reduce transitions:

((p, ε, αR), (p, A)), for each rule A → α in G.

(3) The finish up transition (accept):

((p, $, S), (q, ε))

(This is the “bottom-up” CFG to PDA conversion algorithm.)

Lecture Notes 18 Bottom Up Parsing 1

 M for Expressions
0 (p, a, ε), (p, a) for each a ∈ Σ
1 (p, ε, T + E), (p, E)
2 (p, ε, T), (p, E)
3 (p, ε, F * T), (p, T)
4 (p, ε, F), (p, T)
5 (p, ε, “)”E”(“ ), (p, F)
6 (p, ε, id), (p, F)
7 (p, $, E), (q, ε)

trans (action) state unread input stack

p id + id * id$ ε
0 (shift) p + id * id$ id
6 (reduce F → id) p + id * id$ F
4 (reduce T → F) p + id * id$ T
2 (reduce E → T) p + id * id$ E
0 (shift) p id * id$ +E
0 (shift) p * id$ id+E
6 (reduce F → id) p * id$ F+E
4 (reduce T → F) p * id$ T+E (could also reduce)
0 (shift) p id$ *T+E
0 (shift) p $ id*T+E
6 (reduce F → id) p $ F*T+E (could also reduce T → F)
3 (reduce T → T * F) p $ T+E
1 (reduce E → E + T) p $ E
7 (accept) q $ ε

The Parse Tree

E T

T T F

F F

id + id * id $

Producing the Rightmost Derivation

We can reconstruct the derivation that we found by reading the results of the parse bottom to top, producing:

EÞ E+ id* idÞ
E+ T Þ T+ id*idÞ
E+ T* FÞ F+ id*idÞ
E+ T* idÞ id+ id*id
E+ F* idÞ

This is exactly the rightmost derivation of the input string.

Lecture Notes 18 Bottom Up Parsing 2

 Possible Solutions to the Nondeterminism Problem

1) Modify the language

• Add a terminator $

2) Change the parsing algorithm

• Add one character look ahead
• Use a parsing table
• Tailor parsing table entries by hand
• Switch to a bottom-up parser

3) Modify the grammar

• Left factor
• Get rid of left recursion

Solving the Shift vs. Reduce Ambiguity With a Precedence Relation

Let's return to the problem of deciding when to shift and when to reduce (as in our example).

We chose, correctly, to shift * onto the stack, instead of reducing T+E to E.

This corresponds to knowing that “+” has low precedence, so if there are any other operations, we need to do them first.

Solution:
1. Add a one character lookahead capability.
2. Define the precedence relation
P⊆ ( V × {Σ ∪ $} )
top next
stack input
symbol symbol

If (a,b) is in P, we reduce (without consuming the input) . Otherwise we shift (consuming the input).

How Does It Work?

We're reconstructing rightmost derivations backwards. So suppose a rightmost derivation contains

βγabx
Ý corresponding to a rule A →γa and not some rule X → ab
βAbx
Ý*
S

We want to undo rule A. So if the top of the stack is

a
γ and the next input character is b, we reduce
now, before we put the b on the stack.

To make this happen, we put (a, b) in P. That means we'll try to reduce if a is on top of the stack and b is the next character. We
will actually succeed if the next part of the stack is γ.

Lecture Notes 18 Bottom Up Parsing 3

 Example

T*F
Ý corresponding to a rule T→T*F
T
Ý* Input: id * id * id
E

We want to undo rule T. So if the top of the stack is

F
* and the next input character is anything legal, we reduce.
T

The precedence relation for expressions:

V\Σ ( ) id + * $
(
) • • • •
id • • • •
+
*
E
T • • •
F • • • •

A Different Example
E+T
Ý* corresponding to a rule E→E+T
E

We want to undo rule E if the input is E+T$

or E + T + id
but not E + T * id

The top of the stack is

T
+
E

The precedence relation for expressions:

V\Σ ( ) id + * $
(
) • • • •
id • • • •
+
*
E
T • • •
F • • • •

Lecture Notes 18 Bottom Up Parsing 4

 What About If Then Else?

ST → if C then ST else ST
ST → if C then ST

What if the input is

if C1 then if C2 then ST1 else ST2

1 2

Which bracketing (rule) should we choose?

We don't put (ST, else) in the precedence relation, so we will not reduce at 1. At 2, we reduce:

ST2 2
else
ST1 1
then
C2
if
then
C1
if

Resolving Reduce vs. Reduce Ambiguities

0 (p, a, ε), (p, a) for each a ∈ Σ
1 (p, ε, T + E), (p, E)
2 (p, ε, T), (p, E)
3 (p, ε, F * T), (p, T)
4 (p, ε, F), (p, T)
5 (p, ε, “)” E “(“ ), (p, F)
6 (p, ε, id), (p, F)
7 (p, $, E), (q, ε)

trans (action) state unread input stack

p id + id * id$ ε
0 (shift) p + id * id$ id
6 (reduce F → id) p + id * id$ F
4 (reduce T → F) p + id * id$ T
2 (reduce E → T) p + id * id$ E
0 (shift) p id * id$ +E
0 (shift) p * id$ id+E
6 (reduce F → id) p * id$ F+E
4 (reduce T → F) p * id$ T+E (could also reduce)
0 (shift) p id$ *T+E
0 (shift) p $ id*T+E
6 (reduce F → id) p $ F*T+E (could also reduce T → F)
3 (reduce T → T * F) p $ T+E
1 (reduce E → E + T) p $ E
7 (accept) q $ ε

Lecture Notes 18 Bottom Up Parsing 5

 The Longest Prefix Heuristic

A simple to implement heuristic rule, when faced with competing reductions, is:

Choose the longest possible stack string to reduce.

Example:
T
Ý
Suppose the stack has F * T + E
ß
T

We call grammars that become unambiguous with the addition of a precedence relation and the longest string reduction heuristic
weak precedence grammars.

Possible Solutions to the Nondeterminism Problem in a Bottom Up Parser

1) Modify the language

• Add a terminator $

2) Change the parsing algorithm

• Add one character lookahead
• Use a precedence table
• Add the longest first heuristic for reduction
• Use an LR parser

3) Modify the grammar

LR Parsers

LR parsers scan each input Left to right and build a Rightmost derivation. They operate bottom up and deterministically using a
parsing table derived from a grammar for the language to be recognized.

A grammar that can be parsed by an LR parser examining up to k input symbols on each move is an LR(k) grammar. Practical
LR parsers set k to 1.

An LALR ( or Look Ahead LR) parser is a specific kind of LR parser that has two desirable properties:
• The parsing table is not huge.
• Most useful languages can be parsed.

Another big reason to use an LALR parser:

There are automatic tools that will construct the required parsing table from a grammar and some optional additional
information.

We will be using such a tool: yacc

Lecture Notes 18 Bottom Up Parsing 6

 How an LR Parser Works

Input String

state
state Lexical Analyzer
state
state
Output Token
Stack

Parsing Table

In simple cases, think of the "states" on the stack as corresponding to either terminal or nonterminal characters.

In more complicated cases, the states contain more information: they encode both the top stack symbol and some facts about
lower objects in the stack. This information is used to determine which action to take in situations that would otherwise be
ambiguous.
The Actions the Parser Can Take

At each step of its operation, an LR parser does the following two things:

1) Based on its current state, it decides whether it needs a lookahead token. If it does, it gets one.
2) Based on its current state and the lookahead token if there is one, it chooses one of four possible actions:
• Shift the lookahead token onto the stack and clear the lookahead token.
• Reduce the top elements of the stack according to some rule of the grammar.
• Detect the end of the input and accept the input string.
• Detect an error in the input.

Lecture Notes 18 Bottom Up Parsing 7

 A Simple Example
0: S → rhyme $end ;
1: rhyme → sound place ;
2: sound → DING DONG ;
3: place → DELL

state 0 (empty)
$accept : _rhyme $end ⇐ the rule this came from
DING shift 3 state 3
. error current position of input
rhyme goto 1 if none of the others match
sound goto 2 push state 2
state 1 (rhyme)
$accept : rhyme_$end
$end accept if we see EOF, accept
. error
state 2 (sound)
rhyme : sound_place
DELL shift 5
. error by rule 1
place goto 4
state 3 (DING)
sound : DING_DONG state 5 (DELL)
DONG shift 6 place : DELL_ (3)
. error . reduce 3
state 4 (place) state 6 (DONG)
rhyme : sound place_ (1) sound : DING DONG_ (2)
. reduce 1 . reduce 2

When the States Are More than Just Stack Symbols

[1] <stmt> → procname ( <paramlist>)

[2] <stmt> → <exp> := <exp>
[3] <paramlist> → <paramlist>, <param> | <param>
[4 ] <param> → id
[5] <exp> → arrayname (<subscriptlist>)
[6] <subscriptlist> → <subscriptlist>, <sub> | <sub>
[7] <sub> → id

Example:
id
procname ( id) (
procname

Should we reduce id by rule 4 or rule 7?

procid
proc(
procname

The parsing table can get complicated as we incorporate more stack history into the states.

Lecture Notes 18 Bottom Up Parsing 8

 The Language Interpretation Problem:

Input: -(17 * 83.56) + 72 / 12

Output: -1414.52

The Language Interpretation Problem:

Input: -(17 * 83.56) + 72 / 12

Compute the answer

Output: -1414.52

The Language Interpretation Problem:

Input: -(17 * 83.56) + 72 / 12

Parse the input ✸2

A tree of actions, whose structure corresponds to the structure of the input.

Compute the answer

Output: -1414.52

Lecture Notes 18 Bottom Up Parsing 9

 The Language Interpretation Problem:

Input: -(17 * 83.56) + 72 / 12

Lexical analysis of the input ✸1

A string of input tokens, corresponding to the primitive objects of which the input is composed:
-(id * id) + id / id

Parse the input ✸2

A tree of actions, whose structure corresponds to the structure of the input.

Compute the answer

Output: -1414.52

yacc and lex

Lexical analysis of the input ✸1

Parse the input ✸2

Where do the procedures to do these things come from?

regular expressions that describe patterns

lex

lexical analyzer ✸1

grammar rules and other facts about the language

yacc

parser ✸2

Lecture Notes 18 Bottom Up Parsing 10

 lex

The input to lex: definitions

%%
rules
%%
user routines

All strings that are not matched by any rule are simply copied to the output.

Rules:

[ \t]+; get rid of blanks and tabs

[A-Za-z][A-Za-z0-9]* return(ID); find identifiers

[0-9]+ { sscanf(yytext, "%d", &yylval);

return (INTEGER); } return INTEGER and put the value in yylval

How Does lex Deal with Ambiguity in Rules?

lex invokes two disambiguating rules:

1. The longest match is prefered.

2. Among rules that matched the same number of characters, the rule given first is preferred.

Example:
integer action 1
[a-z]+ action 2

input: integers take action 2

integer take action 1

yacc
(Yet Another Compiler Compiler)
The input to yacc:

declarations
%%
rules
%%
#include "lex.yy.c"
any other programs

This structure means that lex.yy.c will be compiled as part of y.tab.c, so it will have access to the same token names.

Declarations:

%token name1 name2 …

Rules:

V :a b c
V :a b c {action}
V :a b c {$$ = $2} returns the value of b

Lecture Notes 18 Bottom Up Parsing 11

 Example
Input to yacc:
%token DING DONG DELL
%%
rhyme : sound place ;
sound : DING DONG ;
place : DELL
%%
#include "lex.yy.c"

state 0 (empty) state 3 (DING)

$accept : _rhyme $end sound : DING_DONG
DING shift 3 DONG shift 6
. error . error
rhyme goto 1 state 4 (place)
sound goto 2 rhyme : sound place_ (1)
state 1 (rhyme) . reduce 1
$accept : rhyme_$end state 5 (DELL)
$end accept place : DELL_ (3)
. error . reduce 3
state 2 (sound) state 6 (DONG)
rhyme : sound_place sound : DING DONG_ (2)
DELL shift 5 . reduce 2
. error
place goto 4
How Does yacc Deal with Ambiguity in Grammars?

The parser table that yacc creates represents some decision about what to do if there is ambiguity in the input grammar rules.
How does yacc make those decisions? By default, yacc invokes two disambiguating rules:
1. In the case of a shift/reduce conflict, shift.
2. In the case of a reduce/reduce conflict, reduce by the earlier grammar rule.
yacc tells you when it has had to invoke these rules.

Shift/Reduce Conflicts - If Then Else

ST → if C then ST else ST
ST → if C then ST

What if the input is

if C1 then if C2 then ST1 else ST2

1 2
Which bracketing (rule) should we choose?

yacc will choose to shift rather than reduce.

ST2 2
else
ST1 1
then
C2
if
then
C1
if

Lecture Notes 18 Bottom Up Parsing 12

 Shift/Reduce Conflicts - Left Associativity

We know that we can force left associativity by writing it into our grammars.

Example:

E→E+T E
E→T
T → id E T

E T

id + id + id

What does the shift rather than reduce heuristic if we instead write:

E→E+E id + id + id
E → id
Shift/Reduce Conflicts - Operator Precedence

Recall the problem: input: id + id * id

T Should we reduce or shift on * ?

+
E

The "always shift" rule solves this problem.

But what about: id * id + id

T Should we reduce or shift on + ?

*
E This time, if we shift, we'll fail.

One solution was the precedence table, derived from an unambiguous grammar, which can be encoded into the parsing table of an
LR parser, since it tells us what to do for each top-of-stack, input character combination.

Operator Precedence

We know that we can write an unambiguous grammar for arithmetic expressions that gets the precedence right. But it turns out
that we can build a faster parser if we instead write:

E → E + E | E * E | (E) | id

And, in addition, we specify operator precedence. In yacc, we specify associativity (since we might not always want left) and
precedence using statements in the declaration section of our grammar:

%left '+' '-'

%left '*' '/'

Operators on the first line have lower precedence than operators on the second line, and so forth.

Lecture Notes 18 Bottom Up Parsing 13

 Reduce/Reduce Conflicts
Recall:

2. In the case of a reduce/reduce conflict, reduce by the earlier grammar rule.

This can easily be used to simulate the longest prefix heuristic, "Choose the longest possible stack string to reduce."

[1] E→E+T
[2] E→T
[3] T→T*F
[4] T→F
[5] F → (E)
[6] F → id

Generating an Executable System

Step 1: Create the input to lex and the input to yacc.

Step 2:
$ lex ourlex.l creates lex.yy.c
$ yacc ouryacc.y creates y.tab.c
$ cc -o ourprog y.tab.c -ly -ll actually compiles y.tab.c and lex.yy.c, which is included.
-ly links the yacc library, which includes main and yyerror.
-ll links the lex library
Step 3: Run the program
$ ourprog

Runtime Communication Between lex and yacc-Generated Modules

Parser read the value of the token

ask return
for a a yylval
token token

Lexical Analyer
set the value of the token

Summary

Efficient parsers for languages with the complexity of a typical programming language or command line interface:

• Make use of special purpose constructs, like precedence, that are very important in the target languages.

• May need complex transition functions to capture all the relevant history in the stack.

• Use heuristic rules, like shift instead of reduce, that have been shown to work most of the time.

• Would be very difficult to construct by hand (as a result of all of the above).

• Can easily be built using a tool like yacc.

Lecture Notes 18 Bottom Up Parsing 14

 Languages That Are and Are Not Context-Free
Read K & S 3.5, 3.6, 3.7.
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: Closure Properties of Context-Free
Languages
Read Supplementary Materials: Context-Free Languages and Pushdown Automata: The Context-Free Pumping Lemma.
Do Homework 16.
Deciding Whether a Language is Context-Free

Theorem: There exist languages that are not context-free.

Proof:
(1) There are a countably infinite number of context-free languages. This true because every description of a context-free
language is of finite length, so there are a countably infinite number of such descriptions.

(2) There are an uncountable number of languages.

Thus there are more languages than there are context-free languages.

So there must exist some languages that are not context-free.

Example: {anbncn}
Showing that a Language is Context-Free

Techniques for showing that a language L is context-free:

1. Exhibit a context-free grammar for L.

2. Exhibit a PDA for L.
3. Use the closure properties of context-free languages.

Unfortunately, these are weaker than they are for regular languages.

The Context-Free Languages are Closed Under Union

Let G1 = (V1, Σ1, R1, S1) and

G2 = (V2, Σ2, R2, S2)

Assume that G1 and G2 have disjoint sets of nonterminals, not including S.

Let L = L(G1) ∪ L(G2)

We can show that L is context-free by exhibiting a CFG for it:

The Context-Free Languages are Closed Under Concatenation

Let G1 = (V1, Σ1, R1, S1) and

G2 = (V2, Σ2, R2, S2)

Assume that G1 and G2 have disjoint sets of nonterminals, not including S.

Let L = L(G1) L(G2)

We can show that L is context-free by exhibiting a CFG for it:

Lecture Notes 19 Languages That Are and Are Not Context Free 1
 The Context-Free Languages are Closed Under Kleene Star

Let G1 = (V1, Σ1, R1, S1)

Assume that G1 does not have the nonterminal S.

Let L = L(G1)*

We can show that L is context-free by exhibiting a CFG for it:

What About Intersection and Complement?

We know that they share a fate, since

L1 ∩ L2 = L1 ∪ L2

But what fate?

We proved closure for regular languages two different ways. Can we use either of them here:
1. Given a deterministic automaton for L, construct an automaton for its complement. Argue that, if closed under complement
and union, must be closed under intersection.
2. Given automata for L1 and L2, construct a new automaton for L1 ∩ L2 by simulating the parallel operation of the two original
machines, using states that are the Cartesian product of the sets of states of the two original machines.

More on this later.

The Intersection of a Context-Free Language and a Regular Language is Context-Free

L = L(M1), a PDA = (K1, Σ, Γ1, ∆1, s1, F1)

R = L(M2), a deterministic FSA = (K2, Σ, δ, s2, F2)

We construct a new PDA, M3, that accepts L ∩ R by simulating the parallel execution of M1 and M2.

M = (K1 × K2, Σ, Γ1, ∆, (s1, s2), F1 × F2)

Insert into ∆:

For each rule ((q1, a, β), (p1, γ)) in ∆1,

and each rule (q2, a, p2) in δ,
(((q1, q2), a, β), ((p1, p2), γ))

For each rule ((q1, ε, β), (p1, γ) in ∆1,

and each state q2 in K2,
(((q1, q2), ε, β), ((p1, q2), γ))

This works because: we can get away with only one stack.

Lecture Notes 19 Languages That Are and Are Not Context Free 2
 Example

L= a nb n ∩ (aa)*(bb)*

b/a/ a
A B 1 2
a//a b/a/ a
b
b
3 4
b
((A, a, ε), (A, a)) (1, a, 2)
((A, b, a), (B, ε)) (1, b, 3)
((B, b, a), (B, ε)) (2, a, 1)
(3, b, 4)
(4, b, 3)
A PDA for L:
Don’t Try to Use Closure Backwards

One Closure Theorem:

If L1 and L2 are context free, then so is

L3 = L1 ∪ L2.

But what if L3 and L1 are context free? What can we say about L2?

L3 = L1 ∪ L2.

Example:

anbnc* = anbnc* ∪ anbncn

The Context-Free Pumping Lemma

This time we use parse trees, not automata as the basis for our argument.

u v x y z

If L is a context-free language, and if w is a string in L where |w| > K, for some value of K, then w can be rewritten as uvxyz,
where |vy| > 0 and |vxy| ≤ M, for some value of M.

uxz, uvxyz, uvvxyyz, uvvvxyyyz, etc. (i.e., uvnxynz, for n ≥ 0) are all in L.

Lecture Notes 19 Languages That Are and Are Not Context Free 3
 Some Tree Basics

root

height
nodes

leaves

yield

Theorem: The length of the yield of any tree T with height H and branching factor (fanout) B is ≤ BH.

Proof: By induction on H. If H is 1, then just a single rule applies. By definition of fanout, the longest yield is B.
Assume true for H = n.
Consider a tree with H = n + 1. It consists of a root, and some number of subtrees, each of which is of height ≤ n (so induction
hypothesis holds) and yield ≤ Bn. The number of subtrees ≤ B. So the yield must be ≤ B(Bn) or Bn+1.

What Is K?
S

u v x y z

Let T be the number of nonterminals in G.

If there is a tree of height > T, then some nonterminal occurs more than once on some path. If it does, we can pump its yield.
Since a tree of height = T can produce only strings of length ≤ BT, any string of length > BT must have a repeated nonterminal and
thus be pumpable.

So K = BT, where T is the number of nonterminals in G and B is the branching factor (fanout).

What is M?

u v x y z

Assume that we are considering the bottom most two occurrences of some nonterminal. Then the yield of the upper one is at
most BT+1 (since only one nonterminal repeats).

So M = BT+1.

Lecture Notes 19 Languages That Are and Are Not Context Free 4
 The Context-Free Pumping Lemma

Theorem: Let G = (V, Σ, R, S) be a context-free grammar with T nonterminal symbols and fanout B. Then any string w ∈ L(G)
where |w| > K (BT) can be rewritten as w = uvxyz in such a way that:
• |vy| > 0,
• |vxy| ≤ M (BT+1), (making this the "strong" form),
• for every n ≥ 0, uvnxynz is in L(G).

Proof:
Let w be such a string and let T be the parse tree with root labeled S and with yield w that has the smallest number of leaves
among all parse trees with the same root and yield. T has a path of length at least T+1, with a bottommost repeated nonterminal,
which we'll call A. Clearly v and y can be repeated any number of times (including 0). If |vy| = 0, then there would be a tree with
root S and yield w with fewer leaves than T. Finally, |vxy| ≤ BT+1.

An Example of Pumping

L = {anbncn : n≥ 0}

Choose w = aibici where i > éK/3ù (making |w| > K)

u v x y z

Unfortunately, we don't know where v and y fall. But there are two possibilities:
1. If vy contains all three symbols, then at least one of v or y must contain two of them. But then uvvxyyz contains at least one
out of order symbol.
2. If vy contains only one or two of the symbols, then uvvxyyz must contain unequal numbers of the symbols.

Using the Strong Pumping Lemma for Context Free Languages

If L is context free, then
There exist K and M (with M ≥ K) such that
For all strings w, where |w| > K,
(Since true for all such w, it must be true for any paricular one, so you pick w)
(Hint: describe w in terms of K or M)

there exist u, v, x, y, z such that w = uvxyz and |vy| > 0, and

|vxy| ≤ M, and
for all n ≥ 0, uvnxynz is in L.

We need to pick w, then show that there are no values for uvxyz that satisfy all the above criteria. To do that, we just need to
focus on possible values for v and y, the pumpable parts. So we show that all possible picks for v and y violate at least one of
the criteria.

Write out a single string, w (in terms of K or M) Divide w into regions.

For each possibility for v and y (described in terms of the regions defined above), find some value n such that uvnxynz is not in L.
Almost always, the easiest values are 0 (pumping out) or 2 (pumping in). Your value for n may differ for different cases.

Lecture Notes 19 Languages That Are and Are Not Context Free 5
 v y n why the resulting string is not in L

[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]

Convince the reader that there are no other cases.

Q. E. D.

A Pumping Lemma Proof in Full Detail

Proof that L = {anbncn : n≥ 0} is not context free.

Suppose L is context free. The context free pumping lemma applies to L. Let M be the number from the pumping lemma.
Choose w = aMbMcM. Now w ∈ L and |w| > M ≥ K. From the pumping lemma, for all strings w, where |w| > K, there exist u, v, x,
y, z such that w = uvxyz and |vy| > 0, and |vxy| ≤ M, and for all n ≥ 0, uvnxynz is in L. There are two main cases:
1. Either v or y contains two or more different types of symbols (“a”, “b” or “c”). In this case, uv2xy2z is not of the form
a*b*c* and hence uv2xy2z ∉L.
2. Neither v nor y contains two or more different types of symbols. In this case, vy may contain at most two types of
symbols. The string uv0xy0z will decrease the count of one or two types of symbols, but not the third, so uv0xy0z ∉L
Cases 1 and 2 cover all the possibilities. Therefore, regardless of how w is partitioned, there is some uvnxynz that is not in L.
Contradiction. Therefore L is not context free.

Note: the underlined parts of the above proof is “boilerplate” that can be reused. A complete proof should have this text or
something equivalent.

Context-Free Languages Over a Single-Letter Alphabet

Theorem: Any context-free language over a single-letter alphabet is regular.

Examples:

L = {anbn}
L′ = {anan}
= {a2n}
= {w ∈ {a}* : |w| is even}

L = {wwR : w ∈ {a, b}*}

L′ = {wwR : w ∈ {a}*}
= {ww: w ∈ {a}*}
= {w ∈ {a}* : |w| is even}

L = {anbm : n, m ≥ 0 and n ≠ m}
L′ = {anam : n, m ≥ 0 and n ≠ m}
=

Proof: See Parikh's Theorem

Lecture Notes 19 Languages That Are and Are Not Context Free 6
 Another Language That Is Not Context Free
L = {an : n ≥ 1 is prime}

Two ways to prove that L is not context free:

1. Use the pumping lemma:

Choose a string w = an such that n is prime and n > K.
w = aaaaaaaaaaaaaaaaaaaaaaa
u v x y z
Let vy = ap and uxz = ar. Then r + kp must be prime for all values of k. This can't be true, as we argued to show that L was not
regular.

2. |ΣL| = 1. So if L were context free, it would also be regular. But we know that it is not. So it is not context free either.

Using Pumping and Closure

L = {w ∈ {a, b, c}* : w has an equal number of a's, b's, and c's}

L is not context free.

Try pumping: Let w = aKbKcK

Now what?

Using Intersection with a Regular Language to Make Pumping Tractable

L = {tt : t ∈ {a, b}* }

Let's try pumping: |w| > K

t t
u v x y z

What if u is ε,
v is w,
x is ε,
y is w, and
z is ε

Then all pumping tells us is that tntn is in L.

Lecture Notes 19 Languages That Are and Are Not Context Free 7
 L = {tt : t ∈ {a, b}* }

What if we let |w| > M, i.e. choose to pump the string aMbaMb:

Now v and y can't be t, since |vxy| ≤ M:

t t
u v x y z

Suppose |v| = |y|. Now we have to show that repeating them makes the two copies of t different. But we can’t.

L = {tt : t ∈ {a, b}* }

But let's consider L' = L ∩ a*b*a*b*

This time, we let |w| > 2M, and the number of both a's and b's in w >M:

1 2 3 4
aaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbb
t t
u v x y z

Now we use pumping to show that L' is not context free.

First, notice that if either v or y contains both a's and b's, then we immediately violate the rules for L' when we pump.

So now we know that v and y must each fall completely in one of the four marked regions.

L' = {tt : t ∈ {a, b}* } ∩ a*b*a*b*

|w| > 2M, and the number of both a's and b's in w >M:

1 2 3 4
aaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbb
t t
u v x y z

Consider the combinations of (v, y):

(1,1)
(2,2)
(3,3)
(4,4)
(1,2)
(2,3)
(3,4)
(1,3)
(2,4)
(1,4)

Lecture Notes 19 Languages That Are and Are Not Context Free 8
 The Context-Free Languages Are Not Closed Under Intersection

Proof: (by counterexample)

Consider L = {anbncn: n ≥ 0}

L is not context-free.

Let L1 = {anbncm: n, m ≥ 0} /* equal a's and b's

L2 = {ambncn: n, m ≥ 0} /* equal b's and c's

Both L1 and L2 are context-free.

But L = L1 ∩ L2.

So, if the context-free languages were closed under intersection, L would have to be context-free. But it isn't.

The Context-Free Languages Are Not Closed Under Complementation

Proof: (by contradiction)

By definition:

L1 ∩ L2 = L1 ∪ L2

Since the context-free languages are closed under union, if they were also closed under complementation, they would necessarily
be closed under intersection. But we just showed that they are not. Thus they are not closed under complementation.

The Deterministic Context-Free Languages Are Closed Under Complement

Proof:

Let L be a language such that L$ is accepted by the deterministic PDA M. We construct a deterministic PDA M' to accept (the
complement of L)$, just as we did for FSMs:

1. Initially, let M' = M.

2. M' is already deterministic.
3. Make M' simple. Why?
4. Complete M' by adding a dead state, if necessary, and adding all required transitions into it, including:
• Transitions that are required to assure that for all input, stack combinations some transition can be followed.
• If some state q has a transition on (ε, ε) and if it does not later lead to a state that does consume something then
make a transiton on (ε, ε) to the dead state.
5. Swap final and nonfinal states.
6. Notice that M′ is still deterministic.

Lecture Notes 19 Languages That Are and Are Not Context Free 9
 An Example of the Construction

L = anbn M accepts L$ (and is deterministic):

a//a b/a/
b/a/ $/ε/
1 2 3

$/ε/

Set M = M'. Make M simple.

a/a/aa
a/Z/aZ b/a/
ε/ε/Z b/a/ $/Z/
0 1 2 3

$/Z/

The Construction, Continued

Add dead state(s) and swap final and nonfinal states:

a/a/aa
a/Z/aZ b/a/
ε/ε/Z b/a/ $/Z/
0 1 2 3

$/Z/

b/Z/, $/a/
a//, $/a/, b/Z/
4

a//, b//, $//, ε/a/, ε/Z/

Issues: 1) Never having the machine die

2) ¬(L$) ≠ (¬L)$
3) Keeping the machine deterministic

Deterministic vs. Nondeterministic Context-Free Languages

Theorem: The class of deterministic context-free languages is a proper subset of the class of context-free languages.

Proof: Consider L = {anbmcp : m ≠ n or m ≠ p} L is context free (we have shown a grammar for it).

But L is not deterministic. If it were, then its complement L1 would be deterministic context free, and thus certainly context free.
But then
L2 = L1 ∩ a*b*c* (a regular language)
would be context free. But
L2 = {anbncn : n ≥ 0}, which we know is not context free.

Thus there exists at least one context-free language that is not deterministic context free.

Note that deterministic context-free languages are not closed under union, intersection, or difference.

Lecture Notes 19 Languages That Are and Are Not Context Free 10
 Decision Procedures for CFLs & PDAs

Decision Procedures for CFLs

There are decision procedures for the following (G is a CFG):

• Deciding whether w ∈ L(G).
• Deciding whether L(G) = ∅.
• Deciding whether L(G) is finite/infinite.

Such decision procedures usually involve conversions to Chomsky Normal Form or Greibach Normal Form. Why?

Theorem: For any context free grammar G, there exists a number n such that:
1. If L(G) ≠ ∅, then there exists a w ∈ L(G) such that |w| < n.
2. If L(G) is infinite, then there exists w ∈ L(G) such that n ≤ |w| < 2n.

There are not decision procedures for the following:

• Deciding whether L(G) = Σ*.
• Deciding whether L(G1) = L(G2).

If we could decide these problems, we could decide the halting problem. (More later.)

Decision Procedures for PDA’s

There are decision procedures for the following (M is a PDA):

• Deciding whether w ∈ L(M).
• Deciding whether L(M) = ∅.
• Deciding whether L(M) is finite/infinite.

Convert M to its equivalent PDA and use the corresponding CFG decision procedure. Why avoid using PDA’s directly?

There are not decision procedures for the following:

• Deciding whether L(M) = Σ*.
• Deciding whether L(M1) = L(M2).

If we could decide these problems, we could decide the halting problem. (More later.)

Lecture Notes 19 Languages That Are and Are Not Context Free 11
 Comparing Regular and Context-Free Languages

Regular Languages Context-Free Languages

• regular exprs. • context-free grammars

• or
• regular grammars
• recognize • parse
• = DFSAs • = NDPDAs
• recognize • parse
• minimize FSAs • find deterministic grammars
• find efficient parsers
• closed under: • closed under:
∗ concatenation ∗ concatenation
∗ union ∗ union
∗ Kleene star ∗ Kleene star
∗ complement
∗ intersection • intersection w/ reg. langs
• pumping lemma • pumping lemma
• deterministic = nondeterministic • deterministic ≠ nondeterministic

Languages and Machines

Recursively Enumerable
Languages

Recursive
Languages

Context-Free
Languages

Regular
Languages

FSMs
D ND

PDAs

Lecture Notes 19 Languages That Are and Are Not Context Free 12
 Turing Machines
Read K & S 4.1.
Do Homework 17.

Grammars, Recursively Enumerable Languages, and Turing Machines

Recursively
L
Enumerable
Language

Unrestricted
Grammar Accepts

Turing
Machine

Turing Machines

Can we come up with a new kind of automaton that has two properties:
• powerful enough to describe all computable things
unlike FSMs and PDAs
• simple enough that we can reason formally about it
like FSMs and PDAs
unlike real computers
Turing Machines

❑ à ❑ a b b a ❑ ❑ ❑

At each step, the machine may:

• go to a new state, and Finite State Control
• either
• write on the current square, or s1, s2, … h1, h2
• move left or right

A Formal Definition
A Turing machine is a quintuple (K, Σ, δ, s, H):
K is a finite set of states;
Σ is an alphabet, containing at least ❑ and à, but not → or ←;
s ∈ K is the initial state;
H ⊆ K is the set of halting states;
δ is a function from:
(K - H) × Σ to K × (Σ ∪ {→, ←})
non-halting state × input symbol state × action (write or move)
such that
(a) if the input symbol is à, the action is →, and
(b) à can never be written .

Lecture Notes 20 Turing Machines 1

 Notes on the Definition

1. The input tape is infinite to the right (and full of ❑), but has a wall to the left. Some definitions allow infinite tape in both
directions, but it doesn't matter.

2. δ is a function, not a relation. So this is a definition for deterministic Turing machines.

3. δ must be defined for all state, input pairs unless the state is a halt state.

4. Turing machines do not necessarily halt (unlike FSM's). Why? To halt, they must enter a halt state. Otherwise they loop.

5. Turing machines generate output so they can actually compute functions.

A Simple Example

A Turing Machine Odd Parity Machine:

❑ à ❑ 0 1 1 0 ❑ ❑ ❑

Σ = 0, 1, à, ❑
s=
H=
δ=

Formalizing the Operation

à a a b b ❑ ❑ ❑ (1)

à ❑ a a b b ❑ ❑ ❑ (2)

A configuration of a Turing machine

M = (K, Σ, δ, s, H) is a member of

K × àΣ* × (Σ*(Σ - {❑})) ∪ ε

state input up input after
to scanned scanned square
square

The input after the scanned square may be empty, but it may not end with a blank. We assume the entire tape to the right of the
input is filled with blanks.

(1) (q, àaab, b) = (q, àaabb)

(2) (h, à❑aabb, ε) = (h, à❑aabb) a halting configuration

Lecture Notes 20 Turing Machines 2

 Yields

(q1, w1a1u1) |-M (q2, w2a2u2), a1 and a2 ∈ Σ, iff ∃ b ∈ Σ ∪ {←, →}, δ(q1, a1) = (q2, b) and either:

(1) b ∈ Σ, w1 = w2, u1 = u2, and a2 = b (rewrite without moving the head)

| w1 | a1 | u1 |
à ❑ a a b b ❑ ❑ ❑ à❑aabb

| w2 | a2 | u2 |
à ❑ a a a b ❑ ❑ ❑ à❑aaab

Yields, Continued

(2) b = ←, w1 = w2a2, and either

(a) u2 = a1u1, if a1 ≠ ❑ or u1 ≠ ε,

| w1 | a1 | u1 |
à ❑ a a a b ❑ ❑ ❑ à❑aaab

| w2 | a2 | u2 |
à ❑ a a a b ❑ ❑ ❑ à❑aaab

or (b) u2 = ε, if a1 = ❑ and u1 = ε
| w1 | a1 |u1|
à ❑ a a a b ❑ ❑ ❑ à❑aaab❑

| w1 | a1 |u1|
à ❑ a a a b ❑ ❑ ❑ à❑aaab

If we scan left off the first square of the blank region, then drop that square from the configuration.

Yields, Continued

(3) b = →, w2 = w1a1, and either

(a) u1 = a2u2

| w1 | a1 | u1 |
à ❑ a a a b ❑ ❑ ❑ à❑aaab

| w2 | a2 | u2 |
à ❑ a a a b ❑ ❑ ❑ à❑aaab

or (b) u1 = u2 = ε and a2 = ❑
| w1 | a1 |u1|
à ❑ a a a b ❑ ❑ ❑ à❑aaab

| w2 | a2 |u2|
à ❑ a a a b ❑ ❑ ❑ à❑aaab❑

If we scan right onto the first square of the blank region, then a new blank appears in the configuration.

Lecture Notes 20 Turing Machines 3

 Yields, Continued

For any Turing machine M, let |-M* be the reflexive, transitive closure of |-M.

Configuration C1 yields configuration C2 if

C1 |-M* C2.

A computation by M is a sequence of configurations C0, C1, …, Cn for some n ≥ 0 such that

C0 |-M C1 |-M C2 |-M … |-M Cn.

We say that the computation is of length n or that it has n steps, and we write
C0 |-Mn Cn
A Context-Free Example

M takes a tape of a's then b's, possibly with more a's, and adds b's as required to make the number of b's equal the number of a's.

à ❑ a a a b ❑ ❑ ❑

K = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
Σ = a, b, à, ❑, 1, 2
s= 0 H = {9} δ=

0 a/1

❑ /→
a,1,2/→ 1,2/←
a/1 1/→ b/2 2/←
1 2 3 4 5

❑/2 2/←
❑/❑ 6 ❑/→

1/a;2/b
7 8
∀/→
❑/❑

An Example Computation

à ❑ a a a b ❑ ❑ ❑

(0, à❑aaab) |-M

(1, à❑aaab) |-M
(2, à❑1aab) |-M
(3, à❑1aab) |-M
(3, à❑1aab) |-M
(3, à❑1aab) |-M
(4, à❑1aa2) |-M
...

Lecture Notes 20 Turing Machines 4

 Notes on Programming

The machine has a strong procedural feel.

It's very common to have state pairs, in which the first writes on the tape and the second move. Some definitions allow both
actions at once, and those machines will have fewer states.

There are common idioms, like scan left until you find a blank.

Even a very simple machine is a nuisance to write.

A Notation for Turing Machines

(1) Define some basic machines

• Symbol writing machines

For each a ∈ Σ - {à}, define Ma, written just a, = ({s, h}, Σ, δ, s, {h}),
for each b ∈ Σ - {à}, δ(s, b) = (h, a)
δ(s, à) = (s, →)
Example:
a writes an a

• Head moving machines

For each a ∈ {←, →}, define Ma, written R(→) and L(←):
for each b ∈ Σ - {à}, δ(s, b) = (h, a)
δ(s, à) = (s, →)
Examples:
R moves one square to the right
aR writes an a and then moves one square to the right.

A Notation for Turing Machines, Cont'd

(2) The rules for combining machines: as with FSMs

>M1 a M2
b

M3

• Start in the start state of M1.

• Compute until M1 reaches a halt state.
• Examine the tape and take the appropriate transition.
• Start in the start state of the next machine, etc.
• Halt if any component reaches a halt state and has no place to go.
• If any component fails to halt, then the entire machine may fail to halt.

Lecture Notes 20 Turing Machines 5

 Shorthands

a
M1 M2 becomes M1 a, b M2
b

M1 all elems of Σ M2 becomes M1 M2

or
M1M2

MM becomes M2

M1 all elems of Σ M2 becomes M1 x≠a M2

except a
and x takes on the value of the current square

M1 a, b M2 becomes M1 x = a, b M2

and x takes on the value of the current square

M x?y M2

if x = y then take the transition

e.g., > x≠❑ Rx if the current square is not blank, go right and copy it.

Some Useful Machines

>R ¬❑ find the first blank square to the right of the current square

R❑

>L ¬❑ find the first blank square to the left of the current square

L❑

>R ❑ find the first nonblank square to the right of the current square

R¬❑

>L ❑ find the first nonblank square to the left of the current square

L¬❑

Lecture Notes 20 Turing Machines 6

 More Useful Machines

La find the first occurrence of a to the left of the current square

Ra,b find the first occurrence of a or b to the right of the current square

La,b a M1 find the first occurrence of a or b to the left of the current square, then go to M1 if the detected
b character is a; go to M2 if the detected character is b

M2

Lx=a,b find the first occurrence of a or b to the left of the current square and set x to the value found

Lx=a,bRx find the first occurrence of a or b to the left of the current square, set x to the value found, move one
square to the right, and write x (a or b)

An Example
Input: à❑w w ∈ {1}*
Output: à❑w3

Example: à ❑111❑❑❑❑❑❑❑❑❑❑❑❑❑

>R1,❑ 1 #R❑#R#L❑
❑

L # 1
❑

H
A Shifting Machine S←
Input: ❑❑w❑
Output: ❑w❑

Example: ❑❑abba❑❑❑❑❑❑❑❑❑❑❑❑❑

> L❑ R x≠❑ ❑LxR

x=❑

Lecture Notes 20 Turing Machines 7

 Computing with Turing Machines
Read K & S 4.2.
Do Homework 18.

Turing Machines as Language Recognizers

Convention: We will write the input on the tape as:

à❑w❑ , w contains no ❑s
The initial configuration of M will then be:
(s, à❑w)
A recognizing Turing machine M must have two halting states: y and n
Any configuration of M whose state is:
y is an accepting configuration
n is a rejecting configuration
Let Σ0, the input alphabet, be a subset of ΣM-{❑,à}
Then M decides a language L ⊆ Σ0* iff for any string
w ∈ Σ0*it is true that:
if w ∈ L then M accepts w, and
if w ∉ L then M rejects w.
A language L is recursive if there is a Turing machine M that decides it.

A Recognition Example
L = {anbncn : n ≥ 0}

Example: à❑aabbcc❑❑❑❑❑❑❑❑❑

Example: à❑aaccb❑❑❑❑❑❑❑❑❑

a’ a, b’ b, c’
> R a a’ R b b’ R c c’ L❑
❑, b’, c’ c, a’, c’, ❑
b,c ❑, a, b’, a’
b’,c’ R a, b, c, a’ n

❑
y

Lecture Notes 21 Computing with Turing Machines 1

 Another Recognition Example
L = {wcw : w ∈ {a, b}*}

Example: à❑abbcabb❑❑❑

Example: à❑acabb❑❑❑

>R x=a,b ❑ Rc,❑

c ❑ ❑ c

R¬# ¬❑ n ¬(y ? x ) Ry=¬#

❑ y?x

y #L❑

Do Turing Machines Stop?

FSMs Always halt after n steps, where n is the length of the input. At that point, they either accept or reject.

PDAs Don't always halt, but there is an algorithm to convert any PDA into one that does halt.

Turing machines Can do one of three things:

(1) Halt and accept
(2) Halt and reject
(3) Not halt

And now there is no algorithm to determine whether a given machine always halts.

Computing Functions

Let Σ0 ⊆ Σ - {à, ❑} and let w ∈ Σ0*

Convention: We will write the input on the tape as: à❑w❑

The initial configuration of M will then be: (s, à❑w)

Define M(w) = y iff:

• M halts if started in the input configuration,
• the tape of M when it halts is à❑y❑, and
• y ∈ Σ0*

Let f be any function from Σ0* to Σ0*.

We say that M computes f if, for all w ∈ Σ0*, M(w) = f(w)

A function f is recursive if there is a Turing machine M that computes it.

Lecture Notes 21 Computing with Turing Machines 2

 Example of Computing a Function

f(w) = ww

Input: à❑w❑❑❑❑❑❑ Output: à❑ww❑

Define the copy machine C: à❑w❑❑❑❑❑❑ à à❑w❑w❑

Remember the S← machine:

à❑w❑w❑ à à❑ww❑

> L❑ R x≠❑ ❑LxR

x=❑

L
Then the machine to compute f is just >C S L❑←

Computing Numeric Functions

We say that a Turing machine M computes a function f from Nk to N provided that

num(M(n1;n2;…nk)) = f(num(n1), … num(nk))

Example: Succ(n) = n + 1

We will represent n in binary. So n∈ 0 ∪ 1{0,1}*

Input: à❑n❑❑❑❑❑❑ Output: à❑n+1❑

à❑1111❑❑❑❑ Output: à❑10000❑

Why Are We Working with Our Hands Tied Behind Our Backs?

Turing machines are more powerful than any of the other formalisms we have studied so far.

Turing machines are a lot harder to work with than all the real computers we have available.

Why bother?

The very simplicity that makes it hard to program Turing machines makes it possible to reason formally about what they can do.
If we can, once, show that anything a real computer can do can be done (albeit clumsily) on a Turing machine, then we have a
way to reason about what real computers can do.

Lecture Notes 21 Computing with Turing Machines 3

 Recursively Enumerable and Recursive Languages
Read K & S 4.5.
Recursively Enumerable Languages

Let Σ0, the input alphabet to a Turing machine M, be a subset of ΣM - {❑, à}

Let L ⊆ Σ0*.

M semidecides L iff
for any string w ∈ Σ0*,
w∈LÞ M halts on input w
w∉L Þ M does not halt on input w
M(w) = ↑

L is recursively enumerable iff there is a Turing machine that semidecides it.

Examples of Recursively Enumerable Languages

L = {w ∈ {a, b}* : w contains at least one a}

¬a
>R

❑b b b b b b❑❑❑❑❑

L = {w ∈ {a, b, (, ) }* : w contains at least one set of balanced parentheses}

❑
> R),❑ ) ❑L(,❑
❑

L❑

❑b b b b b b) ❑❑❑❑❑

Recursively Enumerable Languages that Aren't Also Recursive

A Real Life Example:

L = {w ∈ {friends} : w will answer the message you've just sent out}

Theoretical Examples
L = {Turing machines that halt on a blank input tape}
Theorems with valid proofs.

Lecture Notes 22 Recursively Enumerable and Recursive Languages 1

 Why Are They Called Recursively Enumerable Languages?
Enumerate means list.

We say that Turing machine M enumerates the language L iff, for some fixed state q of M,
L = {w : (s, à❑) |-M* (q, à❑w)}

q w

A language is Turing-enumerable iff there is a Turing machine that enumerates it.

Note that q is not a halting state. It merely signals that the current contents of the tape should be viewed as a member of L.

Recursively Enumerable and Turing Enumerable

Theorem: A language is recursively enumerable iff it is Turing-enumerable.

Proof that Turing-enumerable implies RE: Let M be the Turing machine that enumerates L. We convert M to a machine M' that
semidecides L:
1. Save input w.
2. Begin enumerating L. Each time an element of L is enumerated, compare it to w. If they match, accept.

=w? halt
w3, w2, w1

M M'

The Other Way

Proof that RE implies Turing-enumerable:
If L ⊆ Σ* is a recursively enumerable language, then there is a Turing machine M that semidecides L.
A procedure to enumerate all elements of L:
Enumerate all w ∈ Σ* lexicographically.
e.g., ε, a, b, aa, ab, ba, bb, …
As each string wi is enumerated:
1. Start up a copy of M with wi as its input.
2. Execute one step of each Mi initiated so far, excluding only those that have previously halted.
3. Whenever an Mi halts, output wi.

ε [1]
ε [2] a [1]
ε [3] a [2] b [1]
ε [4] a [3] b [2] aa [1]
ε [5] a [4] b [3] aa [2] ab [1]
ε [6] a [5] aa [3] ab [2] ba [1]

Lecture Notes 22 Recursively Enumerable and Recursive Languages 2

 Every Recursive Language is Recursively Enumerable

If L is recursive, then there is a Turing machine that decides it.

From M, we can build a new Turing machine M' that semidecides L:

1. Let n be the reject (and halt) state of M.
2. Then add to δ'
((n, a), (n, a)) for all a ∈ Σ

a/a
y n y n

What about the other way around?

Not true. There are recursively enumerable languages that are not recursive.

The Recursive Languages Are Closed Under Complement

Proof: (by construction) If L is recursive, then there is a Turing machine M that decides L.

We construct a machine M' to decide L by taking M and swapping the roles of the two halting states y and n.
M: M':

y n n y

This works because, by definition, M is

• deterministic
• complete
Are the Recursively Enumerable Languages Closed Under Complement?

M: M':

Lemma: There exists at least one language L that is recursively enumerable but not recursive.

Proof that M' doesn't exist: Suppose that the RE languages were closed under complement. Then if L is RE, L would be RE. If
that were true, then L would also be recursive because we could construct M to decide it:
1. Let T1 be the Turing machine that semidecides L.
2. Let T2 be the Turing machine that semidecides L.
3. Given a string w, fire up both T1 and T2 on w. Since any string in Σ* must be in either L or L, one of the two machines will
eventually halt. If it's T1, accept; if it's T2, reject.
But we know that there is at least one RE language that is not recursive. Contradiction.

Lecture Notes 22 Recursively Enumerable and Recursive Languages 3

 Recursive and RE Languages

Theorem: A language is recursive iff both it and its complement are recursively enumerable.

Proof:
• L recursive implies L and ¬L are RE: Clearly L is RE. And, since the recursive languages are closed under complement,
¬L is recursive and thus also RE.
• L and ¬L are RE implies L recursive: Suppose L is semidecided by M1 and ¬L is semidecided by M2. We construct M to
decide L by using two tapes and simultaneously executing M1 and M2. One (but not both) must eventually halt. If it's M1,
we accept; if it's M2 we reject.
Lexicographic Enumeration

We say that M lexicographically enumerates L if M enumerates the elements of L in lexicographic order. A language L is
lexicographically Turing-enumerable iff there is a Turing machine that lexicographically enumerates it.

Example: L = {anbncn}

Lexicographic enumeration:
Proof

Theorem: A language is recursive iff it is lexicographically Turing-enumerable.

Proof that recursive implies lexicographically Turing enumerable: Let M be a Turing machine that decides L. Then M'
lexicographically generates the strings in Σ* and tests each using M. It outputs those that are accepted by M. Thus M'
lexicographically enumerates L.

Σ*3, Σ*2, Σ*1 ∈L? yes Σ*k

no

M
M'

Proof, Continued

Proof that lexicographically Turing enumerable implies recursive: Let M be a Turing machine that lexicographically enumerates
L. Then, on input w, M' starts up M and waits until either M generates w (so M' accepts), M generates a string that comes after w
(so M' rejects), or M halts (so M' rejects). Thus M' decides L.

= w? yes
L3, L2, L1
> w? no
M
no more Lis? no

M'

Lecture Notes 22 Recursively Enumerable and Recursive Languages 4

 Partially Recursive Functions

Languages Functions
Tm always halts recursive recursive
Tm halts if yes recursively ?
enumerable

domain range

Suppose we have a function that is not defined for all elements of its domain.

Example: f: N → N, f(n) = n/2

Partially Recursive Functions

domain range

One solution: Redefine the domain to be exactly those elements for which f is defined:

domain
range

But what if we don't know? What if the domain is not a recursive set (but it is recursively enumerable)? Then we want to define
the domain as some larger, recursive set and say that the function is partially recursive. There exists a Turing machine that halts
if given an element of the domain but does not halt otherwise.

Lecture Notes 22 Recursively Enumerable and Recursive Languages 5

 Language
Summary

IN OUT

Semidecidable Recursively
Enumerable Enumerable
Unrestricted grammar

Decision procedure Recursive Diagonalization

Lexicicographically enumerable Reduction
Complement is recursively enumer.

CF grammar Context Free Pumping

PDA Closure
Closure

Regular expression Regular Pumping

FSM Closure
Closure

Lecture Notes 22 Recursively Enumerable and Recursive Languages 6

 Turing Machine Extensions
Read K & S 4.3.1, 4.4.
Do Homework 19.

Turing Machine Definitions

An alternative definition of a Turing machine:

(K, Σ, Γ, δ, s, H):

Γ is a finite set of allowable tape symbols. One of these is ❑.

Σ is a subset of Γ not including ❑, the input symbols.

δ is a function from:
K×Γ to K× (Γ - {❑}) × {←, →}
state, tape symbol, L or R

❑ ❑ a b b a ❑ ❑ ❑

Example transition: ((s, a), (s, b, →))

Do these Differences Matter?

Remember the goal:

Define a device that is:

• powerful enough to describe all computable things,
• simple enough that we can reason formally about it

Both definitions are simple enough to work with, although details may make specific arguments easier or harder.

But, do they differ in their power?

Answer: No.

Consider the differences:

• One way or two way infinite tape: we're about to show that we can simulate two way infinite with ours.
• Rewrite and move at the same time: just affects (linearly) the number of moves it takes to solve a problem.

Turing Machine Extensions

In fact, there are lots of extensions we can make to our basic Turing machine model. They may make it easier to write Turing
machine programs, but none of them increase the power of the Turing machine because:

We can show that every extended machine has an equivalent basic machine.

We can also place a bound on any change in the complexity of a solution when we go from an extended machine to a basic
machine.

Some possible extensions:

• Multiple tapes
• Two-way infinite tape
• Multiple read heads
• Two dimensional “sheet” instead of a tape
• Random access machine
• Nondeterministic machine

Lecture Notes 23 Turing Machine Extensions 1

 Multiple Tapes

❑ ❑ a b b a ❑ ❑ ❑

❑ b a b b a ❑ ❑ ❑

❑ ❑ 1 2 2 1 ❑ ❑ ❑

The transition function for a k-tape Turing machine:

((K-H) , Σ1 to (K, Σ1' ∪ {←, →}

, Σ2 , Σ2' ∪ {←, →}
, . , .
, . , .
, Σk) , Σk' ∪ {←, →})

Input: input as before on tape 1, others blank

Output: output as before on tape 1, others ignored

An Example of a Two Tape Machine

Copying a string

❑ ❑ a b b a ❑ ❑ ❑

❑ ❑ ❑ ❑ ❑ ❑ ❑ ❑ ❑

❑ ❑ a b b a ❑ ❑ ❑

❑ ❑ a b b a ❑ ❑ ❑

❑ ❑ a b b a ❑ ❑ ❑

❑ ❑ a b b a ❑ ❑ ❑

Lecture Notes 23 Turing Machine Extensions 2

 Another Two Tape Example - Addition

❑ 1 0 1 ; 1 1 0 ❑

❑ ❑ ❑ ❑ ❑ ❑ ❑ ❑ ❑

❑ 0 0 0 0 1 1 0 ❑

❑ 1 0 1 ❑ ❑ ❑ ❑ ❑

Adding Tapes Adds No Power

Theorem: Let M be a k-tape Turing machine for some k ≥ 1. Then there is a standard Turing machine M' where Σ ⊆ Σ', and such
that:
• For any input string x, M on input x halts with output y on the first tape iff M' on input x halts at the same halting state and
with the same output on its tape.
• If, on input x, M halts after t steps, then M' halts after a number of steps which is O(t ⋅ (|x| + t)).
Proof: By construction

à ❑ a b a ❑ ❑
à 0 0 1 0 0 0 0 ❑ ❑
à a b b a b a
0 1 0 0 0 0 0

Alphabet (Σ') of M' = Σ ∪ (Σ × {0, 1})k

e.g., à, (à, 0, à, 0), (❑, 0, a, 1)
The Operation of M'
à ❑ a b a ❑ ❑
à 0 0 1 0 0 0 0 ❑ ❑
à a b b a b a
0 1 0 0 0 0 0

1. Set up the multitrack tape:

1) Shift input one square to right, then set up each square appropriately.
2. Simulate the computation of M until (if) M would halt: (start each step to the right of the divided tape)
1) Scan left and store in the state the k-tuple of characters under the read heads. Move back right.
2) Scan left and update each track as required by the transitions of M. Move back right.
i) If necessary, subdivide a new square into tracks.
3. When M would halt, reformat the tape to throw away all but track 1, position the head correctly, then go to M's halt
state.
How Many Steps Does M' Take?
Let: x be the input string, and
t be the number of steps it takes M to execute.
Step 1 (initialization) O(|x|)
Step 2 ( computation)
Number of passes = t
Work at each pass: 2.1 = 2 ⋅ (length of tape)
= 2 ⋅ (|x| + 2 + t)
2.2 = 2 ⋅ (|x| + 2 + t)
Total = O(t ⋅ (|x| + t))
Step 3 (clean up) O(length of tape)
Total = O(t ⋅ (|x| + t))

Lecture Notes 23 Turing Machine Extensions 3

 Two-Way Infinite Tape
Our current definition:

à a b c d ❑ ❑

Proposed definition:

❑ ❑ g f e a b c d ❑

Simulation:

Track 1 à a b c d ❑ ❑

Track 2 à e f g ❑ ❑ ❑

Simulating a PDA
The components of a PDA:
• Finite state controller
• Input tape
• Stack
The simulation:
• Finite state controller:
• Input tape:
• Stack:

Track 1 à a a a b b ❑
(Input)

Track 2 à ❑ a a ❑ ❑ ❑

Corresponding to
a
a

Simulating a Turing Machine with a PDA with Two Stacks

à a b a a # a a b a
Ý

a #
a a
b a
a b
à a

Lecture Notes 23 Turing Machine Extensions 4

 Random Access Turing Machines
A random access Turing machine has:
• a fixed number of registers
• a finite length program, composed of instructions with operators such as read, write, load, store, add, sub, jump
• a tape
• a program counter
Theorem: Standard Turing machines and random access Turing machines compute the same things. Furthermore, the number of
steps it takes a standard machine is bounded by a polynomial in the number of steps it takes a random access machine.
Nondeterministic Turing Machines

A nondeterministic Turing machine is a quintuple (K, Σ, ∆, s, H)

where K, Σ, s, and H are as for standard Turing machines, and ∆ is a subset of
((K - H) × Σ) × (K × (Σ ∪ {←, →}))

à❑abab

à❑abab à❑abab

à❑abab à❑bbab

What does it mean for a nondeterministic Turing machine to compute something?

• Semidecides - at least one halts.
• Decides - ?
• Computes - ?
Nondeterministic Semideciding

Let M = (K, Σ, ∆, s, H) be a nondeterministic Turing machine. We say that M accepts an input

w ∈ (Σ - {à, ❑})* iff
(s, à❑w) yields a least one accepting configuration.

We say that M semidecides a language

L ⊆ (Σ - {à, ❑})* iff
for all w ∈ (Σ - {à, ❑})*:
w ∈ L iff
(s, à❑w) yields a least one halting configuration.

An Example
L = {w ∈ {a, b, c, d}* : there are two of at least one letter}
¬a/→

2 a

∀/→ a/→ ¬b/→

→
0 ❑/→ 1 b/→ 3 b h

c/→ ¬c/→ c

d/→ 4

¬d/→ d

Lecture Notes 23 Turing Machine Extensions 5

 Nondeterministic Deciding and Computing

M decides a language L if, for all w ∈ (Σ - {à, ❑})* :

1. all of M's computations on w halt, and
2. w ∈ L iff at least one of M's computations accepts.

M computes a function f if, for all w ∈ (Σ - {à, ❑})* :

1. all of M's computations halt, and
2. all of M's computations result in f(w)

Note that all of M's computations halt iff:

There is a natural number N, depending on M and w, such that there is no configuration C satisfying
(s, à❑w) |-MN C.
An Example of Nondeterministic Deciding

L = {w ∈ {0, 1}* : w is the binary encoding of a composite number}

M decides L by doing the following on input w:

1. Nondeterministically choose two binary numbers 1 < p, q, where |p| and |q| ≤ |w|, and write them on the tape, after w,
separated by ;.

à❑110011;111;1111❑❑

2. Multiply p and q and put the answer, A, on the tape, in place of p and q.

à❑110011;1011111❑❑

3. Compare A and w. If equal, go to y. Else go to n.

Equivalence of Deterministic and Nondeterministic Turing Machines

Theorem: If a nondeterministic Turing machine M semidecides or decides a language, or computes a function, then there is a
standard Turing machine M' semideciding or deciding the same language or computing the same function.

Note that while nondeterminism doesn’t change the computational power of a Turing Machine, it can exponentially increase its
speed!

Proof: (by construction)

For semideciding: We build M', which runs through all possible computations of M. If one of them halts, M' halts

Recall the way we did this for FSMs: simulate being in a combination of states.

Will this work here?

What about: Try path 1. If it accepts, accept. Else

Try path 2. If it accepts, accept. Else
•
•

Lecture Notes 23 Turing Machine Extensions 6

 The Construction

At any point in the operation of a nondeterministic machine M, the maximum number of branches is
r= |K| ⋅ (|Σ| + 2)
states actions

So imagine a table:

1 2 3 r
(q1,σ1) (p-,σ-) (p-,σ-) (p-,σ-) (p-,σ-) (p-,σ-)
(q1,σ2) (p-,σ-) (p-,σ-) (p-,σ-) (p-,σ-) (p-,σ-)
(q1,σn)
(q2,σ1)

(q|K|,σn)

Note that if, in some configuration, there are not r different legal things to do, then some of the entries on that row will repeat.

The Construction, Continued

Md: (suppose r = 6)

Tape 1: Input

Tape 2: 1 3 2 6 5 4 3 6

Md chooses its 1st move from column 1

Md chooses its 2nd move from column 3
Md chooses its 3rd move from column 2
•
•
until there are no more numbers on Tape 2

Md either:
• discovers that M would accept, or
• comes to the end of Tape 2.

In either case, it halts.

The Construction, Continued
M' (the machine that simulates M):

Tape 1: Input

Tape 2: Copy of Input

Md
Tape 3: 1 3 2 6 5 4 3 6

Steps of M':
write ε on Tape 3
until Md accepts do
(1) copy Input from Tape 1 to Tape 2
(2) run Md
(3) if Md accepts, exit
(4) otherwise, generate lexicographically next string on Tape 3.

Pass 1 2 3 7 8 9
Tape3 ε 1 2 ⋅⋅⋅ 6 11 12 ⋅⋅⋅ 2635

Lecture Notes 23 Turing Machine Extensions 7

 Nondeterministic Algorithms

Other Turing Machine Extensions

Multiple heads (on one tape)

Emulation strategy: Use tracks to keep track of tape heads. (See book)

Multiple tapes, multiple heads

Emulation strategy: Use tracks to keep track of tapes and tape heads.

Two-dimensional semi-infinite “tape”

Emulation strategy: Use diagonal enumeration of two-dimensional grid. Use second tape to help you keep track of
where the tape head is. (See book)

Two-dimensional infinite “tape” (really a sheet)

Emulation strategy: Use modified diagonal enumeration as with the semi-infinite case.

What About Turing Machine Restrictions?

Can we make Turing machines even more limited and still get all the power?

Example:

We allow a tape alphabet of arbitrary size. What happens if we limit it to:

• One character?
• Two characters?
• Three characters?

Lecture Notes 23 Turing Machine Extensions 8

 Problem Encoding, TM Encoding, and the Universal TM
Read K & S 5.1 & 5.2.

Encoding a Problem as a Language

A Turing Machines deciding a language is analogous to the TM solving a decision problem.

Problem: Is the number n prime?

Instance of the problem: Is the number 9 prime?
Encoding of the problem, ánññ: n as a binary number. Example: 1001

Problem: Is an undirected graph G connected?

Instance of the problem: Is the following graph connected?

1 2 3

4 5

Encoding of the problem, áGññ:

1) |V| as a binary number
2) A list of edges represented by pairs of binary numbers being the vertex numbers that the edge connects
3) All such binary numbers are separated by “/”.
Example: 101/1/10/10/11/1/100/10/101

Problem View vs. Language View

Problem View: It is unsolvable whether a Turing Machine halts on a given input. This is called the Halting Problem.

Language View: Let H = {áM, wñ : TM M halts on input string w}

H is recursively enumerable but not recursive.

The Universal Turing Machine

Problem: All our machines so far are hardwired.

Question: Does it make sense to talk about a programmable Turing machine that accepts as input
program input string
executes the program, and outputs
output string

Yes, it's called the Universal Turing Machine.

Notice that the Universal Turing machine semidecides H = {áM, wñ : TM M halts on input string w} = L(U).

To define the Universal Turing Machine U we need to do two things:

1. Define an encoding operation for Turing machines.
2. Describe the operation of U given an input tape containing two inputs:
• encoded Turing machine M,
• encoded input string to be given to M.

Lecture Notes 24 Problem Encoding, Turing Machine Encoding, and the Universal Turing Machine 1
 Encoding a Turing Machine M

We need to describe M = (K, Σ, δ, s, H) as a string. To do this we must:

1. Encode δ
2. Specify s.
3. Specify H (and y and n, if applicable)

1. To encode δ, we need to:

1. Encode the states
2. Encode the tape alphabet
3. Specify the transitions

1.1 Encode the states as

qs : s ∈ {0, 1}+ and
|s| = i and
i is the smallest integer such that 2i ≥ |K|

Example: 9 states i=4

s = q0000,
remaining states: q0001, q0010, q0011,
q0100, q0101, q0110, q0111, q1000

Encoding a Turing Machine M, Continued

1.2 Encode the tape alphabet as
as : s ∈ {0, 1}+ and
|s| = j and
j is the smallest integer such that 2j ≥ |Σ| + 2 (the + 2 allows for ← and →)
Example: Σ = {à, ❑, a, b} j = 3
❑= a000
à= a001
←= a010
→= a011
a= a100
b= a101

Encoding a Turing Machine M, Continued

1.3 Specify transitions as (state, input, state, output)
Example: (q00,a000,q11,a000)
2. Specify s as q0i
3. Specify H:
• States with no transitions out are in H.
• If M decides a language, then H = {y, n}, and we will adopt the convention that y is the lexicographically smaller of
the two states.
y = q010 n = q011
Encoding Input Strings

We encode input strings to a machine M using the same character encoding we use for M.
For example, suppose that we are using the following encoding for symbols in M:

symbol representation
❑ a000
à a001
← a010
→ a011
a a100

Then we would represent the string s = àaa❑a as "s" = ásñ = a001a100a100a000a100

Lecture Notes 24 Problem Encoding, Turing Machine Encoding, and the Universal Turing Machine 2
 An Encoding Example
Consider M = ({s, q, h}, {❑, à,a}, δ, s, {h}), where δ =

state symbol δ state/symbol representation

s a (q, ❑) s q00
s ❑ (h, ❑) q q01
s à (s, →) h q11
q a (s, a) ❑ a000
q ❑ (s, →) à a001
q à (q, →) ← a010
→ a011
a a100

The representation of M, denoted, "M", áMñ, or sometimes ρ(M) =

(q00,a100,q01,a000), (q00,a000,q11,a000), (q00,a001,q00,a011),
(q01,a100,q00,a100), (q01,a000,q00,a011), (q01,a001,q01,a011)

Another Win of Encoding

One big win of defining a way to encode any Turing machine M:

• It will make sense to talk about operations on programs (Turing machines). In other words, we can talk about some
Turing machine T that takes another Turing machine (say M1) as input and transforms it into a different machine
(say M2) that performs some different, but possibly related task.

Example of a transforming TM T:
Input: a machine M1 that reads its input tape and performs some operation P on it.
Output: a machine M2 that performs P on an empty input tape:

>R x≠❑ ❑

Là R M1

The Universal Turing Machine

The specification for U:
U("M" "w") = "M(w)"

"M ------------------------------ M" "w------------------------w"

1 0 0 0 0 0 0
à ❑ ❑ ❑ ❑ ❑ ❑ ❑ ❑ ❑
❑ ❑ ❑ ❑ ❑ ❑ ❑
❑ ❑ ❑ ❑ ❑ ❑ ❑ÿ
❑ ❑ ❑ ❑ ❑ ❑ ❑

"à ❑" "w--------------------w" ❑ ❑

1 0 0 0 0 0 0
à "M ---------------------------- M" ❑ ❑ ❑ ❑ ❑
1 0 0 0 0 0 0
q 0 0 0 ❑ ❑ ❑
1 ❑ ❑ ❑ ❑ ❑ ❑

Initialization of U:
1. Copy "M" onto tape 2
2. Insert "à❑" at the left edge of tape 1, then shift w over.
3. Look at "M", figure out what i is, and write the encoding of state s on tape 3.

Lecture Notes 24 Problem Encoding, Turing Machine Encoding, and the Universal Turing Machine 3
 The Operation of U

a 0 0 1 a 0 0
1 0 0 0 0 0 0
à "M ---------------------------- M" ❑ ❑ ❑ ❑ ❑
1 0 0 0 0 0 0
q 0 0 0 ❑ ❑ ❑
1 ❑ ❑ ❑ ❑ ❑ ❑

Simulate the steps of M:

1. Start with the heads:

tape 1: the a of the character being scanned,
tape 2: far left
tape 3: far left

2. Simulate one step:

1. Scan tape 2 for a quadruple that matches current state, input pair.
2. Perform the associated action, by changing tapes 1 and 3. If necessary, extend the tape.
3. If no quadruple found, halt. Else go back to 2.

An Example

Tape 1: a001a000a100a100a000a100
à ❑ a a ❑ a

Tape 2: (q00,a000,q11,a000), (q00,a001,q00,a011),

(q00,a100,q01,a000), (q01,a000,q00,a011),
(q01,a001,q01,a011), (q01,a100,q00,a100)

Tape 3: q01

Result of simulating the next step:

Tape 1: a001a000a100a100a000a100
à ❑ a a ❑ a

Tape 3: q00

If A Universal Machine is Such a Good Idea …

Could we define a Universal Finite State Machine?

Such a FSM would accept the language

L = {"F" "w" : F is a finite state machine, and w ∈ L(F) }

Lecture Notes 24 Problem Encoding, Turing Machine Encoding, and the Universal Turing Machine 4
 Grammars and Turing Machines
Do Homework 20.

Grammars, Recursively Enumerable Languages, and Turing Machines

L
Recursively
Enumerable
Language

Unrestricted
Grammar Accepts

Turing
Machine

Unrestricted Grammars

An unrestricted, or Type 0, or phrase structure grammar G is a quadruple

(V, Σ, R, S), where

• V is an alphabet,
• Σ (the set of terminals) is a subset of V,
• R (the set of rules) is a finite subset of
• (V* (V-Σ) V*) × V*,
context N context → result
• S (the start symbol) is an element of V - Σ.
We define derivations just as we did for context-free grammars.
The language generated by G is

{w ∈ Σ* : S ÞG* w}
There is no notion of a derivation tree or rightmost/leftmost derivation for unrestricted grammars.

Unrestricted Grammars
Example: L = anbncn, n > 0
S → aBSc
S → aBc
Ba → aB
Bc → bc
Bb → bb
Another Example

L = {w ∈ {a, b, c}+ : number of a's, b's and c's is the same}

S → ABCS CA → AC
S → ABC CB → BC
AB → BA A→a
BC → CB B→b
AC → CA C→c
BA → AB

Lecture Notes 25 Grammars and Turing Machines 1

 A Strong Procedural Feel

Unrestricted grammars have a procedural feel that is absent from restricted grammars.

Derivations often proceed in phases. We make sure that the phases work properly by using nonterminals as flags that we're in a
particular phase.

It's very common to have two main phases:

• Generate the right number of the various symbols.
• Move them around to get them in the right order.

No surprise: unrestricted grammars are general computing devices.

Equivalence of Unrestricted Grammars and Turing Machines

Theorem: A language is generated by an unrestricted grammar if and only if it is recursively enumerable (i.e., it is semidecided
by some Turing machine M).

Proof:
Only if (grammar → TM): by construction of a nondeterministic Turing machine.

If (TM → grammar): by construction of a grammar that mimics backward computations of M.

Proof that Grammar → Turing Machine

Given a grammar G, produce a Turing machine M that semidecides L(G).

M will be nondeterministic and will use two tapes:

à ❑ a b a ❑ ❑
à 0 1 0 0 0 0 0 ❑ ❑
à a S T a b ❑
0 1 0 0 0 0 0

For each nondeterministic "incarnation":

• Tape 1 holds the input.
• Tape 2 holds the current state of a proposed derivation.

At each step, M nondeterministically chooses a rule to try to apply and a position on tape 2 to start looking for the left hand side
of the rule. Or it chooses to check whether tape 2 equals tape 1. If any such machine succeeds, we accept. Otherwise, we keep
looking.

Lecture Notes 25 Grammars and Turing Machines 2

 Proof that Turing Machine → Grammar

Suppose that M semidecides a language L (it halts when fed strings in L and loops otherwise). Then we can build M' that halts in
the configuration (h, à❑).

We will define G so that it simulates M' backwards.

We will represent the configuration (q, àuaw) as
>uaqw<

M'
goes from

à ❑ a b b a ❑ ❑ ❑

à ❑ ❑ ❑ ❑ ❑ ❑ ❑ ❑

Then, if w ∈ L, we require that our grammar produce a derivation of the form

S ÞG >❑h< (produces final state of M')
ÞG* >❑abq< (some intermediate state of M')
ÞG* >❑sw< (the initial state of M')
ÞG w< (via a special rule to clean up >❑s)
ÞG w (via a special rule to clean up <)

The Rules of G
S → >❑h< (the halting configuration)

>❑s → ε (clean-up rules to be applied at the end)

<→ε

Rules that correspond to δ:

If δ(q, a) = (p, b) : bp → aq

If δ(q, a) = (p, →) : abp → aqb ∀b ∈ Σ

a❑p< → aq<

If δ(q, a) = (p, ←), a ≠ ❑ pa → aq

If δ(q, ❑) = (p, ←) p❑b → ❑qb ∀b ∈ Σ

p< → ❑q<

Lecture Notes 25 Grammars and Turing Machines 3

 A REALLY Simple Example
M' = (K, {a}, δ, s, {h}), where
δ ={ ((s, ❑), (q, →)), 1
((q, a), (q, →)), 2
((q, ❑), (t, ←)), 3
((t, a), (p, ❑)), 4
((t, ❑), (h, ❑)), 5
((p, ❑), (t, ←)) 6

L = a*

S →>❑h< (3) t❑❑ → ❑q❑

>❑s → ε t❑a → ❑qa
<→ε t< → ❑q<
(4) ❑p → at
(1) ❑❑q→ ❑s❑ (5) ❑h → ❑t
❑aq → ❑sa (6) t❑❑ → ❑p❑
❑❑q< → ❑s< t❑a → ❑pa
(2) a❑q → aq❑ t< → ❑p<
aaq → aqa
a❑q< → aq<
Working It Out

S →>❑h< 1 (3) t❑❑ → ❑q❑ 10

>❑s → ε 2 t❑a → ❑qa 11
<→ε 3 t< → ❑q< 12
(4) ❑p → at 13
(1) ❑❑q→ ❑s❑ 4 (5) ❑h → ❑t 14
❑aq → ❑sa 5 (6) t❑❑ → ❑p❑ 15
❑❑q< → ❑s< 6 t❑a → ❑pa 16
(2) a❑q → aq❑ 7 t< → ❑p< 17
aaq → aqa 8
a❑q< → aq< 9

>❑saa< 1 S Þ >❑h< 1
>❑aqa< 2 Þ >❑t< 14
>❑aaq< 2 Þ >❑❑p< 17
>❑aa❑q< 3 Þ >❑at< 13
>❑aat< 4 Þ >❑a❑p< 17
>❑a❑p< 6 Þ >❑aat< 13
>❑at< 4 Þ >❑aa❑q< 12
>❑❑p< 6 Þ >❑aaq< 9
>❑t< 5 Þ >❑aqa< 8
>❑h< Þ >❑saa< 5
Þ aa< 2
Þ aa 3

Lecture Notes 25 Grammars and Turing Machines 4

 An Alternative Proof

An alternative is to build a grammar G that simulates the forward operation of a Turing machine M. It uses alternating symbols
to represent two interleaved tapes. One tape remembers the starting string, the other “working” tape simulates the run of the
machine.

The first (generate) part of G:

Creates all strings over Σ* of the form
w = à à ❑ ❑ Qs a1 a1 a2 a2 a3 a3 ❑ ❑ …

The second (test) part of G simulates the execution of M on a particular string w. An example of a partially derived string:
à à ❑ ❑ a 1 b 2 c c b 4 Q3 a 3

Examples of rules:
b b Q 4 → b 4 Q 4 (rewrite b as 4)
b 4 Q 3 → Q 3 b 4 (move left)

The third (cleanup) part of G erases the junk if M ever reaches h.

Example rule:
#ha1→a#h (sweep # h to the right erasing the working “tape”)

Computing with Grammars

We say that G computes f if, for all w, v ∈Σ*,

SwS ÞG* v iff v = f(w)
Example:
S1S ÞG* 11
S11S ÞG* 111 f(x) = succ(x)
A function f is called grammatically computable iff there is a grammar G that computes it.

Theorem: A function f is recursive iff it is grammatically computable.

In other words, if a Turing machine can do it, so can a grammar.

Example of Computing with a Grammar

f(x) = 2x, where x is an integer represented in unary

G = ({S, 1}, {1}, R, S), where R =

S1 → 11S
SS → ε

Example:

Input: S111S

Output:

Lecture Notes 25 Grammars and Turing Machines 5

 More on Functions: Why Have We Been Using Recursive as a Synonym for Computable?
Primitive Recursive Functions

Define a set of basic functions:

• zerok (n1, n2, … nk) = 0
• identityk,j (n1, n2, … nk) = nj
• successor(n) = n + 1
Combining functions:
• Composition of g with h1, h2, … hk is
g(h1( ), h2( ), … hk( ))
• Primitive recursion of f in terms of g and h:
f(n1,n2,…nk, 0) = g(n1,n2,…nk)
f(n1,n2,…nk,m+1) = h(n1,n2,…nk, m, f(n1, n2,…nk,m))

Example: plus(n, 0) = n
plus(n, m+1) = succ(plus(n, m))

Primitive Recursive Functions and Computability

Trivially true: all primitive recursive functions are Turing computable.

What about the other way: Not all Turing computable functions are primitive recursive.

Proof:
Lexicographically enumerate the unary primitive recursive functions, f0, f1, f2, f3, ….
Define g(n) = fn(n) + 1.
G is clearly computable, but it is not on the list. Suppose it were fm for some m. Then
fm(m) = fm(m) + 1, which is absurd.

0 1 2 3 4
f0
f1
f2
f3 27
f4

Suppose g is f3. Then g(3) = 27 + 1 = 28. Contradiction.

Functions that Aren't Primitive Recursive

Example: Ackermann's function: A(0, y) = y + 1

A(x + 1, 0) = A(x, 1)
A(x + 1, y + 1) = A(x, A(x + 1, y))

0 1 2 3 4
0 1 2 3 4 5
1 2 3 4 5 6
2 3 5 7 9 11
3 5 13 29 61 125
4 13 65533 265536-3 * 265536
−3
65536
22 # 22 −3 %

* 19,729 digits 1017 seconds since big bang

# 105940 digits 1087 protons and neutrons
% 1010
5939
digits 10-23 light seconds = width
of proton or neutron
Thus writing digits at the speed of light on all protons and neutrons in the universe (all lined up) starting at the big bang would
have produced 10127 digits.

Lecture Notes 25 Grammars and Turing Machines 6

 Recursive Functions

A function is µ-recursive if it can be obtained from the basic functions using the operations of:
• Composition,
• Recursive definition, and
• Minimalization of minimalizable functions:

The minimalization of g (of k + 1 arguments) is a function f of k arguments defined as:

f(n1,n2,…nk) = the least m such at g(n1,n2,…nk,m)=1, if such an m exists,
0 otherwise

A function g is minimalizable iff for every n1,n2,…nk, there is an m such that g(n1,n2,…nk,m)=1.

Theorem: A function is µ-recursive iff it is recursive (i.e., computable by a Turing machine).

Partial Recursive Functions

Consider the following function f:
f(n) = 1 if TM(n) halts on a blank tape
0 otherwise

The domain of f is the natural numbers. Is f recursive?

domain range

Theorem: There are uncountably many partially recursive functions (but only countably many Turing machines).

Functions and Machines

Partial Recursive
Functions

Recursive
Functions

Primitive Recursive
Functions

Turing Machines

Lecture Notes 25 Grammars and Turing Machines 7

 Languages and Machines

Recursively Enumerable
Languages

Recursive
Languages

Context-Free
Languages

Deterministic
Context-Free
Languages

Regular
Languages

FSMs

DPDAs

NDPDAs

Turing Machines

Is There Anything In Between CFGs and Unrestricted Grammars?

Answer: yes, various things have been proposed.

Context-Sensitive Grammars and Languages:

A grammar G is context sensitive if all productions are of the form

x→y
and |x| ≤ |y|

In other words, there are no length-reducing rules.

A language is context sensitive if there exists a context-sensitive grammar for it.

Examples:
L = {anbncn, n > 0}
L = {w ∈ {a, b, c}+ : number of a's, b's and c's is the same}

Lecture Notes 25 Grammars and Turing Machines 8

 Context-Sensitive Languages are Recursive

The basic idea: To decide if a string w is in L, start generating strings systematically, shortest first. If you generate w, accept. If
you get to strings that are longer than w, reject.

Linear Bounded Automata

A linear bounded automaton is a nondeterministic Turing machine the length of whose tape is bounded by some fixed constant k
times the length of the input.

Example: L = {anbncn : n ≥ 0}

à❑aabbcc❑❑❑❑❑❑❑❑❑

a’ a,b’ b,c’
> R a a’ R b b’ R c c’ L❑
❑,b’,c’ c,a’,c’,❑
b,c ❑,a,b’,a’
b’,c’ R a,b,c,a’ n

❑
y

Context-Sensitive Languages and Linear Bounded Automata

Theorem: The set of context-sensitive languages is exactly the set of languages that can be accepted by linear bounded automata.

Proof: (sketch) We can construct a linear-bounded automaton B for any context-sensitive language L defined by some grammar
G. We build a machine B with a two track tape. On input w, B keeps w on the first tape. On the second tape, it
nondeterministically constructs all derivations of G. The key is that as soon as any derivation becomes longer than |w| we stop,
since we know it can never get any shorter and thus match w. There is also a proof that from any lba we can construct a context-
sensitive grammar, analogous to the one we used for Turing machines and unrestricted grammars.

Theorem: There exist recursive languages that are not context sensitive.

Lecture Notes 25 Grammars and Turing Machines 9

 Languages and Machines

Recursively Enumerable
Languages

Recursive
Languages

Context-Sensitive
Languages

Context-Free
Languages

Deterministic
Context-Free
Languages

Regular
Languages

FSMs

DPDAs

NDPDAs

Linear Bounded Automata

Turing Machines

Lecture Notes 25 Grammars and Turing Machines 10

 The Chomsky Hierarchy

Recursively Enumerable
Languages

Context-Sensitive
Languages

Context-Free
Languages

Regular
Type 0 Type 1 Type 2 (Type 3)
Languages
FSMs

PDAs

Linear Bounded Automata

Turing Machines

Lecture Notes 25 Grammars and Turing Machines 11

 Undecidabilty
Read K & S 5.1, 5.3, & 5.4.
Read Supplementary Materials: Recursively Enumerable Languages, Turing Machines, and Decidability.
Do Homeworks 21 & 22.
Church's Thesis
(Church-Turing Thesis)

An algorithm is a formal procedure that halts.

The Thesis: Anything that can be computed by any algorithm can be computed by a Turing machine.

Another way to state it: All "reasonable" formal models of computation are equivalent to the Turing machine.

This isn't a formal statement, so we can't prove it. But many different computational models have been proposed and they all turn
out to be equivalent.

Examples:
§ unrestricted grammars
§ lambda calculus
§ cellular automata
§ DNA computing
§ quantum computing (?)

The Unsolvability of the Halting Problem

Suppose we could implement the decision procedure

HALTS(M, x)
M: string representing a Turing Machine
x: string representing the input for M
If M(x) halts then True
else False
Then we could define
TROUBLE(x)
x: string
If HALTS(x, x) then loop forever
else halt

So now what happens if we invoke TROUBLE(“TROUBLE”), which invokes HALTS(“TROUBLE”, “TROUBLE”)

If HALTS says that TROUBLE halts on itself then TROUBLE loops. IF HALTS says that TROUBLE loops, then TROUBLE
halts. Either way, we reach a contradiction, so HALTS(M, x) cannot be made into a decision procedure.

Lecture Notes 26 Undecidability 1

 Another View

The Problem View: The halting problem is undecidable.

The Language View: Let H =

{"M" "w" : TM M halts on input string w}
H is recursively enumerable but not recursive.

Why?

H is recursively enumerable because it can be semidecided by U, the Universal Turing Machine.

But H cannot be recursive. If it were, then it would be decided by some TM MH. But MH("M" "w") would have to be:
If M is not a syntactically valid TM, then False.
else HALTS("M" "w")

But we know cannot that HALTS cannot exist.

If H were Recursive

H = {"M" "w" : TM M halts on input string w}

Theorem: If H were also recursive, then every recursively enumerable language would be recursive.

Proof: Let L be any RE language. Since L is RE, there exists a TM M that semidecides it.

Suppose H is recursive and thus is decided by some TM O (oracle).

We can build a TM M' from M that decides L:

1. M' transforms its input tape from à❑w❑ to à❑"M""w"❑.
2. M' invokes O on its tape and returns whatever answer O returns.

So, if H were recursive, all RE languages would be. But it isn't.

Undecidable Problems, Languages that Are Not Recursive, and Partial Functions

The Problem View: The halting problem is undecidable.

The Language View: Let H =

{"M" "w" : TM M halts on input string w}
H is recursively enumerable but not recursive.

The Functional View: Let f (w) = M(w)

f is a partial function on Σ*

"M""w" pairs

Lecture Notes 26 Undecidability 2

 Other Undecidable Problems About Turing Machines

• Given a Turing machine M, does M halt on the empty tape?

• Given a Turing machine M, is there any string on which M halts?
• Given a Turing machine M, does M halt on every input string?
• Given two Turing machines M1 and M2, do they halt on the same input strings?
• Given a Turing machine M, is the language that M semidecides regular? Is it context-free? Is it recursive?

Post Correspondence Problem

Consider two lists of strings over some alphabet Σ. The lists must be finite and of equal length.

A = x1, x2, x3, …, xn

B = y1, y2, y3, …, yn

Question: Does there exist some finite sequence of integers that can be viewed as indexes of A and B such that, when elements of
A are selected as specified and concatenated together, we get the same string we get when elements of B are selected also as
specified?

For example, if we assert that 1, 3, 4 is such a sequence, we’re asserting that x1x3x4 = y1y3y4

Any problem of this form is an instance of the Post Correspondence Problem.

Is the Post Correspondence Problem decidable?

Post Correspondence Problem Examples

i A B
1 1 111
2 10111 10
3 10 0

i A B
1 10 101
2 011 11
3 101 011

Some Languages Aren't Even Recursively Enumerable

A pragmatically non RE language: L1={ (i, j) : i, j are integers where the low order five digits of i are a street address number
and j is the number of houses with that number on which it rained on November 13, 1946 }

An analytically non RE language: L2={x : x = "M" of a Turing machine M and M("M") does not halt}

Why isn't L2 RE? Suppose it were. Then there would be a TM M* that semidecides L2. Is "M*" in L2?
• If it is, then M*("M*") halts (by the definition of M* as a semideciding machine for L2)
• But, by the definition of L2, if "M*" ∈ L2, then M*("M*") does not halt.
Contradiction. So L2 is not RE.

Another Non RE Language

Why not?

Lecture Notes 26 Undecidability 3

 Reduction

Let L1, L2 ⊆ Σ* be languages. A reduction from L1 to L2 is a recursive function τ: Σ* → Σ* such that

x ∈ L1 iff τ(x) ∈ L2.
Example:
L1 = {a, b : a,b ∈ N : b = a + 1}

ß τ = Succ

ß a, b becomes Succ(a), b

L2 = {a, b : a,b ∈ N : a = b}

If there is a Turing machine M2 to decide L2, then I can build a Turing machine M1 to decide L1:
1. Take the input and apply Succ to the first number.
2. Invoke M2 on the result.
3. Return whatever answer M2 returns.

Reductions and Recursive Languages

Theorem: If there is a reduction from L1 to L2 and L2 is recursive, then L1 is recursive.

x
M1 x ∈ L 1?

y = M2 yes yes
τ y ∈ L2?
τ(x) no no

Theorem: If there is a reduction from L1 to L2 and L1 is not recursive, then L2 is not recursive.

Reductions and RE Languages

Theorem: If there is a reduction from L1 to L2 and L2 is RE, then L1 is RE.

x
M1 x ∈ L1?

y = M2 halt halt
τ y ∈ L2?
τ(x)

Theorem: If there is a reduction from L1 to L2 and L1 is not RE, then L2 is not RE.

Lecture Notes 26 Undecidability 4

 Can it be Decided if M Halts on the Empty Tape?

This is equivalent to, "Is the language L2 = {"M" : Turing machine M halts on the empty tape} recursive?"

L1 =H= {s = "M" "w" : Turing machine M halts on input string w}

ß τ

(?M2) L2 = {s = "M" : Turing machine M halts on the empty tape}

Let τ be the function that, from "M" and "w", constructs "M*", which operates as follows on an empty input tape:
1. Write w on the tape.
2. Operate as M would have.

If M2 exists, then M1 = M2(Mτ(s)) decides L1.

A Formal Reduction Proof

Prove that L2 = {áMñ: Turing machine M halts on the empty tape} is not recursive.

Proof that L2 is not recursive via a reduction from H = {áM, wñ: Turing machine M halts on input string w}, a non-recursive
language. Suppose that there exists a TM, M2 that decides L2. Construct a machine to decide H as M1(áM, wñ) = M2(τ(áM, wñ)).
The τ function creates from áMñ and áwñ a new machine M*. M* ignores its input and runs M on w, halting exactly when M halts
on w.
• áM, wñ ∈ H Þ M halts on w Þ M* always halts Þε ∈ L(M*) Þ áM*ñ ∈ L2 Þ M2 accepts Þ M1 accepts.
• áM, wñ ∉ H Þ M does not halt on w Þ ε ∉ L(M*) Þ áM*ñ ∉ L2 Þ M2 rejects Þ M1 rejects.

Thus, if there is a machine M2 that decides L2, we could use it to build a machine that decides H. Contradiction. ∴L2 is not
recursive.

Important Elements in a Reduction Proof

• A clear declaration of the reduction “from” and “to” languages and what you’re trying to prove with the reduction.
• A description of how a machine is being constructed for the “from” language based on an assumed machine for the “to”
language and a recursive τ function.
• A description of the τ function’s inputs and outputs. If τ is doing anything nontrivial, it is a good idea to argue that it is
recursive.
• Note that machine diagrams are not necessary or even sufficient in these proofs. Use them as thought devices, where
needed.
• Run through the logic that demonstrates how the “from” language is being decided by your reduction. You must do both
accepting and rejecting cases.
• Declare that the reduction proves that your “to” language is not recursive.

The Most Common Mistake: Doing the Reduction Backwards

The right way to use reduction to show that L2 is not recursive:

1. Given that L1 is not recursive, L1
2. Reduce L1 to L2, i.e. show how to solve L1 (the known one) in terms of L2 (the unknown one)
L2
Example: If there exists a machine M2 that solves L2, the problem of deciding whether a Turing machine halts on a blank tape,
then we could do H (deciding whether M halts on w) as follows:
1. Create M* from M such that M*, given a blank tape, first writes w on its tape, then simulates the behavior of M.
2. Return M2("M*").

Doing it wrong by reducing L2 (the unknown one to L1): If there exists a machine M1 that solves H, then we could build a
machine that solves L2 as follows:
1. Return (M1("M", "")).

Lecture Notes 26 Undecidability 5

 Why Backwards Doesn't Work

Suppose that we have proved that the following problem L1 is unsolvable: Determine the number of days that have elapsed since
the beginning of the universe.

Now consider the following problem L2: Determine the number of days that had elapsed between the beginning of the universe
and the assassination of Abraham Lincoln.

Reduce L1 to L2: L1
L1 = L2 + (now - 4/9/1865)
L2

Reduce L2 to L1: L2
L2 = L1 - (now - 4/9/1865)
L1

Why Backwards Doesn't Work, Continued

L1 = days since beginning of universe

L2 = elapsed days between the beginning of the universe and the assassination of Abraham Lincoln.
L3 = days between the assassination of Abraham Lincoln and now.

Considering L2: L1
Reduce L1 to L2:
L1 = L2 + (now - 4/9/1865) L2

Reduce L2 to L1: L2
L2 = L1 - (now - 4/9/1865)
L1

Considering L3: L1
Reduce L1 to L3:
L1 = oops L3

Reduce L3 to L1: L3
L3 = L1 - 365 - (now - 4/9/1866)
L1

Is There Any String on Which M Halts?

L1 =H= {s = "M" "w" : Turing machine M halts on input string w}

ß τ

(?M2) L2 = {s = "M" : there exists a string on which Turing machine M halts}

Let τ be the function that, from "M" and "w", constructs "M*", which operates as follows:
1. M* examines its input tape.
2. If it is equal to w, then it simulates M.
3. If not, it loops.

Clearly the only input on which M* has a chance of halting is w, which it does iff M would halt on w.

If M2 exists, then M1 = M2(Mτ(s)) decides L1.

Lecture Notes 26 Undecidability 6

 Does M Halt on All Inputs?

L1 = {s = "M" : Turing machine M halts on the empty tape}

ß τ

(?M2) L2 = {s = "M" : Turing machine M halts on all inputs}

Let τ be the function that, from "M", constructs "M*", which operates as follows:
1. Erase the input tape.
2. Simulate M.

Clearly M* either halts on all inputs or on none, since it ignores its input.

If M2 exists, then M1 = M2(Mτ(s)) decides L1.

Rice's Theorem

Theorem: No nontrivial property of the recursively enumerable languages is decidable.

Alternate statement: Let P: 2Σ*→{true, false} be a nontrivial property of the recursively enumerable languages. The language
{“M”: P(L(M)) = True} is not recursive.

By "nontrivial" we mean a property that is not simply true for all languages or false for all languages.

Examples:
• L contains only even length strings.
• L contains an odd number of strings.
• L contains all strings that start with "a".
• L is infinite.
• L is regular.

Note:
Rice's theorem applies to languages, not machines. So, for example, the following properties of machines are decidable:
• M contains an even number of states
• M has an odd number of symbols in its tape alphabet
Of course, we need a way to define a language. We'll use machines to do that, but the properties we'll deal with are properties of
L(M), not of M itself.

Proof of Rice's Theorem

Proof: Let P be any nontrivial property of the RE languages.

L1 = H = {s = "M" "w" : Turing machine M halts on input string w}

ß τ

(?M2) L2 = {s = "M" : P(L(M)) = true}

Either P(∅) = true or P(∅) = false. Assume it is false (a matching proof exists if it is true). Since P is nontrivial, there is some
language LP such that P(LP) is true. Let MP be some Turing machine that semidecides LP.

Let τ construct "M*", which operates as follows:

1. Copy its input y to another track for later.
2. Write w on its input tape and execute M on w.
3. If M halts, put y back on the tape and execute MP.
4. If MP halts on y, accept.

Claim: If M2 exists, then M1 = M2(Mτ(s)) decides L1.

Lecture Notes 26 Undecidability 7

 Why?

Two cases to consider:

• "M" "w" ∈ H Þ M halts on w Þ M* will halt on all strings that are accepted by MP Þ L(M*) = L(MP) = LP Þ P(L(M*)) =
P(LP) = true Þ M2 decides P, so M2 accepts "M*" Þ M1 accepts.

• "M" "w" ∉ H Þ M doesn’t halt on w Þ M* will halt on nothing Þ L(M*) = ∅ Þ P(L(M*)) = P(∅) = false Þ M2 decides
P, so M2 rejects "M*" Þ M1 rejects.

Using Rice’s Theorem

Theorem: No nontrivial property of the recursively enumerable languages is decidable.

To use Rice’s Theorem to show that a language L is not recursive we must:

• Specify a language property, P(L)

• Show that the domain of P is the set of recursively enumerable languages.

• Show that P is nontrivial:

Ø P is true of at least one language
Ø P is false of at least one language

Using Rice’s Theorem: An Example

L = {s = "M" : there exists a string on which Turing machine M halts}.

= {s = "M" : L(M) ≠ ∅ }

• Specify a language property, P(L):

P(L) = True iff L ≠ ∅

• Show that the domain of P is the set of recursively enumerable languages.

The domain of P is the set of languages semidecided by some TM. This is exactly the set of RE languages.

• Show that P is nontrivial:

P is true of at least one language: P({ε}) = True
P is false of at least one language: P(∅) = False

Inappropriate Uses of Rice’s Theorem

Example 1:
L = {s = "M" : M writes a 1 within three moves}.

• Specify a language property, P(L)

P(M?) = True if M writes a 1 within three moves,
False otherwise
• Show that the domain of P is the set of recursively enumerable languages.
??? The domain of P is the set of all TMs, not their languages

Example 2:
L = {s = "M1" "M2": L(M1) = L(M2)}.

• Specify a language property. P(L)

P(M1?, M2?) = True if L(M1) = L(M2)
False otherwise
• Show that the domain of P is the set of recursively enumerable languages.
??? The domain of P is RE × RE

Lecture Notes 26 Undecidability 8

 Given a Turing Machine M, is L(M) Regular (or Context Free or Recursive)?

Is this problem decidable?

No, by Rice’s Theorem, since being regular (or context free or recursive) is a nontrivial property of the recursively enumerable
languages.

We can also show this directly (via the same technique we used to prove the more general claim contained in Rice’s Theorem):

Given a Turing Machine M, is L(M) Regular (or Context Free or Recursive)?

L1 = H = {s = "M" "w" : Turing machine M halts on input string w}

ß τ
(?M2) L2 = {s = "M" : L(M) is regular}

Let τ be the function that, from "M" and "w", constructs "M*", whose own input is a string
t = "M*" "w*"
M*("M*" "w*") operates as follows:
1. Copy its input to another track for later.
2. Write w on its input tape and execute M on w.
3. If M halts, invoke U on "M*" "w*".
4. If U halts, halt and accept.
If M2 exists, then ¬M2(M*(s)) decides L1 (H).

Why?
If M does not halt on w, then M* accepts ∅ (which is regular).
If M does halt on w, then M* accepts H (which is not regular).

Undecidable Problems About Unrestricted Grammars

• Given a grammar G and a string w, is w ∈ L(G)?

• Given a grammar G, is ε ∈ L(G)?
• Given two grammars G1 and G2, is L(G1) = L(G2)?
• Given a grammar G, is L(G) = ∅?

Given a Grammar G and a String w, Is w ∈ L(G)?

L1 =H= {s = "M" "w" : Turing machine M halts on input string w}

ß τ

(?M2) L2 = {s = "G" "w" : w ∈ L(G)}

Let τ be the construction that builds a grammar G for the language L that is semidecided by M. Thus
w ∈ L(G) iff M(w) halts.

Then τ("M" "w") = "G" "w"

If M2 exists, then M1 = M2(Mτ(s)) decides L1.

Lecture Notes 26 Undecidability 9

 Undecidable Problems About Context-Free Grammars

• Given a context-free grammar G, is L(G) = Σ*?

• Given two context-free grammars G1 and G2, is L(G1) = L(G2)?
• Given two context-free grammars G1 and G2, is L(G1) ∩ L(G2) = ∅?
• Is context-free grammar, G ambiguous?
• Given two pushdown automata M1 and M2, do they accept precisely the same language?
• Given a pushdown automaton M, find an equivalent pushdown automaton with as few states as possible.

Given Two Context-Free Grammars G1 and G2, Is L(G1) = L(G2)?

L1 = {s = "G" a CFG G and L(G) = Σ*}

ß τ

(?M2) L2 = {s = "G1" "G2" : G1 and G2 are CFGs and L(G1) = L(G2)}

Let τ append the description of a context free grammar GΣ* that generates Σ*.

Then, τ("G") = "G" "GΣ*"

If M2 exists, then M1 = M2(Mτ(s)) decides L1.

Non-RE Languages
There are an uncountable number of non-RE languages, but only a countably infinite number of TM’s (hence RE languages).
∴The class of non-RE languages is much bigger than that of RE languages!

Intuition: Non-RE languages usually involve either infinite search or knowing a TM will infinite loop to accept a string.

{áMñ: M is a TM that does not halt on the empty tape}

{áMñ: M is a TM and L(M) = Σ*}
{áMñ: M is a TM and there does not exist a string on which M halts}

Proving Languages are not RE

§ Diagonalization
§ Complement RE, not recursive
§ Reduction from a non-RE language
§ Rice’s theorem for non-RE languages (not covered)

Diagonalization

L={áMñ: M is a TM and M(áMñ) does not halt} is not RE

Suppose L is RE. There is a TM M* that semidecides L. Is áM*ñ in L?

• If it is, then M*(áM*ñ) halts (by the definition of M* as a semideciding machine for L)
• But, by the definition of L, if áM*ñ ∈ L, then M*(áM*ñ) does not halt.
Contradiction. So L is not RE.

(This is a very “bare-bones” diagonalization proof.)

Diagonalization can only be easily applied to a few non-RE languages.

Lecture Notes 26 Undecidability 10

 Complement of an RE, but not Recursive Language

Example: H = {áM, wñ: M does not accept w}

Consider H = {áM, wñ: M is a TM that accepts w}:
§ H is RE—it is semidecided by U, the Universal Turing Machine.
§ H is not recursive—it is equivalent to the halting problem, which is undecidable.
From the theorem, H is not RE.

Reductions and RE Languages

Theorem: If there is a reduction from L1 to L2 and L2 is RE, then L1 is RE.

x
M1 x ∈ L1?

y = M2 halt halt
τ y ∈ L2?
τ(x)

Theorem: If there is a reduction from L1 to L2 and L1 is not RE, then L2 is not RE.

Reduction from a known non-RE Language

Using a reduction from a non-RE language:

L1 = H = {áM, wñ: Turing machine M does not halt on input string w}

ß τ

(?M2) L2 = {áMñ: there does not exist a string on which Turing machine M halts}

Let τ be the function that, from áMñ and áwñ, constructs áM*ñ, which operates as follows:
1. Erase the input tape (M* ignores its input).
2. Write w on the tape
3. Run M on w.

áM, wñ

M1
áM*ñ halt halt
τ M2

M*
x w halt halt
M

áM, wñ ∈ H Þ M does not halt on w Þ M* does not halt on any input Þ M* halts on nothing Þ M2 accepts (halts).
áM, wñ ∉ H Þ M halts on w Þ M* halts on everything Þ M2 loops.

If M2 exists, then M1(áM, wñ) = M2(Mτ(áM, wñ)) and M1 semidecides L1. Contradiction. L1 is not RE. ∴ L2 is not RE.

Lecture Notes 26 Undecidability 11

 Language
Summary

IN OUT

Semidecidable Recursively
Enumerable Enumerable
Unrestricted grammar

Decision procedure Recursive Diagonalization

Lexicicographically enumerable Reduction
Complement is recursively enumer.

CF grammar Context Free Pumping

PDA Closure
Closure

Regular expression Regular Pumping

FSM Closure
Closure

Lecture Notes 26 Undecidability 12

 Introduction to Complexity Theory
Read K & S Chapter 6.

Most computational problems you will face your life are solvable (decidable). We have yet to address whether a problem is
“easy” or “hard”. Complexity theory tries to answer this question.

Recall that a computational problem can be recast as a language recognition problem.

Some “easy” problems:

§ Pattern matching
§ Parsing
§ Database operations (select, join, etc.)
§ Sorting

Some “hard” problems:

§ Traveling salesman problem
§ Boolean satisfiability
§ Knapsack problem
§ Optimal flight scheduling

“Hard” problems usually involve the examination of a large search space.

Big-O Notation

§ Gives a quick-and-dirty measure of function size

§ Used for time and space metrics

A function f(n) is O(g(n)) whenever there exists a constant c, such that |f(n)| ≤ c⋅|g(n)| for all n ≥ 0.

(We are usually most interested in the “smallest” and “simplest” function, g.)

Examples:
2n3 + 3n2⋅log(n) + 75n2 + 7n + 2000 is O(n3)
75⋅2n + 200n5 + 10000 is O(2n)

A function f(n) is polynomial if f(n) is O(p(n)) for some polynomial function p.

If a function f(n) is not polynomial, it is considered to be exponential, whether or not it is O of some exponential function
(e.g. n log n).

In the above two examples, the first is polynomial and the second is exponential.

Comparison of Time Complexities

Speed of various time complexities for different values of n, taken to be a measure of problem size. (Assumes 1 step per
microsecond.)
f(n)\n 10 20 30 40 50 60
n .00001 sec. .00002 sec. .00003 sec. .00004 sec. .00005 sec. .00006 sec.
n2 .0001 sec. .0004 sec. .0009 sec. .0016 sec. .0025 sec. .0036 sec.
n3 .001 sec. .008 sec. .027 sec. .064 sec. .125 sec. .216 sec.
n5 .1 sec. 3.2 sec. 24.3 sec. 1.7 min. 5.2 min. 13.0 min.
2n .001 sec. 1.0 sec. 17.9 min. 12.7 days 35.7 yr. 366 cent.
3n .059 sec. 58 min. 6.5 yr. 3855 cent. 2x108 cent. 1.3x1013 cent.

Faster computers don’t really help. Even taking into account Moore’s Law, algorithms with exponential time complexity are
considered intractable. ∴Polynomial time complexities are strongly desired.

Lecture Notes 27 Complexity Theory 1

 Polynomial Land

If f1(n) and f2(n) are polynomials, then so are:

§ f1(n) + f2(n)
§ f1(n) ⋅ f2(n)
§ f1(f2(n))

This means that we can sequence and compose polynomial-time algorithms with the resulting algorithms remaining polynomial-
time.
Computational Model

For formally describing the time (and space) complexities of algorithms, we will use our old friend, the deciding TM (decision
procedure).

There are two parts:

§ The problem to be solved must be translated into an equivalent language recognition problem.
§ A TM to solve the language recognition problem takes an encoded instance of the problem (of size n symbols) as input
and decides the instance in at most TM(n) steps.

We will classify the time complexity of an algorithm (TM) to solve it by its big-O bound on TM(n).

We are most interested in polynomial time complexity algorithms for various types of problems.

Encoding a Problem

Traveling Salesman Problem: Given a set of cities and the distances between them, what is the minimum distance tour a
salesman can make that covers all cities and returns him to his starting city?

Stated as a decision question over graphs: Given a graph G = (V, E), a positive distance function for each edge d: E→N+, and a
bound B, is there a circuit that covers all V where Σd(e) ≤ B? (Here a minimization problem was turned into a bound problem.)

A possible encoding the problem:

§ Give |V| as an integer.
§ Give B as an integer.
§ Enumerate all (v1, v2, d) as a list of triplets of integers (this gives both E and d).
§ All integers are expressed as Boolean numbers.
§ Separate these entries with commas.

Note that the sizes of most “reasonable” problem encodings are polynomially related.

What about Turing Machine Extensions?

Most TM extensions are can be simulated by a standard TM in a time polynomially related to the time of the extended machine.

§ k-tape TM can be simulated in O(T2(n))

§ Random Access Machine can be simulated in O(T3(n))

(Real programming languages can be polynomially related to the RAM.)

BUT… The nondeterminism TM extension is different.

A nondeterministic TM can be simulated by a standard TM in O(2p(n)) for some polynomial p(n).

Some faster simulation method might be possible, but we don’t know it.

Recall that a nondeterministic TM can use a “guess and test” approach, which is computationally efficient at the expense of
many parallel instances.

Lecture Notes 27 Complexity Theory 2

 The Class P

P = { L : there is a polynomial-time deterministic TM, M that decides L }

Roughly speaking, P is the class of problems that can be solved by deterministic algorithms in a time that is polynomially related
to the size of the respective problem instance.

The way the problem is encoded or the computational abilities of the machine carrying out the algorithm are not very important.

Example: Given an integer n, is there a positive integer m, such that n = 4m?

Problems in P are considered tractable or “easy”.

The Class NP

NP = { L: there is a polynomial time nondeterministic TM, M that decides L }

Roughly speaking, NP is the class of problems that can be solved by nondeterministic algorithms in a time that is polynomially
related to the size of the respective problem instance.

Many problems in NP are considered “intractable” or “hard”.

Examples:
§ Traveling salesman problem: Given a graph G = (V, E), a positive distance function for each edge d: E→N+, and a
bound B, is there a circuit that covers all V where Σd(e) ≤ B?
§ Subgraph isomorphism problem: Given two graphs G1 and G2, does G1 contain a subgraph isomorphic to G2?

The Relationship of P and NP

Recursive

NP

We’re considering only solvable (decidable) problems.

Clearly P ⊆ NP.

P is closed under complement.

NP probably isn’t closed under complement. Why?

Whether P = NP is considered computer science’s greatest unsolved problem.

Lecture Notes 27 Complexity Theory 3

 Why NP is so Interesting

§ To date, nearly all decidable problems with polynomial bounds on the size of the solution are in this class.

§ Most NP problems have simple nondeterministic solutions.

§ The hardest problems in NP have exponential deterministic time complexities.

§ Nondeterminism doesn’t influence decidability, so maybe it shouldn’t have a big impact on complexity.

§ Showing that P = NP would dramatically change the computational power of our algorithms.

Stephen Cook’s Contribution (1971)

§ Emphasized the importance of polynomial time reducibility.

§ Pointed out the importance of NP.

§ Showed that the Boolean Satisfiability (SAT) problem has the property that every other NP problem can be
polynomially reduced to it. Thus, SAT can be considered the hardest problem in NP.

§ Suggested that other NP problems may also be among the “hardest problems in NP”.

This “hardest problems in NP” class is called the class of “NP-complete” problems.

Further, if any of these NP-complete problems can be solved in deterministic polynomial time, they all can and, by implication,
P = NP.

Nearly all of complexity theory relies on the assumption that P ≠ NP.

Polynomial Time Reducibility

A language L1 is polynomial time reducible to L2 if there is a polynomial-time recursive function τ such that ∀x ∈ L1 iff τ(x) ∈
L2.

If L1 is polynomial time reducible to L2, we say L1 reduces to L2 (“polynomial time” is assumed) and we write it as L1 ∝ L2.

Lemma: If L1 ∝ L2, then (L2 ∈ P) Þ (L1 ∈ P). And conversely, (L1 ∉ P) Þ (L2 ∉ P).

Lemma: If L1 ∝ L2 and L2 ∝ L3 then L1 ∝ L3.

L1 and L2 are polynomially equivalent whenever both L1 ∝ L2 and L2 ∝ L1.

Polynomially equivalent languages form an equivalence class. The partitions of this equivalence class are related by the partial
order ∝.
P is the “least” element in this partial order.

What is the “maximal” element in the partial order?

Lecture Notes 27 Complexity Theory 4

 The Class NP-Complete

A language L is NP-complete if L ∈ NP and for all other languages L’ ∈ NP, L’ ∝ L.

NP-Complete problems are the “hardest” problems in NP.

Lemma: If L1 and L2 belong to NP, L1 is NP-complete and L1 ∝ L2, then L2 is NP-complete.

Thus to prove a language L2 is NP-complete, you must do the following:

1. Show that L2 ∈ NP.
2. Select a known NP-complete language L1.
3. Construct a reduction τ from L1 to L2.
4. Show that τ is polynomial-time function.

M1
w τ(w) y
τ M2
n

How do we get started? Is there a language that is NP-complete?

Boolean Satisfiability (SAT)

Given a set of Boolean variables U = {u1, u2, …, um} and a Boolean expression in conjunctive normal form (conjunctions of
clauses—disjunctions of variables or their negatives), is there a truth assignment to U that makes the Boolean expression true
(satisfies the expression)?

Note: All Boolean expressions can be converted to conjunctive normal form.

Example: (x1∨ ¬x2 ∨ x3) ∧ (¬x3 ∨ x4 ∨ ¬x2)

Cook’s Theorem: SAT is NP-complete.

1. Clearly SAT ∈ NP.
2. The proof constructs a complex Boolean expression that satisfied exactly when a NDTM accepts an input string x
where |w| = n. Because the NDTM is in NP, its running time is O(p(n)). The number of variables is polynomially
related to p(n).

SAT is NP-complete because SAT ∈ NP and for all other languages L’ ∈ NP, L’ ∝ SAT.

Reduction Roadmap

SAT

3SAT

3DM VC

PARTITION HC CLIQUE
The early NP-complete reductions took this structure. Each phrase represents a problem. The arrow represents a reduction from
one problem to another.

Today, thousands of diverse problems have been shown to be NP-complete.

Let’s now look at these problems.

Lecture Notes 27 Complexity Theory 5

 3SAT (3-satisfiability)

Boolean satisfiability where each clause has exactly 3 terms.

3DM (3-Dimensional Matching)

Consider a set M ⊆ X × Y × Z of disjoint sets, X, Y, & Z, such that |X| = |Y| = |Z| = q. Does there exist a matching, a subset
M’⊆ M such that |M’| = q and M’ partitions X, Y, and Z?

This is a generalization of the marriage problem, which has two sets men & women and a relation describing acceptable
marriages. Is there a pairing that marries everyone acceptably?

The marriage problem is in P, but this “3-sex version” of the problem is NP-complete.

PARTITION

Given a set A and a positive integer size, s(a) ∈ N+, for each element, a ∈ A. Is there a subset A’ ⊆ A such that

Σ s(a) = Σ s(a) ?
a∈A’ a∈A-A’

VC (Vertex Cover)

Given a graph G = (V, E) and an integer K, such that 0 < K ≤ |V|, is there a vertex cover of size K or less for G, that is, a subset
V’ ⊆ V such that |V’| ≤ K and for each edge, (u, v) ∈ E, at least one of u and v belongs to V’?

CLIQUE

Given a graph G = (V, E) and an integer J, such that

0 < J ≤ |V|, does G contain a clique of size J or more, that is a subset V’ ⊆ V such that |V’| ≥ J and every two vertices in V’ are
joined by an edge in E?

HC (Hamiltononian Circuit)

Given a graph G = (V, E), does there exist a Hamiltonian circuit, that is an ordering <v1, v2, …, vn> of all V such that
(v|V|, v1) ∈ E and (vi, vi+1) ∈ E for all i, 1 ≤ i < |V|?

Traveling Salesman Prob. is NP-complete

Given a graph G = (V, E), a positive distance function for each edge d: E→N+, and a bound B, is there a circuit that covers all V
where Σd(e) ≤ B?

To prove a language TSP is NP-complete, you must do the following:

1. Show that TSP ∈ NP.
2. Select a known NP-complete language L1.
3. Construct a reduction τ from L1 to TSP.
4. Show that τ is polynomial-time function.

TSP ∈ NP: Guess a set of roads. Verify that the roads form a tour that hits all cities. Answer “yes” if the guess is a tour and the
sum of the distances is ≤ B.

Reduction from HC: Answer the Hamiltonian circuit question on G = (V, E) by constructing a complete graph where “roads”
have distance 1 if the edge is in E and 2 otherwise. Pose the TSP problem, is there a tour of length ≤ |V|?

Lecture Notes 27 Complexity Theory 6

 Notes on NP-complete Proofs

The more NP-complete problems are known, the easier it is to find a NP-complete problem to reduce from.

Most reductions are somewhat complex.

It is sufficient to show that a restricted version of the problem is NP-complete.

More Theory

NP has a rich structure that includes more than just P and NP-complete. This structure is studied in later courses on the theory of
computation.

The set of recursive problems outside of NP (and including NP-complete) are called NP-hard. There is a proof technique to
show that such problems are at least as hard as NP-complete problems.

Space complexity addresses how much tape does a TM use in deciding a language. There is a rich set of theories surrounding
space complexity.

Recursive
NP-hard

NP

NP-complete
(part of NP-hard)

Dealing with NP-completeness

You will likely run into NP-complete problems in your career. For example, most optimization problems are NP-complete.

Some techniques for dealing with intractable problems:

§ Recognize when there is a tractable special case of the general problem.
§ Use other techniques to limit the search space.
§ For optimization problems, seek a near-optimal solution.

The field of linear optimization springs out of the latter approach. Some linear optimization solutions can be proven to be “near”
optimal.

A branch of complexity theory deals with solving problems within some error bound or probability.

For more: Read Computers and Intractability: A Guide to the Theory of NP-Completeness by Michael R. Garey and David S.
Johnson, 1979.

Lecture Notes 27 Complexity Theory 7