COMPUTER SCIENCE

S E D G E W I C K / W A Y N E

18. Turing Machines

Section 7.4

http://introcs.cs.princeton.edu

Universality and computability Context: Mathematics and logic

Principia Mathematics
Fundamental questions Mathematics. Any formal system powerful enough to express arithmetic. Peano arithmetic
• What is a general-purpose computer? Zermelo-Fraenkel set theory
.
• Are there limits on the power of digital computers? .
.
• Are there limits on the power of machines we can build? Complete. Can prove truth or falsity of any arithmetic statement.
Consistent. Cannot prove contradictions like 2 + 2 = 5.
Decidable. An algorithm exists to determine truth of every statement.
Pioneering work at Princeton in the 1930s.

Q. (Hilbert, 1900) Is mathematics complete and consistent?

A. (Gödel's Incompleteness Theorem, 1931) NO (!!!)

David Hilbert Kurt Gödel Alsonzo Church Alan Turing Q. (Hilbert's Entscheidungsproblem) Is mathematics decidable?
1862 1943 1906 1978 1903 1995 1912 1954

Solved the math Provided THE answers

A. (Church 1936, Turing 1936) NO (!!)
Asked the questions Solved the decision
problem problem
3 4
 COMPUTER SCIENCE Starting point
S E D G E W I C K / W A Y N E

Goals
• Develop a model of computation that encompasses all known computational processes.
• Make the model as simple as possible.

Example: A familiar computational process.

18. Turing Machines
• A simple model of computation
1 0 1 0
• Universality
• Computability 3 1 4 2
Characteristics
• Implications 7 1 8 2
• Discrete.
1 0 3 2 4 • Local.
http://introcs.cs.princeton.edu • States.

Previous lecture: DFAs This lecture: Turing machines

A DFA is an abstract machine that solves a pattern matching problem. A Turing machine (TM) is an abstract model of computation.
• A string is specified on an input tape (no limit on its length). • A string is specified on a tape (no limit on its length).
• The DFA reads each character on input tape once, moving left to right. • The TM reads and writes characters on the tape, moving left or right.
• The DFA lights "YES" if it recognizes the string, "NO" otherwise. • The TM lights "YES" if it recognizes the string, "NO" otherwise.
Each DFA defines a set of strings (all the strings that it recognizes). • The TM may halt, leaving the result of the computation on the tape.

NO NO

HALT

YES YES

b b a a b b a b b ... ...

7 8
Previous lecture: DFA details and example This lecture: Turing Machine details and example

A DFA is an abstract machine with a finite number of states, each labelled Y or N and A Turing Machine is an abstract machine with a finite number of states, each labelled Y, N,
transitions between states, each labelled with a symbol. One state is the start state. H, L, or R and transitions between states, each labelled with a read/write pair of symbols.
• Begin in the start state. • Begin in the designated start state.
• Read an input symbol and move to the indicated state. • Read an input symbol, move to the indicated state and write the indicated output.
• Repeat until the last input symbol has been read. • Move tape head left if new state is labelled L, right if it is labelled R.
• Turn on the "YES" or "NO" light according to the label on the current state. • Repeat until entering a state labelled Y, N, or H ( and turn on associated light).

NO NO
1:0 #:1
a a a
Does this DFA recognize
Y b N b N this string? R #:# L H HALT

b 0:1
YES YES

b b a a b b a b b # # # 1 0 1 1 0 0 1 1 1 # # #
9 10

DFAs vs TMs TM example 1: Binary decrementer

Similarities
• Simple model of computation.
• Input on tape is a finite string with symbols from a finite alphabet.
NO
• Finite number of states.
0:1
• State transitions determined by current state and input symbol.
1:0
R #:# L H HALT

Differences
YES

DFAs TMs # # # 0
1 0 1 0 1 0 0 0 0 # # #
• Can read input symbols from the tape. • Can read from or write onto the tape.
• Can only move tape head to the right. • Can move tape head either direction.
• Tape is finite (a string). • Tape does not end (either direction).
Input 1 0 1 0 1 0 0 0 0
• One state transition per input symbol. • No limit on number of transitions.
• Can recognize (turn on "YES" or "NO"). • Can also compute (with output on tape). Output 1 0 1 0 0 1 1 1 1
11 12
TM example 1: Binary decrementer TM example 2: Binary incrementer

Q. What happens when we try to decrement 0? Note: This adds a 1 at the left as the
last step when incrementing 111...1

NO NO
0:1 #:# 1:0 #:1

R #:# L 1:0 H HALT R #:# L H HALT

0:1
YES YES

# # # 1
# # # # # 0 0 0 0 # # # # # # 1
0 0 1 0 0 1 1 1 1 # # #

A. Doesn't halt! TMs can have bugs, too.

Input 1 0 1 0 0 1 1 1 1
Fix to avoid infinite loop. Check for #. Output 1 0 1 0 1 0 0 0 0
13 14

TM example 3: Binary adder (method) TM example 3: Binary adder

# 1 0 1 1 + 1 0 1 0 # NO
To compute x + y Find +
0:1
# 1 0 1 1 + 1 0 0 1 # Decrement y L 1:0 L HALT
• Move right to right end of y. ... # # 1 0 1 1 + 1 0 1 0 # # ...
+:# +:+
1:# Halt
# 1 0 1 1 + 1 0 0 1 # YES
• Decrement y. ... # # 1 0 1 1 + 1 0 0 1 # # ...
R #:# H
# 1 1 0 0 + 1 0 0 1 # Clean Up
• Move left to right end of x (left of +) . ... # # 1 0 1 1 + 1 0 0 1 # # ... #:#
0:1
... Increment x
• Increment x. ... # # 1 1 0 0 + 1 0 0 1 # # ... Find right end R L
#:1
1 0 1 0 1 + 1 1 1 1 # 1:0
• Continue until y = 0 is decremented. ... # 1 0 1 0 1 + 1 1 1 1 # # ...
Found + when seeking 1? Just decremented 0.
1 0 1 0 1 # # # # # #
• Clean up by erasing + and 1s. ... # 1 0 1 0 1 # # # # # # # ...
# # # 1 0 1 1 + 1 0 1 0 # # #
Clean up

15 16
Simulating an infinite tape with two stacks Simulating the operation of a Turing machine

public class TM fixes bug

Q. How can we simulate a tape that is infinite on both ends? { 0:1 #:#
private int state;
A. Use two stacks, one for each end. private int start; 0
R #:#
1
L 1:0
2
H
private String[] action;
private ST<Character, Integer>[] next;
private ST<Character, Character>[] out; action[] next[] out[]
private Stack<Character> left;
private Stack<Character> right; /* Stack code from previous slide */
0 1 # 0 1 #
private char read() # # # 1 0 1 1 + 1 0 1 0 # # # public TM(In in)
{ /* Fill in data structures */ } 0 R 0 0 0 1 0 0 1 #
{
if (right.isEmpty()) return '#'; public String simulate(String input) 1 L 1 1 2 2 1 1 0 #
return right.pop(); "tape head" is top of right stack {
} state = start; 2 H 2 2 2 2 2 0 1 #
assumes
write just for (int i = input.length()-1; i >= 0; i--) entries in gray are implicit in graphical representation
private char write(char c)
1 right.push(input.charAt(i);
after each
{ right.push(c); } read 1 move 1 while (action(state).equals("L") ||
right action(state).equals("R")) % more dec.txt
private void moveRight() + + { 3 01# 0
{ char c = read();
1 1 state = next[state].get(c);
R 0 0 1 0 1 #
char c = '#'; move L 1 2 2 1 0 #
if (!right.isEmpty()) c = right.pop(); 0 1 0 write(out[state].get(c));
H 2 2 2 0 1 #
left.push(c);
left if (action[state].equals("R") moveRight();
0 0 0 1 if (action[state].equals("L") moveLeft(); % java TM dec.txt
}
} 000111
1 1 1 0 return action[state];
private void moveLeft() 000110
{ # # # # } 010000
char c = '#'; public static void main(String[] args) 001111
if (!left.isEmpty()) c = left.pop();
empty? assume # is there { /* Similar to DFA's main() */ } 000000
}
right.push(c); 111111
}
17 18

COMPUTER SCIENCE COMPUTER SCIENCE

S E D G E W I C K / W A Y N E S E D G E W I C K / W A Y N E

18. Turing Machines 18. Turing Machines

• A simple model of computation • A simple model of computation
• Universality • Universality
• Computability • Computability
• Implications • Implications

http://introcs.cs.princeton.edu http://introcs.cs.princeton.edu
Representing a Turing machine Universal Turing machine (UTM)

NO
Turing's key insight. A TM is nothing more than a finite sequence of symbols. Universal Turing machine. A TM that takes as input
any TM and input for that TM on a TM tape. UTM HALT

dec.txt YES

decrementer TM
0:1 #:# 3 01# 0 ... # # # 1 1 0 0 0 3 0 1 # 0 R 0 0 1 0 1 $
R 0 0 1 0 1 #
R #:# L 1:0 H L 1 2 2 1 0 #
0 1 2 H 2 2 2 0 1 # input to decrementer TM decrementer TM

Result. Whatever would happen if that TM were to

NO

Implication. Can put a TM and its input on a TM tape. run with that input (could loop or end in Y, N or H). UTM HALT

YES

... # # # 1 0 1 1 1 # # #
1 1 0 0 0 3 0 1 # 0 R 0 0 1 0 1 # L 1 2 1 1 0 # H 2 2 2 0 1 #

result that decrementer TM would produce

Turing. Simulating a TM is a simple computational task, so there exists a TM to do it: A UTM.

Profound implication. We can use a TM to simulate the operation of any TM.
Easier for us to think about. Implement Java simulator as a TM.
21 22

Implementing a universal Turing machine Universality

Note: This booksite UTM uses a
Java simulator gives a roadmap transition-based TM representation UTM: A simple and universal model of computation.
that is easier to simulate than the
• No need for constructor because everything state-based one used in this lecture.
is already on the tape. Definition. A task is computable if a Turing machine exists that computes it.
A 24-state UTM
• Simulating the infinite tape is a bit easier
NO
because TM has an infinite tape. Theorem (Turing, 1936). It is possible to invent a single UTM HALT

• Critical part of the calculation is to update machine which can be used to do any computable task. YES

state as indicated. ... ...

Profound implications
• Any machine that can simulate a TM can simulate a UTM.
Want to see the details or build your own TM? • Any machine that can simulate a TM can do any computable task.
Use the booksite's TM development environment.

Warning. TM development may be addictive. Amazed that it's only 24 states?

The record is 4 states, 6 symbols.
23 24
A profound connection to the real world Evidence in favor of the Church-Turing thesis

Church-Turing thesis. Turing machines can do anything that can be described by any Evidence. Many, many models of computation have turned out to be equivalent (universal).
physically harnessable process of this universe: All computational devices are equivalent.
model of computation description

enhanced Turing machines multiple heads, multiple tapes, 2D tape, nondeterminism

Remarks New model of computation or new physical process? untyped lambda calculus method to define and manipulate functions
• A thesis, not a theorem. • Use simulation to prove equivalence.
recursive functions functions dealing with computation on integers
• Not subject to proof. • Example: TOY simulator in Java.
unrestricted grammars iterative string replacement rules used by linguists
• Is subject to falsification. • Example: Java compiler in TOY.
extended Lindenmayer systems parallel string replacement rules that model plant growth
programming languages Java, C, C++, Perl, Python, PHP, Lisp, PostScript, Excel
random access machines registers plus main memory, e.g., TOY, Pentium

= cellular automata cells which change state based on local interactions

Implications
quantum computer compute using superposition of quantum states
• No need to seek more powerful machines or languages.
DNA computer compute using biological operations on DNA
NO

• Enables rigorous study of computation (in this universe).

=
HALT

YES
PCP systems string matching puzzles (stay tuned)

... ...

8 decades without a counterexample, and counting.

25 26

Example of a universal model: Extended Lindenmayer systems for synthetic plants COMPUTER SCIENCE
S E D G E W I C K / W A Y N E

18. Turing Machines

• A simple model of computation
• Universality
• Computability
• Implications

http://introcs.cs.princeton.edu

http://astronomy.swin.edu.au/~pbourke/modelling/plants
27
 COMPUTER SCIENCE Post's correspondence problem (PCP)
S E D G E W I C K / W A Y N E

PCP. A family of puzzles, each based on a set of cards.

• N types of cards.
• No limit on the number of cards of each type.
• Each card has a top string and bottom string.
Does there exist an arrangement of cards with matching top and bottom strings?

18. Turing Machines

• A simple model of computation
BAB A AB BA
• Universality Example 1 (N = 4).
A ABA B B
• Computability
0 1 2 3
• Implications
A BA BAB AB A
Solution 1 (easy): YES.
http://introcs.cs.princeton.edu ABA B A B ABA
1 3 0 2 1 30

Post's correspondence problem (PCP) Post's correspondence problem (PCP)

PCP. A family of puzzles, each based on a set of cards. PCP. A family of puzzles, each based on a set of cards.
• N types of cards. • N types of cards.
• No limit on the number of cards of each type. • No limit on the number of cards of each type.
• Each card has a top string and bottom string. • Each card has a top string and bottom string.
Does there exist an arrangement of cards with matching top and bottom strings? Does there exist an arrangement of cards with matching top and bottom strings?

Example 3 (created by Andrew Appel).

BAB A AB BA S[ X BAB 11A 1 [A ] [ B1 B] [1A]E

Example 2 (N = 4).
A BAB B A S[11111X][ 1X A A1 1 [B ] [ 1B A] E
0 1 2 3 0 1 2 3 4 5 6 7 8 9 10

Solution 2 (easy): NO. No way to match even the first character! Challenge for the bored: Find a solution that starts with a card of type 0.

31 32
Post's correspondence problem (PCP) Another impossible problem

PCP. A family of puzzles, each based on a set of cards. Halting problem. Write a Java program that reads in code for Java static method f()
• N types of cards. and an input x, and decides whether or not f(x) results in an infinite loop.
• No limit on the number of cards of each type.
• Each card has a top string and bottom string.
Does there exist an arrangement of cards with matching top and bottom strings? Example 1 (easy). Example 2 (difficulty unknown).
public void f(int x) public void f(int x)
{ {
while (x != 1) while (x != 1)
Involves Collatz conjecture
{ {
(see Recursion lecture)
... if (x % 2 == 0) x = x / 2; if (x % 2 == 0) x = x / 2;
else x = 2*x + 1; else x = 3*x + 1;
} }
} }
0 1 2 3 4 N

f(7): 7 22 11 34 17 52 26 13 40 20 10 5 16 8 4 2 1
Halts only if x is a positive power of 2 f(-17): -17 -50 -25 -74 -37 -110 -55 -164 -82 -41 -122 ... -17 ...
A reasonable idea. Write a program to take N card types as input and solve PCP.

A surprising fact. It is not possible to write such a program. Next. A proof that it is not possible to write such a program.

33 34

Undecidability of the halting problem Warmup: self-referential statements

Definition. A yes-no problem is undecidable if no Turing machine exists to solve it. Liar paradox (dates back to ancient Greek philosophers).
(A problem is computable if there does exist a Turing machine that solves it.) • Divide all statements into two categories: true and false.
• Consider the statement "This statement is false."
• Is it true? If so, then it is false, a contradiction.
• Is it false? If so, then it is true, a contradiction.
Theorem (Turing, 1936). The halting problem is undecidable. Logical conclusion. Cannot label all statements as true or false.

Source of the difficulty: Self-reference.

Profound implications
• There exists a problem that no Turing machine can solve.
• There exists a problem that no computer can solve.
• There exist many problems that no computer can solve (stay tuned).

35 36
 37 38

Proof of the undecidability of the halting problem Proof of the undecidability of the halting problem

Theorem (Turing, 1936). The halting problem is undecidable. Theorem (Turing, 1936). The halting problem is undecidable.

Proof outline. Proof. Solution to the problem

• Assume the existence of a function halt(f, x) that solves the problem. • Assume the existence of a function halt(f, x) public boolean halt(String f, String x)
{
that solves the problem. if ( /* f(x) halts */ ) return true;
public boolean halt(String f, String x) else return false;
{ • Create a function strange(f) that goes into an }
By universality, may as well use Java.
if ( /* something terribly clever */ ) return true;
(If this exists, we could simulate it on a TM.)
infinite loop if f(f) halts and halts otherwise.
else return false;
} • Call strange() with itself as argument. A client
public void strange(String f)
• If strange(strange) halts, then {
• Arguments: A function f and input x, encoded as strings. strange(strange) goes into an infinite loop. if (halt(f, f))
while (true) { } // infinite loop
• Return value: true if f(x) halts and false if f(x) does not halt. • If strange(strange) does not halt, then }

• Always halts. strange(strange) halts.

• Proof idea: Reductio ad absurdum: if any logical argument based on an • Reductio ad absurdum. A contradiction
halts?
assumption leads to an absurd statement, then the assumption is false. strange(strange)
• halt(f,x) cannot exist. does not halt?

39 40
 COMPUTER SCIENCE COMPUTER SCIENCE
S E D G E W I C K / W A Y N E S E D G E W I C K / W A Y N E

18. Turing Machines 18. Turing Machines

• A simple model of computation • A simple model of computation
• Universality • Universality
• Computability • Computability
• Implications • Implications

http://introcs.cs.princeton.edu http://introcs.cs.princeton.edu

Implications of undecidability Implications for programming systems

Primary implication. If you know that a problem is undecidable... Q. Why is debugging difficult?
A. All of the following are undecidable.
...don't try to solve it!
Hey, Alice. We came up with a
great idea at our hackathon.
We're going for startup funding. Halting problem. Give a function f, does it halt on a given input x?
What's the idea? Totality problem. Give a function f, does it halt on every input x?
No-input halting problem. Give a function f with no input, does it halt?
An app that you can use to
make sure that any app you Program equivalence. Do two functions f() and g() always return same value?
download won't hang your phone! Uninitialized variables. Is the variable x initialized before it's used? LE
AB
Dead-code elimination. Does this statement ever get executed? C ID
DE
Ummm. I think UN
that's undecidable.
Prove each by reduction to the halting problem: A solution would solve the halting problem.
?
Will your app
work on itself ? Q. Why are program development environments complicated?
??? A. They are programs that manipulate programs.
43 44
Another undecidable problem Another undecidable problem

The Entscheidungsproblem (Hilbert, 1928) "Decision problem"

• Given a first-order logic with a finite number of additional axioms.

E
• Is the statement provable from the axioms using the rules of logic?
DA
BL
I
D EC
UN David Hilbert
1862 1943

Lambda calculus E
BL
• DA
Formulated by Church in the 1930s to address the Entscheidungsproblem. E CI
U ND
• Also the basis of modern functional languages.

Alsonso Church
1903 1995

Theorem (Church and Turing, 1936). The Entscheidungsproblem is undecidable. Theorem (Post, 1946). Post's correspondence problem is undecidable.

45 46

Examples of undecidable problems from computational mathematics Examples of undecidable problems from computer science

Hilbert's 10th problem Ex. 1 ( , , )= +

Optimal data compression
• Given a multivariate polynomial f (x, y, z, ...). YES ( , , )=
• Find the shortest program to produce a given string.
• Does f have integral roots ? (Do there exist
integers x, y, z, such that f (x, y, z, ...) = 0 ? ) Ex. 2 ( , )= + NO • Find the shortest program to produce a given picture.
E
A BL
ID
DE
C LE
AB
UN C ID
DE
UN produced by a 34-line Java program

Definite integration Private Sub AutoOpen()

On Error Resume Next
cos( ) cos( ) If System.PrivateProfileString("", CURRENT_USER\Software

• Given a rational function f (x ) composed of Ex. 1 YES = Virus identification \Microsoft\Office\9.0\Word\Security",

+ + "Level") <> "" Then

CommandBars("Macro").Controls("Security...").Enabled = False
polynomial and trigonometric functions. • Is this code equivalent to this known virus?
. . .
For oo = 1 To AddyBook.AddressEntries.Count
Peep = AddyBook.AddressEntries(x)
BreakUmOffASlice.Recipients.Add Peep
x = x + 1
LE If x > 50 Then oo = AddyBook.AddressEntries.Count
• Does ( ) exist? AB cos( )
• Does this code contain a virus?
Next oo

C ID Ex. 2 NO E
. . .

DE BL
BreakUmOffASlice.Subject = "Important Message From " &

UN DA
Application.UserName

CI
BreakUmOffASlice.Body = "Here is that document you asked
E for ... don't show anyone else ;-)"

U ND . . .

47 Melissa virus (1999) 48

Turing's key ideas Alan Turing: the father of computer science

Turing's paper in the Proceedings of the London Mathematical Society

"On Computable Numbers, With an Application to the Entscheidungsproblem" It was not only a matter of abstract mathematics, not only a play of symbols,
was one of the most impactful scientific papers of the 20th century. for it involved thinking about what people did in the physical world…. It was a
play of imagination like that of Einstein or von Neumann, doubting the axioms
rather than measuring effects…. What he had done was to combine such a
Alan Turing
The Turing machine. A formal model of computation. 1912 1954 naïve mechanistic picture of the mind with the precise logic of pure
mathematics. His machines – soon to be called Turing machines – offered
Equivalence of programs and data. Encode both as strings and compute with both. a bridge, a connection, between abstract symbols and the physical world.

Universality. Concept of general-purpose programmable computers. — John Hodges, in Alan Turing, the Enigma

Church-Turing thesis. If it is computable at all, it is computable with a Turing machine.

Computability. There exist inherent limits to computation. NO

UTM
=
HALT

YES

Turing's paper was published in 1936, ten years before Eckert and Mauchly worked on ENIAC (!) ... ...

A Universal Turing Machine

Suggestion: Now go back and read the beginning
John von Neumann read the paper... of the lecture on von Neumann machines A Google data center
49 50

COMPUTER SCIENCE COMPUTER SCIENCE

S E D G E W I C K / W A Y N E S E D G E W I C K / W A Y N E

18. Turing Machines

• A simple model of computation
• Universality
• Computability
18. Turing Machines
• Implications
Section 7.4

http://introcs.cs.princeton.edu http://introcs.cs.princeton.edu