Turing Machines

The Next Level of Computation

• Turing Machines– From Alan Turing, 1936

• Turns out to be the LAST– The only machine you'll ever need

• The Church-Turing Thesis– All algorithms have a Turing Machine

equivalent• Some TMs are not algorithms, however

But First…

• We have shown that anbncn is not context free– By the Pumping Lemma

• What if we add an extra stack to a PDA?

A 2PDA

Trace “aabbcc”…

- q0 q1

+ q2

a^e/A,e/e

b^A/e,e/B

b^A/e,e/B

c^e/e,B/e

c^e/e,B/e

e^e/e,e/e

An Observation

• “an” is accepted by an FA

• “anbn” is accepted by a PDA

• “anbncn” is accepted by a 2PDA

• What about “anbncndn”?

A 2PDA for “anbncndn”

- q0 q1 q2

+ q3

a^e/A,e/e

b^A/e,e/B

b^A/e,e/B

c^e/C,B/e

c^e/C,B/e

e^e/e,e/e d^C/e,e/e

d^C/e,e/e

Trace “aabbccdd” …

Interesting Fact

• 2PDA = nPDA = TM = (many things…)

Perspective

• An FA is a machine with no memory• A PDA is like an FA but with unlimited, restricted

memory– Restricted because we access it LIFO

• A Turing Machine is like an FA but with unlimited, unrestricted memory– We can move to any memory cell

• By moving one cell at a time over and over• Can move back and forth

– We can both read and write there

Turing MachinesTake some states and add…

• A one-way-infinite, read/write “tape”– Initialized with input string at left; blanks everywhere

else– Machine can replace one character with another– Then it can move to the right or left

• Except it can't fall off the left end (“crash”)– Can contain instances of a special “blank” symbol (not

part of input alphabet)• A “program”

– A set of transitions, plus…– Specifies the character replacements and tape

movements

Example

• The following TM swaps a's and b's:– q3 is a “halt state”

-q0 q1 q2B,B,R

a,b,Rb,a,R

B,B,L

a,a,Lb,b,L

+q3B,B,R

NOTE: B == “blank”; input is preceded by B here

Trace “bab”…

Tracing a TM Execution

• BbabB (character about to be read)BbabBBaabBBabbBBabaBBabaBBabaBBabaBBabaBBabaB (halts pointing at 1st output character)

Turing Machines

• A set of states– At least one is a “halting” state– Program halts (and optionally accepts input)

• If it crashes, the input is rejected– May leave output on tape

• Therefore, can also compute functions

• An input alphabet• An output alphabet

– Usually contains input alphabet– And a “blank symbol” (not part of input)

• A Program

anbncn

-q0

q1 q5

q2 +q6

q3

q4

B,B,R

a,X,R

Y,Y,R

Y,Y,RZ,Z,R

B,B,RB,B,R

a,a,RY,Y,R

b,b,RZ,Z,R

b,Y,R

c,Z,L

a,a,Lb,b,LY,Y,LZ,Z,L

X,X,R

Trace “aabbcc”…

Tracing “aabbcc”

BaabbccBBaabbccBBXabbccBBXabbccBBXaYbccBBXaYbccBBXaYbZcBBXaYbZcBBXaYbZcBBXaYbZcBBXaYbZcBBXXYbZcBBXXYbZcB

BXXYYZcBBXXYYZcBBXXYYZZBBXXYYZZBBXXYYZZBBXXYYZZBBXXYYZZBBXXYYZZBBXXYYZZBBXXYYZZBBXXYYZZBBXXYYZZBB (halt)

Subroutines

• A TM can be used as a subroutine– Just place an arrow from the calling machine to the

start state of the subroutine machine• After tape is positioned at the string to be copied

– Then every halt state of the subroutine moves back to where you want it to (must copy/paste)

• See INSERT on page 451 (next slide)• See DELETE on pp. 452-453 (second slide)• Can also just chain TMs in a sequence

1

2

45

3

67

In

Out

(a,a,L)(b,b,L)(X,X,L)

(X,X,R)

(X,Q,R)

(a,Q,R)

(b,Q,R)

(a,b,R)

( ,b,R) (Q,b,R)

( ,a,R)

(a,X,R)

(X,a,R)

(a,a,R)

( ,b,R)

(b,b,R)

(b,a,R)

( ,X,R)

(b,X,R) (X,b,R)

( , ,L)

Insert

1 2 3In

(a,a,R)

( , ,L)

Delete( c, ,R)( b, ,R)

(b,b,R)(c,c,R)( a, ,R)

4 5

7

Out

(a,a,L) 6

(b,b,L)

( ,a,R)

( a, ,L) ( b, ,L) ( c, ,L)

(c,c,L)(b,a,L) (c,b,L)

(a,b,,L) (b,c,L)

(a,c,L)

(c,a,L)

( ,c,R)

COPY

Trace “ab”… (in class, right now)

-q0 q1

q2

q5

q3

q6

q4

q7

B,B,R

a,X,R

a,a,Rb,b,R

B,B,R

a,a,Rb,b,R

B,a,L

a,a,Lb,b,LB,B,L

b,Y,R

a,a,Rb,b,R

B,B,R

a,a,Rb,b,R

B,b,L

X,X,RY,Y,R

B,B,L

X,a,LY,b,L

+q8 B,B,R

Copying “ab”

BabB

BabB

BXbB

BXbB

BXbBB

BXbBa

BXbBa

BXbBa

BXbBa

BXYBa

BXYBa

BXYBaB

BXYBab

BXYBab

BXYBab

BXYBab

BXYBab

BXbBab

BabBab

BabBab (halt)

Mechanisms equivalent to Standard TMs

• PDAs with 2 or more stacks• TMs with 2-way infinite tapes• TMs with n tapes or n heads or n-dim tapes• TMs with various “move” options

– Move by a number of cells (including 0), random cell access by position

• Adding non-determinism to a Standard TM• Church's Lambda Calculus (LISP)• Unrestricted Phrase Structure Grammars• Matrix Grammars• Post Systems• Markov Algorithms• Structured Programming with unlimited memory• Any abstract machine yet imagined by man!

Interesting Fact About TMs

• They may not halt on every possible input!• And not just because the creator of a specific

TM was a doofus• As we shall see, there are languages that are

recognizable (if a valid string is input, the TM halts and accepts)– But not decidable (because if an invalid string is input,

it will not halt!)• This is the major mathematical/computational

discovery of the 20th century!– There are propositions that cannot be decided

(“proven”)

“Algorithms” that may not Halt

• Let g(x,y) be some computable function• Let f(x) =

the smallest p where g(x, p) = 1,0, if such a p does not exist

• “Pseudo-algorithm” for f(x):m = 0;while (g(x, m) != 1)

++m;cout << m;

• If there is no m for a given g and x, then we hang!

A language L over the alphabet is called recursive (aka “Turing computable” or “Turing decidable”) if there is a TM T that accepts every word in L and rejects every word in L'; that is,

accept(T) = L

reject(T) = L'

A TM that recognizes a recursive set is a model of an algorithm. It always halts.

Halting

• It is possible that TMs will not halt for certain inputs– Infinite loop

• Due to the nature of the language or computation• This is the price we pay for such a flexible computation

model as TMs

• 3 possibilities when a TM processes an input string from a non-recursive language:– Accepts (goes to a halt state)– Rejects (crashes)– Hangs (infinite loop)

A language L over the alphabet is called recursively enumerable (aka “Turing recognizable”) if there is a TM T that accepts every word in L and either rejects (crashes) or loops forever for every word in the language L', the complement of L:

accept(T) = L reject(T) + loop(T) = L'

Recursive sets are r.e, of course.

Just like regular languages are vacuously context-free.

Important Facts

• There exist languages that are r.e. but not recursive– You'll see one shortly

• There exist languages that aren't even r.e.!– (Again, you'll see one soon)

• All are “contrived”– They don't arrive in “practice”

• They just make mathematicians feel smart :-)

– Languages generated by grammars are r.e. or “better”

Closure Properties

• All r.e. languages are closed under union, intersection, concatenation, and Kleene*– Everything but complement!

• Recursive languages are also closed under complement

• Also: If L and L' are r.e., then L is recursive

• Here come the Proofs…

Complements of Recursive Languages

• Let M be a machine that decides a recursive language, L

• Form the machine M' by inverting the acceptability of M– Goes to a reject state instead

• Then M' decides L'– So L' is recursive

Complements of r.e. Languages

• Suppose L and L' are both r.e.• Let M recognize L, and M' recognize L'

– M may hang on elements of L', but M' doesn't

• Form a new machine, M* that calls M and M' in parallel (non-deterministically)– If M accepts w, so does M*– If M' accepts w, reject w– There are no other possibilities! (No hanging)– Therefore, L is recursive, by definition

Pit Stop/Review

• TMs can recognize strings from certain languages and/or compute functions

• If there is a TM, M, that accepts a language, L, and M always halts, then L is recursive– Also called decidable– M constitutes an algorithm for L

• If there is no such M for a language, L1, but there is instead a machine M1 that accepts every string in L1, but M1 may hang on strings not in L1, then L1 is recursively enumerable

Pit Stop(continued)

• The complement of a recursive language is recursive– Recursive languages are closed under all

operations, like regular languages are

• r.e. languages are closed under intersection

• The complement of a r.e. language may not be r.e.– If it is, then both languages are recursive!

Universal Turing Machines

• TMs that process partitioned input:– A “code” section and a “data” section

• A UTM simulates the actions of the input TM (the “code”) on the input data

• Requires encoding TM instructions as data– Using separator characters

B <TM Encoding>Current

State<Input> B ………………………...

A Sample TM Encoding

• a = 1• b = 11• B = 111

• q0 = 1

• q1 = 11

• qn = 1(n+1)

• L = 1• R = 11

• Use 000 to delimit machine components

• Use 00 to delimit transitions

• A TM is just a big binary string! (So they're enumerable!)

• The UTM reads a character, searches for the transition that applies, updates the state and tape

The Totality of TMs is Enumerable

• In other words, they are countable– They can be arranged in a sequence

• TM1, TM2, etc.

• Why?– Because they can be encoded as strings fed

unto a Universal TM– Strings can be ordered lexicographically– So there you have it!

Interlude

• The Halting Dog Problem

The Halting Problem

• Question: Is there a TM (H) that takes another TM (M) as input, along with an input string (w), and decides whether or not M halts on w?– In other words, is there such an algorithm?

• This would be very useful– Could detect infinite loops in bad code!– Or could save us from wasting precious time

The Halting Problem

• Suppose there is such a machine, H– This is the dog, Meg, before retraining

• Since all TMs are enumerable, we could apply H to all such TMs

• Construct a table of the outputs of applying H to all TMs

• If the machine won't halt, insert 0 or some other special value

The Halting Problem

• The halt detector is depicted below:

H

M

w

Accept if M halts on w

Reject if M doesn’t halt on w

The Halting Problem (cont.)

• Question: What happens when we replace M and w with H and its encoding, resp.?

H

H

H

Accept if H halts on H

Reject if H doesn’t halt on H

So far so good…

The Halting Problem (cont.)(“Retraining Meg”)

• Now, define a machine D as follows…

DM

Accept if M doesn’t halt on M

Reject if M halts on M

That is, D(x) calls H(x,x) and inverts the result.

Question: What is the result of D(D)?

Contradiction

• D accepts D iff D doesn’t halt on D

• Huh?

The Halting Problem (cont.)

• Conclusion: there can be no such machine, H• The Halting Problem is undecidable• Many problems are shown to be unsolvable by

showing them to be logically equivalent to the Halting Problem– i.e., show that H => P

• There are unsolvable problems• Just like there are unprovable propositions

– Godel's Incompleteness Theorem

A Non-recursive Language

• Let P be the set of all pairs (M,w) such that the machine M halts on input w

• This is r.e. because we can just input M and w into a UTM, U– U will either halt (accept), crash (reject), or hang (this

is the very definition of r.e.)– In fact, U will halt on each element of P

• By definition!

• But we just showed that it is not recursive– Because the Halting Problem is undecidable

A non-r.e. Language!

• Consider P':– P' = {Set of all (M,w) where either M is not a

TM encoding, or M does not halt on w}

• If P' were r.e., then P and P' would both be recursive– Since we proved that if a language and its

complement are r.e., they are both recursive!

• So P‘ is not r.e.!

Yet Another Look

• There are more languages than there are Turing Machines– this will prove the existence of non r.e.

languages

• How can you prove this?– By showing that the number of languages is

uncountable (not “enumerable”)

Which Set is Larger?

• A = {0, 1, 2, …}

• B = {1, 2, 3, …}

A 1-to-1 Correspondence

• The function f: A => B defined as

f(n) = n+1

is a 1-to-1 correspondence between A and B

• Therefore, their cardinality is the same– Welcome to infinity!

How “Big” is a Language?

• A Language is a subset of Σ*

• We can write the elements of Σ* in lexicographical order

• Therefore, there is a first one, a second one, etc.– So, the strings of a language are enumerable– Every language is a countable set of strings

How Many Real Numbers are There?

• The real numbers are uncountable– They cannot be mapped in a 1-to-1 fashion to

the counting numbers

• Proof:– Assume they can be– Arrange their digits in a table a[ ][ ]– The sequence a[n][n]+1 is not in the table!

How Many Bit Strings are There?

• This also has to be uncountable– Real numbers can be seen as bit strings

• But if you need convincing, use diagonalization again:– Arrange the supposedly countable number of

bits strings in a table– The bit string where you flip the diagonal bits

is not in the table!

How Many Languages Are There?

• It is an uncountable number!• Associate an infinite bit string with each

subset of Σ*– Bit-n is 1 if string sn is in some language L

• Each bit string represents a language!– But the number of such bit strings is

uncountable

• Therefore, there are an uncountable number of languages over Σ*

There are More Languages than TMs!

• Follows from the previous slide– Another way to say it:– There is no 1-to-1 mapping from N => 2N

• TMs are countable, the # of languages is not• Therefore, some languages cannot be

recognized by a TM– There aren't enough TMs to go around– Just like there are more reals than integers

• So, non-r.e. languages must exist!– “More” than are r.e.!

Undecidable Problems

• Does a TM halt on all input?

• Does an unrestricted grammar generate any strings?

• Is the language of a TM finite?

• Many more…

The Chomsky HierarchyLanguage Grammar Accepting Machine

Recursively Enumerable(“Turing Recognizable”)

Unrestricted Phrase Structure (Type 0)

TM (that may not halt)

Recursive(“Turing Decidable”)

" TM (that always halts)

Context-Sensitive Context Sensitive (Type 1)(monotonic/non-contracting)

Linear-bounded automata(bounded TM)

Context Free Context Free (Type 2) PDA

Deterministic Context Free

" DPDA

Regular Regular (Type 3)(Right-linear or Left-linear)

FA

Unrestricted Phrase Structure Grammars

• Left-hand side of the rule can have multiple symbols– Must have at least one non-terminal, though

• Introduces context-sensitive replacement

• “Type 0” Grammar

A Grammar for anbncn

S → aAbc | abc | e

Ab → bA

Ac → Bbcc

bB → Bb

aB → aa | aaA

S aAbc

abAc

abBbcc

aBbbcc

aabbcc

An A is created as the left-most variable. It travels to the right until it finds a 'c', then creates a new b and c, and becomes a B. The B moves back to create the extra needed a. The last rule allows the option to do it all over again (by introducing another A). This is similar to what the Turing machine for the same language does, except the TM recognizes instead of generates the letters.

About Unrestricted Grammars

• Cannot use the decidability algorithms for CFGs– CYK algorithm does not apply

• No “Normal Form”

• Non-null productions may create shorter strings

• Terminals can disappear!

Context-Sensitive Grammars

• Like Unrestricted Grammars except:– Right-hand side must be no shorter than left-hand

side• So strings never shrink

– Hence the name “non-contracting” or “monotonic”– Cannot contain lambda (the empty string)

• Context-sensitive languages don't need unlimited memory– Since intermediate sentential forms never contract,

the largest memory requirement is |w|– Accepting machine: Linear Bounded Automaton

• A TM with limited memory

anbncn, n>0 is Context Sensitive

S → aAbc | abc

Ab → bA

Ac → Bbcc

bB → Bb

aB → aa | aaA

Matrix Grammars

• Like Phrase Structure Grammars, but sets of rules must be applied in sequence

• P1: S → S1S2

P2: S1 → aS1,S2 → bS2c

P3: S1 → e, S2 → e

• Generates anbncn

• S S1S2 aS1bS2c aaS1bbS2cc aabbcc

Church-Turing Thesis

• "It is believed that there are no functions that can be defined by humans, whose calculation can be described by any well-defined mathematical algorithm that people can be taught to perform, that cannot be computed by TMs. The TM is believed to be the ultimate calculating mechanism."