introduction to intractability p, np, np-completeness how to approach intractable problems monday,...

1

Introduction to Intractability

P, NP, NP-completeness

How To Approach Intractable Problems

Monday, August 3rd

2

Outline For Today

1. Intractability: P, NP, NP-completeness

2. How to Approach Intractable Problems?

3

Outline For Today



Focus of CS161

4

Practical Algorithms to Fundamental Problems

Almost-linear time super-fast algorithms

Sorting: O(nlogn)

Dijkstra: O(mlog(n))

MST: O(mlog(n))

Superlinear but still practical

Strassen’s Algorithm: O(n2.83)

Bellman-Ford: O(mn)

Floyd-Warshall: O(n3)

Karger, etc…

Sad Fact

5

Many other important problems seem

impossible to solve very efficiently.

Ex: TSP, Max-Cut

A critical skill that we all have to acquire is to

recognize such problems.

Formalizing Tractability: Class P

6

Given a computational problem C

P: C is ∈ P (polynomial-time solvable) if ∃an

algorithm solving C with O(nk) run-time, for some

constant k.

where n is the input length in bits

k is some constant, say, 1, 2, 5, 1M, etc.

Ex poly-times: O(n), O(nlog(n)), O(n3), O(n100000)

Ex: Almost all problems we have seen so far

except TSP, Independent Set, Max-Cut

Interpretation of P and Its Caveats

7

A rough test for **tractability**.

Caveat 1: n1000 is not tractable in practice

Caveat 2: Some intractable problems can actually be

solved in some restricted form efficiently

But generally speaking, has been

**extremely successful** in classifying

problems.

Example of Intractable Problem: TSP

8

TSP: Traveling Salesman Problem

Input: Complete graph G(V, E) with non-negative

edge costs

Output: Min-cost tour [cycle visiting each vertex

once]

D

CB

A2

14

3

5

6

Example of Intractable Problem: TSP

9

TSP: Traveling Salesman Problem

Input: Complete graph G(V, E) with non-negative

edge costs


once]

D

CB

A2

14

3

5

6

Answer: A->B->D->C->A

= 13

Conjecture from 1965 (Jack Edmonds)

10

We have been looking for a fast algorithm for

TSP for > 80 years, and no one has succeeded.

After 30 years or so in 1965, Jack Edmonds

made the following conjecture:

There is no poly-time algorithm for

TSP.

(equivalent to conjecture: P != NP)

To this day, no one has been able to

prove/disprove this conjecture.

Making a Case For TSP’s Intractability?

11

(So far) We have not been able to argue for

TSP’s intractability in an absolute sense.

Instead accumulate evidence of intractability

Idea From Early 1970s:

Instead show “relative” intractability

(i.e. show TSP is as hard as bunch of

other problems.)

Reductions

12

What does it mean for problem C2 to be as hard

as C1?

Definition: C1 reduces to C2, (C1 ≤p C2), if given a

poly-time algorithm for C2, we can solve C1 in

poly-time.

If C1 reduces to C2 => C2 is “as hard as”

C1

Poly-time Alg for C2

Π1 ∈ C1

Poly-time C1 -> C2

ConverterΠ2 ∈ C2

Solution to Π2

Poly-time C2 Solution -> C1

Solution Converter

Solution to Π1

High-level Idea of Reduction

13

Efficient Alg A for C2

Π1 ∈ C1 Efficient Π2 ∈ C2

Solution to Π2

EfficientC2 Solution -> C1

Solution Converter

Solution to Π1

If we can solve C2 efficiently, we can also solve C1

efficiently

C2 as hard as C1

Effectively an efficient algorithm for C1

Reductions are Transitive

14

Claim: If C1 ≤p C2, and C2 ≤p C3, then C1 ≤p C3

Proof: We have to argue if we have a poly-time algorithm

for C3,

we can solve C1 efficiently. Or C3 is as hard as C1.

If we had a poly-time algorithm for C3

we could solve C2 in poly-time (since C2 ≤ C3)

we could solve C1 in poly-time (since C1 ≤ C2)

Q.E.D.

Contrapositive

15

Efficient Alg A for C2

Π1 ∈ C1 Efficient Π2 ∈ C2

Solution to Π2

EfficientC2 Solution -> C1

Solution Converter

Solution to Π1

Then if C1 ≤p C2, there cannot be an efficient algorithm A

for C2

Assume we knew NO efficient algorithm for C1 exists

X

Making a Case for TSP’s Intractability

16

Efficient Alg A for

TSPΠ1 ∈ C1 Efficient Π2 ∈ TSP

Solution to Π2

EfficientTSP Solution -> C1 Solution Converter

Solution to Π1

If we can solve TSP efficiently, we can also solve C1 efficiently

TSP is as hard as C1

Goal: Argue TSP is as hard as a large C of problems

Definition: Completeness

17

Let C be a set of problems.

Ci ∈ C

If ∀ Ck ∈ C , Ck ≤p Ci, then Ci is C-complete.

(i.e. Ci is as hard as every other problem in C )

(i.e. Ci is the hardest problem in C ) Goal: Argue TSP is C-complete for as large a

set C as possible.

The larger C, the more evidence we

accumulate for TSP’s intractability.

Arguing TSP is C-complete for a large C

18

C

C1

TSP

C3

C4 C5

C6

Ck

C2

Picking an Appropriate COption 1: Let C be any problem

Q: Could we hope to prove that TSP is as hard

as any other computational problem?

A: No, there are problems, such as the Halting

Problem, which we know are simply not

solvable!

(i. e, do not have any algorithms solving them,

let alone an efficient one).

But TSP is solvable: if nothing, by brute-force

search. Exponential-time but solvable.

Option 2: C = NP: Brute-force Solvable Problems

Option 2: Let C be all brute-force solvable problems

Definition (NP or brute-force solvable problems):

A problem C ∈ NP if:

1. Correct solutions have polynomial length.

2. Claimed solutions are verifiable in poly-time.

***All that is required to be in NP is that we

can verify the solution to the problem

efficiently.***

Example NP problems Every problem we have seen in CS 161.

Every problem in P.

(Most-likely) Every problem you will ever see in

practice.

Ex: Decision version of TSP: ∃ T of size ≤ k?

1. Each tour T is a cycle of size n, so poly-length.

2. We can verify length of T in poly-time: just sum

each edge in T.

What is an NP-complete Problem?

22

NP

C1

C*: NP-complete

C3 C4C5

C6

Ck

C2

C* is as hard as any NP problem!

Solving an NP-complete Problem

23

NP

C1

C*: NP-complete

C3 C4C5

C6

Ck

C2

If we can solve C* efficiently, we can solve every

single

NP problem (basically all problems in practice)!!!

Do NP-complete Problems Exist?

24

YES! They’re Everywhere!

History of NP-completeness

25

< 1970: lots of unsolved problems

1971: Cook-Levin Thm: SAT is NP-complete

(just from the definition of NP)

1972: Karp: By showing SAT reduces to 21

other problems, showed the existence of 21

other NP-complete problems. (why?)

Since 1972: 1000s of problems are NP-

complete (including TSP).


26

NP

C2

SAT

C3 C4C5

Ck

C6

C1

K1

K2… K20

K21

1971: Cook-Levin1972: Karp


27

NP

C2

SAT

C3 C4C5

Ck

C6

C1

K1

K2… K20

K21

NP-complete

Since 1972

Reducing C* to C** => C** is NP-complete

28

NP

C2

C*: NP-complete

C3 C4C5

Ck

C6

C*

*

If we can solve C** efficiently => we solve C*

efficiently

=> we solve all NP problems efficiently

C** is NP-

complete!

Overall “Intractability” Argument for TSP

29

Can’t prove TSP is hard in an absolute sense.

Argue that TSP is “hard” in a relative sense.

Show that TSP is NP-complete => i.e. TSP is as

hard as any other NP problem

This is our evidence that TSP is intractable.

Why is TSP NP-complete?

30

HAM-CYCLE Problem:

Input: Undirected Graph G(V, E)

Output: YES if G contain a Hamiltonian cycle/NO

o.w

Dfn: A Ham. Cycle is a simple cycle that contain

each vertex of the graph once.

Fact: Hamiltonian Cycle is NP-complete.

SAT≤p3-SAT≤pCLIQUE≤pVERTEX-COVER≤p

HAM-CYCLE

Hamiltonian Cycle Example (1)

31

F

CB

A

D

E

G

Hamiltonian Cycle Example

32

F

CB

A

D

E

G

Answer: YES.


33

F

CB

A

E


34

F

CB

A

E

Answer: NO. Any complete cycle has to visit A twice

HAM-CYCLE ≤p TSP

35

Let G(V, E) be the input to HAM-CYCLE

Let G*(V, E*) be a complete graph with edge

weights:

w((u, v)) = 0 if (u, v) ∈ E

w((u,v)) = 1 if (u, v) ∉ E

Claim:∃a Hamiltonian

Cycle in G ⇔∃a TSP Tour

with weight 0

Proof of Claim

36

=> If∃a Hamiltonian Cycle C in G, then since

each edge of C has weight 0 in E*, then C is a

tour in G* with weight 0

<= If∃a tour T with weight 0 in G*, then all of its

edges must be of weight 0, and hence from E, so

T is a hamiltonian cycle in G

Q.E.DTherefore, if we can solve TSP efficiently, we can

solve HAM-CYCLE efficiently,

Completing TSP’s NP-completeness Proof

37

If we can solve TSP efficiently, we can solve HAM-

CYCLE efficiently

(by just transforming the input G to G* in poly-

time)

since HAM-CYCLE is NP-complete

we can solve every single NP problem

efficiently

TSP is NP-complete

Q.E.D

Does P = NP?

38

We know P ⊆ NP. Two possibilities:

NP

P

P=N

P

Is every problem, whose solutions are efficiently

verifiable, is also efficiently solvable?

We don’t know. But most people believe P ≠ NP.

Why Do People Believe P ≠ NP?

39

1. There are 1000s of NP-complete problems and

for none of them is there a poly-time

algorithm.

2. Counter to General Human Experience

3. Also some weird mathematical consequences,

such as polynomial hierarchy collapsing, and

others…

But We Simply Don’t Know (Yet)!

40

Outline For Today



Your Problem is NP-complete. Now What?

41

Option 1: Focus to special-case inputs.

Ex: Independent Set is NP-complete.

Focusing on line graphs, had a O(n) DP alg.

Option 2: Find an approximate answer.

Ex: Max-cut is NP-complete

We had O(n)-time expected ½-approximation

Option 3: Be exponential time but better than

brute-force search.

Option 4: Heuristics: fast algorithms that are not

always correct (or even approximate)

Option 5: Mix some of these options

Vertex Cover (Ex of Options 1 & 3)

42

Input: Undirected Graph G(V, E)

Output: Minimum Vertex Cover of G

Vertex Cover: S ⊆ V, s.t. for each (u, v) ∈ E:

either u ∈ S, or v ∈ S.

Fact: Vertex Cover is NP-complete:

3-SAT≤pCLIQUE≤pVERTEX-COVER

Vertex Cover Example (1)

43


44


45

B

DC

A

E

F


46

B

DC

A

E

F


47

B

DC

A F

Min Vertex Cover: {A, B}

F

Let’s Hope that There is a VC That’s Small

48

Suppose we know that ∃VC of size ≤ k, with k

small

So, we’re focusing on special case inputs

Bad Brute-force Search:

Look at all choose(n, k) possible VCs = Ω(nk)

So now we have a poly-time algorithm

Problem 1: we didn’t understand the problem

at all

Problem 2: even with k=5, we’re n5, not

practical

Smarter Brute-Force Search (1)

49

Consider an edge (u, v). Let S* be a VC with |S*|

≤ k

Q: What’s a claim about VCs that doesn’t need a

proof?

A: S* contains either (1) u or (2) v (or both)

Idea: Guess which case we’re in and recurse


50

u

G`

v

G``

Case 1

Case 2

Q: Consider case 1. What can you assert about

S*-{u}?

A: S*-{u} is a VC in G` with size ≤ k -1.


51

u

G`

v

G``

Case 1

Case 2

Q: Consider case 2. What can you assert about

S*-{v}?

A: S*-{v} is a VC in G`` with size ≤ k -1.


52

G has a VC

of size ≤ k ⇔G` or G`` has

a VC of size ≤

k-1

Smarter Brute-Force Search Pseudocode

53

procedure Smarter-BruteFS (G(V,E), k):if k == 0:

if (G is empty): return empty-set else return null

pick an arbitrary edge (u, v)Let G` = (V – {u}, E- all (u, w))Let G`` = (V – {v}, E- all (v, w))

Let T1 = Smarter-BruteFS(G`, k-1)Let T2 = Smarter-BruteFS (G``, k-1)if (both T1 T2 are null) return nullelse return T1∪u or T2∪v

Runtime:

54

T(k) ≤ 2T(k-1) + m

Branching factor 2, depth k: O(2km)

Compared to the first brute-force search

alg k=5 => n5 vs 25m

k=8 => n8 vs 28m

k=15 => n15 vs 215m

Could be much better than the first brute-

fs!

TSP with Triangle Inequality (Options 1& 2)

55

Input: Complete graph G(V, E) non-negative edge

costs, where triangle inequality holds


once]

Triangle Inequality: ∀u, v, w: d(u, v) ≤ d(u, w) + d(w,

v)

Upshot: We will get at 2-approximation to OPT Tour

C*

Option 1 b/c we’re restricting to a subclass of all

inputs

Option 2 b/c we only approximate the exact solution

TSP with Triangle Inequality Example

56

D

CB

A2

11

32

2


v)


57

D

CB

A2

11

32

2


v)


58

D

CB

A2

11

32

2


v)

d(B, D) = 3 ≤ d(B, A) + d(A, D)=3


59

D

CB

A2

11

32

2


v)

D(B, D) = 3 ≤ d(B, C) + d(C, D)=4

Similarly holds for all other edges


60

D

CB

A2

11

32

2


v)

OPT C*= 1+2+2+2 = 7

“Taking Shortcuts” Lemma

61

Assume the triangle inequality holds in a complete

graph.

Let P=u->x1 -> … -> xk ->v be any path in G, then:

Claim: d(u, v) ≤ d(u, x1) + d(x1, x2) + … d(xk-1, xk)

+d(xk, v)

u v

xkx1

P

(u,v) Claim: d(u, v) ≤

cost(P)

Proof by Induction on path lengths k

62

Base case: k=1 => This is true by the triangle

inequality

IH: Holds for path lengths k-1

d(u, v) ≤ d(u, xk) + d(xk, v) (by triangle inequality)

u v

xkx1

d(u, xk) ≤ d(u, x1) + d(x1, x2) + … d(xk-1, xk)

(by IH)d(u, v) ≤ d(u, x1) + d(x1, x2) + … d(xk-1, xk) +

d(xk, v)

Removing any Edge of TSP Tours

63

D

CB

A2

11

32

2

Any tour C is a simple cycle in the graph

Consider removing any edge of C: C-(u, v)

Ex: Consider C*, and remove (A, D) (any edge in

general)

Removing any Edge of TSP Tours

64

D

CB

A2

11

32

2

Any tour C is a simple cycle in the graph

Consider removing any edge of C: C-(u, v)

Ex: Consider C*, and remove (A, D) (any edge in

general)

Q: What other graph structure is C` = C –(A, D)?

A: C` is a spanning tree

B/c it has n-1 edges in it’s

connected

Cost(MST) ≤ Cost(C* - (u,v)) ≤ Cost(C*)

65

D

CB

A2

11

32

2

MST T* = 1 + 1 + 2 =4

Let’s Run a DFS On MST T*

66

D

CB

A2

11

32

2


67

D

CB

A2

11

32

2

D


68

D

CB

A2

11

32

2

D -> A


69

D

CB

A2

11

32

2

D -> A -> C


70

D

CB

A2

11

32

2

D -> A -> C


71

D

B

A2

11

32

2

D -> A -> C -> A

C


72

D

B

A2

11

32

2

D -> A -> C -> A -> B

C


73

D

B

A2

11

32

2

D -> A -> C -> A -> B

C


74

D

B

A2

11

32

2

D -> A -> C -> A -> B -> A

C


75

D

B

A2

11

32

2

D -> A -> C -> A -> B -> A

C


76

D

B

A2

11

32

2

D -> A -> C -> A -> B -> A -> D

C


77

D

CB

A2

11

32

2

D -> A -> C -> A -> B -> A -> D

Cost of X=(D -> A -> C -> A -> B -> A -> D)?

78

D

CB

A2

11

32

2

A: 2*cost(T*) But it’s not a valid tour, because it

visits some vertices twice.

Trimming X=(D -> A -> C -> A -> B -> A -> D)

79

D -> A -> C -> A -> B -> A -> D

Trim X by taking “shortcuts”.

D


80

D -> A -> C -> A -> B -> A -> D


D


81

D -> A -> C -> A -> B -> A -> D


D -> A


82

D -> A -> C -> A -> B -> A -> D


D -> A


83

D -> A -> C -> A -> B -> A -> D


D -> A -> C


84

D -> A -> C -> A -> B -> A -> D


D -> A -> C


85

D -> A -> C -> A -> B -> A -> D


D -> A -> C


86

D -> A -> C -> A -> B -> A -> D


D -> A -> C -> B


87

D -> A -> C -> A -> B -> A -> D


D -> A -> C -> B


88

D -> A -> C -> A -> B -> A -> D


1. Trimmed path Y is now a valid tour.

2. Shortcuts can only decrease the

cost

=> cost(Y) ≤ 2cost(T*) ≤ 2C*

D -> A -> C -> B -> D

2-Approx TSP with Triangle Inequality

89

procedure 2-ApproxTSP(G(V,E), edge weights):

Let T* = MST(G(V, E)) // Kruskal/PrimRun DFS(T*) and get a non-simple tour

XY = trim(X)return YCorrectness (already argued)

Run-time: O(mlog(n))

Summary

90

1. Took TSP, an NP-complete problem (don’t

hope to solve exactly in general form)

2. Restricted inputs to those with triangle

ineq.

3. Got a 2-approximation in O(mlog(n)) time

Attacking NP-complete Problems

91








brute-force search.




Attacking Vertex Cover

92








brute-force search.




Attacking TSP

93








brute-force search.




94

More on Approaching NP-complete

problems on Wednesday (Knapsack)

introduction to intractability p, np, np-completeness how to approach intractable problems monday,...

Documents