complexity theory primer - freie universität · complexity theory primer we will discuss:...

Complexity Theory Primer

We will discuss:

• Optimization and decision problems

• The classes P, EXP, and NP

• NP-completeness, reductions, and NP-hardness

1

Source and References

This lecture is mainly based on a script by Prof. Heinz Schmitz, FH Trier.

Further reading:

• U. Schoning: Theoretische Informatik kurz gefaßt . B. I. Wissenschaftsverlag,Mannheim, 1992. ISBN 3-411-15641-4

• M. Garey, D. Johnson: Computers and Intractability: a Guide to the Theory ofNP-Completeness. Freeman, San Francisco, 1979. ISBN 0-7167-1044-7

• C. Papadimitriou: Computational Complexity . Addison Wesley, Reading, 1994.ISBN 0-201-53082-1

• W. Cook, W. Cunningham, W. Pulleyblank, A. Schrijver: Combinatorial Opti-mization. John Wiley & Sons, New York, 1998. ISBN 0-471-55894-X

2

Complexity theory

Algorithms with polynomial and algorithms with exponential running time differ dra-matically with respect to their applicability to practical problems:

n 20 40 60 100 300timen 10−8 sec 10−8 sec 10−7 sec 10−7 sec 10−7 secn2 10−7 sec 10−6 sec 10−6 sec 10−5 sec 10−4 secn3 10−5 sec 10−4 sec 10−4 sec 10−3 sec 10−2 sec

2n/ log n 10−7 sec 10−6 sec 10−5 sec 10−3 sec 79 days2n 10−3 sec 19 min 37 years 1013 years 1073 years

Assumption: 109 operations/sec

Complexity theory (2)

The field of complexity theory investigates classes of problems of different complex-ity and their relations.

⇒ Do problems exist that can be solved in time O(n logn), but not in time O(n)?

⇒ Can all problems for which we can compute a solution in O(logn) space besolved in polynomial running time?

Answers to these fundamental questions have considerable practical impact.

3

Optimization and decision problems

Given a certain problem P , we denote by IP the set of problem instances and bySP the set of feasible solutions.

Example.

IP := (m, D) |m ∈ N and D is m×m matrix

with D(i, j) ∈ N for 1 ≤ i, j ≤ mSP := (c1, . . . , ck) | k ∈ N and

c1, . . . , ck = 1, . . . , k

The set of feasible solutions of a certain instance x ∈ IP is a subset of SP .

In the example: If x = (m, D), then all m-ary tuples in SP are feasible.

In this manner we are able to define the therm optimization problem in a formal way.

4

Optimization and decision problems (2)

Definition. An optimization problem P is a tuple (IP , SP , solP , mP , goal), where

1. IP is the set of instances,

2. solP : IP 7→ 2SP is a function that returns the set of feasible solutions for agiven instance x ∈ IP

3. mP : IP × SP 7→ N is a function, which, given a pair (x, y(x)) with x ∈ IPand y(x) ∈ solP(x), returns the value mP(x, y(x)) ∈ N of solution y(x) ofthe instance x, and

4. goal ∈ min,max.

5


Example. Minimum traveling salesman (MIN-TSP)

Sets IMIN-TSP and SMIN-TSP are already defined in the previous example.

solMIN-TSP(m, D) := SMIN-TSP ∩ Nm

mMIN-TSP((m, D), (c1, . . . , cm)) :=m−1∑i=1

D(ci, ci+1) + D(cm, c1)

goalMIN-TSP := min

If the context is clear, we omit the indices (e.g., MIN-TSP) and write y instead ofy(x).

Example. Maximum similarity multiple sequence alignment with WSOP score(MAX-MSA-WSOP).Blackboard.

6


We need some terminology to deal with optimal solutions:

Definition. A solution y∗(x) is optimal for the instance x, if

m(x, y∗(x)) = goalv | v = m(x, z) and z ∈ sol(x).

The set of all optimal solutions y∗(x) of x is referred to as sol∗(x).

The optimal value of an instance x is denoted by m∗(x).

⇒ it follows directly that all optimal solutions have the same value

7


There are three important variants of optimization problems:

1. constructive problem PC

input: x ∈ IPoutput: a y∗(x) and m∗(x)

2. evaluation problem PE

input: x ∈ IPoutput: m∗(x)

3. decision problem PD

input: x ∈ IP and k ∈ Noutput: if goalP = max (the case goalP = min is analogous)

1, if m∗(x) ≥ k

0, otherwise

Unless otherwise stated, an “optimization problem” is a constructive problem.8


We call PD the decision problem corresponding to P . It is “easier” than PC orPE, since solving the constructive or evaluation problem also solves the decisionproblem:

• calculate m∗(x) with an algorithm for PC or PE

• compare k and m∗(x)

Example. TRAVELING SALESMAN (TSP) is the decision problem corresponding toMIN-TSP. Here, the input consists of additional maximal costs k ∈ N of a tour.

input: m ∈ N, an m×m distance matrix with D(i, j) ∈ N and k ∈ N.output: 1, if there exists a permutation (c1, . . . , cm) of 1, . . . , m with cost≤ k,

0, otherwise.

MSA-WSOP example. Blackboard.

9


The following example shows a typical packing problem, where the aim is to max-imize the profit of the chosen items under the restriction that they not exceed amaximum size S.

Example. MAXIMUM KNAPSACK

input: m items with sizes s1, . . . , sm ∈ N, profits v1, . . . , vm ∈ N, and amaximal size S ∈ N.

output: I ⊆ 1, . . . , m with∑i∈I

si ≤ S

measure:∑i∈I

vi

KNAPSACK is the corresponding decision problem. Here, an additional minimumtotal profit that has to be achieved is given:

input: m items with sizes s1, . . . , sm ∈ N and profits v1,. . . ,vm ∈ N, maximalsize S ∈ N and k ∈ N.

output: 1, if there exists a set I ⊆ 1, . . . , m with∑i∈I

si ≤ S and∑i∈I

vi ≥ k,

0, otherwise. 10


A decision problem only asks for a yes-no-answer. This can also be expressed interms of a set E ⊆ IP for which the characteristic function

cE(x) :=

1, if x ∈ E,0, otherwise

has to be computed for a given instance x ∈ IP (membership problem).

An equivalent, often used, definition is in terms of languages: Given an alphabet Σ,a language L is a subset of Σ∗. The language that corresponds to a decision prob-lem contains those words (encodings of input instances) that yield a yes-answer.

We also say that algorithms that solve a decision problem decide the correspondinglanguage.

Example. PRIME := x ∈ N | x prime number= 2,3,5,7,11,13, . . . ⊆ 1,2,3,4, . . .

11


Example.problem: MINIMUM PATH

input: graph G = (V, E), two vertices s, t ∈ Voutput: a path (v0, v1, . . . , vk) from s = v0 to t = vk with minimum k

problem: PATH

input: graph G = (V, E), two vertices s, t ∈ V and k ∈ N.output: 1, if there exists a path (v0, v1, . . . , vl) from s = v0 to t = vl with

l ≤ k,0, otherwise.

12

The classes P and NP

We introduce some notation from complexity theory in order to classify optimizationproblems. For systematic reasons, we focus on decision problems.

Polynomial time.

We call algorithmic problems for which we know an algorithm that solves them intime O(nk) for some k ∈ N efficiently solvable (cf. table of running times at thebeginning of the primer). The motivates the definition of the complexity class P:

P := E decision problem | there exists a k ∈ N and an algorithm A

such that A solves E in time O(nk)

An algorithm solves a decision problem E, if it computes cE.

13

The classes P and NP (2)

Example.

We already know an algorithm that solves PATH in time O(n2):∗

• Each vertex has at most |V | direct successors.

• We only have to consider paths up to length k < |V |.

Thus, PATH ∈ P.

∗The algorithm sketched above is a very short summary of the Bellman-Ford algorithm to computeshortest paths in a graph.

14


Example.

A naıve algorithm for PRIME checks, given an input x ∈ N, whether one of thenumbers in 2, . . . ,

√x is a factor of x. This results in a running time of Ω(

√x) =

Ω(2n−12 ).

In 2002, M. Agrawal, N. Kayal, and N. Saxena gave a polynomial time algorithm forPRIME, thus PRIME ∈ P.

15


How could we solve KNAPSACK? We have to find a subset of items 1,2, . . . , mwith certain properties.

We could proceed as follows:

1. Consider every subset I = ∅, 1, 2, . . . of 1,2, . . . , m

2. Check whether∑i∈I

si ≤ S

3. Check whether∑i∈I

vi ≥ k.

Step 1 yields a complexity of Ω(2m), i.e., the algorithm is only practicable for verysmall instances.

We cannot use the above algorithm to show that KNAPSACK is in P.

16


At least we know that KNAPSACK belongs to the complexity class EXP:

EXP := E decision problem | there exists a k ∈ Nand an algorithm A such that

A solves E in time O(2nk)

Since every function in O(nk) is also in O(2nk), we have P ⊆ EXP. There is even

a proof that problems in EXP exist that do not belong to P. Whether KNAPSACK

belongs to these problems∗ is an open problem.

Theorem. P ( EXP.

∗As we will argue later, most complexity theoreticians think so

17


Non-deterministic algorithms.

The knowledge that problems such as KNAPSACK belong to the class EXP is notvery satisfactory, because EXP also contains problems whose solution requires ex-ponential running time due to tedious/long-winded calculations.

But KNAPSACK exhibits a special problem structure that differs from other problemsin EXP:

• The size of the solution space is exponential.(KNAPSACK: all 2m subsets are solution candidates.)

• We can check in polynomial time whether a solution candidate is feasible.(KNAPSACK: we just have to compute the sums∑

i∈I

si and∑i∈I

vi

and compare them to S and k.)18


The reason for the complexity of KNAPSACK are thus not expensive calculations butthe difficulty of finding a solution in the exponentially large solution space.

We classify problems that exhibit such a structure by establishing the complexityclass NP.

The problem structure “large solution space / easy candidate test” suggests tosearch the solution space in parallel .

⇒ If we were able to check all candidates I ⊆ 1, . . . , m in parallel, we couldsolve KNAPSACK in in polynomial time.

19


To be able to do parallel calculations in our pseudocode model, we add the possi-bility of non-deterministic branching:

begin expr1 | expr2 | · · · | exprk end

Executing such a statement splits the computation path in k independent computa-tion paths, which cannot communicate with each other. Each such path computesan independent result.

Non-deterministic branching changes the traditional linear computation path into acomputation tree. We can use this tree to generate and check candidate solutions.

20


Example.

algorithm: non-deterministic knapsackinput: m, s1, . . . , sm, v1, . . . , vm, S, k ∈ N

I := ∅;for i := 1 to m do

begin I := I ∪ i | I := I end ; // non-deterministic branchingif (

∑i∈I

si ≤ S) & (∑i∈I

vi ≥ k) return 1; // check candidates

return 0;

21


Let’s look at the instance m := 3, s := (3,2,2), v := (5,1,3), S := 6, k := 6.

SI = 7 SI = 5VI = 6

SI = 5VI = 8

SI = 3VI = 5

SI = 4VI = 4

SI = 2VI = 1

SI = 2VI = 3

SI = 0VI = 0

0 1 0 00 01 0

1, 2, 3 1, 31, 2 1 2, 3 2 3 ∅

1, 2 1 2 ∅

1 ∅

By means of the non-deterministic branching operator we can adequately capturethe problem structure of KNAPSACK.

22


Non-deterministic computations yield a 1-answer if and only if at least one of thecomputation paths yields a 1-answer.

Remark. Non-determinism is a theoretical model , and currently no computer canrealize this model for realistic applications. This is due to the fact that the numberof independent computation paths grows exponentially with the input size. In acomputer, however, the number of parallel processes or processors is bounded bya constant.

Example. One million processors are not sufficient to simulate all independentcomputation paths for the KNAPSACK problem for an instance with only 20 items.

23


NP := E decision problem | there exists a k ∈ N such that

a non-deterministic algorithm A solves E

in time O(nk)

The definition of the class NP (= Non-deterministic Polynomial time) captures thefollowing aspects:

• We allow non-determinism by adding the non-deterministic branching operatorto the pseudocode instruction set.

• The running time has to be polynomially bounded on each path of the compu-tation tree.

• The whole computation returns 1 if and only if one path yields a 1-answer.

We have already seen that KNAPSACK ∈ NP.24


We can characterize NP also in terms of languages:

Theorem. Let L ⊆ Σ∗ be a language. L ∈ NP if and only if there is a polynomial-time decidable relation R such that

L = x | (x, y) ∈ R for some y and |y| ≤ |x|k .

Merkregel: Problems in P are “easy to solve” , problems in NP are “easy to check” .

Proof sketch. (⇐) Assume such an R exists. Then L can be decided by a non-deterministic algorithm that, given an input x, “guesses” a y of length at most |x|k

and then decides whether (x, y) ∈ R in polynomial time.

The forward direction (⇒) of the proof is similar.

A detailed proof is part of the assignments (Ex. 7.3).

25


The class NP contains all problems with the mentioned structure:

• the size of the solution space is bounded by 2p(n) (for a polynomial p)

• checking a candidate needs polynomial time

The class is very interesting since the above characterization comprises many prac-tically relevant problems.

Theorem. P ⊆ NP ⊆ EXP.

Proof. This is easy to see: Every deterministic polynomial-time algorithm is also anon-deterministic polynomial time algorithm.

On the other hand we can fully deterministically simulate a computation tree ofpolynomial depth in time O(2p(n)) (e.g., with DFS) to check whether one of theleaves returns a 1-answer.

26


The question to find an efficient algorithm for KNAPSACK and many other problemsin NP can be formulated shortly as

P-NP question : Does P = NP hold or does P 6= NP hold? ∗

This is one of the seven millenium problems identified by the Clay MathematicsInstitute. There is a video (see Section “Links” on the class website) with a verynice lecture by Vijaya Ramachandran on this problem.

∗proven answer worth 106 $

27

Reductions

⇒ Comparing problems with respect to their complexity

The concept of reducability captures the following intuitive idea:

If we can solve a problem P by means of an algorithm for a problem P ′

without “significant” additional effort , we call P reducible to P ′.

Thus, we relate and compare problems P and P ′ with respect to their algorithmictractability. An algorithm A for P can use an algorithm A′ for P ′ as a subroutine.

If P is reducible to P ′, it cannot be algorithmically harder than P ′, which we expressby writing P ≤ P ′ .

28

Reductions (2)

Let P and P ′ be two decision problems. The following picture illustrates the relationP ≤ P ′:

algorithmfor P ′f

algorithm for P

x ∈ IPf(x) ∈ IP′

0

1

Definition. Problem P is reducible to P ′ if there exists a polynomial-time functionf : IP 7→ IP ′ with the property

x ∈ P ←→ f(x) ∈ P ′ .

29

Reductions (3)

To show P ≤ P ′ we have to find a suitable reduction f and show

x ∈ P ←→ f(x) ∈ P ′ .

• Note that we do not necessarily need to know an algorithm for P ′ to showP ≤ P ′.

• If, however, this is the case (now or later), we immediately get an algorithm forP (with a bit of∗ overhead).

Lemma. The relation ≤ is reflexive and transitive.

Proof. Blackboard.

∗polynomial30

Reductions (4)

Example. Let P = SUM OF SUBSETS and P ′ = KNAPSACK.

problem: SUM OF SUBSETS

input: a1, . . . , am, b ∈ Nquestion: Does an I ⊆ 1, . . . , m exists with

∑i∈I

ai = b?

Let x = (a1, . . . , am, b) ∈ IP . How can we decide this instance of SUM OF SUB-SETS with an algorithm for KNAPSACK?

Idea: Treat every number ai ∈ N as an item of size ai and profit ai. The algorithmfor KNAPSACK should choose a subset of items (numbers) whose values sum upexactly to b. The trick is to split the equality into two inequalities. Thus, we requirethe sizes of the choice to be at most b and the profits to be at least b.

b

feasible sizefeasible profit

31

Reductions (5)

Formally, we define the reduction f as

f(a1, . . . , am, b) := (a1, . . . , am, a1, . . . , am, b, b).

Obviously, f runs in polynomial time since we just have to copy the input. Now weshow (blackboard) the property

x ∈ SUM OF SUBSETS ←→ f(x) ∈ KNAPSACK .

32

Reductions (6)

The example shows

• If we have an efficient algorithm for KNAPSACK, we can use it to solve SUM OF

SUBSETS efficiently.

• Vice versa: If we can prove that is impossible to solve SUM OF SUBSETS effi-ciently this must also hold for KNAPSACK.

This is a general observation:

Theorem. (proof omitted, it is easy)

1. If P ≤ P ′ and P ′ ∈ P then P ∈ P.

2. If P ≤ P ′ and P ′ ∈ NP then P ∈ NP.

Another way to put this is to say that P and NP are closed under reduction.

33

NP-completeness

Using reductions we can compare all problems in NP. The following natural questionarises:

Does NP contain “hardest” problems? That is, problems, to which all otherproblems in NP are reducible.

These are extremely interesting with respect to the P-NP-question.

Definition. A decision problem P is NP-complete, if

1. P ∈ NP and

2. for all P ′ ∈ NP holds P ′ ≤ P.

34

NP-completeness (2)

Thus, an algorithm for an NP-complete problem P can solve every other problem inNP without significant overhead.

Such a problem P can be seen as a representative for the whole class NP in termsof complexity.

In particular: A polynomial time algorithm only for one single NP-complete problemleads to an efficient algorithm for every problem in NP, and we have P = NP.

Theorem. Let P be an NP-complete problem. If P ∈ P, then also P = NP.

Proof. Blackboard

35

NP-completeness (3)

problem: SAT

input: Boolean expression φ over variables x1, . . . , xn in conjunctive normalform, e.g.,

((x1 ∨ ¬x2) ∧ ¬x1

)question: Is φ satisfiable?

Theorem (Cook, 1970). SAT is NP-complete.

Rough proof sketch: blackboard.

Not every NP-completeness proof has to be this ingenuous. In most cases, the bestway to show NP-completeness of a problem is to use reductions.

Let P be an NP-complete problem. If we can show

P ≤ P ′ and P ′ ∈ NP

then P ′ is NP-complete as well.∗

∗why?36

Complexity of MSA

And what about multiple sequence alignment (MSA)?

Theorem. MSA is NP-complete.

Proof.

(a) MSA ∈ NP: Ex. 7.4.(b) Reduction from the following problem:

problem: SHORTEST COMMON SUPERSE-QUENCE WITH |Σ| = 2 (SCS2)

input: Finite set S of sequences overalphabet Σ with |Σ| = 2 andpositive integer m.

question: Is there a sequence s with |s| ≤m such that each t ∈ S is a sub-sequence of s?

Remainder of proof: blackboard.

(Maier, 1978)

(Karp, 1972)

(Cook, 1971)

sat

3sat

vertex cover

shortest common supersequence

(Middendorf, 1994)

shortest common supersequence with |Σ| = 2

≤

≤

≤

≤

37

NP-hardness

A language/problem L′ is NP-hard if ∀L ∈ NP : L ≤ L′. This class contains thushard problems that do not even have to be in NP.

A famous example of an NP-hard but not NP-complete∗ problem is the halting prob-lem H “given a program and its input, will it terminate?”.

E.g., SAT ≤ H, because we can take the non-deterministic code for SAT and modifyit in such a way that we

• output 1 if the input formula is satisfiable and

• go into an infinite loop otherwise.

The term NP-hardness is often extend to the corresponding constructive or evalua-tion problems of a decision problem (in the sense of Ex. 6.3)

∗Remark: H 6= NP, because all problems in NP are decidable, but the halting problem is not.38

complexity theory primer - freie universität · complexity theory primer we will discuss:...

Documents