2 ALGORITHMS AND COMPLEXITY

ALGORITHMS AND THE MODEL OF COMPUTATION

Combinatorial optimization problems are always algorithmicallysolvable because the solution space is finite.

Measure of algorithmic EFFICIENCY = COMPUTATION TIME

(Space requirements not considered here.)

Computation time is linearly proportional to the elementarycomputational steps. We can define:

Computation time = the number of elementary steps / operations of the algorithm

Computational model for algorithms: the Turing machine

Every algorithm for any type of computer can be implemented by auniversal Turing machine so that the numbers of operations arepolynomially related.

Single tape Turing machine

Operations of a single tape Turing machine are determined by thestate of the machine, symbols on the tape and a transition function:

"If the state is s and the symbol in the current tape position is x, moveto state s', write symbol x' and move the read-write head one step tothe right/left".

The transition function that determines the steps of the machinecorresponds to a computer program.

The Turing machine solves the given problem if it halts (i.e. stops in aspecific final state or halting state) after a finite number of steps withthe correct solution written on its tape.

The solution to a given problem can be- final state qY = yes or qN = no- the answer written to the tape when a halting state has been reached.

CLASSES OF COMPUTATIONAL COMPLEXITY

OPTIMIZATION PROBLEM:A class of similar problems with different input parameters.

Each individual case with fixed parameter values is called aninstance of the problem.

Example: Input for an instance of the TSP problem is given bythe number of vertices and the distances between vertices.

The computation time required to obtain a solution depends on• the problem• size of the instance• parameter values of the instance• the algorithm• the computer and implementation of the algorithm

In complexity theory, algorithms and problems are classified bythe growth order of computation time as a function of instancesize.

Definition 2.1The worst case time complexity or a complexity of an algorithmA is the time TA(n) that is needed at most for solving a probleminstance of size n.

Exact number is impossible or difficult to determine asymptotic upper bound will be used.

Definition 2.2The order of growth of a function f(n) is O(g(n)), if there arepositive constants a and N such that f(n) a g(n) for every n N

Examples:• f1(n) = a0+a1n+a2n2+...+ aknk f1(n) = O(nk)• f2(n) = 2n3+106n log n f2(n) = O(n3).

Symbol O is used for an upper bound for complexity TA(n).

Notation: TA(n) = O(g(n)) means the complexity of algorithm Ais of order O(g(n)).

Essential categories:

Definition 2.3• Algorithm A is polynomial, if its complexity TA(n) = O(nk)

for some constant k.• Algorithm A is exponential, if it is not polynomial.

Examples of orders of exponential algorithms: O(nlog n), O(2n),O(kn) where k>1, O(n!) and O(nn).

Term “superpolynomial” considered exponential.

Logarithmic order of complexity TA(n)=O(logkn): lower thanpolynomial.

Polynomial algorithms can be combined, concatenated or nestedand the resulting algorithm is still polynomial.

For instance, when a polynomial algorithm A uses polynomialalgorithm B as a subroutine so that the number that B is invokedis bounded by a polynomial, then A is polynomial.

SIZE OF THE PROBLEM = number of symbols needed topresent the problem input, length of the input string

• Size is not unique: dependent on the encoding scheme of aproblem.

• Encoding should be as concise (short) as possible.• Only the order of the size is relevant.• Sizes of all "reasonable", nonredundant encodings are

polynomially related.• Size may be a polynomial function of few problem

parameters, e.g. number of vertices and/or edges in a graph,number of variables, matrix dimensions Complexity canbe given as a function of these parameters only.

COMPARISON OF SOME COMPLEXITY CLASSES

f(n) n=10 n=100 n=1000n 10 100 1000n log n 33 664 9966n3 1000 106 109

106n8 1014 1022 1030

nlog n 2099 1.93· 1013 7.89· 1029

2n 1024 1.27· 1030 1.05· 10301

n! 3628800 10158 4· 102567

(log = log2)

Assume that the computer can execute 106 operations / second.

Table: Maximum size n of problems solvable in 1 second and 3years by algorithms having computation time shown on the left:

f(n) 1 s 108 s 3 yrsn 106 1014

n log n 6 · 104 2 · 1012

n3 100 46415106n8 1 10nlog n 22 1122n 19 46n! 9 16

Example 2.1Comparison of complexity orders:Assume a problem can be solved by three different algorithmsA1, A2 and A3.

An instance of size n=10 takes (in the worst case)• one hour by the O(log n)-algorithm A1• one minute by the O(n5)-algorithm A2• one second by the O(2n)-algorithm A3

If the size of an instance is n=100, an upper bound for theexecution time by

• the logarithmic algorithm A1 is 2 hours• the polynomial algorithm A2 is 69.4 days• the exponential algorithm A3 is 1019 years

Is O(106n8) better than O(1.2n)?

A polynomial algorithm doesn't always beat an exponential one,because- the upper bounds may not be tight- worst cases may be rare- the implementation of the method may have a significant influence in the computation time- the problem size may be moderate enough

Example: 106n8 < 1.2n when n 331

Example: Linear programming problem (LP) can be solved by• simplex method: exponential• Karmarkar's interior point method: polynomial

Simplex works well even for large problems:- computer codes developed for decades- hard instances are rare - the average computation time does not show exponential growth.

COMPLEXITY CLASSIFICATION OF OPTIMIZATIONPROBLEMS

Definition 2.4The complexity of an optimization problem X is polynomial,if there is a polynomial algorithm that solves problem X.

To show polynomial complexity• define the algorithm• verify that it gives the correct solution to every instance of

the problem• show that the computation time for instances of size n is

always O(nk) for some k.

Instead of worst case complexity, average case complexity couldbe defined, based on a distribution of problems or solution times.

TYPES OF PROBLEMS

The algorithm complexity theory was originally developed fordecision or recognition problems.

DECISION PROBLEM = RECOGNITION PROBLEM:

Decide whether or not an instance, encoded as a string ofsymbols, satisfies the given statement or condition

That is,

Decide/recognize whether the instance belongs to the set of yesinstances of the problem.

Examples of decision or recognition problems:

1) Input: Integer MQuestion: Is M a prime number?

2) Input: n integers x1, x2,...,xnQuestion: Can the numbers be partitioned into two sets such thatthe sums of the integers in both sets are the same?(Partition problem)

3) Input: Graph G=(V,E).Question: Does the graph G have a Hamiltonian path (i.e. a paththat passes through every vertex exactly once)?

4) Input: Number of vertices n, distances between all vertices cij,integer L.Question: Does the graph contain a cycle through all the vertices(a Hamiltonian cycle) with total length L?

5) Input: n objects with weights wi, i=1,...,n, weight capacity Wof a container, number of containers k.Question: Can the objects be placed in the k containers withinthe capacity limit?

6) Input: Graph G=(V,E), integer k.Question: Does the graph contain a clique of k vertices?

7) Input: m×n matrix of integers A, m×1 vector of integers b.Question: Is there an integer (or binary) vector x = (x1,...,xn)T

such that Ax = b?

Many of these decision problems have an optimizationcounterpart:

Problem 4: TSPProblem 5: Bin Packing ProblemProblem 6: Maximum Clique ProblemProblem 7: Integer Linear Programming Problem

Three different versions of the combinatorial optimizationproblem

min f(x)subject to x∈S

(1) Optimization problem:Find a feasible solution x* that minimizes the objectivefunction f.

(2) Evaluation problem:Find the minimum value f* of the objective function.

(3) Decision problem, recognition problem:Is there a feasible solution x∈S such that f(x) L (a giventhreshold value).

The first is the “hardest”, the third is the “easiest”, in thefollowing sense:

If the optimization problem is solved then the solution to theevaluation problem can be calculated and the answer to thedecision / recognition problem is obvious.

Is there any real difference in the difficulty of these problems?

COMPLEXITY CLASSES P AND NP

Definition 2.5. The class P consists of those decision problemsthat can be solved with a polynomial algorithm.

P = the set of all polynomially solvable decision problems.

The solution to a recognition problem is yes or no. If the solutionis yes, the algorithm is said to accept the given input.

The problem belongs to the class P if and only if the followingcondition holds:

There is a deterministic Turing machine such that when an instance of size n is given as its input, the machine stops in an accepting state (yes state) after a polynomial number p(n) of steps if and only if answer for the instance is yes.

Decision problems in P:

- LP (Linear Programming problem)- Assignment Problem- Minimum Spanning Tree Problem- Shortest Path Problem- several one machine scheduling problems- Perfect Matching- Edge Cover

If the answer to the posed yes/no question is yes, a verificationof the result is a proof based on a feasible arrangement x.

For instance, if a TSP solution x, i.e. a cycle through all verticeswith length L is given or guessed, the verification means:

1. Check that the solution is a cycle (if not obvious) andevery vertex is included exactly once.

2. Calculate the length of the cycle and compare it with thethreshold value L.

The given solution x acts as a certificate of the answer. Thenumber of steps needed is a polynomial function of the numberof vertices.

Informally:

The class NP (Nondeterministic Polynomial problems) consists of decision problems for which a yes instance can be verified in polynomial time.

More formally:

Definition 2.6.a The class NP consists of decision problems for which every yes instance y has a polynomial size certificate c(y) that can be checked in polynomial time to give answer yes.

Still more formally:

Definition 2.6.b The class NP consists of decision problems for which there exists a polynomial p(n) and an algorithm A such that an instance, encoded as a string y, is a yes instance if and only if • there exists a string c(y), the certificate, with size |c(y)| p(|y|) • given input (y, c(y)), the algorithm A will stop with the answer yes after at most p(|y|) steps.

Nondeterministic algorithm (nondeterministic Turing machine)

Guessing (nondeterministic) stage:For a given instance string y, the algorithm generates a randomclue or guess c(y) and uses the concatenated string (y, c(y)) as itsinput.

Verification / checking (deterministic) stage:When the right certificate c(y) is generated for a yes instance, thealgorithm stops with the answer yes. The right certificate mustalways have a positive probability.

There are other descriptions for the nondeterministic Turingmachine.

Definition using the nondeterministic TM:

Definition 2.6.c The problem belongs to NP if there is a nondeterministic algorithm that 1) stops at an accepting state if and only if the instance

satisfies the given statement and the right polynomial sizecertificate is given as a clue

2) takes a number of steps that is bounded by a fixedpolynomial function of the size of the instance.

P ⊆ NP:Polynomially solvable problems can be solved by anondeterministic algorithm: use the polynomial algorithm as thechecking algorithm and ignore the clue.

The class NP contains the decision versions of virtually allcombinatorial optimization problems.That is because the solution to a combinatorial optimizationproblem is usually an arrangement, design or sequence of objectsthat can be coded as a concise certificate to a given question.

NP-COMPLETENESS

Definition 2.7.aTURING REDUCTIONDecision problem Q1 reduces polynomially to decision problem Q2, ifthe following condition holds:If there exists a polynomial algorithm A2 for problem Q2, then thereexists a polynomial algorithm A1 for problem Q1.

Denoted by Q1∝T Q2.

Definition 2.7.bKARP REDUCTION or POLYNOMIAL TRANSFORMATIONDecision problem Q1 transforms polynomially to decision problemQ2, if there exists an algorithm that transforms an instance y1 ofproblem Q1 to an instance y2 of problem Q2 such that1) the transformation algorithm is polynomial in the size of y12) y1 is a yes instance of Q1 if and only if y2 is a yes instance of Q2

Denoted by Q1∝ Q2.

Open question:

Are the definitions 2.7.a and 2.7.b equivalent?

Obviously: 2.7.b. ⇒2.7.a.

Definition 2.8 Decision problem Q is NP-hard, if every problem of the class NP reduces polynomially to it.

Definition 2.9 Decision problem Q is NP-complete, if Q∈ NP and Q is NP-hard. The class of NP-complete problems is denoted by NPC.

Problem Q is NP-complete, if1) Q ∈ NP i.e. its solution can be verified in polynomial time2) every problem in the class NP reduces polynomially to it.

An equivalence class NPC:Every NPC-problem reduces polynomially to every other NPCproblem.

Theorem 2.1 PROPERTIES OF NP-COMPLETE PROBLEMS:

(1) If there exists a polynomial algorithm to one NP-completeproblem, then there is a polynomial algorithm to everyNP-complete problem, that is,

EITHER all NP-complete problems are polynomially solvableOR none of them is.

(2) If decision problem Q is NP-complete, then Q ∈ P if and onlyif NP = P.

(3) Assume Q is NP-complete. If R∈ NP and Q∝R, then R isNP-complete.

• No polynomial algorithm has been found to any NP-completeproblem.

• Neither has it been proved that they didn't exist.

Is P = NP ?

Probably not!

The first provably NP-complete problem:

Satisfiability Problem:

SATInput: Boolean expression, that consists of• literals from a set of boolean (logical) variables x1,...,xn and theircomplements ix• connectives ∧ (AND), ∨(OR) and parentheses ( , ).Question: Is the expression satisfiable i.e. is there a truth valueassignment of the variables such that the expression is TRUE?

S.A.Cook (1971):Every decision problem in NP reduces polynomially to thesatisfiability problem SAT.

SAT ∈ NP:If a correct truth value assignment for a satisfiable expression is given,the truth value can be calculated with a number of steps that ispolynomial in the lenght of the expression.

⇒SAT is NP-complete

Example of a satisfiable boolean expression:

)xxx()xxx()xxx()xxx( 321421431321 ∨∨∧∨∨∧∨∨∧∨∨

A truth value assignment that makes the expression TRUE (=1):x1=1, x2=0, x3=1, x4=1.

Every boolean expression C can be formulated in a conjunctivenormal form

C = c1 ∧ c2 ∧ ... ∧ ck,

in which the clauses ci are disjunctions of the literals i.e. of the form

y1∨y2∨ ...∨ym

where yj∈{ n21n21 x,...,x,x,x,...,x,x }.

NP-complete decision problems:

- Traveling Salesman Problem (TSP)- Hamiltonian Cycle Problem- Chinese Postman Problem in mixed graphs- Knapsack Problem- Bin Packing Problem- Set Covering- Clique Problem- Vertex Cover Problem- Vertex Coloring Problem- Partition Problem- Integer Linear Programming (ILP), Binary Integer Programming (BIP)

How to prove that a decision problem R is NP-complete?

1) Show that the problem is in NP.

2) Make up a polynomial transformation from a known NPC problem.Use transitivity: If P∝Q and Q∝R, then P∝R.

Reduction chains: SAT ∝ Q1 ∝ Q2...

Every NP-problem reduces polynomially to SAT⇒All problems in the chain Q1, Q2,... are NP-complete (when in NP).

Garey & Johnson: Computers and Intractability: A Guide to theTheory of NP-Completeness (1972)lists over 300 NPC problems.

Example 2.2Decision problem CLIQUE is NP-complete:

CLIQUEInput: Graph G=(V,E) and integer k.Question: Does there exist clique of size k in the graph i.e. a

completely connected subgraph of k vertices?

Proof:

1) CLIQUE ∈ NP:

If a subset of vertices is given as a guess, it can be checked in apolynomial time that the number of vertices is k and they are allpairwise connected.

2) Proof of SAT ∝ CLIQUE:

An arbitrary instance of SAT problem is polynomially transformed toan instance of the CLIQUE problem so that the SAT-instance issatisfiable if and only if a clique of specified size exists in theconstructed graph.

Set of boolean variables X = {x1,x2,...,xn}

Given expression in a conjunctive normal formC = c1 ∧ c2 ∧ ... ∧ ck

Construct a graph G = (V,E), with• vertex set V = {( ,i) | is a literal in clause ci }• edge set E = {[( ,i), ( ,j)] | i j and β}.

The number of vertices |V| is of the same order as length of theexpression C. The number of edges |E| is of order |V|2.⇒The transformation can be executed in polynomial time with respectto the size of the instance of SAT.

The question for the CLIQUE problem will be:

Is there a clique of size k (the number of clauses) in our graph?

Proof of "SAT yes ⇒ CLIQUE yes":

Assume C is satisfiable with a given truth value assignment

⇒ Every expression ci is TRUE

⇒ Each ci must have at least one literal with value TRUE.

Let j(i) be a true literal of clause ci⇒Vertices ( j(1),1), ( j(2),2),...,( j(k),k) form a clique of size k.

Explanation:

When i m, there is an edge between vertices ( j(i),i) and ( j(m),m),because if both literals are true, j(i) )m(jα .

Proof of "CLIQUE yes ⇒ SAT yes"

Assume a k-clique in G.⇒Every vertex in the clique corresponds to a unique clause ci with itsliteral j(i).

Truth value assignment:• Literals corrsponding to the members of the clique := TRUE• Their complements := FALSE• Nonexistent variables := anything

Then every clause ci is TRUE and there are no conflicting clausesbecause j(i) )m(jα for all pairs of vertices in the clique.

Example: SAT instance for variables X = {x1, x2, x3}

C = )xx()xx()xx( 313221 ∨∧∨∧∨

Corresponding graph G=(V,E):

The graph has a 3-cliques, e.g. of nodes {(x1,1), (x2,2), ( 3x ,3)}.Set x1:=TRUE, x2:=TRUE and x3:=FALSE expression is TRUE.

Conclusion:

We have proved that the boolean expression C = c1 ∧ c2 ∧... ∧ ckis satisfiable if and only if the corresponding graph G(V,E) has ak-clique.

Because SAT is NP-complete, so is CLIQUE, by Theorem 2.1. (3).

COMPLEXITY OF OPTIMIZATION PROBLEMS

NP-completeness was defined for decision problems. Many of thedecision problems originate from an optimization problem:the decision problem is a threshold value analog of the optimizationproblem.

If the optimization problem is solved, the solution to the decisionproblem is straightforward.

Example: If an optimization algorithm gives the optimal TSP tour,we can compute the length of the tour and compare it to the giventhreshold value L in polynomial time.

Turing reduction: If the optimization problem could be solved bya polynomial algorithm, then we could solve the decision analog inpolynomial time.

All decision problems in NP reduce polynomially to an optimizationversion of an NP-complete problem.

⇒

An optimization version of an NP-complete decision problem isNP-hard.

If a polynomial solution algorithm to one of these NP-hardoptimization problems would exist, then all decision problemsin the class NP would be solved in polynomial time.

Optimization problems are at least as hard as their decisioncounterparts.

Opposite question:

Does an optimization problem reduce polynomially to thecorresponding decision problem?

Turing reduction: polynomial transformation is not suitable!

Evaluation problem ∝ Decision problem by binary search:

Suppose the optimum value f* ∈ [0, 2p(n)],where p(n) is a polynome of the instance size n.

The value f* can be solved by a binary search = interval halvingmethod, using p(n) calls of the decision algorithm.

Example 2.3Suppose an upper bound for a TSP tour is 32768 = 215.

A given algorithm AQ will answer the question "Does the graphcontain a cycle through all the vertices with total length L?"

Set L := 32768 / 2 = 16384.If the answer is•yes, set L:= 16384 / 2 = 8192•no, set L:= 16384 + (16384 / 2) = 24576.

Continuing the process, f* will be found in log2(32768) = 15 calls ofthe algorithm.

There is no general procedure for reducing the optimization problemto the evaluation (or decision) problem; it depends on the problem.

In some cases, dynamic programming can be used to determine thesolution x* by means of the optimum value f*.

Example 2.4 The Knapsack problem:Choose a subset of given n object with weights w1,...,wn and valuesv1,...,vn such that the total weight does not exceed the capacity M ofthe knapsack and the total value is maximized.

Decision variables:

xi =

notif0chosenisiobjectif1

A binary linear optimization model:

max f = v1x1 + v2x2 + ... + vnxns.t.

w1x1 + w2x2 + ... + wnxn M

xi = 0 or 1

Assume evaluation algorithm Af solves the maximum f value f* of theknapsack.

Set the value of each variable in turn to xi=0 and xi=1 and solve theknapsack value for the remaining n–1 variables using capacities Mand M–wi.

Resulting objective function values: f1 (for xi=0) and f2 (for xi=1).

If f1=f*, then xi*=0.If f2+vi = f*, then xi*=1.

The optimum solution vector x* is determined by 2n executions of theevaluation algorithm Af.

Alternative: After fixing x1, modify the problem for the remainingvariables and solve for x2=0 and x2=1, etc. In each subsequentproblem set M as the remaining capacity after the fixed variables.

If the evalution algorithm Af were polynomial, then the optimizationalgorithm would be polynomial.

Knapsack optimization ∝T Knapsack evaluation

Transitivity of reduction ⇒

Knapsack optimization ∝T Knapsack decision

Most of the NP-complete optimization problems can be polynomiallyreduced to the corresponding decision problems.

Example 2.5TSP optimization ∝T TSP decision

Instance of TSP:n vertices, distances in n×n matrix C, threshold value L.

Algorithm TSPDECISION(n,C,L) solves the decision problem

"Does there exist a TPS tour with length L?"

The following algorithm solves the optimal TSP tour using subroutineTSPDECISION.

Algorithm TSPTOUR(n,C)

low:=0high:=n max {ci,j}while low high do L:= 2/)highlow( + if TSPDECISION(n,C,L)='yes'

then high:=L else low:=L+1

opt:=high

for i:=1 to n do for j:=1 to n do

remember:=cij cij:= n max {cij} + 1 if TSPDECISION(n,C,opt)='no'

then cij:=remember

First part solves the optimum tour length by a binary search.

The other part tests for each edge if that edge belongs to the optimumtour.

In the resulting matrix C, the edges that keep their original values cijbelong to the optimal tour. For other edges, cij = n max {cij} + 1.

The subroutine TSPDECISION is called 2max2 n)cn(log + times

which is polynomial in the size of the input.

CONCLUSION:

Optimization problems associated with certain NP-complete decisionproblems are Turing reducible to their decision counterparts, andvice versa.

These optimization problems are called

NP-complete (optimization problems)or

NP-equivalent = NP-hard & NP-easy

HOW TO SOLVE AN NP-COMPLETE PROBLEM:GENERAL PRICIPLES

When confronted with a known NP-complete problem, large or small,a solution is needed anyway.

For instance, a vehicle routing plan or a flight crew schedule must beavailable for the decision makers.

1) Exact exponential algorithms•Worth trying if an implemented algorithm is readily available.• May be exponential in the worst case, but resonable in the average(e.g. several branch & bound methods).• If a so called pseudopolynomial algorithm exists, it is preferablealthough not polyomial.

2) Efficient approximation algorithms• Polynomial time methods.• Bound for the objective value given.• With good luck, optimal solution may be found.

3) Heuristics• Based on reasonable principles and rules of thumb but cannotguarantee optimum.• Should be fast and easy to implement.• If the solution is not satisfactory, use it as a starting point for ahigher level method.• Belongs to the previous category if a bound for the solution valuecan be given.

4) Tractable special cases• Some of the NP-complete problems have special cases that arepolynomially solvable. Does your problem have special characteristicsor structure that make it easier?

E.g. the clique problem for planar graphs can be decided efficiently,because a planar graph cannot have a clique of size 5.

5) Tractable relaxations• Some of the restrictions of the original problem have been droppedto make the problem tractable.• The solution set includes the solutions of the original problem.• The solution to the relaxed problem can be used as a starting point.