computational complexity - heuristic algorithms complexity heuristic algorithms ... polynomial and...

Download Computational complexity - Heuristic Algorithms complexity Heuristic Algorithms ... Polynomial and exponential complexity In combinatorial optimization, the main distinction is between

Post on 06-Feb-2018




0 download

Embed Size (px)


  • Computational complexityHeuristic Algorithms

    Giovanni Righini

    University of Milan

    Department of Computer Science (Crema)

  • Definitions: problems and instances

    A problem is a general question expressed in mathematical terms.Usually the same question can be expressed on many examples:they are instances of the problem.For instance:

    Problem: Is n prime?

    Instance: Is 7 prime?

    A solution S is the answer corresponding to a specific instance.Formally, a problem P is a function that maps instances from a set Iinto solutions (set S):

    P : I S

    A priori, we do not know how to compute it: we need an algorithm.

  • Definitions: algorithms

    An algorithm is a procedure with the following properties:

    it is formally defined

    it is deterministic

    it made of elementary operations

    it is finite.

    An algorithm for a problem P is an algorithm whose steps aredetermined by an instance I I of P and produce a solution S S

    A : I S

    An algorithm defines a function and it also computes it.

    If the function is the same, the algorithm is exact; otherwise, it isheuristic.

  • Algorithms characteristics

    A heuristic algorithm should be

    1. effective: it should compute solutions with a value close to theoptimum;

    2. efficient: its computational complexity should be low, at leastcompared with an exact algorithm;

    3. robust: it should remain effective and efficient for any possibleinput.

    To compute a solution, an algorithm needs some resources. The twomost important ones are

    space (amount of memory required to store data);

    time (number of elementary steps to be performed to computethe final result).

  • Complexity

    Time is usually considered as the most critical resource because:

    time is subtracted from other computations more often thanspace;

    it is often possible to use very large amounts of space at a verylow cost, but not the same for time;

    the need of space is upper bounded by the need for time,because space is re-usable.

    It is intuitive that in general the larger is an instance, the larger is theamount of resources that are needed to compute its solution.However how the computational cost grows when the instance size

    grows is not always the same: it depends on the problem and on thealgorithm.By computational complexity of an algorithm we mean the speed withwhich the consumption of computational resources grows when thesize of the instance grows.

  • Measuring the time complexity

    The time needed to solve a problem depends on:

    the specific instance to be solved

    the algorithm used

    the machine that executes the algorithm

    . . .

    We want a measure of the time complexity with the followingcharacteristics:

    independent of the technology, i.e. it must be the same when thecomputation is done on different hardware;

    synthetic and formally defined, i.e. it must be represented by asimple and well-defined mathematical expression;

    ordinal, i.e. it must allow to rank the algorithms according to theircomplexity.

    The observed computing time, does not satisfy these requirements.

  • Time complexity

    The asymptotic worst-case time complexity of an algorithm providesthe required measure in this way:

    1. we measure the number T of elementary operations executed(which is computer-independent);

    2. we compute a number n which determines the number of bitsneeded to define the size of any instance (e.g., the number ofelements in the ground set in a combinatorial optimizationproblem);

    3. we find the maximum number of elementary operations neededto solve instances of size n

    T (n) = maxIIn

    T (I) n N

    (this reduces the complexity to a function T : N N)

    4. we approximate T (n) with a simpler funcion f (n), for which weare only interested in the asymptotic trend for n +(complexity is more important when instances are larger)

    5. finally we can collect these functions in complexity classes.

  • Notation:

    T (n) (f (n))

    means that

    c1, c2 R+, n0 N : c1 f (n) T (n) c2 f (n) for all n n0

    where c1, c2 and n0 are constant values, independent on n.

    T (n) is between c1f (n) and c2f (n)

    for a suitable small value c1 for a suitable large value c2 for any size larger than n0

    T (n)





    c f(n)


    c f(n)



    Asymptotically, f (n) is an estimate of T (n) within a constant factor:

    for large instances, the computing time is proportional to f (n).

  • Notation: O

    T (n) O (f (n))

    means that

    c R+, n0 N : T (n) c f (n) for all n n0

    where c and n0 do not depend on n.

    T (n) is upper bounded by cf (n)

    for a suitable large value c

    for any n larger than a suitablen0

    T (n)



    c f(n)



    n0Asymptotically, f (n) is an upper bound for T (n) within a constantfactor:

    for large instances the computing time is at most proportional tof (n).

  • Notation:

    T (n) (f (n))

    means that

    c > 0, n0 N : T (n) c f (n) for all n n0

    where c and n0 do not depend on n.

    T (n) is lower bounded by cf (n)

    for some suitable small valuedi c

    for any n larger than n0

    T (n)





    c f(n)

    n0Asymptotically, f (n) is a lower bound of T (n) within a constant factor:

    for large instances the computing time is at least proportional tof (n)

  • Combinatorial optimization

    In combinatorial optimization problems it is natural to define the sizeof an instance as the cardinality of its ground set. An explicit

    enumeration algorithm

    considers each subset S E ,

    evaluates whether it is feasible (x X ) in (n) time,

    evaluates the objective function f (x) in (n) time,

    records the best value found.

    Since the number of solutions is exponential in n, its complexity is atleast exponential, even if (n) and (n) are polynomials (as oftenoccurs).

  • Polynomial and exponential complexity

    In combinatorial optimization, the main distinction is between polynomial complexity: T (n) O



    for a constant d > 0 exponential complexity: T (n) (dn) for a constant d > 1

    The algorithms of the former type are efficient; those of the latter typeare inefficient.

    In general, heuristic algorithms are polynomial and they are usedwhen the corresponding exact algorithms are exponential.

    Assuming 1 operation/secn n2 op. 2n op.1 1 sec 2 sec10 0.1 msec 1 msec20 0.4 msec 1 sec30 0.9 msec 17.9 min40 1.6 msec 12.7 days50 2.5 msec 35.7 years60 3.6 msec 366 centuries

  • Problem transformations and reductions

    Some times it is possible and convenient to reformulate an instanceof a problem P into an instance of a problem Q and then to transformback the solution of the latter into a solution of the former.

    Polynomial transformation P Q: given any instance of P

    a corresponding instance of Q is defined in polynomial time

    the instance of Q is solved by a suitable algorithm, providing asolution SQ

    from SQ a corresponding solution SP is obtained in polynomialtime

    Example: VCP SCP, MCP MISP and MISP MCP.

  • Problem transformations and reductions

    Polynomial reduction P Q: given any instance of P

    an algorithm A is executed a polynomial number of times;

    to solve instances of a problem Q obtained in polynomial timefrom the instance of Pand from the results of the previous runs;

    from the solutions computed, a solution of the instance of P isobtained.

    Examples: BPP PMSP and PMSP BPP.

    In both cases

    if A is polynomial/exponential, the overall algorithm turns out tobe polynomial/exponential

    if A is exact/heuristic, the overall algorithm turns out to beexact/heuristic

  • Optimization vs. decision

    A polynomial reduction links optimization and decision problems.

    Optimization problem: given a function f and a feasible region X ,what is the minimum of f in X?

    f = minxX

    f = ?

    Decision problem: given a function f , a value k and a feasibleregion X , do solutions with a value not larger than k exist?

    x X : f (x) k?

    The two problems are polynomially equivalent:

    the decision problem can be solved by solving the optimizationproblem and then comparing the optimal value with k ;

    the optimization problem can be solved by repeatedly solving thedecision problem for different values of k , tuned by dichotomoussearch.

  • Drawbacks of worst-case analysis

    The worst-case time complexity has some relevant drawbacks:

    it does not consider the performance of the algorithm on theeasy/small instances; in practice the most difficult instancescould be rare or unrealistic;

    it provides a rough estimate of the computing time growth, not ofthe computing time itself;

    the estimate can be very rough, up to the point it becomesuseless;

    it may be misleading: algorithms with worse worst-casecomputational complexity can be very efficient in practice, evenmore than algorithms with better worst-case computationalcomplexity.

  • Other complexity measures

    To overcome these drawbacks one could employ different definitionsof computational complexity:

    parameterized complexity expresses T as a function of someother relevant parameter k besides the size of the instance n:T (n, k)

    average-case complexity assumes a probability distribution on Iand it evaluates the expected value of T (I) on In

    T (n) = E [T (I) |I


View more >