COMP3600/6466 – Algorithms Introduction & Asymptotic Analysis

[CLRS ch. 1, sec. 2.1, 2.2, 3.1]

Hanna Kurniawati

https://cs.anu.edu.au/courses/comp3600/

Today•What is Algorithms?• The Problems we’ll focus on:• Sorting• Searching•Model of Computation• Asymptotic Analysis

What is Algorithms?• An algorithm is a well-defined

computational procedure (i.e., computational steps) that transforms inputto output for solving a well-specifiedcomputational problem•Well-defined: Can specify when & where

the procedure works and when it is not• Correctness• Space & time requirements

What is this class about?

The basics of Algorithms Design & Analysis

• Know and understand various algorithm design techniques

• Develop the basic skills to design correct & efficient (in time & space requirements) algorithms

• Is an algorithm correct?• How to predict the time & space an algorithm

required given various input size?

Why bother? • Computer capability improves fast (Moore’s

Law: Doubled every 18 month), why don’t we just wait?• Just parallelize, right?

An Example: Breadth First Search depth # Nodes Time Memory

2 110 .11 msec 107 Kbytes4 11,110 11 msec 10.6 Mbytes6 ~106 1 sec 1 Gbytes8 ~108 ~2 min 103 Gbytes10 ~1010 ~2.8 hours 10 Tbyte12 ~1012 ~11.6 days 1 Pbytes14 ~1014 ~3.2 years 99 Pbytes

Assume: Branching factor = 10, can visit 1 million nodes/sec, use 1 Kbytes/node

Taken from: Russell & Norvig, Artificial Intelligence: A Modern Approach

Why bother? • Computer capability improves fast (Moore’s Law:

Doubled every 18 month), why don’t we just wait?• The waiting time for the hardware may be more than the

waiting time to solve the problem…• Just parallelize, right? • Parallelization provides increased computation linearly, while

many problem complexity increases exponentially• Note: Of course, the above does not mean that hardware

advances do NOT matter. The two (algorithms + its software implementation and hardware) must go hand in hand. But, in this class, we will focus on the algorithms.

Why bother?•We are good at utilizing available

resources and more• We now have sensors that generates

8GB data/sec (e.g., Velodyne LiDAR)• Even if somehow we can have

unlimited time and memory requirements, we still need to ensure correctness

TodayüWhat is Algorithms?• The Problems we’ll focus on:• Sorting• Searching•Model of Computation• Asymptotic Analysis

The Problems• Recall: An algorithm is a well-defined

procedure to solve well-specified problems• In this class, we will focus only on 2

classes of problems:• Sorting • SearchingThese problems are building blocks of almost any program you see in practice today

Sorting – Problem Specification• Input: A sequence of n numbers [a1, a2, …, an]• Output: A reordering [ a1’, a2’, …, an’] of the

input sequence such that a1 ≤ a2 ≤… ≤ an (for ascending order)

Where is sorting used?• Often sorting is used as a building block of a bigger

program. For instance:• Searching. Binary search can run in O(lg n) once the keys

are sorted. • Closest pair. Given n numbers, which two have the smallest

different between them? Once the numbers are sorted, the closest pair must be next to each other. Therefore, a linear-time scan will find the solution. Closest pair problem is often encountered in various applications, including in machine learning, computational geometry, etc.

• More in ApplicationsOfSorting.pdf (see website, schedule, notes)

Searching – Problem Specification• Input: A bounded set 𝑋 with relation 𝑅 among its

elements, and a solution criteria 𝐶• Output: An element of 𝑋, 𝑥 ∈ 𝑋, that satisfies the

criteria 𝐶• Note: The above definition is very generic. In many

problems, we’ll be more specific. For instance:• What type of space is 𝑋? (e.g., is it finite or infinite?, is it

countable or uncountable?) What type of relation exists among the elements? Different algorithms are more suitable for different types of 𝑋 and 𝑅.

• The solution criteria will also influence the type of algorithms to use to be efficient.

Searching – A Problem Specification• An example for a specific search problem that will be

used often in this class• Input: A sequence of n numbers A = (a1, a2, …, an)

and a value v• Output: An index i such that v = A[i] or the special

value null if v does not appear in A

Where is searching used?•More visible than sorting. For instance:• Internet search, search in your email, search in a

document, etc.• Optimization. Computation wise, optimization

problem is a search problem (i.e., finding a value that is the largest/smallest). Many are search in continuous space (e.g., in real number space) but, still a search problem.Since almost (if not all) problems can be framed as an optimization problem, they will eventually become a search problem J

A bit more about problems• Recall: An algorithm solves well-specified problems• In reality, problems do not come as well-specified.

Someone (usually the algorithm designer) needs to formulate the problem into a well-specified problem that an algorithm can solve • In fact, good problem formulation is usually half the

solution• In this class, esp. from tutorial questions and

assignments, we’ll learn how to formulate problems• Yes, you’ll need to do this in your assignments J

TodayüWhat is Algorithms?üThe Problems we’ll focus on:üSortingüSearching•Model of Computation• Asymptotic Analysis

Model of Computation• To analyze time & memory consumption,

theoretically, we need to assume the type of machine the algorithms will be executed on• For perhaps all of this class, we’ll assume:• An abstract generic one-processor Random Access

Machine (RAM) • Abstract: We don’t consider memory swapping, garbage

collection, etc.• One-processor RAM: Instructions are executed one after

another (no concurrent operations)

Model of Computation• The following instructions take constant time

Arithmetic: Add, subtract, multiply, divide, mod, floor, ceil

Data movement: Load, store, copyControl: Conditional & unconditional branching,

subroutine call and returnReal computers have more instructions than above. We will introduce them in the RAM model if & when necessary

• Each data type in RAM has a limit. We will introduce as necessary

TodayüWhat is Algorithms?üThe Problems we’ll focus on:üSortingüSearching

üModel of Computation• Asymptotic Analysis

What is Asymptotic Analysis?• In general: Asymptotic analysis means

understanding the behavior of a function in the limit• In our class: Asymptotic analysis means

understanding how the running time (or memory consumption) of an algorithms increases with the size of the input in the limit –that is, when the input size increases without bound.

Why bother?• In general, algorithms and computer programs

are not for a single input case. Rather, for solving problems with varying input• Asymptotic analysis helps in:• Deciding which algorithms are better for solving a

specific problem• Predicting time & memory requirements as input size

grow. This could relate to the computers we need to purchase (the highest spec is not always the best, esp. when starting a business w. limited funding)

Some asymptotic notation1. Big-Oh: 𝑂 𝑔 𝑛Def.:

𝑂 𝑔 𝑛 = 𝑓 𝑛 ∃ 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠 𝑐 𝑎𝑛𝑑 𝑛8𝑠. 𝑡. 0 ≤ 𝑓 𝑛 ≤ 𝑐𝑔 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛8}

Intuitively:𝑂 𝑔 𝑛 is a set of functions that is upper bounded by a constant multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛8.We write 𝑓 𝑛 = 𝑂 𝑔 𝑛 to mean 𝑓 𝑛 ∈ 𝑂 𝑔 𝑛 , and call 𝑔 𝑛 to be an asymptotic upper bound of 𝑓 𝑛Note: this bound can be tight or not tight. The function 𝑔 𝑛is an asymptotically tight upper bound of 𝑓 𝑛 whenever for ∀ 𝑛 ≥ 𝑛8, 𝑓 𝑛 is equal to 𝑔 𝑛 to within a constant factor

Some asymptotic notation1. Big-Oh: 𝑂 𝑔 𝑛• How to figure out if 𝑓 𝑛 ∈ 𝑂 𝑔 𝑛 ? • Find the constants 𝑐 and 𝑛8• Use limit• 𝑓 𝑛 ∈ 𝑂 𝑔 𝑛 means 𝑓 𝑛 has smaller or equal growth

rate, compared to 𝑔 𝑛• 𝑓 𝑛 has smaller growth rate than 𝑔 𝑛 means lim

B→D

E BF B

= 0

• 𝑓 𝑛 has the same growth rate as 𝑔 𝑛 means limB→D

E BF B

= 𝑐′for a positive constant 𝑐′

• Example: 2𝑛 = 𝑂 𝑛 ; 2𝑛 = 𝑂 𝑛I ; 2𝑛I≠ 𝑂 𝑛 ; 2𝑛I = 𝑂 𝑛I

Just in case…• How to compute the limit?• Recall L’Hopital’s rule: lim

B→D

E BF B

= limB→D

EL BFL B

Some asymptotic notation2. Little-Oh: 𝑜 𝑔 𝑛Def.: 𝑜 𝑔 𝑛 = 𝑓 𝑛 𝑓𝑜𝑟 𝑎𝑛𝑦 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑐 > 0, ∃ 𝑎 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑛8 > 0

𝑠. 𝑡. 0 ≤ 𝑓 𝑛 < 𝑐𝑔 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛8}Intuitively:

𝑜 𝑔 𝑛 is a set of functions that is upper bounded by a constant multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛8– in a non tight mannerWe write 𝑓 𝑛 = 𝑜 𝑔 𝑛 to mean 𝑓 𝑛 ∈ 𝑜 𝑔 𝑛 , and call 𝑔 𝑛 to be a non asymptotically tight upper bound of 𝑓 𝑛

Some asymptotic notation2. Little-oh: 𝑜 𝑔 𝑛• How to figure out if 𝑓 𝑛 ∈ 𝑜 𝑔 𝑛 ? • Find the constants 𝑐 and 𝑛8• Use limit• 𝑓 𝑛 ∈ 𝑜 𝑔 𝑛 means 𝑓 𝑛 has smaller growth rate

compared to 𝑔 𝑛 , i. e., limB→D

E BF B

= 0

• Example: 2𝑛 ≠ 𝑜 𝑛 ; 2𝑛 = 𝑜 𝑛I ; 2𝑛I≠ 𝑜 𝑛 ; 2𝑛I ≠ 𝑜 𝑛I

Some asymptotic notation3. Big-Omega: Ω 𝑔 𝑛Def.:

Ω 𝑔 𝑛 = 𝑓 𝑛 ∃ 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠 𝑐 𝑎𝑛𝑑 𝑛8𝑠. 𝑡. 0 ≤ 𝑐𝑔 𝑛 ≤ 𝑓 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛8}

Intuitively:Ω 𝑔 𝑛 is a set of functions that is lower bounded by a constant multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛8.We write 𝑓 𝑛 = Ω 𝑔 𝑛 to mean 𝑓 𝑛 ∈ Ω 𝑔 𝑛 , and call 𝑔 𝑛 to be an asymptotic lower bound of 𝑓 𝑛

Some asymptotic notation3. Big-Omega: Ω 𝑔 𝑛• How to figure out if 𝑓 𝑛 ∈ Ω 𝑔 𝑛 ? • Find the constants 𝑐 and 𝑛8• Use limit• 𝑓 𝑛 ∈ Ω 𝑔 𝑛 means 𝑓 𝑛 has larger or equal growth rate,

compared to 𝑔 𝑛• 𝑓 𝑛 has larger growth rate than 𝑔 𝑛 means lim

B→D

E BF B

= ∞

• 𝑓 𝑛 has the same growth rate as 𝑔 𝑛 means limB→D

E BF B

= 𝑐′for a positive constant 𝑐′

• Example: 2𝑛 = Ω 𝑛 ; 2𝑛 ≠ Ω 𝑛I ; 2𝑛I = Ω 𝑛 ; 2𝑛I = Ω 𝑛I

Some asymptotic notation4. Little-omega: 𝜔 𝑔 𝑛Def.: 𝜔 𝑔 𝑛 = 𝑓 𝑛 𝑓𝑜𝑟 𝑎𝑛𝑦 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑐 > 0, ∃ 𝑎 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑛8 > 0

𝑠. 𝑡. 0 ≤ 𝑐𝑔 𝑛 < 𝑓 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛8}Intuitively:

𝜔 𝑔 𝑛 is a set of functions that is lower bounded by a constant multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛8– in a non tight mannerWe write 𝑓 𝑛 = 𝜔 𝑔 𝑛 to mean 𝑓 𝑛 ∈ 𝜔 𝑔 𝑛 , and call 𝑔 𝑛 to be a non asymptotically tight lower bound of 𝑓 𝑛

Some asymptotic notation4. Little-omega: 𝜔 𝑔 𝑛• How to figure out if 𝑓 𝑛 ∈ 𝜔 𝑔 𝑛 ? • Find the constants 𝑐 and 𝑛8• Use limit• 𝑓 𝑛 ∈ 𝜔 𝑔 𝑛 means 𝑓 𝑛 has larger growth rate

compared to 𝑔 𝑛 , i. e., limB→D

E BF B

= ∞

• Example: 2𝑛 ≠ 𝜔 𝑛 ; 2𝑛 ≠ 𝜔 𝑛I ; 2𝑛I = 𝜔 𝑛 ; 2𝑛I ≠ 𝜔 𝑛I

Some asymptotic notation5. Big-Theta: Θ 𝑔 𝑛Def.:

Θ 𝑔 𝑛 = 𝑓 𝑛 ∃ 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠 𝑐 𝑎𝑛𝑑 𝑛8𝑠. 𝑡. 0 ≤ 𝑐U𝑔 𝑛 ≤ 𝑓 𝑛 ≤ 𝑐I𝑔 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛8}

Intuitively:Θ 𝑔 𝑛 is a set of functions that is asymptotically upper and lower bounded by constants multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛8.We write 𝑓 𝑛 = Θ 𝑔 𝑛 to mean 𝑓 𝑛 ∈ Θ 𝑔 𝑛 , and call 𝑔 𝑛 to be an asymptotically tight bound of 𝑓 𝑛

Some asymptotic notation5. Big-Theta:Θ 𝑔 𝑛• How to figure out if 𝑓 𝑛 ∈ Θ 𝑔 𝑛 ? • Find the constants 𝑐U, 𝑐I, and 𝑛8• Use limit• 𝑓 𝑛 ∈ Θ 𝑔 𝑛 means 𝑓 𝑛 has the same growth rate as 𝑔 𝑛 , i. e., lim

B→D

E BF B

= 𝑐V for a positive 𝑐V

• Example: 2𝑛 = Θ 𝑛 ; 2𝑛 ≠ Θ 𝑛I ; 2𝑛I≠ Θ 𝑛 ; 2𝑛I = Θ 𝑛I

Some asymptotic notation6. Little-theta: ?• Actually, little-theta does not exist. And the

previous notation is usually called “Theta” rather than “Big-Theta” • Why?

Is this information useful?• The running-time complexity of an

algorithm is at least 𝑂 𝑛I

• No. Because 𝑂 𝑛I includes constant and even zero function. Stating “the running-time complexity of an algorithm is at least 𝑂 𝑛I ” can mean “the running-time complexity of an algorithm is at least epsilon or zero”, which does not provide any information.

Something to be careful about• Is 8B = 𝑂 2B ?

No. Because: 8B= 2XB and limB→D

IY Z

IY= limB→D

2B I = ∞

Therefore, 8B ≠ 𝑂 2B

• Is 𝑙𝑜𝑔 2𝑛 = 𝑂 𝑙𝑛 𝑛 ?Yes. Because:

𝑙𝑜𝑔 2𝑛 = 𝑙𝑜𝑔 2 + 𝑙𝑜𝑔 𝑛 = 𝑐U +]B B]B U8

= 𝑐U + 𝑐I𝑙𝑛 𝑛

And using L’Hopital rule, limB→D

^_`^a]B B]B B

= limB→D

baY_Y= 𝑐I

Therefore, 𝑙𝑜𝑔 2𝑛 = 𝑂 𝑙𝑛 𝑛

Example: Asymptotic Analysis of Insertion Sort• Sorting problem:• Input: A sequence of n numbers [a1, a2, …, an]• Output: A reordering [ a1’, a2’, …, an’] of the input

sequence such that a1 ≤ a2 ≤ … ≤ an

• InsertionSort(A)1. for j = 2 to A.length2. Key = A[j]3. i = j-14. While i > 0 and A[i] > key5. A[i+1] = A[i]6. i = i-17. A[i+1] = key

Comparing Algorithms: Some Useful Properties of Asymptotic Notation

[Taken from:CLRS pp. 51-52]

TodayüWhat is Algorithms?üThe Problems we’ll focus on:üSortingüSearching

üModel of ComputationüAsymptotic Analysis

Next: Recurrent Analysis and Analysis of Randomized Algorithms