comp3600/6466 –algorithms · admin •lecture time changes (should not clash with top 5 other...
TRANSCRIPT
COMP3600/6466 – Algorithms Asymptotic Analysis Cont.
[CLRS sec. 3.1]
Hanna Kurniawati
https://cs.anu.edu.au/courses/comp3600/[email protected]
Admin• Lecture time changes (should not clash with top 5
other courses students in this class take)• Monday 11.00-11.55am• Tuesday 3.00-4.55pm• Tutorial starts this week on MS Teams• 22 students have not selected their group. Please select
your group ASAP• 3 class rep. opening• Please send an email to [email protected]
to apply
Assignment 1 is Out• Due date: Monday, 17 August 23:59• Grace Period: 13 hours• Late Penalty: 100%• Issues? Contact teaching staff
@[email protected]• Submission: Via Wattle (pls. submit as draft)• If you submit as draft, you can keep updating until the
grace period ends. Once the grace period ends, the last draft in Wattle will automatically become your formal submission
Last Week• Introduction• What is Algorithms• Problem Definitions (Sort & Search)• Model of Computation• Some Math Refresher• Asymptotic Analysis• What it is• Big-Oh notation
Def.: 𝑜 𝑔 𝑛 = 𝑓 𝑛 𝑓𝑜𝑟 𝑎𝑛𝑦 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑐 > 0,∃ 𝑎 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑛! > 0 𝑠. 𝑡. 0 ≤ 𝑓 𝑛 < 𝑐𝑔 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛!}
Intuitively:𝑜 𝑔 𝑛 is a set of functions that is upper bounded by a constant multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛!– in a non tight mannerWe write 𝑓 𝑛 = 𝑜 𝑔 𝑛 to mean 𝑓 𝑛 ∈ 𝑜 𝑔 𝑛 , and call 𝑔 𝑛 to be a non asymptotically tight upper bound of 𝑓 𝑛
Asymptotic notations: Little-Oh 𝑜 𝑔 𝑛
• How to figure out if 𝑓 𝑛 ∈ 𝑜 𝑔 𝑛 ? • Find the constants 𝑐 and 𝑛!• Use limit• 𝑓 𝑛 ∈ 𝑜 𝑔 𝑛 means 𝑓 𝑛 has smaller growth rate
compared to 𝑔 𝑛 , i. e., lim"→$
% "& "
= 0
• Example:2𝑛 ? 𝑜 𝑛 ; 2𝑛 ? 𝑜 𝑛' ; 2𝑛' ? 𝑜 𝑛 ; 2𝑛' ? 𝑜 𝑛'
For comparison2𝑛 = 𝑂 𝑛 ; 2𝑛 = 𝑂 𝑛' ; 2𝑛'≠ 𝑂 𝑛 ; 2𝑛'= 𝑂 𝑛'
Membership test for Little-Oh 𝑜 𝑔 𝑛
Def.: Ω 𝑔 𝑛 = 𝑓 𝑛 ∃ 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠 𝑐 𝑎𝑛𝑑 𝑛!
𝑠. 𝑡. 0 ≤ 𝑐𝑔 𝑛 ≤ 𝑓 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛!}Intuitively:
Ω 𝑔 𝑛 is a set of functions that is lower bounded by a constant multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛!.We write 𝑓 𝑛 = Ω 𝑔 𝑛 to mean 𝑓 𝑛 ∈ Ω 𝑔 𝑛 , and call 𝑔 𝑛 to be an asymptotic lower bound of 𝑓 𝑛
Asymptotic notations: Big-Omega Ω 𝑔 𝑛
Visually,
• How to figure out if 𝑓 𝑛 ∈ Ω 𝑔 𝑛 ? • Find the constants 𝑐 and 𝑛!• Use limit• 𝑓 𝑛 ∈ Ω 𝑔 𝑛 means 𝑓 𝑛 has larger or equal growth rate,
compared to 𝑔 𝑛• 𝑓 𝑛 has larger growth rate than 𝑔 𝑛 means lim
"→$
% "& "
= ∞
• 𝑓 𝑛 has the same growth rate as 𝑔 𝑛 means lim"→$
% "& "
= 𝑐′for a positive constant 𝑐′
• Example: 2𝑛 ? Ω 𝑛 ; 2𝑛 ? Ω 𝑛! ; 2𝑛! ? Ω 𝑛 ; 2𝑛! ? Ω 𝑛!
For comparison2𝑛 = 𝑂 𝑛 ; 2𝑛 = 𝑂 𝑛' ; 2𝑛'≠ 𝑂 𝑛 ; 2𝑛'= 𝑂 𝑛'
Membership test for Big-Omega Ω 𝑔 𝑛
Def.: 𝜔 𝑔 𝑛 = 𝑓 𝑛 𝑓𝑜𝑟 𝑎𝑛𝑦 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑐 > 0,
∃ 𝑎 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡 𝑛! > 0 𝑠. 𝑡. 0 ≤ 𝑐𝑔 𝑛 < 𝑓 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛!}Intuitively:
𝜔 𝑔 𝑛 is a set of functions that is lower bounded by a constant multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛!– in a non tight mannerWe write 𝑓 𝑛 = 𝜔 𝑔 𝑛 to mean 𝑓 𝑛 ∈ 𝜔 𝑔 𝑛 , and call 𝑔 𝑛 to be a non asymptotically tight lower bound of 𝑓 𝑛
Asymptotic notations: Little-Omega 𝜔 𝑔 𝑛
• How to figure out if 𝑓 𝑛 ∈ 𝜔 𝑔 𝑛 ? • Find the constants 𝑐 and 𝑛!• Use limit• 𝑓 𝑛 ∈ 𝜔 𝑔 𝑛 means 𝑓 𝑛 has larger growth rate
compared to 𝑔 𝑛 , i. e., lim"→$
% "& "
= ∞
• Example:2𝑛 ? 𝜔 𝑛 ; 2𝑛 ? 𝜔 𝑛' ; 2𝑛' ? 𝜔 𝑛 ; 2𝑛' ? 𝜔 𝑛'
For comparison2𝑛 = 𝑂 𝑛 ; 2𝑛 = 𝑂 𝑛' ; 2𝑛'≠ 𝑂 𝑛 ; 2𝑛'= 𝑂 𝑛'
Membership test for Little-Omega 𝜔 𝑔 𝑛
Def.: Θ 𝑔 𝑛 = 𝑓 𝑛 ∃ 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠 𝑐 𝑎𝑛𝑑 𝑛!
𝑠. 𝑡. 0 ≤ 𝑐(𝑔 𝑛 ≤ 𝑓 𝑛 ≤ 𝑐'𝑔 𝑛 𝑓𝑜𝑟 ∀ 𝑛 ≥ 𝑛!}Intuitively:
Θ 𝑔 𝑛 is a set of functions that is asymptotically upper and lower bounded by constants multiplication of 𝑔 𝑛 for ``large” 𝑛 –that is, whenever 𝑛 ≥ 𝑛!.We write 𝑓 𝑛 = Θ 𝑔 𝑛 to mean 𝑓 𝑛 ∈ Θ 𝑔 𝑛 , and call 𝑔 𝑛 to be an asymptotically tight bound of 𝑓 𝑛
Asymptotic notations: Big-Theta Θ 𝑔 𝑛
Visually,
• How to figure out if 𝑓 𝑛 ∈ Θ 𝑔 𝑛 ? • Find the constants 𝑐(, 𝑐', and 𝑛!• Use limit• 𝑓 𝑛 ∈ Θ 𝑔 𝑛 means 𝑓 𝑛 has the same growth rate as 𝑔 𝑛 , i. e., lim
"→$
% "& "
= 𝑐) for a positive 𝑐)
• Example: 2𝑛 ? Θ 𝑛 ; 2𝑛 ? Θ 𝑛' ; 2𝑛' ? Θ 𝑛 ; 2𝑛' ? Θ 𝑛'
For comparison2𝑛 = 𝑂 𝑛 ; 2𝑛 = 𝑂 𝑛' ; 2𝑛'≠ 𝑂 𝑛 ; 2𝑛'= 𝑂 𝑛'
Membership test for Big-Theta Θ 𝑔 𝑛
Asymptotic notations: Little-Theta?• Actually, little-theta does not exist. And the
previous notation is usually called “Theta” rather than “Big-Theta” • Why?
Summary of Asymptotic notationsAsymptotic
bounds of 𝒇 𝒏 Upper bound Lower bound
May be tight or not 𝑂 𝑔 𝑛 Ω 𝑔 𝑛Non-tight 𝑜 𝑔 𝑛 𝜔 𝑔 𝑛
Tight Θ 𝑔 𝑛
• These notations are essentially sets of functions• Membership test:
• Finding the appropriate constants• Limit definition
Summary of Membership Tests for Asymptotic notations
Asymptotic bounds of 𝒇 𝒏 & membership test
based on limit
Upper bound Lower bound
May be tight or not𝑂 𝑔 𝑛
lim"→$
% "& "
= 0 or 𝑐′Ω 𝑔 𝑛
lim"→$
% "& "
= ∞ or 𝑐′
Non-tight𝑜 𝑔 𝑛
lim"→$
𝑓 𝑛𝑔 𝑛
= 0
𝜔 𝑔 𝑛lim"→$
% "& "
= ∞
Tight Θ 𝑔 𝑛 lim"→$
𝑓 𝑛𝑔 𝑛
= 𝑐′
𝑐′ : Positive constant
Asymptotic Analysis of Algorithms: Back to Insertion Sort Example• Recall Insertion Sort and its time complexity
InsertionSort(A)1. for j = 2 to A.length2. Key = A[j]3. i = j-14. While i > 0 and A[i] > key5. A[i+1] = A[i]6. i = i-17. A[i+1] = key
Time complexity of Insertion Sort: 𝑇(𝑛) = 𝐶𝑛' + 𝐶′𝑛 − 𝐶′′Tight asymptotic bound of the running time of Insertion Sort: ?
Useful Properties of Asymptotic Notation
[Taken from: CLRS pp. 51-52]
• The transitivity, reflexivity, symmetry, and transpose symmetry properties allows us to think of the relation between 𝑓 and 𝑔 in a way that is analogous to when we think of the relation between two real numbers 𝑎 and 𝑏, that is:• 𝑓 𝑛 = 𝑂 𝑔 𝑛 ≈ 𝑎 ≤ 𝑏• 𝑓 𝑛 = 𝑜 𝑔 𝑛 ≈ 𝑎 < 𝑏• 𝑓 𝑛 = Ω 𝑔 𝑛 ≈ 𝑎 ≥ 𝑏• 𝑓 𝑛 = 𝜔 𝑔 𝑛 ≈ 𝑎 > 𝑏• 𝑓 𝑛 = Θ 𝑔 𝑛 ≈ 𝑎 = 𝑏
A Simpler View of Asymptotic Notations
• Real numbers ordering has the trichotomy property • Exactly one of a < b, a = b, or a > b must hold• In contrast, not all functions are asymptotically
comparable. For example:• Suppose 𝑓 𝑛 = 𝑛 and 𝑔 𝑛 = 𝑛(*+," " , then 𝑓 𝑛 ≠𝑂 𝑔 𝑛 and 𝑓 𝑛 ≠ Ω 𝑔 𝑛
• The two inequality above means, 𝑓 𝑛 ≠ Θ 𝑔 𝑛 , 𝑓 𝑛 ≠𝑜 𝑔 𝑛 , and 𝑓 𝑛 ≠ 𝜔 𝑔 𝑛
• Why 𝑓 𝑛 ≠ 𝑂 𝑔 𝑛 and 𝑓 𝑛 ≠ Ω 𝑔 𝑛 ?
Though, there’s a catch…
𝑔 𝑛 = 𝑛(*+," "
𝑓 𝑛 = 𝑛
𝑛
𝑔(𝑛)
𝑓(𝑛)
Comparing Algorithms• Recall asymptotic analysis provides
information about the growth trend of computational resources required by an algorithm as the input grow• Such information and the simpler view of
asymptotic analysis in the previous slides means we can use asymptotic analysis to compare algorithms easily
Comparing Algorithms: An Example• Suppose someone proposes an algorithm called
AltSort as an alternative to InsertionSort, and suppose the running-time of AltSort is 𝑇"#$%&'$ 𝑛• How do we compare InsertionSort and AltSort?• We can think of 𝑓 𝑛 = 𝑇-"+./0,1"21/0 𝑛 and 𝑔 𝑛 =𝑇34021/0 𝑛 , and check if any of the asymptotic relation between 𝑓 𝑛 and 𝑔 𝑛 holds.
• If 𝑓 𝑛 = 𝑜 𝑔 𝑛 , we say that the running-time of InsertionSort is asymptotically lower than AltSort
• If 𝑓 𝑛 = Θ 𝑔 𝑛 , we say that the running-time of InsertionSort is asymptotically equivalent to AltSort, or they are in the same efficiency class
Basic Asymptotic Efficiency Class(From Small to Large)
Class Name1 constant
𝑙𝑜𝑔 𝑛 logarithmic
𝑛 linear
𝑛 𝑙𝑜𝑔 𝑛 n-log-n
𝑛' quadratic
𝑛5 cubic
2" exponential
𝑛! factorialTaken from: [Lev] pp. 59
Basic Asymptotic Efficiency Class(From Small to Large)
Class Name1 constant
𝑙𝑜𝑔 𝑛 logarithmic
𝑛 linear
𝑛 𝑙𝑜𝑔 𝑛 n-log-n
𝑛' quadratic
𝑛5 cubic
2" exponential
𝑛! factorialTaken from: [Lev] pp. 59
Where should 𝑙𝑜𝑔 𝑛! be?
Where should poly-log function be?
Floors & Ceilings • Most analysis of algorithms are concerns only with
integer number as input• This often means, we need to use floor/ceiling functions• Below are some arithmetic manipulations of floor/ceiling
functions
Taken from [CLRS] sec. 3.2
But, we’ll be a bit Sloppy• In most of our analysis in this class, we’ll be a bit
sloppy and just assume the numbers are always integer (i.e., ignore the ceil/floor). • This will help convey the main component of the
analysis clearer• And, since we are concern more with the trend of the
function’s growth and on upper/lower bound, this “sloppiness” will in general be fine
Next: Correctness