lesson 1 .1 - introduction
TRANSCRIPT
-
8/11/2019 Lesson 1 .1 - Introduction
1/115
Why Study Algorithms
CS 854 Advance Algorithm Analysis
Anis ur RahmanDepartment of Computing
NUST-SEECSIslamabad
September 10, 2014
1/115
-
8/11/2019 Lesson 1 .1 - Introduction
2/115
Course Information
1 Course Information
2 Algorithm analysis
3 Algorithm analysis
4 Big-O notation
2/115
-
8/11/2019 Lesson 1 .1 - Introduction
3/115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithmsBig-Oh notationhigh level reasoning about algorithmsgeneral analysis methods (Master Method)
3/115
-
8/11/2019 Lesson 1 .1 - Introduction
4/115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithmsBig-Oh notationhigh level reasoning about algorithmsgeneral analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,closest pair, etc.
4/115
-
8/11/2019 Lesson 1 .1 - Introduction
5/115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithmsBig-Oh notationhigh level reasoning about algorithmsgeneral analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
5/115
-
8/11/2019 Lesson 1 .1 - Introduction
6/115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithmsBig-Oh notationhigh level reasoning about algorithmsgeneral analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
4 Primitives for reasoning about graphs
connectivity information, shortest paths, structure of informationand social networks.
6/115
C I f i C T i
-
8/11/2019 Lesson 1 .1 - Introduction
7/115
Course Information Course Topics
This course is about
1 Vocabulary for design and analysis of algorithmsBig-Oh notationhigh level reasoning about algorithmsgeneral analysis methods (Master Method)
2 Divide and conquer algorithm design paradigm
applied to integer multiplication, sorting, matrix multiplication,closest pair, etc.
3 Randomization in algorithm design
applied to QuickSort, primality testing, graph partitioning, hashing.
4 Primitives for reasoning about graphs
connectivity information, shortest paths, structure of informationand social networks.
5 Use and implementation of data structures
Heaps, balanced binary search trees, hashing, and many more
7/115
C I f ti C T i
-
8/11/2019 Lesson 1 .1 - Introduction
8/115
Course Information Course Topics
Skills Youll Learn
1 Become a better programmer2 Sharpen your mathematical and analytical skills
3 Start thinking algorithmically
4 Literacy with computer sciences greatest hits
5 Ace your technical interviews
8/115
Course Information Course Topics
-
8/11/2019 Lesson 1 .1 - Introduction
9/115
Course Information Course Topics
Who Are You?
1 Ideally, you know some programming.2 Doesnt matter which language(s) you know.
But you should be capable of translating high-level algorithmdescriptions into working programs in some programming language.
3 Some mathematical experience.
basic discrete math, proofs by induction, etc.Check out Mathematics for Computer Science, by Eric Lehman andTom Leighton.
9/115
Course Information Course Topics
-
8/11/2019 Lesson 1 .1 - Introduction
10/115
Course Information Course Topics
Logistics
Class meets.
Monday 18:00 20:00
Thursdays 17:00 18:00
Web Site.
lms.nust.edu.pkE-mail.
10/115
Course Information Course Topics
-
8/11/2019 Lesson 1 .1 - Introduction
11/115
Course Information Course Topics
Source Material
No required textbook. A few of the many good ones:
1 Cormen/Leiserson/Rivest/Stein, Introduction to Algorithms, 2009(3rd edition).
2 Kleinberg/Tardos, Algorithm Design, 2005.
3 Dasgupta/Papadimitriou/Vazirani, Algorithms, 2006.
4 Mehlhorn/Sanders, Data Structures and Algorithms: The BasicToolbox, 2008.
5 Web is the best and greenest textbook
No specific development environment required.
But you should be able to write and execute programs.
11/115
Course Information Course Topics
-
8/11/2019 Lesson 1 .1 - Introduction
12/115
Course Information Course Topics
General information
Prerequisites.
CS110, CS250
Assessment.
5-6 Quizzes (15%).
2-3 Assignments (10%).Term paper (5%).
OHTs (30%).
ESE (40%).
12/115
Course Information Course Topics
-
8/11/2019 Lesson 1 .1 - Introduction
13/115
p
A barrage of show-of-hands questions
13/115
Course Information Course Topics
-
8/11/2019 Lesson 1 .1 - Introduction
14/115
Tiny assignment for today
Q. A good example problem for showing algorithms with different
orders of magnitude is the classic anagram detection problem forstrings. One string is an anagram of another if the second is simply a
rearrangement of the first. For example, heart and earth areanagrams; python and typhon are anagrams as well. For the sake of
simplicity, assume that the two strings in question are of equal lengthand are made up of the set of 26 lowercase alphabetic characters. Ourgoal is to write a boolean function that will take two strings and return
whether they are anagrams.
14/115
Algorithm analysis
-
8/11/2019 Lesson 1 .1 - Introduction
15/115
1 Course Information
2 Algorithm analysis
3 Algorithm analysis
4 Big-O notation
15/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
16/115
Algorithm: background
What is an Algorithm?
In mathematics, computer science, and related subjects, an algorithmis aneffectivemethod for solving a problem expressed as afinite
sequence of instructions.
16/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
17/115
Algorithm: background
What is an Algorithm?
In mathematics, computer science, and related subjects, an algorithmis aneffectivemethod for solving a problem expressed as afinite
sequence of instructions.
Abu Abdullah Muhammad bin Musa al-Khwarizmi
a Muslim mathematician.
well-known work: Al-Jabr.
invented algorithms to solve quadratic equations.
word algorithm derived by his Latin name Algorithmi.
17/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
18/115
Algorithm: question
Which one is an algorithm?
A recipe for making tomato soup?
A procedure to sort 1000 numbers?
A procedure to recognize a particular face in a crowd?
A procedure to judge beauty?
18/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
19/115
Algorithm: designing algorithms
Four steps:
What is a Problem?
Correctness. is whether the proposed solution works.
Efficiency.is the desired running time of the algorithm comparedto some benchmark.
How to express an Algorithm
19/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
20/115
Algorithm: solving a problem
Q. How to solve a new problem?
20/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
21/115
Algorithm: solving a problem
Q. How to solve a new problem?
use basic building blocks
to achieve correctness then to worry about efficiency
so simple only a genius could have thought of it"Albert Einstein
21/115
-
8/11/2019 Lesson 1 .1 - Introduction
22/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
23/115
Algorithm: designing algorithms
Start with simple building blocksUse tested tools and techniques
Initial design should be as simple as possible
Worry about correctness
Then worry about efficiency
23/115
-
8/11/2019 Lesson 1 .1 - Introduction
24/115
-
8/11/2019 Lesson 1 .1 - Introduction
25/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
26/115
Algorithm analysis: problem
Find the eldest person six billion people.
a) We need to perform six billion operations?
b) We can do it by doing less than six billion operations by usingfast computers?
c) We can do it by doing about three billion operations by usingintelligent algorithms?
26/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
27/115
Algorithm: how to express one?
InputOutput
Basic idea in simple language
Complexity calculations
27/115
-
8/11/2019 Lesson 1 .1 - Introduction
28/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
29/115
Algorithm: how to express one?
Precondition: An imageXof sizeWH
1 function StaticPathway(X)
2 X RetinalFilter(X)3 Mu,v GaborFilterBank(X
)4 Mu,v Interactions(Mu,v)
5 Mu,v Normalizations(Mu,v)6 Ms Fusion(Mu,v)
7 returnMs
29/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
30/115
Algorithm: problem
Input:An Array A of k numbers. A [1...k]. The firstk1 numbers are
sorted in ascending order, the last number is out of place.
1,3,7,9,13,17,23,4
Output: An arrayA [1...k], which is sorted in ascending order.
1,3,4,7,9,13,17,23
Q. Write the algorithm, verify its correctness.
30/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
31/115
Algorithm: moveMin
f o r i = k t o 1
if A[i] < A[i-1]
SWAP(A[i],A[i-1])
Q. Will it swap till the first element or stop?
Q. Is this enough for moveMin?
31/115
-
8/11/2019 Lesson 1 .1 - Introduction
32/115
Algorithm analysis Algorithms
-
8/11/2019 Lesson 1 .1 - Introduction
33/115
Algorithm: moveMin
f o r i = k t o 1
if A[i] < A[i-1]
SWAP(A[i],A[i-1])
Q. Will it swap till the first element or stop?
For correctness, it is important to check the entire arrayto be sure.Also, for simplicity of the algorithm.
Q. Is this enough for moveMin?
This is not a general algorithm to sort any unordered array. It is just abuilding block for the sorting algorithm.
33/115
Algorithm analysis Algorithms
S
-
8/11/2019 Lesson 1 .1 - Introduction
34/115
Summary
Algorithms are a finite set of operations to solve a problem.Initial design of an algorithm should be:
Simple.using known concepts and principles.Clear.described in plain English, or simple notation.Correct. solve the problem.
34/115
-
8/11/2019 Lesson 1 .1 - Introduction
35/115
Algorithm analysis Overview
Al ith l i b k d
-
8/11/2019 Lesson 1 .1 - Introduction
36/115
Algorithm analysis: background
Why is a algorithm analysis important?
Computer scientists learn by experience.
Learn techniques to solve for one problem, and use it for another.
Algorithm design helps to solve more challenging problems.
36/115
Algorithm analysis Overview
Al ith l i b k d
-
8/11/2019 Lesson 1 .1 - Introduction
37/115
Algorithm analysis: background
Why is a algorithm analysis important?
Computer scientists learn by experience.
Learn techniques to solve for one problem, and use it for another.
Algorithm design helps to solve more challenging problems.
For example,Consider two solutions for one problem.
Both give the correct result.
One might be resource efficient compared to the other.
37/115
-
8/11/2019 Lesson 1 .1 - Introduction
38/115
-
8/11/2019 Lesson 1 .1 - Introduction
39/115
-
8/11/2019 Lesson 1 .1 - Introduction
40/115
Algorithm analysis Overview
Algorithm analysis: background
-
8/11/2019 Lesson 1 .1 - Introduction
41/115
Algorithm analysis: background
It is convenient to have a measure of the amount of work involved in a
computing process, even though it may be a very crude one. We may
count up the number of times that various elementary operations are
applied in the whole process..."
Alan Turing (1947)
41/115
-
8/11/2019 Lesson 1 .1 - Introduction
42/115
-
8/11/2019 Lesson 1 .1 - Introduction
43/115
-
8/11/2019 Lesson 1 .1 - Introduction
44/115
Algorithm analysis Overview
Algorithm analysis: quantitative analysis
-
8/11/2019 Lesson 1 .1 - Introduction
45/115
Algorithm analysis: quantitative analysis
What to use quantitative, or qualitative analysis.
As an engineer, for the proposed algorithm:one must determine the actual costs (memory, time, monetary).
comparison of qualitiesfaster, less memoryare unimportant.
45/115
Algorithm analysis Overview
Algorithm analysis: running time
-
8/11/2019 Lesson 1 .1 - Introduction
46/115
Algorithm analysis: running time
Running time of the program is a functionT(N),
represents thenumber of units of timetaken by an algorithm onany input of sizeN, or
the totalnumber of statementsexecuted by the program.
For example, a program may have a running time T(N) =cN, wherecis
some constant.the program is linearly proportional to the size of the input on
which it is run.
running time is linear time, or just linear.
46/115
Algorithm analysis Overview
Algorithm analysis: empirical analysis
-
8/11/2019 Lesson 1 .1 - Introduction
47/115
Algorithm analysis: empirical analysis
Q. How to time a program?
manual.using a stopwatch
automatic. using some timer function
Run the program for various input sizes and measure running time.
data size time (ms)
250 5500 8
1000 10
2000 154000 308000 5016000 75
47/115
-
8/11/2019 Lesson 1 .1 - Introduction
48/115
Algorithm analysis Overview
Algorithm analysis: question
-
8/11/2019 Lesson 1 .1 - Introduction
49/115
g y q
What are the different factor affecting therunning-time of an algorithm?
49/115
Algorithm analysis Overview
Algorithm analysis: running time
-
8/11/2019 Lesson 1 .1 - Introduction
50/115
g y g
Different factors influencing the running time of an algorithm are:
processing power of computer
algorithm used
size of the input
structure of the input
amount of available memory
implementation language
50/115
Algorithm analysis Overview
Algorithm analysis: running time
-
8/11/2019 Lesson 1 .1 - Introduction
51/115
g y g
We considerthree simplificationsto analyze algorithms:
without any implementation
taking the worst possible inputs
ignoring details
51/115
Algorithm analysis Overview
Algorithm analysis: running time
-
8/11/2019 Lesson 1 .1 - Introduction
52/115
We considerthree simplificationsto analyze algorithms:
without any implementation => RAM model
taking the worst possible inputs => worst-case running time
ignoring details => Big-O or Landau notation
52/115
Algorithm analysis Overview
Algorithm analysis: running time
-
8/11/2019 Lesson 1 .1 - Introduction
53/115
We considerthree simplificationsto analyze algorithms:
without any implementation=> RAM model
taking the worst possible inputs => worst-case running time
ignoring details => Big-O or Landau notation
53/115
Algorithm analysis RAM model
RAM model: overview
-
8/11/2019 Lesson 1 .1 - Introduction
54/115
Random access machine.
Considers a very simple computer.
Motivation is to analyze an algorithm ona minimal computermodel.
Q. What do you consider to be a minimal computer model for analysis?
54/115
Algorithm analysis RAM model
RAM model: overview
-
8/11/2019 Lesson 1 .1 - Introduction
55/115
A simple model to find how an algorithm will perform on a real
computer.
Mouse
Keyboard
Display
CD-ROM
Graphics processor
Working memory
Processor
Operating system
File system
Input/Output
Programming capability
CD-ROM
55/115
-
8/11/2019 Lesson 1 .1 - Introduction
56/115
Algorithm analysis RAM model
RAM model: question
-
8/11/2019 Lesson 1 .1 - Introduction
57/115
Simple operations (+,-,*,/) take 1 timestep
Loops comprise a condition plus multiple operations in eachiteration
Read/write to memory is free
57/115
Algorithm analysis RAM model
RAM model: question
-
8/11/2019 Lesson 1 .1 - Introduction
58/115
1 a=12 b=2 *a
1 s=02 while s
-
8/11/2019 Lesson 1 .1 - Introduction
59/115
1 a=12 b=2 *a
# of timesteps = 1
1 s=02 while s
-
8/11/2019 Lesson 1 .1 - Introduction
60/115
1 a=12 b=2 *a
# of timesteps = 1
1 s=02 while s
-
8/11/2019 Lesson 1 .1 - Introduction
61/115
We observe some problem with RAM,
memory access times depends whether from cache or disk.
unlimited memory is unrealistic.
a+ b a b.
However, itssimplicitymakes it an excellent tool to analyze algorithms.
61/115
Algorithm analysis Worst case running time
Algorithm analysis: running time
-
8/11/2019 Lesson 1 .1 - Introduction
62/115
We consider three simplifications to analyze algorithms:
without any implementation => RAM model
taking the worst possible inputs=> worst-case running time
ignoring details => Big-O or Landau notation
62/115
Algorithm analysis Worst case running time
Worst case running time: overview
-
8/11/2019 Lesson 1 .1 - Introduction
63/115
RAM simply counts operations for any input.
Worst case running timecombinesnotion of running time withstructure of input.
determinesgood or bad algorithmbased on all instances of inputs.
63/115
Algorithm analysis Worst case running time
Worst case running time: question
-
8/11/2019 Lesson 1 .1 - Introduction
64/115
Consider an example,
1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':
4 co unt += 1
In the example,
N: length of string
A: number of as in strRunning time = N+ A+
64/115
Algorithm analysis Worst case running time
Worst case running time: question
-
8/11/2019 Lesson 1 .1 - Introduction
65/115
Consider an example,
1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':
4 co unt += 1
In the example,
N: length of string
A: number of as in strRunning time = 2N+ 1A+ 0 or 1
65/115
Algorithm analysis Worst case running time
Worst case running time: the cases
-
8/11/2019 Lesson 1 .1 - Introduction
66/115
Consider a sample problem, e.g. sorting (arranging items in order)
66/115
-
8/11/2019 Lesson 1 .1 - Introduction
67/115
Algorithm analysis Worst case running time
Worst case running time: the challenge
-
8/11/2019 Lesson 1 .1 - Introduction
68/115
Actual data might not match input running time model?
Need to understand input to effectively process it.Two directions to follow:
Approach 1.design for the worst case.Approach 2.randomize, depend on probabilistic guarantee.
68/115
Algorithm analysis Worst case running time
Worst case running time: the outcomes
-
8/11/2019 Lesson 1 .1 - Introduction
69/115
Goals.
establishdifficultyof a problem.developoptimalalgorithms.
Approach.
suppress detailsin analysis; analyze to within a constant factor.
eliminate variabilityin input model by focusing on the worst case.Optimal algorithm.
performance guarantee (to within a constant factor) for any input.
no other algorithm can provide a better performance guarantee.
69/115
Algorithm analysis Worst case running time
Worst case running time: question
-
8/11/2019 Lesson 1 .1 - Introduction
70/115
Q. Which one do you consider is used? And why?
70/115
Algorithm analysis Worst case running time
Worst case running time: question
-
8/11/2019 Lesson 1 .1 - Introduction
71/115
Q. Which one do you consider is used? And why?
Best case is improbable.Average case is difficult to determine.
Worst-case only guarantees running times.
71/115
Algorithm analysis Worst case running time
Worst case running time: question
-
8/11/2019 Lesson 1 .1 - Introduction
72/115
Consider an example,
1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':4 co unt += 1
In the example consider,N: length of string
A: number of as in str
Question?
Worst case running time= N+ A+
Best case running time= N+ A+
Average case running time=
72/115
Algorithm analysis Worst case running time
Worst case running time: question
-
8/11/2019 Lesson 1 .1 - Introduction
73/115
Consider an example,
1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':4 co unt += 1
In the example consider,N: length of string
A: number of as in str
Question?
Worst case running time= 3N+ 0A+ 1
Best case running time= 2N+ 0A+ 1
Average case running time= ?
73/115
Algorithm analysis Worst case running time
Worst case running time: the problem
-
8/11/2019 Lesson 1 .1 - Introduction
74/115
Sample code counts ab substring in stringstr.
1 count = 02 fo r i i n r an ge ( N ):3 if s tr [ i] = = 'a ':
4 if
str[i+1] == 'b':5 co unt += 1
As a function grows larger,
counting exact running times becomes tedious.
determining complexities becomes difficult.To further simplify we use Big-O notation.
74/115
Algorithm analysis Worst case running time
Algorithm analysis: running time (optional)
-
8/11/2019 Lesson 1 .1 - Introduction
75/115
According to Donald E. Knuth in the book,The Art of Computer
Programming",Total running time = sum of cost frequency for all operations.
The mathematical model for analysis is defined as,
analyze the program to determine set of operations.
determine cost and frequency,frequency. depends on algorithm, input data.cost.depends on machine, compiler.
75/115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
-
8/11/2019 Lesson 1 .1 - Introduction
76/115
The power law. a Nb
N is the size of the data.b is the slope, determines the order of growth.
a is a constant.
76/115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
-
8/11/2019 Lesson 1 .1 - Introduction
77/115
System independent factors, or frequencies.
Algorithm.
Input data.
System dependent factors, or costs.
Hardware: CPU, memory, cache, ...
Software: compiler, interpreter, garbage collector, ...
System: operating system, network, other apps, ...
77/115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
-
8/11/2019 Lesson 1 .1 - Introduction
78/115
System independent factors, or frequencies.
Algorithm.
Input data.
determines exponent b
in power law.
System dependent factors, or costs.
Hardware: CPU, memory, cache, ...Software: compiler, interpreter, garbage collector, ...
System: operating system, network, other apps, ...
78/115
Algorithm analysis Worst case running time
Algorithm analysis: the power law
S
-
8/11/2019 Lesson 1 .1 - Introduction
79/115
System independent factors, or frequencies.
Algorithm.
Input data.
System dependent factors, or costs.
Hardware: CPU, memory, cache, ...Software: compiler, interpreter,
garbage collector, ...
System: operating system,
network, other apps, ...
determines constant ain power law.
79/115
Algorithm analysis Worst case running time
Algorithm analysis: mathematical models
I i i l h i h i l d l f l i h
-
8/11/2019 Lesson 1 .1 - Introduction
80/115
In principle, there exist accurate mathematical models for algorithm
analysis.In practice, such models,
use complicate formulas.
require advanced mathematics.
defined and used only for experts.
80/115
Algorithm analysis Worst case running time
Algorithm analysis: mathematical models
An example model could be
-
8/11/2019 Lesson 1 .1 - Introduction
81/115
An example model could be,
T(N) =c1
A+ c2
B+ c3
C+ c4
D+ c5
E
A = array accessB= integer addC= integer compareD= incrementE= variable assignment
where,
costs[c1,c2,c3,c4,c5](depend on machine, compiler).
frequencies[A ,B,C,D,E](depend on algorithm, input).
81/115
Algorithm analysis Worst case running time
Algorithm analysis: mathematical models
An example model could be
-
8/11/2019 Lesson 1 .1 - Introduction
82/115
An example model could be,
T(N) =c1
A+ c2
B+ c3
C+ c4
D+ c5
E
A = array accessB= integer addC= integer compareD= incrementE= variable assignment
where,
costs[c1,c2,c3,c4,c5](depend on machine, compiler).
frequencies[A ,B,C,D,E](depend on algorithm, input).
Much easier and cheaper to determine approximate running times,
can run huge number of experiments
However, it is difficult to get precise measurements.
In this course, we will use approximate models, i.e. for matrix multiplicationT(N) cN3.
82/115 Big-O notation
-
8/11/2019 Lesson 1 .1 - Introduction
83/115
1 Course Information
2 Algorithm analysis
3 Algorithm analysis
4 Big-O notation
83/115 Big-O notation Running time
Algorithm analysis: running time
We consider three simplifications to analyze algorithms:
-
8/11/2019 Lesson 1 .1 - Introduction
84/115
We consider three simplifications to analyze algorithms:
without any implementation => RAM modeltaking the worst possible inputs => worst-case running time
ignoring details=> Big-O or Landau notation
84/115 Big-O notation Running time
Algorithm analysis: quadratic growth
Consider the two functions
-
8/11/2019 Lesson 1 .1 - Introduction
85/115
Consider the two functions,
f(n) =n 2
g(n) =n 2 3n+ 2
Around n = 3, they look very different.
85/115 Big-O notation Running time
Algorithm analysis: quadratic growth
Around n = 10 they look similar
-
8/11/2019 Lesson 1 .1 - Introduction
86/115
Around n = 10, they look similar.
86/115 Big-O notation Running time
Algorithm analysis: quadratic growth
Around n = 100 they are almost same
-
8/11/2019 Lesson 1 .1 - Introduction
87/115
Around n = 100, they are almost same.
87/115
-
8/11/2019 Lesson 1 .1 - Introduction
88/115
Big-O notation Running time
Algorithm analysis: quadratic growth
The absolute difference is large, such that,
-
8/11/2019 Lesson 1 .1 - Introduction
89/115
The absolute difference is large, such that,
f(1000) = 1000000
g(1000) = 997002
but the relative diff
erence is very small,
f(1000)g(1000)
f(1000)
= 0.002998 < 0.3%
and this difference goes to zero asn .
89/115 Big-O notation Running time
Algorithm analysis: polynomial growth
To demonstrate with another example,
-
8/11/2019 Lesson 1 .1 - Introduction
90/115
o e o s e o e e p e,
f(n) =n 6
g(n) =n 6 76n5 + 343n4 345n3 + 45n2 344n
Around n = 10, they are very different
90/115 Big-O notation Running time
Algorithm analysis: polynomial growth
Around n = 100, they are not similar.
-
8/11/2019 Lesson 1 .1 - Introduction
91/115
, y
91/115 Big-O notation Running time
Algorithm analysis: polynomial growth
Around n = 1000, they seem similar.
-
8/11/2019 Lesson 1 .1 - Introduction
92/115
, y
92/115 Big-O notation Running time
Algorithm analysis: polynomial growth
Both function pairs of polynomials are similar, they each had the same
-
8/11/2019 Lesson 1 .1 - Introduction
93/115
leading term:
n2 in the first case
n6 in the second case
In each case, both functions exhibit the same rate of growth,
one is always proportionally larger than the other.
93/115 Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
-
8/11/2019 Lesson 1 .1 - Introduction
94/115
Algorithm 13n2 + 2n+ 33 Algorithm 22n 5n+ 5
Here,n is the problem size.
Question
Which one will you prefer?
94/115 Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
-
8/11/2019 Lesson 1 .1 - Introduction
95/115
Algorithm 13n2 + 2n+ 33 Algorithm 22n 5n+ 5
Here,n is the problem size.
Question
Which one will you prefer?
When answering, you ignore all unnecessary details.
Onlyn 2 < 2n considered.
95/115 Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
-
8/11/2019 Lesson 1 .1 - Introduction
96/115
Algorithm 13n2 + 2n+ 33f(n)
Algorithm 22n 5n+ 5g(n)
Here,n is the problem size.
n0andc, wherec g(n0) f(n0) n >n0, wherec g(n) f(n)
96/115 Big-O notation Asymptotic analysis
Big-O notation
Consider two algorithms to solve some problem,
-
8/11/2019 Lesson 1 .1 - Introduction
97/115
Algorithm 13n2 + 2n+ 33f(n)
Algorithm 22n 5n+ 5g(n)
Here,n is the problem size.
n0andc, wherec g(n0) f(n0) n >n0, wherec g(n) f(n)
f(n) O(g(n)), orf(n) =O(g(n))It means the functionf(n)has growth rate no greater than g(n).
97/115 Big-O notation Asymptotic analysis
Big-O notation
-
8/11/2019 Lesson 1 .1 - Introduction
98/115
Upper and lower bounds valid for n >n0smooth out the behavior of complexfunctions.
98/115 Big-O notation Asymptotic analysis
Big-O notation
Formally,
-
8/11/2019 Lesson 1 .1 - Introduction
99/115
f(
n) =
O(
g(
n))means
c g(
n)is an upper bound on
f(
n). Thus,there exists some constantcsuch thatf(n)is always cg(n), for
large enough n (i.e. ,n n0for some constantn0).
f(n) =(g(n))meansc g(n)is a lower bound onf(n). Thusthere exists some constantcsuch thatf(n)is always cg(n), for
alln n0.f(n) = (g(n))meansc1 g(n)is an upper bound onf(n)andc2 g(n)is a lower bound onf(n), for alln n0. Thus there existconstantsc1andc2such thatf(n) c1 g(n)andf(n) c2 g(n).
This means thatg(n)provides a nice, tight bound onf(n).
99/115 Big-O notation Asymptotic analysis
Big-O notation
-
8/11/2019 Lesson 1 .1 - Introduction
100/115
100/115 Big-O notation Asymptotic analysis
Big-O notation
f (n) g(n)
-
8/11/2019 Lesson 1 .1 - Introduction
101/115
f(n) g(n)
g(n)grows at least as fast asf(n)
101/115 Big-O notation Asymptotic analysis
Big-O notation
f(n) g(n)
-
8/11/2019 Lesson 1 .1 - Introduction
102/115
( ) g( )
g(n)grows at least as fast asf(n)
5n+ 2 O(n)
102/115 Big-O notation Asymptotic analysis
Big-O notation
f(n) g(n)
-
8/11/2019 Lesson 1 .1 - Introduction
103/115
( ) g( )
g(n)grows at least as fast asf(n)
5n+ 2 O(n)12n2 10 O(n2)
103/115
Big-O notation Asymptotic analysis
Big-O notation
f(n) g(n)
-
8/11/2019 Lesson 1 .1 - Introduction
104/115
( ) g( )
g(n)grows at least as fast asf(n)
5n+ 2 O(n)12n2 10 O(n2)3n 3n8 + 23 O(3n )
104/115
Big-O notation Asymptotic analysis
Big-O notation
f(n) g(n)
-
8/11/2019 Lesson 1 .1 - Introduction
105/115
( ) ( )
g(n)grows at least as fast asf(n)
5n+ 2 O(n)12n2 10 O(n2)3n 3n8 + 23 O(3n )
3n n3 3n8 + 23 O(3n n3)
105/115
Big-O notation Asymptotic analysis
Big-O notation
f(n) g(n)
-
8/11/2019 Lesson 1 .1 - Introduction
106/115
g(n)grows at least as fast asf(n)
5n+ 2 O(n)=> 5n+ 2 O(n2)12n2 10 O(n2)=> 12n2 10 O(n3)3n 3n8 + 23 O(3n )
3n n3 3n8 + 23 O(3n n3)
However, it is preferable to keep the bound as tight as possible.
106/115
Big-O notation Asymptotic analysis
Big-O notation
-
8/11/2019 Lesson 1 .1 - Introduction
107/115
The most common Big-O functions are given names:
O(1)=> constant
O(ln(n))=> logarithmic
O(n)=> linear
O(nln(n))=> log linear
O(n2)=> quadratic
O(n3)=> cubic
2
n ,en ,4
n
=> exponential
107/115
Big-O notation Asymptotic analysis
Big-O notation
Plot for common Big-O functions:
-
8/11/2019 Lesson 1 .1 - Introduction
108/115
108/115
Big-O notation Asymptotic analysis
Big-O notation
-
8/11/2019 Lesson 1 .1 - Introduction
109/115
Sample code counts ab substring in stringstr.
1 count = 02 f or c ha r in s tr :3 i f c ha r == 'a ':
4 co unt += 1
Using Big-O simplification,
We require no counting of operations.
It can be simply written as,O(n), a linear algorithm.
109/115
Big-O notation Asymptotic analysis
Big-O notation: problem
Given the following code fragment, what is its Big-O running time?
-
8/11/2019 Lesson 1 .1 - Introduction
110/115
1 test = 02 fo r i i n r an ge ( n ):3 fo r j i n r an ge ( n) :4 test = test + i * j
a)O
(n
)b)O(n2)c)O(logn)d)O(n3)
110/115
Big-O notation Asymptotic analysis
Big-O notation: problem
Given the following code fragment what is its Big-O running time?
-
8/11/2019 Lesson 1 .1 - Introduction
111/115
1 test = 02 fo r i i n r an ge ( n ):3 test = test + 145 fo r j i n r an ge ( n ):6 test = test - 1
a)O(n)b)O(n2)c)O(logn)d)O(n3)
111/115
Big-O notation Asymptotic analysis
Big-O notation: problem
Given the following code fragment what is its Big-O running time?
-
8/11/2019 Lesson 1 .1 - Introduction
112/115
1 i = n2 while i > 0 :3 k = 2 + 24 i = i / 25 }
a)O(n)b)O(n2)c)O(logn)d)O(n3)
112/115
Big-O notation Asymptotic analysis
Big-O notation: problem
Running time of a sorting algorithm.
Computer A:
-
8/11/2019 Lesson 1 .1 - Introduction
113/115
p
Speed is one billion instructions per second.Uses Insertion Sort and requires 2n2 instructions to sort n numbers.
Computer B:
Speed is 10 million instructions per second.Uses Merge Sort and requires 50nlgninstructions.
113/115
Big-O notation Asymptotic analysis
Big-O notation: problem
Computer A takes.
-
8/11/2019 Lesson 1 .1 - Introduction
114/115
2 (107)2instructions
1010instructions/second= 20,000secs(> 5.5hrs)
Computer B takes.
50 (107)lg107instructions
107instructions/second= 1,163secs(< 20mins)
Computer B runs 17 times faster than computer A,
using an algorithm that grows slowly.
even if uses a slow processor.
114/115
Big-O notation Asymptotic analysis
Summary
We will use Landau symbols to describe the complexity of algorithms.
-
8/11/2019 Lesson 1 .1 - Introduction
115/115
e.g., adding a list of n numbers in linear time
(n)algorithm.An algorithm is said to have polynomial time complexity if its runtimemay be described byO(nd)for some fixedd 0.
We will consider such algorithms to be efficient.
Problems that have no known polynomial-time algorithms are said tobeintractable.
e.g. Traveling salesman problem. find the shortest path that visits
n cities.
Best run time: O(n2
2n
)In general, you dont want to implement exponential-time or
exponential-memory algorithms.
115/115