cse 450/598 design and analysis of algorithms
DESCRIPTION
CSE 450/598 Design and Analysis of Algorithms. Instructor: Arun Sen Office: BYENG 530 Tel: 480-965-6153 E-mail: [email protected] Office Hours: MW 3:30-4:30 or by appointment TA: TBA Office : TBA Tel: TBA E-mail: TBA Office Hours : TBA. Introduction (1) Growth of functions - PowerPoint PPT PresentationTRANSCRIPT
CSE 450/598 Design and Analysis of
Algorithms Instructor: Arun Sen Office: BYENG 530 Tel: 480-965-6153 E-mail: [email protected] Office Hours: MW 3:30-4:30 or by appointment TA: TBA Office: TBA Tel: TBA E-mail: TBA Office Hours: TBA
Textbook and Course OutlineText: Algorithm Design by Kleinberg & Tardos
Note: A significant amount of course material will come from sources other than the textbook. As such, class
attendance is absolutely essential. Introduction (1)
Growth of functions Complexity of computation Recurrence relations
Divide and Conquer(2) MaxMin in a sequence Binary search Quicksort Mergesort Strassen’s matrix multiplication
Dynamic Programming(2) Matrix chain multiplication Optimal polygon triangulation Optimal binary tree Longest common subsequence Traveling Salesman Problem
Greedy Algorithms(2) Chromatic number Knapsack Set cover Minimum spanning tree Event scheduling
Network Flows(1) Max-flow Min-cut Theorem Ford-Fulkerson Algorithm
Backtracking(1) N-Queens Problem
Branch and Bound(1) Traveling Salesman Problem
NP-Completeness (2) Problem transformation No-wait flow shop scheduling 3-Satisfiability Traveling Salesman Problem Node Cover
Approximation Algorithms(2) Node Cover Bin Packing Scheduling Steiner Trees
Probabilistic Algorithms (1) *** The course outline may be modified if
necessary, depending on progress in class.
Grading Policy for CSE 450 There will be one mid-term and a final. In addition, there will be
two quizzes and programming and homework assignments
90% will ensure A, 80% will ensure B, 70% will ensure C and so on
Loss of points due to late submission of assignments 1 day 50% 2 days 75% 3 days 100%
Assignment CSE 450 CSE 598Mid-term 20% 15%
Final 30% 25%Quizzes 1 & 2 20% 20%Programming
Assg.20% 20%
Homework Assg. 10% 10%Project 0% 10%
Cheating Policy Any case of cheating will be severely
dealt with.
Penalty for cheating will be in accordance with the policies of the Fulton School of Engineering and Arizona State University.
Multiple offenders may be removed from the program and the University.
An algorithm may be broadly defined as a step by step procedure for solving a problem or accomplishing some end. It is a finite sequence of unambiguous, executable steps that ultimately terminate if followed.
What is not an algorithm?1. Make a list of all positive integers2. Arrange this list in descending order (from
largest to smallest)3. Extract the first integer from the resulting
list4. Stop.
What is an algorithm?
Example in Origami: Algorithm for making a bird
Algorithms = Problem Solving
Example in Manufacturing:Various wafers (tasks) are to be processed in a series of stations. The processing time of the wafers in different stations is different. Once a wafer is processed on a station it needs to be processed on the next station immediately, i.e., there cannot be any wait. In what order should the wafers be supplied to the assembly line so that the completion time of processing of all wafers is minimized?
S1 S2 S8
w1 t11 t12 t18
w2 t21 t22 t28
w3 t31 t32 t38
w1 : t11 = 4, t12= 5;w2 : t21 = 2, t22 = 4;w1 : w2 : w2:
w2 : w1: Completion Time in the first ordering =
13Completion Time in the second ordering
= 11
S1 : 4 S2 : 5S1: 2
S2 : 4S1:2
S2 : 4
S1: 2
S2 : 4S1 : 4 S2 : 5
Search Space
The solution is somewhere here
Solution can be found by exhaustive search in the search space
Search space for the solution may be very large Large search space implies long computation
time to find solution (?) Not necessarily true Search space for the sorting problem is very
large The trick in the design of efficient algorithms
lies in finding ways to reduce the search space
The Central Role of Algorithms in Computer Science
ALGORITHMS
Limitations of
Discovery of Representation of
Communication of
Execution of
Properties of Algorithms Finiteness: An algorithm must always
terminate after a finite number of steps Definiteness: Each step must be precisely
defined; the actions must be unambiguous Input: An algorithm has zero or more inputs
Offline Algorithms: All input data is available before the execution of the algorithm begins
Online Algorithms: Input data is made available during the execution of the algorithm
Output: An algorithm has one or more outputs
Effectiveness: All operations must sufficiently basic to be done exactly and within a finite length of time by a man using pencil and paper
Evaluating Quality of Algorithms Often there are several different ways to
solve a problem, i.e., there are several different algorithms to solve a problem
What is the “best” way to solve a problem?
What is the “best” algorithm? How do you measure the “goodness” of
an algorithm? What metric(s) should be used to
measure the “goodness” of an algorithm?
Time Space *** What about Power?
Problem and Instance Algorithms are designed to solve
problems What is a problem?
A problem is a general question to be answered, usually processing several parameters, or free variables, whose values are left unspecified. A problem is described by giving (i) a general description of all its parameters and (ii) a statement of what properties the answer, or the solution, required to satisfy.
What is an instance? An instance of a problem is obtained by specifying
particular values for all the problem parameters.
Traveling Salesman ProblemInstance: A finite set C={c1, c2, …, cm} of
cities, a distance d(ci, cj) є Z+ for each pair of cities ci, cj є C and a bound B є Z+ (where Z+ denotes the positive integers).
Question: Is there a tour of all cities in C having total length no more than B, that is an ordering <cπ(1), cπ(2), …, cπ(m)> of C such that,
1
1
)1(),()1(),( )()(m
i
mii BCCdCCd
Measuring efficiency of algorithms
One possible way to measure efficiency may be to note the execution time on some machine
Suppose that the problem P can be solved by two different algorithms A1 and A2.
Algorithms A1 and A2 were coded and using a data set D, the programs were executed on some machine M
A1 and A2 took 10 and 15 seconds to run to completion
Can we now say that A1 is more efficient that A2?
Measuring efficiency of algorithms
What happens if instead of data set D we use a different dataset D’? A1 may end up taking more time than A2
What happens if instead of machine M we use a different machine M’? A1 may end up taking more time than A2
If one want to make a statement about the efficiency of two algorithms based on timing values, it should read “A1 is more efficient that A2 on machine M, using data set D”, instead of an unqualified statement like “A1 is more efficient that A2”
Measuring efficiency of algorithms
The qualified statement “A1 is more efficient that A2 on machine M, using data set D” is of limited value as someone may use different data set or a different machine
Ideally, one would like to make an unqualified statement like “A1 is more efficient that A2” , that is independent of data set and machine
We cannot make such an unqualified statement by observing execution time on a machine
Data and Machine independent statement can be made if we note the number of “basic operations” needed by the algorithms The “basic” or “elementary” operations are operations of the
form addition, multiplication, comparison etc
Analysis of Algorithms Size= nTimeCompl Func
10 20 30 40 50 60
(A1) n.00001
sec
(A2) n2
(A3) n3
(A4) n5
(A5) 2n
(A6) 3n
.00003 sec
.00002 sec
.00004 sec
.00005 sec
.00006 sec
.0001 sec
.0004 sec
.0009 sec
.0016 sec
.0025 sec
.0036 sec
.001 sec
.008 .027 .064 .125 .216 sec sec sec sec sec
58 6.5 3855 2*108 1.3*1013
min years cents. cents. cents.
3.2 sec 24.3 sec 1.7 min 5.2 min 13.0 min
.1 sec
.001 sec.059 sec
1.0 sec
17.9
min
12.7
days
35.7
years
366 centuri
es
Size of Largest Problem Instance Solvable in 1 Hour
Time complexity
function
With present
computer
With computers 100 times
faster
With computer
1000 times faster
n N1
n2 N2
n3 N3
n5 N4
2n N5
3n N6
100 N1 1000 N1
10 N2 31.6 N2 4.64 N3 10 N3
2.5 N4 3.98 N4
N5 + 6.64 N5 + 9.97N6 + 4.19 N6 + 6.29
Growth of Functions: Asymptotic Notations
O(g(n)) = {f(n): there exists positive constants c and n0 such that 0<=f(n)<=c * g(n) for all n >= n0}
Ω(g(n)) = {f(n): there exists positive constants c and n0 such that 0<=c * g(n)<=f(n) for all n >= n0}
Q(g(n)) = {f(n): there exists positive constants c1, c2 and n0 such that 0<= c1 * g(n)<=f(n)<=c2*g(n) for all n >= n0}
o(g(n) = {f(n): for any positive constant c>0 there exists a constant n0 such that 0<=f(n)<c * g(n) for all n >= n0}
w(g(n)) = {f(n): for any positive constant c>0 there exists a constant n0 such that 0<=<c * g(n)< f(n) for all n >= n0}
A function f(n) is said to be of the order of another function g(n) and
is denoted by O(g(n)) if there exists positive constants c andn0 such
that 0<=f(n)<=c * g(n) for all n >= n0}
Basic Operations and Data Set
To evaluate efficiency of an algorithm, we decided to count the number of basic operations performed by the algorithm
This is usually expressed as a function of the input data size
The number of basic operations in an algorithm Is it dependent or independent of the data
set ?
Given a set of records R1, …, Rn with keys k1, …,kn. Sort the records in ascending order of the keys.
Basic Operations and Data Set
The number of basic operations in an algorithm Is it independent of the data set ? Is it dependent on the data set?
If the number of basic operations in an algorithm depends on the data set then one needs to consider Best case complexity Worst case complexity Average case complexity
What does “average” mean? Average over what?
Given n elements X[1], …, X[n], the algorithm finds m and j such that m = X[j] = max 1<=k<=n X[k], and for which j is as large as possible.
Algorithm FindMaxStep 1. Set j n, k n – 1, m X[n]Step 2. If k=0, the algorithm terminates.
Step 3. If X[k] <= m, go to step 5.Step 4. Set j k, m X[k].Step 5. Decrease k by 1, and return to step 2
Moore’s law says that computing power (hardware speed) doubles every eighteen months
How long will it take to have a thousand-fold speed-up in computation, if we rely on hardware speed alone? Answer: 15 years Expected cost: significant
How long will it take to have a thousand-fold speed-up in computation, if we rely on the design of clever algorithms? Thousand-fold speed-up can be attained if currently used O(n5)
complexity algorithm is replaced by a new algorithm with complexity O(n2) for n=10.
How long will it take to develop a O(n2) complexity algorithm which does the same thing as the currently used O(n5) complexity algorithm?
Answer: May be as little as one afternoon Ingredients needed
Pencil Paper A beautiful mind
Expected cost: significantly less than what will be needed if we rely on hardware alone
Computational Speed-up and the Role of Algorithms
Computational Speed-up and the Role of Algorithms
A clever algorithm can achieve overnight what progress in hardware would require decades to accomplish
“The algorithm things are really startling, because when you get those right you can jump three orders of magnitude in one afternoon.”
William PulleyblankSenior Scientist, IBM Research
Algorithm Design Techniques
Divide and Conquer Dynamic Programming Greedy Algorithms Backtracking Branch and Bound Approximation Algorithms Probabilistic Algorithms Mathematical Programming Parallel and Distributed Algorithms Simulated Annealing Genetic Algorithms Tabu Search
How do you “prove” a problem to be “difficult”?
Suppose that the algorithm you developed for the problem to be solved (after many sleepless nights) turned out to be very time consuming
Possibilities You haven’t designed an efficient algorithm for the problem
May be you are not that great an algorithm designer May be you are a better fashion designer May be you have not taken CSE 450/598
May be the problem is difficult and more efficient algorithm cannot be designed How do you know that more efficient algorithm cannot be designed? It is difficult to substantiate a claim that more efficient algorithm cannot be designed Your inability to design an efficient algorithm does not necessarily mean that the
problem is “difficult” It may be easier to claim that the problem “probably” is “difficult” How do you substantiate the claim that the problem “probably” is “difficult”? What if you line up a bunch of “smart” people who will testify that they also think that
the problem is difficult? Theory of NP-Completeness
Theory of NP-Completeness Complexity of an algorithm for a problem says
more about the algorithm and less about the problem If a low complexity algorithm can be found for the
solution of a problem, we can say that the problem is not difficult
If we are unable to find a low complexity algorithm for the solution of a problem, can we say that the problem is difficult?
Answer: No
NP-Completeness of a problem says something about the problem
Problems may or may not be NP-Complete – not the algorithms
Problems and Algorithms for their solution
Problem P
Algorithm 1Complexity: O(n)
Algorithm 3Complexity: O(2n)
Algorithm 2Complexity: O(n4)
Complexity of a Problem
How to prove a problem difficult?
Is the approach of lining up a group of famous people really going to work?
Answer: Probably not Why would a group of famous people be interested in
working on your problem? “If the mountain does not come to Mohammed, Mohammed
goes to the mountain” If the famous people are not interested in working on your
problem, you transform their problem into yours. If such a transformation is possible, you can now claim that
if your problem can easily be solved, so can be theirs. In other words, if their problem is difficult, so is yours.
Problem Transformation – Hamiltonian Cycle Problem
A cycle in a graph G = (V, E) is a sequence <v1, v2, …, vk> of distinct vertices of V such that {vi, vi+1} e E for 1 <= i < k and such that {vk, v1} e E.
A Hamiltonian cycle in G is a simple cycle that includes all the vertices of G.
Hamiltonian Cycle Problem Instance: A graph G = (V, E) Question: Does G contain a Hamiltonian
cycle?
Traveling Salesman ProblemInstance: A finite set C={c1, c2, …, cm} of
cities, a distance d(ci, cj) є Z+ for each pair of cities ci, cj є C and a bound B є Z+ (where Z+ denotes the positive integers).
Question: Is there a tour of all cities in C having total length no more than B, that is an ordering <cπ(1), cπ(2), …, cπ(m)> of C such that,
1
1
)1(),()1(),( )()(m
i
mii BCCdCCd
No-wait Flow-shop Scheduling Problem
S1 S2 S8 w1 t11 t12 t18
w2 t21 t22 t28
w3 t31 t32 t38
Problem Transformation No-wait Flow-shop Scheduling Problem can be
transformed into Traveling Salesman Problem How? We will see it later
Hamiltonian Cycle problem can be transformed to Traveling Salesman Problem How? From an instance of the HC Problem, the graph G =
(V, E), (|V| = n), construct an instance of the TSP problem as follows: Construct a completely connected graph G’ = (V’, E’) where (|V’| = |V|). Associate a distance with each edge of E’. For each edge e’ e E’, if e’ e E then dist(e’) = 1, otherwise dist(e’) = 2. Set B, a problem parameter of the TSP problem, equal to n.
Problem Transformation Claim: Graph G contains a Hamiltonian Cycle, if and
only if there is a tour of all the cities in G’, that has a total length no more than B.
If G has a HC <v1, v2, …, vn>, then G’ has a TSP tour of length n = B, because each intercity distance traveled in the tour corresponds to an edge in G and hence has length 1.
If G’ has a TSP tour of length n = B, then each edge e that contributes to the tour must have dist(e) = 1 (because the tour is made up of n edges). It implies that these edges are present in G as well. These set of edges makes up a Hamiltonian Cycle in G.
Algorithms and their Complexities
N-th Fibonacci Number
2n1nn FFF 1F0,F 10
Reference Books For solution of Linear Homogeneous Recurrence
Relations with Constant Coefficients: Elements of Discrete Mathematics by C. L. Liu
For Problem Transformation and NP-Completeness: Computers and Intractability by Garey and Johnson
How to Compute Fibonacci Number in O(log n) time?
Transform Fibonacci number computation problem to a matrix chain multiplication problem.
Matrix Chain Multiplication Problem• P = A1 * A2 * A3 * … * Ap, where Ai is an n x n
matrix.• A1 * A2, where Ai is an n x n matrix, can be
done in O(n3) complexity.• If dimensions of A1 & A2 are constant, the
product A1 * A2 can be done in constant time.• The matrix chain A1 * A2 * … * An, where A1 =
A2 = … = An, can be computed in O(log n) time.
Computation of the n-th Fibonacci Number, Fn
2
2-n3-n
2-n3-n
1-n2-n
1-n2-n
2-n1-n1-n
n1-n
A*FF
A*A*FF1110
A whereA,*FF
1110
FF
FFFFF
1110
A and
01 where,A *FF
Hence,
A *01
A0111
10
A1110
10
A*A*A*FF
A*FF
....
....A *FF
nn1-n
n
n
n1
1-n1-10
1-n10
33-n4-n
xx
212
21
448
224
2
xxxxx
form theof isProduct
34212113
5332
5332
A .AA
5332
2111
2111
A .AA
2111
1110
1110
A A.A
1110
A
jhjkihj
jhiki
jihkhhk
ji
hkhhkhkhh
hkk
hkhhk
hkhhk
hkhhk
hk
'
'
''
22'
22'
'''
''
21
2
where,
x, xalgorithm In the
Example 1: Computation of F7
[F6 F7] = x * A7
x’’’ = x A7
n = 7 n = 3 n = 1x’ = x A
A’ = A2
x’’ = x’ . A’
= x A . A2
= x A3
A’’ = A’ 2
= A4
x’’’ = x’’ . A’’
= x A3 . A4
= x A7
A’’’ = A’’ 2
= A8
Example 2: Computation of F8
[ F7 F8] = x . A8
x’ = x A8
n = 8 n = 4 n = 2 n = 1
A’ = A2 A’’ = A’ 2
= A4A’’’ = A’’ 2
= A8x’ = x . A’’’
= x . A8
A’’’’ = A’’’ 2
The longest path