HOW TO SOLVE IT?

Algorithms

An Algorithm

An algorithm is any well-defined (computational) procedure that takes some value, or set of values, as input and produces some value, or set of values, as output.

An algorithm is thus a sequence of (computational) steps that transforms the input into the output.

Algorithms

The output we would like to obtain as the result of the algorithm is usually the solution of a problem.

Many problems do not have closed-form solutions that we can just plug-in the input values to obtain the solution.

To obtain the solution requires the performance of a procedure.

Algorithms

Even the simplest of computational tasks requires algorithms when one thinks of it.

Multiple digit addition. Input: Two integers a and b Output: An integer that is the sum of a and b Algorithm?

The sorting problem Input: A sequence of n numbers (a1, a2, …, an) Output: A permutation (reordering) ( a1’, a2’, …, an’ )

of the input sequence such that a1’<= a2’<= …<= an’ Algorithm?

Algorithms

Consider the input sequence (33,22,11, 45,42)

When sorting algorithm is applied, it returns the output sequence (11,22,33,42,45)

Such an input sequence is called an instance of the sorting problem.

In general an instance of a problem consists of the input (satisfying the input requirements of the problem) needed to compute a solution to the problem

Algorithms

Sorting is a fundamental operation that are used in many computer programs

A large number of good sorting algorithms have been developed

The best algorithm for a given application depends on The number of times to be sorted The extent to which the items are already somehow sorted Possible restrictions on the values The kind of storage device used to keep the sequence

(RAM, hard drive, tapes, …) …

Algorithms

An algorithm is called correct if, for every input sequence it stops with the correct output.

A correct algorithm solves any given instance of the problem.

Examples of Important Algorithms

Sorting algorithmsDNA sequencing algorithmsData mining algorithmsCryptology algorithmsOptimization algorithms

Simplex Algorithm Metaheuristics Algorithms for TSP (Travelling Salesman Problem)

Algorithms to analyze Markov chainsSimulation algorithms….

Efficiency of Algorithms

The devices that execute the algorithms (?) are not infinitely fast and the memory capacity is limited

Algorithms devised to solve the same problem often differ in their efficiency.

Efficiency is about how many basic operations are done and how much memory is used

The number of basic operations required by the algorithm determines the execution time

Insertion Sort

An efficient algorithm for sorting a small number of elements

It works the way many people sort a hand of playing cards

We start with an empty left hand and the cards face down on the table

We then remove one card at a time from the table and insert it into the correct position between the cards already in the hand

Insertion Sort

How can we adapt this to a sequence?Assume there is a barrier that separates the

sorted part of the sequence from the unsorted part (the left hand from the cards on the table)

This barrier will move by one at each step of the algorithm.

Insertion Sort - Example

Input: (5, 2, 4, 6, 1, 3)Step 1: (5,| 2, 4, 6, 1, 3)Step 2: (2, 5,| 4, 6, 1, 3)Step 3: (2, 4, 5,| 6, 1, 3)Step 4: (2, 4, 5, 6,| 1, 3)Step 5: (1, 2, 4, 5, 6,| 3)Step 6: (1, 2, 3, 4, 5, 6|)

Insertion Sort- Pseudocode

Input: Array A

1 for to {2 3 % Insert into the sorted sequence 4 5 while and {6 7 8 }

Loop Invariant for Insertion Sort

At the start of each iteration of the for loop of lines 1-8, the subarray consists of the elements in the original subarray but in sorted order.

Loop Invariants

We use loop invariants to help us understand why an algorithm is correct.

We must show three things to check if a loop invariant is true1. Initialization: It is true prior to the first iteration of

the loop.2. Maintenance: If it is true before an iteration of the

loop, it remains true before the next iteration.3. Termination: When the loop terminates, the invariant

gives us useful property that helps us to show that the algorithm is correct.

Is it similar to another concept you know?

Analysis of Insertion Sort

Let and be the number of times the while loop is executed for that value of j.

Cost Times1 for to { n2 n-13 % Insert into the sorted 0 n-1

sequence 4 n-15 while and { 6 7 8 } n-1

Analysis of Insertion Sort

Even for inputs of a given size (length(A)), an algorithm’s running time may depend on the actual input.

Let us assume that the array A is already sorted.For each j, we then find .Hence, for all j.This would give the best running time which is The running time can be expressed as The running time is a linear function of n. ( )

Analysis of Insertion Sort

What about the worst time?Assume that the array is in reverse sorting

order.Then, for all j.(Why?)Noting and

The worst-case running time can be expressed as .

It is a quadratic function of n. ( )

Analysis of Algorithms

In our previous analysis, we looked at both the best case, in which the input array was already sorted, and the worst case, in which the input array was reverse sorted.

Usually the worst-case analysis is the one that merits most attention, because:1. It gives an upper bound on the running time for any

input.2. For some algorithm, worst case occurs very

frequently (search algorithms)3. The average case behaves usually the same with the

worst case.

Merge-Sort Algorithm

Another algorithm for sortingInstead of the incremental approach of

insertion sort, it makes use of a divide-and-conquer approach

Divide: Divide the n-element sequence into two subsequences of n/2 elements each.

Conquer: Sort the two subsequences recursively using merge sort.

Combine: Merge the two sorted sequences to produce the sorted answer.

Merge-Sort Algorithm

In order to execute this algorithm we need to merge to sorted (in order) sequences in the combine step.

For that we are going propose another algorithm:

MERGE (A, p, q, r)Where A is an array, p, q, and r are indices

such that.The algorithm assumes and are already

sorted and merges them into to sorted subarray that replaces the current .

MERGE (A, p, q, r)

1 2 3 % C and 4 for to {5 } 6 for to {7 }8 9

MERGE (A, p, q, r)

10 % M and into 111213 for to {13 if {14 15 }16 else {17 18 }

Loop Invariant for Merge Algorithm

At the start of each iteration of the for loop of lines 13-18, the subarray contains smallest elements of and , in sorted order.

Moreover, and are the smallest elements of their arrays that have not been copied back into A. Initialization: (?) Maintenance (?) Termination: (?)

Running Time of Merge Algorithm

Let number of operations computational complexity

Lines 1-3 constant o(1)Lines 8-12 constant o(1)Lines 4-7 times o(n)Lines 13-18 n times o(n)

The computational complexity is o(n). (Linear function of n)

MERGE-SORT (A, p, r)

1 if {2 3 MERGE-SORT (A, p, q)4 MERGE-SORT (A, q+1, r)5 MERGE(A, p, q, r)}

Initial call: MERGE-SORT (A, 1, length )

Computational Complexity

Computational complexity of merge sort is Why?This is better than of insertion sort.

Counting Sort

It assumes that each of the n input elements is an integer in the range 1 to k, for some integer k.

If k is a linear function of n, then the algorithm runs in time.

Input: A(1:n)Temporary Storage: C(1:k) (Counts)

COUNTING-SORT (A)

1 2 3 for to { 4 5 % Now we have the count vector C6 7 for { 8 while C {910 C C-111 }}

Counting Sort

It is a very good algorithmIt is able to run in linear time since it does

not do comparisonsYet beware of its limitations

Project Work

Write MATLAB functions for the three sorting algorithms presented: Insertion Sort Merge Sort (In order to implement this sort you will need

to write a function for the Merge algorithm) Counting Sort

Create random arrays of elements between 0 and 10 of sizes 10, 1000, 1000000

Compare the times it takes to for each algorithm to sort these three arrays. (Don’t forget that the counting sort necessitates integers)

Present your findings in a report.