# Computation, Complexity, and Algorithm IST 501 Fall 2014 Dongwon Lee, Ph.D

Post on 03-Jan-2016

215 views

Embed Size (px)

TRANSCRIPT

Computation, Complexity, and Algorithm

IST 501Fall 2014

Dongwon Lee, Ph.D.

IST 501

Learning ObjectivesUnderstand what computation and algorithm areBe able to compare alternative algorithms for a problem w. r. t their complexityGiven an algorithm, be able to describe its big O complexityBe able to explain NP-complete

IST 501

*

IST 501

IST 501

*Computational ComplexityWhy complexity?Primarily for evaluating difficulty in scaling up a problemHow will the problem grow as resources increase?Knowing if a claimed solution to a problem is optimal (best)Optimal (best) in what sense?

IST 501

IST 501

*Why do we have to deal with this?Moores lawHwangs lawGrowth of information and information resourcesStoring stuffFinding stuff

IST 501

IST 501

*ImpactThe efficiency of algorithms/methods The inherent "difficulty" of problems of practical and/or theoretical importanceA major discovery in the science was that computational problems can vary tremendously in the effort required to solve them precisely. The technical term for a hard problem is _________________" which essentially means: "abandon all hope of finding an efficient algorithm for the exact (and sometimes approximate) solution of this problem".

IST 501

IST 501

*OptimalityA solution to a problem is sometimes stated as optimalOptimal in what sense?Empirically?Theoretically? (the only real definition)

IST 501

IST 501

*Youre given a problemData informatics: How will our system scale if we store everybodys email every day and we ask everyone to send a picture of themselves and to everyone else. We will also hire a new employ each week for a year

Social informatics: We want to know how the number of social connections in a growing community increase over time.

IST 501

We will use algorithmsIST 501

*

IST 501

Eg. Sorting AlgorithmsProblem: Sort a set of numbers in ascending order[1 7 3 5 9]Many variations of sorting algorithmsBubble SortInsertion SortMerge SortQuick SortIST 501

*

IST 501

ScenariosIve got two algorithms that accomplish the same taskWhich is better?I want to store some dataHow do my storage needs scale as more data is storedGiven an algorithm, can I determine how long it will take to run?Input is unknownDont want to trace all possible paths of executionFor different input, can I determine how system cost will change?IST 501

*

IST 501

IST 501

*Measuring the Growth of WorkWhile it is possible to measure the work done by an algorithm for a given set of input, we need a way to:Measure

Compare

IST 501

IST 501

*Basic unit of the growth of workInput NDescribing the relationship as a function f(n)f(N): the number of steps required by an algorithm

IST 501

Why Input Size N?IST 501

*

IST 501

Comparing the growth of workSize of input NWork doneIST 501

*

IST 501

IST 501

*Time vs. SpaceVery often, we can trade space for time:

For example: maintain a collection of students with SSN information.Use an array of a billion elements and have immediate access (better time)Use an array of number of students and have to search (better space)

IST 501

IST 501

*Introducing Big O NotationUsed in a sense to put algorithms and problems into families

IST 501

IST 501

*Size of Input (measure of work)In analyzing rate of growth based upon size of input, well use a variableWhy?For each factor in the size, use a new variableN is most common

Examples:A linked list of N elementsA 2D array of N x M elementsA Binary Search Tree of P elements

IST 501

IST 501

*Formal Definition of Big-OFor a given function g(n), O(g(n)) is defined to be the set of functions

IST 501

IST 501

*Visual O( ) Meaningf(n)n0f(n) = O(g(n))Size of inputWork doneOur Algorithm

IST 501

IST 501

*Simplifying O( ) AnswersWe say Big O complexity of 3n2 + 2 = O(n2) drop constants!

IST 501

Correct but MeaninglessIST 501

*

IST 501

IST 501

*Comparing AlgorithmsNow that we know the formal definition of O( ) notation (and what it means)If we can determine the O( ) of algorithms and problemsThis establishes the worst they perform.

Thus now we can compare them and see which has the better performance.

IST 501

IST 501

*

Correctly Interpreting O( )O(1) or Order OneO(N) or Order N

IST 501

IST 501

*Complex/Combined FactorsAlgorithms and problems typically consist of a sequence of logical steps/sections

We need a way to analyze these more complex algorithms

Its easy analyze the sections and then combine them!

IST 501

IST 501

*Eg: Insert into Linked ListTask: Insert an element into an ordered listFind the right locationDo the steps to create the node and add it to the list1738142head//Inserting 75

IST 501

IST 501

*Eg: Insert into Linked ListInsert an element into an ordered listFind the right locationDo the steps to create the node and add it to the list1738142head//75

IST 501

Combine the AnalysisIST 501

*

IST 501

IST 501

*Eg: Search a 2D ArrayTask: Search an unsorted 2D array (row, then column)Traverse all rowsFor each row, examine all the cells (changing columns)

IST 501

IST 501

*Search an unsorted 2D array (row, then column)Traverse all rowsFor each row, examine all the cells (changing columns)RowColumn 123451 2 3 4 5 6 7 8 9 10O(M)Eg: Search a 2D Array

IST 501

Combine the AnalysisIST 501

*

IST 501

IST 501

*Sequential StepsIf steps appear sequentially (one after another), then add their respective O().

loop. . .endlooploop. . .endloopNM

IST 501

IST 501

*Embedded Steps

If steps appear embedded (one inside another), then multiply their respective O().

loop loop . . . endloopendloopMN

IST 501

IST 501

*Correctly Determining O( )Can have multiple factors:O(N*M)O(logP + N2)But keep only the dominant factors:O(N + NlogN) O(N*M + P) O(V2 + VlogV) Drop constants:O(2N + 3N2)

IST 501

SummaryIST 501

*

IST 501

Poly-Time vs. Expo-TimeSuch algorithms with running times of orders O(log n), O(n ), O(n log n), O(n2), O(n3) etc. Are called polynomial-time algorithms.

On the other hand, algorithms with complexities which cannot be bounded by polynomial functions are called exponential-time algorithms. These include "exploding-growth" orders which do not contain exponential factors, like n!.

IST 501

*

IST 501

IST 501

*The Towers of HanoiA B CGoal: Move stack of rings to another pegRule 1: May move only 1 ring at a timeRule 2: May never have larger ring on top of smaller ring

IST 501

IST 501

*Original StateMove 1Move 2Move 3Move 4Move 5Move 6Move 7Towers of Hanoi: Solution

IST 501

IST 501

*Towers of Hanoi - ComplexityFor 3 rings we have 7 operations.

In general, the cost is 2N 1 = O(2N)

Each time we increment N, we double the amount of work.

This grows incredibly fast!

IST 501

IST 501

*Towers of Hanoi (2N) RuntimeFor N = 642N = 264 = 18,450,000,000,000,000,000

If we had a computer that could execute a billion instructions per second

It would take 584 years to complete

But it could get worse

The case of 4 or more pegs: Open Problem

IST 501

IST 501

*The Traveling Salesman ProblemA traveling salesman wants to visit a number of cities and then return to the starting point. Of course he wants to save time and energy, so he wants to determine the shortest path for his trip.We can represent the cities and the distances between them by a weighted, complete, undirected graph.The problem then is to find the round-trip of minimum total weight that visits each vertex exactly once.

IST 501

IST 501

*The Traveling Salesman ProblemExample: What path would the traveling salesman take to visit the following cities?

IST 501

Traveling Salesman Problem (TSP): one of the best-known computational problems that is known to be extremely hard to solveNP-hardA brute force solution takes O(n!)A dynamic programming solution takes O(cn), c>2Harder to solve than Tower of Hanoi problem

The Traveling Salesman ProblemIST 501

*

IST 501

IST 501

*Where Does this Leave Us?Clearly algorithms have varying runtimes or storage costs.

Wed like a way to categorize them:

Reasonable, so it may be useful Unreasonable, so why bother running

IST 501

IST 501

*Performance Categories

IST 501

IST 501

*Two Categories of Algorithms2 4 8 16 32 64 128 256 512 1024Size of Input (N)10351030102510201015trillionbillionmillion100010010Runtime

IST 501

*Reasonable vs. UnreasonableReasonable algorithms feature polynomial factors in their O( ) and may be usable depending upon input size.O (Log N), O (N), O (NK) where K is a constantUnreasonable algorithms feature exponential factors in their O( ) and have little practical utilityO (2N), O (N!), O (NN)IST 501

*

IST 501

IST 501

*Eg: Computational ComplexityBig O complexity in

Recommended