i501 – introduction to informatics [email protected] info rm atics and computing lecture 8 –...

I501 – Introduction to Informatics

[email protected]://informatics.indiana.edu/jbollen/I501F11/

Informatics and computing

Lecture 8 – Fall 2011


Computational Complexity and the limits of computation






Today’s class

1) Lecture2) Blackbox presentations3) Guest Lecture: Jonathan Mills






Organized complexity

organized complexity study of organization

whole is more than sum of parts Systemhood properties Holism vs. reductionism

Need for new mathematical and computational tools Massive combinatorial searches Problems that can only be tackled with

computers: “computer as lab

Understanding function Of wholes

Systems biology Evolutionary thinking Systems thinking

Emergence How do elements combine to form

new unities?

Organizedsimplicity

Disorganized complexity

Organized Complexity

Complexity

Ran

do

mn

ess

From systems science to informatics






What is complexity?

Dictionary Having many varied or interrelated parts, patterns or elements Quantity of parts and extent of interrelations

Subjective or epistemic connotation Ability to understand or cope

Complexity is in the eyes of the observer Brain to a neuroscientist and to a butcher

Quantity of information required to describe a system






complexity and information

Descriptive complexity: Proportional to the amount of information required to describe

the system In a syntactic way

Measure number of entities (variables, states, components) and variety relationships among them

General requirements for indicators: Nonnegative quantity If system A is a homomorphic image of B, then the complexity of A should not

be greater than B If A and B are isomorphic, then their complexity should be the same If system C consists of two non-interacting subsystems B and neither is a

homomorphic image of the other, then the complexity of C should be equal to the sum of the complexities of A and B

Size of shortest description or program in a standard language or universal computer Applicable to any system Difficult to determine shortest description aka Kolmogorov complexity






Uncertainty-based complexity and information

Proportional to the amount of information needed to resolve any uncertainty with the system involved In a syntactic way

Related to number of alternatives left undecided to characterize a particular element

Examples Hartley Measure Shannon Entropy

AAH 2log)( in

iiS xpxpAH 2

1

log)(






Trade-off between descriptive and uncertainty-based complexity:

When one is reduced, the other is likely to increase Trade certainty for acceptable descriptive complexity Models of phenomena in the realm of organized complexity require large

descriptive complexity But to be manageable, we must simplify by accepting larger uncertainty (and

smaller descriptive complexity)






Computational complexity of algorithms

Descriptive and uncertainty-based complexity pertain to systems Characterized by information

Computational complexity pertains to systems problems Characterization of the time or space (memory) requirements

for solving a problem by a particular algorithm






Types of Computational Problems

Algorithms are Procedures for Solving Problems Types of Problems

Search Find an X in input satisfying property Y

Find a prime number in a random sequence of numbers Structuring Problems

Transform the input to satisfy property Y Sort a random sequence of numbers

Construction Problems Build an X satisfying property Y

Generate a random sequence of numbers with a given mean and standard deviation

Optimization Problems Find the best X satisfying property Y

Find the largest prime number in a given sequence? Decision Problems

Decide whether the input satisfies property Y Is the input number a prime number?

Adaptive Problems Maintain property Y over time

Grow a sequence of numbers such that there are always m prime numbers with a given mean and standard deviation






Problem Difficulty

Conceptually Hard Problem No algorithm is known to solve the problem

Analytically Hard Problem An algorithm exists to solve the problem, but we

don’t know how long it will take to solve every instance of the problem

Computationally Hard Problem An algorithm exists to solve the problem, but

relatively few instances take millions of years (or forever) to solve

Computationally unsolvable Problem No algorithm can exist to solve the problem. Computability, decidability

Artificial Intelligence

Algorithmic Complexity Theory

Computability Theory






Hanoi Problem

Invented by French Mathematician Édouard Lucas in 1883 At the Tower of Brahma in India, there are three

diamond pegs and sixty-four gold disks. When the temple priests have moved all the disks, one at a time preserving size order, to another peg the world will come to an end.

If the priests can move a disk from one peg to another in one second, how long does the World have to exist?

See Clarke’s “The Nine Billion Names of God”






Solving the Hanoi Problem

Solve for the smallest instances and then try to generalize N=2

N=3

0 43 7

Use Hanoi_2 (H2) as building block (3 moves)

H3 uses H2 twice, plus 1 move of the largest disk: 2 x 3 + 1 moves

1 2 3






Hanoi Problem for n disks

Algorithm to move n disks from A to C Move top n-1 disks from A to B Move biggest disk to C Move n-1 disks on B to C

Recursion Until H2

Use Hanoi_2 (H2) as building block (of 3 moves)

H3 uses H2 twice, plus 1 move of the largest disk

Use Hanoi_2 (H2) as building block (of 3 moves)

H3 uses H2 twice, plus 1 move of the largest disk

An Algorithm that uses itself to solve a problem

An Algorithm that uses itself to solve a problem






Pseudocode for Hanoi Problem

Hanoi (Start, Temp, End, n) If n = 1 then

Move Start’s top disk to End Else

Hanoi (Start, End, Temp, n-1) Move Start’s top disk to End Hanoi (Temp, Start, End, n-1)

Start Temp End






Computational Complexity• Resources required during computation

of an algorithm to solve a given problem– Time

• how many steps does it take to solve a problem?

– Space• how much memory does it take to solve a

problem?

• The Hanoi Towers Problem– f(n) is the number of times the HANOI

algorithm moves a disk for a problem of n disks

• f(1)=1, f(2)=3, f(3)=7• f(n)= f(n-1) + 1 + f(n-1) = 2 f(n-1) + 1

– Every time we add a disk, the time to compute is at least double

• f(n) = 2n - 1

616,551,709,073,744,446,182

4906,842,621,125,899,)(2

776,627,511,099,1)(2

824,741,073,1)(2

576,048,1)(2

024,1)(2

64

50

40

30

20

10

PETA

TERA

GIGA

MEGA

KILO

585 billion years in seconds!!!!!!!!

Earth: 5 billion years

Universe: 15 billion years

Fastest Computer: 1petaflops - 1015 (approx 250 instructions a second)

214 s needed = 4.6 hours

"FLOPS"

(FLoating Point Operations Per Second)






Fastest Computers

Fastest Computer (early 2005): 135.5 teraflops - 135.5 trillion calculations a second --- Approaching petaflops: 250

Fastest Computer (late 2005): 280.6 teraflops - 280.6 trillion calculations a second --- Approaching petaflops: 3 petaflops in late 2006????

Fastest Computer (june 2006): 1 petaflop !!! – 1 quadrillion calculations per second --- MDGRAPE-3 @ Riken, Japan

MDGRAPE-3: Not a general-purpose computer

Fastest general-purpose computer (may 2008): 1 petaflop !!! – 1 quadrillion calculations per second --- Roadrunner @ Los Alamos--- aprox 214 s needed = 4.6 hours for Hanoi problem (assuming one disk change per operation)

IBM BlueGene/L

IBM Roadrunner

RIKEN & Fujitsu “K” Computer: 8.162 petaflops 672 racks, 68544 CPUs…

K computer






Bremermann's Limit

Physical Limit of Computation Hans Bremmermann in 1962 “no data processing system, whether artificial

or living, can process more than 2 × 1047 bits per second per gram of its mass.”

Based on the idea that information could be stored in the energy levels of matter

Calculated using Heisenberg's uncertainty principle, the Hartley measure, Planck's constant, and Einstein's famous E = mc2 formula

A computer with the mass of the entire Earth and a time period equal to the estimated age of the Earth

would not be able to process more than about 1093 bits

= transcomputational problems






Transcomputational Problems

• A system of n variables, each of which can take k different states– kn possible system states– When is it larger than 1093?

– Pattern Recognition• Grid of n = q2 squares of k colors• Blackbox: 10100 possible states!

– The human retina contains millions of light-sensitive cells– Large scale integrated digital circuits

• K= 2 (bits): a circuit with 308 inputs and one output!• Complex problems need simplification/compression!

k 2 3 4 5 6 7 8 9 10

n 308 194 154 133 119 110 102 97 93






What happens to Moore’s law ?






BUT!

“Even more remarkable – and even less widely understood – is that in many areas, performance gains due to improvements in algorithms have vastly exceeded even the dramatic performance gains due to increased processor speed” http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-nitrd-report-2010.pdf






Is the singularity near?• Ray Kurtzweil, Vernor Vinge• Technological progress grows exponentially and reaches

infinity in finite time (=singularity)• Sigmoidal function?

http://www.shirky.com/herecomeseverybody/2008/04/looking-for-the-mouse.html

Also, noteworthy: metasystem transition (Turchin/Heylighen), Clay Shirky’s “gin analogy”






computational complexity (algorithms) Computational complexity pertains to systems problems

Characterization of the time or space (memory) requirements for solving a problem by a particular algorithm

Time complexity function f(n) is the largest amount of time needed for an algorithm to solve a problem instance of

size n. Polynomial time algorithms

Complexity :O(nk): can be computed by deterministic turing machine in a polynomial time function of order k.

“Efficient” or “tractable” algorithms, P class Exponential time algorithms

Exponential time function, e.g. O(2n), O(10n), O(nn), O(n!) Inefficient or intractable algorithms

Non-deterministic Polynomial time algorithms (NP): Solvable by non-deterministic Turing machine in polynomial time (“guessing”) Answers verifiable by deterministic Turing Machine in polynomial time (“verification”)

NP-complete: no polynomial time solution is known, although answers can be verified by deterministic Turing machine in polynomial time (e.g. traveling salesman problem) P \subset NP, but P = NP? Need for approximation algorithms






Computational complexityGrowth rates for time complexity functions

Assuming a million operations per second






Computational complexityEffect of increasing computing speed

i501 – introduction to informatics [email protected] info rm atics and computing lecture 8 –...

Documents

computing complexity

informatics slide

computing lecture

informatics jbollen

i501 introduction

computational complexity

complexity of c

kolmogorov complexity