Computational complexity theory abcFrom Wikipedia, the free encyclopedia

Contents

1 Aanderaa–Karp–Rosenberg conjecture 11.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Query complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Deterministic query complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 Randomized query complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.6 Quantum query complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.9 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Advice (complexity) 62.1 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Analysis of algorithms 83.1 Cost models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 Run-time analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2.1 Shortcomings of empirical metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2.2 Orders of growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2.3 Empirical orders of growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2.4 Evaluating run-time complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2.5 Growth rate analysis of other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3 Relevance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.4 Constant factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.5 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4 Approximation algorithm 144.1 Performance guarantees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2 Algorithm design techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.3 Epsilon terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.4 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

i

ii CONTENTS

4.5 Citations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.7 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5 Approximation-preserving reduction 175.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.2 Types of approximation-preserving reductions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.2.1 Strict reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.2.2 L-reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.2.3 PTAS-reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.2.4 A-reduction and P-reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185.2.5 E-reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.2.6 AP-reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.2.7 Gap reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6 Asymptotic computational complexity 216.1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.2 Types of algorithms considered . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

7 Averaging argument 227.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227.2 Formalized definition of averaging argument . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227.3 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

8 Best, worst and average case 248.1 Best-case performance for algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248.2 Worst-case versus average-case performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248.3 Practical consequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

8.4.1 Sorting algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258.4.2 Data structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258.4.3 Graph search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

8.5 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

9 Boolean circuit 279.1 Formal definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279.2 Computational complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

CONTENTS iii

9.2.1 Evaluation of a circuit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279.2.2 Complexity measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279.2.3 Complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

9.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289.4 Footnotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

10 Certificate (complexity) 2910.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2910.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2910.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2910.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

11 Circuit complexity 3011.1 Uniformity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

11.1.1 Polynomial-time uniform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3111.1.2 Logspace uniform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

11.2 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3111.3 Circuit lower bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3111.4 Complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3211.5 Relation to time complexity[1] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3211.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

12 Circuits over sets of natural numbers 3412.1 Formal definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

12.1.1 Algorithmic problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3412.1.2 Restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3512.1.3 Quickly growing set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

12.2 Complexity results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3512.2.1 Membership problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3512.2.2 Equivalence problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

12.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3512.4 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

13 Cobham’s thesis 3613.1 Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3713.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

14 Combinatorial optimization 3814.1 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3914.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3914.3 Specific problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3914.4 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

iv CONTENTS

14.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4114.6 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

15 Combinatorial search 4215.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4215.2 Lookahead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4215.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4315.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

16 Communication complexity 4416.1 Formal definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

16.1.1 Example: EQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4516.2 Randomized communication complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

16.2.1 Example: EQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4516.2.2 Public coins versus private coins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

16.3 Quantum communication complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4716.4 Nondeterministic communication complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4716.5 Open problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4716.6 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4816.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4816.8 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

17 Complement (complexity) 4917.1 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

18 Complete (complexity) 5118.1 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

19 Complexity class 5219.1 Important complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5219.2 Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5319.3 Closure properties of classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5319.4 Relationships between complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

19.4.1 Hierarchy theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5419.5 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5419.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5419.7 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

20 Complexity index 5520.1 Definition of sentry function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

20.1.1 Definition of detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5620.1.2 Example: continuous spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5720.1.3 Example: discrete spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

20.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

CONTENTS v

21 Compression theorem 5821.1 Compression theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5821.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

22 Computable topology 5922.1 Computational topology from λ-calculus topology . . . . . . . . . . . . . . . . . . . . . . . . . . 5922.2 Computational algebra from λ-calculus algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5922.3 Scott topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6022.4 Böhm trees and computational topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

22.4.1 Böhm tree and tree topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6322.5 Algebraic structure of computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6322.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

23 Computation tree 6623.1 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

24 Computational complexity of mathematical operations 6724.1 Arithmetic functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6724.2 Algebraic functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6724.3 Special functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

24.3.1 Elementary functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6724.3.2 Non-elementary functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6724.3.3 Mathematical constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

24.4 Number theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6824.5 Matrix algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6824.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6824.7 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

25 Computational complexity theory 7025.1 Computational problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

25.1.1 Problem instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7025.1.2 Representing problem instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7125.1.3 Decision problems as formal languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7125.1.4 Function problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7325.1.5 Measuring the size of an instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

25.2 Machine models and complexity measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7325.2.1 Turing machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7325.2.2 Other machine models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7425.2.3 Complexity measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7425.2.4 Best, worst and average case complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . 7525.2.5 Upper and lower bounds on the complexity of problems . . . . . . . . . . . . . . . . . . . 75

25.3 Complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7625.3.1 Defining complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

vi CONTENTS

25.3.2 Important complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7625.3.3 Hierarchy theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7725.3.4 Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

25.4 Important open problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7825.4.1 P versus NP problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7825.4.2 Problems in NP not known to be in P or NP-complete . . . . . . . . . . . . . . . . . . . . 7925.4.3 Separations between other complexity classes . . . . . . . . . . . . . . . . . . . . . . . . 79

25.5 Intractability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8025.6 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8025.7 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8125.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

25.8.1 Textbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8225.8.2 Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

25.9 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

26 Computational resource 8326.1 Describing generally accessible computing equipment . . . . . . . . . . . . . . . . . . . . . . . . 8326.2 Formal quantification of computing capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8326.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

27 Computational topology 8427.1 Major algorithms by subject area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

27.1.1 Algorithmic 3-manifold theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8427.1.2 Algorithmic knot theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8527.1.3 Computational homotopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8527.1.4 Computational homology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

27.2 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8527.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8627.4 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8627.5 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

28 Computationally bounded adversary 8728.1 Comparison to other models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

28.1.1 Worst-case model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8728.1.2 Stochastic noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

28.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8828.2.1 Comparison to stochastic noise channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

28.3 Specific applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8828.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

29 Computing the permanent 9129.1 Definition and naive algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9129.2 Ryser formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

CONTENTS vii

29.3 Balasubramanian-Bax/Franklin-Glynn formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9229.4 Special cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

29.4.1 Planar and K₃,₃-free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9229.4.2 Computation modulo a number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

29.5 Approximate computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9329.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9329.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

30 Configuration graph 9530.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9530.2 Useful property . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9530.3 Size of the graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9530.4 Use of this object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9630.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

31 Constructible function 9731.1 Time-constructible definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9731.2 Space-constructible definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9731.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9731.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

32 Context of computational complexity 9932.1 Definitions of variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9932.2 Abstract machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10032.3 Metric being measured . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

33 Padding argument 10133.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10133.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

34 The Complexity of Songs 10234.1 Article summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10234.2 Further developments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10234.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10334.4 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10334.5 Text and image sources, contributors, and licenses . . . . . . . . . . . . . . . . . . . . . . . . . . 104

34.5.1 Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10434.5.2 Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10634.5.3 Content license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Chapter 1

Aanderaa–Karp–Rosenberg conjecture

In theoretical computer science, theAanderaa–Karp–Rosenberg conjecture (also known as theAanderaa–Rosenbergconjecture or the evasiveness conjecture) is a group of related conjectures about the number of questions of theform “Is there an edge between vertex u and vertex v?" that have to be answered to determine whether or not anundirected graph has a particular property such as planarity or bipartiteness. They are named after Stål Aanderaa,Richard M. Karp, and Arnold L. Rosenberg. According to the conjecture, for a wide class of properties, no algo-rithm can guarantee that it will be able to skip any questions: any algorithm for determining whether the graph hasthe property, no matter how clever, might need to examine every pair of vertices before it can give its answer. Aproperty satisfying this conjecture is called evasive.More precisely, the Aanderaa–Rosenberg conjecture states that any deterministic algorithm must test at least a con-stant fraction of all possible pairs of vertices, in the worst case, to determine any non-trivial monotone graph property;in this context, a property is monotone if it remains true when edges are added (so planarity is not monotone, butnon-planarity is monotone). A stronger version of this conjecture, called the evasiveness conjecture or the Aanderaa–Karp–Rosenberg conjecture, states that exactly n(n − 1)/2 tests are needed. Versions of the problem for randomizedalgorithms and quantum algorithms have also been formulated and studied.The deterministic Aanderaa–Rosenberg conjecture was proven by Rivest & Vuillemin (1975), but the strongerAanderaa–Karp–Rosenberg conjecture remains unproven. Additionally, there is a large gap between the conjec-tured lower bound and the best proven lower bound for randomized and quantum query complexity.

1.1 Example

The property of being non-empty (that is, having at least one edge) is monotone, because adding another edge toa non-empty graph produces another non-empty graph. There is a simple algorithm for testing whether a graph isnon-empty: loop through all of the pairs of vertices, testing whether each pair is connected by an edge. If an edgeis ever found in this way, break out of the loop, and report that the graph is non-empty, and if the loop completeswithout finding any edges, then report that the graph is empty. On some graphs (for instance the complete graphs) thisalgorithm will terminate quickly, without testing every pair of vertices, but on the empty graph it tests all possiblepairs before terminating. Therefore, the query complexity of this algorithm is n(n − 1)/2: in the worst case, thealgorithm performs n(n − 1)/2 tests.The algorithm described above is not the only possible method of testing for non-emptiness, but the Aanderaa–Karp–Rosenberg conjecture implies that every deterministic algorithm for testing non-emptiness has the same querycomplexity, n(n − 1)/2. That is, the property of being non-empty is evasive. For this property, the result is easy toprove directly: if an algorithm does not perform n(n − 1)/2 tests, it cannot distinguish the empty graph from a graphthat has one edge connecting one of the untested pairs of vertices, and must give an incorrect answer on one of thesetwo graphs.

1

2 CHAPTER 1. AANDERAA–KARP–ROSENBERG CONJECTURE

1.2 Definitions

In the context of this article, all graphs will be simple and undirected, unless stated otherwise. This means that theedges of the graph form a set (and not a multiset) and each edge is a pair of distinct vertices. Graphs are assumed tohave an implicit representation in which each vertex has a unique identifier or label and in which it is possible to testthe adjacency of any two vertices, but for which adjacency testing is the only allowed primitive operation.Informally, a graph property is a property of a graph that is independent of labeling. More formally, a graph propertyis a mapping from the set of all graphs to 0,1 such that isomorphic graphs are mapped to the same value. Forexample, the property of containing at least 1 vertex of degree 2 is a graph property, but the property that the firstvertex has degree 2 is not, because it depends on the labeling of the graph (in particular, it depends on which vertex isthe “first” vertex). A graph property is called non-trivial if it doesn't assign the same value to all graphs. For instance,the property of being a graph is a trivial property, since all graphs possess this property. On the other hand, theproperty of being empty is non-trivial, because the empty graph possesses this property, but non-empty graphs donot. A graph property is said to be monotone if the addition of edges does not destroy the property. Alternately, ifa graph possesses a monotone property, then every supergraph of this graph on the same vertex set also possessesit. For instance, the property of being nonplanar is monotone: a supergraph of a nonplanar graph is itself nonplanar.However, the property of being regular is not monotone.The big O notation is often used for query complexity. In short, f(n) is O(g(n)) if for large enough n, f(n) ≤ c g(n)for some positive constant c. Similarly, f(n) is Ω(g(n)) if for large enough n, f(n) ≥ c g(n) for some positive constantc. Finally, f(n) is Θ(g(n)) if it is both O(g(n)) and Ω(g(n)).

1.3 Query complexity

The deterministic query complexity of evaluating a function on n bits (x1, x2, ..., xn) is the number of bits xᵢ that haveto be read in the worst case by a deterministic algorithm to determine the value of the function. For instance, if thefunction takes value 0 when all bits are 0 and takes value 1 otherwise (this is the OR function), then the deterministicquery complexity is exactly n. In the worst case, the first n − 1 bits read could all be 0, and the value of the functionnow depends on the last bit. If an algorithm doesn't read this bit, it might output an incorrect answer. (Such argumentsare known as adversary arguments.) The number of bits read are also called the number of queries made to the input.One can imagine that the algorithm asks (or queries) the input for a particular bit and the input responds to this query.The randomized query complexity of evaluating a function is defined similarly, except the algorithm is allowed to berandomized, i.e., it can flip coins and use the outcome of these coin flips to decide which bits to query. However,the randomized algorithm must still output the correct answer for all inputs: it is not allowed to make errors. Suchalgorithms are called Las Vegas algorithms, which distinguishes them from Monte Carlo algorithms which are allowedto make some error. Randomized query complexity can also be defined in the Monte Carlo sense, but the Aanderaa–Karp–Rosenberg conjecture is about the Las Vegas query complexity of graph properties.Quantum query complexity is the natural generalization of randomized query complexity, of course allowing quantumqueries and responses. Quantum query complexity can also be defined with respect to Monte Carlo algorithms or LasVegas algorithms, but it is usually taken to mean Monte Carlo quantum algorithms.In the context of this conjecture, the function to be evaluated is the graph property, and the input is a string of sizen(n − 1)/2, which gives the locations of the edges on an n vertex graph, since a graph can have at most n(n − 1)/2possible edges. The query complexity of any function is upper bounded by n(n − 1)/2, since the whole input is readafter making n(n − 1)/2 queries, thus determining the input graph completely.

1.4 Deterministic query complexity

For deterministic algorithms, Rosenberg (1973) originally conjectured that for all nontrivial graph properties on nvertices, deciding whether a graph possesses this property requires Ω(n2) queries. The non-triviality condition isclearly required because there are trivial properties like “is this a graph?" which can be answered with no queries atall.The conjecture was disproved by Aanderaa, who exhibited a directed graph property (the property of containing a“sink”) which required only O(n) queries to test. A sink, in a directed graph, is a vertex of indegree n−1 and outdegree

1.5. RANDOMIZED QUERY COMPLEXITY 3

0. This property can be tested with less than 3n queries (Best, van Emde Boas & Lenstra 1974). An undirected graphproperty which can also be tested with O(n) queries is the property of being a scorpion graph, first described in Best,van Emde Boas & Lenstra (1974). A scorpion graph is a graph containing a three-vertex path, such that one endpointof the path is connected to all remaining vertices, while the other two path vertices have no incident edges other thanthe ones in the path.Then Aanderaa and Rosenberg formulated a new conjecture (the Aanderaa–Rosenberg conjecture) which says thatdeciding whether a graph possesses a non-trivial monotone graph property requires Ω(n2) queries.[1] This conjecturewas resolved by Rivest & Vuillemin (1975) by showing that at least n2/16 queries are needed to test for any nontrivialmonotone graph property. The bound was further improved to n2/9 by Kleitman & Kwiatkowski (1980), then to n2/4- o(n2) by Kahn, Saks & Sturtevant (1983), then to (8/25)n2 - o(n2) by Korneffel & Triesch (2010), and then to n2/3- o(n2) by Scheidweiler & Triesch (2013).Richard Karp conjectured the stronger statement (which is now called the evasiveness conjecture or the Aanderaa–Karp–Rosenberg conjecture) that “every nontrivial monotone graph property for graphs on n vertices is evasive.”[2]

A property is called evasive if determining whether a given graph has this property sometimes requires all n(n − 1)/2queries.[3] This conjecture says that the best algorithm for testing any nontrivial monotone property must (in the worstcase) query all possible edges. This conjecture is still open, although several special graph properties have shown to beevasive for all n. The conjecture has been resolved for the case where n is a prime power by Kahn, Saks & Sturtevant(1983) using a topological approach. The conjecture has also been resolved for all non-trivial monotone properties onbipartite graphs by Yao (1988). Minor-closed properties have also been shown to be evasive for large n (Chakrabarti,Khot & Shi 2001).

1.5 Randomized query complexity

Richard Karp also conjectured that Ω(n2) queries are required for testing nontrivial monotone properties even ifrandomized algorithms are permitted. No nontrivial monotone property is known which requires less than n2/4queries to test. A linear lower bound (i.e., Ω(n)) follows from a very general relationship between randomized anddeterministic query complexities. The first superlinear lower bound for this problem was given by Yao (1991) whoshowed that Ω(n log1/12 n) queries are required. This was further improved by King (1988) to Ω(n5/4), and then byHajnal (1991) to Ω(n4/3). This was subsequently improved to the current best known lower bound of Ω(n4/3 log1/3 n)by Chakrabarti & Khot (2001).Some recent results give lower bounds which are determined by the critical probability p of the monotone graphproperty under consideration. The critical probability p is defined as the unique p such that a random graph G(n, p)possesses this property with probability equal to 1/2. A random graph G(n, p) is a graph on n vertices where eachedge is chosen to be present with probability p independent of all the other edges. Friedgut, Kahn & Wigderson(2002) showed that any monotone property with critical probability p requires Ω

(min

n

min(p,1−p) ,n2

log n

)queries.

For the same problem, O'Donnell et al. (2005) showed a lower bound of Ω(n4/3/p1/3).As in the deterministic case, there are many special properties for which an Ω(n2) lower bound is known. Moreover,better lower bounds are known for several classes of graph properties. For instance, for testing whether the graphhas a subgraph isomorphic to any given graph (the so-called subgraph isomorphism problem), the best known lowerbound is Ω(n3/2) due to Gröger (1992).

1.6 Quantum query complexity

For bounded-error quantum query complexity, the best known lower bound is Ω(n2/3 log1/6 n) as observed by AndrewYao.[4] It is obtained by combining the randomized lower bound with the quantum adversary method. The bestpossible lower bound one could hope to achieve is Ω(n), unlike the classical case, due to Grover’s algorithm whichgives an O(n) query algorithm for testing the monotone property of non-emptiness. Similar to the deterministicand randomized case, there are some properties which are known to have an Ω(n) lower bound, for example non-emptiness (which follows from the optimality of Grover’s algorithm) and the property of containing a triangle. Moreinterestingly, there are some graph properties which are known to have an Ω(n3/2) lower bound, and even someproperties with an Ω(n2) lower bound. For example, the monotone property of nonplanarity requires Θ(n3/2) queries(Ambainis et al. 2008) and the monotone property of containing more than half the possible number of edges (alsocalled the majority function) requires Θ(n2) queries (Beals et al. 2001).

4 CHAPTER 1. AANDERAA–KARP–ROSENBERG CONJECTURE

1.7 Notes[1] Triesch (1996)

[2] Lutz (2001)

[3] Kozlov (2008, pp. 226–228)

[4] The result is unpublished, but mentioned in Magniez, Santha & Szegedy (2005).

1.8 References

• Ambainis, Andris; Iwama, Kazuo; Nakanishi, Masaki; Nishimura, Harumichi; Raymond, Rudy; Tani, Sei-ichiro; Yamashita, Shigeru (2008), “Quantum query complexity of Boolean functions with small on-sets”,Proceedings of the 19th International Symposium on Algorithms and Computation, Lecture Notes in ComputerScience 5369, Gold Coast, Australia: Springer-Verlag, pp. 907–918, doi:10.1007/978-3-540-92182-0_79,ISBN 978-3-540-92181-3.

• Beals, Robert; Buhrman, Harry; Cleve, Richard; Mosca, Michele; de Wolf, Ronald (2001), “Quantum lowerbounds by polynomials”, Journal of the ACM 48 (4): 778–797, doi:10.1145/502090.502097.

• Best, M.R.; van Emde Boas, P.; Lenstra, H.W. (1974), A sharpened version of the Aanderaa-Rosenberg con-jecture, Report ZW 30/74, Mathematisch Centrum Amsterdam, hdl:1887/3792.

• Chakrabarti, Amit; Khot, Subhash (2001), “Improved Lower Bounds on the Randomized Complexity of GraphProperties”, Proceedings of the 28th International Colloquium on Automata, Languages and Programming,Lecture Notes in Computer Science 2076, Springer-Verlag, pp. 285–296, doi:10.1007/3-540-48224-5_24,ISBN 978-3-540-42287-7.

• Chakrabarti, Amit; Khot, Subhash; Shi, Yaoyun (2001), “Evasiveness of subgraph containment and relatedproperties”, SIAM Journal on Computing 31 (3): 866–875, doi:10.1137/S0097539700382005.

• Friedgut, Ehud; Kahn, Jeff; Wigderson, Avi (2002), “Computing Graph Properties by Randomized SubcubePartitions”, Randomization and Approximation Techniques in Computer Science, Lecture Notes in ComputerScience 2483, Springer-Verlag, p. 953, doi:10.1007/3-540-45726-7_9, ISBN 978-3-540-44147-2.

• Gröger, Hans Dietmar (1992), “On the randomized complexity of monotone graph properties” (PDF), ActaCybernetica 10 (3): 119–127.

• Hajnal, Péter (1991), “An Ω(n4/3) lower bound on the randomized complexity of graph properties”,Combinatorica11 (2): 131–143, doi:10.1007/BF01206357.

• Kahn, Jeff; Saks, Michael; Sturtevant, Dean (1983), “A topological approach to evasiveness”, 24th AnnualSymposium on Foundations of Computer Science (sfcs 1983), Los Alamitos, CA, USA: IEEE Computer Society,pp. 31–33, doi:10.1109/SFCS.1983.4, ISBN 0-8186-0508-1.

• King, Valerie (1988), “Lower bounds on the complexity of graph properties”, Proc. 20th ACM Symposiumon Theory of Computing, Chicago, Illinois, United States, pp. 468–476, doi:10.1145/62212.62258, ISBN0-89791-264-0.

• Kleitman, D.J.; Kwiatkowski, DJ (1980), “Further results on the Aanderaa-Rosenberg conjecture”, Journal ofCombinatorial Theory. Series B 28: 85–95, doi:10.1016/0095-8956(80)90057-X.

• Kozlov, Dmitry (2008), Combinatorial Algebraic Topology, Springer-Verlag, ISBN 978-3-540-73051-4.

• Lutz, Frank H. (2001), “Some results related to the evasiveness conjecture”, Journal of Combinatorial Theory,Series B 81 (1): 110–124, doi:10.1006/jctb.2000.2000.

• Korneffel, Torsten; Triesch, Eberhard (2010), “An asymptotic bound for the complexity of monotone graphproperties”, Combinatorica (Springer-Verlag) 30 (6): 735–743, doi:10.1007/s00493-010-2485-3, ISSN 0209-9683.

1.9. FURTHER READING 5

• Magniez, Frédéric; Santha, Miklos; Szegedy, Mario (2005), “Quantum algorithms for the triangle prob-lem”, Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms, Vancouver, BritishColumbia: Society for Industrial and Applied Mathematics, pp. 1109–1117, arXiv:quant-ph/0310134.

• O'Donnell, Ryan; Saks, Michael; Schramm, Oded; Servedio, Rocco A. (2005), “Every decision tree has an in-fluential variable”, Proc 46th IEEE Symposium on Foundations of Computer Science, pp. 31–39, doi:10.1109/SFCS.2005.34,ISBN 0-7695-2468-0.

• Rivest, Ronald L.; Vuillemin, Jean (1975), “A generalization and proof of the Aanderaa-Rosenberg conjec-ture”, Proc. 7th ACM Symposium on Theory of Computing, Albuquerque, New Mexico, United States, pp.6–11, doi:10.1145/800116.803747.

• Rosenberg, Arnold L. (1973), “On the time required to recognize properties of graphs: a problem”, SIGACTNews 5 (4): 15–16, doi:10.1145/1008299.1008302.

• Scheidweiler, Robert; Triesch, Eberhard (2013), “A Lower Bound for the Complexity of Monotone GraphProperties”, SIAM Journal on Discrete Mathematics 27 (1): 257–265, doi:10.1137/120888703

• Triesch, Eberhard (1996), “On the recognition complexity of some graph properties”, Combinatorica 16 (2):259–268, doi:10.1007/BF01844851.

• Yao, Andrew Chi-Chih (1988), “Monotone bipartite graph properties are evasive”, SIAM Journal on Computing17 (3): 517–520, doi:10.1137/0217031.

• Yao, Andrew Chi-Chih (1991), “Lower bounds to randomized algorithms for graph properties”, Journal ofComputer and System Sciences 42 (3): 267–287, doi:10.1016/0022-0000(91)90003-N.

1.9 Further reading• Bollobás, Béla (2004), “Chapter VIII. Complexity and packing”, Extremal Graph Theory, New York: Dover

Publications, pp. 401–437, ISBN 978-0-486-43596-1.

• László Lovász; Young, Neal E. (2002). “Lecture Notes on Evasiveness of Graph Properties”. arXiv:cs/0205031v1[cs.CC].

• Chronaki, Catherine E (1990), A survey of Evasiveness: Lower Bounds on the Decision-Tree Complexity ofBoolean Functions, CiteSeerX: 10 .1 .1 .37 .1041.

• Michael Saks. “Decision Trees: Problems and Results, Old and New” (PDF).

Chapter 2

Advice (complexity)

In computational complexity theory, an advice string is an extra input to a Turing machine which is allowed todepend on the length n of the input, but not on the input itself. A decision problem is in the complexity class P/f(n)if there is a polynomial time Turing machine M with the following property: for any n, there is an advice string A oflength f(n) such that, for any input x of length n, the machine M correctly decides the problem on the input x, givenx and A.The most common complexity class involving advice is P/poly where advice length f(n) can be any polynomial in n.P/poly is equal to the class of decision problems such that, for every n, there exists a polynomial size Boolean circuitcorrectly deciding the problem on all inputs of length n. One direction of the equivalence is easy to see. If, for everyn, there is a polynomial size Boolean circuit A(n) deciding the problem, we can use a Turing machine that interpretsthe advice string as a description of the circuit. Then, given the description of A(n) as the advice, the machine willcorrectly decide the problem on all inputs of length n. The other direction uses a simulation of a polynomial-timeTuring machine by a polynomial-size circuit as in one proof of Cook’s Theorem. Simulating a Turing machine withadvice is no more complicated than simulating an ordinary machine, since the advice string can be incorporated intothe circuit.[1]

Because of this equivalence, P/poly is sometimes defined as the class of decision problems solvable by polynomialsize Boolean circuits, or by polynomial-size non-uniform Boolean circuits.P/poly contains both P and BPP (Adleman’s theorem). It also contains some undecidable problems, such as theunary version of every undecidable problem, including the halting problem. Because of that, it is not contained inDTIME (f(n)) or NTIME (f(n)) for any f.Advice classes can be defined for other resource bounds instead of P. For example, taking a non-deterministic poly-nomial time Turing machine with an advice of length f(n) gives the complexity class NP/f(n). If we are allowed anadvice of length 2n, we can use it to encode whether each input of length n is contained in the language. Therefore,any boolean function is computable with an advice of length 2n and advice of more than exponential length is notmeaningful.Similarly, the class L/poly can be defined as deterministic logspace with a polynomial amount of advice.Known results include:

• The classes NL/poly and UL/poly are the same, i.e. nondeterministic logarithmic space computation withadvice can be made unambiguous.[2] This may be proved using an isolation lemma.[3]

• It is known that coNEXP is contained in NEXP/poly.[4]

• If NP is contained in P/poly, then the polynomial time hierarchy collapses (Karp-Lipton theorem).

2.1 References[1] Arora, Sanjeev; Barak, Boaz (2009), Computational Complexity: A Modern Approach, Cambridge University Press, p. 113,

ISBN 9780521424264, Zbl 1193.68112.

6

2.1. REFERENCES 7

[2] Reinhardt, Klaus; Allender, Eric (2000). “Making nondeterminism unambiguous”. SIAM J. Comput. 29 (4): 1118–1131.doi:10.1137/S0097539798339041. Zbl 0947.68063.

[3] Hemaspaandra, Lane A.; Ogihara, Mitsunori (2002). The complexity theory companion. Texts in Theoretical ComputerScience. An EATCS Series. Berlin: Springer-Verlag. ISBN 3-540-67419-5. Zbl 0993.68042.

[4] Lance Fortnow, A Little Theorem

Chapter 3

Analysis of algorithms

In computer science, the analysis of algorithms is the determination of the amount of resources (such as time andstorage) necessary to execute them. Most algorithms are designed to work with inputs of arbitrary length. Usually,the efficiency or running time of an algorithm is stated as a function relating the input length to the number of steps(time complexity) or storage locations (space complexity).Algorithm analysis is an important part of a broader computational complexity theory, which provides theoreticalestimates for the resources needed by any algorithm which solves a given computational problem. These estimatesprovide an insight into reasonable directions of search for efficient algorithms.In theoretical analysis of algorithms it is common to estimate their complexity in the asymptotic sense, i.e., to estimatethe complexity function for arbitrarily large input. Big O notation, Big-omega notation and Big-theta notation areused to this end. For instance, binary search is said to run in a number of steps proportional to the logarithm of thelength of the list being searched, or in O(log(n)), colloquially “in logarithmic time". Usually asymptotic estimatesare used because different implementations of the same algorithm may differ in efficiency. However the efficienciesof any two “reasonable” implementations of a given algorithm are related by a constant multiplicative factor called ahidden constant.Exact (not asymptotic) measures of efficiency can sometimes be computed but they usually require certain assump-tions concerning the particular implementation of the algorithm, called model of computation. A model of com-putation may be defined in terms of an abstract computer, e.g., Turing machine, and/or by postulating that certainoperations are executed in unit time. For example, if the sorted list to which we apply binary search has n elements,and we can guarantee that each lookup of an element in the list can be done in unit time, then at most log2 n + 1 timeunits are needed to return an answer.

3.1 Cost models

Time efficiency estimates depend on what we define to be a step. For the analysis to correspond usefully to the actualexecution time, the time required to perform a step must be guaranteed to be bounded above by a constant. One mustbe careful here; for instance, some analyses count an addition of two numbers as one step. This assumption may notbe warranted in certain contexts. For example, if the numbers involved in a computation may be arbitrarily large, thetime required by a single addition can no longer be assumed to be constant.Two cost models are generally used:[1][2][3][4][5]

• the uniform cost model, also called uniform-cost measurement (and similar variations), assigns a constantcost to every machine operation, regardless of the size of the numbers involved

• the logarithmic cost model, also called logarithmic-cost measurement (and variations thereof), assigns acost to every machine operation proportional to the number of bits involved

The latter is more cumbersome to use, so it’s only employed when necessary, for example in the analysis of arbitrary-precision arithmetic algorithms, like those used in cryptography.

8

3.2. RUN-TIME ANALYSIS 9

A key point which is often overlooked is that published lower bounds for problems are often given for a model ofcomputation that is more restricted than the set of operations that you could use in practice and therefore there arealgorithms that are faster than what would naively be thought possible.[6]

3.2 Run-time analysis

Run-time analysis is a theoretical classification that estimates and anticipates the increase in running time (or run-time)of an algorithm as its input size (usually denoted as n) increases. Run-time efficiency is a topic of great interest incomputer science: A program can take seconds, hours or even years to finish executing, depending on which algorithmit implements (see also performance analysis, which is the analysis of an algorithm’s run-time in practice).

3.2.1 Shortcomings of empirical metrics

Since algorithms are platform-independent (i.e. a given algorithm can be implemented in an arbitrary programminglanguage on an arbitrary computer running an arbitrary operating system), there are significant drawbacks to usingan empirical approach to gauge the comparative performance of a given set of algorithms.Take as an example a program that looks up a specific entry in a sorted list of size n. Suppose this program wereimplemented on Computer A, a state-of-the-art machine, using a linear search algorithm, and on Computer B, a muchslower machine, using a binary search algorithm. Benchmark testing on the two computers running their respectiveprograms might look something like the following:Based on these metrics, it would be easy to jump to the conclusion that Computer A is running an algorithm that is farsuperior in efficiency to that of Computer B. However, if the size of the input-list is increased to a sufficient number,that conclusion is dramatically demonstrated to be in error:Computer A, running the linear search program, exhibits a linear growth rate. The program’s run-time is directlyproportional to its input size. Doubling the input size doubles the run time, quadrupling the input size quadruples therun-time, and so forth. On the other hand, Computer B, running the binary search program, exhibits a logarithmicgrowth rate. Quadrupling the input size only increases the run time by a constant amount (in this example, 50,000ns). Even though Computer A is ostensibly a faster machine, Computer B will inevitably surpass Computer A inrun-time because it’s running an algorithm with a much slower growth rate.

3.2.2 Orders of growth

Main article: Big O notation

Informally, an algorithm can be said to exhibit a growth rate on the order of a mathematical function if beyond acertain input size n, the function f(n) times a positive constant provides an upper bound or limit for the run-time ofthat algorithm. In other words, for a given input size n greater than some n0 and a constant c, the running time of thatalgorithm will never be larger than c × f(n). This concept is frequently expressed using Big O notation. For example,since the run-time of insertion sort grows quadratically as its input size increases, insertion sort can be said to be oforder O(n2).Big O notation is a convenient way to express the worst-case scenario for a given algorithm, although it can also beused to express the average-case — for example, the worst-case scenario for quicksort is O(n2), but the average-caserun-time is O(n log n).

3.2.3 Empirical orders of growth

Assuming the execution time follows power rule, t ≈ k na, the coefficient a can be found [7] by taking empiricalmeasurements of run time t1, t2 at some problem-size points n1, n2 , and calculating t2/t1 = (n2/n1)

a sothat a = log(t2/t1)/ log(n2/n1) . In other words, this measures the slope of the empirical line on the log–log plotof execution time vs. problem size, at some size point. If the order of growth indeed follows the power rule (and sothe line on log–log plot is indeed a straight line), the empirical value of a will stay constant at different ranges, and if

10 CHAPTER 3. ANALYSIS OF ALGORITHMS

not, it will change (and the line is a curved line) - but still could serve for comparison of any two given algorithms asto their empirical local orders of growth behaviour. Applied to the above table:It is clearly seen that the first algorithm exhibits a linear order of growth indeed following the power rule. Theempirical values for the second one are diminishing rapidly, suggesting it follows another rule of growth and in anycase has much lower local orders of growth (and improving further still), empirically, than the first one.

3.2.4 Evaluating run-time complexity

The run-time complexity for the worst-case scenario of a given algorithm can sometimes be evaluated by examiningthe structure of the algorithm and making some simplifying assumptions. Consider the following pseudocode:1 get a positive integer from input 2 if n > 10 3 print “This might take a while...” 4 for i = 1 to n 5 for j = 1 to i 6print i * j 7 print “Done!"A given computer will take a discrete amount of time to execute each of the instructions involved with carryingout this algorithm. The specific amount of time to carry out a given instruction will vary depending on which in-struction is being executed and which computer is executing it, but on a conventional computer, this amount will bedeterministic.[8] Say that the actions carried out in step 1 are considered to consume time T1, step 2 uses time T2,and so forth.In the algorithm above, steps 1, 2 and 7 will only be run once. For a worst-case evaluation, it should be assumed thatstep 3 will be run as well. Thus the total amount of time to run steps 1-3 and step 7 is:

T1 + T2 + T3 + T7.

The loops in steps 4, 5 and 6 are trickier to evaluate. The outer loop test in step 4 will execute ( n + 1 ) times (notethat an extra step is required to terminate the for loop, hence n + 1 and not n executions), which will consume T4( n +1 ) time. The inner loop, on the other hand, is governed by the value of i, which iterates from 1 to i. On the first passthrough the outer loop, j iterates from 1 to 1: The inner loop makes one pass, so running the inner loop body (step 6)consumes T6 time, and the inner loop test (step 5) consumes 2T5 time. During the next pass through the outer loop,j iterates from 1 to 2: the inner loop makes two passes, so running the inner loop body (step 6) consumes 2T6 time,and the inner loop test (step 5) consumes 3T5 time.Altogether, the total time required to run the inner loop body can be expressed as an arithmetic progression:

T6 + 2T6 + 3T6 + · · ·+ (n− 1)T6 + nT6

which can be factored[9] as

T6 [1 + 2 + 3 + · · ·+ (n− 1) + n] = T6

[1

2(n2 + n)

]The total time required to run the outer loop test can be evaluated similarly:

2T5 + 3T5 + 4T5 + · · ·+ (n− 1)T5 + nT5 + (n+ 1)T5

= T5 + 2T5 + 3T5 + 4T5 + · · ·+ (n− 1)T5 + nT5 + (n+ 1)T5 − T5

which can be factored as

T5 [1 + 2 + 3 + · · ·+ (n− 1) + n+ (n+ 1)]−T5 =

[1

2(n2 + n)

]T5+(n+1)T5−T5 = T5

[1

2(n2 + n)

]+nT5 =

[1

2(n2 + 3n)

]T5

Therefore the total running time for this algorithm is:

f(n) = T1 + T2 + T3 + T7 + (n+ 1)T4 +

[1

2(n2 + n)

]T6 +

[1

2(n2 + 3n)

]T5

3.3. RELEVANCE 11

which reduces to

f(n) =

[1

2(n2 + n)

]T6 +

[1

2(n2 + 3n)

]T5 + (n+ 1)T4 + T1 + T2 + T3 + T7

As a rule-of-thumb, one can assume that the highest-order term in any given function dominates its rate of growthand thus defines its run-time order. In this example, n² is the highest-order term, so one can conclude that f(n) =O(n²). Formally this can be proven as follows:

Prove that[12 (n

2 + n)]T6 +

[12 (n

2 + 3n)]T5 + (n+1)T4 + T1 + T2 + T3 + T7 ≤ cn2, n ≥ n0[

12 (n

2 + n)]T6 +

[12 (n

2 + 3n)]T5 + (n+ 1)T4 + T1 + T2 + T3 + T7

≤ (n2 + n)T6 + (n2 + 3n)T5 + (n+ 1)T4 + T1 + T2 + T3 + T7 (for n ≥ 0)Let k be a constant greater than or equal to [T1..T7]

T6(n2 +n) + T5(n

2 +3n) + (n+1)T4 + T1 + T2 + T3 + T7 ≤ k(n2 +n) + k(n2 +3n) + kn+5k= 2kn2 + 5kn+ 5k ≤ 2kn2 + 5kn2 + 5kn2 (for n ≥ 1) = 12kn2

Therefore[12 (n

2 + n)]T6 +

[12 (n

2 + 3n)]T5 + (n+ 1)T4 + T1 + T2 + T3 + T7 ≤ cn2, n ≥ n0

for c = 12k, n0 = 1

A more elegant approach to analyzing this algorithm would be to declare that [T1..T7] are all equal to one unit oftime, in a system of units chosen so that one unit is greater than or equal to the actual times for these steps. Thiswould mean that the algorithm’s running time breaks down as follows:[10]

4 +∑n

i=1 i ≤ 4 +∑n

i=1 n = 4 + n2 ≤ 5n2 (for n ≥ 1) = O(n2).

3.2.5 Growth rate analysis of other resources

The methodology of run-time analysis can also be utilized for predicting other growth rates, such as consumption ofmemory space. As an example, consider the following pseudocode which manages and reallocates memory usage bya program based on the size of a file which that program manages:while (file still open) let n = size of file for every 100,000 kilobytes of increase in file size double the amount of memoryreserved

In this instance, as the file size n increases, memory will be consumed at an exponential growth rate, which is orderO(2n). This is an extremely rapid and most likely unmanageable growth rate for consumption of memory resources.

3.3 Relevance

Algorithm analysis is important in practice because the accidental or unintentional use of an inefficient algorithmcan significantly impact system performance. In time-sensitive applications, an algorithm taking too long to run canrender its results outdated or useless. An inefficient algorithm can also end up requiring an uneconomical amount ofcomputing power or storage in order to run, again rendering it practically useless.

3.4 Constant factors

Analysis of algorithms typically focuses on the asymptotic performance, particularly at the elementary level, but inpractical applications constant factors are important, and real-world data is in practice always limited in size. Thelimit is typically the size of addressable memory, so on 32-bit machines 232 = 4 GiB (greater if segmented memoryis used) and on 64-bit machines 264 = 16 EiB. Thus given a limited size, an order of growth (time or space) can bereplaced by a constant factor, and in this sense all practical algorithms are O(1) for a large enough constant, or forsmall enough data.This interpretation is primarily useful for functions that grow extremely slowly: (binary) iterated logarithm (log*) isless than 5 for all practical data (265536 bits); (binary) log-log (log log n) is less than 6 for virtually all practical data

12 CHAPTER 3. ANALYSIS OF ALGORITHMS

(264 bits); and binary log (log n) is less than 64 for virtually all practical data (264 bits). An algorithm with non-constant complexity may nonetheless be more efficient than an algorithm with constant complexity on practical dataif the overhead of the constant time algorithm results in a larger constant factor, e.g., one may have K > k log lognso long as K/k > 6 and n < 22

6

= 264 .For large data linear or quadratic factors cannot be ignored, but for small data an asymptotically inefficient algorithmmay be more efficient. This is particularly used in hybrid algorithms, like Timsort, which use an asymptotically effi-cient algorithm (here merge sort, with time complexity n logn ), but switch to an asymptotically inefficient algorithm(here insertion sort, with time complexity n2 ) for small data, as the simpler algorithm is faster on small data.

3.5 See also• Amortized analysis

• Analysis of parallel algorithms

• Asymptotic computational complexity

• Best, worst and average case

• Big O notation

• Computational complexity theory

• Master theorem

• NP-Complete

• Numerical analysis

• Polynomial time

• Program optimization

• Profiling (computer programming)

• Scalability

• Smoothed analysis

• Termination analysis — the subproblem of checking whether a program will terminate at all

• Time complexity — includes table of orders of growth for common algorithms

3.6 Notes[1] Alfred V. Aho; John E. Hopcroft; Jeffrey D. Ullman (1974). The design and analysis of computer algorithms. Addison-

Wesley Pub. Co., section 1.3

[2] Juraj Hromkovič (2004). Theoretical computer science: introduction to Automata, computability, complexity, algorithmics,randomization, communication, and cryptography. Springer. pp. 177–178. ISBN 978-3-540-14015-3.

[3] Giorgio Ausiello (1999). Complexity and approximation: combinatorial optimization problems and their approximabilityproperties. Springer. pp. 3–8. ISBN 978-3-540-65431-5.

[4] Wegener, Ingo (2005), Complexity theory: exploring the limits of efficient algorithms, Berlin, New York: Springer-Verlag,p. 20, ISBN 978-3-540-21045-0

[5] Robert Endre Tarjan (1983). Data structures and network algorithms. SIAM. pp. 3–7. ISBN 978-0-89871-187-5.

[6] Examples of the price of abstraction?, cstheory.stackexchange.com

[7] How To Avoid O-Abuse and Bribes, at the blog “Gödel’s Lost Letter and P=NP” by R. J. Lipton, professor of ComputerScience at Georgia Tech, recounting idea by Robert Sedgewick

3.7. REFERENCES 13

[8] However, this is not the case with a quantum computer

[9] It can be proven by induction that 1 + 2 + 3 + · · ·+ (n− 1) + n = n(n+1)2

[10] This approach, unlike the above approach, neglects the constant time consumed by the loop tests which terminate theirrespective loops, but it is trivial to prove that such omission does not affect the final result

3.7 References• Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L. & Stein, Clifford (2001). Introduction to Al-gorithms. Chapter 1: Foundations (Second ed.). Cambridge, MA: MIT Press and McGraw-Hill. pp. 3–122.ISBN 0-262-03293-7.

• Sedgewick, Robert (1998). Algorithms in C, Parts 1-4: Fundamentals, Data Structures, Sorting, Searching (3rded.). Reading, MA: Addison-Wesley Professional. ISBN 978-0-201-31452-6.

• Knuth, Donald. The Art of Computer Programming. Addison-Wesley.

• Greene, Daniel A.; Knuth, Donald E. (1982). Mathematics for the Analysis of Algorithms (Second ed.).Birkhäuser. ISBN 3-7643-3102-X.

• Goldreich, Oded (2010). Computational Complexity: A Conceptual Perspective. Cambridge University Press.ISBN 978-0-521-88473-0.

Chapter 4

Approximation algorithm

In computer science and operations research, approximation algorithms are algorithms used to find approximatesolutions to optimization problems. Approximation algorithms are often associated with NP-hard problems; sinceit is unlikely that there can ever be efficient polynomial-time exact algorithms solving NP-hard problems, one set-tles for polynomial-time sub-optimal solutions. Unlike heuristics, which usually only find reasonably good solutionsreasonably fast, one wants provable solution quality and provable run-time bounds. Ideally, the approximation isoptimal up to a small constant factor (for instance within 5% of the optimal solution). Approximation algorithms areincreasingly being used for problems where exact polynomial-time algorithms are known but are too expensive dueto the input size. A typical example for an approximation algorithm is the one for vertex cover in graphs: find anuncovered edge and add both endpoints to the vertex cover, until none remain. It is clear that the resulting cover is atmost twice as large as the optimal one. This is a constant factor approximation algorithm with a factor of 2.NP-hard problems vary greatly in their approximability; some, such as the bin packing problem, can be approximatedwithin any factor greater than 1 (such a family of approximation algorithms is often called a polynomial time approxi-mation scheme or PTAS). Others are impossible to approximate within any constant, or even polynomial factor unlessP = NP, such as the maximum clique problem.NP-hard problems can often be expressed as integer programs (IP) and solved exactly in exponential time. Manyapproximation algorithms emerge from the linear programming relaxation of the integer program.Not all approximation algorithms are suitable for all practical applications. They often use IP/LP/Semidefinite solvers,complex data structures or sophisticated algorithmic techniques which lead to difficult implementation problems.Also, some approximation algorithms have impractical running times even though they are polynomial time, forexample O(n2156 )[1] . Yet the study of even very expensive algorithms is not a completely theoretical pursuit as theycan yield valuable insights. A classic example is the initial PTAS for Euclidean TSP due to Sanjeev Arora which hadprohibitive running time, yet within a year, Arora refined the ideas into a linear time algorithm. Such algorithms arealso worthwhile in some applications where the running times and cost can be justified e.g. computational biology,financial engineering, transportation planning, and inventory management. In such scenarios, they must compete withthe corresponding direct IP formulations.Another limitation of the approach is that it applies only to optimization problems and not to “pure” decision prob-lems like satisfiability, although it is often possible to conceive optimization versions of such problems, such as themaximum satisfiability problem (Max SAT).Inapproximability has been a fruitful area of research in computational complexity theory since the 1990 result ofFeige, Goldwasser, Lovász, Safra and Szegedy on the inapproximability of Independent Set. After Arora et al. provedthe PCP theorem a year later, it has now been shown that Johnson’s 1974 approximation algorithms for Max SAT,Set Cover, Independent Set and Coloring all achieve the optimal approximation ratio, assuming P != NP.

4.1 Performance guarantees

For some approximation algorithms it is possible to prove certain properties about the approximation of the optimumresult. For example, a ρ-approximation algorithm A is defined to be an algorithm for which it has been proven thatthe value/cost, f(x), of the approximate solution A(x) to an instance x will not be more (or less, depending on the

14

4.2. ALGORITHM DESIGN TECHNIQUES 15

situation) than a factor ρ times the value, OPT, of an optimum solution.

OPT ≤ f(x) ≤ ρOPT, if ρ > 1;

ρOPT ≤ f(x) ≤ OPT, if ρ < 1.

The factor ρ is called the relative performance guarantee. An approximation algorithm has an absolute performanceguarantee or bounded error c, if it has been proven for every instance x that

(OPT − c) ≤ f(x) ≤ (OPT + c).

Similarly, the performance guarantee, R(x,y), of a solution y to an instance x is defined as

R(x, y) = max(OPT

f(y),f(y)

OPT

),

where f(y) is the value/cost of the solution y for the instance x. Clearly, the performance guarantee is greater than orequal to 1 and equal to 1 if and only if y is an optimal solution. If an algorithm A guarantees to return solutions with aperformance guarantee of at most r(n), thenA is said to be an r(n)-approximation algorithm and has an approximationratio of r(n). Likewise, a problem with an r(n)-approximation algorithm is said to be r(n)-approximable or have anapproximation ratio of r(n).[2][3]

One may note that for minimization problems, the two different guarantees provide the same result and that formaximization problems, a relative performance guarantee of ρ is equivalent to a performance guarantee of r = ρ−1 .In the literature, both definitions are common but it is clear which definition is used since, for maximization problems,as ρ ≤ 1 while r ≥ 1.The absolute performance guarantee PA of some approximation algorithm A, where x refers to an instance of aproblem, and where RA(x) is the performance guarantee of A on x (i.e. ρ for problem instance x) is:

PA = infr ≥ 1 | RA(x) ≤ r,∀x.

That is to say that PA is the largest bound on the approximation ratio, r, that one sees over all possible instances ofthe problem. Likewise, the asymptotic performance ratio R∞

A is:

R∞A = infr ≥ 1 | ∃n ∈ Z+, RA(x) ≤ r,∀x, |x| ≥ n.

That is to say that it is the same as the absolute performance ratio, with a lower bound n on the size of probleminstances. These two types of ratios are used because there exist algorithms where the difference between these twois significant.

4.2 Algorithm design techniques

By now there are several standard techniques that one tries to design an approximation algorithm. These include thefollowing ones.

1. Greedy algorithm2. Local search3. Enumeration and dynamic programming4. Solving a convex programming relaxation to get a fractional solution. Then converting this fractional solution

into a feasible solution by some appropriate rounding. The popular relaxations include the following.(a) Linear programming relaxation(b) Semidefinite programming relaxation

5. Embedding the problem in some simple metric and then solving the problem on the metric. This is also knownas metric embedding.

16 CHAPTER 4. APPROXIMATION ALGORITHM

4.3 Epsilon terms

In the literature, an approximation ratio for a maximization (minimization) problem of c - ϵ (min: c + ϵ) means thatthe algorithm has an approximation ratio of c ∓ ϵ for arbitrary ϵ > 0 but that the ratio has not (or cannot) be shownfor ϵ = 0. An example of this is the optimal inapproximability — inexistence of approximation — ratio of 7 / 8 +ϵ for satisfiable MAX-3SAT instances due to Johan Håstad.[4] As mentioned previously, when c = 1, the problem issaid to have a polynomial-time approximation scheme.An ϵ-term may appear when an approximation algorithm introduces a multiplicative error and a constant error whilethe minimum optimum of instances of size n goes to infinity as n does. In this case, the approximation ratio is c ∓ k/ OPT = c ∓ o(1) for some constants c and k. Given arbitrary ϵ > 0, one can choose a large enough N such that theterm k / OPT < ϵ for every n ≥ N. For every fixed ϵ, instances of size n < N can be solved by brute force , therebyshowing an approximation ratio — existence of approximation algorithms with a guarantee — of c ∓ ϵ for every ϵ >0.

4.4 See also• Domination analysis considers guarantees in terms of the rank of the computed solution.

• PTAS - a type of approximation algorithm that takes the approximation ratio as a parameter

• APX is the class of problems with some constant-factor approximation algorithm

• Approximation-preserving reduction

4.5 Citations[1] Zych, Anna; Bilò, Davide (2011). “New Reoptimization Techniques applied to Steiner Tree Problem”. Electronic Notes in

Discrete Mathematics 37: 387–392. doi:10.1016/j.endm.2011.05.066. ISSN 1571-0653.

[2] G. Ausiello, P. Crescenzi, G. Gambosi, V. Kann, A. Marchetti-Spaccamela, and M. Protasi (1999). Complexity andApproximation: Combinatorial Optimization Problems and their Approximability Properties.

[3] Viggo Kann (1992). On the Approximability of NP-complete Optimization Problems (PDF).

[4] Johan Håstad (1999). “Some Optimal Inapproximability Results”. Journal of the ACM.

4.6 References• Vazirani, Vijay V. (2003). Approximation Algorithms. Berlin: Springer. ISBN 3-540-65367-8.

• Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms,Second Edition. MIT Press and McGraw-Hill, 2001. ISBN 0-262-03293-7. Chapter 35: ApproximationAlgorithms, pp. 1022–1056.

• Dorit S. Hochbaum, ed. Approximation Algorithms for NP-Hard problems, PWS Publishing Company, 1997.ISBN 0-534-94968-1. Chapter 9: Various Notions of Approximations: Good, Better, Best, and More

• Williamson, David P.; Shmoys, David B. (April 26, 2011), TheDesign of Approximation Algorithms, CambridgeUniversity Press, ISBN 978-0521195270

4.7 External links• Pierluigi Crescenzi, Viggo Kann, Magnús Halldórsson, Marek Karpinski and Gerhard Woeginger, A com-pendium of NP optimization problems.

Chapter 5

Approximation-preserving reduction

In computability theory and computational complexity theory, especially the study of approximation algorithms, anapproximation-preserving reduction is an algorithm for transforming one optimization problem into another prob-lem, such that the distance of solutions from optimal is preserved to some degree. Approximation-preserving reduc-tions are a subset of more general reductions in complexity theory; the difference is that approximation-preservingreductions usually make statements on approximation problems or optimization problems, as opposed to decisionproblems.Intuitively, problem A is reducible to problem B via an approximation-preserving reduction if, given an instance ofproblem A and a (possibly approximate) solver for problem B, one can convert the instance of problem A into aninstance of problem B, apply the solver for problem B, and recover a solution for problem A that also has someguarantee of approximation.

5.1 Definition

Unlike reductions on decision problems, an approximation-preserving reduction must preserve more than the truthof the problem instances when reducing from one problem to another. It must also maintain some guarantee on therelationship between the cost of the solution to the cost of the optimum in both problems. To formalize:Let A and B be optimization problems.Let x be an instance of problem A , with optimal solution OPT (x) . Let cA(x, y) denote the cost of a solution y toan instance x of problem A . This is also the metric used to determine which solution is considered optimal.An approximation-preserving reduction is a pair of functions (f, g) (which often must be computable in polyno-mial time), such that:

• f maps an instance x of A to an instance x′ of B .

• g maps a solution y′ of B to a solution y of A .

• g preserves some guarantee of the solution’s performance, or approximation ratio, defined as RA(x, y) =

max(

cA(x,OPT (x))cA(x,y) , cA(x,y)

cA(x,OPT (x))

).

5.2 Types of approximation-preserving reductions

There are many different types of approximation-preserving reductions, all of which have a different guarantee (thethird point in the definition above). However, unlike with other reductions, approximation-preserving reductionsoften overlap in what properties they demonstrate on optimization problems (e.g. complexity class membership orcompleteness, or inapproximability). The different types of reductions are used instead as varying reduction tech-niques, in that the applicable reduction which is most easily adapted to the problem is used.Not all types of approximation-preserving reductions can be used to show membership in all approximability com-plexity classes, the most notable of which are PTAS and APX. A reduction below preserves membership in a

17

18 CHAPTER 5. APPROXIMATION-PRESERVING REDUCTION

complexity class C if, given a problem A that reduces to problem B via the reduction scheme, and B is in C, thenA is in C as well. Some reductions shown below only preserve membership in APX or PTAS, but not the other.Because of this, careful choice must be made when selecting an approximation-preserving reductions, especially forthe purpose of proving completeness of a problem within a complexity class.Crescenzi suggests that the three most ideal styles of reduction, for both ease of use and proving power, are PTASreduction, AP reduction, and L-reduction. [1] The reduction descriptions that follow are from Crescenzi’s survey ofapproximation-preserving reductions.

5.2.1 Strict reduction

Strict reduction is the simplest type of approximation-preserving reduction. In a strict reduction, the approximationratio of a solution y' to an instance x' of a problem B must be at most as good as the approximation ratio of thecorresponding solution y to instance x of problem A. In other words:

RA(x, y) ≤ RB(x′, y′) for x′ = f(x), y = g(y′) .

Strict reduction is the most straightforward: if a strict reduction from problem A to problem B exists, then problemA can always be approximated to at least as good a ratio as problem B. Strict reduction preserves membership in bothPTAS and APX.There exists a similar concept of a S-reduction, for which cA(x, y) = cB(x

′, y′) , and the optima of the twocorresponding instances must have the same cost as well. S-reduction is a very special case of strict reduction, and iseven more constraining. In effect, the two problems A and B must be in near perfect correspondence with each other.The existence of an S-reduction implies not only the existence of a strict reduction but every other reduction listedhere.

5.2.2 L-reduction

Main article: L-reduction

L-reductions preserve membership in PTAS as well as APX (but only for minimization problems in the case of thelatter). As a result, they cannot be used in general to prove completeness results about APX, Log-APX, or Poly-APX,but nevertheless they are valued for their natural formulation and ease of use in proofs.[1]

5.2.3 PTAS-reduction

Main article: PTAS reduction

PTAS-reduction is another commonly used reduction scheme. Though it preserves membership in PTAS, it does notdo so for APX. Nevertheless, APX-completeness is defined in terms of PTAS reductions.PTAS-reductions are a generalization of P-reductions, shown below, with the only difference being that the functiong is allowed to depend on the approximation ratio r .

5.2.4 A-reduction and P-reduction

A-reduction and P-reduction are similar reduction schemes that can be used to show membership in APX and PTASrespectively. Both introduce a new function c , defined on numbers greater than 1, which must be computable.In an A-reduction, we have that

RB(x′, y′) ≤ r → RA(x, y) ≤ c(r)

In a P-reduction, we have that

5.3. SEE ALSO 19

RB(x′, y′) ≤ c(r) → RA(x, y) ≤ r

The existence of a P-reduction implies the existence of a PTAS-reduction.

5.2.5 E-reduction

E-reduction, which is a generalization of strict reduction but implies both A-reduction and P-reduction, is an exampleof a less restrictive reduction style that preserves membership not only in PTAS and APX, but also the larger classesLog-APX and Poly-APX. E-reduction introduces two new parameters, a polynomial p and a constant β . Its definitionis as follows.In an E-reduction, we have that for some polynomial p and constant β ,

• cB(OPTB(x′)) ≤ p(|x|)cA(OPTA(x)) , where |x| denotes the size of the problem instance’s description.

• For any solution y′ to B , we have RA(x, y) ≤ 1 + β · (RB(x′, y′)− 1) .

To obtain an A-reduction from an E-reduction, let c(r) = 1 + β · (r − 1) , and to obtain a P-reduction from anE-reduction, let c(r) = 1 + (r − 1)/β .

5.2.6 AP-reduction

AP-reductions are used to define completeness in the classes Log-APX and Poly-APX. They are a special case ofPTAS reduction, meeting the following restrictions.In an AP-reduction, we have that for some constant α ,

RB(x′, y′) ≤ r → RA(x, y) ≤ 1 + α · (r − 1)

with the additional generalization that the function g is allowed to depend on the approximation ratio r , as in PTAS-reduction.AP-reduction is also a generalization of E-reduction. An additional restriction actually needs to be imposed forAP-reduction to preserve Log-APX and Poly-APX membership, as E-reduction does: for fixed problem size, thecomputation time of f, g must be non-increasing as the approximation ratio increases.

5.2.7 Gap reduction

Main article: Gap reduction

A gap reduction is a type of reduction that, while useful in proving some inapproximability results, does not resemblethe other reductions shown here. Gap reductions deal with optimization problems within a decision problem con-tainer, generated by changing the problem goal to distinguishing between the optimal solution and solutions somemultiplicative factor worse than the optimum.

5.3 See also• Reduction (complexity)

• PTAS reduction

• L-reduction

• Approximation algorithm

20 CHAPTER 5. APPROXIMATION-PRESERVING REDUCTION

5.4 References[1] Crescenzi, Pierluigi (1997). “A Short Guide To Approximation Preserving Reductions”. Proceedings of the 12th Annual

IEEE Conference on Computational Complexity (Washington, D.C.: IEEE Computer Society): 262–.

Chapter 6

Asymptotic computational complexity

In computational complexity theory, asymptotic computational complexity is the usage of asymptotic analysis forthe estimation of computational complexity of algorithms and computational problems, commonly associated withthe usage of the big O notation.

6.1 Scope

With respect to computational resources, asymptotic time complexity and asymptotic space complexity are com-monly estimated. Other asymptotically estimated behavior include circuit complexity and various measures of parallelcomputation, such as the number of (parallel) processors.Since the ground-breaking 1965 paper by Juris Hartmanis and Richard E. Stearns[1] and the 1979 book by MichaelGarey and David S. Johnson on NP-completeness,[2] the term "computational complexity" (of algorithms) has becomecommonly referred to as asymptotic computational complexity.Further, unless specified otherwise, the term “computational complexity” usually refers to the upper bound for theasymptotic computational complexity of an algorithm or a problem, which is usually written in terms of the big O no-tation, e.g.. O(n3).Other types of (asymptotic) computational complexity estimates are lower bounds ("Big Omega"notation; e.g., Ω(n)) and asymptotically tight estimates, when the asymptotic upper and lower bounds coincide (writ-ten using the "big Theta"; e.g., Θ(n log n)).A further tacit assumption is that the worst case analysis of computational complexity is in question unless statedotherwise. An alternative approach is probabilistic analysis of algorithms.

6.2 Types of algorithms considered

In most practical cases deterministic algorithms or randomized algorithms are discussed, although theoretical com-puter science also considers nondeterministic algorithms and other advanced models of computation.

6.3 See also• Asymptotically optimal algorithm

6.4 References[1] Hartmanis, J.; Stearns, R. E. (1965). “On the computational complexity of algorithms”. Transactions of the American

Mathematical Society 117: 285–306. doi:10.1090/S0002-9947-1965-0170805-7.

[2] Michael Garey, and David S. Johnson: Computers and Intractability: A Guide to the Theory of NP-Completeness. NewYork: W. H. Freeman & Co., 1979.

21

Chapter 7

Averaging argument

In computational complexity theory and cryptography, averaging argument is a standard argument for provingtheorems. It usually allows us to convert probabilistic polynomial-time algorithms into non-uniform polynomial-sizecircuits.

7.1 Example

To simplify, let’s first consider an example.Example: If every person likes at least 1/3 of the books in a library, then, there exists a book, which at least 1/3 ofpeople liked it.Proof: Suppose there are N people and B books. Each person likes at least B/3 of the books. Let people leave amark on the book they like. Then, there will be at least M = (NB)/3 marks. The averaging argument claims thatthere exists a book with at leastN/3 marks on it. Assume, to the contradiction, that no such book exists. Then, everybook has fewer than N/3 marks. However, since there are B books, the total number of marks will be fewer than(NB)/3 , contradicting the fact that there are at least M marks.

7.2 Formalized definition of averaging argument

Consider two sets: X and Y, a proposition p : X × Y → TRUE/FALSE , and a fraction f (where 0 ≤ f ≤ 1 ).If for all x ∈ X and at least a fraction f of y ∈ Y , the proposition p(x, y) holds, then there exists a y ∈ Y , forwhich there exists a fraction f of x ∈ X that the proposition p(x, y) holds.Another formal (and more complicated) definition is due to Barak:[1]

Let f be some function. The averaging argument is the following claim: if we have a circuit C such that C(x, y) =f(x) with probability at least ρ , where x is chosen at random and y is chosen independently from some distributionY over 0, 1m (which might not even be efficiently sampleable) then there exists a single string y0 ∈ 0, 1m suchthat Prx[C(x, y0) = f(x)] ≥ ρ .Indeed, for every y define py to be Prx[C(x, y) = f(x)] then

Prx,y

[C(x, y) = f(x)] = Ey[py]

and then this reduces to the claim that for every random variable Z , if E[Z] ≥ ρ then Pr[Z ≥ ρ] > 0 (this holdssince E[Z] is the weighted average of Z and clearly if the average of some values is at least ρ then one of the valuesmust be at least ρ ).

22

7.3. APPLICATION 23

7.3 Application

This argument has wide use in complexity theory (e.g. proving BPP ⊊ P/poly ) and cryptography (e.g. proving thatindistinguishable encryption results in semantic security). A plethora of such applications can be found in Goldreich'sbooks.[2][3][4]

7.4 References[1] Boaz Barak, “Note on the averaging and hybrid arguments and prediction vs. distinguishing.”, COS522, Princeton Uni-

versity, March 2006.

[2] Oded Goldreich, Foundations of Cryptography, Volume 1: Basic Tools, Cambridge University Press, 2001, ISBN 0-521-79172-3

[3] Oded Goldreich, Foundations of Cryptography, Volume 2: Basic Applications, Cambridge University Press, 2004, ISBN0-521-83084-2

[4] Oded Goldreich, Computational Complexity: A Conceptual Perspective, Cambridge University Press, 2008, ISBN 0-521-88473-X

Chapter 8

Best, worst and average case

In computer science, best, worst, and average cases of a given algorithm express what the resource usage is at least,at most and on average, respectively. Usually the resource being considered is running time, i.e. time complexity,but it could also be memory or other resources.In real-time computing, the worst-case execution time is often of particular concern since it is important to knowhow much time might be needed in the worst case to guarantee that the algorithm will always finish on time.Average performance and worst-case performance are the most used in algorithm analysis. Less widely found isbest-case performance, but it does have uses: for example, where the best cases of individual tasks are known, theycan be used to improve the accuracy of an overall worst-case analysis. Computer scientists use probabilistic analysistechniques, especially expected value, to determine expected running times.The terms are used in other contexts; for example the worst- and best-case outcome of a planned-for epidemic, worst-case temperature to which an electronic circuit element is exposed, etc. Where components of specified toleranceare used, devices must be designed to work properly with the worst-case combination of tolerances and externalconditions.

8.1 Best-case performance for algorithm

The term best-case performance is used in computer science to describe an algorithm’s behavior under optimal con-ditions. For example, the best case for a simple linear search on a list occurs when the desired element is the firstelement of the list.Development and choice of algorithms is rarely based on best-case performance: most academic and commercialenterprises are more interested in improving Average-case complexity and worst-case performance. Algorithms mayalso be trivially modified to have good best-case running time by hard-coding solutions to a finite set of inputs, makingthe measure almost meaningless.[1]

8.2 Worst-case versus average-case performance

Worst-case performance analysis and average case performance analysis have some similarities, but in practice usuallyrequire different tools and approaches.Determining what average input means is difficult, and often that average input has properties which make it difficultto characterise mathematically (consider, for instance, algorithms that are designed to operate on strings of text).Similarly, even when a sensible description of a particular “average case” (which will probably only be applicable forsome uses of the algorithm) is possible, they tend to result in more difficult analysis of equations.Worst-case analysis has similar problems: it is typically impossible to determine the exact worst-case scenario. In-stead, a scenario is considered such that it is at least as bad as the worst case. For example, when analysing analgorithm, it may be possible to find the longest possible path through the algorithm (by considering the maximumnumber of loops, for instance) even if it is not possible to determine the exact input that would generate this path

24

8.3. PRACTICAL CONSEQUENCES 25

(indeed, such an input may not exist). This gives a safe analysis (the worst case is never underestimated), but onewhich is pessimistic, since there may be no input that would require this path.Alternatively, a scenario which is thought to be close to (but not necessarily worse than) the real worst case may beconsidered. This may lead to an optimistic result, meaning that the analysis may actually underestimate the true worstcase.In some situations it may be necessary to use a pessimistic analysis in order to guarantee safety. Often however, apessimistic analysis may be too pessimistic, so an analysis that gets closer to the real value but may be optimistic(perhaps with some known low probability of failure) can be a much more practical approach.When analyzing algorithms which often take a small time to complete, but periodically require a much larger time,amortized analysis can be used to determine the worst-case running time over a (possibly infinite) series of operations.This amortized worst-case cost can be much closer to the average case cost, while still providing a guaranteed upperlimit on the running time.The worst-case analysis is related to the worst-case complexity.[2]

8.3 Practical consequences

Many problems with bad worst-case performance have good average-case performance. For problems we want tosolve, this is a good thing: we can hope that the particular instances we care about are average. For cryptography,this is very bad: we want typical instances of a cryptographic problem to be hard. Here methods like random self-reducibility can be used for some specific problems to show that the worst case is no harder than the average case,or, equivalently, that the average case is no easier than the worst case.On the other hand some algorithms like hash tables have very poor worst case behaviours, but a well written hashtable of sufficient size will statistically never give the worst case; the average number of operations performed followsan exponential decay curve, and so the run time of an operation is statistically bounded.

8.4 Examples

8.4.1 Sorting algorithms

• Insertion sort applied to a list of n elements, assumed to be all different and initially in random order. Onaverage, half the elements in a list A1 ... Aj are less than elementAj₊₁, and half are greater. Therefore thealgorithm compares the j+1-st element to be inserted on the average with half the already sorted sub-list, so tj= j/2. Working out the resulting average-case running time yields a quadratic function of the input size, justlike the worst-case running time.

• Quicksort applied to a list of n elements, again assumed to be all different and initially in random order. Thispopular sorting algorithm has an average-case performance of O(n log(n)), which contributes to making it avery fast algorithm in practice. But given a worst-case input, its performance degrades to O(n2). Also, whennot implemented with the “shortest first” policy, the worst-case space complexity degrades to O(log(n)).

8.4.2 Data structures

• Linear search on a list of n elements. In the worst case, the search must visit every element once. This happenswhen the value being searched for is either the last element in the list, or is not in the list. However, on average,assuming the value searched for is in the list and each list element is equally likely to be the value searched for,the search visits only n/2 elements.

26 CHAPTER 8. BEST, WORST AND AVERAGE CASE

8.4.3 Graph search

8.5 See also• Sorting algorithm – an area where there is a great deal of performance analysis of various algorithms.

• Worst-case circuit analysis

• Smoothed analysis

• Interval finite element

• Big O notation

8.6 References[1] Introduction to Algorithms (Cormen, Leiserson, Rivest, and Stein) 2001, Chapter 2 “Getting Started”.

[2] Worst-case complexity

• http://bigocheatsheet.com/

• http://www.algorithmiccomplexity.com/

Chapter 9

Boolean circuit

In computational complexity theory and circuit complexity, a Boolean circuit is a mathematical model for digitallogic circuits. A formal language can be decided by a family of Boolean circuits, one circuit for each possible inputlength. Boolean circuits are also used as a formal model for combinational logic in digital electronics.Boolean circuits are defined in terms of the logic gates they contain. For example, a circuit might contain binaryAND and OR gates and unary NOT gates, or be entirely described by binary NAND gates. Each gate correspondsto some Boolean function that takes a fixed number of bits as input and outputs a single bit.Boolean circuits provide a model for many digital components used in computer engineering, including multiplexers,adders, and arithmetic logic units.

9.1 Formal definition

In giving a formal definition of Boolean circuits, Vollmer starts by defining a basis set B of Boolean functions, corre-sponding to the gates allowable in the circuit model. A Boolean circuit over a basis B, with n inputs and m outputs, isthen defined as a finite directed acyclic graph. Each vertex corresponds to either a basis function or one of the inputs,and there are a set of exactly m nodes which are labeled as the outputs.[1] The edges must also have some ordering,to distinguish between different arguments to the same Boolean function.[2]

As a special case, a propositional formula or Boolean expression is a Boolean circuit with a single output node inwhich every other node has fan-out of 1. Thus, a Boolean circuit can be regarded as a generalization that allowsshared subformulas and multiple outputs.A common basis for Boolean circuits is the set AND, OR, NOT, from which all other Boolean functions can beconstructed.

9.2 Computational complexity

9.2.1 Evaluation of a circuit

The Circuit Value Problem, the problem of computing the output of a given Boolean circuit on a given input string,is a P-complete decision problem.[3] Therefore, this problem is considered to be “inherently sequential” in the sensethat there is likely no efficient, highly parallel algorithm that solves the problem.

9.2.2 Complexity measures

See also: Circuit complexity

Several important complexity measures can be defined on Boolean circuits, including circuit depth, circuit size, andnumber of alternations between AND gates and OR gates. For example, the size complexity of a Boolean circuit is

27

28 CHAPTER 9. BOOLEAN CIRCUIT

the number of gates.

9.2.3 Complexity classes

Main article: Circuit complexity § Complexity classes

Several important complexity classes are defined in terms of Boolean circuits, including NC. NC is defined to bethe set of Boolean functions that can be decided by uniform Boolean circuits of polynomial size and polylogarithmicdepth. Here, the word uniform means that there must be some condition on the circuit family so that a description ofa circuit can be computed from only the number of inputs to the circuit.

9.3 See also• Circuit satisfiability

• Logic gate

• Boolean logic

9.4 Footnotes[1] Vollmer 1999, p. 8.

[2] Vollmer 1999, p. 9.

[3] S. Arora and B. Barak. Computational complexity, a modern approach. p. 119.

9.5 References• Vollmer, Heribert (1999). Introduction to Circuit Complexity. Berlin: Springer. ISBN 3-540-64310-9.

Chapter 10

Certificate (complexity)

In computational complexity theory, a certificate (also called a witness) is a string that certifies the answer to acomputation, or certifies the membership of some string in a language. A certificate is often thought of as a solutionpath within a verification process, which is used to check whether a problem gives the answer “Yes” or “No”.In the decision tree model of computation, certificate complexity is the minimum number of the n input variables ofa decision tree that need to be assigned a value in order to definitely establish the value of the Boolean function f .

10.1 Definition

Certificate is generally used to prove semi-decidability as following:[1]

L ∈ SD iff there is a two-place predicate R ⊆ Σ∗ × Σ∗ such that R is computable, and such that for all x ∈ Σ∗:x ∈ L ⇔ there exists y such that R(x, y)and to prove NP as following:L ∈ NP iff there is a polytime verifier V such that:x ∈ L ⇔ there exists y such that |y| <= |x|c and V accepts (x, y)

10.2 Example

L = <<M>, x, w> | does <M> accept x in |w| steps? Show L ∈ NP. verifier: gets string c = <M>, x, w such that|c| <= P(|w|) check if c is an accepting computation of M on x with at most |w| steps |c| <= O(|w|3) if we have acomputation of a TM with k steps the total size of the computation string is k2 Thus, <<M>, x, w> ∈ L ⇔ thereexists c <= a|w|3 such that <<M>, x, w, c> ∈ V ∈ P

10.3 See also• Witness (mathematics), an analogous concept in mathematical logic

10.4 References[1] Cook, Stephen. “Computability and Noncomputability” (PDF). Retrieved 7 February 2013.

• Buhrman, Harry; Wolf, Ronald (2002), Complexity Measures and Decision Tree Complexity:A Survey.

• Computational Complexity: a Modern Approach by Sanjeev Arora and Boaz Barak

29

Chapter 11

Circuit complexity

In theoretical computer science, circuit complexity is a branch of computational complexity theory in which Booleanfunctions are classified according to the size or depth of Boolean circuits that compute them. One speaks of the circuitcomplexity of a Boolean circuit. A related notion is the circuit complexity of a recursive language that is decided bya family of circuits C1, C2, . . . (see below).A Boolean circuit with n input bits is a directed acyclic graph in which every node (usually called gates in this context)is either an input node of in-degree 0 labeled by one of the n input bits, an AND gate, an OR gate, or a NOT gate.One of these gates is designated as the output gate. Such a circuit naturally computes a function of its n inputs. Thesize of a circuit is the number of gates it contains and its depth is the maximal length of a path from an input gate tothe output gate.There are two major notions of circuit complexity (these are outlined in Sipser (1997)[1]:324). The circuit-size com-plexity of a Boolean function f is the minimal size of any circuit computing f . The circuit-depth complexity of aBoolean function f is the minimal depth of any circuit computing f .These notions generalize when one considers the circuit complexity of a recursive language: A formal language maycontain strings with many different bit lengths. Boolean circuits, however, only allow a fixed number of input bits.Thus no single Boolean circuit is capable of deciding such a language. To account for this possibility, one considersfamilies of circuits C1, C2, . . . where each Cn accepts inputs of size n . Each circuit family will naturally generate arecursive language by outputting 1 when a string is a member of the family, and 0 otherwise. We say that a familyof circuits is size minimal if there is no other family that decides on inputs of any size, n , with a circuit of smallersize than Cn (respectively for depth minimal families).Hence, the circuit-size complexity of a recursive language A is defined as the function t : N → N , that relates abit length of an input, n , to the circuit-size complexity of a minimal circuit Cn that decides whether inputs of thatlength are in A . The circuit-depth complexity is defined similarly.Complexity classes defined in terms of Boolean circuits include AC0, AC, TC0 and NC.

11.1 Uniformity

Boolean circuits are one of the prime examples of so-called non-uniform models of computation in the sense thatinputs of different lengths are processed by different circuits, in contrast with uniform models such as Turing machineswhere the same computational device is used for all possible input lengths. An individual computational problem isthus associated with a particular family of Boolean circuits C1, C2, . . . where each Cn is the circuit handling inputsof n bits. A uniformity condition is often imposed on these families, requiring the existence of some resource-bounded Turing machine which, on input n, produces a description of the individual circuit Cn . When this Turingmachine has a running time polynomial in n, the circuit family is said to be P-uniform. The stricter requirement ofDLOGTIME-uniformity is of particular interest in the study of shallow-depth circuit-classes such as AC0 or TC0.

30

11.2. HISTORY 31

11.1.1 Polynomial-time uniform

A family of Boolean circuits Cn : n ∈ N is polynomial-time uniform if there exists a deterministic Turing machineM, such that

• M runs in polynomial time

• For all n ∈ N , M outputs a description of Cn on input 1n

11.1.2 Logspace uniform

A family of Boolean circuits Cn : n ∈ N is logspace uniform if there exists a deterministic Turing machine M,such that

• M runs in logarithmic space

• For all n ∈ N , M outputs a description of Cn on input 1n

11.2 History

Circuit complexity goes back to Shannon (1949), who proved that almost all Boolean functions on n variables requirecircuits of size Θ(2n/n). Despite this fact, complexity theorists have not been able to prove superpolynomial circuitlower bounds for specific Boolean functions.On the other hand, superpolynomial lower bounds have been proved under certain restrictions on the family of circuitsused. The first function for which superpolynomial circuit lower bounds were shown was the parity function, whichcomputes the sum of its input bits modulo 2. The fact that parity is not contained in AC0 was first establishedindependently by Ajtai (1983)[2] and by Furst, Saxe and Sipser (1984).[3] Later improvements by Håstad (1987)in fact establish that any family of constant-depth circuits computing the parity function requires exponential size.Smolensky (1987) proved that this is true even if the circuit is augmented with gates computing the sum of its inputbits modulo some odd prime p.The k-clique problem is to decide whether a given graph on n vertices has a clique of size k. For any particular choiceof the constants n and k, the graph can be encoded in binary using

(n2

)bits which indicate for each possible edge

whether it is present. Then the k-clique problem is formalized as a function fk : 0, 1(n2) → 0, 1 such that fk

outputs 1 if and only if the graph encoded by the string contains a clique of size k. This family of functions is monotoneand can be computed by a family of circuits, but it has been shown that it cannot be computed by a polynomial-sizefamily of monotone circuits (that is, circuits with AND and OR gates but without negation). The original result ofRazborov (1985) was later improved to an exponential-size lower bound by Alon and Boppana (1987). Rossman(2008) shows that constant-depth circuits with AND, OR, and NOT gates require size Ω(nk/4) to solve the k-cliqueproblem even in the average case. Moreover, there is a circuit of size nk/4+O(1) which computes fk .Raz and McKenzie later showed that the monotone NC hierarchy is infinite (1999).The Integer Division Problem lies in uniform TC0 (Hesse 2001).

11.3 Circuit lower bounds

Circuit lower bounds are generally difficult. Known results include

• Parity is not in nonuniform AC0, proved by Ajtai (1983) and by Furst, Saxe and Sipser.

• Uniform TC0 is not contained in PP, proved by Allender.

• The classes SP2, PP[4] and MA/1[5] (MA with one bit of advice) are not in SIZE(nk) for any constant k.

32 CHAPTER 11. CIRCUIT COMPLEXITY

• While it is suspected that the nonuniform class ACC0 does not contain the majority function, it was only in2010 that Williams proved that NEXP ⊆ ACC0 .[6]

It is open whether NEXPTIME has nonuniform TC0 circuits.Proofs of circuit lower bounds are strongly connected to derandomization. A proof that P = BPP would imply thateither NEXP ⊆ P/poly or that permanent cannot be computed by nonuniform arithmetic circuits (polynomials) ofpolynomial size and polynomial degree.[7]

11.4 Complexity classes

Many circuit complexity classes are defined in terms of class hierarchies. For each nonnegative integer i, there is aclass NCi, consisting of polynomial-size circuits of depth O(logi(n)) , using bounded fan-in AND, OR, and NOTgates. We can talk about the union NC of all of these classes. By considering unbounded fan-in gates, we constructthe classes ACi and AC. We construct many other circuit complexity classes with the same size and depth restrictionsby allowing different sets of gates.

11.5 Relation to time complexity[1]

Say that a certain language, A , belongs to the time-complexity class TIME(t(n)) for some function t : N → N .Then A has circuit complexity O(t2(n))

11.6 References[1] Sipser, M. (1997). 'Introduction to the theory of computation.' Boston: PWS Pub. Co.

[2] Ajtai, Miklós; Komlós, János; Szemerédi, Endre (1983). “An 0(n log n) sorting network”. STOC '83 Proceedings of thefifteenth annual ACM symposium on Theory of computing: 1–9. ISBN 0-89791-099-0.

[3] Furst, Merrick; Saxe, James B.; Sipser, Michael (1984). “Parity, circuits, and the polynomial-time hierarchy”. Math. Syst.Theory 17: 13–27. doi:10.1007/bf01744431. ISSN 0025-5661. Zbl 0534.94008.

[4] See proof

[5] Santhanam, Rahul (2007). “Circuit lower bounds for Merlin-Arthur classes”. STOC 2007: Proceedings of the thirty-ninthannual ACM symposium on Theory of computing. pp. 275–283. doi:10.1145/1250790.1250832.

[6] Williams, Ryan (2011). “Non-Uniform ACC Circuit Lower Bounds” (PDF). CCC 2011: Proceedings of the 26th AnnualIEEE Conference on Computational Complexity. pp. 115–125. doi:10.1109/CCC.2011.36.

[7] Kabanets, V.; Impagliazzo, R. (2004). “Derandomizing polynomial identity tests means proving circuit lower bounds”.Computational Complexity 13 (1): 1–46. doi:10.1007/s00037-004-0182-6.

• Ajtai, Miklós (1983). " Σ11 -formulae on finite structures”. Annals of Pure and Applied Logic 24: 1–24.

doi:10.1016/0168-0072(83)90038-6.

• Alon, Noga; Boppana, Ravi B. (1987). “The monotone circuit complexity of Boolean functions”. Combina-torica 7 (1): 1–22. doi:10.1007/bf02579196.

• Furst, Merrick L.; Saxe, James B.; Sipser, Michael (1984). “Parity, circuits, and the polynomial-time hierar-chy”. Mathematical Systems Theory 17 (1): 13–27. doi:10.1007/bf01744431.

• Håstad, Johan (1987), Computational limitations of small depth circuits (PDF), Ph.D. thesis, MassachusettsInstitute of Technology.

• Hesse, William (2001). “Division is in uniform TC0". Proc. 28th International Colloquium on Automata,Languages and Programming. Springer. pp. 104–114.

11.6. REFERENCES 33

• Raz, Ran; McKenzie, Pierre (1999). “Separation of the monotone NC hierarchy”. Combinatorica 19 (3):403–435. doi:10.1007/s004930050062.

• Razborov, Alexander A. (1985). “Lower bounds on the monotone complexity of some Boolean functions”.Mathematics of the USSR, Doklady 31: 354–357.

• Rossman, Benjamin (2008). “On the constant-depth complexity of k-clique”. STOC 2008: Proceedings of the40th annual ACM symposium on Theory of computing. ACM. pp. 721–730. doi:10.1145/1374376.1374480.

• Shannon, Claude E. (1949). “The synthesis of two-terminal switching circuits”. Bell System Technical Journal28 (1): 59–98. doi:10.1002/j.1538-7305.1949.tb03624.x.

• Smolensky, Roman (1987). “Algebraic methods in the theory of lower bounds for Boolean circuit complexity”.Proc. 19th Annual ACM Symposium on Theory of Computing. ACM. pp. 77–82. doi:10.1145/28395.28404.

• Vollmer, Heribert (1999). Introduction to Circuit Complexity: a Uniform Approach. Springer Verlag. ISBN3-540-64310-9.

• Wegener, Ingo (1987). The Complexity of Boolean Functions. John Wiley and Sons Ltd, and B. G. Teubner,Stuttgart. ISBN 3-519-02107-2. At the time an influential textbook on the subject, commonly known as the“Blue Book”. Also available for download (PDF) at the Electronic Colloquium on Computational Complexity.

• Lecture notes for a course of Uri Zwick on circuit complexity

• Circuit Complexity before the Dawn of the New Millennium, a 1997 survey of the field by Eric Allender slides.

Chapter 12

Circuits over sets of natural numbers

Circuits over natural numbers are a mathematical model used in studying computational complexity theory. Theyare a special case of circuits. The object is a labeled directed acyclic graph the nodes of which evaluate to sets ofnatural numbers, the leaves are finite sets, and the gates are set operations or arithmetic operations.As an algorithmic problem, the problem is to find if a given natural number is an element of the output node or iftwo circuits compute the same set. Decidability is still an open question.

12.1 Formal definition

A natural number circuit is a circuit, i.e. a labelled directed acyclic graph of in-degree at most 2. The nodes ofin-degree 0, the leaves, are finite sets of natural numbers, the labels of the nodes of in-degree 1 are −, where A =x ∈ N|x ∈ A and the labels of the nodes of in-degree 2 are +, ×, ∪ and ∩, where A+B = a+ b|a ∈ A, b ∈ B, A×B = a× b|a ∈ A, b ∈ B and ∪ and ∩ with the usual set meaning.The subset of circuits which do not use all of the possible labels are also studied.

12.1.1 Algorithmic problems

One can ask:

• Is a given number n a member of the output node.

• Is the output node empty, does it contain a specific element, is it equal to N ?

• Is one node is a subset of another.

For circuits which use all the labels, all these problems are equivalent.

Proof

The first problem is reducible to the second one, by taking the intersection of the output gate and n. Indeed the newoutput get will be empty if and only if n was not an element of the former output gate.The first problem is reducible to the third one, by asking if the node n is a subset of the output node.The second problem is reducible to the first one, it suffices to multiply the output gate by 0, then 0 will be in theoutput gate if and only if the former output gate were not empty.The third problem is reducible to the second one, checking if A is a subset of B is equivalent to ask if there is anelement in A ∩B .

34

12.2. COMPLEXITY RESULTS 35

12.1.2 Restrictions

Let O be a subset of ∪,∩,−,+,×, then we call MC(O) the problem of finding if a natural number is inside the outputgate of a circuit the gates’ labels of which are in O, and MF(O) the same problem with the added constraint that thecircuit must be a tree.

12.1.3 Quickly growing set

One difficulty comes from the fact that the complement of a finite set is infinite, and a computer has got only a finitememory. But even without complementation, one can create double exponential numbers. Let E0 = 2, Ei+1 =

Ei × Ei , then one can easily prove by induction on i that Ei = 22i , indeed E0 = 2 = 21 = 220 and byinduction Ei+1 = Ei × Ei = 22i × 22i = (22i)2 = 22i×2 = 22i+1 .And even double exponential—sized sets: let S0 = 0, 1, 2, Si+1 = (Si×Si)+Si , then x|0 < x < 22

i ⊂ Si ,i.e. Si contains the 22i firsts number. Once again this can be proved by induction on i , it is true for S0 by definitionand let x ∈ x|0 < x < 22

i+1 , dividing x by 22i we see that it can be written as x = 22

i × d+ r where d, r < 22i

, and by induction, 22i , d and r are in Si , so indeed x ∈ (Si × Si) + Si .These examples explains why addition and multiplication are enough to create problems of high complexity.

12.2 Complexity results

12.2.1 Membership problem

The membership problem asks if, given an element n and a circuit, n is in the output gate of the circuit.When the class of authorized gates is restricted, the membership problem lies inside well known complexity classes.

12.2.2 Equivalence problem

The equivalence problem asks if, given two gates of a circuit, they evaluate to the same set.When the class of authorized gates is restricted, the equivalence problem lies inside well known complexity classes.[1]

We call EC(O) and EF(O) the problem of equivalence over circuits and formulae the gates of which are in O.

12.3 References[1] Christian Glaßer, Katrin Herr, Christian Reitwießner, Stephen Travers and Matthias Waldherr (2007), “Equivalence Prob-

lems for Circuits over Sets of Natural Numbers”, Lecture Notes in Computer Science ((what is called “number” in bibtex)ed.) (Berlin / Heidelberg: Springer), Volume 4649/2007: 127–138, doi:10.1007/978-3-540-74510-5, ISBN 978-3-540-74509-9

• Travers, Stephen (2006), The Complexity of Membership Problems for Circuits over Sets of Natural Numbers389 (1), Theoretical Computer Science, pp. 211–229, ISSN 0304-3975

• Pierre McKenzie and Klaus W. Wagner (2003), “The Complexity of Membership Problems for Circuits overSets of Natural Numbers”, Lecture Notes In Computer Science (Springer-Verlag) 2607: 571–582, ISBN 3-540-00623-0

12.4 External links• Pierre McKenzie, The complexity of circuit evaluation over the natural numbers

Chapter 13

Cobham’s thesis

The graph shows time of solution of problem in milliseconds (msec) vs. problem size, n, for knapsack problems solved by a state-of-the-art specialized algorithm using a 933 MHz Pentium III computer (average of 100 instances, data from:[1]). The fit of thequadratic equation suggests that empirical algorithmic complexity for instances with 50–10,000 variables is O((log n)2).

Cobham’s thesis, also known as Cobham–Edmonds thesis (named after Alan Cobham and Jack Edmonds),[2][3][4]

asserts that computational problems can be feasibly computed on some computational device only if they can becomputed in polynomial time; that is, if they lie in the complexity class P.[5]

Formally, to say that a problem can be solved in polynomial time is to say that there exists an algorithm that, givenan n-bit instance of the problem as input, can produce a solution in time O(nc), where c is a constant that depends onthe problem but not the particular instance of the problem.Alan Cobham’s 1965 paper entitled “The intrinsic computational difficulty of functions”[6] is one of the earliestmentions of the concept of the complexity class P, consisting of problems decidable in polynomial time. Cobhamtheorized that this complexity class was a good way to describe the set of feasibly computable problems. Any problemthat cannot be contained in P is not feasible, but if a real-world problem can be solved by an algorithm existing in P,generally such an algorithm will eventually be discovered.The class P is a useful object of study because it is not sensitive to the details of the model of computation: forexample, a change from a single-tape Turing machine to a multi-tape machine can lead to a quadratic speedup, butany algorithm that runs in polynomial time under one model also does so on the other.

36

13.1. REASONING 37

In similar spirit, NC complexity class can be thought to capture problems “effectively solvable” on a parallel computer.

13.1 Reasoning

The thesis is widely considered to be a good rule of thumb for real-life problems. Typical input lengths that users andprogrammers are interested in are approximately between 100 and 1,000,000. Consider an input length of n=100and a polynomial algorithm whose running time is n2. This is a typical running time for a polynomial algorithm.(See the “Objections” section for a discussion of atypical running times.) The number of steps that it will require, forn=100, is 1002=10000. A typical CPU will be able to do approximately 109 operations per second (this is extremelysimplified). So this algorithm will finish on the order of (10000 ÷109) = .00001 seconds. A running time of .00001seconds is reasonable, and that’s why this is called a practical algorithm. The same algorithm with an input length of1,000,000 will take on the order of 17 minutes, which is also a reasonable time for most (non-real-time) applications.Meanwhile, an algorithm that runs in exponential time might have a running time of 2n. The number of operationsthat it will require, for n=100, is 2100. It will take (2100 ÷ 109) ≈ 1.3×1021 seconds, which is (1.3×1021 ÷ 31556926)≈ 4.1×1013 years, longer than the age of the universe. The largest problem this algorithm could solve in a day wouldhave n=46, which seems very small.Mathematically speaking, for big enough inputs, any polynomial time algorithm will beat any exponential time algo-rithm, and by arbitrarily large amounts. The only question is how big the input must be for this crossover to occur.

13.2 References[1] D. Pisinger, 2003. “Where are the hard knapsack problems?" Technical Report 2003/08, Department of Computer Science,

University of Copenhagen, Copenhagen, Denmark, see , accessed 31 January 2015.

[2] Oded Goldreich (2008), Computational complexity: a conceptual perspective, Cambridge University Press, p. 128, ISBN978-0-521-88473-0

[3] Dexter Kozen (2006), Theory of computation, Birkhäuser, p. 4, ISBN 978-1-84628-297-3

[4] Egon Börger (1989), Computability, complexity, logic, Elsevier, p. 225, ISBN 978-0-444-87406-1

[5] Steven Homer and Alan L. Selman (1992), “Complexity Theory”, in Alan Kent and James G. Williams, Encyclopedia ofComputer Science and Technology 26, CRC Press

[6] Alan Cobham (1965), “The intrinsic computational difficulty of functions”, Proc. Logic, Methodology, and Philosophy ofScience II, North Holland

Chapter 14

Combinatorial optimization

8

3

8

7

2

2

4

31

A minimum spanning tree of a weighted planar graph. Finding a minimum spanning tree is a common problem involving combina-torial optimization.

In applied mathematics and theoretical computer science, combinatorial optimization is a topic that consists offinding an optimal object from a finite set of objects.[1] In many such problems, exhaustive search is not feasible. Itoperates on the domain of those optimization problems, in which the set of feasible solutions is discrete or can bereduced to discrete, and in which the goal is to find the best solution. Some common problems involving combinatorialoptimization are the traveling salesman problem (“TSP”) and the minimum spanning tree problem (“MST”).Combinatorial optimization is a subset of mathematical optimization that is related to operations research, algorithmtheory, and computational complexity theory. It has important applications in several fields, including artificial intel-ligence, machine learning, mathematics, auction theory, and software engineering.

38

14.1. APPLICATIONS 39

Some research literature[2] considers discrete optimization to consist of integer programming together with combi-natorial optimization (which in turn is composed of optimization problems dealing with graph structures) althoughall of these topics have closely intertwined research literature. It often involves determining the way to efficientlyallocate resources used to find solutions to mathematical problems.

14.1 Applications

Applications for combinatorial optimization include, but are not limited to:

• Developing the best airline network of spokes and destinations

• Deciding which taxis in a fleet to route to pick up fares

• Determining the optimal way to deliver packages

• Determining the right attributes of concept elements prior to concept testing

14.2 Methods

There is a large amount of literature on polynomial-time algorithms for certain special classes of discrete optimiza-tion, a considerable amount of it unified by the theory of linear programming. Some examples of combinatorialoptimization problems that fall into this framework are shortest paths and shortest path trees, flows and circulations,spanning trees, matching, and matroid problems.For NP-complete discrete optimization problems, current research literature includes the following topics:

• polynomial-time exactly solvable special cases of the problem at hand (e.g. see fixed-parameter tractable)

• algorithms that perform well on “random” instances (e.g. for TSP)

• approximation algorithms that run in polynomial time and find a solution that is “close” to optimal

• solving real-world instances that arise in practice and do not necessarily exhibit the worst-case behavior inherentin NP-complete problems (e.g. TSP instances with tens of thousands of nodes[3]).

Combinatorial optimization problems can be viewed as searching for the best element of some set of discrete items;therefore, in principle, any sort of search algorithm or metaheuristic can be used to solve them. However, genericsearch algorithms are not guaranteed to find an optimal solution, nor are they guaranteed to run quickly (in polynomialtime). Since some discrete optimization problems are NP-complete, such as the traveling salesman problem, this isexpected unless P=NP.

14.3 Specific problems

• Assignment problem

• Closure problem

• Constraint satisfaction problem

• Cutting stock problem

• Integer programming

• Knapsack problem

• Minimum spanning tree

• Nurse scheduling problem

40 CHAPTER 14. COMBINATORIAL OPTIMIZATION

An optimal traveling salesperson tour through Germany’s 15 largest cities. It is the shortest among 43,589,145,600[nb 1] possible toursvisiting each city exactly once.

• Traveling salesman problem

• Vehicle routing problem

• Vehicle rescheduling problem

• Weapon target assignment problem

14.4 Further reading

• Schrijver, Alexander. Combinatorial Optimization: Polyhedra and Efficiency. Algorithms and Combinatorics24. Springer.

14.5. REFERENCES 41

14.5 References[1] Schrijver, p. 1

[2] “Discrete Optimization”. Elsevier. Retrieved 2009-06-08.

[3] Cook, William. “Optimal TSP Tours”. University of Waterloo. Retrieved 2009-06-08.

[1] Take one city, and take all possible orders of the other 14 cities. Then divide by two because it does not matter in whichdirection in time they come after each other: 14!/2 = 43,589,145,600.

14.6 External links• Alexander Schrijver. On the history of combinatorial optimization (till 1960).

Lecture notes

• Integer programming notes, J E Beasley.

Source code

• Java Combinatorial Optimization Platform open source project.

Workshops

• The Aussois Combinatorial Optimization Workshop

Others

• Alexander Schrijver; A Course in Combinatorial Optimization February 1, 2006 (© A. Schrijver)

• William J. Cook, William H. Cunningham, William R. Pulleyblank, Alexander Schrijver; Combinatorial Op-timization; John Wiley & Sons; 1 edition (November 12, 1997); ISBN 0-471-55894-X.

• Eugene Lawler (2001). Combinatorial Optimization: Networks and Matroids. Dover. ISBN 0-486-41453-1.

• Jon Lee; A First Course in Combinatorial Optimization; Cambridge University Press; 2004; ISBN 0-521-01012-8.

• Pierluigi Crescenzi, Viggo Kann, Magnús Halldórsson, Marek Karpinski, Gerhard Woeginger, A Compendiumof NP Optimization Problems.

• Christos H. Papadimitriou and Kenneth Steiglitz Combinatorial Optimization : Algorithms and Complexity;Dover Pubns; (paperback, Unabridged edition, July 1998) ISBN 0-486-40258-4.

• Arnab Das and Bikas K Chakrabarti (Eds.) Quantum Annealing and Related Optimization Methods, LectureNote in Physics, Vol. 679, Springer, Heidelberg (2005)

• Journal of Combinatorial Optimization

• Arnab Das and Bikas K Chakrabarti, Rev. Mod. Phys. 80 1061 (2008)

Chapter 15

Combinatorial search

In computer science and artificial intelligence, combinatorial search studies search algorithms for solving instancesof problems that are believed to be hard in general, by efficiently exploring the usually large solution space of theseinstances. Combinatorial search algorithms achieve this efficiency by reducing the effective size of the search spaceor employing heuristics. Some algorithms are guaranteed to find the optimal solution, while others may only returnthe best solution found in the part of the state space that was explored.Classic combinatorial search problems include solving the eight queens puzzle or evaluating moves in games with alarge game tree, such as reversi or chess.A study of computational complexity theory helps to motivate combinatorial search. Combinatorial search algorithmsare typically concerned with problems that are NP-hard. Such problems are not believed to be efficiently solvablein general. However, the various approximations of complexity theory suggest that some instances (e.g. “small”instances) of these problems could be efficiently solved. This is indeed the case, and such instances often haveimportant practical ramifications.

15.1 Examples

Common algorithms for solving combinatorial search problems include:

• A* search algorithm

• Alpha-beta pruning

• Branch-and-bound

• Minimax

15.2 Lookahead

Lookahead is an important component of combinatorial search, which specifies, roughly, how deeply the graph rep-resenting the problem is explored. The need for a specific limit on lookahead comes from the large problem graphsin many applications, such as computer chess and computer Go. A naive breadth-first search of these graphs wouldquickly consume all the memory of any modern computer. By setting a specific lookahead limit, the algorithm’s timecan be carefully controlled; its time increases exponentially as the lookahead limit increases.More sophisticated search techniques such as alpha-beta pruning are able to eliminate entire subtrees of the searchtree from consideration. When these techniques are used, lookahead is not a precisely defined quantity, but insteadeither the maximum depth searched or some type of average.

42

15.3. SEE ALSO 43

15.3 See also• Brute-force search

• Combinatorial explosion

• Combinatorial optimization

• Search algorithm

• State space search

15.4 References• Russell and Norvig. Artificial Intelligence: A Modern Approach.

Chapter 16

Communication complexity

The notion of communication complexitywas introduced by Yao in 1979,[1] who investigated the following probleminvolving two separated parties (Alice and Bob). Alice receives an n-bit string x and Bob another n-bit string y, andthe goal is for one of them (say Bob) to compute a certain function f(x,y) with the least amount of communicationbetween them. Note that here we are not concerned about the number of computational steps, or the size of thecomputer memory used. Communication complexity tries to quantify the amount of communication required forsuch distributed computations.Of course they can always succeed by having Alice send her whole n-bit string to Bob, who then computes thefunction, but the idea here is to find clever ways of calculating f with fewer than n bits of communication.This abstract problem is relevant in many contexts: in VLSI circuit design, for example, one wants to minimize energyused by decreasing the amount of electric signals required between the different components during a distributedcomputation. The problem is also relevant in the study of data structures, and in the optimization of computernetworks. For a survey of the field, see the book by Kushilevitz and Nisan.

16.1 Formal definition

Let f : X × Y → Z where we assume in the typical case that X = Y = 0, 1n and Z = 0, 1 . Alice draws ann-bit string x ∈ X while Bob draws an n-bit string y ∈ Y. By communicating to each other one bit at a time (adoptingsome communication protocol), Alice and Bob want to compute the value of f(x, y) such that at least one party knowsthe value at the end of the communication. At this point the answer can be communicated back so that at the cost ofone extra bit, both parties will know the answer. The worst case communication complexity of this communicationprotocol, denoted as D(f) , is then defined to be

D(f) =

Using the above definition, it is useful to think of the function f as a matrix A (called the input matrix) where eachrow of the matrix corresponds to x ∈ X and each column corresponds to y ∈ Y. An entry in the input matrix isAx,y = f(x, y) . Initially both Alice and Bob have a copy of the entire matrix A (assuming the function f is knownto both). Then, the problem of computing the function value can be rephrased as “zeroing-in” on the correspondingmatrix entry. This problem can be solved if either Alice or Bob knows both x and y . At the start of communication,the number of choices for the value of the function on the inputs is the size of matrix, i.e. 22n . Then, as and wheneach party communicates a bit to the other, the number of choices for the answer reduces as this eliminates a set ofrows/columns resulting in a submatrix of A.More formally, a set R ⊆ X × Y is called a (combinatorial) rectangle if whenever (x1, y1) ∈ R and (x2, y2) ∈ R then(x1, y2) ∈ R. Equivalently, R can also be viewed as a submatrix of the input matrix A such that R = M × N whereM ⊆ X and N ⊆ Y. Consider the case when k bits are already exchanged between the parties. Now, for a particularh ∈ 0, 1k , let us define a matrix

Th = (x, y) : the k-bits exchanged on input (x, y) is h

44

16.2. RANDOMIZED COMMUNICATION COMPLEXITY 45

Then, Th ⊆ X × Y, and Th is a rectangle and a submatrix of A.

16.1.1 Example: EQ

We consider the case where Alice and Bob try to determine if they both have the same string. That is, we are trying todetermine if x is equal to y . It is easy to prove that the equality problem (EQ) will always require you to communicaten bits in the worst case if you want to be absolutely sure x and y are equal. Consider the simple case of x and ybeing 3 bits. The equality function in this case can be represented by the matrix below. The rows representing all thepossibilities of x , the columns those of y .

As you can see, the function only evaluates to 1 when x equals y (i.e., on the diagonal). It is also fairly easy to seehow communicating a single bit divides your possibilities in half. If you know that the first bit of y is 1, you onlyneed to consider half of the columns (where y can equal 100, 101, 110, or 111).Theorem: D(EQ) = n .Proof. Assume thatD(EQ) ≤ n−1 . This means that there exists an (x, x) and an (x′, x′) having the same history,h . Since this history defines a rectangle, f(x, x′) must also be 1. By definition x = x′ and we know that equality isonly true for (a, b) when a = b . Thus, we have a contradiction.Intuitively, for D(EQ) less than n , we need to be able to define a rectangle in the EQ matrix greater in size than asingle cell. All of the cells in this rectangle must contain 1 for us to be able to generalize that this rectangle equals 1.It is not possible to form such a rectangle in the equality matrix.

16.2 Randomized communication complexity

In the above definition, we are concerned with the number of bits that must be deterministically transmitted betweentwo parties. If both the parties are given access to a random number generator, can they determine the value of fwith much less information exchanged? Yao, in his seminal paper[1] answers this question by defining randomizedcommunication complexity.A randomized protocol R for a function f has two-sided error.

Pr[R(x, y) = 0] >1

2, if f(x, y) = 0

Pr[R(x, y) = 1] >1

2, if f(x, y) = 1

A randomized protocol is a deterministic protocol that uses an extra random string in addition to its normal input.There are two models for this: a public string is a random string that is known by both parties beforehand, while aprivate string is generated by one party and must be communicated to the other party. A theorem presented belowshows that any public string protocol can be simulated by a private string protocol that uses O(log n) additional bitscompared to the original.Note that in the probability inequalities above, the outcome of the protocol is understood to depend only on therandom string; both strings x and y remain fixed. In other words, if R(x,y) yields g(x,y,r) when using random stringr, then g(x,y,r) = f(x,y) for at least half of all choices for the string r.The randomized complexity is simply defined as the number of bits exchanged in such a protocol.Note that it is also possible to define a randomized protocol with one-sided error, and the complexity is definedsimilarly.

16.2.1 Example: EQ

Returning to the previous example of EQ, if certainty is not required, Alice and Bob can check for equality usingonly O(log n) messages. Consider the following protocol: Assume that Alice and Bob both have access to the same

46 CHAPTER 16. COMMUNICATION COMPLEXITY

random string z ∈ 0, 1n . Alice computes z · x and sends this bit (call it b) to Bob. (The (·) is the dot productin GF(2).) Then Bob compares b to z · y . If they are the same, then Bob accepts, saying x equals y. Otherwise, herejects.Clearly, if x = y , then z ·x = z ·y , so Probz[Accept] = 1 . If x does not equal y, it is still possible that z ·x = z ·y, which would give Bob the wrong answer. How does this happen?If x and y are not equal, they must differ in some locations:

x = c1c2 . . . p . . . p′ . . . xn

y = c1c2 . . . q . . . q′ . . . yn

z = z1z2 . . . zi . . . zj . . . zn

Where x and y agree, zi ∗ xi = zi ∗ ci = zi ∗ yi so those terms affect the dot products equally. We can safely ignorethose terms and look only at where x and y differ. Furthermore, we can swap the bits xi and yi without changingwhether or not the dot products are equal. This means we can swap bits so that x contains only zeros and y containsonly ones:

x′ = 00 . . . 0

y′ = 11 . . . 1

z′ = z1z2 . . . zn′

Note that z′ · x′ = 0 and z′ · y′ = Σiz′i . Now, the question becomes: for some random string z′ , what is the

probability that Σiz′i = 0 ? Since each z′i is equally likely to be 0 or 1 , this probability is just 1/2 . Thus, when x

does not equal y , Probz[Accept] = 1/2 . The algorithm can be repeated many times to increase its accuracy. Thisfits the requirements for a randomized communication algorithm.This shows that if Alice and Bob share a random string of length n, they can send one bit to each other to computeEQ(x, y) . In the next section, it is shown that Alice and Bob can exchange only O(log n) bits that are as good assharing a random string of length n. Once that is shown, it follows that EQ can be computed in O(log n) messages.

16.2.2 Public coins versus private coins

It is easier to create random protocols when both parties have access to the same random string (shared string pro-tocol). It is still possible to use these protocols even when the two parties don't share a random string (private stringprotocol) with a small communication cost. Any shared string random protocol using an n -bit string can be simulatedby a private string protocol that uses an extra O(log n) bits.Intuitively, we can find some set of strings that has enough randomness in it to run the random protocol with onlya small increase in error. This set can be shared beforehand, and instead of drawing a random string, Alice andBob need only agree on which string to choose from the shared set. This set is small enough that the choice can becommunicated efficiently. A formal proof follows.Consider some random protocol P with a maximum error rate of 0.1. Let R be 100n strings of length n, numberedr1, r2, . . . , r100n . Given such an R , define a new protocol P ′

R which randomly picks some ri and then runs P usingri as the shared random string. It takes O(log 100n) = O(log n) bits to communicate the choice of ri .Let us define p(x, y) and p′R(x, y) to be the probabilities that P and P ′

R compute the correct value for the input(x, y) .For a fixed (x, y) , we can use Hoeffding’s inequality to get the following equation:

PrR[|p′R(x, y)− p(x, y)| ≥ 0.1] ≤ 2 exp(−2(0.1)2 · 100n) < 2−2n

Thus when we don't have (x, y) fixed:

16.3. QUANTUM COMMUNICATION COMPLEXITY 47

PrR[∃(x, y) : |p′R(x, y)− p(x, y)| ≥ 0.1] ≤

∑(x,y)

PrR[|p′R(x, y)− p(x, y)| ≥ 0.1] <

∑(x,y)

2−2n = 1

The last equality above holds because there are 22n different pairs (x, y) . Since the probability does not equal 1,there is some R0 so that for all (x, y) :

|p′R0(x, y)− p(x, y)| < 0.1

Since P has at most 0.1 error probability, P ′R0

can have at most 0.2 error probability.

16.3 Quantum communication complexity

Quantum communication complexity tries to quantify the communication reduction possible by using quantum effectsduring a distributed computation.At least three quantum generalizations of communication complexity have been proposed; for a survey see the sug-gested text by G. Brassard.The first one is the qubit-communication model, where the parties can use quantum communication instead of classicalcommunication, for example by exchanging photons through an optical fiber.In a second model the communication is still performed with classical bits, but the parties are allowed to manipulatean unlimited supply of quantum entangled states as part of their protocols. By doing measurements on their entangledstates, the parties can save on classical communication during a distributed computation.The third model involves access to previously shared entanglement in addition to qubit communication, and is theleast explored of the three quantum models.

16.4 Nondeterministic communication complexity

In nondeterministic communication complexity, Alice and Bob have access to an oracle. After receiving the ora-cle’s word, the parties communicate to deduce f(x,y). The nondeterministic communication complexity is then themaximum over all pairs (x,y) over the sum of number of bits exchanged and the coding length of the oracle word.Viewed differently, this amounts to covering all 1-entries of the 0/1-matrix by combinatorial 1-rectangles (i.e., non-contiguous, non-convex submatrices, whose entries are all one (see Kushilevitz and Nisan or Dietzfelbinger et al.)).The nondeterministic communication complexity is the binary logarithm of the rectangle covering number of thematrix: the minimum number of combinatorial 1-rectangles required to cover all 1-entries of the matrix, withoutcovering any 0-entries.Nondeterministic communication complexity occurs as a means to obtaining lower bounds for deterministic commu-nication complexity (see Dietzfelbinger et al.), but also in the theory of nonnegative matrices, where it gives a lowerbound on the nonnegative rank of a nonnegative matrix.[2]

16.5 Open problems

Considering a 0/1 input matrix Mf = [f(x, y)]x,y∈0,1n , the minimum number of bits exchanged to compute fdeterministically in the worst case, D(f) , is known to be bounded from below by the logarithm of the rank of thematrix Mf . The log rank conjecture proposes that the communication complexity, D(f) , of Mf is bounded fromabove by a constant power of the logarithm of the rank of Mf . Since D(f) is bounded from above and below bypolynomials of log rank (Mf ) , we can say D(f) is polynomially related to log rank (Mf ) . Since the rank of a matrixis polynomial time computable in the size of the matrix, such an upper bound would allow the matrix’s communicationcomplexity to be approximated in polynomial time. Note, however, that the size of the matrix itself is exponential inthe size of the input.

48 CHAPTER 16. COMMUNICATION COMPLEXITY

For a randomized protocol, the number of bits exchanged in the worst case, R(f), is conjectured to be polynomiallyrelated to the following formula:

log min(rank(M ′f ) :M

′f ∈ R2n×2n , (Mf −M ′

f )∞ ≤ 1/3).

Such log rank conjectures are valuable because they reduce the question of a matrix’s communication complexity toa question of linearly independent rows (columns) of the matrix. This reveals that the essence of the communicationcomplexity problem, for example in the EQ case above, is figuring out where in the matrix the inputs are, in order tofind out if they're equivalent.

16.6 Applications

Lower bounds in communication complexity can be used to prove lower bounds in decision tree complexity, VLSIcircuits, data structures, streaming algorithms, space–time tradeoffs for Turing machines and more.

16.7 References• Kushilevitz, E. and N. Nisan. Communication complexity. Cambridge University Press, 1997.

• Brassard, G. Quantum communication complexity: a survey. http://arxiv.org/abs/quant-ph/0101005

• Dietzfelbinger, M., J. Hromkovic, J., and G. Schnitger, “A comparison of two lower-bound methods for com-munication complexity”, Theoret. Comput. Sci. 168, 1996. 39-51.

• Raz, Ran. “Circuit and Communication Complexity.” In Computational Complexity Theory. Steven Rudichand Avi Wigderson, eds. American Mathematical Society Institute for Advanced Study, 2004. 129-137.

• A. C. Yao, “Some Complexity Questions Related to Distributed Computing”, Proc. of 11th STOC, pp. 209–213, 1979. 14

• I. Newman, Private vs. Common Random Bits in Communication Complexity, Information Processing Letters39, 1991, pp. 67–71.

16.8 Notes[1] Yao, A. C. (1979), “Some Complexity Questions Related to Distributed Computing”, Proc. of 11th STOC 14: 209–213

[2] Yannakakis, M. (1991). “Expressing combinatorial optimization problems by linear programs”. J. Comput. System Sci. 43(3): 441–466. doi:10.1016/0022-0000(91)90024-y.

Chapter 17

Complement (complexity)

In computational complexity theory, the complement of a decision problem is the decision problem resulting fromreversing the yes and no answers.[1] Equivalently, if we define decision problems as sets of finite strings, then thecomplement of this set over some fixed domain is its complement problem.[2]

For example, one important problem is whether a number is a prime number. Its complement is to determine whethera number is a composite number (a number which is not prime). Here the domain of the complement is the set of allintegers exceeding one.[3]

There is a Turing reduction from every problem to its complement problem.[4] The complement operation is aninvolution, meaning it “undoes itself”, or the complement of the complement is the original problem.One can generalize this to the complement of a complexity class, called the complement class, which is the set ofcomplements of every problem in the class.[5] If a class is called C, its complement is conventionally labelled co-C.Notice that this is not the complement of the complexity class itself as a set of problems, which would contain a greatdeal more problems.A class is said to be closed under complement if the complement of any problem in the class is still in the class.[6]

Because there are Turing reductions from every problem to its complement, any class which is closed under Turingreductions is closed under complement. Any class which is closed under complement is equal to its complementclass. However, under many-one reductions, many important classes, especially NP, are believed to be distinct fromtheir complement classes (although this has not been proven).[7]

The closure of any complexity class under Turing reductions is a superset of that class which is closed under com-plement. The closure under complement is the smallest such class. If a class is intersected with its complement, weobtain a (possibly empty) subset which is closed under complement.Every deterministic complexity class (DSPACE(f(n)), DTIME(f(n)) for all f(n)) is closed under complement,[8]

because one can simply add a last step to the algorithm which reverses the answer. This doesn't work for nondeter-ministic complexity classes, because if there exist both computation paths which accept and paths which reject, andall the paths reverse their answer, there will still be paths which accept and paths which reject — consequently, themachine accepts in both cases.Some of the most surprising complexity results shown to date showed that the complexity classes NL and SL are infact closed under complement, whereas before it was widely believed they were not (see Immerman–Szelepcsényitheorem). The latter has become less surprising now that we know SL equals L, which is a deterministic class.Every class which is low for itself is closed under complement.

17.1 References[1] Itô, Kiyosi (1993), Encyclopedic Dictionary of Mathematics, Volume 1, MIT Press, p. 269, ISBN 9780262590204.

[2] Schrijver, Alexander (1998), Theory of Linear and Integer Programming, Wiley Series in Discrete Mathematics & Opti-mization, John Wiley & Sons, p. 19, ISBN 9780471982326.

[3] Homer, Steven; Selman, Alan L. (2011), Computability and Complexity Theory, Texts in Computer Science, Springer,ISBN 9781461406815.

49

50 CHAPTER 17. COMPLEMENT (COMPLEXITY)

[4] Singh, Arindama (2009), Elements of Computation Theory, Texts in Computer Science, Springer, Exercise 9.10, p. 287,ISBN 9781848824973.

[5] Bovet, Daniele; Crescenzi, Pierluigi (1994), Introduction to the Theory of Complexity, Prentice Hall International Series inComputer Science, Prentice Hall, pp. 133–134, ISBN 9780139153808.

[6] Vollmer, Heribert (1999), Introduction to Circuit Complexity: A Uniform Approach, Texts in Theoretical Computer Science.An EATCS Series, Springer, p. 113, ISBN 9783540643104.

[7] Pruim, R.; Wegener, Ingo (2005), Complexity Theory: Exploring the Limits of Efficient Algorithms, Springer, p. 66, ISBN9783540274773.

[8] Ausiello, Giorgio (1999), Complexity and Approximation: Combinatorial Optimization Problems and Their ApproximabilityProperties, Springer, p. 189, ISBN 9783540654315.

Chapter 18

Complete (complexity)

In computational complexity theory, a computational problem is complete for a complexity class if it is, in a technicalsense, among the “hardest” (or “most expressive”) problems in the complexity class.More formally, a problem p is called hard for a complexity class C under a given type of reduction, if there exists areduction (of the given type) from any problem in C to p. If a problem is both hard for the class and a member ofthe class, it is complete for that class (for that type of reduction).A problem that is complete for a class C is said to be C-complete, and the class of all problems complete for C isdenoted C-complete. The first complete class to be defined and the most well-known is NP-complete, a class thatcontains many difficult-to-solve problems that arise in practice. Similarly, a problem hard for a class C is calledC-hard, e.g. NP-hard.Normally it is assumed that the reduction in question does not have higher computational complexity than the classitself. Therefore it may be said that if a C-complete problem has a “computationally easy” solution, then all problemsin “C” have an “easy” solution.Generally, complexity classes that have a recursive enumeration have known complete problems, whereas those thatdo not, don't have any known complete problems. For example, NP, co-NP, PLS, PPA all have known naturalcomplete problems, while RP, ZPP, BPP and TFNP do not have any known complete problems (although such aproblem may be discovered in the future).There are classes without complete problems. For example, Sipser showed that there is a languageM such thatBPPM

(BPP with oracle M) has no complete problems.[1]

18.1 References[1] M. Sipser. On relativization and the existence of complete sets, Proceedings of ICALP'82, Springer-Verlag Lecture Notes

in Computer Science volume 140, pp. 523-531, 1982.

51

Chapter 19

Complexity class

In computational complexity theory, a complexity class is a set of problems of related resource-based complexity.A typical complexity class has a definition of the form:

the set of problems that can be solved by an abstract machine M using O(f(n)) of resource R, where nis the size of the input.

For example, the class NP is the set of decision problems whose solutions can be determined by a non-deterministicTuring machine in polynomial time, while the class PSPACE is the set of decision problems that can be solved by adeterministic Turing machine in polynomial space.The simpler complexity classes are defined by the following factors:

• The type of computational problem: The most commonly used problems are decision problems. However,complexity classes can be defined based on function problems (an example is FP), counting problems (e.g.#P), optimization problems, promise problems, etc.

• The model of computation: The most common model of computation is the deterministic Turing machine, butmany complexity classes are based on nondeterministic Turing machines, boolean circuits, quantum Turingmachines, monotone circuits, etc.

• The resource (or resources) that are being bounded and the bounds: These two properties are usually statedtogether, such as “polynomial time”, “logarithmic space”, “constant depth”, etc.

Many complexity classes can be characterized in terms of the mathematical logic needed to express them; seedescriptive complexity.Bounding the computation time above by some concrete function f(n) often yields complexity classes that depend onthe chosen machine model. For instance, the language xx | x is any binary string can be solved in linear time on amulti-tape Turing machine, but necessarily requires quadratic time in the model of single-tape Turing machines. Ifwe allow polynomial variations in running time, Cobham-Edmonds thesis states that “the time complexities in anytwo reasonable and general models of computation are polynomially related” (Goldreich 2008, Chapter 1.2). Thisforms the basis for the complexity class P, which is the set of decision problems solvable by a deterministic Turingmachine within polynomial time. The corresponding set of function problems is FP.The Blum axioms can be used to define complexity classes without referring to a concrete computational model.

19.1 Important complexity classes

Many important complexity classes can be defined by bounding the time or space used by the algorithm. Someimportant complexity classes of decision problems defined in this manner are the following:It turns out that PSPACE = NPSPACE and EXPSPACE = NEXPSPACE by Savitch’s theorem.

52

19.2. REDUCTION 53

Other important complexity classes include BPP, ZPP and RP, which are defined using probabilistic Turing machines;AC and NC, which are defined using boolean circuits and BQP and QMA, which are defined using quantum Turingmachines. #P is an important complexity class of counting problems (not decision problems). Classes like IP andAM are defined using Interactive proof systems. ALL is the class of all decision problems.

19.2 Reduction

Main article: Reduction (complexity)

Many complexity classes are defined using the concept of a reduction. A reduction is a transformation of one probleminto another problem. It captures the informal notion of a problem being at least as difficult as another problem. Forinstance, if a problemX can be solved using an algorithm for Y,X is no more difficult than Y, and we say thatX reducesto Y. There are many different types of reductions, based on the method of reduction, such as Cook reductions, Karpreductions and Levin reductions, and the bound on the complexity of reductions, such as polynomial-time reductionsor log-space reductions.The most commonly used reduction is a polynomial-time reduction. This means that the reduction process takespolynomial time. For example, the problem of squaring an integer can be reduced to the problem of multiplying twointegers. This means an algorithm for multiplying two integers can be used to square an integer. Indeed, this can bedone by giving the same input to both inputs of the multiplication algorithm. Thus we see that squaring is not moredifficult than multiplication, since squaring can be reduced to multiplication.This motivates the concept of a problem being hard for a complexity class. A problem X is hard for a class ofproblems C if every problem in C can be reduced to X. Thus no problem in C is harder than X, since an algorithmfor X allows us to solve any problem in C. Of course, the notion of hard problems depends on the type of reductionbeing used. For complexity classes larger than P, polynomial-time reductions are commonly used. In particular, theset of problems that are hard for NP is the set of NP-hard problems.If a problem X is in C and hard for C, then X is said to be complete for C. This means that X is the hardest problem inC (Since there could be many problems which are equally hard, one might say that X is one of the hardest problemsin C). Thus the class of NP-complete problems contains the most difficult problems in NP, in the sense that they arethe ones most likely not to be in P. Because the problem P = NP is not solved, being able to reduce another problem,Π1, to a known NP-complete problem, Π2, would indicate that there is no known polynomial-time solution for Π1.This is because a polynomial-time solution to Π1 would yield a polynomial-time solution to Π2. Similarly, becauseall NP problems can be reduced to the set, finding an NP-complete problem that can be solved in polynomial timewould mean that P = NP.

19.3 Closure properties of classes

Complexity classes have a variety of closure properties; for example, decision classes may be closed under negation,disjunction, conjunction, or even under all Boolean operations. Moreover, they might also be closed under a varietyof quantification schemes. P, for instance, is closed under all Boolean operations, and under quantification overpolynomially sized domains. However, it is most likely not closed under quantification over exponential sized domains.Each class X that is not closed under negation has a complement class co-Y, which consists of the complements ofthe languages contained in X. Similarly one can define the Boolean closure of a class, and so on; this is however lesscommonly done.One possible route to separating two complexity classes is to find some closure property possessed by one and not bythe other.

19.4 Relationships between complexity classes

The following table shows some of the classes of problems (or languages, or grammars) that are considered in com-plexity theory. If class X is a strict subset of Y, then X is shown below Y, with a dark line connecting them. IfX is a subset, but it is unknown whether they are equal sets, then the line is lighter and is dotted. Technically, the

54 CHAPTER 19. COMPLEXITY CLASS

breakdown into decidable and undecidable pertains more to the study of computability theory but is useful for puttingthe complexity classes in perspective.

19.4.1 Hierarchy theorems

Main articles: time hierarchy theorem and space hierarchy theorem

For the complexity classes defined in this way, it is desirable to prove that relaxing the requirements on (say) compu-tation time indeed defines a bigger set of problems. In particular, although DTIME(n) is contained in DTIME(n2), itwould be interesting to know if the inclusion is strict. For time and space requirements, the answer to such questionsis given by the time and space hierarchy theorems respectively. They are called hierarchy theorems because theyinduce a proper hierarchy on the classes defined by constraining the respective resources. Thus there are pairs ofcomplexity classes such that one is properly included in the other. Having deduced such proper set inclusions, wecan proceed to make quantitative statements about how much more additional time or space is needed in order toincrease the number of problems that can be solved.More precisely, the time hierarchy theorem states that

DTIME(f(n)

)⊊ DTIME

(f(n) · log2(f(n))

)The space hierarchy theorem states that

DSPACE(f(n)

)⊊ DSPACE

(f(n) · log(f(n))

)The time and space hierarchy theorems form the basis for most separation results of complexity classes. For instance,the time hierarchy theorem tells us that P is strictly contained in EXPTIME, and the space hierarchy theorem tells usthat L is strictly contained in PSPACE.

19.5 See also• List of complexity classes

19.6 References

19.7 Further reading• The Complexity Zoo: A huge list of complexity classes, a reference for experts.

• Diagram by Neil Immerman showing the hierarchy of complexity classes and how they fit together.

• Michael Garey, and David S. Johnson: Computers and Intractability: AGuide to the Theory of NP-Completeness.New York: W. H. Freeman & Co., 1979. The standard reference on NP-Complete problems - an importantcategory of problems whose solutions appear to require an impractically long time to compute.

Chapter 20

Complexity index

Besides complexity intended as a difficulty to compute a function (see computational complexity), in modern computerscience and in statistics another complexity index of a function stands for denoting its information content, in turnaffecting the difficulty of learning the function from examples. Complexity indices in this sense characterize the entireclass of functions to which the one we are interested in belongs. Focusing on Boolean functions, the detail of a classC of Boolean functions c essentially denotes how deeply the class is articulated.To identify this index we must first define a sentry function of C . Let us focus for a moment on a single function c,call it a concept defined on a set X of elements that we may figure as points in a Euclidean space. In this framework,the above function associates to c a set of points that, since are defined to be external to the concept, prevent it fromexpanding into another function of C . We may dually define these points in terms of sentinelling a given concept cfrom being fully enclosed (invaded) by another concept within the class. Therefore we call these points either sentinelsor sentry points; they are assigned by the sentry function S to each concept of C in such a way that:

1. the sentry points are external to the concept c to be sentineled and internal to at least one other including it,

2. each concept c′ including c has at least one of the sentry points of c either in the gap between c and c′ , oroutside c′ and distinct from the sentry points of c′ , and

3. they constitute a minimal set with these properties.

The technical definition coming from (Apolloni 2006) is rooted in the inclusion of an augmented concept c+ madeup of c plus its sentry points by another (c′)+ in the same class.

20.1 Definition of sentry function

For a concept class C on a space X , a sentry function is a total function S : C∪∅,X 7→ 2X satisfying the followingconditions:

1. Sentinels are outside the sentineled concept ( c ∩ S(c) = ∅ for all c ∈ C ).

2. Sentinels are inside the invading concept (Having introduced the sets c+ = c ∪ S(c) , an invading conceptc′ ∈ C is such that c′ ⊆ c and c+ ⊆ (c′)

+ . Denoting up(c) the set of concepts invading c, we must have thatif c2 ∈ up(c1) , then c2 ∩ S(c1) = ∅ ).

3. S(c) is a minimal set with the above properties (No S′ = S exists satisfying (1) and (2) and having theproperty that S′(c) ⊆ S(c) for every c ∈ C ).

4. Sentinels are honest guardians. It may be that c ⊆ (c′)+ but S(c) ∩ c′ = ∅ so that c′ ∈ up(c) . This however

must be a consequence of the fact that all points of S(c) are involved in really sentineling c against otherconcepts in up(c) and not just in avoiding inclusion of c+ by (c′)+ . Thus if we remove c′,S(c) remainsunchanged (Whenever c1 and c2 are such that c1 ⊂ c2 ∪S(c2) and c2 ∩S(c1) = ∅ , then the restriction of Sto c1 ∪ up(c1)− c2 is a sentry function on this set).

55

56 CHAPTER 20. COMPLEXITY INDEX

A schematic outlook of outer sentineling functionality

S(c) is the frontier of c upon S .With reference to the picture on the right, x1, x2, x3 is a candidate frontier of c0 against c1, c2, c3, c4 . All pointsare in the gap between a ci and c0 . They avoid inclusion of c0 ∪ x1, x2, x3 in c3 , provided that these points arenot used by the latter for sentineling itself against other concepts. Vice versa we expect that c1 uses x1 and x3 asits own sentinels, c2 uses x2 and x3 and c4 uses x1 and x2 analogously. Point x4 is not allowed as a c0 sentry pointsince, like any diplomatic seat, it should be located outside all other concepts just to ensure that it is not occupied incase of invasion by c0 .

20.1.1 Definition of detail

The frontier size of the most expensive concept to be sentineled with the least efficient sentineling function, i.e. thequantity

DC = supS,c

#S(c)

is called detail of C . S spans also over sentry functions on subsets of X sentineling in this case the intersections ofthe concepts with these subsets. Actually, proper subsets of X may host sentineling tasks that prove harder than thoseemerging with X itself.The detail DC is a complexity measure of concept classes dual to the VC dimension DVC . The former uses pointsto separate sets of concepts, the latter concepts for partitioning sets of points. In particular the following inequalityholds (Apolloni 1997)

20.2. REFERENCES 57

DC ≤ DVC + 1

See also Rademacher complexity for a recently introduced class complexity index.

20.1.2 Example: continuous spaces

Class C of circles in R2 has detail DC = 2 , as shown in the picture on left below. Similarly, for the class of segmentson R , as shown in the picture on right.

20.1.3 Example: discrete spaces

The class C = c1, c2, c3, c4 on X = x1, x2, x3 whose concepts are illustrated in the following scheme, where “+ ” denotes an element xj belonging to ci , “ − ” an element outside ci and ⃝ a sentry point:This class has DC = 2 . As usual we may have different sentineling functions. A worst case S , as illustrated, is:S(c1) = x1, x2,S(c2) = x1,S(c3) = x2,S(c4) = ∅ . However a cheaper one is S(c1) = x3,S(c2) =x1,S(c3) = x2,S(c4) = ∅ :

20.2 References• Apolloni, B; Malchiodi, D.; Gaito, S. (2006). Algorithmic Inference in Machine Learning. International Series

on Advanced Intelligence 5 (2nd ed.). Adelaide: Magill. Advanced Knowledge International

• Apolloni, B.; Chiaravalli, S. (1997). “PAC learning of concept classes through the boundaries of their items”.Theoretical Computer Science 172 (1–2): 91–120. doi:10.1016/S0304-3975(95)00240-5.

Chapter 21

Compression theorem

In computational complexity theory the compression theorem is an important theorem about the complexity ofcomputable functions.The theorem states that there exists no largest complexity class, with computable boundary, which contains all com-putable functions.

21.1 Compression theorem

Given a Gödel numbering φ of the computable functions and a Blum complexity measure Φ where a complexity classfor a boundary function f is defined as

C(f) := φi ∈ R(1)|(∀∞x)Φi(x) ≤ f(x).

Then there exists a total computable function f so that for all i

Dom(φi) = Dom(φf(i))

and

C(φi) ⊊ C(φf(i)).

21.2 References• Salomaa, Arto (1985), “Theorem 6.9”, Computation and Automata, Encyclopedia of Mathematics and Its

Applications 25, Cambridge University Press, pp. 149–150, ISBN 9780521302456.

• Zimand, Marius (2004), “Theorem 2.4.3 (Compression theorem)", Computational Complexity: A QuantitativePerspective, North-Holland Mathematics Studies 196, Elsevier, p. 42, ISBN 9780444828415.

58

Chapter 22

Computable topology

Computable topology is a discipline in mathematics that studies the topological and algebraic structure of computation.Computable topology includes algorithmic topology and therefore encompasses computer science. It is not to be con-fused with computational topology, which is equivalent to the topology of λ-calculus. Within computer science com-putational forms can be reduced to λ-calculus’s functional based mathematics. As shown by Alan Turing and AlonzoChurch, the λ-calculus is strong enough to describe all mechanically computable functions (see Church–Turing the-sis).[1][2][3] Lambda-calculus is then a foundational mathematics easily made into a principal programming languagefrom which other languages can be built. For this reason when considering the topology of computation it is suitableto focus on the topology of λ-calculus. Functional programming, e.g. type free lambda calculus, originated as a the-oretical foundation of mathematics. The premise relies on functional computability, where objects and functions areof the same type. The topology of λ-calculus is the Scott topology, and when restricted to continuous functions thetype free λ-calculus amounts to a topological space reliant on the tree topology. Both the Scott and Tree topologiesexhibit continuity with respect to the binary operators of application ( f applied to a = fa ) and abstraction ((λx.t(x))a= t(a)) with a modular equivalence relation based on a congruency. The algebraic structure of computation may alsobe considered as equivalent to the algebraic structure of λ-calculus, meaning the λ-algebra. The λ-algebra is found tobe an extension of the combinatory algebra, with an element introduced to accommodate abstraction.A primary concern of algorithmic topology, as its name suggests, is to develop efficient algorithms for solving topo-logical problems, or using topological methods to solve algorithmic problems from other fields.

22.1 Computational topology from λ-calculus topology

Type free λ-calculus treats functions as rules and does not differentiate functions and the objects which they are appliedto, meaning λ-calculus is type free. A by-product of type free λ-calculus is an effective computability equivalent togeneral recursion and Turing machines.[4] The set of λ-terms can be considered a functional topology in which afunction space can be embedded, meaning λ mappings within the space X are such that λ:X → X.[4][5] IntroducedNovember 1969, Dana Scott's untyped set theoretic model constructed a proper topology for any λ-calculus modelwhose function space is limited to continuous functions.[4][5] The result of a Scott continuous λ-calculus topology is afunction space built upon a programming semantic allowing fixed point combinatorics, such as the Y combinator, anddata types.[6][7] By 1971, λ-calculus was equipped to define any sequential computation and could be easily adaptedto parallel computations.[8] The reducibility of all computations to λ-calculus allows these λ-topological properties tobecome adopted by all programming languages.[4]

22.2 Computational algebra from λ-calculus algebra

Based on the operators within lambda calculus, application and abstraction, it is possible to develop an algebra whosegroup structure uses application and abstraction as binary operators. Application is defined as an operation betweenlambda terms producing a λ-term, e.g. the application of λ onto the lambda term a produces the lambda term λa.Abstraction incorporates undefined variables by denoting λx.t(x) as the function assigning the variable a to the lambdaterm with value t(a) via the operation ((λ x.t(x))a = t(a)). Lastly, an equivalence relation emerges which identifiesλ-terms modulo convertible terms, an example being beta normal form.

59

60 CHAPTER 22. COMPUTABLE TOPOLOGY

22.3 Scott topology

The Scott topology is essential in understanding the topological structure of computation as expressed through theλ-calculus. Scott found that after constructing a function space using λ-calculus one obtains a Kolmogorov space, aTo topological space which exhibits pointwise convergence, in short the product topology.[9] It is the ability of selfhomeomorphism as well as the ability to embed every space into such a space, denoted Scott continuous, as previouslydescribed which allows Scott’s topology to be applicable to logic and recursive function theory. Scott approacheshis derivation using a complete lattice, resulting in a topology dependent on the lattice structure. It is possible togeneralise Scott’s theory with the use of complete partial orders. For this reason a more general understanding of thecomputational topology is provided through complete partial orders. We will re-iterate to familiarize ourselves withthe notation to be used during the discussion of Scott topology.Complete partial orders are defined as follows:First, given the partially ordered set D=(D,≤) where a subset X of D, X ≤ D is directed, i.e.:

if X ≠ ∅ and∀ x,y ∈ X ∃ z ∈ X where x≤ z & y ≤ z

D is a complete partial order (cpo) if:

∃ bottom element ⊥ such that ⊥ ∈ D & ∀ x ∈ D ⊥ ≤ x

· Every directed X ⊆ D there exists a supremum.

We are now able to define the Scott topology over a cpo (D, ≤ ).O ⊆ D is open if:

(1) for x ∈ O, and x ≤ y, then y ∈ O, i.e. O is an upper set.

(2) for a directed set X ⊆ D, and supremum(X) ∈ O, then X ∩ O = ∅ .

Using the Scott topological definition of open it is apparent that all topological properties are met.

· ∅ and D, i.e. the empty set and whole space, are open.

· Open sets are open under arbitrary unions and under intersection:Proof: Assume Ui is open where i ∈ I, I being the index set. We define U= ∪ Ui ; i ∈ I. Take b as an element of the upper set of U, therefore a≤ b for some a ∈ U It must be that a ∈ Ui for some i, likewise b ∈ upset(Ui ). U must therefore be upper as well since Ui ∈ U.

Lastly, if D is a directed set with a supremum in U, then by assumptionsup(D) ∈Ui whereUi is open. There is necessarily a b ∈ D where upper(b)∩ D ⊆ Ui ⊆ U. The union of open sets Ui is therefore open.

· Open sets under intersection are open:Proof: Given two open sets, U and V, we define W = U ∩ V. If W = ∅then W is open. If non-empty say b ∈ upset(W) (the upper set of W), thenfor some a ∈ W, a ≤ b. Since a ∈ U ∩ V, and b an element of the upperset of both U and V, then b ∈ W. W being open implies the intersection ofopen sets is open.

Though not shown here, it is the case that the map f : D → D′ is continuous if and only if f(sup(X)) = sup(f(X))

for all directed X ⊆ D, where f(X) = f(x) | x ∈ X and the second supremum in D′ .[4]

Before we begin explaining that application as common to λ-calculus is continuous within the Scott topology werequire a certain understanding of the behavior of supremums over continuous functions as well as the conditionsnecessary for the product of spaces to be continuous namely

22.3. SCOTT TOPOLOGY 61

(1) With fii ⊆ [D → D′] be a directed family of maps, then f(x) = ∪ifi(x) if well defined and

continuous.

(2) If F ⊆ [D → D′] is directed and cpo and [D → D

′] a cpo where sup(f(x) | f ∈ F).

We now show the continuity of application. Using the definition of application as follows:

Ap: [D → D′]×D → D

′ where Ap(f,x) = f(x).

Ap is continuous with respect to the Scott topology on the product ( [D → D′]×D → D

′ ) :

Proof: λx.f(x) = f is continuous. Let h = λ f.f(x). For directed F ⊆ [D → D′]

h(sup(F)) = sup(F)(x)

= sup( f(x) | f ∈ F )

= sup( h(f) | f ∈ F )

= sup( h(F) )

By definition of Scott continuity h has been shown continuous. All that is now requiredto prove is that application is continuous when it’s separate arguments are continuous, i.e.[D → D

′] and D → D

′ are continuous, in our case f and h.

Now abstracting our argument to show f : D × D′ → D

′′ with g = λx.f(x, x0) andd = λx

′.f(x0, x

′) as the arguments for D and D′ respectively, then for a directed X ⊆ D

g(sup(X)) = f(sup(X), x′

0))

= f( sup( (x, x′

0 ) | x ∈ X ))

(since f is continuous and (x, x′

0 ) | x ∈ X) is directed):

= sup( f(x, x′

0 ) | x ∈ X )

= sup(g(X))

g is therefore continuous. The same process can be taken to show d is continuous.It has now been shown application is continuous under the Scott topology.

In order to demonstrate the Scott topology is a suitable fit for λ-calculus it is necessary to prove abstraction remainscontinuous over the Scott topology. Once completed it will have been shown that the mathematical foundation ofλ-calculus is a well defined and suitable candidate functional paradigm for the Scott topology.With f ∈ [D ×D

′ → D′′] we define f (x) =λ y ∈ D′ f(x,y)We will show:

(i) f is continuous, meaning f ∈ [D → [D′ → D

′′]

(ii) λ f.f : [D ×D′ → D

′′] → [D → [D

′ → D′′] is continuous.

Proof (i): Let X ⊆ D be directed, then

62 CHAPTER 22. COMPUTABLE TOPOLOGY

f (sup(X)) = λ y.f( sup(X),y )

= λ y. supx∈X ( f(x,y) )

= supx∈X ( λy.f(x,y) )

f

Proof (ii): Defining L = λ f.f then for F ⊆ [D ×D′ → D

′′] directed

L(sup(F)) = λ x λ y. (sup(F))(x,y))

= λ x λ y. supy∈F f(x,y)

= supy∈F λx λy.f(x,y)

= sup(L(F))

It has not been demonstrated how and why the λ-calculus defines the Scott topology.

22.4 Böhm trees and computational topology

Böhm trees, easily represented graphically, express the computational behavior of a lambda term. It is possible topredict the functionality of a given lambda expression from reference to its correlating Böhm tree.[4] Böhm trees canbe seen somewhat analogous to R where the Böhm tree of a given set is similar to the continued fraction of a realnumber, and what is more, the Böhm tree corresponding to a sequence in normal form is finite, similar to the rationalsubset of the Reals.Böhm trees are defined by a mapping of elements within a sequence of numbers with ordering (≤, lh) and a binaryoperator * to a set of symbols. The Böhm tree is then a relation among a set of symbols through a partial mapping ψ.Informally Böhm trees may be conceptualized as follows:

Given: Σ = ⊥ ∪ λ x_1 · · · x_n . y | n ∈ N, x1...xn y are variables and denoting BT(M) as theBöhm tree for a lambda term M we then have:

⊥

BT(M) = λ.y/ \BT( BT( ) ; if M is solvableMore formally:Σ is defined as a set of symbols. The Böhm tree of a λ term M, denoted BT(M), is the Σ labelled tree defined asfollows:

If M is unsolvable:

BT (M)(⟨ ⟩) =⊥,

BT(M)( ⟨k⟩ ∗ α ) is unsolvable ∀k, α

22.5. ALGEBRAIC STRUCTURE OF COMPUTATION 63

If M is solvable, where M = λ x_1 · · ·xn.yM0 · · ·Mm−1 :

BT(M)(< >) = λ x_1 · · ·xn.y

BT(M)( ⟨k⟩ ∗ α ) = BT(M_k)( α ) ∀α and k < m

= undefined ∀α and k ≥ m

We may now move on to show that Böhm trees act as suitable mappings from the tree topology to the scott topology.Allowing one to see computational constructs, be it within the Scott or tree topology, as Böhm tree formations.

22.4.1 Böhm tree and tree topology

It is found that Böhm tree's allow for a continuous mapping from the tree topology to the Scott topology. Morespecifically:We begin with the cpo B = (B, ⊆ ) on the Scott topology, with ordering of Böhm tree’s denoted M ⊆ N, meaning M,N are trees and M results from N. The tree topology on the set Γ is the smallest set allowing for a continuous map

BT: Γ → B.

An equivalent definition would be to say the open sets of Γ are the image of the inverse Böhm tree BT−1 (O) whereO is Scott open in B.The applicability of the Bömh trees and the tree topology has many interesting consequences to λ-terms expressedtopologically:

Normal forms are found to exist as isolated points.

Unsolvable λ-terms are compactification points.

Application and abstraction, similar to the Scott topology, are continuous on the tree topology.

22.5 Algebraic structure of computation

New methods of interpretation of the λ-calculus are not only interesting in themselves but allow new modes of thoughtconcerning the behaviors of computer science. The binary operator within the λ-algebra A is application. Applicationis denoted · and is said to give structure A = (X, ·) . A combinatory algebra allows for the application operator andacts as a useful starting point but remains insufficient for the λ-calculus in being unable to express abstraction. The λalgebra becomes a combinatory algebra M combined with a syntactic operator λ* that transforms a term B(x,y), withconstants in M, into C( y ) ≡ λ* x.B(x, y ). It is also possible to define an extension model to circumvent the need ofthe λ* operator by allowing ∀ x (fx =gx) → f =g . The construction of the λ-algebra through the introduction of anabstraction operator proceeds as follows:We must construct an algebra which allows for solutions to equations such as axy = xyy such that a = λ xy.xyy thereis need for the combinatory algebra. Relevant attributes of the combinatory algebra are:Within combinatory algebra there exists applicative structures. An applicative structure W is a combinatory algebraprovided:

· W is non-trival, meaning W has cardinality > 1· W exhibits combinatory completeness (see completeness of the S-K basis). More specifi-cally: for every term A ∈ the set of terms of W, and x1, ..., xn with the free variables of Awithin x1, ..., xn then:

∃f∀x1 · · · xn where fx1 · · · xn = A

64 CHAPTER 22. COMPUTABLE TOPOLOGY

The combiniatory algebra is:

·

···Combinatory algebras remain unable to act as the algebraic structure for λ-calculus, the lack of recursion being amajor disadvantage. However the existence of an applicative term A(x, y ) provides a good starting point to build aλ-calculus algebra. What is needed is the introduction of a lambda term, i.e. include λx.A(x, y ).We begin by exploiting the fact that within a combinatory algebra M, with A(x, y ) within the set of terms, then:

∀y ∃b s.t. bx = A(x, y ).

We then require b have a dependence on y resulting in:

∀x B( y )x = A(x, y ).

B( y ) becomes equivalent to a λ term, and is therefore suitably defined as follows: B( y) ≡ λ*.A pre-λ-algebra (pλA) can now be defined.

pλA is an applicative structure W = (X, · ) such that for each term A within the set of termswithin W and for every x there is a term λ*x.A ∈ T(W) (T(W) ≡ the terms of W) where(the set of free variables of λ*x.A) = (the set of free variables of A) - x. W must alsodemonstrate:(β) (λ*x.A)x = Aα1 λ*x.A ≡ λ*x.A[x:=y] provided y is not a free variable of Aα2 (λ*x.A)[y:=z] ≡ λ*x.A[x:=y] provided y,z ≠ x and z is not a free variable of A

Before defining the full λ-algebra we must introduce the following definition for the set of λ-terms within W denotedΓ(W ) with the following requirements:

a ∈ W ⇒ ca ∈ Γ(W )

x ∈ Γ(W ) for x ∈ ( v0, v1, ... )M,N ∈ Γ(W ) ⇒ (MN) ∈ Γ(W )

M ∈ Γ(W ) ⇒ (λx.M) ∈ Γ(W )

A mapping from the terms within Γ(W ) to all λ terms within W, denoted * : Γ(W ) → T(W ) , can then be designedas follows:

v∗i = wi, c∗a = ca

(MN)* = M* N*(λx.M)* = λ* x*.M*

We now define λ(M) to denote the extension after evaluating the terms within Γ(W ) .

λx.(λy.yx) ca = λx. ca x in λ(W).

Finally we obtain the full λ-algebra through the following definition:

(1) A λ-algebra is a pλA W such that for M,N ∈ Γ (W):λ(W) ⊢ M = N ⇒ W ⊨ M = N.

Though arduous, the foundation has been set for a proper algebraic framework for which the λ-calculus, and thereforecomputation, may be investigated in a group theoretic manner.

22.6. REFERENCES 65

22.6 References[1] Church 1934:90 footnote in Davis 1952

[2] Turing 1936–7 in Davis 1952:149

[3] Barendregt, H.P., The Lambda Calculus Syntax and Semantics. North-Holland Publishing Company. 1981

[4] Barendregt, H.P., The Lambda Calculus Syntax and Semantics. North-Holland Publishing Company. 1981.

[5] D. S. Scott. Models for the λ-calculus. Informally distributed, 1969. Notes, December 1969, Oxford Univ.

[6] Gordon, M.J.C., The Denotational Description of Programming Languages. Springer Verlag, Berlin. 1979.

[7] Scott, D. S. and Strachey, C. Toward a Mathematical Semantics for Computer Languages, Proc. Symp. on Computers andAutomata, Polytechnic Institute of Brooklyn, 21, pp. 19 46. 1971.

[8] G. Berry, Sequential algorithms on concrete data structures, Theoretical Computer Science 20, 265 321 (1982).

[9] D. S. Scott. “Continuous Lattices.” Oxford University Computing Laboratory August, 1971.

Chapter 23

Computation tree

A computation tree is a representation for the computation steps of a non-deterministic Turing machine on a spec-ified input.[1] A computation tree is a rooted tree of nodes and edges. Each node in the tree represents a singlecomputational state, while each edge represents a transition to the next possible computation. The number of nodesof the tree is the size of the tree and the length of the path from the root to a given node is the depth of the node.The largest depth of an output node is the depth of the tree. The output nodes of the tree are called leaves.In a computation tree for a decision problem, each output node is labeled Yes or No. If a tree, T, with an input spaceX, if x ∈ X and the path for x ends in node labeled yes, then the input x is accepted. Else it is rejected.[2]

The depth of the computation tree for a given input is the computation time for the Turing machine on that input.[1]

Computation trees have also been used to study the computational complexity of problems in computational geometryand real number calculations.[3][4]

23.1 References[1] Griffor, E. R. (1999), Handbook of Computability Theory, Studies in Logic and the Foundations of Mathematics 140,

Elsevier, p. 595, ISBN 9780080533049.

[2] Moret, Bernard M. E. (1998), The Theory of Computation, Addison-Wesley, p. 338, ISBN 9780201258288.

[3] Ben-Or, M. (1983), “Lower bounds for algebraic computation trees”, Proc. 15th Annu. Symp. Theory of Computing, pp.80–86, doi:10.1145/800061.808735.

[4] Grigoriev, Dima; Vorobjov, Nicolai (1996), “Complexity lower bounds for computation trees with elementary transcen-dental function gates”, Theor. Comput. Sci. 157: 185–214, doi:10.1016/0304-3975(95)00159-X.

66

Chapter 24

Computational complexity ofmathematical operations

The following tables list the running time of various algorithms for common mathematical operations.Here, complexity refers to the time complexity of performing computations on a multitape Turing machine.[1] Seebig O notation for an explanation of the notation used.Note: Due to the variety of multiplication algorithms, M(n) below stands in for the complexity of the chosen multi-plication algorithm.

24.1 Arithmetic functions

24.2 Algebraic functions

24.3 Special functions

Many of the methods in this section are given in Borwein & Borwein.[5]

24.3.1 Elementary functions

The elementary functions are constructed by composing arithmetic operations, the exponential function (exp), thenatural logarithm (log), trigonometric functions (sin, cos), and their inverses. The complexity of an elementaryfunction is equivalent to that of its inverse, since all elementary functions are analytic and hence invertible by meansof Newton’s method. In particular, if either exp or log in the complex domain can be computed with some complexity,then that complexity is attainable for all other elementary functions.Below, the size n refers to the number of digits of precision at which the function is to be evaluated.It is not known whether O(M(n) log n) is the optimal complexity for elementary functions. The best known lowerbound is the trivial bound Ω(M(n)).

24.3.2 Non-elementary functions

24.3.3 Mathematical constants

This table gives the complexity of computing approximations to the given constants to n correct digits.

67

68 CHAPTER 24. COMPUTATIONAL COMPLEXITY OF MATHEMATICAL OPERATIONS

24.4 Number theory

Algorithms for number theoretical calculations are studied in computational number theory.

24.5 Matrix algebra

The following complexity figures assume that arithmetic with individual elements has complexity O(1), as is the casewith fixed-precision floating-point arithmetic.In 2005, Henry Cohn, Robert Kleinberg, Balázs Szegedy and Chris Umans showed that either of two different con-jectures would imply that the exponent of matrix multiplication is 2.[19]

^* Because of the possibility of blockwise inverting a matrix, where an inversion of an n×n matrix requires inversionof two half-sized matrices and six multiplications between two half-sized matrices, and since matrix multiplication hasa lower bound of Ω(n2 log n) operations,[20] it can be shown that a divide and conquer algorithm that uses blockwiseinversion to invert a matrix runs with the same time complexity as the matrix multiplication algorithm that is usedinternally.[21]

24.6 References[1] A. Schönhage, A.F.W. Grotefeld, E. Vetter: Fast Algorithms—AMultitape TuringMachine Implementation, BI Wissenschafts-

Verlag, Mannheim, 1994

[2] D. Knuth. The Art of Computer Programming, Volume 2. Third Edition, Addison-Wesley 1997.

[3] Martin Fürer. Faster Integer Multiplication. Proceedings of the 39th Annual ACM Symposium on Theory of Computing,San Diego, California, USA, June 11–13, 2007, pp. 55–67.

[4] http://planetmath.org/fasteuclideanalgorithm

[5] J. Borwein & P. Borwein. Pi and the AGM: A Study in Analytic Number Theory and Computational Complexity. John Wiley1987.

[6] David and Gregory Chudnovsky. Approximations and complex multiplication according to Ramanujan. Ramanujan re-visited, Academic Press, 1988, pp 375–472.

[7] Richard P. Brent,Multiple-precision zero-finding methods and the complexity of elementary function evaluation, in: AnalyticComputational Complexity (J. F. Traub, ed.), Academic Press, New York, 1975, 151–176.

[8] J. Sorenson. (1994). “Two Fast GCD Algorithms”. Journal of Algorithms 16 (1): 110–144. doi:10.1006/jagm.1994.1006.

[9] R. Crandall & C. Pomerance. Prime Numbers – A Computational Perspective. Second Edition, Springer 2005.

[10] Möller N (2008). “On Schönhage’s algorithm and subquadratic integer gcd computation” (PDF). Mathematics of Compu-tation 77 (261): 589–607. doi:10.1090/S0025-5718-07-02017-0.

[11] Bernstein D J. “Faster Algorithms to Find Non-squares Modulo Worst-case Integers”.

[12] Richard P. Brent; Paul Zimmermann (2010). “An O(M(n) log n) algorithm for the Jacobi symbol”. arXiv:1004.2091.

[13] P. Borwein. “On the complexity of calculating factorials”. Journal of Algorithms 6, 376-380 (1985)

[14] Davie, A.M.; Stothers, A.J. (2013), “Improved bound for complexity of matrix multiplication”, Proceedings of the RoyalSociety of Edinburgh 143A: 351–370, doi:10.1017/S0308210511001648

[15] Vassilevska Williams, Virginia (2011), Breaking the Coppersmith-Winograd barrier (PDF)

[16] Le Gall, François (2014), “Powers of tensors and fast matrix multiplication”, Proceedings of the 39th International Sympo-sium on Symbolic and Algebraic Computation (ISSAC 2014), arXiv:1401.7714

[17] Aho, Alfred V.; Hopcroft, John E.; Ullman, Jeffrey D. (1974), The Design and Analysis of Computer Algorithms, Addison-Wesley, Theorem 6.6, p. 241

[18] J. B. Fraleigh and R. A. Beauregard, “Linear Algebra,” Addison-Wesley Publishing Company, 1987, p 95.

24.7. FURTHER READING 69

[19] Henry Cohn, Robert Kleinberg, Balazs Szegedy, and Chris Umans. Group-theoretic Algorithms for Matrix Multiplication.arXiv:math.GR/0511460. Proceedings of the 46th Annual Symposium on Foundations of Computer Science, 23–25 October2005, Pittsburgh, PA, IEEE Computer Society, pp. 379–388.

[20] Ran Raz. On the complexity of matrix product. In Proceedings of the thirty-fourth annual ACM symposium on Theoryof computing. ACM Press, 2002. doi:10.1145/509907.509932.

[21] T.H. Cormen, C.E. Leiserson, R.L. Rivest, C. Stein, Introduction to Algorithms, 3rd ed., MIT Press, Cambridge, MA, 2009,§28.2.

24.7 Further reading• Brent, Richard P.; Zimmermann, Paul (2010). Modern Computer Arithmetic. Cambridge University Press.

ISBN 9780521194693.

• Knuth, Donald Ervin (1997). The Art of Computer Programming. Volume 2: Seminumerical Algorithms (3rded.). Addison-Wesley. ISBN 0201896842.

Chapter 25

Computational complexity theory

Computational complexity theory is a branch of the theory of computation in theoretical computer science andmathematics that focuses on classifying computational problems according to their inherent difficulty, and relatingthose classes to each other. A computational problem is understood to be a task that is in principle amenable to beingsolved by a computer, which is equivalent to stating that the problem may be solved by mechanical application ofmathematical steps, such as an algorithm.A problem is regarded as inherently difficult if its solution requires significant resources, whatever the algorithm used.The theory formalizes this intuition, by introducing mathematical models of computation to study these problems andquantifying the amount of resources needed to solve them, such as time and storage. Other complexity measures arealso used, such as the amount of communication (used in communication complexity), the number of gates in acircuit (used in circuit complexity) and the number of processors (used in parallel computing). One of the roles ofcomputational complexity theory is to determine the practical limits on what computers can and cannot do.Closely related fields in theoretical computer science are analysis of algorithms and computability theory. A keydistinction between analysis of algorithms and computational complexity theory is that the former is devoted toanalyzing the amount of resources needed by a particular algorithm to solve a problem, whereas the latter asks amore general question about all possible algorithms that could be used to solve the same problem. More precisely,it tries to classify problems that can or cannot be solved with appropriately restricted resources. In turn, imposingrestrictions on the available resources is what distinguishes computational complexity from computability theory: thelatter theory asks what kind of problems can, in principle, be solved algorithmically.

25.1 Computational problems

25.1.1 Problem instances

A computational problem can be viewed as an infinite collection of instances together with a solution for every instance.The input string for a computational problem is referred to as a problem instance, and should not be confused with theproblem itself. In computational complexity theory, a problem refers to the abstract question to be solved. In contrast,an instance of this problem is a rather concrete utterance, which can serve as the input for a decision problem. Forexample, consider the problem of primality testing. The instance is a number (e.g. 15) and the solution is “yes” ifthe number is prime and “no” otherwise (in this case “no”). Stated another way, the instance is a particular input tothe problem, and the solution is the output corresponding to the given input.To further highlight the difference between a problem and an instance, consider the following instance of the decisionversion of the traveling salesman problem: Is there a route of at most 2000 kilometres passing through all of Germany’s15 largest cities? The quantitative answer to this particular problem instance is of little use for solving other instancesof the problem, such as asking for a round trip through all sites in Milan whose total length is at most 10 km. Forthis reason, complexity theory addresses computational problems and not particular problem instances.

70

25.1. COMPUTATIONAL PROBLEMS 71

A traveling salesman tour through Germany’s 15 largest cities.

25.1.2 Representing problem instances

When considering computational problems, a problem instance is a string over an alphabet. Usually, the alphabet istaken to be the binary alphabet (i.e., the set 0,1), and thus the strings are bitstrings. As in a real-world computer,mathematical objects other than bitstrings must be suitably encoded. For example, integers can be represented inbinary notation, and graphs can be encoded directly via their adjacency matrices, or by encoding their adjacency listsin binary.Even though some proofs of complexity-theoretic theorems regularly assume some concrete choice of input encoding,one tries to keep the discussion abstract enough to be independent of the choice of encoding. This can be achievedby ensuring that different representations can be transformed into each other efficiently.

25.1.3 Decision problems as formal languages

Decision problems are one of the central objects of study in computational complexity theory. A decision problemis a special type of computational problem whose answer is either yes or no, or alternately either 1 or 0. A decision

72 CHAPTER 25. COMPUTATIONAL COMPLEXITY THEORY

A decision problem has only two possible outputs, yes or no (or alternately 1 or 0) on any input.

problem can be viewed as a formal language, where the members of the language are instances whose output is yes,and the non-members are those instances whose output is no. The objective is to decide, with the aid of an algorithm,whether a given input string is a member of the formal language under consideration. If the algorithm deciding thisproblem returns the answer yes, the algorithm is said to accept the input string, otherwise it is said to reject the input.An example of a decision problem is the following. The input is an arbitrary graph. The problem consists in decidingwhether the given graph is connected, or not. The formal language associated with this decision problem is then theset of all connected graphs—of course, to obtain a precise definition of this language, one has to decide how graphsare encoded as binary strings.

25.2. MACHINE MODELS AND COMPLEXITY MEASURES 73

25.1.4 Function problems

A function problem is a computational problem where a single output (of a total function) is expected for every input,but the output is more complex than that of a decision problem, that is, it isn't just yes or no. Notable examplesinclude the traveling salesman problem and the integer factorization problem.It is tempting to think that the notion of function problems is much richer than the notion of decision problems.However, this is not really the case, since function problems can be recast as decision problems. For example, themultiplication of two integers can be expressed as the set of triples (a, b, c) such that the relation a × b = c holds.Deciding whether a given triple is a member of this set corresponds to solving the problem of multiplying two numbers.

25.1.5 Measuring the size of an instance

To measure the difficulty of solving a computational problem, one may wish to see how much time the best algorithmrequires to solve the problem. However, the running time may, in general, depend on the instance. In particular,larger instances will require more time to solve. Thus the time required to solve a problem (or the space required,or any measure of complexity) is calculated as a function of the size of the instance. This is usually taken to be thesize of the input in bits. Complexity theory is interested in how algorithms scale with an increase in the input size.For instance, in the problem of finding whether a graph is connected, how much more time does it take to solve aproblem for a graph with 2n vertices compared to the time taken for a graph with n vertices?If the input size is n, the time taken can be expressed as a function of n. Since the time taken on different inputs ofthe same size can be different, the worst-case time complexity T(n) is defined to be the maximum time taken over allinputs of size n. If T(n) is a polynomial in n, then the algorithm is said to be a polynomial time algorithm. Cobham’sthesis says that a problem can be solved with a feasible amount of resources if it admits a polynomial time algorithm.

25.2 Machine models and complexity measures

25.2.1 Turing machine

An artistic representation of a Turing machine

Main article: Turing machine

A Turing machine is a mathematical model of a general computing machine. It is a theoretical device that manipulates

74 CHAPTER 25. COMPUTATIONAL COMPLEXITY THEORY

symbols contained on a strip of tape. Turing machines are not intended as a practical computing technology, butrather as a thought experiment representing a computing machine—anything from an advanced supercomputer to amathematician with a pencil and paper. It is believed that if a problem can be solved by an algorithm, there exists aTuring machine that solves the problem. Indeed, this is the statement of the Church–Turing thesis. Furthermore, itis known that everything that can be computed on other models of computation known to us today, such as a RAMmachine, Conway’s Game of Life, cellular automata or any programming language can be computed on a Turingmachine. Since Turing machines are easy to analyze mathematically, and are believed to be as powerful as any othermodel of computation, the Turing machine is the most commonly used model in complexity theory.Many types of Turing machines are used to define complexity classes, such as deterministic Turing machines,probabilistic Turing machines, non-deterministic Turing machines, quantum Turing machines, symmetric Turingmachines and alternating Turing machines. They are all equally powerful in principle, but when resources (such astime or space) are bounded, some of these may be more powerful than others.A deterministic Turing machine is the most basic Turing machine, which uses a fixed set of rules to determine itsfuture actions. A probabilistic Turing machine is a deterministic Turing machine with an extra supply of random bits.The ability to make probabilistic decisions often helps algorithms solve problems more efficiently. Algorithms thatuse random bits are called randomized algorithms. A non-deterministic Turing machine is a deterministic Turingmachine with an added feature of non-determinism, which allows a Turing machine to have multiple possible futureactions from a given state. One way to view non-determinism is that the Turing machine branches into many possiblecomputational paths at each step, and if it solves the problem in any of these branches, it is said to have solved theproblem. Clearly, this model is not meant to be a physically realizable model, it is just a theoretically interestingabstract machine that gives rise to particularly interesting complexity classes. For examples, see non-deterministicalgorithm.

25.2.2 Other machine models

Many machine models different from the standard multi-tape Turing machines have been proposed in the literature,for example random access machines. Perhaps surprisingly, each of these models can be converted to another withoutproviding any extra computational power. The time and memory consumption of these alternate models may vary.[1]

What all these models have in common is that the machines operate deterministically.However, some computational problems are easier to analyze in terms of more unusual resources. For example, anon-deterministic Turing machine is a computational model that is allowed to branch out to check many differentpossibilities at once. The non-deterministic Turing machine has very little to do with how we physically want tocompute algorithms, but its branching exactly captures many of the mathematical models we want to analyze, so thatnon-deterministic time is a very important resource in analyzing computational problems.

25.2.3 Complexity measures

For a precise definition of what it means to solve a problem using a given amount of time and space, a computationalmodel such as the deterministic Turing machine is used. The time required by a deterministic Turing machine M oninput x is the total number of state transitions, or steps, the machine makes before it halts and outputs the answer(“yes” or “no”). A Turing machine M is said to operate within time f(n), if the time required by M on each input oflength n is at most f(n). A decision problem A can be solved in time f(n) if there exists a Turing machine operatingin time f(n) that solves the problem. Since complexity theory is interested in classifying problems based on theirdifficulty, one defines sets of problems based on some criteria. For instance, the set of problems solvable within timef(n) on a deterministic Turing machine is then denoted by DTIME(f(n)).Analogous definitions can be made for space requirements. Although time and space are the most well-known com-plexity resources, any complexity measure can be viewed as a computational resource. Complexity measures arevery generally defined by the Blum complexity axioms. Other complexity measures used in complexity theory in-clude communication complexity, circuit complexity, and decision tree complexity.The complexity of an algorithm is often expressed using big O notation.

25.2. MACHINE MODELS AND COMPLEXITY MEASURES 75

Visualization of the quicksort algorithm that has average case performance Θ(n logn) .

25.2.4 Best, worst and average case complexity

The best, worst and average case complexity refer to three different ways of measuring the time complexity (or anyother complexity measure) of different inputs of the same size. Since some inputs of size n may be faster to solvethan others, we define the following complexities:

• Best-case complexity: This is the complexity of solving the problem for the best input of size n.

• Worst-case complexity: This is the complexity of solving the problem for the worst input of size n.

• Average-case complexity: This is the complexity of solving the problem on an average. This complexity isonly defined with respect to a probability distribution over the inputs. For instance, if all inputs of the samesize are assumed to be equally likely to appear, the average case complexity can be defined with respect to theuniform distribution over all inputs of size n.

For example, consider the deterministic sorting algorithm quicksort. This solves the problem of sorting a list ofintegers that is given as the input. The worst-case is when the input is sorted or sorted in reverse order, and thealgorithm takes time O(n2) for this case. If we assume that all possible permutations of the input list are equallylikely, the average time taken for sorting is O(n log n). The best case occurs when each pivoting divides the list inhalf, also needing O(n log n) time.

25.2.5 Upper and lower bounds on the complexity of problems

To classify the computation time (or similar resources, such as space consumption), one is interested in provingupper and lower bounds on the minimum amount of time required by the most efficient algorithm solving a givenproblem. The complexity of an algorithm is usually taken to be its worst-case complexity, unless specified otherwise.Analyzing a particular algorithm falls under the field of analysis of algorithms. To show an upper bound T(n) on

76 CHAPTER 25. COMPUTATIONAL COMPLEXITY THEORY

the time complexity of a problem, one needs to show only that there is a particular algorithm with running time atmost T(n). However, proving lower bounds is much more difficult, since lower bounds make a statement about allpossible algorithms that solve a given problem. The phrase “all possible algorithms” includes not just the algorithmsknown today, but any algorithm that might be discovered in the future. To show a lower bound of T(n) for a problemrequires showing that no algorithm can have time complexity lower than T(n).Upper and lower bounds are usually stated using the big O notation, which hides constant factors and smaller terms.This makes the bounds independent of the specific details of the computational model used. For instance, if T(n) =7n2 + 15n + 40, in big O notation one would write T(n) = O(n2).

25.3 Complexity classes

Main article: Complexity class

25.3.1 Defining complexity classes

A complexity class is a set of problems of related complexity. Simpler complexity classes are defined by the followingfactors:

• The type of computational problem: The most commonly used problems are decision problems. However,complexity classes can be defined based on function problems, counting problems, optimization problems,promise problems, etc.

• The model of computation: The most common model of computation is the deterministic Turing machine, butmany complexity classes are based on non-deterministic Turing machines, Boolean circuits, quantum Turingmachines, monotone circuits, etc.

• The resource (or resources) that are being bounded and the bounds: These two properties are usually statedtogether, such as “polynomial time”, “logarithmic space”, “constant depth”, etc.

Of course, some complexity classes have complicated definitions that do not fit into this framework. Thus, a typicalcomplexity class has a definition like the following:

The set of decision problems solvable by a deterministic Turing machine within time f(n). (This com-plexity class is known as DTIME(f(n)).)

But bounding the computation time above by some concrete function f(n) often yields complexity classes that dependon the chosen machine model. For instance, the language xx | x is any binary string can be solved in linear time ona multi-tape Turing machine, but necessarily requires quadratic time in the model of single-tape Turing machines.If we allow polynomial variations in running time, Cobham-Edmonds thesis states that “the time complexities in anytwo reasonable and general models of computation are polynomially related” (Goldreich 2008, Chapter 1.2). Thisforms the basis for the complexity class P, which is the set of decision problems solvable by a deterministic Turingmachine within polynomial time. The corresponding set of function problems is FP.

25.3.2 Important complexity classes

Many important complexity classes can be defined by bounding the time or space used by the algorithm. Someimportant complexity classes of decision problems defined in this manner are the following:The logarithmic-space classes (necessarily) do not take into account the space needed to represent the problem.It turns out that PSPACE = NPSPACE and EXPSPACE = NEXPSPACE by Savitch’s theorem.Other important complexity classes include BPP, ZPP and RP, which are defined using probabilistic Turing machines;AC and NC, which are defined using Boolean circuits and BQP and QMA, which are defined using quantum Turingmachines. #P is an important complexity class of counting problems (not decision problems). Classes like IP andAM are defined using Interactive proof systems. ALL is the class of all decision problems.

25.3. COMPLEXITY CLASSES 77

EXPSPACE

EXPTIME

NP

P

PSPACE

=?

=?

=?

=?

NL=?

A representation of the relation among complexity classes

25.3.3 Hierarchy theorems

Main articles: time hierarchy theorem and space hierarchy theorem

For the complexity classes defined in this way, it is desirable to prove that relaxing the requirements on (say) compu-tation time indeed defines a bigger set of problems. In particular, although DTIME(n) is contained in DTIME(n2), itwould be interesting to know if the inclusion is strict. For time and space requirements, the answer to such questionsis given by the time and space hierarchy theorems respectively. They are called hierarchy theorems because theyinduce a proper hierarchy on the classes defined by constraining the respective resources. Thus there are pairs ofcomplexity classes such that one is properly included in the other. Having deduced such proper set inclusions, wecan proceed to make quantitative statements about how much more additional time or space is needed in order toincrease the number of problems that can be solved.More precisely, the time hierarchy theorem states that

DTIME(f(n)

)⊊ DTIME

(f(n) · log2(f(n))

)The space hierarchy theorem states that

DSPACE(f(n)

)⊊ DSPACE

(f(n) · log(f(n))

)

78 CHAPTER 25. COMPUTATIONAL COMPLEXITY THEORY

The time and space hierarchy theorems form the basis for most separation results of complexity classes. For instance,the time hierarchy theorem tells us that P is strictly contained in EXPTIME, and the space hierarchy theorem tells usthat L is strictly contained in PSPACE.

25.3.4 Reduction

Main article: Reduction (complexity)

Many complexity classes are defined using the concept of a reduction. A reduction is a transformation of one probleminto another problem. It captures the informal notion of a problem being at least as difficult as another problem. Forinstance, if a problemX can be solved using an algorithm for Y,X is no more difficult than Y, and we say thatX reducesto Y. There are many different types of reductions, based on the method of reduction, such as Cook reductions, Karpreductions and Levin reductions, and the bound on the complexity of reductions, such as polynomial-time reductionsor log-space reductions.The most commonly used reduction is a polynomial-time reduction. This means that the reduction process takespolynomial time. For example, the problem of squaring an integer can be reduced to the problem of multiplying twointegers. This means an algorithm for multiplying two integers can be used to square an integer. Indeed, this can bedone by giving the same input to both inputs of the multiplication algorithm. Thus we see that squaring is not moredifficult than multiplication, since squaring can be reduced to multiplication.This motivates the concept of a problem being hard for a complexity class. A problem X is hard for a class ofproblems C if every problem in C can be reduced to X. Thus no problem in C is harder than X, since an algorithmfor X allows us to solve any problem in C. Of course, the notion of hard problems depends on the type of reductionbeing used. For complexity classes larger than P, polynomial-time reductions are commonly used. In particular, theset of problems that are hard for NP is the set of NP-hard problems.If a problem X is in C and hard for C, then X is said to be complete for C. This means that X is the hardest problem inC. (Since many problems could be equally hard, one might say that X is one of the hardest problems in C.) Thus theclass of NP-complete problems contains the most difficult problems in NP, in the sense that they are the ones mostlikely not to be in P. Because the problem P = NP is not solved, being able to reduce a known NP-complete problem,Π2, to another problem, Π1, would indicate that there is no known polynomial-time solution for Π1. This is becausea polynomial-time solution to Π1 would yield a polynomial-time solution to Π2. Similarly, because all NP problemscan be reduced to the set, finding an NP-complete problem that can be solved in polynomial time would mean that P= NP.[2]

25.4 Important open problems

25.4.1 P versus NP problem

Main article: P versus NP problem

The complexity class P is often seen as a mathematical abstraction modeling those computational tasks that admitan efficient algorithm. This hypothesis is called the Cobham–Edmonds thesis. The complexity class NP, on theother hand, contains many problems that people would like to solve efficiently, but for which no efficient algorithmis known, such as the Boolean satisfiability problem, the Hamiltonian path problem and the vertex cover problem.Since deterministic Turing machines are special non-deterministic Turing machines, it is easily observed that eachproblem in P is also member of the class NP.The question of whether P equals NP is one of the most important open questions in theoretical computer sciencebecause of the wide implications of a solution.[2] If the answer is yes, many important problems can be shown tohave more efficient solutions. These include various types of integer programming problems in operations research,many problems in logistics, protein structure prediction in biology,[4] and the ability to find formal proofs of puremathematics theorems.[5] The P versus NP problem is one of the Millennium Prize Problems proposed by the ClayMathematics Institute. There is a US$1,000,000 prize for resolving the problem.[6]

25.4. IMPORTANT OPEN PROBLEMS 79

NP Problems

P Problems

NP Complete

Diagram of complexity classes provided that P ≠ NP. The existence of problems in NP outside both P and NP-complete in this casewas established by Ladner.[3]

25.4.2 Problems in NP not known to be in P or NP-complete

It was shown by Ladner that if P ≠NP then there exist problems inNP that are neither in P norNP-complete.[3] Suchproblems are called NP-intermediate problems. The graph isomorphism problem, the discrete logarithm problem andthe integer factorization problem are examples of problems believed to be NP-intermediate. They are some of thevery few NP problems not known to be in P or to be NP-complete.The graph isomorphism problem is the computational problem of determining whether two finite graphs are isomorphic.An important unsolved problem in complexity theory is whether the graph isomorphism problem is in P, NP-complete, or NP-intermediate. The answer is not known, but it is believed that the problem is at least not NP-complete.[7] If graph isomorphism is NP-complete, the polynomial time hierarchy collapses to its second level.[8]

Since it is widely believed that the polynomial hierarchy does not collapse to any finite level, it is believed that graphisomorphism is not NP-complete. The best algorithm for this problem, due to Laszlo Babai and Eugene Luks hasrun time 2O(√(n log(n))) for graphs with n vertices.The integer factorization problem is the computational problem of determining the prime factorization of a giveninteger. Phrased as a decision problem, it is the problem of deciding whether the input has a factor less than k.No efficient integer factorization algorithm is known, and this fact forms the basis of several modern cryptographicsystems, such as the RSA algorithm. The integer factorization problem is in NP and in co-NP (and even in UP andco-UP[9]). If the problem is NP-complete, the polynomial time hierarchy will collapse to its first level (i.e., NPwill equal co-NP). The best known algorithm for integer factorization is the general number field sieve, which takestime O(e(64/9)1/3(n.log 2)1/3(log (n.log 2))2/3 ) to factor an n-bit integer. However, the best known quantum algorithm for thisproblem, Shor’s algorithm, does run in polynomial time. Unfortunately, this fact doesn't say much about where theproblem lies with respect to non-quantum complexity classes.

25.4.3 Separations between other complexity classes

Many known complexity classes are suspected to be unequal, but this has not been proved. For instance P ⊆ NP ⊆PP ⊆ PSPACE, but it is possible that P = PSPACE. If P is not equal to NP, then P is not equal to PSPACE either.Since there are many known complexity classes between P and PSPACE, such as RP, BPP, PP, BQP, MA, PH,etc., it is possible that all these complexity classes collapse to one class. Proving that any of these classes are unequalwould be a major breakthrough in complexity theory.

80 CHAPTER 25. COMPUTATIONAL COMPLEXITY THEORY

Along the same lines, co-NP is the class containing the complement problems (i.e. problems with the yes/no answersreversed) of NP problems. It is believed[10] that NP is not equal to co-NP; however, it has not yet been proven. Ithas been shown that if these two complexity classes are not equal then P is not equal to NP.Similarly, it is not known if L (the set of all problems that can be solved in logarithmic space) is strictly contained inP or equal to P. Again, there are many complexity classes between the two, such as NL and NC, and it is not knownif they are distinct or equal classes.It is suspected that P and BPP are equal. However, it is currently open if BPP = NEXP.

25.5 Intractability

See also: Combinatorial explosion

Problems that can be solved in theory (e.g., given large but finite time), but which in practice take too long for theirsolutions to be useful, are known as intractable problems.[11] In complexity theory, problems that lack polynomial-time solutions are considered to be intractable for more than the smallest inputs. In fact, the Cobham–Edmonds thesisstates that only those problems that can be solved in polynomial time can be feasibly computed on some computationaldevice. Problems that are known to be intractable in this sense include those that are EXPTIME-hard. If NP is not thesame as P, then the NP-complete problems are also intractable in this sense. To see why exponential-time algorithmsmight be unusable in practice, consider a program that makes 2n operations before halting. For small n, say 100,and assuming for the sake of example that the computer does 1012 operations each second, the program would runfor about 4 × 1010 years, which is the same order of magnitude as the age of the universe. Even with a much fastercomputer, the program would only be useful for very small instances and in that sense the intractability of a problem issomewhat independent of technological progress. Nevertheless, a polynomial time algorithm is not always practical.If its running time is, say, n15, it is unreasonable to consider it efficient and it is still useless except on small instances.What intractability means in practice is open to debate. Saying that a problem is not in P does not imply that alllarge cases of the problem are hard or even that most of them are. For example, the decision problem in Presburgerarithmetic has been shown not to be in P, yet algorithms have been written that solve the problem in reasonable timesin most cases. Similarly, algorithms can solve the NP-complete knapsack problem over a wide range of sizes inless than quadratic time and SAT solvers routinely handle large instances of the NP-complete Boolean satisfiabilityproblem.

25.6 History

An early example of algorithm complexity analysis is the running time analysis of the Euclidean algorithm done byGabriel Lamé in 1844.Before the actual research explicitly devoted to the complexity of algorithmic problems started off, numerous foun-dations were laid out by various researchers. Most influential among these was the definition of Turing machines byAlan Turing in 1936, which turned out to be a very robust and flexible simplification of a computer.As Fortnow & Homer (2003) point out, the beginning of systematic studies in computational complexity is attributedto the seminal paper “On the Computational Complexity of Algorithms” by Juris Hartmanis and Richard Stearns(1965), which laid out the definitions of time and space complexity and proved the hierarchy theorems. Also, in1965 Edmonds defined a “good” algorithm as one with running time bounded by a polynomial of the input size.[12]

Earlier papers studying problems solvable by Turing machines with specific bounded resources include [13] JohnMyhill's definition of linear bounded automata (Myhill 1960), Raymond Smullyan's study of rudimentary sets (1961),as well as Hisao Yamada's paper[14] on real-time computations (1962). Somewhat earlier, Boris Trakhtenbrot (1956),a pioneer in the field from the USSR, studied another specific complexity measure.[15] As he remembers:

However, [my] initial interest [in automata theory] was increasingly set aside in favor of computa-tional complexity, an exciting fusion of combinatorial methods, inherited from switching theory, withthe conceptual arsenal of the theory of algorithms. These ideas had occurred to me earlier in 1955 whenI coined the term “signalizing function”, which is nowadays commonly known as “complexity measure”.—Boris Trakhtenbrot, From Logic to Theoretical Computer Science – An Update. In: Pillars of Com-

25.7. SEE ALSO 81

puter Science, LNCS 4800, Springer 2008.

In 1967, Manuel Blum developed an axiomatic complexity theory based on his axioms and proved an importantresult, the so-called, speed-up theorem. The field really began to flourish in 1971 when the US researcher StephenCook and, working independently, Leonid Levin in the USSR, proved that there exist practically relevant problemsthat are NP-complete. In 1972, Richard Karp took this idea a leap forward with his landmark paper, “ReducibilityAmong Combinatorial Problems”, in which he showed that 21 diverse combinatorial and graph theoretical problems,each infamous for its computational intractability, are NP-complete.[16]

25.7 See also• Category:Computational problems

• Context of computational complexity

• Descriptive complexity theory

• Game complexity

• List of complexity classes

• List of computability and complexity topics

• List of important publications in theoretical computer science

• List of unsolved problems in computer science

• Parameterized complexity

• Proof complexity

• Quantum complexity theory

• Structural complexity theory

• Transcomputational problem

25.8 References[1] See Arora & Barak 2009, Chapter 1: The computational model and why it doesn't matter

[2] See Sipser 2006, Chapter 7: Time complexity

[3] Ladner, Richard E. (1975), “On the structure of polynomial time reducibility” (PDF), Journal of the ACM (JACM) 22 (1):151–171, doi:10.1145/321864.321877.

[4] Berger, Bonnie A.; Leighton, T (1998), “Protein folding in the hydrophobic-hydrophilic (HP) model is NP-complete”,Journal of Computational Biology 5 (1): 27–40, doi:10.1089/cmb.1998.5.27, PMID 9541869.

[5] Cook, Stephen (April 2000), The P versus NP Problem (PDF), Clay Mathematics Institute, retrieved 2006-10-18.

[6] Jaffe, Arthur M. (2006), “The Millennium Grand Challenge in Mathematics” (PDF), Notices of the AMS 53 (6), retrieved2006-10-18.

[7] Arvind, Vikraman; Kurur, Piyush P. (2006), “Graph isomorphism is in SPP”, Information and Computation 204 (5): 835–852, doi:10.1016/j.ic.2006.02.002.

[8] Schöning, Uwe. “Graph isomorphism is in the low hierarchy”. Proceedings of the 4th Annual Symposium on TheoreticalAspects of Computer Science 1987: 114–124. doi:10.1007/bfb0039599.; also: Journal of Computer and System Sciences37: 312–323. 1988. doi:10.1016/0022-0000(88)90010-4. Missing or empty |title= (help)

[9] Lance Fortnow. Computational Complexity Blog: Complexity Class of the Week: Factoring. September 13, 2002. http://weblog.fortnow.com/2002/09/complexity-class-of-week-factoring.html

82 CHAPTER 25. COMPUTATIONAL COMPLEXITY THEORY

[10] Boaz Barak’s course on Computational Complexity Lecture 2

[11] Hopcroft, J.E., Motwani, R. and Ullman, J.D. (2007) Introduction to Automata Theory, Languages, and Computation,Addison Wesley, Boston/San Francisco/New York (page 368)

[12] Richard M. Karp, “Combinatorics, Complexity, and Randomness”, 1985 Turing Award Lecture

[13] Fortnow & Homer (2003)

[14] Yamada, H. (1962). “Real-Time Computation and Recursive Functions Not Real-Time Computable”. IEEE Transactionson Electronic Computers. EC-11 (6): 753–760. doi:10.1109/TEC.1962.5219459.

[15] Trakhtenbrot, B.A.: Signalizing functions and tabular operators. Uchionnye Zapiski Penzenskogo Pedinstituta (Transac-tions of the Penza Pedagogoical Institute) 4, 75–87 (1956) (in Russian)

[16] Richard M. Karp (1972), “Reducibility Among Combinatorial Problems” (PDF), in R. E. Miller and J. W. Thatcher (edi-tors), Complexity of Computer Computations, New York: Plenum, pp. 85–103

25.8.1 Textbooks

• Arora, Sanjeev; Barak, Boaz (2009), Computational Complexity: A Modern Approach, Cambridge, ISBN 978-0-521-42426-4, Zbl 1193.68112

• Downey, Rod; Fellows, Michael (1999), Parameterized complexity, Berlin, New York: Springer-Verlag

• Du, Ding-Zhu; Ko, Ker-I (2000), Theory of Computational Complexity, John Wiley & Sons, ISBN 978-0-471-34506-0

• Goldreich, Oded (2008), Computational Complexity: A Conceptual Perspective, Cambridge University Press

• van Leeuwen, Jan, ed. (1990), Handbook of theoretical computer science (vol. A): algorithms and complexity,MIT Press, ISBN 978-0-444-88071-0

• Papadimitriou, Christos (1994), Computational Complexity (1st ed.), Addison Wesley, ISBN 0-201-53082-1

• Sipser, Michael (2006), Introduction to the Theory of Computation (2nd ed.), USA: Thomson Course Technol-ogy, ISBN 0-534-95097-3

• Garey, Michael R.; Johnson, David S. (1979), Computers and Intractability: A Guide to the Theory of NP-Completeness, W. H. Freeman, ISBN 0-7167-1045-5

25.8.2 Surveys

• Khalil, Hatem; Ulery, Dana (1976), A Review of Current Studies on Complexity of Algorithms for Partial Differ-ential Equations, ACM '76 Proceedings of the 1976 Annual Conference, p. 197, doi:10.1145/800191.805573

• Cook, Stephen (1983), “An overview of computational complexity”, Commun. ACM (ACM) 26 (6): 400–408,doi:10.1145/358141.358144, ISSN 0001-0782

• Fortnow, Lance; Homer, Steven (2003), “A Short History of Computational Complexity” (PDF), Bulletin ofthe EATCS 80: 95–133

• Mertens, Stephan (2002), “Computational Complexity for Physicists”, Computing in Science and Engg. (Piscat-away, NJ, USA: IEEE Educational Activities Department) 4 (3): 31–47, arXiv:cond-mat/0012185, doi:10.1109/5992.998639,ISSN 1521-9615

25.9 External links• The Complexity Zoo

• Hazewinkel, Michiel, ed. (2001), “Computational complexity classes”, Encyclopedia ofMathematics, Springer,ISBN 978-1-55608-010-4

Chapter 26

Computational resource

Not to be confused with Resource (computer science).

In computational complexity theory, a computational resource is a resource used by some computational models inthe solution of computational problems.The simplest computational resources are computation time, the number of steps necessary to solve a problem, andmemory space, the amount of storage needed while solving the problem, but many more complicated resources havebeen defined.A computational problem is generally defined in terms of its action on any valid input. Examples of problems mightbe “given an integer n, determine whether n is prime”, or “given two numbers x and y, calculate the product x*y".As the inputs get bigger, the amount of computational resources needed to solve a problem will increase. Thus, theresources needed to solve a problem are described in terms of asymptotic analysis, by identifying the resources as afunction of the length or size of the input. Resource usage is often partially quantified using Big O notation.Computational resources are useful because we can study which problems can be computed in a certain amount ofeach computational resource. In this way, we can determine whether algorithms for solving the problem are optimaland we can make statements about an algorithm’s efficiency. The set of all of the computational problems that canbe solved using a certain amount of a certain computational resource is a complexity class, and relationships betweendifferent complexity classes are one of the most important topics in complexity theory.

26.1 Describing generally accessible computing equipment

The term “Computational resource” is commonly used to describe accessible computing equipment and software.See Utility computing.

26.2 Formal quantification of computing capability

There has been some effort to formally quantify computing capability. A bounded Turing machine has been usedto model specific computations using the number of state transitions and alphabet size to quantify the computationaleffort required to solve a particular problem.[1][2]

26.3 References[1] Gregory J., Chaitin (1966). “On the Length of Programs for Computing Finite Binary Sequences” (PDF). Journal of the

ACM (JACM) 13 (4): 547–569. doi:10.1145/321356.321363. Retrieved 2007-09-25.

[2] Sow, Daby; Eleftheriadis, Alexandros (1998). “Representing Information with Computational Resource Bounds” (PDF).Signals, Systems & Computers. Conference Record of the Thirty-Second Asilomar. Volume 1. pp. 452–456. ISBN 0-7803-5148-7. 10.1109/ACSSC.1998.750904. Retrieved 2007-09-25.

83

Chapter 27

Computational topology

Algorithmic topology, or computational topology, is a subfield of topology with an overlap with areas of computerscience, in particular computational geometry and computational complexity theory.A primary concern of algorithmic topology, as its name suggests, is to develop efficient algorithms for solving topo-logical problems using topological methods from computable topology to solve algorithmic problems from otherfields.

27.1 Major algorithms by subject area

27.1.1 Algorithmic 3-manifold theory

A large family of algorithms concerning 3-manifolds revolve around normal surface theory, which is a phrase thatencompasses several techniques to turn problems in 3-manifold theory into integer linear programming problems.

• Rubinstein and Thompson’s 3-sphere recognition algorithm. This is an algorithm that takes as input a triangulated3-manifold and determines whether or not the manifold is homeomorphic to the 3-sphere. It has exponentialrun-time in the number of tetrahedral simplexes in the initial 3-manifold, and also an exponential memoryprofile. Moreover, it is implemented in the software package Regina.[1] Saul Schleimer went on to show theproblem lies in the complexity class NP.[2]

• The connect-sum decomposition of 3-manifolds is also implemented in Regina, has exponential run-time andis based on a similar algorithm to the 3-sphere recognition algorithm.

• Determining that the Seifert-Weber 3-manifold contains no incompressible surface has been algorithmicallyimplemented by Burton, Rubinstein and Tillmann [3] and based on normal surface theory.

• The Manning algorithm is an algorithm to find hyperbolic structures on 3-manifolds whose fundamental grouphave a solution to the word problem.[4]

At present the JSJ decomposition has not been implemented algorithmically in computer software. Neither has thecompression-body decomposition. There are some very popular and successful heuristics, such as SnapPea whichhas much success computing approximate hyperbolic structures on triangulated 3-manifolds. It is known that the fullclassification of 3-manifolds can be done algorithmically.[5]

Conversion algorithms

• SnapPea implements an algorithm to convert a planar knot or link diagram into a cusped triangulation. Thisalgorithm has a roughly linear run-time in the number of crossings in the diagram, and low memory profile. Thealgorithm is similar to the Wirthinger algorithm for constructing presentations of the fundamental group of linkcomplements given by planar diagrams. Similarly, SnapPea can convert surgery presentations of 3-manifoldsinto triangulations of the presented 3-manifold.

84

27.2. SEE ALSO 85

• D.Thurston and F.Costantino have a procedure to construct a triangulated 4-manifold from a triangulated 3-manifold. Similarly, it can be used to construct surgery presentations of triangulated 3-manifolds, althoughthe procedure is not explicitly written as an algorithm in principle it should have polynomial run-time in thenumber of tetrahedra of the given 3-manifold triangulation.[6]

• S. Schleimer has an algorithm which produces a triangulated 3-manifold, given input a word (in Dehn twistgenerators) for the mapping class group of a surface. The 3-manifold is the one that uses the word as theattaching map for a Heegaard splitting of the 3-manifold. The algorithm is based on the concept of a layeredtriangulation.

27.1.2 Algorithmic knot theory

• Determining whether or not a knot is trivial is known to be in the complexity class NP [7]

• The problem of determining the genus of a knot is known to have complexity class PSPACE.

• There are polynomial-time algorithms for the computation of the Alexander polynomial of a knot.[8]

27.1.3 Computational homotopy

• Computational methods for homotopy groups of spheres.

• Computational methods for solving systems of polynomial equations.

• Brown has an algorithm to compute the homotopy groups of spaces that are finite Postnikov complexes,[9]

although it is not widely considered implementable.

27.1.4 Computational homology

Computation of homology groups of cell complexes reduces to bringing the boundary matrices into Smith normalform. Although this is a completely solved problem algorithmically, there are various technical obstacles to efficientcomputation for large complexes. There are two central obstacles. Firstly, the basic Smith form algorithm has cubiccomplexity in the size of the matrix involved since it uses row and column operations which makes it unsuitablefor large cell complexes. Secondly, the intermediate matrices which result from the application of the Smith formalgorithm get filled-in even if one starts and ends with sparse matrices.

• Efficient and probabilistic Smith normal form algorithms, as found in the LinBox library.

• Simple homotopic reductions for pre-processing homology computations, as in the Perseus software package.

• Algorithms to compute persistent homology of filtered complexes.

27.2 See also

• Computable topology (the study of the topological nature of computation)

• Computational geometry

• Digital topology

• Topological data analysis

• Spatial-temporal reasoning

• Experimental mathematics

86 CHAPTER 27. COMPUTATIONAL TOPOLOGY

27.3 References[1] B.~Burton. Introducing Regina, the 3-manifold topology software, Experimental Mathematics 13 (2004), 267–272.

[2] http://www.warwick.ac.uk/~masgar/Maths/np.pdf

[3] B. A. Burton, J. H. Rubinstein and S. Tillmann, The Weber–Seifert dodecahedral space is non-Haken, Transactions of theAmerican Mathematical Society 364 (2012), 911–932, http://arxiv.org/abs/0909.4625

[4] J.Manning, Algorithmic detection and description of hyperbolic structures on 3-manifolds with solvable word problem,Geometry and Topology 6 (2002) 1–26

[5] S.Matveev, Algorithmic topology and the classification of 3-manifolds, Springer-Verlag 2003

[6] F. Costantino, D.Thurston. 3-manifolds efficiently bound 4-manifolds. Journal of Topology 2008 1(3):703–745

[7] Hass, Joel; Lagarias, Jeffrey C.; Pippenger, Nicholas (1999), “The computational complexity of knot and link problems”,Journal of the ACM 46 (2): 185–211, arXiv:math/9807016, doi:10.1145/301970.301971.

[8] "Main_Page", The Knot Atlas.

[9] E H Brown’s “Finite Computability of Postnikov Complexes” annals of Mathematics (2) 65 (1957) pp 1–20

27.4 External links• CompuTop software archive

• Workshop on Application of Topology in Science and Engineering

• Computational Topology at Stanford University

• Computational Homology Software (CHomP) at Rutgers University.

• Computational Homology Software (RedHom) at Jagellonian University.

• The Perseus software project for (persistent) homology.

• The javaPlex Persistent Homology software at Stanford.

• PHAT: persistent homology algorithms toolbox.

27.5 Books• Tomasz Kaczynski, Konstantin Mischaikow, Marian Mrozek (2004). Computational Homology. Springer.

ISBN 0-387-40853-3.

• Afra J. Zomorodian (2005). Topology for Computing. Cambridge. ISBN 0-521-83666-2.

• Computational Topology: An Introduction, Herbert Edelsbrunner, John L. Harer, AMS Bookstore, 2010,ISBN 978-0-8218-4925-5

Chapter 28

Computationally bounded adversary

In information theory, the computationally bounded adversary problem is a different way of looking at the problemof sending data over a noisy channel. In previous models the best that could be done was ensuring correct decodingfor up to d/2 errors, where d was the Hamming distance of the code. The problem with doing it this way is that itdoes not take into consideration the actual amount of computing power available to the adversary. Rather, it onlyconcerns itself with how many bits of a given code word can change and still have the message decode properly. In thecomputationally bounded adversary model the channel – the adversary – is restricted to only being able to perform areasonable amount of computation to decide which bits of the code word need to change. In other words, this modeldoes not need to consider how many errors can possibly be handled, but only how many errors could possibly beintroduced given a reasonable amount of computing power on the part of the adversary. Once the channel has beengiven this restriction it becomes possible to construct codes that are both faster to encode and decode compared toprevious methods that can also handle a large number of errors.

28.1 Comparison to other models

28.1.1 Worst-case model

At first glance, the worst-case model seems intuitively ideal. The guarantee that an algorithm will succeed no matterwhat is, of course, highly alluring. However, it demands too much. A real-life adversary cannot spend an indefiniteamount of time examining a message in order to find the one error pattern which an algorithm would struggle with.As a comparison, consider the Quicksort algorithm. In the worst-case scenario, Quicksort makes O(n2) comparisons;however, such an occurrence is rare. Quicksort almost invariably makes O(n log n) comparisons instead, and evenoutperforms other algorithms which can guarantee O(n log n) behavior. Let us suppose an adversary wishes to forcethe Quicksort algorithm to make O(n2) comparisons. Then he would have to search all of the n! permutations of theinput string and test the algorithm on each until he found the one for which the algorithm runs significantly slower.But since this would take O(n!) time, it is clearly infeasible for an adversary to do this. Similarly, it is unreasonableto assume an adversary for an encoding and decoding system would be able to test every single error pattern in orderto find the most effective one.

28.1.2 Stochastic noise model

The stochastic noise model can be described as a kind of “dumb” noise model. That is to say that it does not havethe adaptability to deal with “intelligent” threats. Even if the attacker is bounded it is still possible that they might beable to overcome the stochastic model with a bit of cleverness. The stochastic model has no real way to fight againstthis sort of attack and as such as unsuited to dealing with the kind of “intelligent” threats that would be preferable tohave defenses against.Therefore, a computationally bounded adversarial model has been proposed as a compromise between the two. [1]

This forces one to consider that messages may be perverted in conscious, even malicious ways, but without forcingan algorithm designer to worry about rare cases which likely will never occur.

87

88 CHAPTER 28. COMPUTATIONALLY BOUNDED ADVERSARY

28.2 Applications

28.2.1 Comparison to stochastic noise channel

Since any computationally bounded adversary could in O(n) time flip a coin for each bit, it is intuitively clear that anyencoding and decoding system which can work against this adversary must also work in the stochastic noise model.The converse is less simple; however, it can be shown that any system which works in the stochastic noise model canalso efficiently encode and decode against a computationally bounded adversary, and only at an additional cost whichis polynomial in n. [1] The following method to accomplish this was designed by Dick Lipton, and is taken from:[1]

Let E() be an encoder for the stochastic noise model and D() be a simple decoder for the same, each of which runsin polynomial time. Furthermore, let both the sender and receiver share some random permutation function π and arandom pattern R .For encoding: 1. Let X = E(M) .2. Let Y = π(X)⊕R .3. Transmit YThen for decoding: 1. Receive Y ′ . Compute Z = π−1(Y ′ ⊕R) .2. Calculate M = D(Z) .Similarly to the Quicksort comparison above, if the channel wants to do something smart, it must first test all thepermutations. However, this is infeasible for a computationally bounded adversary, so the most it can do is make arandom error pattern N . But then:

Z = π−1(Y ′ ⊕R) = π−1(Y ⊕N ⊕R),

since Y ′ = Y ⊕N by definition.

= π−1(Y ⊕R)⊕N ′ , where N ′ = π−1(N),

since any permutation is linear with respect to XOR,

= X ⊕N ′,

as per the definition of Y above.Since π is random, N ′ is just random noise and we can use the simple decoder to decode the received message andget back M .

28.3 Specific applications

By assuming a computationally bounded adversary, it is possibly to design a locally decodable code which is bothefficient and near-optimal, with a negligible error probability. These codes are used in complexity theory for thingslike self-correcting computations, probabilistically checkable proof systems, and worst-case to average-case hardnessreductions in the constructions of pseudo-random generators. They are useful in cryptography as a result of theirconnection with private information retrieval protocols. They are also in a number of database applications likefault-tolerant data storage. [2]

Furthermore, it is possible to construct codes which surpass known bounds for worst-case codes--specifically, uniquedecoding with a 1−R error rate. [3] This can be done by concatenating timestamped digital signatures onto messages.A computationally bounded channel cannot forge a signature; and while it may have valid past signatures, the receivercan use list decoding and select a message only if its signature has the correct timestamp.

28.4 References[1] Lipton (6 May 2009). “Worst Case Complexity”. Gödel’s Lost Letter and P=NP. Retrieved 2013-04-01.

28.4. REFERENCES 89

[2] Ostrovsky, Pandey, Sahai. “Private Locally Decodable Codes”. Retrieved 2013-04-01.

[3] Micali, Peikert, Sudan, A. Wilson. “Optimal Error Correction for Computationally Bounded Noise.”. Retrieved 2013-04-01.

90 CHAPTER 28. COMPUTATIONALLY BOUNDED ADVERSARY

An illustration of the method. The first row gives the initial encoded message; the second, after random permutation and randomR; the third, after the adversary adds N; the fourth, after unpermuting; the fifth, the encoded message with the adversary’s errorremoved.

Chapter 29

Computing the permanent

In mathematics, the computation of the permanent of a matrix is a problem that is known to be more difficultthan the computation of the determinant of a matrix despite the apparent similarity of the definitions.The permanent is defined similarly to the determinant, as a sum of products of sets of matrix entries that lie in distinctrows and columns. However, where the determinant weights each of these products with a ±1 sign based on the parityof the set, the permanent weights them all with a +1 sign.While the determinant can be computed in polynomial time by Gaussian elimination, the permanent cannot. Incomputational complexity theory, a theorem of Valiant states that computing permanents is #P-hard, and even #P-complete for matrices in which all entries are 0 or 1.Valiant (1979) This puts the computation of the permanent in aclass of problems believed to be even more difficult to compute than NP. It is known that computing the permanentis impossible for logspace-uniform ACC0 circuits.(Allender & Gore 1994)The development of both exact and approximate algorithms for computing the permanent of a matrix is an activearea of research.

29.1 Definition and naive algorithm

The permanent of an n-by-n matrix A = (ai,j) is defined as

perm(A) =∑σ∈Sn

n∏i=1

ai,σ(i).

The sum here extends over all elements σ of the symmetric group Sn, i.e. over all permutations of the numbers 1, 2,..., n. This formula differs from the corresponding formula for the determinant only in that, in the determinant, eachproduct is multiplied by the sign of the permutation σ while in this formula each product is unsigned. The formulamay be directly translated into an algorithm that naively expands the formula, summing over all permutations andwithin the sum multiplying out each matrix entry. This requires n! n arithmetic operations.

29.2 Ryser formula

The best known[1] general exact algorithm is due to H. J. Ryser (1963). Ryser’s method is based on an inclusion–exclusion formula that can be given[2] as follows: Let Ak be obtained from A by deleting k columns, let P (Ak) bethe product of the row-sums of Ak , and let Σk be the sum of the values of P (Ak) over all possible Ak . Then

perm(A) =

n−1∑k=0

(−1)kΣk.

It may be rewritten in terms of the matrix entries as follows[3]

91

92 CHAPTER 29. COMPUTING THE PERMANENT

perm(A) = (−1)n∑

S⊆1,...,n

(−1)|S|n∏

i=1

∑j∈S

aij .

Ryser’s formula can be evaluated using O(2nn2) arithmetic operations, or O(2nn) by processing the sets S in Graycode order.

29.3 Balasubramanian-Bax/Franklin-Glynn formula

Another formula that appears to be as fast as Ryser’s (or perhaps even twice as fast) is to be found in the two Ph.D.theses; see (Balasubramanian 1980), (Bax 1998); also (Bax 1996). The methods to find the formula are quite different,being related to the combinatorics of the Muir algebra, and to finite difference theory respectively. Another way,connected with invariant theory is via the polarization identity for a symmetric tensor (Glynn 2010). The formulageneralizes to infinitely many others, as found by all these authors, although it is not clear if they are any faster thanthe basic one. See (Glynn 2013).The simplest known formula of this type (when the characteristic of the field is not two) is

perm(A) =

∑δ

(m∏

k=1

δk

)m∏j=1

m∑i=1

δiaij

/2m−1,

where the outer sum is over all 2m−1 vectors δ = (δ1 = 1, δ2, . . . , δm) ∈ ±1m .

29.4 Special cases

29.4.1 Planar and K₃,₃-free

The number of perfect matchings in a bipartite graph is counted by the permanent of the graph’s biadjacency matrix,and the permanent of any 0-1 matrix can be interpreted in this way as the number of perfect matchings in a graph.For planar graphs (regardless of bipartiteness), the FKT algorithm computes the number of perfect matchings inpolynomial time by changing the signs of a carefully chosen subset of the entries in the Tutte matrix of the graph, sothat the Pfaffian of the resulting skew-symmetric matrix (the square root of its determinant) is the number of perfectmatchings. This technique can be generalized to graphs that contain no subgraph homeomorphic to the completebipartite graph K₃,₃.[4]

George Pólya had asked the question[5] of when it is possible to change the signs of some of the entries of a 01 matrixA so that the determinant of the new matrix is the permanent of A. Not all 01 matrices are “convertible” in thismanner; in fact it is known (Marcus & Minc (1961)) that there is no linear map T such that per T (A) = detA forall n × n matrices A . The characterization of “convertible” matrices was given by Little (1975) who showed thatsuch matrices are precisely those that are the biadjacency matrix of bipartite graphs that have a Pfaffian orientation:an orientation of the edges such that for every even cycle C for which G \C has a perfect matching, there are an oddnumber of edges directed along C (and thus an odd number with the opposite orientation). It was also shown thatthese graphs are exactly those that do not contain a subgraph homeomorphic to K3,3 , as above.

29.4.2 Computation modulo a number

Modulo 2, the permanent is the same as the determinant, as (−1) ≡ 1 (mod 2). It can also be computed modulo 2k

in time O(n4k−3) for k ≥ 2 . However, it is UP-hard to compute the permanent modulo any number that is not apower of 2. Valiant (1979)There are various formulae given by Glynn (2010) for the computation modulo a prime p . Firstly there is one usingsymbolic calculations with partial derivatives.Secondly, for p = 3 there is the following formula (Grigoriy Kogan, 1996) using the determinants of the principalsubmatrices of the matrix:

29.5. APPROXIMATE COMPUTATION 93

perm(A) = (−1)mΣU⊆1,...,m det(AU ). det(AU ),

where AU is the principal submatrix of A induced by the rows and columns of A indexed by U , and U is thecomplement of U in 1, . . . ,m.(The determinant of an empty submatrix is defined to be 1).This formula implies the following identities over fields of Characteristic 3 (Grigoriy Kogan, 1996):for any invertible A

perm(A−1) det(A)2 = perm(A)

for any unitary U , i.e. a square matrix U such that UTU = I ,

perm(U)2 = det(U + V ) det(U)

where V is the matrix whose entries are the cubes of the corresponding entries of U .

29.5 Approximate computation

When the entries of A are nonnegative, the permanent can be computed approximately in probabilistic polynomialtime, up to an error of εM, where M is the value of the permanent and ε > 0 is arbitrary. In other words, there existsa fully polynomial-time randomized approximation scheme (FPRAS) (Jerrum, Vigoda & Sinclair (2001)).The most difficult step in the computation is the construction of an algorithm to sample almost uniformly from theset of all perfect matchings in a given bipartite graph: in other words, a fully polynomial almost uniform sampler(FPAUS). This can be done using a Markov chain Monte Carlo algorithm that uses a Metropolis rule to define andrun a Markov chain whose distribution is close to uniform, and whose mixing time is polynomial.It is possible to approximately count the number of perfect matchings in a graph via the self-reducibility of thepermanent, by using the FPAUS in combination with a well-known reduction from sampling to counting due toJerrum, Valiant & Vazirani (1986). Let M(G) denote the number of perfect matchings in G . Roughly, for anyparticular edge e in G , by sampling many matchings in G and counting how many of them are matchings in G \ e ,one can obtain an estimate of the ratio ρ = M(G)

M(G\e) . The number M(G) is then ρM(G \ e) , where M(G \ e) canbe approximated by applying the same method recursively.

29.6 Notes[1] As of 2008, see Rempala & Wesolowski (2008)

[2] van Lint & Wilson (2001) p. 99

[3] CRC Concise Encyclopedia of Mathematics ()

[4] Little (1974), Vazirani (1988)

[5] Pólya (1913), Reich (1971)

29.7 References• Allender, Eric; Gore, Vivec (1994), “A uniform circuit lower bound for the permanent”, SIAM Journal onComputing 23 (5): 1026–1049, doi:10.1137/s0097539792233907

• Balasubramanian, K. (1980), Combinatorics and Diagonals ofMatrices, Ph.D. Thesis, Department of Statistics,Loyola College, Madras, India T073, Indian Statistical Institute, Calcutta

94 CHAPTER 29. COMPUTING THE PERMANENT

• Bax, Eric (1998), Finite-difference Algorithms for Counting Problems, Ph.D. Dissertation 223, California Insti-tute of Technology

• Bax, Eric; Franklin, J. (1996), A finite-difference sieve to compute the permanent, CalTech-CS-TR-96-04,California Institute of Technology

• Glynn, David G. (2010), “The permanent of a square matrix”, European Journal of Combinatorics 31 (7):1887–1891, doi:10.1016/j.ejc.2010.01.010

• Glynn, David G. (2013), “Permanent formulae from the Veronesean”, Designs Codes and Cryptography 68(1-3): 39–47, doi:10.1007/s10623-012-9618-1

• Jerrum, M.; Sinclair, A.; Vigoda, E. (2001), “A polynomial-time approximation algorithm for the perma-nent of a matrix with non-negative entries”, Proc. 33rd Symposium on Theory of Computing, pp. 712–721,doi:10.1145/380752.380877, ECCC TR00-079

• Mark Jerrum; Leslie Valiant; Vijay Vazirani (1986), “Random generation of combinatorial structures from auniform distribution”, Theoretical Computer Science 43: 169–188, doi:10.1016/0304-3975(86)90174-X

• Kogan, Grigoriy (1996), “Computing permanents over fields of characteristic 3: where and why it becomesdifficult”, 37th Annual Symposium on Foundations of Computer Science (FOCS '96)

• van Lint, Jacobus Hendricus; Wilson, Richard Michale (2001),ACourse in Combinatorics, ISBN 0-521-00601-5

• Little, C. H. C. (1974), “An extension of Kasteleyn’s method of enumerating the 1-factors of planar graphs”,in Holton, D., Proc. 2nd Australian Conf. Combinatorial Mathematics, Lecture Notes in Mathematics 403,Springer-Verlag, pp. 63–72

• Little, C. H. C. (1975), “A characterization of convertible (0, 1)-matrices”, Journal of Combinatorial Theory,Series B 18 (3): 187–208, doi:10.1016/0095-8956(75)90048-9

• Marcus, M.; Minc, H. (1961), “On the relation between the determinant and the permanent”, Illinois Journalof Mathematics 5: 376–381

• Pólya, G. (1913), “Aufgabe 424”, Arch. Math. Phys. 20 (3): 27

• Reich, Simeon (1971), “Another solution of an old problem of pólya”, American Mathematical Monthly 78 (6):649–650, doi:10.2307/2316574, JSTOR 2316574

• Rempała, Grzegorz A.; Wesolowski, Jacek (2008), Symmetric Functionals on Random Matrices and RandomMatchings Problems, p. 4, ISBN 0-387-75145-9

• Ryser, Herbert John (1963), Combinatorial Mathematics, The Carus mathematical monographs, The Mathe-matical Association of America

• Vazirani, Vijay V. (1988), “NC algorithms for computing the number of perfect matchings in K₃,₃-free graphsand related problems”, Proc. 1st Scandinavian Workshop on Algorithm Theory (SWAT '88), Lecture Notes inComputer Science 318, Springer-Verlag, pp. 233–242, doi:10.1007/3-540-19487-8_27

• Valiant, Leslie G. (1979), “The Complexity of Computing the Permanent”, Theoretical Computer Science (El-sevier) 8 (2): 189–201, doi:10.1016/0304-3975(79)90044-6

• “Permanent”, CRC Concise Encyclopedia of Mathematics, Chapman & Hall/CRC, 2002

Chapter 30

Configuration graph

Configuration graphs are a theoretical tool used in computational complexity theory to prove a relation betweengraph reachability and complexity classes.

30.1 Definition

A theoretical computational model, like Turing machine or finite automata, explains how to do a computation. Themodel explains both what is an initial configuration of the machine and which steps can be taken to continue thecomputation, until we eventually stop. A configuration, also called an Instantaneous Description(ID) is a finite repre-sentation of the machine at a given time. For example, for a finite automata and a given input, the configuration willbe the current state and the number of read letters, for a Turing machine it will be the state, the content of the tapeand the position of the head. A configuration graph is a directed labeled graph where the label of the vertices are thepossible configurations of the models and where there is an edge from one configuration to another if it correspondto a computational step of the model.The initial and accepting configuration(s) of the machine are special vertices of the configuration graph. The com-putation accepts if and only if there is a path from an initial vertex to an accepting vertex.

30.2 Useful property

If the computation is deterministic then from any configuration there is at most one possible step, so the graph is ofout-degree 1, and there is exactly one initial state.Once we add a dummy initial vertex with an edge to every initial vertex and a dummy accepting vertex with an edgefrom every accepting vertex, checking if there is an accepting computation only requires to check if there is a pathfrom the initial vertex to the accepting vertex, which is the reachability problem.A cycle in the graph means that there is a possible infinite loop in the computation.

30.3 Size of the graph

The computational graph can be of infinite size if there are no restrictions on possible configurations; indeed, it iseasy to see that there are Turing machines which can reach arbitrarily large configurations.It is also possible to have finite graphs: on Deterministic finite automaton with s states, for a given word of size n theconfiguration is composed of the position of the head and the current state. So the graph is of size (n+1)s , and theaccessible part from the initial state is of size n+ 1 .

95

96 CHAPTER 30. CONFIGURATION GRAPH

30.4 Use of this object

This notion is useful because it reduces computational problems to graph reachability problems.For example, since reachability is in NL when we can represent configurations in space which is logarithmic in thesize of the input, and since the configuration of the Turing Machine in NL is indeed of logarithmic size, then graph-reachability is complete for NL.[1]

In the other direction, it helps to verify the complexity of a computation model; the decision problem for a (deter-ministic) model whose configuration are of space which is logarithmic in the size of the input is in (L) NL. This isfor example the case of finite automata and finite automata with one counter.

30.5 References[1] Papadimitriou, Christos H. (1994). Computational Complexity, Reading, Massachusetts: Addison-Wesley. ISBN 0-201-

53082-1.

• Sanjeev Arora and Boaz Barak (2009). Computational complexity, a modern approach. Cambridge UniversityPress. ISBN 978-0-521-42426-4. Section 4.3: NL-completeness, p. 87.

Chapter 31

Constructible function

In complexity theory, a time-constructible function is a function f from natural numbers to natural numbers withthe property that f(n) can be constructed from n by a Turing machine in the time of order f(n). The purpose of sucha definition is to exclude functions that do not provide an upper bound on the runtime of some Turing machine.

31.1 Time-constructible definitions

There are two different definitions of a time-constructible function. In the first definition, a function f is called time-constructible if there exists a positive integer n0 and Turing machineM which, given a string 1n consisting of n ones,stops after exactly f(n) steps for all n ≥ n0. In the second definition, a function f is called time-constructible if thereexists a Turing machineM which, given a string 1n, outputs the binary representation of f(n) in O(f(n)) time (a unaryrepresentation may be used instead, since the two can be interconverted in O(f(n)) time).There is also a notion of a fully time-constructible function. A function f is called fully time-constructible if thereexists a Turing machine M which, given a string 1n consisting of n ones, stops after exactly f(n) steps. This definitionis slightly less general than the first two but, for most applications, either definition can be used.

31.2 Space-constructible definitions

Similarly, a function f is space-constructible if there exists a positive integer n0 and a Turing machine M which,given a string 1n consisting of n ones, halts after using exactly f(n) cells for all n ≥ n0. Equivalently, a function f isspace-constructible if there exists a Turing machine M which, given a string 1n consisting of n ones, outputs thebinary (or unary) representation of f(n), while using only O(f(n)) space.Also, a function f is fully space-constructible if there exists a Turing machine M which, given a string 1n consistingof n ones, halts after using exactly f(n) cells.

31.3 Examples

All the commonly used functions f(n) (such as n, nk, 2n) are time- and space-constructible, as long as f(n) is at leastcn for a constant c > 0. No function which is o(n) can be time-constructible unless it is eventually constant, sincethere is insufficient time to read the entire input. However, log(n) is a space-constructible function.

31.4 Applications

Time-constructible functions are used in complexity theory results such as the time hierarchy theorem. They areimportant because the time hierarchy theorem relies on Turing machines that must determine inO(f(n)) time whetheran algorithm has taken more than f(n) steps. This is, of course, impossible without being able to calculate f(n) in that

97

98 CHAPTER 31. CONSTRUCTIBLE FUNCTION

time. Such results are typically true for all natural functions f but not necessarily true for artificially constructed f.To formulate them precisely, it is necessary to have a precise definition for a natural function f for which the theoremis true. Time-constructible functions are often used to provide such definition.Space-constructible functions are used similarly, for example in the space hierarchy theorem.This article incorporates material from constructible on PlanetMath, which is licensed under the Creative CommonsAttribution/Share-Alike License.

Chapter 32

Context of computational complexity

In computational complexity theory and analysis of algorithms, a number of metrics are defined describing the re-sources, such as time or space, that a machine needs to solve a particular problem. Interpreting these metrics mean-ingfully requires context, and this context is frequently implicit and depends on the field and the problem underconsideration. This article describes a number of important pieces of context and how they affect metrics.

32.1 Definitions of variables

Metrics are usually described in terms of variables that are a function of the input. For example, the statement thatinsertion sort requires O(n2) comparisons is meaningless without defining n, which in this case is the number ofelements in the input list.Because many different contexts use the same letters for their variables, confusion can arise. For example, the com-plexity of primality tests and multiplication algorithms can be measured in two different ways: one in terms of theintegers being tested or multiplied, and one in terms of the number of binary digits (bits) in those integers. Forexample, if n is the integer being tested for primality, trial division can test it in Θ(n1/2) arithmetic operations; but ifn is the number of bits in the integer being tested for primality, it requires Θ(2n/2) time. In the fields of cryptographyand computational number theory, it is more typical to define the variable as the number of bits in the input integers.In the field of computational complexity theory, the input is usually specified as a binary string (or a string in somefixed alphabet), and the variable is usually the number of bits in this string. This measure depends on the specificencoding of the input, which must be specified. For example, if the input is an integer specified using unary coding,trial division will require only Θ(n1/2) arithmetic operations; but if the same input is specified in binary (or any largerbase) the complexity rises to Θ(2n/2) operations, not because the algorithm is taking any additional time, but becausethe number of bits in the input n has become exponentially smaller. In the other direction, succinct circuits are compactrepresentations of a limited class of graphs that occupy exponentially less space than ordinary representations likeadjacency lists. Many graph algorithms on succinct circuits are EXPTIME-complete, whereas the same problemsexpressed with conventional representations are only P-complete, because the succinct circuit inputs have smallerencodings.Output-sensitive algorithms define their complexity not only in terms of their input but also their output. For example,Chan’s algorithm can compute the convex hull of a set of points in O(n log h) time, where n is the number of pointsin the input and h is the number of points in the resulting convex hull, a subset of the input points. Because everyinput point might be in the convex hull, an analysis in terms of the input alone would yield the less precise O(n log n)time.The complexity of some algorithms depends not only on parameters of the input but also parameters of the machinethe algorithm is being run on; as mentioned in #Metric being measured below, this is typical in analyzing algorithmsthat run on systems with fixed cache hierarchies, where the complexity may depend on parameters such as cache sizeand block size.

99

100 CHAPTER 32. CONTEXT OF COMPUTATIONAL COMPLEXITY

32.2 Abstract machine

To analyze an algorithm precisely, one must assume it is being executed by a particular abstract machine. For example,on a random access machine, binary search can be used to rapidly locate a particular value in a sorted list in onlyO(log n) comparisons, where n is the number of elements in the list; on a Turing machine, this is not possible, sinceit can only move one memory cell at a time and so requires Ω(n) steps to even reach an arbitrary value in the list.Moreover, different abstract machines define different primitive operations, which are operations that can be per-formed in constant time. Some machines, like Turing machines, only permit one bit at a time to be read or written;these are called bit operations, and the number of bit operations required to solve a problem is called its bit com-plexity. Bit complexity generalizes to any machine where the memory cells are of a fixed size that does not dependon the input; for this reason, algorithms that manipulate numbers much larger than the registers on ordinary PCs aretypically analyzed in terms of their bit complexity. Put another way, the bit complexity is the complexity when theword size is a single bit, where word size is the size of each memory cell and register.Another commonly used model has words with log n bits, where n is a variable depending on the input. For example,in graph algorithms, it is typical to assume that the vertices are numbered 1 through n and that each memory cell canhold any of these values, so that they can refer to any vertex. This is justified in problems where the input uses Ω(n)words of storage, since on real computers, the memory cell and register size is typically selected in order to be ableto index any word in memory. Operations on these words, such as copies and arithmetic operations, are assumed tooperate in constant time, rather than O(log n) time. The number of word operations required to solve a problem inthis model is sometimes called its word complexity.In computational complexity theory, researchers intentionally define complexity classes in a way intended to makethem machine-independent - that is, if a problem lies in a particular class relative to a particular “reasonable” machine,it will lie in that class relative to any “reasonable” machine. For example, as mentioned above, the time complexity ofbinary search depends on whether a Turing machine or a random access machine is used; but regardless of the machinechoice, it lies in P, the class of polynomial-time algorithms. In general, P is considered a machine-independent classbecause any operation that can be simulated in polynomial time can be assumed to require constant time, since it canbe treated as a subroutine without exceeding the polynomial-time bound.Oracle machines are machines that have a specific operation that they can perform in constant time; this operationcan be arbitrarily complex and can dramatically affect the complexity of algorithms performed on the machine.For example, if one has an oracle to solve any NP-complete problem, then any problem in NP can be solved inpolynomial time (whereas without the oracle, no polynomial-time algorithm is known for many of these problems).Oracle machines are impractical to construct but useful in theory for determining which proof techniques will beeffective.

32.3 Metric being measured

It’s typical to say without qualification that insertion sort requires O(n2) time; however, it doesn't make sense tosay that the bit complexity of insertion sort is O(n2), unless the elements being sorted are of constant size. If theelements are assumed to be integers between 1 and n, then the word complexity where words have log n bits wouldbe O(n2), but it’s preferable to have a model that allows sorting of lists other than lists of small integers, such as listsof strings. In this case, instead of measuring the number of time steps the abstract machine takes, it’s preferable todefine a particular metric appropriate to the problem at hand. For comparison sort algorithms, which examine theinput using only comparisons and modify it using only exchanges (swaps), the typical metric is either the numberof element comparisons performed, the number of element exchanges performed, or the sum of these. Differentcomparison sort algorithms can be compared using these metrics, but for useful comparison with non-comparisonsorting algorithms, such as radix sort, a different metric must be used, and the set of elements must be restricted.Because disk operations are orders of magnitude slower than accesses to main memory, the typical metric used indisk-based algorithms is the number of disk seeks or the number of blocks read from the disk, which depend onboth the input and the parameters of the disk. RAM accesses and CPU operations are “free.” Similarly, in manymodels used to study data structures, such as the cache-oblivious model, operations on cached data are considered“free” because they are typically orders of magnitude faster in practice than access to main memory. Consequently,the typical metric used is the number of cache misses, which depends on both the input and parameters of the cache.

Chapter 33

Padding argument

In computational complexity theory, the padding argument is a tool to conditionally prove that if some complexityclasses are equal, then some other bigger classes are also equal.

33.1 Example

The proof that P = NP implies EXP = NEXP uses “padding”. EXP ⊆ NEXP by definition, so it suffices to showNEXP ⊆ EXP .Let L be a language in NEXP. Since L is in NEXP, there is a non-deterministic Turing machine M that decides L intime 2n

c for some constant c. Let

L′ = x12|x|c

| x ∈ L,

where 1 is a symbol not occurring in L. First we show that L′ is in NP, then we will use the deterministic polynomialtime machine given by P = NP to show that L is in EXP.L′ can be decided in non-deterministic polynomial time as follows. Given input x′ , verify that it has the formx′ = x12

|x|c and reject if it does not. If it has the correct form, simulateM(x). The simulation takes non-deterministic2|x|

c time, which is polynomial in the size of the input, x′ . So, L′ is in NP. By the assumption P = NP, there is alsoa deterministic machine DM that decides L′ in polynomial time. We can then decide L in deterministic exponentialtime as follows. Given input x , simulate DM(x12

|x|c

) . This takes only exponential time in the size of the input, x .The 1d is called the “padding” of the language L. This type of argument is also sometimes used for space complexityclasses, alternating classes, and bounded alternating classes.

33.2 References• Arora, Sanjeev; Barak, Boaz (2009), Computational Complexity: A Modern Approach, Cambridge, p. 57,

ISBN 978-0-521-42426-4

101

Chapter 34

The Complexity of Songs

"The Complexity of Songs" was a journal article published by computer scientist Donald Knuth in 1977,[1] as anin-joke about computational complexity theory. The article capitalizes on the tendency of popular songs to devolvefrom long and content-rich ballads to highly repetitive texts with little or no meaningful content.[2] The article noteshow some songs can reach a complexity level, for a song of length N words, as formula: O(log N). The gist of thearticle is repeated below, maintaining the wit of the key concepts.

34.1 Article summary

Knuth writes that “our ancient ancestors invented the concept of refrain" to reduce the space complexity of songs,which becomes crucial when a large number of songs is to be committed to one’s memory. Knuth’s Lemma 1 statesthat if N is the length of a song, then the refrain decreases the song complexity to cN, where the factor c < 1.[1]

Knuth further demonstrates a way of producing songs with O(√N ) complexity, an approach “further improved by

a Scottish farmer named O. MacDonald".[1]

More ingenious approaches yield songs of complexity O( logN ), a class known as "m bottles of beer on the wall".Finally, the progress during the 20th century — stimulated by the fact that “the advent of modern drugs has led todemands for still less memory” — leads to the ultimate improvement: Arbitrarily long songs with space complexityO(1), e.g. for a song to be defined by the recurrence relation[1]

S0 = ϵ, Sk = VkSk−1, k ≥ 1,

Vk = 'That’s the way,' U 'I like it,' U , for all k ≥ 1

U = 'uh huh,' 'uh huh'

34.2 Further developments

Prof. Kurt Eisemann of San Diego State University in his letter to the Communications of the ACM[3] further improvesthe latter seemingly unbeatable estimate. He begins with an observation that for practical applications the value ofthe “hidden constant” c in the Big Oh notation may be crucial in making the difference between the feasibility andunfeasibility: for example a constant value of 1080 would exceed the capacity of any known device. He furthernotices that a technique has already been known in Mediaeval Europe whereby textual content of an arbitrary tunecan be recorded basing on the recurrence relation Sk = C2Sk−1 , where C2 =′ la′ , yielding the value of the big-Ohconstant c equal to 2. However it turns out that another culture achieved the absolute lower bound of O(0). As Prof.Eisemann puts it:

“When the Mayflower voyagers first descended on these shores, the native Americans proud of theirachievement in the theory of information storage and retrieval, at first welcomed the strangers with thecomplete silence. This was meant to convey their peak achievement in the complexity of songs, namelythe demonstration that a limit as low as c = 0 is indeed obtainable.”

102

34.3. REFERENCES 103

However the Europeans were unprepared to grasp this notion, and the chiefs, in order to establish a common groundto convey their achievements later proceeded to demonstrate an approach described by the recurrent relation Sk =C1Sk−1 , where C1 =′ i′ , with a suboptimal complexity given by c = 1.[2][3]

The O(1) space complexity result was also implemented by Guy L. Steele, Jr., perhaps challenged by Knuth’s article.[4]

Dr. Steele’s TELNET Song used a completely different algorithm based on exponential recursion, a parody on someimplementations of TELNET.[5][6][7]

It has been suggested that the complexity analysis of human songs can be a useful pedagogic device for teachingstudents complexity theory.[8]

The article On Superpolylogarithmic Subexponential Functions by prof. Alan Sherman[9] writes that Knuth’s articlewas seminal for analysis of a special class of functions.

34.3 References[1] Knuth, Donald (Summer 1977). “The Complexity of Songs”. SIGACT News: 17–24. Reprinted in: Knuth, Donald (1984).

“The Complexity of Songs”. Communications of the ACM 27 (4): 344–346. doi:10.1145/358027.358042. MR 784131.

[2] Steven Krantz (2005) “Mathematical Apocrypha Redux”, ISBN 0-88385-554-2, pp.2, 3.

[3] Kurt Eisemann, “Further Results on the Complexity of Songs”, Communications of the ACM, vol 28 (1985), no. 3, p. 235.

[4] Peter G. Neumann, “A further view of the first quarter century” ,Communications of the ACM, Volume 27, Issue 4, April1984, p. 343

[5] Guy L. Steele, Jr., “The Telnet Song”, Communications of the ACM, April 1984

[6] Text of the TELNET Song (retrieved January 5, 2012)

[7] Telnet song in MIDI format

[8] Chavey, Darrah (1996). “Songs and the analysis of algorithms”. SIGCSE '96: 4–8. doi:10.1145/236452.236475. Retrieved7 January 2013.

[9] Alan Sherman, “On Superpolylogarithmic Subexponential Functions:, ACM SIGACT News, vol. 22, no. 1, 1991, p. 65

34.4 External links• "The Complexity of Songs", Knuth, Donald E. (1984).

104 CHAPTER 34. THE COMPLEXITY OF SONGS

34.5 Text and image sources, contributors, and licenses

34.5.1 Text• Aanderaa–Karp–Rosenberg conjecture Source: https://en.wikipedia.org/wiki/Aanderaa%E2%80%93Karp%E2%80%93Rosenberg_

conjecture?oldid=646610062 Contributors: Michael Hardy, Phil Boswell, Tobias Bergemann, Giftlite, Bender235, Matt Cook, Mya-suda, A3nm, David Eppstein, Coppertwig, GirasoleDE, Eeekster, Dthomsen8, Dekart, Yobot, Citation bot, Thore Husfeldt, Yewang315,Citation bot 1, RobinK, RjwilmsiBot, Helpful Pixie Bot, CitationCleanerBot, Ynaamad and Anonymous: 4

• Advice (complexity) Source: https://en.wikipedia.org/wiki/Advice_(complexity)?oldid=628842244Contributors: Michael Hardy, Ixfd64,Dcoetzee, MathMartin, Andris, Gadfium, Creidieki, ReyBrujo, SmackBot, Commander Keane bot, Sadeq, CRGreathouse, Simeon, DavidEppstein, Addbot, Luckas-bot, Yobot, Erik9bot, RobinK, BattyBot, ChrisGualtieri, Deltahedron, Cerabot~enwiki, Tesujikekx and Anony-mous: 7

• Analysis of algorithms Source: https://en.wikipedia.org/wiki/Analysis_of_algorithms?oldid=669243362 Contributors: Bryan Derksen,Seb, Arvindn, Hfastedge, Edward, Nixdorf, Kku, McKay, Pakaran, Murray Langton, Altenmann, MathMartin, Bkell, Tobias Berge-mann, David Gerard, Giftlite, Jao, Brona, Manuel Anastácio, Beland, Andreas Kaufmann, Liberlogos, Mani1, Bender235, Ashewmaker,MCiura, Gary, Terrycojones, Pinar, Ruud Koot, Ilya, Qwertyus, Miserlou, DVdm, YurikBot, PrologFan, SmackBot, Vald, Nbarth,Mhym, GRuban, Radagast83, Cybercobra, Kendrick7, Spiritia, Lee Carre, Amakuru, CRGreathouse, ShelfSkewed, Hermel, Magio-laditis, VoABot II, David Eppstein, User A1, Maju wiki, 2help, Cometstyles, The Wilschon, BotKung, Groupthink, Keilana, Xe7al,Ykhwong, Alastair Carnegie, Ivan Akira, Roux, Jarble, Legobot, Yobot, Fraggle81, Pcap, GateKeeper, AnomieBOT, Materialscientist,Miym, Charvest, Fortdj33, 124Nick, RobinK, WillNess, RjwilmsiBot, Uploader4u, Jmencisom, Wikipelli, The Nut, Tirab, Tijfo098,ClueBot NG, Ifarzana, Satellizer, Tvguide1234, Helpful Pixie Bot, Intr199, Manuel.mas12, Liam987, AlexanderZoul, Jochen Burghardt,Cubism44, Vieque, PNattrass and Anonymous: 75

• Approximation algorithm Source: https://en.wikipedia.org/wiki/Approximation_algorithm?oldid=667970915 Contributors: Danny,Dcoetzee, Fredrik, Ojigiri~enwiki, Giftlite, Mellum, Andris, NathanHurst, Jnestorius, Haham hanuka, Arthena, Culix, Oleg Alexan-drov, Decrease789, Ruud Koot, GregorB, Rjwilmsi, Chobot, YurikBot, Bota47, Tribaal, Bmatheny, SmackBot, DKalkin, Pnamjoshi,Myasuda, NotQuiteEXPComplete, LachlanA, Dricherby, Tiagofassoni, David Eppstein, LordAnubisBOT, Ashishgoel.1973, Ratfox,Brvman, Vavi~enwiki, Paolo.dL, Whodoesthis, RMFan1, Kolyma, C. lorenz, Addbot, CarsracBot, Numbo3-bot, Yobot, Citation bot,Anonash, Kiefer.Wolfowitz, RobinK, EmausBot, ZéroBot, Howard nyc, Helpful Pixie Bot, BattyBot, Olonic, Mariolucic, Mark viking,Metin.balaban, Infinitestory, KasparBot and Anonymous: 30

• Approximation-preserving reduction Source: https://en.wikipedia.org/wiki/Approximation-preserving_reduction?oldid=664545324Contributors: Edemaine, Npinsker, Infinitestory and Anonymous: 1

• Asymptotic computational complexity Source: https://en.wikipedia.org/wiki/Asymptotic_computational_complexity?oldid=662061990Contributors: Altenmann, ZeroOne, BD2412, Logan, Taemyr, Sharonidith, Buenasdiaz, NOrbeck, VladimirReshetnikov, FrescoBot,RobinK, Mo ainm, Dcirovic, CeraBot, Brirush, Sol1 and Anonymous: 3

• Averaging argument Source: https://en.wikipedia.org/wiki/Averaging_argument?oldid=536874098Contributors: Michael Hardy, Sadeq,Yobot, Nageh, DrilBot, RobinK and Anonymous: 4

• Best, worst and average case Source: https://en.wikipedia.org/wiki/Best%2C_worst_and_average_case?oldid=676068151 Contribu-tors: Lee Daniel Crocker, The Anome, Ed Poor, Dieter Simon, DWay, AdSR, Michael Hardy, TakuyaMurata, Angela, Charles Matthews,Timwi, Dcoetzee, Malcohol, Populus, Sabbut, Altenmann, Paul G, Wolfkeeper, Nayuki, Pgan002, Octalc0de, Andreas Kaufmann, MikeRosoft, Liso, Nard the Bard, Reinyday, Phils, Jumbuck, Dsm, Fasten, Localh77, Pol098, Ruud Koot, Brazzy, Brighterorange, Crazy-computers, DevastatorIIC, Spl, Gardar Rurak, Arthur Rubin, Cedar101, SmackBot, Ianb1469, Gilliam, Ohnoitsjamie, Bluebot, Dfletter,Radagast83, Rory096, Grinning Fool, Alphachimpbot, Cic, SwiftBot, EagleFan, Jacobko, JavierMC, Muro de Aguas, SieBot, Taemyr,ClueBot, Mostargue, Little Mountain 5, Scasa155, Addbot, Some jerk on the Internet, Dyadron, ,ماني LuK3, AnomieBOT, Jim1138,Materialscientist, Pownuk, Spidern, Charvest, Beekeepingschool, RobinK, Kgautam28, Wiki.Tango.Foxtrot, Brycehughes, Avocato-Bot, Dark Silver Crow, Cadillac000, BonzaiThePenguin, Melonkelon, Gautham tpsz, JaconaFrere, , Basstrekker87, Peppy Paneer,Kebede447 and Anonymous: 63

• Boolean circuit Source: https://en.wikipedia.org/wiki/Boolean_circuit?oldid=605665639 Contributors: The Anome, AC, Creidieki,Eclecticos, EAderhold, J. Finkelstein, Nicko6, CBM, Pascal.Tesson, Libertyernie2, JAnDbot, D-rew, A3nm, David Eppstein, EnJx,Classicalecon, Mild Bill Hiccup, Addbot, OlEnglish, Yobot, Citation bot, Sae1962, Arthur MILCHIOR, Tijfo098, ClueBot NG, Snotbot,Usher14, Ben.d.zimmer, Ngshoy, Hartzm, USER-CSF-540 and Anonymous: 9

• Certificate (complexity) Source: https://en.wikipedia.org/wiki/Certificate_(complexity)?oldid=641963475Contributors: Andreas Kauf-mann, Jérôme, SmackBot, Jonatan Swift, Fabrictramp, David Eppstein, Coppertwig, Andy Dingley, Jim1138, RobinK, Ngshoy, YiFeiBot,Gabrielamota29 and Anonymous: 4

• Circuit complexity Source: https://en.wikipedia.org/wiki/Circuit_complexity?oldid=659743998 Contributors: Michael Hardy, Dcoet-zee, TimR, Creidieki, EmilJ, Rjwilmsi, MarSch, Jpbowen, Ott2, Gnp, Ylloh, Pascal.Tesson, Thijs!bot, Hermel, Krishnachandranvn,Jamelan, Daveagp, Pichpich, Kolyma, Algebran, Addbot, Download, RobertHannah89, Citation bot, Twri, Miym, RobinK, Orenburg1,BG19bot and Anonymous: 17

• Circuits over sets of natural numbers Source: https://en.wikipedia.org/wiki/Circuits_over_sets_of_natural_numbers?oldid=638712207Contributors: Michael Hardy, CRGreathouse, CmdrObot, Electron9, A3nm, Yobot, Arthur MILCHIOR and Anonymous: 2

• Cobham’s thesis Source: https://en.wikipedia.org/wiki/Cobham’s_thesis?oldid=645095025 Contributors: Timwi, Dcoetzee, Mat-tflaschen, Gdr, DcoetzeeBot~enwiki, Shreevatsa, Kolbasz, Msikma, Froth, Black-Velvet, Banus, SmackBot, Bluebot, Emurphy42, LouSchef-fer, Cybercobra, Macha, Vyznev Xnebara, Electron9, Navigatr85, Bhudson, Adrianwn, Mild Bill Hiccup, AnomieBOT, 45Factoid44,Twri, Citation bot 1, RobinK, Staszek Lem, Helpful Pixie Bot, Max Longint, Leprof 7272, Alexbrandts and Anonymous: 10

• Combinatorial optimization Source: https://en.wikipedia.org/wiki/Combinatorial_optimization?oldid=672406949Contributors: Tomo,Michael Hardy, Docu, Silverfish, Hike395, Aliekens, Altenmann, Giftlite, Mellum, Diomidis Spinellis, Remuel, Giraffedata, Kinema,Ruud Koot, Josh Parris, Tizio, Jeffrey Henning, Chobot, Bgwhite, Estr4ng3d, Wavelength, Ksyrie, RKUrsem, Tribaal, LeonardoRob0t,SmackBot, Moxon, Eskimbot, Jcc1, Ctbolt, Lepikhin, Brunato, SilkTork, StoneIsle, Pjrm, Sdorrance, NotQuiteEXPComplete, Arnab das,Brunner ru, Ben1220, Ben pcc, Extropian314, Bonniesteiglitz, David Eppstein, Akhil999in, Gwern, Nocklas, Maurice Carbonaro, Cobi,

34.5. TEXT AND IMAGE SOURCES, CONTRIBUTORS, AND LICENSES 105

ThomHImself, CharlesGillingham, Cngoulimis, Deanlaw, Daveagp, D.scain.farenzena, Dmyersturnbull, Addbot, MrVanBot, Yobot,Amirobot, AnomieBOT, Unara, Isheden, Miym, Eiro06, FrescoBot, Kiefer.Wolfowitz, RobinK, Jonkerz, Duoduoduo, Nmillerche, Johnof Reading, Ebe123, Chewings72, Trevor x1968, Sammy1007, Helpful Pixie Bot, Rjpbi, Brirush, AustinBuchanan, U2fanboi, Amoshag,Whatfoxsays and Anonymous: 34

• Combinatorial search Source: https://en.wikipedia.org/wiki/Combinatorial_search?oldid=581728038 Contributors: Patrick, MichaelHardy, Minesweeper, Ams80, Maxomai, Bart Massey, Klemen Kocjancic, Rd232, Alai, Oleg Alexandrov, Ruud Koot, Moxon, Dfletter,BernardZ, AnomieBOT and Anonymous: 6

• Communication complexity Source: https://en.wikipedia.org/wiki/Communication_complexity?oldid=605611231Contributors: LC~enwiki,Eloquence, Edward, Michael Hardy, Booyabazooka, Kku, Dcoetzee, Altenmann, Captain Segfault, Ancheta Wis, Neilc, SimonArlott,Marudubshinki, Rjwilmsi, Siddhant, Ott2, SmackBot, Elonka, Bigbluefish, Karmastan, ForgeGod, Ppadala, Thingamagic, Bwpach, Cy-debot, Hermel, David Eppstein, DirkOliverTheis, Coppertwig, Legobot, Yobot, Miym, RedBot, Σ, RobinK, Ambrevar, Chester Markel,9vijay and Anonymous: 21

• Complement (complexity) Source: https://en.wikipedia.org/wiki/Complement_(complexity)?oldid=611347401 Contributors: MichaelHardy, Dcoetzee, Oleg Alexandrov, Woohookitty, SmackBot, David Eppstein, Dorftrottel, DragonBot, Addbot, JackieBot, Materialsci-entist and Anonymous: 1

• Complete (complexity) Source: https://en.wikipedia.org/wiki/Complete_(complexity)?oldid=543888767 Contributors: Toby Bartels,Lemming, Dcoetzee, Mattflaschen, Creidieki, Diego Moya, SmackBot, Nr9, Electron9, Justin W Smith, Addbot, Tide rolls, Twri, Grey-isthenewBlack, RobinK, Ebrambot and Anonymous: 5

• Complexity class Source: https://en.wikipedia.org/wiki/Complexity_class?oldid=676149537 Contributors: The Anome, Ixfd64, Ce-sarB, Schneelocke, Timwi, Dcoetzee, Fredrik, Altenmann, MathMartin, Tobias Bergemann, Giftlite, Andris, Gdr, Bosmon, Ascánder,Ben Standeven, Chalst, Peter M Gerdes, Craptree, Ruud Koot, Chobot, YurikBot, TodorBozhinov, Gene.arboit, Ksyrie, Ott2, ArthurRubin, Netrapt, Tom Morris, SmackBot, Stux, Pkirlin, Tsca.bot, Duncancumming, Zero sharp, CRGreathouse, Pascal.Tesson, Thijs!bot,AlexAlex, Electron9, JAnDbot, Willp4139, Entropy, Jamelan, Addbot, Luckas-bot, Yobot, Reshadipoor, ArthurBot, Miym, FrescoBot,Roi 1986, RobinK, Anonymann, EmausBot, Dima david, Bomazi, Frietjes, Wiker63 and Anonymous: 26

• Complexity index Source: https://en.wikipedia.org/wiki/Complexity_index?oldid=662062807 Contributors: Michael Hardy, Discospin-ster, Scjessey, BD2412, SmackBot, Soap, SMasters, CmdrObot, T@nn, Cerebellum, Melcombe, Addbot, Legobot II, Bassis, Citationbot 1, PigFlu Oink, Diannaa, Monkbot and Anonymous: 2

• Compression theorem Source: https://en.wikipedia.org/wiki/Compression_theorem?oldid=653420767Contributors: Altenmann, Math-Martin, Giftlite, SmackBot, RDBury, J. Finkelstein, David Eppstein, Draconx, Mrmcdonnell, Erik9bot and Anonymous: 2

• Computable topology Source: https://en.wikipedia.org/wiki/Computable_topology?oldid=607163443 Contributors: Michael Hardy,Bearcat, Aude, BD2412, Iamfscked, Myasuda, R'n'B, Katharineamy, Sun Creator, SchreiberBike, Addbot, Yobot, AnomieBOT, Johnof Reading, Khazar2, Samuel leventhal, Mark viking, CaptainMurphy1989 and Anonymous: 2

• Computation tree Source: https://en.wikipedia.org/wiki/Computation_tree?oldid=646913097 Contributors: Creidieki, Cmdrjameson,Pearle, Bluemoose, SmackBot, Elonka, Saint Midge, Sadads, Neharungta, Pascal.Tesson, David Eppstein, Erik9bot and Anonymous: 1

• Computational complexity ofmathematical operations Source: https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations?oldid=676835805 Contributors: Mm, Jitse Niesen, Phil Boswell, Fredrik, Lowellian, Tobias Bergemann, Giftlite, Mike40033,Prosfilaes, Fangz, WhiteDragon, Halo, TerraFrost, EmilJ, Coma28, Qwertyus, Rjwilmsi, HappyCamper, Kri, Hmonroe, Bhny, Smack-Bot, InverseHypercube, Optikos, Berland, Kuleebaba, Jafet, CRGreathouse, Cydebot, Barticus88, Headbomb, Marek69, Hermel, Fbahr,Magioladitis, David Eppstein, Pdokj, Branger~enwiki, Gpap.gpap, Addbot, Zorrobot, Liuhb86, Legobot, Yobot, Jalal0, AnomieBOT,Citation bot 1, RobinK, Kaluppollo, JumpDiscont, Blackbombchu, AustinBuchanan, Clayzermk1, Mikhaelkh and Anonymous: 29

• Computational complexity theory Source: https://en.wikipedia.org/wiki/Computational_complexity_theory?oldid=676228384 Con-tributors: AxelBoldt, LC~enwiki, Mav, Robert Merkel, The Anome, Arvindn, DavidSJ, Ryguasu, Youandme, Stevertigo, Hfastedge,Michael Hardy, Booyabazooka, Nixdorf, Ixfd64, Chinju, Looxix~enwiki, Docu, Charles Matthews, Timwi, David Newton, Dcoet-zee, Jitse Niesen, Daniel Quinlan, Arthaey, Doradus, Prumpf, GulDan, E23~enwiki, Jimbreed, Populus, Wernher, McKay, Pakaran,David.Monniaux, Phil Boswell, Robbot, Chealer, Fredrik, Altenmann, MathMartin, Henrygb, Bkell, Jleedev, Tobias Bergemann, DavidGerard, Tdgs, Giftlite, N12345n, Aphaia, Dissident, Everyking, Andris, Déjà Vu, Quackor, Chowbok, Gdr, Knutux, Beland, Ehsan~enwiki,APH, Alotau, Creidieki, Blokhead, Rich Farmbrough, Guanabot, Leibniz, ArnoldReinhold, Talldean, Ascánder, Ben Standeven, Chalst,Barcex, Themusicgod1, John Vandenberg, Shenme, Flammifer, Obradovic Goran, Mdd, Jhertel, SpaceMoose, Walkerma, Scottcraig,Oleg Alexandrov, Postrach, Xiaoyang, Linas, Decrease789, Rend~enwiki, Ruud Koot, Orz, Mpatel, Wikiklrsc, Bluemoose, GregorB,Bruno Unna, Pete142, Graham87, Emallove, Rjwilmsi, Koavf, Nneonneo, Bubba73, RainR, FlaBot, Mathbot, RexNL, Intgr, Der-Graph~enwiki, Chobot, Hmonroe, Bgwhite, Roboto de Ajvol, Siddhant, Wavelength, RussBot, Gaius Cornelius, Trovatore, Tejas81,PrologFan, Larry laptop, Mikeblas, Auminski, Klutzy, Cesarsorm~enwiki, Zipcube, Ripper234, Arthur Rubin, GrEp, Bo Jacoby, Smack-Bot, InverseHypercube, Powo, Gilliam, Hegariz, Kurykh, Droll, Nbarth, DHN-bot~enwiki, Konstable, Readams, Contrasedative, Steven-mitchell, Bsotomay, Battamer, Henning Makholm, Andrei Stroe, Wvbailey, Harryboyles, J. Finkelstein, 16@r, Ryulong, Jaksmata, JR-Spriggs, Ylloh, CRGreathouse, Clecio~enwiki, GPhilip, Pascal.Tesson, ErrantX, Dragonflare82, Omicronpersei8, Egriffin, Epbr123,ConceptExp, Hazmat2, Young Pioneer, Headbomb, Klausness, WikiSlasher, VictorAnyakin, Hermel, JAnDbot, The Transhumanist,Drizzd~enwiki, Yill577, Four Dog Night, Jeff Dahl, David Eppstein, Ekotkie, Abatasigh, Jamesd9007, FANSTARbot, Adavidb, Mau-rice Carbonaro, Bot-Schafter, Deflagg, Tarotcards, Bouke~enwiki, Krishnachandranvn, Policron, Christer.berg, Tkgd2007, JohnBlack-burne, LokiClock, Toddy1, Philip Trueman, Getonyourfeet, Magmi, Calculuslover800, Rogerdpack, Groupthink, Dmcq, Van Parunak,SieBot, Flyer22, Faradayplank, SimonTrew, Manway, Skippydo, CharlesGillingham, Anchor Link Bot, Triwas, Grsbmd, Cngoulimis,ClueBot, The Thing That Should Not Be, Jlpinar83, Huynl, Erudecorp, D.scain.farenzena, DragonBot, Tim32, PixelBot, Estirabot, Dmy-ersturnbull, Brianbjparker, Dmitri pavlov, Johnuniq, Pichpich, C. lorenz, WikHead, Dsimic, Multipundit, Addbot, DOI bot, Bassbone-rocks, SpBot, Numbo3-bot, שי ,דוד Gail, Zorrobot, Mik01aj, Yobot, OrgasGirl, Ptbotgourou, Pcap, Nallimbot, Waltnmi, Rubinbot,Hiihammuk, Materialscientist, Citation bot, Twri, Xqbot, Miym, RibotBOT, Charvest, FrescoBot, Mycer1nus, Sae1962, Citation bot 1,MarcelB612, Eser.aygun, RedBot, RobinK, AvnishIT, Ink-Jetty, D climacus, Quotient group, Rednas1234, EmausBot, Ethereal-Blade,Fuujuhi, Jmencisom, Carbo1200, Bethnim, MassimoLauria, Mastergreg82, AvicAWB, Muditjai, Wayne Slam, Staszek Lem, OrangeSuede Sofa, Dlu776, ClueBot NG, LJosil, Braincricket, Helpful Pixie Bot, CitationCleanerBot, Brad7777, MichiHenning, AidaFernan-daUFPE, Aaron Nitro Danielson, JYBot, Dexbot, Deltahedron, Djhulme, Julianiacoponi, Alexbrandts, Jpmunz, Monkbot, Amitkumarp,Mona Borham, Ja49cs69, KasparBot and Anonymous: 213

106 CHAPTER 34. THE COMPLEXITY OF SONGS

• Computational resource Source: https://en.wikipedia.org/wiki/Computational_resource?oldid=665265964 Contributors: Murray Lang-ton, Altenmann, Piotrus, Creidieki, Abdull, Gene Nygaard, Bovineone, EngineerScotty, SmackBot, Oerjan, Remember the dot, Jamelan,Softtest123, Addbot, DOI bot, Jarble, AnomieBOT, Citation bot, Miym, Locobot, Trappist the monk, RjwilmsiBot, Alfredo ougaowen,Helpful Pixie Bot, Solomon7968, Klilidiplomus, Monkbot and Anonymous: 5

• Computational topology Source: https://en.wikipedia.org/wiki/Computational_topology?oldid=652990378Contributors: Michael Hardy,Hyacinth, Altenmann, Henrygb, C S, Mathbot, Iamfscked, Trovatore, SmackBot, David Eppstein, Wildknot, DirkOliverTheis, Woood,JohnBlackburne, Rybu, Timhoooey, Addbot, Delaszk, Yobot, Citation bot, Charvest, FrescoBot, ZéroBot, CitationCleanerBot, Brad7777,Samuel lev, Phillipmates, Samuel leventhal, Mark viking and Anonymous: 15

• Computationally bounded adversary Source: https://en.wikipedia.org/wiki/Computationally_bounded_adversary?oldid=637704144Contributors: Edward, Michael Hardy, Smarteralec, Chris the speller, Dekart, AnomieBOT, I dream of horses, Jim dobler and Rfs23

• Computing the permanent Source: https://en.wikipedia.org/wiki/Computing_the_permanent?oldid=664367627 Contributors: MichaelHardy, McKay, Altenmann, Giftlite, Xezbeth, Zaslav, Shreevatsa, Ruziklan, Rjwilmsi, Mukadderat, Headbomb, Hermel, David Eppstein,Laudak, Bender2k14, Kbdankbot, Yobot, Kilom691, AnomieBOT, Citation bot, Twri, Thore Husfeldt, Citation bot 1, RjwilmsiBot,Circularargument, Helpful Pixie Bot, Gregory Kogan, Ynaamad and Anonymous: 17

• Configuration graph Source: https://en.wikipedia.org/wiki/Configuration_graph?oldid=645923124Contributors: Michael Hardy, Rjwilmsi,SmackBot, Sadads, A3nm, Sphilbrick, Bender2k14, Addbot, LaaknorBot, LilHelpa, Arthur MILCHIOR, John of Reading, Elaz85 andAnonymous: 2

• Constructible function Source: https://en.wikipedia.org/wiki/Constructible_function?oldid=625042515 Contributors: Zundark, Andris,Neilc, CryptoDerk, Helios369, Kotniski, Bender2k14, Nyrox, Addbot, 777sms, Brad7777, Diogoan, Simkieu and Anonymous: 7

• Context of computational complexity Source: https://en.wikipedia.org/wiki/Context_of_computational_complexity?oldid=568811596Contributors: Dcoetzee, Andreas Kaufmann, Mcld, Chris the speller, Addbot, Yobot, LucienBOT and RobinK

• Padding argument Source: https://en.wikipedia.org/wiki/Padding_argument?oldid=670647071 Contributors: Edward, Michael Hardy,Auric, J. Finkelstein, Arthur MILCHIOR, Gareth Griffith-Jones, Upsidedowntophat, Kephir and Anonymous: 5

• The Complexity of Songs Source: https://en.wikipedia.org/wiki/The_Complexity_of_Songs?oldid=672027327 Contributors: Nealmcb,Michael Hardy, Tango, Tregoweth, Kaihsu, Furrykef, Altenmann, AceMyth, Andycjp, D6, Rich Farmbrough, Rajah, Anders Kaseorg,Keenan Pepper, Woohookitty, Shreevatsa, Thruston, BD2412, Pegship, Argyll Lassie, Ccshan, Gutworth, JonHarder, Henning Makholm,Curly Turkey, Mukadderat, Ser Amantio di Nicolao, Loadmaster, Snoopyjc, GiantSnowman, CRGreathouse, ,הסרפד Wikid77, Agix,Michig, Maximuspower, PopMechanic, Kevinkor2, Addbot, Balabiot, Luckas-bot, Yobot, Darolew, Popa-vask-gamma, RjwilmsiBot,ZéroBot, Marcus.brenneman, Helpful Pixie Bot, Mark viking, François Robere and Anonymous: 14

34.5.2 Images• File:Ambox_important.svg Source: https://upload.wikimedia.org/wikipedia/commons/b/b4/Ambox_important.svg License: Public do-

main Contributors: Own work, based off of Image:Ambox scales.svg Original artist: Dsmurat (talk · contribs)• File:Complexity_classes.svg Source: https://upload.wikimedia.org/wikipedia/commons/b/bc/Complexity_classes.svg License: Public

domain Contributors: ? Original artist: ?• File:Complexity_subsets_pspace.svg Source: https://upload.wikimedia.org/wikipedia/commons/6/6e/Complexity_subsets_pspace.svgLicense: Public domainContributors: Own work by uploader, intended to replace bitmap image illustrating same thingOriginal artist: Handdrawn in Inkscape Qef

• File:Decision_Problem.svg Source: https://upload.wikimedia.org/wikipedia/commons/0/06/Decision_Problem.svgLicense: GFDLCon-tributors: Derivative work based on http://en.wikipedia.org/wiki/File:Decision_Problem.png Original artist: RobinK

• File:DottedLine.png Source: https://upload.wikimedia.org/wikipedia/commons/c/c7/DottedLine.png License: Public domain Contrib-utors: ? Original artist: ?

• File:Edit-clear.svg Source: https://upload.wikimedia.org/wikipedia/en/f/f2/Edit-clear.svg License: Public domain Contributors: TheTango! Desktop Project. Original artist:The people from the Tango! project. And according to the meta-data in the file, specifically: “Andreas Nilsson, and Jakub Steiner (althoughminimally).”

• File:Illustration_of_Proof_for_Computationally_Bounded_Adversary.png Source: https://upload.wikimedia.org/wikipedia/commons/d/d4/Illustration_of_Proof_for_Computationally_Bounded_Adversary.png License: CC BY-SA 3.0 Contributors: Own work Originalartist: Jim dobler

• File:Internet_map_1024.jpg Source: https://upload.wikimedia.org/wikipedia/commons/d/d2/Internet_map_1024.jpg License: CC BY2.5 Contributors: Originally from the English Wikipedia; description page is/was here. Original artist: The Opte Project

• File:KnapsackEmpComplexity.GIF Source: https://upload.wikimedia.org/wikipedia/commons/1/11/KnapsackEmpComplexity.GIFLicense: Public domain Contributors: Own work Original artist: Cngoulimis

• File:Maquina.png Source: https://upload.wikimedia.org/wikipedia/commons/3/3d/Maquina.png License: Public domain Contributors:en.wikipedia Original artist: Schadel (http://turing.izt.uam.mx)

• File:Max_paraboloid.svg Source: https://upload.wikimedia.org/wikipedia/commons/7/72/Max_paraboloid.svg License: CC BY-SA4.0 Contributors: Own work Original artist: IkamusumeFan

• File:Minimum_spanning_tree.svg Source: https://upload.wikimedia.org/wikipedia/commons/d/d2/Minimum_spanning_tree.svg Li-cense: Public domain Contributors: ? Original artist: ?

• File:Question_book-new.svg Source: https://upload.wikimedia.org/wikipedia/en/9/99/Question_book-new.svg License: Cc-by-sa-3.0Contributors:Created from scratch in Adobe Illustrator. Based on Image:Question book.png created by User:Equazcion Original artist:Tkgd2007

34.5. TEXT AND IMAGE SOURCES, CONTRIBUTORS, AND LICENSES 107

• File:SentinelCircle.png Source: https://upload.wikimedia.org/wikipedia/commons/1/17/SentinelCircle.png License: CC BY-SA 3.0Contributors: Own work Original artist: B. Apolloni, S. Bassis, D. Malchiodi

• File:SentinelEx.png Source: https://upload.wikimedia.org/wikipedia/commons/d/db/SentinelEx.png License: CC BY-SA 3.0 Contrib-utors: Own work Original artist: B. Apolloni, S. Bassis, D. Malchiodi

• File:SentinelSegment.png Source: https://upload.wikimedia.org/wikipedia/commons/4/4d/SentinelSegment.png License: CC BY-SA3.0 Contributors: Own work Original artist: B. Apolloni, S. Bassis, D. Malchiodi

• File:SolidLine.png Source: https://upload.wikimedia.org/wikipedia/commons/2/2d/SolidLine.png License: Public domain Contributors:? Original artist: ?

• File:Sorting_quicksort_anim.gif Source: https://upload.wikimedia.org/wikipedia/commons/6/6a/Sorting_quicksort_anim.gif License:CC-BY-SA-3.0 Contributors: originally upload on the English Wikipedia Original artist: Wikipedia:en:User:RolandH

• File:TSP_Deutschland_3.png Source: https://upload.wikimedia.org/wikipedia/commons/c/c4/TSP_Deutschland_3.png License: Pub-lic domain Contributors: https://www.cia.gov/cia/publications/factbook/maps/gm-map.gif Original artist: The original uploader wasKapitän Nemo at German Wikipedia

• File:Text_document_with_red_question_mark.svg Source: https://upload.wikimedia.org/wikipedia/commons/a/a4/Text_document_with_red_question_mark.svg License: Public domain Contributors: Created by bdesham with Inkscape; based upon Text-x-generic.svgfrom the Tango project. Original artist: Benjamin D. Esham (bdesham)

34.5.3 Content license• Creative Commons Attribution-Share Alike 3.0