anatomy of a bit - university of california,...

Anatomy of a Bit

Reading for this lecture:

CMR articles Yeung and Anatomy.

Lecture 20: Natural Computation & Self-Organization, Physics 256A (Winter 2014); Jim Crutchfield

1Monday, March 10, 14

• Alternative rationale for entropy and entropy rate.• measurement outcomes• Stored in decimal digits or bits.

• Series of measurements stored in bits.

• Number of possible sequences of measurements with counts:

Anatomy of a Bit ...


klog10 k log2 k

n n log2 k

nn1, n2, . . . , nk

M =

✓n

n1n2 · · ·nk

◆

=n!

n1!n2! · · ·nk!

Information in a single measurement:

M


• To specify/code which particular sequence occurred store:

bits

• Apply Stirlings approximation:

• To :



k log2 n+ log2 M + log2n+ · · ·Maximum number of bits to store the count ni of each of the k output values

Number of bits to specify particular observed sequence within the class of sequences that have counts n1, n2,..., nk

Bits to specify the number of bits in n itself

n! =p2⇡n(n/e)n

log2 M ⇡ �nkX

i=1

nin log2

nin

= nH[

n1n , n2

n , . . . , nkn ]

= nH[X]

M


• Can compress if



H[X] < log2 k

log2 k

(a)

H[X]

R1

(b)

hµ

R1

(c)

R1 = log2 k �H[X]

1-Symbol Redundancy


• Typical sources are not IID, therefore use:



log2 k

(a)

H[X]

R1

(b)

hµ

R1

(c)

hµ = limL!1

H(L)L

R1 = log2 k � hµ

Intrinsic Redundancy

Intrinsic Randomness

(Total Predictability)


A semantically rich decomposition of a single measurement:



H[X]

(a)

hµ

⇢µ

(b)

rµ

bµ

qµ

bµ

(c)

rµ

wµ

(d)

In addition, measurements occur in the context of past and future.


A semantically rich decomposition of a single measurement ...



H[X]

(a)

hµ

⇢µ

(b)

rµ

bµ

qµ

bµ

(c)

rµ

wµ

(d)

Anticipated from prior observations

Random component




H[X]

(a)

hµ

⇢µ

(b)

rµ

bµ

qµ

bµ

(c)

rµ

wµ

(d)

Relevant for predicting the future

Ephemeral: Exists only for the present

Shared between the past and the current observation, but relevance stops there (stationary process only)

Anticipated by the past, present currently, and plays a role in future behavior. Can be negative!





H[X]

(a)

hµ

⇢µ

(b)

rµ

bµ

qµ

bµ

(c)

rµ

wµ

(d)

Ephemeral: exists only for the present

Structural information



Settings:

Before: Predict the present from the past,



· · · X�3 X�2 X�1 X0 X1 X2 X3 · · ·

X:0 X0 X1:

hµ = H[X0| �X ]


Settings:

Before: Predict the present from the past,



· · · X�3 X�2 X�1 X0 X1 X2 X3 · · ·

X:0 X0 X1:

Today, prediction from the omniscient view: The present in the context of the past and the future.

· · · X�3 X�2 X�1 X0 X1 X2 X3 · · ·

X:0 X0 X1:

hµ = H[X0| �X ]


Notations:

l-blocks:

Past:Present:Future:

Any information measure :



F(`) = F [X0:`]

F

X:0 = . . . X�2X�1

X0

X1: = X1X2 . . .

Xt:t+` = XtXt+1 . . . Xt+`�1


Recall block entropy:



H(`) ⇡ E+ hµ`

E = I[X:0;X0:]

Recall total correlation:T [X0:N ] =

X

A2P (N)|A|=1

H[XA]�H[X0:N ]

For stationary time series, get block total correlation:

T (`) = `H[X0]�H(`) T (0) = 0 T (1) = 0

Compares process’s block entropy to the case of IID RVs.

where:

with


Block entropy versus total correlation ...



T (`) ⇡ �E+ ⇢µ`

Monotone increasing, with linear asymptote:

Convex up.

0 1 2 3 4 5 6

Block length ` [symbols]

�E

0

E

Info

rmation

[bits]

H(`)

T (`)


H[X]

(a)

hµ

⇢µ

(b)

rµ

bµ

qµ

bµ

(c)

rµ

wµ

(d)

Anticipated from prior observations

Random component

Block entropy versus total correlation ...



Note:

H(`) + T (`) = `H(1)

H(1) = hµ + ⇢µ

Plug in asymptotes gives first decomposition:




B(`) = H(`)�R(`)

Q(`) = T (`)�B(`)

W (`) = B(`) + T (`)

0 2 4 6 8 10 12


ER

EB

EQ,EW

Info

rmat

ion

[bits]

R(`)

B(`)

Q(`)

W (`)

Asymptotes:

Scaling of other block informations:

Bound information

Local ExogenousEnigmatic

R(`) ⇡ ER + `rµ

B(`) ⇡ EB + `bµ

Q(`) ⇡ EQ + `qµ

W (`) ⇡ EW + `wµ


H[X]

(a)

hµ

⇢µ

(b)

rµ

bµ

qµ

bµ

(c)

rµ

wµ

(d)

Relevant for predicting the future

Ephemeral: Exists only for the present

Shared between the past and the current observation, but relevance stops there

Anticipated by the past, present currently, and plays a role in future behavior. Can be negative!

Scaling of other block informations:Anatomy of a Bit ...


Thus, and .hµ = bµ + rµBlock total correlation:

Thus, and .⇢µ = bµ + qµ

Block entropy:

E = EB +ER

E = �EB �EQ

Gives seconddecomposition of .H[X]

H(`) = B(`) +R(`)

= (EB +ER) + `(bµ + rµ)

T (`) = B(`) +Q(`)

= (EB +EQ) + `(bµ + qµ)




Scaling of other block informations:

H(1) = wµ + rµ

From definitions:

Plug in asymptotes gives third decomposition of :

R(`) +W (`) = `H(1)

H[X]

Block local exogenous:

Thus, .wµ = bµ + ⇢µ

H[X]

(a)

hµ

⇢µ

(b)

rµ

bµ

qµ

bµ

(c)

rµ

wµ

(d)

Ephemeral: exists only for the present

Structural information

W (`) = B(`) + T (`)

= (EB �E) + `(bµ + ⇢µ)


Block multivariate mutual information:



0 2 4 6 8 10 12


I

Info

rmat

ion

[bit

s]

I(`)

I(`) = H(`)�X

A2P (`)0<|A|<`

I[XA|XA]

Observations of finite-state process tend to independence:lim`!1

I(`) = 0 iµ = 0 I = 0

I(`) ⇡ I+ `iµAsymptote: ?

hµ > 0 :


Measurement Decomposition in the Process I-diagram:



H[X:0] H[X1:]

H[X0]

rµ

bµbµqµ

�µ


Predictive decomposition:



⇢µ

hµ

H[X0]


Atomic decomposition:



rµ

bµ bµqµ

H[X0]


Structural decomposition:



rµ

wµ

H[X0]


What is ? Not a component of the information in a single observation. Information shared between past and future, not in present! Hidden state information!



H[X:0] H[X1:]

H[X0]

rµ

bµbµqµ

�µ

�µ

Measurement Decomposition in the Process I-diagram ...




H[X:0] H[X1:]

H[X0]

rµ

bµbµqµ

�µ

Measurement Decomposition in the Process I-diagram:

Decomposition of Excess Entropy:

E = bµ + qµ + �µ


Example Processes:Anatomy of a Bit ...


A

B

12 |0

12 |11|1

A

B

12 |1

12 |01|1

C

B

A

E

D

12 |0

12 |1

1|0

1|1

12 |0

12 |1

1|0

Even Golden Mean

Noisy Random Phase Slip


Example: Parametrized Golden Mean Process:



A

B

12 |1

12 |01|1

0.0 0.2 0.4 0.6 0.8 1.0Self loop probability p

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Entr

opy

rate

[bits/

sym

bol

]

rµ

bµ

p

1-p

hµ ⇡ bµ

~ Period 2: Randomness affects phase and so is remembered.

rµ ⇡ 0

hµ ⇡ rµ

~ Period1: Randomness disrupts all 1s, but is not remembered.

bµ ⇡ 0

hµ


Entropy-rate Decomposition: Logistic Map




Entropy-rate decomposition: Tent MapAnatomy of a Bit ...


hµ = log2 a



What is information?

Depends on the question!

Uncertainty, surprise, randomness, .... Compressibility. Transmission rate. “Memory”, apparent stored information, .... Synchronization. Ephemeral. Bound. ...








hµH(X)







hµH(X)R = log2 |A|� hµ








I[X;Y ]








I[X;Y ]E








I[X;Y ]ET








I[X;Y ]ETrµ








I[X;Y ]ETrµbµ



01

1

1 0

...001011101000...

ModellerSystem

A

B

C

ProcessInstrument

!

"

# $1

0

0

00

0

1

1 11

Analysis Pipeline:




01

1

1 0

...001011101000...

ModellerSystem

A

B

C

ProcessInstrument

!

"

# $1

0

0

00

0

1

1 11

1. An information source:a. Dynamical System:

deterministic or stochasticlow-dimensional or high-dimensional (spatial?)

b. Design instrument (partition)2 . Information-theoretic analysis:

a. How much information produced?b. How much stored information?c. How does observer synchronize?

Pr(sL)

H(L)hµ

ET

Calculate or estimate

Analysis Pipeline:




Course summary:

Dynamical systems as sources of complexity:1. Chaotic attractors (State)2. Basins of attraction (Initial conditions)3. Bifurcation sequences (Parameters)

Dynamical systems as information processors:1. Randomness:

Entropy hierarchy: block entropy, entropy convergence, rate, ...2. Information storage:

1. Total predictability2. Excess entropy3. Transient information




Preview of 256B: Physics of Computation

Answers a number of questions 256A posed:

What is a model?What are the hidden states and equations of motion?Are these always subjective, depending on the observer?Or is there a principled way to model processes?To discover states?

What are cause and effect?

To what can we apply these ideas?How much can we calculate analytically?How much numerically?How much from data?




Preview of 256B: Physics of Computation

Punch line:

How nature is organized

is

How nature computes



Reading for next lecture:

CMR articles RURO (Introduction), CAO, and ROIC.

... next quarter!




anatomy of a bit - university of california,...

Documents