introduction to information theorydkumar/soi/introtomtc-part1.pdf · shannonshannons’s...
TRANSCRIPT
Introduction to Information Theory
1
Claude E. Shannon 2
3
A General Communication SystemA General Communication System
4
Shannon’s Information TheoryShannon s Information Theory
• Conceptualization of information & modeling• Conceptualization of information & modeling of information sourcesS di f i f ti th h l• Sending of information across the channel:What are the limits on the amount of information th t b t?that can be sent?What is the effect of noise on this communication. 5
A General Communication SystemA General Communication System
6
A General Communication SystemA General Communication System
CHANNEL
• Information Source• Transmitter• Channel• Receiver• Destination
7
“Th f d l bl f i i“The fundamental problem of communication is that of reproducing at one point either
l i l l dexactly or approximately a message selected at another point.”
From: A Mathematical Theory of Communication, Shannon, 1948.
8
Examples of Communication SystemsExamples of Communication Systems
9
Motivating Noise…Motivating Noise…
XTransmittedSymbol
YReceivedSymbol
0 0
Symbol Symbol1 1
10
Motivating Noise…Motivating Noise…
XTransmittedSymbol
YReceivedSymbol
0 0
Symbol Symbol1 1
P(Y=1|X=0) = fP(Y=0|X=1) = f
P(Y=0|X=0) = 1 ‐ fP(Y=1|X=1) = 1 ‐ f
11
Motivating Noise…Motivating Noise…
f 0 1 10 000f = 0.1, n = ~10,000
1 f0 0
1 ‐ f
f
1 11 ‐ f
12
QuestionQuestion
How can we achieve perfect communication over anHow can we achieve perfect communication over an imperfect, noisy communication channel?
Use more reliable components; Stabilize the environment; Use larger areas; Use power/cooling to reduce thermal noise.
These are all costly solutions.
13
Alternately…Alternately…
How can we achieve perfect communication over an imperfect, p p ,noisy communication channel?
Accept that there will be noise Accept that there will be noise Add error detection and correction Introduce the concepts of ENCODER/DECODER
Information TheoryTheoretical limitations of such systemsTheoretical limitations of such systems
Coding TheoryCreation of practical encoding/decoding systemsCreation of practical encoding/decoding systems
14
Alternately…Alternately…
How can we achieve perfect communication over an imperfect, p p ,noisy communication channel?
Accept that there will be noise Accept that there will be noise Add error detection and correction Introduce the concepts of ENCODER/DECODER
REDUNDANCY IS KEY!
Information TheoryTheoretical limitations of such systemsTheoretical limitations of such systems
Coding TheoryCreation of practical encoding/decoding systemsCreation of practical encoding/decoding systems
15
Shannon’s Insight:Shannon s Insight:
High Reliability → Low Transmission Rateg e ab ty→ o a s ss o ate
I.e. Perfect reliability → Zero Transmission Ratey
For a given level of noise there is an associated rate gof transmission that can be achieved with arbitrarily good reliability.
e.g. Sending a lone T versus THIS…
16
Meaning? What meaning?
“Frequently the messages havemeaning; that is
g g
Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual y p y pentities.
These semantic aspects of communication are irrelevant to the engineering problem.”g g p
From: A Mathematical Theory of Communication Shannon 1948From: A Mathematical Theory of Communication, Shannon, 1948.
17
What is information?What is information?
• Just the physical aspects Shannon 1948• Just the physical aspects…Shannon, 1948• The General Definition of Information…Floridi, 2010.
GDI) σ is an instance of information, understood as semantic content, if and only if:
GDI.1) σ consists of n data, for n ≥ 1;GDI.2) the data are well formed;
) h ll f d d f lGDI.3) the well‐formed data are meaningful.
18
What is information?What is information?
• Just the physical aspects Shannon 1948• Just the physical aspects…Shannon, 1948• The General Definition of Information…Floridi, 2010.
GDI) σ is an instance of information, understood as semantic content, if and only if:
GDI.1) σ consists of n data, for n ≥ 1;GDI.2) the data are well formed;
) h ll f d d f lGDI.3) the well‐formed data are meaningful.
19
Shannon Information is…Shannon Information is…
• UncertaintyCan be measured by counting the number of possible messages.
• SurpriseSome messages are more likely than others.
• DifficultWhat is significant is the difficulty of transmitting the message from one point to the other.
E t• EntropyA fundamental measure of information.
20
Understanding EntropyUnderstanding Entropy
l f d !It is sunny in California today!21
Understanding EntropyUnderstanding Entropy Information is quantified using probabilities. Given a finite set of possible messages, associate a probability with
each message. A message with low probability represents more information than g p y p
one with high probability.
Definition of Information:Definition of Information:
I = log(1/p) ≡ ‐log(p)
Where p is the probability of the messageBase 2 is used for the logarithm so I is measured in bits
22
Example: Information in a coin flipExample: Information in a coin flip
P(HEADS) = ½
I = ‐log(½) = 1 bit
23
Example: Text AnalysisExample: Text Analysis
a 0.06428b 0.01147c 0.02413d 0.03188e 0.10210f 0.01842
0 01543
ab
c
dSPC
g 0.01543h 0.04313i 0.05767j 0.00082k 0.00514l 0.03338m 0.01959
e
fu
vw
x yz
n 0.05761o 0.06179p 0.01571q 0.00084r 0.04973s 0.05199t 0 07327
g
ht
u
t 0.07327u 0.02201v 0.00800w 0.01439x 0.00162y 0.01387z 0.00077
i
j
k
lm
nop
r
s
24
SPC 0.20096o
q
Example Text AnalysisExample Text Analysis
25
Example Text AnalysisExample Text AnalysisLetter Freq. I, h(pi)
a 0.06428 3.95951b 0.01147 6.44597c 0.02413 5.37297d 0.03188 4.97116e 0.10210 3.29188f 0.01842 5.76293g 0.01543 6.01840h 0.04313 4.53514i 0.05767 4.11611j 0.00082 10.24909k 0.00514 7.60474l 0.03338 4.90474m 0.01959 5.67385m 0.01959 5.67385n 0.05761 4.11743o 0.06179 4.01654p 0.01571 5.99226q 0.00084 10.21486r 0.04973 4.32981s 0 05199 4 26552s 0.05199 4.26552t 0.07327 3.77056u 0.02201 5.50592v 0.00800 6.96640w 0.01439 6.11899x 0.00162 9.26697
26
y 0.01387 6.17152z 0.00077 10.34877
SPC 0.20096 2.31502
Definition of EntropyDefinition of Entropy
Information (I) is associated with knownInformation (I) is associated with known events/messagesEntropy (H) is the average information w r toEntropy (H) is the average information w.r.to all possible outcomes
27
Example Text AnalysisExample Text AnalysisLetter Freq. I, h(pi)
a 0.06428 3.95951b 0.01147 6.44597c 0.02413 5.37297d 0.03188 4.97116e 0.10210 3.29188f 0.01842 5.76293g 0.01543 6.01840h 0.04313 4.53514i 0.05767 4.11611j 0.00082 10.24909k 0.00514 7.60474l 0.03338 4.90474m 0.01959 5.67385n 0.05761 4.11743o 0.06179 4.01654p 0.01571 5.99226q 0.00084 10.21486r 0.04973 4.32981s 0.05199 4.26552t 0.07327 3.77056u 0.02201 5.50592v 0.00800 6.96640w 0.01439 6.11899x 0.00162 9.26697y 0.01387 6.17152
28
y 0.01387 6.17152z 0.00077 10.34877
SPC 0.20096 2.31502
Entropy (2 outcomes)Entropy (2 outcomes)
29
Entropy: PropertiesEntropy: Properties
Entropy is maximized if p is uniform. 30
ReferencesReferences• Eugene Chiu, Jocelyn Lin, Brok Mcferron, Noshirwan Petigara, Satwiksai
Seshasai: Mathematical Theory of Claude Shannon: A study of the style and t t f hi k t th i f i f ti th MIT 6 933J /context of his work up to the genesis of information theory. MIT 6.933J /
STS.420J The Structure of Engineering Revolutions• Luciano Floridi, 2010: Information: A Very Short Introduction, Oxford
University Press, 2011.• Luciano Floridi, 2011: The Philosophy of Information, Oxford University Press,
2011.• James Gleick, 2011: The Information: A History, A Theory, A Flood, Pantheon
Books 2011Books, 2011.• David Luenberger, 2006: Information Science, Princeton University Press,
2006.• David J.C. MacKay, 2003: Information Theory, Inference, and Learning
Algorithms, Cambridge University Press, 2003.• Claude Shannon & Warren Weaver, 1949: The Mathematical Theory of
Communication, University of Illinois Press, 1949.• W N Francis and H Kucera: Brown University Standard Corpus of Present‐DayW. N. Francis and H. Kucera: Brown University Standard Corpus of Present Day
American English, Brown University, 1967.
31
32