lecture 27: entropy and information - zhejiang...
TRANSCRIPT
![Page 1: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/1.jpg)
General Physics IGeneral Physics I
Lecture 27: Entropy and Lecture 27: Entropy and InformationInformation
![Page 2: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/2.jpg)
OutlineOutline
Introducing entropy The meaning of entropy
– Reversibility
– Disorder
– Information
Selected topics on information
![Page 3: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/3.jpg)
1st & 2nd Laws of Thermodynamics1st & 2nd Laws of Thermodynamics
The 1st law specifies that we cannot get more energy out of a cyclic process by work than the amount of energy we put in.
The 2nd law states that we cannot break even because we must put more energy in, at the higher temperature, than the net amount of energy we get out by work.
WQU
h
c
h
c
h T
T
Q
Q
Q
W 11 carnot
![Page 4: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/4.jpg)
Carnot’s EngineCarnot’s Engine
![Page 5: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/5.jpg)
Efficiency of a Carnot EngineEfficiency of a Carnot Engine
All Carnot engines operating between the same two temperatures have the same efficiency.
![Page 6: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/6.jpg)
An EqualityAn Equality
Now putting in the proper signs,
0c
c
h
h
T
Q
T
Q
0CycleCarnot
T
dQ
negativepositive
![Page 7: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/7.jpg)
A Sum of Carnot CyclesA Sum of Carnot Cycles
0C T
dQ
V
Padiabats Th,i
Tc,i
Any reversible process can be approximated by a sum of Carnot cycles, hence
0,
,
,
, i ic
ic
ih
ih
T
Q
T
Q
![Page 8: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/8.jpg)
Clausius Definition of EntropyClausius Definition of Entropy
Entropy is a state function, the change in entropy during a process depends only on the end points and is independent of the actual path followed.
T
dQdS reversible
012,21, 21
CCC
dSdSdS
21,12,21,
12
221 CCC
dSdSdSSS1
2
C1
C2
![Page 9: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/9.jpg)
Return to Inexact DifferentialReturn to Inexact Differential
12ln)2,2(
)2,1(
)2,1(
)1,1(
dy
y
xdx
0lnln),( fyxyxf
2ln21)2,2(
)1,2(
)1,2(
)1,1(
dy
y
xdx
dyy
xdxdg Assume
Note: is an exact differential.
Integrating factor
y
dy
x
dx
x
dgdf
![Page 10: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/10.jpg)
Back to the First LawBack to the First Law
Heat is path dependent.
Therefore, 1/T is really the integrating factor for the differential form of heat. Now we can recast the 1st law of thermodynamics as
Entropy is also a state function, as is the internal energy or volume.
PdVdUdQ
PdVTdSdU
![Page 11: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/11.jpg)
Entropy of an Ideal Gas (1 mole)Entropy of an Ideal Gas (1 mole)
TfR
TCTU molV 2
)(
000 lnln),(
V
VR
T
TCSVTS mol
V
V
RTVTp ),(
V
RdV
T
dTCpdVdU
TdS
molV
1
Integrating from (T0,V0) to (T, V)
![Page 12: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/12.jpg)
Carnot’s TheoremCarnot’s Theorem
No real heat engine operating between two energy reservoirs can be more efficient than Carnot’s engine operating between the same two reservoirs.
0''
1'
'1'
c
c
h
h
h
c
h
c
T
Q
T
Q
T
T
Q
Qe
0 dS
What does this mean? Still, for any engine in a cycle (S is a state function!)
negativepositive
![Page 13: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/13.jpg)
Counting the Heat Baths inCounting the Heat Baths in
h
hh T
QS
'
0 dSSgas
c
cc T
QS
'
after a cycle
Q'h > 0
Q'c < 0
0'
0'
c
c
h
hcgash T
Q
T
QSSSS
![Page 14: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/14.jpg)
Counting the Heat Baths inCounting the Heat Baths in
h
hh T
QS
'
0 dSSgas
c
cc T
QS
'
after a cycle
Q'h > 0
Q'c < 0
The total entropy of an isolated system that undergoes a change can never decrease.
![Page 15: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/15.jpg)
Example 1: Clausius StatementExample 1: Clausius Statement
hh T
QS
cc T
QS
0ch
ch T
Q
T
QSSS
Irreversible!
![Page 16: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/16.jpg)
Example 2: Kelvin StatementExample 2: Kelvin Statement
0T
QS
Irreversible!
![Page 17: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/17.jpg)
Specific HeatSpecific Heat
Note: Last time we defined molar specific heat. In physics, we also use specific heat per particle.
![Page 18: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/18.jpg)
Example 3: Mixing WaterExample 3: Mixing Water
TA
A
TB
BQTA TB T T
Q
TA< TB
BA
BBAA
B
A
B
A
mm
TmTmT
TT
TT
m
m
AAA TTcmQTT :
TTcmQTT BBB :
![Page 19: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/19.jpg)
Example 3: Mixing WaterExample 3: Mixing Water
Irreversible!
TA
A
TB
BQTA TB T T
Q
TA< TB
0ln2
BABA TT
TcmSSS
0ln:
AA
T
T
AAA T
TcmT
dTcmSTT
A
2/, BABA TTTmmm
0ln:
BB
T
T
BBB T
TcmT
dTcmSTT
B
For simplicity, assume
![Page 20: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/20.jpg)
The Second Law in terms of EntropyThe Second Law in terms of Entropy
The total entropy of an isolated system that undergoes a change can never decrease.
– If the process is irreversible, then the total entropy of an isolated system always increases.
– In a reversible process, the total entropy of an isolated system remains constant.
The change in entropy of the Universe must be greater than zero for an irreversible process and equal to zero for a reversible process.
ΔSSUniverse ≥ 0
![Page 21: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/21.jpg)
Order versus DisorderOrder versus Disorder
Isolated systems tend toward disorder and that entropy is a measure of this disorder.
Ordered: all molecules on the left side
Disordered: molecules on the left and right
![Page 22: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/22.jpg)
We can only calculate S with a reversible process! In this case, we replace the free expansion by the isothermal process with the same initial and final states.
Example 4: Free ExpansionExample 4: Free Expansion
00 SWQU
0ln
i
fV
V
V
V
V
V VV
nRV
nRdV
T
PdV
T
dQS
f
i
f
i
f
i
?
Irreversible!
![Page 23: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/23.jpg)
Entropy: A Measure of DisorderEntropy: A Measure of Disorder
2lnln Bi
fB NkV
VNkS
WkS B ln
N
m
ff V
VW
N
m
ii V
VW
N
i
f
i
f
VV
WW
We assume that each molecule occupies some microscopic volume Vm.
suggesting (Boltzmann)
![Page 24: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/24.jpg)
Information, An ExampleInformation, An Example
Consider the following three statements.1. Newton’s birthday falls on a particular day of the year. (P = 1)
2. Newton’s birthday falls in the second half of the year. (P = 1/2)
3. Newton’s birthday falls on the 25th of a month. (P = 12/365)
As P decreases, the information content increases.
Combining 2 and 3, P =1/2 * 12/365 = 6/365.
It is natural to assume information content is additive. The information content Q of a statement (with probability P) can then be defined by
Q = −log P
![Page 25: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/25.jpg)
Information and EntropyInformation and Entropy
(1927) Bell Labs, Ralph Hartley– Measure for information in a message– Logarithm: 8 bit = 28 = 256 different numbers
(1948) Bell Labs, Claude Shannon– “A mathematical theory of communication”– Probability of a particular message– Average information ~ entropy?
i
ii PPS loginfomation
Shannon entropy
![Page 26: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/26.jpg)
Shannon EntropyShannon Entropy
“It is already in use under that name. … and besides, it will give you great edge in debates because nobody really knows what entropy is anyway.”
– John von Neumann
![Page 27: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/27.jpg)
Entropy of A Bernoulli Trial Entropy of A Bernoulli Trial
Consider the simplest case:
Bernoulli trial –
a two-outcome random
variable with probability
p and 1-p of the two
outcomes.
S = −∑iPi log Pi = − p log p − (1−p)log (1−p)
P
S
![Page 28: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/28.jpg)
The Entropy of MixingThe Entropy of Mixing
Consider two different ideal gases (call them 1 and 2) which are in separate vessels with volume xV and (1-x)V respectively at the same pressure p and temperature T. If the pipe that connects the two vessels is opened, the gas will spontaneously mix, resulting an increase in entropy.
Δ S = xNkB∫xV
V dV 1V 1
+ (1−x)NkB∫(1−x)V
V dV 2V 2
= −NkB [x ln x + (1−x ) ln(1−x )]
Distinguishability is an important piece of information!
![Page 29: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/29.jpg)
Entropy, from Information ViewpointEntropy, from Information Viewpoint
Entropy is a measure of our uncertainty of a system, based on our limited knowledge of its properties and ignorance about which of its microstates it is in.
In making inferences on the basis of partial information, we can assign probabilities on the basis that we maximize entropy subject to the constraints provided by what is known about the system.
Exercise: Maximize entropy S = - kB Si Pi ln Pi subject to 1 = Si Pi and U = Si Ei Pi. What do you expect?
![Page 30: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/30.jpg)
SolutionSolution
Using the method of Lagrange multipliers, in which we maximize
S /kB − α (∑iPi−1) − β (∑i
P iEi−U )
where a and b are Lagrange multipliers. We vary this expression with respect to one of the probability P
j and get
∂∂P j
(∑i−Pi ln Pi − α Pi − β PiE i ) = 0
so that −ln P j − 1 − α − β E j = 0
This can be rearranged to give P j =e−β E j
e1+α
Boltzmann probability
![Page 31: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/31.jpg)
Selected TopicsSelected Topics
Maxwell’s demon Data compression Relative entropy and cross entropy Quantum information
![Page 32: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/32.jpg)
I. Maxwell’s DemonI. Maxwell’s Demon
To determine whether to let a molecule through, the demon must acquire information about the state of the molecule. However well prepared, the demon will eventually run out of information storage space and must begin to erase the information it has previously gathered. Erasing information is a thermodynamically irreversible process that increases the entropy of a system.
![Page 33: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/33.jpg)
Information is PhysicalInformation is Physical
Consider a device which has stored 1 bit of information and is connected to a thermal reservoir of temperature T.
The bit can be either 1 or 0. We can erase it by setting the bit to zero.
– Before erasure, p1 = 1/2, p2 = 1/2. S = log 2.
– After erasure, p1 = 1, p2 = 0. S = 0.
– Erasure is irreversible. Entropy goes down by kB ln 2.
For the total entropy of the universe not to decrease, the entropy of the surroundings must go up by at least kB ln 2 and we must dissipate heat in the surroundings.
![Page 34: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/34.jpg)
Landauer’s Principle & VerificationLandauer’s Principle & Verification
Computation needs to involve heat dissipation only when you do something irreversible with the information.
Lutz group (2012)
693.02ln Tk
Q
B
![Page 35: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/35.jpg)
For Those Who Are InterestedFor Those Who Are Interested
Reading (downloadable from my website): – Charles Bennett and Rolf Landauer, The fundamental
physical limits of computation.
– Antoine Bérut et al., Experimental verification of Landauer’s principle linking information and thermodynamics, Nature (2012).
– Seth Lloyd, Ultimate physical limits to computation, Nature (2000).
Dare to adventure where you have not been!
![Page 36: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/36.jpg)
II. Data CompressionII. Data Compression
Information must be stored/transmitted. Compressible?
Example: Classical information is stored in the form of binary digits 0 (with probability p) and 1 (with probability 1-p). Suppose p = 0.9, the data contain less information than in the case of p = 0.5.
Two-bit encoding:
00 → 0 p2 = 0.81
10 → 10 p(1-p) = 0.09
01 → 110 p(1-p) = 0.09
11 → 1110 (1-p)2 = 0.01
Average length: 0.81 + 2*0.09 + 3*0.09 + 4*0.01 = 1.3
Code efficiently the typical sequence!
![Page 37: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/37.jpg)
Generalization to n-bit EncodingGeneralization to n-bit Encoding
The probability of finding a sequence x1, …, x
n is
P(x1 , x2 , ... , xn) = P(x1)P(x2)...P(xn) ≈ pnp (1−p)n(1− p)
The information content is
−log P(x1 , x2 , ... , xn) = −np log p−n(1−p) log (1−p) = nS
where S is the entropy for a Bernoulli trial with probability p. Hence
P(x1 , x2 , ... , xn) =1
2nS
This shows that there are at most 2nS typical sequences and hence it only requires nS bits to code them.
![Page 38: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/38.jpg)
Noiseless Channel Coding TheoremNoiseless Channel Coding Theorem
A compression algorithm will take a typical sequence of n terms x1, …, xn and turn them into a string of length nR. Hence the smaller R is, the greater the compression.
If we have a source of information with entropy S, and if R > S, then there exists a reliable compression scheme of compression factor R. Conversely, if R < S, then any compression scheme will not be reliable.
The entropy S sets the ultimate compression limit on a set of data.
![Page 39: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/39.jpg)
III. Relative EntropyIII. Relative Entropy
The Kullback–Leibler divergence (also called relative entropy) is a measure of how one probability distribution diverges from a second, expected probability distribution.
– The KL divergence is always non-negative (Gibbs’ inequality), with DKL = 0 if and only if P = Q almost everywhere.
– KL divergence and cross entropy can be used to define the loss function in machine learning and optimization. The true probability P is the true label, and the given distribution Q is the predicted value of the current model.
DKL(P||Q) = −∑iPi log
QiPi
= −∑iPi logQi−(−∑i
Pi log Pi )
cross entropy (Shannon) entropy
![Page 40: Lecture 27: Entropy and Information - Zhejiang Universityzimp.zju.edu.cn/~xinwan/courses/physI19/handouts/lecture... · 2019-06-17 · Entropy, from Information Viewpoint Entropy](https://reader034.vdocuments.site/reader034/viewer/2022042219/5ec543eb2ce9c46c4f070288/html5/thumbnails/40.jpg)
IV. Quantum InformationIV. Quantum Information
Quantum information is composed of quantum bits (known as qubits), which are two-level quantum systems that can be represented by linear combinations of the states |0 and |1.
Quantum entanglement: no classical counterpart
– Bell state
– Entanglement entropy
– Information transmission faster than the speed of light? No!
Quantum no-cloning theorem: It is impossible to make a copy a non-orthogonal quantum mechanical states.
(|01 ⟩+|10 ⟩)/√2