09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 1
CSCI 6990.002: Data Compression
09: Scalar Quantization
Vassil Roussev<vassil@ cs.uno.edu>
UNIVERSITY of NEW ORLEANSDEPARTMENT OF COMPUTER SCIENCE
2
Quantization
Definition:The process of representing a large—possibly infinite—set of values with a much smaller set.Example:
Source: Real numbers in the [-10.0, 10.0]
QuantizationQ(x) = ⎣x+0.5⎦[-10.0, -10.0] {-10, -9, …, -1, 0, 1, 2, …, 9, 10}
Scalar vs. vector quantizationScalar: applied to scalarsVector: applied to vectors
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 2
3
The Quantization Process
Two aspectsEncoder mapping
Map a range of values to a codewordIf source is analog A/D converterKnowledge of the source can help pick more appropriate ranges
Decoder mappingMap the codeword to a value in the rangeIf output is analog D/A converterKnowledge of the source distribution can help pick better approximations
Quantizer = encoder + decoder
4
Quantization Example
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 3
5
Quantizer Input-Output Map
6
Quantization Problem Formulation
Input:X – random variablefX(x) – probability density function (pdf)
Output:{bi}i=0..M decision boundaries
{yi}i=1..M reconstruction levels
Discrete processes are often approximated by continuous distributions
E.g.: Laplacian model of pixel differenceIf source is unbounded, then first/last decision boundaries = ±∞
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 4
7
Quantization Error
Mean squared quantization error
( )
( ) dxfyx
dxfxQx
X
M
i
b
b i
Xq
i
i∑∫
∫
=
∞
∞−
−
−=
−=
1
2
22
1
)(σ
iii bxbyxQ ≤<= −1iff)(
Quantization error is a.k.a.Quantization noiseQuantizer distortion
8
Quantization Problem Formulation (2)
Bit rates w/ fixed-length codewordsR = ⎡log2M⎤E.g.: M = 8 R = 3
Quantizer design problemGiven:
input pdf fX(x) & MFind:
decision boundaries {bi} and
Reconstruction levels {yi}Such that:
MSQE is minimized
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 5
9
Quantization Problem Formulation (3)
Bit rates w/ variable-length codewordsR depends on boundary selectionExample
∑ ==
M
i ii yPlR1
)(
∫−
=i
i
b
b Xi dxxfyP1
)()(
∑ ∫=−
=M
i
b
b Xii
i
dxxflR1
1
)(
10
Quantization Problem Formulation (4)
Rate-optimization formulation
Given:
Distortion constraint σq2 ≤ D*
Find:{bi},{yi}, binary codes
Such that:R is minimized
Distortion-optimization formulation
Given:Rate constraint R ≤ R*
Find:{bi},{yi}, binary codes
Such that:
σq2 is minimized
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 6
11
Uniform Quantizer
All intervals of the same sizeI.e., boundaries are evenly spaced (∆)Outer intervals may be an exception
ReconstructionUsually the midpoint is selected
Midrise quantizerZero is not an output level
Midtread quantizer Zero is an output level
12
Midrise vs. Midtread Quantizer
Midrise Midtread
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 7
13
Uniform Quantization of Uniform Source
Input:Uniform [‐Xmax, Xmax]
OutputM‐level uniform quantizer
MXmax2
=∆
1221
2122
22
1)1(
max
22 ∆
=⎟⎠⎞
⎜⎝⎛ ∆
−−= ∑∫
=
∆
∆−
M
i
i
iq dxX
ixσ
14
Alternative MSQE Derivation
Consider quantization error instead:q = x – Q(x)q ∈ [‐∆/2, ∆/2]
∫∆
∆−
∆=
∆=
2
2
222
121 dqqqσ
( ) ( )
dB02.6)2(log20)(log10
212
122log1012
122log10log10)dB(SNR
102
10
2max
2max
102
2max
102
2
10
nM
MX
XX
n
q
s
===
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎟⎠⎞
⎜⎝⎛
=⎟⎟⎠
⎞⎜⎜⎝
⎛∆
=⎟⎟⎠
⎞⎜⎜⎝
⎛=
σσ
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 8
15
Examples (8 1, 2, 3 bits/pixel)
Darkening, contouring & dithering
16
Uniform Quantization of Nonuniform Sources
Example nonuniform source:x ∈ [‐100, 100], P(x ∈ [‐1, 1]) = 0.95
ProblemDesign an 8‐level quantizer
The naïve approach leads to95% of samples values represented by two numbers:
-12.5 and 12.5
Max quantization error (QE)= 12.5Min QE = 11.5 (!)
Consider an alternativeStep = 0.3Max QE = 98.5, however 95% of the time QE < 0.15
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 9
17
Optimizing MSQE
Numerically solvable for specific PDF
18
Example Optimum Step Sizes
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 10
19
QE for 3-bit Midrise Quantizer
20
Overload/Granular Regions
"4"4,stdev
valuegranularmax11 loadingff σ→==
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 11
21
Variance Mismatch Effects
22
Variance Mismatch Effects (2)
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 12
23
Distribution Mismatch Effects
8-level quantizers, SNR
24
Adaptive Quantization
IdeaInstead of a static scheme, adapt to the actual data:
Mean, variance, pdf
Forward adaptive (off-line)Divide source in blocksAnalyze block statisticsSet quantization scheme
Side channel
Backward adaptive (on-line)Adaptation based on quantizer outputNo side channel necessary
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 13
25
Forward Adaptive Quantization (FAQ)
Choosing block sizeToo large
Not enough resolutionIncreased latency
Too smallMore side channel information
Assuming a mean of zeroVariance estimate:
∑ −
= +=1
022 1ˆ N
i inq xN
σ
26
Speech Quantization Example
16-bit speech 3-bit fixed
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 14
27
Speech Quantization Example (2)
16-bit speech 3-bit FAQBlock = 128 samples8‐bit variance quantization
28
FAQ Refinement
So far we assumed uniform pdfRefinement
Assume uniform pdf butRecord min/max values for each block
Example:Sena image8x8 blocks3‐bit quantizationOverhead = 16/8x8 = 0.25 bits/pixel
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 15
29
FAQ Refinement Example
Original: 8 bits/pixel Quantized: 3.25 bits/pixel
30
Backward Adaptive Quantization (BAQ)
ObservationOnly encoder sees inputAdaptation can only be based on quantized output
ProblemHow do we deduce mismatch information from output only?It is possible, if we know the pdf …… and we are very patient
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 16
31
Jayant Quantizer
IdeaIf input falls in the outer levels
Expand step size
If input falls in the inner levelsContract step size
The product of expansions & contraction should be 1
Multipliers: MkIf input Sn‐1 falls in the kth interval, thenstep is multiplied by Mk
Inner Mk < 1, outer Mk > 1 1)1( −− ∆=∆ nnln M
32
3-bit Jayant Quantizer Output Levels
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 17
33
Jayant Example
M0 = M4 = 0.8, M1 = M5 = 0.9M2 = M6 = 1.0, M3 = M7 = 1.2, ∆0 = 0.5Input: 0.1 -0.2 0.2 0.1 -0.3 0.1 0.2 0.5 0.9 1.5
34
Picking Jayant Multipliers
∆min / ∆max to prevent under/overflow
⎣ ⎦
∑∏
∏
∏
∏
==
=
=
=
=⇒=
=>=
==
=
=
M
kkk
M
k
Pl
kkll
k
M
k
kk
P
M
k
Nn
M
k
n
Pl
llMNnPM
M
M
kk
kk
k
k
k
k
k
k
00
0
0
0
01
,1where,
where,1
1
1
γ
γγ
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 18
35
Jayant Example
“Ringing”
36
Jayant Performance
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 19
37
Non-uniform Quantization
Idea: Pick the boundaries such that error is minimized
I.e., smaller/bigger step for smaller/bigger values
E.g.:
38
pdf-optimized Quantization
Problem: Given fX, minimize MSQE:
( ) dxfyx X
M
i
b
b iqi
i∑∫
= −
−=1
22
1
σ
Set derivative w.r.t. yj to zero and solve for yj:
( )
( )dxxf
dxxfxy
j
j
j
j
b
b X
b
b X
j
∫∫
−
−=
1
1
Set derivative w.r.t. bj to zero and solve for bj:( ) 21 jjj yyb += +
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 20
39
Lloyd-Max Algorithm
Observation:Circular dependency b/w bj and yj
Lloyd/Max/Lukaszewics/Steinhaus approach:Solve the two iteratively until an acceptable solution is found
Example:
40
Lloyd-Max Algorithm (2)
Boundaries: { b1, b2, …, bM/2-1}b0 = 0, bM/2‐1 = MAX_INPUT
Reconstruction levels: { y1, y2, …, yM/2-1}
( ) ( )dxxfdxxfxyb
b X
b
b X ∫∫=1
0
1
01
One equation, two unknowns: b1, y1Pick a value for b1 (e.g. b1 = 1), solve for y1 and continue:
112 2 yby += ( ) ( )dxxfdxxfxyb
b X
b
b X ∫∫=2
1
2
12
… and so on until all { bn} and { ym} are found
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 21
41
Lloyd-Max Algorithm (3)
Terminating condition:
12122 2ˆ −− += MMM yby
( ) ( )dxxfdxxfxy M
M
M
M
b
b X
b
b XM ∫∫−−
=2/
12/
2/
12/2
ε≤− 22 ˆMM yy
Else: pick a different b1 & repeat
42
Example: pdf-Optimized Quantizers
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 22
43
Lloyd-Max Quantizer Properties
1. MeanOUTPUT = MeanINPUT
2. VarianceOUTPUT ≤ VarianceINPUT
3. MSQE:
[ ]jj
M
jjxq bxbPy ≤≤−= −
=∑ 1
1
222 σσ
4. If N is a random variable representing QE:
[ ] 2qXNE σ−=
5. Quantizer output and QE are orthogonal:[ ] 0,,,|)( 10 =MbbbNXQE K
44
Mismatch Effects
4-bit Laplacian pdf-optimized quantizer
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 23
45
Companded Quantization (CQ)
Compressor Uniform quantizer Expander
46
CQ Example: Compressor
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 24
47
CQ Example: Uniform Quantizer
48
CQ Example: Expander
09: Scalar Quantization CSCI 6990 Data Compression
Vassil Roussev 25
49
CQ Example: Equivalent Non-uniform Quantizer
50
Problems & Extra
Preparation problems (Sayood 3rd, pp. 270-271)
Minimum: 3, 6Recommended: 5, 7
ExtraBennett integral derivation (pp. 261‐263)Entropy‐encoded quantization /9.7/