344 ieee transactions on information theory, vol. … · codewords deﬁne a quantizer decoder....

344 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 54, NO. 1, JANUARY 2008

Quantization as Histogram Segmentation: OptimalScalar Quantizer Design in Network Systems

Dan Muresan and Michelle Effros, Senior Member, IEEE

Abstract—An algorithm for scalar quantizer design on dis-crete-alphabet sources is proposed. The proposed algorithm canbe used to design fixed-rate and entropy-constrained conventionalscalar quantizers, multiresolution scalar quantizers, multiple de-scription scalar quantizers, and Wyner–Ziv scalar quantizers. Thealgorithm guarantees globally optimal solutions for conventionalfixed-rate scalar quantizers and entropy-constrained scalar quan-tizers. For the other coding scenarios, the algorithm yields thebest code among all codes that meet a given convexity constraint.In all cases, the algorithm run-time is polynomial in the size of thesource alphabet. The algorithm derivation arises from a demon-stration of the connection between scalar quantization, histogramsegmentation, and the shortest path problem in a certain directedacyclic graph.

Index Terms—Optimal design, multiple descriptions, multireso-lution, scalar quantizer, successive refinement, Wyner–Ziv.

I. INTRODUCTION

ANY lossy source code can be described as a quantizer fol-lowed by a lossless code. The quantizer maps the space

of possible data values to a collection of allowed reproductionvalues, and the lossless code maps the space of allowed repro-duction values to some uniquely decodable family of binary de-scriptions. This paper treats the problem of quantizer design.

A quantizer consists of a set of reproduction values and amapping from the source alphabet to the reproductions. Eachreproduction value is called a codeword, and the set of repro-ductions is called a codebook. The set of alphabet symbols thatare mapped to a certain codeword is called the codecell associ-ated with that codeword. The set of all codecells forms a par-tition of the source alphabet. From a communications systempoint of view, the codecells define a quantizer encoder, and thecodewords define a quantizer decoder. Together, the quantizerencoder and decoder define the quantizer. We focus on scalarquantizers, where the codewords are scalar source reproductionvalues and the codecells partition the scalar alphabet.

In designing a quantizer, one can first design the decoder (thecodewords) and then define the encoder (the codecells) with re-spect to the decoder, or one can design the encoder and then

Manuscript received October 19, 2004; revised September 15, 2007. The ma-terial in this paper was presented at The Data Compression Conference, Snow-bird, UT, March 2002.

D. Muresan was an undergraduate at the California Institute of Technology,Pasadena, and an M.S. student at Stanford University, Stanford, CA, when thiswork was performed (e-mail: [email protected]).

M. Effros is with the Department of Electrical Engineering, MC 136-93, Cal-ifornia Institute of Technology, Pasadena, CA 91125 USA (e-mail: [email protected]).

Communicated by V. A. Vaishampayan, Associate Editor At Large.Digital Object Identifier 10.1109/TIT.2007.911170

define the decoder with respect to the encoder. Most design al-gorithms take what we call a “codewords-first” approach—fo-cusing on codebook design and defining the codecells as func-tions of the codebook. In this paper, we explore the alternativeapproach.

Our main contribution is a low-complexity scalar quantizerdesign algorithm for finite-alphabet sources. (The algorithm canalso be used to design scalar quantizers for continuous-alphabetsources using either a finite training set or a discretization ofthe continuous alphabet.) The algorithm applies to a varietyof types of scalar quantizers and in some cases guarantees aglobally optimal design. The code types treated include fixed-and variable-rate (entropy-constrained) scalar quantizers forscenarios with a single encoder and one or more decoders.Examples include conventional scalar quantizers, multiresolu-tion scalar quantizers, multiple-description scalar quantizers,Wyner–Ziv scalar quantizers, and any combination of these.

In each coding scenario, the algorithm finds the best codeamong all possible codes with convex codecells. Under mildconstraints on the distortion measure, there always exists aglobally optimal conventional scalar quantizer with convexcodecells; thus, our algorithm yields globally optimal fixed- andvariable-rate conventional scalar quantizers. In multiresolution,multiple-description, and Wyner–Ziv scalar quantization, theremay or may not exist a globally optimal code with convexcodecells; thus, our algorithm does not guarantee a globallyoptimal solution in these cases.

The runtime for our algorithm is polynomial in the size ofthe source alphabet. The order of the polynomial varies withthe coding scenario. In addition to the optimal convex-codecelldesign, we also present a fast variation on the code design al-gorithm; the fast algorithm improves runtime (at the expenseof some rate-distortion performance) by reducing the effectivealphabet size. (That such a complexity savings is possible isnot immediately obvious since direct alphabet reduction is sub-optimal in general.)

Previous work in low-complexity, globally optimal scalarquantizer design treats fixed-rate, conventional (single-encoder,single-decoder) scalar quantizers for finite-alphabet sources. Infixed-rate coding, the number of codewords is fixed, all code-words are described at the same rate, and the design goal is tominimize the code’s expected distortion with respect to a givenprobability mass function (pmf) or training set. In his 1964doctoral dissertation [1], Bruce describes a polynomial-timedynamic programming algorithm for globally optimal fixed-rateconventional scalar quantizer design. In [2], Sharma showshow to reduce the complexity of Bruce’s algorithm by usingFibonacci heaps. Wu and Zhang [3], [4] further refine the

0018-9448/$25.00 © 2008 IEEE

MURESAN AND EFFROS: QUANTIZATION AS HISTOGRAM SEGMENTATION 345

dynamic programming algorithm by incorporating a number ofother optimization techniques.

Our results, which first appeared in [5]–[7], generalize theearlier techniques to allow variable-rate scalar quantizer designand to treat quantizers with more than one decoder and quan-tizers with side information at the decoder. In particular, we:

• develop a polynomial-time algorithm for designing glob-ally optimal entropy-constrained scalar quantizers, givingthe optimal tradeoff between rate and distortion for scalarcodes;

• generalize the dynamic-programming strategy to a familyof single-source multiple-receiver and side-informa-tion source coding applications including fixed- andvariable-rate conventional, multiresolution, and mul-tiple-description scalar quantizers with or without decoderside information—giving a single strategy that for eachscenario yields codes that are optimal with respect to theconvex codecell constraint;

• clarify the connection between quantization, segmentation,and the shortest path problem in a certain directed acyclicgraph; and

• derive conditions under which the optimal convex-codecellcode fails and succeeds in guaranteeing the same solutionas the corresponding unconstrained optimization.

In [8], Dumitrescu and Wu describe an algorithm designedto minimize the weighted sum of distortions of all subtreesof a fixed-depth tree-structured scalar quantizer; the distortionof each subtree with the same number of leaves is weightedequally in the optimization. Like our algorithm (which wasfirst presented in [5]), that algorithm applies a dynamic pro-gramming approach to convex-codecell multiresolution codedesign; the unusual performance measure in [8] makes itdifficult to compare the outcomes of the two strategies.1 In[9], Dumitrescu and Wu describe an algorithm similar to theentropy-constrained multiresolution code design algorithm pre-sented in [5], [7]. The same authors later describe a fixed-ratemultiresolution design algorithm in [10]. In [11]–[13], theyconsider fixed-rate multiple-description scalar quantizer designfor the special case where there are only two descriptions andthose descriptions have the same rate. Like this paper, all of thepapers of Dumitrescu and Wu are restricted to convex-codecelldesign.

The remainder of the paper is organized as follows. Section IIsets up the problem and gives a precise description of the convexcodecell constraint. Sections III–VI describe the code design al-gorithm for conventional, multiple-description, multiresolution,and side-information scalar quantizers. Each section describesboth the optimal algorithm and its fast variation and includes aproof of the algorithm’s optimality (subject to the convex-code-cell constraint) when the algorithm is applied to the full sourcealphabet. Section VII derives the algorithm runtime for eachscenario. Section VIII summarizes prior results and derives newtheorems on codecell convexity. Section IX contains experi-mental results.

1By [8, Proposition 1], their performance measure can be made to mimic oursonly for a very restricted collection of parameters.

II. PROBLEM SETUP

Each code design aims to optimize the tradeoff between ex-pected rates and distortions. The distortion measure isassumed to be nonnegative and the source description is re-quired to be uniquely decodable. The expectation is taken rel-ative either to a known pmf on a finite alphabet or to a finitetraining set on an arbitrary alphabet. (In some special cases, wecan also approximate the optimal codes for continuous alphabetsources; we treat these cases under our discussion of fast ap-proximations.) Given this effective restriction to finite alphabetsand the assumption that the alphabet is ordered, in the remainderof this work we refer to scalar source alphabetcontaining symbols by the symbol indices .The full alphabet is written , and the histogram de-scribing the pmf is denoted by .

Once either a quantizer encoder or a quantizer decoder isdesigned, optimization of the remaining portion is straightfor-ward. We focus on optimal encoder design for conventional,multiresolution, multiple-description, and Wyner–Ziv scalarquantizers. In each of these scenarios, designing an encoderis equivalent to designing one or more partitions on thesource alphabet . Each element of partition

of describes a collection ofsource symbols given the same binary description (a codecell).A conventional scalar quantizer encoder requires one partition.An -description scalar quantizer encoder uses partitions,one for each description. An -resolution scalar quantizerencoder similarly requires partitions.

Codecell Convexity

Scalar quantizer has convex codecells if for each encoderpartition of there exists an increasing se-

quence of natural numbers with ,and . The scalar quantizer design algorithm pro-posed in this work finds, in each coding scenario, the optimalscalar quantizer among all scalar quantizers with convex code-cells.

Fast Approximations

We consider both true partition optimization relative to sourcealphabet and approximate optimization where par-titions are constrained to a coarse grid. Prior work on locallyoptimal code design for such quantized source distributions ap-pears in [14].

We achieve the coarse grid by dividing the sym-bols of into convex, nonoverlapping cells

with . Thefast approximation algorithm finds the best partition on thisfixed, ordered set of indivisible cells. Using this coarse grid,the codecell comprising cells is

Example 1: Given source alphabet , let. This breaks the source alphabet

into four indivisible cells:


,. The search for an optimal segmen-

tation on symbols is approximated by a searchfor an optimal partition on cells . One possiblepartition on this coarse grid is . Thecorresponding codecells are and

, giving partition onthe original alphabet.

While there are convex-codecell partitions of the orig-inal alphabet, only of these are compatible with the -cellgrid. Thus, guaranteeing optimality requires (with eachalphabet symbol being allotted a separate grid cell), but using

allows for faster code design. All results in this workare described in terms of the most general case ; foroptimal convex-codecell design, is required.

It is tempting to believe that a fast approximation canbe achieved by replacing pmf on alphabet

by a new pmf on some alphabetwith the hopes that the best convex-codecell

scalar quantizer for is an optimal fast ap-proximation for the best convex-codecell scalar quantizer for

. For example, we might set

and

While code design for is faster than code de-sign for , the best code forfails to achieve the best convex-codecell scalar quantizer con-sistent with our grid cells in general. The impediment is thatfor some distortion measures, we cannot assign a single sourcesymbol to each of the grid cells in a manner that preservesthe accuracy of the distortion calculations. (Code design forthe squared-error distortion measure is a notable exception, asshown in [14].) Thus, it is sometimes necessary to use the fullsource alphabet when evaluating distortions. In cases where thesource alphabet can be replaced by an -symbol coarse al-phabet or cases where the distortion can be calculated analyt-ically, the coarse grid approach is useful for approximating op-timal code design for continuous alphabet sources.

Segmentation and Quantization

The performance of a scalar quantizer depends on the sourcevalues and probabilities of the elements in each codecell. Underthe convexity assumption, each codecell is a single segment ofthe source alphabet. We therefore view the problem of code de-sign as a signal segmentation problem on pmf .We want to design a segmentation of thatachieves an optimal tradeoff between expected rates and ex-pected distortions, and we want the design algorithm to be fast.

Optimal signal segmentation can be solved as a single-sourceshortest path problem in a weighted directed acyclic graph(WDAG). The WDAG is designed so that each possible seg-ment is represented by a single edge in the WDAG, and eachpath from the graph’s source node to its terminal node repre-sents a distinct segmentation.

We demonstrate how to build a WDAG for each type of scalarquantizer. We call this WDAG the partial RD graph since itdescribes the rate and distortion contributions made by eachpotential codecell. For conventional scalar quantizer design,the graph vertices are , the graph edges are

, the weight of edge equals thecontribution to the rate–distortion performance made by code-cell , and each pathfrom node to node represents a distinct partition

of the source alphabet. Formultiple-description and multiresolution scalar quantizers, thegraph vertices and edges become a bit more complicated (seethe corresponding sections for details), but there is again aone-to-one correspondence between the paths from the graph’ssingle source to its single sink and the partitions that define thegiven form of scalar quantizer. In each scenario, we prove thatoptimal convex-codecell scalar quantizer design is equivalentto solving a single-source shortest path problem on that graph.

A number of authors have previously exploited the relation-ship between segmentation and shortest path problems to ob-tain rate–distortion optimality in source codes that rely on seg-mentation. In [15], Chou and Lookabough use dynamic pro-gramming to segment a source sequence for a variable-rate codewith codewords of varying dimensions (“variable-to-variable”length coding). Effros, Chou, and Gray take a similar approachto signal segmentation for variable-dimension weighted uni-versal coding in [16]. In [17], Xiong, Herley, Ramchandran, andOrchard use a dynamic programming algorithm for signal seg-mentation in a source code with time-varying wavelet packets;this algorithm is generalized and refined in [18]. Schuster andKatsaggelos use the directed acyclic graph (DAG) shortest pathalgorithm to separate objects from background for video codingin [19]. In each of these examples, segmentation is part of the en-coding procedure, and the optimal segmentation is the segmen-tation that yields the best tradeoff between rate and distortion inthe resulting code. These results differ fundamentally from ourapproach since they segment the sequence of source observa-tions during encoding while we segment the source pmf duringcode design.

Single-Source Shortest Path Algorithms

Once we have built the partial RD graph, we use standardtechniques to solve the shortest path problem on that WDAG.As outlined in [20], the single-source shortest path problem in aWDAG with vertices and edges can be solved intime. The algorithm relies on the observation that if a shortestpath from to stops at intermediate vertex , then its pathfrom to must also be a shortest path. Using this observa-tion, the algorithm iteratively calculates a shortest path fromto for every . Iteration “relaxes” (improves) the shortestpath estimate to each vertex with by comparing thebest path found so far to the path that traverses a shortest pathfrom to and then the edge .

The dynamic programming algorithm in [18] (not presentedexplicitly as a graph algorithm in that paper) has the same com-plexity but is simpler and more natural in that no relaxation isinvolved. Rather, in the th step, a shortest path to the th vertex


is computed (definitively) based on previous shortest paths. Pre-cisely, a shortest path from to must be a shortest path from

to followed by the edge . Comparing thesealternatives (and relying on the previous calculation of a shortestpath from to for all ) allows the precise solution tothe shortest path problem from to .

The first step in both algorithms is to order the vertices ofthe graph in a sequence such that if , then thereare no arcs from to (this is always possible in an acyclicgraph since the edge set induces a partial order on the vertexset).

The following algorithm takes a graph as its input and pro-duces a topological order on the vertices of that graph in

steps. We number the vertices according to the order inwhich they are placed in some sorted list , which is initializedto be empty. For each vertex , keep a counter , initializedwith the in-degree of (the number of incoming arcs). We willdecrement each time one of its predecessors is added to .When becomes , vertex becomes eligible for addition tolist since all of its predecessors have already been numbered.We therefore maintain a stack of “free vertices” initializedwith the set of vertices with in-degree zero. At each time ,remove a vertex from stack , append it to the sorted sequence

, and decrement for all successors of . Whenever somebecomes zero, place on . The algorithm stops when

contains all vertices; if becomes empty at any point beforehas filled up, the graph has a cycle. This problem cannot arise

in our algorithm since our graphs are acyclic by construction;thus, our algorithm is guaranteed to run to completion.

Given the sorted vertex sequence , the dynamic pro-gramming algorithm proceeds as follows. At each step

, compute the weight of a shortest path to vertex as

where is the weight on arc and the minimizationis performed over the set of indices for which is inour edge set; if there are no such indices . Since thesequence is sorted topologically, if is an edge,then and is known when we calculate .

III. CONVENTIONAL SCALAR QUANTIZATION

We begin by considering the conventional source coding par-adigm, where a single encoder describes information to be in-terpreted by a single decoder. Since the solution to the fixed-rateproblem appears in [1]–[4], this section treats only variable-ratecode design.

Optimization Criterion

As discussed in Section II, we describe the encoder of a scalarquantizer by the partition that defines its codecells. Letand be the expected distortion and output entropy, re-spectively, for the variable-rate scalar quantizer correspondingto partition . (Measuring rate as entropy in variable-rate codesseparates the code design from the specific entropy code im-plementation.) For any partition achieving a point on the oper-ational distortion-rate function

, the corresponding variable-rate scalar quantizer is optimalin the sense that no other partition can achieve a lower distortion

Fig. 1. Partial RD graph for an alphabet with N̂ = 3 cells. The path describingthe partition f(0; 2]; (2;3]g appears in bold.

using the same or lower entropy. For any withlying on the lower convex hull of , there exists a La-grangian multiplier for which minimizes

(1)

We therefore use the Lagrangian as the optimizationcriterion for our partition design. The resulting code is called anentropy-constrained scalar quantizer (ECSQ) [21].

The key observation is that rate and distortion (and thus theLagrangian cost) are additive over codecells. That is, for a parti-tion and . Here

is called the partial rate of codecell,2 and is called the partial

distortion of codecell ; and are theprobability and optimal codeword for codecell . (The optimalcodeword is the reproduction value that minimizes ; thisvalue is not necessarily in the finite source alphabet. For ex-ample, when , the optimal codeword isthe centroid of codecell .) Theadditivity of our Lagrangian performance measure overcodecells is critical for defining the partial RD graph.

Partial RD Graph

The partial RD graph for ECSQ is shown in Fig. 1. Thevertex set is . For every vertex pair with

there is an arc (directed edge) from to . The arccorresponds to a codecell comprising coarse alphabetcells . The weight of arc is theLagrangian cost for codecell .

Each path from nodeto node represents a distinct partition

of the source alphabet. The total weight of pathis

Thus, the shortest path from node to node describes thepartition with the optimal Lagrangian performance. Before pro-ceeding to a formal proof of this assertion, we illustrate theseideas with an example.

2All logarithms in this paper are base-2.


Fig. 2. The 1-DSQ (description scalar quantization) partial RD graph for thesource in Example 2. Edge (u; v) is labeled as r(C ) : d(C ) to showthe rate and distortion of the corresponding codecell.

Example 2: As a simple example, we design a conventionalECSQ for a source with a -symbol alphabetand the squared-error distortion measure. Our source alphabetis with pmf , and

.We begin by computing the partial rates and distortions on the

partial RD graph for this source. Arc corresponds to code-cell which has partial rateand partial distortion . For ex-ample, arc represents the codecell which containssource symbols and . The probability of thiscodecell is , and the cell centroid is

, givingand . Fig. 2

shows the partial rate and distortion for each arc.When . Thus,

. The shortest path to node 1 has weight ; theshortest path to node 2 has weight ;the shortest path to node 3 has weight

. This shortest path is achieved by parti-tion ; thus, the optimal ECSQ for

breaks the source alphabet into codecells .The code achieves entropy 1.0 bit per symbol and expectedsquared-error distortion .

Proof of Correctness

We now formally prove that optimal quantizer design isequivalent to finding the shortest path between vertices and

in the partial RD graph. We do this by first demonstrating aone-to-one correspondence between partitions and paths in thegraph. We then show that the total weight on a path from to

equals the Lagrangian for the partition corresponding to thatpath.

A path of length from to is a chain of edgeswith and .

Because of the structure of our WDAG, is an arc ifand only if . Therefore, we can associate bijectivelyto each arc the codecell . For a partition, con-secutive codecells must be adjacent; for a path, consecutivearcs must share one vertex. Therefore, an enumeration ofthe arcs corresponding to the codecells in a partition forms apath, and vice versa. This establishes the required two-waycorrespondence.

The equivalence between the total weight on a path fromto and the Lagrangian for the partition corresponding to the

path follows directly from our construction. The weight on edgeis the Lagrangian cost . The

weight of a path is the sum of theweights of its edges

where is a unique par-tition of .

Complexity

The partial RD graph has vertices and arcs.The vertex labels give the topological order of the vertices, so inthis case no vertex ordering is required in the shortest path algo-rithm. Building the partial RD graph requires finding each of the

weights. Each weight calculation requires oper-ations in general. Thus, the graph construction takestime. When , this can be reduced totime by computing the cumulative moments of order , , andfor the source pmf so that the partial rates and distortions can becomputed using differences of these cumulative moments [3].Likewise, when is monotonic in , the graph con-struction can be reduced to [4]. (This result cannot takeadvantage of a coarse grid, so we set in this case.) Giventhe partial RD graph, the single-source shortest path problemcan be solved in time. Performing the weight calcula-tions during the shortest path algorithm (rather than in an inde-pendent preprocessing step) yields a total complexity for glob-ally optimal ECSQ design of , or ,depending on the distortion measure.

IV. MULTIPLE-DECODER SYSTEMS

The algorithm described in Section III treats conventionalscalar quantizer design—that is, scalar quantization for thesource coding paradigm, where a single encoder describesinformation to be decoded by a single decoder. We next gen-eralize that algorithm to scenarios where a single source isdescribed to multiple decoders. Examples of single-source,multiple-decoder systems include multiresolution and mul-tiple-description scalar quantizers.

Multiresolution scalar quantization (MRSQ) [22]–[26] pro-duces a binary source description that can be described in in-crements. Reading only the first increment gives a reproduc-tion of the poorest quality; each subsequent increment improvesthe quality of the source reproduction. The increments have anatural order, and each incremental description can only be de-coded if all preceding increments have been received before it.Since the binary description may be decoded at a variety of rates,the MRSQ decoder effectively contains a collection of decoders,one for each rate at which the binary sequence may be decoded.MRSQ is useful in applications where the compressed data se-quence is intended for multiple users with differing rate and re-production quality needs.


Fig. 3. Side partitions P and P and central partition P .

Multiple-description scalar quantization (MDSQ) [27]–[30]is another single-source, multiple-decoder scalar quantizationsystem. In this case, the source description is broken into a col-lection of packets, and reconstruction under any combination ofreceived and lost packets is required. MDSQ is useful in diver-sity systems, where the encoder sends separate descriptions ofthe source over different channels and reconstruction is re-quired using the successfully received descriptions. MDSQ isalso useful for packet-based communication systems and dis-tributed data storage systems, where reconstruction of the datausing a subset of the original set of source descriptions may berequired.

In both MRSQ and MDSQ, the increments or packets areassumed to be labeled, so that the decoder can distinguish whichincrements or packets it receives.

We next describe the optimal code design algorithm forsingle-source, multiple-decoder systems. We focus our de-scription on variable-rate MDSQ design with (thus2DSQ instead of MDSQ) for simplicity. The solutions forfixed-rate coding and are natural extensions of thevariable-rate solution, as described at the end of thissection. The optimal MDSQ design algorithm leads to anoptimal MRSQ design algorithm since MRSQ is a specialcase of MDSQ where decoders are designed for a subset ofthe possible packet-loss scenarios rather than all packet-lossscenarios. We can, however, achieve lower rates for some typesof MDSQ with restricted packet-loss scenarios. MRSQ is aprime example, as discussed in Section V.


The encoder of a 2DSQ gives two separate data descriptionsand is therefore defined by two distinct partitions, and , ofthe source alphabet. The codecells of each partition are convexby assumption. The decoder may receive either of the two de-scriptions alone or both descriptions together. When the decoderreceives only a single description, say the index of codecellin partition , the decoder knows that the source symbol sat-isfies . When the decoder receives two descriptions, itknows that the source symbol satisfies for somecodecells and , yielding an effective under-lying partition of thedata. Partition is called the central partition, while and

are called side partitions. Fig. 3 shows an example. Likeand has convex codecells.

Since either description of a 2DSQ may be received withoutthe other, each partition must be uniquely decodable on itsown, and the rate required to describe partition is

. The distortion for partition is. Thus, is the expected

rate–distortion performance when only the th description isreceived, and is the expected distortion when both descrip-tions are received. (Rate is not used since partitionis described only through the description of and .) Again

and , withand

as defined in Section III.Let

and induce

be the set of rate–distortion vectors achievable through vari-able-rate coding. Given a pair of partitions achievinga variable-rate coding point isoptimal if there does not exist another pair of partitionswith for all and forall for which at least one of the inequalities isstrict. For any with on thelower convex hull of , there exist nonnegative Lagrangianvectors and for whichminimizes

We therefore use as the optimization cri-terion for partition design in variable-rate 2DSQs.

Unfortunately, is a function ofand varies with both partitions. As a result, we cannot dividethis optimization into separate optimizations of and . In-stead, a joint optimization is required, which complicates ourdesign of the partial RD graph.

Partial RD Graph

Our goal is to design a partial RD graph with a one-to-onecorrespondence between paths through the RD graph and parti-tion pairs. The total weight of a path should be the Lagrangiancost of the corresponding partition pair.If we can build such a graph, then running a shortest path algo-rithm on this graph guarantees an optimal 2DSQ.

To begin, we define the vertex set as . Thiscollection will be reduced in the discussion that follows. In thepartial RD graph for conventional scalar quantization, the pathsfrom vertex to vertex specify all partitions that span the cellrange . Similarly, in the MDSQ partial RD graph, thepaths from vertex to vertex represent all parti-tion pairs such that spans and spans

(we ignore the central partition for the time being).For example, the paths from vertex to vertex des-ignate all partition pairs for the full source alphabet. Note thatthe partitions in a pair are always aligned on the left side.

Construction of the edge set presents a certain difficulty;naively, there should be an arc from vertex to allvertices and such that and , asshown in the example in Fig. 4, since we can extend a partitionpair by adding a codecell to either of the two side partitions. (In


Fig. 4. Partial RD graph, two descriptions, 2 � 2 cells.

the discussion that follows, we remove the shaded vertices anddotted arcs from Fig. 4, but for now they should be treated liketheir solid counterparts.) We emphasize the fact that only oneside partition is extended at a time, thus each arc correspondsto a single codecell being added to one of the side partitions.Extending one side partition by one codecell may cause one orpossibly multiple codecells to be added to the central partition.To be able to assign a unique weight to the corresponding arc,the total Lagrangian cost of the newly added codecells must bea function only of the labels of the two vertices that the givenarc joins. The codecell added to one of the side partitions isclearly determined by the two vertex labels only, but the centralcodecells may not be. The problem arises from uncertaintyabout the central partition as shown by the following example.

When the side partitions have equal length (i.e., for anyvertex ) computing from and is straightforward(see Fig. 3). When the side partitions have different lengths,however, it is not clear how to determine the codecells in thecentral partition. For example, consider the partitionpair with shown in Fig. 5. Naively, we would findthe codecells of by merging the list of codecell thresholds;by this procedure, the last central codecell would be .Notice, however, that adding a new side codecell to ,as illustrated in the lower portion of Fig. 5, breaks the centralcodecell into two new central codecells and

that depend on . Thus, central codecells containingare not influenced by the subsequent

addition of side codecells, but central codecells containingmay vary as a function of

future additions to the shorter side partition.The uncertainty about demonstrated in the previous ex-

ample threatens our ability to devise a unique arc labeling. Forexample, consider the situation shown in Fig. 6. In this example,

is a partition spanning and is initially a par-tition spanning . We extend the shorter partitionby adding codecell . The figure shows two examples ofpossible values for partition . The value of is the same inboth examples, so the Lagrangian cost of extending must inboth cases be captured in the weight of the edge from to

. Unfortunately, the central codecells created by the ex-tension are different in the two cases, and thus no single weight

Fig. 5. The central partition is determined only to the end of the shorter sidepartition. In this example, the central partition changes in (v ; v ] when cell(v ; v ] is added to P . The central partition on values less than minfv ; v gcannot change with cell additions.

Fig. 6. The shorter side partition is too short. In this example, v < u

implies that the central partition on (v ; v ] (and thus the weight on the arcfrom (v ; v ) to (v ; v )) varies with u .

can actually capture the Lagrangian cost of the given arc. Thisproblem seems to arise from the fact that is “too short”—oneand a half codecells shorter than . We need a way to keep sidepartitions more “even” to avoid this type of problem.

The codecell lag, defined using the threshold sequencefor partition andfor partition pro-

vides a measure of how “even” two partitions are; partitionsmust be aligned on the left side for the followingdefinition to apply. We say that lags by codecells ifthere are codecells in not lying entirely inside the rangecovered by (i.e., ). We want to keepthe codecell lag between the two side partitions at most . (Alag of zero means that the partitions are of equal total length

, a case which we handle later in this section.) Call apartition pair with codecell lag less than or equal to valid.

Suppose that is the shorter partition . Then bydefinition, a lag of implies that the rangeis included in the last codecell of (i.e., ).Adding codecell to results in a single centralcodecell being generated, regardless ofthe structure of the side partitions; therefore, we achieve our ob-jective. The weight on each arc is defined as the Lagrangian costassociated with one new side codecell and one new central code-cell.


We now have a condition (validity) that guarantees thatadding side codecells results in predictable central codecellsbeing generated; we next constrain the partial RD graph to obeythat condition. We want to keep partition pairs with codecell lagmore than unrepresented in the vertices of the graph. Sincethe partition pairs represented in each vertex are defined bythe paths between and that vertex, it is enough to breakcertain malignant paths by removing arcs from the graph.

Our goal is accomplished by eliminating arcs that extend thelonger side partition in a partition pair: extending the longer sidepartition always increases codecell lag by , thus creating unde-sirable partitions with a lag of or more; by contrast, extendingthe shorter side partition only increases lag when (theexact change in lag depends on the exact structure of the parti-tions), and changes lag from to when , therefore (byan inductive argument), lag can be kept at or . When ,no central codecells are added regardless of which side partitionis extended and by how much. To avoid building more than onepath for a single pair of partitions, we choose to extend if

.The result is a partition extension rule that allows only two

types of arcs: with andwith . In the example of a

partial RD graph for 2DSQ given in Fig. 4, the disallowed arcsare dotted. A consequence of eliminating these arcs is that somevertices become inaccessible from and therefore cannotbe visited; we eliminate these vertices and the arcs originatingfrom them. (To be precise, the inaccessible vertices are exactlythose of the form with ; they are shaded in Fig. 4.)

After removing inaccessible vertices, we are left with a graphof vertices. To count arcs, we divide vertices

in two categories: those with and those with. If , vertex has

outgoing arcs; thus the graph contains a total ofarcs from the first category. In the second category, either

or . If , vertex hasoutgoing arcs; for each there are

such vertices with ; thus, if we had notremoved inaccessible vertices, the graph would contain

second-category arcs from vertices with and an equalnumber from vertices with . We therefore bound thetotal number of arcs by (removing theinaccessible vertices eliminates of these edges).

The weight on an arc is as follows.• When , one codecell

is added to and none are added to ; thus,.

• When , one codecell is addedto and one central codecell isgenerated; thus, .

• When , one codecell is addedto and one central codecell isgenerated; thus, .

When and we pre-compute the cumulativemoments as in [3], each weight can be computed in constant

Fig. 7. The partial RD graph for the 2DSQ in Example 3; inaccessible verticeshave been removed.

time since our construction guarantees that at most two code-cells are generated by adding an arc to a given path.

This completes the description of the 2DSQ partial RDgraph. The graph design ensures a one-to-one correspondencebetween paths and partitions, and each path weight equals theLagrangian cost of the corresponding partition. Thus, runningone of the shortest path algorithms described in Section III withsource and sink yields an optimal 2DSQ amongall 2DSQs with convex codecells. We again work a simpleexample before tackling a formal proof of this correspondence.

Example 3: We design a 2DSQ for the source considered inExample 2. We set , and forthe side partitions, and for the central partition.

We begin by sorting the vertices of the 2DSQ graph shownin Fig. 7. The only requirement for this list is that if thereexists an edge from to , then vertex comes beforein the sorted list. Many possible orderings meet this con-straint. The sorted list created by our algorithm is

. For each , let be the th vertexin this list.

Next, we calculate the weights for graph edges. For eachedge, we compute the corresponding side and intersection code-cells and their weighted partial rates and distortion. For ex-ample, the arc from to generates code-cell in side-partition 2 and codecell in the centralpartition. The partial rates and distortions for these codecellswere computed in Example 2. We therefore read the partialrates and distortion off of the graph in Fig. 2, givingand , respectively. The resulting Lagrangian cost is theedge weight for the graph in Fig. 7. Precisely, the archas weight

. (Recall that the partial rategenerated by the central codecell does not come into play.)

We next run the shortest path algorithm. In step , the algo-rithm finds the shortest path from node to node . The can-didate solutions are all of the form: “shortest path to node ”(for some for which there exists an edge )


Fig. 8. Partitions created by the 2DSQ designed in Example 3.

followed by the edge . The ordering rule guarantees thatthe shortest path to is known before it is needed for the cal-culation of the shortest path to .

For our example, the first vertex in the sorted list is the origin. The shortest path from to has weight .

When , we seek the shortest path from to . Thereis only one index for which there is an arc .Thus, the shortest path from to comprises the shortestpath from node to node (weight ) followed bythe edge from to (weight ). Thus, the shortestpath from to has weight . For ,the process is similar; in each case, has only one incomingedge. Thus, the shortest path from to equals the shortestpath from to that single predecessor followed by the edgefrom that predecessor to . The resulting weights are

. Thecase for requires the first nontrivial calculation. Node

has two incoming edges originating at and . Thus, theshortest path to is either the shortest path to followed bythe edge or the shortest path to followed by the edge

. The resulting weights areand . Thus, the shortestpath to node is . The algorithm proceeds inthe same manner for each vertex in turn. The final result is theshortest path from to which describes the optimal 2DSQpartitions. The weight of this path is and theshortest path is . Recall that

.Since edges that travel from top to bottom in the graph describepartition 1 codecells while edges that travel from left to rightdescribe partition 2 codecells, this shortest path corresponds tothreshold sequence for partition 1 and threshold se-quence for partition 2—giving side partitions

and . The resulting cen-tral partition is . These partitionsare shown in Fig. 8. The resulting code achieves entropies 0.81and 1.0 bit per symbol and expected squared-error distortions

and in descriptions 1 and 2, respectively. The distor-tion of the central partition is since when both descriptions arereceived there is no uncertainty about the symbol transmitted.

Proof of Correctness

As in Section III, we must first exhibit a bijection betweenpaths from to and partition pairs on the alphabet

and then show that the weight of a path equals theLagrangian performance of the corresponding pair of partitions.Note that since both final partitions have the same alphabet, thelag is zero, and all partition pairs are valid.

For the bijection, one direction is immediate: each path de-scribes how to “grow” a partition pair by successively extendingone side partition at a time. We map each arc to the corre-sponding codecell being added to either or ; the codecellfor an arc is chosen based on the labels of the origin and desti-nation vertices of the arc.

Showing that for each partition pair there is aunique path in the graph is a little more involved. First someterminology: given two partitions ,

we say that is a truncated version ofif and for . Similarly, for two

partition pairs and , we define to bea truncated version of if and are truncated ver-sions of and , respectively. (Since the partitions in eachpair are presumed to be aligned on the left side, it follows thatall four partitions are aligned on the left side.) We also say thatsome codecell added to extendsaccording to if the extension is still a truncated versionof . Note that if is not a truncated version of

, none of its extensions are either, regardless of howmany codecells are added, since a mismatch in the thresholdsequence cannot be repaired by adding more elements.

Now consider an arbitrary valid partition pair and letbe the sum of the number of codecells in and the number

of codecells in . We next show by induction that for anythe graph contains exactly one length- path that

starts at and describes a truncated version of .For , this is immediate. For length , any path oflength can be decomposed into a path of length and anextra arc. If the length- path is to correspond to a truncated ver-sion of , we have seen that the length- path mustalso. There exists only one path of length- that satisfiesthis condition, and the extension rule guarantees that there ex-ists a unique arc that extends it according to . (The latterstatement uses the fact that both and the partition paircorresponding to the length- path are valid.) Therefore,the set of paths of length satisfying the condition has cardi-nality exactly one. Note that our language implicitly uses theone-way correspondence from paths to partitions (which has al-ready been demonstrated) but not its inverse.

The equivalence between path weight and Lagrangian perfor-mance is immediate by construction of the partial RD graph.

Extension From 2DSQ to MDSQ

An MDSQ with descriptions may be described by sidepartitions which induce nontrivial intersection par-titions (the analogues of the central partition from 2DSQ). In thissection, we assume that the packet-loss scenarios correspondingto all of these intersection partitions occur with nonzero prob-ability and thus that all are constrained; that assumption is re-laxed in Section V. We describe each possible packet-loss sce-nario by a binary vector of length (called the packet-lossvector and denoted by , where ); in thisvector, bit is if packet is received and if packet is lostin the corresponding scenario. (For example, packet-loss vector

describes the scenario where, of an initial fourpackets, all but packet three are received.) Side and intersectionpartitions can be indexed according to the same scheme (e.g.,


is the side partition used to describe packet three out offour, while is the intersection partition corresponding tocombined knowledge of packets one, two, and four).

In this case, the achievable rate–distortion points are

and side-partitions that together achieve a point on the lowerconvex hull of minimize the Lagrangian

for some nonnegative distortion and rate multipliers and ofdimensions (for the packet-loss scenarios, here indexedby ) and (for the description rates), respec-tively. (For trivial intersection partition isindependent of the side partitions and is arbitrary.) Weuse as the optimization criterion for variable-rateMDSQ design.

To generalize our graph construction, we must generalize ourpartition extension rule and arc weight definitions. The partitionextension rule generalization keeps the maximum lag betweenany two partitions less than or equal to and fixes an orderfor breaking ties; we extend the shortest side partition, and ifthere are ties, we pick the competitor with the smallest index.Extending exactly one partition at a time according to the aboveextension rule implies that one side codecell is added and atmost intersection codecells are generated, resultingin at most distortion terms and one rate term in each arcweight.

More specifically, consider an arc from vertexto vertex . The exten-

sion rule implies that and may differ in only one compo-nent ( for ). Fur-thermore, must be the shortest coordinate of vertex (upto a possible tie), giving for all . If there are ties( for some ), then necessarily . Ineither case, side codecell is added (at a Lagrangian costof ). Moreover, for each packet lossscenario , if , then intersection codecell ,where , is generated in intersection par-tition (at cost ); otherwise, no codecells are gener-ated in . Adding up the Lagrangian costs from the side code-cell and any possible intersection codecells gives the weight forarc ; note that only the side codecell contributes rate to thetotal cost (thus, there is a single rate term in the arc weight).

Some of the vertices are inaccessible due to theextension rule. Consider vertex . If for some , theneither for all , or else the vertex is inacces-sible since access to that vertex would require extending par-tition before extending partition , even though (contra-dicting the extension rule). It turns out that this necessary con-dition is also sufficient for accessibility. Any accessible vertex

can be obtained in steps startingfrom : at step , extend side partition from to ,for from to .

In the course of the shortest path algorithm, one of the re-quired steps is to obtain the ancestor set of a vertex . Assumethat side partition has been extended. Since , and

, we get . If there are no ties,the inequality must be strict. If there are ties, then one of thetiers, namely, , has been extended, but the other tiers are un-changed, so and must be the tier with the leastindex in . In summary, if is the shortest component of (ifthere are ties, choose the one with the least index, i.e., minimize). Then has ancestors: “untied” ancestors

( and ) and “tied”ancestors ( and ). When allancestors are “tied.”

The size of the ancestor set of a given vertex equals its in-de-gree; therefore, the number determined above can be pluggeddirectly into the topological sorting algorithm. A slight simplifi-cation is possible. If we eliminate inaccessible vertices from thegraph, then the origin is the only remaining vertex with in-de-gree zero, making initialization trivial for both the topologicalsorting and the dynamic programming stages of the shortest pathalgorithm. We also need to do away with inaccessible ancestorsin our account. If , this means that is permittedonly for , thus leaving accessible an-cestors; if but , then only one ancestor is accessible,with and ; finally if and then isthe origin, which has no ancestors.

Complexity

The computational complexity of the algorithm depends onthe size of the vertex set and edge set of the partial RD graph.With descriptions, there are vertices; each vertexhas at most outgoing arcs, thus there areedges. The complexity of computing the arc weights dominatesall the other steps of the code design process. In general, up to

codecells are generated for each edge, and we must cal-culate the boundaries for those codecells. Each boundary calcu-lation involves taking the intersection of some subset of theside partitions. Computing the intersection of partitions and

requires only operations since most pairs ofcodecells do not intersect. Specifically, we can find the intersec-tion partition using a merge–sort on the threshold sequencesof the side partitions. That is, ifand , then is the partitioncorresponding to the merge sort of thresholds and

. In general, computing the boundary of a codecellrequires examining at most coordinates; the algorithm runsin time .

Fixed-Rate MDSQ

In fixed-rate MDSQ design, the side-partition sizes are fixedand the rate of each description equals the logarithm of the cor-responding side-partition size. Let be the fixed size of parti-tion . Then

describes the set of achievable distortion points at fixed rate. In this case, side-partitions achieving


a point on the lower convex hull of minimize the Lagrangianperformance measure

for some vector of nonnegative Lagrangian parameters . Weuse as the optimization criterion for fixed-rate MDSQdesign.

The problem of fixed-rate MDSQ design differs from that ofvariable-rate MDSQ design both in the optimization criterionand in the constraint on the size of each side partition. Whilethe first difference affects only the edge weights of the partialRD graph, the second difference affects the node definitions.Let node represent the set of all pos-sible collections of side partitions such thatspans with segments. The partition extensionrule is generalized accordingly, again keeping the lag betweenany two side partitions to at most and here updating the parti-tion size parameters appropriately.

Specifically, we have vertices. (The ex-

tension rule makes some of the vertices inaccessible.) Forany arc from vertex tovertex , the extension rule againimplies that and for all forall , and for any such that . Further,since arc corresponds to the addition of a single sidecodecell to the th partition, . For anywith , arc implies the addition of codecell

to partition at Lagrangiancost . Adding these Lagrangian costs givesthe weight for arc . Fixed-rate MDSQ optimization isequivalent to finding the shortest path fromto . The complexity of the resultingalgorithm is .

V. MRSQ AND OTHER RESTRICTED MDSQS

In this section, we replace the MDSQ assumption that anypacket-loss scenario occurs with nonzero probabilityby the restricted MDSQ assumption that only in some ad-missible scenario set occur with nonzero probability.Under this new assumption, the optimal rate for describing aside partition with fixed- and variable-rate codes varies with .We begin by studying multiresolution codes

and then generalize to a broader class of ad-missible scenario sets.

Again, we focus initially on variable-rate codes, leaving thediscussion of fixed-rate coding to the end of the section. The al-gorithms that follow again constrain every codecell of every sidepartition to be convex, which creates convex codecells in all in-tersection partitions as well. Any MRSQ with convex codecellsat all resolutions can be represented with a collection of parti-tions that satisfies these constraints.

MRSQ Overview

In MRSQ, packet is never received without packets, giving .

Fig. 9. Successive refinement of partitions in an MRSQ. In this example,P (C ) = fc ; c g and P (C ) = fc ; c ; c g. The encoder describes anM -step path from the root to a leaf in the tree structure, uniquely describingthe representations associated with all M resolutions.

Given this constraint, for any , packet need notbe uniquely decodable on its own. Instead, it suffices forpacket to be conditionally uniquely decodable given packets

.Let for each . We begin

by examining the partitions . By definition of the inter-section partitions, for each for some

. Thus, for each ,partition is a refinement of ; that is, the threshold se-quence of is a subsequence of the threshold sequence of

. (Here by definition.) Since refines(written ), for any there exists a col-

lection of cells so thatis a partition of . Fig. 9 shows the refinement relationship be-tween partitions and .

The restriction that satisfies Kraft’s in-equality for each is necessary in MDSQ sinceit allows the decoder to uniquely decode description when thatis the only description received. This restriction is not neces-sary for MRSQ. For any , the decoder uses the firstdescriptions to determine some codecell ; sincerefines , the th incremental description need only distin-guish between the members of rather than all of the mem-bers of . Thus unique decodability in an MRSQ is achievedif and only if satisfies Kraft’s inequality foreach and . The optimal rate for de-scribing cell given a prior description of the cell

that it refines is , giving optimal partial rate, where .

The expected th-resolution rate is

This conditional entropy bound on the expected rate is approx-imated in practice by conditional entropy codes.


The rate-distortion points achievable by variable-rate MRSQsare


and successively refined partitions that together achieve apoint on the lower convex hull of minimize the Lagrangian

(2)

for some positive vectors and . Wetherefore use as our design criterion. Note that

where and expand the terms in (2); follows from; and uses the definition

and the fact that has only one cell with . Weexpress the design criterion recursively as

(3)

with for all and

Algorithm

The design algorithm finds the set of successively refined par-titions (compatible with the coarse grid ) that minimizes

for a fixed collection of Lagrangian param-eters and . Notice that for to be optimal,must be optimal for each ; each of the termsin the equation for can be optimized independently ofthe others since the cells in do not overlap. These ob-servations suggest a bottom-up dynamic programming solution.For each possible codecell , we find the partition of

that minimizes

using a shortest path algorithm. Given for all -com-patible codecells, we similarly find for each byminimizing

Fig. 10. Weights and resulting Lagrangian costs for resolution 2 in Example 4.Edge (v; v ) has label w (v; v ) : J (v; v ).

Iterating back from to yields

More formally, we begin by initializingfor all with . We then proceedthrough iterations, indexed from to . Each iteration relieson a different partial RD graph. In iteration , for all -compat-ible codecells we compute and store the subpartitionthat minimizes . The optimal is found by run-ning the shortest path algorithm between each pair of distinctvertices in a graph with weight

on the arc from node to node . Hereis the weight on the shortest path from node

to node using arc weights ; that shortestpath is known from the previous iteration. (We implicitly usethe two-way correspondence between arc and codecell

.) The final results of the algorithm are andthe successively refined partitions that achieveit. To construct the partitions, we iterate back from to . Par-tition is constructed from by pasting togetherfor all .

For each resolution, we find the shortest paths from node tonode for all simultaneously using an all-pairs shortest path algorithm (see [20]) rather than calculatingthe shortest path between each pair separately. This reduces thecomplexity from per resolution, toper resolution. Thus, the overall complexity is .

Example 4: We design a 2RSQ using the pmf from Ex-ample 2. Recall that Fig. 2 describes the partial rates anddistortions for this example. Our Lagrangian multipliers are

and for the first resolution and andfor the second resolution.

The algorithm begins by finding all-pair shortest paths for res-olution , in a graph with weights

. Fig. 10 shows the weight of edgeand weight of the shortest path from to . Thefollowing table shows a shortest path from node to node . Inthis example, it turns out that each shortest path uses the max-imum number of possible edges. For example, the shortest pathfrom 0 to 2 comprises the edge from 0 to 1 (weight ) fol-lowed by the edge from node 1 to node 2 (weight ) to give


Fig. 11. Weights and resulting Lagrangian costs for resolution–1 in Example 4.Edge (v; v ) has label w (v; v ) : J (v; v ).

Fig. 12. The 2-resolution tree corresponding to the code designed inExample 4.

a total weight of . This is the shortest path because the alter-native—a single edge, passing directly from node 0 to node 2without stopping at node 1—has a higher Lagrangian weight,

.

For resolution , the all-pairs shortest path algorithm runs ona graph with weights

. Fig. 4 shows these weights and the resulting shortestpath weights . The following table describes the cor-responding shortest paths.

For instance, let us again find the shortest path betweennodes 0 and 2, this time at resolution . The relevant weightsare and

. Therefore, the shortestpath is the direct edge with Lagrangian cost . (Thepath has a larger total cost of .)

The optimal 2RSQ with convex codecells is described by theshortest path from 0 to 3 in the resolution graph (see Fig. 11).This path has Lagrangian weight and is achieved by

path , which corresponds to resolution- code-cells and . Resolution- code-cell is partitioned in resolution by the codecells cor-responding to the shortest path from node to node in theresolution- graph. Thus, is partitioned byin resolution while is partitioned by in resolu-tion . Figs. 12 and 13 show the corresponding MRSQ tree andpartitions.

Fig. 13. Optimal partitions for the 2RSQ in Example 4.

Restricted MDSQ

We next modify the general MDSQ design algorithm foruse with more general scenario sets. The modifications greatlyincrease computational complexity, but the algorithm re-mains polynomial in the size of the source alphabet. Sincethere exist more efficient specialized algorithms for somerestricted-MDSQ scenarios (e.g., the above bottom-up dynamicprogramming algorithm for MRSQ design), the generalizationis not of interest for all values of . It does, however, increasethe collection of scenario sets for which we can find the bestcode among all codes that meet the convexity constraint.

In MRSQ, packet never arrives without packets, so we can save rate by allowing packet ’s

description to depend on the descriptions of prior packets. Thedependency sets capture a similar property for generalrestricted MDSQs. Packet depends on packetif is never received without (i.e., impliesfor all admissible scenarios ). We call anya requisite of , and we say that packet is independent if

. For example, in a 3RSQ (three-resolution scalar quan-tizer) , , and

, packet 1 is independent, packet 1 is a requisiteof packet 2, and packets 1 and 2 are requisites of packet 3.

In MRSQ, if packet depends on , then partition must bea refinement of , and the optimal rates are the correspondingconditional entropies. Similar refinement relationships and con-ditional lossless codes also arise in other restricted MDSQs, andthe algorithm that follows uses the dependency sets todetermine these relationships.

While knowing the dependency sets is sufficient for findingthe optimal restricted MDSQ for many scenario sets, depen-dency sets capture only the simplest type of dependence, andthus the algorithm discussed below does not guarantee thebest convex-codecell code for all scenario sets. For example,consider the scenario set .Here packet 1 is guaranteed to appear with either packet 2or packet 3. Yet , and thus the algorithm that fol-lows treats packet 1 as an independent packet, calculatingits rate as the full entropy of Since this entropy is notnecessarily optimal [31], the algorithm guarantees an op-timal solution only for examples characterized entirely bythe simple dependencies captured by . Examples of scenariosets covered here but not covered previously include

(a 2RSQ plus an in-dependent packet), (packets2 and 3 never appear without 1), and

(amultiresolution multiple description code).

One way to compute the dependency sets is to check whetherfor all pairs with by examining


Fig. 14. The dependency graph for � = f10000; 01000;11100;01010;

11110;11111g; dashed lines show indirect dependencies.

all ; this procedure takes operations, givingin the worst case. There is not much room for im-

provement here since any algorithm must visit each admissiblescenario at least once.

The dependency function implicitly defines a (directed) de-pendency graph with vertices and an arc from to ifdepends on . An example appears in Fig. 14. (Weremove the dotted arcs in the discussion that follows, but fornow they are part of the graph). There are no restrictions on thein- and out-degrees of each vertex (i.e., one packet can dependon many others and multiple packets can depend on the samepacket). For any two mutually dependent packets and , wereplace and with a new “joint” packet ; since all packetsare jointly encoded and and can always be jointly decoded,the rate needed to describe the new packet can be broken uparbitrarily between the two original packets and .

We sort the vertices of the graph (the packets) topologically,renumbering them so that if is dependent on , then .(Since all mutually dependent packets are merged, the graph isguaranteed to be acyclic.) The ordering is useful for the designstage, where we again use a tie-breaking rule that extends thecompetitor with the least index.

We calculate the rate of dependent packets using the corre-sponding conditional entropies. To calculate these conditionalentropies, we enforce refinement relationships on some of theside partitions. In MRSQs, if packet depends on , thenmust be a refinement of . In general restricted MDSQs, ifpacket depends on , then for each with

, and must be a refinement of. Unfortunately, this refinement relationship between inter-

section partitions implies no refinement relationships betweenside partitions. In fact, refinement relationships are not neces-sary in the side partitions of dependent packets. We next show,however, that considering only side partitions for whichimplies refines causes no loss of generality.

For any , the set of side partitions thatyields the full partition set is not necessarily unique.We next show that for any full partition set derivedfrom arbitrary side partition set , there exists anotherside partition set for whichand implies . We constructas , where for all and

otherwise. Then since packetnever appears without its requisite packets.Since refinement is transitive ( and

implies ), some of the refinement relation-ships contained in the initial dependency graph are superfluous.We say that depends directly on (and write ) if andonly if depends on and does not depend on

any other node that also depends on (there does not exist ansuch that ). We remove all but the direct depen-

dencies. From a graph-theoretical perspective, this operation isthe inverse of the transitive closure problem on the dependencygraph.

We are now ready to extend the MDSQ design algorithm.First, we modify the partition extension rule to ensure succes-sive refinement according to . Since the packets are sortedtopologically, dependent packets have higher indices, and theirpartitions are extended only after those of their requisites. Topreserve refinement, we make sure that cells added to depen-dent partitions never “cross” boundaries of cells already presentin requisite partitions. Since codecell lag is always or , theonly place where such a crossing might occur is at the end ofa requisite partition. Therefore, we remove from the partial RDgraph any arc that extends a dependent partition beyond the endof any of its requisite partitions.

Since arc costs for codecells added to dependent partitionsmeasure rate as conditional entropy, we must modify the partialRD graph to include information about the “parent” codecellsin all requisite partitions. Given the bound on codecell lag, each“parent” is the last codecell in its respective partition. Thus, foreach packet with dependents, we modify the vertex label to de-scribe not only the end of the corresponding partition but also thebeginning of the final codecell in that partition; thus, instead ofkeeping track of only the last element in the threshold sequencefor that partition, we keep track of the last two elements. Thisdouble labeling increases the number of vertices fromto in the worst case.

Each (single or double) partition label is a component in the-dimensional label for a vertex; partition labels are elements

of , so vertex labels are elements of. As an example, if

and (packets 1 and 3 are independent, while packet 2depends on 1), a vertex label might look like

—corresponding to partitions extending upto the fifth cell of the coarse alphabet, with its final codecellstarting at the third cell, extending up to the second cellof the coarse alphabet, and up to the seventh cell of thecoarse alphabet. As illustrated in this example, double labels

consist of the last two elements in the threshold sequenceof partition (corresponding to a final codecell );an exceptional situation arises when the partition is empty (thethreshold sequence has only one element) in which case we usethe label .

All other partition extension rules remain valid once werealize that the second element in a double label denotes theend of a partition—in particular, this is how we measure thelength of a partition when determining which partition in atuple is the shortest. The introduction of double labels raisesthree new issues. First, the successor set of a vertex may in-volve extending a double-labeled partition, in which case thefollowing rule applies: double-labeled partition can beextended from to where (provided allthe other extension rules are obeyed). Second, the ancestor setof a vertex may involve “shortening” a double-labeled partition

; can be shortened to where eitheror .


Finally, the weight on the arcs associated with dependentpackets in the partial RD graph must be changed to use con-ditional entropy in the rate calculations (distortion terms areleft unchanged). As noted in Section IV, only side codecellscontribute rate terms to the Lagrangian cost. Given the doublelabels on requisite packets and the enforced refinement condi-tion for dependent side partitions, for any packet , any codecell

, and any requisite packet , the “parent” code-cell with is described by the double-labelindex for packet . Since this parent codecell is known, we cancalculate the optimal rate for describing packet given descrip-tions of as

giving a total rate contribution for group equal to

Here

and

Fixed-Rate Coding

The extension to fixed-rate code design proceeds in a mannersimilar to the one described in Section IV. Again, the side-parti-tion sizes are fixed, the nodes of the partial RD graph are modi-fied to include partition sizes, and the entropies and conditionalentropies are replaced by the corresponding log-cardinalities.

VI. DECODER SIDE INFORMATION

We next consider the design of scalar quantizers with decoderside information. The proposed approach allows the incorpo-ration of decoder side information in any of the above codingscenarios, yielding fixed- and variable-rate conventional scalarquantizers with decoder side information (Wyner–Ziv codes),MRSQs with decoder side information, and MDSQs with de-coder side information. We begin by describing the design algo-rithm for ECSQ with decoder side information—giving side-in-formation ECSQ (SECSQ). We then generalize to other codetypes. In each case, the algorithm finds the best code among allcodes of the given type with convex codecells.

ECSQ With Decoder Side Information

Let and denote the ordered, scalar source and side-in-formation alphabets, respectively. By assumption, iseffectively finite, and the joint pmf on is known. Letthe source- and side-information-alphabet sizes be and

, respectively. Then and. We refer to source symbol by its index

and side-information symbol by itsindex . Given the joint probabilityof is the probability of .We again consider both optimal design subject to the convexityconstraint and fast suboptimal design achieved by partitioning

into cells and into cells. As in the previousdiscussion, and are required toguarantee optimal design subject to the convexity constraint.

Like the ECSQ encoder of Section III, the SECSQ encoderpartitions into convex codecells. Thus, finding the best codethat meets the convexity constraint is equivalent to designingthe best partition with convex elements. Further, since the ex-pected distortion and expected rate for an encoder defined bypartition of are additive over the codecells of , we canonce again define a partial RD graph to make optimal partitiondesign equivalent to a shortest path problem. The only differ-ence between the partial RD graph for SECSQ and the partialRD graph for ECSQ lies in the weight calculations.

The expected distortion of the SECSQ defined by partitionis . Here

where is the optimal codeword for codecell andside information . When , the centroid

of with respect topmf yields the optimal performance;here .

In calculating the partial rate, we assume the use of arate- Slepian–Wolf code to describe to a decoderthat knows [32]. While rate may not be achievablewith probability of error , we can design codes thatapproach rate arbitrarily closely and achievefor any [33]. (Given assumptions of a finite alphabet andbounded distortion measure, the increase in distortion causedby a nonzero error probability can be made arbitrarily small.)Thus, the expected rate of the SECSQ defined by partition is

with partial rate

where .The SECSQ optimization criterion is the Lagrangian

The weight on arc of the partial RD graph is thereforedefined as . Given these modifiedweights, the design procedure is identical to the ECSQ designprocedure.

Other Side-Information Codes

The inclusion of decoder side-information in fixed-rateconventional scalar quantizers and fixed- and variable-rateMDSQs, MRSQs, and restricted MDSQs proceeds simi-larly. In each case, the design criterion and algorithm forthe side-information code are identical to those of the codewithout side information. The only differences are the par-tial rate and distortion calculations used to calculate arcweights in the partial RD graph. Without side informa-tion, codecell has a single codeword and contributes


to its partition’s distortion;with side information, codecell has codewords andcontributes toits partition’s distortion. Without side information, variable-ratecoding rates are entropies and conditional entropies; with sideinformation, those entropies and conditional entropies are(further) conditioned on the decoder side information. The ratesfor fixed-rate codes do not change as a result of decoder sideinformation.

VII. COMPUTATIONAL COMPLEXITY

The proof of correctness in each section demonstrates thateach algorithm yields a globally optimal solution subject to theconstraint of codecell convexity. We next summarize the com-plexities of these algorithms.

All design algorithms consist of two steps: the preprocessingstep, where we calculate the partial rates and distortions neededto define the edge weights, and the shortest path step, where wefind the shortest path through the graph. The complexity of thefirst step depends only on the size of the source alphabet (orits reduction) and distortion measure, while the complexity ofthe second step also depends on the code type. The followingresults bound the complexity of the preprocessing and shortestpath steps, respectively.

Theorem 1: For non-mean-square error (MSE) distortionmeasures, the preprocessing step takes time and

space. For MSE distortion, the preprocessing step takesspace and time.

Proof: For a general distortion measure, the preprocessingstep involves computing and storing the partial rates and distor-tion. There are values to compute, computing each re-quires operations in general, which gives the first result.

In the MSE case, we do not compute or store the edgeweights; given the source pmf restricted to the coarse grid,we merely calculate the source pmf’s cumulative momentsof order and . From these, we can compute any partialrate or distortion on the fly in constant time. The moments are

real numbers that can be computed in linear time andstored in an array.

Theorem 2: The shortest path step complexities are:

VR-MDSQ: space,time.

FR-MDSQ: space,

time,where is the fixed length for side partition .

VR-MRSQ: space, time.FR-MRSQ: space,

time.

Restricted MDSQ (possibly with some FR descriptions):

space,

time,

where is the fixed length for side partition

or for VR side partitions, and

is the number of partitions with dependents.

These complexities apply whether the code designer has sideinformation or not.

Proof: For all algorithms except MRSQ-VR, the shortestpath step involves first generating and storing the topologicallysorted list of vertices for the partial RD graph, computing theedge weights, and then finding the shortest path. If the graph has

vertices and edges, then the first step takes space andtime. The second step takes constant space and

time proportional to , but the proportionality factor is nota constant and depends on the algorithm. The third step takes

time. Thus, the space complexity is , while the timecomplexity needs to be settled in each case.

For variable-rate MDSQ, there are vertices;each vertex has at most outgoing arcs, thus there are

edges. The number of edges dominates thenumber of vertices. In general, up to codecells are gen-erated for each edge, and determining the boundaries of eachcodecell requires examining at most coordinates.

For fixed-rate MDSQ, there are ver-tices; each vertex has edges. Again, there are more edgesthan vertices. As with MDSQ, computing the weight of eachedge requires steps.

In a restricted MDSQ, each partition may have one, two, orthree labels. Precisely, each partition has one extra label per fea-ture, where the features considered are: “fixed-rate” and “hasdependents.” The basic label and the label for partitions withdependents ranges from to . The fixed-rate label rangesfrom to . Therefore, the number of vertices increases to

. The number of outgoing edgesper vertex is again ; computing each edge weight still re-quires steps.

The fixed-rate MRSQ result relies on the restricted MDSQresult. In this case, every partition is fixed rate, and all but thefinest one have dependents.

In variable-rate MRSQ, we iterate over resolution levels.At each level, we run an all-pairs shortest path algorithm in a1DSQ-like graph with vertices, which takes time, andstore the resulting costs, which takes space. The costsbecome the weights for the next level up, and are overwrittenwith new costs after that level completes.

VIII. CODECELL CONVEXITY: CONDITIONS FOR OPTIMALITY

The preceding sections describe algorithms for choosing theoptimal quantizer among all quantizers with convex codecells.We here explore the consequences of the convexity constraintunder the assumption that for somenondecreasing . Briefly put, the conclu-sions are as follows. Under reasonably mild constraints on ,convex codecells are sufficient for optimality in conventionalscalar quantization and ECSQ. In contrast, we give examples todemonstrate that even for the squared error distortion measure

, convex codecells preclude optimality for MRSQ,MDSQ, and side-information scalar quantizers for some sourcesat some rates.


The benefit of the convexity constraint is the low-complexityalgorithms that it enables. The cost is the performance degrada-tion that results when convexity precludes optimality. A moreprecise understanding of this tradeoff is an important topic forfuture research. For example, demonstrating the existence of ex-amples where convexity precludes optimality does not provethat convex codecells are never optimal for these code types.In fact, for MRSQ it seems to be far easier to design examplesfor which convex codecells are optimal than to design examplesrequiring nonconvex codecells. Further, even when convexityprecludes optimality, the gap between the performance of theconstrained optimum and the unconstrained optimum is not wellunderstood.

The discussion that follows begins with fixed-rate and en-tropy-constrained conventional scalar quantization and thentreats first fixed-rate and entropy-constrained MRSQ, MDSQ,and restricted MDSQ and then decoder side information.

Conventional Scalar Quantizers

Let anddescribe the sets of expected rate–distortion points achiev-able through fixed-rate and entropy-constrained scalar coding(without decoder side-information) on pmf . A con-ventional fixed-rate scalar quantizer (SQ) that achieves a pointon the lower boundary ofis optimal in the sense that it has the lowest possible distortionamong all fixed-rate conventional scalar quantizers satisfyingsome rate constraint. Similarly, an ECSQ that achieves apoint on the lower boundaryof is optimal in the sense that it has the lowest possibledistortion among all ECSQs satisfying some rate constraint.While and are not convex in general and anypoint on the lower convex hull of and can beachieved through time sharing, the above definitions for optimalfixed-rate SQs and ECSQs does not require performance onthe convex hull of or ; time-sharing strategies,while low in complexity, are not strict scalar codes.

Theorem 3 proves the optimality of convex codecells forfixed-rate scalar quantizers when is nondecreasing. Theresult is a very slight generalization of [34, Lemma 6.2.1](which treats increasing) and the earlier [35] (which treats

). Our proof differs from the ones that precede it; wegeneralize this approach in the discussion that follows.

Theorem 3: Given a pmf and a distortionmeasure for some nondecreasing function

, any point is achiev-able by a fixed-rate SQ with convex codecells.

Proof: We prove that for any partitionwith optimal codewords , there exists an-other partition with convex codecells that satisfies

Since implies, by definition of , the ex-istence of an encoder partition such that and

, this observation gives the desired result.

For each , letifif .

Then and nondecreasing imply that theare half-lines. For example, if is strictly increasing, then

for each andfor each . The

set describes all for which is the closestcodeword, with ties broken in favor of the smallest codeword.Since the are half-lines, each must be an interval. Finally,

partitions (every has a uniqueclosest codeword) and minimizes expected distortion, giving thedesired result.

We make two modifications in generalizing Theorem 3 fromfixed-rate to entropy-constrained codes. First, we consider onlypoints on the convex hull of . Second, we assume that

is convex. The first constraint is practically motivated sincethe Lagrangian performance measure used for ECSQ designfinds the lower convex hull of ; it is also theoreticallymotivated since there exist points on (but not on itsconvex hull) that cannot be achieved with convex codecells[36]. In the absence of the second constraint, codecell convexitymay preclude encoder optimality for some source distributions.Codecell convexity for ECSQ on continuous source distri-butions under the squared error and th-power distortionmeasures is noted without proof in [36, p. 418]. Interestingly,that observation also applies to optimal variable-rate codes withperformance not on the lower convex hull of achievable rates.

Theorem 4: Given a pmf and a distortionmeasure where isconvex and nondecreasing, any point onthe lower convex hull of is achievable by an ECSQ withconvex codecells.

Proof: For any point , there exists apartition with and . We as-sume, without loss of generality, that since anypartition with more than codecells must include empty code-cells and empty codecells cannot improve ECSQ performance.If is on the lower convex hull of ,then there exists a such that minimizesover all partitions on , as discussed in Section III.Let with codecell probabilities

and optimal codewords .We next construct a convex codecell partitionthat satisfies

While this property is sufficient to prove the desired result, wefurther note that

giving


For any and , let. Then for each ,

let

ifif

From [36, Lemma 1], if is convex and nondecreasing in, then is monotonic in . Since

is monotonic inis monotonic in for each . As a result, each nonempty

is a half-line. The set describes all forwhich is the “closest” codeword by this modified nearestneighbor distortion measure, with ties broken in favor of thesmallest codeword. Since the are half-lines, each mustbe an interval. Again partitions , andthus we have the desired result.

If is nondecreasing and not convex, then codecell convexitymay preclude encoder optimality for some source distributionssince guaranteeing a unique solution to

requires monotonicity ofin , which in turn requires convexity of , as shown next.

For any fixed , let

ififif

If is nondecreasing, then is nondecreasing for all. This leaves two possibilities: either is constant onor . The first case cannot be true for all

(that is, all codewords) unless for all , which isa convex function. In the second case, monotonicity ofrequires that is nondecreasing for all

and that is nondecreasing for all. The second condition gives

or

for all and any . While achieving this result forsome does not require the convexity of , achieving thisresult for all requires the convexity of . Thus, when is notconvex, there exists a pmf with optimal codewords and suchthat is not monotonic. The existence of such a pmf opensthe door for an optimal code that requires nonconvex codecells.

MRSQ, MDSQ, and Restricted MDSQ

In MRSQ, MDSQ, and restricted MDSQ, codecell convexitymeans that all codecells of all partitions are convex. To achievethis constraint in MDSQ, it suffices to choose side partitions

with convex codecells.We begin by considering the simplest form of (restricted)

MDSQ, namely MRSQ. The rate–distortion points achievableby fixed-rate- and entropy-constrained RSQs are

respectively, where

An MRSQ that achieves a point on a lower boundary offor any nonnegative (here is the incremental rate ofresolution ) or is in some sense optimal. (In fixed-rate-coding, each lower boundary describes the minimal value ofsome subject to constraints on . In entropy-con-strained coding, each lower boundary describes the minimalvalue of some or subject to constraints on the remainingrates and distortions.) While the lower boundaries are notconvex in general, our design technique focuses on achievingpoints on the lower convex hull of and . For anysuccessively refined partitions that together achievea point on the lower convex hull of , there exists apositive vector for which minimizes theLagrangian

over all . Similarly, any successively refinedpartitions that together achieve a point on the lower convex hullof minimize the Lagrangian

for some positive vectors and .The algorithm described in Section V finds the optimal

MRSQ among MRSQs with convex codecells at all resolu-tions. That is, the design algorithm considers only MRSQswith convex partitions . Unfortunately, there existpmfs for which the constraint of codecell convexity inprecludes optimality even for points on the lower convex hullof with and convex. One suchexample follows. Due to the popularity of the squared-errordistortion measure for practical coding applications, we hereuse . We can construct similar examples forother distortion measures.

Example 5: Let , and consider pmfon alphabet . Then for

there exists a point on the lower convex hull ofthat cannot be achieved with codecell convexity.

In particular, in order to minimize withand , we must use nonconvex codecells at resolution .The optimal partitions (in terms of the symbol alphabet ratherthan the symbol indices) are and

. Fig. 15 shows the corre-sponding optimal 2RSQ codebook. The intuition here is asfollows. By setting the incremental rates as , wefix the maximal depth of our binary tree-structured codebookto two. By further setting the second-resolution distortionto , we require each of our four alphabet symbols to occupya different leaf of the binary codebook. The question that


Fig. 15. An optimal fixed-rate 2RSQ codebook for which P requiresnonconvex codecells. Given pmf f1=8; 1=8;3=8;3=8g on alphabetf20; 40;60;140g, the optimal 2RSQ for all � ; � such that � =� < 0:02695must have nonconvex codecells at resolution 1. The codebook’s resolution-1partition is P = ff20;60g;f40;140gg with �(f20;60g) = 50 and�(f40;140g) = 115.

Fig. 16. An optimal entropy-constrained 2RSQ codebook with non-convex codecells in P . Given pmf f1=8;3=8;1=8;3=8g on alphabetf20;40;60;80g, the optimal 2RSQ for all (� ; � ; � ; � ) such that� + 31:36� < � < � + 5951:16� and � is sufficiently largeto force D to 0 must have nonconvex codecells at resolution 1. HereP = ff20;60g;f40;80gg with �(f20;60g) = 40; �(f40;80g) = 60;R(P ) = :1414, and R(P jP ) = 1.

remains, then, is which pairing of our four symbols into twogroups of two minimizes the first-resolution distortion .While it seems natural to group the symbols lexicographically,with symbols 20 and 40 descending from one first-resolutionnode and 60 and 140 from the other, this turns out not to bethe optimal choice. The key here is that most of the probabilitylies in symbols 60 and 140 and these symbols are quite farapart. Pairing 20 with 60 and pairing 40 with 140 (see Fig. 15)gives lower expected distortion since it allows more accuratereproduction of the high probability events.

The same problem can arise for entropy-constrained MRSQsachieving performance on the lower convex hull of , asshown in Example 6.

Example 6: Consider ,pmf , and alphabet . If

and is sufficientlylarge to force to , then achieving the optimal performancerequires nonconvex codecells at resolution . The optimalpartitions (again in terms of the symbol alphabet rather thanthe symbol indices) are and

. Fig. 16 shows the optimal2RSQ codebook. The intuition in this case differs from thatin the prior example. In this case, the argument is a bit morecomplicated. Setting implies a depth- codebook, andmaking sufficiently large to force to implies that eachsource symbol must occupy a distinct leaf in that codebook.In this case, however, the tree structure need not be binary.Luckily, the space of possible solutions is sufficiently small,that we can test each possible solution. The key to our non-convex solution is that, under the given constraints on and ,the rate advantage associated with pairing the high-probabilitysymbols (40 and 80) together and pairing the low-probabilitysymbols (20 and 60) together outweighs the distortion costassociated with their greater distortion.

While the preceding examples demonstrate that convexitysometimes precludes optimality, it is important to note thatconvex codecells are also optimal for some distributions. Forexample, violating the condition in Example5 or the condition inExample 6 yields examples where there exists an optimal codewith convex codecells.

Since MRSQ is a special case of a restricted MDSQ, the pos-sible suboptimality of the optimal code with convex codecellsgeneralizes to MDSQ and restricted MDSQ as well. The resultis the following theorem.

Theorem 5: Requiring codecell convexity in partition of afixed-rate or entropy-constrained RSQ, DSQ, or restricted

DSQ with precludes optimality for some finite-al-phabet sources, even when with

nondecreasing and convex.

The cost of convex codecells in MDSQ is exacerbated bythe additional constraints it imposes on the size of the intersec-tion partitions. For example, an MDSQ with convex code-cells in partition , has at most

codecells in . In contrast, whenthe codecells of each are not constrained, partition

can have as many as codecells. This observa-tion leads to two crude bounds on the cost of convexity in thecentral distortion of a fixed-rate MDSQ. Let bethe first-order fixed-rate operational rate-distortion bound at rate

. Then and for all other implies that thedifference between the performance of a fixed-rate MDSQ withconvex codecells and the performance of a fixed-rate MDSQwith codecells that need not be convex is

This difference in distortion increases with increasing rate andmay be arbitrarily large. In contrast, the cost of convexity equalszero when for all andfor all other . While these bounds are not entirely meaningful(the 1DSQ design algorithm yields globally optimal codes ineach of these cases) they do help build intuition about the algo-rithm’s performance when the packet loss probability is eitherextremely low or extremely high.

The preceding discussion demonstrates the problem with as-suming codecell convexity at the coarsest level partitions (in MRSQ, in MDSQ). The approach also sug-gests similar difficulties with codecell convexity at other levelssince we can construct from our 2RSQ examples 3RSQ exam-ples that mimic the observed behavior in resolutions and ,and so on. The approach lends little insight, however, into thefinest level partition ( in an MRSQ, in MDSQ).We next show there exist distortion measures for which the as-sumption of codecell convexity at the finest level partition doesnot preclude optimality. Precisely, Theorem 6 proves this re-sult for fixed-rate MRSQ with the squared-error distortion mea-sure. The same argument extends immediately to entropy-con-strained MRSQ and fixed-rate and entropy-constrained MDSQ


and restricted MDSQ. These results generalize the earlier ob-servation of Vaishampayan, which treated the central partitionof a fixed-rate 2DSQ under the squared error distortion measure[27, Sec.III-A].

Theorem 6: Given pmf and distortion mea-sure , any point that sitson the lower convex hull of is achievable by a fixed-rateMRSQ with convex codecells in .

Proof: For any point on the lower convexhull of implies the existence of parti-tions with for each .Further, since is on the lower convex hull of ,there exists a nonnegative vector for which minimizes

over all . Label the codecells ofeach partition in aswith optimal codewords for each

. Without loss of generality, index the codecells ofso that

Since refines for each , letdenote the index of the resolution- codecell from which code-cell descends. (Here and

for all .) We next construct partitionsso that has convex codecells and the ex-

pected Lagrangian performance based on is no worse thanthat based on . For any and any with

, let .Then for each , let

ifif

Since , the differenceis linear in for all (the quadratic terms cancel).As a result, each nonempty is a half line, and the set

, describes all for whichis the -resolution reproduction with the best Lagrangianperformance. (Ties are broken in favor of the smallest res-olution- codeword.) Since the are half-lines, each

must be an interval. The partitiontogether with the ancestor relationships described bydescribe a partition with and

. (Herefor all by construction.) Since minimizes the Lagrangianperformance with respect to the given codebook

giving the desired result.

Generalizing Theorem 6 to MDSQ tells us that, for thesquared-error distortion measure, the codecells of the partition

must be convex. This observation may turn out tobe the key to unconstrained, globally optimal code design, atleast in some cases. For example, in fixed-rate MDSQ design,combining the fixed-rate constraints with the convex-codecell

constraint on partition gives a bound on the maximalnumber of codecells in , which in turn constrains thenumber of intervals that may be observed in the nonconvexcodecells of the side partitions from which derives.Generalizing our algorithm to efficiently design MDSQs forwhich each codecell is a union of at most convex subcellsand the resulting intersection partition meets the aboveconstraint on the maximal number of codecells is a topic ofongoing research.

Side-Information Codes

Unfortunately, the codecell convexity constraint can alsodegrade the performance of codes with decoder side infor-mation. This problem can occur even in simple fixed-rate SQwith for a well-behaved function . Thefollowing example illustrates the problem.

Example 7: Consider , pmf, and alphabet . Suppose

that the side information takes on values from with

. The side information is availableonly to the decoder. Then the optimal fixed-rate- SQ withdecoder side information requires nonconvex codecells. Inparticular, .

Theorem 7: Requiring codecell convexity in fixed-rate orvariable-rate SQ, MRSQ, MDSQ, or restricted MDSQ withdecoder side information precludes optimality for some fi-nite-alphabet sources, even when with

nondecreasing and convex.

IX. EXPERIMENTAL RESULTS

A. Gaussian Data

Fig. 17 compares the performance of the proposed technique(pluses) with the experimental results printed in [27] (squares)on a Gaussian data set. The algorithm in [27] combines iterativedescent with an index assignment algorithm. In each case, wedesign a fixed-rate 2DSQ with 2 bits per description and plotthe code’s central distortion as a function of its average sidedistortion. The results are comparable for the small range of sidedistortions where they overlap. The region of overlap is verysmall, as our algorithm finds many points with low average sidedistortion (but high central distortion), while index assignmentfinds some points with low central distortion, but high averageside distortion.

The explanation for this lies in the convexity constraint. Givenfour convex codecells per side partition, the central partition hasat most codecells. Running our algorithm for aseven-cell fixed-rate 1DSQ confirms that the lowest achievabledistortion for this scenario is about (which is the lowestcentral distortion achieved by the 2DSQ optimization). The factthat [27] achieves central distortions as low proves thatthe index assignment method can generate nonconvex side par-titions. For completeness, the largest number of central code-cells achievable with nonconvex side partitions is ,and those could be used to design a code with distortion as lowas .


Fig. 17. Histogram segmentation versus index assignment, for a fixed-rate 2DSQ with 2 bits per description of Gaussian data.

Fig. 18. Experimental results for a fixed-rate 2DSQ on the image Barbara.

B. Image Data

The results that follow show experimental results on the well-known Barbara image.

Fig. 18 shows the central distortion as a function of theaverage side distortion for 2DSQs with equal rates of 1, 2, and

3 bits per symbols (bps) in each description. Increasing the rateper description decreases all expected distortions. Decreasingthe central distortion causes the average side distortion toincrease.

Fig. 19 shows the experimental results for a fixed-rate 3DSQwith 1 bit per description. The graph shows the largest and


Fig. 19. Experimental results for a fixed-rate 3DSQ with 1 bps per description on the image Barbara.

smallest distortion values achieved by side partitions (labeledmaximal and minimal side distortions), the largest and smallestdistortion values achieved by intersection partitions (labeledmaximal and minimal intersection distortions), the distortionof the central partition, and the Lagrangian performance as afunction of the probability that each packet is received. When

, no description ever arrives at the decoder. As a result,the decoder’s expected distortion, , equals the source variance2982. In this case, the partitions are optimized independently.As a result, each achieves an expected distortion of ,which is the optimal distortion for a rate- 1DSQ on the givensource. Since the partitions are identical, there is no advantageto receiving more than one description, and the distortionsassociated with receiving two or three descriptions (the “inter-section” and “central” distortions, respectively) are identicalto the distortion for one description (the “side” distortion) As

increases from zero, so too does the probability of receivingmore than one description. As a result, the partitions beginto differentiate in order to improve the distortion associatedwith receiving more than one description. We observe a slowrise in the distortion of any one description accompanied by adecrease in the distortions for any two or all three descriptions.By , the probabilities of receiving zero, one, two, andthree descriptions are , and , respectively,giving an expected number of received descriptions of one.The average distortion from one description is , while theaverage distortion for two and three descriptions decreases to

and , respectively. (The distortion for zero descriptionsalways equals the variance, .)

As continues to increase, so too does the expectednumber of received descriptions. By , zero, one,two, and three descriptions are received with probabilities

, respectively, and the expected number of

received descriptions is two. The average distortions achievedwith one, two, and three descriptions are and .

Finally, when , the probability of receiving fewer thanthree descriptions equals , and the expected distortion observedat the decoder equals . While the decoder has received a totalof 3 bps, the achieved distortion equals the optimal distortion ofa rate- 1DSQ. This is, again, the price of convex codecells:the intersection of three rate- partitions with convex codecellscan form at most four distinct cells. The given code achievesthe optimal performance subject to this constraint on the centralpartition size.

The points and are the only places where we cantightly evaluate the cost of codecell convexity. We cannot findthis quantity for any other values of since the problem of op-timal MDSQ design (without the convexity constraint) remainsunsolved.

These results use the fixed-rated MDSQ design algorithm onan approximate coarse grid of 100 segments (the original imagehad 239 gray levels). This is only an approximation, but in ourjudgment a good one: we have experimented with 25, 50, 75,and 100 segments and found out that, while more segments leadto more data points on the convex hull, the performance does notchange much. That is, the 25-segment results closely interpolatebetween the more numerous 50-segment results, and so on.

ACKNOWLEDGMENT

The authors are enormously grateful to anonymous reviewerA, whose detailed and insightful suggestions improved this doc-ument significantly.

REFERENCES

[1] J. D. Bruce, “Optimum Quantization,” Ph.D. dissertation, MIT, Cam-bridge, MA, May 1964.


[2] D. K. Sharma, “Design of absolutely optimal quantizers for a wide classof distortion measures,” IEEE Trans. Inf. Theory, vol. IT-24, no. 6, pp.693–702, Nov. 1978.

[3] X. Wu, “Algorithmic Approach to Mean-Square Quantization,” Ph.D.dissertation, University of Calgary, Calgary, AB, Canada, 1988.

[4] X. Wu and K. Zhang, “Quantizer monotonicities and globally optimalscalar quantizer design,” IEEE Trans. Inf. Theory, vol. 39, no. 3, pp.1049–1053, May 1993.

[5] D. Muresan and M. Effros, “Quantization as histogram segmentation:Globally optimal scalar quantizer design in network systems,” in Proc.Data Compression Conf., Snowbird, UT, Mar. 2002, pp. 302–311.

[6] M. Effros and D. Muresan, “Codecell contiguity in optimal scalar quan-tizers,” in Proce. Data Compression Conf., Snowbird, UT, Mar. 2002,pp. 312–321.

[7] D. Muresan and M. Effros, “Data Compression Method and SystemUsing Globally Optimal Scalar Quantization,” U.S. Patent 6771831,Provisional application filed November 2001. Non-Provisional appli-cation filed November 2002. Patent Granted August 2004.

[8] S. Dumitrescu and X. Wu, “On Optimal Multi-Resolution Scalar Quan-tization,” in Proc. Data Compression Conf., Snowbird, UT, Mar. 2002,pp. 322–331.

[9] S. Dumitrescu and X. Wu, “Optimal multiresolution quantization forscalable multimedia coding,” in Proc. Information Theory Workshop,Bangalore, India, Oct. 2002, pp. 139–142.

[10] S. Dumitrescu and X. Wu, “Algorithms for optimal multi-resolutionquantization,” J. Algorithms, vol. 50, pp. 1–22, 2004.

[11] S. Dumitrescu and X. Wu, “Lagrangian global optimization of two-description scalar quantizers,” in Proc. IEEE Int. Symp. InformationTheory, Chicago, IL, Jun./Jul. 2004, p. 300.

[12] S. Dumitrescu, X. Wu, and G. Bahl, “Fast algorithms for optimal two-description scalar quantizer design,” in Proc. Data Compression Conf.,Snowbird, UT, Mar. 2004, pp. 42–51.

[13] S. Dumitrescu and X. Wu, “Optimal two-description scalar quantizerdesign,” Algorithmica, vol. 41, no. 4, pp. 269–287, Feb. 2005.

[14] N. Moayeri, D. L. Neuhoff, and W. Stark, “Fine-coarse vector quan-tization,” IEEE Trans. Signal Process., vol. 39, no. 7, pp. 1503–1515,July 1991.

[15] P. A. Chou and T. Lookabaugh, “Locally optimal variable-to-variablelength source coding with respect to a fidelity criterion,” in Proc. IEEEInt. Symp. Information Theory, Budapest, Hungary, Jun. 1991, p. 238.

[16] M. Effros, P. A. Chou, and R. M. Gray, “Variable dimension weighteduniversal vector quantization and noiseless coding,” in Proc. DataCompression Conf., Snowbird, UT, Mar. 1994, pp. 2–11, IEEE.

[17] Z. Xiong, C. Herley, K. Ramchandran, and M. T. Orchard, “Flexibletime segmentations for time-varying wavelet packets,” IEEE Trans.Signal Process., vol. 3, no. 1, pp. 9–12, Jan. 1994.

[18] P. Prandoni, M. Goodwin, and M. Vetterli, “Optimal time segmentationfor signal modeling and compression,” in Proc. IEEE Int. Conf. Acous-tics, Speech, and Signal Processing, Munich, Germany, Apr. 1997, vol.3, pp. 2029–2032.

[19] G. M. Schuster and A. K. Katsaggelos, “An optimal segmentation en-coding scheme in the rate distortion sense,” in Proc. IEEE Int. Symp.Circuits and Systems, Atlanta, GA, May 1996, vol. 2, pp. 640–643.

[20] T. H. Cormen, C. E. Leiserson, and R. L. Rivest, Introduction to Algo-rithms. Cambridge, MA: MIT Press/ McGraw-Hill, 1990.

[21] P. A. Chou, T. Lookabaugh, and R. M. Gray, “Entropy-constrainedvector quantization,” IEEE Trans. Acoust. Speech Signal Process., vol.37, no. 1, pp. 31–42, Jan. 1989.

[22] S. Herman and K. Zeger, “Variable fanout trimmed tree-structuredvector quantization for multirate channels,” in Proc. IEEE Int. Symp.Information Theory and Its Applications, Victoria, BC, Canada, Sep.1996, vol. 1, pp. 417–421.

[23] H. Brunk and N. Farvardin, “Fixed-rate successively refinable scalarquantizers,” in Proc. Data Compression Conf., Snowbird, UT, Apr.1996, pp. 250–259.

[24] H. Jafarkhani, H. Brunk, and N. Farvardin, “Entropy-constrained suc-cessively refinable scalar quantization,” in Proc. Data CompressionConf., Snowbird, UT, Mar. 1997, pp. 337–346.

[25] M. Effros, “Practical multi-resolution source coding: TSVQ revisited,”in Proc. Data Compression Conf., Snowbird, UT, Mar. 1998, pp.53–62.

[26] M. Effros and D. Dugatkin, “Multi-resolution vector quantization,”IEEE Trans. Inf. Theory, vol. 50, no. 12, pp. 3130–3145, Dec. 2004.

[27] V. A. Vaishampayan, “Design of multiple description scalar quan-tizers,” IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 821–834, May1993.

[28] V. A. Vaishampayan and J. Domaszewicz, “Design of entropy-con-strained multiple description scalar quantizers,” IEEE Trans. Inf.Theory, vol. 40, no. 1, pp. 245–250, Jan. 1994.

[29] M. Fleming and M. Effros, “Generalized multiple description vectorquantization,” in Proc. Data Compression Conf., Snowbird, UT, Mar.1999, pp. 3–12.

[30] M. Fleming, Q. Zhao, and M. Effros, “Network vector quantization,”IEEE Trans. Inf. Theory, vol. 50, no. 8, pp. 1584–1604, Aug. 2004.

[31] R. M. Gray and A. D. Wyner, “Source coding for a simple network,”Bell Syst. Tech. J., vol. 53, no. 9, pp. 1681–1721, Nov. 1974.

[32] D. Slepian and J. K. Wolf, “Noiseless coding of correlated informationsources,” IEEE Trans. Inf. Theory, vol. IT-19, no. 4, pp. 471–480, Jul.1973.

[33] Q. Zhao and M. Effros, “Lossless and near-lossless source coding formultiple access networks,” IEEE Trans. Inf. Theory, vol. 49, no. 1, pp.112–128, Jan. 2003.

[34] A. Gersho and R. M. Gray, Vector Quantization and Signal Compres-sion. Boston, MA: Kluwer, 1992.

[35] S. P. Lloyd, “Least squares quantization in PCM,” IEEE Trans. Inf.Theory, vol. IT-28, no. 2, pp. 129–136, Mar. 1982, Previously an un-published Bell Laboratories Technical Note (1957).

[36] A. György and T. Linder, “On the structure of optimal entropy-con-strained scalar quantizers,” IEEE Trans. Inf. Theory, vol. 48, no. 2, pp.416–427, Feb. 2002.

344 ieee transactions on information theory, vol. … · codewords deﬁne a quantizer decoder....

Documents