recent progress in stochastic processes- a surveywong/wong_pubs/wong30.pdf · recent progress in...

262 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-19, NO. 3, M A Y 1973

[36] F. Schweppe, “Evaluation of likelihood functions for Gaussian s$nnls,” IEEE Trans. Inform. Theory, vol. IT-11, pp. 61-70, Jan.

[37] T. Kailath, “A general likelihood-ratio formula for random signals in Gaussian noise,” IEEE Trans. Inform. Theory, vol. IT-5, pp. 350-361, May 1969.

[38] P. Swerling, “Parameter estimation accuracy formulas,” IEEE Trans. Inform. Theory, vol. IT-IO, pp. 302-314, Oct. 1964.

[39] A. V. Balakrishnan, “Filtering and prediction theory,” in Com- munication Theory. New York: McGraw-Hill, 1968.

[40] J. Ziv and M. Zakai, “Some lower bounds on signal parameter estimation,” IEEE Trans. Inform. Theory, vol. IT-15 pp. 386-391, May 1969.

[41] J. W. Carlyle, “Nonparametric methods in detection theory,” in A. V. Balakrishnan’s Communication Theory. New York:

McGraw-Hill, 1968. [42] C. W. Helstrom, “Quantum mitations on the detection of

coherent and incoherent signals,” IEEE Trans. Inform, Theory, vol. IT-11, pp. 4822490, Oct. 1965.

[43] R. W. Lucky, J. Salz, and E. J. Weldon, Principles of Data Com- munication. New York: McGraw-Hill, 1968.

[44] J. Aein, “Error rate for peak-limited coherent binary channels,” IEEE Trans. Commun. Technol., vol. COM-16, pp. 35-44, Feb. 1968.

[45] W. W. Peterson, Error Correcting Codes. Cambridge, Mass.: M.I.T. Press. New York: Wiley, 1961.

[46] J. M. Wozencraft and B. Reiffen, Sequential Decoding. Cam- bridge, Mass. : M.I.T. Press. New York: Wiley, 1961.

[47] J. M. Wozencraft and I. M. Jacobs, Principles of Communication Engineering. New York: Wiley, 1965.

Recent Progress in Stochastic Processes- A Survey

Invited Paper

EUGENE ‘WONG

Abstract-This paper surveys some of the literature on applications of stochastic processes published during the period 196&1972. This survey highlights recent developments in the application of martingale theory. Because this subject is relatively new to the engineering community, a tutorial exposition of some aspects of the martingale calculus is also included.

I. INTRODUCTION

TOCHASTIC processes cover such a large area of S diverse interests that a comprehensive and exhaustive survey is neither possible (at least for this author) nor even useful. In selecting the topics and the papers to be included in this survey, I have been guided by many considerations, of which the following three are of particular importance.

1) This survey should be biased.in favor of the audience of this TRANSACTIONS with respect to both the topics and the specific papers.

2) This survey should emphasize those topics that have enjoyed particularly dramatic advances in the recent period, at the expense of both more mature topics and embryonic ones.

3) Although the survey is not restricted to American papers, the bias is heavily American. No systematic coverage of the foreign literature has been attempted.

Manuscript received December 20, 1972. This work was supported by the National Science Foundation under Grant GK-10656X2 and the U.S. Army Research Office, Durham, NC., under Contract DAHCO4-67-C-0046.

The author is with the Department of Electrical Engineering and Computer Sciences and the Electronics Research Laboratory, Univer- sity of California, Berkeley, Calif.

Finally, this survey is intended to cover the period 1968- 1972. Earlier papers referred to in the survey will be listed separately at the end of the References.

In classifying the literature covered by this survey, I have found Slepian’s 1963 survey [44] very useful, even though some aspects of the earlier survey will not be included in this one. The titles of some of my categories have been borrowed directly from Slepian’s survey, and others represent only m inor modifications. My first three categories are : 1) descriptive features, 2) effects of operations and related topics, and 3) representations. Roughly speaking, these are classical categories which have been taken intact from the 1963 survey. Recent results in these areas will be discussed in Section II. The “related topics” referred to in category 2) are principally zero crossings and expansion of bivariate distributions, both of which are closely related to problems in nonlinear operations.

Category 4) covers likelihood ratios, martingales, and stochastic calculus. This has been a big growth area, in which pure and applied aspects of stochastic processes have interacted in a particularly fruitful way. The result has been the development of a dynamical theory of many important aspects of stochastic processes.

Category 5) is point processes and processes with sample discontinuities. Interest in point processes and processes with jumps has always been great. However, until recently most applications have been outside of the interest of information theorists. This is rapidly changing. First, point processes play an important role in a number of new areas

W O N G : RECENT PROGRESS IN STOCHASTIC PROCESSES 263

that are, or should be, of great interest to the information theory audience; e.g., photon channels and neural signal processing. Second, some techniques developed in connection with communication-theoretic problems may well lead to new views of point processes and their applications. In particular, it appears that a martingale approach will yield for processes with sample discontinuities the same kind of elegant results that have been obtained for sample-con- t inuous processes.

Section III will be devoted to an intuitive (and non- r igorous) exposit ion of the calculus of cont inuous martingales, as well as a survey of the literature of categories 4) and 5). In including extensive explicative material, I have perhaps departed from my role as the author of a survey paper. The length and the level of mathematical detail of Section III are also disproportionately great. However, I think this is an opportune time for a tutorial exposit ion on martingales which is oriented toward an engineering audience. I hope that the usefulness of such an exposit ion will justify the imbalance that it may have caused.

F inally, random fields are stochastic processes with a mu ltidimensional time parameter. From both the theoretical and the appl ied points of view, this is an area of growing importance. Unfortunately, results that are not straightforward extensions of one-dimensional counterparts are hard to come by. Some of the most interesting ideas and concepts in stochastic processes, e.g., martingales and Markov processes, depend on the well-ordered property of the time parameter. How these concepts must be mod ified to be appl ied to random fields is a promising and chal lenging area of research. A few papers have appeared already and will be reviewed in Section IV.

II. DESCRIPTIVE FEATURES, EFFECTS OF OPERATIONS, AND REPRESENTATIONS

Papers under the heading of “descriptive features” general ly deal with the derivation of some properties of an interesting class of processes. For example, the derivation of the covariance function of a Gauss-Markov process was a classic problem of this type. Blake and Thomas [I] studied a class of processes called “spherically invariant processes.” These processes, first introduced by Vershik [146], have the interesting property that their least-mean- square estimators are linear estimators, thus general izing an important property of Gaussian processes. Blake and Thomas showed that the characteristic function of a spherically invariant process (assumed to have zero mean) must have the form

Eew (i iI %A) = 4 (j$ I U ju,(EXtJtJ) (2.1)

and used it to prove a number of properties. Picinbono [2] showed that compound Gaussian processes (roughly speaking, processes which are conditionally Gaussian given their variances) formed a subclass of spherically invariant processes. For compound Gaussian processes 4 has the simple

characterization of being the Laplace-Stieltjes transform of a probability distribution on [O,co).

Jamison [3] studied “reciprocal processes,” which are symmetrized generalizations of Markov processes. A process is said to be reciprocal if, given X,,,X,,, the inside {X,, t, < t c tz} and outside {X,, t > t2 or t < tI} are independent. All Markov processes are reciprocal. The only stationary Gaussian reciprocal processes which are not Markov turned out to be sinusoids with a Rayleigh am- plitude and an independent uniformly distributed phase. It is interesting to note that Levy’s definition of Markovian random fields [137] reduces to that of reciprocal processes rather than that of Markov processes when the parameter is one dimensional [c.f. 1151.

“Unit processes” are two-valued (say, rt: 1) processes. W h ile every positive definite function is the covariance function of some process (in particular a Gaussian process), not every positive definite function is the covariance function of a unit process. Masry [4] has obtained a characterization of covariance functions for unit processes with a certain prescribed zero-crossing structure.

“Linear processes” are processes of the form

x, = s

&A dz, (2.2) A

where h is a deterministic function and 2 is a process with stationary and independent increments. Pierre [5], [6] has studied a number of properties of l inear processes, including sample function behavior, quadratic variation, and Kar- hunen-Lo&e representations.

Power spectral distributions have been considered by several authors [7]-[12], especially in connect ion with Gaussian processes.

Not surprisingly, the various aspects of Gaussian processes cont inued to occupy the attention of many workers. McGee [ 131, [ 141 has studied circularly complex Gaussian processes, which can be viewed as pairs of jointly Gaussian real processes {X,, Y,, t E T} satisfying the condit ion EXtxt+, = EY,Y,+, and EX,Y,+, = EYtXt+,.

Two topics which arose out of nonl inear operat ions on stochastic processes have enjoyed a remarkable amount of interest through the years. One is Price’s theorem [141], and the other is the diagonal expansion of a bivariate distribution, first considered by Barrett and Lampard [127]. Besides being extremely useful, there is something extra- ordinarily satisfying about them. Price’s theorem can be stated as follows: Let X = (X1,X,; * -,X,) be a collection of jointly Gaussian random variables (say, with zero mean for simplicity). Let pjj = EXiXj. Then for a reasonable function f we have the remarkable result

where

J;:jCx) = &?;'a,, f(x)*

264 IEEE TRANSACTIONS ON INFORMATION THEORY, M A Y 1973

Roughly speaking, this result is true because it is true for the class of functions

f(v) = exp (i C ukxt) k (2.4)

and hence for any function which can be approximated by linear combinations of f(u,x), e.g., in a Fourier-Stieltjes integral,

s f(v) dE(u). (2.5)

R”

This shows that Price’s theorem is true onZy for Gaussian random variables since if one takes f to be given by (2.4), then (2.3) results in a differential equation for the characteristic function which yields the Gaussian case as its unique solution.

McGee [13] obtained a version of Price’s theorem for circularly complex Gaussian processes. If the vector X = (X1,X,,* * * ,X,) is allowed to be replaced by a continuum of random variables (i.e., a Gaussian process {X,, t E T)) in Price’s theorem, then the resulting formula which replaces (2.3) must necessarily be one of functional calculus. Gorman and Zaborszky [19] have obtained such a formula and developed its connection with both Novikoff’s formula in turbulence theory and Hermite-Wiener functionals.

Some 16 years ago, Barrett and Lampard [127] observed that a number of important bivariate density functions admitted an expansion of the form

P(X,Y) = PI(X)P c GPn(x)Qrt(~) (2.6) ”

where p1 and pz are the marginal densities and P,,Q, are polynomials of degree n orthonormalized relative to p1 and pz, respectively. The most familiar example of such an expansion is Mehler’s expansion for two-dimensional Gaus- sian densities. Because the summation in (2.6) has a single index II even though it is a function of two variables x and y, Barrett and Lampard called these “diagonal expansions.” Stochastic processes with two-dimensional densities which admit diagonal expansions have important properties, especially in connection with nonlinear operations. During the period of this survey, Blachman [20], Brown [21], Cambanis and Liu [22], and Haddad [23] have all made contributions to some aspects of this problem. Other results on the effects of operations on stochastic processes appear in [24]-[41]. Pawula and Tsai [34], for example, considered the classic problem of finding the distribution of the output of a hard lim iter/RC filter combination with an Ornstein- Uhlenbeck process as input. They made an interesting conjecture about the output probability density function which was known to be correct for one value of the ratio between the two time constants (i.e., those of the input process and the RC filter). Even though this conjecture has since been disproved [149], [150], it appears to be a good approxima- tion over a wide range of values of the ratio of time constants.

Like the problem of finding the distribution of nonlinear functionals, first-passage time and zero-crossing distributions for stochastic processes are perennial problems. Every

once in a while a special problem gets solved, but any general solution even for the Gaussian case is still nowhere in sight. References [42]-[48] pertain to some aspects of these problems. Some years ago, the first-passage time problem for Gaussian processes with a triangular covariance function was partially solved by Slepian [143]. Shepp [46] has now completed the solution to this problem. Unfortunately, his solution involves multidimensional integrals which do not appear to yield closed-form expressions.

Most representation problems covered in this survey can be viewed in one or both of the following two ways.

A) Given a process {X,, t E T}, we want to find a simpler process {Y,, I E A} such that X and Y can be obtained from each other uniquely. “Simpler” can mean many things. For example, A may be a countable set, or Y may exhibit some orthoganality properties.

B) Given a second-order process (say, with zero mean for simplicity) {X,, t E T}, we want to find a Hilbert space s and a family of vectors {V,, t E T} in 2 such that

<v,,K> = Ex,X,, t,sET (2.7)

where (e) denotes inner product in x and the overbar denotes complex conjugate. The following examples are familiar to all.

1) Series Expansions: These are representations of type A) with a countable A and with X and Y related by a linear map. Special cases include the Karhunen-Loeve expansions and the sampling representations. Series expansions in a fairly general context have been considered in several papers [49]-[52]. Karhunen-Loeve expansions have been studied in [53]-[56], and sampling representations in [57]-[58].

2) Representations of the Karhunen Type: The following elegant result is due to Karhunen [ 1351, and it unifies the two types of representations A) and B). Suppose the covariance function of a second-order process admits a bilinear expansion of the form

R(t,s) = EX,X, = s

4(bw(4MW. cw A

Then there exists a process {Y,, 2 E A} such that

x, = s &tA dK (2.9) A

and Y has the orthogonality property EdY, dFA, = 6,,+(dA). Representations of the Karhunen type are obviously of

type A). They are also of type B) because one can take # to be the space of functions on A square integrable with respect to CL, take (Q,$) to be equal to j,, $(A)$(@(dA), and take V, = +(t;). We then have a representation of type W.

The two best-known special cases are the Karhunen- Loeve expansion, which stems from Mercer’s theorem

R(4s) = c M t>6”(s>? S, tET (2.10) n

where the 4” are the eigenfunctions of the kernel R, and the spectral representation, which comes from Bochner’s


theorem for stationary covariance functions, cont inuous Markov process with

R(t - s) = s

m exp [i2nA(t - s)] dF(A). (2.11) -cc

Generalizations of the spectral representation can be made to processes which are not stationary, and these have been considered in [59], [60].

YE; E(-&+A - x, I x,> = m (m)

y; ; ECW,+A - X,)2 1 X,] = a”(X,,t).

3) Canonical Representat ions (Filtered W h ite Noise) : G iven a second-order process, under what condit ions can it be represented as the output of a causal and causally invertible l inear filter with a white noise input? For stationary processes a complete answer is given by the Paley-Wiener condition, viz., a necessary and sufficient condit ion is that the spectral density function (which must exist) satisfy

Then under quite general condit ions X, can be represented as the unique solution to the stochastic differential equat ion w91, [1321

dX, = m(X,,t) dt + a(X,,t) dY,

where Y, is a Brownian motion (Wiener process). Haddad [23] considered some processes of this type in connect ion with Barrett-Lampard expansions (c.f. some similar results in [147]). s m b ‘(.f)l d f < co

-m1+f2 . (2.12)

The filter involved can be specified by a factorization of S(e). This problem can be general ized in a number of ways, the three most important ones being a) to allow nonstationary processes, b) to allow more than one white noise input, and c) to allow vector-valued processes. Results pertaining to these generalizations are reported in [61]-[65].

4) State-Equation Representations: An important special case of 3) is to represent a vector-valued process Xt by a pair of equat ions

$2, = A(t)Z, + B(t)Y,

and

x, = c(t)z(t) (2.13)

where the components of Y, are uncorrelated white noise processes. Results along these lines are of vital importance in recursive linear estimation and linear control, and important contributions have been made by Anderson et al. [61] and by Geesey and Kailath [63], [64].

5) Reproducing Kernel Hilbert Space Representat ions (RKHS): Let {X,, t E T} be a second-order process with covariance function R(t,s) = EX,x,. It turns out that there exists a representation of type B) with the following characteristics: a) 2 is a space of functions on T; b) for each t, R(t;) E &“; c) (R(t;),R(s;)) = EX,x,. That is, the family of vectors {V,, t E T} which represent {X,, t E T} is just {R(t, *), t E T}. RKHS representations were made popular by Parzen [140]. Campbel l [SO] discussed the relation between RKHS representations and series expansions. Kailath et al. [66] derived the form of the RKHS inner product for processes which admit state equat ion representations (2.13). Kailath and his co-workers have written a series of papers, some not yet published, on the applications of RKHS representations to estimation and detection problems. The details of these applications, however, are a little beyond the scope of this survey (see, e.g., [67]).

6) It8 Equation Representations: The representations discussed in l)-5) are all linear. An important nonl inear representation is the following. Let {X,, t 2 to} be a sample-

III. MARTINGALES AND ALL THAT

A stochastic process {X,, t E T} is said to be a martingale if

with probability 1 for all s I t. (3.1)

At first glance, it may seem rather surprising that processes of such a very special type should have extensive applications in stochastic problems of systems theory. I think there are two basic reasons’ for this. F irst, we shall see that martingales arise naturally whenever one needs to consider conditional expectations with respect to increasing information patterns. Second, a recently developed calculus of martingales provides us with a powerful analytical tool for deal ing with such conditional expectati0ns.l

Let FGt be a collection of events representing the known information at time t. Mathematically, g-t is a o field of events. In practice, 9, is typically the collection of events generated by one or more stochastic processes up to time t. Now, let {F6,, t E T} be a family of such information collec- tions. W e shall always assume that (9,) is increasing, i.e., t 2 s =z. Ft 2 FGs. In other words, we shall only deal with those situations in which information is cumulative. A process {X,, t E T} is said to be “adapted” to {pt, t E T} if, for every t, X, is completely determined by F-t. In mathematical terms, X, is g-r measurable for each t E T. This implies that E(X, I FGt) = X,, for each t. W e can now define X, as a martingale with respect to {F”,, t E T} if

E(X, 1 9J = A!,, with probability 1 for every t 2 s. (3.2)

Every martingale in the sense of (3.2) is necessari ly a martingale in the sense of (3.1), but a martingale in the sense of (3.1) need not be a martingale in the sense of (3.2) for every {Ft} to which it is adapted. The terminologies “{X,, Ft,

’ The concept of condit ional expectat ion as a random variable is of crucial importance to our subsequent discussion. Readers not already familiar with this concept can find this topic d iscussed in some detail in [126] or [129].


t E T} is a martingale” and “X, is an 9, martingale” are also often used.

Henceforth, we shall always assume T to be an interval, say, 0 I t I 1. Condition (3.2) is equivalent to a pair of conditions :

{X,, t E T} is adapted to {%-,, t E T}

E(dXt I %J = 0, for every t E (0,l).

(3.3a)

(3.3b)

(In martingale calculus, differentials dX, are always defined by forward differences: dX, = X,,,, - X,, dt > 0.)

Although everything that we are going to say has a natural and interesting generalization to discontinuous martingales, technical difficulties are m inimized by restricting ourselves to sample-continuous processes. It also simplifies matters to assume that the increasing families {%-,, 0 I t 5 l} are always continuous in the sense that

p-t = f-j %s S>f

% -, = CT u %s ( 1 S<f

where a( .) denotes the 0 field generated by the quantity within the parentheses. These technicalities, though important in ascertaining the precise results that can be obtained, will not play much of a role at the level of our rather intuitive discussion.

Historically, early mathematical interest in martingales centered around the martingale convergence theorem and martingale inequalities. Neither was of much interest in applications because their roles were primarily to facilitate proofs and not to facilitate calculations. The new interest in martingales among applications-oriented people is due to a large and highly interconnected body of recent results which have made martingale calculus an important analytical tool. I think that one can identify the following components in this body of results:

1) decomposition of a process into the sum of a martingale and a process of bounded variation;

2) quadratic variations of martingales; 3) stochastic calculus; 4) representation of martingales as stochastic integrals; 5) how martingales change under a change in probability

law.

For each of these topics I shall give a brief exposition combined with a survey of the relevant literature. Somewhat lengthy examples are also included. Although I think that these examples are interesting and useful in explaining details, they may be skipped for better continuity on the first reading.

A. Decomposition Let {X,, 0 I t I l} be a process, not necessarily adapted

to {%t, 0 I t I l} and not necessarily a martingale. Define 2, = E(X, 1 %J and construct a process B, by setting

dB, = E(dX, I % t). (3.4)

Because {%“t} is increasing, we can apply the role of iterated conditional expectation (see, e.g., [129]), and find

E @ % I @ -J = E{CEK+,, I % -t+d - Wh I @ --,>I I % r> !’ = EK,,, I%2 - EV, I %J = dB,.

Therefore, X, - B, = M , is a martingale with respect to {%f}. The most useful cases are those where the resulting process B is of bounded variation. In such cases X, is the sum of a martingale and a process of bounded variation, and such a process will be called a semi-martingale.2

If we take % t = 0(X,, 0 I z 5 t), then d&f, = dX, - E(dX, I X,, 0 I z I t) represents the new information in dX,. For this reason the martingale M , is often called the innovations process for X,. Innovations processes and their many applications have been developed by Frost and Kailath [76], [83], [121]. ii

Example 3.1: Let 5, = S, + qr, where S, is a process satisfying

s

1 EIS,I dt < 00

0

and qt is a Gaussian white noise independent of S. We can set

t x, = s

5, dz 0

s

t y,’ vr dz.

0

Then

s

f x, = S, dz + r,

0

and Y, is now a Brownian motion process. Now, define the following information fields :

xt k a(X,, z I t)

gt = a(r,, z I t)

% t = a(X,, Y,, z < t)

= a(& Y,, z < t)

and consider the decomposition of the X, process with respect to each of the families. We note that Y, is an 9, martingale and a g* martingale, but not an X, martingale.

a) {%t}: 8, = E(X, I %J = X,

dB, = E(dX, I 9,)

= E(S, I %J dt + E(dY, I %J

= S, dt.

* We follow Meyer [94] in our use of the term “semi-martingale.” It is more or less equivalent to the “quasi-martingale” of Fisk [130] and the “F-process” of Orey [139]. However, it should not be con- fused with the “semi-martingale” and “lower semi-martingale” of Doob [129], which are now almost universally referred to as “sub” and “super” martingales.

WONG: RECENT PROGRESS IN STOCHASTIC PROCESSES 261

Hence M , = Y, and

s

f x, = S, dz + r,

0

is already the decomposit ion with respect to (Ft}.

b) {.Z2}: 8, = X,

dB, = E(dX, 1 2-J

= E(S, 1 X,) dt + E(dY, I % ‘3

Note that we have used the rule of iterated conditional expectation (see, e.g., [129]). If we define 5, as E(S, I L!TJ, then

s

f

M , = X, - s^, dz 0

must be an % ‘t martingale. W e have already ment ioned that the process M , is called the innovations process for X,. It will turn out that M , is a Brownian motion process.

4 PA: 2, = EGG I gu,) dB, = E(S, I g/3 dt

= (ES,) dt. Hence

s t 2 , - ES, dz = M , 0

is a C!/4/t martingale. It is easy to show that in this case M , is equal to Y,, so that the conclusion is indeed true.

B. Quadratic Variation Let {M,, F-t, 0 5 t < l} be a sample-cont inuous second-

order martingale and consider the decomposit ion of X, = M ,’ with respect to {g-,}. Here, 8, = X, and

dB, = E(dX, I LFJ

= El?%,, - M t2 I F”,l = El@ft+dt - 4)’ I F tl + 2312 [W, +-c,t - M tNft I g”,l = E[(dMJ2 1 F”,]. (3.5)

If we take B, = 0, then, because X, - B, is a martingale, EB, = E(X, - X0) = E(M, - MO)‘.

Since B, is nondecreasing (dB, 2 0), EB, = 0 implies EB, = 0, which implies M , = M O , for each t. Therefore, for a sample-cont inuous second-order martingale which is not a constant, the increasing process B, is always positive. W e shall denote the increasing process associated with a cont inuous second-order martingale M , by (M,M),. If M t and Nt are two Ft martingales, then M t + Nt and M t - Nt are both gGt martingales, and we can define the product-variation process (M,N), by

(M,N), = f[(M + N, M + N), - (A4 - N, M - N),]. (3.6)

An important characterization of (M,M), is that it can be constructed by partitioning the interval [O,t] with a sequence of partitions {t,(“)} which refines to zero suffi- ciently rapidly and by comput ing

(M,M), = lim c [M(t,(‘$J - M(t,(“))12. (3.7) n-tee Y

Now suppose, as often happens, M , is a martingale with respect to two different families of CT fields, {gt} and (9,); then the construction (3.7) shows that (M&V), is an in- trinsic property of M , (i.e., is independent of the (r fields), even though the definition (3.5) makes use of {Ft}. W e note that (3.7) makes precise the notion that (dM,)2 N d(M,M),, which interestingly enough is nonnegligible. Since M t is sample continuous, (3.7) shows that it cannot be of bounded variation.

If X, is a sample-cont inuous semi-martingale, X, = B, + M ,, where B, is of bounded variation, M , is a martingale, and both are sample continuous, then it is easy to show

c [X<t!“!A - W ,(“))12 n-m’ W ,M),. (3.8) v In other words, the quadratic variation of X, is the same as that of M ,, with B, contributing nothing to it. This has interesting consequences, as the following example shows.

Example 3.2: Take the situation of example 3.1 with

s

f x, = S, dz + r,.

0

Decomposit ion with respect to {Ft} and {Xt} yields f f

x, = s

S, dz + r, = s

s, dz + M , 0 0

where 9, = E(S, I .!Zt). Equation (3.8) shows that (Y,Y), = (M,M),. Since Y, is a Brownian motion process independent of S,, we have

d( Y,Y), = E[(dYJ’ I S,, y,, z I t] = dt.

Hence, (M,M), = t. Later we will show that this implies that M , is a Brownian motion. This result was derived by Frost [121] and by Kailath [87].

Results on quadratic variation for Gaussian processes were derived by Baxter [128] quite independently of any consideration of martingales. Baxter’s theorem was used by Slepian in his well-known work on singular Gaussian detection [142]. The observation that the martingale components of two different decomposit ions of the same process must have the same quadratic variation was explicitly made by Wong [96]. The quadratic variation of r ight-continuous martingales has been studied by Doleans [69].

C. Stochastic Calculus Let (M,, F’t, 0 I t I 1 } be a sample-cont inuous martin-

gale. Let {& 0 I t I I} be a process adapted to {F-,, 0 I t I l} such that

s

1 4,” d(M,M), < co with probability 1. (3.9)

0


One can define the stochastic integral JA c$, d&it by definition as

s ’ 4, dM, = lim c &@“[M,l”ll - M tY(“)] (3.10)

0 n-cc Y

for a suitable sequence of partitions (t,(“)} of the interval [O,l]. We note that this definition is consistent with our interpretation of dM, as a forward difference.

A more interesting way of looking at stochastic integrals is to consider them as transformations of martingales into martingales, or more naturally as local martingales into local martingales. What, then, are local martingales? Let {M,, 0 I t I l} be a sample-continuous process adapted to {%,, 0 < t I 11. Define

s ’ $J, dX, = ’ 4, dB, + 1 4, dM,

0 s (3.16)

0 s 0

where sh 4, dBS is a Stieltjes integral and Jh 4, dM, is a stochastic integral. One of the most important tools of stochastic calculus is the transformation rule for local semimartingales which can be stated as follows: Let X = (~I,~,,- * - ,X,) be a vector-valued process, the components of which are continuous local semi-martingales. Let f(x,t)

be a complex-valued function twice continuously differen- tiable with respect to the components of x (including m ixed partials), and once with respect to t. Then

Z,(W) = m in {t: IMt(o)l 2 n}. (3.11) f(XJ> = f(X,,O) + We say that3 M is an % t local martingale if, for every ~1,

1’ f<Xd> ds + T 1’ fi(XsJ> dxis 0 0

M m in(t,r,) is an %-t martingale.4 In other words, every truncated version of M is a martingale. The quadratic + i C [ fij(xSJ> d<Mi,Mj)s (3.17)

hJ 0 variation process of a local martingale can be defined by

where (MA >t = h <Main (t,r,),Mmin (t,d (3.12)

“-.a,

and the product variation process (M,N) can again be defined using (3.6).

Now, denote by &reloC the collection of all sample-continuous local martingales with respect to the same family {%“t, 0 I t I l}, such that they vanish at t = 0. Let ME A,oc and let {c$~, 0 I t I l} be adapted to {%,, 0 I t < 1} and satisfy (3.9). Then, the stochastic integral

f,!T f.2 at’ ’ axi '

and M i denotes the local martingale component of Xi. The outline of stochastic calculus just given is based on

the material in the now famous paper of Kunita and Watanabe [136], who generalized Ito’s original results on stochastic integrals with respect to Brownian motion processes. Equation (3.17), in particular, is a generalization of the famous differention rule of Ito [ 1331.

Example 3.3: Suppose (M,, % t, 0 I t < l} is a sample- continuous martingale with (M,M), = t. We shall show that M must be a Brownian motion and, for each t,

Wt+s - M ,, s 2 0} is independent of %-r. To show this, takef(M,) = exp (iuM,). Then

Z, = s

t $,dM, 0

defines a local martingale which is uniquely determined by the property

V,V, = s

‘4, d<M,Y), (3.13) 0

for every YE JZloC. We note that (3.13) implies

<Z&f), = s

* 4, &M&f), (3.14) 0

<Z,Zh = s * 4, &M,Z), 0

= s

’ 4, d<ZM >, 0

= s

t +S2 d(M,M),. (3.15) 0

exp (iuM,) - 1 = s

f iu exp (iuM,) dM,

0

s

t -3 u2 exp (iuM,) ds.

0

The martingale property of the stochastic integral term implies that

E(exp (iuM,) I % t,) = exp (iuM,,)

s

f - +M’ E (exp (iu M ,) 1 B,,) ds

which can be solved for E(exp (iuM, 1 %J to yield

We say X = B + M is a local semi-martingale if B is of E(exp (iuM,) I %J = exp [iuM,,) exp (-3u2(t - to>]

bounded variation and M is a local martingale. For a local semi-martingale X, the integral i: $, d/Y, has a natural

or

E[exp [iu(M, - M ,,)] ) %-J = exp [-+u2(t - to>] 3 From this point we write M interchangeably with Mt. 4 Local martingales which are not sample continuous can also be

which proves the assertion. Returning to the innovations defined (see [94], [1381), but not quite so simply. process M studied in examples 3.1 and 3.2, we see that M


must be a Brownian motion process since it is a sample- cont inuous martingale with (M,M), = t.

Example 3.4: Take X, = M , -- &(M,M), and F(X,) = ext. Then

t I+ - 1 =

s exs dX, + +

0 s

f exS d(M,M),

0

s

f = exs dM,.

0

In other words, 2, = exp (M, - +(M,M),) is a solution to the equat ion

s

t z, = 1 + Z, dM,.

0

In fact, Doltans-Dade [70] has proved that it is the only solution. (See also [124].) It follows that 2 must be a local martingale. It turns out that {Z,, 0 4 t I l} is a martingale if and only if EZ, = 1.

D. Representat ion of Local Martingales Let {IV,, 0 I t I 1 } be a Brownian motion process and

let (%-,, 0 I t I l} be generated by W . This means that % t = a(W,, z I t), and every event in 8, can be expressed in terms of {IV,, r I t }. Under these condit ions one can show that every %-, local martingale is of the form

Z, = Z, + s

’ C#J, d W ,. (3.18) 0

This very important result can be inferred from works of Ito [134], Kunita and W a tanabe [136], and Clark [68].

The preceding result can be general ized as follows. Sup- pose now {X,, 0 I t I l} is a local semi-martingale which generates {%“,, 0 < t I l}. Let X have a decomposit ion X = B + W with respect to {%t}, where B is of bounded variation, W is a Brownian motion, and both are adapted to {%t}. Then, every % “lt local martingale is again of the form given by (3.18). This recent result is due to Fujisaki et al. [77]. It has been conjectured (probably first by Kailath, cited by Frost in [121]) that in fact {IV, - IV,, 0 I s I t } and {X, - X0, 0 I s I t } generate the same information, i.e., the same G field, for every t. This result, if true (it almost certainly is under quite general conditions), is much stronger than the representation (3.18) and will be important in many applications. (This result, under suitable conditions, has been proved by Clark [ 1511.)

Example 3.5: Suppose that X and Y are two semi-martingales, representing the state and the observation for a dynamical system, which satisfy equat ions of the form

s

f x, = X, ds + M ,

0

s

f y,’ X, ds + W ,

0

where X0 = Y, = 0. Let % , = o(X,,M,,W,; 0 2 s I t) and g, = o(Y,, s < t). W e assume M , and W , are % “lt

martingales and IV, is also a Brownian motion. F inally, let 2, denote E(X, I g’,). W e note that 8, and Y, are 9/t semimartingales with representations

s

f

x, = 2, ds + M , 0

s

f

r,= r?, ds + pr. 0

Since fi, is a gy, local martingale it must be of the form

s

f Ia, = K, dfiS.

0

To find K,, we use the transformation rule on the gt local semi-martingale g,Y, and on the %* local semi-martingale X,Y, and find

X,y, = 1; X, dY, + 1; r, dX, + (M,W),.

Since ztYt = E(X,Y, I g/,), we have

E(d(%Yt) I gt) = W W J ’J I gt> which yields

8,E(dY, I g/3 + Y,E(d$ I g/,) + Kt dt

= EGG dK I g/3 + Y,W X , I gy,) + E[&M,JO, I qtl. Since dY, = X, dt + d W ,, we obtain

2,’ dt + K, dt = E(Xt2 I gyt) dt + E[d(M,W), 1 gyt]

or

K, = E[(X, - X,>’ I gt] + E it <M,W), I g/t . I This example solves a simple form of the recursive filtering problem.

E. Change in Probability Measure Let fi be a basic space and % a collection of events. W e

assume that % satisfies the axioms of a r~ field, viz., closure under Boolean set operat ions and sequential lim its. Let 9 and PO be two probability measures on % . W e are interested in how processes on (Q,%) behave under the two different probability measures. Such questions arise naturally in many situations, the most familiar one to communicat ion theorists being signal detection. In a detection problem, we have the following situation: {X,, 0 I t I r} is a process representing our observation, % is a collection of events generated by X, and 9 and 8, are probability measures on % corresponding to X being “signal plus noise,” or “noise only,” respectively.

W e say B is absolutely cont inuous with respect to PO if B,(A) = 0 implies 9(A) = 0, for every A in % . This situation is denoted by 9 << PO, a notation suggested by


the fact that if go(A) = 0, then B(A) cannot be bigger. If B and 8, are mutually absolutely continuous, then we shall say % and PO are equivalent (written B - PO). If B << PO, then the celebrated theorem of Radon-Nikodym states that there exists a nonnegative random variable A on (R,%) such that

P(A) = s A(w)po(do) (3.19)

A

for every event A in % . The random variable A is called the Radon-Nikodym derivative (of % with respect to go) and is denoted by dP/dPo. It is also a generalization of the likelihood ratio encountered in statistics, and we shall call it the likelihood ratio. If % is generated by a stochastic process {X,, 0 < t < l}, then A must be a functional of (X,, 0 I t 5 l}; i.e., A(w) = f(X,(o), 0 5 t 5 1). A ques- tion important in practice is to determine this functional. Here, martingale theory plays an important role.

Let % = 0(X,, 0 < s 5 1) and %* = 0(X,, 0 5 s 5 t). We shall assume B - PO. Denote by E. expectation with respect to 8,. Then

the rules for taking conditional expectation with respect to two probability measures, we have

Hence, {Z,, %-t, 0 I t 5 l} being a local martingale under B is equivalent to {Z,A,, % ;t, 0 5 t I l} being a local martingale under 8,. Because of the transformation rule (3.17), we have

Z,A, = Z, + 1; Z, d/i, + 1; As dZ, + (Z,A),.

Since PO Z, dA, + Z, is already a local martingale, this implies that

s

f As dz, + C-W>,

0

is a local martingale. If A is given by (3.23) and (3.24), then

(.&A>, = s

’ A, d(Z,M >, 0

A, = Eo[A I % ,I (3.20) so that Z,A, being a B, local martingale implies that defines a martingale with respect to (%O,{%^f}). If, with respect to PO, X is a Brownian motion, then the representa- s

'&[dZ, + &-CM),1 tion result we quoted in the last section implies that 0

is a PO local martingale. Thus we have the following result. At = 1 +

s ’ 4, dX, (3.21) Let {Z,, % ,, 0 I t < l} be a continuous % local martin-

0 gale and let

for some $. If we set q5, = $,A, and

M , = s

’ $, dX, (3.22) 0

then (3.21) becomes

A, = 1 + s

’ A, dM,. (3.23) 0

The results outlined in example 3.4 now imply that A must have the form

4 = exp CM, - SW,MM

= exp (3.24)

E, ($ I % t) = exp(M, - ‘2 (M,M),). (3.26)

Then Z, + (ZM), = K (3.27)

is a local martingale with respect to (PO,{%,}). We note that since terms of bounded variation do not contribute to quadratic variation or product variation, we have

(ZW, = <X&O, (3.28)

under either probability. Therefore, our results can also be stated as follows. If X is a 8, local martingale and % is specified by (3.26), then

Z = Xt - (XJO, (3.29)

What has been shown is that the likelihood ratio of any is a B local martingale. The preceding result is a generaliza- probability measure equivalent to the Wiener measure must tion of a theorem due to Girsanov [131], who dealt with have the exponential form given by (3.24). It remains only the special case where X is a stochastic integral with respect to give an interpretation of the process $. Before interpreting to some Brownian motion under PO. This generalization $, we must digress a moment and consider how martingales appears to be new and is being published here for the first under S and PO are related. time.

Let B - PO on (Q,%). Let (%-,, 0 I t I l} be an in- The development leading to (3.24) showed that if under creasing family in % , and define as before P0 {X,, 0 I t < l} is a Brownian motion process (Wiener

process), then any probability measure B which is ab-

A, = E, ($f$%t). (3’25) solutely continuous with respect to PO on 8 = a(X,, 0 _< s I 1) must be of a form specified by

Suppose (Z,, % t, 0 5 t < 1 } is a sample-continuous local dB martingale under 9’; what is it like under PO? First, under z = exp 0

( j-’ II/, dXs - ; 1’ $s2 ds) . 0

WONG: RECENT PROGRESS IN STOCHASTIC PROCESSES 271

Take M , = & $, dX,, so that (3.29) becomes

X, - s

’ II/, ds = Z,. 0

Now suppose 2, is a local martingale under 8. If it is a martingale, then we have

E(dX, 1 9-J - i+bt dt = E(dZ, 1 FJ = 0 (3.30)

which specifies $. Example 3.6: Consider a detection situation where the

observation consists of either a signal plus a Gaussian white noise or a Gaussian white noise only. W e can avoid the technical difficulties associated with white noise by consider- ing the integrated version of the observation process. Let {X,, 0 I t < l} be the integrated observation process. Let 9 and 8, be two probability measures such that under So, 1, is a Brownian motion; and under 9, X, - r. S, dz is a Brownian motion. Let 8” and 8,” denote the restrictions of B and PO to X = 0(X,, 0 I s < 1). Suppose we assume P’” N 8,“. What is d~xld.Pox? W e note from (3.24) we must have

d9”” ~ = exp dBoX

$, dX, - ; 1’ $s2 ds) 0

and from (3.30) we have

$, dt = E(dX, 1 X,) = E(S, dt 1 X,) = St dt

provided we assume that under 9 the future of the noise is independent of the past of the observation (surely a reasonable assumption). It follows that

di??” dcc9’oX - ev $, dt - ;

where 3, = E(S, 1 X,, z I t). This very well-known result relates estimation and detection in a very neat way; and is due, in the first instance, to Sosulin and Stratonovich [ 1451, who derived it for a restricted class of signals. Ma jor steps in general izing this result were taken by Duncan [72]-[74] and Kailath [89]-[90]. A martingale approach to the problem, first suggested by the work of Kunita and W a tanabe, was effectively exploited by Duncan [73]. The full generality of the result was first suggested by Kailath [87]. O ther results on this topic can be found in [80]-[82], [91].

I have restricted the discussion thus far to sample- cont inuous martingales. Many of the results can be extended to martingales which may have discontinuities, especially in those cases where “{F”,} has no fixed points of discontinuities” [136]. From a practical point of view such extensions are of great importance because point processes and discrete-state processes arise frequently in practice. Results pertaining to those processes are discussed in [97]-[108]. Frost [121], Snyder [107], [lOS], and others [106], [118] have obtained results for discontinuous processes concerning innovations processes and likelihood ratio formulas. A recent dissertation of Brtmaud [ 1181 considered estimation, detection, mode ling, and some optimization problems involving point processes, all in the context of martingale

theory. Although the power of the martingale approach as developed by Kunita and W a tanabe [136] and by Doleans- Dade and Meyer [71] is yet to be fully exploited, this situation is not likely to remain for long in view of the number of talented workers active in the field.

IV. RANDOM FIELDS

The topic of random fields does not really deserve a special section on the basis of the number of results that have been obtained. It is being treated in a separate section, first because it does not seem to fit anywhere else, and second because it gives us an opportunity to look at the broad spectrum of activities and see how many can be general ized or mod ified to include random fields. By a random field I shall mean a family of random variables {X,, t E T} where the parameter space T is mu ltidimensional. In contrast, we have taken stochastic processes to mean families of random variables with a one-dimensional parameter space. The difference in parameter space man- ifests itself in two important ways : through the geometry of T and through the ordering property of T. Thus one expects that the generalization of results which depend on one or the other of these two properties would be both interesting and difficult. On the other hand, the generalization of results which involve neither geometry nor ordering can be expected to be quite straightforward.

Examples in “representations” illustrate these points rather well. The spectral representation is associated with stationarity, which in turn is associated with the geometry of the real line, viz., translation invariance. Spectral representation for random fields not only exists but is far more interesting because the variety of invariances for the parameter space is much richer. Here it is not the dimensionality which matters but the geometric structure. Spectral representations of random fields have been treated fairly ex- haustively (see, e.g., [148]). On the other hand, the representation of a process as a white noise filtered by a causal and causally invertible filter involves the well-ordered property of the time parameter. There appears to be no natural generalization of such representations, nor of spectral factorization. F inally, the Karhunen-Loeve expansion makes no specific reference to either the geometry or the ordering of the parameter space. Thus except for the difficulty of solving the associated integral equation, generalization to the mu ltiparameter case is relatively trivial.

O f those properties of stochastic processes which depend on the ordering of the parameter, two particularly important ones are the Markovian and martingale properties. It is interesting to note that their generalizations to the mu ltiparameter case have taken different routes. Although several ways of general izing the Markovian concept are possible, one due to Levy has enjoyed the greatest interest. Starting with a definition of the Markovian property as “conditional independence of the future and the past given the present,” Levy [137] identified the present as an (n - l)- dimensional surface separat ing the n-dimensional parameter space into two parts which were identified as the past and the future. Markovian fields in Levy’s sense have been


studied by Wong [I 141, [115], who focused on the characterization of Gauss-Markov fields which are also stationary (with reference to a suitable geometry). Woods [117] has considered a modification of Levy’s definition for discrete- parameter fields.

A simple way of generalizing the martingale concept to random fields is to define a partial ordering on the parameter. For example, if the parameter space is the positive quadrant {z = (x,~), x,y 2 0}, an obvious partial ordering is defined by

A natural way in which such a partial ordering arises is to consider a martingale as the integral of white noise on a class of sets in an n-dimensional space and use set inclusion to induce a partial ordering. Such an approach has the additional advantage of connecting the concept of martingales to the way they are to be used. Although some preliminary results in this direction have appeared [109], [l 16],5 it is not yet clear at this point whether martingales and stochastic calculus will play the same fruitful role that they have for one-parameter processes.

Other topics in random fields that have received attention include random surfaces [113], scattering [ill], Wiener processes [112], and likelihood ratios [ 1 lo], [112].

V. CONCLUSION

Early successes in the application of stochastic processes were often associated with the removal of one or both of the “dynamical” and the “probabilistic” aspects of the problem. For example, wide-sense stationarity gives rise to a frequency-domain (or spectral) analysis in which the dynamics all but disappear. Problems involving probability are often successfully transformed into problems of analysis by dealing with averages or with density functions. These approaches, though lim ited in one way or another, gave answers which were often in closed form. There has been a growing realization in recent years that for many problems the solution should explicitly exhibit the evolution of the system structure, of the various processes, and of the information pattern. One often prefers an algorithmic solution, especially a recursive one, to a closed-form solution, even if the latter were possible. This dynamical approach to problems in stochastic processes, which started with the recursive linear filtering theory of Kalman and Bucy, has been given a powerful tool in the form of the martingale calculus that has been developed by Kunita and Watanabe and by Meyer and his co-workers. I think that this development has been a most important event which warrants the coverage that I have given it in this survey.

The role of the likelihood ratio in detection problems is familiar to most of us. It is less widely recognized, however, that likelihood ratios can play a role in the study of stochastic processes similar to that of a probability density

5 I do not want to include unpublished works in the References, especially those of my own. However, I shall be happy to make avail- able to any interested reader a technical report by M. Zakai and myself on this subject.

function for a finite number of random variables. In either case one can evaluate an expectation by integrating with respect to a fixed-reference measure. In the case of a finite number of random variables, each of which possesses an absolutely continuous distribution function, the reference is the Lebesgue measure. For a process absolutely continuous with respect to a Wiener process (i.e., its probability law is absolutely continuous with respect to the Wiener measure), the reference is the Wiener measure. Of course, integrating with respect to the Lebesgue measure in R” is at least moderately easier than integrating with respect to the Wiener measure in a function space. However, the establishment of an exponential formula for the likelihood ratios as outlined in Section III cannot help but improve the situation.

It may be useful to make explicit a number of open problems and new directions which were indicated in Sec- tions III and IV.

1) How does the calculus of martingales generalize when the assumption of sample continuity is dropped, and how can the generalized technique be applied to problems in detection and filtering? As was mentioned earlier, a number of papers on this subject have already appeared, and to my knowledge several workers are currently active in the field. However, I have not yet seen the problems stated with the degree of generality that is possible, nor have the approaches exploited the full power of the martingale theory. In particular, I believe that the crux of the matter is closely associated with the “Levy system” [136, p. 2281, which characterizes the discontinuous components of a martingale. Thus far, I have not seen this explored in connection with problems of filtering and detection.

2) The innovations problem is crying out to be solved. We recall that, the innovations process associated with {X,, 0 5 t 5 t } is the martingale {M,} defined by

dM, = dX, - E[dX, 1 A’,, 0 5 z I t].

The innovations problem can be stated as follows: Under what condition is every functional of (X,, 0 I t < T} necessarily representable as a functional of {M,, 0 I t I T}? In other words, when does {M,, 0 5 t 5 T} contain all the information contained in {X,, 0 I t I T}?

3) By the use of likelihood ratios, we can often transform a process into a simpler process, at the cost of carrying along the likelihood ratio. If the form of the likelihood ratio is simple, this procedure may be an effective one. This idea has already been exploited for Wiener processes in connection with stochastic control problems. There are many applications involving processes of other types (e.g., Poisson) for which this idea may be useful.

4) An obvious application of random fields is to image processing problems. Thus far, however, results have been disappointing. Martingales with a partially ordered parameter may yield simple expressions for likelihood ratios and an effective formulation of recursive estimation problems. Recursive processing in the sense of partial ordering may be of some importance in image processing.

In conclusion, I think the period covered by this survey has been a most interesting one for those of us working on


the applications of stochastic processes. There have been some ma jor changes in directions and jumps in levels of mathematical sophistication which have left us a little breathless. The result has been a closer interplay between pure and appl ied aspects of probability theory, a greater interaction between information theorists and control theorists, and a new approach to some important problems.

ACKNOWLEDGMENT

I am grateful to Drs. G . David Forney, Jr., and Thomas Kailath for many valuable comments on an earlier draft. Had I been able to incorporate all their suggestions, this would have been a much better paper.

REFERENCES

I, Descriptive Features [l] I. F. Blake and J. B. Thomas, “On a class of processes arising

in l inear estimation theory,” ZEEE Trans. Inform. Theory, vol. IT-14, pp. 12-21, Jan. 1968.

[2] B. Picinbono, “Spherically invariant and compound Gaussian stochastic processes,” IEEE Trans. Inform. Theory (Coriesp.), vol. IT-17, pp. 77-79,. Jan. 1970.

[3] B. Jamison, “The stationary Gaussian case,” Ann. Math. Statist., vol. 41, pp. 1624~163?, 1970.

[4] E. Masry, “On covartance functions of unit processes,” SIAM J. Appl. Math., vol. 23, pp. 28-33, 1972.

151 P. A. Pierre. “On the indeuendence of l inear functionals of l inear _ - processes,” ‘SIAM J. Appi. Math., vol. 17, pp. 624-637, 1969.

[6] -, “The sample function regularity of l inear random processes,” SIAM J. Appl. Math vol. 17, pp. 1070-1077, 1969.

[7] A. Arcese and E. W. Trombini: “ Variances of spectral parameters with a Gaussian shape,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-17, pp. 200-201, Mar. 1971.

[8] J. Badcock, “Angle-modulat ion power spectra,” IEEE Trans. Commun. Technol., vol. COM-18, pp. 313-318, Aug. 1970.

1 N. M. Blachman and G. A. McAlpine, “The spectrum of a high- index FM waveform: Woodward’s theorem revisited.” IEEE

[91

HOI 1111 W I

H31

v41

1151

1161 u71

1181

Trans. Commun. Technol., vol. COM-17, pp. 201-208, Apr. 1969. F. Donati, “Finite-time averaged power spectra,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 7-16, Jan. 1971. V. K. Prabhu and H. E. Rowe, “Spectral density bounds of a PM wave,” Bell Syst. Tech. J., vol. 48, pp. 769-816, 1969. M. I. Schwartz, “Distribution of the t ime-average power of a Gaussian process,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 17-26, Jan. 1970. W. F. McGee, “Circularly complex Gaussian noise-a Price theorem and a Mehler expansion,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-15, pp. 317-319, Mar. 1969. -, “Complex Gaussian noise moments,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 149-157, Mar. 1971. R. A. Gonsalves, “Bounds on the variation and bandwidth of a bandlimited signal,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-15, pp. 499-500, July 1969. A. H. Gray, Jr., “On Gaussian noise envelopes,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 522-528, Sept. 1970. A. Papoulis, “Narrow-band systems and Gaussianity,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 20-27, Jan. 1972. H. Sato, “A note on stationarity of some random processes,” Inform. Contr., vol. 13, pp. 274-279, 1968.

II. Eflects of Operat ions and Related Topics [I91

1201

1211

C. D.. G&man and J. Zaborszky, “Functional calculus in the theory of nonl inear systems with stochastic signals,” IEEE Trans. Inform. Theory, vol. IT-14, pp. 528-531, July 1968. N. M. Blachman, “The uncorrelated output components of a nonlinearity,” IEEE Trans. Inform. Theory, vol. IT-14, pp. 250- 255, Mar. 1968. J. L. Brown, Jr., “On the expansion of the bivariate Gaussian probabil ity density using results of nonl inear theory,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-14, pp. 158-159, Jan. 1968.

[22] S. Cambanis and B. Liu, “On the expansion of a bivariate distribution and its relationship to the output of a nonlinearity,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 17-25, Jan. 1971.

[23] A. H. Haddad, “Dynamical representat ion of Markov processes of the separable class,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 529-534, Sept. 1970.

[251

1261

~271

1281 [291

[301

1311

[321

I331

[341

[351

[36] S. 0. Rice, “Second and third order modulat ion terms in the distortion produced when noise modulated FM waves are filtered,” Bell Syst. Tech. J., vol. 48, pp. 87-142r 1969.

I W. Sollfrev. “Hard limiting of three and four smusoidal sianals.” [371

[381

[391

[401

[411

1421

[431

[441

[451

[461

I471

IEEE Trans. Inform. Theory, vol. IT-15, pp. 2-7, Jan. 196-9. ’ F. G. Stremler and T. Jensen, “The probabil ity density function for the output of an analog crosscorrelator with bandpass inputs,” IEEE Trans. Znform. Theory (Corresp.), vol. IT-16, pp. 627-629, Sept. 1970. A. N. Venetsanopoulos and F. B. Tuteur, “Upper bound on the signal-to-noise ratio at the output of a threshold receiver,” IEEE Trans. Znform. Theory (Corresp.), vol. IT-17, pp. 753-755, Nov. 1971. D. D. Weiner, J. F. Spina, and A. W. Fitch, “On the correlation function of signal plus Gaussian noise passed through nonl inear devices,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-17, pp. 613-614, Sept. 1971. W. D. Wynn,, “A second order statistical analysis of the opera- tion of a limtter phase detecter-fi lter cascade,” Bell. Syst. Tech. J., vol. 48, pp. 233-248, 1969. R. G. Gal lager and C. W. Helstrom, “A bound on the probabil ity that a Gaussian process exceeds a given function,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-15, pp. 163-166, Jan. 1969. M. Goldman, “On the first passage of the integrated W iener process,” Ann. Math. Statist., vol. 42, pp. 2150-2155, 1971. A. J. Rainal, “Approximate and exact results concerning zeros of a Gaussian noise,” Bell Syst. Tech. J., vol. 47, pp. 461-463, 1968. -, “First and second passage of a sine wave plus noise,” Bell Syst. Tech. J., vol. 47, pp. 2239-2258, 1968. L. A. Shepp, “First passage time for a particular Gaussian process,” Ann. Math. Statist., vol. 42, pp. 946-951, 1971. N. A. Strakhov and L. Kurz, “An upper bound on the zero- crossing distribution,” Bell Syst. Tech. J., vol. 47, pp. 529-548,

I481 E. Wong, “The distribution of intervals between zeros for a stationary Gaussian process,” SIAM J. Appl. Math., vol. 18, pp. 67-73, 1970.

through nonl inear devices,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 448456, July 1969. N. M. Blachman, “The signal x signal, noise x noise, and signal x noise output of a nonlinearity,” IEEE Trans. Inform. Theory, vol. IT-14, pp. 21-27, Jan. 1968. T. L. Fine, “The response of a particular nonl inear system with feedback to each of two random processes,” IEEE Trans. Inform. Theory, vol. IT-14, pp. 255-264, Mar. 1968. P. A. Franaszek, “On linear systems which preserve wide sense stationarity,” SIAM J. Appl. Math., vol. 15, pp. 1480-1484, 1967. P. C. Jain, “Limiting of signals in random noise,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 332-340, May 1972. R. E. Langseth and R. F. Lambert, “Inf luence of bandwidth on some nonl inear transformations of a Gaussian random process,” IEEE Trans. Inform, Theory, vol. IT-14, pp. 88-93, Jan. 1968. N. Morinaga, H. Osawa, T. Namekawa, and K. Aoyagi, “Anal- ysis of the output of FM correlation systems,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 7-16, Jan. 1969. S. Neal, “Spectral density of a nonl inear function of a Gaussian process,” Bell Syst. Tech. J., vol. 50, pp. 1025-1047, 1971. D. S. Newman, “On the probabil ity distribution of a filtered random telegraph signal,” Ann. Math. Statist., vol. 39, pp. 890- 896, 1968. H. Osawa, N. Morinaga, and T. Namekawa, “Output signal-to- noise ratios of FM correlation systems,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 32-36, Jan. 1971. R. F. Pawula and A. Y. Tsai, “Theoretical and experimental results for the distribution of a certain nonl inear functional of the Ornstein-Uhlenbeck process,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 532-535, Sept. 1969. D. A. Pitassi, “Second-order propert ies of products of cl ipped Gaussian processes,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 535-540, Sept. 1969.

III. Representat ions [49] B. H. Bharucha and T. T. Kadota, “On the representat ion of

cont inuous parameter processes by a sequence of random variables,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 139-141, Mar. 1970.

[50] L. L. Campbell , “Series expansions for random processes,” in Probability and Information Theory (Lecture Notes in Math- ematics) vol. 89. Berlin: Springer, 1969, pp. 77-95.

[51] E. Masry, B. Liu, and K. Steiglitz, “Series expansion of wide- sense stationary random processes,” IEEE Trans. Inform. Theory, vol. IT-14, pp. 792-796, Nov. 1968.

1521 J. Pearl, “Basis-restricted transformations and performance measures for spectral representat ions,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-17, pp. 751-752, Nov. 1971.

1531 V. K. Algazi and D. J. Sakrison, “On the optimality of the [24] R. F. Baum, “The correlation function of Gaussian noise passed


Karhunen-Loeve expansion,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-15, pp. 319-321, Mar. 1969.

[54] P. A. Pierre, “Asymptotic eigenfunctions of covariance kernels with rational spectral densities,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-16, pp. 346-347, May 1970.

[55] -, “Characteristics of Gaussian random processes by representations in terms of independent random variables,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 648-658, Nov. 1969. W. D. Rav and R. M. Driver. “Further decomnosit ion of the [561

I571

P81

[591

[601

[611

[Ql

WI

1641

[651

1661

Karhunen-Lotve series representation of a stationary random process,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 663-668, Nov. 1970. F. J. Beutler, “Alias-free randomly timed sampling of stochastic processes, ” IEEE Trans. Inform. Theory, vol. IT-16, pp. 147-152, Mar. 1970. J. L. Brown, Jr., “Bounds for truncation error in sampling expansions of band-limited signals,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 440444, July 1969. S. Cambanis and B. Liu, “On harmonizable stochastic process,” Inform. Contr., vol. 17, pp. 183-202, 1970. H. Ogura, “Spectral representation of a periodic nonstationary random process,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 143-149,-Mar. 1971. B. D. 0. Anderson, J. B. Moore, and S. G. Loo, “Spectral factorization of time-carrvina covariance functions.” IEEE Trans. Inform. Theory, vol. IT-15 pp. 550-557, Sept. 1969. L. H. Brandenburg and H. E. Meadows, “Shaping filter representation of nonstationary colored noise,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 26-31, Jan. 1971. R. Geesey and T. Kailath, Proc. 19th Polytech. Inst. Brooklyn Symp: Computer Processing in Communications. Brooklyn, N.Y.: Polytechnic Press, 1969.

[671

T. Kailath, “Fredholm resolvents, Wiener-Hopf equations, and Riccati differential equations,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 665-672, Nov. 1972. M. G. Strintzis, “A solution to the matrix factorization prob- F9yi’ IEEE Trans. Znform. Theory, vol. IT-18, pp. 225-232, Mar.

T. Kailath, R. T. Geesey, and H. L. Weinert, “Some relations among RKHS norms, Fredholm equations, and innovations representations,” IEEE Trans. Znform. Theory, vol. IT-18, pp. 341-348, May 1972. T. Kailath, “RKHS approach to detection and estimation problems-part I: deterministic signals in Gaussian noise,” IEEE Trans. Znform. Theory, vol. IT-17, pp. 530-549, Sept. 1971.

IV. Likelihood Ratios, Martingales, and Stochastic Calculus [68] J. M. C. Clark. “The renresentation of functionals of Brownian

t691

[701

[711

1721

I731

[741

[751

[761

motion by stochastic integrals,” Ann. Math. Statist., vol. 41, pp. 1282-1295, 1970. C. Doleans, “Variation quadratique des martingales continues a droite,” Ann. Math. Statist., vol. 40, pp. 284-289, 1969. C. Doleans-Dade, “Quelques application de la formule de changement de variables pour les semimartingales,” Z. Wahr- scheinlichkeits. uerw. Geb., vol. 16, pp. 181-194, 1970. C. Doleans-Dade and P. A. Meyer, “Integrales stochastiques par rapport aux martingales locales,” in Seminaire de Prob- abilities IV (Universite de Strasbourg, Lecture Notes in Math- ematics), vol. 124. Berlin: Springer, 1970, pp. 77-107. T. E. Duncan. “Evaluation of likelihood functions.” Inform. Contr., vol. 13,‘~~. 62-74, 1968.

_

__ “On the absolute continuity of measures,” Ann. Math. St&t vol. 41, pp. 30-3’8, 1970. - “‘Likelihood functions for stochastic signals in white noise’,” Inform. Corm., vol. 16, pp. 303-310, 1970. B. Etsenberg, “Translating Gaussian processes,” Ann. Math. Statist., vol. 41, pp. 888-893, 1971. P. A. Frost and T. Kailath, “An innovations approach to least- squares estimation-part III: nonlinear estimation in white Gaussian noise,” IEEE Trans. Automat. Contr., vol. AC-16, pp. 217-226, June 1971.

[771

[781

t791

P301

WI

M. Fujisaki, G. Kallianpur, and H. Kunita, “Stochastic differential equations for the nonlinear filtering problem,” Osaka J. Math., vol. 9, pp. 19-40, 1972. R. 0. Harger, “On the characterization and likelihood functional of lognormal random processes,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-16, pp. 63%632, Sept. 1970. C. P. Hatsell and L. W. Nolte, “Some geometric properties of the likelihood ratio,” IEEE Trans. Inform. Theory (Corresp.), vol. IT-17, pp. 616618, Sept. 1971. M. Hitsuda, “Representation of Gaussian processes equivalent to a Wiener process,” Osaka J. Math., vol. 5, pp. 299-312, 1968. T. T. Kadota, “Nonsingular detection and likelihood ratio for random signals in white Gaussian noise,” IEEE Trans. Inform.

Wahrscheinlichkeits. uerw. Geb., vol. 16, pp. 250-260, 1970. [83] T. Kailath, “An innovations approach to least-squares estima-

tion-part I: linear filtering in additive white noise,” IEEE Trans. Automat. Contr., vol. AC-13, pp. 646655, Dec. 1968.

[84] T. Kailath and P. Frost, “An innovations approach to least- squares estimation-part II: linear smoothing in additive white noise,” IEEE Trans. Automat. Contr., vol. AC-13, pp. 655-660,

B51

WI

B71

w31

F391

[901

Dec. 1968. T. Kailath, “The innovations approach to detection and estimation theory,” Proc. IEEE, vol. 58, pp. 680-695, May 1970. - “Some extensions of the innovations theorem,” Bell Syst. Tech: J., vol. 50, pp. 1487-1494, 1971. -, :‘A general likelihood-ratto formula for random signals in Gausstan noise,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 350-361, May 1969. - “Likelihood ratios for Gaussian processes,” IEEE Trans. Infoim. Theory, vol. IT-16, pp. 276-288, May 1970. - “A further note on a general likelihood formula for randbm signals in Gaussian noise,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 393-396, July 1970. -, “The structure of Radon-Nikodym derivatives with respect to Wiener and related measures.” Ann. Math. Statist., vol. 42, pp. 10541067, 1971.

[91] T. Kailath and M. Zakai, “Absolute continuity and Radon- Nikodym derivatives for certain measures relative to Wiener measure,” Ann. Math. Statist., vol. 42, pp. 130-140, 1971.

[92] E. J. McShane, “Toward a stochastic calculus, II,” Proc. Nut. Acud. Sci., vol. 63, pp. 10841087, 1969.

[93] P. A. Meyer, “Square integrable martingales, a survey,” in Martingales (Lecture Notes in Mathematics), vol. 190. Berlin: Springer, 1971, pp. 32-37.

[94] -, “Non-square integrable martingales, etc.,” in Martingales (Lecture Notes in Mathematics), vol. 190. Berlin: Springer, 1971, pp. 3843.

[95] P. A. Pierre, “Singular non-Gaussian measures in detection and estimation theory,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 266-272, Mar. 1969.

[96] E. Wang, “Representation of martingales, quadratic variation and apphcations,” SIAM J. Contr., vol. 9, pp. 621-633, 1971.

V. Point Processes [971

[981

[991

[lOOI

[loll

t1021

HO31

DO41

PO51

DO61

DO71

[lo81

I. Bar-David, “Communication under the Poisson regime,” IEEE Trans. Inform. Theory, vol. IT-15, pp. 31-37, Jan. 1969. I. Bar-David and A. Nemirovsky, “Level crossings of nondif- ferentiable shot processes,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 27-34, Jan. 1972. F. J. Beutler and 0. A. Z. Leneman, “The spectral analysis of impulse processes,” Inform. Contr., vol. 12, pp. 236-258, 1968. -. “On the statistics of random nulse urocesses.” Inform. Co&., vol. 18,,pp. 326341, 1971. - _ - M. Brown, “Drscrimination of Poisson processes,” Ann. Math. Statist., vol. 42, pp. 773-776, 1971. S. Karp and R. M. Gagliardi, “On the representation of a continuous stochastic intensity by Poisson shot noise,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 142-147, Mar. 1970. S. Karp and J. R. Clark, “Photon counting: a problem in classical noise theory,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 672-680, Nov. 1970. 0. Macchi, “Stochastic point processes and multicoincidences,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 2-7, Jan. 1971. H. Ogura, “Orthogonal functionals of the Poisson process,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 473481, July 1972. I. Rubin, “Regular point processes and their detection,” ZEEE Trans. Inform. Theory, vol. IT-18, pp. 547-557, Sept. 1972. D. L. Snyder, “Filtering and detection for doubly stochastic Poisson nrocesses.” IEEE Trans. Inform. Theory, vol. IT-18, pp. 91-102, Jan. 1972: - “Smoothing for doubly stochastic Poisson processes,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 558-562, Sept. 1972.

VI. Random Fields DO91

[1101

I1111 Ann. Math. Statist., vol. 40, pp. 197-202, 1969. D. Middleton, “A statistical theory of reverberation and similar first-order scattered fields,” IEEE Trans. Inform. Theory: “Part I, Waveforms and the general process,” vol. IT-13, pp. 372-392, July 1967; “Part II, Moments, spectra, and special distribution,” vol. IT-13, pp. 393-414, July 1967; “Part III, Waveforms and fields,” vol. IT-18, pp. 35-67, Jan. 1972; “Part IV, Statistical models,” vol. IT-18, pp. 68-90, Jan. 1972. W. J. Park, “A multi-parameter Gaussian process,” Ann. Math. Statist., vol. 41, pp. 1582-1595, 1971. R. F. Pawula, “A proof of Corrsin’s theorem concerning stationarv random surfaces,” IEEE Trans. Inform. Theory

Theory, vol. IT-16, pp. 291-298, May 1970. [82] T. T. Kadota and L. A. Shepp, “Conditions for absolute con-

tinuity between a certain pair of probability measures,” Z.

ill21

ill31

R. Cairoli, “Sur une equation differentielle stochastique,” C. R. Acnd. Sci. (Paris), vol. 274, pp. 1739-1742, 1972. T. R. Chow. “Eauivalence of Gaussian stationary processes,”

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-19, NO. 3, MAY 1973 275

(Corresp.), vol. IT-14, pp. 77(X772, Sept. 1968. [114] E. Wong, “Two-dimensional random fields and representat ion [1321

of images,” SIAM J. Appl. Math., vol. 16, pp. 756770, 1968. [115] -, “Homogeneous Gauss-Markov random fields,” Ann. Math. [133]

Statist., vol. 40, pp. 1625-1634, 1969. [116] -, “The dynamics of random fields,” in Proc. 3rd U.S.-Japan [134]

Seminar Stochastic Methods in Dynamical Problems, 1971. [117] J. W. Woods, “Two-dimensional discrete Markovian fields,” 11351

IEE Trans. Inform. Theory, vol. IT-18, pp. 232-240, Mar. 1972.

VII. Books and Dissertations [I361 [118] P. M. Bremaud, “A mart ingale approach to point. processes,”

y$h.F. dissertation, Dep. Elec. Eng., Univ. Cahforma, Berkeley, t1371

[119] W. B. Davenport , Jr., Probability and Random Processes. New York: McGraw-Hill, 1970.

[120] D. L. Duttweiler, “Reproducing kernel Hilbert space techniques 11381 for detect ion and estimation problems,” Ph.D. dissertation, Dep. Elec. Eng., Stanford Univ., Stanford, Calif., 1970. [ I391

[121] P. A. Frost, “Estimation in continuous-t ime nonl inear systems,” Ph.D. dissertation, Dep. Elec. Eng., Stanford Univ., Stanford, Calif., 1968. [1401

[122] R. Geesey, “Proper canonical representat ions of second order processes,” Ph.D. dissertation, Dep. Elec. Eng., Stanford Univ., 11411 Stanford, Calif., 1969.

[123] I. I. Gikhman and A. V. Skorokhod, Introduction to the Theory of Random Processes. Philadelphia, Pa.: Saunders, 1969. n421

[124] H. P. McKean, Jr., Stochastic Integrals. New York: Academic Press, 1969.

Theory Prob. Appl. (USSR!, vol. 5? pp. 285-301, 1960. K. It& “On stochastic dlfferentral equat ions,” Mem. Amer. Math. Sot., vol. 4, 1951. -, “On a formula concerning stochastic differentials,” Nugoyu Math. J., vol. 3, pp. 55-65, 1951. -, “Multiple W iener integral,” J. Math. Sot. Jup., vol. 3, pp. 157-169, 1951. K. Karhunen, “Uber l ineare methoden in der wahrscheinl ich- kei tsrechnung,” Ann. Acad. Sci. Fenn., Ser. Al: Math. Phys., vol. 37, p. 79, 1947. H. Kunita and S. Watanabe, “On square integrable martingales,” Nagoya Math. J., vol. 30, pp. 209-245, 1967. P. L&v. “A snecial nroblem of Brownian motion. and a general theory-of Gaussian-random functions,” in Pro;. 3rd Bzrkeley Symp. Mathematical Statistics and Probability, vol. 2. Berkeley, Calif.: Univ. California Press, 1956, pp. 133-175. P. A. Meyer, Probability and Potentials. Waltham, Mass.: Blaisdell, 1966. S. Orey, “F-processes,” in Proc. 5th Berkeley Symp. Math- ematical Statistics and Probability, vol. II-l. Berkeley, Calif. : Univ. California Press, 1967, pp. 301-313. E. Parzen, Time Series Analysis Papers. San Francisco, Calif.: Holden-Day, 1967. R. Price, “A useful theorem for nonl inear devices having Gaus- sian inputs,” IRE Trans. Inform. Theory, vol. IT-4, pp. 69-72, June 1958. D. Slepian, “Some comments on the detect ion of Gaussian signals in Gaussian noise,” IRE Trans. Inform. Theory, vol. IT-4,

[125] J. B. Thomas, An Introduction to Applied Probability and Random [143] Processes. New York: W iley, 1971.

[125a] H. L. Weinert, “A reproducing kernel Hilbert space approach to [144] spline problems with applications in estimation and control,” Ph.D. dissertation, Dep. Elec. Eng., Stanford Univ., Stanford, u451 Calif., 1972.

pp. 65-68, June 1958. - “First passage time for a particular Gaussian process,” Ann’Math. Statist., vol. 32, pp. 610-612, 1961. -, ‘Contributions to signal and noise theory,” IEEE Trans.

[126] E. Wong, Stochastic Processes in Information and Dynamical Systems. New York: McGraw-Hill, 1971.

VIII. General References

Inform. Theory, vol. IT-9, pp. 229-233, Oct. 1963. 1 Y. Sosulin and R. L. Stratonovich, “Opt imum detect ion of a

diffusion process in white noise,” Radio Eng. Electron. Phys. (USSR), vol. 10, pp. 704714, 1965.

[146] A. M. Vershik, “Some characteristic propert ies of Gaussian processes,” Theory Prob. Appl. (USSR), vol. 9, pp. 353-356, 1964.

[127] J. F. Barrett and D. G. Lampard, “An expansion.for. some second-order pt;obability distributions and Its apphcatron to g2re T&&ems, IRE Truns. Inform. Theory, vol. IT-l, pp. 10-15,

[128] G. Baxter; “A strong limit theorem for Gaussian processes,” Proc. Amer. Math. Sot., vol. 7, pp. 522-528, 1956.

[129] J. L. Doob, Stochastic Processes. New York: W iley, 1953. [130] D. L. Fisk, “Quasi-mart ingales,” Trans. Amer. Math. Sot., vol.

120, pp. 369-389, 1965. [131] I. V. Girsanov, “On transforming a certain class of stochastic

processes by absolutely cont inuous substitution of measurse,”

__ _ [147] E. Wong, “The construct ion of a class of stationary Markoff

processes.” Proc. Amer. Math. Sot. Symp. A&. Math., vol. 16, pp. 264-273, 1963.

. _ __ [148] A. M. Yaglom, “Second-order homogeneous random fields,”

in Proc. 4th Berkeley Symp Mathematical Statistics and Prob- ability, vol. 2. Berkeley, Calif.: Univ. California Press, 1961, pp. 593-620.

[149] J. E. Mazo, P. F. Pawula, and S. 0. Rice, to be publ ished. [150] L. D. Davisson and P. Papantoni-Kazakos, to be publ ished. [151] J. M. C. Clark, “A measurabil i ty property of the solutions of

stochastic differential equat ions,” to be publ ished.

On Functionals Satisfying a Data-Processing Theorem

JACOB ZIV AND MOSHE ZAKAI

Abstraci-It is shown that the rate-distortion hound (R(d) I C) remains true when -log x in the definition of mutual information is re- p laced by an arbitrary concave (u) nonincreasing function satisfying some technical condit ions. Examples are given showing that for certain choices of the concave functions, the bounds obtained are better than the classical rate-distortion bounds.

Manuscript received April 24, 1972; revised November 4, 1972. The authors are with the Faculty of Electrical Engineering, Tech-

nion-Israel Institute of Technology, Haifa, Israel.

I. INTRODUCTION

E start with a rough and very short outline of the derivation of the rate-distortion inequality of in-

formation theory Cl]-[3]. In order to simplify the representation, we will base our exposition, in this section only, on the case of finite-dimensional random vectors. Let (X, Y) be a pair of random n-vectors with a joint probability density p(x,y). The mutual information between the pair

recent progress in stochastic processes- a surveywong/wong_pubs/wong30.pdf · recent progress in...

Documents