ee565 advanced image processing copyright xin li 2008 1 wavelet-based texture synthesis...

EE565 Advanced Image Processing Copyright Xin Li 2008

1

Wavelet-based Texture Synthesis Pyramid-based (using steerable pyramids)

• Facilitate the statistical modeling

Histogram matching• Enforce the first-order statistical constraint

Texture matching• Alternate histogram matching in spatial and wavelet

domain

Boundary handling: use periodic extension Color consistency: use color transformation

Basic idea: two visually similar textures will also have similar statistics


2

Histogram MatchingGeneralization of histogram equalization (the target is the histogramof a given image instead of uniform distribution)


3

Histogram Equalization

x

t

thLy0

)(Uniform

Quantization

L

t

th0

1)(Note:

L

1

x

t

ths0

)(

x

L

y

0

cumulative probability function


4

MATLAB Implementationfunction y=hist_eq(x)

[M,N]=size(x);for i=1:256 h(i)=sum(sum(x= =i-1));End

y=x;s=sum(h);for i=1:256 I=find(x= =i-1); y(I)=sum(h(1:i))/s*255;end

Calculate the histogramof the input image

Perform histogramequalization


5

Histogram Equalization Example


6

Histogram Specification

ST

S-1*T

histogram1 histogram2

?


7

Texture MatchingObjective: the histogram of both subbands and synthesized image matches the given template

Basic hypothesis: if two texture images visually look similar, then theyhave similar histograms in both spatial and wavelet domain


8

Image Examples


9

I.I.D. Assumption Challenged

If wavelet coefficients of each subband are indeed i.i.d., then random permutation of pixels should produce another image of the same class (natural images)

The fundamental question here: does WT completely decorrelate image signals?


10

Image Example

High-band coefficientspermutation

You can run the MATLAB demo to check this experiment


11

Another Experiment

-4 -3 -2 -1 0 1 2 3 4-4

-3

-2

-1

0

1

2

3

4

5

Joint pdf of two uncorrelated random variables X and Y

X

Y


12

Joint PDF of Wavelet Coefficients

Neighborhood I(Q): {Left,Up,cousin and aunt}

X=

Y=

Joint pdf of two correlated random variables X and Y


13

Texture Synthesis via Parametric Models in the Wavelet Space

Instead of matching histogram (nonparametric models), we can buildparametric models for wavelet coefficients and enforce the synthesizedimage to inherit the parameters of given image

Model parameters: 710 parameters were used in Portilla and Simoncelli’s experiment (4 orientations, 4 scales, 77 neighborhood)

Basic idea: two visually similar textures will also have similar statistics


14

Statistical Constraints

Four types of constraints• Marginal Statistics

• Raw coefficient correlation

• Coefficient magnitude statistics

• Cross-scale phase statistics

Alternating Projections onto the four constraint sets• Projection-onto-convex-set (POCS)


15

Convex Set

A set Ω is said to be convex if for any two point yx ,We have 10,)1( ayaax

Convex set examples

Non-convex set examples


16

Projection Operator

f

g

Projection onto convex set C

C ||}||min||||{ fxfxCxPfgCx

In simple words, the projection of f onto a convex set C is theelement in C that is closest to f in terms of Euclidean distance


17

Alternating Projection

X0

X1

X2

X∞

Projection-Onto-Convex-Set (POCS) Theorem: If C1,…,Ck areconvex sets, then alternating projection P1,…,Pk will convergeto the intersection of C1,…,Ck if it is not empty

Alternating projection does not always converge in the caseof non-convex set. Can you think of any counter-example?

C1

C2


18

Convex Constraint Sets

● Non-negative set

}0|{ ff

● Bounded-value set

}2550|{ ff

● Bounded-variance set

}||||{ 2 Tgff

A given signal

}|{ BfAf or


19

Non-convex Constraint Set

Histogram matching used in Heeger&Bergen’1995

Bounded Skewness and Kurtosis

skewness kurtosis

The derivation of projection operators onto constraint sets are tediousare referred to the paper and MATLAB codes by Portilla&Simoncelli.


20

Image Examples

original

synthesized


21

Image Examples (Con’d)

original

synthesized


22

When Does It Fail?

original

synthesized


23

Summary on Wavelet-based Texture Synthesis

Textures represent an important class of structures in natural images – unlike edges characterizing object boundaries, textures often associate with the homogeneous property of object surfaces

Wavelet-domain parametric models provide a parsimonious representation of high-order statistical dependency within textural images

EE591b Advanced Image Processing Copyright Xin Li 2006

24

Image Denoising

Theory of linear estimation Spatial domain denoising techniques

• Conventional Wiener filtering

• Spatially adaptive Wiener filtering Wavelet domain denoising

• Wavelet thresholding: hard vs. soft

• Wavelet-domain adaptive Wiener filtering Latest advances (we will cover it after patch-

based modeling)• Patch-based image denoising

• Learning-based image denoising


25

Denoising Problem

Noisy measurements

signal N(0,σw2)

MMSE estimator

aYX ˆ

Wiener’s idea To simplify estimation by computing the bestestimator that is a linear scaling of Y

Difficulty: we need to know conditional pdf

WXY

]|[ˆ YXEX


26

Orthogonality Principle

A linear estimator X of a random variable X aYX ˆ

0})ˆ{( YXXE

^

Minimizes E{(X-X)2} if and only if ^

Geometric Interpretation

X

Y

X-X̂

X̂


27

Linear MMSE Estimation

YXwx

x22

2

ˆ

),0(~ 2xNX For Gaussian signal

The optimal LMMSE estimation is given by

22

22

wx

xwMMSE

And it achieves

Note: it can be shown such linear estimator is indeedE[X|Y] for Gaussian signal


28

What if Signal Variance is Unknown?

222ˆ wx y

Maximum-likelihood estimation of 2x is given by

Since variance is nonnegative, we modify it

],0max[ˆ 222wx y

When multiple observations yi’s are available, we have

]1

,0max[ˆ 2

1

22w

N

iix y

N


29

From Scalar to Vector Case

Suppose X is a Gaussian process whose covariance matrix is adiagonalized matrix RX=diag{ηm}(m=0,…,N-1), the linear MMSEestimator is given by

1

022

2

][][ˆN

m wm

m mYnX

(A)

1

022

222}||ˆ{||

N

m wm

wmXXE

and the minimal MSE is given by


30

Decorrelating

Q: What if X={X[0],…,X[N-1]} is correlated (i.e., Rx is not diagonalized)?

A: We need to transform X into a set of uncorrelated basis and then apply the above result.

The celebrated Karhunen-Loeve Transform does this job!

XT R*

Diagonal matrix

XX T*'

Karhunen-Loeve Transform


31

Transform-Domain Denoising

ForwardTransform

InverseTransform

Denoisingoperation

e.g.,KLTDCTWT

e.g.,Linear Wiener filteringNonlinear Thresholding

Noisysignal

denoisedsignal

The performance of such transform-domain denoising is determinedby how well the assumed probability model in the transform domainmatches the true statistics of source signal (optimality can only beestablished for the Gaussian source so far).


32

Image Denoising





• Wavelet-domain adaptive Wiener filtering Latest advances

• Patch-based image denoising



33

Conventional Wiener Filtering

Basic assumption: image source is modeled by a stationary Gaussian process

Signal variance is estimated from the noisy observation data

Can be interpreted as a linear frequency weighting


34

Linear Frequency Weighting

),(),(

),(),(

),,(),(),(ˆ

2121

2121

212121

wwSwwS

wwSwwH

wwYwwHwwX

WX

X

22

2

,ˆwx

xaaYX

FT

Power spectrum |X|2


35

Image Example

Noisy, =50 (MSE=2500) denoised (MSE=1130)


36

Image Example (Con’d)



37

Conclusions from the Experiments

Why did it Fail? • Nonstationary

• NonGaussian

• Poor modeling

How to improve?• Achieve spatial adaptation

• Use linear transform

• Putting them together


38

Spatially Adaptive Wiener Filtering

Basic assumption: image source is modeled by a nonstationary Gaussian process

Signal variance is locally estimated from the windowed noisy observation data

]1

,0max[ˆ 2

1

22w

N

iix y

N

T

T

N=T2

Recall


39

Image Example

Noisy, =10 (MSE=100) denoised (T=3,MSE=56)


40

Image Example (Con’d)



41

Image Denoising





• Wavelet-domain adaptive Wiener filtering Latest advances

• Patch-based image denoising



42

A Toy Example

18

2 2 N

MSE

01 N-1

σ2=4/N

Wiener Filter Estimator:

22sin4

)( mX N

m

NmP

12

1 1

0

21

022

22

N

mm

N

m m

mMSE

otherwise

nYifnYnX0

2

1|][|][][

~

Nonlinear thresholding Estimator:

]mod)[(][ NPnfnX f(n)

An integer uniformlydistributed on [0,N-1]

Conclusion: for sparse signals, linear filtering is not as effective asNonlinear thresholding


43

Why Wavelet Denoising?

We need to distinguish spatially-localized events (edges) from noise components

More about noise components

Wavelet is such a basis because exceptional event generates identifiable exceptional coefficients due to its good localization property in both spatial and frequency domain

As long as it does not generate exceptions

Additive White Gaussian Noise is just one of them


44

Wavelet Thresholding

DWT IWTThresholdingY X

~

otherwise

TnYifnYnX

0

|][|][][

~Hard thresholding

Soft thresholding

TnY

TnYTnY

TnYTnY

nX

|][|0

][][

][][

][~

Noisysignal

denoisedsignal


45

Choice of Threshold

NT elog2

Donoho and Johnstone’1994

Gives denoising performance close to the “ideal weighting”

][][

][][

~22

2

nYn

nnX

Reference: S. Mallat, “A Wavelet Tour of Signal Processing”, Section 10.2 (pp. 435-453)


46

Soft vs. Hard thresholding

|][||][~

|1][

][],[][

~22

2

nYnXn

nanaYnX

● It can be also viewed as a computationally efficient approximationof ideal weighting

|][||][~

|][,][][~

nYnXTnYTnYnX soft

ideal

● Soft-thresholding has the same upper bound as hard-thresholding asymptotically and larger error than hard-thresholding at the same threshold value, but perceptually it works better.

● Shrinking the amplitude by T guarantees with a high probability that.

|][||][~

| nXnX


47

Denoising Example

noisy image(σ2=100)

Wiener-filtering (ISNR=2.48dB)

Wavelet-thresholding (ISNR=2.98dB)

2

2

10||

~||

||||log10

XX

YXISNR

X: original, Y: noisy, X: denoised~

Improved SNR


48

Duality with Image Coding*

DWT IWTThresholding

DWT IWTQ Q-1Channel

Image denoising system

Image coding system


49

Basis Selection Problem

G0

G1

)(ˆ nxx(n)

H0

H1

y0(n)

y1(n)

x(n)

H0

H1

22 G0

22 G1

)(ˆ nx

s(n)

d(n)

complete expansion (with decimation)

overcomplete expansion (without decimation)

TceTce

-1

Toe Toe-1


50

What do We Buy from Redundancy?

0 1 N-1… …

x(n)

H1

T

-T


51

Translation Invariance (TI) Denoising

Toe Toe-1Thresholding

Tce Tce-1Thresholding

Tce Tce-1Thresholding

z

+

x(n) )(ˆ nx

x(n) )(ˆ1 nx

)(ˆ2 nx

2

)(ˆ)(ˆ)(ˆ 21 nxnx

nx

Implementation based on overcomplete expansion

Implementation based on complete expansion

z-1


52

2D Extension

Noisy image

Tce Tce-1ThresholdingWD =

shift(mK,nK) WD shift(-mK,-nK)

shift(m1,n1) WD shift(-m1,-n1)

Avg

denoised image

(mk,nk): a pair of integers, k=1-K (K: redundancy ratio)


53

Example

Wavelet-thresholding (ISNR=2.98dB)

Translation-Invariant thresholding (ISNR=3.51dB)


54

Go Beyond Thresholding

Challenges with wavelet thresholding• Determination of a global optimal threshold

• Spatially adjusting threshold based on local statistics

How to go beyond thresholding?• We need an accurate modeling of wavelet

coefficients – nonlinear thresholding is a computationally efficient yet suboptimal solution


55

Spatially Adaptive Wiener Filtering in Wavelet Domain

Wavelet high-band coefficients are modeled by a Gaussian random variable with zero mean and spatially varying variance

Apply Wiener filtering to wavelet coefficients, i.e.,

][][

][][

~22

2

nYn

nnX

estimated in the same wayas spatial-domain (Slide 15)


56

Practical Implementation

]1

,0max[ˆ 2

1

22w

N

iix y

N

T

T

N=T2

Recall

Conceptually very similar to its counterpart in the spatial domain

YXwx

x22

2

ˆ

ˆˆ

In demo3.zip, you can find a C-coded example (de_noise.c)

(ML estimation of signal variance)


57

Example

Translation-Invariant thresholding (ISNR=3.51dB)

Spatially-adaptive wiener filtering (ISNR=4.53dB)


58

Further Improvements

Gaussian scalar mixture (GSM) based denoising (Portilla et al.’ 2003)• Instead of estimating the variance, it explicitly

addresses the issue of uncertainty with variance estimation

Hidden Markov Model (HMM) based denoising (Romberg et al.’ 2001)• Build a HMM for wavelet high-band

coefficients (refer to the posted paper)


59

Gaussian Scalar Mixture (GSM)

Model definition: u~N(0,1)

Noisy observation model

Gaussian pdf

scale (variance) parameter


60

Basic IdeaIn spatially adaptive Wiener filtering, we estimate the variancefrom the data of a local window. The uncertainty with such varianceestimation is ignored. In GSM model, such uncertainty is addressedthrough the scalar z (it determines the variance of GSM). Instead of using a single z (estimated variance), we build a probabilitymodel over z, i.e., E{x|y}=Ez{E{x|y,z}}


61

Posterior Distribution

where

Due to

is so-called Jeffery’s prior

Question: What is E{xc|y,z}?

Bayesian formula

(proof left as exercise)


62

GSM Denoising Algorithm

http://decsai.ugr.es/~javier/denoise/index.htmlMATLAB codes available at:


63

Image Examples



64

Image Examples (Con’d)

Noisy, =10 (MSE=100) denoised (MSE=31.7)


65

Early Attempts

Each band is modeled by a Guassian random variable with zero mean and unknown variance (e.g., WSQ)

Only modest gain over JPEG (DCT-based) is achieved

Question: is this an accurate model?

and how can we test it?


66

Proof by Contradiction (I)

Suppose each coefficient X in a high band does observeGaussian distribution, i.e., X~N(0,σ2), then flip the sign ofX (i.e., replace X with –X) should not matter and generatesanother element in Ω (i.e., a different but meaningful image)

Assumption: our modeling target Ω is the collection of natural images

Let’s test it!


67

Proof by Contradiction (II)

DWT

sign flip

IWT


68

What is wrong with that?

Think of two coefficients: one in smooth region and the other around edge, do they observe the same probabilistic distribution?

Think of all coefficients around the same edge, do they observe the same probabilistic distribution?

Ignorance of topology and geometry


69

The Importance of Modeling Singularity Location Uncertainty Singularities carry critical visual

information: edges, lines, corners … The location of singularities is important

• Recall locality of wavelets in spatial-frequency domain

Singularities in spatial domain → significant coefficients in wavelet domain


70

Where-and-What Coding

Communication context

Where The location of significant coefficients

What The sign and magnitude of significant

coefficients

Alice Bob

communicationchannelpicture


71

1993-2003

Embedded Zerotree Wavelet (EZW)’1993 Set Partition In Hierarchical Tree

(SPIHT)’1995 Space-Frequency Quantization (SFQ)’ 1996 Estimation Quantization (EQ)’1997 Embedded Block Coding with Optimal

Truncation (EBCOT)’2000 Least-Square Estimation Quantization

(LSEQ)’2003


72

Zerotree Data Structure


73

Zerotree Data Structure

Parent-and-Children Ancestor-and-Descendent


74

Zerotree Terminology

Zerotree root (ZRT): it and its all descendants are insignificant

Isolated zero (IZ): it is insignificant but its descendant is not

Positive significant (POS): it is significant and have a positive sign

Negative significant (NEG): it is significant and have a negative sign


75

Coding Process


76

Toy Example


77

Toy Example

Note: T=32

LH1 contains POS

LH1 contains POS


78

Where-and-What Interpretation

Zerotree data structure effectively resolves the location uncertainty (where) of insignificant coefficients

The dominant and subordinate passes defined in EZW can be viewed as “where” and “what” coding respectively

Dyadic choice of T values (i.e., T=128,64, 32,16,…) renders embedded coding


79

A Simpler Two-Stage Coding

Position coding stage (where)• Generate a binary map indicating the location

of significant coefficients (|X|>T)

• Use context-based adaptive binary arithmetic coding (e.g., JBIG) to code the binary map

Intensity coding stage (what)• Code the sign and magnitude of significant

coefficients


80

A Different Interpretation

Two-class modeling of high-band coefficients• Significant class: |X|>T

• Insignificant class: |X|<T

Why does classification help?• Nonstationarity of image source

• A probabilistic modeling perspective


81

Classification-based Modeling

),0(~ 200 NX

Insignificant class

),0(~ 211 NX

Significant class

Mixture

20

21

2201 )1(),,0(~)1( aaNXaaXX


82

Classification Gain

RRD 22 2)(

Without classification

With classification

RaaRD 221

)1(20 2)('

Classification gain

0)1(

log10)('

)(log10

21

)1(20

20

21

1010

dBaa

dBRD

RDG

aa


83

Example

100,1 21

20


84

Summary of Wavelet Coding

SPIHT: a simpler yet more efficient implementation of EZW coder

SFQ: Rate-Distortion optimized zerotree coder

EQ: Rate-Distortion optimized backward classification strategy

EBCOT (adopted by JPEG2000): a versatile embedded coder

ee565 advanced image processing copyright xin li 2008 1 wavelet-based texture synthesis...

Documents