today: non-linear filters, and · 2002-10-12 · 4 mid-term exam problem set 3 given out today –...
TRANSCRIPT
![Page 1: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/1.jpg)
1
![Page 2: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/2.jpg)
2
Today: non-linear filters, and uses for the filters and
representations from last time
• Review pyramid representations• Non-linear filtering• Textures
![Page 3: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/3.jpg)
3
Reading
• Related to today’s lecture: – Chapter 9, Forsyth&Ponce..
• For next Thursday’s lecture:– Horn, Ch. 12– Bishop chapter 1 (handout from last lecture)
![Page 4: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/4.jpg)
4
Mid-term exam
Problem set 3 given out today– Open book, open web.– Work by yourself. This problem set is a mid-term
exam, and you can’t: talk about it, e-mail about it, give hints, etc, with others.
– Due Tuesday, Oct. 22 (in 12 days).
![Page 5: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/5.jpg)
5
Image representations
• Fourier basis• Image pyramids
![Page 6: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/6.jpg)
6
Image pyramids
Shows the information added in Gaussian pyramid at each spatial scale. Useful for noise reduction & coding.
Progressively blurred and subsampled versions of the image. Adds scale invariance to fixed-size algorithms.
Shows components at each scale and orientation separately. Non-aliased subbands. Good for texture and feature analysis.
Bandpassed representation, complete, but with aliasing and some non-oriented subbands.
• Gaussian
• Laplacian
• Wavelet/QMF
• Steerable pyramid
![Page 7: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/7.jpg)
7
Wavelet/QMF representation
![Page 8: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/8.jpg)
8
Linear image transformations
• In analyzing images, it’s often useful to make a change of basis.
Fourier transform, orWavelet transform, or
Steerable pyramid transform
fUFrr
=transformed image
Vectorized image
![Page 9: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/9.jpg)
9
Schematic pictures of each matrix transform
• Shown for 1-d images• The matrices for 2-d images are the same
idea, but more complicated, to account for vertical, as well as horizontal, neighbor relationships.
![Page 10: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/10.jpg)
10
Fourier transform
= *
Fourier transform
Fourier bases are global: each transform coefficient depends on all pixel locations.
pixel domain image
![Page 11: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/11.jpg)
11
Gaussian pyramid
=
Overcomplete representation. Low-pass filters, sampled appropriately for their blur.
*Gaussian pyramid
pixel image
![Page 12: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/12.jpg)
12
Laplacian pyramid
=
Overcomplete representation. Transformed pixels represent bandpassed image information.
*Laplacian
pyramidpixel image
![Page 13: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/13.jpg)
13
Wavelet (QMF) transform
Wavelet pyramid = *
Ortho-normal transform (like Fourier transform), but with localized basis functions.
pixel image
![Page 14: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/14.jpg)
14
=Multiple
orientations at one scale
Multiple orientations at the next scale
the next scale…
Steerable pyramid
*Steerablepyramid
pixel image
Over-complete representation, but non-aliased subbands.
![Page 15: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/15.jpg)
15
Matlab resources for pyramids (with tutorial)http://www.cns.nyu.edu/~eero/software.html
![Page 16: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/16.jpg)
16
Why use these representations?
• Handle real-world size variations with a constant-size vision algorithm.
• Remove noise• Analyze texture• Recognize objects• Label image features
![Page 17: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/17.jpg)
17
Image statistics (or, mathematically, how can you tell image from noise?)
![Page 18: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/18.jpg)
18
P(x)
Bayesian MAP estimator for clean bandpasscoefficient values
Let x = bandpassed image value before adding noise.Let y = noise-corrupted observation.
By Bayes theorem
P(x|y) = k P(y|x) P(x)
P(y|x)
y
P(y|x)
P(x|y)P(x|y)
![Page 19: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/19.jpg)
19
P(x)
Let x = bandpassed image value before adding noise.Let y = noise-corrupted observation.
By Bayes theorem
P(x|y) = k P(y|x) P(x)
P(y|x)
y
P(y|x)
P(x|y)P(x|y)
Bayesian MAP estimator
![Page 20: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/20.jpg)
20
P(x)
Let x = bandpassed image value before adding noise.Let y = noise-corrupted observation.
By Bayes theorem
P(x|y) = k P(y|x) P(x)
P(y|x)
y
P(y|x)
P(x|y)
P(x|y)
Bayesian MAP estimator
![Page 21: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/21.jpg)
21
Noise removal results
http://www-bcs.mit.edu/people/adelson/pub_pdfs/simoncelli_noise.pdfSimoncelli and Adelson, Noise Removal via Bayesian Wavelet Coring
![Page 22: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/22.jpg)
22
Image texture
![Page 23: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/23.jpg)
23
Texture
• Key issue: representing texture– Texture based matching
• little is known
– Texture segmentation• key issue: representing texture
– Texture synthesis• useful; also gives some insight into quality of representation
– Shape from texture• cover superficially
![Page 24: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/24.jpg)
24
The Goal of Texture Synthesis
True (infinite) texture
SYNTHESIS
generated image
input image
• Given a finite sample of some texture, the goal is to synthesize other samples from that same texture– The sample needs to be "large enough“
![Page 25: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/25.jpg)
25
The Goal of Texture Analysis
True (infinite) texture
ANALYSIS
generated image
input image
“Same” or “different”
Compare textures and decide if they’re made of the same “stuff”.
![Page 26: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/26.jpg)
26
Pre-attentive texture discrimination
![Page 27: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/27.jpg)
27
Pre-attentive texture discrimination
![Page 28: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/28.jpg)
28
Pre-attentive texture discrimination
Same or different textures?
![Page 29: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/29.jpg)
29
Pre-attentive texture discrimination
![Page 30: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/30.jpg)
30
Pre-attentive texture discrimination
![Page 31: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/31.jpg)
31
Pre-attentive texture discrimination
Same or different textures?
![Page 32: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/32.jpg)
32
Julesz• Textons: analyze the texture in terms of
statistical relationships between fundamental texture elements, called “textons”.
• It generally required a human to look at the texture in order to decide what those fundamental units were...
![Page 33: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/33.jpg)
33
Influential paper:
![Page 34: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/34.jpg)
34
Learn: use filters.
Bergen and Adelson, Nature 1988
![Page 35: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/35.jpg)
35
Malik and PeronaLearn: use lots of filters, multi-ori&scale.
Malik J, Perona P. Preattentive texture discrimination with early vision mechanisms. J OPT SOC AM A 7: (5) 923-932 MAY 1990
![Page 36: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/36.jpg)
36
Representing textures
• Textures are made up of quite stylised subelements, repeated in meaningful ways
• Representation:– find the subelements, and
represent their statistics• But what are the
subelements, and how do we find them?– recall normalized
correlation– find subelements by
applying filters, looking at the magnitude of the response
• What filters?– experience suggests spots
and oriented bars at a variety of different scales
– details probably don’t matter
• What statistics?– within reason, the more the
merrier.– At least, mean and standard
deviation– better, various conditional
histograms.
![Page 37: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/37.jpg)
37
![Page 38: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/38.jpg)
38
image
Squared responses Spatially blurred
vertical filter
horizontal filter
Threshold squared, blurred responses, then categorize texture based on those two bits
![Page 39: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/39.jpg)
39
![Page 40: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/40.jpg)
40
![Page 41: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/41.jpg)
41
![Page 42: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/42.jpg)
42
SIGGRAPH 1994
![Page 43: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/43.jpg)
43
Learn: use filter marginal statistics.
Bergen and Heeger
![Page 44: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/44.jpg)
44
Bergen and Heeger results
![Page 45: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/45.jpg)
45
Bergen and Heeger failures
![Page 46: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/46.jpg)
46
De Bonet (and Viola)SIGGRAPH 1997
![Page 47: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/47.jpg)
47
Learn: use filter conditional statistics across scale.
DeBonet
![Page 48: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/48.jpg)
48
DeBonet
![Page 49: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/49.jpg)
49
DeBonet
![Page 50: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/50.jpg)
50
Portilla and Simoncelli• Parametric representation.• About 1000 numbers to describe a texture.• Ok results; maybe as good as DeBonet.
![Page 51: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/51.jpg)
51
Portilla and Simoncelli
![Page 52: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/52.jpg)
52
Zhu, Wu, & Mumford, 1998
• Principled approach.• Synthesis quality not great, but ok.
![Page 53: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/53.jpg)
53
Zhu, Wu, & Mumford
• Cheetah Synthetic
![Page 54: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/54.jpg)
54
![Page 55: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/55.jpg)
55
Efros and Leung
![Page 56: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/56.jpg)
56
![Page 57: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/57.jpg)
57
What we learned from Efros and Leung regarding texture synthesis
• Don’t need conditional filter responses across scale
• Don’t need marginal statistics of filter responses.
• Don’t need multi-scale, multi-orientation filters.
• Don’t need filters.
![Page 58: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/58.jpg)
58
Efros & Leung ’99• The algorithm
– Very simple– Surprisingly good results– Synthesis is easier than analysis!– …but very slow
• Optimizations and Improvements– [Wei & Levoy,’00] (based on [Popat & Picard,’93]) – [Harrison,’01]– [Ashikhmin,’01]
![Page 59: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/59.jpg)
59
Efros & Leung ’99 extended
pp
• Observation: neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = blockunit of synthesis = block• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
• Not the same as multi-scale!
Synthesizing a block
![Page 60: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/60.jpg)
60
Image Quilting• Idea:
– let’s combine random block placement of Chaos Mosaic with spatial constraints of Efros & Leung
• Related Work (concurrent):– Real-time patch-based sampling [Liang et.al. ’01]– Image Analogies [Hertzmann et.al. ’01]
![Page 61: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/61.jpg)
61
Input texture
block
B1 B2
Random placement of blocks
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
![Page 62: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/62.jpg)
62
Minimal error boundaryoverlapping blocks vertical boundary
__ ==22
overlap error min. error boundary
![Page 63: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/63.jpg)
63
Our Philosophy• The “Corrupt Professor’s Algorithm”:
– Plagiarize as much of the source image as you can– Then try to cover up the evidence
• Rationale: – Texture blocks are by definition correct samples of
texture so problem only connecting them together
![Page 64: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/64.jpg)
64
Algorithm– Pick size of block and size of overlap– Synthesize blocks in raster order
– Search input texture for block that satisfies overlap constraints (above and left)
• Easy to optimize using NN search [Liang et.al., ’01]
– Paste new block into resulting texture• use dynamic programming to compute minimal error
boundary cut
![Page 65: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/65.jpg)
65
![Page 66: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/66.jpg)
66
![Page 67: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/67.jpg)
67
![Page 68: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/68.jpg)
68
![Page 69: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/69.jpg)
69
![Page 70: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/70.jpg)
70
![Page 71: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/71.jpg)
71
![Page 72: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/72.jpg)
72
Failures(Chernobyl
Harvest)
![Page 73: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/73.jpg)
73
Texture Transfer• Take the texture from one
object and “paint” it onto another object– This requires separating texture
and shape– That’s HARD, but we can cheat – Assume we can capture shape by
boundary and rough shading
•Then, just add another constraint when sampling: Then, just add another constraint when sampling: similarity to underlying image at that spotsimilarity to underlying image at that spot
![Page 74: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/74.jpg)
74
parmesan
++ ==
++ ==rice
![Page 75: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/75.jpg)
75
++ ==
![Page 76: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/76.jpg)
76
==++
![Page 77: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/77.jpg)
77
Sourcetexture
Target image
Sourcecorrespondence
image
Targetcorrespondence image
![Page 78: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/78.jpg)
78
++ ==
![Page 79: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/79.jpg)
Portilla & Simoncelli Xu, Guo & Shum
input image
Wei & Levoy Image Quilting
![Page 80: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/80.jpg)
Portilla & Simoncelli Xu, Guo & Shum
input image
Wei & Levoy Image Quilting
![Page 81: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/81.jpg)
Homage to Shannon!
Portilla & Simoncelli Xu, Guo & Shum
input image
Wei & Levoy Image Quilting
![Page 82: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/82.jpg)
82
Summary of image quilting• Quilt together patches of input image
– randomly (texture synthesis) – constrained (texture transfer)
• Image Quilting – No filters, no multi-scale, no one-pixel-at-a-time! – fast and very simple– Results are not bad
![Page 83: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/83.jpg)
83
Median filterReplace each pixel by the median over N pixels (5 pixels, for these examples). Generalizes to “rank order” filters.
Spike noise is removed
In: Out:
5-pixel neighborhood
Monotonic edges remain unchanged
In: Out:
![Page 84: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/84.jpg)
84
Degraded image
![Page 85: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/85.jpg)
85
Radius 1 median filter
![Page 86: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/86.jpg)
86
Radius 2 median filter
![Page 87: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/87.jpg)
87
CCD color sampling
![Page 88: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/88.jpg)
88
Color sensing, 3 approaches
• Scan 3 times (temporal multiplexing)• Use 3 detectors (3-ccd camera, and color
film)• Use offset color samples (spatial
multiplexing)
![Page 89: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/89.jpg)
89
Typical errors in temporal multiplexing approach
• Color offset fringes
![Page 90: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/90.jpg)
90
Typical errors in spatial multiplexing approach.
• Color fringes.
![Page 91: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/91.jpg)
91
CCD color filter pattern
detector
![Page 92: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/92.jpg)
92
The cause of color moire
detector
Fine black and white detail in imagemis-interpreted as color information.
![Page 93: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/93.jpg)
93
Black and white edge falling on color CCD detector
Black and white image (edge)
Detector pixel colors
![Page 94: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/94.jpg)
94
Color sampling artifact
Interpolated pixel colors, for grey edge falling on coloreddetectors (linear interpolation).
![Page 95: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/95.jpg)
95
Typical color moire patterns
Blow-up of electronic cameraimage. Notice spuriouscolors in the regionsof fine detail in the plants.
![Page 96: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/96.jpg)
96
Color sampling artifacts
![Page 97: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/97.jpg)
97
Human Photoreceptors
(From Foundations of Vision, by Brian Wandell, Sinauer Assoc.)
![Page 98: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/98.jpg)
98
Brewster’s colors example (subtle).
Scale relativeto humanphotoreceptorsize: each linecovers about 7photoreceptors.
![Page 99: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/99.jpg)
99
Median Filter Interpolation
• Perform first interpolation on isolated color channels.
• Compute color difference signals.• Median filter the color difference signal.• Reconstruct the 3-color image.
![Page 100: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/100.jpg)
100
Two-color sampling of BW edge
Sampled data
Linear interpolation
Color difference signal
Median filtered color difference signal
![Page 101: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/101.jpg)
101
R-G, after linear interpolation
![Page 102: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/102.jpg)
102
R – G, median filtered (5x5)
![Page 103: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/103.jpg)
103
Recombining the median filtered colors
Linear interpolation Median filter interpolation
![Page 104: Today: non-linear filters, and · 2002-10-12 · 4 Mid-term exam Problem set 3 given out today – Open book, open web. – Work by yourself. This problem set is a mid-term exam,](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f0fd83a7e708231d4462ab5/html5/thumbnails/104.jpg)
104
Didn’t get a chance to show:
Local gain control.