learning low-level vision william t. freeman egon c. pasztor owen t. carmichael
DESCRIPTION
Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael. Model image and scene patches as nodes in a Markov network. image patches. scene patches. image. F ( x i , y i ). Y ( x i , x j ). scene. Network joint probability. 1. Õ. Õ. =. Y. F. y. P. (. x. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/1.jpg)
Learning Low-Level Vision
William T. Freeman Egon C. Pasztor
Owen T. Carmichael
![Page 2: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/2.jpg)
Model image and scene patches as nodes in a Markov network
image patches
(xi, yi)
(xi, xj)
image
scene
scene patches
![Page 3: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/3.jpg)
Network joint probability
scene
image
Scene-scenecompatibility
functionneighboringscene nodes
local observations
Image-scenecompatibility
function
i
iiji
ji yxxxZ
yxP ),(),(1
),(,
![Page 4: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/4.jpg)
Super-resolution
• Image: low resolution image
• Scene: high resolution image
imag
esc
ene
ultimate goal...
![Page 5: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/5.jpg)
True high freqsLow-band input
(contrast normalized, PCA fitted)
Full freq. originalRepresentationZoomed low-freq.
(to minimize the complexity of the relationships we have to learn,we remove the lowest frequencies from the input image,
and normalize the local contrast level).
![Page 6: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/6.jpg)
Training images, ~100,000 image/scene patch pairs
Images from two Corel database categories: “giraffes” and “urban skyline”.
![Page 7: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/7.jpg)
Training data samples (magnified)
......
Gather ~100,000 patches
low freqs.
high freqs.
![Page 8: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/8.jpg)
Input low freqs.
Training data samples (magnified)
......
Nearest neighbor estimate
low freqs.
high freqs.
Estimated high freqs.
![Page 9: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/9.jpg)
![Page 10: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/10.jpg)
Image-scene compatibility function, (xi, yi)
Assume Gaussian noise takes you from observed image patch to synthetic sample:
y
x
![Page 11: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/11.jpg)
![Page 12: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/12.jpg)
Scene-scene compatibility function, (xi, xj)
Assume overlapped regions, d, of hi-res. patches differ by Gaussian observation noise:
d
Uniqueness constraint,not smoothness.
![Page 13: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/13.jpg)
Form linking matrices between nodes
scene samplesat node xj
scene samplesat node xk (xk, xj)
0.16 0.14 0.23 0.40 0.380.72 0.61 0.58 0.13 0.050.60 0.55 0.52 0.11 0.070.48 0.32 0.29 0.03 0.000.09 0.04 0.03 0.01 0.00
Linking matrix:(xk,xj)at samples
Local likelihoods are
all 1 for the scene samples
![Page 14: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/14.jpg)
Markov network
image patches
(xi, yi)
(xi, xj)
scene patches
![Page 15: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/15.jpg)
),,,,,(sumsummean 3213211321
yyyxxxPxxxx
MMSE
y1
Derivation of belief propagation
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3
![Page 16: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/16.jpg)
The posterior factorizes
y1
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3),(),(sum
),(),(sum
),(mean
),(),(
),(),(
),(sumsummean
),,,,,(sumsummean
3233
2122
111
3233
2122
111
3213211
3
2
1
321
321
xxyx
xxyx
yxx
xxyx
xxyx
yxx
yyyxxxPx
x
x
xMMSE
xxxMMSE
xxxMMSE
![Page 17: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/17.jpg)
Propagation rules
y1
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1
y2
x2
y3
x3
),(),(sum
),(),(sum
),(mean
3233
2122
111
3
2
1
xxyx
xxyx
yxx
x
x
xMMSE
)( ),( ),(sum)( 23222211
21
2
xMyxxxxMx
![Page 18: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/18.jpg)
Belief, and message updates
jii =
ij( )\
( ) ( , ) ( , ) ( )j
j ki i i j i j j j
x k N j i
M x x x x y M x
j
( )
( ) ( , ) ( )kj j j j j j
k N j
b x x y M x
ˆ argmax ( )j
j j jx
x b x
![Page 19: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/19.jpg)
Optimal solution in a chain or tree:Belief Propagation
• “Do the right thing” Bayesian algorithm.
• For Gaussian random variables over time: Kalman filter.
• For hidden Markov models: forward/backward algorithm (and MAP variant is Viterbi).
![Page 20: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/20.jpg)
No factorization with loops!
y1
x1
y2
x2
y3
x3
),(),(sum
),(),(sum
),(mean
3233
2122
111
3
2
1
xxyx
xxyx
yxx
x
x
xMMSE
31 ),( xx
![Page 21: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/21.jpg)
Justification for running belief propagation in networks with loops
• Experimental results:
– Error-correcting codes
– Vision applications
• Theoretical results:
– For Gaussian processes, means are correct.
– Large neighborhood local maximum for MAP.
– Equivalent to Bethe approx. in statistical
physics.
Weiss and Freeman, 2000
Yedidia, Freeman, and Weiss, 2000
Freeman and Pasztor, 1999;Frey, 2000
Kschischang and Frey, 1998;McEliece et al., 1998
Weiss and Freeman, 1999
![Page 22: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/22.jpg)
VISTA--Vision by Image-Scene TrAining
image patches
(xi, yi)
(xi, xj)
image
scene
scene patches
![Page 23: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/23.jpg)
Super-resolution application
image patches
(xi, yi)
(xi, xj)
scene patches
![Page 24: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/24.jpg)
Iter. 3
Iter. 1
Belief PropagationInput
Iter. 0
After a few iterations of belief propagation, the algorithm selects spatially consistent high resolution
interpretations for each low-resolution patch of the input image.
![Page 25: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/25.jpg)
Zooming 2 octaves
85 x 51 input
Cubic spline zoom to 340x204 Max. likelihood zoom to 340x204
We apply the super-resolution algorithm recursively, zooming
up 2 powers of 2, or a factor of 4 in each dimension.
![Page 26: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/26.jpg)
Generic training images
Next, train on a generic set of training images. Using the same camera
as for the test image, but a random collection of
photographs.
![Page 27: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/27.jpg)
Cubic Spline
Original70x70
Markovnet, training:generic
True280x280
![Page 28: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/28.jpg)
Training image
![Page 29: Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael](https://reader035.vdocuments.site/reader035/viewer/2022081603/5681585f550346895dc5bc57/html5/thumbnails/29.jpg)
Processed image