more details on presentations - computer sciencelazebnik/research/fall08/lec02...more details on...
TRANSCRIPT
![Page 1: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/1.jpg)
More details on presentations• Aim to speak for ~50 min (after 15 min review,
leaving 10 min for discussions)• Try to plan discussion topics
• It’s fine to “steal” slides from the Web, but be sure to acknowledge sources• Include SIGGRAPH videos, demos (if any)
• Send me your slides afterwards for including on the class webpage
• Feel free to include related papers and background material in the presentation, or make a presentation based on two (or more) closely related papers• Actually, this can make your life easier!
• Feel free to propose papers not on the list• Email me four presentation preferences and any
scheduling constraints by the end of Sunday• Come and talk to me about any doubts or questions
![Page 2: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/2.jpg)
Texture Synthesis
Most slides from A. Efros
![Page 3: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/3.jpg)
Last Time: Beauty in Complexity
![Page 4: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/4.jpg)
Today: Texture
• Texture is “stuff” (as opposed to “things”)• Characterized by spatially repeating patterns• Texture lacks the full range of complexity of
photographic imagery, but makes a good starting point for study of image-based techniques
radishes rocks yogurt
![Page 5: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/5.jpg)
Texture Synthesis• Goal of Texture Synthesis: create new samples of
a given texture• Many applications: virtual environments, hole-
filling, texturing surfaces
![Page 6: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/6.jpg)
The Challenge
• Need to model the whole spectrum: from repeated to stochastic texture
repeated
stochastic
Both?
![Page 7: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/7.jpg)
Statistical modeling of texture• Assume stochastic model of texture (Markov
Random Field)• Stationarity: the stochastic model is the same
regardless of position
stationary texture non-stationary texture
![Page 8: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/8.jpg)
Statistical modeling of texture• Assume stochastic model of texture (Markov
Random Field)• Stationarity: the stochastic model is the same
regardless of position• Markov property:
p(pixel | rest of image) = p(pixel | neighborhood)
?
![Page 9: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/9.jpg)
Motivation from Language• Shannon (1948) proposed a way to generate
English-looking text using N-grams– Assume a Markov model– Use a large text to compute probability distributions
of each letter given N–1 previous letters – Starting from a seed repeatedly sample the
conditional probabilities to generate new letters– One can use whole words instead of letters too
![Page 10: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/10.jpg)
Mark V. Shaney (Bell Labs)
• Results (using alt.singles corpus):– “As I've commented before, really relating to
someone involves standing next to impossible.”– “One morning I shot an elephant in my arms and
kissed him.”– “I spent an interesting evening recently with a
grain of salt.”• Notice how well local structure is preserved!
– Now let’s try this in 2D...
![Page 11: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/11.jpg)
Efros & Leung Algorithm
Idea initially proposed in 1981 (Garber ’81), but dismissed as too computationally expensive!
![Page 12: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/12.jpg)
Efros & Leung Algorithm
• Assume Markov property, sample from P(p|N(p))– Building explicit probability tables infeasible
pp
Synthesizing a pixel
non-parametricsampling
Input image
– Instead, we search the input image for all sufficiently similar neighborhoods and pick one match at random
![Page 13: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/13.jpg)
|| – ||2
Finding matches
• Sum of squared differences (SSD)
![Page 14: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/14.jpg)
|| *( – )||2
Finding matches
• Sum of squared differences (SSD)– Gaussian-weighted to make sure closer
neighbors are in better agreement
![Page 15: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/15.jpg)
Details• Random sampling from the set of candidates vs.
picking the best candidate• Initialization
– Start with a few rows of white noise and grow in scanline order– Start with a “seed” in the middle and grow outward in layers
• Hole filling: growing is in “onion skin” order– Within each “layer”, pixels with most neighbors are
synthesized first– Normalize error by the number of known pixels– If no close match can be found, the pixel is not synthesized
until the end
![Page 16: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/16.jpg)
Varying Window Size
input
![Page 17: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/17.jpg)
Varying Window Size
Increasing window size
![Page 18: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/18.jpg)
Synthesis Resultsfrench canvas raffia weave
![Page 19: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/19.jpg)
More Resultswhite bread brick wall
![Page 20: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/20.jpg)
Homage to Shannon
![Page 21: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/21.jpg)
Hole Filling
![Page 22: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/22.jpg)
Extrapolation
![Page 23: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/23.jpg)
Failure Cases
Growing garbage Verbatim copying
![Page 24: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/24.jpg)
Summary• The Efros & Leung algorithm
– Very simple– Surprisingly good results– …but very slow
![Page 25: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/25.jpg)
Accelerating texture synthesis• For textures with large-scale
structures, use a Gaussian pyramid to reduce required neighborhood size
Li-Yi Wei and Marc Levoy, "Fast Texture Synthesis using Tree-structured Vector Quantization," SIGGRAPH 2000
![Page 26: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/26.jpg)
Accelerating texture synthesis• For textures with large-scale
structures, use a Gaussian pyramid to reduce required neighborhood size– Low-resolution image is
synthesized first– For synthesis at a given pyramid
level, the neighborhood consists of already generated pixels at this level plus all neighboring pixels at the lower level
Li-Yi Wei and Marc Levoy, "Fast Texture Synthesis using Tree-structured Vector Quantization," SIGGRAPH 2000
![Page 27: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/27.jpg)
Accelerating texture synthesis
• Perform fast approximate nearest neighbor search using tree-structured vector quantization
Li-Yi Wei and Marc Levoy, "Fast Texture Synthesis using Tree-structured Vector Quantization," SIGGRAPH 2000
![Page 28: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/28.jpg)
Tree-structured Vector Quantization
Slide credit: Li-Yi Wei
![Page 29: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/29.jpg)
Accelerating texture synthesis• Perform fast approximate nearest neighbor search
using tree-structured vector quantization– Use all neighborhoods of the exemplar texture to build
a tree-structured codebook– To find a match for a new neighborhood, follow the
tree in best-first order (at each level, choose child codeword closest to the query)
– Example running times from the paper:• Exhaustive search: 360 sec• Building codebook: 22 sec, synthesis: 7.5 sec
– Shortcomings?
Li-Yi Wei and Marc Levoy, "Fast Texture Synthesis using Tree-structured Vector Quantization," SIGGRAPH 2000
![Page 30: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/30.jpg)
Accelerating texture synthesis• Use original position of
already synthesized neighborhood pixels to create a “short list” of candidates for the current pixel
Ashikhmin, M. "Synthesizing Natural Textures," 2001 ACM Symposium on Interactive 3D Graphics
![Page 31: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/31.jpg)
Accelerating texture synthesis
Ashikhmin, M. "Synthesizing Natural Textures," 2001 ACM Symposium on Interactive 3D Graphics
Original sample Wei & Levoy
Ashikhmin Boundaries
![Page 32: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/32.jpg)
pp
Image Quilting [Efros & Freeman]
• Observation: neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = blockunit of synthesis = block• Exactly the same as Efros & Leung but now we want
P(B|N(B))
• Much faster: synthesize all pixels in a block at once
Synthesizing a block
![Page 33: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/33.jpg)
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
![Page 34: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/34.jpg)
min. error boundary
Minimal error boundaryoverlapping blocks vertical boundary
__ ==22
overlap error
![Page 35: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/35.jpg)
The Philosophy
• The “Corrupt Professor’s Algorithm”:– Plagiarize as much of the source image as you can– Then try to cover up the evidence
• Rationale: – Texture blocks are by definition correct samples of
texture so problem only connecting them together
![Page 36: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/36.jpg)
Algorithm– Pick size of block and size of overlap– Synthesize blocks in raster order
– Search input texture for block that satisfies overlap constraints (above and left)
– Paste new block into resulting texture• use dynamic programming to compute minimal error
boundary cut
![Page 37: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/37.jpg)
![Page 38: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/38.jpg)
![Page 39: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/39.jpg)
![Page 40: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/40.jpg)
![Page 41: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/41.jpg)
![Page 42: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/42.jpg)
![Page 43: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/43.jpg)
Failures(Chernobyl
Harvest)
![Page 44: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/44.jpg)
input image Wei & Levoy Efros & Freeman
![Page 45: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/45.jpg)
Wei & Levoy Efros & Freemaninput image
![Page 46: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/46.jpg)
Wei & Levoy Efros & Freemaninput image
![Page 47: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/47.jpg)
Political Texture Synthesis!
![Page 48: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/48.jpg)
Texture Transfer• Take the texture from one
object and “paint” it onto another object– This requires separating texture
and shape– That’s HARD, but we can cheat – Assume we can capture shape by
boundary and rough shading
• Then, just add another constraint when sampling: similarity to underlying image at that spot
![Page 49: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/49.jpg)
Sourcetexture
Target image
Sourcecorrespondence
image
Targetcorrespondence image
![Page 50: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/50.jpg)
++ ==
![Page 51: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/51.jpg)
++ ==
++ ==
parmesan
rice
![Page 52: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/52.jpg)
++ ==
![Page 53: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/53.jpg)
==++
![Page 54: More details on presentations - Computer Sciencelazebnik/research/fall08/lec02...More details on presentations • Aim to speak for ~50 min (after 15 min review, leaving 10 min for](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec886b1fa146116dd23a0ea/html5/thumbnails/54.jpg)