Download - Data-driven methods: Video Textures
![Page 1: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/1.jpg)
Data-driven methods: Video Textures
Cs195g Computational PhotographyJames Hays, Brown, Spring 2010Many slides from Alexei Efros
![Page 2: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/2.jpg)
Weather Forecasting for Dummies™ Let’s predict weather:
• Given today’s weather only, we want to know tomorrow’s• Suppose weather can only be {Sunny, Cloudy, Raining}
The “Weather Channel” algorithm:• Over a long period of time, record:
– How often S followed by R
– How often S followed by S
– Etc.
• Compute percentages for each state: – P(R|S), P(S|S), etc.
• Predict the state with highest probability!• It’s a Markov Chain
![Page 3: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/3.jpg)
4.04.02.0
3.03.04.0
1.06.03.0
Markov Chain
What if we know today and yesterday’s weather?
![Page 4: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/4.jpg)
Text Synthesis
[Shannon,’48] proposed a way to generate English-looking text using N-grams:• Assume a generalized Markov model• Use a large text to compute prob. distributions of
each letter given N-1 previous letters • Starting from a seed repeatedly sample this Markov
chain to generate new letters • Also works for whole words
WE NEED TO EAT CAKE
![Page 5: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/5.jpg)
Mark V. Shaney (Bell Labs)
Results (using alt.singles corpus):• “As I've commented before, really relating to
someone involves standing next to impossible.”
• “One morning I shot an elephant in my arms and kissed him.”
• “I spent an interesting evening recently with a grain of salt”
![Page 6: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/6.jpg)
Data Driven PhilosophyThe answer to most non-trivial questions is “Let’s see what the data says”, rather than “Let’s build a parametric model” or “Let’s simulate it from the ground up”.
This can seem unappealing in its simplicity (see Chinese Room thought experiment), but there are still critical issues of representation, generalization, and data choices.
In the end, data driven methods work very well.
“Every time I fire a linguist, the performance of our speech recognition system goes up.” “ Fred Jelinek (Head of IBM's Speech Recognition Group), 1988
![Page 7: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/7.jpg)
Video TexturesVideo Textures
Arno Schödl
Richard Szeliski
David Salesin
Irfan Essa
Microsoft Research, Georgia Tech
![Page 8: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/8.jpg)
Still photosStill photos
![Page 9: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/9.jpg)
Video clipsVideo clips
![Page 10: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/10.jpg)
Video texturesVideo textures
![Page 11: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/11.jpg)
Problem statementProblem statement
video clip video texture
![Page 12: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/12.jpg)
Our approachOur approach
• How do we find good transitions?
![Page 13: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/13.jpg)
Finding good transitions Finding good transitions
Compute L2 distance Di, j between all frames
Similar frames make good transitions
frame ivs.
frame j
![Page 14: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/14.jpg)
Markov chain representationMarkov chain representation
2 3 41
Similar frames make good transitions
![Page 15: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/15.jpg)
Transition costs Transition costs
• Transition from i to j if successor of i is similar to j
• Cost function: Cij = Di+1, j
• i
j
i+ 1
j-1
i j D i+ 1, j
![Page 16: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/16.jpg)
Transition probabilitiesTransition probabilities
•Probability for transition Pij inversely related
to cost:•Pij ~ exp ( – Cij / 2 )
high low
![Page 17: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/17.jpg)
Preserving dynamicsPreserving dynamics
![Page 18: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/18.jpg)
Preserving dynamics Preserving dynamics
![Page 19: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/19.jpg)
Preserving dynamics Preserving dynamics
• Cost for transition ij
• Cij = wk Di+k+1, j+kk = -N
N -1
i
j j+ 1
i+ 1 i+ 2
j-1j-2
i jD i, j - 1 D Di+ 1, j i+ 2, j+ 1
i-1
D i-1, j-2
![Page 20: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/20.jpg)
Preserving dynamics – effect Preserving dynamics – effect
• Cost for transition ij
• Cij = wk Di+k+1, j+kk = -N
N -1
![Page 21: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/21.jpg)
2 3 41
Dead endsDead ends
• No good transition at the end of sequence
![Page 22: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/22.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fij = Cij + mink Fjk
![Page 23: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/23.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fij = Cij + mink Fjk
![Page 24: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/24.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fij = Cij + mink Fjk
![Page 25: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/25.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fij = Cij + mink Fjk
![Page 26: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/26.jpg)
2 3 41
• Propagate future transition costs backward
• Iteratively compute new cost
• Fij = Cij + mink Fjk
• Q-learning
Future costFuture cost
![Page 27: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/27.jpg)
Future cost – effectFuture cost – effect
![Page 28: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/28.jpg)
Finding good loopsFinding good loops
• Alternative to random transitions
• Precompute set of loops up front
![Page 29: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/29.jpg)
Video portraitVideo portrait
• Useful for web pages
![Page 30: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/30.jpg)
Region-based analysisRegion-based analysis
• Divide video up into regions
• Generate a video texture for each region
![Page 31: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/31.jpg)
Automatic region analysisAutomatic region analysis
![Page 32: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/32.jpg)
Discussion Discussion
• Some things are relatively easy
![Page 33: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/33.jpg)
Discussion Discussion
• Some are hard
![Page 34: Data-driven methods: Video Textures](https://reader036.vdocuments.site/reader036/viewer/2022062314/56813dbd550346895da78828/html5/thumbnails/34.jpg)
Michel Gondry train videoMichel Gondry train video
http://www.youtube.com/watch?v=0S43IwBF0uM