video textures arno schödl richard szeliski david salesin irfan essa microsoft research, georgia...
TRANSCRIPT
![Page 1: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/1.jpg)
Video TexturesVideo Textures
Arno SchödlRichard Szeliski
David SalesinIrfan Essa
Microsoft Research, Georgia Tech
![Page 2: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/2.jpg)
Still photosStill photos
![Page 3: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/3.jpg)
Video clipsVideo clips
![Page 4: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/4.jpg)
Video texturesVideo textures
![Page 5: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/5.jpg)
Related workRelated work
Image-based rendering
• Novel still views from still images
• View Morphing [Seitz 96]
• Lightfields / Lumigraph [Levoy 96, Gortler 96]
• Façade [Debevec 96]
• Virtualized Reality [Kanade 97]
• many more…
![Page 6: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/6.jpg)
Image-based rendering (IBR)Image-based rendering (IBR)
• From View Morphing by Seitz and Dyer
Slide added by D.Brogan
![Page 7: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/7.jpg)
Related workRelated work
Texture synthesis
• Infinite extension of 2D pattern
• [Heeger 95]
• [de Bonet 97]
• [Efros 99]
![Page 8: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/8.jpg)
Related workRelated work
Temporal texture synthesis
• Video as a spatio-temporal texture
• Statistical Learning of Multidimensional Textures [Bar-Joseph 99][Wei 00]
![Page 9: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/9.jpg)
Related workRelated work
Video Rewrite
• Assemble video snippets to generate lip motion
• [Bregler 97]
![Page 10: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/10.jpg)
Periodic motion analysis
• Uses similar analysis tools
Related workRelated work
• [Niyogi 94]
• [Seitz 97]
• [Polana 97]
• [Cutler 00]
![Page 11: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/11.jpg)
Problem statementProblem statement
video clip video texture
![Page 12: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/12.jpg)
Our approachOur approach
How do we find good transitions?
![Page 13: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/13.jpg)
Finding good transitions Finding good transitions
Compute L2 distance Di, j between all frames
Similar frames make good transitions
frame ivs.
frame j
![Page 14: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/14.jpg)
Markov chain representationMarkov chain representation
2 3 41
Similar frames make good transitions
![Page 15: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/15.jpg)
Transition costs Transition costs
Transition from i to j if successor of i is similar to j
Cost function: Cij = Di+1, j
i
j
i+ 1
j-1
i j D i+ 1, j
![Page 16: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/16.jpg)
Transition probabilitiesTransition probabilities
Probability for transition Pij inversely related
to cost:Pij ~ exp ( – Cij / 2 )
high low
![Page 17: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/17.jpg)
Preserving dynamicsPreserving dynamics
![Page 18: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/18.jpg)
Preserving dynamics Preserving dynamics
![Page 19: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/19.jpg)
Preserving dynamics Preserving dynamics
Cost for transition ij
Cij = wk Di+k+1, j+kk = -N
N -1
i
j j+ 1
i+ 1 i+ 2
j-1j-2
i jD i, j - 1 D Di+ 1, j i+ 2, j+ 1
i-1
D i-1, j-2
![Page 20: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/20.jpg)
Preserving dynamics – effect Preserving dynamics – effect
Cost for transition ij
Cij = wk Di+k+1, j+kk = -N
N -1
![Page 21: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/21.jpg)
2 3 41
Dead endsDead ends
No good transition at the end of sequence
![Page 22: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/22.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
![Page 23: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/23.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
![Page 24: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/24.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
![Page 25: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/25.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
![Page 26: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/26.jpg)
2 3 41
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
• Q-learning
Future costFuture cost
![Page 27: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/27.jpg)
Future cost – effectFuture cost – effect
![Page 28: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/28.jpg)
Fixed-length videosFixed-length videos
• Alternative to random transitions
• Precompute set of loops up front
Slide edited by D.Brogan
![Page 29: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/29.jpg)
Fixed-length videosFixed-length videos
• Creating fixed-length videos• Enumerate all sets of transitions with a total
length L (multiloops)• Select the ones that are legal (the loops
overlap so video can jump from one to another)
• Find the best one
• Exponential in number of transitions
Slide added by D.Brogan
![Page 30: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/30.jpg)
Fixed-length videosFixed-length videos
• Create an L x N table
• L = length of longest loop being considered
• N = number of loops being considered
Slide added by D.Brogan
![Page 31: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/31.jpg)
Fixed-length videosFixed-length videos
• Populate the table
• For each (loopn, lengthl) tuple, find best loop (possibly compound) of length l that includes at least one instance of loop n
Slide added by D.Brogan
![Page 32: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/32.jpg)
Fixed-length videosFixed-length videos
• Populate the table with dynamic programming
• Start from row one, find best loops
• Proceed through the rows
• (length5, loopC) is found from (length3, loopC) + (length2, loopD)
Slide added by D.Brogan
![Page 33: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/33.jpg)
Fixed-length videosFixed-length videos
• Each cell requires inspection of at most L-1 compound loops from same column and N-1 entries from other columns
• Total Cost: O(L2N2)
Slide added by D.Brogan
![Page 34: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/34.jpg)
Fixed-length videosFixed-length videos
• Organizing the loops
• You cannot jump from any loop to any other loop… sequence order matters
• Start with loop with largest last frame num.
• Remove it from the pool
• Identify isolated loops
• Add loops that require first loop for access
Slide added by D.Brogan
![Page 35: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/35.jpg)
Visual discontinuitiesVisual discontinuities
• Problem: Visible “Jumps”
![Page 36: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/36.jpg)
Crossfading Crossfading
• Solution: Crossfade from one sequence to the other.
A i-2
A i-2
B j-2
15
…
…
3
1 2
2 1
3
4
4 4
4 4
4+ + +
A i-1
A i-1 / B j-2 A i-1 / B j-2 A i-1 / B j-2
B j-1
A i
B j
A i+1
B j+1
B j+1
![Page 37: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/37.jpg)
Morphing Morphing
• Interpolation task:
A25 B2
5 C15+ +
![Page 38: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/38.jpg)
Morphing Morphing
• Interpolation task:
• Compute correspondencebetween pixels of all frames
• Optical flow
A25 B2
5 C15+ +
Slide edited by D.Brogan
![Page 39: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/39.jpg)
Morphing Morphing
• Interpolation task:
• Compute correspondence between pixels of all frames
• Interpolate pixel position andcolor in morphed frame
• based on [Shum 2000]
A25 B2
5 C15+ +
![Page 40: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/40.jpg)
Results – crossfading/morphingResults – crossfading/morphing
![Page 41: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/41.jpg)
Results – crossfading/morphingResults – crossfading/morphing
Jump Cut Crossfade Morph
![Page 42: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/42.jpg)
CrossfadingCrossfading
![Page 43: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/43.jpg)
Frequent jump & crossfadingFrequent jump & crossfading
![Page 44: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/44.jpg)
Video portraitVideo portrait
Useful for web pages
![Page 45: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/45.jpg)
Combine with IBR techniques
Video portrait – 3DVideo portrait – 3D
![Page 46: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/46.jpg)
Region-based analysisRegion-based analysis
• Divide video up into regions
• Generate a video texture for each region
![Page 47: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/47.jpg)
Automatic region analysisAutomatic region analysis
![Page 48: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/48.jpg)
User selects target frame range
User-controlled video texturesUser-controlled video textures
slow variable fast
![Page 49: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/49.jpg)
Lengthen / shorten video without affecting speed
Time warpingTime warping
shorter original longer
![Page 50: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/50.jpg)
Video-based animationVideo-based animation
• Like spritescomputer games
• Extract spritesfrom real video
• Interactively control desired motion
©1985 Nintendo of America Inc.
![Page 51: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/51.jpg)
Video sprite extractionVideo sprite extraction
blue screen m attingand velocity estim ation
![Page 52: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/52.jpg)
C i j = + angle C i j
vector tom ouse pointer
S im ilarity term Contro l term
velocity vector
Animation{ {
Video sprite controlVideo sprite control
• Augmented transition cost:
![Page 53: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/53.jpg)
F i j
F i j
F i j F i j
F i j
F i jF i j
S W
W
N W
N
N E
E
S E
S
G oal
Video sprite controlVideo sprite control
• Need future cost computation
• Precompute future costs for a few angles.
• Switch between precomputed angles according to user input
• [GIT-GVU-00-11]
![Page 54: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/54.jpg)
Interactive fishInteractive fish
![Page 55: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/55.jpg)
Summary Summary
• Video clips video textures
• define Markov process
• preserve dynamics
• avoid dead-ends
• disguise visual discontinuities
![Page 56: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/56.jpg)
Summary Summary
• Extensions
• regions
• external constraints
• video-based animation
![Page 57: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/57.jpg)
Discussion Discussion
• Some things are relatively easy
![Page 58: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/58.jpg)
Discussion Discussion
• Some are hard
![Page 59: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/59.jpg)
Future workFuture work
• Better analysis of complex motions
• New optimality metrics
• More powerful video-based animation
• Other kinds of video-based rendering
![Page 60: Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649e735503460f94b7252b/html5/thumbnails/60.jpg)
A final exampleA final example