video magnification...robust video registration • find feature points with harris corner detector...
TRANSCRIPT
![Page 1: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/1.jpg)
12/07/17
Video Magnification
Computational Photography
Derek Hoiem, University of Illinois
Magritte, “The Listening Room”
![Page 2: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/2.jpg)
Today
1. Video Magnification
– Lagrangian (point tracking) approach
– Eulerian (signal within a pixel) approach
2. Video Microphone
![Page 3: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/3.jpg)
Imperceptible Motions and Changes
[Liu et al. 2005] [Wu et al. 2012]
![Page 4: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/4.jpg)
MAGNIFIED Imperceptible Motions and Changes
[Wu et al. 2012][Liu et al. 2005]
![Page 5: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/5.jpg)
Motion Magnification
Goal: exaggerate selected motions
Ideas?
![Page 6: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/6.jpg)
Approach 1: Point Tracking
Motion Magnification (SIGGRAPH 2005)
Ce Liu Antonio Torralba William T. Freeman Frédo Durand Edward
H. Adelson
Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology
Following slides based on SG 2005 presentation:
http://people.csail.mit.edu/celiu/motionmag/motionmag.html
![Page 7: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/7.jpg)
Naïve Approach
• Magnify the estimated optical flow field
• Rendering by warping
Original sequence Magnified by naïve approach
![Page 8: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/8.jpg)
Tracking-based Motion Magnification
+ +
++ +
Liu et al. Motion Magnification, 20058
![Page 9: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/9.jpg)
Robust Video Registration
• Find feature points with Harris corner detector on the reference frame
• Track feature points
• Select a set of robust feature points with inlier and outlier estimation (most from the rigid background)
• Warp each frame to the reference frame with a global affine transform
![Page 10: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/10.jpg)
Feature tracking trick 1: Adaptive Region of Support
• SSD patch matching search
• Learn adaptive region of support using expectation-maximization (EM) algorithm
region of support
Confused by
occlusion !
time
time
![Page 11: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/11.jpg)
Feature tracking trick 2: trajectory pruning
• Tracking with adaptive region of support
• Outlier detection and removal by interpolation
Nonsense at full occlusion!
time
inlier
probability Outliers
![Page 12: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/12.jpg)
Without adaptive region of support and trajectory pruningWith adaptive region of support and trajectory pruning
Comparison
![Page 13: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/13.jpg)
Cluster trajectories based on normalized complex correlation
• The similarity metric should be independent of phase and magnitude
• Normalized complex correlation
tt
t
tCtCtCtC
tCtCCCS
)()()()(
|)()(|),(
2211
2
21
21
![Page 14: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/14.jpg)
Spectral Clustering
Affinity matrix Clustering Reordering of affinity matrix
Two clustersTrajectory
Tra
jecto
ry
![Page 15: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/15.jpg)
Clustering Results
![Page 16: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/16.jpg)
Flow vectors of
clustered sparse
feature points
Dense optical flow
field of cluster 1
(leaves)
Dense optical flow
field of cluster 2
(swing)
From Sparse Feature Points to Dense Optical Flow Field
Cluster 1: leaves
Cluster 2: swing
Interpolate dense optical flow field using locally weighted linear regression
![Page 17: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/17.jpg)
Motion Layer Assignment
• Assign each pixel to a motion cluster layer, using four cues:
– Motion likelihood—consistency of pixel’s intensity if it moves with the
motion of a given layer (dense optical flow field)
– Color likelihood—consistency of the color in a layer
– Spatial connectivity—adjacent pixels favored to belong the same group
– Temporal coherence—label assignment stays constant over time
• Energy minimization using graph cuts
![Page 18: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/18.jpg)
Segmentation Results
Two additional layers: static background and outlier
![Page 19: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/19.jpg)
Layered Motion Representation for Motion Processing
Background Layer 1 Layer 2
Layer mask
Occluding layers
Appearance for each
layer before texture
filling-in
Appearance for each
layer after texture
filling-in
Appearance for each
layer after user
editing
![Page 20: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/20.jpg)
![Page 21: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/21.jpg)
![Page 22: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/22.jpg)
Discussion of point tracking approach
• Good: applies to any motion
• Bad: requires accurate point tracking, clustering and texture synthesis, so likely to fail
![Page 23: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/23.jpg)
Eulerian Video Magnification for Revealing Subtle Changes in the WorldHao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Fredo Durand, William T. Freeman
ACM Transactions on Graphics, Volume 31, Number 4 (Proc. SIGGRAPH) 2012
Following slides based on Siggraph presentations:
http://people.csail.mit.edu/mrub/vidmag/
http://people.csail.mit.edu/nwadhwa/phase-video/
Approach 2: pixelwise processing
Phase-based Video Motion ProcessingNeal Wadhwa, Michael Rubinstein, Fredo Durand, William T. Freeman
ACM Transactions on Graphics, Volume 32, Number 4 (Proc. SIGGRAPH) 2013
![Page 24: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/24.jpg)
Lagrangian and Eulerian Perspectives: Fluid Dynamics
Lagrangian Eulerian
24
![Page 25: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/25.jpg)
Eulerian Perspective: Videos
• Each pixel is processed independently
• Treat each pixel as a time series and apply signal processing to it
y
xtime
Eulerian
25
![Page 26: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/26.jpg)
Method Overview
Spat
ial
Dec
om
posi
tion
Spat
ial
Dec
om
posi
tion
Tem
pora
l fi
lter
ing
Spat
ial
Dec
om
posi
tion
Tem
pora
l fi
lter
ing
α1
α2
αn-1
αn
Σ
Σ
Σ
Σ
Spat
ial
Dec
om
posi
tion
Tem
pora
lfi
lter
ing
α1
α2
αn-1
Rec
ons
truct
ion
αn
Σ
Σ
Σ
Σ
26
Laplacian
Pyramid
Bandpass filter
intensity at each
pixel over time
Amplify
bandpassed
signal and add
back to original
![Page 27: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/27.jpg)
Subtle Color Variations
• The face gets slightly redder when blood flows
• Unfortunately usually below the per pixel noise level
Input frame Luminance trace (zero
mean)
27
![Page 28: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/28.jpg)
Subtle Color Variations
1. Average spatially to overcome sensor and quantization noise
Input frame
Spatially averaged luminance trace
Luminance trace (zero
mean)pulses
28
![Page 29: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/29.jpg)
Amplifying Subtle Color Variations
2. Filter temporally to extract the signal of interest
Temporally bandpassed trace
⊗
Temporal filter
=
Spatially averaged luminance trace
29
![Page 30: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/30.jpg)
Color Amplification Results
Source Color-amplified (x100)
0.83-1 Hz (50-60 bpm)
30
![Page 31: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/31.jpg)
Heart Rate Extraction
Temporally bandpassed trace
(one pixel)
Peak detection Pulse locations
31
![Page 32: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/32.jpg)
Heart Rate Extraction
2.33-2.67 Hz (140-160 bpm)Thanks to Dr. Donna Brezinski and the Winchester Hospital staff
EKG
32
![Page 33: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/33.jpg)
Why It Amplifies Motion
33
![Page 34: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/34.jpg)
Relating Temporal and Spatial Changes
34Space
Inte
nsity
0
𝐵(𝑥, 𝑡)
Implied
Translation
𝟏 + 𝜶 𝜹
(1 + 𝑎)𝐵(𝑥, 𝑡)
𝟏 + 𝜶 𝑩 𝒙, 𝒕 ≈ 𝟏 + 𝜶 𝜹(𝒕)𝑰′ 𝒙, 𝒕(1st order Taylor expansion)
𝐼’(𝑥, 𝑡)
𝑩 𝒙, 𝒕 ≈ 𝜹(𝒕)𝑰′ 𝒙, 𝒕(1st order Taylor expansion)
![Page 35: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/35.jpg)
Relating Temporal and Spatial Changes
35
Signal at time 𝑡
Signal at time 𝑡 + 1
Motion-magnified
Courtesy of Lili Sun
![Page 36: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/36.jpg)
Synthetic 2D Example
Source Motion-magnified
36
![Page 37: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/37.jpg)
Selective Motion Magnification
Source
(Single video with 4 blobs)
Temporal filter:
1-3 Hz
Motion-magnified (2 Hz)
7 Hz
3 Hz
5 Hz
2 Hz
7 Hz
3 Hz
5 Hz
2 Hz
37
![Page 38: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/38.jpg)
Selective Motion Magnification
Motion-magnified (3 Hz)
Temporal filter:
2-4 Hz
7 Hz
3 Hz
5 Hz
2 Hz
7 Hz
3 Hz
5 Hz
2 Hz
Source
(Single video with 4 blobs)
38
![Page 39: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/39.jpg)
Selective Motion Magnification
Motion-magnified (5 Hz)
Temporal filter:
4-6 Hz
7 Hz
3 Hz
5 Hz
2 Hz
7 Hz
3 Hz
5 Hz
2 Hz
Source
(Single video with 4 blobs)
39
![Page 40: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/40.jpg)
Selective Motion Magnification
Motion-magnified (7 Hz)
Temporal filter:
6-8 Hz
7 Hz
3 Hz
5 Hz
2 Hz
7 Hz
3 Hz
5 Hz
2 Hz
Source
(Single video with 4 blobs)
40
![Page 41: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/41.jpg)
When Does It Break?
Inte
nsity
0Space
Signal at time 𝑡
Signal at time 𝑡 + 1
Motion-magnified
0
255
Clipped
Clipped41
![Page 42: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/42.jpg)
Motion Magnification Artifacts
Source Motion-magnified (3.6-6.2 Hz, x60)
Artifact
Artifact
42
![Page 43: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/43.jpg)
Scale-varying Amplification
• The amplification is more accurate for low spatial frequencies– Images are smoother
– Motions are smaller
• Use the desired 𝛼 for lower spatial frequencies, and attenuate for the higher spatial frequencies
43
Spatial wavelength (2𝜋/freq)
Amplification
Desired amplificationLinear
falloff
![Page 44: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/44.jpg)
Motion Magnification Results
Source Motion-magnified (0.4-3 Hz, x10)
44
![Page 45: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/45.jpg)
Motion Magnification
Source Motion-magnified (0.4-3 Hz, x10)
Radial
Ulnar
45
![Page 46: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/46.jpg)
Discussion of pixelwise intensity amplification approach
• Good: – Does not require explicit motion estimation or
texture synthesis (robust)
– Very fast (real time)
• Bad:– Can only handle very small motions
– Amplifies noise
![Page 47: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/47.jpg)
Limitations of Linear Motion Processing
Inte
nsity
Space
(𝑥)
• Noise amplified with signal
Signal at time 𝑡 + Δ𝑡
Motion-magnified
𝜕𝑥
𝜕𝑡
𝜕𝑓
𝜕𝑡
𝜕𝑓
𝜕𝑥
![Page 48: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/48.jpg)
Limitations of Linear Motion Processing
Source Linear
SIGGRAPH’12Phase-based
SIGGRAPH’13
Overshoot
Overshoot
![Page 49: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/49.jpg)
Eulerian approach part 2: shift phase instead of amplifying intensity
Translation in space is equivalent to a shift in phase
• Linear Motion Processing
– Assumes images are locally linear
– Translate by changing intensities
• Phase-Based Motion Processing
– Represents images as collection of local sinusoids
– Translate by shifting phase
𝜕𝑥
𝜕𝑡
𝜕𝑓
𝜕𝑡
𝜕𝑓
𝜕𝑥
![Page 50: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/50.jpg)
Linear vs. Phase-Based Motion Processing
Source Linear
SIGGRAPH’12Phase-based
SIGGRAPH’13
![Page 51: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/51.jpg)
Time (t)
Phase over Time
Radia
ns
Time (t)
Time (t)
Radia
ns
Phase over Time
…
WaveletsInput
Space(x)
Inte
nsity
Inte
nsity
Space(x)
a
Inte
nsity
Space(x)
![Page 52: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/52.jpg)
Phase over Time
Time (t)
Phase over TimeInput Motion-magnified
Space(x)
Radia
ns
Time (t)
Time (t)
Radia
ns
Space(x)
Inte
nsity
Inte
nsity
…Space(x)
Inte
nsity
Inte
nsity
Space(x)
Wavelets
![Page 53: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/53.jpg)
2D Complex Steerable Pyramid
Filter Bank
Orientation 2
Real Imag
Scale
1
Orientation 1
Sca
le 2
Orientation 1
Orientation 2F
requency (𝜔𝑦)
Frequency (𝜔𝑥)
Idealized Transfer Functions
Scale
Orientation
FFT
Highpass
Residual
Lowpass
Residual
![Page 54: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/54.jpg)
Phase over Time
Amplitude
Sub-bands
Filter Bank
Orientation 2
Real Imag
Scale
1
Orientation 1
Sca
le 2
Orientation 1
Orientation 2
Phase
Phase
Time (s)
Phase over time
Phase
Bandpassed Phase over time
Time (s)
Temporal
Filtering
![Page 55: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/55.jpg)
New Phase-Based Pipeline
Amplitude
Sub-bands
Filter Bank
Complex steerable
pyramid
[Simoncelli et al. 1992]
Bandpassed
Phase
Orientation 2
Real Imag
Scale
1
Orientation 1
Sca
le 2
Orientation 1
Orientation 2
Phase
Re
co
nstr
uctio
n
𝛼 ⊗
𝛼 ⊗
𝛼 ⊗
𝛼 ⊗
Te
mp
ora
l F
ilte
rin
g
Temporal filtering on
phases
![Page 56: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/56.jpg)
Improvement #1: Less Noise
Noise amplified Noise translated
Source (IID
Noise, std=0.1)
Linear [Wu et al.
2012] (x50)Phase-based
(x50)
![Page 57: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/57.jpg)
Improvement #2: More Amplification
Amplification factor Motion in the sequence
Range of linear method:
Range of phase-based method:
4 times the
amplification!
![Page 58: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/58.jpg)
Attenuated
• Local phase can move image features, but only within the filter window
Limits of Phase Based Magnification
Amplification factor
![Page 59: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/59.jpg)
Comparison with [Wu et al. 2012]
Wu et al. 2012 Phase-Based (this paper)
![Page 60: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/60.jpg)
Eye Movements
Source (500FPS) Motion magnified x150 (30-50 Hz)
![Page 61: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/61.jpg)
Expressions
Low frequency motions Mid-range frequency motions
Source
![Page 62: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/62.jpg)
Ground Truth Validation
• Induce motion (with hammer)
• Record with accelerometer
AccelerometerHammer Hit
![Page 63: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/63.jpg)
Ground Truth Validation
![Page 64: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/64.jpg)
Motion Attenuation
Source Turbulence Removed
Sequence courtesy Vimeo user Vincent Laforet
![Page 65: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/65.jpg)
Car Engine
Source
![Page 66: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/66.jpg)
Car Engine
22Hz Magnified
![Page 67: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/67.jpg)
Car Engine
Source
![Page 68: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/68.jpg)
Car Engine
22Hz Magnified
![Page 69: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/69.jpg)
Neck Skin Vibrations
Frequency (Hz)0 500 1000
Pow
er
Source (2 KHz)
![Page 70: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/70.jpg)
Source (2 KHz)
Frequency (Hz)0 500 1000
Pow
er
Source (2 KHz) 100 Hz Amplified x100
Fundamental
frequency: ~100Hz
![Page 71: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/71.jpg)
Source (2 KHz) Amplified (x100)
![Page 72: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/72.jpg)
Discussion of pixelwise phase magnification approach
• Good: – Does not require explicit motion estimation– Produces more direct translations (instead of perceived
motion)– Does not amplify noise
• Bad:– Limited in range of amplication (compared to pointwise
approach)– May have difficulty with non-periodic motion and large
motions
![Page 73: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/73.jpg)
Non-periodic Motions and Large Motions
Source (300 FPS) Motion Magnification x50 Motion Magnification x50
Large Motions Unmagnified
Non-periodic motion
![Page 74: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/74.jpg)
The Visual Microphone:Passive Recovery of Sound from Video
Abe Davis Michael Rubinstein Neal Wadhwa
Gautham Mysore Fredo Durand William T. Freeman
(slides adopted from Siggraph presentation)
![Page 75: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/75.jpg)
Remote Sound Recovery
![Page 76: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/76.jpg)
Sound and Motion
Source: mediacollege.com
![Page 77: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/77.jpg)
Air pressure
(Pa)
Object
motion
(mm disp.)
Video
(pixels)
Recovered
Signal
(~Pa)
Object response (A)
Frequency
RM
S D
isp
lace
men
t
Processing (B)
Camera (Projection)
Input
The Visual Microphone
79
![Page 78: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/78.jpg)
Processing
• Extract local motion signals
• Average and Align
• Post-process
80
![Page 79: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/79.jpg)
Some materials are better microphones than others
81
Object response (A)
Frequency
RM
S D
isp
lace
men
t
Air pressure
(Pa)
Object
motion
(mm disp.)
![Page 80: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/80.jpg)
Sound Recovered from Video
Source sound in the roomWaveform Spectrogram
Recovered sound
2200Hz video
![Page 81: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/81.jpg)
Sound Recovered from Video
Source sound in the roomWaveform Spectrogram
2200Hz video
Recovered sound
![Page 82: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/82.jpg)
Sound Recovered from Video
20 kHz video
Source sound in the roomWaveform Spectrogram
(small patch on the chip bag)Recovered sound
![Page 83: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/83.jpg)
85
![Page 84: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/84.jpg)
86
![Page 85: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/85.jpg)
Rolling Shutter
Artifacts
references
87
https://www.flickr.com/photos/sorenragsdale/3904937619/
http://www.flickr.com/photos/boo66/5730668979/
![Page 86: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/86.jpg)
Rolling Shutter
88
![Page 87: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/87.jpg)
Rolling Shutter
89
![Page 88: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/88.jpg)
Rolling Shutter
90
Input video (60 fps)
Input
Recovered Sound
![Page 89: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/89.jpg)
Rolling Shutter
91
Input video (60 fps)
Input
Recovered Sound
400Hz!
![Page 90: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/90.jpg)
Summary
• Several ways to magnify motion
– Directly measure and exaggerate point motions
– Amplify intensity changes after temporal filtering (creating apparent motion)
– Amplify local phase variations after temporal filtering
• Micro-motion estimates can be used to measure sound
![Page 91: Video Magnification...Robust Video Registration • Find feature points with Harris corner detector on the reference frame • Track feature points • Select a set of robust feature](https://reader033.vdocuments.site/reader033/viewer/2022053123/60ad12ea276527030d5220cb/html5/thumbnails/91.jpg)
Next week
• Final class
– A few examples of cutting edge applications, inc.deep network based approaches
– Where to learn more
– Course feedback (important for me)