![Page 1: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/1.jpg)
![Page 2: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/2.jpg)
Wikipedia - Mysid
![Page 3: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/3.jpg)
![Page 4: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/4.jpg)
![Page 5: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/5.jpg)
![Page 6: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/6.jpg)
![Page 7: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/7.jpg)
[Akiyoshi Kitaoka, Ritsumeikan University, Kyoto]
![Page 8: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/8.jpg)
![Page 9: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/9.jpg)
![Page 11: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/11.jpg)
Project 1 Feedback
Thank you all.
- There are no late days on proj0
- Numpy gearup beyond proj0 tutorial – OK
- Short time frame [my fault]
- Axes3D - OK
- Improvements to code description – OK
![Page 12: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/12.jpg)
Project 1 Convolution SpeedsConvolution speed (seconds)
Jake Chanan 0.91
Alice Marbach 0.94
Andrew Cooke 1.07
Albert Webson 1.2
Reza Esfandiarpoor 1.2
Andrew Levy 1.22
James White 1.23
Da Huo 1.24
Troy Moo Penn 1.27
Michael T Lincoln 1.29
FFT convolution (seconds)
James Wang 0.45
Matthew Kovoor 0.75
Isaiah Liu 0.85
![Page 13: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/13.jpg)
Hybrid images
![Page 14: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/14.jpg)
![Page 15: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/15.jpg)
![Page 16: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/16.jpg)
![Page 17: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/17.jpg)
![Page 18: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/18.jpg)
![Page 19: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/19.jpg)
![Page 20: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/20.jpg)
![Page 21: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/21.jpg)
![Page 22: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/22.jpg)
![Page 23: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/23.jpg)
Corners
Slides from Rick Szeliski, Svetlana Lazebnik, Derek Hoiem and Grauman&Leibe 2008 AAAI Tutorial
Szeliski 4.1
Also called interest points, key points, etc.
Often described as ‘local’ features.
Filtering Edges
Feature points
![Page 24: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/24.jpg)
Correspondence across views
Matching points, patches, edges, or regions across images.
Sparse or local correspondence vs.
dense correspondence (at every pixel).
≈
Hays
![Page 25: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/25.jpg)
Fundamental to Applications
• Image alignment
• 3D reconstruction
• Motion tracking (robots, drones, AR)
• Indexing and database retrieval
• Object recognition
Hays
![Page 26: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/26.jpg)
Example application: Panorama stitching
We have two images –
how do we estimate how to overlay them?
![Page 27: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/27.jpg)
Local features: main components
1) Detection:Find a set of distinctive key points.
2) Description: Extract feature descriptor around
each interest point as vector.
3) Matching: Compute distance between feature
vectors to find correspondence.
],,[ )1()1(
11 dxx =x
],,[ )2()2(
12 dxx =x
Td )x,x( 21
1x
2x
K. Grauman, B. Leibe
![Page 28: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/28.jpg)
Goal: Distinctiveness
We want to be able to reliably determine which
point goes with which.
?
Kristen Grauman
May be difficult in structured environments
with repeated elements
![Page 29: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/29.jpg)
Goal: Repeatability
We want to detect (at least some of)
the same points in both images.
With these points, there’s no chance to find true matches!
Kristen Grauman
Under geometric and
photometric variations.
![Page 30: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/30.jpg)
Example: Object Detection
Finding distinctive and repeatable feature points can be difficult when
we want our features to be invariant to large transformations:
- geometric variation (translation, rotation, scale, perspective)
- appearance variation (reflectance, illumination)
Keypoint Descriptors James Hays
![Page 31: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/31.jpg)
Goal: Compactness and Efficiency
We want the representation to be as small and
as fast as possible
– Much smaller than a whole image
We’d like to be able to run the detection
procedure independently per image
- Match just the compact descriptors for speed.
- Difficult! We don’t get to see ‘the other image’ at
match time, e.g., object detection.
Kristen Grauman
![Page 32: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/32.jpg)
Characteristics of good features
DistinctivenessEach feature can be uniquely identified
RepeatabilityThe same feature can be found in several images despite differences:
- geometrically (translation, rotation, scale, perspective)
- photometrically (reflectance, illumination)
Compactness and efficiencyMany fewer features than image pixels; run independently per image
Kristen Grauman
![Page 33: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/33.jpg)
Local features: main components
1) Detection:Find a set of distinctive key points.
2) Description:Extract feature descriptor around
each interest point as vector.
3) Matching:Compute distance between feature
vectors to find correspondence.
![Page 34: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/34.jpg)
Detection: Basic Idea
We do not know which other image
locations the feature will end up being
matched against.
But we can compute how stable a location
is in appearance with respect to small
variations in its position.
Strategy: Compare image patch
against local neighbors.
A. Efros
![Page 35: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/35.jpg)
Corner Detection: Basic Idea
Recognize corners by looking at small window.
We want a window shift in any direction to give a large change in intensity.
“Edge”:
no change
along the edge
direction
“Corner”:
significant
change in all
directions
“Flat” region:
no change in
all directions
A. Efros
![Page 36: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/36.jpg)
Corner Detection by Auto-correlation
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y= + + −
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
Source: R. Szeliski
Change in appearance of window w(x,y) for shift [u,v]:
![Page 37: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/37.jpg)
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y= + + −
I(x, y)E(u, v)
E(0,0)
w(x, y)
Change in appearance of window w(x,y) for shift [u,v]:
Corner Detection by Auto-correlation
![Page 38: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/38.jpg)
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y= + + −
Change in appearance of window w(x,y) for shift [u,v]:
I(x, y)E(u, v)
E(3,2)
w(x, y)
Corner Detection by Auto-correlation
![Page 39: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/39.jpg)
As a surface
Think-Pair-Share:
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y= + + −
Correspond the three
red crosses to (b,c,d).
![Page 40: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/40.jpg)
Corner Detection by Auto-correlation
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y= + + −
We want to discover how E behaves for small shifts
Change in appearance of window w(x,y) for shift [u,v]:
But this is very slow to compute naively.
O(window_width2 * shift_range2 * image_width2)
O( 112 * 112 * 6002 ) = 5.2 billion of these
14.6k ops per image pixel
![Page 41: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/41.jpg)
Corner Detection by Auto-correlation
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y= + + −
But we know the response in E that we are looking
for – strong peak.
Change in appearance of window w(x,y) for shift [u,v]:
We want to discover how E behaves for small shifts
![Page 42: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/42.jpg)
Strategy:
Approximate E(u,v) locally by a quadratic surface,
and look for that instead.
≈
![Page 43: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/43.jpg)
Recall: Taylor series expansion
A function f can be represented by an infinite series
of its derivatives at a single point a:
Approximation of
f(x) = ex
centered at f(0)
Wikipedia
As we care about window
centered, we set a = 0
(MacLaurin series)
![Page 44: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/44.jpg)
Local quadratic approximation of E(u,v) in the
neighborhood of (0,0) is given by the
second-order Taylor expansion:
+
+
v
u
EE
EEvu
E
EvuEvuE
vvuv
uvuu
v
u
)0,0()0,0(
)0,0()0,0(][
2
1
)0,0(
)0,0(][)0,0(),(
Notation: partial derivative
![Page 45: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/45.jpg)
Local quadratic approximation of E(u,v) in the
neighborhood of (0,0) is given by the
second-order Taylor expansion:
+
+
v
u
EE
EEvu
E
EvuEvuE
vvuv
uvuu
v
u
)0,0()0,0(
)0,0()0,0(][
2
1
)0,0(
)0,0(][)0,0(),(
Ignore function
value; set to 0 Ignore first
derivative,
set to 0
Just look at shape of
second derivative
(2D quadratic surface)
![Page 46: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/46.jpg)
Corner Detection: Mathematics
The quadratic approximation simplifies to
2
2,
( , )x x y
x y x y y
I I IM w x y
I I I
=
where M is a second moment matrix computed from image
derivatives:
v
uMvuvuE ][),(
M
v
u
EE
EEvuvuE
vvuv
uvuu
)0,0()0,0(
)0,0()0,0(][),(
![Page 47: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/47.jpg)
=
yyyx
yxxx
IIII
IIIIyxwM ),(
x
II x
y
II y
y
I
x
III yx
Corners as distinctive interest points
2 x 2 matrix of image derivatives
(averaged in neighborhood of a point)
Notation:
James Hays
![Page 48: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/48.jpg)
E(u,v) is locally approximated by a quadratic
surface. Let’s try to understand how its shape
relates to M.
Interpreting the second moment matrix
v
uMvuvuE ][),(
=
yx yyx
yxx
III
IIIyxwM
,2
2
),(
James Hays
![Page 49: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/49.jpg)
Let’s take horizontal “slices” of our approximation of E(u, v):
Interpreting the second moment matrix
[𝑢 𝑣] 𝑀𝑢𝑣
= constant
James Hays
Each coloured line in the diagram below is where
![Page 50: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/50.jpg)
Visualization of second moment matrix
Simple image:
Checkerboard.
1 corner
4 edges
4 flat regions
Note: Edges show ‘wide’ response
because image derivatives were
blurred by Gaussian 𝜎 = 1 before
visualizing.
![Page 51: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/51.jpg)
Visualization of second moment matrices
James Hays
![Page 52: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/52.jpg)
Visualization of second moment matrices
James Hays
For cornerness,
we only care about
the ‘steepness’,
not the rotation.
Can we ignore
this somehow?
![Page 53: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/53.jpg)
Linear algebra review
𝑀 is symmetric. Symmetric matrices have
orthogonal eigenvectors (i.e., a basis).
𝑀 is square. Square matrices are
diagonalizable if some matrix 𝑃 exists s.t.
𝑀 = 𝑃−1𝐴𝑃, where 𝐴 has only diagonal
entries and 𝑃 represents a change of basis
(in 2D, a rotation).
![Page 54: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/54.jpg)
Consider a horizontal “slice” of E(u, v):
Interpreting the second moment matrix
This defines an ellipse.
RRM
= −
2
11
0
0
The axis lengths of the ellipse are determined by the eigenvalues,
and the orientation is determined by a rotation matrix 𝑅.
direction of the
slowest change
direction of the
fastest change
(max)-1/2
(min)-1/2
const][ =
v
uMvu
Diagonalization of M:
James Hays
Note inverse relationship:
larger eigenvalue =
steeper slope; smaller
ellipse in visualization
𝐴
![Page 55: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/55.jpg)
Classification of image points using eigenvalues of M
1
2
“Corner”
1 and 2 are large,
1 ~ 2;
E increases in all
directions
1 and 2 are small;
E is almost constant
in all directions
“Edge”
1 >> 2
“Edge”
2 >> 1
“Flat”
region
![Page 56: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/56.jpg)
Classification of image points using eigenvalues of M
“Corner”
C > 0
“Edge”
C < 0
“Edge”
C < 0
|C| small
1
2Cornerness score:
α: some constant (~0.04 to 0.06)
“Flat”
region
𝐶 = 𝜆1𝜆2 − 𝛼(𝜆1 + 𝜆2)2
![Page 57: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/57.jpg)
Linear algebra review
Determinant of a diagonal matrix is the
product of all eigenvalues. 𝐴 is diagonal.
Trace of a square matrix is the sum of its
diagonal entries; and is the sum of its
eigenvalues.
![Page 58: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/58.jpg)
Classification of image points using eigenvalues of M
“Corner”
C > 0
“Edge”
C < 0
“Edge”
C < 0
|C| small
2)(trace)det( MMC −=
Determinant:
Trace:
1
2
Remember your linear algebra:
Cornerness score:
α: some constant (~0.04 to 0.06)
“Flat”
region
(diagonal matrices)
𝐶 = 𝜆1𝜆2 − 𝛼(𝜆1 + 𝜆2)2
Avoids explicit eigenvalue computation!
![Page 59: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/59.jpg)
This is the Harris corner detector!
1) Compute M matrix for each window to recover a
cornerness score 𝐶.Note: We can find M purely from the per-pixel image derivatives!
2) Threshold to find pixels which give large corner
response (𝐶 > threshold).
3) Find the local maxima pixels,
i.e., non-maximal suppression.
C.Harris and M.Stephens. “A Combined Corner and Edge Detector.” Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
![Page 60: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/60.jpg)
Harris Corner Detector [Harris88]
1. Compute image derivatives (optionally, blur first).
2. Compute 𝑀 components
as squares of derivatives.
3. Gaussian filter g() with width s
= 𝑔 𝐼𝑥2 , 𝑔 𝐼𝑦
2 , 𝑔 𝐼𝑥 ∘ 𝐼𝑦
𝐼𝑥 𝐼𝑦
𝑔(𝐼𝑥2) 𝑔(𝐼𝑦2) 𝑔(𝐼𝑥 ∘ 𝐼𝑦)
4. Compute cornerness
𝑅 5. Threshold on 𝐶 to pick high cornerness
6. Non-maximal suppression to pick peaks.
James Hays
0. Input image
We want to compute M at each pixel.𝐼
𝐼𝑥𝑦𝐼𝑥2 𝐼𝑦
2
𝐶 = det 𝑀 − 𝛼 trace 𝑀 2
= 𝑔 𝐼𝑥2 ∘ 𝑔 𝐼𝑦
2 − 𝑔 𝐼𝑥 ∘ 𝐼𝑦2
−𝛼 𝑔 𝐼𝑥2 + 𝑔 𝐼𝑦
2 2
Reminder: 𝑎 ∘ 𝑏 is
Hadamard product
(element-wise
multiplication)
![Page 61: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/61.jpg)
Harris Detector: Steps
![Page 62: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/62.jpg)
Harris Detector: Steps
Compute corner response 𝐶
![Page 63: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/63.jpg)
Harris Detector: Steps
Find points with large corner response: 𝐶 > threshold
![Page 64: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/64.jpg)
Harris Detector: Steps
Take only the points of local maxima of 𝐶
![Page 65: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/65.jpg)
Harris Detector: Steps
![Page 66: Wikipedia - Mysid · 2020. 2. 6. · Reza Esfandiarpoor 1.2 Andrew Levy 1.22 James White 1.23 Da Huo 1.24 Troy Moo Penn 1.27 Michael T Lincoln 1.29 FFT convolution (seconds) James](https://reader033.vdocuments.site/reader033/viewer/2022060819/6098b48059040b1e2977dba5/html5/thumbnails/66.jpg)
Live Harris Demo