mtx4700: experimental roboticsweb.aeromech.usyd.edu.au/mtrx4700/course_documents/... · linear...
TRANSCRIPT
![Page 1: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/1.jpg)
MTX4700: Experimental Robotics Robot Vision and Image Processing
Dr. Thierry Peynot Dr. Stefan B. Williams
[with material from Gonzalez & Woods, Chieh-Chih (Bob) Wang, Martial Hebert, Sebastian Thrun,…]
![Page 2: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/2.jpg)
Course Outline
MTRX4700: Experimental Robotics | Robot Vision 2
![Page 3: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/3.jpg)
Goals • To familiarize you with the basic techniques of Robot/
Computer Vision.
• To get you excited!
• To let you experience (and appreciate!) the difficulties of real-world computer vision
MTRX4700: Experimental Robotics | Robot Vision 3
![Page 4: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/4.jpg)
MTRX4700: Experimental Robotics | Robot Vision 4
![Page 5: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/5.jpg)
Vision in Robotics: Applications • SLAM
Movie courtesy of Andrew Davison
MTRX4700: Experimental Robotics | Robot Vision 5
![Page 6: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/6.jpg)
Vision in Robotics: Applications • Multiview Structure from Motion
Movie courtesy of N. Snavely et al. at the University of Washington MTRX4700: Experimental Robotics | Robot Vision 6
![Page 7: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/7.jpg)
Vision in Robotics: Applications
Movie courtesy of Jing Xiao MTRX4700: Experimental Robotics | Robot Vision 7
![Page 8: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/8.jpg)
Vision in Robotics: Applications • Tracking
Lookingbill, Lieb, Stavens and Thurn, ICRA2005 MTRX4700: Experimental Robotics | Robot Vision 8
![Page 9: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/9.jpg)
Vision in Robotics: Applications • Path-following
[ 2004, LAAS-CNRS ]
MTRX4700: Experimental Robotics | Robot Vision 9
![Page 10: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/10.jpg)
Vision in Robotics: Applications • (Dense)
3D Mapping from Stereovision
[ Johnson-Robertson et al., 2009] MTRX4700: Experimental Robotics | Robot Vision 10
![Page 11: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/11.jpg)
(Some) Major References: Vision
MTRX4700: Experimental Robotics | Robot Vision 11
![Page 12: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/12.jpg)
MTRX4700: Experimental Robotics | Robot Vision
Computer Vision [Trucco&Verri’98]
12
![Page 13: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/13.jpg)
Topics we will (hardly) cover… • Digital Imaging: Acquisition, Basic Definitions • Camera Geometry/Model
– Ideal pinhole model – Geometric models
• Introduction to Camera Calibration • Feature Extraction (Edge & Corner)
– Sobel and Canny Edge Detectors – Harris Corners – Interest Points
• Image Matching using detected corners
MTRX4700: Experimental Robotics | Robot Vision 13
![Page 14: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/14.jpg)
Image Acquisition
MTRX4700: Experimental Robotics | Robot Vision 14
![Page 15: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/15.jpg)
Digital Image Fundamentals • Digital image = 2D function: (x,y) è f(x,y) (1D or 3D)
© 1992–2008 R. C. Gonzalez & R. E. Woods
MTRX4700: Experimental Robotics | Robot Vision 15
![Page 16: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/16.jpg)
Digital Image Fundamentals • Sampling and Quantization
© 1992–2008 R. C. Gonzalez & R. E. Woods
MTRX4700: Experimental Robotics | Robot Vision 16
![Page 17: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/17.jpg)
Digital Image Fundamentals • Representation of a digital image
© 1992–2008 R. C. Gonzalez & R. E. Woods MTRX4700: Experimental Robotics | Robot Vision 17
![Page 18: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/18.jpg)
Digital Image Fundamentals • Aliasing Effect
© 2007 Scientific Volume Imaging b.v.
Example in 1 dimension
MTRX4700: Experimental Robotics | Robot Vision 18
![Page 19: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/19.jpg)
Digital Image Fundamentals • Aliasing Effect
Aliasing Effect Original image: 200x200
pixels
Sampled image: 100x100 pixels
19 MTRX4700: Experimental Robotics | Robot Vision
![Page 20: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/20.jpg)
Digital Image Fundamentals • Aliasing Effect
Aliasing Effect
Original image: 622x756 pixels 205x250 pixels
“Moiré pattern”
MTRX4700: Experimental Robotics | Robot Vision 20
![Page 21: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/21.jpg)
Basic Image Filtering • Modify the pixels in an image based on some function
of a local neighborhood of the pixels
10 5 3
4 5 1
1 1 7
7 Some function
MTRX4700: Experimental Robotics | Robot Vision 21
![Page 22: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/22.jpg)
Linear Filtering • Linear case is simplest and most useful
– Replace each pixel with a linear combination of its neighbors.
• The prescription for the linear combination is called the convolution kernel.
10 5 3
4 5 1
1 1 7
7
0 0 0
0 0.5 0
0 1.0 0.5
kernel
⊗ =
MTRX4700: Experimental Robotics | Robot Vision 22
![Page 23: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/23.jpg)
Linear Filter = Convolution
g22
+ g22 I(i,j)
I(.)
f (i,j) =
I(.)
g12
+ g12 I(i-1,j)
I(.)
g13
+ g13 I(i-1,j+1) +
I(.)
g21
g21 I(i,j-1)
I(.)
g23
+ g23 I(i,j+1) +
I(.)
g31
g31 I(i+1,j-1)
I(.)
g32
+ g32 I(i+1,j)
I(.)
g33
+ g33 I(i+1,j+1)
I(.)
g11
g11 I(i-1,j-1)
MTRX4700: Experimental Robotics | Robot Vision 23
![Page 24: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/24.jpg)
Linear Filter = Convolution
• Example on the web: www.jhu.edu/~signals/convolve
• Matlab function: conv(1D) or conv2(2D)
∑ −−=⊗=lk
lkglnkmIgInmf,
],[],[],[
1],[with,
=∑lk
lkg
MTRX4700: Experimental Robotics | Robot Vision 24
![Page 25: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/25.jpg)
Original Image
MTRX4700: Experimental Robotics | Robot Vision 25
![Page 26: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/26.jpg)
Slight Blurring
MTRX4700: Experimental Robotics | Robot Vision 26
![Page 27: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/27.jpg)
More Blurring
MTRX4700: Experimental Robotics | Robot Vision 27
![Page 28: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/28.jpg)
Lots of Blurring
MTRX4700: Experimental Robotics | Robot Vision 28
![Page 29: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/29.jpg)
Gaussian
MTRX4700: Experimental Robotics | Robot Vision 29
![Page 30: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/30.jpg)
Smoothing with Gaussian
Slide credit: Marc Pollefeys
Gaussian Averaging
MTRX4700: Experimental Robotics | Robot Vision 30
![Page 31: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/31.jpg)
Gaussian Smoothing to Remove Noise
MTRX4700: Experimental Robotics | Robot Vision 31
![Page 32: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/32.jpg)
Edge Detection with Smoothed Images Image Blurred Image
- =
MTRX4700: Experimental Robotics | Robot Vision 32
![Page 33: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/33.jpg)
Smoothing to Reduce Aliasing
Scaling: 50%
Blurring prior to Scaling: 50%
© 1992–2008 R. C. Gonzalez & R. E. Woods
MTRX4700: Experimental Robotics | Robot Vision 33
![Page 34: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/34.jpg)
Camera Model & Geometry • Why needed • What is needed
MTRX4700: Experimental Robotics | Robot Vision 34
![Page 35: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/35.jpg)
Pinhole Imaging Model
MTRX4700: Experimental Robotics | Robot Vision 35
![Page 36: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/36.jpg)
Perspective Projection
Apparent sizes of objects depend on distance
MTRX4700: Experimental Robotics | Robot Vision 36
![Page 37: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/37.jpg)
Perspective Projection
x
f Z ZfXx =
X O -x
Apparent sizes of objects depend on distance (Z)
MTRX4700: Experimental Robotics | Robot Vision 37
![Page 38: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/38.jpg)
Special Case: Weak Perspective (Affine Projection)
MTRX4700: Experimental Robotics | Robot Vision 38
![Page 39: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/39.jpg)
Special Case: Weak Perspective (Affine Projection)
MTRX4700: Experimental Robotics | Robot Vision 39
![Page 40: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/40.jpg)
Pictorial Comparison
Weak perspective Perspective
MTRX4700: Experimental Robotics | Robot Vision 40
![Page 41: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/41.jpg)
Limitations of the pinhole model
MTRX4700: Experimental Robotics | Robot Vision 41
![Page 42: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/42.jpg)
Limitations of the pinhole model
MTRX4700: Experimental Robotics | Robot Vision 42
![Page 43: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/43.jpg)
Thin Lens Model
Real Lenses
Geometric Distortion
MTRX4700: Experimental Robotics | Robot Vision 43
![Page 44: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/44.jpg)
44
Thin Lens Model
Geometric Distortion
No distortion Pincushion Distortions Barrel Distortions
44
![Page 45: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/45.jpg)
Camera Calibration • Determine the intrinsic parameters of a camera (with
lens)
• What are Intrinsic Parameters? – Focal Length f – Pixel size sx , sy (ku , kv) – Distortion coefficients k1 , k2… – Image center u0 , v0
• It is important because …
MTRX4700: Experimental Robotics | Robot Vision 45
![Page 46: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/46.jpg)
Coord. Of Point in the “World” frame
Coord. of Pixel in the image
s = depth of 3D point in the scene (scale factor)
I E
Transf. between Camera frame and “World” frame [Devy 2003]
Camera Model Camera Model
MTRX4700: Experimental Robotics | Robot Vision 46
![Page 47: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/47.jpg)
• M = Matrix of Perspective Projection
• I = Matrix of Intrinsic Parameters
• E = Matrix of Extrinsic Parameters (Rotation + Translation)
[Devy 2003]
Camera Model Camera Model
MTRX4700: Experimental Robotics | Robot Vision 47
![Page 48: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/48.jpg)
Camera Calibration
Camera Model and Parameters
Locations in the image Locations
in the world
A Designed Object
MTRX4700: Experimental Robotics | Robot Vision 48
![Page 49: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/49.jpg)
Example of Calibration Procedure
• Camera Calibration Toolbox for Matlab – http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
MTRX4700: Experimental Robotics | Robot Vision 49
![Page 50: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/50.jpg)
Example of Calibration Procedure
Output: Calib_Results.mat
MTRX4700: Experimental Robotics | Robot Vision
![Page 51: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/51.jpg)
MTRX4700: Experimental Robotics | Robot Vision 51
![Page 52: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/52.jpg)
What is a Feature?
• Local, meaningful, detectable parts of the image
MTRX4700: Experimental Robotics | Robot Vision 52
![Page 53: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/53.jpg)
Features in Computer Vision • What is a feature?
– Location of sudden change
• Why use features? – Information content high – Invariant to change of view point, illumination – Reduces computational burden
MTRX4700: Experimental Robotics | Robot Vision 53
![Page 54: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/54.jpg)
Features in Computer Vision
Computer Vision
Algorithm
Feature 1 Feature 2
: Feature N
Feature 1 Feature 2
: Feature N
Image 2
Image 1
MTRX4700: Experimental Robotics | Robot Vision 54
![Page 55: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/55.jpg)
What makes for GOOD features? • Invariance
– View point (scale, orientation, translation) – Lighting conditions – Object deformations – Partial occlusion
• Other Characteristics – Uniqueness – Sufficiently many – Tuned to the task
MTRX4700: Experimental Robotics | Robot Vision 55
![Page 56: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/56.jpg)
First Feature: Edge
• Depth discontinuity
• Surface orientation discontinuity
• Reflectance discontinuity (i.e., change in surface material properties)
• Illumination discontinuity (e.g., shadow)
Slide credit: Christopher Rasmussen MTRX4700: Experimental Robotics | Robot Vision 56
![Page 57: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/57.jpg)
How to find Edges?
Basic Filtering à Edge Detectors à Corner Detectors à SIFT
MTRX4700: Experimental Robotics | Robot Vision 57
![Page 58: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/58.jpg)
Sobel Edge Detector
MTRX4700: Experimental Robotics | Robot Vision 58
![Page 59: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/59.jpg)
Sobel Operator/Kernel
Sobel operators :
Vertical edges
Horizontal edges
S1 = = S2
Edge Magnitude =
Edge Direction =
S1 + S1 2 2
tan-1 S1
S2
MTRX4700: Experimental Robotics | Robot Vision 59
![Page 60: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/60.jpg)
Robinson Compass Masks
-1 0 1 -2 0 2 -1 0 1
0 1 2 -1 0 1 -2 -1 0
1 2 1 0 0 0 -1 -2 -1
2 1 0 1 0 -1 0 -1 -2
1 0 -1 2 0 -2 1 1 -1
0 -1 -2 -1 0 -1 2 1 0
-1 -2 -1 0 0 0 1 2 1
-2 -1 0 -1 0 1 0 1 2
MTRX4700: Experimental Robotics | Robot Vision 60
![Page 61: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/61.jpg)
These kernels are Gradient operators
• Edges are discontinuities of intensity in images • Correspond to local maxima of image gradient • Gradient computed by convolution
• General principle applies: – Slight smoothing: Good localization, poor detection – More smoothing: Poor localization, good detection
MTRX4700: Experimental Robotics | Robot Vision 61
![Page 62: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/62.jpg)
MTRX4700: Experimental Robotics | Robot Vision
General principle
62
![Page 63: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/63.jpg)
Canny Edge Detector
• Canny edge detector shows that Gaussian derivatives yield good compromise between localization and detection.
MTRX4700: Experimental Robotics | Robot Vision 63
![Page 64: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/64.jpg)
Canny’s Result • Given a filter f, define the two objective functions:
– A(f) large if f produces good localization – B(f) large if f produces good detection
• Problem: Find a family of f that maximizes the compromise criterion A(f)B(f) under the constraint that a single peak is generated by a step edge.
• Solution: Unique solution, a close approximation is the Gaussian derivative.
MTRX4700: Experimental Robotics | Robot Vision 64
![Page 65: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/65.jpg)
Next Steps • The gradient magnitude enhances the edges but 2 problems remain:
– What threshold should we use to retain only the “real” edges? – Even if we had a perfect threshold, we would still have poorly localized
edges. How to optimally localize contours? • Solution:
– Non-local maxima suppression – Hysteresis thresholding
MTRX4700: Experimental Robotics | Robot Vision 65
![Page 66: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/66.jpg)
Non-Local Maxima Suppression
MTRX4700: Experimental Robotics | Robot Vision 66
![Page 67: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/67.jpg)
Non-Local Maxima Suppression
Select the single maximum point across the width of an edge.
MTRX4700: Experimental Robotics | Robot Vision 67
![Page 68: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/68.jpg)
Hysteresis Thresholding
MTRX4700: Experimental Robotics | Robot Vision 68
![Page 69: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/69.jpg)
MTRX4700: Experimental Robotics | Robot Vision 69
![Page 70: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/70.jpg)
Canny Edge Detector Algorithm Steps:
1. Apply derivative of Gaussian • Smooth image using Gaussian filter • Compute gradient magnitude and angle images
2. Non-maximum suppression • Thin multi-pixel wide “ridges” down to single pixel width
3. Double thresholding and Linking • Low, high edge-strength thresholds • Accept all edges over low threshold that are connected to
edge over high threshold
MTRX4700: Experimental Robotics | Robot Vision 70
![Page 71: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/71.jpg)
Corner Effect
• We need a way to detect corners. • Harris Corner Detector MTRX4700: Experimental Robotics | Robot Vision 71
![Page 72: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/72.jpg)
Finding Corners • Edge detectors perform poorly at corners. • Corners provide repeatable points for
matching, so are worth detecting.
Idea:
• Exactly at a corner, gradient is ill defined.
• However, in the region around a corner, gradient has two or more different values.
MTRX4700: Experimental Robotics | Robot Vision 72
![Page 73: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/73.jpg)
The Harris Corner Detector
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
∑∑∑∑
2
2
yyx
yxx
IIIIII
C
Form the second-moment matrix:
Sum over a small region around the hypothetical corner
Matrix is symmetric
Slide credit: David Jacobs
Gradient with respect to x, times gradient with respect to y
MTRX4700: Experimental Robotics | Robot Vision 73
![Page 74: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/74.jpg)
Simple Case
⎥⎦
⎤⎢⎣
⎡=
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
∑∑∑∑
2
12
2
00λ
λ
yyx
yxx
IIIIII
C
First, consider case where:
This means dominant gradient directions align with x or y axis
If either λ is close to 0, then this is not a corner, so look for locations where both are large.
Slide credit: David Jacobs MTRX4700: Experimental Robotics | Robot Vision 74
![Page 75: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/75.jpg)
General Case
It can be shown that since C is symmetric:
RRC ⎥⎦
⎤⎢⎣
⎡= −
2
11
00λ
λ
So every case is like a rotated version of the one on last slide.
Slide credit: David Jacobs MTRX4700: Experimental Robotics | Robot Vision 75
![Page 76: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/76.jpg)
So, to detect corners • Filter image with Gaussian to reduce noise • Compute magnitude of the x and y gradients at each pixel • Construct C in a window around each pixel (Harris uses a
Gaussian window – just blur) • Solve for product of λs (determinant of C) • If λs are both big (product reaches local maximum and is
above threshold), we have a corner (Harris also checks that ratio of λs is not too high)
MTRX4700: Experimental Robotics | Robot Vision 76
![Page 77: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/77.jpg)
Gradient Orientation
Closeup
MTRX4700: Experimental Robotics | Robot Vision 77
![Page 78: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/78.jpg)
Corner Detection
Corners are detected where the product of the ellipse axis lengths reaches a local maximum.
MTRX4700: Experimental Robotics | Robot Vision 78
![Page 79: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/79.jpg)
Harris Corners
• Originally developed as features for motion tracking • Greatly reduces amount of computation compared to tracking
every pixel • Translation and rotation invariant (but not scale invariant)
MTRX4700: Experimental Robotics | Robot Vision 79
![Page 80: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/80.jpg)
Scaling, Rotation, Illustration, Deformation
Want to find … in here
MTRX4700: Experimental Robotics | Robot Vision 80
![Page 81: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/81.jpg)
Scale Invariant Feature Transform (SIFT)
• Scale Invariant Feature Tracking (SIFT) features proposed by Lowe
• Successful matching over large change in range, lighting and orientation
• False positive rejection also verified
• Each keypoint: 128 element feature vector
81
![Page 82: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/82.jpg)
State of the art: SIFT = Scale Invariant Feature Transform
• David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60, 2 (2004), pp. 91-110.
• Invariances: • Scaling • Rotation • Illumination • Deformation
• Provides • Good localization
Yes
Yes
Yes
Maybe Yes
MTRX4700: Experimental Robotics | Robot Vision 82
![Page 83: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/83.jpg)
Finding Correspondences Between Images
• First step toward 3-D reconstruction • First step toward tracking • Object Recognition: finding correspondences between
feature points in “training” and “test” images.
MTRX4700: Experimental Robotics | Robot Vision 83
![Page 84: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/84.jpg)
MTRX4700: Experimental Robotics | Robot Vision [Jung 2004]
Application to Visual Odometry
84
Right
Stereo correlation
Temporal correlation
Left
Localisation (visual odometry)
![Page 85: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/85.jpg)
Application to Visual SLAM
[Brunner, Peynot and Vidal-Calleja, 2011]
MTRX4700: Experimental Robotics | Robot Vision 85
![Page 86: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/86.jpg)
SIFT • Extensively used, for:
– Camera localisation (Visual Odometry, SLAM…) – Structure from Motion – Tracking – Recognition – Etc…
• Computationally expensive – But: SURF (Bay, CVIU 2007) etc.
MTRX4700: Experimental Robotics | Robot Vision 86
![Page 87: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/87.jpg)
(Some) References • Ballard and Brown, Computer Vision, Prentice Hall, 1982 • D. A. Forsyth and J. Ponce, Computer Vision - A Modern
Approach, Prentice Hall, 2002 • R.C. Gonzalez and R.E. Woods, Digital Image Processing, 3rd
Edition, Prentice Hall, 2008 • R. I. Hartley and A. Zisserman, Multiple View Geometry in
Computer Vision, Cambridge University Press, 2000 • Countless Conference and Journal papers
MTRX4700: Experimental Robotics | Robot Vision 87
![Page 88: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/88.jpg)
References • Lots of books & papers, including:
MTRX4700: Experimental Robotics | Robot Vision 88
![Page 89: MTX4700: Experimental Roboticsweb.aeromech.usyd.edu.au/MTRX4700/Course_Documents/... · Linear Filtering • Linear case is simplest and most useful – Replace each pixel with a](https://reader033.vdocuments.site/reader033/viewer/2022060420/5f16ef96040ddd2f13497ec2/html5/thumbnails/89.jpg)
If you want to know more….
AMME4710: Image Processing and Computer Vision
Coming to a theatre near you…
in Semester 2!
MTRX4700: Experimental Robotics | Robot Vision 89