como observa un robot

38
C18 2013 1 / 38 C18 Computer Vision David Murray [email protected] www.robots.ox.ac.uk/dwm/Courses/4CV Michaelmas 2013

Upload: nerinconq

Post on 21-Jul-2016

226 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Como Observa Un Robot

C18 2013 1 / 38

C18 Computer Vision

David Murray

[email protected]/∼dwm/Courses/4CV

Michaelmas 2013

Page 2: Como Observa Un Robot

C18 2013 2 / 38

Computer Vision: This time ...

1. Introduction; imaging geometry; camera calibration2. Salient feature detection – edges, line and corners3. Recovering 3D from two images I: epipolar geometry.4. Recovering 3D from two images II: stereo correspondence

algorithms; triangulation.

Page 3: Como Observa Un Robot

C18 2013 3 / 38

Lecture 2

2.1 Cameras as photometric devices (just a note)

2.2 Image convolution

2.3 Edge detection

2.4 Edges to strings, strings to lines

2.5 Corner detection

Page 4: Como Observa Un Robot

C18 2013 4 / 38

2.1 Cameras as photometric devicesIn Lecture 1, we considered the camera as a geometric abstractiongrounded on the rectilinear propagation of light.

But they are also photometric devices.

Important to consider the way image formation depends on:the nature of scene surface (reflecting, absorbing ...)the relative orientations surface, light source and camerathe power and spectral properties of sourcethe spectral properties of the imaging system.

The important overall outcome (eg, Forsyth & Ponce, p62) is that

image irradiance is proportional to the scene radiance

A relief! This means the image really can tell us about the scene.

Page 5: Como Observa Un Robot

C18 2013 5 / 38

Cameras as photometric devices /ctdBut the study of photometry (often called physics-based vision)requires detailed models of the reflectance properties of the sceneand of the imaging process itself — the sort of models that underpinphoto-realistic graphics too.

Eg, understanding how light scatters on water droplets allowed thisimage to be de-fogged

Original De-fogged

Can we avoid going into such detail? ... Yes — by consideringaspects of scene geometry that are revealed in step changes inimage irradiance.

Page 6: Como Observa Un Robot

C18 2013 6 / 38

Step irradiance changes are due to ...

1. Changes in Scene Radiance.Natural (eg shadows) ordeliberately introduced via artificialillumination: light stripes aregenerated either by a mask orfrom a laser.

2. Changes in the scenereflectance at sudden changes insurface orientation. These arise atthe intersection of two surfaces,and thus represent geometricalentities fixed on the object in thescene.3. Changes in reflectance properties due to changes in surfacealbedo. The reflectance properties are scaled by a changing albedoarising from surface markings. Again these are fixed to the object.

Page 7: Como Observa Un Robot

C18 2013 7 / 38

Feature detection

We are after step spatial changes in image irradiance because(i) they are likely to be tied to scene geometry; and(ii) they are likely to be salient (have high info content)

A simple classification of changes in image irradiance I(x , y) is intoareas that, locally, have

1D structure 2D structure

⇒ Edge Detectors ⇒ Corner Detectors

Page 8: Como Observa Un Robot

C18 2013 8 / 38

2.2 Image operations for Feature detection

Feature detection should be a local operation, working withoutknowledge of higher geometrical entities or objects ...We should use pixel values I(x , y) and derivatives ∂I/∂x and∂I/∂y , and so on.

It would be useful to have a non-directional combination of these, sothat a feature map of a rotated image is identical to the rotatedfeature map of the original image: F(RI) = RF(I)

Considering edge detection, two possibilities are:

• Search for maxima in the gradient magnitude√(∂I/∂x)2 + (∂I/∂y)2 — 1st order, but non-linear

• Search for zeroes in the Laplacian∇2I = ∂2I/∂x2 + ∂2I/∂y2 — linear, but 2nd order

Page 9: Como Observa Un Robot

C18 2013 9 / 38

Which to choose?The gradient magnitude is attractive because it is first order in thederivatives. Differentiation enhances noise, and the 2nd derivatives inthe Laplacian operator introduces even more.

The Laplacian is attractive because it is linear, which means it can beimplemented by a succession of fast linear operations – effectivelymatrix operations as we are dealing with a pixelated image.

Both approaches have been used.

For both methods we need to consider* how to compute gradients, and* how to suppress noiseso that insignificant variations in pixel intensity are not flagged up asedges.

This will involve spatial convolution of the image with an impulseresponse function that smooths the image and computes thegradients.

Page 10: Como Observa Un Robot

C18 2013 10 / 38

Preamble: spatial convolutionYou are familiar with the 1D convolution integral in the time domainbetween a input signal i(t) and impulse response function h(t)

o(t) = i(t) ∗ h(t) =

∫ +∞

−∞i(t − τ)h(τ)dτ =

∫ +∞

−∞i(τ)h(t − τ)dτ .

The second equality reminds us that convolution commutesi(t) ∗ h(t) = h(t) ∗ i(t). It also associates.

In the frequency domain we would write O(s) = H(s)I(s).

Now in a continuous 2D domain, the spatial convolution integral is

o(x , y) = i(x , y) ∗ h(x , y) =

∫ +∞

−∞

∫ +∞

−∞i(x − a, y − b)h(a,b)dadb

In the spatial domain you’ll often see h(x , y) referred to as thepoint spread function or the convolution mask.

Page 11: Como Observa Un Robot

C18 2013 11 / 38

Spatial convolution /ctdFor pixelated images I(x , y) we need a discrete convolution

O(x , y) = I(x , y) ∗ h(x , y) =∑

i

∑j

I(x − i , y − j)h(i , j)

for x , y ranging over the image width and height respectively, and i , jensuring that access is made to any and all non-zero entries in h.

Many authors rewrite the convolution by replacing h(i , j) with h(−i ,−j)

O(x , y) =∑

i

∑j

I(x − i , y − j)h(i , j) =∑

i

∑j

I(x + i , y + j)h(−i ,−j)

=∑

i

∑j

I(x + i , y + j)h(i , j)

This looks more like the expression for a cross-correlation but,confusingly, it is still called a convolution.

Page 12: Como Observa Un Robot

C18 2013 12 / 38

Computing partial derivatives using convolutionWe can approximate ∂I/∂x at image pixel (x , y)using a central difference

∂I/∂x ≈ (1/2) [I(x + 1, y)− I(x , y)] + [I(x , y)− I(x − 1, y)]= (1/2) I(x + 1, y)− I(x − 1, y)

Writing this as a “proper” convolution we would set

h(−1) = +12

h(0) = 0 h(1) = −12

D(x , y) = I(x , y) ∗ h(x) =1∑

i=−1

I(x − i , y)h(i)

Notice how the “proper” mask is reversed from what we might naïvelyexpect from the expression.

Page 13: Como Observa Un Robot

C18 2013 13 / 38

Computing partial derivatives using convolution /ctdNow, as ever,

(∂I/∂x) ≈ (1/2)[I(x + 1, y)− I(x − 1, y)]

Writing this as a “sort of correlation”

h(−1) = −1/2 h(0) = 0 h(1) = +1/2

D(x , y) = I(x , y) ∗ h(x) =1∑

i=−1

I(x + i , y)h(i)

Note how we can just lay this mask directly on the pixels to bemultiplied and summed ...

Page 14: Como Observa Un Robot

C18 2013 14 / 38

Example Results

The actual image used isgrey-level, not colour

x-gradient “image” y -gradient “image”

Page 15: Como Observa Un Robot

C18 2013 15 / 38

In 2 dimensions ...As before, one imagines the flipped “correlation-like” mask centred ona pixel, and the sum of products computed.

Often a 2D mask is “separable” in that it can be broken up in to twoseparate 1D convolutions in x and y

O = h2d ∗ I = fy ∗ gx ∗ I

The computational complexity is lower — but intermediate storage isrequired, and so for a small mask it may be cheaper to use it directly.

Page 16: Como Observa Un Robot

C18 2013 16 / 38

Example result — Laplacian (non-directional)

The actual image used isgrey-level, not colour Laplacian

Page 17: Como Observa Un Robot

C18 2013 17 / 38

Noise and smoothingDifferentiation enhancesnoise — the edge appearsclear enough in the image,but less so in the gradientmap.If we knew the noise spectrum, we might find an optimal brickwallfilter g(x , y)⇔ G(s) to suppress noise edges outside the signal edgeband. We would applied this g to the image (g ∗ I) before findingderivatives.

But the sharper the cut-off in spatial-frequency, the wider the spatialsupport of g(x , y) has to be (it tends to an Infinite Impulse Response(IIR) filter). This is expensive to compute.

Conversely, if g(x , y) has finite size (an FIR filter) it will not beband-limited, spatially blurring of the “signal” edges.

Can we compromise spread in space and spatial-frequency in someoptimal way?

Page 18: Como Observa Un Robot

C18 2013 18 / 38

Compromise in space and spatial-frequencySuppose our IR function is h(x) and h H is a Fourier transformpair.

Define the spreads in space and spatial-frequency as X and Ω where

X 2 =

∫(x − xm)2h2(x)dx∫

h2(x)dxwith xm =

∫xh2(x)dx∫h2(x)dx

Ω2 =

∫(ω − ωm)2H2(ω)dω∫

H2(ω)dωwith ωm =

∫ωH2(ω)dω∫H2(ω)dω

Now vary h to minimize the product of the spreads U = XΩ.An uncertainty principle indicates that Umin = 1/2 when

h(x) = A Gaussian function =1√2πσ

exp(−x2

σ2

)

Page 19: Como Observa Un Robot

C18 2013 19 / 38

2.3 The Canny Edge Detector (JF Canny 1986)2D Gaussian smoothing is applied to the image

S(x , y) = G(x , y) ∗ I =1

2πσ2 exp[− (x2 + y2)

2σ2

]∗ I(x , y)

But separated into two separable masks

S(x , y) =1√2πσ

exp[− (x2)

2σ2

]∗ 1√

2πσexp

[− (y2)

2σ2

]∗ I(x , y)

Then 1st derivatives in x and y are found by two further convolutions

Dx (x , y) = ∂S/∂x = hx ∗ SDy (x , y) = ∂S/∂y = hy ∗ S.

Interestingly Canny’s decision to use a Gaussian is made from aseparate study of how to maximize robustness, localization, anduniqueness in edge detection.

Page 20: Como Observa Un Robot

C18 2013 20 / 38

Canny/ Implementation

Step 1: Gaussian smoothimage S = G ∗ I

This is separableS = Gx ∗Gy ∗ I

Often 5× 1 or 7× 1 masks

Step 2: Compute gradients∂S/∂x , ∂S/∂yusing derivative masks.

Page 21: Como Observa Un Robot

C18 2013 21 / 38

Canny/ ImplementationStep 3: Non-maximal suppressiona. Use ∂S/∂x and ∂S/∂y to find gradient magnitude and direction ata pixel.

b. Track along direction to markedpositions on neighbourhood perimeter.c. Linearly interpolate gradient magnitudeat positions using magnitudes at arrowpixels.d. If gradient magnitude at central pixelgreater than both interpolated values,declare an edgel, and fit parabola to find(x , y) to sub-pixel acuity along the edgedirection.

Step 4:Store edgel’s sub-pixel position (x , y); gradient strength; andorientation (full 360as light/dark distinguished),

Page 22: Como Observa Un Robot

C18 2013 22 / 38

Canny/ Implementation: Hysterical Thresholding

Step 5:Link edgestogether intostrings usingorientation to aidsearch forneighbouringedgels.

Step 6:Performthresholding withhysteresis, byrunning alongthe linked strings

Page 23: Como Observa Un Robot

C18 2013 23 / 38

Example of Edgel and String Outputs

Page 24: Como Observa Un Robot

C18 2013 24 / 38

2.4 From Strings to Straight LinesSplit algorithm for lines:

1. Fit a straight line to all edgels in string2. If RMS error less than threshold accept

and stop.3. Otherwise find point of highest curvature

on edge string and split into two. Repeatfrom 1 for each substring.

Merge algorithm for lines:1. Fit straight lines to each pair of

consecutive edgels in a string2. Compute RMS error for each potential

merger of an adjacent pair of lines into asingle line, and find the pair for whichthe RMS error is minimum

3. If RMS error less than threshold, mergeand repeat from 2.

Page 25: Como Observa Un Robot

C18 2013 25 / 38

Actually fitting lines and finding that RMS errorConsider n “edgels” that have been linked into an edge (sub-)string.

Fitting a line requires orthogonal regression.See B14, or the variant here ...

Find

min E(a,b, c) =n∑

i=1

(axi + byi + c)2

subject to a2 + b2 = 1

Hence a 2-dof problem.

Page 26: Como Observa Un Robot

C18 2013 26 / 38

From edge elements to lines /ctd

Now min E(a,b, c) = minn∑

i=1

(axi + byi + c)2 requires

∂E/∂c = 0.⇒ 2

∑ni=1 (axi + byi + c) = 0 ⇒ c = −(ax + by) .

Therefore E becomes

E =n∑

i=1

[a(xi − x) + b(yi − y)]2 =[a b

]UT U

[ab

]where the 2nd-moment matrix is

UT U =

[ ∑ni=1 x2

i − nx2 ∑ni=1 xiyi − nxy∑n

i=1 xiyi − nxy∑n

i=1 y2i − ny2

],

Solution a,b given by the unit eigenvector of UT U corresponding tosmaller eigenvalue. (Eigen-decomposition or SVD.)

RMS error given by the eigenvalue itself.

Page 27: Como Observa Un Robot

C18 2013 27 / 38

Example of String and Line Outputs

Page 28: Como Observa Un Robot

C18 2013 28 / 38

Problems using 1D image structure for GeometryComputing edges make the feature map sparse but interpretable.Much of the salient information is retained.

If the camera motion is known, feature matching is a 1D problem, forwhich edges are very well suited (see Lecture 4).

However, matching is much harder whenthe camera motion is unknown: known asthe Aperture problem.

End points are unstable, hence linematching is largely uncertain. (Indeedonly line orientation is useful for detailedgeometrical work.)

In general, matching requires the unambiguity of 2D image featuresor “corners”.

Corners defined as sharp peaks in the 2D autocorrelation oflocal patches in the image.

Page 29: Como Observa Un Robot

C18 2013 29 / 38

2.5 Corner detection: preamble on auto-correlation

Suppose that we are interested incorrelating a (2n + 1)2 pixel patchat (x , y) in Image I with a similarpatch displaced from it by (u, v).

We would write the correlation between the patches as

Cuv (x , y) =n∑

i=−n

n∑j=−n

I(x + i , y + j)I(x + u + i , y + v + j)

As we keep (x , y) fixed, but change (u, v), we build up theauto-correlation surface around (x , y).

Page 30: Como Observa Un Robot

C18 2013 30 / 38

Preamble: Auto-correlationReminder correlation:

Cuv (x , y) =n∑

i=−n

n∑j=−n

I(x + i , y + j)I(x + u + i , y + v + j)

Here are the surfaces around 3 different patches

Plain corner Straight edge 3-way corner

A pixel in a uniform region will have a flat autocorrelation

A pixel on an edge will have a ridge-like autocorrelation, but

A pixel at a corner has a peak.

Page 31: Como Observa Un Robot

C18 2013 31 / 38

Results ...So a simple corner detector might havethe following steps at each pixel

a. Determine the auto-correlationCuv (x , y) around the pixel positionx , y .

b. Find positions x , y where Cuv (x , y) ismaximum in two directions.

c. If Cuv (x , y) > threshold mark as acorner.

There is an expression which can be computed more cheaply thanCuv which gives comparable qualitative results.This is the sum of squared differences

Euv (x , y) =+n∑

i=−n

+n∑j=−n

[I(x + u + i , y + v + j)− I(x + i , y + j)]2

Page 32: Como Observa Un Robot

C18 2013 32 / 38

Harris Corner Detector (Chris Harris, 1987)Earlier we estimated the gradient from the pixels values.Now assume we know the the gradients, and estimate a pixeldifference using a 1st order Taylor expansion ...

I(x + u, y + v)− I(x , y) ≈ u∂I(x , y)

∂x+ v

∂I(x , y)

∂y

So the sum of squared differences can be approximated as

Euv (x , y) =+n∑

i=−n

+n∑j=−n

(I(x + u + i , y + v + j)− I(x + i , y + j))2

≈∑

i

∑j

(u∂I∂x

+ v∂I∂y

)2

=∑

i

∑j

(u2(∂I∂x

)2

+ 2uv∂I∂x

∂I∂y

+ v2(∂I∂y

)2)

Note, ∂I/∂x etc are computed at the relevant (x + i , y + j).

Page 33: Como Observa Un Robot

C18 2013 33 / 38

Harris Corner Detector /ctdThe double sum over i ,j is replaced by a convolution with

W =

1 1 1 1 11 1 1 1 11 1 1 1 11 1 1 1 11 1 1 1 1

Euv (x , y) =∑

i

∑j

(u2(∂I∂x

)2

+ 2uv∂I∂x

∂I∂y

+ v2(∂I∂y

)2

)

= u2(W ∗(∂I∂x

)2

) + 2uv(W ∗ ∂I∂x

∂I∂y

) + v2(W ∗(∂I∂y

)2

)

= ( u v )

(p rr q

)(uv

)where

p(x , y) = W ∗ (∂I/∂x)2

q(x , y) = W ∗ (∂I/∂y)2

r(x , y) = W ∗ (∂I/∂x)(∂I/∂y)

Page 34: Como Observa Un Robot

C18 2013 34 / 38

Harris Corner Detector /ctd

We can introduce smoothing by replacing W with a Gaussian G, sothat

Euv (x , y) = ( u v )

(p rr q

)(uv

)where

p(x , y) = G ∗ (∂I/∂x)2

q(x , y) = G ∗ (∂I/∂y)2

r(x , y) = G ∗ (∂I/∂x)(∂I/∂y)

The quantities p, q and r computed at each (x , y) define the shape ofthe auto-correlation function Euv (x , y) at (x , y)

Page 35: Como Observa Un Robot

C18 2013 35 / 38

Harris Corner Detector /ctdRecall that

Euv (x , y) = ( u v )

(p rr q

)(uv

)

Now (u, v) defines a direction, so an estimate of Euv a unit distanceaway from (x , y) along a vector direction n is then

E ≈ n>Snn>n

where S =

(p rr q

)

Now recall (eg, from A1 Eng Comp notes) that if λ1 and λ2 are thelarger and smaller eigenvalues of S, respectively, that

λ2 ≤n>Snn>n

≤ λ1 ⇒ λ2 ≤ E ≤ λ1

Page 36: Como Observa Un Robot

C18 2013 36 / 38

/ctd Harris Corner DetectorThis allows a classification of image structure:• Both λ1, λ2 ≈ 0 — the autocorrelation is small in all directions:

image must be flat.• λ1 0, λ2 ≈ 0 — the autocorrelation is high in just one direction⇒a 1D edge

• λ1 0, λ2 0 — the autocorrelation is high in all directions⇒a2D corner.

Harris’ original “interest” score was later modified by Harris andStephens

SHarris =λ1λ2

λ1 + λ2SHS = λ1λ2 − α

(λ1 + λ2)2

4

α is a positive constant 0 ≤ α ≤ 1 which decreases the response toedges, sometimes called the “edge-phobia”.With α > 0 edges give negative scores, and corners positive. Scores close to zeroindicate flat surfaces or “T” features which are intermediary between edges andcorners. The size of α determines how much edges are penalised.

Page 37: Como Observa Un Robot

C18 2013 37 / 38

Examples

Page 38: Como Observa Un Robot

C18 2013 38 / 38

Summary

In this lecture we have consideredAs enabling techniques, we have described convolution andcorrelation.

Described how to smooth imagery and detect gradients

Described how to recover 1D structure (edges) in an imageIn particular, have developed the Canny Edge detector.

Considered how to join edgels into strings, and to fit lines.

Described how to recover 2D structure (corners) in an image.Have developed the Harris cor ner detector