cs 395/495-26: spring 2003 ibmr: week 4a from chapter 3: point correspondence and direct linear...

CS 395/495-26: Spring 2003CS 395/495-26: Spring 2003

IBMR: Week 4AIBMR: Week 4A

from Chapter 3:from Chapter 3:

Point Correspondence Point Correspondence and and

Direct Linear Transform (DLT) Direct Linear Transform (DLT)

Jack TumblinJack Tumblin

[email protected]@cs.northwestern.edu

Proj_A WorkaroundProj_A Workaround• Programming ‘Pick & Drag’ is messy!Programming ‘Pick & Drag’ is messy!• KeyDown, KeyUp is easy (see ProjA_06)KeyDown, KeyUp is easy (see ProjA_06)

We have only a few points to drag;We have only a few points to drag;– Use number keys to select points:Use number keys to select points:

• ‘‘1’ for 11’ for 1stst point, ‘2’ for 2 point, ‘2’ for 2ndnd point, etc. point, etc.• ‘‘SHIFT 1’ for 1SHIFT 1’ for 1stst line, ‘SHIFT 2’ for 2 line, ‘SHIFT 2’ for 2ndnd line... line...

– Use arrow keys to change point / line coords:Use arrow keys to change point / line coords:• left/right for x1 value, left/right for x1 value, • up/down for x2 value, up/down for x2 value, • SHIFT up/down for x3 value, etc.SHIFT up/down for x3 value, etc.

Most Important IdeasMost Important Ideas

SVD solves Null-Space problems SVD solves Null-Space problems Ax = 0Ax = 0 (‘flatten, (‘flatten,

stack, find null space’ to solve) stack, find null space’ to solve) – MANY problems can be written this wayMANY problems can be written this way– Expandable! Expandable!

--more data points--more data points tall, ‘overconstrained’ A matrix tall, ‘overconstrained’ A matrix --SVD finds optimal ‘least squares’ solution.--SVD finds optimal ‘least squares’ solution.

Robustness: bane of all inverse methodsRobustness: bane of all inverse methodsAccuracy vs. # of measurements Accuracy vs. # of measurements (more isn’t always better! Outliers...)(more isn’t always better! Outliers...)

Data-rich images: Data-rich images: quantityquantity easier than easier than qualityquality moremore is easier than is easier than betterbetter) )

?What links measurement errors?What links measurement errorsHH errors? errors??How can more measurements reduce error??How can more measurements reduce error?

Recall: Rectification Undo parts of HRecall: Rectification Undo parts of H

x’ = H xx’ = H x where where H = HH = HS S HHA A HHPP

GOAL: GOAL: Put world plane x’ into view plane xPut world plane x’ into view plane x– Affine Rect.; Affine Rect.; (find only (find only HHPP (2DOF)); (2DOF)); – Metric Rect.;Metric Rect.; (find (find HHA A and and HHP P (6DOF));(6DOF));– Full Rect.;Full Rect.; (find all: (find all: HHSSHHAAHHPP (8DOF)); (8DOF));

METHODS:METHODS:1.1. Affine: ‘Vanishing Point’, Horizon line methodsAffine: ‘Vanishing Point’, Horizon line methods2.2. Metric: Conics & Circular Points Metric: Conics & Circular Points (!?!?)(!?!?)3.3. Full: 4-point correspondence Full: 4-point correspondence

Finding H from Image Space Finding H from Image Space

– Affine Rect; Affine Rect; (find only (find only HHPP (2DOF)); (2DOF));

2 parallel line pairs2 parallel line pairs2 points on horizon 2 points on horizon lineline

– Metric Rect.Metric Rect. (find (find HHA A and and HHP P (4DOF));(4DOF));

5 perpendicular line pairs5 perpendicular line pairs C*C*’ ’

– Full Rect.;Full Rect.; (find all: (find all: HHSSHHAAHHPP (8DOF)); (8DOF));

4 (or more point pairs) 4 (or more point pairs) all all HH coeffs coeffs

2D world space 2D world space

2D image 2D image space space

HHS S HHA A HHPP

LL MMmmLL

x’x’

Apply the Apply the 3x3 matrix 3x3 matrix HH

x’ = Hx x’ = Hx

Recall: Projective Transform Recall: Projective Transform HH

xx33

xx22

xx11

xx

yy

x’x’

y’y’

(x(x/w/w,y,y/w/w)) xx

(x’(x’/w’/w’,y’,y’/w’/w’))

Homog. coordsHomog. coords x’ =x’ = [x’,y’,[x’,y’,w’w’]]TT

w’=1w’=1

2D image 2D image (x’,y’)(x’,y’)

2D image 2D image (x,y)(x,y)

Homog. coords Homog. coords

[x,y,[x,y,ww]]TT = x = x w=1w=1

Recall: Projective Transform: Recall: Projective Transform: Hx = x’Hx = x’

xx11

xx22

xx33

hh11 11 h h1212 h h1313

hh2121 h h2222 h h2121

hh3131 h h32 32 h h3333

= = x’x’11

x’x’22

x’x’33

H defines point pairs (x,x’) H defines point pairs (x,x’) (correspondences)(correspondences)

x’ = =x’ = =

y’ = =y’ = =

= = xx

yy

ww

Input (or output) Input (or output) image is on image is on central planecentral plane

xx11

xx22

xx33

hh21 21 x + hx + h22 22 y + hy + h2121

hh31 31 x + hx + h32 32 y + hy + h3333

x’x’11

x’x’33

x’x’22

x’x’33

Write RWrite R22 expressions; expressions;(for each x,x’ pair)(for each x,x’ pair)

hh11 11 x + hx + h12 12 y + hy + h1313

hh31 31 x + hx + h32 32 y + hy + h3333

x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))Rearrange, solve as Rearrange, solve as a matrix problem…a matrix problem…

let w=1let w=1

Naïve Method: (pg 13-14)Naïve Method: (pg 13-14)

4 point correspondence:4 point correspondence:Book shows:Book shows:

Rearrange: Rearrange: knownknown vector (dot) vector (dot) unknownunknown vector vector

stack, solve for null space…stack, solve for null space…

But this assumes But this assumes xx11=x, x=x, x22=y, =y, xx33=w=1=w=1,,

There is a better way (DLT)…There is a better way (DLT)…


y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

hh1111

hh1212

hh13 13

hh2121 hh2222

hh2323

hh3131 hh3232

hh3333

[x y 1 0 0 0 -x’x -x’y -x’] = 0[x y 1 0 0 0 -x’x -x’y -x’] = 0[0 0 0 x y 1 -y’x -y’y -y’][0 0 0 x y 1 -y’x -y’y -y’]

Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)

• Naïve method: Solves Hx – x’ = 0Naïve method: Solves Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 in measurement)=1 in measurement)

• compare:compare:

• include the w, w’ terms:include the w, w’ terms:– look what happens when w’look what happens when w’0!0!

[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ]

hh1111

hh1212

hh1313

hh2121

hh2222

hh2323

hh3131

hh3232

hh3333

[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]


• Naïve method: Solves Hx – x’ = 0Naïve method: Solves Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 in measurement)=1 in measurement)


• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– (a different identity: x’ (a different identity: x’ x’ = 0 x’ = 0))

– Accepts any scale (xAccepts any scale (x33=w=w1 OK, xx1 OK, xx33))


hh1111

hh1212

hh1313

hh2121

hh2222

hh2323

hh3131

hh3232

hh3333



• Naïve method: Solve Hx – x’ = 0Naïve method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)


• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts Accepts anyany scale (x scale (x33=w=w1, x1, x33’=w’’=w’1)1)


hh11

hh22

hh33

[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w][w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

[[ 00TT --ww’’i i xxiiTT yy’’I I xxii

TT ]][[ ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT ]]


• DLT method: Solve H DLT method: Solve H x = 0x = 0– Accepts any scale, any point (xAccepts any scale, any point (x33=w=w1 is OK)1 is OK)

– ‘‘Pure’ and Compatible -- Pure’ and Compatible -- PP22 terms only, no Rterms only, no R22

– MuchMuch better suited to error measurements. better suited to error measurements.

• Subtlety:Subtlety:– Won’t divide-by-zero if Won’t divide-by-zero if w=0w=0 or or hh3333=0=0 (it happens!)(it happens!)

– has a 3has a 3rdrd row; it is row; it is notnot degenerate if w’=0 degenerate if w’=0– OK to use it…OK to use it…

(Solve 8x12)(Solve 8x12) hh11

hh22

hh33

00TT - -ww’’i i xxiiTT yy’’I I xxii

TT ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT =0 =0

--yy’’i i xxiiTT - -xx’’i i xxii

TT 00TT

Deceptive ‘Robustness’Deceptive ‘Robustness’

• Suppose we have 4 pt-correspondences:Suppose we have 4 pt-correspondences:– Use DLT to write 8x9 matrix Use DLT to write 8x9 matrix AA: : Ah=0Ah=0– Solve for Solve for h h null space. null space. ALWAYSALWAYS gives gives HH matrix matrix

• But what if points are bad / fictional?But what if points are bad / fictional?– 3 collinear input pts, crooked out: 3 collinear input pts, crooked out: IMPOSSIBLE!IMPOSSIBLE!

– Yet we get an Yet we get an HH solution! Why? solution! Why?

• AA matrix rank is 7 or 6 matrix rank is 7 or 6 rank 1 rank 1 HH result(s) result(s)– ‘‘Null Space’ of Null Space’ of AA may contain >1 may contain >1 HH solution! solution!

– ‘‘Degenerate’ Degenerate’ HH solution of form solution of form aaHH11 + + bbHH22……

• Answer: SVD ranks Answer: SVD ranks AA; ; rejectreject bad point sets. bad point sets.

Actual ‘Robustness’Actual ‘Robustness’

• Vectorizing Vectorizing (‘Flatten, Stack, Null Space’ method) (‘Flatten, Stack, Null Space’ method)

works for almost ANY input! works for almost ANY input! (Points, lines, planes, ||, (Points, lines, planes, ||, , conics, quadrics, , conics, quadrics, cross-ratios, vanishing points, twisted cubics…cross-ratios, vanishing points, twisted cubics…

• Use DLT formulation: (Hx Use DLT formulation: (Hx x’) = 0 x’) = 0

• Rearrange as dot product: Rearrange as dot product: (known)(known)··(unknown) = 0(unknown) = 0

• Be careful to have ENOUGH constraintsBe careful to have ENOUGH constraints tricky when you mix types: points, lines,… tricky when you mix types: points, lines,… (pg 75)(pg 75)

measured inputmeasured inputtransformedtransformed

measured outputmeasured output

= 0= 0

Adding More MeasurementsAdding More Measurements

How can we use How can we use >4>4 point correspondences? point correspondences?• Easy:Easy:

– Add more rows to our 8x9 matrix A: Add more rows to our 8x9 matrix A: AA··h = 0h = 0– Use SVD to find Null space Use SVD to find Null space (Always gives an answer!)(Always gives an answer!)

– Result:Result: ‘Least squares’ solution; error= ‘Least squares’ solution; error=AA··h = h = • minimizes || minimizes || A hA h || ||22 = = ii 22

i i

wherewhere,,ii is error for is error for ii-th pt. -th pt. correspondence: correspondence:

i i = || Hx= || Hxii x xii’ ||’ ||2 2 = = || (2 rows of A)*h |||| (2 rows of A)*h ||2 2 = ‘algebraic distance’= ‘algebraic distance’

– ‘‘Algebraic Distance’ ?Algebraic Distance’ ? No geometric meaning! No geometric meaning!


• 2D‘Algebraic Distance’ ?2D‘Algebraic Distance’ ? No geometric meaning! No geometric meaning!

• 2D ‘Geometric Distance’ d2D ‘Geometric Distance’ d(a,b)(a,b)22 is Better: is Better: measurable length in input or output spacemeasurable length in input or output space

if a = (aif a = (a11 a a22 a a33) and b = (b) and b = (b11 b b2 2 b b33), then define), then define

Turns out that:Turns out that:

( - )( - )22 + + ( - )( - )22 dd(a,b)(a,b)22 = = aa11

aa33

bb22

bb33

bb11

bb33

aa22

aa33

dd(a,b)(a,b)22 = d = dalgebraicalgebraic(a,b)(a,b)aa33··bb33 (Not very surprising)(Not very surprising)


Overall Strategy:Overall Strategy:• Overconstrain the answer Overconstrain the answer HH

– Collect extra measurements (>4 point pairs, etc. …)Collect extra measurements (>4 point pairs, etc. …)– expect errors; call them ‘estimates’ xexpect errors; call them ‘estimates’ x

• Compute a 1Compute a 1stst solution (probably by SVD) solution (probably by SVD)

• Compute error Compute error d(Hx, x’)d(Hx, x’)22, and use this to…, and use this to…

• ‘‘Tweak’ answer Tweak’ answer HH andand estimates estimates xx

• Compute new answerCompute new answer

• Stop when error < useful thresholdStop when error < useful threshold

^̂

^̂

^̂

^̂

Using EstimatesUsing Estimates

• Simplest: ‘one image’ transfer method:Simplest: ‘one image’ transfer method:– Assume inputs are a perfect test pattern:Assume inputs are a perfect test pattern:

onlyonly output output pts are in error pts are in error– Adjust output estimates x’ until d(Adjust output estimates x’ until d(HHx, x’)x, x’)22 00

(note we re-compute H as x’ changes)(note we re-compute H as x’ changes)

• Better: ‘Symmetric’ transfer method:Better: ‘Symmetric’ transfer method:– Assume BOTH inputs and outputs have error.Assume BOTH inputs and outputs have error.– Adjust BOTH input and output ests x x’Adjust BOTH input and output ests x x’

(note we re-compute H as x, x’ change)(note we re-compute H as x, x’ change)– Stop when d(Hx,x’)Stop when d(Hx,x’)22 + d(H + d(H-1-1x’, x)x’, x)22 00

^̂ ^̂

^̂ ^̂

^ ^ ^ ^ ^ ^ ^ ^

ENDEND

DLT: Four-Point CorrespondenceDLT: Four-Point Correspondence

• 4 point correspondence:4 point correspondence:– Book shows:Book shows:

– Rearrange: Rearrange: knownknown vector (dot) vector (dot) unknownunknown vector vector

– stack, solve for null space…stack, solve for null space…


y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

hh1111

hh1212

hh1313

hh2121

hh2222

hh2323

hh3131

hh3232

hh3333

[x y 1 0 0 0 -x’x -x’y -x’] = 0[x y 1 0 0 0 -x’x -x’y -x’] = 0[0 0 0 x y 1 -y’x -y’y -y’][0 0 0 x y 1 -y’x -y’y -y’]


• Project 2 method: Solve Hx – x’ = 0Project 2 method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 in measurement)=1 in measurement)


• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts any scale (xAccepts any scale (x33=w=w1 OK, xx1 OK, xx33))


hh1111

hh1212

hh1313

hh2121

hh2222

hh2323

hh3131

hh3232

hh3333



• Project 2 method: Solve Hx – x’ = 0Project 2 method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)


• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts Accepts anyany scale (x scale (x33=w=w1, x1, x33’=w’’=w’1)1)


hh11

hh22

hh33

[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w][w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

[[ 00TT --ww’’i i xxiiTT yy’’I I xxii

TT ]][[ ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT ]]


• DLT method: Solve H DLT method: Solve H x = 0x = 0– Accepts any scale, any point (xAccepts any scale, any point (x33=w=w1 is OK)1 is OK)

– ‘‘Pure’, Compatible -- Pure’, Compatible -- PP22 terms onlyterms only– Much better suited to error measurements.Much better suited to error measurements.

• Subtlety:Subtlety:– Won’t divide-by-zero if Won’t divide-by-zero if w=0w=0 or or hh3333=0=0 (it happens!)(it happens!)

– has a 3has a 3rdrd row; it is not degenerate if w’=0 row; it is not degenerate if w’=0– OK to use it…OK to use it…

(Solve 8x12)(Solve 8x12) hh11

hh22

hh33

00TT - -ww’’i i xxiiTT yy’’I I xxii

TT ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT =0 =0

--yy’’i i xxiiTT - -xx’’i i xxii

TT 00TT

Deceptive ‘Robustness’Deceptive ‘Robustness’

• Suppose we have 4 pt-correspondences:Suppose we have 4 pt-correspondences:– Use DLT to write 8x9 (or 12x9) matrix Use DLT to write 8x9 (or 12x9) matrix AA: : Ah=0Ah=0– Solve for Solve for h h null space. null space. ALWAYSALWAYS gives gives HH matrix matrix

• But what if points are bad / fictional?But what if points are bad / fictional?– 3 collinear input pts, crooked out: 3 collinear input pts, crooked out: IMPOSSIBLE!IMPOSSIBLE!

– Yet we get an Yet we get an HH solution! Why? solution! Why?

• AA matrix rank is 7 or 6 matrix rank is 7 or 6 rank 1 rank 1 HH result(s) result(s)– ‘‘Null Space’ of Null Space’ of AA may contain >1 may contain >1 HH solution! solution!

– ‘‘Degenerate’ Degenerate’ HH solution of form solution of form aaHH11 + + bbHH22……

• Answer: SVD ranks Answer: SVD ranks AA; ; rejectreject bad point sets. bad point sets.

Actual ‘Robustness’Actual ‘Robustness’

• Vectorizing Vectorizing (‘Flatten, Stack, Null Space’ method) (‘Flatten, Stack, Null Space’ method)

works for almost ANY input! works for almost ANY input! (Points, lines, planes, ||, (Points, lines, planes, ||, , conics, quadrics, , conics, quadrics, cross-ratios, vanishing points, twisted cubics…cross-ratios, vanishing points, twisted cubics…

• Use DLT formulation: (Hx Use DLT formulation: (Hx x’) = 0 x’) = 0

• Rearrange as dot product: Rearrange as dot product: (known)(known)··(unknown) = 0(unknown) = 0

• Be careful to have ENOUGH constraintsBe careful to have ENOUGH constraints tricky when you mix types: points, lines,… tricky when you mix types: points, lines,… (pg 75)(pg 75)

measured inputmeasured inputtransformedtransformed

measured outputmeasured output

= 0= 0

cs 395/495-26: spring 2003 ibmr: week 4a from chapter 3: point correspondence and direct linear...

Documents