motion from image and inertial measurements dennis strelow honeywell advanced technology lab

Motion from image and inertial measurements

Dennis Strelow

Honeywell Advanced Technology Lab

Dennis Strelow -- Motion estimation from image and inertial measurements -- March 22. 2006 2

On the web

Related materials:

these slides

related papers

movies

VRML models

at:

http://www.dennis-strelow.com/umn


Introduction (1)

From an image sequence, we can recover:

6 degree of freedom (DOF) camera motion

without knowledge of the camera’s surroundings

without GPS


Introduction (2)

Fitzgibbon


Introduction (3)

Potential applications include:

modeling from video

Yuji Uchida


Introduction (4)

micro air vehicles (MAVs)

AeroVironment Black Widow AeroVironment Microbat


Introduction (5)

rover navigation

Hyperion Nister, et al.


Introduction (6)

search and rescue robots

Rhex (movies: http://ai.eecs.umich.edu/Rhex/Movies)


Introduction (7)

NASA Personal Satellite Assistant (PSA)


Introduction (8)

For these problems, we want:

6 DOF motion

in unknown environments

without GPS or other absolute positioning


Introduction (8)


6 DOF motion



using small, light, and cheap sensors


Introduction (8)


6 DOF motion



using small, light, and cheap sensors

over the long term


Introduction (9)

Long-term motion estimation:

absolute distance or time is long

only a small fraction of the scene is visible at any one time


Introduction (10)

given these requirements, cameras are promising sensors…

…and many algorithms for motion from images already exist


Introduction (11)

But, where are the systems for estimating the motion of:

over the long term?


Introduction (12)

…and for automatically modeling

rooms

buildings

cities

from a handheld camera?


Introduction (13)

Motion from images suffers from some long-standing difficulties

This work attacks these problems by…

exploiting image and inertial measurements

robust image feature tracking

recognizing previously mapped locations

exploiting omnidirectional images


Outline

Motion from images

refresher

bundle adjustment

difficulties


Robust image feature tracking

Long-term motion estimation

Conclusion


Motion from images: refresher (1)

A two-step process is common:

sparse feature tracking

estimation



A two-step process is common:

sparse feature tracking

estimation

Sparse feature tracking:

inputs: raw images

outputs: projections



Template matching:

correlation tracking

Lucas-Kanade (Lucas and Kanade, 1981)

Extraction and matching:

Harris features (Harris, 1992)

Scale Invariant Feature Transform (SIFT) keypoints (Lowe, 2004)



The second step is estimation:

inputs:

projections

outputs:

6 DOF camera position at the time of each image

3D position of each tracked point



bundle adjustment (various, 1950’s)

Kalman filtering (Broida, Chandrashekhar, and Chellappa, 1990)

variable state dimension filter (VSDF) (McLauchlan, 1996)

two- and three-frame methods(Hartley and Zisserman, 2000; Nister, et al. 2004)


Motion from images: bundle adjustment (1)

From tracking, we have the image locations xij for each point j and each image i



Suppose we also have estimates of:

the camera rotation ρi and translation ti at time of each image

3D point positions Xj of each tracked point

Then, we can compute reprojections:

))(( iji tXR



So, minimize:

with respect to all the ρi, ti, Xj

)),)(((,

image ijijji

i xtXRDE


Motion from images: difficulties (1)

Estimation step can be very sensitive to…

incorrect or insufficient image feature tracking

camera modeling and calibration errors

outlier detection thresholds

sequences with degenerate camera motions



Iterative batch methods have poor convergence or may fail to converge if:

observations are missing

the initial estimate is poor



Recursive methods suffer from:

poor prior assumptions on the motion

poor approximations in state error modeling



Resulting errors are:

gross local errors

long term drift



151 images, 23 pointsmanually corrected Lucas-Kanade



squares: ground truth points dash-dotted line: accurate estimate solid line: image-only, bundle adjustment estimate


Outline

Motion from images


inertial sensors

algorithms and results

related work



Conclusion


Motion from image and inertial measurements: inertial sensors (1)

inertial sensors can be integrated to estimate six degree of freedom motion



But many applications require small, light, and cheap sensors



Integrating the outputs of these low grade sensors will produce drifting motion because of:

noise

unmodeled nonlinearities



And, we can’t even integrate until we can separate the effects of…

rotation ρ

gravity g

acceleration a

slowly changing bias ba

noise n

…in the accelerometer measurements



Image and inertial measurements are highly complementary

With inertial measurements we can:

decrease sensitivity in image-only estimates

establish two rotation angles without drift

establish the global scale



Image and inertial measurements are highly complementary

With inertial measurements we can:

decrease sensitivity in image-only estimates

establish two rotation angles without drift

establish the global scale

…even with our low-grade sensors



With image measurements, we can:

reduce the drift in integrating inertial data

distinguish between… rotation ρ

gravity g

acceleration a

bias ba

noise n

…in accelerometer measurements


Motion from image and inertial measurements: algorithms and results (1)

This work has developed both:

batch

recursive

algorithms for motion from image and inertial measurements



Gyro measurements:

ω’, ω: measured and actual angular velocity

bω: gyro bias

n: gaussian noise

nb '



Accelerometer measurements:

ρ: rotation

a’, a: measured and actual acceleration

g: gravity vector

ba: accelerometer bias

n: gaussian noise

nbgaRa' aT )((



batch algorithm minimizes a combined error:

inertialimagecombined EEE



image term Eimage is the same as before



inertial error term Einertial is:

1

1n,translatio

1

1velocity,

1

1rotation,

ntranslatiovelocityrotationinertial

f

ii

f

ii

f

ii

E

E

E

EEEE




1

1n,translatio

1

1velocity,

1

1rotation,


f

ii

f

ii

f

ii

E

E

E

EEEE



,...))(,( 1n,translatio itii tItDE

timeτi-1 (time of image i - 1)

ti-1

ti

I(ti-1, …)

τi (time of image i)

tran

slat

ion

( : translation estimate for image i – 1)

( : translation estimate for image i)

( : translation integrated from previous estimate)



timeτ0

tran

slat

ion

τ1 τ2 τ5 τ3 τ4 τf-3 τf-2 τf-1

1

1n,translationtranslatio

f

iiEE



1

1n,translatio

1

1velocity,

1

1rotation,


f

ii

f

ii

f

ii

E

E

E

EEEE



It(τi-1, τi ,…, ti-1) depends on:

τi-1, τi (known)

all inertial measurements for times τi-1< τ < τi (known)

ρi-1, ti-1

g

bω, ba

camera linear velocities: vi



dash-dotted line: batch estimate from image and inertial solid line: image-only, bundle adjustment estimate squares: ground truth points



IEKF for the same sensors, unknowns dash-dotted line: batch estimate solid line: IEKF estimate



Difficulties with IEKF for this application:

prior assumptions about motion smoothness

cannot model relative error between adjacent camera positions

So, converting the batch algorithm into a variable state dimension filter (VSDF) is a promising future direction



IEKF assumptions on motion smoothness dash-dotted line: batch estimate solid line: IEKF estimate

right: IEKF propagation variances too strict

left: IEKF propagation variances just right



Recap:

image, gyro, and accelerometer measurements

batch algorithm

recursive algorithm

experiments

evaluate batch and recursive algorithms

establish basic facts about motion from image and inertial measurements


Outline

Motion from images



Lucas-Kanade and real sequences

The “smalls” tracker


Conclusion


Robust image feature tracking: Lucas-Kanade and real sequences (1)

Combining image and inertial measurements improves our situation, but…

we still need accurate feature tracking tracking

some sequences do not come with inertial measurements



better feature tracking for improved 6 DOF motion estimation

remaining results will be image-only



Lucas-Kanade has been the go-to feature tracker for shape-from-motion

minimizes a correlation-like matching error

using general minimization

evaluates the matching error at only a few locations

subpixel resolution



Additional heuristics used to apply Lucas-Kanade to shape-from-motion:

task: heuristic:

choose features to track high image texture

identify mistracked, occluded, no-longer-visible

convergence, matching error

handle large motions image pyramid



But Lucas-Kanade performs poorly on many real sequences…


Robust image feature tracking: the “smalls” tracker (1)

smalls is a new feature tracker targeted at 6 DOF motion estimation

exploits the rigid scene assumption

eliminates the heuristics normally used with Lucas-Kanade

SIFT is an enabling technology here



First step: epipolar geometry estimation

use SIFT to establish matches between the two images

get the 6 DOF camera motion between the two images

get the epipolar geometry relating the two images



Second step: track along epipolar lines

use nearby SIFT matches to get initial position on epipolar line

exploits the rigid scene assumption

eliminates heuristic: pyramid



Third step: prune features

geometrically inconsistent features are marked as mistracked and removed

clumped features are pruned

eliminates heuristic: detecting mistracked features based on convergence, error



Fourth step: extract new features

spatial image coverage is the main criterion

required texture is minimal when tracking is restricted to the epipolar lines

eliminates heuristic: extracting only textured features



left: odometry only right: images only

average error: 1.74 m

maximum error: 5.14 m

total distance: 230 m



Recap:

exploits the rigid scene and eliminates heuristics

allows hands-free tracking for real sequences

can still be defeated by textureless areas or repetitive texture


Outline

Motion from images




proof of concept system

experiment

Conclusion


Long-term motion estimation: proof of concept system (1)

Image-based motion estimates from any system will drift:

if the features we see are always changing

given sufficient time

if we don’t recognize when we’ve revisited a location



To limit drift:

recognize when we’ve returned to a previous location

exploit the return

A proof of concept system demonstrates these capabilities



“smalls” tracker state: 2D feature history for images in I

variable state dimension filter (VSDF) state for images in I: 6 DOF camera positions, covariances for images in I 3D positions for features visible in I

SIFT keypoints for image in

system state S

image indices: I = {i1, …, in}



0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}


rollback


0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}

non-rollback States:


rollback


0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}

8

non-rollback States:


rollback


0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2}

8

{0, 1, 2, 3, 8}

non-rollback pruned States:


rollback


0 1 2 3 4 5 6 7 8

8 9 10 11

11 12 13 14

14 15 16 17

17 18 19 20

{0, …, 6, 11, 12, 17, …, 20}




When to “roll back”?

examine the camera covariances for the current state and the candidate rollback state

check the number of SIFT matches

extend from the candidate state

examine the camera covariances for the current state and the resulting extended state


Long-term motion estimation: experiment (1)

CMU FRC highbay views; 945 images total



CMU FRC highbay



CMU FRC highbay

(first forward pass: images 0-213)



CMU FRC highbay




CMU FRC highbay

(first backward pass: images 214-380)


rollback


0 1 2 3 4 5 6 7 8

8 9 10 11

11 12 13 14

14 15 16 17

17 18 19 20


normally, the system produces a general tree of states



…0 1 2 3 4 5 6 7

13 14 15 14

16 17 18 17

non-rollback rollback pruned States:

for this example, the “rollback” states are restricted to the first forward pass

8 9

10 11 12 14

14

213



movie…bottom half is smalls output:



movie…top half is motion estimates:


Outline

Motion from images




Conclusion

remaining issues


Conclusion: remaining issues

all: system is experimental, not optimized for speed

image and inertial: VSDF

“smalls”: integration of gyro, more robustness to poor texture needed

long-term: “roll back” space, computation grow with sequence length


Thanks!

Related materials:

these slides

related papers

movies

VRML models

at:

http://www.dennis-strelow.com/umn


Motion from omnidirectional images (1)



left: non-rigid camera right: rigid camera

squares: ground truth points solid: image-only estimates

dash-dotted: image-and-inertial estimates



In this experiment:

omni images

conventional images + inertial

have roughly the same advantages

But in general:

inertial has some advantages that omni images alone can’t produce

omni images can be harder to use

motion from image and inertial measurements dennis strelow honeywell advanced technology lab

Documents

motion estimation

dof motion

gps slide

time slide

image sequence

edurhexmovies slide

fitzgibbon slide

absolute positioning