motion from image and inertial measurements

Motion from image and inertial measurements

Dennis Strelow

Honeywell Advanced Technology Lab

Dennis Strelow -- Motion estimation from image and inertial measurements -- March 22. 2006 2

On the web

Related materials:

these slides

related papers

movies

VRML models

http://www.dennis-strelow.com/umn

Introduction (1)

From an image sequence, we can recover:

6 degree of freedom (DOF) camera motion

without knowledge of the camera’s surroundings

without GPS

Introduction (2)

Fitzgibbon

Introduction (3)

Potential applications include:

modeling from video

Yuji Uchida

Introduction (4)

micro air vehicles (MAVs)

AeroVironment Black Widow AeroVironment Microbat

Introduction (5)

rover navigation

Hyperion Nister, et al.

Introduction (6)

search and rescue robots

Rhex (movies: http://ai.eecs.umich.edu/Rhex/Movies)

Introduction (7)

NASA Personal Satellite Assistant (PSA)

Introduction (8)

For these problems, we want:

6 DOF motion

in unknown environments

without GPS or other absolute positioning

Introduction (8)

6 DOF motion

using small, light, and cheap sensors

Introduction (8)

6 DOF motion

using small, light, and cheap sensors

over the long term

Introduction (9)

Long-term motion estimation:

absolute distance or time is long

only a small fraction of the scene is visible at any one time

Introduction (10)

given these requirements, cameras are promising sensors…

…and many algorithms for motion from images already exist

Introduction (11)

But, where are the systems for estimating the motion of:

over the long term?

Introduction (12)

…and for automatically modeling

buildings

cities

from a handheld camera?

Introduction (13)

Motion from images suffers from some long-standing difficulties

This work attacks these problems by…

exploiting image and inertial measurements

robust image feature tracking

recognizing previously mapped locations

exploiting omnidirectional images

Outline

Motion from images

refresher

bundle adjustment

difficulties

Robust image feature tracking

Long-term motion estimation

Conclusion

Motion from images: refresher (1)

A two-step process is common:

sparse feature tracking

estimation

A two-step process is common:

sparse feature tracking

estimation

Sparse feature tracking:

inputs: raw images

outputs: projections

Template matching:

correlation tracking

Lucas-Kanade (Lucas and Kanade, 1981)

Extraction and matching:

Harris features (Harris, 1992)

Scale Invariant Feature Transform (SIFT) keypoints (Lowe, 2004)

The second step is estimation:

inputs:

projections

outputs:

6 DOF camera position at the time of each image

3D position of each tracked point

bundle adjustment (various, 1950’s)

Kalman filtering (Broida, Chandrashekhar, and Chellappa, 1990)

variable state dimension filter (VSDF) (McLauchlan, 1996)

two- and three-frame methods(Hartley and Zisserman, 2000; Nister, et al. 2004)

Motion from images: bundle adjustment (1)

From tracking, we have the image locations xij for each point j and each image i

Suppose we also have estimates of:

the camera rotation ρi and translation ti at time of each image

3D point positions Xj of each tracked point

Then, we can compute reprojections:

))(( iji tXR

So, minimize:

with respect to all the ρi, ti, Xj

)),)(((,

image ijijji

i xtXRDE

So, minimize:

with respect to all the ρi, ti, Xj

)),)(((,

image ijijji

i xtXRDE

Motion from images: difficulties (1)

Estimation step can be very sensitive to…

incorrect or insufficient image feature tracking

camera modeling and calibration errors

outlier detection thresholds

sequences with degenerate camera motions

Iterative batch methods have poor convergence or may fail to converge if:

observations are missing

the initial estimate is poor

Recursive methods suffer from:

poor prior assumptions on the motion

poor approximations in state error modeling

Resulting errors are:

gross local errors

long term drift

151 images, 23 pointsmanually corrected Lucas-Kanade

squares: ground truth points dash-dotted line: accurate estimate solid line: image-only, bundle adjustment estimate

Outline

Motion from images

inertial sensors

algorithms and results

related work

Conclusion

Motion from image and inertial measurements: inertial sensors (1)

inertial sensors can be integrated to estimate six degree of freedom motion

But many applications require small, light, and cheap sensors

Integrating the outputs of these low grade sensors will produce drifting motion because of:

unmodeled nonlinearities

And, we can’t even integrate until we can separate the effects of…

rotation ρ

gravity g

acceleration a

slowly changing bias ba

noise n

…in the accelerometer measurements

Image and inertial measurements are highly complementary

With inertial measurements we can:

decrease sensitivity in image-only estimates

establish two rotation angles without drift

establish the global scale

Image and inertial measurements are highly complementary

With inertial measurements we can:

decrease sensitivity in image-only estimates

establish two rotation angles without drift

establish the global scale

…even with our low-grade sensors

With image measurements, we can:

reduce the drift in integrating inertial data

distinguish between… rotation ρ

gravity g

acceleration a

bias ba

noise n

…in accelerometer measurements

Motion from image and inertial measurements: algorithms and results (1)

This work has developed both:

recursive

algorithms for motion from image and inertial measurements

Gyro measurements:

ω’, ω: measured and actual angular velocity

bω: gyro bias

n: gaussian noise

Accelerometer measurements:

ρ: rotation

a’, a: measured and actual acceleration

g: gravity vector

ba: accelerometer bias

n: gaussian noise

nbgaRa' aT )((

batch algorithm minimizes a combined error:

inertialimagecombined EEE

image term Eimage is the same as before

inertial error term Einertial is:

1n,translatio

1velocity,

1rotation,

ntranslatiovelocityrotationinertial

1n,translatio

1velocity,

1rotation,

1n,translatio

1velocity,

1rotation,

,...))(,( 1n,translatio itii tItDE

timeτi-1 (time of image i - 1)

I(ti-1, …)

τi (time of image i)

( : translation estimate for image i – 1)

( : translation estimate for image i)

( : translation integrated from previous estimate)

timeτ0

τ1 τ2 τ5 τ3 τ4 τf-3 τf-2 τf-1

1n,translationtranslatio

1n,translatio

1velocity,

1rotation,

It(τi-1, τi ,…, ti-1) depends on:

τi-1, τi (known)

all inertial measurements for times τi-1< τ < τi (known)

ρi-1, ti-1

bω, ba

camera linear velocities: vi

dash-dotted line: batch estimate from image and inertial solid line: image-only, bundle adjustment estimate squares: ground truth points

IEKF for the same sensors, unknowns dash-dotted line: batch estimate solid line: IEKF estimate

Difficulties with IEKF for this application:

prior assumptions about motion smoothness

cannot model relative error between adjacent camera positions

So, converting the batch algorithm into a variable state dimension filter (VSDF) is a promising future direction

IEKF assumptions on motion smoothness dash-dotted line: batch estimate solid line: IEKF estimate

right: IEKF propagation variances too strict

left: IEKF propagation variances just right

Recap:

image, gyro, and accelerometer measurements

batch algorithm

recursive algorithm

experiments

evaluate batch and recursive algorithms

establish basic facts about motion from image and inertial measurements

Outline

Motion from images

Lucas-Kanade and real sequences

The “smalls” tracker

Conclusion

Robust image feature tracking: Lucas-Kanade and real sequences (1)

Combining image and inertial measurements improves our situation, but…

we still need accurate feature tracking tracking

some sequences do not come with inertial measurements

better feature tracking for improved 6 DOF motion estimation

remaining results will be image-only

Lucas-Kanade has been the go-to feature tracker for shape-from-motion

minimizes a correlation-like matching error

using general minimization

evaluates the matching error at only a few locations

subpixel resolution

Additional heuristics used to apply Lucas-Kanade to shape-from-motion:

task: heuristic:

choose features to track high image texture

identify mistracked, occluded, no-longer-visible

convergence, matching error

handle large motions image pyramid

But Lucas-Kanade performs poorly on many real sequences…

Robust image feature tracking: the “smalls” tracker (1)

smalls is a new feature tracker targeted at 6 DOF motion estimation

exploits the rigid scene assumption

eliminates the heuristics normally used with Lucas-Kanade

SIFT is an enabling technology here

First step: epipolar geometry estimation

use SIFT to establish matches between the two images

get the 6 DOF camera motion between the two images

get the epipolar geometry relating the two images

Second step: track along epipolar lines

use nearby SIFT matches to get initial position on epipolar line

exploits the rigid scene assumption

eliminates heuristic: pyramid

Third step: prune features

geometrically inconsistent features are marked as mistracked and removed

clumped features are pruned

eliminates heuristic: detecting mistracked features based on convergence, error

Fourth step: extract new features

spatial image coverage is the main criterion

required texture is minimal when tracking is restricted to the epipolar lines

eliminates heuristic: extracting only textured features

left: odometry only right: images only

average error: 1.74 m

maximum error: 5.14 m

total distance: 230 m

Recap:

exploits the rigid scene and eliminates heuristics

allows hands-free tracking for real sequences

can still be defeated by textureless areas or repetitive texture

Outline

Motion from images

proof of concept system

experiment

Conclusion

Long-term motion estimation: proof of concept system (1)

Image-based motion estimates from any system will drift:

if the features we see are always changing

given sufficient time

if we don’t recognize when we’ve revisited a location

To limit drift:

recognize when we’ve returned to a previous location

exploit the return

A proof of concept system demonstrates these capabilities

“smalls” tracker state: 2D feature history for images in I

variable state dimension filter (VSDF) state for images in I: 6 DOF camera positions, covariances for images in I 3D positions for features visible in I

SIFT keypoints for image in

system state S

image indices: I = {i1, …, in}

0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}

rollback

0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}

non-rollback States:

rollback

0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}

non-rollback States:

rollback

0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2}

{0, 1, 2, 3, 8}

non-rollback pruned States:

rollback

0 1 2 3 4 5 6 7 8

8 9 10 11

11 12 13 14

14 15 16 17

17 18 19 20

{0, …, 6, 11, 12, 17, …, 20}

When to “roll back”?

examine the camera covariances for the current state and the candidate rollback state

check the number of SIFT matches

extend from the candidate state

examine the camera covariances for the current state and the resulting extended state

Long-term motion estimation: experiment (1)

CMU FRC highbay views; 945 images total

CMU FRC highbay

(first forward pass: images 0-213)

CMU FRC highbay

(first backward pass: images 214-380)

rollback

0 1 2 3 4 5 6 7 8

8 9 10 11

11 12 13 14

14 15 16 17

17 18 19 20

normally, the system produces a general tree of states

…0 1 2 3 4 5 6 7

13 14 15 14

16 17 18 17

non-rollback rollback pruned States:

for this example, the “rollback” states are restricted to the first forward pass

10 11 12 14

movie…bottom half is smalls output:

movie…top half is motion estimates:

Outline

Motion from images

Conclusion

remaining issues

Conclusion: remaining issues

all: system is experimental, not optimized for speed

image and inertial: VSDF

“smalls”: integration of gyro, more robustness to poor texture needed

long-term: “roll back” space, computation grow with sequence length

Thanks!

Related materials:

these slides

motion from image and inertial measurements

Documents

inertial sensors for navigation · pdf filethe system will...

motion tracking algorithms for inertial measurement

arxiv:2003.12516v1 [physics.atom-ph] 27 mar 2020 ·...

vision aided inertial navigation using single video camera...

inertial and magnetic sensing of human motion

motion from image and inertial measurements dennis strelow...

motion estimation from image and inertial measurements

inertial motion tracking on mobile and wearable devices

human motion tracking and orientation estimation...

global positioning and inertial measurements · pdf...

inertial motion, explanation, and the foundations of

machine learning for placement-insensitive inertial motion...

limitations on inertial sensor testing produced by … on...

inertial sensing to determine movement disorder motion

university of cambridge - non-inertial motion of charged...

observation of inertial particle motion in laminar flow in

determining anatomical frames via inertial motion capture: a...

3dsuit inertial motion capture systems for biomechanical...

motion capture: from radio signals to inertial signals ·...

hybrid inertial-robotic motion tracking for posture...