1 motion estimation from image and inertial measurements dennis strelow and sanjiv singh carnegie...
TRANSCRIPT
![Page 1: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/1.jpg)
1
Motion estimation from image and inertial measurements
Dennis Strelow and Sanjiv Singh
Carnegie Mellon University
![Page 2: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/2.jpg)
2
Introduction (1)
micro air vehicle (MAV) navigation
AeroVironment Black Widow AeroVironment Microbat
![Page 3: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/3.jpg)
3
Introduction (2)
mars rover navigation
Mars Exploration Rovers (MER) Hyperion
![Page 4: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/4.jpg)
4
Introduction (3)
robotic search and rescue
RhexCenter for Robot-Assisted Search and Rescue, U. of South Florida
![Page 5: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/5.jpg)
5
Introduction (4)
NASA ISS personal satellite assistant
![Page 6: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/6.jpg)
6
Introduction (5)
Each of these requires:
six degree of freedom motion
in unknown environments
without GPS or other absolute positioning
over the long term
…and in many cases…
small, light, and cheap sensors
![Page 7: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/7.jpg)
7
Introduction (6)
If we adopt a camera as our sensor, estimate…
the vehicle (i.e., sensor) motion
…and as a by-product:
the sparse structure of the environment
![Page 8: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/8.jpg)
8
Introduction (7)
One paradigm for estimating the motion and sparse structure uses two steps:
tracking: track image features through the image sequence
estimation: find 6 DOF camera positions and 3D point positions consistent with the image features
![Page 9: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/9.jpg)
9
Introduction (8)
![Page 10: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/10.jpg)
10
Introduction (8)
Lucas-Kanade
![Page 11: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/11.jpg)
11
Introduction (9)
![Page 12: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/12.jpg)
12
Introduction (10)
![Page 13: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/13.jpg)
13
Introduction (10)
bundle adjustment, a.k.a. nonlinear shape-from-motion
![Page 14: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/14.jpg)
14
Introduction (11)
This example sequence is benign:
small number of images
large number of easily tracked image features
all points visible in all frames
…and the estimated motion and scene structure are still ambiguous
![Page 15: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/15.jpg)
15
Introduction (12)
Sequences from navigation applications much harder:
long observations sequences
small number of features, which may be poorly tracked
each feature visible in a small section of the image stream
![Page 16: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/16.jpg)
16
Introduction (13)
We have been working on both:
tracking
estimation
This talk:
mostly estimation
![Page 17: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/17.jpg)
17
Introduction (14)
![Page 18: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/18.jpg)
18
Introduction (15)
Image measurements only
often the only option
Image and inertial measurements
can disambiguate image-only estimates
more complex: requires more calibration, estimation of additional unknowns
![Page 19: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/19.jpg)
19
Introduction (16)
Batch estimation:
uses all of the observations at once
all observations must be available before computation begins
Online estimation:
observations are incorporated as they arrive
suitable for long or “infinite” sequences
![Page 20: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/20.jpg)
20
Introduction (17)
Conventional images:
images from standard cameras
model as pinhole projection with radial distortion
Omnidirectional images:
images that result from combining a conventional camera with a convex mirror
requires a more complex projection model
![Page 21: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/21.jpg)
21
Introduction (18)
![Page 22: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/22.jpg)
22
Introduction (19)
![Page 23: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/23.jpg)
23
Outline
Background: image-only batch estimation
Image-and-inertial batch estimation
Online estimation
Experiments
Future work
![Page 24: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/24.jpg)
24
Background: image-only batch estimation (1)
![Page 25: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/25.jpg)
25
Background: image-only batch estimation (1)
Tracking provides image location xij for each point j that appears in image i
![Page 26: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/26.jpg)
26
Background: image-only batch estimation (2)
Suppose we also have estimates of:
the camera rotation ρi and translation ti at time of each image
Three dimensional point positions Xj of each tracked point
Then the reprojections are:))(( iji tXR
![Page 27: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/27.jpg)
27
Background: image-only batch estimation (3)
![Page 28: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/28.jpg)
28
Background: image-only batch estimation (4)
![Page 29: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/29.jpg)
29
Background: image-only batch estimation (5)
So, minimize:
with respect to all the ρi, ti, Xj
“Bundle adjustment”: use iterative nonlinear minimization on this error
)),)(((,
image ijijji
i xtXRDE
![Page 30: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/30.jpg)
30
Background: image-only batch estimation (6)
Bundle adjustment pros:
any projection model can be used
some missing points can be tolerated
can be extended
Bundle adjustment cons:
requires some initial estimate
slow
![Page 31: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/31.jpg)
31
Background: image-only batch estimation (7)
Batch estimation cons:
all observations must be in hand before estimation begins
Image-only estimation cons:
sensitivity to mistracking
cannot recover global scale
![Page 32: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/32.jpg)
32
Outline
Background: image-only batch estimation
Image-and-inertial batch estimation
Online estimation
Experiments
Future work
![Page 33: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/33.jpg)
33
Image-and-inertial batch estimation (1)
![Page 34: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/34.jpg)
34
Image-and-inertial batch estimation (2)
Image and inertial measurements are highly complementary
![Page 35: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/35.jpg)
35
Image-and-inertial batch estimation (2)
Image and inertial measurements are highly complementary
Inertial measurements can:
establish the global scale
provide robustness if:• too few features• features infinitely far away• features in an “accidental” configuration
![Page 36: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/36.jpg)
36
Image-and-inertial batch estimation (3)
Image measurements can:
reduce inertial integration drift
separate effects of:• rotation• gravity• acceleration• biasin inertial readings
![Page 37: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/37.jpg)
37
Image-and-inertial batch estimation (4)
Gyro measurements:
ω’, ω: measured and actual angular velocity
bω: gyro bias
n: gaussian noise
nb '
![Page 38: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/38.jpg)
38
Image-and-inertial batch estimation (5)
Accelerometer measurements:
a’, a: measured and actual acceleration
g: gravity vector
ba: accelerometer bias
n: gaussian noise
nbgaRa' aT )((
![Page 39: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/39.jpg)
39
Image-and-inertial batch estimation (6)
Minimizes a combined error:
inertialimagecombined EEE
![Page 40: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/40.jpg)
40
Image-and-inertial batch estimation (7)
Image term Eimage is the same as before
![Page 41: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/41.jpg)
41
Image-and-inertial batch estimation (8)
Inertial error term Einertial is:
1
1n,translatio
1
1velocity,
1
1rotation,
ntranslatiovelocityrotationinertial
f
ii
f
ii
f
ii
E
E
E
EEEE
![Page 42: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/42.jpg)
42
Image-and-inertial batch estimation (8)
Inertial error term Einertial is:
1
1n,translatio
1
1velocity,
1
1rotation,
ntranslatiovelocityrotationinertial
f
ii
f
ii
f
ii
E
E
E
EEEE
![Page 43: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/43.jpg)
43
Image-and-inertial batch estimation (8)
Inertial error term Einertial is:
1
1n,translatio
1
1velocity,
1
1rotation,
ntranslatiovelocityrotationinertial
f
ii
f
ii
f
ii
E
E
E
EEEE
![Page 44: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/44.jpg)
44
Image-and-inertial batch estimation (9)
)),...,,(,( 11n,translatio iiitii tItDE
timeτi-1 (time of image i - 1)
ti-1 = t(τi-1)
ti = t(τi)
I(τi-1, τi, …, ti-1)
τi (time of image i)
tran
slat
ion
![Page 45: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/45.jpg)
45
Image-and-inertial batch estimation (10)
timeτ0
tran
slat
ion
τ1 τ2 τ5 τ3 τ4 τf-3 τf-2 τf-1
1
1n,translationtranslatio
f
iiEE
![Page 46: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/46.jpg)
46
Image-and-inertial batch estimation (11)
1
1n,translatio
1
1velocity,
1
1rotation,
ntranslatiovelocityrotationinertial
f
ii
f
ii
f
ii
E
E
E
EEEE
![Page 47: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/47.jpg)
47
Image-and-inertial batch estimation (11)
1
1n,translatio
1
1velocity,
1
1rotation,
ntranslatiovelocityrotationinertial
f
ii
f
ii
f
ii
E
E
E
EEEE
![Page 48: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/48.jpg)
48
Image-and-inertial batch estimation (12)
It(τi-1, τi ,…, ti-1) depends on:
τi-1, τi (known)
all inertial measurements for times τi-1< τ < τi (known)
ρi-1, ti-1
camera linear velocities: vi
g
bω, ba
![Page 49: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/49.jpg)
49
Image-and-inertial batch estimation (13)
The combined error function (image-and-inertial) is then minimized with respect to:
ρi, ti
Xj
![Page 50: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/50.jpg)
50
Image-and-inertial batch estimation (14)
The combined error function (image and inertial) is then minimized with respect to:
ρi, ti
Xj
vi
g
bω, ba
![Page 51: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/51.jpg)
51
Image-and-inertial batch estimation (15)
![Page 52: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/52.jpg)
52
Image-and-inertial batch estimation (16)
accurate motions even when image-only and inertial-only motions are poor
![Page 53: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/53.jpg)
53
Image-and-inertial batch estimation (17)
In addition:
recovers good gravity values, even with poor initialization
global scale often recovered with less than 5% error
![Page 54: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/54.jpg)
54
Outline
Background: image-only batch estimation
Image-and-inertial batch estimation
Online estimation
image measurements only
image and inertial measurements
Experiments
Future work
![Page 55: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/55.jpg)
55
Online algorithms (1)
![Page 56: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/56.jpg)
56
Online algorithms (2)
Batch estimation:
uses all of the observations at once
all observations must be available before computation begins
Online estimation:
observations are incorporated as they arrive
suitable for long or “infinite” sequences
![Page 57: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/57.jpg)
57
Online algorithms (3)
Our online algorithms are iterated extended Kalman filters (IEKFs)
Some components of the IEKF when applied to vehicle motion:
state estimate distribution gaussian: mean and covariance
motion model
measurement model
![Page 58: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/58.jpg)
58
Online algorithms (4)
initializationprior distribution on unknowns
timestamped observations
time propagationprior distribution on unknowns
timestamp kinit + 1
measurement updateprior distribution on unknowns
observation kinit + 1
prior distribution on unknowns
time propagationprior distribution on unknowns
timestamp i
measurement updateprior distribution on unknowns
observation i
1, …, kinit
![Page 59: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/59.jpg)
59
Online estimation (5): image-only
For the image-only online algorithm, the unknowns (state) are:
ρ(τ), t(τ)
currently visible point positions Xj
![Page 60: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/60.jpg)
60
Online estimation (6): image-only, cont.
initializationprior distribution on unknowns
timestamped observations1, …, kinit
apply the batch image-only algorithm to observations 1, …, kinit
![Page 61: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/61.jpg)
61
Online estimation (7): image-only, cont.
prior distribution on unknowns
time propagationprior distribution on unknowns
timestamp i
assume that ρ, t are perturbed by gaussian noise
assume that Xj are unchanged
![Page 62: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/62.jpg)
62
Online estimation (8): image-only, cont.
n
tXR
tXR
x
x
z
pp
))()((
))()((
1
0
1
0
prior distribution on unknowns
measurement updateprior distribution on unknowns
observation i
To update given an image measurement z, use the reprojection equation:
![Page 63: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/63.jpg)
63
Online estimation (9): image-only, cont.
For points that become visible after online operation has begun…
…adapt Smith, Self, and Cheeseman’s method for SLAM
![Page 64: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/64.jpg)
64
Online estimation (10): image-and-inertial
initializationprior distribution on unknowns
timestamped observations
time propagationprior distribution on unknowns
timestamp kinit + 1
measurement updateprior distribution on unknowns
observation kinit + 1
prior distribution on unknowns
time propagationprior distribution on unknowns
timestamp i
measurement updateprior distribution on unknowns
observation i
1, …, kinit
![Page 65: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/65.jpg)
65
Online estimation (11): image-and-inertial, cont.
Augmented state includes:
ρ(τ), t(τ)
currently visible point positions Xj
![Page 66: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/66.jpg)
66
Online estimation (11): image-and-inertial, cont.
Augmented state includes:
ρ(τ), t(τ)
currently visible point positions Xj
v(τ)
g
bω, ba
![Page 67: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/67.jpg)
67
Online estimation (11): image-and-inertial, cont.
Augmented state includes:
ρ(τ), t(τ)
currently visible point positions Xj
v(τ)
g
bω, ba
camera system angular velocity: ω(τ)
world system linear acceleration: a(τ)
![Page 68: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/68.jpg)
68
Online estimation (12): image-and-inertial, cont.
initializationprior distribution on unknowns
timestamped observations1, …, kinit
apply the batch image-and-inertial algorithm
![Page 69: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/69.jpg)
69
Online estimation (13): image-and-inertial, cont.
assume that ω, a are perturbed by gaussian noise
assume that Xj, g, bω, ba are unchanged
prior distribution on unknowns
time propagationprior distribution on unknowns
timestamp i
![Page 70: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/70.jpg)
70
Online estimation (14): image-and-inertial, cont.
nbgaR
b
az
aT
)()('
'
prior distribution on unknowns
measurement updateprior distribution on unknowns
observation i
use the inertial sensor model we’ve seen:
![Page 71: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/71.jpg)
71
Outline
Background: image-only batch estimation
Image and inertial batch estimation
Online estimation
Experiments
Hyperion
CMU crane
Future work
![Page 72: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/72.jpg)
72
Experiments (1): Hyperion
![Page 73: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/73.jpg)
73
Experiments (2): Hyperion, cont.
![Page 74: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/74.jpg)
74
Experiments (3): Hyperion, cont.
![Page 75: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/75.jpg)
75
Experiments (4): Hyperion, cont.
![Page 76: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/76.jpg)
76
Experiments (4): Hyperion, cont.
![Page 77: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/77.jpg)
77
Experiments (5): Hyperion, cont.
![Page 78: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/78.jpg)
78
Experiments (6): Hyperion, cont.
![Page 79: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/79.jpg)
79
Experiments (7): CMU crane
Crane capable of translating a platform…
…through x, y, z…
…through a workspace of about 10 x 10 x 5 m
![Page 80: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/80.jpg)
80
Experiments (8): CMU crane, cont.
y tr
ansl
atio
n (m
eter
s)
x translation (meters)3.00.0-3.0
-3.0
0.0
3.0
(x, y) translation ground truth
![Page 81: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/81.jpg)
81
Experiments (9): CMU crane, cont.z
(m)
time
3.5
4.5
z translation ground truth
No change in rotation
![Page 82: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/82.jpg)
82
Experiments (10): CMU crane, cont.
![Page 83: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/83.jpg)
83
Experiments (11): CMU crane, cont.
Hard sequence:
• Each image contains an average of 56.0 points
• Each point appears in an average of 62.3 images (4.4% of sequence)
• Image-and-inertial online algorithm applied
• 40 images used in batch initialization
![Page 84: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/84.jpg)
84
Experiments (12): CMU crane, cont.
![Page 85: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/85.jpg)
85
Experiments (13): CMU crane, cont.
Estimated z camera translations
![Page 86: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/86.jpg)
86
Experiments (14): CMU crane, cont.
6 DOF errors, after scaled rigid alignment:
Rotation: 0.14 radians average
Translation: 31.5 cm average (0.9% of distance traveled)
Global scale error: -3.4%
![Page 87: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/87.jpg)
87
Outline
Background: image-only batch estimation
Image and inertial batch estimation
Online estimation
Experiments
Future work
![Page 88: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/88.jpg)
88
Future work (1)
For long sequences:
no single external point is always visible
Estimated positions will drift due to…
random and gross observation errors
modeling errors
suboptimality in online estimation
![Page 89: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/89.jpg)
89
Future work (2)
![Page 90: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/90.jpg)
90
Future work (3)
![Page 91: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/91.jpg)
91
Future work (4)
Need to incorporate some advances from the SLAM community to deal with this issue…
…in particular, reacquiring revisited features
![Page 92: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/92.jpg)
92
Future work (5)
Two issues:
(1) recognizing revisited features
(2) exploiting revisited features in the estimation
Lowe’s SIFT features a good candidate for (1)
![Page 93: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/93.jpg)
93
![Page 94: 1 Motion estimation from image and inertial measurements Dennis Strelow and Sanjiv Singh Carnegie Mellon University](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f4e5503460f94c6f684/html5/thumbnails/94.jpg)
94
Thanks!
Pointers to:
these slides
related work by others
related publications by our group
feature tracking movies
VRML models
at:http://www.cs.cmu.edu/~dstrelow/uw