real-time 3d modeling of vehicles in low-cost monocamera systems
DESCRIPTION
ThisTRANSCRIPT
![Page 1: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/1.jpg)
1
Real-time 3D modeling of vehicles in
low-cost monocamera systems
1
M. Nieto, L. Unzueta, A. Cortés, A. Barandiaran, O. Otaegui, and P. Sánchez
![Page 2: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/2.jpg)
2
Outline
Introduction Context
Objectives
Motivation
System overview Examples
Proposed solution
Calibration
Segmentation
2D blob extraction
3D tracking
Bayesian framework Likelihood
Prior
Real-time operation MH
From 3D to 1D search
Finite posterior evaluations
Results
2Algarve (Portugal) - VISAPP 2011/03/05-07
![Page 3: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/3.jpg)
3
Introduction
Context
Algarve (Portugal) - VISAPP 2011/03/05-07 3
“Intrusive” technology (ILD) Non-intrusive technology (video)
![Page 4: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/4.jpg)
4
Introduction
Conditions
Real-time solution
High speed roads
Objectives
Count and classify vehicles (light and heavy)
Algarve (Portugal) - VISAPP 2011/03/05-07 4
“Any” perspective
Low-cost camera
![Page 5: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/5.jpg)
5
Introduction
Motivation
Projective distortion
Algarve (Portugal) - VISAPP 2011/03/05-07 5
![Page 6: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/6.jpg)
6
Introduction
Motivation
Length error (2D vs 3D)
Algarve (Portugal) - VISAPP 2011/03/05-07 6
![Page 7: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/7.jpg)
7
Introduction
Algarve (Portugal) - VISAPP 2011/03/05-07 7
Motivation
Width error (2D vs 3D)
![Page 8: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/8.jpg)
8
System overview
Algarve (Portugal) - VISAPP 2011/03/05-07 8
Examples
![Page 9: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/9.jpg)
9
System overview
Proposed solution
Algarve (Portugal) - VISAPP 2011/03/05-07 9
![Page 10: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/10.jpg)
10
System overview
Calibration and rectification
Algarve (Portugal) - VISAPP 2011/03/05-07 10
Radial distortion
Perspective distortion
Metric information
We can obtain the approximate intrinsic and extrinsic parameters
of the calibration from the homography
![Page 11: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/11.jpg)
11
System overview
Segmentation
Pixel-level classification
Colour (IHLS) and gradient
Algarve (Portugal) - VISAPP 2011/03/05-07 11
![Page 12: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/12.jpg)
12
System overview
2D blob extraction
Algarve (Portugal) - VISAPP 2011/03/05-07 12
Remove small blobs
Merge (it. #1) Merge (it. #2)
![Page 13: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/13.jpg)
13
System overview
3D tracking
Infer 3D boxes that projects onto detected 2D blobs
Propagate through time
Algarve (Portugal) - VISAPP 2011/03/05-07 13
![Page 14: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/14.jpg)
14
Bayesian framework
Algarve (Portugal) - VISAPP 2011/03/05-07 14
Likelihood
(Motion) Prior
Models
State (t-1) State (t)
![Page 15: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/15.jpg)
15
Bayesian framework
MCMC-based particle filter Recursive Bayes’ rule
Approximate with a set of samples (particles)
Obtain samples with Metropolis-Hastings
Algarve (Portugal) - VISAPP 2011/03/05-07 15
Posterior LikelihoodPrediction
Sample-basedapproach
![Page 16: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/16.jpg)
16
2D blob
3D best box
3D hypotheses
Bayesian framework
Likelihood model
Algarve (Portugal) - VISAPP 2011/03/05-07 16
Plane perpendicular to the raypassing through
Projection of into the ray
![Page 17: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/17.jpg)
17
Bayesian framework
Prior vehicle models
Algarve (Portugal) - VISAPP 2011/03/05-07 17
![Page 18: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/18.jpg)
18
Real-time operation
Real-time is a hard requirement
High speed roads + short stretchs = few frames
MCMC particle filter
Propagate a chain of samples using Metropolis Hastings
It might be slow (depend on the processor)
Simplifications are welcome
One single sample
From 3D to 1D search
Check discrete posterior values and compute argmax
Algarve (Portugal) - VISAPP 2011/03/05-07 18
![Page 19: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/19.jpg)
19
Real-time operation
Metropolis-Hastings
Full computation
Single-sample posterior representation
Algarve (Portugal) - VISAPP 2011/03/05-07 19
Previous set of particles
Single previoushypothesis
![Page 20: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/20.jpg)
20
Real-time operation
From 3D to 1D search
Algarve (Portugal) - VISAPP 2011/03/05-07 20
s.t.
Measures the distance of the hypothesis
to the ray
Let us consider onlypoints on the ray
Need to sample 3D space WHL
Need to sample 1D ray
![Page 21: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/21.jpg)
21
Real-time operation
Finite (small) number of posterior evaluations
Approximate MAP as product of most likely model and prior
Project each vehicle model to the ray given by the 2D observation
Algarve (Portugal) - VISAPP 2011/03/05-07 21
We reduce the search to the
evaluation of M points of the ray
These points are the projection of the set of prior models on the ray
![Page 22: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/22.jpg)
22
Results
Remarkable capabilities
Flexibility to adapt to large vehicles
Algarve (Portugal) - VISAPP 2011/03/05-07 22
![Page 23: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/23.jpg)
24
Results
Capabilities
Multiple / simultaneous vehicles
Static / moving vehicles
Algarve (Portugal) - VISAPP 2011/03/05-07 24
![Page 24: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/24.jpg)
25
Results
Capabilities
Adapts to visible part of vehicles
Better fits when several frames have led the filter to converge
Algarve (Portugal) - VISAPP 2011/03/05-07 25
![Page 25: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/25.jpg)
26
Results
Known problems / difficulties
Algarve (Portugal) - VISAPP 2011/03/05-07 26
Visual artifacts
Merging vehicles
Shadows
Occlusions / perspectiveNight time
Dark vehicles
![Page 26: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/26.jpg)
27
Results
Classification
Algarve (Portugal) - VISAPP 2011/03/05-07 27
![Page 27: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/27.jpg)
28
Results
Recall & precision in challenging scenarios
6+ hours
13000+ vehicles
Algarve (Portugal) - VISAPP 2011/03/05-07 28
Sequence N_L N_P R_L (%) P_L (%) R_P (%) P_P (%)
Dusk 1662 118 98.01 98.61 94.92 95.73
Rain & Shadow 4516 627 95.70 94.84 92.98 87.80
Traffic jam 4796 563 98.33 98.91 91.47 91.80
Dusk and rain 968 115 92.25 88.42 87.83 81.45
Perspective 614 101 91.86 92.16 86.14 84.47
Color noise 561 23 99.82 99.12 86.96 86.96
Average 13117 1547 95.99 95.34 90.05 88.03
![Page 28: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/28.jpg)
29
Results
Algarve (Portugal) - VISAPP 2011/03/05-07 29
0,8
0,85
0,9
0,95
1
0,8 0,85 0,9 0,95 1
Pre
cisi
on
Recall
Dusk
Rain and shadows
Traffic jam
Dusk and rain
Perspective
Color noise
0,8
0,85
0,9
0,95
1
0,8 0,85 0,9 0,95 1
Pre
cisi
on
Recall
Dusk
Rain and shadows
Traffic jam
Dusk and rain
Perspective
Color noise
Light vehicles Heavy vehicles
Dusk Rain and shadows Traffic jam Dusk and rain Perspective Colour noise
![Page 29: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/29.jpg)
30
Conclusions
Vehicle detection and tracking 3D volume estimation
Real-time operation Using some approximations within the particle filter framework
Flexible Automatically adapts to illumination changes
Requires calibration to work under different perspectives
Average counting and classification rates above 90% in challengingscenarios
Future work: interaction between objects, autocalibration, automatic selection of parameters…
Algarve (Portugal) - VISAPP 2011/03/05-07 30
![Page 30: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/30.jpg)
3131
Dr.-Ing. Marcos Nieto DoncelInvestigador/Researcher
http://marcosnieto.net/
Algarve (Portugal) - VISAPP 2011/03/05-07
![Page 31: REAL-TIME 3D MODELING OF VEHICLES IN LOW-COST MONOCAMERA SYSTEMS](https://reader034.vdocuments.site/reader034/viewer/2022051817/548f91fab479598e6a8b4e81/html5/thumbnails/31.jpg)
32
Results
Night time performance
Algarve (Portugal) - VISAPP 2011/03/05-07 32