augmented vehicular reality - microsoft...augmented vehicular reality (avr) hang qiu, fawad ahmad,...
TRANSCRIPT
Augmented Vehicular Reality (AVR)
Hang Qiu, Fawad Ahmad, Fan Bai, Marco Gruteser, Ramesh Govindan
Reliable Autonomous Driving
Human drivers can achieve high reliability• ~ 100,000,000 miles between fatalities
Can autonomous cars reach the same level of reliability?2
AVR Introduction AVR Design AVR Evaluation AVR Future Work
Stereo Camera
Reliability with Advanced Sensing
LiDAR
Radar
3
AVR Introduction AVR Design AVR Evaluation AVR Future Work
4
ShadowSensors limited by
Line-of-sight
5
Car
6
7
Extended Vision
8
Extended Vision Improves Safety
9
AVR Introduction AVR Design AVR Evaluation AVR Future Work
10
AVR Introduction AVR Design AVR Evaluation AVR Future Work
11
12
13
AVR Introduction AVR Design AVR Evaluation AVR Future Work
14
AVR: Augmented Vehicular Reality
• Accurate Positioning
• Extremely Limited V2V Bandwidth
• Low Latency
AVR: Challenges
dx
r
15
AVR Introduction AVR Design AVR Evaluation AVR Future Work
AVR Contributions
• Accurate Positioning• Relative Positioning using 3D Feature Map
• Perspective Transformation
• Extremely Limited V2V bandwidth • Dynamic Object Extraction
• Adaptive Motion Vector Transmission
• Low Latency• Pipeline Optimization
• Motion Prediction to Hide Latency
16
AVR Introduction AVR Design AVR Evaluation AVR Future Work
AVR Contributions
• Accurate Positioning• Relative Positioning using 3D Feature Map
• Perspective Transformation
• Extremely Limited V2V bandwidth • Dynamic Object Extraction
• Adaptive Motion Vector Transmission
• Low Latency• Pipeline Optimization
• Motion Prediction to Hide Latency
17
AVR Introduction AVR Design AVR Evaluation AVR Future Work
3D Perception: The Point Cloud
. Voxel • Position (x, y, z)• Color (R, G, B)• Reflection Intensity
18
AVR Introduction AVR Design AVR Evaluation AVR Future Work
Relative Positioning
?
AVR Introduction AVR Design AVR Evaluation AVR Future Work
19
Sparse Features for LocalizationStatic Features
20AVR Introduction AVR Design AVR Evaluation AVR Future Work
20
Collecting the 3D feature map
21AVR Introduction AVR Design AVR Evaluation AVR Future Work
21
AVR uses SLAM to build a 3D feature map
Localization using sparse 3D map
22AVR Introduction AVR Design AVR Evaluation AVR Future Work
22
Perspective Transformation
23AVR Introduction AVR Design AVR Evaluation AVR Future Work
23
AVR Contributions
• Accurate Positioning• Relative Positioning using 3D Feature Map
• Perspective Transformation
• Extremely Limited V2V bandwidth • Dynamic Object Extraction
• Adaptive Motion Vector Transmission
• Low Latency• Pipeline Optimization
• Motion Prediction to Hide Latency
24
24AVR Introduction AVR Design AVR Evaluation AVR Future Work
The Bandwidth
Data Size Bandwidth at 30fps
Full Point Cloud 4.91 MB 1178.4 Mbps
A Single Car 0.33 MB 79.2 Mbps
DSRC 802.11g
6 ~ 27 Mbps 6 ~ 54 Mbps
25AVR Introduction AVR Design AVR Evaluation AVR Future Work
25
Dynamic Object Detection
C1
C2
26AVR Introduction AVR Design AVR Evaluation AVR Future Work
26
Dynamic Object Detection
V C2+VC2
27AVR Introduction AVR Design AVR Evaluation AVR Future Work
27
Estimating Absolute Motion
C2+VC1
Object Motion Vector
28AVR Introduction AVR Design AVR Evaluation AVR Future Work
28
Isolating Dynamics from Statics
B2+V
C2+V
B1
C1
29AVR Introduction AVR Design AVR Evaluation AVR Future Work
29Static Object
Object Motion Vector
Adaptive Transmission
30AVR Introduction AVR Design AVR Evaluation AVR Future Work
30
Transmitting Motion Vectors Adaptively based on Bandwidth Availability.
AVR Contributions
• Accurate Positioning• Relative Positioning using 3D Feature Map
• Perspective Transformation
• Extremely Limited V2V bandwidth • Dynamic Object Extraction
• Adaptive Motion Vector Transmission
• Low Latency• Pipeline Optimization
• Motion Prediction to Hide Latency
31
31AVR Introduction AVR Design AVR Evaluation AVR Future Work
Pipeline Optimization
Decouple computation and reuse intermediate results to cut latency.Distribute computation evenly to 3 stages for high throughput.
32AVR Introduction AVR Design AVR Evaluation AVR Future Work
32
Stage 1: Preprocessing Stage 2: Localization Stage 3: Postprocessing
AVR Evaluation
• The Benefits of AVR in ADAS and Autonomous Driving
• End-to-end Performance
• Reconstruction Accuracy
• Bandwidth / Accuracy Tradeoff
• The Impact of Delay
• Motion Estimation Accuracy
• Throughput And Latency
33
33AVR Introduction AVR Design AVR Evaluation AVR Future Work
AVR Evaluation
• The Benefits of AVR in ADAS and Autonomous Driving
• End-to-end Performance
• Reconstruction Accuracy
• Bandwidth / Accuracy Tradeoff
• The Impact of Delay
• Motion Estimation
• Throughput And Latency
34
34AVR Introduction AVR Design AVR Evaluation AVR Future Work
AVR Setup
Alienware Laptop• I7 CPU, 4.4GHz• 16GB DDR4 RAM• Nvidia 1080p GPU, 2560 CUDA cores.
ZED Stereo Camera
TP-Link Talon 7200• 802.11 g/n/ac/ad• Wireless Distributed System (WDS)
35AVR Introduction AVR Design AVR Evaluation AVR Future Work
35
AVR Setup
3636AVR Introduction AVR Design AVR Evaluation AVR Future Work
36
AVR Setup
3737AVR Introduction AVR Design AVR Evaluation AVR Future Work
37
AVR Setup
3838AVR Introduction AVR Design AVR Evaluation AVR Future Work
38
Building an extended vision by merging the leader’s view.
Benefits in Autonomous Driving
3939AVR Introduction AVR Design AVR Evaluation AVR Future Work
39
Drivable Space (in blue)
Planned Path (in green) Follower can see oncoming vehicle (in red circle)
Benefits in Autonomous Driving
AVR can double the drivable space detected.
4040AVR Introduction AVR Design AVR Evaluation AVR Future Work
40
Benefits in Autonomous Driving
AVR can help path planner to avoid dangerous / wasteful lane change maneuver.
4141AVR Introduction AVR Design AVR Evaluation AVR Future Work
41
End-to-end Result
• Full: sharing the point cloud of the whole scene.
• Dynamic: sharing only the dynamic part of the scene.
AVR reduces 10x of bandwidth requirement in dynamic mode. AVR adapts smoothly to available bandwidth.
MeanThroughput
(Mbps)Motion Vector
StreakMotion Vector /
Frame
Full 367.02 2.80 1.17
Dynamic 34.95 1 0.003
4242AVR Introduction AVR Design AVR Evaluation AVR Future Work
42
End-to-end Result
AVR achieves 30 fps using a 3-stage pipeline AVR induces 96 ms processing delay
4343AVR Introduction AVR Design AVR Evaluation AVR Future Work
43
Preprocessing
Localization
Postprocessing
Caveat: perfect calibration, more details in the paper.
Reconstruction Accuracy
Error @ 20mph Static Objects Moving Objects
Median 0.027 m 0.070 m
90th Percentile 0.045 m 0.193 m
AVR achieves high reconstruction fidelity.Major source of error: motion estimation, camera calibration.
4444AVR Introduction AVR Design AVR Evaluation AVR Future Work
44
• Stereo Camera vs. LiDAR
• Accuracy
• Range
• Robustness to poor visibility
• Cooperative AVR
Limitation & Future Work
4545AVR Introduction AVR Design AVR Evaluation AVR Future Work
45
AVR Demo
46
Augmented Vehicular Reality❑Breaks the LOS limit to see beyond occlusion
❑Extends 3D perception range
❑Inspires more informed driving decisions
Thank You!Q&AQ&A
Q&A47
Code & Datasethttps://github.com/hangqiu/AVR16