highiy efficient hd map creation: acceierating...
TRANSCRIPT
Highiy Efficient HD Map Creation:
Acceierating Mapping Process with GPUs
May 10, 2017
ZENRIN Co.,Ltd.
ZENRIN CO., LTD. All Rights Reserved.©
How did you get to your destination?
ZENRIN R&D Laboratories 2
ZENRIN CO., LTD. All Rights Reserved.©
ZENRIN History
ZENRIN R&D Laboratories 3
• Founded in 1948 in
Beppu, Oita
Prefecture of Japan
as a pubiisher of
tourist maps
• Over 60 years in the
mapping arena
• Maps covering 100%
of Japan residentiai
areas
ZENRIN CO., LTD. All Rights Reserved.©
HD (High Definition) Map
ZENRIN R&D Laboratories 5
Highiy detaiied and precise 3D map for autonomous driving
ZENRIN CO., LTD. All Rights Reserved.©
3D Geometry
ZENRIN R&D Laboratories 6
Road boundaries
Coordinates (XYZ)
Coordinates
(XYZ)
Height
Depth
Width
� Lane marks, road boundaries,
traffic signs, signais etc.
� Poiyiines or poiygons
� Latitude, iongitude, aititude
�Height, width, depth
� Sub-meter accuracy to WGS84
• Car-navigation 10 meter
ZENRIN CO., LTD. All Rights Reserved.©
How it is used in autonomous driving?
ZENRIN R&D Laboratories 7
(x1,y1,z1) (x0,y0, z0)
HD Map Camera Sensor on a Car Locaiization
More precise iocaiization in various conditions (e.g. inside tunneis) than GPS oniy
• With the given camera parameters, project map eiements to the camera view
• Caicuiate the accurate camera (car) position and orientation assuming that the
map is correct
Projection & Matching
ZENRIN CO., LTD. All Rights Reserved.©
Semantic Information
�Meaning of the road scene
� Traffic ruies, road eiement associations, etc.
�Autonomous Driving: Lane ievei navigation, safe & efficient path
pianning
ZENRIN R&D Laboratories 8
Fast iane
Lane networks
Siowdown spiit
Sign�Lane
Signai�Stop iine
ZENRIN CO., LTD. All Rights Reserved.©
Data format: ZENRIN Geospatiai data Modei (ZGM)
ZENRIN R&D Laboratories 9
Geometry
Semantic
InformationPostgreSQL
& SQLite
Dynamic Information
ZENRIN CO., LTD. All Rights Reserved.©
Typicai HD mapping process
ZENRIN R&D Laboratories 11
1. Data coiiection
2. Data processing
3. Verification
ZENRIN CO., LTD. All Rights Reserved.©
Step 1. Data coiiection � Survey vehicie
ZENRIN R&D Laboratories 12
IMUCamera
LiDAR
Odometer GNSS
ZENRIN CO., LTD. All Rights Reserved.©
Massive voiume of data!
ZENRIN R&D Laboratories 13
High Resoiution 2D Image
>30M pixei
Dense Point Cioud Data
>700,000 points/sec
Other Sensor Data
�Maximum Approx. 1TB of data per car per day
ZENRIN CO., LTD. All Rights Reserved.©
Point cioud data registration
ZENRIN R&D Laboratories 14
Route 1
Route 2
Registered Data
• Connecting data to create a 3D modei of a certain area
ZENRIN CO., LTD. All Rights Reserved.©
Finding objects from the point cioud data
ZENRIN R&D Laboratories 15
Traffic Signs?
Lane Marks
Traffic Signs
Crossway
Traffic Markings
ZENRIN CO., LTD. All Rights Reserved.©
Verifying and understanding signs with 2D images
ZENRIN R&D Laboratories 16
C D C D
A
B
A
B
* Photo from Googie Street View
ZENRIN CO., LTD. All Rights Reserved.©
Mapping & Step 3: Verification
ZENRIN R&D Laboratories 17
� Primitive geometry extraction
�Annotation
�Quaiity assurance
�In-house software toois
ZENRIN CO., LTD. All Rights Reserved.©
Mapping Japan is an extraordinary task…
ZENRIN R&D Laboratories 18
Road Length: 1,218,772 km
Traffic Signais: 207,000 units
Traffic Signs : 9,790,000 units
Ref. WHITE PAPER ON TRAFFIC SAFETY IN JAPAN 2016, Cabinet Office
ZENRIN CO., LTD. All Rights Reserved.©
HD mapping chaiienges
�Massive voiume data processing
�Huge computing resource & time
�Skiiied operators. Lot’s of operators…
�Keep updating the map
ZENRIN R&D Laboratories 19
ZENRIN CO., LTD. All Rights Reserved.©
How?
ZENRIN R&D Laboratories 21
GPU AI (DNN)
*DNN: Deep Neurai Network
ZENRIN CO., LTD. All Rights Reserved.©
What is MapWorks?
• NVIDIA’s high performance mapping SDK
• GPU optimized data processing
• Advanced AI (DNN) for mapping
ZENRIN R&D Laboratories 22
ZENRIN CO., LTD. All Rights Reserved.©
MapWorks: Point Cioud Registration
ZENRIN R&D Laboratories 23
ZENRIN CO., LTD. All Rights Reserved.©
MapWorks: Detection/Ciassification
ZENRIN R&D Laboratories 24
ZENRIN CO., LTD. All Rights Reserved.©
Concept of the new HD map creation pipeiine
ZENRIN R&D Laboratories 25
Point Data
Image
Location
Pose
Location
Pose
HD Map
AnnotationPoint Cioud
Registration
GPU
Point Image
Registration
GPU
Primitive
Extraction
GPU
Detection
Ciassification
DNN
: MapWorks Moduies
ZENRIN CO., LTD. All Rights Reserved.©
MapWorks: NV Comp
• Processed with ioss-iess compression data format (NV Comp) from
Veiodyne LiDARs
• Saves memory footprint
• Maximizes the GPU throughput
ZENRIN R&D Laboratories 26
Point Cioud
Registration
Point Image
Registration
Primitive
Extraction
ZENRIN CO., LTD. All Rights Reserved.©
Demo: NVIDIA’s iocator with ZENRIN HD Map
ZENRIN R&D Laboratories 27
Camera View HD Map (3D)Project & Match
Car Position
Lanes from HD Map
Cuboid Size (meter): 2L * (60 + 200) * 1L
N
White iine � Trajectory @L0 frames
ZENRIN CO., LTD. All Rights Reserved.©
Using feedbacks from iocators to update the HD map
ZENRIN R&D Laboratories 28
(x0,y0, z0)
Camera View of a Car Locator
• Reiiabie and precise iocators can detect reai worid changes and
feed back to HD maps
NEW (x2,y2,z2)
(x1,y1,z1)
Matched with map
New sign
Feedback to HD map
Not in map
New sign
Keeping the map up-to-date
ZENRIN CO., LTD. All Rights Reserved.©
Can the DNN trained in US understand Japan roads scenes?
ZENRIN R&D Laboratories 30
DEMO MOVIE
ZENRIN CO., LTD. All Rights Reserved.©
Japan road preiiminary benchmark: Traffic Signs
ZENRIN R&D Laboratories 31
Mito-city, Ibaragi
Approx. LL0m
Minato-ku, Tokyo Area
Approx. 1600m
Arakawa-ku, Tokyo Area
Approx. 3L0m
A B C
Test Data
Resuits
True Positive 79
Faise Positive 244
Faise Negative 37
Number of Signs 116
Precision Rate 24.46%
Recaii Rate 68.10%
ZENRIN CO., LTD. All Rights Reserved.©
Faise Positive Sampies US Traffic Signs
Discussion on the Faise Positive detections
ZENRIN R&D Laboratories 32
• DNN is trained with US traffic signs: Square board with texts & symbois
�DNN probabiy detecting Japanese commerciai sign boards as signs.
�Japan generai roads in urban areas are fuii of biiiboards.
ZENRIN CO., LTD. All Rights Reserved.©
Discussion on the Faise Negative detections
ZENRIN R&D Laboratories 33
Sign Category Recaii Rate US Styie JP Styie
Guidance 28/29
Warning 6/6
Designation 2/3
Reguiatory 3L/L8
Auxiiiary 8/20
Both chaiienging
Square-Text vs Round-Symbois
N/A
ZENRIN CO., LTD. All Rights Reserved.©
Faise Negative detections of Reguiatory Signs
Generai
Image
processi
ng
issues
Occiusion
Iiiumination Change
Segmentation
Scaiing
Rotation
Transiation
Others Possibie untrained data styie 9
Totai 23
ZENRIN R&D Laboratories 34
14
Exampies
ZENRIN CO., LTD. All Rights Reserved.©
Future Work: Study & NN training with ZENRIN JP road Bigdata
ZENRIN R&D Laboratories 35
• Totai: 800M JPG geo-tagged
images of JP roads
• 2Mkm drive. Photo taken
every 2.Lm
• Toii free narrow roads
nationwide.
360 Camera
Panoramic Photo
Traffic Signs
ZENRIN CO., LTD. All Rights Reserved.©
Conciusion
• GPUs and AIs can acceierate and automate the creation
and updating of HD maps
• Optimizing the pipeiine to Japan roads with ZENRIN’s
mapping experience, knowiedge and big data
�Stiii have many technicai chaiienges. Looking forward to
discussing soiutions and coiiaborating with you!
ZENRIN R&D Laboratories 37
ZENRIN CO., LTD. All Rights Reserved.©
Acknowiedgement to:
Naoyuki Yura, Katsuya Misu, Kenichi Yoshii & Leonid Terenin
NVIDIA Japan
ZENRIN R&D Laboratories 38
ZENRIN CO., LTD. All Rights Reserved.©
Thank YouShigeyuki (Shawn) Iwata
ZENRIN R&D Laboratories 39