design 3d scenes for automated driving simulation · design scenes – roads, lane markings –...
TRANSCRIPT
1© 2020 The MathWorks, Inc.
Design 3D Scenes for Automated Driving Simulation
Shashank Sharma
Peter Fryscak
2
Some common questions from automated driving engineers
How can I
analyze & synthesize
scenarios?
How can I
design & deploy
algorithms?
How can I
integrate & test
systems?
Control
Planning
Perception
3
Some common questions from automated driving engineers
How can I
analyze & synthesize
scenarios?
How can I
design & deploy
algorithms?
How can I
integrate & test
systems?
Control
Planning
Perception
4
Synthetic scenario workflows
Real-world data workflows
Analyze and synthesize scenarios
Connect Visualize Label
Model actors Model sensors
Enables
open loop
workflows
Enables
open loop and
closed loop
workflowsCreate scenes
5
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
Update 1
6
Update 1
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
7
Design scenes with road, marking, and prop assets
▪ Roads and markings
▪ Traffic signals
▪ Guard rails
▪ Trees
▪ Signs
▪ Elevation data
Assets
RoadRunnerTM
Update 1
8
Design scenes and export to driving simulator
Design
scenes
Export
meshes
Import to
simulatorSimulate
▪ Edit roads
▪ Edit road materials
▪ Add road markings
Exporting to CARLA
RoadRunnerTM
Update 1
9
Design scenes and export to driving simulator
Design
scenes
Export
meshes
Import to
simulatorSimulate
▪ Install plugin
▪ Export from RoadRunner
▪ Import into CARLA/Unreal
Exporting to CARLA
RoadRunnerTM
Update 1
10
Design scenes and export to driving simulator
Design
scenes
Export
meshes
Import to
simulatorSimulate
▪ Move vehicle in automated driving
simulation
▪ Visualize pixels IDs for semantic
segmentation
Exporting to CARLA
RoadRunnerTM
Update 1
11
Design scenes with hundreds of premade assets
▪ Road and highway signs
▪ Traffic signals
▪ Road surface markings
▪ Trees
▪ Barriers
▪ Road damage textures
– Cracks, oil spills
Asset Library
RoadRunnerTM Asset Library
Update 1
12
Export scenes to file formats and driving simulators
▪ Export to common file formats for
use in third-party applications
– Filmbox (.fbx), OpenDRIVE (.xodr)
– Unreal Engine®, CARLA
– Unity®, LGSVL
– VIRES Virtual Test Drive, Metamoto
– IPG Carmaker, Cognata, Baidu Apollo
– Tesis Dynaware, TaSS PreScan
– Universal Scene Description (USD)
Exporting
RoadRunnerTM
FBX
(meshes)
OpenDRIVE
(semantics)
Update 1
13
Import, visualize, and edit OpenDRIVE files
Import
OpenDRIVEVisualize Edit Export
▪ Validate OpenDRIVE file
▪ Import and visualize
▪ Edit roads and scene
▪ Export to common driving simulator
formats (including OpenDRIVE)
Importing OpenDRIVE Files
RoadRunnerTM
Update 1
14
Get started designing scenes by watching tutorial videos
Update 1
▪ Add roads and junctions
▪ Add lane markings
▪ Add traffic signals
▪ Add traffic signs
https://www.mathworks.com/videos/sear
ch.html?q=roadrunner
RoadRunnerTM
15
Update 1
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
16
Synthesize scenarios to test algorithms and systems
Scenes Cuboid
Testing Controls, sensor fusion, planning
Sensing Probabilistic vision (detection list)
Probabilistic lane (detection list)
Probabilistic radar (detection list)
Lidar (point cloud)
17
Synthesize scenarios to test algorithms and systems
Scenes Cuboid Unreal Engine
Testing Controls, sensor fusion, planning Controls, sensor fusion, planning, perception
Sensing Probabilistic vision (detection list)
Probabilistic lane (detection list)
Probabilistic radar (detection list)
Lidar (point cloud)
Monocular camera (image, labels, depth)
Fisheye camera (image)
Probabilistic radar (detection list)
Lidar (point cloud)
18
Graphically author scenarios with Driving Scenario Designer
▪ Design scenes
– Roads, Lane markings
– Pre-built scenes (Euro NCAP)
▪ Import roads
– OpenDRIVE, HERE HD Live Map
▪ Add actors
– Size, Radar cross-section (RCS)
– Trajectories
▪ Export scenarios
– MATLAB code, Simulink model
Driving Scenario Designer
Automated Driving ToolboxTM
Updated
19
Synthesize driving scenarios from recorded data
▪ Import roads from OpenDRIVE
▪ Create ego trajectory from GPS
▪ Create target trajectories object lists
Scenario Generation from Recorded
Vehicle Data
Automated Driving ToolboxTM
Visualize
video
Import
roads
Create ego
trajectory
Create target
trajectories
Simulate
scenario
20
Design with cuboid and Unreal Engine driving scenarios
Trajectories
Specify Vehicle Trajectories
for 3D Simulation
Automated Driving ToolboxTM
Scenes
Cuboid Versions of 3D Simulation
Scenes in Driving Scenario Designer
Automated Driving ToolboxTM
Customize scenes
Customize 3D Scenes for
Automated Driving
Automated Driving ToolboxTM
21
Integrate RoadRunner with MATLAB and Simulink workflows
Unreal EngineRoadRunner
RoadRunner
scene
Export to
Unreal Engine(.FBX, .XML)
Import and
configure game
Export to
OpenDRIVE(.XODR)
MATLAB & Simulink
Simulink
model
Connect to
game
Add
vehicles
Import to
driving scenario
22
Unreal EngineRoadRunner MATLAB & Simulink
Integrate RoadRunner with MATLAB and Simulink workflows
Export to
Unreal Engine(.FBX, .XML)
Simulink
model
Import and
configure game
Connect to
game
Add
vehicles
Export to
OpenDRIVE(.XODR)
Import to
driving scenario
RoadRunner
scene
23
Create and explore scene in RoadRunner
Create
Roads
Add
guard rails
Add
trees
Change
lane markings
Change
road textures
▪ Design scene to test behavior of
lane following algorithm under
different road conditions
– Shadows, guard rails
– Lane markings, road textures
– Target vehicles
24
Unreal EngineRoadRunner MATLAB & Simulink
Integrate RoadRunner with MATLAB and Simulink workflows
RoadRunner
scene
Export to
Unreal Engine(.FBX, .XML)
Simulink
model
Import and
configure game
Connect to
game
Add
vehicles
Export to
OpenDRIVE(.XODR)
Import to
driving scenario
25
Integrate RoadRunner scene with driving scenario
Export scene to
OpenDRIVE
Import to
driving scenario
Add
vehiclesSimulate
▪ Export scene from RoadRunner to
OpenDRIVE file (.XODR)
26
Integrate RoadRunner scene with driving scenario
Export scene to
OpenDRIVE
Import to
driving scenario
Add
vehiclesSimulate
▪ Create scenario with Driving
Scenario Designer App
▪ Import road network from
OpenDRIVE
▪ Add vehicle
▪ Specify trajectory waypoints and
speed
27
Unreal EngineRoadRunner MATLAB & Simulink
Integrate RoadRunner with MATLAB and Simulink workflows
RoadRunner
scene
Add
vehicles
Export to
OpenDRIVE(.XODR)
Import to
driving scenario
Simulink
model
Export to
Unreal Engine(.FBX, .XML)
Import and
configure game
Connect to
game
28
Integrate RoadRunner scene with Unreal Engine scenario
▪ Export scene from RoadRunner to Unreal
Engine game (.FBX & .XML)
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
29
Integrate RoadRunner scene with Unreal Engine scenario
▪ Import game into Unreal Engine
▪ Add plugin to communicate with Simulink
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
30
Integrate RoadRunner scene with Unreal Engine scenario
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
▪ Create Simulink model
▪ Configure connection to Unreal Engine
▪ Read driving scenario
▪ Add vehicle
▪ Add sensors
Scenario
reader
Sensors
VehicleGame
configuration
31
Integrate RoadRunner scene with Unreal Engine scenario
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
▪ Create Simulink model
▪ Configure connection to Unreal Engine
▪ Read driving scenario
▪ Add vehicle
▪ Add sensors
32
Model sensors in Unreal Engine driving scenarios
▪ Monocular camera
– Image
– Depth
– Labels
▪ Fisheye camera image
▪ Lidar point cloud
▪ Radar detections
3D Simulation for Automated Driving
Automated Driving ToolboxTM
Updated
Monocular image Lidar
Radar
FisheyeLabels
Depth
33
Integrate RoadRunner scene with lane following scenario
Explore highway lane
following system
Integrate with driving scenario
and Unreal EngineSimulate
▪ Integrate algorithms
– Vision detection
– Sensor fusion
– Decision logic and controls
– Use MathWorks solutions or your ownScenario
&
sensors
Vehicle
dynamics
Vision
detection
Sensor
fusion
Decision
logic
& controls
34
Integrate RoadRunner scene with lane following scenario
Explore highway lane
following system
Integrate with driving scenario
and Unreal EngineSimulate
▪ Create driving scenario from scene
▪ Read driving scenario
▪ Configure Unreal Engine game
▪ Add ego and target vehicles
▪ Connect sensors to ego vehicle
Scenario
Reader
Game
configuration
Vehicles
Sensors
35
Integrate RoadRunner scene with lane following scenario
Explore highway lane
following system
Integrate with driving scenario
and Unreal EngineSimulate
▪ Explore system behavior under different
road conditions
– Shadows, guard rails
– Lane markings, road textures
– Target vehicles
36
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
Update 1
37
Let us know how we can help you…
analyze & synthesize
scenarios
design & deploy
algorithms
integrate & test
systems
Control
Planning
Perception
Contact us : [email protected]