software enablement blue box v2.0: adas sensor …

NXP and the NXP logo are trademarks of NXP B.V. All other product or service names are the property

of their respective owners. © 2017 NXP B.V.

PUBLIC

FIELD SOFTWARE SOLUTIONS

REV 0.5

MARK DOUGLAS

SOFTWARE ENABLEMENT

BLUE BOX V2.0: ADAS SENSOR

FUSION PLATFORM

AMF-AUT-T2666 | JUNE 2017

PUBLIC 1

AGENDA

• ADAS sensor fusion needs

• The Blue Box v2.0

• Linux enablement

• Discussion of middleware

• Level 2/3 ADAS Proof of Concept (POC)

• Forthcoming NXP ADAS development kit

PUBLIC 2

Goals of this session:

• Understanding of ADAS sensor fusion needs

• What the Blue Box v2.0 can support

• Linux enablement for this ADAS platform

• What Middleware/HALs run on it

• 3rd Party and NXP POC running level 2+

• Forthcoming NXP ADAS development kit

PUBLIC 3

Access/ Door/.. Control

PUBLIC 4

ADAS Sensor Fusion needs

01.

PUBLIC 5

Levels of Autonomy

Responsibility for safe operation Control of complete vehicle Control of steering Control of vehicle speed

LEVEL 1

Driver Assistance

LEVEL 2

Partial Automation

LEVEL 3

Conditional

Automation

• Adaptive cruise control

• Automatic braking

• Lane keeping

• Partial automated parking

• Traffic jam assistance

• Emergency brake with steer

• Semi autonomous:

−Highway chauffeur

−Self parking

• Human driver can regain

control

ADAS

Driver

Vehicle or

Driver

Vehicle

Driver

Vehicle

LEVEL 4

High Automation

• Autonomous driving in

some driving modes

• Human driver may not

respond to request to

intervene

Driver

Vehicle

Safe Autonomous System

LEVEL 5

Full Automation

• Fully autonomous under

all driving modes

• Human driver not

expected to respond to

request to intervene

Driver

Vehicle

SAE. (2014). AUTOMATED DRIVING LEVELS OF DRIVING AUTOMATION. SAE INTERNATIONAL STANDARD J3016.

Feet Off

Hands Off

Eyes Off

Brain Off

Here is where we are going

to focus in this presentation

http://www.sae.org/misc/pdfs/automated_driving.pdf

PUBLIC 6

Typically what is needed for ADAS

What is ADAS? –“Advanced Driver Assist System” that includes

− Level 2/3 autonomy or what we are calling “Level 2+”

Examples are highway autonomy:

lane detection lane centering intelligent lane change

intelligent cruise emergency breaking

− Forward facing type applications

− Intelligent sensors pre-processed data (the data seen is not raw)

− Combines mostly: Radar and Camera data (can support LiDAR)

− Not that complex in the sensor fusion domain due to pre-processed data

PUBLIC 7

ADAS Sensor needsOur focus is on the forward-facing

Camera and RADAR sensors

PUBLIC 8

Basic ADAS Sensor needs (aggregation flow)

PUBLIC 9

Typical ADAS software Sensor needs

• hw/sw platform to accommodate Level 2/3 type autonomy

− Network (Ethernet), MIPI, CAN and other interfaces

− Functionally Safe + high performance fusion processing

• Operating System enablement for I/O

• HAL middleware to isolate your fusion layers from the O/S and other target

specifics

− bringing in messaging and other features

• Sensor hander nodes and fusion

• Connection to the drive by wire or vehicle domain interface(s)

PUBLIC 10

The Blue Box Mini: ADAS Dev Platform

02.

PUBLIC 11

BlueBox v2.0 – standard case, sheet metal with fan

PUBLIC 12

NXP BlueBox 2.0 (Mini)

• A prototyping platform for Level 2/3 ADAS applications

• Covers scaled software support and processing needs

− High performance compute with 16GB DDR4 and 250GB SSD

− ASIL-B compute, automotive interfaces, with vision acceleration

− ASIL-D subsystem, with dedicated interfaces

• Numerous interfaces

− Ethernet 100M/1G/10G, SFP+, 8*100BASE-T1,

• Ships with near desktop Linux for ease of use

− Allows for ease of porting from PC platforms

− Also Ubuntu-like environment for the LS2084A (possibly for the S32V)

PUBLIC 13

Automotive + Ethernet I/O

• Separated I/O for ASIL-B/ASIL-D

domains

• Automotive Ethernet partitioned through

switches

• Desktop convenience connectors for

convenience Ethernets

• 10G Ethernet for logging purposes

• SFP+ cage for fiber based system links

PUBLIC 14

Infrastructure

• Large Boot Flashes

• Standard JTAG debug accessibility

• Slots for PCIe x4 and MIPI signals

• Console UARTs available on front panel

• MicroSD cards in latching slots

• Convenience connectors for prototyping

PUBLIC 15

Classic SDKs + ADAS support SW

03.

PUBLIC 16

What is a Linux SDK/BSP

Device DriversU-Boot

Linux Kernel

User’s

Apps

PUBLIC 17

Overview of sensor fusion software stack ADAS

• What is needed for a software platform to get started?

COMMUNICATION

UART

ENET

CAN

PCIEXPRESS

I2C

DSPI

STORAGE

SD

EMMC

QSPI

GRAPHICS

DCU

FRAMEBUFFER

GPU

VIU

ENCODERS

SECURITY

CRYPTO

SECUREBOOT

BLUEBOX Mini (LS2084A and S32V)

SOC

TIMERS

PINMUX

GPIO

CLOCKS

INTERRUPTS

WDG

MIDDLEWARE

ROS or 3rd Party Middleware

TOOLS

Sensor Fusion SAMPLES3rd party and NXP APPLICATIONS

ICC

ROOTFS (Ubuntu rootfs) UTILITIES

LIBRARIES

DOCUMENTATION

BOOTLOADERSUBOOT

UEFI

PUBLIC 18

ADAS Middleware Services

04.

PUBLIC 19

Why are these middleware stacks needed for ADAS?

Operating Systems do not usually include these features:

• Sensor node messaging

− Communication bridge between sensor hander nodes

− Inter-node or process communications

• Time synchronization

− Relative sensor timing

• Distributed and coordinated functionality

• Addition sensor data synchronization with algorithm development tools

• (key feature) Application Hardware Abstraction Layer (HAL) which allows for fusion mobility

PUBLIC 20

Robot Operating System (ROS) Features

Node 1

Laser Scanning

Node 2:

Map Building

Node 3:

Planning

(C) File-system level: ROS Tools for managing source code, build instructions,

and message definitions.

Node 4 (B) Computation Graph: Peer-to-Peer Network of

ROS nodes (processes).

Node 5

Node 6Node7

Carnegie Mellon

(A) ROS Community: ROS Distributions, Repositories

PUBLIC 21

Key ROS Features (continued)

• publish/subscribe anonymous message passing

− communication between distributed nodes via the anonymous publish/subscribe mechanism

− message interfaces is defined in the message IDL (Interface Description Language)

• recording and playback of messages

− publish/subscribe system is anonymous and asynchronous

− the data can be easily captured and replayed without any changes to code

• request/response remote procedure calls

− synchronous request/response interactions between processes

• distributed parameter system

− share configuration information

PUBLIC 22

LEVEL 2+ AUTONOMY POC

05.

PUBLIC 23


NXP/3rd Party POC - Demo

Highway Level 2+

Features:

highway speeds: Lane centering, Adaptive cruise, Traffic jam assist & Intelligent lane

change

All processing done on the A72 cores NXP LS2084ARDB target under 8-way SMP Linux

Sensors: ESR 2.5, Mobileye 660, and the Vehicle DBW interface

All processed data was sent from the 2088 to the PC via Ethernet

PUBLIC 24

NXP & 3rd Party: software stack & demo

3rd party ARMv8-based sensor nodes

SMP Linux + Ubuntu root file system + HAL support packages

LS2084ARDB Target

ROS Kinetic (Master)

NovaTel

ProPak 6

EthernetEthernetEthernet

ROS Kinetic (Slave)

PC for displays (HMI) Ethernet EthernetUSB

ROS_MASTER_URI

multi machine

connection over Ethernet

3rd party ROS sensor nodes


Kvaser single

CAN

USB to CAN

Vehicle Drive by Wire interface

Delphi ESR 2.5

Radar

Mobileye

660

USB Hub

Kvaser Single CAN

channel device for each

USB to CAN

Delphi SRR2 Radars

NOTE: Integrated

into the Vehicle

Ethernet

Delphi SRR2 Radars

NOTE: Integrated

into the Vehicle

Displays

PUBLIC 25

NXP Blue Box Mini

Moving the Highway Software to the Blue Box Mini

LS2084A

3rd party ARMv8-based sensor fusion nodes

SMP Linux + Ubuntu root file system +

HAL support packages

ROS Kinetic

USB

Ethernet

P0

Ethernet

P2

Ethernet

P3

PCIe Host

S32V SoC on BBM Target PCIe End Point seen as

a network interface

Single

Camera MIPI

(replacing

MobileEye

Virtual Ethernet

Interface (PCIe0)

Ubuntu Linux

User-Space

Ubuntu Linux kernel-space

MIPI

Driver

3rd party ADAS

ROS Kinetic

3rd party ARMv8-based camera nodeVirtual Ethernet

Interface (PCIe0)

NovaTel

ProPak 6

USB Hub Kvaser USB to

CAN

Drive by wire RADAR

3rd party ADAS APIs accessed to

3rd party nodes for lane and

vehicle detection

ROS multi-machine messaging is

used to share camera node results

vEthernet provides

standard API

PUBLIC 26

Virtual Ethernet – Use Case

• BlueBox v2.0

• Communication between LS2

and S32V inside BlueBox

• Standard endpoint

configuration: “ifconfig”

• Ping test

• ROS multi-machine test

$ ifconfig pcie0 192.168.1.1 $ ifconfig pcie0 192.168.1.2

LS2 S32V

Linux LinuxPCIe

Communication

$ ping 192.168.1.2

PING 192.168.1.2 56(84) bytes of data.

64 bytes from 192.168.1.2: time=0.34 ms

64 bytes from 192.168.1.2: time=0.35 ms

PUBLIC 27

Virtual Ethernet

• Standard API

• Simplify the inter-communication

specification

• Ease of use using standard

network socket

• No difference than standard

ETH interfaceIPFC

ARM/

PowerPC

Data-link abstraction layer

USB SPIUARTShared

mem

MCAPIVirtual ETH

TCP/IP stack

ETH

vSomeIP/ OpenSSL

Applications

PUBLIC 28

Future Integration of the research

development platform: possible

use-case options

06.

PUBLIC 29

Distributive Use Cases Multi-SoC sensor handlers controlled across networked SoCs

3rd party ARMv8-based

Master sensor nodes


LS2/LS1 Target/other ARM-64

USB

USB to CAN


PCIe

middleware (Master)

Ethernet Ethernet


Slave sensor nodes


USBPCIe

middleware (Slave)

Ethernet Ethernet


Slave sensor nodes


USBPCIe

middleware (Slave)

Ethernet Ethernet


Slave sensor nodes


USBPCIe

middleware (Slave)

Ethernet Ethernet

LS2/LS1 Target/other ARM-64 LS2/LS1 Target/other ARM-64 LS2/LS1 Target/other ARM-64

3rd party sensor node management across the SoC-based ECUs

Key #1 management of sensor handlers across networked

ECUs: management node can enable/disable selective

monitors

sensors sensors sensors sensors

USB to CAN USB to CAN USB to CAN

Key #2: multi-monitoring sensor nodes for maximum up-

time and collaboration

Key #3 field up-grade management with

max up-time: monitor node manages

application upgrades

PUBLIC 30

Redundant sensor monitor exampleTwo independent modules operating on the single Ethernet sensor data



Master sensor nodes


USBPCIe

ROS 2.0 (Master)

Ethernet Ethernet


Slave sensor nodes


USBPCIe

ROS 2.0 (slave)

Ethernet Ethernet

LS2/LS1 Target LS2/LS1 Target

sensorUSB to CAN USB to CAN

Sensor X module instance 1 Sensor X module instance 2

Sensor X module

fusion/manager

Taskset = 1 on physical

cpu 1 on target 1

One logical or

physical interface

One of two

(2) sensor

modules

Taskset = 3 on physical

cpu 3 on target 2QoS Messaging

Other 3rd party

and ROS v2.0 can

support this

PUBLIC 31

Production Use Case Platform: Key stack software replaced with auto qualify-able stacks


Master sensor nodes


LS2/LS1 Target/other ARM-64

USB

USB to CAN


PCIe

middleware (Master)

Ethernet Ethernet


Slave sensor nodes


USBPCIe

middleware (Slave)

Ethernet Ethernet


Slave sensor nodes


USBPCIe

Middleware (Slave)

Ethernet Ethernet


Slave sensor nodes


USBPCIe

Middleware (Slave)

Ethernet Ethernet

LS2/LS1 Target/other ARM-64 LS2/LS1 Target/other ARM-64 LS2/LS1 Target/other ARM-64

sensor node management across the SoC-based devices

sensors sensors sensors sensors

USB to CAN USB to CAN USB to CAN

Key: Linux replaced by an Auto Qualified O/S

PUBLIC 32

ADAS Development Kit

07.

PUBLIC 33

Fully Integrated SW/HW Platform: Research to Production

Autonomous Solutions

Comprehensive NXP Platform

Key Value Points:

Start Fusion Development Immediately

Complete Software Solutions

No Domain Limitations

Sensors Used in Fusion

Integrated Middleware and Stacks

Optimized BSP Components

Production Grade Commercial Software

Customer Focus on Fusion Integration

Leveraging Examples and Tools

One Stop Shop!

Reducin

g D

evelo

pm

en

t T

ime a

nd C

ost

Accele

rate

d R

esearc

h to

Pro

ductio

n

Customer Development Fusion

Drivers

AV Application Tools

Autonomous HAL

AV IPC Build Tools

Kernel Boot Firmware User Land

Example Fusion Applications

PUBLIC 34

Fully Integrated SW/HW Platform

12

3

DEVELOPMENT

• Use Full Dev Kit form NXP

• Focus on Fusion and Applications

Test and Experiment • All invested coding

directly transferable to production

DELIVERYOnly focus on application delivery

Drivers

AV Application Tools

Autonomous HAL

AV IPCBuild Tools

KernelBoot

FirmwareUser Land

Customer Development Fusion

Example Fusion Applications

PUBLIC 35


NXP Autonomous Development Platform: goals

• Deliver a complete ADAS Stack on the NXP target

• Get the customer up and running in under 2 hours

• Deliver in Phases with a usable “roll-out” offering

− Provide a hardware/software platform with O/S, middleware and example code for sensor

development.

• All that is needed to get a potential sensor integrator and ADAS system going.

• Based on an actual demo running in an actual vehicle

• 3 sensors with driver-by-wire actuation interface

− RADAR, Camera and LiDAR

PUBLIC 36

F A

B

CD

E

Start Immediately with Sensor Fusion Kit

Experiment with Sensor Domains on the Bench

Move your Software to Vehicle Swap non-production SW to Production SW!

Keep ALL work including HAL

Test with Production

Platform

Deliver to Production!

Au

ton

om

ou

s Deve

lop

me

nt P

roce

ss

PUBLIC 37

For more information please contact:

Fernanda Prata

Account Manager

[email protected]

(408) 613 5355

Adrian Koh, PhD

[email protected]

(408) 821 5469

mailto:[email protected]

mailto:[email protected]

PUBLIC 39

Back-Up Slides

08.

PUBLIC 40

ADAS Sensor needs (external)

PUBLIC 41

NXP Solution for OS

Prototype or POC - OS Production OS

Purpose Reference/Prototyping/Fast

Enablement

Runs in the car

Target Disty Market, Tier1s or Research Tier 1s

Development Internal External (through partnership)

Pricing Free Commercial

Certification N/A ISO26262 Safety

Automotive SPICE

Examples Linux QNX

GHS

PUBLIC 42

72-b

it D

DR

4

Me

mo

ry

Co

ntr

oll

er

1 M

B

Pla

tfo

rm

Ca

ch

e

Performance

• ARM A72 x 8 @ 2.0 GHz

• 72K DMIPS

• SpecInt2k6 – 14, Rate -84

• Neon SIMD in all CPUs

• 2x72b (including ECC) DDR4 up to 2.1GT/s

• 33GB/s memory BW

• High Speed IO

• Multiple PCIe Gen3 controllers

• Multiple Ethernet MACs (up to 10G)

Auto quality

• AEC Q100 Grade 3 (105C Tj)

• 15 years product longevity

• ZD-like approach to reduce risk of DPPM or Life failures

• Expected Operating Life fail rate <10 FIT

• Mission Profile: 10 years, 90C Tj-effective

Security

• 20Gbps Crypto Acceleration

• MACSEC, IPsec, SSL

• Trust Architecture

• Secure Boot

• Secure Debug

• Secure Storage

• Tamper Detection

• HW Enforced Partitioning

• ARM Trust Zone

Functional Safety

• Target ASIL-B

• ECC protected memories

• Fault localization, containment and recovery

• Soft lockstep with determinism

• Excellent support for virtualization, containerization

1MB Banked L2

ARM

A72

ARM

A72

Interconnect

72-b

it D

DR

4

Me

mo

ry

Co

ntr

oll

er

SA

TA

3S

AT

A3

x8 G

en3 P

EX

x4 G

en3 P

EX

x4 G

en3 P

EX

x8 G

en3 P

EX

Queue Manager

Buffer Manager

SEC – 20G

DCE – 20G

Secure Boot

Trust Zone

Power Mgt

SD/eMMC

2x DUART

4x I2C

SPI, GPIO, JTAG

2x USB3.0 + PHY

SERDES 16 lanes @ up to 10GHz

Wire Rate IO Processor

2MB Packet Buffer

8x1/10 + 8x1 Ethernet MACs

L2 Switching

Process & Package• 28HPM, ~40W Thermal Max @ 105C

• 37.5 x 37.5 mm, lidded FCBGA, 1mm pitch, 1292 pins

1MB Banked L2

ARM

A72

ARM

A72

1MB Banked L2

ARM

A72

ARM

A72

1MB Banked L2

ARM

A72

ARM

A72

PME – 10G

Samples: Now

Telecom Production: Q3 2017

AEC-Q100 Grade 3 Qual: Q1 2018

Auto Production: Q1 2018

QorIQ Layerscape LS2084A

PUBLIC 43

Some Hardware Abstraction Layers (HALs) “middleware” and

Highly Automated Driving (HAD) Frameworks

• Robot Operating System: ROS

• PolySync run-time

• Intempora: RTmaps run-time

• Elektrobit: EB-robinos

PUBLIC 44


What does PolySync run-time bring to “Autonomy”

Features needed that application level developers would need to be responsible for:

− “a set of services common across autonomous driving applications”

− 3 types of inter-process communications (IPC)

− Hardware abstraction, Host abstraction, Time services, Coordinate frame transformation, Record and replay, Diagnostics & Analytics

PUBLIC 45


ROS features Vs. PolySync

• How does ROS compare to PolySync?

− does much of the same thing

IPC, recording/playback, request/response remote calls and distributive

system

Differences

− Open Source project

− BSD licensed

− alternative to a propriety middleware like PolySync and IP licensing

− large community of shared example code

PUBLIC 46

dSPACE + Intempora RTMaps

• Recording, synchronization

and playback of data from

numerous sensors and

communication buses

• Prototyping, testing and

benchmarking of perception

and data fusion algorithms

Leading producer of

engineering tools for developing

and testing mechatronic control

systems.

PUBLIC 47

Trajectory Control

Longitudinal Control

Lateral Control

Positioning

Object Fusion

Grid Fusion

Behavior

Road and Lane Fusion

Vehicle Database

Fun

ctio

n S

pe

cifi

c V

iew

s

Ve

hic

le A

bst

ract

ion

-Se

nso

rs

Ve

hic

le A

bst

ract

ion

-A

ctu

ato

rs

MotionManagement

Safety Management

Behavior

Behavior

Situative Behavior ArbitrationSensorData Fusion

HMI Management

SituationAnalysis

SituationAnalysis

SituationAnalysis

PathPlanning

PathPlanning

PathPlanning

EB robinos - DNA for automated driving

Key Features:

• architecture for ADAS up to SAE

level 5

• runs on AUTOSAR

• central ECU as well as distributed

systems

• incorporates existing or new,

customer or third party subsystems

Open robinos

• specifies a reference platform for

automated driving up to Level 5

(SAE)• architecture

• interfaces

• data flow

• control mechanisms

• software modules

• functional safety aspects

• freely available and licensed as

Creative Commons

• Available for download on

www.eb-robinos.com

EB robinos

• Lean on to the open robinos

specification

• provides software modules• for prototyping

in EB Assist ADTF

• for rapid embedding

on AUTOSAR / Linux for NXP

BlueBox

• for production

on vehicle ECU

• Prepared for functional safety

relevant systems

• EB robinos: EB‘s Autonomous Driving Software Components

PUBLIC 48


NXP Autonomous Development Platform: goals continued

• Customer will receive all components from AutonomouStuff in one package

• Evaluation support within the provided SW blocks will be provided by NXP

− Beyond demonstrator platform customization will need 3rd party involvement

• 30-60 evaluation with follow-up status

• 3rd party involvement

software enablement blue box v2.0: adas sensor …

Documents