hadoop and friends as key enabler of the ioe - continental's dynamic ehorizon

Bitte decken Sie die schraffierte Fläche mit einem Bild ab.Please cover the shaded area with a picture.

(24,4 x 7,6 cm)

Hadoop and Friends as Key Enabler of the IoE – Continental’s Dynamic eHorizonDr. Thomas Beer, Continental AG (Interior Electronics Solutions)

www.continental-corporation.comInterior Division

Public

Hadoop and Friends as Key Enabler of the IoEContinental’s Dynamic eHorizon

Challenges in Today’s MobilityComfort

Public


Challenges in Today’s Mobility Efficiency

Public


Challenges in Today’s MobilitySafety

Public

Hadoop and Friends as Key Enabler of the IoEContinental’s Dynamic eHorizon Hadoop Summit, Dublin - 13th April, 2016

5Dr. Thomas Beer © Continental AG

Automated DrivingA gradual introduction

Public



EfficiencyComfort Safety

On the Road to 2025 …

Hadoop Summit, Dublin - 13th April, 2016

› Comfort: e.g. Cruise Control

› Efficiency: e.g. Traffic Light Assist

› Safety: e.g. Hazard Warning

Public



Public


The Dynamic eHorizon ….

Hadoop Summit, Dublin - 13th April, 20168Dr. Thomas Beer © Continental AG

…is the Extension to Everything Beyond the Vehicle’s Sensor Range

Vehicle-Sensors0-300m

… is Connected via the Continental Backend Platform

Public


Dynamic eHorizon – Key Players


Continental Backend Platform

Public


Continental Backend Platform – Core Components


CC

CC

CC

Continental Backend Platform

Secure VGW

Data & Map Services

› eHorizon Cloud Connector (CC)

› Vehicle Gateway’s counterpart inside the vehicle

› Services

› Data services: Analytics, data ingest, data storage, long-term monitoring, …

› Map services: map matching, map updates, map enhancement, …

› Secure Vehicle Gateway (sVGW)

› Handles communication with vehicles (collection & distribution services)

› AAA (Authentication, Authorization, Accounting)

› Data En-/Decryption

Public


Example: Learning Map


Public



Public


Learning Map – System Overview


SecureVehicle

GatewayVehicle

Data Services

Map Services

Public


Secure Vehicle Gateway

Ingestion

«generic»Cental Landing Zone

VGW :CoBePaKafkaConnector

BackendDatalake (BEDL)

«generic»PDBValidation RawdataStorage

«gene...KafkaBEDL Connector

«gene...HDFS

Sequencefile

Data Processing

DeseralizePDB

«refinesGeneric»LM

Analytics

Clustering Data Fusion

View Creation

TileStorage

Map Services

Data Services – Functional View

Hadoop Summit, Dublin - 13th April, 2016 14Dr. Thomas Beer © Continental AG

Protocol Buffers

Public


Hadoop Cluster (simplified)

Data Services – Execution View


Spark

HDFS

Spark

Storm

HDFS

Kafka

Storm

Spark

HDFS

Kafka

Storm

HDFS

› self-operated

› hosted in the “cloud” (IaaS)

Public


We See Improvement Potential Within Our Current Solution Architecture


Batch Computing

Ingesting

Storage

PaaS (simplified)Hadoop Cluster (simplified)

Spark

HDFS

Spark

Storm

HDFS

Kafka

Storm

Spark

HDFS

Kafka

Storm

HDFS

Public


We See Improvement Potential Regarding …


Operations Costs Flexibility

Public


Secure Vehicle Gateway

Ingestion

«generic»Cental Landing Zone

VGW :CoBePaKafkaConnector

BackendDatalake (BEDL)

«generic»PDBValidation RawdataStorage

Data Processing

DeseralizePDB

«refinesGeneric»LM

Analytics

Clustering Data Fusion

View Creation

TileStorage

Map Services

Solution Architecture Refined – PaaS Style

Hadoop Summit, Dublin - 13th April, 2016 18Dr. Thomas Beer © Continental AG

Protocol Buffers

Protocol Buffers

S3

EMR

«gene...KafkaBEDL Connector

«gene...HDFS

Sequencefile

Kinesis

Kinesis

Public


Our first experiences are promising


› Spark jobs could be 100% re-used without modifications

› Zero maintenance of infrastructure for batch jobs

› High flexibility of cluster size for batch jobs

› You don’t have to – but you can

› Not for all – but for specific scenarios (e.g. batch)

PaaS services offer architectural flexibility – especially for batch jobs.We will further analyze the opportunities.

Public


› take care of your metadata

› selection of Hadoop distributions – do not evaluate – do it!

› no “one size fits all” solution – be prepared for change

› automate the deployment and be prepared for any extension (storage, compute, etc.) in your cluster - you have to do the work quite often

› strong interdisciplinary collaboration: developers, ops, mathematician, domain experts, legal

Lessons Learned (non technical)


Public


My Recommendation Is:Don’t Do It Yourself – Let’s Get In Touch!


Public


› Department: Pre Development / Research & Development

› Location: Regensburg (Germany), Troy (USA)

› Team: Developer, Data Scientist, SW-Architect, Domain-/Vehicle experts…

Continental Backend Platform – The Team


Public


ContinentalAlready a software company


1990 2000 2010 2014

01,6

10,2

>12

Sales generated with digitalized technologies (€ billion) Digitalization at Continental today

software developersContinental is one of the major "software firms" in Germany

60% of automotive sales

generated with electronics,

sensors, and software

€ billion sales generated with digitalized technologies

of innovationsin automotive thanks to software functionality

Public


Thank you for your attention.


hadoop and friends as key enabler of the ioe - continental's dynamic ehorizon

Technology