iot arhitecture, database and big data - aceiot · gateway –centric iot cloud (1/2) iot...
TRANSCRIPT
IoT Arhitecture, database and big
data
23th- 27th April,
2018
Rwanda
Abdur Rahim
Open IoT
FBK CREATE-NET
IoT Architecture and platform
Platform overview 3
Two main platforms
IoT Gateway platform (Gateway-Centric
Cloud-Bring Cloud functionalities in Gateway)
IoT integrated Cloud/PaaS platform (Cloud-
centric IoT- Bring IoT functionalities in Cloud)
Gateway – Centric IoT Cloud (1/2)
IoT infrastructure will provide the opportunities to take services,
workloads, applications and large amounts of data and deliver
it all to the edge of the network.
Processing and storage of data close to users/near to sources
To distribute data to move it closer to the end-users to eliminate latency,
numerous hop, and support mobile computing and data streaming
Creating dense geographical distribution
Gateway- Centric IoT Cloud (2/2)
This approach are useful when service is provisioned from the
data coming from same location
Highly real time applications
Reduce network traffic and cost
Supporting end-users security
Data process and service execute locally (distributed cloud
processing, sub-work flow, data aggregation locally)
What is gateway platform?7
IoT Gateway centric architecture 8
Driving the functionalities to the edge
Intelligent gateway9
Data acquisition, integration and rules activation,
providing dynamic intelligence at the edge
Industrial IoT gateway10
Intel IoT Gateway11
Gateway comparison summary 12
https://www.loriot.io/gateways.html
Kura architecture 13
http://eclipse.githu
b.io/kura/doc/intr
o.html
Open source gateway14
Kura and camel integration
https://camel.apache.org/kura.html
Agile micorservice Gateway15
Cloud-centric IoT PaaS platform
Bring IoT data in the cloud
Processing and computing the data and deploy
management tools in cloud
This approach this good if service are provided
among objects located in multiple location
Giving several example (market platform, project
platform..)
Example of IoT platform
Bosch IoT Cloud18
https://www.bosch-si.com/products/
Most known IoT cloud platform19
Architecture
Architecture: functional domains and
components
Functional domain Component Technology choice
Application platform Orchestrator Deis
Execution environments Docker
RAD Node-RED
UI manager Elastic search, Kibana (Freeboard, D3.js)
IoT platform IoT bridge Specific protocols/libraries
Pre-Process FI-WARE CEPHEUS
Sensor registry Not selected yet
Sensor discovery Not selected yet
Big data platform Data streaming Orion
Storage manager MongoDB
Data analytic FlinkML, Mlib
Security & privacy Identity manager OpenAM/Gluu
Authorization manager OpenAM/Gluu
Privacy manager OpenAM/Gluu
Big data platform
orchestrator
App
compileApp
Read
manifest
Request
sensorStart
app
deployIoT PF
Manifest
reader
App Data
stream
Status
manager
Web
GUI
WorkersBig data
runtime
Stream
broker
Analytic
IoT platform
Sensor
finder
Sensor
registry
IoT
brokerIoT
bridge
Subscribe
Raw
data
data
Conn. req.Big
data PF
stream
sensor.
req.
Status
manager
Web
GUI
Pre-
process
PaaS platform
Developer
App +
manifest
WAZIUP Cloud platform
orchestrator
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
Service
create push
deploy
deploy
Local deployment
App
Gateway Sensor
Local
PC
uses
With/Without internet
24
App deploy: local and global
Developer
App +
manifes
t
WAZIUP Cloud platform
orchestrator
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
Service
create push
deploy
deploy
Local deployment
Ap
p
Gateway Sensor
Local
PC
uses
Data flows 1
WAZIUP Cloud platform
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
access
Gateway
Data flow 2
WAZIUP Cloud platform
orchestrator
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
Service
access
deploy
deploy
Data flows 2
WAZIUP Cloud platform
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
access
Gateway
Databases requirements
• Run in gateway (low resources)
• Run in Cloud
• Run in local PC
• Synchronize in best-effort
• Support queries from local app
Big data technologies
• Data processing:
• Spark
• Flink
• Data broker:
• Kafka
• Orion
• Data Mining:
• Spark Mlib
• H20.ai
• Data Storage
• MongoDB
• Apache HBase
Databases
Two basic types
NoSQL
SQL
31
Data model
Key/Value
Memcached, Dynamo
Tabular
Big Table
Document Orieneted
MongoDB, CouchdB, JSON stores
32
NoSQL
A form of database management system
that is non-relational
System are often schema less, avoid joins
& are easy to scale
33
But why Choose NoSQL
Amount of data stored is on the
up and up
The data we store is more
complete than before
All the data is need to be easy to
be able to add/remove servers
without any disruption of services
34
MongoDB
Document base
Schema-less
Highly scalable
Easy replication & sharing
35
Basic of Big data and database
BIG data is always not well understood
Smart Data
Big data: 3V’s (volume)
Volume
Large data size• What does mean size?
– Not gigabytes
– Most likely not a few terabytes
– Possibly not 10’s of terabytes
– Probably 100’s of terabytes
– Definitely petabytes
Big Data: 3V’s (velocity)
Velocity
real-time
near-real time
streaming data flow
Big Data: 3V’s (variety)
Variety diverse data (structure and
unstructured, diverse data models and
query languages, diverse data sources
Some make it 4V’s
IoT in BIG data
IoT presents challenges in combination of all BIG data characteristics (3Vs/4Vs)
Most challenging IoT applications impact both Velocity & Volume and sometimes also Variety (situation and context)
Today..
GE each day gathers 50 million pieces of data from 10 million sensors, off equipment worth $1 trillion
A wearable sensor produces about 55 million data points pro day (challenge for storage), whereas some medical wearable's (like ECG) produce up to 1000 events per second (challenge for real-time processing)
IoT real-time Big data
Real-time Big Data (Fast data)
Real-time Data coming from (mainly)
Monitoring systems (e.g. Nagios)
Sensors
Stationary sensors (environment)
Embedded (e.g. in mobile devices)
Wearable sensors (e.g. HR monitors)
extreme velocity
mobile data streams
extreme resource
constraints
EMERGING
GREAT BUSINESS OPPORTUNITIES
e.g. there should be about 250 Million Wearable Health & Fitness Sensing Devices by 2017.
The market for sports and fitness apps will cross $400 million in 2016
BIG Data is nothing without BIG
business value insight
IoT is a huge opportunity for BIG data
..and opposite
IoT without BiG DATA is first generation IoT 1.0
IoT with BIG DATA is the third generation IoT 3.0 (vision of future world)
=+
IoT BIG Data BIG Value
© 2016 Abdur Rahim Biswas
IoT and Big Data
Content
Big data introduction
IoT big data requirements and platform
IoT big data technologies and tools
IoT BIG data applications
Deep understanding (observe of behavior of many thing””, gain important insight
Health example (understanding the cause of diseases/comorbidities/indicators)
Real-time actionable insight (Real-time analytic, detect and react in real-time)
Health example (real-time fall detection and potential reaction for aging population)
Performance optimization (configuration, energy, health-care)
Health example (Improve overall healthcare efficiency)
Proactive and predictive functional applications
Health example (proactive and prediction identification of diagnostic in healthcare applications (before thing occur)
Deep understanding applications
challenges
This vision boils down to solve multiple challenges:
to store all the events (Velocity & Volume Challenge);
to run analytical queries over the stored events; (Velocity & Volume Challenge)
to perform analytics (data mining and machine learning) over the data to gain insights (Velocity & Volume & Variety Challenge);
Real-time actionable Insights
Real-time detection and action represent multiple challenges
How to make reliable knowledge and decision from BIG Data? (Veracity and Verity)
How to process (real-time process and data interpretation) the streaming/real-time events on the fly (Velocity challenges)
How to store the events in the operational database (Velocity challenges)
How to correlate streaming events with store data in the operational database (velocity and Volume challenges)
Performance optimization challenges
When the “old way” of processing data just doesn’t work effectively
How we store the diverse set of BIG data, from mobile/sensors/server? (much data
How we move that much data
How we extract, load & transform that much data
How we explore and analyze that much data
How we process and get meaningful insights from that much data
Prediction challenges
Context
Long-period of time
IoT Big data platform requirements
CognitiveScalable
Real-time Unified view
Data
sourceData
source
Architecture of IoT Big data platform
Major big data analytic platform
IoT Big data framework- INTEL
Traditional methods Big data
Centralize Distributed
More power More machines
Summarize data Keep all data
Transform and store Transform on demand
Pre-define schema Flexible/no-schema
Move data toward compute Move compute towards data
Less data/more complex algorithms More data/simple algorithms
Philosophical differences of Big data analytic
© 2016 Abdur Rahim Biswas
IoT and Big Data
Content
Big data introduction
IoT big data requirements and platform
IoT big data technologies and tools
BIG data tools
Hadoop ecosystem- well known tools
Hadoop: The disruptive technology at
the core of Big data
Batch and stream processing
Hadoop MapReduce Batch Processing
Storm Streaming Stream Processing
Stream Processing
Handles data at high velocity
If Hadoop is the ocean, streams are the firehose
Processing in near real-time
Storm
Storm
Storm is a distributed data processing system whose processing is based on elements called spouts and bolts
The topology consists out of spouts which are message originators for the rest of the topology
The processing elements in the topology are called bolts which can be interconnected with an internal pub/sub mechanism
Bolts can also deliver their results to other systems such as DBMS’s, legacy systems or applications.
Lambda architecture combine
Complex Architectures Using Many Big
Data Technologies
