designing and implementing your iot solutions with open source
TRANSCRIPT
![Page 1: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/1.jpg)
© 2016 MapR Technologies 1© 2016 MapR Technologies
Designing & Implementing your IOT Solutions with Open Source
Sunil Patil Solutions Architect, MapRSridhar Reddy Director, Solutions Engineering, MapR
2016
![Page 2: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/2.jpg)
© 2016 MapR Technologies 2
Agenda• Use Case• Demo• Layers
– Sensor – Controller– Protocol– Messaging– Computation– Persistence– Dashboard
![Page 3: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/3.jpg)
© 2016 MapR Technologies 3© 2015 MapR Technologies
Use Case – Temperature Monitoring
![Page 4: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/4.jpg)
© 2016 MapR Technologies 4
Use Case – Temperature monitoring dashboard
![Page 5: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/5.jpg)
© 2016 MapR Technologies 5
Do I need big data for IOT ?Sensor reads temperature once every second
• Number of readings per year3600*24*365 = 31,536,000 (31 million)
• Every reading is 100 bytes 31,536,000 * 100 = 3,153,600,000 (3.1 GB)• Every store has 50 aisel(4 sensors) and 6000+ stores
200 * 6000 = 1,200,000 (1.2 million sensors)3 * 200 * 6000 = 3,600,000 (3.6 PetaBytes)
![Page 6: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/6.jpg)
© 2016 MapR Technologies 6© 2015 MapR Technologies
Demo
![Page 7: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/7.jpg)
© 2016 MapR Technologies 7
Demo – Data Flow
![Page 8: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/8.jpg)
© 2016 MapR Technologies 8
Demo – Data Flow
![Page 9: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/9.jpg)
© 2016 MapR Technologies 9
IOT Big Data Reference Architecture
![Page 10: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/10.jpg)
© 2016 MapR Technologies 10© 2015 MapR Technologies
Sensors/Actuators
![Page 11: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/11.jpg)
© 2016 MapR Technologies 11
Why do we need Sensor/Actuator Layer ?Sensor/actuator will be responsible for1. Interacting with physical world
1. Sensor – Measures or detects physical property
1. Thermistor2. Temperature Oscillators
2. Actuator – Move or control mechanism or system
1. LED2. Servo Motor
![Page 12: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/12.jpg)
© 2016 MapR Technologies 12
Temperature Sensors
DS18B20 DHT22
![Page 13: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/13.jpg)
© 2016 MapR Technologies 13
Sensor Pin Layout
DS18B20 DHT22
![Page 14: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/14.jpg)
© 2016 MapR Technologies 14
Actuator Pin Layout
![Page 15: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/15.jpg)
© 2016 MapR Technologies 15
Home Automation
![Page 16: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/16.jpg)
© 2016 MapR Technologies 16
IOT Home AutomationYou can build home automation1. Temp Monitoring – For AC2. Auto turning of lights on and off3. Sprinkler System4. Primitive home security5. Gas detection
![Page 17: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/17.jpg)
© 2016 MapR Technologies 17© 2015 MapR Technologies
Device
![Page 18: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/18.jpg)
© 2016 MapR Technologies 18
Why do we need Device Layer ?The controller layer is responsible for1. Reading data from sensor2. Transform sensor data into
message3. Communicate with IOT Gateway
using suitable protocol4. Listening to message from IOT Gateway5. Activating with actuators based on message from Gateway
![Page 19: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/19.jpg)
© 2016 MapR Technologies 19
Arduino
Arduino Uno NodeMCU V2 –ESP8266
![Page 20: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/20.jpg)
© 2016 MapR Technologies 20
Arduino Application
![Page 21: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/21.jpg)
© 2016 MapR Technologies 21
Install Sensor Library
![Page 22: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/22.jpg)
© 2016 MapR Technologies 22
Temperature sensor – Sensor Library
![Page 23: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/23.jpg)
© 2016 MapR Technologies 23
Temperature sensor – Start Sensor
![Page 24: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/24.jpg)
© 2016 MapR Technologies 24
Arduino – Read temperature
![Page 25: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/25.jpg)
© 2016 MapR Technologies 25
Upload code to Arduino
![Page 26: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/26.jpg)
© 2016 MapR Technologies 26
Raspberry Pi
![Page 27: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/27.jpg)
© 2016 MapR Technologies 27
Raspberry pi - Logic for reading temperature
![Page 28: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/28.jpg)
© 2016 MapR Technologies 28
Raspberry Pi – Development Life Cycle
![Page 29: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/29.jpg)
© 2016 MapR Technologies 29
IOT Device comparison
Arduino Raspberry Pi
Architecture Micro controller Micro Processor
Operating System No Yes
Power Can go to sleep Continuous 5V
Network No Yes
Sensor Analog + digital Digital
Development Environment
Arduino IDE – C code Java, Python,… etc
![Page 30: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/30.jpg)
© 2016 MapR Technologies 30© 2015 MapR Technologies
Protocols
![Page 31: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/31.jpg)
© 2016 MapR Technologies 31
Why do we need Protocol Layer ?The protocol layer is responsible for 1. Communication between IOT Device and IT infrastructure2. The IOT Devices operate in constrained environment
1. Limited processing power2. Limited amount of memory (Less than 100K)3. Network usage
1. Less Reliable network2. Optimized usage of network
4. Operate on battery1. Sleep mode
![Page 32: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/32.jpg)
© 2016 MapR Technologies 32
MQTT – Message Queue Telemetry TransportMQTT is publish/subscribe messaging protocol designed for lightweight M2M communication.• Open: MQTT is OASIS standard created by IBM • Lightweight Client: Client libraries less than 30 KB• Optimized Network usage: Small packet size 2 bytes (header)
– TCP Based. MQTT-SN is UDP based– Easy to parse packets
• Reliable: Three QoS patterns to avoid packet loss on client disconnect– At-most-once – does not survive failures, but is never duplicated– At-least-once – survives connection loss and other failures, but can be duplicated– Exactly-once – survives connection loss and other failures, and is never
duplicated
![Page 33: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/33.jpg)
© 2016 MapR Technologies 33
MQTT vs. HTTPS 1. Performance is 80 times better2. Battery usage 10 times less
http://stephendnicholas.com/archives/1217
![Page 34: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/34.jpg)
© 2016 MapR Technologies 34
MQTT
Device 1
Device 2
Device 3
MQTTBroker
Subscriber 1
Subscriber 2
/store/fremont/aisel1
/store/fremont/aisel2
/store/sanjose/aisel1
/store/fremont/#
/store/#
![Page 35: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/35.jpg)
© 2016 MapR Technologies 35
MQTT – Server & Broker
Subscriber 1
Subscriber 2
/store/fremont/aisel1
/store/fremont/aisel2
/store/sanjose/aisel1
/store/fremont/#
/store/#
![Page 36: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/36.jpg)
© 2016 MapR Technologies 36
MQTT – Include MQTT client library
![Page 37: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/37.jpg)
© 2016 MapR Technologies 37
MQTT – Setup MQTT Connection
![Page 38: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/38.jpg)
© 2016 MapR Technologies 38
Arduino – Send Receive Message
![Page 39: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/39.jpg)
© 2016 MapR Technologies 39
Arduino – Memory usage
![Page 40: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/40.jpg)
© 2016 MapR Technologies 40
CoAP – Constrained Application ProtocolThe CoAP is request/response protocol similar to HTTP + REST• Open: CoAP is IETF standard - RFC 7252 IETF• Lightweight Client: Client libraries less than 30 KB• Optimized Network usage
– Based on UDP (6LowPAN), SMS in addition to TCP– Compact 4 bytes header, compressed– CoAP uses familiar HTTP stuff like Method (Get, Put, Post, Put, Delete), status
codes, URIs, content types/mapping– Can be used to carry different type of data. It uses content type to identify type
of data being carried– HTTP Proxy can be used to map COAP to HTTP
• Reliable: The HTTP proxy takes care of reliability
![Page 41: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/41.jpg)
© 2016 MapR Technologies 41
Arduino – Read Temp. + Publish MQTT Message
http://slideplayer.com/slide/9365664/
![Page 42: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/42.jpg)
© 2016 MapR Technologies 42
COAP
Device1
Device 2
Device 3
HTTPProxy
Subscriber 1
Subscriber 2
/store/fremont/aisel1
/store/fremont/aisel2
/store/sanjose/aisel1 coap://192.168.1.2:5683:/store/sanjose/aisel1
http://192.168.1.2:5683:/store/fremont/store1
![Page 43: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/43.jpg)
© 2016 MapR Technologies 43
Arduino – Read Temp. + Publish COAP Message
![Page 44: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/44.jpg)
© 2016 MapR Technologies 44© 2015 MapR Technologies
Messaging
![Page 45: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/45.jpg)
© 2016 MapR Technologies 45
Why do we need Messaging Layer ?The Messaging layer is responsible for 1. Highly available MQTT
1. Scale MQTT layer horizontally2. Provide high availability
2. Decouple IOT layer from the computation layer
3. Allow multiple consumers to process the IOT data to meet different business requirements
![Page 46: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/46.jpg)
© 2016 MapR Technologies 46
MQTT load balancing – No Kafka
Device
Device
Device
Device
MQTT
MQTT
MQTT
Spark
Spark
Spark
![Page 47: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/47.jpg)
© 2016 MapR Technologies 47
MQTT load balancing - Kafka
Device
Device
Device
Device
MQTT
MQTT
MQTT
Spark
Spark
Spark
Kafka
![Page 48: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/48.jpg)
© 2016 MapR Technologies 48
MQTT Kafka Bridge
MQTT Kafka
Bridge 1
Bridge 2
![Page 49: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/49.jpg)
© 2016 MapR Technologies 49© 2015 MapR Technologies
Computation
![Page 50: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/50.jpg)
© 2016 MapR Technologies 50
Why do we need computation layer ?The computation layer is responsible for 1. Process huge amount of data in linear
scalable manner1. Read streaming data 2. Transform/Enrich/Analyze data3. Store the incoming data into persistence storage
2. Analyze IOT data1. Business Rules2. Predictive model
![Page 51: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/51.jpg)
© 2016 MapR Technologies 51
Spark Streaming
![Page 52: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/52.jpg)
© 2016 MapR Technologies 52
Business rules
![Page 53: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/53.jpg)
© 2016 MapR Technologies 53© 2015 MapR Technologies
Storage
![Page 54: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/54.jpg)
© 2016 MapR Technologies 54
Why do we need Storage Layer ?The storage layer is responsible for 1. Providing persistent store for IOT
data2. Store data in time series format
1. Efficient storage of data2. Ingest data at high speed3. Retrieve/Query data
![Page 55: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/55.jpg)
© 2016 MapR Technologies 55
Time Series Database• Data Point
(time, temperature)
• Metricssensor.temp.1s
• TagssensorId=2, location=bed1
• Metrics + Tags = Time Series
![Page 56: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/56.jpg)
© 2016 MapR Technologies 56
InfluxDB
![Page 57: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/57.jpg)
© 2016 MapR Technologies 57© 2015 MapR Technologies
Dashboard
![Page 58: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/58.jpg)
© 2016 MapR Technologies 58
Why do we need Dashboard Layer ?The dashboard layer is responsible for 1. Interact with TSDB2. Visualize TSDB data in easy to
understand format
![Page 59: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/59.jpg)
© 2016 MapR Technologies 59
Configure InfluxDB connection
![Page 60: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/60.jpg)
© 2016 MapR Technologies 60
Temperature Variation Graph
![Page 61: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/61.jpg)
© 2016 MapR Technologies 61
Temperature Variation Graph
![Page 62: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/62.jpg)
© 2016 MapR Technologies 62
Resources • Streaming Processing for Real-Time Analytics and Dashboard• MapR Streams• MapR DB• Eclipse IOT • COAP Tutorial• GitHub For source code
![Page 64: Designing and Implementing your IOT Solutions with Open Source](https://reader036.vdocuments.site/reader036/viewer/2022070516/586f751c1a28ab10258b5ee9/html5/thumbnails/64.jpg)
© 2016 MapR Technologies 64
Arduino - Logic for reading temperature