big data hadoop apex app for device to mobile, gps tracking with datatorrent
TRANSCRIPT
![Page 1: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/1.jpg)
Big Data Hadoop Apex App for device to mobile, GPS tracking @DataTorrent
Venkatesh Kottapalli. Software Engineer
Vikram Patil. Software Engineer
![Page 2: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/2.jpg)
Agenda:
● Introduction to Apex
● Use cases for GPS Tracking
● General requirements for GPS Tracking App
● Application Architecture using Apache Apex
● Further App Details
● Resources
![Page 3: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/3.jpg)
Apache Apex - Stream Processing
Easily Operable - Exposes an easy API for developing Operators (part of an
application) and Applications
Highly Scalable - Scales statically as well as dynamically
Highly Performant - Can reach single digit millisecond end-to-end latency
Fault Tolerant - Automatically recovers from failures - without manual
intervention
Stateful - Guarantees that no state will be lost
Apex Malhar library
YARN - Native - Uses Hadoop YARN framework for resource negotiation
![Page 4: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/4.jpg)
Apache Apex Platform Overview
![Page 5: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/5.jpg)
An Apex Application is a DAG(Directed Acyclic Graph)
A DAG is composed of vertices (Operators) and edges (Streams).
A Stream is a sequence of data tuples which connects operators at end-points called Ports
An Operator takes one or more input streams, performs computations & emits one or more output
streams
● Each operator is USER’s business logic, or built-in operator from our open source
library
● Operator may have multiple instances that run in parallel
![Page 6: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/6.jpg)
Apex - Native Hadoop Integration
• YARN is the resource manager
• HDFS used for storing any persistent state
![Page 7: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/7.jpg)
Usecases:
● Track fleet vehicles while they are in transit for path safety or any kind of
frauds.
● Bus tracking for Government / Private Transportations to adjust routes
dynamically according to traffic situations.
● Track wild animals using gps enabled collars or devices
● Track inventory of items including cars, refrigerators, expensive retail goods
etc.
● Location based transportation apps. Ex - Uber, Lyft
● Location based gaming apps. Ex - Pokemon go
● Location based utility apps. Ex - Find my friends
![Page 8: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/8.jpg)
General Requirements:
● Accept data from millions of devices through Tcp sockets
or over MQTT protocol.
● Once data is ingested, it need to be processed in realtime
to identify trends or events.
● Based on event priority, customer need to be informed
about it as well historical data need to be stored for
analysis or further review.
![Page 9: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/9.jpg)
Overall Application Architecture:
![Page 10: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/10.jpg)
App Details:
● Http Rest API support
● Websocket Support for clients to receive real-time
updates from App.
● Receive device data from millions of devices using tcp
socket at configured time interval.
[ Device data = location and device identification + (
temperature / pressure / battery status etc ) ]
● Device data parsing + processing to make it actionable in
real-time.
![Page 11: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/11.jpg)
GPS Data Processing App
![Page 12: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/12.jpg)
Websocket App
![Page 13: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/13.jpg)
Http Server App
![Page 14: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/14.jpg)
Communication between apps● Any config updates by the end user will be received by the http load receiver and
published onto a kafka topic which is then consumed by the GPS tracking app and the configuration is updated in memory in real time
![Page 15: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/15.jpg)
Data Persistence
● Cassandra Output Operator● Cassandra Input Operator● Event Archival
![Page 16: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/16.jpg)
Resources●http://apex.apache.org/
●Learn more: http://apex.apache.org/docs.html
●Subscribe - http://apex.apache.org/community.html
●Download - http://apex.apache.org/downloads.html
●Follow @ApacheApex - https://twitter.com/apacheapex
●Meetups – http://www.meetup.com/pro/apacheapex/
●More examples: https://github.com/DataTorrent/examples
●Slideshare: http://www.slideshare.net/ApacheApex/presentations
●https://www.youtube.com/results?search_query=apache+apex
●Free Enterprise License for Startups -
https://www.datatorrent.com/product/startup-accelerator/
![Page 17: Big Data Hadoop Apex App for Device to Mobile, GPS Tracking with DataTorrent](https://reader031.vdocuments.site/reader031/viewer/2022021421/5a6490947f8b9a76568b4aff/html5/thumbnails/17.jpg)
Q&A
Thank you