aws real-time event processing
TRANSCRIPT
![Page 1: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/1.jpg)
April 21, 2015
Seattle
AWS Big Data Platform
![Page 2: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/2.jpg)
Agenda Overview
10:00 AM Registration
10:30 AM Introduction to Big Data @ AWS
12:00 PM Lunch + Registration for Technical Sessions
12:30 PM Use Case Technical Deep Dive Sessions • Data Collection and Storage
• Real-time Event Processing
• Analytics
![Page 3: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/3.jpg)
Collect Process Analyze Store
Data Collection and Storage
Data Processing
Data Analysis
Event Processing
Primitive Patterns
S3 Kinesis DynamoDB RDS (Aurora)
MySQL
AWS Lambda
KCL Apps EMR Redshi?
Machine Learning
![Page 4: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/4.jpg)
Real-Time Event Processing
• Examples:
![Page 5: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/5.jpg)
Processing framework
![Page 6: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/6.jpg)
Two main processing patterns
![Page 7: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/7.jpg)
Real-time event processing frameworks
Kinesis Client Library
AWS Lambda
![Page 8: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/8.jpg)
Amazon KCL
![Page 9: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/9.jpg)
Shard 1
Shard 2
Shard 3
Shard n
Shard 4
KCL Worker 1
KCL Worker 2
EC2 Instance
KCL Worker 3
KCL Worker 4
EC2 Instance
KCL Worker n
EC2 Instance
Kinesis
Kinesis Client Library (KCL)
![Page 10: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/10.jpg)
KCL Design Components
KCL restarts the processing of the shard at the last known processed record if a worker fails
![Page 11: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/11.jpg)
Processing with Kinesis Client Library • Connects to the stream and enumerates the shards
• Instantiates a record processor for every shard it manages
• Checkpoints processed records in Amazon DynamoDB
• Balances shard-worker associations when the worker instance count changes
• Balances shard-worker associations when shards are split or merged
![Page 12: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/12.jpg)
Best practices for KCL applications
![Page 13: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/13.jpg)
Amazon Kinesis Connector
S3 Dynamo DB Redshift
Kinesis
![Page 14: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/14.jpg)
Amazon Kinesis connector application
Connector Pipeline
Transformed
Filtered
Buffered
Emitted
Incoming Records
Outgoing to Endpoints
![Page 15: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/15.jpg)
Real-time Monitoring dashboard with KCL
Amazon Kinesis
Kinesis-enabled Application
Producer on Amazon EC2
Amazon DynamoDB
Dashboard on Amazon EC2
2 sec sliding-window analysis over streaming clickstream data
![Page 16: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/16.jpg)
Monitoring Demo Kinesis Client Library
![Page 17: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/17.jpg)
AWS Lambda
![Page 18: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/18.jpg)
Event-Driven Compute in the Cloud
![Page 19: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/19.jpg)
No Infrastructure to Manage
![Page 20: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/20.jpg)
Automatic Scaling
![Page 21: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/21.jpg)
Bring your own code
![Page 22: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/22.jpg)
Fine-grained pricing
Free Tier 1M requests and 400,000 GB-s of compute.
Every month, every customer.
Never pay for idle.
![Page 23: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/23.jpg)
Data Triggers: Amazon S3
Amazon S3 Bucket Events AWS Lambda
Original image Thumbnailed image
1
2
3
![Page 24: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/24.jpg)
Data Triggers: Amazon DynamoDB
AWS Lambda Amazon DynamoDB Table and Stream
Send Amazon SNS notifications
Update another table
![Page 25: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/25.jpg)
Calling Lambda Functions
![Page 26: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/26.jpg)
Writing Lambda Functions
![Page 27: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/27.jpg)
How can you use these features?
“I want to send customized messages to
different users”
SNS + Lambda
“I want to send an offer when a user runs out of lives in
my game”
Amazon Cognito + Lambda + SNS
“I want to transform the
records in a click stream or an IoT
data stream”
Amazon Kinesis + Lambda
![Page 28: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/28.jpg)
Real-Time Alerting Demo AWS Lambda
![Page 29: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/29.jpg)
Stream Processing Apache Spark Apache Storm Amazon EMR
![Page 30: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/30.jpg)
Read Data Directly into Hive, Pig, Streaming and Cascading Real time sources into Batch Oriented Systems Multi-Application Support & Check-pointing
Amazon EMR integration
![Page 31: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/31.jpg)
CREATE TABLE call_data_records ( start_time bigint, end_time bigint, phone_number STRING, carrier STRING, recorded_duration bigint, calculated_duration bigint, lat double, long double ) ROW FORMAT DELIMITED FIELDS TERMINATED BY "," STORED BY 'com.amazon.emr.kinesis.hive.KinesisStorageHandler' TBLPROPERTIES("kinesis.stream.name"=”MyTestStream");
Amazon EMR integration: Hive
![Page 32: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/32.jpg)
DStream
RDD@T1 RDD@T2
Messages
Receiver
Spark Streaming – Basic concepts
http://spark.apache.org/docs/latest/streaming-kinesis-integration.html
![Page 33: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/33.jpg)
Spark Streaming
![Page 34: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/34.jpg)
Processing Amazon Kinesis streams
Amazon Kinesis
Spark-Streaming
![Page 35: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/35.jpg)
Weblog Demo Kinesis + Spark Streaming
![Page 36: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/36.jpg)
Storm
![Page 37: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/37.jpg)
Apache Storm: Basic Concepts
https://github.com/awslabs/kinesis-storm-spout
![Page 38: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/38.jpg)
Launches Workers
Storm architecture
Master Node
Cluster Coordination
Worker Processes
Worker
Nimbus
Zookeeper
Zookeeper
Zookeeper
Supervisor
Supervisor
Supervisor
Supervisor
Worker
Worker
Worker
![Page 39: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/39.jpg)
Real-time: Event-based processing
Kinesis Storm Spout
Producer Amazon Kinesis
Apache Storm
ElastiCache (Redis) Node.js Client
(D3)
http://blogs.aws.amazon.com/bigdata/post/Tx36LYSCY2R0A9B/Implement-a-Real-time-Sliding-Window-Application-Using-Amazon-Kinesis-and-Apache
![Page 40: AWS Real-Time Event Processing](https://reader037.vdocuments.site/reader037/viewer/2022102723/55a69daf1a28abf67d8b47fa/html5/thumbnails/40.jpg)
Thank You