fast data at scale with amazon elasticache for redis
TRANSCRIPT
![Page 1: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/1.jpg)
Fast DataActing on Information
µs are the new ms
![Page 2: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/2.jpg)
What is Fast Data?• Generated fast
• Actionable fast
![Page 3: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/3.jpg)
Source: Financial TradingBids, asks, trades, equities, bonds, derivatives …
![Page 4: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/4.jpg)
Source: SensorsAny Thing in the Internet of Things
• Temperature• Location• Velocity• Acceleration• Radiation• Electricity
![Page 5: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/5.jpg)
Source: ClickstreamYou click, someone tracks…
![Page 6: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/6.jpg)
Source: Log Aggregation• Servers• Storage• Applications• Network• Security
![Page 7: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/7.jpg)
Capturing the Data
AWS IoT
IoT Device AmazonElasticsearchAWS
Lambda
Hot Data
AmazonElastiCache
AmazonDynamoDB
Longer Retention
Data Lake
Amazon S3
Amazon Glacier
Cold Data
Amazon Kinesis Firehose
7
Data Sources
101 0
Amazon Kinesis Streams
![Page 8: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/8.jpg)
8
In-Memory Key-Value Store
High-performance
Redis and Memcached
Fully managed; Zero admin
Highly Available and Reliable
Hardened by Amazon
![Page 9: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/9.jpg)
AmazonRDS
Amazon S3
Request RateHigh Low
LatencyLow High
Data VolumeLow High
AmazonGlacier
AmazonCloudSearch & ElasticsearchSt
ruct
ure
Low
High
AmazonDynamoDB
AmazonElastiCache
HDFS
![Page 10: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/10.jpg)
Redis – the fast in-memory database
Powerful ~200 commands + Lua scripting
In-memory database
Utility data structuresstrings, lists, hashes, sets, sorted sets, bitmaps & HyperLogLogs
Simple
Atomic operationssupports transactionshas ACID properties
Ridiculously fast!<500microsecond latency for most commands
Highly Availablereplication
Persistentsnapshots or append-only log
Open Source
10
![Page 11: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/11.jpg)
One Simple Example ….
+on-demand c3.8xlarge
3,000,000 objects
+
100 bytes eachmemtier benchmark50 read : 50 write
1,228,432 TPSSource: http://highscalability.com/blog/2014/8/27/the-12m-opssec-redis-cloud-cluster-single-server-unbenchmar.html
![Page 12: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/12.jpg)
12
Fully managed service = Automated Operations
Redis datastore hosted on Amazon EC2 Amazon ElastiCache for Redis
![Page 13: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/13.jpg)
ElastiCache for Redis Key AttributesExtreme
Performance
Sub-millisecond access latencies
4.5 million writes / sec
20 millionreads / sec
Open Source Compatible
Compatible with Open Source
Redis
Works with any Redis Client
Fully Managed
Automates node
replacement, software patching,
upgrades and backups
CloudWatch automates
monitoring of cache
performance metrics
Secure and Hardened
Supports Amazon VPC and IAM for
secure and fine grained access.
Monitors nodes and applies
security patches when needed
Highly Available and
Scalable
Clusters of up to 15 shards, each with a primary node and up to 5 replica nodes.
Multi-AZ with rapid automatic failover, with no
human intervention or code changes
needed.
Cost Effective
Pay as low as US$0.017 per
hour. Get started with
AWS Free Tier.
Zero data transfer costs for cross-AZ replication
![Page 14: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/14.jpg)
Amazon ElastiCache
Enhanced Redis Engine on ElastiCacheHardened by Amazon
RDB Without Fork•Mitigate the risk of increased swap usage during syncs and snapshots.
Dynamic write throttling•To improve output buffer management when the node’s memory is close to being exhausted
Smoother failovers•Clusters recover faster as replicas will avoid flushing their data to do a full re-sync with the primary.
![Page 15: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/15.jpg)
15
GEO Commands Store and retrieve longitude, latitude and
radius as a Sorted Set
• GEOADD
• GEODIST
• GEOHASH
• GEOPOS
• GEORADIUS
• GEORADIUSBYMEMBER
Amazon ElastiCache for Redis
![Page 16: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/16.jpg)
16
BITFIELD• Memory-efficient approach to
storing many small integers
as a large bitmap
• Treats a Redis string as an array of bits
• GET, SET on INCRBY any group of bits within a string
• Control increment / decrement behavior with OVERFLOW
Amazon ElastiCache for Redis
![Page 17: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/17.jpg)
17
Clusters! • Enables Horizontal Scaling• In ElastiCache for Redis 3.2:
• a Cluster is one or more Shards• a Shard is a primary node plus up to 5
replica nodes• Up to 15 Shards per Cluster
• 15 r3.8xlarge = 3.5TiB!• Cluster-level backup and restore
S5
S1
S2
S4 S3Client
Amazon ElastiCache for Redis
Clusters!
![Page 18: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/18.jpg)
18
Autosharding • 16384 hash slots per Cluster• Slot for a key is CRC16 modulo {key}
• Slots are distributed across the Clusterinto Shards
• Developers must use a Redis cluster client!• Clients are redirected to the correct shard• Smart clients store a mapS5
S1
S3
S4 S3Client
Shard S1 = slots 0 – 3276Shard S2 = slots 3277 – 6553Shard S3 = slots 6554 – 9829Shard S4 = slots 9830 – 13106Shard S5 = slots 13107 - 16383
Amazon ElastiCache for Redis
![Page 19: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/19.jpg)
Faster Recovery
• A Cluster consists of 1 – 15 Shards• Each Shard has a primary node and up to 5
replica nodes• In the event of a primary node outage, a replica node
in that Shard will be promoted
PrimaryS1a
S1b S1c
S2a
S2b S2c
S3a
S3b S3c
ReplicaShard Shard Shard
XS2c
S2a
Amazon ElastiCache for Redis
![Page 20: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/20.jpg)
20
Open Source Distributed Index
Managed Service using Elasticsearch and Kibana
Fully managed; Zero admin
Highly Available and Reliable
RESTful API for easy integrationAmazon Elasticsearch
Service
![Page 21: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/21.jpg)
Logstash
REST
CWL Agent
EC2 Instances
Amazon Kinesis
AmazonRDS
AmazonDynamoDB
AmazonSQS
Queue
Logstash Cluster
Amazon Elasticsearch Service
Amazon CloudWatch
AWSLambda
AWSCloudTrail
Access Logs
Amazon VPC Flow
Logs
Amazon S3 bucket
AWS IoT
Amazon Kinesis Firehose
Integration with the AWS ecosystem
Amazon ECS
![Page 22: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/22.jpg)
Using Fast Data: FinanceFraud Detection
Regulatory Compliance
![Page 23: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/23.jpg)
Using Fast Data: RetailRecommendations
![Page 24: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/24.jpg)
Using Fast Data: Utilities
Incident Response
![Page 25: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/25.jpg)
Using Fast Data: Health CareDrug Interactions
![Page 26: Fast Data at Scale with Amazon ElastiCache for Redis](https://reader036.vdocuments.site/reader036/viewer/2022070509/589dc7c81a28abf45d8b49c1/html5/thumbnails/26.jpg)
What’s Next?How will Fast Data change your industry?
Featured Tools:Amazon ElastiCache for RedisAmazon Elasticsearch Service
Darin Briskman ~ [email protected]