redis labcamp
TRANSCRIPT
REDIS Labcamp
Angelo Simone ScottoCluster Reply
2
• Developed by Salvatore Sanfilippo for his startup Merzia S.r.l.• Aimed as a fast replacement of MySQL for web analytics (LLOOGG)• Released as open source
in 2009.• Sponsored by VMWare
since 2010.• Sponsored by Pivotal
since 2013.• Currently part of backend
of
An italian story
• Twitter• Github• Pinterest• StackOverflow• Flickr• …
3
What is REDIS (REmote DIctionary Service)
Redis is a DSL (Domain Specific Language) that manipulates abstract data types and implemented as a TCP daemon. Commands manipulate a key space where keys are binary-safe strings and values are different kinds of abstract data types.
REDIS Manifesto
4
Available Abstract Data Types
Key-Value Store:• Keys
Printable ASCII• Values
Primitives Strings
Containers (of strings) Hashes Lists Sets Sorted Sets
Other Functionalities Pub/Sub HyperLogLog Bitmaps
5
Where to find Redis
• Obviosuly at redis website : http://redis.io• Redis written in ANSI C and POSIX compliant:
• Linux (supported and recommended)• Mac OS X (supported)• *BSD (supported)• Solaris (support defined «Best effort»)
• Windows is not directly supported by REDIS Project but• Microsoft mantains and support a version at http
://msopentech.com/opentech-projects/redis/• Used by Microsoft in Azure (Azure Redis Cache Service) therefore stable
and usable also on windows.
6
How to Install Redis
• On Windows, very simple:• Download binaries from https://
github.com/MSOpenTech/redis/tree/2.8/bin/release, unzip and run.• On Linux or MacOSX:
• Your preferred Package Manager (apt-get, YUM …) should have it but it’s usually not recommended because often lagging behind current version.
• Recommended way is to recompile binaries directly: http://redis.io/download
7
Strings
Redis Key
Value: Redis String
• Max Value Size: 512 MB• Binary Strings
Commands:• GET, SET• INCR, DECR, INCRBY, DECRBY• GETBIT, SETBIT, BITCOUNT, BITPOS, BITOP• APPEND, GETRANGE, SETRANGE• MGET, MSET
8
Hashes
• Max Value Size: 512 MB• Binary Strings
Redis Key
Value: Redis Hash
Field 1 Value 1
Field 2 Value 2
Field 3 Value 3
Commands:• HGET, HSET• HINCRBY, HINCRBYFLOAT• HLEN, HGETALL, HVALS, HKEYS• HMGET, HMSET
9
Lists
• LPUSH, LPOP (Left, first)• RPUSH, RPOP (Right, last)• Queues are simply lists
when using LPUSH & RPOP• Stack are simply lists when
using LPUSH & LPOP• Index (list) operations are
obviously available (LINDEX)
Redis Key
Value: Redis List
HE
AD
Valu
e 1
Valu
e n
TAIL
Commands:• LPUSH, LPOP• RPUSH, RPOP, RPOPLPUSH• BLPOP, BRPOP, BRPOPLPUSH• LINDEX, LRANGE, LREM
10
Sets• Unsorted set• Unique Values (duplicates
are non inserted) Redis Key
Value: Redis Set
Value 2Value 1
Value 4
Value 5 Value 3
Commands:• SADD, SPOP, SREM• SCARD, SISMEMBER• SUNION, SDIFF, SINTER• SUNIONSTORE, SDIFFSTORE, SINTERSTORE
11
Sorted Sets• Score is a float (double precision
floating number).• Unique Values (duplicates are non
inserted) • Sorted sets are sorted by their score in
ascending way
Redis Key
Value: Redis Sorted Set
Valu
e 1
Sco
re 1
00
Valu
e 2
Sco
re 2
00
Valu
e 3
Sco
re 2
00
Valu
e 4
Sco
re 2
50
Commands:• ZADD, ZREM• ZCARD, ZRANGE, ZREVRANGE• ZSCORE, ZCOUNT, ZINCRBY, ZRANGEBYSCORE• ZLEXCOUNT, ZRANGEBYLEX, ZREVRANGEBYLEX, ZREMRANGEBYLEX• ZUNIONSTORE, ZINTERSTORE
12
Other Functionalities
Bitmaps: Not really a type, just a set of operators on strings.
Publish / Subscribe: Not a type, it implements a fire&forget publish/subscribe engine. PUBLISH, SUBSCRIBE, UNSUBSCRIBE PSUBSCRIBE, PUNSUBSCRIBE
HyperLogLog: Value contains an HyperLogLog algorithm implementation. REDIS implementation uses 12K per key with a std error of 0.81% to
count 2^64 items. PFADD, PFCOUNT, PFMERGE
13
REDIS Architecture
Key points:• Extremely Fast.
• i.e. Data is always in RAM.
The Redis data set, composed of defined key-value pairs, is
primarily stored in the computer's memory. The amount of
memory in all kinds of computers, including entry-level
servers, is increasing significantly each year. Memory is fast,
and allows Redis to have very predictable performance. REDIS Manifesto
14
REDIS Architecture
Key points:• Extremely Fast.
• i.e. Data is always in RAM.• Extremely Simple.
• i.e. REDIS is single-threaded.
We're against complexity. We believe designing systems is a fight
against complexity. We'll accept to fight the complexity when it's
worthwhile but we'll try hard to recognize when a small feature is not
worth 1000s of lines of code. Most of the time the best way to fight
complexity is by not creating it at all.
REDIS Manifesto
15
REDIS Architecture
Key points:• Extremely Fast.
• i.e. Data is always in RAM.• Extremely Simple.
• i.e. REDIS is single-threaded.• Extremely Light.
16
REDIS Architecture
Key points:• Extremely Fast.
• i.e. Data is always in RAM.• Extremely Simple.
• i.e. REDIS is single-threaded.• Extremely Light.• Extremely Robust.
17
REDIS Architecture
Security:«Redis is not optimized for maximum security but for maximum performance and simplicity»• AUTH with password.• BIND to specific network interface.• No encryption (use SSL proxy)• No ACS (each connection is admin)• Possible to RENAME sensible
commands…
18
REDIS Architecture
Persistence:• None.• RDB (Redis DB) single-file dump.• AOF (Append Only File), a log file.
• No fsync (fast, not safe).• Fsync each second.• Fsync each query (slow but safe).
• Both
Memory Usage:• Data is always in memory, you cannot have more data than RAM in REDIS.
• Virtual Memory deprecated.• Disk Store abandoned.
• Several policies to manage overflow:• noeviction: no action, just error.• allkeys-lru: remove older data• allkeys-random: remove rnd data
• volatile-lru: remove expirable• volatile-random: remove expirable• volatile-ttl: remove expirable
19
REDIS Architecture
Transactions:• Not an ACID Transactions.• All commands are serialized and
executed sequentially (MULTI).• Either all commands or no commands
are processed (EXEC – DISCARD).• Keys must be explicitely specified in
Redis transactions (WATCH)• Less useful since lua scripting appeared.
• Commands Available:• WATCH / UNWATCH• MULTI• EXEC / DISCARD
20
REDIS Architecture
Lua Scripting:• Lua is a small, fast, simple embeddable
scripting language.• EVAL command takes script, keys and
arguments:
• Redis calls are done through redis.call or redis.pcall functions:
• Scripts are executed sequentially on the server in atomic way
21
A Swiss Army Knife for Programmers
Redis gives you a small, light, efficient remote datastore with persistence, high-availability, scalability and publish/subscribe functionalities.
Use is limited just by your imagination:• Shared Configuration.• Distributed Lock.• Caching.• Work Queues.• Load Balancers.• Service Discovery.• …
Workgroup and Contest
23
A super-simple blog platform
• Use the frontend and the language you prefer:• Frontend:
• Console Application• Web Application• Form Application
• Language:• List of supported clients: http://redis.io/clients
• 5 Levels of features (10 points):1. User Management (2 points)2. Post Management (2 points)3. Tags / Categories (2 points)4. Votes (2 points)5. New posts notification (2 points)
24
User Management
• System should be able to register new users, login existing users and show registered users.
• Each user should have a unique id (number), a username, a password
• Conventions:• A common convention is to use hierarchical names for keys
separated by ‘:’ such as blog:users:1 or blog:users:1:posts
• Tip:• Use String and INCR to manage unique ids.• Use Hashes to manage user objects• Use Set to guarantee uniqueness of usernames
25
Post Management
• A logged in user should be able to add a blog post.• Each post will have a unique id (number), a content, a timestamp and a
subject.• Given a user the system should return the list of items posted by the
user.
• Tip:• Use String and INCR to manage unique ids.• Use Hashes to manage post objects.• Use Lists to relate posts with owners
26
Tags / Categories
• Modify the previous post structure to keep also a set of tag for each post.
• Given a tag the system should return all posts tagged with it.• Given multiple tags the system should return all posts, if any, tagged
with them.
• Tip:• Add Tags field to post Hash.• Use Sets to manage categories, and SINTER to search for multiple
tags.
27
Votes
• A logged in user should be able to vote posts.• Each user must be able to vote just once for a post.• System should be able to show posts ordered by votes.
• Tip:• Use Sorted Sets to manage post ranks.• Use Sets to enforce one vote per user per post.
28
Push Notifications
• A user could express interest for a category (tag) or another user activity.
• System should notify user if an article is posted in a category he expressed interest into.
• System should notify user if an article is posted by a user he expressed interest into.
• Tip:• Use PubSub engine to manage notifications.• Use one channel per category and one channel per user.