postgresql and redis - talk at pgcon 2013
TRANSCRIPT
![Page 2: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/2.jpg)
Topics
● What is Redis?● The Redis Foreign Data Wrapper● The Redis Command wrapper for Postgres● Case study – a high performance Ad server
using Postgres and Redis
![Page 3: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/3.jpg)
What is Redis?
● High performance in-memory key/value data store
![Page 4: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/4.jpg)
Redis is easy to use
● Almost no configuration● On Fedora
sudo yum install redissudo systemctl enable redis.servicesudo systemctl start redis.serviceredis-cli
![Page 5: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/5.jpg)
Redis keys
● Are just strings
![Page 6: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/6.jpg)
Redis data values
● Values can be scalars● Strings● Integers
● Values can be structured● Lists● Sets● Ordered sets● Hashes – name value pairs
– c.f. Hstore
![Page 7: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/7.jpg)
Simple command set
● Nothing like SQL, table joins● Command set is large but most commands only
take 2 or 3 parameters● http://redis.io/commands
![Page 8: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/8.jpg)
Examples - adding values
● SET mykey myvalue● HMSET myhashkey prop1 val1 prop2 val2● SADD mysetkey val1 val2 val3● LPUSH mylist val1 val2● ZADD myzsetkey 1 val1 5 val2
![Page 9: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/9.jpg)
No creation command
● You create an object by setting or adding to it● Almost schema-less
● Can't use a command for one object to another
![Page 10: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/10.jpg)
Redis keys all live in a single global namespace
● No schemas● No separation by object type● Very common pattern is to use fine grained
keys, like (for a web session)
web:111a7c9ff5afa0a7eb598b2c719c7975
● KEYS command can find keys by pattern:● KEYS web:*
– Dangerous
![Page 11: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/11.jpg)
How Redis users do “tables”
● They use a prefix:● INCR hits:2013.05.25
● They can find all these by doing● KEYS hits:*
● Or they keep a set with all the keys for a given type of data
● SADD hitkeyset hits:2013.05.25● The application has to make use of these keys –
Redis itself won't
![Page 12: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/12.jpg)
Redis Client library
● “hiredis”● Moderately simple● https://github.com/redis/hiredis
![Page 13: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/13.jpg)
Redis Foreign Data Wrapper
● https://github.com/pg-redis-fdw/redis_fdw● Originally written by Dave Page● Brought up to date and extended by me
![Page 14: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/14.jpg)
Originally
● Only supported scalar data● No support for segmenting namespace or use
of key sets
![Page 15: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/15.jpg)
Updates by me
● All data types supported● Table key prefixes supported● Table key sets supported● Array data returned as a PostgreSQL array
literal
![Page 16: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/16.jpg)
Hash tables
● Most important type● Most like PostgreSQL tables● Best to define the table as having array of text
for second column● Turn that into json, hstore or a record.
![Page 17: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/17.jpg)
Example
● CREATE FOREIGN TABLE web_sessions(key text,values text[])
SERVER localredisOPTIONS (tabletype hash,
tablekeyprefix 'web:');
SELECT * from web_sessions;
![Page 18: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/18.jpg)
Use with hstore
● CREATE TYPE websession AS (id text, browser text, username text);
SELECT populate_record(null::websession,hstore(values))
FROM websessions;
![Page 19: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/19.jpg)
Use with json_object
● https://bitbucket.org/qooleot/json_object● CREATE EXTENSION json_object;
SELECT json_object(values)FROM websessions;
![Page 20: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/20.jpg)
Key prefix vs Key Set
● Key sets are much faster● Ad server could not meet performance goals
until it switched to using key sets● Recommended by Redis docs
![Page 21: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/21.jpg)
Using a key set to filter rows
● Sort of “where” clause● Put the keys of the entries you want in a set
somehow● Can use command wrapper
● Define a new foreign table that uses that set as the keyset
![Page 22: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/22.jpg)
9.3 notes
● In 9.3 there is json_populate_record()● Could avoid use of hstore● For post 9.3, would be a good idea to have a
function converting an array of key value pairs to a record directly
![Page 23: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/23.jpg)
Brand new – Singleton Key tables
● Each object is a table, not a row● Sets and lists come back as single field rows● Ordered sets come back as one or two field
rows – second field can be score
● Hashes come back as rows of key/value
![Page 24: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/24.jpg)
Coming soon
● Writable tables● Supported in upcoming release 9.3
![Page 25: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/25.jpg)
Redis Command Wrapper
● Fills in the missing gaps in functionality● Sponsored by IVC: http://www.ivc.com● https://bitbucket.org/qooleot/redis_wrapper
![Page 26: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/26.jpg)
Redis wrapper functionality
● Thin layer over hiredis library● Four basic functions
● redis_connect()● redis_disconnect()● redis_command()● redis_command_argv()
![Page 27: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/27.jpg)
redis_connect()
● First argument is “handle”● Remaining arguments are all optional
● con_host text DEFAULT '127.0.0.1'::text● con_port integer DEFAULT 6379● con_pass text DEFAULT ''::text● con_db integer DEFAULT 0● ignore_duplicate boolean DEFAULT false
![Page 28: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/28.jpg)
Redis wrapper connections are persistent
● Unlike FDW package, where they are made at the beginning of each table fetch
● Makes micro operations faster
![Page 29: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/29.jpg)
redis_command and redis_command_argv
● Thin layers over similarly named functions in client library
● redis_command has max 4 arguments after command string – for more use redis_command_argv
● Might switch from VARIADIC text[] to VARIADIC “any”
![Page 30: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/30.jpg)
Uses
● Push data into redis● Redis utility statements from within Postgres
![Page 31: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/31.jpg)
Higher level functions
● redis_push_record● con_num integer● data record● push_keys boolean● key_set text● key_prefix text● key_fields text[]
![Page 32: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/32.jpg)
Why use Redis?
● Did I mention it's FAST?● But not safe
![Page 33: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/33.jpg)
Our use case
● An ad server for the web● If Redis crashes, not a tragedy● If it's slow, it's a tragedy
![Page 34: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/34.jpg)
Ad Server Project by IVC
http://www.ivc.com
Remaining slides are mostly info from IVC
![Page 35: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/35.jpg)
System Goals
● Serve 10,000 ads per second per application server cpu
● Use older existing hardware
● 5 ms for Postgres database to filter from 100k+ total ads to ~ 30 that can fit a page and meet business criteria
● 5 ms to filter to 1-5 best ads per page using statistics from Redis for freshness, revenue maximization etc.
● Record ad requests, confirmations and clicks.
● 24x7 operation with automatic fail over
![Page 36: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/36.jpg)
Physical View
802.3ad
/4
/2ea
/4 /4 /4
/4
/4
Cisco 3750 stacked
1G HSRP
Xen Hosts
SLES 11.2Dell R810128GIntel e654024cores
SLES 11.2Dell 295032GIntel e54308 cores
![Page 37: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/37.jpg)
Redundancy View
www.draw-shapes..de
www.draw-shapes.de
Cisco HSRP
Keepalived
NGINX
Node
Sentinel
Tier 1 Client Tier 2 Web Tier 3 Application
Tier 4 Database
Shorewall
KeepalivedRedis
MultipleInstances
Postgres 9.2
Sentinel
TransactionDBPgpool
Hot Replication
Business DBHot Replication
Data Warehouse DBHot Replication
SkytoolsLondiste3
![Page 38: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/38.jpg)
Postgres databases
● 6 Postgres databases
● Two for business model – master and streaming hot standby (small VM)
● Two for serving ads – master and streaming hot standby (physical Dell 2950)
● Two for for storing clicks and impressions – master and hot standby (physical Del 2950)
● Fronted by redundant pg pool load balancers with fail over and automated db fail over.
![Page 39: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/39.jpg)
Business DB
● 30+ tables
● Example tables: ads, advertisers, publishers, ip locations
● Small number of users that manipulate the data (< 100)
● Typical application and screens
● Joining too slow to serve ads
● Tables get materialized into 2 tables in the ad serving database
![Page 40: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/40.jpg)
● Two tables
● First has ip ranges so we know where the user is coming from. Ad serving is often by country, region etc.
● Second has ad sizes, ad types, campaigns, keywords, channels, advertisers etc.
● Postgres inet type and index was a must have to be successful for table one
● Tsquery/tsvector, boxes, arrays were all a must have for table two (with associated index types)
Ad Serving Database
![Page 41: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/41.jpg)
Ad serving Database
● Materialized and copied from Business database every 3 minutes
● Indexes are created and new tables are vacuum analyzed then renamed.
● Performance goals were met.● We doubt this could be done without Postgres
data types and associated indexes● Thanks
![Page 42: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/42.jpg)
Recording Ad requests/confirmations and clicks
● At 10k/sec/cpu recording ads one row at a time + updates on confirmation is too slow
● Approach: record in Redis, update in Redis and once every six minutes we batch load from Redis to Postgres. - FDW was critical.
● Partitioning (inheritance) with constraint exclusion to segregate data by day using nightly batch job. One big table with a month's worth of data would not work.
● Table partitioning is not cheap in the leading commercial product.
● Thanks
![Page 43: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/43.jpg)
Recording DB continued.
● Used heavily for reporting.● Statistics tables (number of clicks, impressions
etc.) are calculated every few minutes on today's data
● Calculated nightly for the whole day tables● For reporting we needed some business data
so we selectively replicate business tables in the ad recording database using Skytools. DB linking tables is too slow when joining.
![Page 44: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/44.jpg)
Recording DB cont'd
● Another usage is fraud detection. ● Medium and long term frequency fraud
detection is one type of fraud that this database is used for.
![Page 45: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/45.jpg)
Redis
● In memory Database.
● Rich type support.
● Multiple copies and replication.
● Real time and short term fraud detection
● Dynamic pricing
● Statistical best Ad decision making
● Initial place to record and batch to Postgres
● Runs on VM with 94Gb of dedicated RAM.
![Page 46: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/46.jpg)
Redis cont'd
● FDW and commands reduce the amount of code we had to write dramatically
● FDW good performance characteristics.● Key success factor: In memory redis DB +
postgres relational DB.
![Page 47: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/47.jpg)
Postgres – Redis interaction
● Pricing data is pushed to Redis from Business DB via command wrapper
● Impression and Click data is pulled from Redis into Recording DB via Redis FDW
![Page 48: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/48.jpg)
Current Status
● In production with 4 significant customers since March 1
● Scaling well
![Page 49: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/49.jpg)
Conclusions
● Postgres' rich data types and associated indexes were absolutely essential
● Redis + Postgres with good FDW integration was the second key success factor
● Node.js concurrency was essential in getting good application throughput
● Open source allowed the system to be built for less than 2% of the cost of a competing commercial system
![Page 50: PostgreSQL and Redis - talk at pgcon 2013](https://reader035.vdocuments.site/reader035/viewer/2022081403/554f7889b4c905d25b8b4635/html5/thumbnails/50.jpg)
Questions?