postgrenosql - hagander · postgrenosql opensource days 2013 copenhagen, denmark magnus hagander...
TRANSCRIPT
![Page 1: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/1.jpg)
PostgreNoSQLOpenSource Days 2013Copenhagen, Denmark
Magnus [email protected]
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
![Page 2: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/2.jpg)
Magnus Hagander•PostgreSQL
•Core Team member•Committer•PostgreSQL Europe
•Redpill Linpro•Infrastructure services•Principal database consultant
![Page 3: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/3.jpg)
PostgreSQL•Relational database•"Old world"•Nobody wants SQL?
![Page 4: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/4.jpg)
Nobody wants SQL•Really?
![Page 5: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/5.jpg)
What do people actuallydislike?
•SQL•Or relational•Or "ACID guarantees"•or...?
![Page 6: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/6.jpg)
What can we do•We can't take away SQL•But we can minimize use of it•Still very useful for many things
•Even in typical NoSQL scenarios
![Page 7: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/7.jpg)
ACID guarantees•By default, PostgreSQL guarantees everything•Turning off fsync risks your data
•Almost guaranteed data corruption•Hard to detect, even harder to fix
![Page 8: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/8.jpg)
Almost-ACID•Instead, use synchronous_commit=off•Guarantees data integrity•Guarantees data consistency•But may loose small amounts of durability
![Page 9: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/9.jpg)
Eventual consistency•Often used for write scaling•Don't really care about data consistency•We don't do this at this point
•There are people experimenting...
![Page 10: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/10.jpg)
Relational model•Tables and relations
•And scary joins!•Does not always match application model
•Though more often than you think•Something we can work with
![Page 11: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/11.jpg)
Potential issues withrelational
•Relational model has columns•Need to know which•Very sparse data sets are bad
•Anti-solution: EAV•Please don't go there
![Page 12: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/12.jpg)
Dealing withnon-relational
•key/value store•General schemaless data
•Put all requirements on application•Mix and match!
![Page 13: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/13.jpg)
hstore•Generic key-value store•Fully indexable!•Typeless•Available in all supported versions
![Page 14: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/14.jpg)
Installing hstorepostgres=# CREATE EXTENSION hstore;CREATE EXTENSION
![Page 15: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/15.jpg)
Defining hstore columnspostgres=# CREATE TABLE items (postgres(# itemid serial NOT NULL PRIMARY KEY,postgres(# itemname text NOT NULL,postgres(# tags hstore);CREATE TABLE
![Page 16: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/16.jpg)
Creating hstore valuespostgres=# INSERT INTO items (itemname, tags)postgres-# VALUES ('item1', 'color => red, category => stuff');INSERT 0 1
![Page 17: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/17.jpg)
Query by hstorepostgres=# SELECT itemname FROM itemspostgres-# WHERE tags->'color' = 'red'; item1
![Page 18: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/18.jpg)
Indexed access•Create normal expression index on columnCREATE INDEX foo ON ((items->'color'))•Requires one index per key•That's what we wanted to avoid...
![Page 19: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/19.jpg)
Dynamic GiST indexing•Create index covering all keysCREATE INDEX hstoreidx ON items USING gist(tags)•Available for multiple operators
•All types of containment•Must use these operators
![Page 20: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/20.jpg)
Querying with GiSTpostgres=# EXPLAINpostgres-# SELECT itemname FROM itemspostgres-# WHERE tags @> 'color=>red';
Index Scan using hstoreidx on items (cost=0.12..8.14 rows=1 width=32) Index Cond: (tags @> '"color"=>"red"'::hstore)
![Page 21: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/21.jpg)
Querying for tag presencepostgres=# EXPLAINpostgres-# SELECT itemname FROM itemspostgres-# WHERE tags ? 'color';
Index Scan using hstoreidx on items (cost=0.12..8.14 rows=1 width=32) Index Cond: (tags ? 'color'::text)
![Page 22: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/22.jpg)
Downsides of hstore•Values are not typed
•Just strings•No hierarchy•No key compression•Still slower than "normal columns"
•But very useful with sparse data!
![Page 23: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/23.jpg)
Fully schemaless•"Document storage"•And of course, processing•"Everybody uses JSON"
•Yup, they stopped using XML!
![Page 24: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/24.jpg)
JSON•JavaScript Object Notation•Text-based data•Schemaless•Hierarchical•PostgreSQL has native support (since 9.2)!
![Page 25: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/25.jpg)
JSON in PostgreSQLCREATE TABLE jsontable ( id serial PRIMARY KEY, j json);
![Page 26: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/26.jpg)
Storing JSONpostgres=# INSERT INTO jsontable (j) VALUES ('{postgres'# "id":"mha",postgres'# "name":"Magnus Hagander",postgres'# "country": "Sweden"postgres'# }');INSERT 0 1
•Validates json syntax•Maintains formatting
![Page 27: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/27.jpg)
Mapping JSONpostgres=# SELECT row_to_json(schedule)postgres-# FROM schedule WHERE id=1; {"id":1,"employee_id":1,"t":"[\"2013-02-08 13:00:00+00\", \"2013-02-08 17:00:00+00\")"}
![Page 28: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/28.jpg)
Mapping JSONpostgres=# SELECT row_to_json(schedule)postgres-# FROM schedule WHERE id=1; {"id":1,"employee_id":1,"t":"[\"2013-02-08 13:00:00+00\", \"2013-02-08 17:00:00+00\")"}
postgres=# SELECT row_to_json(t) FROM (postgres-# SELECT id, employee_idpostgres-# FROM schedule) t; {"id":2,"employee_id":1} {"id":3,"employee_id":2} {"id":1,"employee_id":1}
![Page 29: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/29.jpg)
Using JSON•That's really all there is to JSON
•At least in 9.2•For full power, use with pl/v8
![Page 30: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/30.jpg)
PL/V8•Combines PostgreSQL and V8•Outside extensionpostgres=# CREATE EXTENSION plv8;CREATE EXTENSION
![Page 31: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/31.jpg)
JSON extractionCREATE or replace FUNCTION jmember (j json, key text ) RETURNS text LANGUAGE plv8 IMMUTABLEAS $function$ if (typeof j != 'object') return NULL; return JSON.stringify(j[key]);$function$;
![Page 32: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/32.jpg)
JSON extractionv8test=# SELECT jmember(j, 'name')v8test-# FROM jsontable; "Magnus Hagander"
![Page 33: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/33.jpg)
JSON indexingv8test=# CREATE INDEX idx_nameidv8test-# ON jsontable (jmember(j, 'id'));CREATE INDEX
![Page 34: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/34.jpg)
JSON indexingv8test=# EXPLAINv8test-# SELECT * FROM jsontablev8test-# WHERE jmember(j, 'id') = 'mha';
Index Scan using idx_nameid on jsontable (cost=0.38..8.39 rows=1 width=36) Index Cond: (jmember(j, 'id'::text) = 'mha'::text)
![Page 35: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/35.jpg)
JSON indexing•Still need one index per key•Can use arbitrary expression
•Including processing hierarchical data•Can transform key on lookup as well
![Page 36: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/36.jpg)
Javascript functions•Uses "javascript way"•E.g., subtransactions by:CREATE OR REPLACE FUNCTION public.st() RETURNS void LANGUAGE plv8AS $function$plv8.subtransaction(function() { plv8.execute("INSERT INTO tbl VALUES(1)"); plv8.execute("INSERT INTO tbl VALUES(2)");});$function$
![Page 37: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/37.jpg)
JSON/JS future inPostgreSQL
•More built-in operators for JSON•Likely showing up in 9.3•Extraction etc
•No current plans to include PL/V8 in core•C++, etc•But we like extensions!
![Page 38: PostgreNoSQL - Hagander · PostgreNoSQL OpenSource Days 2013 Copenhagen, Denmark Magnus Hagander ... non-relational •key/value store •General schemaless data •Put all requirements](https://reader035.vdocuments.site/reader035/viewer/2022062603/5f0a06557e708231d429a623/html5/thumbnails/38.jpg)
JSON/JS future inPostgreSQL
•Generic JSON indexing•With hierarchy support•No code yet•Plans are in progress•Not in 9.3, but maybe 9.4