postgresql - pgconf india · 2019-02-20 · logical replication: pglogical postgresql core now has...
TRANSCRIPT
![Page 1: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/1.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
![Page 2: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/2.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
PostgreSQL is extensible● Hyperconverged Database!
● PostgreSQL is extensible because its operation is catalog-driven and it stores information about data types, functions, access methods etc in the catalogs.
● The catalogs can be modified by the users and that makes PostgreSQL highly extensible.
● PostgreSQL also allows users to dynamically load arbitrary code in the engine
● And remember, it’s Open Source!
![Page 3: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/3.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
What is an Extension?● A package of functions, operators, data types, index
types, that can be installed and removed as a unit.● First appeared in PostgreSQL 9.1, though similar
capability existed even before in form of modules.● Special SQL commands such as CREATE
EXTENSION, DROP EXTENSION.● A few important extensions are bundled with the core,
several others are written and managed by third party developers.
![Page 4: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/4.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
What can be extended?● PostgreSQL provides a bunch of hooks to incept and
override default behaviour of planner, executor, transaction manager, DDLs, start/stop background worker processes, request system resources such as shared memory, low-level locks etc
● Define your own data types, aggregate functions, operators, operator classes
● Indexes and storage systems.
![Page 5: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/5.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
What can be extended?● Write Foreign Data Wrappers to talk to a completely
different datasource (including another database engine too).
● Define your own user-defined-functions in the language of your choice.
● Doesn’t stop there - implement your own language as well.
![Page 6: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/6.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Why Extensions are important?● Nearly impossible to handle all workloads in core
PostgreSQL○ Development process is often slow and
conservative○ Extensions allow rapid development and
experimentation● PostgreSQL can be modified without forking it (though
our liberal licensing allows that)
![Page 7: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/7.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
SyntaxCREATE EXTENSION [ IF NOT EXISTS ] extension_name [ WITH ] [ SCHEMA schema_name ] [ VERSION version ] [ FROM old_version ]
DROP EXTENSION [ IF EXISTS ] extension_name [, ...] [ CASCADE | RESTRICT ]
![Page 8: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/8.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
SyntaxALTER EXTENSION extension_name UPDATE [ TO new_version ]
ALTER EXTENSION extension_name SET SCHEMA new_schema
ALTER EXTENSION extension_name ADD member_object
ALTER EXTENSION extension_name DROP member_object
![Page 9: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/9.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Built-In Extensions
![Page 10: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/10.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Built-In Extensions● Bundled with PostgreSQL source code, standard
packages● Full supported and maintained by PostgreSQL
development team● Examples -
○ hstore
○ pg_stat_statements
○ auto_explain
● Some are moved to the server core
○ pg_rewind, pg_waldump, full text search
![Page 11: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/11.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
auto_explain● The auto_explain module provides a means for
logging execution plans of slow statements automatically, without having to run EXPLAIN by hand.
● The module provides no SQL-accessible functions. To use it, simply load it into the server.○ shared_preload_libraries○ session_preload_libraries○ LOAD ‘auto_explain’;
![Page 12: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/12.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
auto_explain ● auto_explain.log_min_duration (integer)
○ minimum statement execution time, in milliseconds, that will cause the statement's plan to be logged
● auto_explain.log_analyze (boolean)○ log EXPLAIN ANALYZE output, rather than just EXPLAIN
output● auto_explain.log_nested_statements (boolean)
○ log nested statements, rather than just the top level statement● auto_explain.sample_rate (real)
○ Control logging rate
![Page 13: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/13.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
pg_stat_statements● provides a means for tracking execution statistics of
all SQL statements executed by a server.● The module must be loaded by adding
pg_stat_statements to shared_preload_libraries in postgresql.conf, because it requires additional shared memory. This means that a server restart is needed to add or remove the module.
● Provides functions and views to access/manipulate stats. Requires CREATE EXTENSION
![Page 14: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/14.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
pg_stat_statements
Name Type Description
userid oid OID of user who executed the statement
dbid oid OID of database in which the statement was executed
queryid bigint Internal hash code, computed from the statement's parse tree
query text Text of a representative statement
calls bigint Number of times executed
![Page 15: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/15.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
pg_stat_statements
Name Type Description
total_time double precision Total time spent in the statement, in milliseconds
min_time double precision Minimum time spent in the statement, in milliseconds
max_time double precision Maximum time spent in the statement, in milliseconds
mean_time double precision Mean time spent in the statement, in milliseconds
stddev_time double precision Population standard deviation of time spent in the statement, in milliseconds
![Page 16: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/16.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
postgres_fdw● A data wrapper to speak to remote PostgreSQL
databases (replaces dblink extension)● Push WHERE clauses, JOINS, ORDER BY,
aggregates● Use transaction hooks to control remote transactions
● ANALYZE remote tables
● You have a distributed database!
![Page 17: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/17.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
HStore: A Key-Value Store● Implements the hstore data type for storing sets
of key/value pairs within a single PostgreSQL value
● A set of operators to operate on the hstore data type
● A set of functions● New index types● Integration with JSON and JSONB types
![Page 18: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/18.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
And More..● pg_buffercache
● pg_prewarm
● pg_visibility
● pg_trgm
● pg_crypto
● pgstattuple
![Page 19: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/19.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Third Party Extensions
![Page 20: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/20.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Geospatial Database - PostGIS● Several geometrical datatypes
● Point, Line, Rectangle, Polygon
● Associated operators, functions
● Associated index access methods
● Maintained and developed by PostGIS community
● You have a fully OpenGIS compatible geospecial database!
![Page 21: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/21.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Logical Replication: pglogical● PostgreSQL core now has logical replication
● Publisher-subscriber model
● Pglogical extends the in-core features
○ Connects to different data sources
○ Row and column filtering
○ Seamlessly replicate DDLs on the subscriber nodes
● You have a complete logical replication in PostgreSQL!
![Page 22: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/22.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Bi-Directional Replication● Uses built-in logical replication, the pglogical
extension to create a multi-master, bi-directional replication solution
● Always-on architecture
● Rolling upgrades
● Geographically distributed database
● You have a multi-master clustering solution!
![Page 23: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/23.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
And many more..● Miss planner hints?
○ pg_hint_plan
● Columnar store?
○ Cstore_fdw
● Timeseries data?
○ TimescaleDB
● Distributed data?
○ Citus
![Page 24: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/24.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Write Your Own Extension?● Faced with PostgreSQL’s limitation?
○ Check if a work-around is available
○ Check someone else has already solved the problem for you (and if the solution is publicly available)
○ Talk to your PostgreSQL support provider.
○ Roll out your own?
![Page 25: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/25.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Knowing PGXS● Build infrastructure provided by PostgreSQL for
building/distributing extensions● Mainly used for extensions which include C code (as
most extensions would do), but can be used otherwise too
● Automates simple build rules
● For very complex extensions, you may need to write your own
![Page 26: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/26.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Sample Makefile# contrib/pg_prewarm/Makefile
MODULE_big = pg_prewarm
OBJS = pg_prewarm.o autoprewarm.o
EXTENSION = pg_prewarm
DATA = pg_prewarm--1.1--1.2.sql pg_prewarm--1.1.sql
PGFILEDESC = "pg_prewarm - preload relation data into system buffer cache"
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
![Page 27: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/27.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Server-side Hooks● Parser Hooks
● Planner Hooks
● Executor Hooks
● Transaction Control Hooks
● Utility command hooks
![Page 28: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/28.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
An Example● Our customer reported TOAST corruption
● Queries started failing with ERRORs; no easy way to find the extent of corruption and the problematic rows○ Sequential scan of the table ends at the first error
○ Index scan on each PK is very costly
![Page 29: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/29.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Simple WayDO $$
DECLARE
baddata TEXT;
badid INT;
BEGIN
FOR badid IN SELECT id FROM badtable LOOP
BEGIN
SELECT badcolumn
INTO columndata
FROM badtable where id = badid;
EXCEPTION
WHEN OTHERS THEN
RAISE NOTICE 'Data for ID % is corrupt', badid;
CONTINUE;
END;
END LOOP;
END;
$$
![Page 30: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/30.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
More hackish (superfast) wayA toastcheck extension
/* * toast_check(relid regclass) * * Verify integrity of toast table. */Datumtoast_check(PG_FUNCTION_ARGS){ Oid relid = PG_GETARG_OID(0);
![Page 31: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/31.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Toastcheck: Scan the Heap /* * Scan all tuples in the base relation. Uses a global heapTuple pointer to * track the heap tuple so that toast routine can quickly know the current * TID.
*/ while ((state.heapTuple = heap_getnext(state.scan, ForwardScanDirection)) != NULL) { ... heap_deform_tuple(state.heapTuple, tupdesc, values, nulls);
![Page 32: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/32.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Toastcheck: ERROR -> NOTICE if (residx != nextidx) { elog(NOTICE, "unexpected chunk number %d (expected %d) for toast value %u in %s ctid (%u,%u), column %s", residx, nextidx, toast_pointer.va_valueid, RelationGetRelationName(state->heaprel), ItemPointerGetBlockNumber(&state->heapTuple->t_self), ItemPointerGetOffsetNumber(&state->heapTuple->t_self), state->colName);
![Page 33: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/33.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Toastcheck: Makefile# contrib/toastcheck/Makefile MODULE_big = toastcheckOBJS = verify_toast.o
EXTENSION = toastcheckDATA = toastcheck--1.0.sqlPGFILEDESC = "toastcheck - function for verifying toast relation integrity"
REGRESS = toastcheck
PG_CONFIG = pg_configPGXS := $(shell $(PG_CONFIG) --pgxs)include $(PGXS)
![Page 34: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/34.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Summary● Significantly extend PostgreSQL capabilities, outside
the core● A huge open source community which is contributing
significantly to PostgreSQL and its adoption● New businesses are being built purely on PostgreSQL
extensions
![Page 35: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/35.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Website https://www.2ndquadrant.com/Blog https://blog.2ndquadrant.com/ Email [email protected]
2ndQuadrantPostgreSQL Solutions
![Page 36: PostgreSQL - PGConf India · 2019-02-20 · Logical Replication: pglogical PostgreSQL core now has logical replication Publisher-subscriber model Pglogical extends the in-core features](https://reader035.vdocuments.site/reader035/viewer/2022081405/5f0ce9d07e708231d437c057/html5/thumbnails/36.jpg)
https://www.2ndQuadrant.com
PostgreSQLExtensions
Thank you!