Managing Resources with PostgreSQL

FOSDEM PGDay 2015

Cédric Villemain cedric@2ndQuadrant.fr

30th January 2015

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 1 / 24

Cédric Villemain

PostgreSQL Expertise

Development, Support, Training

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 2 / 24

What resources ?

• CPU

• RAM

• Disk

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 3 / 24

How ?

• with .... Concurrency

• with .... Quota or Limit

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 4 / 24

Where ?

Operating System

• user process (limits.conf, ...)

• container (cgroup, jail, ...)

• virtualization (VMWare, Xen, ...)

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 5 / 24

Where ?

PostgreSQL

• Main con�guration (postgresql.conf)

• per object

• per role

• per database

• per role in database

• pg_service

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 6 / 24

GUCs with restart

• max_connections

• shared_bu�ers

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 7 / 24

GUCs without restart

• temp_bu�ers

• work_mem

• synchronous_commit

• temp_tablespace

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 8 / 24

More GUCs without restart

• temp_�le_limit

• statement_timeout

• lock_timeout

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 9 / 24

Resources per object

• FUNCTION

• ROLE

• DATABASE

• SYSTEM

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 10 / 24

Function example

ALTER FUNCTION update_datamart()

SET temp_file_limit = -1;

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 11 / 24

Database example

ALTER DATABASE datawarehouse

WITH CONNECTION LIMIT 10; --hard limit

ALTER DATABASE datawarehouse

SET temp_tablespace TO dwh_tblspc;

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 12 / 24

Role example

ALTER ROLE web_user

WITH CONNECTION LIMIT 10; -- hard limit but ...

ALTER ROLE dba

SET work_mem = '128MB';

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 13 / 24

Role in Database example

ALTER ROLE ALL IN DATABASE devel

SET synchronous_commit TO off;

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 14 / 24

System example

ALTER SYSTEM

SET temp_buffers = '12MB';

SELECT pg_reload_conf();

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 15 / 24

Connection Limit

• PostgreSQL �ood & DoS

• pooling & bouncing

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 16 / 24

PgBouncer

• pool_size = 20

• max_client_conn = 2000

my_db = user=web_user host=remote_host pool_size=4

• bonus: PAUSE / RESUME

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 17 / 24

top-like

• pg_activity (python, system only)

• pg_top (C, PostgreSQL extension)

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 18 / 24

pg_proctab

• pg_cputime()

• pg_loadavg()

• pg_memusage()

• pg_proctab()

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 19 / 24

Execution depends on loadavg

if (select load1<1 from pg_loadavg())

then update_datamart();

end if;

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 20 / 24

What about e�ective_cache_size ?

select pg_size_pretty(memcached * 4096)

from pg_memusage;

-- don't forget to count shared_buffers

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 21 / 24

Are we writing a lot on disk?

select pg_size_pretty(wchar) as requested_write,

pg_size_pretty(writes) as really_written,

(writes * 100 / wchar) as percent_really_written

from pg_proctab()

where pid = pg_backend_pid())

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 22 / 24

And next ?

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 23 / 24

Questions ?

Now is the time to ask!

Cédric Villemain cedric@2ndQuadrant.fr () Managing Resources with PostgreSQL 30th January 2015 24 / 24