pg_proctab: accessing system stats in postgresql
TRANSCRIPT
pg proctabAccessing System Stats in PostgreSQL
Mark Wong [email protected] Roth [email protected]
PGWest Seattle (JDCon) 2009
Oct 16-18, 2009
Soooo . . .
You can query the PostgreSQL system catalog tables (e.g.pg stat activity, pg stat all tables,pg stat all indexes) to find out which queries are taking a longtime, which indexes are being scanned an unreasonable number oftimes, etc.
Example:
portal=# SELECT datname, procpid, usename, current_query
FROM pg_stat_activity;
datname | procpid | usename | current_query
---------+---------+----------+-------------------------------------------------
portal | 5412 | markwkm | <IDLE>
portal | 5437 | postgres | SELECT datname, procpid, usename, current_query
: FROM pg_stat_activity;
(2 rows)
What if you want to know about the OS?
pg proctab provides a collection of four C stored functions:
◮ pg cputime
◮ pg loadavg
◮ pg memusage
◮ pg proctab
What can do you with pg proctab?
◮ Query operating system process table
◮ Query operating system statistics◮ Processor time◮ Load averages◮ Memory usage
◮ Without escaping out to a shell!
◮ ...plus generate reports about timeslices
pg cputime() Example
SELECT *
FROM pg_cputime();
user | nice | system | idle | iowait
--------+--------+--------+------------+--------
681317 | 109924 | 395481 | 1466101128 | 462661
(1 row)
pg cputime() Column Description
From Linux kernel source code atDocumentation/filesystems/proc.txt:user: normal processes executing in user modenice: niced processes executing in user modesystem: processes executing in kernel modeidle: processes twiddling thumbsiowait: waiting for I/O to complete
pg loadavg() Example
SELECT *
FROM pg_loadavg();
load1 | load5 | load15 | last_pid
-------+-------+--------+----------
0.99 | 0.78 | 0.67 | 27719
(1 row)
pg loadavg() Column Description
load1: load average of last minuteload5: load average of last 5 minutesload15: load average of last 15 minuteslast pid: last pid running
pg memusage() Example
SELECT *
FROM pg_memusage();
memused | memfree | memshared | membuffers | memcached | swapused | swapfree | swapcached
---------+---------+-----------+------------+-----------+----------+----------+------------
3809140 | 224084 | 0 | 60656 | 2389700 | 76 | 8385844 | 0
(1 row)
pg memusage() Column Description
Paraphrased from Linux kernel source code atDocumentation/filesystems/proc.txt:memused: Total physical RAM usedmemfree: Total physical RAM not usedmemshared: Not used, always 0. (I don’t remember why. . . )membuffers: Temporary storage for raw disk blocksmemcached: In-memory cache for files read from diskswapused: Total swap space usedswapfree: Memory evicted from RAM that is now temporary ondiskswapcached: Memory that was swapped out, now swapped in butstill in swap
pg proctab() Example 1
SELECT datname, procpid, usesysid, usename, uid, username
FROM pg_stat_activity, pg_proctab()
WHERE procpid = pid;
datname | procpid | usesysid | usename | uid | username
---------+---------+----------+----------+-----+----------
markwkm | 27801 | 10 | markwkm | 500 | markwkm
dbt3 | 27787 | 16770 | postgres | 500 | markwkm
(2 rows)
pg proctab() Example 2
SELECT datname, procpid, processor, state, fullcomm
FROM pg_stat_activity, pg_proctab()
WHERE procpid = pid;
datname | procpid | processor | state | fullcomm
---------+---------+-----------+-------+------------------------------------------
markwkm | 27801 | 0 | R | postgres: markwkm markwkm [local] SELECT
dbt3 | 29325 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29327 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29333 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29328 | 2 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29329 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29324 | 3 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 29331 | 0 | R | postgres: markwkm dbt3 [local] SELECT
dbt3 | 27787 | 1 | S | postgres: postgres dbt3 [local] idle
(9 rows)
pg proctab() Partial Column Description
Everything from the operating system such as /proc/<pid>/stat,/proc/<pid>/io and /proc/<pid>/cmdline as well as datafrom PostgreSQL system catalog such as pg stat activity tableare available but we’ll only cover some of the fields here:Informative:
◮ pid
◮ comm - filename of the executable
◮ fullcomm (/proc/<pid>/cmdline)
◮ uid
◮ username
Processor:
◮ utime - user mode jiffies
◮ stime - kernel mode jiffies
. . .
pg proctab() Partial Column Description (cont.)
Memory:
◮ vsize - virtual memory size
◮ rss - resident set memory size
I/O:
◮ syscr - number of read I/O operations
◮ syscw - number of write I/O operations
◮ reads - number of bytes which this process really did cause tobe fetched from the storage layer
◮ writes - number of bytes which this process really did cause tobe sent from the storage layer
◮ cwrites - number of bytes which this process caused to nothappen, by truncating pagecache
__ __ / \
/ \~~~/ \ . o O | Let’s try something |
,----( oo ) | more useful. |
/ \__ __/ \ /
/| (\ |(
^ \ /___\ /\ |
|__| |__|-"
__ __ / \
/ \~~~/ \ . o O | Measuring performance |
,----( oo ) | of a query. |
/ \__ __/ \ /
/| (\ |(
^ \ /___\ /\ |
|__| |__|-"
(You can find the following examples in the pg proctab contribdirectory.)
Create snapshot tables.
(Only need to do this once.)
\i create-ps_procstat-tables.sql
Identify yourself.
SELECT *
FROM pg_backend_pid();
pg_backend_pid
----------------
4590
(1 row)
Take a snapshot before running the query
\i ps_procstat-snap.sql
BEGIN
ps_snap_stats
---------------
1
(1 row)
COMMIT
Execute the query
Don’t focus too much on the actual query, the idea is that is youwant to collect statistics for a single query:
SELECT nation,
o_year,
Sum(amount) AS sum_profit
FROM (SELECT n_name AS nation,
Extract(YEAR FROM o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount
FROM part,
supplier,
lineitem,
partsupp,
orders,
nation
WHERE s_suppkey = l_suppkey
AND ps_suppkey = l_suppkey
AND ps_partkey = l_partkey
AND p_partkey = l_partkey
AND o_orderkey = l_orderkey
AND s_nationkey = n_nationkey
AND p_name LIKE ’%white%’) AS profit
GROUP BY nation,
o_year
ORDER BY nation,
o_year DESC;
Take a snapshot after running the query
\i ps_procstat-snap.sql
BEGIN
ps_snap_stats
---------------
2
(1 row)
COMMIT
Calculate Processor Utilization
$ ./ps-processor-utilization.sh [pid] [before] [after]
$ ./ps-processor-utilization.sh 4590 1 2
Processor Utilization = 1.00 %
What’s going on (partially):
SELECT stime, utime, stime + utime AS total,
extract(epoch FROM time)
FROM ps_snaps a, ps_procstat b
WHERE pid = ${PID}
AND a.snap = b.snap
AND a.snap = ${SNAP1}
TIMEDIFF=‘echo "scale = 2; (${TIME2} - ${TIME1}) * ${HZ}" | bc -l‘
U=‘echo "scale = 2; (${TOTAL2} - ${TOTAL1}) / ${TIMEDIFF} * 100" | bc -l‘
Calculate Disk Utilization
$ ./ps-io-utilization.sh 4590 1 2
Reads = 276981
Writes = 63803
Reads (Bytes) = 2164604928
Writes (Bytes) = 508166144
Cancelled (Bytes) = 36880384
SELECT syscr, syscw, reads, writes, cwrites
FROM ps_snaps a, ps_procstat b
WHERE pid = ${PID}
AND a.snap = b.snap
AND a.snap = ${SNAP1}
TIMEDIFF=‘echo "scale = 2; (${TIME2} - ${TIME1}) * ${HZ}" | bc -l‘
U=‘echo "scale = 2; (${TOTAL2} - ${TOTAL1}) / ${TIMEDIFF} * 100" | bc -l‘
__ __ / \
/ \~~~/ \ . o O | Creating Custom |
,----( oo ) | Reports! |
/ \__ __/ \ /
/| (\ |(
^ \ /___\ /\ |
|__| |__|-"
__ __ / \
/ \~~~/ \ . o O | Warning! Too much data |
,----( oo ) | to fit on screen! |
/ \__ __/ \ /
/| (\ |(
^ \ /___\ /\ |
|__| |__|-"
Creating Reports: Section 1
Database : dbt3
Snapshot Start : 2009-04-18 00:43:56.716034-07
Snapshot End : 2009-04-18 00:45:17.031167-07
-------------------
Database Statistics
-------------------
Commits : 0
Rollbacks : 2
Blocks Read : 213295
Blocks Hit : 1679509
Creating Reports: Section 2
================
Table Statistics
================
------------------------------------------ -------- ------------ -------- ------------- --------- ---------
Schema.Relation Seq Scan Seq Tup Read Idx Scan Idx Tup Fetch N Tup Ins N Tup Upd
------------------------------------------ -------- ------------ -------- ------------- --------- ---------
information_schema.sql_features 0 0 0 0 0
information_schema.sql_implementation_info 0 0 0 0 0
information_schema.sql_languages 0 0 0 0 0
information_schema.sql_packages 0 0 0 0 0
information_schema.sql_parts 0 0 0 0 0
information_schema.sql_sizing 0 0 0 0 0
information_schema.sql_sizing_profiles 0 0 0 0 0
pg_catalog.pg_aggregate 0 0 2 2 0
pg_catalog.pg_am 1 1 0 0 0
pg_catalog.pg_amop 0 0 19 46 0
pg_catalog.pg_amproc 0 0 11 11 0
pg_catalog.pg_attrdef 0 0 1 2 0
pg_catalog.pg_attribute 0 0 137 331 0
pg_catalog.pg_auth_members 0 0 0 0 0
pg_catalog.pg_authid 3 2 0 0 0
pg_catalog.pg_autovacuum 0 0 0 0 0
pg_catalog.pg_cast 0 0 160 51 0
pg_catalog.pg_class 3 747 101 88 0
pg_catalog.pg_constraint 0 0 0 0 0
pg_catalog.pg_conversion 0 0 0 0 0
pg_catalog.pg_database 5 12 0 0 0
pg_catalog.pg_depend 0 0 0 0 0
pg_catalog.pg_description 0 0 0 0 0
pg_catalog.pg_index 2 200 39 50 0
...
Creating Reports: Section 2 - Falling off the right side...
◮ N Tup Upd
◮ N Tup Del
◮ Last Vacuum
◮ Last Autovacuum
◮ Last Analyze
◮ Last Autoanalyze
Creating Reports: Section 3
================
Index Statistics
================
------------------------------------------------------------ -------- ------------ -------------
Schema.Relation.Index Idx Scan Idx Tup Read Idx Tup Fetch
------------------------------------------------------------ -------- ------------ -------------
pg_catalog.pg_aggregate.pg_aggregate_fnoid_index 2 2 2
pg_catalog.pg_am.pg_am_name_index 0 0 0
pg_catalog.pg_am.pg_am_oid_index 0 0 0
pg_catalog.pg_amop.pg_amop_opc_strat_index 12 36 36
pg_catalog.pg_amop.pg_amop_opr_opc_index 7 10 10
pg_catalog.pg_amproc.pg_amproc_opc_proc_index 11 11 11
pg_catalog.pg_attrdef.pg_attrdef_adrelid_adnum_index 1 2 2
pg_catalog.pg_attrdef.pg_attrdef_oid_index 0 0 0
pg_catalog.pg_attribute.pg_attribute_relid_attnam_index 0 0 0
pg_catalog.pg_attribute.pg_attribute_relid_attnum_index 137 331 331
pg_catalog.pg_auth_members.pg_auth_members_member_role_index 0 0 0
pg_catalog.pg_auth_members.pg_auth_members_role_member_index 0 0 0
pg_catalog.pg_authid.pg_authid_oid_index 0 0 0
pg_catalog.pg_authid.pg_authid_rolname_index 0 0 0
pg_catalog.pg_autovacuum.pg_autovacuum_vacrelid_index 0 0 0
pg_catalog.pg_cast.pg_cast_oid_index 0 0 0
pg_catalog.pg_cast.pg_cast_source_target_index 160 51 51
pg_catalog.pg_class.pg_class_oid_index 71 71 71
pg_catalog.pg_class.pg_class_relname_nsp_index 30 17 17
pg_catalog.pg_constraint.pg_constraint_conname_nsp_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_conrelid_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_contypid_index 0 0 0
pg_catalog.pg_constraint.pg_constraint_oid_index 0 0 0
pg_catalog.pg_conversion.pg_conversion_default_index 0 0 0
...
What else can we do with pg proctab?
Enable pg top to monitor remote databases by providing access tothe database system’s operating system process table.
pg top
__ __
/ \~~~/ \ . o O ( Thank you! )
,----( oo )
/ \__ __/
/| (\ |(
^ \ /___\ /\ |
|__| |__|-"
. . . the fine print . . .
◮ Linux-only
◮ Developed on 8.3; still works on 8.4
◮ Download it from:http://git.postgresql.org/gitweb?p=pg_proctab.git
◮ Change it:git clone
git://git.postgresql.org/git/pg_proctab.git
◮ Patches welcome! We’ll be in the (Sn—H)ackers’ Lounge!
Acknowledgements
Haley Jane Wakenshaw
__ __
/ \~~~/ \
,----( oo )
/ \__ __/
/| (\ |(
^ \ /___\ /\ |
|__| |__|-"
License
This work is licensed under a Creative Commons Attribution 3.0Unported License. To view a copy of this license, (a) visithttp://creativecommons.org/licenses/by/3.0/us/; or, (b)send a letter to Creative Commons, 171 2nd Street, Suite 300, SanFrancisco, California, 94105, USA.