advanced performance analysis and monitoring of db2 · db2 aktuell 2019 mon_get_pkg_cache_stmt()...

Advanced Performance Analysis and Monitoring of Db2

Matthias Nicola

[email protected]

@mnicolaIBM

Db2 aktuell 2019

Agenda

Identifying high-impact SQL Statements

dbmon Script for Database Monitoring

Analyzing Query Plans / Statistical Views

Section Actuals

Monitoring Backup/Restore operations

Monitoring Enhancements in Db2 11.5

2

Db2 aktuell 2019

Agenda




Section Actuals



3

Db2 aktuell 2019

MON_GET_PKG_CACHE_STMT()

Good entry point for analyzing query problems

A wealth of metrics for any statement in the package cache– Metrics are collected by default

– You can rank and order by any of these metrics

– Aggregate metrics accumulated after each statement execution (no metrics per individual execution)

– Both static and dynamic SQL

– Low overhead

Retains significant workload information with a modest PCKCACHESZ

Package Cache Event Monitor can be configured in cases where cache evictions are causing information to be lost

250+ metrics:

• BP activity

• Prefetching

• Lock waits

• Sort / spilling

• Locking

• Logging

• Execution time

• FCM activity

• Columnar tables

• External tables

• etc.

4

Db2 aktuell 2019

Finding High Impact Queries

SELECT stmt_exec_time, num_executions, stmt_text

FROM TABLE(mon_get_pkg_cache_stmt(null,null,null,-2)) as s

ORDER BY stmt_exec_time DESC

FETCH FIRST 5 ROWS ONLY;

Top 5 queries by statement execution

time in server

STMT_EXEC_TIME NUM_EXECUTIONS STMT

-------------------- -------------------- -----------------------------------------------

3951764 2218111 SELECT s_quantity, s_dist_01, s_dist_02, …

902078 195866 SELECT c_balance, c_delivery_cnt …

619547 212999 DECLARE CUST_CURSOR1 CURSOR FOR SELEC …

480681 221873 SELECT w_tax, c_discount, c_last, c_credit …

441494 20124 SELECT count(distinct S_I_ID) INTO :H …

Statement with thehighest execution time

5

Db2 aktuell 2019

Other High Impact Queries: CPU

SELECT stmt_exec_time, num_executions, total_cpu_time, stmt_text

FROM TABLE(mon_get_pkg_cache_stmt(null,null,null,-2)) AS s

ORDER BY total_cpu_time DESC

FETCH FIRST 5 ROWS ONLY

Top 5 most CPU intensive

queries

6

Db2 aktuell 2019

Other High Impact Queries: CPU and I/O time

Top 5 most CPU intensive

queries

SELECT stmt_exec_time, num_executions,

(pool_read_time + pool_write_time +

direct_read_time + direct_write_time) AS io_time

FROM TABLE(mon_get_pkg_cache_stmt(null,null,null,-2)) AS t

ORDER BY io_time DESC

FETCH FIRST 5 ROWS ONLY Top 5 most I/O intensive

queries

7

SELECT stmt_exec_time, num_executions, total_cpu_time, stmt_text

FROM TABLE(mon_get_pkg_cache_stmt(null,null,null,-2)) AS s

ORDER BY total_cpu_time DESC


Db2 aktuell 2019

CTQ

SORT

SORT

HSJOIN

HSJOIN

Effectiveness of column-organized queries

CTQ operator: switch from columnar processing to row-based processing

Goal: maximize columnar processing, i.e. CTQ very high in the plan!

How can I find queries where a lot of processing is happening "above" the CTQ, in row-based runtime?

8

Db2 aktuell 2019

Effectiveness of column-organized queries

SELECT TOTAL_SECTION_TIME, TOTAL_COL_TIME,

DEC((FLOAT(TOTAL_COL_TIME)/

FLOAT(NULLIF(TOTAL_SECTION_TIME,0)))*100,5,2)

AS PCT_COL_TIME, stmt_text

FROM TABLE(MON_GET_PKG_CACHE_STMT(NULL,NULL,NULL,-1)) AS T

ORDER BY PCT_COL_TIME ASC


TOTAL_SECTION_TIME TOTAL_COL_TIME PCT_COL_TIME STMT_TEXT

-------------------- -------------------- ------------ -----------------------------

5837 2962 50.74 SELECT...

12089 10954 90.61 SELECT...

... ... ...

Compute the ratio of columnar processing time

to overall section processing time to see

how much we’re leveraging the columnar runtime

Only ~50% of the processing has occurred in Db2's optimized columnar runtime

9

Db2 aktuell 2019

Queries with high ratio of wait time to processing time

SELECT total_act_time, total_act_wait_time,

total_act_wait_time / total_act_time

as percentwait,

stmt_text

FROM TABLE (mon_get_pkg_cache_stmt(null,null,null,-2)) as t

ORDER BY percentwait DESC


Find activities that are most impacted

by waits.

Percentage of query time

spent waiting

TOTAL_ACT_TIME TOTAL_ACT_WAIT_TIME PERCENTWAIT STMT_TEXT

----------------- -------------------- ----------- ------------------------------

1481597 1457690 98 DECLARE cur1 CURSOR FOR SELECT …

228 125 55 SELECT s_quantity, s_dist_01,s …

946 234 25 SELECT c_balance, c_delivery_c …

30 5 17 SELECT w_tax, c_discount, c_la …

7645 123 2 SELECT count(distinct S_ID) IN …

10

Db2 aktuell 2019

MON_GET_PKG_CACHE_STMT(): Some Additional Tips

Monitoring data is accumulated and maintained in-memory from point of

database activation until de-activation

Monitoring metrics are incremented globally and do not reset– To compute changes in metrics over a period of time take an initial baseline sample and compute deltas from

that (eg. compute I/O a particular SQL statement has driven over the past 5 mins)

Event monitors can be utilized to capture and persist event based data for

historical analysis– Package cache event monitor for aggregate statement data

– Activity event monitor for individual statement executions

11

Db2 aktuell 2019

Agenda




Section Actuals



12

Db2 aktuell 2019

dbmon Script for Database Monitoring• Calls SQL monitoring functions (mon_get_...) and produces textual report

• Available for 9.7, 10.1, 10.5, 11.1

• Monitors the database by default for 30 seconds (can be changed)

https://www.ibm.com/support/docview.wss?uid=swg21993956

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.admin.perf.doc/doc/t0070377.html

Shell script to call db2mon.sql:./db2mon.sh mydbname >> report.txt

Collects and computes data from monitoring functions

Exports monitoring data to IXF files

Creates tables and imports monitoring data from IXF files

Produces a reports from imported monitoring data

13



Db2 aktuell 2019

dbmon Script for Database Monitoring• Calls SQL monitoring functions (mon_get_...) and produces textual report

• Available for 9.7, 10.1, 10.5, 11.1

• Monitors the database by default for 30 seconds (can be changed)

14



Collects and computes data from monitoring functions

Exports monitoring data to IXF files

Creates tables and imports monitoring data from IXF files

Produces a reports from imported monitoring data

Option to retain mon

data in table format and

collect / analyze in a

different database

Shell script to call db2mon.sql:./db2mon.sh mydbname >> report.txt



Db2 aktuell 2019

dbmon Script – How does db2mon.sql work?

1. Creates DGTTs to hold monitoring data collected at the start and the end of the monitoring interval

2. Collects monitoring information and inserts it into DGTTs (start data)

3. Sleeps for 30 seconds (call dbms_alert.sleep(30);)

4. Collects monitoring information and inserts it into DGTTs (end data)

5. Computes diff between start and end data

6. Productes the report

15

Db2 aktuell 2019

• Currently executing SQL and utilities• Top SQL stmts by exec time, by wait time, by BP activity,…• Rows read, modified, returned• Lock waits, wait times, lock escalations, deadlocks, etc. • Active connections• Overall time spent on compile, execution, connect, rollback, etc.• Sort memory usage and overflows• CPU and I/O utilization• Log reads/writes, etc.• BP and Table Space stats, hit ratios, prefetching, etc.• db cfg, dbm cfg, reg vars• etc.

dbmon Script – How does db2mon.sql work?

1. Creates DGTTs to hold monitoring data collected at the start and the end of the monitoring interval

2. Collects monitoring information and inserts it into DGTTs (start data)

3. Sleeps for 30 seconds (call dbms_alert.sleep(30);)

4. Collects monitoring information and inserts it into DGTTs (end data)

5. Computes diff between start and end data

6. Productes the report

16

Db2 aktuell 2019

dbmon Script – Sample Output

======================================

Top SQL statements by execution time

======================================

select member, integer(num_exec_with_metrics) as num_exec, coord_stmt_exec_time,

decimal(coord_stmt_exec_time / double(num_exec_with_metrics), 10, 2) as avg_coord_exec_time,

decimal( (coord_stmt_exec_time / double(...

NUM_EXEC COORD_STMT_EXEC_TIME AVG_COORD_EXEC_TIME TOTAL_CPU_TIME AVG_CPU_TIME PCT_WAIT_TIME STMT_TEXT

-------- -------------------- ------------------- -------------- ------------ ------------- ----------------------------------------------

61729 301979 4.89 29928189 484 53.67 Se1ect Count(Distinct S_I_ID) from STOCK, ORDE



64562 137670 2.13 3121S907 483 41.59 Se1ect Count(Distinct S_I_ID) from STOCK, ORDE

124839 112563 0.90 4468845 35 89.65 Update STOCK set S_QUANTITY = ?, S_YTD = ?, S_

124833 111863 0.89 3686860 29 90.54 Se1ect S_QUANTITY, S_DIST_01, S_DIST_02, S_DIS

124836 58906 0.46 2807094 22 85.45 Insert into ORDER_LINE va1ues (?, ?, ?, ?, ?,

11350 57011 5.02 1997313 175 93.18 Update ORDER_LINE set OL_DELIVERY_D = ? where






(...)

17

Db2 aktuell 2019

Agenda




Section Actuals



18

Db2 aktuell 2019


DAILY SALES1 Billion

STORE63

CUSTOMER1,507,826

PRODUCT38,450

Time18,250

PROMOTION180

SELECT ...

FROM daily_sales ds,

time t,

product p,

store s

WHERE t.year = 2018

AND t.month = 'March'

AND p.category = 'Phones'

AND s.region = 5

AND ds.datekey = t.datekey

AND ds.storekey = s.storekey

AND ds.prodkey = p.prodkey

GROUP BY ...;

A simple star schema. A simple star join.What could possibly go wrong….?

19

Db2 aktuell 2019

Optimal join order?

20

DAILY SALES

STORE

PRODUCT

TIME

JOIN

JOIN

JOIN

DAILY SALES

STORE

TIME

PRODUCT

JOIN

JOIN

JOIN

…..

The join with the smallest result set should come first, to make subsequent joins cheaper.

Db2 aktuell 2019

• Fact table: Daily Sales• Sales records from the last 3 years• Older data is archived and pruned

• Time dimension• 18,250 dates representing 50 years• 2000-01-01 to 2050-12-31• "Pre-loaded dimension"

• Product dimension• 38,450 products• 22,200 “old” products not sold in last 3 yrs

• Store dimension• 63 stores• Some stores have 10x or 100x more sales

transactions than others

What could possibly go wrong?


STORE63

CUSTOMER1,507,826

PRODUCT38,450

Time18,250

PROMOTION180

21

Db2 aktuell 2019

Cardinality estimation: Time ⨝ DailySales

SELECT ...


time t,

product p,

store s

WHERE t.year = 2018





AND s.region = 5


GROUP BY ...;

Time dimension: 50 years (600 months)

Estimated result of the join:1/600 * 1B = 1.66M rows

But, fact table contains only 3 years (36 months):1/36 * 1B = 27.7M rows

Problem: no stats for t.year+t.month after the join!

~16x

22

Db2 aktuell 2019

Cardinality estimation: Product⨝ DailySales

SELECT ...


time t,

product p,

store s

WHERE t.year = 2018





AND s.region = 5


GROUP BY ...;

Product: 3,921 out of 38,450 products are "Phones".

Estimated result of the join: 3921/38450 * 1B = 102M rows

But, only 114 types of phones were sold in the last 3 years and referenced in the fact table:

114/38450 * 1B = 3M rows

Problem: no stats for p.category after the join!

~34x

24

Db2 aktuell 2019

Optimal join order?

DAILY SALES

STORE

PRODUCT

TIME

JOIN

JOIN

JOIN

DAILY SALES

STORE

TIME

PRODUCT

JOIN

JOIN

JOIN

…..1.66M 102M

Estimate: Time ⨝ DailySales produces fewer rows than Product ⨝ DailySales.

25

Db2 aktuell 2019

Optimal join order?

DAILY SALES

STORE

PRODUCT

TIME

JOIN

JOIN

JOIN

DAILY SALES

STORE

TIME

PRODUCT

JOIN

JOIN

JOIN

…..1.66M27.7M

102M3M

Estimate: Time ⨝ DailySales produces fewer rows than Product ⨝ DailySales. More accurate: the opposite!

26

Db2 aktuell 2019

Optimal join order?

DAILY SALES

STORE

PRODUCT

TIME

JOIN

JOIN

JOIN

DAILY SALES

STORE

TIME

PRODUCT

JOIN

JOIN

JOIN

…..1.66M27.7M

102M3M

With statistical views: Time ⨝ DailySales should not be the first join.

27

Db2 aktuell 2019

Statistical Views

CREATE VIEW stat_sales_time AS

(SELECT t.*

FROM daily_sales ds, time t

WHERE ds.datekey = t.datekey);

CREATE VIEW stat_sales_product AS

(SELECT p.*

FROM daily_sales ds, product p

WHERE ds.prodkey = p.prodkey);

CREATE VIEW stat_sales_store AS

(SELECT s.*

FROM daily_sales ds, store s

WHERE ds.storekey = s.storekey);


STORE63

CUSTOMER1,507,826

PRODUCT38,450

Time18,250

PROMOTION180

1

2

3

1

2

3

29

Db2 aktuell 2019

Using Statistical Views is Easy !

CREATE VIEW stat_sales_time AS

(SELECT t.* FROM daily_sales ds, time t WHERE ds.datekey = t.datekey);

CREATE VIEW stat_sales_product AS

(SELECT p.* FROM daily_sales ds, product p WHERE ds.prodkey = p.prodkey);

CREATE VIEW stat_sales_store AS

(SELECT s.* FROM daily_sales ds, store s WHERE ds.storekey = s.storekey);

ALTER VIEW stat_sales_time ENABLE QUERY OPTIMIZATION;

ALTER VIEW stat_sales_product ENABLE QUERY OPTIMIZATION;

ALTER VIEW stat_sales_store ENABLE QUERY OPTIMIZATION;

RUNSTATS ON VIEW mnicola.stat_sales_time WITH DISTRIBUTION...;

RUNSTATS ON VIEW mnicola.stat_sales_product WITH DISTRIBUTION...;

RUNSTATS ON VIEW mnicola.stat_sales_store WITH DISTRIBUTION...;

1. Create views for the critical joins

2. Enable the views for query optimization

3. Collect statistics on the result set of the views

30

Db2 aktuell 2019

Statistical Views - Summary

• Statistical view can be useful for queries with:• pre-loaded dimensions• heavy skew in join columns• grouping on columns from multiple tables• complex predicates/functions that cannot be estimated otherwise• nick names• etc.

• Create view, enable for optimization, runstats

• View is not materialized, not referenced in user queries

• View statistics are used if query matches the view (like MQT matching)

33

Db2 aktuell 2019

Agenda




Section Actuals



34

Db2 aktuell 2019

Estimation vs Reality

How do I know if or which cardinality estimates in the

chosen execution plan are inaccurate?

35

Db2 aktuell 2019

Estimation (Compile) vs Reality (Execution)

How do I know if or which cardinality estimates in the

chosen execution plan are inaccurate?

Look at the "section actuals"!

Section actuals =

number of rows produced by

each operator in the plan,

measured at execution time

not estimated at compile time!

36

Db2 aktuell 2019

db2caem (db2 capture activity event monitor)

One of multiple different ways to capture explain with section actuals !– For queries without parameter markers.

db2caem -d <db_name> -o <output_dir> -sf <query_file>

Example:

mkdir caem_out

db2caem -d SAMPLE -o caem_out -sf myquery.sql

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0057282.html

37

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0057282.html

Db2 aktuell 2019db2exfmt Output db2caem OutputRows

RETURN

( 1)

Cost

I/O

|

1.0123

NLJOIN

( 2)

570.128

551.026

/-------+-------\

0.034 0.999855

TBSCAN FETCH

( 3) ( 4)

554.792 15.1474

549 2.0013

| /---+----\

1206918 0.999855 16917

TABLE: T1 IXSCAN TABLE: T2

( 5)

7.5741

1

|

16917

INDEX: IDX2

Rows

Rows Actual

RETURN

( 1)

Cost

I/O

|

1.0123

1

NLJOIN

( 2)

570.128

/-------+-------\

0.034 0.999855

25072 1

TBSCAN FETCH

( 3) ( 4)

554.792 15.1474

NA NA

| /---+----\

1206918 0.999855 16917

NA 1 NA


( 5)

7.5741

|

16917

NA

INDEX: IDX2

38


RETURN

( 1)

Cost

I/O

|

1.0123

NLJOIN

( 2)

570.128

551.026

/-------+-------\

0.034 0.999855

TBSCAN FETCH

( 3) ( 4)

554.792 15.1474

549 2.0013

| /---+----\

1206918 0.999855 16917


( 5)

7.5741

1

|

16917

INDEX: IDX2

Rows

Rows Actual

RETURN

( 1)

Cost

I/O

|

1.0123

1

NLJOIN

( 2)

570.128

/-------+-------\

0.034 0.999855

25072 1

TBSCAN FETCH

( 3) ( 4)

554.792 15.1474

NA NA

| /---+----\

1206918 0.999855 16917

NA 1 NA


( 5)

7.5741

|

16917

NA

INDEX: IDX2

Cardinality estimates

Actual #rows(Section Actuals)

39


RETURN

( 1)

Cost

I/O

|

1.0123

NLJOIN

( 2)

570.128

551.026

/-------+-------\

0.034 0.999855

TBSCAN FETCH

( 3) ( 4)

554.792 15.1474

549 2.0013

| /---+----\

1206918 0.999855 16917


( 5)

7.5741

1

|

16917

INDEX: IDX2

Rows

Rows Actual

RETURN

( 1)

Cost

I/O

|

1.0123

1

NLJOIN

( 2)

570.128

/-------+-------\

0.034 0.999855

25072 1

TBSCAN FETCH

( 3) ( 4)

554.792 15.1474

NA NA

| /---+----\

6918 0.999855 16917

NA 1 NA


( 5)

7.5741

|

16917

NA

INDEX: IDX2

0.034

25072

TBSCAN

40

Db2 aktuell 2019

Agenda




Section Actuals



41

Db2 aktuell 2019

Monitoring backup/restore

• "list utilities show detail" for backup and restore operations

• Some monitoring elements for online backups:• TOTAL_BACKUPS: Number of online backups• TOTAL_BACKUP_TIME: Total elapsed time spent on online backups (msec)• TOTAL_BACKUP_PROC_TIME: Total processing time (no wait time) for online

backups

42

Db2 aktuell 2019

Monitoring backup/restore with db2pd -barstats

• Three steps:

1. Find the application handle whose status is "PerformingBackup" or "RestoringDatabase":$ db2pd –db <dbname> -appinfo | grep –B20 "PerformingBackup" | grep "ApplHandl"

AppHandl [nod-index] : 57 [000-00057]

2. Find the agent ID of the agent running the backup or restore operations: $ db2pd –db <dbname> -agents | grep " 57 " | grep "Subagents" | awk '{print $4}'

409

3. Run db2pf with the –barstats option for the identified agent:$ db2pd –db <dbname> -barstats 409

43

11.1.4.4

https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.admin.ha.doc/doc/t0070439.html

https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.admin.ha.doc/doc/t0070439.html

Db2 aktuell 2019

Printing out Backup Runtime Statistics at 2019-05-11-21.13.47.925472

...

[1] tblSpaceName: FACTDATA

tblSpaceID: 3

backupStatus: in progress

Being backed up by: db2bm.18.0

Object being backed up: DMS.TABLESPACE.DATA

Number of pages already backed up: 524288

Total number of pages in table space: 3702752

...

Parallelism = 4

BM# Total I/O MsgQ WaitQ Buffers Bytes

--- -------- -------- -------- -------- -------- --------

000 128.12 39.18 0.00 0.00 1 98304

001 128.31 40.62 0.00 0.00 1 98397

002 128.30 40.63 0.00 0.00 1 98371

003 127.79 40.69 0.00 0.00 1 98401

--- -------- -------- -------- -------- -------- --------

TOT 512.52 161.12 0.00 0.00 1 393473

44

Status info for

each table space

Progress info for

each table space

Perf statistics for

each backup thread

(Bytes or Mbytes)

Sample Output from db2pd -barstats

Db2 aktuell 2019

Progress Monitor:

---------------------------------------------------------------------

Phase #: 1 of 1

Total work units (bytes): 173981696

Completed work units (bytes): 14916218

Size estimates:

---------------------------------------------------------------------

Total size estimate (bytes): 173981696

Pre-adjusted total size estimate (bytes): 173981696

Init data estimate (bytes): 1492516

User data estimate (bytes): 159252480

End data estimate (bytes): 25624

Size estimate for MC1 (bytes): 173981696

Size estimate for remaining MCs (bytes): 172408832

45

Progress info for

entire backup

Total amount of data

being backed up.

Database meta data

Database meta data

https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.admin.ha.doc/doc/c0070441.html

Sample Output from db2pd -barstats

https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.admin.ha.doc/doc/c0070441.html

Db2 aktuell 2019

Agenda




Section Actuals



46

Db2 aktuell 2019

Monitoring enhancements in Db2 11.5

• New monitor elements to monitor error and warning from SQL statements

• Improved table functions provide monitoring details without having to perform joins

• New monitor element ACT_TOP for high water mark of concurrently executing activities

• New monitor interfaces for workloads at the service superclass level

• Improved table functions provide direct access to WLM statistics

• New monitor elements report parallelism and memory usage for workload and service

class objects

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.5.0/com.ibm.db2.luw.wn.doc/doc/c0055019.html

47

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.5.0/com.ibm.db2.luw.wn.doc/doc/c0055019.html

Db2 aktuell 2019

New in Db2 11.5: Monitoring SQL Failures and Warnings It is now possible to monitor the failure rate of an SQL statement using the

MON_GET_PKG_CACHE_STMT table function.

The following columns have been added to MON_GET_PKG_CACHE_STMT

– LAST_EXEC_ERROR: The Error SQLCODE for the last error of this statement

– LAST_EXEC_ERROR_SQLERRMC: The error tokens for the last error of this statement

– LAST_EXEC_ERROR_TIMESTAMP: Time when the last error for this statement occurred

– LAST_EXEC_WARNING: The Warning SQLCODE for the last warning of this statement

– LAST_EXEC_WARNING_SQLERRMC: The tokens for the last warning of this statement

– LAST_EXEC_WARNING_TIMESTAMP: Time when the last warning for this statement occurred

– NUM_EXEC_WITH_ERROR: The number of executions that resulted in an error

– NUM_EXEC_WITH_WARNING: The number of executions that resulted in a warning

48

Db2 aktuell 2019

Summary




Section Actuals



50

Advanced Performance Analysis and Monitoring of Db2 Matthias Nicola

[email protected]

@mnicolaIBM

Questions ?

advanced performance analysis and monitoring of db2 · db2 aktuell 2019 mon_get_pkg_cache_stmt()...

Documents