wdug storage basics

© Copyright 2020 XTIVIA, Inc. The information contained herein is subject to change without notice.

www.xtivia.com 888.685.3101 ext. 2

Storage Basics for Db2: Faster I/O, Fewer Headaches


Agenda

● Db2 Storage Basics

● System Storage Basics

● Db2 “Optimal Filesystem Architecture”

● Key Performance Metrics

● Finding I/O Bottlenecks


Db2 Storage Basics


Logical Hierarchy

Tables


Logical Hierarchy

Tables

Tablespace


Logical Hierarchy

Tables

Tablespace

Tables

Tablespace


Logical Hierarchy

Tables

Tablespace

Tables

Tablespace

Storage Group


Logical Hierarchy


Logical Hierarchy

Storage Path(s) Storage Path(s)


Tablespaces

● Each tablespace has a page size: 4096, 8192, 16384 or 32768 bytes


Tablespaces


● Tablespace Types

● Regular

● Large

● System Temporary

● User Temporary


Tablespaces



● Regular

● Large


● User Temporary

❌ Limits on size, rows/page; not reclaimable


Tablespaces



● Regular

● Large


● User Temporary

● Storage Types

● Automatic Storage

● Managed by Database (DMS)

● Managed by System (SMS)



Tablespaces



● Regular

● Large


● User Temporary

● Storage Types

● Automatic Storage

● Managed by Database (DMS)

● Managed by System (SMS)

✅

❌ Deprecated

❌ Deprecated



Storage Groups

● Storage is defined on one or more directories on the server

● STOGROUP Parameters

● OVERHEAD: Seek time (milliseconds)

● DEVICE READ RATE: Throughput (megabytes/s)

● Cases for using multiple STOGROUPS in a database

● “Multi-Temperature” Storage (SSD vs Disk)

● Limiting Temporary Tablespace Utilization


Where is my database stored?

$ db2 "SELECT type, path FROM sysibmadm.dbpaths ORDER BY 1,2"



$ db2 "SELECT type, path FROM sysibmadm.dbpaths ORDER BY 1,2"

TYPE PATH -------------------- ------------------------------------------------------- DB_STORAGE_PATH /database/data/ DBPATH /database/data/db2inst1/NODE0000/SQL00001/ DBPATH /database/data/db2inst1/NODE0000/SQL00001/MEMBER0000/ LOCAL_DB_DIRECTORY /database/data/db2inst1/NODE0000/sqldbdir/ LOGPATH /database/data/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/ 5 record(s) selected.


SELECT type, path FROM sysibmadm.dbpaths UNION ALL SELECT 'ARCHIVE_LOGPATH', replace(value,'DISK:','') FROM sysibmadm.dbcfg WHERE name in ('logarchmeth1','logarchmeth2') AND value like ‘DISK:%' UNION ALL SELECT upper(name), value FROM sysibmadm.dbmcfg WHERE name = 'keystore_location' AND EXISTS (SELECT 1 FROM sysibmadm.dbcfg WHERE name = 'encrypted_database' AND value = 'YES');



SELECT type, path FROM sysibmadm.dbpaths UNION ALL SELECT 'ARCHIVE_LOGPATH', replace(value,'DISK:','') FROM sysibmadm.dbcfg WHERE name in ('logarchmeth1','logarchmeth2') AND value like 'DISK:%' UNION ALL SELECT upper(name), value FROM sysibmadm.dbmcfg WHERE name = 'keystore_location' AND EXISTS (SELECT 1 FROM sysibmadm.dbcfg WHERE name = 'encrypted_database' AND value = 'YES');



SELECT type, path FROM sysibmadm.dbpaths UNION ALL SELECT 'ARCHIVE_LOGPATH', replace(value,'DISK:','') FROM sysibmadm.dbcfg WHERE name in ('logarchmeth1','logarchmeth2') AND value like ‘DISK:%' UNION ALL SELECT upper(name), value FROM sysibmadm.dbmcfg WHERE name = 'keystore_location' AND EXISTS (SELECT 1 FROM sysibmadm.dbcfg WHERE name = 'encrypted_database' AND value = 'YES');



SELECT type, pathFROM sysibmadm.dbpaths

UNION ALLSELECT 'ARCHIVE_LOGPATH', replace(value,'DISK:','')

FROM sysibmadm.dbcfgWHERE name in ('logarchmeth1','logarchmeth2')

AND value like 'DISK:%'UNION ALLSELECT upper(name), value

FROM sysibmadm.dbmcfgWHERE name = 'keystore_location'

AND EXISTS (SELECT 1 FROM sysibmadm.dbcfgWHERE name = 'encrypted_database' AND value = 'YES');



SELECT 'ARCHIVE_LOGPATH', replace(value,'DISK:','') FROM sysibmadm.dbcfg WHERE name in ('logarchmeth1','logarchmeth2') AND value like 'DISK:%' UNION ALL SELECT upper(name), value FROM sysibmadm.dbmcfg WHERE name = 'keystore_location' AND EXISTS (SELECT 1 FROM sysibmadm.dbcfg WHERE name = 'encrypted_database' AND value = 'YES');

TYPE PATH -------------------- ------------------------------------------------------- ARCHIVE_LOGPATH /database/config/db2inst1/archive_logs/ DB_STORAGE_PATH /database/data/ DBPATH /database/data/db2inst1/NODE0000/SQL00001/ DBPATH /database/data/db2inst1/NODE0000/SQL00001/MEMBER0000/ KEYSTORE_LOCATION /database/config/db2inst1/.security/db2inst1.p12 LOCAL_DB_DIRECTORY /database/data/db2inst1/NODE0000/sqldbdir/ LOGPATH /database/data/db2inst1/NODE0000/SQL00001/LOGSTREAM0000/ 7 record(s) selected.



Important Database Files

Location Requires OS-Level Backup

Storage Path(s) No

Active Transaction Logs No

Archive Logs Yes

Local Database Directory Yes

Keystore - Encryption Yes


Important Database Files

Location Requires OS-Level Backup

Storage Path(s) No

Active Transaction Logs No

Archive Logs Yes

Local Database Directory Yes

Keystore - Encryption Yes

† If using LOGARCHMETH DISK

†


System Storage Basics

Disk


Storage Types and Devices

● Direct-Attached Storage Devices

● Stand-alone Disks

● Local Storage Arrays - RAID

● SAN

● Virtual Machines: VDisks

● Network Attached Storage


Logical Volume Manager

● A tool to create virtual block devices from physical devices

● Eliminates restrictions from partitioning hard drives

● One block device can span multiple physical devices

● Used on AIX, Linux

● Logical Device Manager & Storage Spaces provide similar functionality on Windows


Mapping Filesystems to Physical Disks with LVM

Task Linux AIX

List Volume Groups vgs lsvg

List Logical Volumeslvs <vg> lvdisplay <vg>

lsvg -l <vg>

List Physical Volumes pvslspv lsvg -p <vg>

List Filesystems on a Physical Volume

pvdisplay <pv> -m lsblk

lspv -l <pv>

† Requires root privileges

†


$ lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 96G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 95.5G 0 part ├─vglocal20161128-swap00 253:1 0 2G 0 lvm [SWAP] ├─vglocal20161128-tmp00 253:2 0 12G 0 lvm /tmp └─vglocal20161128-root00 253:3 0 171.5G 0 lvm / sdb 8:16 0 128G 0 disk └─sdb1 8:17 0 128G 0 part └─data_vg-data_lv 253:0 0 128G 0 lvm /data sde 8:64 0 20G 0 disk └─sde1 8:65 0 20G 0 part ├─vglocal20161128-tmp00 253:2 0 128G 0 lvm /tmp └─vglocal20161128-root00 253:3 0 171.5G 0 lvm /

Exploring lsblk


$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 96G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 95.5G 0 part ├─vglocal20161128-swap00 253:1 0 2G 0 lvm [SWAP] ├─vglocal20161128-tmp00 253:2 0 12G 0 lvm /tmp └─vglocal20161128-root00 253:3 0 171.5G 0 lvm / sdb 8:16 0 128G 0 disk └─sdb1 8:17 0 128G 0 part └─data_vg-data_lv 253:0 0 128G 0 lvm /data sde 8:64 0 20G 0 disk └─sde1 8:65 0 20G 0 part ├─vglocal20161128-tmp00 253:2 0 128G 0 lvm /tmp └─vglocal20161128-root00 253:3 0 171.5G 0 lvm /

Exploring lsblk

Volume Group



Exploring lsblk

Logical Volume



Exploring lsblk

Filesystem



Exploring lsblk

Sharing 1 physical device



Exploring lsblk

1 filesystem spanning multiple physical devices


RAID Arrays and Db2

● RAID Arrays are typically N+1 disks

● Strip (aka Element) size is commonly 32kb or 64kb

Disk Disk Disk Disk Disk Disk

RAID-5

Stripe Size

Strip Size

Strip size 64 kbStripe size 320 kb

EXTENTSIZE 320K


RAID Arrays and Db2



● Match Db2 extent size to RAID stripe size


RAID-5

Stripe Size

Strip Size


EXTENTSIZE 320K


RAID Arrays and Db2



● Match Db2 extent size to RAID stripe size

● With SAN virtualization, a single LUN may be spread over multiple RAID arrays

● Use EXTENTSIZE 256K


RAID-5

Stripe Size

Strip Size


EXTENTSIZE 320K


Disk Disk Disk

RAID-5

Hardware Level 1 logical unit on

multiple physical disks

Understanding Striping


Disk Disk Disk

RAID-5

Disk Disk Disk

RAID-5

/db2/data01

LVM Level 1 logical volume on

multiple physical volumes





IBMSTOGROUP

Disk Disk Disk

RAID-5

Disk Disk Disk

RAID-5

/db2/data01 /db2/data02

Db2 Level 1 STOGROUP with

multiple storage paths








IBMSTOGROUP

Disk Disk Disk

RAID-5

Disk Disk Disk

RAID-5


❌









IBMSTOGROUP

Disk Disk Disk

RAID-5

Disk Disk Disk

RAID-5


✅

✅

❌








How Db2 Processes I/O

Disk Disk Disk

RAID-5

Disk Disk Disk

RAID-5


● Db2 allocates new extents to each storage path in a round-robin fashion

IBMSTOGROUP



Disk Disk Disk

RAID-5

Disk Disk Disk

RAID-5


● When reading, multiple prefetchers can run in parallel, 1 per storage path

IBMSTOGROUP




Disk Disk Disk

RAID-5

Disk Disk Disk

RAID-5


● When reading, multiple prefetchers can run in parallel, 1 per storage path

● When DB2_PARALLEL_IO is enabled, multiple prefetchers will be used for each storage path

IBMSTOGROUP

db2set DB2_PARALLEL_IO=*,3



Db2 Optimal Filesystem Architecture


Db2 OFA Purpose

Define a standard directory structure that provides flexibility and support for multiple instances and databases on a single system.

Using this recommendation will provide a solid foundation for building out Db2 servers regardless of database size and performance requirements.


Directory Layout

● System-Level Directories

● Installation path

● Global Registry

● Database Backups

● Instance-Level Directories

● Instance Owner home

● Diagnostic Log Path

● Database-Level Directories

● Database Path

● Storage Path(s)

● Active Log Path

● Archive Log Path

Resolved Path

/opt/ibm/db2/V11.5.4.0

/var/db2

/db2/backups

/db2/db2inst1

/db2/db2dump/db2inst1/DIAG0000

/db2/db2inst1/NODE0000/SQL00001/

/db2/data01/db2inst1/NODE0000/TESTDB/

/db2/active_logs/db2inst1/TESTDB/NODE0000/LOGSTREAM0000

/db2/archive_logs/db2inst1/TESTDB/NODE0000/LOGSTREAM0000




● Global Registry






● Database Path

● Storage Path(s)

● Active Log Path


Resolved Path (Filesystem shown in Red)


/var/db2

/db2/backups

/db2/db2inst1



/db2/data01/db2inst1/NODE0000/TESTDB



FS Size

10 Gb

Default

Varies

8 Gb

8 Gb

Varies

Varies

Varies

Directory – Filesystem Layout




● Global Registry






● Database Path

● Storage Path(s)

● Active Log Path


Resolved Path (Filesystem shown in Red)


/var/db2

/db2/backups

/db2/db2inst1



/db2/data01/db2inst1/NODE0000/TESTDB/



FS Size

Default

Varies

Directory – Filesystem Layout - Flexibility


Creating a Database using Db2 OFA

$ db2 create database testdb \on /db2/data01, /db2/data02 \dbpath on /db2




DB20000I The CREATE DATABASE command completed successfully.




DB20000I The CREATE DATABASE command completed successfully.

$ db2 update db cfg for testdb using \newlogpath /db2/active_logs/db2inst1/TESTDB \logarchmeth1 DISK:/db2/archive_logs

DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.



$ db2 create database testdb \ on /db2/data01, /db2/data02 \ dbpath on /db2DB20000I The CREATE DATABASE command completed successfully.

$ db2 update db cfg for testdb using \ newlogpath /db2/active_logs/db2inst1/TESTDB \ logarchmeth1 DISK:/db2/archive_logsDB20000I The UPDATE DATABASE CONFIGURATION command completed successfully.

$ db2 backup database testdb to /db2/backups/db2inst1

Backup successful. The timestamp for this backup image is : 20200625002633


Filesystem Separation

● Store active transaction logs on separate physical disk(s)

● DW / BLU Workloads may need temporary tablespaces physically separated from primary storage.

● Create separate filesystems:

● /db2/tempNN/db2inst1/NODE0000/TESTDB/

● Create new STOGROUP

● CREATE STOGROUP sg_temp ON /db2/temp01, /db2/temp02

● Move temporary tablespaces

● ALTER TABLESPACE TEMPSPACE1 USING STOGROUP sg_temp

● ALTER TABLESPACE USERTEMP4K USING STOGROUP sg_temp


Key Performance Metrics


Average I/O Read Time

● Represents the average time it takes for Db2 to read a page from disk

● Calculate using these metrics:

● POOL_DATA_P_READS

● POOL_XDA_P_READS

● POOL_COL_P_READS

● POOL_TEMP_INDEX_P_READS

● POOL_TEMP_XDA_P_READS

● POOL_TEMP_COL_P_READS

● POOL_READ_TIME

● Available at database level, tablespace level and from many other MON_GET functions

avg read ms = POOL_READ_TIME∑ POOL_ * _P_READS


SELECTmember,pool_read_time * 1.0 /

NULLIF(pool_data_p_reads + pool_xda_p_reads + pool_col_p_reads +pool_temp_index_p_reads + pool_temp_xda_p_reads +pool_temp_col_p_reads, 0) AS avg_read_ms

FROM TABLE (mon_get_database(-2));



SELECT member, pool_read_time * 1.0 / NULLIF(pool_data_p_reads + pool_xda_p_reads + pool_col_p_reads + pool_temp_index_p_reads + pool_temp_xda_p_reads + pool_temp_col_p_reads, 0) AS avg_read_ms FROM TABLE (mon_get_database(-2));

MEMBER AVG_READ_MS ------ ----------------------------------- 0 0.94642857142 1 record(s) selected.



Average I/O Write Time

● Represents the average time it takes for Db2 to write a page to disk

● Calculate using these metrics:

● POOL_DATA_WRITES

● POOL_INDEX_WRITES

● POOL_XDA_WRITES

● POOL_COL_WRITES

● POOL_WRITE_TIME

● Available at database level, tablespace level and from many other MON_GET functions

avg write ms = POOL_WRITE_TIME∑ POOL_ * _WRITES


SELECTtbsp_name,pool_write_time * 1.0 /

NULLIF(pool_data_writes + pool_index_writes +pool_xda_writes + pool_col_writes,0) AS avg_write_ms

FROMTABLE (mon_get_tablespace(NULL, -2));



SELECT tbsp_name, pool_write_time * 1.0 / NULLIF(pool_data_writes + pool_index_writes + pool_xda_writes + pool_col_writes,0) AS avg_write_ms FROM TABLE (mon_get_tablespace(NULL, -2));

TBSP_NAME AVG_WRITE_MS -------------------- --------------------------------- SYSCATSPACE 0.15384615384 TEMPSPACE1 0.19390190480 USERSPACE1 0.27419473202 SYSTOOLSPACE - 1 record(s) selected.



Average Log I/O Write Time

● Represents the average time for DB2 to write a log page

● Calculate using these monitor elements:

● LOG_WRITES

● LOG_WRITE_TIME

● Monitor elements available in MON_GET_TRANSACTION_LOG

avg log write ms = LOG_WRITE_TIMELOG_WRITES


SELECTmember,log_write_time * 1.0 / NULLIF(log_writes,0) AS log_write_ms

FROMTABLE (mon_get_transaction_log(-2));



SELECT member, log_write_time * 1.0 / NULLIF(log_writes,0) AS log_write_ms FROM TABLE (mon_get_transaction_log(-2));

MEMBER LOG_WRITE_MS ------ --------------------------------- 0 0.35634328358 1 record(s) selected.



dmctop – Overview


dmctop – Logging


Other Considerations for I/O Metrics

● Direct I/O Read and Write Times

● Direct Reads and Writes do not go through the bufferpool

● Includes Temp Tablespaces, LOBs, Database Backups

● Synchronous vs Asynchronous I/O

● Synchronous I/O times are more impactful to workload

● Understand workload characteristics


Finding I/O Bottlenecks


I/O Wait

● I/O Wait time is a type of idle time – occurs when a CPU is idle and there are outstanding I/O requests

● Reported as wa in vmstat output

● Generally good at under 20-30%

$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- --system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 1 0 2574952 142484 837652 0 0 315 199 0 0 4 1 94 1 0 1 1 0 2574952 142492 837652 0 0 2066 7913 22238 4726 2 1 96 2 0 0 2 0 2754268 142500 837652 0 0 1419 7320 30958 5943 3 2 93 1 0 0 1 0 2754528 142508 837652 0 0 1340 8239 23108 5288 2 1 95 2 0 2 1 0 2574292 142516 837652 0 0 1334 8061 24161 5477 3 1 94 2 0


Device-Level Statistics - AIX

● iostat – get statistics for individual disk devices

●% tm_act - percentage of time device was active

●Kbps - transfer rate of data to/from disk

$ iostat -d hdisk46 5 5

Disks: % tm_act Kbps tps Kb_read Kb_wrtn hdisk46 0.0 1809.6 24.0 9048 0



AIX


Device-Level Statistics - Linux

● iostat – get statistics for individual disk devices

● tps - transfers per second (IOPS)

●kB_read/s - transfer rate of data from disk

$ iostat -d -N b2cnaprvg-table04 5 3

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn db2vg-table04 9.00 56.00 0.00 280 0



Linux


nmon - System Performance

● nmon provides a terminal-based interface to system statistics

●Display CPU statistics and I/O statistics on a single screen

●Use disk groups for viewing I/O statistics

$ cat ~/dgvglocal00 dm-42 dm-39 dm-37 dm-41 dm-40 dm-38 dm-33 dm-32 dm-34 dm-36 dm-35db2-base dm-11 dm-12 dm-14db2-tbsp dm-15 dm-16 dm-17 dm-18 dm-19 dm-20db2-logarc dm-21db2-log dm-13backupvg dm-10


nmon - System Performance

● nmon provides a terminal-based interface to system statistics

●Display CPU statistics and I/O statistics on a single screen

●Use disk groups for viewing I/O statistics

$ cat ~/dgvglocal00 dm-42 dm-39 dm-37 dm-41 dm-40 dm-38 dm-33 dm-32 dm-34 dm-36 dm-35 db2-base dm-11 dm-12 dm-14 db2-tbsp dm-15 dm-16 dm-17 dm-18 dm-19 dm-20 db2-logarc dm-21 db2-log dm-13 backupvg dm-10

$ NMON=lg nmon -g ~/dg


nmon


dmctop – Time Spent

Speaker: Ian Bjorhovde

Email Address: [email protected] Phone: 719-309-1127

mailto:[email protected]

mailto:[email protected]

wdug storage basics

Documents