db2 autonomics - autopilot for the dba

May 21, 2008 • 4:00 p.m. – 5:00 p.m.Platform: DB2 for Linux, UNIX, Windows

Jayesh ThakrarIntersect Solutions, LLC

Session: D12

DB2 Autonomics - Autopilot for the DBA

2

Jayesh ThakrarIntersect Solutions, LLC

jthakrar@intersectsolutions.com

Session D12

DB2 Autonomics -Autopilot for the DBA

3

DB2 Autonomics –Autopilot for the DBA

• Describe DB2 LUW Autonomic features

• Show how to implement Autonomic features using GUI as well as DB2 commands / SQL

• Present some of the best practices to implement DB2 Autonomics for recoverability, availability and performance

Make deploying, managing and supportingDB2 LUW a pleasant & scalable experience

Session Objectives –

4

DB2 Autonomics –Autopilot for the DBA

• Introduction

• Discuss DB2 Autonomics FeaturesStorage Management

DB2 Configuration Management

Routine Maintenance

Health Monitoring and Notification

Workload Management

• Discuss Implementation Best Practices

• Questions & Answers

Agenda

5

Autonomics – What and Why?

6

What is Autonomics ?Autonomic Computing is an initiative started by IBM in 2001. Its ultimate aim is to create computer systems capable of self-management, to overcome the rapidly growing complexity of computing systems management, and to reduce the barrier that that complexity poses to further growth.From : Wikipediahttp://en.wikipedia.org/wiki/Autonomic_Computing

Autonomic computing architecture is a range of software technologies that enable you to build an information infrastructure that can, to lesser and greater degrees, manage itself, saving countless hours (and dollars) in human management. And all this without giving up control of the system. From : Hello Autonomic Computing!http://www.ibm.com/developerworks/edu/ac-dw-ac-intro-i.html

In a nutshell, administrators spend too much time doing repetitive tasks that the system should be able to do for itself.From : An autonomic Computing Roadmaphttp://www.ibm.com/developerworks/autonomic/library/ac-roadmap/

7

Why Autonomics and Why Now?

• High database to DBA ratio• Large number of databases on a server• Large, rapidly growing databases• Automation - reduce human intervention• Progressing maturity• Competitive advantage

Bottom line – reduce TCO !

8

Autonomics in DB2 LUW

Storage Management

Configuration Management

Health Monitoring & Actions

Workload Management

Routine Maintenance

9

Autonomics in DB2 LUW

Storage Management

10

Traditional Storage Management

Size database and storage needs upfrontSelectively place tablespace containersMonitor tablespace free space Monitor storage free space

11

Automatic Storage Management

Size database and storage needsSpecify storage locations and create databaseCreate tablespacesSit back and relax !

Tablespaces grow as needed and alert you when filesystem or directory out of space !

12

Automatic Storage Management

• CREATE DATABASE MYDB• CREATE DATABASE MYDB AUTOMATIC STORAGE YES• CREATE DATABASE MYDB AUTOMATIC STORAGE YESON /db2/data01, /db2/data02,

/db2/data03, /db2/data04DBPATH ON /db2/mydb

Examples – Database Creation

13

Automatic Storage Management

<storage_path>/<instance_name>/<node_no>/<tablespace_no>/<container_no>.<ext>

Where

Automatic Storage Container Paths

<storage_path> = Implicitly or explicitly defined storage path

<instance_name> = Name of instance under which database is created

<node_no> = The database partition number

<tablespace_no> = Tablespace id (as in SYSCAT.TABLESPACES)

<container_no> = Container id

<ext> = CAT (catalog) / TMP (temporary) / UTM (user temporary) / USR (user or regular) LRG (large)

14

Automatic Storage Management

• CREATE TABLESPACE Staging_Data• CREATE TABLESPACE Staging_Data

MANAGED BY AUTOMATIC STORAGE• CREATE TABLESPACE Staging_Data

PAGESIZE 32 K MANAGED BY AUTOMATIC STORAGE

• CREATE TEMPORARY TABLESPACE Temp_32K PAGESIZE 32 K

REGULAR / LARGE TABLESPACE = DMS TYPE WITHAUTOMATIC STORAGE

TEMPORARY TABLESPACE = SMS TYPE

Examples – Table Space Creation

15

Automatic Storage Management

• Additional storage paths can be added after database creation• Automatic storage can only be enabled at database creation• Databases with automatic storage allow SMS & DMS tablespaces• Automatic storage also works with partitioned databases (v9.0+)• Database partition expressions can be used e.g.

CREATE DATABASE MYDBON "/data01_ $N", "/data02_ $N"

"/data03_ $N", "/data04_ $N"

• On database restore, storage paths can be preserved or redefined• DB2 v9.5 – can resize automatic tablespace to reduce size• DB2 v9.5 – automatic compression for large tables

Additional Info

16

Autonomics in DB2 LUW

Configuration Management

17

Configuration Management

sqllib/SQLnnnnn/SQLDBCON

sqllib/db2systm

Configuration Parameters – where do they go

18

Configuration ManagementTraditionally - Exasperating and Overwhelming Task

DB2 Registry Level Parameters = 100+

Instance Level Parameters = 50+

Database Level Parameters = 50+

Memory Capacity Management

Agent Management

I/O, Storage and Logging Management

Locking Management

Query Processing & Compiler Management

Communication Management

Security Management & Misc.

Effective at next statement / transaction

Effective at next application connect

Effective on database re-activation

Effective on instance restart

DBA

19

Configuration Management

• Dynamic configuration parameters• Better “default values”

RESET DATABASE MANAGER CONFIGURATIONRESET DATABASE CONFIGURATION FOR MYDB

• STMM - Self-Tuning Memory ManagerUPDATE DBM CFG USING <parameter> AUTOMATICUPDATE DB CFG FOR MYDB USING <parameter> AUTOMATIC

• “Autoconfigure” all dynamic parameters using db2 clp command or Wizard

• GUI Wizard-based interface to autoconfigure & STMM

Autonomic Features – Sit Back and Relax !

20

Configuration ManagementDefault Database Manager Parameters

21

Configuration ManagementAutoconfigure• Autoconfgure at database creation

(via command-line or using database creation wizard)CREATE DATABASE MYDB AUTOCONFIGURE

APPLY DB AND DBM

• Autoconfigure anytime – command-line or Configuration Advisor

22

Configuration ManagementAutoconfigure Keywords

Keyword Valid Values Explanationmem_percent 1–100 Percentage of memory to dedicate. If other applications (other than the

operating system) are running on this server, set this to less than 100

workload_type simple, mixed, complex

Simple workloads tend to be I/O intensive and mostly transactions, whereas complex workloads tend to be CPU intensive and mostly queries.

num_stmts 1–1 000 000 Number of statements per unit of work

tpm 1–200 000 Transactions per minute

admin_priority formance, recovery, both

Optimize for better performance (more transactions per minute) or better recovery time

is_populated yes, no Is the database populated with data?

num_local_apps 0–5 000 Number of connected local applications

num_remote_apps 0–5 000 Number of connected remote applications

isolation RR, RS, CS, UR It is only used to determine values of other configuration parameters. Nothing is set to restrict the applications to a particular isolation level and it is safe to use the default value.

bp_resizeable yes, no Are buffer pools resizeable?

23

Configuration ManagementSTMM - Self-Tuning Memory Manager• STMM = DB2 feature to dynamically manage database

memory among 2 or more of the following “consumers” –BUFFERPOOLS PCKCACHESZ (Package cache)LOCKLIST (Lock list – total memory for locks within the database)MAXLOCKS (max. % of locks per application)SORTHEAP (sort list heap – max. memory allocated per sort)SHEAPTHRES_SHR (sort heap threshold for shared sorts)STMTHEAP (SQL Statement heap)APPLHEAPSZ (Application heap size)APPL_MEMORY (Application memory size)STAT_HEAP_SZ (Statistics heap size – used during runstats)

• STMM – takes over configuring / tuning of database parameters(adjustments done dynamically)

24

Configuration ManagementSTMM – How to Enable And Other Information• Set SELF_TUNING_MEM database parameter to ON• Set 2 or more database parameters (consumers) to AUTOMATIC• Set DATABASE_MEMORY parameter to

AUTOMATIC (Windows and AIX only)COMPUTED (total memory requirements computed at database activation time, with some room for growth/overhead)Explicit value (all consumers “bound” by this upper limit)

• STMM adopts to all of the following –Changing database workload (e.g. OLTP during business hours and batch processing at night)Multiple DB2 instances and database on the database serverNon-DB2 and OS memory needs

• Dynamic adjustments at 30 seconds to 10 minute intervals• Note – STMM works only for “active databases”

25

Configuration ManagementSTMM – How to Monitor STMM Behavior• Detailed STMM activity log files in stmmlog directory

(located in instance/sqllib)Multiple stmm.<NNN>.log files (each about 10 MB)Contains detailed record of “Cost Benefit Analysis” for each consumer

• To examine current values of database memory consumers –Use database snapshot commandget snapshot for database on MYDB

Use db2pd command(many options – see DB2 Command Reference Manual)

• Note - following DB2 registry entries affect STMM’s operations –DB2_LARGE_PAGE_MEMDB2MEMDISCLAIMDB2MEMMAXFREEDB2_MEM_TUNING_RANGEDB2_PINNED_BP

26

Configuration ManagementSTMM – Partitioned Databases• STMM can work in partitioned databases also• STMM started only if database activated explicitly• All STMM monitoring and adjustments based on a single partition• STMM changes propagated to all partitions in database• Individual partitions can be disabled for STMM• Database parameters can be set manually at a partition• Check current database partition used for STMM using –

CALL SYSPROC.ADMIN_CMD( ’get stmm tuning dbpartitionnum’ )

• Change current STMM database partition using –CALL SYSPROC.ADMIN_CMD( ’update stmm tuning dbpartitionnum <db_partition_num>’ )

• STMM advisable if servers with similar configuration & workload• Disable STMM on partitions with dissimilar configuration/workload

(e.g. catalog partition or dedicated co-ordinator partition)

27

Configuration ManagementOther GUI Wizards

28

Autonomics in DB2 LUW

Routine Maintenance

29

Database MaintenanceFeature Overview• Runs DB2 utilities only on “as needed basis” –

BACKUPREORGSRUNSTATSSTATEMENT STATISTICSSTATISTICS PROFILING

• Alternatively can notify to run the following utilities –BACKUPREORGSRUNSTATS

• Enable / disable via database configuration parameters• Can define “online” and “offline” maintenance windows• GUI wizard for database maintenance – more features

Maintenance windows only through WizardSpecify automatic maintenance for subset of tables and indexes Specify backup destinationEnable offline / online maintenance individuallySpecify recipients of notification messages (available as CLP for health monitoring)

30

Database MaintenanceDatabase Configuration Parameters

31

Database MaintenanceGUI Wizard – Introduction

32

Database MaintenanceGUI Wizard – Enable / Disable Maintenance

33

Database MaintenanceGUI Wizard – Define Online / Offline Windows

34

Database MaintenanceGUI Wizard – Define Notification List

35

Database MaintenanceGUI Wizard – Configure Type of Maintenance

36

Database MaintenanceGUI Wizard – Define subset of tables for reorg

37

Database MaintenanceGUI Wizard – Reorg Options

38

Database MaintenanceGUI Wizard – Maintenance Specs Review

39

Database MaintenanceAdditional Information• Automatic Backup

Backup to disk, tape, TSM or vendor DLL / plug-inOnline (if archive logging enabled) or offline backupsFor partitioned databases, backup happens at each partitionDisk-based backup can delete previous/obsolete backupsNo compressionIncremental backups only in DB2 LUW V9.5

• ReorgsOffline reorgs onlyFor partitioned databases, reorg evaluation done on catalog parttion onlyReorg Options include rebuild or retain compression dictionary

• Runstats, Statement Statistics & Statistics ProfilingOnline operationFor partitioned databases, evaluation done at catalog partition onlyEnable runstats only or all of themProfiling – updates individual table runstats profilesStatement statistics – collects SQL details for input to profiling and runstats

40

Autonomics in DB2 LUW

Health Monitoring & Actions

41

Health MonitoringFeature Overview• Monitors “health” of instances and “active” databases• Instance level parameter to enable/disable health monitoring• Pre-defined 29 Key Health Indicators (KHI) monitored continuously

Instance level and database level key health indicators Tablespace level and tablespace container level key health indicators

• User-configurable thresholds for each KHI (where value-based)• Different KHI inspected at different, pre-defined intervals • On threshold – notification and optional user-defined action• DB2 DAS used for notification to email or pager• KHIs can be enabled/disabled individually• Command-line / GUI interface to enable, disable and configure KHIs• Command-line / SQL functions to retrieve monitoring data• “Action” for alerts can be SQL or OS scripts/command• Default and user-defined parameters passed to action scripts

42

Health MonitoringKey Health Indicator Listingdb2.db2_op_status db.lock_escal_rate

db2.sort_privmem_util db.apps_waiting_locks

db2.mon_heap_util db.catcache_hitratio

db.db_op_status db.pkgcache_hitratio

db.db_backup_req db.shrworkspace_hitratio

db.hadr_op_status db.db_heap_util

db.hadr_delay db.auto_storage_util

db.log_util db.sort_shrmem_util

db.log_fs_util ts.ts_auto_resize_status

db.deadlock_rate ts.ts_util_auto_resize

db.locklist_util ts.ts_util

db.spilled_sorts ts.ts_op_status

db.tb_reorg_req tsc.tscont_util

db.tb_runstats_req tsc.tscont_op_status

db.max_sort_shrmem_util

43

Health MonitoringDB2 CLP Commands for Health Monitor• GET / UPDATE ALERT CONFIGURATION• GET / UPDATE CONTACTGROUPS• GET / UPDATE CONTACTS• GET / UPDATE HEALTH NOTIFICATION CONTACT LIST• GET HEALTH SNAPSHOT• GET RECOMMENDATIONS FOR HEALTH INDICATOR

44

Health MonitoringUPDATE ALERT CONFIGURATION - Syntax

Any of the KHI listed earlier

Parameter name can be ALARM, WARNING, SENSITIVITY ACTIONSENABLED, THRESHOLDSCHECKED

See next slide for Script Details

45

Health MonitoringUPDATE ALERT CONFIGURATION - Syntax

46

Health MonitoringUPDATE ALERT CONFIGURATION - Examples• Disable the checking of an individual KHI

UPDATE ALERT CONFIGURATION FOR DATABASE MANAGER USING db2.mon_heap_util SET THRESHOLDSCHECKED NO

• Get & update the threshold for automatic storage path(s) free spaceGET ALERT CONFIGURATION FOR DATABASE ON MYDBUSING db.auto_storage_util UPDATE ALERT CONFIGURATION FOR DATABASE ON MYDBUSING db.auto_storage_util SET WARNING 88, ALARM 95

• Get & update action for tablespace stateGET ALERT CONFIGURATION FOR TABLESPACES ON MYDBUSING ts.ts_op_statusUPDATE ALERT CONFIGURATION FOR TABLESPACES ON MYDBUSING ts.ts_op_status ADD ACTION SCRIPT <full_path_for_script> TYPE OSWORKING DIRECTORY <full_path_for_directory> ON WARNING USER <username> USING <password>

• Delete action script for database stateGET ALERT CONFIGURATION FOR DATABASE ON MYDBUSING db.log_util DELETE ACTION SCRIPT <full_path_for_script> ONWARNING

47

Health MonitoringUPDATE CONTACTS / CONTACTGROUPS - Syntax

Keyword can be ADDRESS (email address), TYPE (email or pager) MAXPAGELEN (max. length for pager message) or DESCRIPTION

48

Health MonitoringGET HEALTH SNAPSHOT - Syntax

Show all history data (usually upto 9 previous data collections). Also helpful in determining “monitoring period”

Shows all objects being examined (e.g. table statistics)

49

Health MonitoringGET RECOMMENDATIONS FOR HEALTH INDICATOR - Syntax

50

Health MonitoringAdditional Information• Disable certain KHI if STTM enabled and relevant

parameters set to AUTOMATICdb.sort_shrmem_utildb.spilled_sortsdb.max_sort_shrmem_utildb.locklist_utildb.lock_escal_ratedb.pkgcache_hitratio

• For operating system (OS) action scripts, following information passed as parameters –

List of user-supplied parameters (if any)Health indicator short nameFully qualified object nameHealth Indicator valueAlert state

51

Autonomics in DB2 LUW

Workload Management (Version 9.5 onwards)

52

Workload ManagementFeature Overview (v9.5 onwards)• Allows classifying and grouping user requests,

database activity and DB2 resources and thresholds / limits • Allows granular management of system load,

system resources and concurrency for better SLA • Allows better response to load spikes and runaway queries• Allows batch processing and OLTP to co-exist• SQL based access to historical and real-time activity

and workload• Activity monitors on service class, workload, etc

(similar to event monitors)• Ability to profile and capture workload

53

Workload ManagementNew Concepts• Service Class

User-definable logical execution environment that has allocated resources(CPU priority and prefetch I/O priority) and thresholdsAll DB2 activity (work) carried out within a service class2-tier hierarchy – service class and subclassDefault service classes present in a new database

• WorkloadWorkload = all “database activity” from user sessions, database maintenance activities, etc.Includes QUERY (SELECT or XQuery) , DDL, DML, LOAD, CALLWorkload can be grouped on various database connection attributesWorkload assigned to specific service class (i.e. resources and thresholds)

• Work class & Work Class SetWork class = individual set of database activity – e.g DDL, DMLWork class set = collection of one or more work classes

54

Workload Management

DBA

SELECT ….

FROM Account_Header

WHERE …….

INSERT INTO Contact(…..)

VALUES (…..)

LOAD FROM Product_Data.CSV OF DEL INSERT INTO …..

Finance Manager

Sales Manager

Different users connecting to database and generating database requests & activities

DB2 LUW

55

Workload Management

DBA

ERP_APP_WORKLOAD

CRM_APP_WORKLOAD

SELECT ….

FROM Account_Header

WHERE …….

INSERT INTO Contact(…..)

VALUES (…..)

LOAD FROM Product_Data.CSV OF DEL INSERT INTO …..

Finance Manager

Sales Manager

DBA_WORKLOAD

Based on (certain) connection and user attributes, the work request is considered part of a specific workload

DB2 LUW

56

Workload Management

DBA

ERP_APP_WORKLOAD

CRM_APP_WORKLOAD

SELECT ….

FROM Account_Header

WHERE …….

INSERT INTO Contact(…..)

VALUES (…..)

LOAD FROM Product_Data.CSV OF DEL INSERT INTO …..

Finance Manager

Sales Manager

DBA_WORKLOAD

ERP_SERVICE_CLASS

CRM_SERVICE_CLASS

UTIL_SERVICE_CLASS

Each workload is assigned to a DB2 Service Class –which has associated DB2 resources and thresholds

DB2 LUW

57

Workload ManagementSQL - CLP, Views, Functions and Procedures - 1• CLP Commands

CREATE / ALTER / DROP WORKLOADCREATE / ALTER / DROP SERVICE CLASSCREATE / ALTER / DROP WORK CLASS SETCREATE / ALTER / DROP THRESHOLDCREATE / ALTER / DROP HISTOGRAM TEMPLATEGRANT USAGE ON WORKLOAD …. TO …..SET WORKLOAD TO ….

• ViewsWORKACTIONSWORKACTIONSETSWORKCLASSESWORKCLASSSETSWORKLOADAUTHWORKLOADCONNATTRWORKLOADS

58

Workload ManagementSQL - CLP, Views, Functions and Procedures - 2• Table Functions

WLM_GET_ACTIVITY_DETAILSWLM_GET_QUEUE_STATSWLM_GET_SERVICE_CLASS_AGENTSWLM_GET_SERVICE_CLASS_WORKLOAD_OCCURRENCESWLM_GET_SERVICE_SUBCLASS_STATSWLM_GET_SERVICE_SUPERCLASS_STATSWLM_GET_WORKLOAD_OCCURRENCE_ACTIVITIESWLM_GET_WORKLOAD_STATSWLM_GET_WORK_ACTION_SET_STATS

• ProceduresWLM_CANCEL_ACTIVITYWLM_CAPTURE_ACTIVITY_IN_PROGRESSWLM_COLLECT_STATS

59

DB2 Autonomics Best Practices

60

DB2 Autonomics – Best PracticesDatabase Setup and Configuration• Avoid micro-managing tablespace containers –

instead, create pools of storage for different content typesStorage pool for data (tables and indexes)Use multiple storage paths to enable I/O parallelism (even with single data storage pool)Storage pool for database directory (optional – can be part of storage pool for data)Storage pool for logs (optional – can be part of storage pool for data)Storage pool for archived logsStorage pool for backup

• Avoid manual configuration - use autoconfigure and STMM• Use / configure DAS – DB2 Administration Server• Setup contacts and contact groups• Enable health monitoring • Disable specific health indicators (KHI) if using STMM• Minor annoyances – tablespace state during backup and

monitor heap size warning at first database activation / startup

61

DB2 Autonomics – Best PracticesDatabase Monitoring and Maintenance• Enable following automatic maintenance parameters

Automatic runstatsAutomatic statement statisticsAutomatic statistics profiling (only non-partitioned & non intra-parallel enabled systems)Automatic profile updates (only non-partitioned & non intra-parallel enabled systems)For profiling and statement statistics create query feedback warehouse –call SYSINSTALLOBJECTS ('ASP', 'C', '', '')ASP = Automatic Stats Profiling

• Collect system state, configuration and workload baseline dataCreate separate schema / database for the dataCollect data on a periodic basis (e.g. once a day or every 4 hours)Use snapshot functions to collect “state” informationUse DB2 catalog views to collect object information (e.g. table and index sizes)Use WLM (Workload Manager) views to collect workload dataIn health monitor action scripts collect configuration and activity snapshots when triggeredIn health monitor action scripts collect recommendations when triggered

• Consider integrating DB2 health monitor with enterprise monitoring

62

DB2 Autonomics – ReferencesInformation Sources• DB2 Online and PDF Documentation

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsphttp://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsphttp://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp

• IBM Developerworks Articleshttp://www-128.ibm.com/developerworks/db2/zones/dba/

• IBM Chat-with-the-Labs Presentationshttp://www-306.ibm.com/software/data/db2/9/labchats.html

• IBM Redbookshttp://www.redbooks.ibm.com/

• IBM Research Publicationshttp://www.research.ibm.com/

• DB2 Newsgroup (available via many providers – e.g.)groups.google.com

63

Questions?

Session D12

DB2 Autonomics -Autopilot for the DBA

64

Jayesh ThakrarIntersect Solutions, LLC

jthakrar@intersectsolutions.com

Session D12

DB2 Autonomics -Autopilot for the DBA

Contact Information