© 2013 ibm corporation information management high availability improvements

© 2013 IBM Corporation

Information Management

High Availability Improvements



Agenda

Recover and Online Schema

Pending alter supports in Sequoia*Drop Column

*Online Alter Limit Key

Defer Define enhancements

Work file enhancements



Recover and Online

Schema Changes

PendingChange PBGUTS PBRUTS XMLTS LOBTS

AlterSegmentSize X X X

AlterDataSetSize X X X X

AlterBPT/SPageSize X X X

AlterBPIndexPage X X

AlterMemberCluster X X

ConvertClassic1-TablePart.T/StoUTSPBR

X X

ConvertClassicSimpleT/S

X

ConvertClassicSegmentedT/S

IBMConfidential

X ©2013IBMCorporation


DB2 10 Online Schema Enhancements



DB2 10 PIT RECOVERy with Pending Alter

5

time

PendingALTER issued

0

1

Change deferred

2 REORGMaterialize changes

3RECOVER to

PIT started

4Earliest PIT

RECOVER point(“Brick Wall”)

© 2013 IBM CorporationIBM Confidential


PIT Recovery with Pending alterDB2 11 PITR with Pending ALTER

* Supports in NFM only* A new record is inserted into SYSPENDINGDDL table after RECOVER

Table spaces supported:* LOB* XML* Partition by Range (PBR) Universal Table Space

At end of RECOVER to PIT prior to the materializing REORG:* Table space is in restrictive status: REORG-pending* REORG to finalise the PIT recovery process is MANDATORY* No other PITR during this period prior to subsequence REORG

Schema definition is not recoverable*Only data is recovered to PIT

© 2013 IBM Corporation 7

time

ALTER issued

0

1

Change deferred


Enhanced PIT RECOVERy with Pending Alter

2 REORGMaterialize changes

Pending

3RECOVER to

PIT start

4

Earliest PITRECOVERy point


time

PendingALTER issued

materialises changes

0

1

Change deferred


PIT RECOVERy with immediate ALTERs

2 REORG

3

PiT start

5

Earliest PiTRECOVERy point

Immediate ALTERs

4 RECOVER to



PITR with non-materialized pending ALTER

Pending alter can be dropped any time before REORG

Pending change is not required to drop when RECOVER starts

PITR would be business as usual

time

REORGmaterializing changes

0

1Pending

ALTER issued

Change deferred

3RECOVER to

PIT start2



After PIT RECOVERy

The subsequent REORG is required

* Must on be on ENTIRE table space

* SHELEVEL NONE is not supported

* SHRLEVEL CHANGE is overridden by SHRLEVEL REFERENCE

Before the subsequent REORG to materialize the RECOVER* No CREATE/ALTER/RENAME/DROP TABLE on the TS or AUX objects

* All other utility jobs fail: DSNU933I (REORG required)

Only REORG/REPAIR DBD/REPORT RECOVERY or RECOVER to the

same PIT are permitted

Where there were pending changes on LOB table space:

* First REORG the LOB table space* Then REORG the base table space

2 REORGmaterializing changes


PIT Recovery after materialized Pending alter

When recover pending alter to PITR after materializing REORG :

*Recovering an index is not supported

*Entire table space must be recovered

*There must be NO other Pending changes when RECOVER starts

*VERIFYSET NO option will always be overridden to

VERIFYSET YES

time


0

1Pending

ALTER issued

Change deferredRECOVER to

PIT start3

2 REORGmaterializing changes


PITR and Pending ALTER restriction TOCOPY/TOLASTCOPY/TOLASTFULLCOPY options can be used for entire table space on a single

object* RECOVER cannot use an image copy with SHRLEVEL CHANGE taken between the change and it being materialised by REORG

* Use TORBA or TOLOGPOINT to reach point after the image copy completed

No clone table may exist at the time RECOVER is issued

RECOVERy to a different PIT cannot commence until the first recovery AND its REORG complete

Recovery to PITR prior to materialize of TS type conversion pending changes is NOT supported

time


0

1Pending

ALTER issued

Change deferredRECOVER to

PIT start3

PendingChanges LOBTableSpace

PartitionbyRangeUTS

XMLTableSpace

AlterSegmentSize AlterDataSetSize AlterBPT/SPage

Size AlterBPIndexPage

AlterMemberCluster

AlterLimitkey© 2013 IBM Corporation


DB2 11 PIT Recovery Prior to Materializing REORG:Supported Changes

sc1.table1inDB1.TS1

Col_1 Col_2 Col_3

Data_1 Data_2 Data_3




Drop Column



ALTER TABLE xxx COLUMN

ALTER TABLE

DB2 10

ADD COLUMN

ALTER COLUMN

RENAME COLUMN

DB2 11

DROP COLUMN

sc1.table1inDB1.TS1

Col_1 Col_2 Col_3 Col_4

Data_1 Data_2 Data_3 Data_4



Columns become obsolete as applications change

Could the column be left?– An abandoned column costs

•

•

•

•

Real space in every row stored in the table

Real space in every Image Copy of the tablespace

Potential space taken up in the log records written for the table

Additional CPU and elapsed time in all aspects of accessing and maintaining thedata

• DBA time “remembering” that the column is redundant/obsolete

• Developer time analyzing usage of the column


Why do you need to DROP COLUMN?

sc1.table1inDB1.TS1




sc1.table1inDB1.TS1

Col_1 Col_2 Col_4


Data_1 Data_2 Data_4©2013IBM Corporation


How do you DROP a COLUMN? Before DB2 11

– You have to DROP and recreate the Table to remove Col_3

ApplicationOutage /

ScheduledMaintenance

ApplicationTable




ApplicationTable

Col_1 Col_2 Col_4


Data_1 Data_2 Data_4X18© 2013 IBM Corporation


Improvements in DB2 11 DB2 11

ALTER TABLE sc1.table1 DROP COLUMN Col_3 RESTRICT

REORG TABLESPACE DB1.TS1 SHRLEVEL REFERENCE

Application Outage /Scheduled

Maintenance

IBM Confidential


SYSPENDINGDDL

Table

CatalogField

Value


DROP COLUMN Process

DBA issuesAlter table

DROP COLUMN

Validation and authorizationchecks are done as the

ALTER is issued.

CatalogTable

Sch ly newem


DBA issuesAlter


Drop column process

CatalogTable

SYSPENDINGDDL

App

a

Applications

schema

REORGTable SpaceWith

Definedschema

Continue to accessdata in the old

Table SpaceWith

New definedschema

ALTER TABLEDROP COLUMN …

At ALTER time : authorization checks

SYSPENDINGDDL entries

TablespacesetinAREOR

REORG

materialized the new definition

Unload

Data

InlineImagecopy


Removing a Pending DROP COLUMN

If a pending ALTER needs to be cancelled before the

materializing REORG...ALTER TABLE DROP

COLUMNALTER TABLESPACE

DROP PENDINGCHANGES

Time

NOTE: This removes ALL pending changes associated

with the table space or any objects within the table space



A column cannot be dropped if* The containing table space is NOT a Universal Table Space

* The table is a created global temporary table, a system-period temporal table,

a history table, MQT or table referenced by MQT plus others....* There are row permissions or column masks dependent on the table* There are triggers defined on the table* The column is an XML column, is part of the table partitioning key, RI, plus

others...* Hash Key column

* DOCID / ROWID column

* Tables with EDITPROC or VALIDPROC

No immediate alters (e.g. Can‘t change data type, add column,add/rotate parts) combines with pending alter


Things to think about...



New SQL CodesSQLCODE -195

LAST COLUMN OF table-name CANNOT BE DROPPED

ExplanationAn attempt was made to drop a column using an ALTER TABLE statement. The columncannot be dropped from table table-name because at least one of the existing columns mustbe preserved when altering a table.

SQLCODE -196

COLUMN table-name.column-name CANNOT BE DROPPED. REASON = reason-code.

ExplanationThe column cannot be dropped for one of the following reasons:

** See Information Centre or Manuals for the full list

© 2013 IBM


RECOVER* RECOVER to a PIT prior to a materializing REORG is not allowed

• DSNU556I and RC8

* An inline copy will be taken by the REORG

• This single image copy is the only base for recovery

UNLOAD* will not be able to unload from an Image Copy which contains data for

dropped columns

• Dropped columns is no longer exist in the catalog after REORG

materialzied data

• DSNU1227I and RC8

* Consider UNLOAD with external formatting

• If data corruption occurs, can be used with LOAD utility if necessary Corporation 24

Impact on Other Utilities



Impact on Other Utilities - continued

RUNSTATs*Should be run after materializing REORG

• Ensures statistics are current for BIND/REBIND or PREPARE

REORG*Only table space level REORG materializes the changes

Partition Logical_Part Limitkey

1 1 200

2 2 300

3 3 400

4 4 500



Alter Partition Limit Key

New key limit

350

ALTER TABLE … ALTER PARTITION integer ENDING AT …

Part-ition

Logical_Part

Limitkey_Internal

2 1 200

3 2 300350

4 3 400

1 4 500


Partition 1DSNUM 2

DSNUM 3

DSNUM 4

DSNUM 1

logial viewphysical view

Partition 2

Partition 3

Partition 4

Alter Table ... Alter Partition 3 Ending At 350 Inclusive

REORP

REORP

DSNUM 2

DSNUM 3

DSNUM 4

DSNUM 1

50, 125, 150, 175, 200

225, 250, 275, 300

325, 350, 375, 400

425, 450, 475 500

50, 125, 150, 175, 200

225, 250, 275, 300,325, 350

375, 400

425, 450, 475500


keysstatusDB2 Catalog

Partition 1

Partition 2

Partition 3

Partition 4

Any Reorg including physical part 3 and 4 must run to remove the REORP status

SYSIBM.SYSTABLEPART

Prior to V11 behavior

* Affected partitions are set to REORP.

* These partitions could not be accessed.

* Any REORG could be run to

redistribute the data and remove the

status.



Alter Limit Key in DB2 11

Alter limit key is treated as a pending alter in NFM

The affected partitions are set to AREOR.

Online REORG must be run to materialize the pending changes.

*REORG SHRLEVEL NONE does not materialize these changes.

Supported table spaces types are:

*UTS – partitioned by range (PBR)

*Classic partitioned table spaces (table controlled partitioning)• Objects, in which the partitioning is index controlled, must be first converted !

* The new limit keys are materialized in SYSTABLEPART in the SWITCH

phase (new message DSNU2916I).

Part-ition

Logical_Part

Limitkey_Internal

3 2 350

Part-ition

Logical_Part

Limitkey_Internal

3 2 300

Option_Value Part-ition

…350 3



DSNUM 2

DSNUM 3

DSNUM 4

DSNUM 1

logial viewphysical view

Partition 1

Partition 2

Partition 3

Partition 4

Alter Table ... Alter Partition 3 Ending At 350 Inclusive

AREOR

AREOR

DSNUM 3

DSNUM 4

50, 125, 150, 175, 200

225, 250, 275, 300

325, 350, 375, 400

425, 450, 475 500

225, 250, 275, 300,325, 350

375, 400

keysstatusDB2 Catalog

Partition 2

Partition 3

SYSIBM.SYSTABLEPART

SYSIBM.SYSPENDINGDDL

Online Reorg including physical part 3and 4 must run to materialize the Alter

SYSIBM.SYSTABLEPART

Materialization takes placein the SWITCH phase

SYSIBM.SYSPENDINGDDL is empty



ALTER TABLE … ALTER PARTITION integerENDING AT …

Removed restriction:

* If the table space is in a materialized query table relation.

* RECOVER PIT is allowed, but requires a subsequent REORG due to setting

of REORP after the recovery.

Changed behavior:

* REORG SHRLEVEL NONE does not materialize the pending changes.

* LOAD REPLACE does not materialize the pending changes.

If special REORG keywords are not allowed, message DSNU2917I is issued

(RC=8):

* REBALANCE

* AUX NO (if the table space has LOB columns)



ALTER TABLESPACE ... DROP PENDINGCHANGES

Current: AREOR setting is not removed.

New: AREOR setting is removed.*Exception: AREOR is still kept if transition to hash organization

is in progress.



Workfile

Enhancements



Workfile Enhancements

Issue a warning message when the total amount of in-use work filestorage (including DGTT) has reached a storage shortage threshold(e.g.10% of total configured storage)

*WFSTGUSE_AGENT_THRESHOLD : agent level space usagealert threshold (percentage range 0 – 100, default = 0)

* WFSTGUSE_SYSTEM_THRESHOLD: system level spaceusage alert threshold (percentage range 0 -100, default = 90)

Once the warning message has been issued, DB2 will wait for 30system check points before issuing another same warning message

Update space map page under S-Latch to reduce space map pagecontention



Workfile Statistics trace records

Add additional instrumentation data in DB2 statistics

trace records (QIST) :* Total configured DASD storage for a work file database (QISTWSTG)

* Current storage used and Maximum total storage ever used (include DGTTs):

available in DB2 10

* Maximum total storage even used by non-DGTT (QISTWFMXU)

* Current storage used by non-DGTTs (QISTWFCTO)

* Total preferred configured storage for DGTTs (QISTDGTTSTG)

* Maximum total storage ever used by DGTTs since DB2 is up

(QISTDGTTMXU)

* Current storage used by DGTTs (QISTDGTTCTO)

0

DB2 11 forz/OS

ApplicationCompatibility

Problem

• Changes to SQL and XML behavior are problematic to adopt

– It is difficult to understand the impact of change to thousands ofapplications

– Change needs to be controlled at the application level

• Service stream incompatibilities cause maintenance issues, especiallywhen key PTFs pre-req the PTFs that cause the changed behavior

• Changes at a release boundary allow for some planning, but it isdifficult

to sync application changes with release migration plans

Solution Details

• BIND/REBIND PACKAGE

Syntax

BIND/REBIND

>----+-----------------------------------------+----------------------------->

'-------- APPLCOMPAT---(--

V10R1--)--------'

(--V11R1--)

• REBIND TRIGGER

Syntax

REBINDER TRIGG

>----+-----------------------------------------6

DSNT298I

csect-name ATTEMPT TO USE COMMAND OR OPTION command WHENTHEAPPLICATION COMPATIBILITY IS SET FOR A PREVIOUS LEVEL

Explanation

A new command or command option was issued when applicationcompatibility bind option is set to a prior DB2 release. Anotherpossibilityis when a command or command option introduced in release n was usedbut the application compatibility bind option is lower than n.

command The command or command option that can be used only whenthe application compatibility bind option is set to the compatible release.

7

Solution Details

• CREATE/ALTER PROCEDURE (SQL native):

Syntax

>>-CREATE/ALTER PROCEDURE-------------------------------------------------------------->

option-list:

>-----+---------------------+-----------------------------------------------------------><

|---APPLCOMPAT V10R1--|

+---APPLCOMPAT V11R1--+

8

• CREATE/ALTER FUNCTION (SQL native):

ColumnsTableName

SYSPACKAGE

ColumnName

APPLCOMPAT

DataType

VARCHAR(10)

Description

TheDB2releaselevelthattheSQLstatementsinthispackageiscompatiblewith.

SYSPACKCOPY APPLCOMPAT VARCHAR(10) TheDB2releaselevelthattheSQLstatementsinthispackageiscompatiblewith.

Updated Catalog

9

–

–

•

•

Solution Details (cont)

• ALL SQL DML and XML changes are fencedby

these options. This includes:• Function Resolution

New rules SQL function

• Result type changesIncompatible SQL changes such as:• SQLCODE changes

BIF changes

…

SQLCODE -4743

-4743

ATTEMPT TO USE A FUNCTION WHEN THE APPLICATION COMPATIBILITY IS SET FOR APREVIOUS LEVEL

Explanation

Functions that this release of DB2® introduces cannot be used when application compatibility setting is foraprior DB2 release. An attempt was made to execute one of these functions. Another possibility is when afunction introduced in release n was used but the application compatibility setting is lower than n.

To use the new functions that have been introduced in this release of DB2, the application compatibilitybindoption or special register CURRENT APPLICATION COMPATIBILITY must be set to the current release.Anattempt to use functions introduced in release n requires the application compatibility bind option or specialregister CURRENT APPLICATION COMPATIBILITY value to be at least n.

11

•

– Ensure

code

level

behavior

Once

Interaction with NFM is intended to fence application and system behavior ina migration/co-existance

environmentthat the “n-m”* level of code can process objects createdin the “n” level of the– Ensure that applications bound on level “n” can run (afterautobind) “n-

m”*

applicationis consistent across releases of DB2– As such, APPLCOMPAT for the current release “n” may notbespecified until NFM

– mode in NFM MODE, either APPLCOMPAT(n)or APPLCOMPAT(n-1) may be specified (see the following

charts)

* n-m where m = 1 or 2. DB2 10 is the lowest level that may bespecified

DB2 11 Enabling New

Function Mode (ENFM10)

DB2 11Catalog

DB2 11 New Function

Mode (NFM)

CATMAINT

UPDATE

(DSNTIJTC)

CATENFM

COMPLETE(DSNTIJNF)

CATENFMSTART

(DSNTIJEN)

DB2 11

Libraries

DB2 10Catalog

DB2 10Libraries

DB2 10 DB2

DB2 11 Conversion

Mode (CM10)

10

Normal Migration11

Mode

(NFM) With

DB2 SPE New Function

1 – 2 months

Data SharingCoexistence1 week

2hours

Bind with APPLCOMPAT(V10R1) option only

Bind with APPLCOMPAT(V10R1) or

APPLCOMPAT(V11R1)

Interaction with NFM (cont)

• What about New DDL and Authorization syntax and semantics in anapplication bound with APPLCOMPAT(n-1)?

– DDL and Authorization are fenced by NFM and are not affectedby

the APPLCOMPAT level

• For example – You can use SPUFI - bound withAPPLCOMPAT(V10R1) to “CREATE” a GLOBAL VARIABLE(proposed DB2 11 function) and authorize a user to “SET” it, but

youcannot use that same version of SPUFI to reference the variable youjust created.

29

Summary

• With the APPLCOMPAT bind option, we are striving to separaterelease

migration from application changes that may be necessary to migrate toa new release.


®

DB2 for z/OS DB2 11

Extended RBA / LRSNReliability, Availability, Serviceability

70

DB2 for z/OS DB2 11

RBA Today

DB2’s Relative Byte Address (RBA) for logging is 6 bytes

– Gives 256TB of log record addressing capacity per DB2 subsystem/member

With faster processors, consolidation of DB2 members, and heavysustained logging rates, DB2 can exhaust the 6-byte RBA

• Warning messages issued at x’FFFF00000000’

– Some customers wrap a log today in 2 months, others just approaching limit

Manual recovery actions are needed to reset RBA to zeros

– Data Sharing: Shut down affected member and start a new one in its place

• Operational issues, such as automation, routing, and others

– Non Data Sharing: reset PGLOGRBA field in every page to zero

• Special form of Image Copy – this is extended outage (>1 day)

71

DB2 for z/OS DB2 11

LRSN Today

The data sharing Log Record Sequence Number (LRSN) is the high order 6 bytesof the 8-byte time-of-day clock which hits end of range in 2042

Some groups have a non-zero LRSN “delta” which gets added to the TOD clock– Then LRSN will hit end of range prior to 2042

– Print Log Map (DSNJU004) shows the LRSN delta value• A “delta” value could be set when data sharing is enabled or re-enabled• Example: data sharing enabled when the highest non shared RBA written is > LRSN

Some non data sharing customers have enabled data sharing to circumvent RBAnearing end-of-range– This causes a non-zero LRSN delta if High Written RBA > LRSN

No current circumvention for end of LRSN

6-byte LRSN value has precision to only 16 microseconds– Can cause LRSN ‘spinning’ which burns extra CPU and aggravates log latch contention

– Changes in DB2 9 and 10 address most LRSN spin situations

– Some still exist, mainly with row update and delete, and occasionally multi row insert

DB2 for z/OS DB2 11

Solution

Expand the RBA and LRSN to 10 bytes

– RBA addressing capacity of 1 yottabyte (2**80)

• X’00000000112233445566’ – add positions to high order RBA to left– LRSN extended on left by 1 byte, on the right by 3 bytes

•

••

x'00LLLLLLLLLLLL000000’>30,000 years and 16M times more precisionNumeric progression after end of 6-byte LRSN:

––––

Next value 01 000000000000 000000.The next value is 01 000000000000 000001.Eventually, it becomes 01 000000000000 FFFFFFAnd next value is 01 000000000001 000000

Reason for 10

– 8 bytes is not enough to solve LRSN issues and insufficient for longer term

••

One byte on lower end of LRSN means DB2 could spin on current processors8-byte RBA -- could exhaust the range in < week with technology now in IBMlabs

72

DB2 for z/OS DB2 11

Solution

NFM only (6 byte RBA/LRSN continues to be used in CM)

In NFM, DB2 uses 6-byte values until you take action to convert

Two conversion tasks (order not important):

1. Convert BSDSes to new format to enable logging with larger RBAs/LRSNs

2. Convert DB2 objects to new page format via REORG in piecemeal fashion overdays/weeks/months

These tasks are optional (but recommended)

– If you don’t care about larger RBAs/LRSNs then you don’t have to convert

– But performance can be better if you convert BSDSes••

Avoid internal conversion overhead on log write (1%)Eliminate LRSN spin

– Even if you are “safe” w/I limits, DB2 11 is 10 byte system and 6-byte RBA will beconverted before processing

73

74

DB2 for z/OS DB2 11

Major areas of change

Lots internal – affects the bones of DB2

Utilities - Most have no external changes

Catalog

Trace records: numerous IFCIDs are changed for extended RBA/LRSN

New page format to handle 10-byte PGLOGRBA equivalent

Utilities can handle either page format, so can SQL, DDL

Definition: “Basic” means 6-byte RBA/LRSN. “Extended” is 10-bytes

Check with your vendor for specific tools you use: they must be able tohandle/tolerate as appropriate:

• Basic and Extended RBA/LRSN• Both page formats

75

DB2 for z/OS DB2 11

Considerations for Replication products

Replication products must be updated for DB2 11

IFCID 306 products will not work in DB2 11 CM

– They always see the 10-byte log format.

• Q-Replication

• CDC

• Others

Same for products that use log capture exit (E-Net / RRDF)

– Interface changed significantly

In DB2 11 CM any coexisting DB2 10 members see log records in oldformat

– Workaround for IFCID 306 based products if customer does not yet have theupdated version installed

76

DB2 for z/OS DB2 11

Modes of DB2 11 implementation

CM – 6-byte RBA/LRSN and pages in current or “basic” format

– New catalog objects for DB2 11 are created in CM in basic page formatmode, but with extended RBA/LRSN columns, where applicable

– RECOVER utility: TORBA, TOLOGPOINT, RESTOREBEFORE keywordsextended to accept either 6- or 10-byte RBA/LRSN input

– Standalone log utilities accept larger RBA/LRSN values

– REPORT RECOVERY output accommodates extended RBA/LRSN values

– Work files are in extended format

ENFM

– DB2 10 RBA/LRSN related catalog columns are extended to 10 bytes

– Catalog objects updated/reorged may be converted to extended page formatbased on DSNZPARM OBJECT_CREATE_FORMAT (Basic/Extended)

– DSNTIJNF SCA converted to handle 10-byte RBA/LRSN

77

DB2 for z/OS DB2 11

Modes of DB2 11 implementation

NFM

– Can convert BSDSes for extended RBA/LRSN

– Begin to convert objects any time you REORG

– Almost all partitioned objects can be converted independently (by part(s))

– Conversion MUST have occurred before All Fs for either RBA/LRSN

• Else read only object since updates cannot be made in basic format

– New job DSNTIJCV converts catalog/directory basic > extended page format

• Optional, if you are not close to end of the log

• Converts any not yet converted; over time all are in extended page format

Any mode: The Unit of Recovery ID (URID) extends to 10 bytes

– DISPLAY THREAD output, restart status table messages

– Padded in CM with zeros

78

DB2 for z/OS DB2 11

Procedure to Convert to Larger LRSN

New warning messages for LRSN reaching end of range

– DSNJ034I - WARNING: one year or less

– DSNJ035E – Critical: 2 weeks of LRSN end (note “E” for “action”)

Convert the BSDSes of all D/S members to activate the extended logrecord format (DSNJCNVT offline utility while DB2 is down)

– Roll through one data sharing member at a time for continuous availability

Convert table spaces and indexes to the new page format

– When LRSN advances past the 6-byte limit (x’FFFFFFFFFFFF’)

• objects in the old format become read only but still opened as RW

No order implied if members not approaching 6-byte limit, but makessense to convert BSDSes first

Procedure for non sharing DB2 subsystems similar – in b/g material

79

DB2 for z/OS DB2 11

Converting the Page Format

An object can be converted to / from extended format by REORG orREBUILD utilities

A partitioned TS or index can be converted one part at a time

– Except PBG hashed TS which must be converted at the TS level (all parts)

CLONE tables cannot be converted

– DROP CLONE

– Convert base table, then ALTER ADD CLONE

– Re add all the cloned authorizations

SYSINDEXPART and SYSTABLEPART indicate the page format

80

DB2 for z/OS DB2 11

Current 4K page format (Basic)

Pghead = PGLOGRBA (6 bytesstored)

E is PGTAIL (2 bytes)

Last byte is PGEND (parity)

– “E” - x’11000101’ or

– “N” - x’11010101’

– Underlined bit called PGBigTailApplies

– Zeros mean off (does not apply)

So DB2 uses 6-byte RBA/LRSNs inpghead area

PGLOGRBA

E

pgheadPGBigTail(20)

Unused(6) Reserved(2)

PGBigRBA(10) PGTAIL(2)

81

DB2 for z/OS DB2 11

4K page with Extended RBA/LRSN

PGBigTail (20 bytes)

1. Unused – 6 bytes

2. Reserved – 2 bytes

3. PGBigRBA – 10 bytes – ExtendedRBA/LRSN stored

4. PGTAIL – 2 bytes

byte 1 reserved

Byte 2 – PGEND either

x’01000010’ or

x’01010010’

Parity is bits 2-4 (100 or 101) and

PGBigTailApplies is bit 7 -If one, it is ON, (Extended formatPGBigRBA)

if zero, it doesn’t (Basic format)All page sizes have same format

Maximum row length unchanged

82

DB2 for z/OS DB2 11

New DSNZPARMs

OBJECT_CREATE_FORMAT

– Parameter states page format new objects use, basic or extended

– Default is BASIC (6-byte RBA/LRSN format)

– EXTENDED only takes effect in NFM (10-byte format)

• If you are going to convert, set it in CM and catalog objects touched inENFM will be converted for you

UTILITY_OBJECT_CONVERSION (ignored during CM)

– Should page conversion be done by default by REORG, LOAD, REBUILD?

• BASIC: the output object will be basic format (default)• EXTENDED: the output object will be extended format (only NFM)• NONE: the output object remains the same format

– RBALRSN_CONVERSION utility parm of same options override DSNZPARM

• Specify the format of the object on completion of the utility– Can’t specify in CM

– Specify for catalog objects with altered columns during ENFM

83

DB2 for z/OS DB2 11

Main Utilities

RECOVER

REORG / REBUILD INDEX / LOAD REPLACE

Other utilities support both basic and extended page formats

DSN1COPY – When source and target have differing page formats,copied data may not reflect target catalog

– REPAIR – new option to correct catalog/OBD for format of actual data setsafter DSN1COPY

LISTDEF – adds Filtering on Basic/Extended page format

Utilities display/messages are in extended format regardless of actualobject format

84

DB2 for z/OS DB2 11

RECOVER

RECOVER accepts longer values for TORBA, TOLOGPOINT,RESTOREBEFORE

– Six-byte RBAs may be specified as ten-byte operands with the six-byte valueright justified in ten-bytes, i.e.x'00000000RRRRRRRRRRRR‘

– Six-byte LRSNs may be specified as ten-byte operands in byte locations twothrough seven, i.e. x'00LLLLLLLLLLLL000000'

– Corrects any mismatch between recovered data and catalog

• i.e. current = Extended, but TORBA = Basic

85

DB2 for z/OS DB2 11

Utilities in General

Use DSNZPARM UTILITY_OBJECT_CONVERSION globally

– Unless overriding Utility statement parms

• RBALRSN_CONVERSION

Mapping table handles extended RBA/LRSN

– Automatically created!

Any SHRLEVEL can convert

Index and underlying object unrelated

– Index can be left in different page format from underlying objects

• Example: 50 parts extended – 500 parts in basic – NPI can be either

Utility Partition PartitionedIndex

NonPartitionedIndex

ReorgTablespace Y Y Y

ReorgPART Y Y N

ReorgINDEX Y Y

ReorgINDEXPART Y N

RebuildINDEX Y Y

RebuildINDEXPART Y N

LoadReplaceTablespace Y Y Y

LoadReplacePART Y Y N

86

DB2 for z/OS DB2 11

Utilities and Page Format Conversion

Tablespace reorg required for only:– Hashed tables (hash algorithm to spread data)

– XML Versioned T/S – to expand START_TS and END_TS from 8 > 10 bytes - Must occur beforeor with conversion to extended page format - then Part Reorg OK

Key: NPI must be converted specifically outside tablespace level REORG

FromDB211Version

ToVersion BasicPageformat

Extendedpageformat

NFM DB211NFM Yes Yes

NFM DB29/10 Yes No

CM DB29/10 Yes No

CM* DB29/10 Yes No

87

DB2 for z/OS DB2 11

DSN1COPY from DB2 11 to another subsystem

Reason for conversion back and forth is porting to another DB2

Table shows the page format for a DB2 11 object that must be copied to a DB2subsystem that is on another version of DB2

A REORG is necessary if an object is already in extended page format

– DB2 9/10 understands only 6-byte basic page formats

88

DB2 for z/OS DB2 11

Recovery Considerations

Example for table space ‘TS1’:

T1 - image copyT2 - quiesce

T3 - convert to extended page formatT4 - quiesce

T5 - image copyT6 - quiesce

Q1: Can I recover TS1 back to time T2?Ans: Yes. And the catalog is updated to indicate “basic” format

Q2: Can I then roll forward to time T4?Ans: No. RECOVER does not allow log apply across a REORG

Messages RBA(D/Sornon-D/S)

LRSND/Sonly

ReasonCode

DSNJ032I X’F00000000000’ n/a n/a

DSNJ033E X’FFFF00000000’ n/a 00D10251

DSNJ034I n/a 1yearorless

n/a

DSNJ035E n/a 2weeks 00D10251

Advisorylimit(‘soft’) X’FFF800000000’ n/a -90400C2026D

AdvisorylimitDS(‘soft’) 2months -90400C2026D

89

DB2 for z/OS DB2 11

Messages and Limits

Messages appear at DB2 restart and every log switch – not just a speed bump!

Soft limit:

– Rollback and abort allowed

– SQL updates not allowed

– DDL ok if DROP/CREATE for extended format tablespace – utilities okay too

91

DB2 for z/OS DB2 11

End of RBA Timeline

F00000000000

DSNJ032I

Every logswitch and

Restart

Begin ConvertingTo Extended

DB2 in MAINT modeConvert BSDS

FFF80000000

Soft limit-904 00C206D

No SQL updatesfor basicobjects

FFFF00000000

DSNJ033ECritical

DB2 stops00D10251

FFFFFFFFFFF

Hard limit

Convert BSDSand objects NOW!

92

DB2 for z/OS DB2 11

End of LRSN Timeline

1 Year left

DSNJ034I

Every logswitch and

Restart

Begin ConvertingTo Extended

DB2 in MAINT modeConvert BSDS

2 months left

Soft limit-904 00C206D

No SQL updatesfor basicobjects

2 weeks left

DSNJ034ICritical

DSNJ035E

DB2 stops00D10251

FFFFFFFFFFF

Hard limit

Convert BSDSand objects NOW!

93

DB2 for z/OS DB2 11

Benefits

Grants permanent relief for operational issues/outages caused byhitting the end of RBA or LRSN

– Overhead of 6-byte logs is about 1%

– Converting logs to 10-byte and tables to extended format may get 3% CPUimprovement (your mileage may vary)

Complete Flexibility

– Conversion of BSDS and DB2 objects can be done in any order

– Conversion of DB2 objects performed non-disruptively with Reorg/Load

••

No extra effort – can be done over timeCan be done either way – Basic to Extended or Extended to Basic

– Conversion optional if you are not close to end of log

1. But DB2 code always converts Basic RBA/LRSN to Extended2. Better update performance with Extended page formats

94

DB2 for z/OS DB2 11

The Message

DB2 11 is a 10-byte RBA/LRSN system

If you think you will be “conservative” or “safe” byremaining in 6-byte RBA/LRSN mode,

Think again

–Every 6-byte RBA/LRSN that is touched is converted to10-bytes before processing, and may need to beconverted back to 6-bytes.


®

DB2 for z/OS DB2 11

Background slides

96

DB2 for z/OS DB2 11

Active Log Data Set Background

Support for active log data sets up to 4GB was introduced in DB2 V6

– The previous maximum was 2GB

– Total active log space: 124GB (31 active logs * 4GB)

– Up to 1000 archive logs, 4TB

DB2 V8 NFM and later allows up to 93 active log data sets

– Maximum active log space: 372GB

– Up to 10000 archive logs, 40TB

In DB2 11, active log formatter (DSNJLOGF) can preformat 10 log datasets per execution

– One in DB2 10

97

DB2 for z/OS DB2 11

Non Data Sharing Procedure to Convert to Large RBA

Convert the BSDS to activate the extended log record format

– New job DSNJCNVT

Convert table spaces and indexes to the new page format

– Once the RBA value has advanced past the 6-byte limit (x’FFFFFFFFFFFF’)then any object still in the old format is forced to R/O access by DB2

– No way to update the RBA now

98

DB2 for z/OS DB2 11

BSDS Conversion

Run new DSNJCNVT utility to convert BSDS after creating new ones.

– Need new output BSDS because CI size is increased

– Rename old BSDS with IDCAMS ALTER NEWNAME (SYSUT1/2)

– Create new BSDS with larger CI size (SYSUT 3/4)

– Typically takes much less than one minute to execute

99

DB2 for z/OS DB2 11

Log Limits and behavior

Advisory logging occurs at RBA “FFF80000000’x or LRSN about 2months before LRSN exhausted. Updates for rollback work

– No SQL updates allowed

Actual limit occurs when RBA/LRSN no longer fit in 6 bytes

– LOAD REPLACE, REBUILD, REORG works for conversion into EXTENDEDformat only – ignores DSNZPARMs and utility settings

– RECOVER of object in BASIC format sets REORP/RBLDP

– RESTORE SYSTEM allowed for both limits for non-D/S. For D/S allowed onlyif SLB created after BSDS conversion or recover from tape..

MERGECOPY – Basic format not allowed

®


IBM Software Group

Archive Transparency

IBM Software Group | DB2 information management software

DB2 11 – Part III: Archive Transparency

Importance and Business Need

Technical Details


DB2 11 Part III – Importance and Business Need

Archive Transparency:

It provides basic archive and retrieval functions using SQL

The retrieval of data from base table or base table plus its associated archivetable is controlled by the setting of a built-in system defined global variablewithout changing SQL in the applications

The majority of applications retrieve data from base table. There should be noperformance degradation by UNION ALL to its associated archive table.

The functionality is primarily of interest to

DBAs

application developers

end users

consultants


DB2 11 Controls of ArchiveTransparent Access and Archive Data

Built-in global variablesSYSIBMADM.GET_ARCHIVE : CHAR(1), value can be either 'Y' or 'N’SYSIBMADM.MOVE_TO_ARCHIVE: CHAR(1), value can be either 'Y' or 'N'

Read/write of the variables: The read authority is granted to PUBLIC. The authority rules forgrant are consistent with user-defined global variables. The administrative authorities thatcan grant global variable privileges are install SYSADM, SYSADM, SECADM, andACCESSCTRL. If the SEPARATE_SECURITY zparm is set, then SYSADM cannot grantglobal variable privileges.

Default: 'N'

Transparent archive data via DELETE: set the built-in global variableSYSIBMADM.MOVE_TO_ARCHIVE to 'Y', the deleted records are propagated to archivetable automatically by DB2 via single DELETE SQL statement

Transparent access archive data: set the built-in global variableSYSIBMADM.GET_ARCHIVE to 'Y' for all subsequent SQL statements including those frominvoked function, stored procedure, and trigger. This allows the application to see bothactive and archive data without modifying the SQL statements in multiple packages. DB2rewrites the query with UNION ALL operator.

DB2 11 – Controls of Archive Transparency

The following bind/routine options are added to control the sensitivity to the settings of the SYSIBMADM.GET_ARCHIVE global variable:

ARCHIVESENSITIVE (default YES) BIND PACKAGE REBIND PACKAGE REBIND TRIGGER PACKAGE CREATE TRIGGER (implicit trigger package) ARCHIVE SENSITIVE (default YES) CREATE FUNCTION (SQL scalar) ALTER FUNCTION (SQL scalar) CREATE PROCEDURE (SQL native) ALTER PROCEDURE (SQL native) ARCHIVE SENSITIVE (DB2I panels) DB2I Panel DSNEBP10 DB2I Panel DSNEBP11 DB2I Panel DSNEBP19


Page 106

DB2 11 – DDL Example of Archive Transparency Two table approach: archive-enabled table stores active data; archive table stores archive

data

CREATE TABLE policy_info_aet(policy_id CHAR(10) NOT NULL,coverage INT NOT NULL);

CREATE TABLE policy_info_arc(policy_id CHAR(10) NOT NULL,coverage INT NOT NULL);

ALTER table to enable archive transparency:

ALTER TABLE policy_info_aet ENABLE ARCHIVE USE policy_info_arc;

ALTER table add column: single ALTER statement on archive-enabled table also adds thecolumn(s) to archive table

ALTER TABLE policy_info_aet ADD COLUMN customer_id;

DB2 also implicitly adds customer_id column to archive table policy_info_arc

Temporal tables (STT, ATT, bitemporal tables) and archive-enabled table (AET) aremutually exclusive; AET cannot be referenced in same SQL statement with STT or ATT.


DB2 11 – Example of Querying Archive Data Assuming --

There are three archive-enabled tables AET1, AET2, and AET3

In an application, there is an SQL statement:

SELECT UDF_1(AET1.C1) FROM AET1, AET2;

In UDF_1, there is an SQL statement:

SELECT * FROM AET3;

The application wants to get the result from both base and archive table

Transparent access to archive data with DB2 11 feature --

NO change of the application and UDF_1, before calling the application

SET SYSIBMADM.GET_ARCHIVE = ‘Y’ ;

If the application just wants to access the active data

SET SYSIBMADM.GET_ARCHIVE = ‘N’ ; *** 'N' is the default ***

Transparent retrieval of archive data: there is no need to change SQL statements in anexisting application to get current data only or to get both current and archive data. Thesetting of the SYSIBMADM.GET_ARCHIVE built-in global variable offers transparentaccess to archive data for queries.

Page 107


DB2 11 – Performance Consideration

To avoid runtime performance degradation of static DML with the built-inSYSIBMADM.GET_ARCHIVE global variable, there is a design of two sections per static DMLwith AET reference.

Given a static SQL SELECT * FROM AET1, and if the package is bound with optionARCHIVESENSITIVE YES, then DB2 binds this SQL twice --

Original section: bind the original SELECT * FROM AET1 for current data.The majority of transparent archive applications request for current data only. There is noperformance degradation caused by UNION ALL query transformation.

Extended section: bind again an extended section by expanding AET1 to UNION ALL of AET1and its associated archive table.

For the application, there is no change.

At execution time:

If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'N' value (the default),DB2 chooses the original section

If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'Y' value, DB2 choosesthe extended section to get both current and archive data.

Page 108


DB2 11 – GET_ARCHIVE and Data Change Operation

INSERT/UPDATE/DELETE/MERGE: if AET is the target of the data-change operation (eitherdirectly or indirectly thru views), NO implicit union all query transformation on the target

AET when SYSIBMADM.GET_ARCHIVE global variable is in effect.

Example: given a static SQL DELETE FROM AET1 WHERE EXISTS (SELECT 1 FROMAET2 WHERE CUSTOMER_ID = 'JOHN'), and if the package is bound with option

ARCHIVESENSITIVE YES, then DB2 binds this SQL twice --

Original section: bind the original delete for current data. The majority of transparent archiveapplications request for current data only. There is no performance degradation caused byUNION ALL query transformation.

Extended section: bind again an extended section by expanding AET2 to UNION ALL with itsassociated archive table. The target table AET1 of DELETE is not expanded with UNION ALL.

For the application, there is no change on the DML statements.

At execution time:

-- If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'N' value (the default),DB2 chooses the original section

-- If the built-in SYSIBMADM.GET_ARCHIVE global variable contains 'Y' value, DB2chooses the extended section to get both current and archive data from AET2 for the

evaluation of EXISTS predicate

Page 109


DB2 11 – MOVE_TO_ARCHIVE and Data Change Operation

DELETE: single DELETE statement can trigger transparent archive when MOVE_TO_ARCHIVEglobal variable is in effect; no additional privilege is required on archive table, only the privilegeson delete of archive-enabled table are needed

INSERT/UPDATE/MERGE (blocked in archive mode):

If SYSIBMADM.MOVE_TO_ARCHIVE = ‘Y’, the INSERT, UPDATE and MERGE will fail. Thisis based on the assumption that in archive mode, only DELETE is meaningful.

If SYSIBMADM.MOVE_TO_ARCHIVE = ‘N’, no archive mode failure

Example -- given a SQL DELETE FROM AET1, regardless whether it is dynamic or static SQL,and whether the package is bound with option ARCHIVESENSITIVE YES or NO:

For the application, there is no change on the data-change SQL statements.

If the built-in SYSIBMADM.MOVE_TO_ARCHIVE global variable contains 'Y' value, for each rowdeleted from archive-enabled table AET1, DB2 will insert it into the corresponding archive table.This is an SQL performance improvement to help archiving data with a single DELETE.

If the built-in SYSIBMADM.MOVE_TO_ARCHIVE global variable contains 'N' value (the default),DB2 does no data propagation to archive table. It is a regular delete statement.

Page 110


DB2 11 – DDLs and DML Error Scenarios

NO Impacts of GET_ARCHIVE and MOVE_TO_ARCHIVE on DDL Statements (NO packagegeneration)

ViewMQTInline UDFRow PermissionColumn Mask… ...

Error Scenarios of GET_ARCHIVE on DML Statements Inline table UDF references: the SQL table UDF contains AET, and the SYSIBMADM.GET_ARCHIVE global

variable is in effect Application of access controls: the access control contains AET, and the SYSIBMADM.GET_ARCHIVE

global variable is in effect

Error Scenarios of MOVE_TO_ARCHIVE on DML Statements

INSERT/UPDATE/MERGE: if SYSIBMADM.MOVE_TO_ARCHIVE = ‘Y’, the INSERT, UPDATE and MERGEon AET will fail. This is based on the assumption that in archive mode, only DELETE is meaningful.

Error Scenarios NOT tied to GET_ARCHIVE and MOVE_TO_ARCHIVE on DML Statements

WHEN clause of trigger: CREATE TRIGGER and REBIND TRIGGER PACKAGE will fail if the trigger hasAET reference in WHEN clause and the trigger package is generated with ARCHIVESENSITIVE YES

Page 111

SYSTEM_TIMEVersioning ArchiveTransparency

Schema--twotableapproach

currenttable&historytablesamecolumn#,columnname,columnattributes(datatype,etc)

archive-enabledtable&archivetablesamecolumn#,columnname,columnattributes(datatype,etc)

ROWBEGIN/ROWEND/TRANSIDcolumns mandatory none

Periodconcept yes–SYSTEM_TIMEperiod none,notcompatiblewithSTT

CompatiblewithATT yes no

Bindoption SYSTIMESENSITIVE ARCHIVESENSITIVE

Implicitunionallquerytransformation

controlledbyCURRENTTEMPORALSYSTEM_TIMEspecialregister:thereareimplicitsystem_timepredicates

controlledbybuilt-inglobalvariableSYSIBMADM.GET_ARCHIVE:union

allonly,noimplicitpredicatesforarchivetransparency

Datapropagationtohistory/archivetable

UPDATEandDELETE DELETE,SYSIBMADM.MOVE_TO_ARCHIVE

StaticDMLs twosectiondesign twosectiondesign


Page 116

sc.: SYSTEM_TIME Versioning vs Archive Transparency

Index Enhancements

• Suppress-null indexes• Index duplicate skipping• Index Skipping • Auto Cleanup of Pseudo-deleted Index

Entries

Migration Planning Workshop

© 2013 IBM Corporation120

DB2 11 for z/OS

Suppress-null indexes Support creation of indexes where the default NULL values are

excluded from the index– Reduced index size, improved insert/update/delete performance, 3rd party

DBMS compatibility

New CREATE INDEX keywords– INCLUDE NULL KEYS (default)

• DB2 will create an index entry even if every key column contains the NULL value– EXCLUDE NULL KEYS

• DB2 will not create an index entry when every key column contains the NULLvalue. If any key column is not null the index entry will be indexed

Conditions:– EXCLUDE NULL KEYS must not be specified if any of the columns

identified by column-name are defined as NOT NULL

– Not supported if the index is defined as a partitioning index

IDX



DB2 11 for z/OS

Auto Cleanup of Pseudo-deleted Index Entries …

Pre DB2 11 pseudo deleted keys can accumulate over time

– Index size grows with increasing number of pseudo-deleted index entries

• More getpages and lock requests required

• Increased CPU cost and longer elapsed times for index search

– Applications may encounter deadlocks and timeouts duringINSERT/UPDATE/DELETE

• Pseudo-deleted index entries are locked to enforce uniqueness

• RID reuse by INSERT following DELETE can lead to deadlock

DB2 11 provides automatic cleanup of pseudo deleted entries

– Clean up both pseudo empty index pages and pseudo deleted index entries

• Reduce impact of pseudo delete entries

• Reduce the need to run the REORG INDEX utility

IDX



DB2 11 for z/OS

Auto Cleanup of Pseudo-deleted Index Entries DB2 11 automatically cleans up pseudo deleted entries

– zIIP eligible and runs in the background

– Designed to have minimal or no disruption to applications

– New ZParm INDEX_CLEANUP_THREADS (0-128) to control number of concurrentcleanup tasks––––

Default is 100 disables index cleanupValue can vary between members of a data sharing groupOnline changeable

– New SYSIBM.SYSINDEXCLEANUP catalog table to control auto cleanup at indexlevel• Day of week/month, start/end time.• By default cleanup is enabled for all indexes

– IFCID 377 written when index is cleaned up

Benefits of automatic pseudo delete cleanup– Reduce size of some indexes, fewer getpages

– Improve SQL performance, Lower CPU, less elapsed time

– Reduce need to run REORG INDEX

IDX



DB2 11 for z/OS

Pseudo-deleted Index Cleanup Cleanup done under system tasks, run as enclave SRBs and zIIP eligible

• Parent thread (one per DB2 member) reads through RTS to find candidates– Runs every 5 minutes

• Eligible indexes sorted based on number of pseudo deleted rids to delete highest first

NAME

IX1

IX2

IX3

IX4

…

n

n

n

n

NPAGES

100

1000

500

2000

…

X

X

X

X

REORGPSEUDODELETES

5000

20000

100000

75000

• Child threads assigned based on ZParm setting

SYSIBM.SYSINDEXSPACESTATS

Parentthread

Index

IX3

SELECT FROM… ORDER BY

Child cleanupthread IX3

Child cleanupthread IX4

– Child cleanup thread only startedif Index already open for INSERT,

UPDATE or DELETE

• ‘X’-type P-lock already held

IDX

IX4 InMemory

IX2

IX1



DB2 11 for z/OS

SYSIBM.SYSINDEXCLEANUP Example All index spaces in DB_1234 are enabled for cleanup on

Sundays from 4:30 until noon, except

– Index space IX_9876 is always disabled for cleanup

All index spaces in DB_XYZ disabled for cleanup onSaturdays, and

– Index space IX_ABC is disabled for cleanup on the 30th of each monthfrom 1:30 to 7:30

DBNAME INDEX-SPACE

ENABLE_DISABLE

MONTH_WEEK

MONTH DAY START_TIME

END_TIME

DB_1234

DB_1234

DB_XYZ

DB_XYZ

NULL

IX_9876

NULL

IX_ABC

E

D

D

D

W

NULL

W

M

NULL

NULL

NULL

NULL

7

NULL

6

30

043000

NULL

NULL

001300

120000

NULL

NULL

073000

SYSIBM.SYSINDEXCLEANUP

IDX

© 2013 ibm corporation information management high availability improvements

Documents