faster fetch

DB2 ATS (Advanced Technical Support)

© 2006 IBM Corporation

Christopher J. Crone

Faster FETCH & INSERT V8

Multiple Row Processing

Heart of America RUGMarch 13, 2006

Updated and presented by John Iczkovits


© 2006 IBM Corporation2

Presentation Topics

Host Variable ArraysMultiple-Row InsertMultiple-Row Fetch

DELETE WHERE CURRENT OFUPDATE WHERE CURRENT OF



Host Variable Arrays



Host Variable Arrays

Host variable array is an array in which each element of the array contains a value for the same column

Changes have been made to allow host variable arrays in:ƒCOBOLƒPL/1ƒC++ƒNOTE: Assembler support is limited to cases where

USING DESCRIPTOR is allowed. Assembler pre-compiler does not recognize declaration of host variable arrays. The programmer is responsible for allocating storage correctly, etc.

Can only be referenced in multi-row fetch or insertIn general, arrays may not be arrays of structures



COBOL



Example 1: Declare a CURSOR C1 and fetch 10 rows using a multi-row FETCH statement

01 OUTPUT-VARS.05 NAME OCCURS 10 TIMES.

49 NAME-LEN PIC S9(4)COMP-4 SY C.49 NAME-DATA PIC X(40).

05 SERIAL-NUMBER PIC S9(9)COMP-4 OCCURS 10 TIMES.

PROCEDURE DIVISION.

EXEC SQLDECLARE C1 CURSOR WITH ROWSET POSITIONING FORSELECT NAME, SERIAL# FROM CORPORATE.EMPLOYEE END-EXEC.

EXEC SQLOPEN C1 END-EXEC.

EXEC SQLFETCH FIRST ROWSET FROM C1 FOR 10 ROWS INTO :NAME,

:SERIAL-NUMBER END-EXEC.

COBOL



PL/I



PL/1

Example 2: You can retrieve 10 rows from the table CORPDATA.DEPARTMENT with:

DCL DEPTNO(10) CHAR(3),DEPTNAME(10) CHAR(29) VAR,MGRNO(10) CHAR(6),ADMRDEPT(10) CHAR(3);

DCL IND_ARRAY1(10)BIN FIXED(15);DCL IND_ARRAY2(10)BIN FIXED(15);...EXEC SQL

DECLARE C1 CURSOR WITH ROWSET POSITIONING FORSELECT * FROM CORPDATA.DEPARTMENT;

...EXEC SQL

FETCH FIRST ROWSET FROM C1 FOR 10 ROWS INTO:DEPTNO :IND_ARRAY1,:DEPTNAME :IND_ARRAY2,:MGRNO :IND_ARRAY3,:ADMRDEPT :IND_ARRAY4;



C/C++



C++

Example 3: Declare an integer and varying character array to holdcolumns retrieved from a multi-row fetch statement

long serial_num(10);struct {

short len;char data [18];}name [10];

...EXEC SQL

DECLARE C1 CURSOR FOR SELECT NAME, SERIAL#FROM CORPDATA.EMPLOYEE WITH ROWSET POSITIONING;

...EXEC SQL OPEN C1;EXEC SQL

FETCH FIRST ROWSET FROM C1 FOR 10 ROWS INTO :NAME,:SERIAL_NUM;



ODBC#define TC 10;

SQLWCHAR SAVE_H1WCHR[TC][1025];

SQLINTEGER LNSAVE_H1WCHR[TC];

/* Main Program */

int main()

{

/* initialize data */

wcscpy(SAVE_H1WCHR[0], (SQLWCHAR *)"abc 1");

...

wcscpy(SAVE_H1WCHR[9], (SQLWCHAR *)“abc 10");

hstmt=0;

rc=SQLAllocStmt(hdbc, &hstmt);

rc=SQLBindParameter(hstmt, 1, SQL_PARAM_INPUT, SQL_C_WCHAR, SQL_DBCLOB

1024, 0, SAVE_H1WCHR, (SQLINTEGER) 2050,

(SQLINTEGER *) LNSAVE_H1WCHR);

/* Set number of rows to insert */

rc=SQLParamOptions(hstmt, TC, NULL);

/* Insert rows into DBCLOB column via SQLBindParameter */

rc=SQLPrepareW(hstmt,(wchar_t *)

"INSERT INTO TABLECU (C1) VALUES (?)",SQL_NTS);

rc=SQLExecute(hstmt);

rc=SQLTransact(henv, hdbc, SQL_COMMIT);

} /* End Main */



JDBCtry {

stmt = con.prepareStatement ("insert into T1 values (?, ?)");stmt.setInt (1, 1);stmt.setInt (2, 1);stmt.addBatch();stmt.setInt (1, 2);stmt.setInt (2, 2);stmt.addBatch();stmt.setInt (1, 3);stmt.setInt (2, 3);stmt.addBatch();

int[] updateCount = stmt.executeBatch();

for (int i = 0; i < updateCount.length; i++)actualResults.println(UpdateCount[" + i + "] = " + updateCount[i]);

stmt.clearBatch();con.commit();stmt.close();

}



Multiple-Row Insert



Different Forms of INSERTƒINSERT via VALUES is used to insert a single row into the table or

view using values provided or referencedƒINSERT via SELECT is used to insert one or more rows into table or

view using values from other tables or viewsƒINSERT via VALUES… FOR "n" ROWS form is used to insert multiple

rows into table or view using values provided in host variable array

FOR "n" ROWSƒFor static, specify FOR "n" ROWS on INSERT statement

(for dynamic INSERT, you may also specify FOR "n" ROWS on EXECUTE statement)

ƒMaximum value of n is 32767 specified as host-variable, parameter marker, or literal value

ƒInput provided with literal, host variable, or host variable array --each array represents cells for multiple rows of a single column

VALUES… FOR “n” ROWS clause allows specification of multiple rows of

data ƒHost variable arrays used to provide values for a column on INSERT ƒExample: VALUES (:hva1, :hva2) FOR 10 ROWS

Multiple Row INSERT



Multiple-Row Insert

Changed StatementsƒINSERTƒPREPAREƒEXECUTE



INSERT



INSERT (cont)



INSERT Example 1

Insert a variable number of rows using host variable arrays for columnvalues. Assume that the table T1 has one column and that a variable(:hv) number of rows of data are to be inserted into T1 table.

EXEC SQL INSERT INTO T1VALUES (:hva :hvind) FOR :hv ROWSATOMIC;

In this example, :hva represents the host variable array and :hvind represents the array of indicator variables



INSERT Example 2

Insert 10 rows into a table T2, and return the rows that have been inserted using a multiple row fetch statement.

DECLARE CS1 ASENSITIVE SCROLL CURSOR WITH RETURN WITH ROWSET POSITIONING FOR SELECT T2.C1, T2.C2 FROM FINAL TABLE

(INSERT INTO T2 VALUES (:hvai1 :hvindi1, :hva2 :hvinid2) FOR 10 ROWS);

EXEC SQL OPEN CS1; /* INSERT OCCURS HERE */EXEC SQL FETCH FIRST ROWSET FROM CS1

FOR 10 ROWS INTO :hvao1 :hvindo1, :hvao2 :hvindo2



Assume that table T3 has two columns: C1 (SMALLINT) and C2(INTEGER)The application allocates two arrays, :hva1 with 5 elements, and :hva2 with 10 elements

:hva1 :hva2

INSERT INTO T3 (C1, C2) VALUES (:hva1, :hva2) FOR 5 ROWS;

INSERT Example 3



Assume that table T3 has three columns: S1 SMALLINT, I1 INTEGER, and M1 CHAR(8)The application allocates one array, :hva2 with 10 elements. :hv1 is a scalar host variable and the value for M1 is a special register In this example, hv1, and CURRENT TIME are used for each row of data, the values for C2 are obtained from host-variable-array :hva2

:hv1 :hva2

INSERT INTO T3 VALUES (:hv1, :hva2,CURRENT TIME) FOR 10 ROWS -- Assume CURRENT TIME = ’ 13:30:05’

INSERT Example 4

T1I1S1

13:30:0510513:30:059513:30:058513:30:057513:30:056513:30:055513:30:054513:30:053513:30:052513:30:0515

5 12345678910

Result of Insert



Local INSERT Flow

1900170013001204

0000

1900170013001204 … …

SQLDAID = SQLDASQLDABC = 104SQLN = 2SQLD = 2SQLTYPE[1] = 496 SQLTYPE[2] = 496SQLLEN[1] = 4 SQLLEN[2] = 4SQLDATA[1] = x00001000 SQLDATA[2] = x00003000SQLIND[1] = x00002000 SQLIND[2] = x00000000SQLNAME[1] = x‘0008000000010004’ SQLNAME[2] =x‘0008000000020000’

x1000 x2000 Internal Buffers

4

Table

4

x3000



Distributed Multiple Row INSERT Flow

ApplicationC1 C2

Client Com-Buffer

IntermediateBuffer

Table Page

ServerCom-Buffer

Distributed Chained INSERT FlowApplicationStmt Data

Client Com-Buffer

IntermediateBuffer

Table Page

ServerCom-Buffer



ATOMIC -vs- NOT ATOMIC

ATOMICƒTraditional behaviorƒAll rows being inserted must successfully be inserted. If the insert for any row fails, all changes made to database by that INSERT statement are undone.

NOT ATOMIC CONTINUE ON SQLEXCEPTIONƒInsert rows that are successfulƒReject rows that are not successful

GET DIAGNOSTICS can be used to determine which rows were not successful

ƒSQLCODE will indicate if all failed, all were successful or at least one failed



NOT ATOMIC CONTINUE ON SQLEXCEPTION – GET DIAGNOSTICS

NOT ATOMIC CONTINUE ON SQLEXCEPTION– Diagnostics are available for each failed row through

GET DIAGNOSTICS

– SQLCODE indicates if:• All failed

– SQLSTATE 22530, SQLCODE -254• All were successful, but warnings

– SQLSTATE 01659, SQLCODE +252• At least one failed

– SQLSTATE 22529, SQLCODE -253



ATOMIC -vs- NOT ATOMIC with Triggers

•Trigger Behavior on Multiple Row Insert•ATOMIC

•The inserts are processed as a single statement. •Any statement level triggers fire once for the statement, and the transition tables will include all of the rows that were inserted.

•NOT ATOMIC CONTINUE ON SQLEXCEPTION•Inserts are processed separately. •Any statement level triggers are processed for each row that is inserted•Transition tables include the individual row that is inserted. •When errors are encountered with this option in effect, processing continues, and some of the specified rows will not beinserted.

•In this case, if an insert trigger is defined on the underlying base table, the trigger transition table will only include rows that were successfully inserted.



PREPARE



EXECUTE



PREPARE and EXECUTE - Example

Assume that the prog table has 9 columns. Prepare and execute a dynamicINSERT statement which inserts 5 rows of data into the prog table.

stmt = 'INSERT INTO prog (iwhid, updated_by, update_ts, name, short_description,orderNo, parmData, parmDataLong, VWProgKey)

VALUES (?,?,?,?,?,?,?,?,?)';

attrvar = ’FOR MULTIPLE ROWS ’;NROWS = 5;

EXEC SQL PREPARE ins_stmt ATTRIBUTES :attrvar FROM :stmt;

EXEC SQL EXECUTE ins_stmt FOR :NROWS ROWSUSING :V1,:V2,:V3,:V4,:V5,:V6,:V7,:V8,:V9;

In this example, each host variable in the USING clause represents an array of values for the corresponding column of the target of the INSERT statement.



SQLDA must contain a valid description of the host variable arrays or buffers which contain the values to be insertedƒEach SQLVAR describes a host variable or host variable

array which represents a buffer which contains value(s) for a column of target table

ƒSQLDA must have enough storage to contain SQLVAR for each target column for which values are provided, plus an additional SQLVAR entry for use by DB2 for z/OS

ƒPrior to the multi-row insert, the SQLDA fields must be set correctly to include number of SQLVAR occurrences, number of variables used, pointer to arrays, indicator variables etc.

INSERT SQLDA Considerations



MRI Performance ConsiderationsUp to 30% faster INSERT performance

Performance improvement largely due to savings of API costsSavings flattens out quickly, for example, savings (as a percentage) was equal • For 100 Rows, 500 Rows, 1000 Rows…• Reasonable “n” for MRI is about 200 – no additional savings above that, and downside (rollback)

increases.

Distributed MRI performance up to 70% Elapsed Time and 50% Server CPU time reductions seenPerformance variable based on • Number of rows INSERTed• Number of columns INSERTed• Number of INDEXes on table• Class 2 accounting (on or off) – savings larger is Class 2 is on

Note: Don’t use MRI to INSERT 1 row due to overhead to set up for MRI

Similar improvement with UPDATE and DELETE WHERE CURRENT OF when updating/deleting the entire rowset. This is in addition to the savings provided by MRF



Multiple-Row Fetch



Multiple-Row Fetch

Changed StatementsƒDECLARE CURSORƒOPEN CURSORƒALLOCATE CURSORƒFETCHƒPositioned UPDATE (UWCO)ƒPositioned DELETE (DWCO)



DECLARE CURSOR



DECLARE CURSOR - Example

Declare C1 as the cursor of a query to retrieve a rowset from the table DEPT.

The prepared statement is MYCURSOR

EXEC SQL DECLARE CURSOR C1 CURSORWITH ROWSET POSITIONINGFOR MYCURSOR;

Rowset positioning specifies whether multiple rows of data can be accessed as a rowset on a single FETCH statement –default is WITHOUT ROWSET POSITIONING



FETCH



FETCH (cont)



Rowsets

A group of rows for the result table of a query which are returned by a single FETCH statement (or inserted by a single (multi-row) INSERT statement)

Program controls how many rows are returned (i.e., size of the rowset)ƒCan be specified on the FETCH statement (maximum rowset size is

32767)

Each group of rows are operated on as a rowset

Ability to intermix row positioned (single row) and rowset (multiple row) positioned fetches when a cursor is declared WITH ROWSET POSITIONING

FETCH FIRST ROWSET STARTING AT ABSOLUTE 10 FROM CURS1

FOR 6 ROWS INTO :hva1, :hva2;



123456789

10 XXXX XXXX XXXX XXXX

11121314151617181920

Fetch Orientation and Cursor Position

Result table

FETCH ABSOLUTE 5

FETCH BEFORE

FETCH FIRST

FETCH RELATIVE - 3

FETCH PRIOR

FETCH RELATIVE 3

FETCH ABSOLUTE - 5

FETCH LAST

FETCH AFTER

FETCH or FETCH NEXT

FETCH CURRENT

Current Cursor Position

Independent ofstart cursor position

Ending position depends on startcursor position

Independent ofstart cursor position



Rowset Positioned Fetches

CUST_NO CUST_TYP CUST_NAME

1 P Ian2 P Mark3 P John4 P Karen5 P Sarah6 M Florence7 M Dylan8 M Bert9 M Jo

10 R Karen11 R Gary12 R Bill13 R Geoff14 R Julia15 R Sally

FETCH FIRST ROWSET FOR 3 ROWS

FETCH NEXT ROWSET

FETCH ROWSET STARTING AT ABSOLUTE 8

FETCH NEXT ROWSET FOR 10 ROWSPartial Result Set

Result table



Partial Rowsets (Result Sets)



Cursor Positioning: Mixing Row and Rowset Positioned Fetches - Multi Row Fetch (V8)



Determining rowset size

If FOR n ROWS is NOT specified and cursor is declared for rowset positioning..

Size of rowset will be the same as the previous rowset fetch as long as ƒIt was the previous fetch for this cursorƒOr the previous fetch was a FETCH BEFORE or FETCH AFTER and the fetch

before that was a rowset fetch

Else rowset is 1

FETCH FIRST ROWSET FOR 5 ROWS FETCH NEXT ROWSETFETCH NEXTFETCH NEXT ROWSETFETCH NEXT ROWSET FOR 3 ROWSFETCH BEFORE FETCH NEXT ROWSET

Returns 5 rowsReturns the next 5 rowsReturns a single rowReturns a single rowReturns 3 rowsReturns 0 rowsReturns 3 rows



FETCH examples

EXAMPLE 1:Fetch the previous rowset and have the cursor positioned on that rowset

EXEC SQL FETCH PRIOR ROWSET FROM C1 FOR 3 ROWS INTO...

-- OR --EXEC SQL

FETCH ROWSET STARTING AT RELATIVE -3 FROM C1 FOR 3 ROWS INTO...

EXAMPLE 2:Fetch 3 rows starting with row 20 regardless of the current position of the cursor

EXEC SQL FETCH ROWSET STARTING AT ABSOLUTE 20

FROM C1 FOR 3 ROWS INTO...



Partial Result Sets

If you fetch beyond the end of the result set, you will receive an end of data conditionƒi.e., When there are only 5 rows left in result table and you

request FETCH NEXT ROWSET FOR 10 ROWS, 5 rows will be returned - SQLCODE +100

ƒSQLERRD(3) will contain the number or rows returnedƒThis includes where FETCH FIRST n ROWS ONLY has been

specified

If you fetch beyond the beginning of the result set, you will receive an end of data conditionƒ i.e., if you are positioned on rows 3,4,5,6, and 7, and you

request FETCH PRIOR ROWSET FOR 10 ROWS, 2 rows will be returned (Rows 1 and 2) - SQLCODE +20237

ƒSQLERRD(3) will contain the number or rows returned



Fetching Beyond the Result Set ABSOLUTE or RELATIVE

If you fetch beyond the end of the result set, or beyond the beginning of the result set, you will receive an end of data conditionƒAssume you are positioned on row 5 in a result set with 10

rows. ƒFETCH ROWSET STARTING AT ABSOLUTE 15ƒFETCH ROWSET STARTING AT RELATIVE -7

ƒNo rows will be returned - SQLCODE +100ƒSQLERRD(3) will contain 0ƒCursor position will be either “BEFORE” or “AFTER” depending

on the direction of the FETCH.



Resulting Cursor Positions

Fetch Orientation

Current Position

before first row

Current Position on

first row

Current Position on

last row

Current Position After

Last row

Resulting Position on

Delete or Update Hole

Resulting Position on normal Row

Resulting Position Before

First row

Resulting Position After

Last Row

NEXT OK OK +100 +100 +222 IF OK N/A IF +100 FROM LAST ROW

PRIOR +100 +100 OK OK +222 IF OK IF+100 FROM LAST ROW

N/A

FIRST OK OK OK OK +222 IF OK N/A N/A

LAST OK OK OK OK N/A IFOK N/A N/A

BEFORE OK OK OK OK N/A N/A IF OK N/A

AFTER OK OK OK OK +222 N/A N/A IF OK

CURRENTRELATIVE 0

+231 OK OK +231 +222 IF OK N/A N/A

ABSOLUTE +N

OK OK OK OK +222 IF OK N/A IF +100 AND N OUT OF RANGE

ABSOLUTE -N

OK OK OK OK +222 IF OK IF +100 AND N OUT OF RANGE

N/A

RELATIVE +N

OK OK +100 +100 +222 IF OK N/A IF +100 AND N OUT OF RANGE

RELATIVE -N +100 +100 OK OK +222 IF OK IF +100 AND N OUT OF

RANGE

N/A



Distributed Multiple Row Fetch Flow

ApplicationC1 C2

Client Com-Buffer

IntermediateBuffer

Table Page

ServerCom-Buffer

DiagnosticInfo

SQLCA DiagnosticInfo

DiagnosticInfo

Limited Block Fetch FlowApplicationC1 C2

Client Com-Buffer

IntermediateBuffer

Table Page

ServerCom-Buffer

DiagnosticInfo Diag Info Diag Info

SQLCA

SQLCA

SQLCA



Local FETCH Flow

1900170013001204

0000

1900170013001204 … …

SQLDAID = SQLDASQLDABC = 104SQLN = 2SQLD = 2SQLTYPE[1] = 496SQLLEN[1] = 4 SQLDATA[1] = x00001000SQLIND[1] = x00002000SQLNAME[1] = x‘0008000000010004’

x1000 x2000 Internal Buffers ‘N’ Table

4



MRF Performance Considerations

Up to 50% faster FETCH performance Performance improvement largely due to savings of API costsPerformance variable based on • Number of rows fetched• Number of columns fetched• Class 2 accounting (on or off) – savings larger is Class 2 is on

Examples– DSNTEP4– 35% improvement on FETCH of 1000 rows with 5 cols and 20 cols– DSNTIAUL– 50% improvement on FETCH of 10000 rows with 5 cols and 20 cols

Up to 50% reduction in CPU cost for LBF –vs- V7



Locking and Isolation Levels

Cursor will be positioned on all rows in current rowset

Locks will be held on all rows in rowset depending on isolation level and whether a result table has been materialized

As normal, these factors affect whether you will see changes made to the table following open cursor

Also affects whether you refetch the same rows when issuing a FETCH CURRENT to refetch current rowset



Using Static Scrollable Cursors

When scrolling between rowsets:

With insensitive fetches, updates by the application itself could cause changes and holes

With sensitive fetches, updates by other applications could cause changes and holes

For example, FETCH PRIOR ROWSET may return update or delete holes in place of rows that were fetched before

Row contents can change between fetches



If using static scrollable cursors...

CUSTNO NULLABLE_COL IND_VAR CUST_TYPE

1000 M P 2000 -1 B

-34000 F P

As holes may occur, ensure at least one indicator variable is defined for a row ƒIf no nullable columns exist add an indicator variable for at least one

columnƒIf multiple nullable columns exist each indicator variable will be updated if a

hole is found

New value of -3 indicates hole

SQLCODE +222 will also be returned (via GET DIAGNOSTICS)



Locks will be held on base table for entire rowset with ISO CS, RR and RS

Starting point and contents of rowsets may change when scrollingback and forth

Refetching current rowset may return different rowsƒIf ISO(RR), other applications cannot affect your rowset ƒYour application can still affect your rowset

FETCH PRIOR ROWSET will return the previous n rows that qualify from the start of the current cursor positionƒTherefore n rows will be returned as long as start of rowset is not reached

Considerations with Dynamic Scrollable Cursors



Multiple Row FETCH/INSERT --DRDA Considerations

For remote client one rowset returned in a network request – this is in effect the blocksize

Updateable cursors will still be blocked.

Supported by any requester or server that supports DRDA Level 3

DB2 for z/OS V8DB2 LUW V8 FP 4 ODBC client

Limit of 10 megabyte buffer for distributed MRF/MRI



Positioned DELETE

Positioned Update



Considerations for positioned update/delete

It is possible for another application process to update or delete a row in the base table of the SELECT statement so that the specified row of the cursor no longer has a corresponding row in the base table

–If the cursor is non-scrollable, this could happen if the cursor is ISOLATION(CS) CURRENTDATA(NO) and lock avoidance succeeds

–If the cursor is static scrollable, this could happen, since the result set is materialized into a temporary table and all underlying locks are released (unless ISOLATION(RR) or ISOLATION(RS)

–If the cursor is dynamic scrollable, this could happen if the cursor is ISOLATION(CS) CURRENTDATA(NO) and lock avoidance succeeds.



Summary

Host Variable ArraysMultiple-Row InsertMultiple-Row Fetch

DELETE WHERE CURRENT OFUPDATE WHERE CURRENT OF

faster fetch

Documents