db2 lobying at arcelormittal gent davy goethals volvo gentdb2 lobying .pdf · db2 lobying at...

DB2 LOBying at ArcelorMittal Gent Davy Goethals

GSE DB2 working group 21/03/2013 Volvo Gent

22/03/2013 1

Agenda

• LOB table space architecture in DB2 V9 • Our first LOB applications with Net.Data as browser facility • More recent applications with dotNET • Populating LOB tables • Unload/Load LOB tables • Online REORG LOB table spaces • Altering tables with LOB data • Future area of importance : pure XML applications

Large objects (LOB’s) in DB2

• Before DB2 V6 : max character field was 32 Kb – SBCS character (EBCDIC or ASCII) – BIT bit format

• Starting with DB2 V6 : support for large objects till 2 Gb

– CLOB character (EBCDIC, ASCII or UNICODE) – BLOB bit format – can be used to store text documents, images, audio and

video objects

LOB support in DB2 : architecture

key col col2 col3 rowid lob colkey akey b

data adata b

data adata b

rowid arowid b

--

Base table space

LOB table space

base table

auxiliary table

auxiliary index

rowid lob datarowid arowid b

lob data alob data b

LOB support in DB2 : LOB indicator

• ROWID column :

– Used to link the LOB column in the base table with the actual LOB data in the auxiliary table via the auxiliary index

– can be explicitly defined (one is enough) or implicitly added by DB2

• Lob indicator : 6 bytes field in the base table – 2 bytes length – 2 bytes flags :

• Null flag : if the LOB has the null value • Zero length flag : if the LOB has a zero length • Valid flag : if the LOB has been marked invalid by CHECK DATA • Other internal flags

– 2 bytes current version • Changed at update of the LOB • Used to check consistency

LOB support in DB2 : DDL

CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10) , VERSION VARCHAR(30) , ……………………………………. ROW_ID ROWID NOT NULL GENERATED ALWAYS , IMAGE BLOB(100M) ) IN DBIMAGE . TSIMAGE

CREATE LOB TABLESPACE IMAGELOB IN DBIMAGE USING STOGROUP SYSDEFLT .. BUFFERPOOL BP1 LOG NO LOCKSIZE LOB DSSIZE 8 G

CREATE AUXILIARY TABLE A.IMAGES_AUX IN DBIMAGE. IMAGELOB STORES A. IMAGES COLUMN IMAGE ( PART . … )

CREATE UNIQUE INDEX A.AUX_IX

ON A.IMAGES_AUX

USING STOGROUP SYSDEFLT …...

LOB support in DB2 : LOB table space

• One LOB table space per LOB column and partition – max 4096 partitions in base table – Each LOB table space can span 254 data sets – Each data set has a DSSIZE of 4,8,16,32 or 64 Gb – 4096*254*64 Gb = 65536 Tb

• No compress • Bufferpool 4,8,16,32 K 4k recommended

– Max 1 LOB per datapage (reduce waste of space) – Dedicated bufferpool ?

• LOG YES or NO (always NO if lobsize > 1 Gb) – consistency always maintained after rollback – but considerations for RECOVER utility and LPL and GBP recovery

• LOCKSIZE TABLESPACE or LOB • GBPCACHE CHANGED (recomm.) or ALL or NONE or SYSTEM

LOB applications in DB2

• How to visualise LOB data ? – Not possible with 3270 applications

– Need some kind of client server application

– Thin client (browser) or fat client ?

22/03/2013 7

Net.Data for z/OS and OS/390

• WHY ? – Good support for DB2 lobs and for free – Browser oriented and 2 tier – Could use web server on mainframe – No need to involve other platform software & specialists – Good for developing pilot and POC

• BUT – Need HTML skills and knowledge of USS environment


• Net.Data was a multiplatform product • on z/OS : part of DB2 (supported till DB2 V8) • follow-on of DB2 W W W (DB2 world wide web connection) • could be used to create simple dynamic Web pages • developer had to create scripts called Web macros • could be executed by the IBM HTTP server for z/OS by

specifying an URL containing the name of the Web macro • returned a dynamically built Web page to the browser


IBM HTTP server

System external program

%HTML( initial page)

%DEFINE

browser

z/OS Net.Data

REXX rexx program

SQL db2 data

own other

language environments

URL

%HTML( report)

%FUNCTION HTML pages

Net.Data : invoked by URL

http://server/netdata-cgi/db2www/macro/block [?var1=value1&var2=value2…….]

macro request

http://server/netdata-cgi/db2www/?LANGENV=DTW_SQL &DB2SSID=DB2P&SQL=select+*+from+sysibm.systables

direct request

AM Gent: Net.Data in the 24/7 sysplex

HTTP

net.data

HTTP

HTTP

net.data

HTTP

DB2 DB2 DB2 DB2

sysplex distr

tcp/ip

network partitions

DB2 partitions

coupling facility

WLM

CGI CGI

browser

HTML

dynamic web page

z/OS host

2 tier architecture

AM Gent : lob applications overview

file

ftp

host appl

.TIF scanner

net.data

z/OS host

DB2 tables

key col col2 col3 rowid lob coldocid adocid b

meta ameta b

meta ameta b

rowid arowid b

Blob aBlob b

search engine based on SQL meta data columns

scanned doc stored in blob tif format

type 1


batch appl net.data

z/OS host

DB2 tables


meta ameta b

meta ameta b

rowid arowid b

Blob aBlob b


reports stored in clob html format expired reports kept in DB2 archive tables (ML2)

factory planning Reports in HTML Initially stored on HFS directory (DFS)

type 2

DATA STORE of HTML reports


invoice appl net.data

z/OS host

DB2 tables


meta ameta b

meta ameta b

rowid arowid b

Blob aBlob b

search engine based on SQL on invoice tables

DJDE invoices converted to pdf and stored in blob column (Elixir) expired invoices kept on ML2

1,500,000 invoices/year

type 2

arcelor intranet

Now replaced by SAP application

AM Gent : more recent lob applications

ZFS file

DFS

FRV

Certificates

pdf

.DB2 Connect

z/OS host

DB2 tables


meta ameta b

meta ameta b

rowid arowid b

Blob aBlob b


Pdf files

type 1

dotNET applications

Populating LOB data in DB2 for z/OS

DB2 table

file

z/OS host

client

DRDA

DFS

FTP client appl

host appl

key col col2 col3 rowid lob col key a key b

data a data b

data a data b

rowid a rowid b

- -

LOBs

LOB support in DB2 : SQL

DCL BLOB_VAR SQL TYPE IS BLOB (32000) DCL CLOB_VAR SQL TYPE IS CLOB(64M)

DCL BLOB_LOC SQL TYPE IS BLOB_LOCATOR DCL CLOB_LOC SQL TYPE IS CLOB_LOCATOR

LOB variable

LOB locator


DCL BLOB_FRV SQL TYPE IS BLOB _FILE DCL CLOB_FRV SQL TYPE IS CLOB(_FILE

LOB file reference variable


• LOB host variable : – used to store a complete LOB in a variable of the application

program – Only used for small LOBs

• LOB locator : – contains reference to a LOB object for manipulating LOB

data without moving it to the application program – avoid materialization of the complete LOB in the (client)

program – avoid memory allocation and data movement and improve

performance – Used when manipulating the content of the lob data in the

program


• LOB file reference variable : – New DB2 V9 – used to copy a complete LOB to/from a file – Sequential file, pds,pdse, open-edition file (HFS/ZFS) – No manipulation/processing of the content of the lob in the

program – Used for loading/unloading LOB data to HFS

• Can also be used by LOAD/UNLOAD utility

– New DB2 V9 but retrofitted to V7/V8 – Load/unload each lob value to a separate file – Before only Load/unload when SLOB < 32K

Lob support in DB2 • LOAD/UNLOAD utility: restrictions before FRV support

– Only SLOBs allowed

• Always unload/load of base table (not auxiliary table) • Lob columns treated as normal data columns • Normal data and lob data in same input/output record with a

limit of 32K (MVS limit for sequential data set) • Not sufficient in most cases !

– Cross loading (INCURSOR) is now also possible to load

data from another DB2 table

Unload/Load Lobs using file ref variables

• Load/Unload data from/to separate file instead of the normal unload file (with max 32K record length)

• Member of a pds or pdse • Hfs file

• Store file name in the normal unload file as char or varchar field • 1 separate file per lob value

Unload Lobs using file ref variables

• Implemented through : – New keywords in the field specification for char and varchar

(file reference variables) – BLOBF template name – CLOBF template name – DBCLOBF template name

– New keywords in TEMPLATE definition : – DIR – DSNTYPE – (&UNIQ) or /&UNIQ. automatically appended

Unload Lobs using file ref variables

CREATE TABLE A. IMAGES

(IMAGE_ID VARCHAR(30) NOT NULL,

IMAGE_TYPE VARCHAR(10) ,

VERSION VARCHAR(30) ,

…………………………………….

ROW_ID ROWID NOT NULL

GENERATED ALWAYS ,

IMAGE BLOB(100M) )

IN IMAGES.IMAGES

TEMPLATE TSYSPUN DSN('WRK.PUNCH')

DISP(MOD,CATLG,CATLG)

TEMPLATE TSYSREC DSN('WRK.SYSREC')

DISP(MOD,CATLG,CATLG)

TEMPLATE TSYSLOB DSN(/u/davy)

DSNTYPE(HFS)

UNLOAD DATA FROM TABLE A. IMAGES

IMAGE_ID, IMAGE_TYPE, ……

IMAGE VARCHAR BLOBF TSYSLOB )

UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN)

Files /u/davy/ AYPPXJRD

/u/davy/ AYPPXJ3H

/u/davy/AYPPXJ6K

…….

Load Lobs using file ref variables

CREATE TABLE A. DOCUMENTS

(DOC_ID VARCHAR(30) NOT NULL,

DOC_TYPE VARCHAR(10) ,

VERSION VARCHAR(30) ,

…………………………………….

ROW_ID ROWID NOT NULL

GENERATED ALWAYS ,

DOCUMENT CLOB(100M) )

IN DOCUMENTS.DOCUMENTS

TEMPLATE TSYSREC DSN('WRK.SYSREC')

DISP(OLD,KEEP,KEEP)

LOAD DATA INDDN TSYSREC LOG NO REPLACE

INTO TABLE A.DOCUMENTS

(DOC_ID POSITION( 1 : 31) VARCHAR ,

…………

DOCUMENT POSITION( 122:378) VARCHAR CLOBF

NULLIF(00121)=X'FF' )

Xload Lob support

• Cross load : feature of the LOAD utility – input is not a sequential file but the result set of a dynamic

SQL query – source data in query are local or remote DB2 table(s) – source can be on any drda server or the result of Websphere

Information Integrator • Replaces unload, file transfer, load

Xload Lob enhancements PQ90263

EXEC SQL

DECLARE c1 CURSOR FOR

select query

ENDEXEC

LOAD DATA

INCURSOR c1

INTO TABLE creator.table

Using V9 DSNTIAUL to unload lobs

• V9 UNLOAD utility : – unload lob fields to PDS(E) or HFS directory – seq. files not supported (performance)

• V9 DSNTIAUL sample program :

– Enhanced to unload lob fields to seq files – PARMS(‘SQL’) : unload lobs as normal data (with truncation if

record > 32K ) – PARMS(‘LOBFILE(prefix)’) : unload lobs to seq files

Online Reorg of LOB table spaces

• V6,V7,V8 REORG : – Only improved prefetch performance of lob data

• Rebuild lob table space in place (no unload-reload) • No reclaim of free space • No delete/define cluster : RECOVER utility or UNLOAD/RELOAD was

needed to change priqty & secqty allocated values and number of extents

• Lot of software problems

– V9 REORG SHRLEVEL NONE (default)


• DB2 9 REORG SHRLEVEL REFERENCE: – Uses shadow datasets – PRIQTY,SECQTY changes are honored – Read access allowed (except switch phase) – Free space is reclaimed – Always LOG NO – Inline image copy mandatory – Primary datasets available after failure


• When to REORG a LOB table space ? – ORGRATIO < 10 – REORGDISLOB/TOTALROWS > 10%

• When to REORG an auxiliary index ?

– LEAFFAR/NLEAF > 10% – PSEUDO_DEL_ENTRIES/NLEAF > 10%

• Monitor FREESPACE also


• Conclusions : – We always use REORG SHRLEVEL REFERENCE in DB2 9

• Better availability (no R/W access ) • Allows free space reclaim • Easier to handle in case of failure • Plan for enough DASD

Altering tables with LOB data

• TABLE changes that require DROP/RECREATE – Column changes that require drop/create table are

painful when the table contains lob columns : • Must also recreate all auxiliary tables and indexes • How to unload/load the lob data ?

– Same for ddl changes to the base table space that require a recreate of the table space

Altering tables with LOB data

• In DB2 V9 2 techniques possible – Use temporary shadow table :

• Create identical shadow table • Copy lob data using insert • Repopulate after drop/create with insert from shadow table

– Use unload/reload with FRV or third-party tool with equivalent

functionality • Only used when dasd space issue on DB2 volumes

DB2 V9 : pure XML

• Support for XML in the relational database :

• XML view of relational data – Document composition (publishing) and shredding

• Performance is a nightmare • Difficult to maintain schema changes

• Store XML documents in a DB2 field ; cfr XML extender

– Monolitical storage (Clob or Blob) ; indexing through side tables.

– Limited query possibilities

DB2 V9 : pure XML

• XML as a new datatype – Efficient storage in parsed format and indexing of XML data – Schema validation through XML schemas (.xsd) (optional) – Strong query possiblities ; Xpath and SQL integration

DB2 V9 : pure XML

• Use of pure XML is linked to LOB’s : – XML table spaces are similar to LOB table spaces

– XMLPARSE : parse and insert CLOB or BLOB into a XML

column – XMLSERIALIZE : convert XML type to CLOB type (“textual

XML”)

DB2 V9 : pure XML • SELECT XMLSERIALIZE ( XMLELEMENT(NAME "NAME", A.NAME)

AS CLOB) AS “RESULT” FROM Q.STAFF A ; “TEXTUAL XML”

RESULT <NAME>SANDERS</NAME> <NAME>PERNAL</NAME> <NAME>MARENGHI</NAME> <NAME>O’BRIEN</NAME>

DB2 V9 : pure XML • SELECT XMLSERIALIZE(XMLELEMENT(NAME "NAME", XMLATTRIBUTES(A.ID AS "ID"), A.NAME) AS CLOB) AS "RESULT" FROM Q.STAFF A ;

RESULT <NAME ID="10“ >SANDERS</NAME> <NAME ID=“20“ >PERNAL</NAME> <NAME ID=“30“ >MARENGHI</NAME> <NAME ID=“40“ >O’BRIEN</NAME>

22/03/2013 44

Questions ?

• Email :

[email protected]

db2 lobying at arcelormittal gent davy goethals volvo gentdb2 lobying .pdf · db2 lobying at...

Documents