data replication implementation
TRANSCRIPT
What is New in IBM Replication Solution
Richard SinnIBM Santa Teresa LabAS/400 Tech Conference 2000
IBM Santa Teresa Lab
Presentation Goals Learn introduction materials to replication Learn IBM’s solution and where to order Understand the concept of Capture/Apply Pickup useful details of Capture/Apply Quick Overview of Advanced Features and
Future Enhancement
IBM DB2 Data Propagator
Product ID: – 5769DP2 for V4R1+– 5769DPX for future release (X is a number)
Product Home Page at: http://www.software.ibm.com/data/dpropr
You could order online OR try it free for 70 days in future release
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features Future Enhancement Conclusion
What is Replication Movement of Data from a
database source system to a target database
Copy only the "Changes" vs Copy "the whole thing"
Redefining Application Development Model
Build Data Warehouse Making Data Available
Replication Requirements
Copy - aCopy - bPrimary - c
Copy - aPrimary - bCopy - c
Primary
Copy Copy
Copy
Primary Primary
Primary Primary
Primary
Data dissemination Data consolidation
Replication of distinct fragments Update anywhere
Primary - aCopy - bCopy - c
Why Replication ? Making Data Available Use Information
Intelligently and improve decision making
Stay competitive with both Data access and Data movement
Speedup application deployment
Reengineer business processes
Increase online throughput
Improve system availability
Support audit requirements
Support Data Warehousing
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features Future Enhancement Conclusion
IBM Data Propagator Overview
Journal
ADMINISTRATION (e.g DJRA)
COPY
CONTROL
COPYCOPY
Base TablesColumn SelectionAfter Image or Before & After image
APPLY
CONTROL
CAPTURE
BASE
Source/Operational System
Target
IBM Data Propagator Overview Capture works on "Source
System" Apply works on "Target
System" To tell Capture what to
work on: Registration To tell Apply what to work
on: Subscription Registrations/
Subscriptions are created by DJRA
IBM Data Propagator Overview DPR runs on multiple
platforms (AIX, NT, AS/400, 95, HP-UNIX, OS/2, etc)
All control info is SQL based
Published Architecture: works with NotesPump, etc
Well established technology
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features Future Enhancement Conclusion
Overview: Admin with DJRA
Overview: Administration
Administration: Registration
Select AS/400 database Select table Select necessary
column only Pick capture policy Pick update capture
policy Pick conflict detection
level
Overview: Capture
Journal
ADMINISTRATION
COPY
CONTROL
COPYCOPY
Base TablesColumn SelectionAfter Image or Before & After image
APPLY
UNIT OF WORK
CHANGE DATA
CONTROL
CAPTURE
BASE
Operational System
Overview: Capture DJRA creates
registration information in the control tables associated with Capture/400
"Changes" are actually put into "Change Data" table
When there is a commit, a row will be put into "Unit of Work" table
Captures base table changes from Journal
Timestamps changes Maintains transaction
consistency Automatically maintains
staging tables Pruning of CD tables
are automatically as well
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features Future Enhancement Conclusion
Overview: Subscription Set
Overview: Subscription Member
Overview: Subscription
Subscription is bounded by set
Set can have multiple members
Event based, or Time based
Blocking factor Can select necessary
columns only
Overview: Apply
Journal
Operational System CONTROL
APPLY
BASE
UNIT OF WORK
CHANGE DATA
CONTROL
CAPTURE
ADMINISTRATION
PIT/User CopyHISTORY
STAGINGREPLICA
Base and Copy TablesInterval and RepetitionColumn and Row SelectionComputed ColumnsAggregationsAppend or Replace
Target
Overview: Apply
DJRA creates control information associated with Apply/400
According to the control info, Apply– Runs at user-specified
intervals or events– Refreshes, updates, and
enhances copies– Do distribution
optimizations
Apply picks up "changes" from the "Change Data" table and "Unit of Work" table
Apply communicates with Capture via Syn point
Apply is a big SQL application
Apply can be "Push" or "Pull"
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features Future Enhancement Conclusion
Useful Internal: Capture
Journal
CONTROL
CAPTURE JOURNALJOB
CHANGE DATA CHANGE DATACHANGE DATAUNIT OF WORKCHANGE DATA CHANGE DATACHANGE DATA
BASEBASE
CAPTURE JOURNALJOB
Journal
BASEBASE
CAPTURE CONTROLJOB
Capture: Control Job Control Job - QDPRCTL5
– Analyze the user options and new registration– Check replication source validity– Sort out all selected replication sources by journals– Determine eligibility– Determine journal job starting time– Start up journal jobs– Prune - CD tables, UOW and AUTHTKN– Handle messages (from commands and journal jobs)
Capture: Journal Jobs Journal Job(s) - Top Program
QZSNCAP2– Issue RCVJRNE command
• Use time to specify which journal entries to process
• Give a list of up to 300 file/members• Give a list of 24+ specific journal entry types• Specify a delay of 60 seconds
– Handles messages from exit program
Capture: Exit Program Journal Job(s) - Exit Program QZSNCAP3
– Receive and analyze changes from the journal– Advance SYNCHPOINT in PRUNCNTL table on a set
basis when necessary– Maintain CD_OLD_SYNCHPOINT and
CD_NEW_SYNCHPOINT in register table– Maintain the global row in register table– Support application commitment control– Handle special journal types (end journal, renamed, etc)– Cancel handling (Warm start table), etc
Capture: Performance Tuning
Latency is 4 to 5 minutes
CPU overhead is about 6 to 8%
Ways to tune performance:– Job priority– Num of replication
sources per journal job– COMMIT_INTERVAL in
CCPPARMS
Apply: Job Information STRDPRAPY APYQUAL(ABC) TRACE(*ALL)
Work with Subsystem Jobs STL400F 01/24/00 15:30:03 Subsystem . . . . . . . . . . : QZSNDPR Opt Job User Type -----Status----- Function ABC SINN BATCH MSGW PGM-QZSNAPV2 KSCHIA KSCHIA BATCH ACTIVE PGM-QZSNAPV2 QDPRCTL5 KSCHIA BATCH MSGW PGM-QZSNCAP1 QSQJRN KSCHIA BATCH ACTIVE PGM-QZSNCV72
Apply: Trace Information
STRDPRAPY APYQUAL(ABC) TRACE(*ALL)
Work with Job Spooled Files Job: ABC User: SINN Number: 048177
Device or Total Current Opt File Queue User Data Status Pages Page Copies QPZSNATRC PRT01 ABC OPN 1818 1
Apply: Apply Trail Table
SQL format trail info is also available STRSQL
select * from asn/ibmsnap_applytrail
APPLY_QUAL SET_NAME WHOS_ON_FIRST ASNLOAD MASS_DELETE EFFECTIVE_MEMBERS STL400F STL400F S N Y 1 STL400F STL400F S N N 0 STL400F STL400F S N N 0 STL400F STL400F S N N 0 STL400F STL400F S N N 1 STL400F STL400F S N N 0 STL400F STL400F S N N 0 STL400F STL400F S N N 1
Apply: Apply Trail Table
Some of the available information are:
APPLY_QUAL, SET_NAME, ASNLOAD, MASS_DELETE, EFFECTIVE_MEMBERS, SET_INSERTED, SET_DELETED, SET_UPDATED, SET_REWORKED, STATUS, LASTRUN, LASTSUCCESS, SOURCE_SERVER, SOURCE_TABLE, SOURCE_VIEW_QUAL, TARGET_SERVER, TARGET_TABLE, SQLSTATE, SQLCODE, SQLERRP, SQLERRM
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features Future Enhancement Conclusion
Advance Replication Features AS/400 Relative Record Number (RRN) Support Remote Journal Replication Support Automated Deletion of Journal Receivers Performance Improvement to Apply/400 Programming with ASNLOAD and ASNDONE Set Subscriptions Update Anywhere IBM Replication Solution Extendibility
RRN Replication Support
Journal
CAPTURE
SOURCE TARGET
COPY
RRN("base-owner".base-table")
TARGET
COPY
Normal differential processing (contents of IBM_QSQRRN)
Differential Refresh
APPLY
UNIT OF WORK
CHANGE DATA
Full Refresh
Base
COL1 COL2 COL3 COL4
I RRN etc.D RRN etc.
APPLY
RRN Replication Support
Allow user to have a unique key for CCD and Point-in-time target tables when no combination of target columns is unique (no SQL key)
DJRA support administration of RRN replication on AS/400
Remote Journal Replication
Remote Journal can be setup using AS/400 commands (V4R3+)
Capture and Apply run on the same system
Free up resource on operational (source) system
Better performance (no network overhead, etc)
Remote Journal Replication
Operational System Target System
Journal
BASE
Remote Journal
CONTROL
CAPTURE
UNIT OF WORK
CHANGE DATA
CONTROL
COPY
COPYCOPY
APPLY
Reg. Ext.
Automated Deletion of JR ReceiversSource System
Journal
CONTROL
CAPTURE
Journal Receivers QZSNDREP
exit program Detached Receiver
Delete Receiver
Keep Receiver
Yes Vote
No Vote
Automated Deletion of JR Receivers Exit program registered at
installation time (QIBM_QJO_DLT_JRNRCV)
DLTJRNRCV calls the exit program before deleting JR receivers
If Capture needs the receiver, it votes no on delete
When receivers are managed by system, it can then be deleted automatically
Performance Improvements to Apply Blocked reading and writing to work files Use of OS/400 API QSQPRCED to perform blocked
insert into local AS/400 target tables during full refresh
Use of CLRPFM (Clear Physical File Member) CL command to clear local AS/400 target tables
Use of Static SQL (instead of Dynamic SQL) when possible
"Direct insert" in remote journal replication
Programming with Data Propagator ASNDONE exit routine is a program that Apply can
optionally call after subscription processing completes, regardless of success or failure
Divided by Apply qualifier Other done logic can be added
Language Source File NameSubscription notify programmember name
Full refresh program member name
C QDPR/QCSRC ASNDONE ASNLOADCOBOL QDPR/QCBLLESRC ASNDONE ASNLOAD
RPG QDPR/QRPGLESRC ASNDONE ASNLOAD
Programming with Data Propagator ASNLOAD full-refresh exit routine is called by Apply
– when a full refresh of a target table is necessary OR – if you specify the name of a full refresh program on the
FULLREFPGM parameter when you start Apply Can be used to do "offline, non-SQL" full refresh Other pre-load logic can also be added ASNDONE/ASNLOAD can be named anything,
located in any library ASNDONE/ASNLOAD can be coded in C, COBOL,
RPG, etc.
Features: Set Subscriptions
Journal
SOURCE CONTROL
BASE
UNIT OF WORK
CHANGE DATA
CONTROL
CAPTURE
TARGET
ADMINISTRATION
APPLY
SET 1, TBL 1
SET 1, TBL 2
SET 1, TBL 3SET 2, TBL 1
SET 2, TBL 2
Features: Set Subscriptions
Maintains referential constraints over replication
Replicates tables within a set in a single unit of work
Coordinates the components of a view subscription
Limits the boundary for cascade rejections due to RI violation or update collision
Features: Update Anywhere
USERTABLE
CDCA
PTU
RE
CD
REPLICA
CA
PTU
RE
AP P
LY
CD
REPLICA
CA
PTU
RE
APP
LY
CD
REPLICA
CA
PTU
RE
APP
LY
Collision detection levelsNoneStandardEnhanced
Reference site
Features: Update Anywhere
Hierarchical Architecture
Three levels of conflict detection
Automatic transaction compensation
Compensation of dependent transactions
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features: Extendibility Future Enhancement Conclusion
Extendibility: DataJoiner + Data PropagatorOS/2
AIX
DOS
Windows
HP-UX
Solaris
Transparency
Full SQL
Global OptimizationSingle-DBMS
Image
DB2 for VSE & VMor DB2 for AS/400
DB2 for OS/2DB2 for AIX
Oracle
Sybase
MicrosoftSQL Server
Informix
DB2 Common Server
NT IPCS
DataJoiner
DB2 for MVS,
DB2 for PE
other Relationalor Non-Relational
DBMS
DB2 for NT
DB2 for HP-UXor DB2 for Solaris
VSAM
IMS
APPLY
Extendibility: DataJoiner + Data Propagator
GLOBALCATALOG
COPY
Journal
TargetCONTROL
APPLY
BASE
UNIT OF WORK
CHANGE DATA
CONTROL
CAPTURE
ADMINISTRATION
DataJoiner
Oracle SybaseInformixSQL Server
Operational System
DB2
Extendibility: DataJoiner + Data Propagator
COPY
Target
BASE
STAGING
Triggers
CONTROL
APPLY
ADMINISTRATION
GLOBALCATALOG
DataJoiner
Oracle SybaseInformixMicrosoft SQL Server
Operational System
DB2
Extendibility: Lotus Notes, IMS, Flat Files
Enterprise Data Propagation to NotesDB2 Family
Source Data
DPropR Staging Tables
Updates
Refresh
Trigger
Multi-vendor
DataRefresher
Notes DBNotesPump
IMS
IMS, VSAM, flat files
DataPropagatorRelational
DataPropagatorNon-relational
Updates
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features: Extendibility Future Enhancement Conclusion
Future Enhancement Enhanced Remote Journal Replication
– Administration– Performance
CD Row Changed Only Support– Generate CD Row when there is change in the
captured column– DASD vs CPU
LOB replication Support– Easy management of LOB object across the network
Future Enhancement Delivered as Keyed Stamped Media
– Every new OS/400 customers will have a copy– Try and Buy– Manual available online
BIG INTEGER Replication Support Support for greater than 18 characters object DataLinks Replication
– Integrated with your database and file system– Customer programming for replication
Agenda What is Replication IBM Data Propagator Overview
– Capture Side– Apply Side
AS/400 Data Propagator Useful Internal Advance Replication Features: Extendibility Future Enhancement Conclusion
Conclusion: IBM Data Propagator Solution Versatile Infrastructure
– Operational application– Informational application– Mobile & occasionally connected systems– Update anywhere with conflict detection
and automatic compensation Leverages Current Assets
– Legacy Sources– Standard SQL
Efficient & Automated Operations– Journal/Log-based capture for DB2– Trigger-based capture for non-DB2
sources – Network optimizations
Scaleable Design– Servers– Networks
Flexible Distribution– Subsetting with join views– Fan-out distribution
Integrated Administration– DB2 UDB Control Center GUI for
administration for V5– CL commands for pre-V5 function– Automated initialization– Tailorable
Robust Data Enhancements– Derivation, Summarization, Translation...
Multi-vendor Interoperability– SQL based architecture– Architected data staging area– Replication among DB2, Oracle, Sybase,
SQL Server, Informix, IMS, IDMS and VSAM
Conclusion: IBM Data Propagator Solution
DB2 Family
LegacyData
Source Data
Staging Tables
Copies
DataPropagatorRelational
DataRefresher
Capture DB2
Capture IMS
DB2 Family orMulti-vendor
RDBMS
Multi-vendorRDBMS
Lotus Notes
Capture Triggers
NotesPump
DataPropagatorRelationalApply DB2
DataPropagatorNonRelational
Refresh for IMS,VSAM, flat files
DataJoiner Multi-vendor
Access
MobileComputing
Apply for Notes or ODBC
Reference
http://www.software.ibm.com/data/dpropr http://www.software.ibm.com/data/datajoiner http://www.software.ibm.com/data http://as400bks.rochester.ibm.com http://www.as400.ibm.com
THANKS MUCH FOR COMING !!!
Trademark Information AIX, AS/400, DataHub, DataJoiner, DataPropagator, DataRefresher, DB2, DProp,
DRDA, IBM, IMS, MVS, OS/2, OS/400, RS/6000, VM/ESA and VSE/ESA are trademarks or registered trademarks of the IBM Corpopration in the United States or other countries or both.
Microsoft, Windows, and the Windows 95, and the Windows 98, and the Windows NT logos are trademarks or registered trademarks of Microsoft Corporation.
UNIX is a registered trademark in the United States and other countries licensed exclusively through X/Open Company Limited.
Lotus Notes is a trademark of the Lotus Development Corporation.
Other company, product, and service names may be trademarks or service marks of
others.