102145823-datastage-dmirror

Using WebSphere DataStage with IBM DataMirrorChange Data CaptureSet up a working environment for Information ServerDataMirrorCDC operator

Skill Level: Intermediate

Indrani Ghatare ([email protected])Software EngineerIBM

15 May 2008

Do you need real-time access to data residing in diverse data sources? Learn how touse the DataMirror® Change Data Capture (CDC) stagetype operator of theDataMirror Transformation Server to capture streaming data changes intoWebSphere® DataStage®.This tutorial includes a step-by-step guide to help you setup your DataMirror environment. You'll learn how to generate a DataStage-specificimport file for using the operator in the DataStage job. Finally, learn how to use theoperator in some example jobs. This tutorial includes a sample DSX file.

Section 1. Before you start

This integration solution is suitable in a business environment where data copies arekept in different databases, and all the databases must be available to accessreal-time data at any point in time. The integration technology is suitable for keepingtwo data copies synced up in heterogeneous databases with different schema.DataMirror's Change Data Capture (CDC) technology show data changes thatoccurred in databases and also minimizes the resources required for maintainingETL data. This gives you access to real-time data for your target databases.

Using WebSphere DataStage with IBM DataMirror Change Data Capture Trademarks© Copyright IBM Corporation 2008. All rights reserved. of 42

mailto:[email protected]

http://www.ibm.com/developerworks/ibm/trademarks/

http://www.ibm.com/legal/copytrade.shtml

About this tutorial

This tutorial teaches you how to use Information Server DataMirrorCDC stagetypeoperator in IBM DataMirror environment. The tutorial provides step-by-stepinstruction to setup the DataMirror environment. It shows you how to set upDataStage DataMirrorCDC operator in a DataStage job. The tutorial includes sampledata and DataStage job definition file.

Objectives

In this tutorial, you will:

• Set up the DataMirror environment

• Generate DataStage job definition file

• Prepare the DataStage job using DataMirrorCDC operator and sampledata

• Run a DataStage job using sample data and example job

Prerequisites

This tutorial is written for Windows® users whose skills and experience are at anintermediate level. You should have a solid understanding of Information ServerDataStage and a working knowledge of DB2® and DataMirror.

System requirements

To use this tutorial, you need to have these installed on the same Windows box:

• Information Server 8.0.1 FP1

• Information Server patch for DataMirrorCDC operator (available oneService)

• IBM DataMirror Transformation Server for UDB

• IBM DataMirror Transformation Server for WebSphere DataStage

• IBM DataMirror Transformation Server Access Manager

• IBM DataMirror Transformation Server Management Console

developerWorks® ibm.com/developerWorks


https://www-01.ibm.com/software/data/support/iis/eservice/login.do;jsessionid=8EBEBC82FBF2D0EFD6FD8BEB72A933F3



Before beginning, you should also do the following:

• Read the document about DataMirrorCDC stagetype operator, refer to theDataMirrorCDCReadme-a4.pdf file from Download section.

• Download the datamirrorTutorial.zip file from the Download section. Thiszip file contains the sample data and job definition file that are used in thistutorial. Extract the zip file in an empty directory.

Section 2. Overview

The DataMirrorCDC stagetype operator in Information Server allows you to utilizethe CDC technology of IBM DataMirror. The scenario describes how DataMirrorCDCoperator allows you to integrate data between a remote source database and atarget database using DataMirror's CDC technology by streaming data changes intoInformation Server DataStage.

Figure 1 shows the integration scenario:

Figure 1. Integration scenario

The scenario follows these steps:

1. DataStage extracts data from source database using standard ETLfunctions

2. DataMirrorCDC operator requests the changed data from DataMirror

ibm.com/developerWorks developerWorks®




3. DataMirror captures changes made to the source database

4. DataMirror sends the captured changes to DataMirrorCDC operator

5. DataMirrorCDC operator passes the data off to downstream stages

6. Updates are written to the target database

In the next section, you'll set up the DataMirror environment needed to implementthe above steps.

Section 3. Set up the DataMirror environment

Follow the steps described in this section to set up your DataMirror environment.

Task 1: Set up the database

To set up the database used in this tutorial:

1. Create database BANKDATA by running the DB2 command:

• db2 create database BANKDATA

2. Enable log retention by running the DB2 commands:

• db2 update db config for BANKDATA using LOGRETAINrecovery

• db2 backup database BANKDATA to c:\backupBANKDATA is the name of the database that you want to enable forreplication, and C:\backup is the directory where BANKDATA is backedup. Make sure that C:\backup exists in the system before running thecommand.

3. If your CLASSPATH does not have db2java.zip, add it to yourCLASSPATH.

4. Create source table BANK.BANKCUSTOMERS using data.db2 found inthe datamirrorTutorial.zip file (see Downloads).

Task 2: Add an instance of Transformation Server for UDB





To add an instance of Transformation Server for UDB, bring up TransformationServer Configuration Tool by clicking Start -> All Programs -> DataMirror ->Transformation Server for UDB -> Configure Transformation Server.

1. Click Add on the Transformation Server Configuration Tool window tocreate a new instance as Figure 2 shows.Figure 2. Add a new instance for UDB

2. Check Local System account for Windows Service. Enter theinformation for the database. Click Apply and then Close. Figure 3 showsthose steps.Figure 3. Enter information for the database





3. Select the new_instance and click Start.Figure 4. Select new_instance





4. The instance new_instance should be in running status now. Click Close.Figure 5. Make sure new_instance is running in UDB





Task 3: Add an instance of Transformation Server for WebSphere DataStage

To add an instance of Transformation Server for WebSphere DataStage, bring upTransformation Server Configuration Tool by clicking Start -> All Programs ->DataMirror -> Transformation Server for WebSphere DataStage -> ConfigureTransformation Server.

1. Click Add on Transformation Server Configuration Tool window to createa new instance.Figure 6. Add a new instance for WebSphere DataStage





2. Check Local System account for Windows Service. Set the passwordfor tsuser in the Transformation Server Authentication box. Click Applyand then Close.Figure 7. Set password





3. Select new_instance and click Start.Figure 8. Start the new instance





4. The instance new_instance should be in running status now. Click Close.Figure 9. Make sure new_instance for Datastage is running

Task 4: Create and configure replication agents and users in Access Manager

1. Click Start -> All Programs -> DataMirror -> Transformation Server





Access Control -> Access Manager.

2. In the Access Manager window, click File -> New -> Replication Agent.Figure 10. Create a new replication agent

3. To create a replication agent for source, go to the General tab of theReplication Agent Properties window:

• In the Name field, type srcagent (or any other unique name)

• In the Description field, type a description for the replication agent(optional)

• In the Hostname field, type the host name or ip address of where theTransformation Server is running

• In the Port field, type a port number to allow the replication agent tocommunicate with Transformation Server. This port number isspecified during the Transformation Server configuration.

Figure 11. Fill in the values for the replication agent





4. Under the Version tab in the Replication Agent Properties window, clickPing Now. If the ping is successful, it will obtain the correct versioninformation of Transformation Server. Click OK. If the ping isunsuccessful, check if the information in the General tab is correct or ifthere is any network issue.Figure 12. Click ping now





5. Similarly, create another replication agent for the target. Under theGeneral tab in the Replication Agent Properties window:

• In the Name field, type tgtagent (or any other unique name)

• In the Description field, type a description for the target agent(optional)

• In the Hostname field, type the host name or ip address where thereplication agent is running

• In the Port field, type a port number that allows the replication agentto communicate with the Transformation Server for DataStage. Thisport number is specified during the Transformation Server forDataStage configuration.

Figure 13. Create a target replication agent





6. Under the Version tab in the Replication Agent Properties window, clickPing Now. If the ping is successful, it will obtain the correct versioninformation of Transformation Server for DataStage. Click OK. If the pingis unsuccessful, check if the information in the General tab is correct or ifthere is any network issue.Figure 14. Click Ping Now





7. In the Access Manager window, click File -> New -> Replication User tocreate the Transformation Server System Administrator.Figure 15. Create the Transformation Server System Administrator





8. Under the General Tab in the User Properties window:

• In the Username field, type username

• In the Description field, type a description (optional).

• In the Password field, create and confirm a password.Figure 16. Create a password for the system administrator





9. Under the Replication Agents table in the User Properties window, clickAdd/Delete..Figure 17. Click the Add/Delete button





Select

both srcagent and tgtagent and click Add to move them to the "Allowedto access" window.Figure 18. Permit access to the srcagent and tgtagent





Click OK. Your screen should look like Figure 19.Figure 19. Access to srcagent and tgtagent

Click OK. You should see both srcagent and tgtagent under theReplication Agents list.Figure 20. srcagent and tgtagent are now replication agents





Select

srcagent and click Parameters. Fill in the information for srcagent thatthe Access Parameters window requests. Click OK.Figure 21. Parameter information for srcagent





Select tgtagent and click Parameters. Fill in the information for tgtagentthat the Access Parameter window requests. Click OK.Figure 22. Parameter information for tgtagent





10. In Options tab, uncheck User must change password at next logon.Figure 23. Disable request to change password





Click OK

in the User Properties window. Close the Access Manager.

Task 5: Setup subscription in the Transformation Server Management Console

1. Click Start -> All Programs -> DataMirror -> Transformation ServerManagement Console -> Management Console

2. Type the user name and password to login to the Management Console.Figure 24. Login to the Management Console





3. In the Management Console, click on File -> Datastore -> Add orRemove Tables...Figure 25. Add tables





4. Select the source table BANKCUSTOMERS from the Add or RemoveTables wizard. Click OK.Figure 26. Add BANKCUSTOMERS





5. Create a new subscription:

• From the menu, click Subscription -> New Subscription.

• In the New subscription window, provide a name for the subscriptionand select replication agents for Datastores. Click OK.Figure 27. Create a new subscription





6. Configure subscription:

• From the menu, click theConfiguration tab and then theSubscription tab. Right-click on the subscription that you havecreated and click on Map Tables.Figure 28. Map tables





• From Map Tables, check WebSphere DataStage mapping type. ClickNext.Figure 29. Map to WebSphere DataStage





• Check Direct Connect as WebSphere DataStage ConnectionMethod. Click Next.Figure 30. Select Direct Connect





• Select the source table. Click Next.Figure 31. Select a source table





• For Connection, enter a port number that is not used by any otherapplication. Under Record Format, check Single Record. Click Next.Figure 32. Select Single Record





• Verify that Review mapping settings has "Before and After images willbe in a single record." Click Finish.Figure 33. Ensure all images will be in a single record





Task 6: Set Access Server parameters

To set the access server parameters:

1. Run C:\Program Files\DataMirror\Transformation Server for WebSphereDataStage\bin\dmsetaccessserverparams.exe and provide the values

Figure 34. Set the access server parameters





Section 4. Generate DataStage job definition file

1. From the Management Console, right-click on the subscription, and clickon Generate WebSphere DataStage Job Definition...Figure 35. Generate the DataStage job definition





Save the .dsx file. The generated job definition fileDATASTAGE_SUB.dsx is attached with this tutorial for your reference.See downloads.

Section 5. Prepare and run DataStage job

Import the DataStage job definition file

To import the DataStage job definition file (.dsx) that you have created in theprevious section, from WebSphere DataStage Designer:

• Click Import -> DataStage Components.

• Browse and select the DataStage job definition file that you have createdin the previous section

• After the import is complete, a job is created under the Jobs ->DataMirror folder in the DataStage designer. In this example, jobDATASTAGE_SUB_BANKCUSTOMERS is created.





• You can open the job by double-clicking Jobs->DataMirror->DATASTAGE_SUB_BANKCUSTOMERS

Figure 36. Import the DataStage job definition file

Verify and set properties for the DataMirrorCDC stage

To verify and set properties for DataMirrorCDC stage:

• Right-click on DatamirrorCDC_Input stage on canvas.

• Click on Properties.

• On the Stage tab, change the value for dsautostartTS to True. Click OK.Figure 37. Set properties for the DataMirror CDC stage





Set properties for Sequential_File_Output stage

To set the Target File property value as the output file where incoming data will bewritten to:

• Right-click Sequential_File_Output stage and click on Properties.

• From the Input tab, provide a file name for the target file. Click OK.Figure 38. Set properties for the Sequential_File_Output stage

Run the DataStage job DATASTAGE_SUB_BANKCUSTOMERS

1. You can insert, update or delete any data in the source table.





2. Compile the job by clicking File -> Compile.

3. Once compilation is successful, run the job by clicking File -> Run.

4. Once the job has run successfully, the target file that you specified in thesequential file stage contains the data changes based on insert, update ordelete operation done on source table.

Run an example job

Once the above job runs successfully, the DataMirrorCDC operator can be used withother stages in DataStage. To run the example job:

1. Import the Example_Job.dsx file in DataStage designer. TheExample_job.dsx is contained in the dataMirrorTutorial.zip file (see theDownload section).

2. Set job properties for $APT_DBNAME and $APT_DB2_INSTANCEHOMEby clicking Edit -> Job Properties -> Parameters.

3. Create a target database BANKCUSTOMERS_TARGET with tabledefinition of BANKCUSTOMERS source table. This example job usesDEMO.BANKCUSTOMERS as the target table, if you use a differenttarget database, change the target database stage properties valuesappropriately.Figure 39. Run an example job





Section 6. Conclusion

Now that you have gone through this tutorial, you can configure the DataMirrorproduct to capture changed data using DataMirror. You have learned how to transferthe data changes captured to WebSphere DataStage for cleansing andtransforming, ready for saving in a target database. Try this technology using yourinstallations of DataMirror and DataStage, and keep your databases in sync.





Downloads

Description Name Size Downloadmethod

Tutorial sample data datamirrorTutorial.zip 17KB HTTP

About DataMirrorCDC operator DataMirrorCDCReadme-a4.pdf 164KB HTTP

Information about download methods



http://www.ibm.com/developerworks/apps/download/index.jsp?contentid=308179&filename=datamirrorTutorial.zip&method=http&locale=

http://www.ibm.com/developerworks/apps/download/index.jsp?contentid=308179&filename=DataMirrorCDCReadme-a4.pdf&method=http&locale=

http://www.ibm.com/developerworks/library/whichmethod.html



Resources

Learn

• In the Information Integration area on developerWorks, get the resources youneed to advance your skills on IBM Information Platform and Solution products.

• Browse the technology bookstore for books on these and other technical topics.

Get products and technologies

• Download IBM product evaluation versions and get your hands on applicationdevelopment tools and middleware products from DB2®, Lotus®, Rational®,Tivoli®, and WebSphere®.

Discuss

• Participate in the discussion forum for this content.

• Check out developerWorks blogs and get involved in the developerWorkscommunity.

About the author

Indrani GhatareIndrani Ghatare is a software engineer in IBM Silicon Valley Lab, USA.Indrani has been working as a software developer in IBM softwaregroup since 2001. Currently, she works in the Information ServerIntegration team and provides integration solutions for IBM productswithin IPS portfolio.



http://www.ibm.com/developerworks/db2/products/integration

http://www.ibm.com/developerworks/apps/SendTo?bookstore=safari

http://www.ibm.com/developerworks/downloads/?S_TACT=105AGX01&S_CMP=ART

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=825

http://www.ibm.com/developerworks/blogs/

http://www.ibm.com/developerworks/community

http://www.ibm.com/developerworks/community



102145823-datastage-dmirror

Documents