datamigrator 7.7 in real time
DESCRIPTION
DataMigrator 7.7 in Real Time. Chris Bevilacqua iWay Solutions Architect. Real Time ETL with DataMigator Agenda. Web Services Change Data Capture File Listener IWAF Adapters. Web Services. What are Web Services for?. What is a Web Service?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/1.jpg)
DataMigrator 7.7 in Real Time
Chris BevilacquaiWay Solutions Architect
![Page 2: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/2.jpg)
Real Time ETL with DataMigatorAgenda
Web ServicesChange Data CaptureFile ListenerIWAF Adapters
![Page 3: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/3.jpg)
Web Services
What are Web Services for?
![Page 4: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/4.jpg)
What is a Web Service?
A software system designed to support interoperable machine-to-machine interaction over a network.
It has an interface described in a machine-processable format (specifically Web Services Description Language WSDL).
– World Wide Web Consortium
![Page 5: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/5.jpg)
.net
Publicly Available Commercial Web Services
Some sources of Public Web ServicesFree and commercial
![Page 6: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/6.jpg)
WSDLL.COMWeb Services – Free and Commercial
![Page 7: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/7.jpg)
Xmethods.net List of public web services
![Page 8: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/8.jpg)
Seekda.com Find WSDL for PhoneVerify Service from cdyne.com
![Page 9: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/9.jpg)
Add adapter for Web Services
![Page 10: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/10.jpg)
Add Connection for a Web Service
![Page 11: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/11.jpg)
Add Connection for Web Service
![Page 12: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/12.jpg)
Create Synonym
![Page 13: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/13.jpg)
Open Synonym
![Page 14: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/14.jpg)
Sample DataSupply values for PHONENUMBER and LICENSEKEY
![Page 15: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/15.jpg)
Sample DataShow all fields returned by Web Service
![Page 16: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/16.jpg)
Create Data FlowAdd input file
![Page 17: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/17.jpg)
User Defined FunctionCleans up phone number
![Page 18: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/18.jpg)
Add transformation Clean up phone number and assign name
![Page 19: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/19.jpg)
Add synonym for Web Service
![Page 20: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/20.jpg)
Join input file to Web ServiceOn PHONENUMBER and LICENSEKEY
![Page 21: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/21.jpg)
Select Columns from Web Service
![Page 22: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/22.jpg)
Test SQLShows “join” from input file to Web Service
![Page 23: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/23.jpg)
Change Data Capture
When would you want to capture changes?
![Page 24: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/24.jpg)
Why Change Data Capture?
Large source databases, small batch windowWant to only copy the changes Want near real-time updates
Need some way to detect what has changed…
![Page 25: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/25.jpg)
Change Data CaptureHow to detect changes to database tables
Change the application? Track timestamp of changes, flag deletes Write changes to a separate table
Compare source to target? Effective for small tables
Poll the database for changes periodically? Detects additions only
Use database triggers? Developer adds to database…. or automatically generated Requires staging table
![Page 26: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/26.jpg)
Change Data CaptureData Adapter to read from database logs
Read the database log… Proprietary formats Every database is different
New Data Adapter for database log tables No actual staging table required Synonym describes a view of database log for one table Provides automatic checkpoint processing
![Page 27: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/27.jpg)
Change Data CaptureDatabase configuration
Platforms and databases supported ORACLE 10g and11g on UNIX and Windows UDB on UNIX and Windows DB2 on IBM zSeries and iSeries MS SQL Server 2008
DBA issues database specific commands to Configure database to use archiving or enable logging Add logging or enable capture mode
Create additional synonym for table log records
![Page 28: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/28.jpg)
Create Synonym For Table Log Records
![Page 29: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/29.jpg)
Create Synonym for Table Log Records
![Page 30: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/30.jpg)
Synonym created for Log TableThree CDC columns added
![Page 31: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/31.jpg)
Synonym for Log TableLayout same as source table… plus three columns
CDC_OPER – Operation Type I = Insert U = Update D = Delete
CDC_TID – Transaction ID CDC_TIMES – Timestamp
![Page 32: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/32.jpg)
Create Synonym for Log TableCDC Configuration
Polling intervalTimeoutCheckpoint processing
![Page 33: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/33.jpg)
CDCConfiguration
POLLING – Log polling interval for LUW
TIMEOUT – Timeout interval polling for LUW
START – CHKPT after last checkpoint saved
CHECKPT_SAVE [YES|NO] – save last checkpoint in file
CHECKPT_FILE – Location and name of file
MAX_LUWS – Maximum number of LUWs processed
![Page 34: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/34.jpg)
UDBSynonym for Log Table
![Page 35: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/35.jpg)
Data FlowLog table as source
![Page 36: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/36.jpg)
Column Selection
![Page 37: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/37.jpg)
Target Transformations
![Page 38: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/38.jpg)
Enabled for IUD ProcessingIUD control column identified
![Page 39: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/39.jpg)
UDB CDCStart with Source and Target the same
Run program to update source
![Page 40: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/40.jpg)
CDC Flow Log shows rows inserted / updated / deleted
![Page 41: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/41.jpg)
Change Data CaptureSource and Target now have same rows
![Page 42: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/42.jpg)
Change Data CaptureReprocessing
For testing or reprocessing…Start at specified timestamp
![Page 43: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/43.jpg)
File Listener
Process files as they arrive in a directory… posted or copied or program output
![Page 44: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/44.jpg)
Refactoring
7.6 Listener based Required creating and managing Listener Only Direct Load flows Only fixed length files, only character data Flow never completed… could not view log
could not send email or run process flow Could not schedule flows
7.7 Adapter based All specifications in synonym, simpler to set up Either SQL or Direct Load flows Any file type, any data type Flow completes when file or time limit reached…
Can send email or run flow or procedure on completion Can schedule flows as needed
![Page 45: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/45.jpg)
File ListenerCreate Sample Data
![Page 46: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/46.jpg)
File Listener ParametersSynonym - dmordfl
![Page 47: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/47.jpg)
Data Allocation Parameters
Connection – Name of a connection for an FTP Server
Data Origin FILE – one time read LISTENER – process files as they arrive TAIL – future development
Identify File(s) Directory – Application Directory to poll for files Name – file name, wildcards allowed Extension – file type or extension
![Page 48: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/48.jpg)
Listening Parameters and Read Limits
Listening Polling – interval in seconds Timeout – time in seconds. If no new files arrive end job
Read Limits Maxfiles – Maximum number of data files Maxrecs – Maximum number of records (only for TAIL)
![Page 49: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/49.jpg)
Pre and Post Processing
Pre Processing - Pickup IMMEDIATE – As soon as files arrives TRIGGER – Wait for trigger file with
EXTENSION
Post Processing - DiscardDELETE ARCHIVE – Copy files to
DIRECTORY KEEP – Delete trigger only and keep file
![Page 50: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/50.jpg)
File Name Field with Alias of INSTANCE
![Page 51: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/51.jpg)
SQL or Direct Load Flow… looks like any other flow
![Page 52: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/52.jpg)
File ListenerCreate directories listen and archive
![Page 53: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/53.jpg)
File ListenerSubmit the Flow – Waits for TIMEOUT interval
![Page 54: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/54.jpg)
File ListenerDrop the Files
![Page 55: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/55.jpg)
File ListenerDrop the Files - within timeout interval (60 seconds)
![Page 56: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/56.jpg)
File ListenerSample Data
![Page 57: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/57.jpg)
Log shows number of rows processedFor all files
![Page 58: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/58.jpg)
IWAF - iWay Adapter Framework
![Page 59: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/59.jpg)
iWay Adapter FrameworkApplication Adapters
Data Adapters for some ERP systems New support for Application Adapters including
Salesforce.com MySAP Seibel Peoplesoft
So that DataMigrator can use Services Events
![Page 60: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/60.jpg)
Configure IWAF Adapter
![Page 61: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/61.jpg)
Add IWAF Adapter
![Page 62: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/62.jpg)
Select IWAF Adapter
![Page 63: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/63.jpg)
Add Connection for IWAFSalesforce Service
![Page 64: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/64.jpg)
Salesforce AdapterCreate Synonym
![Page 65: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/65.jpg)
Salesforce AdapterCreate Synonym
![Page 66: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/66.jpg)
Salesforce AdapterCreate Synonym
![Page 67: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/67.jpg)
Create Synonym for Salesforce Service
![Page 68: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/68.jpg)
Salesforce ServiceSample Data
![Page 69: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/69.jpg)
Salesforce EventSynonym
![Page 70: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/70.jpg)
SalesforceData Flow
![Page 71: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/71.jpg)
Salesforce.com
![Page 72: DataMigrator 7.7 in Real Time](https://reader035.vdocuments.site/reader035/viewer/2022062301/568158e4550346895dc62794/html5/thumbnails/72.jpg)
Salesforce Output table and log