integrating oracle data integrator with oracle goldengate 12c

[email protected] www.rittmanmead.com @rittmanmead !

ODI and GoldenGate 12c: Beyond Data Integration

Edelweiss Kammermann OOW 2014 September 28th

1!


Edelweiss Kammermann

•  Computer engineer, BI and BPM specialist with over 18 years of consulting and project management experience in Oracle technology.

•  Co-founder and Vice President of Uruguayan Oracle User Group (UYOUG)

•  Business Intelligence articles published on OTN site

•  Frequent speaker at international conferences as Collaborate, OTN Tour LA, UKOUG Annual Conference and Rittman Mead BI Forum

•  Oracle ACE

•  Training Materials Specialist in Rittman Mead

email: [email protected] Twitter: @EdelweissK


Uruguay


Rittman Mead •  Oracle BI and DW Gold partner with offices in

USA (Atlanta), Europa, Australia and India •  Winner of five UKOUG Partner of the Year

awards 2013 including BI •  World leading specialist partner for technical

excellence, solutions delivery and innovation in Oracle BI

•  100+ consultants worldwide included 1 Oracle ACE Director, 3 Oracle ACEs and 1 Oracle ACE Associate

•  Skills in broad range of supporting Oracle tools: OBIEE,OBIA, ODI, Essbase, Oracle OLAP, GoldenGate, Exadata, Exalytics, Endeca and BigData

•  Blog : http://www.rittmanmead.com/blog/

•  Twitter : @rittmanmead


Agenda

•  What is Oracle Data Integrator? •  What is Oracle GoldenGate?

•  How the integration between ODI and OGG works?

•  Benefits of this integration

•  ODI and OGG integration steps

•  Conclusions


Oracle Data Integrator ODI is a complete and unified data integration solution for data transformation and integration processes. •  Provides reusability, flexibility and extensibility using Knowledge Modules (KM) •  Part of the Oracle Fusion Middleware 12c Stack •  Platform Independent •  Separates the logical from the physical implementation •  Based in an ELT architecture to leverage target platform capabilities: higher

performance. •  Combines data movement, data synchronization, data transformation, data quality,

data management and data services. •  Supports batch-based, real-time and service-based integration


ETL vs ELT


ETL vs ELT

• Reduce network transfer • Reduce hardware costs • Set-based operations

• Leverage target platform capabilities • Referential Integrity Check


ODI Principal Components

•  ODI Studio •  Agents - the runtime components

•  Repositories

•  Work Repository

•  Master Repository

•  Management Pack for Oracle Data Integrator •  ODI Console


Master and Work Repositories

•  ODI systems connect to a single Master Repository and one or more Work Repositories

•  A Work Repository can be used for:

•  Development

•  Execution (Execution Repository)

•  Development work repositories are full-function and can be developed in

•  Execution work repositories can only import scenarios and execute them


ODI Studio

•  An IDE (Integrated Development Environment) which allows Operators, Developers and Administrators users to access the repositories.

•  Four navigators to perform different tasks.

•  Designer

•  Topology

•  Operator •  Security


ODI Studio: Topology Navigator

•  Defines the physical and logical architecture of the information systems

•  Connections through to databases, OLAP servers, applications, files, etc

•  Logical aliases for these connections

•  Contexts (environments) •  Standalone and JEE agents

•  Manage repositories.

•  This information is stored in the Master Repository


ODI Studio: Designer Navigator

•  Models •  Reverse-engineering metadata on data objects from

source systems

•  Projects

•  Used for designing ODI data, transformation and mapping objects

•  Define variables, sequences, functions, mappings and packages among other objects

•  Import and use Knowledge Modules to check integrity, load and integrate information.

•  Generate Scenarios


ODI Studio: Operator Navigator

•  Monitor the execution of ODI objects -mappings, packages and scenarios- in the sessions

•  Drill-into the actual code executed at each KM step

•  Shows errors, warnings and execution metrics in the execution logs.

•  Import and execute scenarios and load plans

•  This information is stored in the Work Repository


What is Change Data Capture (CDC)?

•  CDC is the process of identify, capture and delivery only the new and modified data from the source to the target tables

•  Incremental Load

•  Typical requirement of Data Warehouses

•  More efficient: less data to move, less data to process

•  Extract data in real time.


CDC process in ODI

•  ODI performs CDC applying a JKM (Journalize Knowledge Module) to a model •  JKM creates the infrastructure to capture change data

•  The Journalizing infrastructure is a set of tables, views and triggers to capture, store and purge the changed data

•  ODI will create

•  A table with the J$ prefix to load the PK of the changed data •  Views that join the J$ table with the original one to get all the columns of the

changed data.


Oracle Golden Gate

•  Oracle GoldenGate 12c is a complete and high-performance application for real-time data integration and data replication with minimal overhead.

•  OGG 12c provides solutions for:

•  High availability

•  Real-time data integration using log-based change data capture (CDC) and replication software platform.

•  Disaster Recovery

•  Data replication


Oracle Golden Gate


How the integration between ODI and OGG works?? •  OGG replicates the source data into staging area. •  OGG capture the changed data (CDC) from the source table, route it and deliver it to

•  the replicated table in the staging area

•  the J$/JV$ structures required by ODI JKMs

•  ODI detects the changed records in its journalizing infrastructure and move the data into the final target table.


How the integration between ODI and OGG works?? •  ODI 12c integrates with Oracle GoldenGate through the GoldenGate Jagent

•  This allows ODI to communicate with OGG and also to remotely deploy configuration settings into GG installation.

•  OGG JAgents are defined in ODI as Data Servers in the Topology Navigator


Benefits of ODI – OGG integration

•  Low impact on the source system •  OGG reads the changes directly from the database redo-logs

•  OGG doesn’t require additional activity as triggers

•  Great performance

•  Integration of changed data in real-time

•  Heterogeneous capabilities •  ODI and GoldenGate can work on heterogenous databases allowing more flexibility

in the data integration infrastructure.


OGG and ODI Integration Steps

Configurations in Oracle GoldenGate:

1.  Install OGG twice for the source and target schema.

2.  Database and Environment Variables Configurations

3.  Configure the JAgent for each GG installation

4.  Start both installations of OGG


Step 1: Install OGG twice

•  We need to install Golden Gate twice •  A source installation is required for capturing the data from the source schema

•  A target installation is required for delivering the data to the target schema


Step 2: Database configuration

•  Oracle GoldenGate uses the redo logs to capture the data that needs to replicate, so they must be configured in the source database.

•  Two commands need to be executed to enable Supplemental Logging

•  ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

•  ALTER SYSTEM SWITCH LOGFILE;

•  Oracle recommends to set also the Database to force logging: •  ALTER DATABASE FORCE LOGGING;

•  If your Oracle Database version is 11.2.0.4 you also need to set explicitly a database initialization parameter to enable GoldenGate

•  ENABLE_GOLDENGATE_REPLICATION=true


Step 2: Environment variables configuration

•  In Unix environments, the Library Path variable that contents the path of the OGG shared libraries must be set

•  Example:

export LD_LIBRARY_PATH=/ggcore_1:$LD_LIBRARY_PATH (*)

•  The JAVA_HOME and PATH variables also required to be setup to point to the JDK folder and to the BIN folder of JDK respectively.

•  Example:

export JAVA_HOME=/u01/app/oracle/product/ogg1/jdk

export PATH=$PATH:$JAVA_HOME/bin (*) The command to set variables depends on the type of Unix. Check the Installing and Configuring OGG document


Step 3: JAgent Configuration for each GG installation

•  To use the OGG JAgent enable monitoring is required. •  Create a parameter file without any extension in the GG home directory and name it

GLOBALS.

•  Add the line ENABLEMONITORING to enable monitoring in GG.


Step 3: JAgent Configuration

•  In order to secure the communication between ODI and OGG an Oracle Wallet must be created

1.  Set the JAVA_HOME environment variable to point to the JDK folder under the OGG home.

2.  Run the pw_agent_util utility which is under the OGG installation folder using the –jagentonly option


Step 3: JAgent Configuration

•  Modify the config.properties file 1.  Agent_type_enabled=OEM. To specify that

we are not using the OGG Monitor (OGGMON) which is the default value

2.  As OEM is used for jagent type, the RMI port will be used. Default value 5559


Step 4: Start GoldenGate

•  Run the executable file ggsci which is under the OGG home folder 1.  Start Manager

2.  Start jagent


OGG and ODI Integration Steps

Configurations in Oracle Data Integrator:

1.  Create both source and target OGG Data Servers in the Topology Navigator.

2.  Create OGG Physical Schemas from the Model 3.  Start Journal 4.  Develop a mapping for initial and incremental load of the final target table 5.  Automate the journalized process


Step 1: Create OGG Data Servers in ODI •  ODI 12c communicates with OGG through the GoldenGate JAgents which are defined

as Data Servers in the Topology navigator.

•  Create a new Data Server under Oracle GoldenGate Technology.

•  Enter values for: host, JMX Port (RMI port), JMI User (root by default), the Password created for the Oracle Wallet and the Installation Path.

•  Test the connection.


Step 2: Select the OGG JKM on the source Model

•  We need to apply the journalizing process to the source model

1.  In the Journalizing tab of the source model, set the Journalizing Mode as Consistent Set

•  This option takes into account referential integrity between datastores involved in the CDC process

2.  Select the JKM Oracle to Oracle Consistent (OGG online)


Step 3: Create OGG Physical Schemas from the Model

•  The Capture and Delivery process are the Physical Schemas in ODI •  They can be defined in the Topology Navigator or directly from the source Model.

•  Press the Create button next to the Capture process. Select the source OGG Data Server as Jagent and enter a Name. That will create the Physical and Logical Schemas in ODI.

•  Repeat those steps for the Delivery Process and select the target schema



•  Once we define the Capture and Delivery process: •  We need to set the ONLINE option as TRUE.

•  In the Topology Navigator we have the Physical and Logical Schemas.



•  When we create the Capture process from the Model, the Trial File Path isn’t set with the path of the Delivery instance.

•  The Remote Trial File Path needs to be modified in the Topology Navigator.


Step 4: Start Journal

1.  Once we configured the model, we need to select which datastores (tables) we want to journalize adding the datastores to the CDC process

2.  Define the Subscribers that have to be notified of those changes

3.  Start the journal process.

1 3

2


Step 5: Create a mapping to use the journalized data

1.  In the Designer Navigator create a New Mapping to load the data from the source datastore to the final target.

2.  In the Physical tab create a Deployment Specification for the initial load. Use an Insert or Control Append KM.

3.  Create another Deployment Specification for the incremental load. Select the source datastore and tick the Journalized Data Only option. Use an Incremental IKM.


Step 6: Automation of the Journalized Process

When we use Consistent Set as Journalizing mode there are some required tasks to perform before and after the consumption of the modified data.

Previous Tasks:

•  Extended Window

•  Lock Subscribers

Post Tasks: •  Unlock Subscribers

•  Purge Journal


Step 6: Automation of the Journalized Process

We need to create a package in ODI with the following steps: •  ODIWaitForLogData Step: waits for changes on the Journalized data

•  Journal Model Step: to Extend Window and Lock Subscriber

•  Mapping Step: executes the mapping (need to select a Deployment Specification)

•  Journal Model Step: to Unlock Subscriber and Purge Journal Data


DEMO

Oracle Data Integrator and GoldenGate 12c

41!


Conclusions

•  OGG is the best option for CDC in ODI

•  Less impact in the source database (OGG reads the redo logs and doesn’t require additional activity in the database as triggers)

•  Great performance in delivering the modified data (real time integration). •  More flexibility in the data integration infrastructure (ODI and OGG support

a wide range of databases)


Questions?


Additional Information

•  ODI 12c Documentation: http://docs.oracle.com/middleware/1212/odi/odi-devandrun.htm

•  ODI 12c OTN Forum : https://community.oracle.com/community/developer/english/business_intelligence/system_management_and_integration/data_integrator

•  A-Team Blog: http://www.ateam-oracle.com/ •  Data Integration Blog: https://blogs.oracle.com/dataintegration/ •  Rittman Mead Blog: www.rittmanmead.com/blog


Thank you!

Twitter: @EdelweissK Mail: [email protected]

integrating oracle data integrator with oracle goldengate 12c

Technology

odi data

data objects

data management

data transformation

integration odi

data quality

data synchronization

data movement