oracle goldengate

Database Monitoring Services

Oracle GoldenGate

www.LamkoDB.com www.MonitorMyDatabase.com

1Jos van LammerenJos.vanLammeren@LamkoDB.com1BackgroundGoldenGateOverviewSetupOperationsTroubleshootingMonitoringChallengesQ&A

www.LamkoDB.com www.MonitorMyDatabase.comAgenda2Using Oracle GoldenGate since 2010At clientsOracle 9i, 10g near-zero downtime migrations to 11g RACOracle 11g RAC setup for DR and EDW loadUpgrade OGG 10.4 to 11.2 for Oracle 11g clusters (no app downtime)Oracle 9i, 10g, 11g EDW loadSQL Server 2008 to Oracle 11g EDW load

At LamkoDBOracle 10g, 11g, 12c, SQL Server 2008, 2012, MySQL tests with OGG 11g and 12c

www.LamkoDB.com www.MonitorMyDatabase.comBackground3What GoldenGate doesContinuous data replication from one database to anotherSource and target databases can be of different vendors/releasesNon-intrusive, low-impact, sub-second latency Maintains transactional integrity - Resilient against interruptions and failures Data transformation on the flyFew or many tables in a schemaDDL replication if desiredUsed for EDW feeds, DR, DB version upgrades, DB migrations

What GoldenGate does notIs not a replacement for Data Guardwww.LamkoDB.com www.MonitorMyDatabase.comOverview44www.LamkoDB.com www.MonitorMyDatabase.comOverview5

5Use of replicationwww.LamkoDB.com www.MonitorMyDatabase.comOverview6

6Source Databases/Platforms supported

www.LamkoDB.com www.MonitorMyDatabase.comOverview7

7Target Databases/Platforms supported

www.LamkoDB.com www.MonitorMyDatabase.comOverview8

8Processes/Files involved

Classic Capture or Integrated Capture modeNonintegrated Replicat or Integrated Replicatwww.LamkoDB.com www.MonitorMyDatabase.comOverview9

9ObservationsNot all data types are supportedNeed data type translation in heterogeneous environmentsEncryption of data in files and across networkOptional compression of data copied by pump processCryptic names for processes: 8 charactersCryptic names of trail files e1xxxxxx (2 characters followed by incrementing number)Administrator must like command line tool (similar to sqlplus)www.LamkoDB.com www.MonitorMyDatabase.comOverview1010Installation GG software (source & target server)Download software from Oracle site. Software to use is Oracle DBMS release specific.On Linux/Unix create account that owns the software & add to dba groupCreate profile to set $ORACLE_HOME, $PATH, etc. for the accountUnzip & untar software or use Oracle Universal InstallerConfigure GLOBALS and mgr.prm filesInstallation in DBDB in archive log mode (at least source instance)Turn off recyclebin (in source instance for DDL replication)Put DB in supplemental logging mode (source instance)Run OGG supplied installation scripts (source & target)

www.LamkoDB.com www.MonitorMyDatabase.comSetup1111How many schemas/tables to replicate & how much data will be replicated?Have tables the same structure in source and target database or do target tables have additional columns?Do tables have PKs?Are there referential constraints to deal with if not all tables are being replicated?Delete Cascade constraintsWhen bi-directional replicationSequencesTriggers firing on either database

www.LamkoDB.com www.MonitorMyDatabase.comSetup items to keep in mind1212Setup replication flow & initial load of dataCreate/start Extract on source databaseCreate/start Pump process on source databaseDo initial data load from source to target databaseCreate/start Replicat on target database

www.LamkoDB.com www.MonitorMyDatabase.comReplication Setup Example13

13ExtractExtract process E1ORA1E1ORA1 extracts data from MMDBORA1 (Oracle) to local trail files in directory/gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1

www.LamkoDB.com www.MonitorMyDatabase.comSetup14

14EXTRACT

E01ORA1.obeyADD EXTRACT E01ORA1, TRANLOG, BEGIN NOW, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/E01ORA1.prmADD EXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, EXTRACT E01ORA1

E01ORA1.prmEXTRACT E01ORA1. . .USERID ggate, PASSWORD abcEXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1 megabytes 100. . .TABLE CHINOOK.*;

start E01ORA1

www.LamkoDB.com www.MonitorMyDatabase.comSetup1515PumpPump process P1ORA1 copies local trail file contents to remote trail files on another server in directory C:\gg\dirdat\MMDBORA1\p1


16PUMP

P01ORA1.obeyADD EXTRACT P01ORA1, EXTTRAILSOURCE /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/P01ORA1.prmADD RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, EXTRACT P01ORA1, MEGABYTES 100

P01ORA1.prmEXTRACT P01ORA1RMTHOST 192.168.21.94, MGRPORT 7809RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, megabytes 100PASSTHRUTABLE CHINOOK.*;

start P01ORA1

www.LamkoDB.com www.MonitorMyDatabase.comSetup1717Initial load of dataExport/ImportData pumpRMANLoad data over a DB linkbcpOGG utilitiesmany more


18ReplicatReplicat process R01ORA1 (SQL Server 2012) reads the remote trail file contents and applies rows to the database


19REPLICAT

R01ORA1.obeyDBLOGIN SOURCEDB GGATE, USERID ggate, PASSWORD abcadd replicat R01ORA1, exttrail C:\gg\dirdat\MMDBORA1\p1 params C:\gg\dirprm\CHINOOK\R01ORA1.prm

R01ORA1.prmREPLICAT R01ORA1SOURCEDB GGATE, USERID ggate, PASSWORD abc. . .MAP CHINOOK.ALBUM, TARGET dbo.ALBUM;MAP CHINOOK.ARTIST, TARGET dbo.ARTIST;MAP CHINOOK.CUSTOMER, TARGET dbo.CUSTOMER;MAP CHINOOK.TRACK, TARGET dbo.TRACK;

start R01ORA1

www.LamkoDB.com www.MonitorMyDatabase.comSetup2020Replication flow complete


21ggsci command line interfaceinfo allinfo * or info E*, etcinfo E01ORA1info E01ORA1 detailstart E01ORA1stop E01ORA1stats E01ORA1send E01ORA1 statusmany more www.LamkoDB.com www.MonitorMyDatabase.comOperations2222ggsci commands

GGSCI (MMDBLNXDB01) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNINGEXTRACT RUNNING E01ORA1 00:00:00 00:00:04EXTRACT RUNNING E02ORA1 00:00:00 00:00:04EXTRACT RUNNING P01ORA1 00:00:00 00:00:09EXTRACT RUNNING P02ORA1 00:00:00 00:00:00REPLICAT RUNNING R01SQL1 00:00:00 00:00:04

GGSCI (MMDBLNXDB01) 2> info E01ORA1

EXTRACT E01ORA1 Last Started 2014-02-20 15:49 Status RUNNINGCheckpoint Lag 00:00:00 (updated 00:00:02 ago)Log Read Checkpoint Oracle Redo Logs 2014-03-13 21:52:56 Seqno 3726, RBA 25596928 SCN 0.127734271 (127734271)

www.LamkoDB.com www.MonitorMyDatabase.comOperations2323Network issues causing pump to crashUsually restart pumpUse AUTORESTART in mgr.prm

www.LamkoDB.com www.MonitorMyDatabase.comOperational Issues2424Network issues causing pump to crashUsually restart pumpUse AUTORESTART in mgr.prmServer reboot and processes not startingUse AUTOSTART in mgr.prm

www.LamkoDB.com www.MonitorMyDatabase.comOperational Issues2525Network issues causing pump to crashUsually restart pumpUse AUTORESTART in mgr.prmServer reboot and processes not startingUse AUTOSTART in mgr.prmExtract performance badUse dedicated volumes where redo/archived logs are storedHave separate volumes for local trail filesUse multiple extracts

www.LamkoDB.com www.MonitorMyDatabase.comOperational Issues2626Network issues causing pump to crashUsually restart pumpUse AUTORESTART in mgr.prmServer reboot and processes not startingUse AUTOSTART in mgr.prmExtract performance badUse dedicated volumes where redo/archived logs are storedHave separate volumes for local trail filesUse multiple extractsPump performance badCompress data when sending data acrossHave separate volumes for local/remote trail filesUse multiple extracts/pumps

www.LamkoDB.com www.MonitorMyDatabase.comOperational Issues2727Replicat performance bad Check for tables with missing PKsUse batchsql in prm file of replicatHave separate volumes for remote trail filesUse multiple replicats

www.LamkoDB.com www.MonitorMyDatabase.comOperational Issues2828Replicat performance bad Check for tables with missing PKsUse batchsql in prm file of replicatHave separate volumes for remote trail filesUse multiple replicatsDisk full for local and/or remote trail filesOccasionally need to setup replication again, with or without initial loadMonitor disk space usage by storage group or sysadminsHave separate volumes for local and remote trail filesUse PURGEOLDEXTRACTS in mgr.prm

www.LamkoDB.com www.MonitorMyDatabase.comOperational Issues2929Crashing replicats when tables added/changedAdd/modify tables on target database, possibly update data definition file, and restart replicatMake extract process stop when it notices DDLwww.LamkoDB.com www.MonitorMyDatabase.comOperational Issues3030Crashing replicats when tables added/changedAdd/modify tables on target database, possibly update data definition file, and restart replicatMake extract process stop when it notices DDLExtract process does not stopCommit open transactions in DBKill session having open transactionForce stop of Extract process if not many archived logs to re-read at next Extract startupwww.LamkoDB.com www.MonitorMyDatabase.comOperational Issues3131Crashing replicats when tables added/changedAdd/modify tables on target database, possibly update data definition file, and restart replicatMake extract process stop when it notices DDLExtract process does not stopCommit open transactions in DBKill session having open transactionForce stop of Extract process if not many archived logs to re-read at next Extract startupCrashing replicats because of manual data changes in target databaseDo more manual changes to fix the dataPossibly reload affected tablesEducate users

www.LamkoDB.com www.MonitorMyDatabase.comOperational Issues3232Where to look for informationReport file of process (view report )Discard fileOGG Error logOracle alert log / SQL Server ERRORLOG

Compare source and target data after fixing issueVeridataselect data from source table MINUS select data from target tableOracle DBMS_COMPARISONOther tools (TOAD, )

www.LamkoDB.com www.MonitorMyDatabase.comTroubleshooting3333Setup Heartbeat in replication flowDocument ID: 1299679.1 OGG_Best_Practice_-_heartbeat_table_using_DBMS_SCHEDULER-_V11_0_ID1299679.1.pdfUpdate heartbeat record in source database (every minute)Extract picks up record in source database, adds information and write to local trail filePump picks up record, adds information and write to remote trail fileReplicat picks up record, adds information and writes to target database (as an insert)

www.LamkoDB.com www.MonitorMyDatabase.comMonitoring3434Monitor using Oracle tool (Director)Need to install Java agent in every GG instance & maintain bunch of servers to keep tool running (Weblogic, )(Remote) Monitor using own developed toolsMonitor status of processesMonitor number of rows replicatedReport on Heartbeat dataQueries against heartbeat table in target databaseGraphs generated based on data in heartbeat table (lag times)ggsci-like GG Dashboard for all GG instances monitored

www.LamkoDB.com www.MonitorMyDatabase.comMonitoring3535OGG Management Pack

www.LamkoDB.com www.MonitorMyDatabase.comMonitoring36

36LamkoDB Monitoring


37Historical stats for Extract process


38Lag times for Extract/Pump/Replicat


39GG Dashboard


40GG process detail information


41GG process relationships


42Who supports GoldenGate?DBAMiddleware group

www.LamkoDB.com www.MonitorMyDatabase.comChallenges4343Who supports GoldenGate?DBAMiddleware groupWhat to do about table structure changessoftware releases need to be evaluated/tested/coordinated

www.LamkoDB.com www.MonitorMyDatabase.comChallenges4444Who supports GoldenGate?DBAMiddleware groupWhat to do about table structure changessoftware releases need to be evaluated/tested/coordinatedWhat do developers, support, clients know about OGG?Educate personnel

www.LamkoDB.com www.MonitorMyDatabase.comChallenges4545Who supports GoldenGate?DBAMiddleware groupWhat to do about table structure changessoftware releases need to be evaluated/tested/coordinatedWhat do developers, support, clients know about OGG?Educate personnelMonitor what is running and what is brokenHave proper monitoring in place with alerting

www.LamkoDB.com www.MonitorMyDatabase.comChallenges4646Who supports GoldenGate?DBAMiddleware groupWhat to do about table structure changessoftware releases need to be evaluated/tested/coordinatedWhat do developers, support, clients know about OGG?Educate personnelMonitor what is running and what is brokenHave proper monitoring in place with alertingKeeping track of new, existing and changing replication flows. Number of processes quickly becomes a spaghetti bowl.Have automated tool to show relationships

www.LamkoDB.com www.MonitorMyDatabase.comChallenges4747Questions?

www.LamkoDB.com www.MonitorMyDatabase.comQ&A48Jos van [email protected]

oracle goldengate

Documents

g racoracle

g clusters

oracle site

source databasesplatforms

data types

g edw loadsql server

source instanceturn

g rac setup