oracle goldengate – irctc - · pdf fileoracle goldengate, irctc (incompatible record cum...

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

ORACLE GOLDENGATE Oracle GoldenGate, IRcTC (Incompatible Record cum Trail Corruption)

Veeratteshwaran Sridhar Oracle GoldenGate, IRcTC (Incompatible Record cum Trail Corruption) Oracle India Pvt Ltd June 10, 2017 [email protected] Blogger - www.oracle-scn.com OTN community - ORASCN

Oracle Confidential – Internal/Restricted/Highly Restricted


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3


EXTRACT REPLICAT

DATAPUMP COLLECTOR

MANAGER MANAGER

N

E

T

W

O

R

K

SOURCE TARGET Transaction Logs Trail Files Transactions

GOLDENGATE ARCHITECTURE FLOW DIAGRAM

STORAGE STORAGE


TRAIL FILE Prior to OGG 12.2


2017/04/10 18:13:52.707.000 FileHeader Len 1395 RBA 0

Name: *FileHeader*

3000 0321 3000 0008 4747 0d0a 544c 0a0d 3100 0002 | 0..!0...GG..TL..1...

0005 3200 0004 2000 0000 3300 0008 02f2 7302 c89d | ..2... ...3.....s...

2bb8 3400 003a 0038 7572 693a 5645 4553 5249 4448 | +.4..:.8uri:VEESRIDH

2d49 4e3a 696e 3a6f 7261 636c 653a 636f 6d3a 6472 | -IN:in:oracle:com:dr

6976 652d 433a 6f67 6731 3232 5f73 6f75 7263 653a | ive-C:ogg121_source:

5245 5854 3600 0025 0023 633a 5c6f 6767 3132 325f | REXT6..%.#c:\ogg121_

736f 7572 6365 5c64 6972 6461 745c 7265 3030 3030 | source\dirdat\re0000

Trail File Header

TRAIL FILE


TRAIL FILE

Hdr-Ind : E (x45) Partition : . (x04) UndoFlag : . (x00) BeforeAfter : A (x41) RecLength : 35 (x0023) IO Time : 2015/09/29 11:58:09.000.000 IOType : 5 (x05) OrigNode : 255 (xff) TransInd : . (x03) FormatType : R (x52) SyskeyLen : 0 (x00) Incomplete : . (x00) AuditRBA : 50 AuditPos : 514576 Continued : N (x00) RecCount : 1 (x01)

Transaction Type

Length of the Record Operation

Type Captured Image Type (Before or

After)

Before / After Image

Archive Log Sequence

No

Position of Record


1 - Abort 2 - Commit 3 - Delete 4 - EndRollBack 5 - Insert 6 - Prepared 7 - TMF-Shutdown 8 - TransBegin 9 - TransRelease 10 - Update 11 - UpdateComp 12 - FileAlter 13 - FileCreate 14 - FilePurge 15 - FieldComp 16 - FileRename 17 - AuxPointer 18 - NetworkCommit

19 - NetworkAbort 20 - CurrentPos 89 - SQL/MX DDL OP 90 - GGSSQLCol 100 - GGSPurgedata 101 - GGSPurgeFile 102 - GGSCreateFile 103 - GGSAlterFile 104 - GGSRenameFile 105 - GGSSetmode 107 - GGSControl 106 - GGSChangeLabel 160 - DDL OP 115 - GGSKeyFieldComp 117 - GGSKeyFieldComp32 161 - RecordFragment 116 - LargeObject 132 - GGSCreateSequence

133 - GGSAlterSequence 134 - GGSDropSequence 150 - RestartAbend 151 - RestartOK 152 - RecoveryEnd 200 - GGSBulkio 201 - GGSFileClose 202 - GGSLoggerTS 203 - GGSExtractTS 204 - GGSCollectTS 205 - GGSComment 250 - LibOpenTrace 251 - LibCloseTrace 252 - LoggerOpenTrace 253 - LoggerCloseTrace 254 - LoggerAddedInfo 249 - LoggerAddedStats

LOGDUMP UTILITY – RECORD OPERATION TYPE


TRANSACTION INDICATOR DESCRIPTION

TransInd x00 First Statement in the Transaction

TransInd x01 Statement in Middle of the Transaction

TransInd x02 Last Statement in the Transaction

TransInd x03 Single Operation or Single record

Transaction

TRAIL FILE


2015/09/29 11:58:09.000.000 Insert Len 35 RBA 1516 Name: SCOTT.TAB1 After Image: Partition 4 G s 0000 000a 0000 0000 0000 0000 0002 0001 0009 0000 | .................... 0005 7669 636b 7900 0200 04ff ff00 00 | ..vicky........ Column 0 (x0000), Len 10 (x000a) 0000 0000 0000 0000 0002 | .......... Column 1 (x0001), Len 9 (x0009) 0000 0005 7669 636b 79 | ....vicky Column 2 (x0002), Len 4 (x0004) ffff 0000 | ....

Timestamp and

Operation Type

Source Object Name

Captured Image Type (Before or

After)

Record Position in the Trail File

Record Length

Data in ASCII Format

Data in Hex Format

Column Information

with Data

TRAIL FILE


GGS tokens: TokenID x52 'R' ORAROWID Info x00 Length 20 4141 4156 6172 4141 4541 4141 4146 4c41 4142 0001 | AAAVarAAEAAAAFLAAB.. TokenID x74 't' ORATAG Info x01 Length 0 TokenID x4c 'L' LOGCSN Info x00 Length 7 3333 3733 3438 33 | 3373483 TokenID x36 '6' TRANID Info x00 Length 9

342e 3235 2e31 3130 34 | 4.25.1104

Row ID

SCN

TRANSACTION ID

TRAIL FILE


HOW TO OPEN THE TRAIL FILES?


LOGDUMP

UTILITY


LOGDUMP UTILITY

It is the tool used to troubleshoot Oracle GoldenGate during Data corruption or trail file corruption.

This tool allows a user to navigate through the trail file.

It is used to compare the information of the trail file with the data extracted and replicated by the processes.

This is mainly used when Data error occurs in the Oracle GoldenGate.


LOGDUMP UTILITY

Transactions Information. Operation type and Time when the Record written. Source Object name. Image type, whether it is a Before Image or After Image. Column information with data and sequence information. Record length, Record data in ASCII format. RBA Information.


LOGDUMP

COMMANDS


LOGDUMP UTILITY

1. To view the record header with the data: Logdump 1 >GHDR ON - The record header contains information about the transaction 2. To add column information: Logdump 2 >DETAIL ON - Column information includes the number and length in hex and ASCII 3. To add hex and ASCII data values to the column information: Logdump 3 >DETAIL DATA 4. To view user tokens: Logdump 4 >USERTOKEN ON - User tokens are custom user-defined information that is specified in a

TABLE or FILE mapping statement and stored in the trail file for specific purposes. 5. To view automatically generated tokens: Logdump 5 >GGSTOKEN ON - Oracle GoldenGate automatically generated tokes include the transaction ID (XID), the

row id for DML operations, the fetching status (if applicable), and tag value. 6. To control how much record data is displayed: Logdump 6 >RECLEN length


LOGDUMP UTILITY – PERFORMING SOME TASKS

7. To Open a trail file, follow the below steps., Logdump 7 >OPEN file_name file_name is either the relative name or fully qualified name of the file, including the file sequence

number. For example: open /home/ggs/dirdat/jd000000 8. To go to the first record and then move through records in sequence Logdump 8 >NEXT Logdump 9 >N


9. To find the next good record header, enter the following command, Logdump 10 >SCANFORHEADER Logdump 11 >SFH 10. To scan for the end of a transaction, enter the following command, Logdump 12 >SCANFORENDTRANS Logdump 13 >SFET 11. To go to a specific RBA in the file, you can do either of the following, To go to an RBA anywhere in the file, Logdump 14 >POS rba Logdump 15 >N To go to the first record in the file, Logdump 16 >POS FIRST Logdump 17 >POS 0



12. The following command shows a count summary followed by counts for each table or data file., Logdump 18 >COUNT Logdump 22 >COUNT LogTrail /vol3/ogg/dirdat/et000000 has 7 records Total Data Bytes 8217 Avg Bytes/Record 1173 Delete 1 DDL 5 Others 1 Before Images 1 After Images 5 Average of 7 Transactions Bytes/Trans ..... 1221 Records/Trans ... 1 Files/Trans ..... 1

13. To save the whole file, enter the following command, where file is the name of the new file Logdump 7 >SAVE file



14. To close the current file and open the next one in the trail, enter the following command., Logdump 18 >NEXTTRAIL Logdump 19 >NT 15. To keep a log of the Session, To start logging, enter the following command Logdump 20 >LOG TO filename.txt 16. To stop logging Logdump 21 >LOG STOP



17. To see the current Logdump environment, enter the following command., Logdump 22 >ENV Version : Linux, x64, 64bit (optimized) on Sep 24 2013 18:18:53 Current Directory : /vol3/ogg LogTrail : /vol3/ogg/dirdat/et000000 Trail Format : New End of File : 9063 Current Position : 0 Forward Next Position : 0 Last Modtime : 2015/09/13 11:38:12.000.000 Display RecLen : 140 Logtrail Filter : On Trans History : 0 Transactions, Records 100, Bytes 100000 LargeBlock I/O : On, Blocksize 57344 Local System : LittleEndian Metadata Byte Order : BigEndian Logtrail Data : BigEndian/ASCII Logtrail Headers : ASCII Dump : ASCII Savefile comments : Off Timeoffset : LOCAL Scan Notify Interval: 10000 records, Scrolling On



18. To exit Logdump, enter either of the following commands., Logdump 23 >EXIT Logdump 24 >QUIT



Trail Files in

OGG 12.2?


Quick Quiz… Identify The Missing Parameter

EXTRACT ext1 USERIDALIAS ggs_admin DDL include mapped RMTTRAIL ./dirdat/et TABLE hr.*;

REPLCAT rep1 USERIDALIAS ggs_replicat DDL include all MAP hr.*, TARGET hr.*;

No SourceDefs No AssumeTargetDefs


No more SOURCEDEFS

Simpler to configure replication

– Eliminates the SOURCEDEFS or ASSUMETARGETDEFS parameters – Supports replication even if source and target have different structures or different databases

Eliminate many manual steps and reduces errors during replication – Metadata information in the trail file is accurate unlike ASSUMETARGETDEFS which assumes

target table has the same internal structure as source table

Self-Describing Trail Files


DDR – Database Definition Record TDR – Table Definition Record

Ref TDR

TDR DDR

DML TDR

Metadata records used to interpret DML records instead of SOURCEDEFS or ASSUMETARGETDEFS

Each trail file contains a Database Definition Record (DDR) before first occurrence of a DML record or a SEQUENCE from a particular database

Each trail file contains a Table Definition Record (TDR) before first occurrence of a DML record for a particular table

- TDR contains table and column definition including column number, data types, column lengths, etc.

DML records have a reference to the TDR and no longer contain the object name

- Typically results in smaller trail files

SEQUENCE records have a reference to the DDR and no longer contain the SEQUENCE name


USE SOURCEDEFS

& ASSUMETARGETDEFS

in OGG 12.2?


You can easily switch between the old and new ways of resolving the

table metadata by:

– Using the new NO_USE_TRAILDEFS in GLOBALS

parameter file.

Default is USE_TRAILDEFS

– Use the OVERRIDE option for individual SOURCEDEFS and

ASSUMETARGETDEFS parameters in the pump or Replicat parameter

files.


2017/04/10 18:13:52.707.000 FileHeader Len 1395 RBA 0

Name: *FileHeader*

3000 0321 3000 0008 4747 0d0a 544c 0a0d 3100 0002 | 0..!0...GG..TL..1...

0005 3200 0004 2000 0000 3300 0008 02f2 7302 c89d | ..2... ...3.....s...

2bb8 3400 003a 0038 7572 693a 5645 4553 5249 4448 | +.4..:.8uri:VEESRIDH

2d49 4e3a 696e 3a6f 7261 636c 653a 636f 6d3a 6472 | -IN:in:oracle:com:dr

6976 652d 433a 6f67 6731 3232 5f73 6f75 7263 653a | ive-C:ogg122_source:

5245 5854 3600 0025 0023 633a 5c6f 6767 3132 325f | REXT6..%.#c:\ogg122_

736f 7572 6365 5c64 6972 6461 745c 7265 3030 3030 | source\dirdat\re0000

Trail File Header


___________________________________________________________________

Hdr-Ind : E (x45) Partition : . (x00)

UndoFlag : . (x00) BeforeAfter: A (x41)

RecLength : 81 (x0051) IO Time : 2017/04/10 18:13:52.926.000

IOType : 170 (xaa) OrigNode : 1 (x01)

TransInd : . (x03) FormatType : R (x52)

SyskeyLen : 0 (x00) Incomplete : . (x00)

DDR/TDR Idx: (001, 000) AuditPos : 9964560

Continued : N (x00) RecCount : 1 (x01)

2017/04/10 18:13:52.926.000 Metadata Len 81 RBA 1403

Name:

*

DDR Version: 1

Database type: ORACLE

Character set ID: we8mswin1252

National character set ID: UTF-16

Locale: neutral

Case sensitivity: 14 14 14 14 14 14 14 14 14 14 14 14 11 14 14 14

TimeZone: GMT

Global name: TESTDB Database Name

Database Type

Database Characterset

DDR – Database Definition Record


Hdr-Ind : E (x45) Partition : . (x00)

UndoFlag : . (x00) BeforeAfter: A (x41)

RecLength : 192 (x00c0) IO Time : 2017/04/10 18:13:52.927.000

IOType : 170 (xaa) OrigNode : 2 (x02)



DDR/TDR Idx: (001, 001) AuditPos : 9964560


2017/04/10 18:13:52.927.000 Metadata Len 192 RBA 1535

Name: SOURCE.T33

*

1)Name 2)Data Type 3)External Length 4)Fetch Offset 5)Scale 6)Level

7)Null 8)Bump if Odd 9)Internal Length 10)Binary Length 11)Table Length 12)Most Sig DT

13)Least Sig DT 14)High Precision 15)Low Precision 16)Elementary Item 17)Occurs 18)Key Column

19)Sub DataType 20)Native DataType 21)Character Set 22)Character Length 23)LOB Type 24)Partial Type

*

TDR version: 1

Definition for table SOURCE.T33

Record Length: 82

Columns: 2

ID 64 50 0 0 0 1 0 50 50 50 0 0 0 0 1 0 1 2 2 -1 0 0 0

NAME 64 20 56 0 0 1 0 20 20 0 0 0 0 0 1 0 0 0 1 -1 0 0 0

End of definition

Table Name

Table Definition

Number of Columns

TDR – Table Definition Record


Hdr-Ind : E (x45) Partition : . (x0c)

UndoFlag : . (x00) BeforeAfter: B (x42)

RecLength : 22 (x0016) IO Time : 2017/04/10 18:10:59.000.000

IOType : 15 (x0f) OrigNode : 255 (xff)



AuditRBA : 420 AuditPos : 9964560


2017/04/10 18:10:59.000.000 FieldComp Len 22 RBA 1788

Name: SOURCE.T33 (TDR Index: 1)

Before Image: Partition 12 G b

0000 0005 0000 0001 3200 0100 0900 0000 0576 6963 | ........2........vic

6b79 | ky

Column 0 (x0000), Len 5 (x0005)

0000 0001 32 | ....2

Column 1 (x0001), Len 9 (x0009)

0000 0005 7669 636b 79 | ....vicky

GGS tokens:

TokenID x52 'R' ORAROWID Info x00 Length 20

4141 4158 6b48 4141 4541 4141 4139 3941 4142 0001 | AAAXkHAAEAAAA99AAB..

TokenID x74 't' ORATAG Info x01 Length 0

TokenID x4c 'L' LOGCSN Info x00 Length 7

3634 3530 3833 34 | 6450834

TokenID x36 '6' TRANID Info x00 Length 9

362e 3235 2e33 3431 39 | 6.25.3419

Transactional Record


IRCTC


OGG IRCTC

INCOMPATIBLE RECORD

CUM

TRAIL CORRUPTION


Trail file corruption

Data Corruption in buffer.

Archive log file corruption.

Data is not properly sync to Disk.

TYPES OF CORRUPTION


Why Corruptions


Network Glitch

Multiple process writing to same trail file

Space issues

Filesystem /Disk Issues

Old trail files not cleaned up properly

Abnormal System Shutdown

Unsupported File system

If FORMAT RELEASE 9.5 is used

REASONS OF CORRUPTION


• When trail file gets corrupted, Pump extract/ Replicat abends with either of the following error

ERROR OGG-01449 Oracle GoldenGate Capture for Oracle, p1.prm: Scan failed in trail file ./dirdat/pk000011, with scan start seqno 11, rba 325929194.

ERROR OGG-01028 Incompatible record (104) in ./dirdat//aa000000, rba 433792090 (getting header).

IMPACTS OF CORRUPTION


Oracle GoldenGate Extract/Replicat has ability to read from buffer before Synced to disk. Due to file system or space issue, if the data is not sync properly to Disk

Input checkpoint position 52755539 for input trail file './dirdat/ca004207' is greater than the size of the file (52617450).

IMPACTS OF CORRUPTION


OPEN SR?????????


Classic Integrated

select group_name, seqno, rba, log_csn, log_cmplt_csn, to_char(last_update_ts, 'DD-MON-YYYY HH24:MI:SS AM') last_update_ts from GGADMIN.GGSCHKPT;

select server_name, applied_low_position, applied_high_position, logbsn from DBA_GG_INBOUND_PROGRESS;

RECOVERY OF CORRUPTION

Note down the Last Applied SCN in the Target by the Replicat process


CHECKPOINT TABLE

GGSCI (Oracle12c.localdomain) 5> info rep1 detail REPLICAT REP1 Initialized 2017-06-17 11:54 Status STOPPED Checkpoint Lag 00:00:00 (updated 00:00:03 ago) Log Read Checkpoint File /ogg/dirdat/target/ft000000000 First Record RBA 0 Current Log BSN value: (requires database login) Last Committed Transaction CSN value: (requires database login) Extract Source Begin End /ogg/dirdat/target/ft000000 * Initialized * First Record Current directory /ogg Report file /ogg/dirrpt/REP1.rpt Parameter file /ogg/dirprm/rep1.prm Checkpoint file /ogg/dirchk/REP1.cpr Checkpoint table ggadmin.ggschkpt Process file /ogg/dirpcs/REP1.pcr Error log /ogg/ggserr.log


CHECKPOINT TABLE FOR CLASSIC REPLICAT

SQL> desc ggadmin.ggschkpt Name Null? Type ----------------------------------------- -------------------------- ---------- GROUP_NAME NOT NULL VARCHAR2(8) GROUP_KEY NOT NULL NUMBER(19) SEQNO NUMBER(10) RBA NOT NULL NUMBER(19) AUDIT_TS VARCHAR2(29) CREATE_TS NOT NULL DATE LAST_UPDATE_TS NOT NULL DATE CURRENT_DIR NOT NULL VARCHAR2(255) LOG_BSN VARCHAR2(128) LOG_CSN VARCHAR2(128) LOG_XID VARCHAR2(128) LOG_CMPLT_CSN VARCHAR2(128) LOG_CMPLT_XIDS VARCHAR2(2000) VERSION NUMBER(3)


CHECKPOINT TABLE FOR INTEGRATED REPLICAT

SQL> desc DBA_GG_INBOUND_PROGRESS Name Null? Type ----------------------------------------- -------- ---------------------- SERVER_NAME NOT NULL VARCHAR2(128) PROCESSED_LOW_POSITION VARCHAR2(4000) APPLIED_LOW_POSITION VARCHAR2(4000) APPLIED_HIGH_POSITION VARCHAR2(4000) SPILL_POSITION VARCHAR2(4000) OLDEST_POSITION VARCHAR2(4000) APPLIED_LOW_SCN NOT NULL NUMBER APPLIED_TIME DATE APPLIED_MESSAGE_CREATE_TIME DATE SOURCE_DATABASE VARCHAR2(128) SOURCE_ROOT_NAME VARCHAR2(128)


Note down the Time Stamp of the Last Applied SCN Record in the Target Side.


Logdump> open <trail file> Logdump>ghdr on Logdump> ggstoken detail Logdump> filter include ggstoken logcsn = <log_cmplt_csn> or <LWM> Logdump> n If reaches EOF of trail file, then nt to search in next trail file Logdump> nt Logdump> n When record is found, for classic mode Logdump> filter clear Logdump> n This is the point of next transaction which we need to start.

SEARCH FOR SCN


For Integrated

When record is found,

Logdump>filter clear

Logdump> n

If this shows High water mark SCN, then do a next and note down the RBA. This means there are no transactions between LWM and HWM

Logdump>n

SEARCH FOR SCN


ggsci> alter extract <pump_name, extseqno <seqno>, extrba <rba> ggsci> alter extract <pump_name>, etrollover ggsci> info <pump_name>, showch and note down write checkpoint seqno

Alter replicat <rep_name>, extseqno<write checkpoint seqno> extrba 0

On Source

On Target

REPOSITION THE PROCESSES


RECOVERY STEPS PRIOR TO OGG 12cR2

• If Classic Replicat, find the LOG_CMPLT_CSN from Checkpoint table. If Integrated, get the Low Watermark SCN (LWM) from the info detail output of the Replicat process.

• Use the Logdump utility and find the Timestamp that contains the SCN on the Target trail.

• Now check the Trail file that covers the timestamp and then use the Logdump to search for the SCN in the Local Trail file.

• Alter the Pump to the respective local trail file and RBA which contains the next new transaction and perform ETROLLOVER.

• Alter the replicat to the new Trail file and start it.


AUTOMATED REMOTE TRAIL FILE

RECOVERY

BY PUMP PROCESS


For use when Replicat abends due to missing or corrupt trail file – If the trail is corrupt, delete all the trail files first from the corrupted seq#. Any missing trails are now automatically rebuilt by bouncing the Extract Pump. Once trail files have been restored, restart the Replicat – Do not use NOFILTERDUPTRANSACTIONS Requires at least 1 valid, complete trail on the target – Due to this, you may want to modify your PURGEOLDEXTRACTS parameter

IMPROVED TRAIL FILE RECOVERY


Causes Replicat to ignore transactions that it has already processed.

Use when Extract was repositioned to a new start point (ATCSN or AFTERCSN

option of "START EXTRACT") and you are confident that there are duplicate

transactions in the trail that could cause Replicat to abend.

This option requires the use of a checkpoint table.

If the database is Oracle, this option is valid only for Replicat in nonintegrated

mode.

The default is FILTERDUPTRANSACTIONS

To override this, use NOFILTERDUPTRANSACTIONS

FILTERDUPTRANSACTIONS | NOFILTERDUPTRANSACTIONS


RECOMMENDATIONS TO AVOID CORRUPTIONS

Use recommended mount options with caching and async io

disabled for file system where OGG resides

Clean up the trail file when reconfiguring extract/pump

Stop OGG process during system shutdown

Configure PURGEOLDEXTRACT parameter in manager parameter file

to delete processed trail files.

oracle goldengate – irctc - · pdf fileoracle goldengate, irctc (incompatible record cum...

Documents