4101

Teradata Data Mover Installation and User Guide

B035-4101-039A

Release 13.0

March 2009

The product or products described in this book are licensed products of Teradata Corporation or its affiliates.

Teradata, BYNET, DBC/1012, DecisionCast, DecisionFlow, DecisionPoint, Eye logo design, InfoWise, Meta Warehouse, MyCommerce, SeeChain, SeeCommerce, SeeRisk, Teradata Decision Experts, Teradata Source Experts, WebAnalyst, and You’ve Never Seen Your Business Like This Before are trademarks or registered trademarks of Teradata Corporation or its affiliates. Adaptec and SCSISelect are trademarks or registered trademarks of Adaptec, Inc. AMD Opteron and Opteron are trademarks of Advanced Micro Devices, Inc. BakBone and NetVault are trademarks or registered trademarks of BakBone Software, Inc. EMC, PowerPath, SRDF, and Symmetrix are registered trademarks of EMC Corporation. GoldenGate is a trademark of GoldenGate Software, Inc. Hewlett-Packard and HP are registered trademarks of Hewlett-Packard Company. Intel, Pentium, and XEON are registered trademarks of Intel Corporation. IBM, CICS, RACF, Tivoli, and z/OS are registered trademarks of International Business Machines Corporation. Linux is a registered trademark of Linus Torvalds. LSI and Engenio are registered trademarks of LSI Corporation. Microsoft, Active Directory, Windows, Windows NT, and Windows Server are registered trademarks of Microsoft Corporation in the United States and other countries. Novell and SUSE are registered trademarks of Novell, Inc., in the United States and other countries. QLogic and SANbox are trademarks or registered trademarks of QLogic Corporation. SAS and SAS/C are trademarks or registered trademarks of SAS Institute Inc. SPARC is a registered trademark of SPARC International, Inc. Sun Microsystems, Solaris, Sun, and Sun Java are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States and other countries. Symantec, NetBackup, and VERITAS are trademarks or registered trademarks of Symantec Corporation or its affiliates in the United States and other countries. Unicode is a collective membership mark and a service mark of Unicode, Inc. UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and company names mentioned herein may be the trademarks of their respective owners.

The information contained in this document is provided on an “as-is” basis, without warranty of any kind, either express or implied, including the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. Some jurisdictions do not allow the exclusion of implied warranties, so the above exclusion may not apply to you. In no event will Teradata Corporation be liable for any indirect, direct, special, incidental, or consequential damages, including lost profits or lost savings, even if expressly advised of the possibility of such damages. The information contained in this document may contain references or cross-references to features, functions, products, or services that are not announced or available in your country. Such references do not imply that Teradata Corporation intends to announce such features, functions, products, or services in your country. Please consult your local Teradata Corporation representative for those features, functions, products, or services available in your country. Information contained in this document may contain technical inaccuracies or typographical errors. Information may be changed or updated without notice. Teradata Corporation may also make improvements or changes in the products or services described in this information at any time without notice. To maintain the quality of our products and services, we would like your comments on the accuracy, clarity, organization, and value of this document. Please e-mail: [email protected] Any comments or materials (collectively referred to as “Feedback”) sent to Teradata Corporation will be deemed non-confidential. Teradata Corporation will have no obligation of any kind with respect to Feedback and will be free to use, reproduce, disclose, exhibit, display, transform, create derivative works of, and distribute the Feedback and derivative works thereof without limitation on a royalty-free basis. Further, Teradata Corporation will be free to use any ideas, concepts, know-how, or techniques contained in such Feedback for any purpose whatsoever, including developing, manufacturing, or marketing products or services incorporating Feedback.

Copyright ©2009 by Teradata Corporation. All Rights Reserved.

Table of Contents

Table of Contents................................................................................................................ 3 Chapter 1: Overview........................................................................................................... 7

Teradata Data Mover ...................................................................................................... 7 System Requirements...................................................................................................... 7 Supported Database Versions ......................................................................................... 8 Metadata Components .................................................................................................... 8 Replication Process Flow................................................................................................ 9

Replicator Process Shell ..................................................................................... 9 Synchronization Validation Shell ....................................................................... 9

Considerations............................................................................................................... 10 Network Bandwidth .......................................................................................... 10 Synchronization Delay...................................................................................... 10

Chapter 2: Installation....................................................................................................... 11 Gather Information........................................................................................................ 11 Install Dependencies ..................................................................................................... 11 Install from CD Media .................................................................................................. 12 Install Script .................................................................................................................. 14

New Setup Option............................................................................................. 14 Install and Overwrite Option ............................................................................ 15 Restore Option .................................................................................................. 15 Uninstall Option................................................................................................ 16

Installed Teradata Data Mover Files............................................................................. 16 Chapter 3: Configuration .................................................................................................. 19

Set Linux Environment Variables................................................................................. 19 Grant Database Privileges............................................................................................. 20

Teradata DM User Privileges............................................................................ 20 Staging Database Privileges.............................................................................. 20 Staging Database User Privileges ..................................................................... 21 Teradata DM Metadata Tables User Privileges ................................................ 21

Populate Metadata Tables ............................................................................................. 22 MDA_DIRECTIVE Table ................................................................................ 22 MDA_TABLE_GROUPINGS Table ............................................................... 22

Set Up Stored Procedure Calls...................................................................................... 23 INS_EVENT Stored Procedure ........................................................................ 23

Set Maximum Number of Concurrent Jobs .................................................................. 23 Chapter 4: Replication and Maintenance.......................................................................... 25

Start Teradata Data Mover............................................................................................ 25 Run rps.ksh as a UNIX Cron Job...................................................................... 25 Run rps.ksh with Command – Alternative 1..................................................... 25 Run rps.ksh with Command – Alternative 2..................................................... 25

Stop Teradata Data Mover ............................................................................................ 25

Teradata Data Mover Installation and User Guide - 3 -

Table of Contents Teradata Data Mover

Output Logs .................................................................................................................. 26 Logs for the Replicator Process Shell............................................................... 26 Logs for Teradata Load and Unload Utilities ................................................... 26

Replication Method Use Cases ..................................................................................... 26 Full Copy .......................................................................................................... 27 Partial Copy ...................................................................................................... 27 Tables with Deletes........................................................................................... 29

Replication Methods ..................................................................................................... 29 Generic Approach ............................................................................................. 29 FastExport and FastLoad Replication............................................................... 30 BTEQ and TPump Replication ......................................................................... 30 BTEQ Execution............................................................................................... 31

Maintenance.................................................................................................................. 31 Metadata Table Maintenance............................................................................ 31 Logs Directory Maintenance............................................................................. 31 Temporary Script File Directory Maintenance ................................................. 32

Chapter 5: Recovery Procedures....................................................................................... 33 Configured Primary Server is Down............................................................................. 33

1. Stop Application Processing......................................................................... 33 2. Stop Dual-Active Processes.......................................................................... 33 3. Reconfigure Secondary System as Configured Primary System.................. 33 4. Manually Load Newly Configured Primary System .................................... 34 5. Start Application Processing: Secondary System as Configured Primary

System....................................................................................................................... 34 6. Fix Previously Configured Primary System ................................................. 35 7. Stop Application Processing......................................................................... 35 8. Replicate from Secondary System to Primary System ................................. 35 9. Stop Dual-Active Processes.......................................................................... 35 10. Reconfigure Primary System as CPS.......................................................... 35 11. Start Application Processing: Primary System as CPS............................... 36

Configured Secondary System is Down ....................................................................... 36 Both Systems are Down................................................................................................ 36 Teradata Data Mover Server is Down .......................................................................... 37

Chapter 6: Advanced Features .......................................................................................... 38 Replication of Tables with Deletes ............................................................................... 38 Batched Replication ...................................................................................................... 38

Batch ID............................................................................................................ 39 Time-Based ....................................................................................................... 40

Copy Statistics .............................................................................................................. 41 Event Processing........................................................................................................... 41 TMSM Integration ........................................................................................................ 43

Chapter 7: Troubleshooting .............................................................................................. 45 Display Version Numbers............................................................................................. 45 Check Error Messages .................................................................................................. 46

Logs Directory .................................................................................................. 46 Metadata Status Table....................................................................................... 46

- 4 - Teradata Data Mover Installation and User Guide

Table of Contents Teradata Data Mover

Error Messages.............................................................................................................. 46 Common Errors............................................................................................................. 49

Appendix A: Teradata Data Mover Metadata Tables ....................................................... 53 Directive Table.............................................................................................................. 53 Directive Event Table ................................................................................................... 55 Primary System Control Table...................................................................................... 58 Table Groupings Table ................................................................................................. 60 TMSMEVENT Table.................................................................................................... 60

Glossary ............................................................................................................................ 65


Chapter 1: Overview

Teradata Data Mover Teradata Data Mover (DM) 13.0 replicates data in a dual system environment. A dual-active architecture uses two independent Teradata systems that have the same data and applications, working together to service users and applications. During normal operations, the systems run production workloads and are actively accessed by users and applications. If one system becomes unavailable, the alternate system continues to transparently service users and applications to provide continuous availability. The primary task of Teradata DM is keeping the data of the primary and secondary Teradata systems closely synchronized in the dual system environment. The primary system is designated the Configured Primary Server (CPS); the secondary system is designated the Configured Secondary System (CSS).

Teradata DM has appliance-like functionality: given a command sequence, Teradata DM runs standard Teradata utilities. Teradata DM exports data from the CPS using FastExport or BTEQ Export. It loads data through a named pipe to the CSS using FastLoad (for SET table), MultiLoad (for Multi-Set table), or TPump Upsert.

Teradata DM supports full- and partial-table replication. After replication, a health check can be performed to ensure both systems' tables are in synch. If differences are found, alerts are generated. Data validation health checks can also be configured to run on a scheduled basis.

Reference: For documentation on the utilities, go to http://www.info.teradata.com/ and click Online Publications>General Search. In Title or Keyword, type FastLoad, FastExport, MultiLoad, or BTEQ.

System Requirements Teradata Managed Server

Teradata DM uses the SAS managed server configuration. The operating system is SUSE Linux SLES 10 SP1.

Reference: On the internal documentation website, http://infocentral.daytonoh.teradata.com/tsd-library, refer to Teradata Managed Server Guide for Model 4XX, B035-5449-098A for hardware information on the SAS server.

The Teradata Managed Server for SAS must have a minimum of 1 gigabyte of free disk space.


http://www.info.teradata.com/

http://infocentral.daytonoh.teradata.com/tsd-library

Chapter 1: Overview Supported Database Versions

Two Teradata Database Servers

The Teradata Database servers should:

Both have Teradata Database version V2R6.1 or later.

Have a minimum of 110 gigabytes of free permanent storage space.

Supported Database Versions Teradata DM is supported on Teradata V2R6.1 or later.

Metadata Components Teradata DM uses metadata tables, stored procedures, and macros for replication. These metadata components are installed on the CPS and CSS. Teradata DM metadata tables are summarized below, and described in more detail in Appendix A. Teradata Data Mover Metadata Tables

Table Name Description

MDA_DIRECTIVE Identifies tables involved in replication.

MDA_DIRECTIVE_EVENT Replication begins when replication-eligible events exist. Records events in the replication process (START, END, and/or ERROR events). Rows inserted into this table with a stored procedure call initiate replication.

MDA_PRI_SYS_CTL Identifies the primary system (CPS) and maximum concurrent jobs (loader slots.

MDA_TABLE_GROUPINGS Assigns groupings and priority to tables for replication.

TMSMEVENT Stores resource, ecosystem, and replication event information. This table is populated when Teradata Multi System Manager (TMSM) is not installed on the Teradata Managed Server for SAS.

Teradata Data Mover Stored Procedures

Stored Procedure Description

INS_EVENT Inserts rows into MDA_DIRECTIVE_EVENT. This stored procedure is called at the end of a load job to insert the replication-eligible event that triggers replication.

PARSE_PK Builds TPump syntax.


Chapter 1: Overview Replication Process Flow

Teradata Data Mover Macros

Macro Name Description

GEN_FASTLOAD_UCODE Builds the FastLoad script when the table for replication contains one or more Unicode columns.

GEN_MLOAD_INSERT Builds the MultiLoad script when the table for replication contains no Unicode column.

GEN_MLOAD_INSERT_UCODE Builds the MultiLoad script when the table for replication contains one or more Unicode columns.

GEN_UPSERT Builds the TPump script when the table for replication contains no Unicode column.

GEN_UPSERT_UCODE Builds the TPump script when the table for replication contains one or more Unicode columns.

Replication Process Flow Teradata DM continuously seeks events that signal the need to replicate a table or a portion of a table, then launches replication as events are added. Refer to the next section for specifics.

Replicator Process Shell The Replicator Process Shell (RPS) is started by a system operator or by the UNIX cron utility. The RPS runs continually on the Teradata Managed Server, but only one instance of the RPS can run at one time. When the RPS begins, it checks to see if another instance is running, and if so, that instance terminates.

The RPS:

Reads the Directive Event table to determine whether replication-eligible events have occurred, and when detected, begins replication.

Conducts health checks on the CPS and the CSS, then reports the results to the Directive Event table. If any health check fails, the RPS sleeps until all health checks are successful or a maximum waiting period elapses. If both health checks are successful before the maximum waiting period elapses, replication begins. Otherwise, an alarm event is sent to the Directive Event table.

Builds the WHERE clause that is used in the export scripts generated for partial-table replication.

Synchronization Validation Shell The Synchronization Validation Shell (SVS) validates synchronization of data. The SVS is manually started by a system operator or by the UNIX cron utility, and runs from the managed server where Teradata DM is installed. It runs on a periodic basis as scheduled by the system operator and terminates upon task completion.


Chapter 1: Overview Considerations

The SVS reads the Directive Event table to determine what needs to be checked and when. Then the SVS writes a row to the Directive Event table to indicate that it has initialized successfully before starting the main program loop.

Under normal circumstances, the CPS and CSS are synchronized after the RPS has completed. The SVS provides an additional level of checking for synchronization by periodically comparing table utility apply-counts, aggregations of summations and averages, table counts, view counts, column counts, and table sizing across the CPS and CSS. If the comparison is a mismatch, a message is sent to the TMSMEVENT table and the comparison metadata is stored on the Teradata DM server.

Considerations Teradata DM does not support replication of:

Tables with identity columns

Tables with large objects (BLOB and CLOB)

Queue tables

The copy statistics feature is valid only when the primary and secondary systems are Teradata Database V2R6.2.

Indices will be maintained on the target table.

Objects other than tables are not supported. Teradata DM does not support replication of table access-rights.

Other considerations include network bandwidth and synchronization delay, which are discussed in the next sections.

Network Bandwidth Network bandwidth can constrain throughput. To verify that throughput will not be affected, validate that the current network can support the additional data load of replication and the ability to move saved changes. For example, aggregate bytes during a 7 day period could total approximately 128 GB, based on the increase in size of the tables in the affected databases. Significant updates increase the amount of data that must be replicated. Tables marked for full replication also increase network usage.

Synchronization Delay After data has been successfully loaded to the CPS, the loaded tables become eligible for replication on the next replication cycle. At that point, there is a time delay between extracting data from the CPS, loading the data into a temporary table on the CSS, and moving the data to the production table.

Use table groupings, which allow the assignment of priority to groups of tables and speed replication of those tables.


Chapter 2: Installation

To install Teradata DM on a Linux Teradata Managed Server, follow the steps below:

1. Gather Information

2. Install Dependencies

3. Do one of the following:

Install from CD Media

Install Script

Gather Information Before installation, identify the:

Configured Primary Server (CPS) and Configured Secondary System (CSS).

Existing user name and password for both the CPS and CSS (the names and passwords will be used for creating the Teradata DM user).

Teradata DM user name and password on the CPS and CSS.

Permanent storage space for new Teradata DM user on the CPS and CSS. No permanent storage space is needed for existing Teradata DM user.

Staging database name for the Teradata DM user on the CPS and CSS.

Permanent storage space for Teradata DM user’s new staging database on the CPS and CSS. No permanent storage space is needed for Teradata DM user's existing staging database.

Install Dependencies Teradata Data Mover requires version 13.00.00.00 or later of the Teradata Tools and Utilities products listed in the next table.


Chapter 2: Installation Install from CD Media

Product Package Name

Basic Teradata Query bteq

Shared Common Components for Internationalization for Teradata tdicu

Teradata Call Level Interface (CLI) V2 cliv2

Teradata Data Connector piom

Teradata FastExport fastexp

Teradata FastLoad fastld

Teradata GSS TeraGSS_redhatlinux-i386

Teradata MultiLoad mload

Teradata Parallel Data Pump tpump

The latest software patches for these utilities are available at: http://tssprod.teradata.com:8080/TSFS/home.do

1. On the Teradata Software Server site, click Teradata Client.

2. On the Welcome to Client Section page, click Search>Client Product.

3. In Search Patch, enter 13.0 in Version and the package name in Full Package Name.

4. Click Submit.

Install from CD Media Use the next procedure to install Teradata DM from the CD media.

Requirement: Privileges are required to mount the installation CD onto the CD-ROM drive on the Teradata Managed Server system. If you do not have those privileges, contact your Teradata Managed Server system administrator to mount the installation CD.

Note: It is not necessary to remove an existing version of Teradata DM on the Teradata Manager Server system.

1. Log on to the Teradata Managed Server system.

2. If a /mnt directory does not exist, create a directory by typing the following at the command prompt:

# mkdir /mnt

The CD-ROM mount point is /mnt, but an alternate directory can be used.

3. Insert the installation CD into the CD-ROM drive, then type the following at the command prompt: # mount /dev/<cdrom device name> /mnt


http://tssprod.teradata.com:8080/TSFS/home.do;jsessionid=E86618B35A91B0CD7F17F13B03925184

Chapter 2: Installation Install from CD Media

For example, if the CD-ROM device name is cdrom, type: # mount /dev/cdrom /mnt

The CD-ROM mount point is /mnt, but an alternate directory can be used.

4. If a $HOME/tdm directory does not exist, create one by typing: # mkdir $HOME/tdm

The directory containing Teradata DM extracted setup files is $HOME/tdm, but an alternate directory can be used.

5. To extract the Teradata DM setup files, type: # cd /mnt # gunzip -c tdm__linux_indep.13.00.00.00-1.tar.gz | tar xvf - -C <extracted files directory>

For example, if the extracted files directory is $HOME/tdm, type: # gunzip -c tdm__linux_indep.13.00.00.00-1.tar.gz | tar xvf - -C $HOME/tdm

6. To begin the installation process, type: # cd <extracted files directory>/tdm.13.00.00.00 # ./tdmsetup

For example, if the extracted files directory is $HOME/tdm, type: # cd $HOME/tdm/tdm.13.00.00.00 # ./tdmsetup

7. Select 1 to do one of the following:

To install a new setup.

To restore (requires Teradata Data Mover version 13.00.00.00 to be installed on the Teradata Managed Server system).

To install Teradata Data Mover version 13.00.00.00 and overwrite the existing version.

The New Setup option installs and sets up Teradata DM on the Teradata Managed Server system and the Teradata primary and secondary servers.

The Restore option restores the:

Teradata DM files and directories that have been deleted on the Teradata Managed Server system.

Teradata DM tables, macros, or stored procedures that have been deleted on the Teradata primary or secondary server.


Chapter 2: Installation Install Script

The Install and Overwrite option installs Teradata Data Mover version 13.00.00.00 and overwrites the existing version on the Teradata Managed Server system. The existing version is saved before it is overwritten.

8. Follow the command prompts to complete the setup. No kernel rebuild or reboot is required after this installation.

Note: Do not delete the tdm__linux_indep.13.00.00.00-1.tar.gz file or the tdm.13.00.00.00 directory on the Teradata Managed Server system. The file and directory are required for installing, restoring, or uninstalling Teradata Data Mover version 13.00.00.00.

Install Script Teradata DM is packaged in a compressed file, tdm__linux_indep.13.00.00.00-1.tar.gz. To run the script:

1. FTP tdm__linux_indep.13.00.00.00-1.tar.gz in binary mode to the Linux Teradata Managed Server.

2. Uncompress tdm__linux_indep.13.00.00.00-1.tar.gz by typing: gunzip –c tdm__linux_indep.13.00.00.00-1.tar.gz | tar xvf –

3. Change to the Teradata DM directory by typing: cd tdm.13.00.00.00

4. Run the Teradata DM install script by typing: ./tdmsetup

5. Follow the command line prompts. Installation and uninstallation options are described in:

New Setup Option

Install and Overwrite Option

Restore Option

Uninstall Option

New Setup Option The New Setup option copies Teradata DM files to a user-defined install directory and creates the Teradata DM metadata objects (tables, macros, and stored procedures) on the CPS and CSS.

This option is only available when Teradata DM is not installed on the system. (If Teradata DM is already installed, see Install and Overwrite Option, Restore Option, or Uninstall Option.)

To run a new setup:

NOTE: You will be asked if you are the correct user to install and run Teradata DM. If you install Teradata DM, you will be granted execution privileges for Teradata


Chapter 2: Installation Install Script

DM. To grant other people execution privileges for Teradata DM, do so manually after installation.

1. Type ./tdmsetup on the command line.

2. Select 1.

3. Follow the command line prompts.

Teradata DM is installed in <BASEDIR>/teradata/client/13.0/datamover. The default for <BASEDIR> is /opt, although you can specify a different location for <BASEDIR>.

A logs directory is created that contains Teradata DM job log files. The directory accumulates job log files when Teradata DM is running. Therefore, ensure there is a minimum of 1 gigabyte of free disk space for the logs directory.

Note: The directory name for <BASEDIR> and the logs directory cannot:

Be a relative directory path

Have the double quote character

Have the equal character “=”

Have whitespace character

Install and Overwrite Option The Install and Overwrite option installs a different version of Teradata DM; the prior version is saved first, then overwritten. This option is available only when a different Teradata DM version is installed. (If Teradata DM is not installed, see New Setup Option.)

To install a different version and overwrite the existing version:

1. Verify that Teradata DM is not currently running on the system.


3. Select 1 to install and overwrite.


Restore Option The Restore option restores one or more of the following: Teradata DM file, directory, table, macro, or stored procedure. This is useful if a Teradata DM file, directory, table, macro, or stored procedure has been deleted.

This option is available only when the version of Teradata DM that is currently installed is the same as the version that will be restored. (If Teradata DM is not installed, see New Setup Option.)


Chapter 2: Installation Installed Teradata Data Mover Files

To restore:



3. Select 1 to restore.


Uninstall Option The Uninstall option uninstalls all Teradata DM files, directories, tables, macros, and stored procedures. The option also deletes all Teradata DM user and staging tables on the CPS and CSS.

This option is only available when Teradata DM is currently installed on the system. If Teradata DM is not installed, see New Setup Option.

To uninstall:



3. Select 2 to uninstall.


5. Manually delete the files in the Teradata DM logs and tmp directories.

Installed Teradata Data Mover Files After installation, Teradata DM installs the files and directories listed in the next table on the Teradata Managed Server.


Chapter 2: Installation Installed Teradata Data Mover Files

Directory File Name File Permission

<INSTALLDIR> .dwh.cfg 644

<INSTALLDIR> colstat.ksh 500

<INSTALLDIR> readme.txt 444

<INSTALLDIR> rpsfefl.ksh 500

<INSTALLDIR> rps.ksh 500

<INSTALLDIR> rpsupsrt.ksh 500

<INSTALLDIR> sendevent 500

<INSTALLDIR> sqlcmd.ksh 500

<INSTALLDIR> svs.ksh 500

<INSTALLDIR> tblcmp 500

<INSTALLDIR>/logn LOGONPRI 600

<INSTALLDIR>/logn LOGONSEC 600

<INSTALLDIR> logs/ 755

<INSTALLDIR> repl_backup/ 755

<INSTALLDIR> tmp/ 755

The default directory for <INSTALLDIR> is /opt/teradata/client/13.0/datamover.


Chapter 3: Configuration

This chapter includes these topics:

Set Linux Environment Variables

Grant Database Privileges

Populate Metadata Tables

Set Up Stored Procedure Calls

Set Maximum Number of Concurrent Jobs

Set Linux Environment Variables After Teradata DM is set up, the .dwh.cfg configuration file is automatically created in the Teradata DM install directory and populated using values entered during setup. The file contains environment variables used by Teradata DM.

Edit this configuration file only if a system variable needs to be changed.

Variable Name Definition Default Value

DWH_ROOT Directory structure for application scripts.

/opt/teradata/client/13.0/datamover

PATH Appends DWH_ROOT to current PATH environment variable.

${PATH}:${DWH_ROOT}

DWH_LOGS Directory structure for Linux logs. ${DWH_ROOT}/logs

MDA_ID Teradata DM user ID. No default value. User must provide this during installation.

DWH_STAGE_DB Staging database for replication. No default value. User must provide this during installation.

TeraPPStdpid TdpId of Configured Primary System (CPS).

No default value. User must provide this during installation.

TeraPSStdpid TdpId of Configured Secondary System (CSS).

No default value. User must provide this during installation.

RPS_SLEEP_SEC Value, in seconds, that Teradata DM sleeps after a successful replication. After the time has passed, Teradata DM wakes up, looking for replication-eligible jobs. If the value is 0, Teradata DM terminates after the current successful replication job.

60 seconds

LOOP_HC_MAX Only one instance of Teradata DM can be running on the system at one time.

5


Chapter 3: Configuration Grant Database Privileges

Variable Name Definition Default Value When Teradata DM is called, it first performs a health check to see if another instance of Teradata DM is running. If so, the instance of Teradata DM that performed the health check terminates. This variable contains the maximum number of sequential times Teradata DM will perform the health check described above.

TMSMVERT System variable that tells Teradata DM what to do with the sendevent. Values N or NO Tells Teradata DM to not send the event. BOTH Tells Teradata DM to send the event to the primary and secondary systems.

BOTH

Note: In the configuration file, there are environment variables in addition to the variables listed in the preceding table. Do not change the additional variables.

Grant Database Privileges

Teradata DM User Privileges The privileges listed in the next table are required for every table that will be replicated. The Teradata user name for Teradata DM must have these privileges on the database where the table is located.

Note: These privileges are required for the user ID that will perform the replication on the CPS and the CSS.

Privilege Description

CREATE TABLE Creates the target table when the target table does not exist

DELETE Deletes rows in the target table

INSERT Inserts rows into the target table

SELECT Selects rows from the source table

UPDATE Updates rows in the target table

Staging Database Privileges For every table that will be replicated, the staging database for Teradata DM must have the following privileges on the database where the table is located.


Chapter 3: Configuration Grant Database Privileges

Note: These privileges are required for the user ID that will perform the replication on the CPS and the CSS.


CREATE TABLE Creates the target table when the target table does not exist

DELETE Deletes rows in the target table

INSERT WITH GRANT OPTION

Inserts rows into the target table

SELECT WITH GRANT OPTION

Selects rows from the source table

UPDATE WITH GRANT OPTION

Updates rows in the target table

Staging Database User Privileges The Teradata user name for Teradata DM must have the following privileges on the staging database.

Note: These privileges are required on the CPS and the CSS.


CREATE MACRO Allows TPump to create macros

CREATE TABLE Creates the staging table, log tables, and error tables

DROP MACRO Allows TPump to drop macros

DROP TABLE Drops the staging table, log tables, and error tables

EXECUTE MACRO Allows TPump to execute macros

INSERT Inserts rows into the staging table, log tables, and error tables

Teradata DM Metadata Tables User Privileges Any Teradata user name that inserts a row into the Teradata DM MDA_DIRECTIVE table or calls the Teradata DM INS_EVENT stored procedure must have the following privileges on the Teradata DM database.

Note: These privileges are required on the CPS and the CSS.


INSERT Inserts rows into the Teradata DM MDA_DIRECTIVE and MDA_DIRECTIVE_EVENT tables

EXECUTE PROCEDURE

Calls the Teradata DM INS_EVENT stored procedure

SELECT Selects rows from the Teradata Data Mover tables


Chapter 3: Configuration Populate Metadata Tables

Populate Metadata Tables

MDA_DIRECTIVE Table The MDA_DIRECTIVE table identifies the tables on the CPS to be replicated. Therefore, to replicate a table, insert a row into the MDA_DIRECTIVE table on the CPS and CSS.

Note: Do not populate the Table Version column.

Here is an example of an INSERT statement that replicates the SALES_DB.SALES_TABLE table: INSERT INTO TDM_USER.MDA_DIRECTIVE ( 'SALES_DB', /* Database name of the table for replication */ 'SALES_TABLE', /* Table name for replication */ '', '', '', , 'F', /* F for full table copy; P for partial table copy */ , , , , 'Y', /* Y means to copy table statistics */ 'Y', /* Y means to compare table definitions */ /* Table version; column intentionally left blank */ );

Refer to Appendix A for more information on MDA_DIRECTIVE table columns.

MDA_TABLE_GROUPINGS Table The MDA_TABLE_GROUPINGS table provides a method for grouping tables and setting priorities in the replication queue. The tables to be replicated are sorted based on the priority assigned in this table.

Populating this table is optional. If this table is empty, Teradata DM will perform replication jobs based on the order in which eligible replication rows are placed into the MDA_DIRECTIVE_EVENT table.

Note: Do not populate the Table Version column.

Here is an example of an INSERT statement that populates the MDA_TABLE_GROUPINGS table: INSERT INTO TDM_USER.MDA_TABLE_GROUPINGS ( 'SALES_DB', /* Database name */ 'SALES_TABLE', /* Table name */ 'COMPANY_SALES', /* Table group name */ 1, /* Priority number */ /* Table version; column intentionally left blank */ );


Chapter 3: Configuration Set Up Stored Procedure Calls

Refer to Appendix A for more information on the columns of the MDA_TABLE_GROUPINGS table.

Set Up Stored Procedure Calls

INS_EVENT Stored Procedure To perform a table replication, Teradata DM queries the Directive Event table (MDA_DIRECTIVE_EVENT) for a row indicating that a table is eligible for replication. Usually, the Directive Event table is populated after the replication-eligible table has been changed. Teradata DM provides a stored procedure to insert a row in the Directive Event table to indicate eligible table replication.

To insert rows into the Directive Event table, call the INS_EVENT stored procedure on the CPS.

Here is an example of the stored procedure call: CALL TDM_USER.INS_EVENT ( 'NOTIFY: ELIGIBLE REPLICATION', /* message to start table copy */ random(0, 999999999)(format '999999999'), /* creates unit of work ID */ 'SALES_DB', /* Database name */ 'SALES_TABLE', /* Table name for replication */ '', /* Field value for partial table copy */ 'SOURCE_TD', /* Teradata primary server */ 'F', /* F for FastExport/FastLoad */ 'ETL_PROCESS_NAME', /* Source calling stored procedure */ CURRENT_TIMESTAMP, /* Use current timestamp */ '', /* Custom SQL for hard delete */ rc, /* Stored procedure return code */ errormessage /* Stored procedure output message */ );

Refer to Appendix A for more information on MDA_TABLE_GROUPINGS columns.

Set Maximum Number of Concurrent Jobs MaxLoadTasks is a Teradata Database control parameter that prevents FastLoad, MultiLoad, and FastExport jobs from using all available AMP worker tasks (AWTs). Running more load or unload utilities can affect applications that run concurrently, such as tactical or decision support system queries, because of the higher demand placed by the number of sessions, CPU, I/O, and memory. Teradata DM provides the ability to limit the number of loader slots that Teradata DM uses.

The maximum number of loader slots for Teradata DM jobs is defined in the MDA_PRI_SYS_CTL table. During Teradata DM setup, the user is asked to supply the value for the maximum number of concurrent jobs (loader slots). To change this value, update the MAX_CONCURRENT_JOBS column in MDA_PRI_SYS_CTL.


Chapter 3: Configuration Set Maximum Number of Concurrent Jobs

For example, to set the maximum number of loader slots to 5 using the UPDATE statement: UPDATE TDM_USER.MDA_PRI_SYS_CTL SET MAX_CONCURRENT_JOBS = 5;

Note: Be aware that these settings should be based on the system configuration and workload. Refer to Teradata Database Utilities – Volume 1 A – F, B035-1102, for guidance on these settings. To access the book at http://www.info.teradata.com/:

1. Click Online Publications>General Search.

2. Type 1102 in Publication Product ID.



Chapter 4: Replication and Maintenance Start Teradata Data Mover

Chapter 4: Replication and Maintenance

This chapter includes these topics:

Start Teradata Data Mover

Stop Teradata Data Mover

Output Logs

Replication Method Use Cases

Replication Methods

Maintenance

Start Teradata Data Mover The Replicator Process Shell, rps.ksh, drives the replication process. Only one instance of rps.ksh can run on a system at one time. If rps.ksh is currently running, subsequent attempts to run rps.ksh will fail.

Run rps.ksh in one of three ways:

Run rps.ksh as a UNIX Cron Job Running rps.ksh as a UNIX cron job is recommended. Run rps.ksh every 15 minutes.

Run rps.ksh with Command – Alternative 1 Type: # cd <BASEDIR>/teradata/client/13.0/datamover # nohup ./rps.ksh &

The default for <BASEDIR> is /opt.

Run rps.ksh with Command – Alternative 2 Type: # cd <BASEDIR>/teradata/client/13.0/datamover # ./rps.ksh &

Stop Teradata Data Mover To stop Teradata DM, type:


Chapter 4: Replication and Maintenance Output Logs # cd <BASEDIR>/teradata/client/13.0/datamover # touch stoprps


Note: Using these commands stops the rps.ksh script, but if the script was set up as a UNIX cron job, the RPS will start again.

Output Logs Teradata DM output logs are written to the directory:

<BASEDIR>/teradata/client/13.0/datamover/logs


Logs for the Replicator Process Shell

Log Name Description

daf.rps.<timestamp>.log Logs information for the rps.ksh script and is generated when checking for eligible replication jobs. Contains health check output plus BTEQ SQL and output.

daf.rpsfefl.<CPS-name>.<databasename>. <table-name>.<timestamp>

Logs information for the rpsfefl.ksh script and is generated when performing full table copy. Contains BTEQ SQL and output.

daf.rpsupsrt.<CPS-name>.<databasename>. <table-name>.<timestamp>

Logs information for the rpsupsrt.ksh script and is generated when performing partial table copy. Contains BTEQ SQL and output.

daf.sqlcmd.<databasename>.<table-name>. <timestamp>

Logs information for the sqlcmd.ksh script and is generated when performing hard deletes. Contains BTEQ SQL and output.

SVS_KSH.<timestamp> Logs information for the svs.ksh script and is generated when validating the table on the CSS after replication. Contains BTEQ SQL and output.

Logs for Teradata Load and Unload Utilities

Log Name Description

<CPS-name>.<databasename>.<table-name>. <timestamp>

Logs information for the Teradata load and unload utilities (BTEQ, FastExport, FastLoad, MultiLoad, and TPump).

Replication Method Use Cases Full copy replication is recommended for small tables; partial copy is recommended for large tables. The next sections describe those methods, plus information on replicating tables with hard deletes.


Chapter 4: Replication and Maintenance Replication Method Use Cases

Note that a temporary staging table is used for FastExport/FastLoad or FastExport/MultiLoad replication. Data is loaded directly to the target table with BTEQ/TPump replication.

Full Copy Copying the entire table (full copy) can be more efficient than selecting a few rows for replication. This is especially true for tables that incur hard deletes: it is simpler to replicate the entire table than incur the overhead of tracking each deleted row and replicating the activity on the CSS.

RECOMMENDATIONS

If the job fails during FastExport or FastLoad replication, rerun the job from the beginning. If the job fails while loading the staging table on the CSS, the restart discards the data in the staging table. Restart processing proceeds normally.

If the job fails while applying the data in the staging table to the target table, any changes to the target table are rolled back by the Teradata Database. Restart processing proceeds normally.

Partial Copy For large tables, it is impractical to replicate the entire table on a regular basis. Instead, select a subset of the data for replication (partial copy).

RECOMMENDATIONS

Use BTEQ and TPump replication for partial copy. The FastExport and FastLoad replication method could be used for partial copy. However, the data volumes do not typically warrant the additional overhead of these utilities, especially if loading mini-batches.

Identify the rows to be replicated from the source table. A column containing the timestamp of when each row was loaded or last modified is required. Using the column’s information, configure Teradata DM to select all rows with a timestamp since the previous replication cycle.

A secondary index (VOI) might be required on the timestamp column to improve the performance when selecting the rows from the CPS.

If a partial copy job fails, rerun the job from the beginning. The BTEQ and TPump replication method upserts the data to the target table on the CSS. Therefore, rerunning the job from the beginning produces the same data in the target table.

The FastLoad and FastExport replication method can also be restarted from the beginning.


Chapter 4: Replication and Maintenance Replication Method Use Cases

The WHERE clause for partial replication is based on parameters defined in the Teradata DM metadata tables. The parameters are described in the next table.

Parameter Description

DAL_METHOD Indicates full or partial replication (‘F’ or ‘P’)

FIELDNAME Column name(s) used in WHERE clause (selected as OR clauses)

FIELD_OPERATOR Boolean operator used in the WHERE clause (for example, ‘=’, ‘>’,’<=’)

FIELDVALUE Value that corresponds to FIELDNAME

Example FIELDNAME= ‘batchid’ FIELD_OPERATOR=’=’ FIELDVALUE=’12345’

The WHERE clause that is generated is: WHERE batchid = 12345;

If FIELDNAME consists of multiple values separated by commas, the multiple columns are treated as multiple OR conditions.

Example FIELDNAME= ‘Insert_dte, last_upd_date’ FIELD_OPERATOR=’>=’ FIELDVALUE=’2008-01-01’

The WHERE clause that is generated is: WHERE Insert_dte >= ‘2008-01-01’ OR Last_upd_date >= ‘2008-01-01’

Example FIELDNAME = ‘batchid’ FIELD_OPERATOR = ’>’

When two INS_EVENT stored procedures are used consecutively, in the first INS_EVENT: FIELDVALUE = ’12345’ In the second INS_EVENT: FIELDVALUE = ’12500’

The WHERE clause that is generated is: WHERE batchid > 12345 AND batchid <= 12500


Chapter 4: Replication and Maintenance Replication Methods

Tables with Deletes The term hard delete indicates that a row has been permanently deleted from a table. A soft delete indicates that a row has been marked as deleted in the table, but has not been actually removed from the table.

Replicating data changes that include hard deletes to the source table pose a special problem because Teradata DM cannot identify the rows deleted from the source table if they do not exist. To handle this situation, the load job must provide a list of the rows to be deleted.

RECOMMENDATIONS

Ensure that the load job populates a permanent staging table containing the primary key values of the rows to be deleted. The Primary Key table is replicated to the CSS. BTEQ then deletes the rows from the target table by performing a join between the base table and the table containing the Delete PK table.

Perform a partial copy to replicate the data from the CPS to the CSS and upsert the rows to the target.

To improve performance if the Primary Key column differs from the Primary Index column, create a secondary index on the Primary Key columns.

Replication Methods

Generic Approach The generic processing described in this section applies to all replication methods. The replication to be performed is configured in the Teradata DM metadata tables by the DBA. Data is extracted from the CPS and loaded to the CSS using standard Teradata utilities. Teradata DM can also run SQL on the CSS, in support of the data replication.

Actions Taken

The export and load utilities are connected through a named pipe on the client server.

When a load job has completed loading the data to Table1 on the CPS, INS_EVENT is called to insert an event into the MDA_DIRECTIVE_EVENT table. This event indicates that data is available for the next Teradata DM replication cycle. The load job calls SendEvent to track the event. There is one INS_EVENT and SENDEVENT for each table that will be replicated. (A SendEvent command is sent to TMSM to report status on a job. If TMSM is not installed, the SendEvent information is sent to the TMSMEvent table.)

At the next replication cycle, Teradata DM detects the event and reads the metadata tables to obtain the details for replicating the data. Then, Teradata DM builds the utility scripts and launches BTEQ and the load and unload utilities.


Chapter 4: Replication and Maintenance Replication Methods

FastExport and FastLoad Replication The FastExport and FastLoad method is intended for large volume data replication from the CPS to the CSS. This method is especially suited for copying entire tables (full copy).

Actions Taken Teradata DM creates the utility scripts and launches FastExport and FastLoad,

connecting them through a named pipe. This allows the data to be extracted from the CPS and immediately loaded into the CSS, bypassing the Teradata DM server.

A temporary staging table is created on the CSS. After FastLoad processing completes, the data is moved into the target table.

If the target table does not exist on the CSS, Teradata DM creates the target table using the same table definition on the CPS.

Considerations FastExport and FastLoad can open multiple sessions on the CPS and CSS, therefore

the overhead for logging on and off can consume a high percentage of the overall resources. For low volume jobs, the time to log on and off sessions might take longer than the time to replicate the data.

FastExport and FastLoad are optimized for high volume data movement, processing as much data as possible in as little time as possible. Because the purpose of Teradata DM is to replicate data between two geographically separated systems, this imposes a large amount of burst traffic on the WAN.

To limit the impact on other users, the Teradata DM logon ID should be closely monitored and controlled through TASM. The utilities are designed to process the data as quickly as possible, but are still subject to the system resources they are allowed to consume, which is controlled through TASM.

The Teradata Database restricts the number of load and unload utilities that can operate concurrently. FastExport and FastLoad use one load or unload utility slot on the CPS and CSS, therefore the effect on other ETL and replication jobs must be considered.

BTEQ and TPump Replication This replication method copies from the CPS directly to the target table on the CSS and uses the BTEQ and TPump utilities. This method is intended for low volume replication and is especially suited for copying selected data from a table (partial copy).

Actions Taken Teradata DM creates the appropriate scripts and launches BTEQ Export and TPump,

connecting them through a named pipe.

TPump is used to load the data directly into the target table on the CSS.

If the target table does not exist on the CSS, Teradata DM creates the target table using the same table definition on the CPS.


Chapter 4: Replication and Maintenance Maintenance

Considerations The Teradata Database restricts the number of load and unload utilities that can operate concurrently (through the MAX_CONCURRENT_JOBS parameter), however BTEQ and TPump are not included in the total.

BTEQ Execution This method runs SQL on the CSS to support post-replication tasks that might be necessary.

Actions Taken Teradata DM creates the script and launches BTEQ to execute the SQL on the CSS.

Considerations The SQL runs under the same logon ID as the data replication; the ID is frequently

given a high priority. Consider this fact when writing the SQL, otherwise processing can consume too many system resources.

This method is useful for data cleanup and system maintenance tasks related to data replication, such as purging logs tables and clearing staging tables.

This method supports the HARD DELETE capability.

Maintenance

Metadata Table Maintenance

Type of Tables

Action Required Owner of Maintenance

Schedule

Dual-active metadata tables

Protect the tables by including them in the backup schedule. These are important tables for proper system operation.

Database administrator

Daily

Directive Event table

Trim the table periodically; the table grows on a continual basis.

System administrator When needed

Logs Directory Maintenance Each time that Teradata DM checks the Directive Event tables for tables that are eligible to be replicated, Teradata DM writes approximately 6 KB to the rps.ksh log file. If there are no jobs eligible for replication, Teradata DM sleeps for 60 seconds (default). After waking up, Teradata DM writes another 6 KB to the log file to check again.


Chapter 4: Replication and Maintenance Maintenance

Assuming a default sleep value of 60 seconds, the next table lists the approximate amount of data written by rps.ksh during a period of time:

Period of Time Written to Log File

1 day 9 MB

7 days 63 MB

30 days 1.9 GB

365 days 24 GB

Therefore, you should:

Allow sufficient space for the logs directory

Archive log files periodically (every month, for example)

Delete older archived log files (every 18 months, for example)

Temporary Script File Directory Maintenance During normal dual-active operations, the RPS creates temporary script files in the tmp directory that is configured in the .dwh.cfg file. The files are used as temporary copies of scripts that are run in conjunction with the load or BTEQ scripts.


Chapter 5: Recovery Procedures Configured Primary Server is Down

Chapter 5: Recovery Procedures

Topics in this chapter include:

Configured Primary Server is Down

Configured Secondary System is Down

Both Systems are Down

Teradata Data Mover Server is Down

Configured Primary Server is Down In a disaster recovery scenario or a long-term planned outage, it might be necessary to reconfigure the secondary system as the primary system. To switch the CPS from one system to another:

1. Stop Application Processing A. Stop all applications that are associated with the dual-active systems (for example,

ETL processes).

B. Stop all source applications.

C. Stop rps.ksh.

2. Stop Dual-Active Processes A. Stop loading and replication processing. This is required to reconfigure and

evaluate the health of the systems.

B. Run the shell script to stop all active dual-active processes (applications accessing both Teradata production systems).

C. Run the termination script to set the spool space to zero and stop other secondary system processes.

D. Query the Directive Event table on the secondary system to list any unfinished replication processes.

3. Reconfigure Secondary System as Configured Primary System A. Record the health of the secondary system and the state of the data.

B. Assess damage.

C. Estimate the level of effort required to reestablish the CPS to service. If the time spent reestablishing the CPS to service is unacceptable, update the Primary System Control table (MDA_PRI_SYS_CTL) on the secondary system.



The next table shows an example of data in the Primary System Control table after the change.

Field Name Field Purpose Example

SYSTEMNAME Designates the Configured Primary Server by name

Secondary system

MAX_CONCURRENT_JOBS Limits the number of concurrent Teradata DM jobs on the Teradata Managed Server

10

D. Update the fields that are shown in the next table in the .dwh.cfg configuration file, located in <BASEDIR>/teradata/client/13.0/datamover. Reconfigure the secondary system as the primary system.


TeraPPStdpid

Designates the Configured Primary Server name Secondary system

TeraPSStdpid

Designates the Configured Secondary Server name Primary system

E. Update the LOGONPRI and LOGONSEC Teradata DM logon files, which are located in <BASEDIR>/teradata/client/13.0/datamover/logn. Reconfigure the secondary system as the primary system.


LOGONPRI Designates the Configured Primary Server logon Secondary system

LOGONSEC Designates the Configured Secondary Server logon Primary system

4. Manually Load Newly Configured Primary System A. Using the list of unfinished replication processes, evaluate the state of the backup

files to determine if they are complete or not.

B. Manually delete incomplete files.

C. Load the following to the secondary system:

a. Any jobs that failed to complete successfully on the primary system

b. Any RPS replication jobs that did not finish replicating

5. Start Application Processing: Secondary System as Configured Primary System Start source applications such as business intelligence and ETL tools. At this point, the secondary system is the new CPS and is ready for normal processing.



6. Fix Previously Configured Primary System Restore the primary system (that was configured as the CPS).

The secondary system is now handling normal emergency operations and users are able to receive reports and queries. Problems with the primary system can be investigated and fixed with minimum disruption to users.

7. Stop Application Processing Temporarily stop application processing such as ETL and business intelligence tools. Do not abort the processes or associated replication jobs will not finish.

At this point, the primary system is healthy again and ready to regain its role as the CPS.

8. Replicate from Secondary System to Primary System After the primary system is running normally, its databases need to catch up to the secondary system before it can again become the CPS. The RPS automatically detects the healthy state of the primary system and starts normal replication processing in the direction of the primary system.

9. Stop Dual-Active Processes A. Stop loading and replication processing. This is necessary to reconfigure and

evaluate the health of the systems.

B. Run a shell to stop all active dual-active processes (applications accessing both Teradata production systems).

C. Set the spool space to zero to stop other secondary system processes.

Do not proceed until all replication is complete and both systems are in synch.

10. Reconfigure Primary System as CPS A. After the dual-active processes and source applications are stopped, update the

Primary System Control table (MDA_PRI_SYS_CTL) on the secondary and primary systems.

The next table shows an example of data in the Primary System Control table after the change.


SYSTEMNAME Designates the Configured Primary Server by name

Primary system

MAX_CONCURRENT_JOBS Limits the number of concurrent Teradata DM jobs on the Teradata Managed Server

10


Chapter 5: Recovery Procedures Configured Secondary System is Down

B. Update the fields that are shown in the next table in the .dwh.cfg configuration file, located in <BASEDIR>/teradata/client/13.0/datamover. Reconfigure the primary system as the CPS.


TeraPPStdpid

Designates the Configured Primary Server name Primary system

TeraPSStdpid

Designates the Configured Secondary Server name Secondary system

C. Update the LOGONPRI and LOGONSEC Teradata DM logon files, located in <BASEDIR>/teradata/client/13.0/datamover/logn. Reconfigure the primary system as the CPS.


LOGONPRI

Designates the Configured Primary Server logon Primary system

LOGONSEC

Designates the Configured Secondary Server logon Secondary system

11. Start Application Processing: Primary System as CPS A. Start the RPS. Refer to Start Teradata Data Mover.

The primary system is ready to regain the role of CPS.

B. Verify that the RPS is running on the ETL machine by typing: ps –ef | grep rps

D. Verify that the SVS is running on the ETL machine by typing: ps –ef | grep SVS

Configured Secondary System is Down When the CSS is down, replication cannot occur. However, the CPS continues to load normally and ETL processes run as usual, continuing to update the Metadata Events table.

Replication resumes when the CSS comes back up. As each table is replicated, the script updates the Metadata Event table (see Batched Replication).

Both Systems are Down If both systems are down, ETL processing cannot continue. The RPS sleeps when its own health check fails until both system health checks succeed or the maximum waiting time


Chapter 5: Recovery Procedures Teradata Data Mover Server is Down

is exceeded. After that point, replication processing resumes.

Teradata Data Mover Server is Down The workaround for this situation is to have a secondary Teradata DM server available to complete all the processes that were being performed when the initial Teradata DM server went down.


Chapter 6: Advanced Features Replication of Tables with Deletes

Chapter 6: Advanced Features

Topics in this chapter include:

Replication of Tables with Deletes

Batched Replication

Copy Statistics

Event Processing

TMSM Integration

Replication of Tables with Deletes Replicating data changes that include hard deletes to the source table pose a special problem. Teradata DM cannot identify the rows deleted from the source table if they do not exist. To handle this situation, the load job must provide a list of the rows to be deleted. The actions taken to accomplish this are:

The load job populates a staging table (the table should be permanent) containing the Primary Key values of the rows to be deleted.

The Primary Key table is replicated to the secondary system.

BTEQ deletes the rows from the target table by doing a join between the base table and the table containing the Delete PK table.

If the Primary Key column(s) differ from the Primary Index column(s), creating a secondary index on the Primary Key columns to improve performance might be necessary.

Batched Replication The term batched replication describes a feature in Teradata DM that provides an automated process allowing more than one partial migration at a time, on the same table, based on multiple events existing in the MDA_DIRECTIVE_EVENT table.

Every time a row or a set of rows is inserted or updated in a CPS table, a row for that event is inserted into the MDA_DIRECTIVE_EVENT table. The row in the MDA_DIRECTIVE_EVENT table describes the rules to use when replicating the rows in the CPS table to the CSS table.

Based on the period of time between two occurrences of Teradata DM processing the MDA_DIRECTIVE_EVENT table, multiple updates might have been made to a source table (for example, if the target system was down for maintenance). This would result in multiple rows being inserted into the MDA_DIRECTIVE_EVENT table.


Chapter 6: Advanced Features Batched Replication

Without the batched replication feature, each row in the MDA_DIRECTIVE_EVENT table would be processed as a separate, unique replication job. With batched replication, Teradata DM processes all rows from the MDA_DIRECTIVE_EVENT table and uses a single utility job to replicate all of the rows from the CPS table to the CSS table.

To use the batched replication feature, set the value in the FIELD_OPERATOR column of the Directive table to ">=". This "greater than or equal to" rule causes Teradata DM to perform special processing of the information in the MDA_DIRECTIVE_EVENT table for the batched replication feature.

Two ways to perform the batched replication feature are batch ID and timestamp processing. The Batch ID feature moves only completed and validated batches. Therefore, if a batch is being processed, it will not be moved prematurely.

Timestamp processing moves any changes that occur to a base timestamp (depending on the qualifier, "=" or ">=").

Batch ID In the Batch ID method, the user must supply a column in the CPS (and CSS) table to hold an identifying integer value. When the CPS table is updated with one or more rows of data, each row must have that identifying column updated with the same value. The MDA_DIRECTIVE_EVENT table is then updated with a row signaling the replication that must take place, and the FieldValue of that row is set to the value placed in the identifying column in the CPS table.

Subsequent updates to the CPS table update the identifying column with a value that is incremented by 1 (or some ascending sequence) more than the preceding update. Each update to the MDA_DIRECTIVE_EVENT table reflects the value set in the CPS table.

When Teradata DM processes the MDA_DIRECTIVE_EVENT table, it encounters one or more rows associated with the CPS table. Because Teradata DM processes the rows in the order in which they were inserted into the MDA_DIRECTIVE_EVENT table, the first FieldValue value is encountered first. This value is used as the first value in a WHERE clause to identify the rows to be replicated from the CPS to the CSS. Teradata DM then takes the MAX value of the FieldValue column for the CPS table and uses that value as the second condition value of the WHERE clause.

The constructed WHERE clause is: SELECT * FROM <source table> WHERE <identifying column> >= <first-value> AND <identifying column> <= <highest-value>;

All of the rows that were inserted into the CPS table across all batches of updates are processed at one time, in one load job.


Chapter 6: Advanced Features Batched Replication

Example In this example:

The CPS table is called daily_sales.

The identifying batch ID column is called batch_id.

Teradata DM is processing the MDA_DIRECTIVE_EVENT table every 30 minutes, but updates are being made to daily_sales every 10 minutes.

The initial batch id value is 10001.

At 9:00 pm, 100 rows are inserted into daily_sales.

The batch_id column for those 100 rows must contain the value 10001.

When a row is inserted into the MDA_DIRECTIVE_EVENT table for the rows inserted into daily_sales at 9:00 p.m., the value for the FieldValue column must be set to 10001.

At 9:10 p.m., 125 rows are inserted into daily_sales.


When a row is inserted into the MDA_DIRECTIVE_EVENT table for the rows inserted into daily_sales at 9:10 p.m., the value for the FieldValue column must be set to 10002.

At 9:20 p.m., 150 rows are inserted into daily_sales.


When a row is inserted into the MDA_DIRECTIVE_EVENT table for the rows inserted into daily_sales at time 9:20 p.m., the value for the FieldValue column must be set to 10003.

Between time 9:20 p.m. and 9:30 p.m., Teradata DM processes the MDA_DIRECTIVE_EVENT table and sees the first entry for daily_sales, with a FieldValue of 10001.

Teradata DM finds the MAX value of the FieldValue column (10003) and creates a SELECT statement: SELECT * FROM daily_sales WHERE FIELDVALUE >= 10001 AND FIELDVALUE <= 10003;

All 375 rows will be selected from daily_sales and replicated from the CPS to the CSS.

Time-Based For the time-based method of batched replication, the user must supply a column in the CPS (and CSS) table to hold an identifying timestamp value.

When the CPS table is updated with one or more rows of data, each row must have that identifying column updated with the same timestamp value. The MDA_DIRECTIVE_EVENT table is then updated with a row signaling the replication


Chapter 6: Advanced Features Copy Statistics

that must take place. The FieldValue of that row is set to the value placed in the identifying column in the CPS table.

Subsequent updates to the CPS table update the identifying column with a timestamp value that is greater than the value used in the previous update. Each update to the MDA_DIRECTIVE_EVENT table reflects the value set in the CPS table.

When Teradata DM processes the MDA_DIRECTIVE_EVENT table, it encounters one or more rows associated with the CPS table. The utility processes the rows in the order they were placed in the Event table, therefore the first FieldValue value is encountered first. This value is used as the value for the WHERE clause to identify the rows to be replicated from the CPS to the CSS.

The constructed WHERE clause is: SELECT * FROM <source table> WHERE <identifying column> >= <first-timestamp-value>;

With this feature, all of the rows that were inserted into the CPS table across all batches of updates are processed at one time, in one load job.

Copy Statistics Teradata DM allows copying of statistics, which can be done after a table has been successfully replicated from the CPS to the CSS.

The copy statistics feature is enabled for each table replication cycle by entering Y into the column COLLSTATS_IND in the MDA_DIRECTIVE table. When a replication job is run and the table is eligible for replication with copy statistics enabled, the replication is performed. When the replication is successful, the existing statistics are copied from the CPS to the CSS.

Note: The copy statistics feature is valid only when the primary and secondary systems are Teradata V2R6.2.

Event Processing When Teradata Multi-System Manager (TMSM) is not configured on the Teradata Managed Server, Teradata DM inserts rows into the TMSMEVENT table about the job status during the replication process.

When replication starts, Teradata DM inserts START events in the TMSMEVENT table. When replication completes successfully, Teradata DM inserts END and HCKH events into the TMSEVENT table. (The HCKH event is for latency.)


Chapter 6: Advanced Features Event Processing

Each row in the TMSMEVENT table contains the following information:

Type of Information Notes

Resource Id Resource name. Always set to "Replication".

Resource Type Teradata DM component inserting the row.

Ecosystem Id ID of the primary or secondary ecosystem.

UOW Id ID for a given Unit of Work.

Event Type Type of event (START, END, HCKH, or ALERT).

Event Timestamp Time stamp when the event was inserted.

Created Timestamp Time stamp when the event was created.

Event Message Message text generated by the event.

UOW Health Amount Number of rows affected on the affected table. This number is only available for END events.

Alert Code Set to 1 for ALERT event; otherwise, NULL.

Severity Level Severity of the event type (1 is NORMAL, 5 is WARNING, or 10 is CRITICAL).

Event Source System Teradata DM script inserting the row.

Event Source User Always blank.

Job Name Always blank.

Job Step Process job step ID.

Event DML Always blank.

Optional1 Always blank.


Optional3 Number of times the replication has attempted.

State Code Always NULL.

Application Id Always blank.

Affected Database Name Database name for the affected table name.

Affected Table Name Table name for replication.

UOW Name Always blank.

UOW Description Always blank.

UOW Time Stamp Always NULL.

UOW "From" Time Stamp Always NULL.

UOW Date Always NULL.

UOW Source System Always blank.

Modified Time Stamp Always the same time stamp as the event timestamp.

MDA_VERSION Version of the TMSMEVENT table. Always the default value.


Chapter 6: Advanced Features TMSM Integration

Teradata DM uses the timestamp on the CPS and CSS.

Note: When TMSM is configured and running on the Teradata Managed Server, Teradata DM inserts rows in the TMSM MSMEVENT table, instead of the TMSMEVENT table.

TMSM Integration When TMSM is configured and running on the Teradata Managed Server, Teradata DM inserts rows in the TMSM MSMEVENT table about the job status during the replication process.

When replication starts, Teradata DM inserts START events in the TMSM MSMEVENT table. When the replication completes successfully, Teradata DM inserts END and HCKH events to the TMSM MSMEVENT table. (The HCKH event is for latency.)

Each row in the TMSM MSMEVENT table contains the following information:

Type of Information Notes

Resource Id Resource name. Always set to "Replication".

Resource Type Teradata DM component inserting the row.

Ecosystem Id ID of the primary or secondary ecosystem.

UOW Id ID for a given Unit of Work.

Event Type Type of event (START, END, HCKH, or ALERT).

Event Timestamp Time stamp when the event was inserted.

Created Timestamp Time stamp when the event was created.

Event Message Message text generated by the event.

UOW Health Amount Number of rows affected on the affected table.

Alert Code Set to 1 for ALERT event; otherwise, NULL.

Severity Level Severity of the event type (1 is NORMAL, 5 is WARNING, or 10 is CRITICAL).

Event Source System Teradata DM script inserting the row.

Event Source User Always blank.

Job Name Always blank.

Job Step Process job step ID.

Event DML Always blank.



Optional3 Number of times the replication has attempted.

State Code Always 0.

Application Id Always blank.

Affected Database Name Database name for the affected table name.

Affected Table Name Table name for replication.


Chapter 6: Advanced Features TMSM Integration

Teradata DM uses the timestamp on the CPS and CSS.

Note: When TMSM is not configured on the Teradata Managed Server, Teradata DM inserts rows into the TMSMEVENT table, instead of the TMSM MSMEVENT table.

Reference: For documentation on TMSM, go to http://www.info.teradata.com/. Click Online Publications>General Search. In Title or Keyword, type Teradata Multi-System Manager User Guide.



Chapter 7: Troubleshooting Display Version Numbers

Chapter 7: Troubleshooting

The topics in this chapter include:

Display Version Numbers

Check Error Messages

Error Messages

Common Errors

Display Version Numbers When troubleshooting, it can be helpful to display Teradata DM version numbers and dependencies. Use the -v option of the Teradata DM rps.ksh script to display the Teradata DM:

Version number

Module version number

Version numbers of dependencies

To use the option, type: # cd <BASEDIR>/teradata/client/13.0/datamover # ./rps.ksh -v

Example > ./rps.ksh -v Teradata Data Mover Version 13.00.00.00 rps.ksh : 13.00.00.03 colstat.ksh : 13.00.00.02 rpsfefl.ksh : 13.00.00.01 rpsupsrt.ksh : 13.00.00.01 sendevent : 13.00.00.01 sqlcmd.ksh : 13.00.00.01 svs.ksh : 13.00.00.01 tblcmp : 13.00.00.01 TeraGSS_redhatlinux-i386 : 13.00.00.00 tdicu : 13.00.00.00 cliv2 : 13.00.00.00 piom : 13.00.00.00 bteq : 13.00.00.00 fastexp : 13.00.00.00 fastld : 13.00.00.00 mload : 13.00.00.00 tpump : 13.00.00.00


Chapter 7: Troubleshooting Check Error Messages

Check Error Messages

Logs Directory When errors are encountered in Teradata DM, they are usually written to the output logs in the output logs directory. See Output Logs for more details.

Metadata Status Table Teradata DM uses sendevent calls to update status during a replication job. Teradata DM is designed to work with Teradata Multi System Manager (TMSM) if it is installed. If TMSM is not installed, Teradata DM uses sendevent calls to update its own metadata status table called the TMSMEVENT table.

TMSM Integration

When errors are encountered in Teradata DM, they are usually updated in the TMSM MSMEVENT table under the EventMsg column. Sort the TMSM MSMEVENT table by EventType in ascending order. The errors will have an ALERT event type. Examine the event message for the error.

TMSMEVENT Teradata DM Metadata Table

When errors are encountered in Teradata DM, they are usually updated in the TMSMEVENT table under the EventMsg column. Sort the TMSMEVENT table by EventType in ascending order. The errors will have an ALERT event type. Examine the event message for the error.

Error Messages Message: rps.ksh has detected itself. Exiting program.

Explanation: This message indicates rps.ksh is already running.

Note: Only one instance of rps.ksh can be running on the Teradata Managed Server.

Remedy: None.

Message: UTY8001 RDBMS failure in Packing Test: 3523. An owner referenced by user does not have <missing database privilege> access to <database name>.<table name>.

Explanation: The Teradata DM staging database does not have the proper privileges on the <database name>.

Note: This error occurs when TPump is used.


Chapter 7: Troubleshooting Error Messages

Remedy: Grant the following privileges on the <database name> to the Teradata DM staging database:

CREATE TABLE

DELETE

INSERT WITH GRANT OPTION

SELECT WITH GRANT OPTION

UPDATE WITH GRANT OPTION

Message: CALL Failed. [3523]. The user does not have EXECUTE PROCEDURE access to TDM_USER.INS_EVENT.

Explanation: Self-explanatory.

Remedy: Grant the following privilege on the Teradata DM database to the user:

EXECUTE PROCEDURE

Message: Error: Table contains LOB data. LOB data are unsupported for replication.

Explanation: Teradata DM does not support table with LOB data.

Remedy: None.

Message: DDL is different on source and target.

Explanation: The table definition on the CPS and CCS are different.

Note: This message occurs when the user sets the TABLE_COMPARE_IND column to Y in the MDA_DIRECTIVE table and the table definition on the CPS and CCS are different.

Remedy: Check the table definition on the CPS and CCS.

Message: FastExport Failed (nonzero return code).

Explanation: There was a failure in the FastExport job.

Remedy: Examine the Teradata unload utility log file, <CPS-name>.<databasename>.<table-name>.<timestamp>, in the logs directory and correct the failure.

Message: Load Failed (nonzero return code).

Explanation: There was a failure in the FastLoad or MultiLoad job.


Chapter 7: Troubleshooting Error Messages

Remedy: Examine the Teradata load utility log file, <CPS-name>.<databasename>.<table-name>.<timestamp>, in the logs directory and correct the failure.

Message: Upsert failed for <databasename>.<table-name>.

Explanation: There was a failure in the TPump UPSERT job.


Message: TPump Failed.

Explanation: There was a failure in the TPump job.


Message: Bteq Failed.

Explanation: There was a failure in the BTEQ job.

Remedy: Examine the Teradata utility log file, <CPS-name>.<databasename>.<table-name>.<timestamp>, in the logs directory and correct the failure.

Message: Bteq Export failed for <databasename>.<table-name> with return code of <return code>.

Explanation: There was a failure in the BTEQ Export job.

Remedy: Examine the Teradata utility log file, <CPS-name>.<databasename>.<table-name>.<timestamp>, in the logs directory and correct the failure.

Message: Cannot Generate Upsert for <database name>.<table name>.

Explanation: Teradata DM was not able to generate the UPSERT statement in the TPump script.

Remedy: Examine the rpsupsrt.ksh log file, daf.rpsupsrt.<CPS-name>.<databasename>.<table-name>.<timestamp>, in the logs directory and correct the failure.


Chapter 7: Troubleshooting Common Errors

Message: Cannot execute SQL for table.

Explanation: Teradata DM was not able to execute the SQL on the target table.

Note: This message occurs when Teradata DM tries to execute the hard delete on the target table.

Remedy: Examine the log file for the sqlcmd.ksh script in the logs directory and correct the failure.

Message: send(21080) : Connection refused. SendMsg(21080) : failed to send message.

Explanation: Teradata Multi System Manager is not running or configured properly.

Note: This message will be logged to a file called sendevent.log. The file will exist in the same directory where rps.ksh was started.

Remedy: Check Teradata Multi System Manager and correct the failure.

Message: WARNING: PRIOR NOTIFY EXISTS.

Explanation: A previous event against the same <database name>.<table name> was already entered and the replication for the <database name>.<table name> has not completed.

Note: This message occurs when the INS_EVENT stored procedure is called multiple times for the same <database name>.<table name>.

Remedy: None.

Message: INSERT EVENT: FAILED.

Explanation: There was a failure calling the INS_EVENT stored procedure.

Remedy: Check and correct the failure.

Common Errors The MDA_DIRECTIVE_EVENT table is empty even after an INS_EVENT is called.

EVENT

(empty)

Possible reasons:

Wrong source tdpid sent to INS_EVENT stored procedure

Wrong system name in MDA_PRI_SYS_CTL table in primary system



No primary key is supplied to the MDA_DIRECTIVE_EVENT table when doing upsert (BTEQ and TPump, that is, when Replication_Method= B)

No FieldName is supplied to the MDA_DIRECTIVE table when doing partial replication (that is, when DAL_Method= P)

Remedy:

Make sure correct values are entered into the corresponding fields in these metadata tables: MDA_DIRECTIVE, MDA_PRI_SYS_CTL and MDA_DIRECTIVE_EVENT. See Appendix A for more information.

In the MDA_DIRECTIVE_EVENT table, there is only one event, “NOTIFY: ELIGIBLE REPLICATION”. Replication does not start.

EVENT

NOTIFY: ELIGIBLE REPLICATION

Possible reasons:

rps.ksh has stopped

Error in MDA_DIRECTIVE table or MDA_PRI_SYS_CTL table

Wrong system name in MDA_PRI_SYS_CTL table in secondary system

Error in INS_EVENT (for example, wrong database name or table name)

Remedy:

Make sure rps.ksh is running and correct values are entered into the corresponding fields in those tables.

In the MDA_DIRECTIVE_EVENT table, there are only “NOTIFY: ELIGIBLE REPLICATION” and “START: REPLICATION” events. There are no other rows.

EVENT


START: REPLICATION

Possible reasons:

Error in INS_EVENT (i.e. error in MDA_DIRECTIVE_EVENT table), such as wrong Replication_Method.



At this stage, there is usually more information available in the log files and TMSMEVENT table.

In the log files, some common error messages are:

UTY0805 RDBMS failure, 3803: Table 'ET_t_<table_name>' already exists. Some error tables are left inside the target database in the Secondary System

UTY1008 RDBMS failure: 2644, No more room in database <database_name> Not enough permanent space is allocated for the corresponding database. It can be the staging database or target database.

Failure 3523 The user does not have any access to database_name>.<table_name> The access rights are not set up correctly.

DDL is different on source and target. Exiting now.

The table definitions are different in the primary and secondary systems.

Remedy:

Clean up any error tables or staging tables left in the secondary system.

As the tables grow in size, watch out for under-allocated perm space.

Whenever there is a change in user or change in database, all the access rights must be set up carefully.

In the MDA_DIRECTIVE_EVENT table, there are “NOTIFY: ELIGIBLE REPLICATION”, “START: REPLICATION”, and “ERROR: REPLICATION” events among other events. EVENT


START: REPLICATION

ERROR: REPLICATION

…..

…..

Possible reasons:

UTY8001 RDBMS failure in Packing Test: 3523, An owner referenced by user does not have SELECT WITH GRANT OPTION access to <database_name>. The staging database may be granted the appropriate access rights.

UTY8713 RDBMS failure, 6760: Invalid timestamp

The timestamp is invalid.

At this stage, there is usually more information available in the log files and TMSMEVENT table.



Remedy:

Please refer to Output Logs for details on log files and Event Processing for information on the TMSMEVENT table.


Appendix A: Teradata Data Mover Metadata Tables

Teradata DM metadata tables control and record the processes that are used to maintain the dual system environment. The tables are queried by the processes to determine which replication paths are available. Table details are available in these appendix topics:

Directive Table

Directive Event Table

Primary System Control Table

Table Groupings Table

TMSMEVENT Table

Directive Table The Directive table, MDA_DIRECTIVE, identifies the tables to be replicated.

There are two methods of replication: full or partial. In the DAL_METHOD table field, the value F represents full replication, P represents partial replication, and N represents no replication.

Refer to the next table for other Directive table columns. The values in the table’s Req. column specify:

Y = Always required

PR = Required for partial replication (with WHERE CLAUSE)

U = Required for updates (fld1, fld2, fld3)

N = Not required


Appendix A: Teradata Data Mover Metadata Tables Directive Table

Field Name Data Type Req. Field Purpose

DATABASENAME VARCHAR(30) Y Database name for source and target.

TABLENAME VARCHAR(30) Y Table name for source and target.

FIELDNAME VARCHAR(255) PR WHERE CLAUSE restrictive field name.

FIELD_OPERATOR CHAR(2) PR BOOLEAN operator used in WHERE CLAUSE. (‘=’,’>=’,’<=’,’<>’,’<’,’>’)

PRIMARY_KEY VARCHAR(255) PRU Primary key of table; used for primary key in upsert.

EVENT_ID INTEGER Y Unique event identifier. Generated by default as an identity column.

DAL_METHOD CHAR(1) Y F=FULL P=PARTIAL N=NONE

TEST_STMT VARCHAR(255) N Query used for svs.ksh verification.

TEST_WHERE_CLAUSE VARCHAR(255) N Periodic validation check. WHERE CLAUSE STATEMENT

TEST_GROUP_BY VARCHAR(255) N Periodic validation check. GROUP_BY STATEMENT

TEST_ORDER_BY VARCHAR(255) N Periodic validation check. ORDER_BY STATEMENT

COLLSTATS_IND

CHAR(1) N Y = Copy statistics to target table. N = Do not copy statistics to target table.

TABLE_COMPARE_IND

CHAR(1) N Y = Verify DDL on the table to be replicated matches on source and target. N = Do not verify DDL.


Appendix A: Teradata Data Mover Metadata Tables Directive Event Table

Table DDL CREATE SET TABLE TDM_USER.MDA_DIRECTIVE, FALLBACK, NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT ( DATABASENAME VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, TABLENAME VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, FIELDNAME VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, FIELD_OPERATOR CHAR(2) CHARACTER SET LATIN NOT CASESPECIFIC, PRIMARY_KEY VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, EVENT_ID INTEGER NOT NULL GENERATED BY DEFAULT AS IDENTITY (START WITH 1 INCREMENT BY 1 MINVALUE -2147483647 MAXVALUE 2147483647 NO CYCLE), DAL_METHOD CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL DEFAULT 'N' COMPRESS ('F','N','P'), TEST_STMT VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, TEST_WHERE_CLAUSE VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, TEST_GROUP_BY VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, TEST_ORDER_BY VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, INDICATOR_FILE VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, COLLSTATS_IND CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC DEFAULT NULL, TABLE_COMPARE_IND CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC DEFAULT NULL, MDA_VERSION INTEGER NOT NULL DEFAULT 1 ) UNIQUE PRIMARY INDEX (DATABASENAME, TABLENAME) UNIQUE INDEX (EVENT_ID);

Directive Event Table The Directive Event table, MDA_DIRECTIVE_EVENT, contains information from the replication events. Teradata DM records “before”, “after”, and “error” events in the Directive Event table. The table also records the full path of the backup file where the data is stored. This facilitates any recovery required by system states.

Refer to the next table for Directive Event table values. The values in the table’s Req. column specify:

Y = Always required

PR = Required for partial replication (with WHERE CLAUSE)

U = Required for updates (fld1, fld2, fld3)

N = Not required



Field Name Data Type Req. Field Purpose

EVENT VARCHAR (32) Y Description of event. Valid Values NOTIFY: ELIGIBLE REPLICATION Triggers replication. START: REPLICATION Indicates replication has begun. END: FASTEXPORT Indicates successful completion of Fast Export step. END: FASTLOAD Indicates successful completion FastLoad step. END: BTEQ Indicates successful completion of BTEQ step. END: REPLICATION Indicates all components of replication are complete. ERROR: HEALTH: NOTIFY: HEALTH OK Example 'NOTIFY: ELIGIBLE REPLICATION'

UOWID DECIMAL (18, 0) Y Unit Of Work ID (or batch ID) that calls sendevent. Example 10000001

DATABASENAME VARCHAR (30) Y Database name for source and target.

TABLENAME VARCHAR (30) Y Table name for source and target.

FIELDVALUE VARCHAR (255) PR Field used in WHERE CLAUSE.

SOURCE_TDPID VARCHAR (30) Y TDPID of source system.

FILENAME VARCHAR (255) N Path of dual load backup file.

REPLICATION_ METHOD

CHAR(1) Y Valid Values



F FastExport/FastLoad or FastExport/MultiLoad B BTEQ/TPump Upsert S Uses sqlcmd.ksh to perform the hard delete.

SCRIPT_NAME VARCHAR (255) N Name of replication script. Valid Values rpsfefl.ksh FastExport/FastLoad script used for full table copy rpsupsrt.ksh BTEQ/Tpump script used for partial table copy SQL_CMD_KSH Script used for hard delete.

APPLY_CNT INTEGER N Count taken from log files at the end of each load run.

INSERT_CNT INTEGER N Insert count taken from log files at the end of each load run.

UPDATE_CNT INTEGER N Update count taken from log files at the end of each load run.

DELETE_CNT INTEGER N Count of deletions to target table for the inserts from staging table.

ET_CNT INTEGER N Rows in the load utility error table.

UV_CNT INTEGER N Rows in the load utility uniqueness value table.

EVENT_TS TIMESTAMP (6) Y Timestamp of the event (to the second).

CUSTOMSQL VARCHAR (1000) N DELETE SQL statement.


Appendix A: Teradata Data Mover Metadata Tables Primary System Control Table

Table DDL CREATE SET TABLE TDM_USER.MDA_DIRECTIVE_EVENT, FALLBACK, NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT ( EVENT VARCHAR(32) CHARACTER SET LATIN NOT CASESPECIFIC, UOWID DECIMAL(18,0), DATABASENAME VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, TABLENAME VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, FIELDVALUE VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, SOURCE_TDPID VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC, TARGET_TDPID VARCHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC, FILENAME VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, REPLICATION_METHOD CHAR(1) CHARACTER SET LATIN NOT CASESPECIFIC, SCRIPTNAME VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC, APPLY_CNT INTEGER, INSERT_CNT INTEGER, UPDATE_CNT INTEGER, DELETE_CNT INTEGER, ET_CNT INTEGER, UV_CNT INTEGER, EVENT_TS TIMESTAMP(6) NOT NULL, CUSTOMSQL VARCHAR(1000) CHARACTER SET LATIN NOT CASESPECIFIC, MDA_VERSION INTEGER NOT NULL DEFAULT 1 ) PRIMARY INDEX ( DATABASENAME ,TABLENAME );

Primary System Control Table The Primary System Control table, MDA_PRI_SYS_CTL, identifies the system to be loaded, the direction of replication, and the maximum number of concurrent replications that can run. Data movement between the primary and secondary systems (in either direction) uses the replication method specified in the Directive Event table.

The Primary System Control table also redirects the system to load the secondary system machine on those occasions when the primary system machine is unavailable for an extended period of time.

The Primary System Control table exists on the primary and secondary systems. Its values (listed in the next table) determine which system is primary. The values are initially set during Teradata DM installation (see Chapter 2: Installation).

Field Name Data Type Field Purpose

SYSTEMNAME VARCHAR (32) Designates the source (CPS) by name.

MAX_CONCURRENT_JOBS INTEGER Limits the number of concurrent replication processes on the target system.

MDA_VERSION INTEGER Version of the MDA_PRI_SYS_CTL table. Default is 1; null is not valid. Note: Do not modify this field.


Appendix A: Teradata Data Mover Metadata Tables Primary System Control Table

Table DDL CREATE SET TABLE TDM_USER.MDA_PRI_SYS_CTL, FALLBACK, NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT ( SYSTEMNAME VARCHAR(32) CHARACTER SET LATIN NOT CASESPECIFIC NOT NULL, MAX_CONCURRENT_JOBS INTEGER, MDA_VERSION INTEGER NOT NULL DEFAULT 1 ) PRIMARY INDEX (SYSTEMNAME);

Examples of outcomes that can occur, based on the values in the Primary System Control table, are listed in the next table.

System States (CPS/CSS)

Primary System (return from health check SQL)

Secondary System (return from health check SQL)

Outcome

UP/UP Primary system Primary system With both primary and secondary systems up, there is normal data load and replication.

UP/DOWN Primary system NULL With the primary system up and the secondary system down, normal loading with synchronization will occur when the secondary system comes up.

DOWN/UP – first stage unscheduled outage

NULL Primary system With the primary system (CPS) down, designate the secondary system as the CPS.

DOWN/UP – second stage unscheduled outage

NULL Secondary system With the primary system (CPS) down, designate the secondary system as the CPS. Load to the secondary system (new CPS) and synchronize the primary system when it comes back up.

UP/UP – post-unscheduled outage

Primary system Secondary system Synchronize to the primary system before it is redesignated as the CPS.

DOWN/DOWN – total outage

NULL NULL Follow normal EDW recovery procedures. No loads can take place, therefore set whichever system recovers as the CPS.

UP/UP – scheduled outage or acknowledgement of uncompleted unscheduled outage

Secondary system Secondary system If the secondary system comes up before the primary system after an outage, the secondary system will be marked as the CPS and will receive data loads that will be


Appendix A: Teradata Data Mover Metadata Tables Table Groupings Table

System States (CPS/CSS)

Primary System (return from health check SQL)

Secondary System (return from health check SQL)

Outcome

replicated to the primary system.

After data is synchronized, switch back to the appropriate primary and secondary systems.

Table Groupings Table The Table Groupings table, MDA_TABLE_GROUPINGS, provides a method for grouping tables, primarily for priority in the replication queue. The tables to be replicated are sorted based on the priority assigned in MDA_TABLE_GROUPINGS.


DATABASENAME CHAR (30) Affected database name.

TABLENAME CHAR (30) Affected table name.

GROUPNAME CHAR (30) Table group name, for example, “Sales Tables” or “Inventory Tables”.

PRIORITY INTEGER Priority of the specified TABLEGROUP.

MDA_VERSION INTEGER Version of the MDA_TABLE_GROUPINGS table. Default is 1; null is not valid. Note: Do not modify this field.

Table DDL CREATE SET TABLE TDM_USER.MDA_TABLE_GROUPINGS, FALLBACK, NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT ( DATABASENAME CHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC, TABLENAME CHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC, GROUPNAME CHAR(30) CHARACTER SET LATIN NOT CASESPECIFIC, PRIORITY INTEGER, MDA_VERSION INTEGER NOT NULL DEFAULT 1 ) UNIQUE PRIMARY INDEX (DATABASENAME, TABLENAME, GROUPNAME);

TMSMEVENT Table The TMSMEVENT table stores resource, ecosystem, and replication event information.


ResourceId

CHAR(72) Primary or secondary system tdpid; used in conjunction with the ResourceType to identify which process is running the UoW.


Appendix A: Teradata Data Mover Metadata Tables TMSMEVENT Table

Set in the configuration file.

ResourceType

CHAR(50) Type of job, for example, FastExport or FastLoad; used in conjunction with the ResourceId to identify which process is running the UoW.

EcosystemId

CHAR(20) ID of the primary or secondary ecosystem.

UOWId

CHAR(50) ID for a given Unit of Work (UoW); this is a 50-byte field. Example Productload092908063010 (job name with the date and timestamp)

EventType

CHAR(10) Type of event that was last received. The events are listed in order of severity, then in chronological order, with the latest events listed first.

EventTS TIMESTAMP(6) Time stamp of when the event was sent.

CreatedTS TIMESTAMP(6) Time stamp of when the event was created

EventMsg VARCHAR(200) Message text generated by the event.

UOWHealthAmt FLOAT Number of rows affected on the affected table.

AlertCode INTEGER Alert code for ALERT events.

SeverityLvl

INTEGER Severity level of the event (1 is normal. 5 is warning. 10 is critical).

EventSourceSystem CHAR(30) Teradata DM script that generated the event.

EventSourceUser CHAR(30) User identifier.

JobName VARCHAR(255) The ETL job name.

JobStep VARCHAR(255) Process job step ID

EventDML CHAR(50) DML code.

Optional1 VARCHAR(50) Optional argument 1.

Optional2 VARCHAR(50) Optional argument 2.

Optional3 VARCHAR(5) Number of times the replication has been attempted.

StateCode INTEGER Status code of the resource or ecosystem.

ApplicationId CHAR(20) Application identifier.

AffectedDatabaseName CHAR(30) Target database

AffectedTableName CHAR(30) Target table

UOWName CHAR(30) Unit of Work name.

UOWDesc VARCHAR(200) Unit of Work description.

UOWTS TIMESTAMP(6) Unit of Work timestamp.

UOWFromTS TIMESTAMP(6) Unit of Work "FROM" timestamp.

UOWDate DATE Unit of Work date marker.

UOWSourceSystem CHAR(30) Unit of Work source system.



ModifiedTS TIMESTAMP(6) Time stamp when the event was modified.

MDA_VERSION INTEGER Version of the TMSMEVENT table. Default is 1; null is not valid. Note: Do not modify this field.

Note: Teradata DM does not update a value for the following field names:

EventSourceUser

JobName

EventDML

Optional1

Optional2

ApplicationId

UOWName

UOWSourceSystem

Teradata DM updates NULL value for the following field names:

StateCode

UOWTS

UOWFromTS

UOWDate



Table DDL CREATE SET TABLE TDM_USER.TMSMEVENT,FALLBACK , NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT ( ResourceId CHAR(72) NOT NULL, ResourceType CHAR(50) NOT NULL, EcosystemId CHAR(20) NOT NULL, UOWId CHAR(50) NOT NULL, EventType CHAR(10) NOT NULL, EventTS TIMESTAMP(6) NOT NULL, CreatedTS TIMESTAMP(6) NOT NULL, EventMsg VARCHAR(200), UOWHealthAmt FLOAT, AlertCode INTEGER, SeverityLvl INTEGER NOT NULL, EventSourceSystem CHAR(30), EventSourceUser CHAR(30), JobName VARCHAR(255), JobStep VARCHAR(255), EventDML CHAR(50), Optional1 VARCHAR(50), Optional2 VARCHAR(50), Optional3 VARCHAR(50), StateCode INTEGER, ApplicationId CHAR(20), AffectedDatabaseName CHAR(30), AffectedTableName CHAR(30), UOWName CHAR(30), UOWDesc VARCHAR(200), UOWTS TIMESTAMP(6), UOWFromTS TIMESTAMP(6), UOWDate DATE, UOWSourceSystem CHAR(30), ModifiedTS TIMESTAMP(6), MDA_VERSION INTEGER NOT NULL DEFAULT 1 ) PRIMARY INDEX Inversion_Entry_1 ( ResourceId, ResourceType, EcosystemId, UOWId, EventType, EventTS );


Glossary

Acronym or Term Definition

CPS Configured Primary Server

CSS Configured Secondary Server

DA Dual Active

DBC Database Computer

DDL Data Definition Language

DNS Domain Name Service

DR Disaster Recovery

EDW Enterprise Data Warehouse

ELT Extract Load and Transform

ETL Extract Transform and Load

GB Gigabyte

Gbps Gigabits per second

HA High Availability

HADA High Availability Dual Active

HA/DR High-Availability/Disaster Recovery

KSH Korn Shell

PPI Partitioned Primary Index

PPS Preferred Primary Server

PSS Preferred Secondary Server

Primary Primary Production Environment Under normal operations, this is the primary system.

Secondary High Availability System Environment Under normal operations, this is the secondary system.

SK Surrogate Key

SQL Structured Query Language

TB Terabyte

TMSM Teradata Multi-System Manager

TMP Temporary

UOWID Unit of Work Identifier

UTP Unit Test Plan


Glossary TMSMEVENT Table


4101

Documents

click online

permanent

unix cron

restart processing

nonzero return

stop application

teradata managed

free disk