teradata parallel transporter scripts with simplified syntax
DESCRIPTION
Teradata Parallel Transporter Scripts with Simplified Syntax. Nam Tran, Teradata Parallel Transporter Teradata Corporation October 6, 2011. Today’s Agenda. Introduction Job Script Example Before Simplified Syntax Using Operator Templates Using Generated Schemas - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/1.jpg)
Teradata Parallel Transporter Scripts with Simplified Syntax
Nam Tran, Teradata Parallel TransporterTeradata Corporation
October 6, 2011
![Page 2: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/2.jpg)
1. Introduction
2. Job Script Example Before Simplified Syntax
3. Using Operator Templates
4. Using Generated Schemas
5. Using Generated SQL INSERT Statements
6. Job Script Example After Simplified Syntax
7. Q & A
Today’s Agenda
![Page 3: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/3.jpg)
Introduction
• Customer ease-of-use is of paramount importance to Teradata’s strategy. Teradata Parallel Transporter 13.10 and 14.0 introduces a simplified script syntax:> Smaller and simpler job scripts> Less job script maintenance with use of operator templates> User-defined templates allow any degree of customization> Generated schema objects> Generated SQL insert statement> Standardized the names of job variables that correspond to operator attributes
• Target audience of this presentation:> Users of script-based TPT> Users who write TPT job scripts> Users who work with TPT job script generation tools
![Page 4: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/4.jpg)
Job Script Example Before Simplified Syntax
DEFINE JOB File_LoadDESCRIPTION ‘Load table from flat file’( DEFINE SCHEMA Customer_Accounts ( Account_Number INTEGER, Account_Name VARCHAR(50), Trans_Number INTEGER, Trans_Date ANSIDATE, Trans_Amount VARCHAR(20) );
DEFINE OPERATOR File_Reader TYPE DATACONNECTOR PRODUCER SCHEMA Customer_Accounts ATTRIBUTES ( VARCHAR FileName = @Filename, VARCHAR Format = @Format, VARCHAR OpenMode = @Openmode, VARCHAR TextDelimiter = @Delimiter, VARCHAR PrivateLogName = @DCLog );
/* continued on next page */
> Declarative Section- Defines the “what”:
DEFINE SCHEMA object(s)
DEFINE OPERATOR object(s)
• Two main script sections
![Page 5: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/5.jpg)
Job Script Example Before Simplified Syntax (continued)
/* continued from previous page */
DEFINE OPERATOR Load_Op TYPE LOAD SCHEMA * ATTRIBUTES ( VARCHAR TdpId = @TdpId, VARCHAR UserName = @UserName, VARCHAR UserPassword = @UserPW, VARCHAR TargetTable = @TarTable, VARCHAR LogTable = @LogTable, VARCHAR PrivateLogName = @LoadLog );
APPLY ‘INSERT INTO TABLE_X VALUES (:Account_Number, :Account_Name, :Trans_Number, :Trans_Date, :Trans_Amount);’ TO OPERATOR( Load_Op[2] ) SELECT * FROM OPERATOR( File_Reader ););
> Executive Section- Defines the “how”
APPLY statement(s)
![Page 6: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/6.jpg)
Job Variables File
Tdpid = ‘drill’,Username = ‘johndoe’,Userpw = ‘janedoe’,TarTable = ‘TABLE_X’,LogTable = ‘TABLE_X_LOG’,LoadLog = ‘load.log’,Filename = ‘flatfile1.txt’,Format = ‘formatted’,Openmode = ‘read’,TextDelimiter = ‘|’,DCLog = ‘dc.log’
• Stores and specifies common operator attributes> Reusable across multiple job scripts
![Page 7: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/7.jpg)
Job Variables
• Defined as <name> = ‘value’ pair in ”-u” command line option, external job variables file, or in job script SET directive
• Once defined, @<name> can be used anywhere in job script to specify ‘value’ (except within quoted strings and comments)
• Most commonly used to assign values to operator attributes
• Script users are encouraged to use job variables for easier script maintenance.
• TPT simplified syntax will rely heavily on predefined job variable names
![Page 8: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/8.jpg)
Simplifying TPT Syntax
• Our goal? To make job scripts simpler by:> Removing the declarative section> Reusing common job variables> Reducing potential keystroke errors
• DEFINE OPERATOR object:> Using Operator Templates instead
• DEFINE SCHEMA object:> Using Generated Schemas instead
• SQL INSERT statement:> Using Generated SQL INSERT Statements instead
![Page 9: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/9.jpg)
Using Operator Templates
• What are operator templates?> Stored DEFINE OPERATOR statements for all TPT operators> Uses formulaic names for the operators they define:
- $EXPORT, $LOAD, $UPDATE, $STREAM, etc.
> All possible operator attributes are declared> Each operator attribute has a job variable reference as its value> Standardizes job variable names that correspond to operator
attributes
![Page 10: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/10.jpg)
Operator Template Example
DEFINE OPERATOR $DDLDESCRIPTION 'Teradata Parallel Transporter DDL Operator'TYPE DDLATTRIBUTES(
VARCHAR UserName = @TargetUserName,VARCHAR UserPassword = @TargetUserPassword,VARCHAR TdpId = @TargetTdpId,VARCHAR AccountId = @TargetAccountId,VARCHAR WorkingDatabase = @TargetWorkingDatabase,VARCHAR LogonMech = @TargetLogonMech,VARCHAR LogonMechData = @TargetLogonMechData,VARCHAR DataEncryption = @DDLDataEncryption,VARCHAR ARRAY ErrorList = @DDLErrorList,VARCHAR LogSQL = @DDLLogSQL,VARCHAR PrivateLogName = @DDLPrivateLogName,VARCHAR ARRAY QueryBandSessInfo = @DDLQueryBandSessInfo,VARCHAR ReplicationOverride = @DDLReplicationOverride,VARCHAR ARRAY TraceLevel = @DDLTraceLevel
);
![Page 11: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/11.jpg)
Referencing Templates in Your Script
• Templates are imported into the job script when operators are referenced in an APPLY statement using their template name, for example:
APPLY ‘INSERT INTO TABLE_X (:col1, :col2);’
TO OPERATOR ($LOAD);
• By referencing the $LOAD operator template name, TPT knows to import the corresponding template
![Page 12: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/12.jpg)
Using Template Job Variables
• Templates contain predefined, conventionally-named job variables assigned to each operator attribute
• Assign predefined job variables as <name> = ‘value’ pair, for example:
SourceUserName = ‘johndoe’
SourceUserPassword = ‘janedoe’
• Unassigned job variables cause TPT to ignore value assignments
![Page 13: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/13.jpg)
Job Variables Naming Convention
• For producer templates: Logon-associated: Source<AttributeName>
VARCHAR UserName = @SourceUserName,
VARCHAR UserPassword = @SourceUserPassword,VARCHAR TdpId = @SourceTdpId
• For consumer templates: Logon-associated: Target<AttributeName>
VARCHAR UserName = @TargetUserName,
VARCHAR UserPassword = @TargetUserPassword,VARCHAR TdpId = @TargetTdpId
• All other job variables names: <TemplateName><AttributeName>
VARCHAR ARRAY ErrorList = @DDLErrorList,VARCHAR PrivateLogName = @DDLPrivateLogName
![Page 14: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/14.jpg)
User-Defined Templates
• Users can create their own templates to maximize usefulness:> Template name must begin with ‘$’ and be unique> Use SCHEMA * for consumer template and SCHEMA $ for
producer template> All operator attributes not assigned fixed values should be
assigned predefined, conventionally-named job variables> Store template in $TWB_ROOT/template directory
![Page 15: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/15.jpg)
User-Defined Operator Template Example
DEFINE OPERATOR $MY_DELIMITED_READERDESCRIPTION ‘My user-defined delimited file reader’TYPE DATACONNECTOR PRODUCERSCHEMA $ATTRIBUTES(
VARCHAR FileName = @SourceFileName,VARCHAR Format = ‘Delimited’, VARCHAR TextDelimiter = ‘,’,INTEGER IOBufferSize = @FileReaderIOBufferSize,
: : : :
VARCHAR PrivateLogName = @FileReaderPrivateLogName,VARCHAR TraceLevel = @FileReaderSkipRows,
);
![Page 16: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/16.jpg)
Using Generated Schemas
• All producer operators require a schema specification. How does TPT simplified syntax overcome this?> Explicitly generated schemas
- A simpler way of defining a schema object via a DBS table name and allowing TPT to auto-generate the column definitions
> Inferred generated schemas- A method by which TPT analyzes job step information to auto-
generate a schema object and its column definitions
![Page 17: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/17.jpg)
Explicitly Generated Schemas
• Explicitly specify the name of a DBS table in DEFINE SCHEMA to:> auto-generate a schema, for example:
DEFINE SCHEMA WEEKLY_TRANSACTIONS ‘Weekly_Trans’;
> auto-generate a delimited-file schema, for example:DEFINE SCHEMA WEEKLY_TRANSACTIONS DELIMITED ‘Weekly_Trans’;
• Explicitly specify the name of a DBS table in the APPLY statement’s producer operator reference to:> auto-generate a schema, for example:
APPLY ‘INSERT INTO TABLE_X (:col1, :col2);’ TO OPERATOR ($LOAD);FROM OPERATOR ($EXPORT (‘TABLE_X’));
> auto-generate a delimited-file schema, for example: APPLY ‘INSERT INTO TABLE_X (:col1, :col2);’ TO OPERATOR ($LOAD);FROM OPERATOR ($EXPORT (DELIMITED ‘TABLE_X’));
![Page 18: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/18.jpg)
Inferred Generated Schemas
• What is an inferred schema?> The schema of a producer template operator whose column
content can be inferred from information in the same job step
• How does TPT infer a schema? > By analyzing the script information of all operators invoked in
any job step that employs one of more producer templates
![Page 19: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/19.jpg)
Inferred Generated Schemas: Example 1
STEP LOAD_2(
APPLY <DML statement> TO OPERATOR( $LOAD )SELECT * FROM OPERATOR( $EXPORT ) UNION ALLSELECT * FROM OPERATOR( EXPORT_OP_2 );
);
• Producer Operator $EXPORT:> is an operator template instantiation> does not have an explicit schema
• Producer Operator EXPORT_OP_2:> is defined previously in the job script> has a script-defined schema
• Source data from both producers is merged into a single input data stream
• Thus, TPT will “infer” the same schema for both operators
![Page 20: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/20.jpg)
Inferred Generated Schemas: Example 2
STEP INSERT_DAILY_TRANS(
APPLY <DML statement> TO OPERATOR( $LOAD )SELECT * FROM OPERATOR( $SELECTOR ATTR( SelectStmt = ‘SELECT * FROM Daily_Trans;‘ ));
);
• Producer Operator $SELECTOR:> is an operator template instantiation> does not have an explicit schema
• The SelectStmt attribute has been assigned an SQL SELECT statement
• Thus, TPT will “infer” the schema based on the columns of the SELECT’s results table
• Works for producer templates $EXPORT and $SELECTOR, since both require the SelectStmt attribute
![Page 21: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/21.jpg)
Inferred Generated Schemas: Example 3
STEP INSERT_MONTHLY_SHIPMENTS(
APPLY <DML statement>TO OPERATOR( $LOAD ATTR( TargetTable = 'Monthly_Shipments‘ ))SELECT * FROM OPERATOR( $FILE_READER );
);
• Consumer operator $LOAD:> is an operator template instantiation> has a TargetTable attribute assigned to ‘Monthly_Shipments’
• Producer operator $FILE_READER:> is an operator template instantiation> does not have an explicit schema
• TPT will assume source data is loaded unchanged and “infer” the source schema based on the TargetTable
• Limitations:> Assumption may not be correct in all cases, resulting in schema mismatch causing job to terminate> TPT cannot infer a template’s schema from a target table when the job step contains multiple target tables
![Page 22: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/22.jpg)
How TPT Generates Schema
• Whenever TPT generates a schema based on a DBS table, it must:> Make a HELP TABLE call to DBS> Construct DEFINE SCHEMA object> Merge generated DEFINE SCHEMA object into job script> Substitute generated schema for the SCHEMA $ in producer
operator template
![Page 23: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/23.jpg)
Generated Schemas Advantages & Limitations
• Generated schemas> Major convenience when number of schema column definitions
is large> Reducing keyboarding time and keystroke errors> TPT job scripts simpler, more compact, easier to read
• Limitations> Requires the name of a DBS table that already exists prior to
running the job> Inferred schemas based on TargetTable attribute assumes
data loaded unchanged
![Page 24: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/24.jpg)
Using Generated SQL INSERT Statements
• TPT supports one additional feature to reduce script size and eliminate keystroke errors: Generating any SQL INSERT statement:> if the target table is specified or can be unambiguously determined
• For example:> DBS table ’Invoice_Counts’ has 4 columns col1, col2, col3, and col4> By specifying:
APPLY $INSERT 'Invoice_Counts' TO OPERATOR( $UPDATE )SELECT * FROM OPERATOR( $SELECTOR );
> TPT will replace $INSERT 'Invoice_Counts' with the following generated SQL INSERT statement:
'INSERT INTO Invoice_Counts VALUES ( :col1, :col2, :col3, :col4 );'
![Page 25: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/25.jpg)
Our Example Job Script – Simplified!
DEFINE JOB File_LoadDESCRIPTION ‘Load table from flat file’( DEFINE SCHEMA Customer_Accounts ( Account_Number INTEGER, Account_Name VARCHAR(50), Trans_Number INTEGER, Trans_Date ANSIDATE, Trans_Amount VARCHAR(20) );
DEFINE OPERATOR File_Reader TYPE DATACONNECTOR PRODUCER SCHEMA Customer_Accounts ATTRIBUTES ( VARCHAR FileName = @Filename, VARCHAR Format = @Format, VARCHAR OpenMode = @Openmode, VARCHAR TextDelimiter = @Delimiter, VARCHAR PrivateLogName = @DCLog ); /* continued on next page */
Before AfterBefore After
DEFINE JOB File_LoadDESCRIPTION ‘Load table from flat file’( APPLY $INSERT TO OPERATOR( $LOAD[2] ) SELECT * FROM OPERATOR( $FILE_READER ););
![Page 26: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/26.jpg)
Our Example Job Script – Simplified! (continued)
/* continued from previous page */
DEFINE OPERATOR Load_Op TYPE LOAD SCHEMA * ATTRIBUTES ( VARCHAR TdpId = @TdpId, VARCHAR UserName = @UserName, VARCHAR UserPassword = @UserPW, VARCHAR TargetTable = @TarTable, VARCHAR LogTable = @LogTable, VARCHAR PrivateLogName = @LoadLog );
APPLY ‘INSERT INTO TABLE_X VALUES (:Account_Number, :Account_Name, :Trans_Number, :Trans_Date, :Trans_Amount);’ TO OPERATOR( Load_Op[2] ) SELECT * FROM OPERATOR( File_Reader ););
Before AfterBefore After
DEFINE JOB File_LoadDESCRIPTION ‘Load table from flat file’( APPLY $INSERT TO OPERATOR( $LOAD[2] ) SELECT * FROM OPERATOR( $FILE_READER ););
![Page 27: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/27.jpg)
Job Variables Files
Tdpid = ‘drill’,Username = ‘johndoe’,Userpw = ‘janedoe’,TarTable = ‘TABLE_X’,LogTable = ‘TABLE_X_LOG’,LoadLog = ‘load.log’ Filename = ‘flatfile1.txt’,Format = ‘formatted’,Openmode = ‘read’,TextDelimiter = ‘|’,DCLog = ‘dc.log’
TargetTdpId = ‘drill’,TargetUserName = ‘johndoe’,TargetUserPassword = ‘janedoe’,LoadTargetTable = ‘TABLE_X’,LoadLogTable = ‘TABLE_X_LOG’,LoadPrivateLogName = ‘load.log’,FileReaderFileName = ‘flatfile1.txt’,FileReaderFormat = ‘formatted’,FileReaderOpenmode = ‘read’,FileReaderTextDelimiter = ‘|’,FileReaderPrivateLogName = ‘dc.log’
Before After, with Conventionalized Names
![Page 28: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/28.jpg)
Q & A
![Page 29: Teradata Parallel Transporter Scripts with Simplified Syntax](https://reader033.vdocuments.site/reader033/viewer/2022061420/56813c69550346895da5fab1/html5/thumbnails/29.jpg)
Thank you!