11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 1/8

Create an AccountLog In

Toolbox for IT Topics Business Intelligence Blogs

Tweet 1 0 1

How to Create, Use and Maintain DataStage 8Parameter SetsVincent McBurney Jun 25, 2008 | Comments (6)

This is a three part DataStage tutorial on using the new version 8 Parameter Setfunctionality that shows how it works and adds some practical advice for how to useit.

DataStage 8.0.1 came with a great new function called Parameter Sets that let yougroup your DataStage and QualityStage job parameters and store default values infiles.

There are some very good things about Parameter Sets:

You can group a set of related parameters together and add them in one click toany Sequence or Server or Parallel DataStage job and maintain them from asingle parameter definition.

Parameter Sets make Sequence jobs and Shared Containers easier and faster tobuild. When you add a Server or Parallel job to a Sequence Job you have a lotless properties to set if both jobs are synchronising parameters via ParameterSets. When you add a Shared Container to a job you only need to link in theParameter Set name and not each individual parameter.

You take the storage of parameter values completely out of the hands ofindividual sequence, server or parallel jobs and put them into centralised files orobjects.

Parameter Help Text is given a new life. This field that is either blank or filledwith rubbish becomes more important with centralised Parameter Sets as youcan turn your Parameter Set into a type of technical glossary of terms that notonly lists a set of parameters but offers a good definition of each.

I like the second benefit, an average project could have 1020 job parameters perjob once you have file, database, processing, date and source system jobparameters. When you add a job onto a Sequence job canvas you've got to passthrough every bloody parameter value via manual clicks, it takes 20 clicks to addand configure a job. Time consuming when you just want to throw together aSequence job for some testing. With the arrival of a ParameterSet you do onevalue per ParameterSet rather than one value per Parameter many less clicks.

There are a couple drawbacks to ParameterSets that we will expand on later:

There still isn't a good interface for maintaining parameter values for supportstaff. ParameterSet values can be modified directly against a text file but not ifthe value is encrypted. The only GUI or browser interfaces for maintainingparameter values is the Designer or Administrator tool neither of which is easyto use for a part time support person.

There is no obvious place to save ParameterSets in the repository, you have tochoose or create a home for them and this could lead to a bit ofdisorganisation. ParamterSet files can only be saved under a DataStageProject directory when it would be better to be able to browse and choose alocation that is easier for support staff to find.

The ParameterSet object doesn't allow comments or description or any type ofhelp text.

Creating Parameter Sets

A Parameter Set is a way to group a set of parameters into a single reusabledefinition that you can add to your jobs. Parameter sets let you save the values for

Get YourCloudTogethersoftlayer.…A Single

1Recommend Share

Your email address FOLLOW

BEGIN NOW

Work With Me

Links

Encryptyourfilesswi5t.comProtectyourprivacy.Easy andfree.

Tooling Around in the IBM InfoSphereby Vincent McBurney

Vincent McBurney is an IBM Champion for InformationIntegration and has been blogging for many years onInfoSphere software and ... more

Receive the latest blog posts:

Share Your PerspectiveShare your professional knowledge andexperience with peers. Start a blog on Toolbox forIT today!

If you are an expert in InfoSphere software and want to workfor the biggest IBM partner in Australia and New Zealandget in touch with me via ITToolbox or Linked In.

Steal This IM Methodology Informatica Data Quality Blog DataFlux Community of Experts Data Governance Blog

11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 2/8

a set in a file or in an object in the DataStage repository.

To create a Parameter Set use the "New" menu or toolbar option and findParameter Sets under the "Other" or "Recently Used" folders. Choose a shortParameter Set name because you will need to use it throughout your job as aparameter prefix: #parameter_set.parameter_name#. Your parameter set nameshould be short and sweet and your parameter names can be longer anddescriptive.

In the Add Parameter Set form type the parameters that belong in a single set intothe Add Parameter Set screen just like normal job parameters, note that you havetwo fields for operators and/or developers, one for the Prompt you see when yourun the job and one for the Help Text when you click "Property Help" from the JobRun screen:

I suggest these Prompt and Help Text fields be used as technical instructions onhow to use the parameter.

On the "Values" tab you can specify one or more files to save values to and bydefault it copies across the values from the previous tab. This is an optional tab,you can put values into a file or keep the values in the Parameter Set repositoryobject:

You can add more than one file. The columns you see are the parameter valuescopied across from the previous tab. When you specify multiple files you arecreating multiple scenarios to be selected at run time. A job can be run with adifferent set of file values depending on the parameter file name passed into the jobor selected from a drop down list by the operator.

You could use this feature if you are in a dev or test environment and you hadmultiple source databases to choose from, say a small database for a quick test or afull database for a performance test. One file could be called DB_SMALL_DEVand the other DB_LARGE_DEV with the DB connection values in each.

I'm not sure what use it could have in production where you want to be quite certainabout what parameters a job should use and don't want an operator trying tochoose the right file.

When you save your Parameter Set you choose a location in your DataStagerepository for them to be saved into the repository. So I don't lose them I create a"ParameterSets" folder under the Job folder to save them close to the DataStagejobs. In DataStage 8.0.1 you can save Parameter Sets anywhere, you can savethem in Jobs or Table Definitions or even Stage Types. There are lots of stupidplaces to save them so create a folder for them that makes sense and put them all

Categories

Big Data GO

dq:view Steve Tuck on Data Quality

Blogs Discussions Research Directory

11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 3/8

there.Â

Under the covers DataStage saves the option Parameter Set Values in the location$PROJECTHOME/ParameterSets/ParameterSetName/ValueFileName. TheValues file is a plain text file where encrypted values are converted to printablemashed text:

MY_DB_SERVER=OVERTHERE MY_DB_NAME=MYDATA MY_DB_LOGIN=MURRAY MY_DB_PASSWORD=L59@A;V1=9JM06E

You can see that this file would be easy to modify directly or manage from a simpleGUI application except for the encrypted password value.

When you run the job that uses a Parameter Set you get three choices for choosingjob parameters values:

1.

File Values you run the job with parameter values from the Parameter Set file bychoosing the Parameter Set file name from the "Value" field next to the ParameterSet name. At this point if you have multiple files to chose from (eg. multiple sourcedatabases or source instances) you can choose the right file.

Â

2.

Parameter Set object Values you can go with the values stored on the ParameterSet object in the repository by choosing "predefined" from the Value field next to theParameter Set name.

3. User override the person running the job can override the value of anyparameter by typing in a value next to the individual parameter names. This workswith either file or predefined usage shown above.

One of the biggest improvements brought by Parameter Sets is the simplification ofSequence Jobs. In DataStage 6 you had to add a Server/Parallel and then setevery single parameter value which could be quite time consuming if you had 20 ormore parameters per job. You could then copy and paste the stage and switch jobnames and keep those parameter values. In DataStage 7.5.x they took away thiscopy and paste feature and each time you changed the job name on a job stage theparameter values got wiped out. So for every job you added to a Sequence jobyou had to painstakingly set all the job parameter values there were no shortcuts,no auto mapping or auto setting.

11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 4/8

Parameter Sets make Sequence Jobs easy again by only requiring you to set adefault behaviour for the Parameter Set to either "UserDefined" (take it from theParameter Set object) or File (take it from the Parameter Set file):

If you have parameters that you want to dynamically generate such as processingdates, last key used, process id you would set this up as a normal job parameter inthe parallel job so it can be retrieved and set in the Sequence job and passed in asa parameter override.Â

Using Parameters from Parameter Sets

When you use a Parameter Set parameter in a job you refer to it using theParameter Set name as a prefix: #parameter_set.parameter_name#. When astage property window has an "Insert Parameter" button with a popup list ofparameters you will see the list with both the parameter set name and parametername in alphabetic order so it's easy to scroll to the right parameter set.

When you call a job from the command line you can specify whether to setParameter Set values from the object or from the file. To override values you needto refer to the full parameter name: dsjob run paramParameterSetName=ParameterFileName param ParameterSet.ParameterName =OverrideValue

If you try to override an encrypted parameter value from outside of a DataStageproduct you will lose the encryption the override will work but the value will show upin the DataStage log in plain text.Â

Maintaining Parameter Sets

The major drawback of Parameter Sets is that a DataStage Support person whomight have only a passing knowledge of the tools has no easy way to changeencrypted values in the Parameter Set file. The primary tool of use for aDataStage Support Dude is the Director and unfortunately IBM have forgotten toput any Parameter Set maintenance tools into the Director. You can set JobParameter Defaults in Director but not if those Parameters are in a Parameter Set. Parameter Sets are kind of invisible to the Director tool.Â

This leaves just two ways to change a Parameter Set default modify the ParameterSet object or modify the underlying file.

If you modify the Parameter Set file directly you open it up to manual mistakes andyou cannot change encrypted values. Putting passwords into a file without anyencryption is a big security no no. Not only will they be exposed in this file but theymay turn up in DataStage Director log messages. So even if you did keep this filesecure, which is difficult to do since DataStage needs read access to it, you can stillexpose it in log messages that almost anyone can get access to.

If you modify the Parameter Set values from the DataStage Designer you can useDataStage encryption to protect passwords and use the grid for data entry. It's a

11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 5/8

technically safe way to do it but it means giving the Designer tool to your productionsupport team which is overkill. They need an operations tool not a complexdesigner tool.

What we need in DataStage 8.1 is a way to change Parameter default values andencrypted values via the Director tool and/or via the Information Server console. So a member of a support team who doesn't really know much about DataStage canfollow a set of instructions to log into a DataStage support tool to change a databasepassword on a regular interval.

Parameter Set Ideas

I covered Job Parameter Ideas in a previous post about some of the uses of normaljob parameters. Now it's time to beef this up and organise parameters intoparameter sets.

Here are some job parameters that should not be put in parameter sets becauseyou want to set them from a Sequence job or hard coded to a default value in aparallel job and not pulled from a central parameter set file:

Process Id

Process Date / Business Date / Transaction Date

Last Surrogate Key Used

Source System ID

File Name

These are the Parameter Set groupings where every parameter set has a prefix ofPS_ to make them easier to find in repository searches, what you can do inParameter Sets is overload a job with parameters, adding parameters the jobdoesn't necessarily need to cut back the number of Parameter Sets that need to befound. The overloading doesn't hurt as you don't need to set every singleparameter value in the Sequence job any more:

PS_DB_????_CONNECTDB_????_SERVER

DB_????_NAME

DB_????_USERID

DB_????_PASSWORD

When you have a database parameter set you have some type of databaseidentifier as a prefix (eg 4 characters) that you use in every parameter setname and every parameter in that set. That makes the parameter namesunique and easier to use, especially when a job connects to severaldatabases.

PS_DIRECTORIESDIR_SOURCE_FILES

DIR_REJECT_FILES

DIR_ARCHIVE_FILES

DIR_DATASETS

DIR_LOOKUP_FILESETS

Put all your directories into one Parameter Set to make them easier to find.

PS_SCD_CONSTANTSSCD_HIGH_DATE

SCD_NULL_STRING_DEFAULT

SCD_NULL_INTEGER_DEFAULT

All constants used in a slowly changing dimension transformer stage.

PS_CUSTOMER_DEFAULTSNULL_CUSTOMER_BIRTHDATE_DEFAULT

NULL_CUSTOMER_SURNAME_DEFAULT

NULL_CUSTOMER_GENDER_DEFAULT

The values that get put into blank fields in a customer record, where ever itgets created.

PS_DB_????_LOOKUP_TABLESDB_????_LOOKUP_CUSTOMERS

DB_????_LOOKUP_PRODUCTS

DB_????_LOOKUP_STAFF

Let's say you have lookups that are used in a lot of different jobs but thetables have names that are hard to remember. A Parameter Set can holdthe name of every lookup table with a default value and hint text that tells youwhat the table is called and how to use it. A Parameter Set then becomes a

11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 6/8

More White Papers

6 Comments

Read 6 comments

type of technical glossary.

In my next Parameter Set Series post I'll look at Environment Variables in ParameterSets with a lot of new ideas for grouping parameters.

Disclaimer: The opinions expressed herein are my own personal opinions and do not representmy employer's view in any way.

Vincent McBurney is an IBM Information Champion for Information Integration.

Popular White Paper On This Topic Business Intelligence with SharePoint 2010

Related White Papers The State of Converged Infrastructure in 2013 Premise PBX Comparison Guide SMB Data Networking Playbook

USER_2195744 Oct 21, 2010

I do not understand because they mentioned a lot. What I need is how and whenwe use the parameter sets in Datastage? Can you please send the solution for thisquestion to my mail?

sumoka Nov 8, 2010

Hi Vincent ,

your article is of great help thanks a lot!

Though I have a query, is it possible to see the password in the director log.. even ifit is encrypted & not overridden like u mentioned ?

If yes.. how to prevent it from being exposed ?

Thansk in advance !

Robert Edis Nov 30, 2010

I like parameter sets as in theory I can create a generalised set of jobs. Forexample I have several lookup tables to be updated from a data source. All of thesetables has the same structure (KEY, CODE, DESC, etc) though the column namesare matched to the table name.

I should in theory be able to call the parallel job from a sequence job in a loop witheach occurence using a different psv file.

Still working though the kinks (I'm a DS novice). Do you see any drawbacks to thisidea?

sarmistha sarkar Jan 16, 2012

Can U provide me a detailed command line example using parameterset file?

USER_2667656 May 30, 2012

is it possible to pass a value to Parameter Set at run time??

haylo75 Jul 31, 2012

Very nice article on the possibilities with Parameter Sets. I'm new to DS, but not tomaintaining and supporting very large projects. I also like your Parameter Setideas, which to me boil down to a couple things:

1 Avoid placing parameters whose values frequently change

11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 7/8

SUBMITPREVIEW

Browse all IT Blogs

We Recommend

Danger Will Robinson: Has My TabletBeen Hacked?

Why PostgreSQL Doesn't Have Query

From Around The Web

Golden Star Thali (Happy Trips)

There's Amazing Content Online. How toKnow What to Read Next (Sphere)

2 Overloading 'nuff said 3 Apply a sort of Apps Hungarian notation to a PS naming scheme

With that said, I have an alternate thought with respect to logically identicalparameters in PS_DB sets. Is it not redundant to also have the "PS_DB" prefixapplied to the parameters within the set? Perhaps this is my newness to DSspeaking, and maybe it's repeated in the parameter name itself to allow for bettersearching?

Again, great read! I need to find your next blog in the Parameter Set series.

Leave a Comment

Connect to this blog to be notified of new entries.

Name Your email address

You are not logged in. Sign In to post unmoderated comments. Join the community to create your free profile today.

Want to read more from Vincent McBurney? Check out the blog archive.

Archive Category: DataStageKeyword Tags: datastage parameter set best practices tutorial advice job parameter

Disclaimer: Blog contents express the viewpoints of their independent authors and are not reviewed forcorrectness or accuracy by Toolbox for IT. Any opinions, comments, solutions or other commentaryexpressed by blog authors are not endorsed or recommended by Toolbox for IT or any vendor. If you feel ablog entry is inappropriate, click here to notify Toolbox for IT.

From Around The Web

Recommended by

Who’s the idiot?(LiveInStyle.com)

Try This Gmail AppFor Two Days YouWon't Regret It(Frank Addante)

Slideshow: 5 WaysFrugal Pet OwnersCan Save(Wably)

Maruti YRA hatchbackto take on the HyundaiElite i20(Zigwheels)

Slideshow: 22Amazing Photobombs& Photo Fails(Wably)

Slideshow: 30Funniest Auto CorrectFails(Wably)

Let my folder go!(Spiraling into control)

How to "Trim the Fat"on eDiscovery Costs(Ricoh Services)

11/21/2014 How to Create, Use and Maintain DataStage 8 Parameter Sets

http://it.toolbox.com/blogs/infosphere/howtocreateuseandmaintaindatastage8parametersets25604 8/8

Hints

Is InMemory Computing CostEffectivefor My Small Business?

Reducing Your Organization's EnergyFootprint by Going Green IT

Boost Windows performance with thesefive apps

CEOtoCEO Top 5 Questions CEOsShould Ask Themselves & Board AboutRisk Management

Slideshow: Best Hometown PeanutButter Recipes (Wably)

Slideshow: 16 Things People Need toStop Saying (Wably)

Sending data from Unity (Spiraling intocontrol)

Recommended by

Collaboration ToolsDiscussion GroupsBlogsWiki

Toolbox for IT

My HomeTopicsPeopleCompaniesJobsWhite Paper Library

Follow Toolbox.comToolbox for IT onTwitterToolbox.com on TwitterToolbox.com onFacebook

Data CenterData Center

DevelopmentC LanguagesJavaVisual BasicWeb Design & Development

Enterprise ApplicationsCRMERPPeopleSoftSAPSCMSiebel

Enterprise Architecture & EAIEnterprise Architecture & EAI

Information ManagementBusiness IntelligenceDatabaseData WarehouseKnowledge ManagementOracle

IT Management & StrategyEmerging Technology & TrendsIT Management & StrategyProject & Portfolio Management

Cloud ComputingCloud Computing

Networking & InfrastructureHardwareNetworkingCommunications Technology

Operating SystemsLinuxUNIXWindows

SecuritySecurity

StorageStorage

Topics on Toolbox for IT Toolbox.com

AboutNewsPrivacyTerms of UseWork at Toolbox.comAdvertiseContact usProvide Feedback

Help TopicsTechnical SupportPCMag Digital Group AdChoices

Other Communities

Toolbox for HRToolbox for Finance

Copyright 19982014 Ziff Davis, LLC (Toolbox.com). All rights reserved. All product names are trademarks of their respective companies. Toolbox.com is notaffiliated with or endorsed by any company listed at this site.