bb2011 (fp) ebb scripting language 01€¦ · the name of the field the description of the field ...

Commission européenne, B-1049 Bruxelles / Europese Commissie, B-1049 Brussel - Belgium. Telephone: (32-2) 299 11 11. Commission européenne, L-2920 Luxembourg. Telephone: (352) 43 01-1.

EUROPEAN COMMISSION DIRECTORATE-GENERAL INFORMATICS Information systems Directorate

European Commission

"Building Blocks Evolutionary and Corrective Maintenance - EBB Scripting Language

Specification"

Date: 28/09/2012

Version: 1.80

Reference Number: D1.7.1

"Building Blocks Evolutionary and Corrective Maintenance - EBB Scripting Language Specification" / 53 Document Version 1.80 dated 28/09/2012

TABLE OF CONTENTS

1. TERMINOLOGY ..................................................................................................................................... 5

1.1. Abbreviations and Acronyms ................................................................................................................... 5

1.2. Syntax Definition ..................................................................................................................................... 5

2. OVERVIEW OF THE SCRIPTING LANGUAGE ............................................................................... 6

3. FUNCTIONALITY OVERVIEW ........................................................................................................... 6

4. DETAILED FUNCTIONALITY DESCRIPTION ................................................................................. 7

4.1. Format Definition ..................................................................................................................................... 7

4.2. Program Definition .................................................................................................................................. 8

4.3. Steps Definition...................................................................................................................................... 11

4.4. Auxiliary Datasets .................................................................................................................................. 13

4.4.1. Simplified Lookup .............................................................................................................................. 13

4.4.2. Simplified Transcode .......................................................................................................................... 13

4.5. Dataset Operation Definition ................................................................................................................. 14

4.5.1. Aggregation ......................................................................................................................................... 14

4.5.2. Alter .................................................................................................................................................... 15

4.5.3. Append ................................................................................................................................................ 17

4.5.4. Copy .................................................................................................................................................... 18

4.5.5. Keysets Template ................................................................................................................................ 18

4.5.6. Merge .................................................................................................................................................. 19

4.5.7. Missing Values .................................................................................................................................... 20

4.5.8. Hierarchical Merge ............................................................................................................................. 21

4.5.9. Transpose ............................................................................................................................................ 22

4.5.10. Reorder ............................................................................................................................................. 23

4.5.11. Select ................................................................................................................................................. 24

4.5.12. Sort .................................................................................................................................................... 25

4.5.13. Split ................................................................................................................................................... 26

4.5.14. Statistics ............................................................................................................................................ 26

4.5.15. Update ............................................................................................................................................... 27

4.5.16. Import................................................................................................................................................ 28

4.5.17. Export ............................................................................................................................................... 29

4.5.18. Comext .............................................................................................................................................. 31

4.5.19. Food Safety ....................................................................................................................................... 32

4.5.20. Growth Rate ...................................................................................................................................... 33

4.5.21. HBHistorical (Hidiroglou Berthelot) ................................................................................................ 34

4.5.22. LFS.................................................................................................................................................... 36

4.5.23. Sigma Gap ......................................................................................................................................... 37


4.5.24. Urban Method 3 ................................................................................................................................ 39

4.5.25. Time Series ....................................................................................................................................... 40

4.5.25.1. Input Format ................................................................................................................................... 40

4.5.25.2. Output Format ................................................................................................................................ 40

4.5.25.3. Timeseries operation ...................................................................................................................... 40

4.6. Validation Rules Definition ................................................................................................................... 41

4.6.1. No Record Rule................................................................................................................................... 41

4.6.2. Horizontal Rule ................................................................................................................................... 42

4.6.3. Vertical Rule ....................................................................................................................................... 43

4.6.4. Hierarchical Rule ................................................................................................................................ 44

4.7. Scripting Language Expressions Definition ........................................................................................... 45

4.7.1. Expression Fields ................................................................................................................................ 45

4.7.2. Data Types .......................................................................................................................................... 46

4.7.3. Operators ............................................................................................................................................. 46

4.7.4. Functions ............................................................................................................................................. 46

4.7.1. Variables ............................................................................................................................................. 49

4.8. Comments .............................................................................................................................................. 50

4.9. Multi-line Strings ................................................................................................................................... 50

4.10. New Line Characters ............................................................................................................................ 50

4.11. Keyword Abbreviations ....................................................................................................................... 51

4.12. Reserved Keywords ............................................................................................................................. 51


Document History

Version Date Comment Modified Pages

01.10 24/02/2011 Updated functions list, added reserved keywords

22, 26

01.20 11/10/2011 Constant definition, Definition of conditions, Export command, updated the sorting of reserved keywords

9-13, 21-22, 26

01.30 12/10/2011 Outlier detection 2,19-20

01.35 16/11/2011 Hierarchical Merge 15

01.40 20/11/2011 Added Transpose Operation, full review -

01.50 03/02/2011 Added NORECORD, Variables; Updated functions

26, 31, 34

01.60 07/02/2011 Added Missing Values, Alter, Keysets Template, Statistics; Sigma Gap, Hidiroglou-Berthelot Historical, Growth Rate, Food Safety, Comext, LFS, Urban Metod 3, Time Series

-

1.70 25/07/2012 Corrected errors in examples 9,12

1.80 28/09/2012 Added auxiliary datasets and simplified lookups

13,14


1. TERMINOLOGY

1.1. Abbreviations and Acronyms

Terms/Abbreviation Description

CLI Command Line Interface

PDL Program Definition Language

FDL Format Definition Language

ESLE EDIT Scripting Language Expression

1.2. Syntax Definition

The following rules will be used during the definition of the syntax of the Program Definition Language:

Definition Description

ItalicText Keyword

[...] Optional element

[...]* Optional repeatable element

[...]+ Repeatable element with at least one occurrence

[ItalicText] Optional Keyword

[ItalicText ...]+ Repeatable keywords, see also repeatable parameters. Elements in the enumeration will not be separated by any special character.

[ItalicText1… | ItalicText2… |… ]

Choice between two or more elements.

[ItalicText1… | ItalicText2… |… ]?

Optional choice between two or more elements.

<Text> Parameter

[<Text>] Optional Parameter

[<Text>]+ Repeatable parameters. The elements in the enumeration will be separated by comma. Ex: Key1, Key2, Key3

[<ESLE>]+ Block of EDIT Scripting Language Expressions. The expressions in the block will be separated by semicolon.

Ex:

product := “testProd”; value := value + 1;


2. OVERVIEW OF THE SCRIPTING LANGUAGE

The EBB System offers the possibility to store the definition of formats or programs in text files. These definition files can be imported into the System through the Command Line Interface or the Web Application. The language used to create format definition files and program definition files is called the EBB Scripting Language.

The language used to define a format in a definition file is called the Format Definition Language. The FDL has similar functional capabilities as the GUI, allowing the user to define a format or a file format which contains various types of fields.

For consistency purposes the users are advised to use “.FDL” for the extension of the files in which they store format or file format definitions.

An FDL file can contain definitions for multiple formats.

The language used to define a program in a definition file is called the Program Definition Language. The PDL has similar functional capabilities as the GUI, allowing the user to define a program which contains various types of steps and rules.

For consistency purposes the users are advised to use “.PDL” for the extension of the files in which they store program definitions.

A PDL file can contain definitions for multiple programs.

3. FUNCTIONALITY OVERVIEW

Format Definition Language:

1. Format Definition

Program Definition Language:

2. Program Definition

3. Steps Definition

4. Dataset Operations Definition

5. Rules Definition

Auxiliary Functions:

6. Comments

7. Multiline Strings

8. Multi-line Commands

9. New Line Characters

10. Keyword Abbreviations


4. DETAILED FUNCTIONALITY DESCRIPTION

4.1. Format Definition

The following syntax is used to define a format:

SYNTAX :

FORMAT <FORMAT_NAME> {

[DESCRIPTION <FORMAT_DESCRIPTION>;]

FIELDS {

<FIELD_NAME> {

[DESCRIPTION <FIELD_DESCRIPTION>;]

CAPTION <CAPTION>;

[DEFAULTVALUE <VALUE>;]

TYPE [STRING | NUMBER | DOUBLE | BOOLEAN];

LENGTH <LENGTH> [.<PRECISION>] [MANDATORY];

}

}

}

PARAMETERS:

<FORMAT_NAME> The name of the new format

<FORMAT_DESCRIPTION> The description of the new format

<FIELD_NAME> The name of the field

<FIELD_DESCRIPTION> The description of the field

<CAPTION> Caption to be used for the field

<VALUE> String or numeric value

<LENGTH> Total length of the field

<PRECISION> Precision to be used by the DOUBLE type

EX:

FORMAT MyFormat {

DESCRIPTION “This is my format“;

FIELDS {

Year {

CAPTION “Year”;

TYPE NUMBER;

LENGTH 4;

}


Value {

CAPTION Value;

TYPE DOUBLE;

LENGTH 6.2 MANDATORY;

}

}

}

4.2. Program Definition

The following syntax is used to define a program:

SYNTAX:

PROGRAM <PROGRAM_NAME> {

[DESCRIPTION <PROGRAM_DESCRIPTION>;]

[INPUT <DATASET_FORMAT_NAME>

<DATASET_REFERENCE_VARIABLE_NAME>

[<CONSTANT_DATASET_INSTANCE>];]+

[KEYSET <KEYSET_NAME> [<FIELD_NAME>]+]*

[PARAMETER <PARAMETER_NAME> <FORMAT_FIELD>;]*

[PARAMETERSET <PARAMETER_SET_NAME> {

[<PARAMETER_NAME>=<PARAMETER_VALUE>;]+

} ]

[CONSTANT [STRING | BOOLEAN | NUMBER | DOUBLE] <CONSTANT_NAME> := <CONSTANT_VALUE>; ]*

STEPS {

<STEPS_DEFINITION_SCRIPT>

}

}

PARAMETERS:

<PROGRAM_NAME> The name of the new program

<PROGRAM_DESCRIPTION> The description of the new program

<DATASET_FORMAT_NAME> Name of the dataset format

<DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset reference variable

< CONSTANT_DATASET_INSTANCE> Name of an existing dataset instance to be used as a constant input for this program


<KEYSET_NAME> Name for a reusable keyset that groups fields from a format

<PARAMETER_NAME> Name of the parameter

<FORMAT_FIELD> Format field representing the type of the parameter

<PARAMETER_SET_NAME> Name of the parameter set

<PARAMETER_VALUE> Value of the parameter

<CONSTANT_NAME> Name of the constant

<CONSTANT_VALUE> Value of the constant

<STEPS_DEFINITION_SCRIPT> Block of script containing the definition of the program steps

EX :

PROGRAM p01 {

DESCRIPTION “Monthly trades program”;

INPUT inputTrades MonthlyTradesFormat;

PARAMETER pContinent continent;

PARAMETER pYear year;

PARAMETERSET mypset {

pContinent=Asia;

pYear=1500;

}

CONSTANT STRING const1 := “test 23”;

STEPS {

DATAOPERATION copyDS {

COPY {

INPUT inputTrades;

OUTPUT tradesCopy;

}

}

VALIDATION {

INPUT tradesCopy;

ERROR errDs;

ERRORDATA errData;

RULES {

RECORD countryNotEmpty {

CONDITION country != "";

ERRMSG 'failed' SEVERITY Error;


}

RECORD checkChinaImports{

PRECONDITION country = "CHINA" AND flow = “I”;

CONDITION quantity = 0 AND value = 0;

ELSE {quantity := 0; value := 0;}

}

RECORD recalculateChina {;

PRECONDITION country = "CHINA" AND flow=”E”;

CONDITION value < 10000000;

THEN {

value := value * 100;

quantity:= quantity *100;

}

ELSE {

value := value * 10;

quantity:=quantity * 10;

}

}

VERTICAL v01 {

EXPRESSION {

KEYS country, flow, month;

TRKEYS country;

CONDITION value[‘GB ‘] > 100000;

}


}

}

}

DATAOPERATION aggregate {

AGGREGATION {

INPUT tradesCopy;

OUTPUT aggregatedTrades;

CREATE NEW FORMAT annualTrades;

KEYS country, flow;

SUM(value); SUM(quantity); DELETE month;


}

}

}

}

4.3. Steps Definition

The steps script contains a series of definitions of steps that will be executed in the program. The steps can be either a dataset operation or a validation step.

Step conditions must be defined before dataset operation scripts for dataset operations and input definition for validation steps.

SYNTAX :

STEPS {

[DATAOPERATION <STEP_NAME> {

[CONDITION { <BOOLEAN_CONDITION>; }]

<DATASET_OPERATION_SCRIPT>

} |

VALIDATION <STEP_NAME> {

[CONDITION { <BOOLEAN_CONDITION>;}]

INPUT <DATASET_REFERENCE_VARIABLE_NAME>;

[LOOKUP <LOOKUP_DATASET_REFERENCE_VARIABLE_NAME>;]*

[AUX <AUXILIARY_DATASET_REFERENCE_VARIABLE_NAME>;]*

[ERROR <ERROR_ DATASET_REFERENCE_VARIABLE_NAME>;]

[ERRORDATA <ERROR_DATA_DATASET_REFERENCE_VARIABLE_NAME>;]

[VALIDDATA <VALID_DATA_DATASET_REFERENCE_VARIABLE_NAME>;]

RULES {

<VALIDATION_RULES_SCRIPT>

}

} ]

}

PARAMETERS : <STEP_NAME> Name of the step

<BOOLEAN_CONDITION> A BOOLEAN type condition (SL expression)

<DATASET_OPERATION_SCRIPT> Block of script containing the definition of a dataset operation


<DATASET_REFERENCE_VARIABLE_NAME> Name of the input dataset reference variable

<LOOKUP_DATASET_REFERENCE_VARIABLE_NAME> Name of a dataset reference variable to be prepared for a lookup operation in the current step

<AUXILIARY_DATASET_REFERENCE_VARIABLE_NAME> Name of a dataset reference variable to be prepared for use as an auxiliary dataset in the current step

<ERROR_ DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset reference variable where the errors will be stored

<ERROR_DATA_DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset reference variable to be used for storing the rows that failed validation

<VALID_DATA_DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset reference variable to be used for storing the rows that passed validation successfully

EX :

STEPS {

DATAOPERATION copyDS {

CONDITION { “a” = “b”;}

COPY {

INPUT inputTrades;

OUTPUT tradesCopy;

}

}

VALIDATION {

CONDITION { “c” <= “d”;}

INPUT tradesCopy;

ERROR errDs;

ERRORDATA errData;

RULES {

RECORD countryNotEmpty {

CONDITION country != "";


}

}

}


4.4. Auxiliary Datasets

Auxiliary datasets can be used for Simplified Lookup and Simplified Transcode operations based on a set of keys defined as the keyset of the auxiliary dataset.

SYNTAX:

VALIDATION validationStep {

INPUT …;

AUX <REFERENCE_NAME> (<AUX_KEY>)+;

…

}

PARAMETERS:

< REFERENCE_NAME > Name of the reference to be loaded as an auxiliary dataset

< AUX_KEY > Field to be used as a key of the auxiliary dataset

EX:

VALIDATION validationStep {

INPUT workData;

AUX lookupData CODE, MONTH;

ERROR errorLog;

RULES {

…

}

}

4.4.1. Simplified Lookup

The Simplified Lookup checks the existence of a combination of keys in an auxiliary dataset.

SYNTAX:

isNull(<AUX_REFERENCE_NAME>([‘<AUX_KEY_VALUE>’])+)

PARAMETERS:

< AUX_REFERENCE_NAME > Name of the auxiliary dataset reference

< AUX_KEY_VALUE > Value of the auxiliary key; can be an explicit value or a field from the current partition

EX:

isNull(lookupData[COUNTRY][‘JAN’])

4.4.2. Simplified Transcode

The Simplified transcode operation fetches a value from an auxiliary dataset corresponding to a set of specified value keys.


SYNTAX:

<AUX_REFERENCE_NAME>.<VALUE_COLUMN>([‘<TRANSPOSITION_VALUE>’])+)

PARAMETERS:

< VALUE_COLUMN > Value column to be returned from the auxiliary dataset

< AUX_REFERENCE_NAME > Name of the auxiliary dataset reference

< TRANSPOSITION_VALUE > Value of the transposition or a field from the current partition

EX:

lookupData.TARGET[COUNTRY][‘JAN’]

4.5. Dataset Operation Definition

4.5.1. Aggregation

The Aggregation operation performs a series of aggregations on an input dataset based on grouping functions and stores the data in the output dataset.

The format of the output for the Aggregation operation is defined as follows:

• The CREATE NEW option means that the dataset definition for the output will be created, and an error is given if the name is already used by a different format.

• The USE EXISTING option means that the format must exist in the database with the specified name.

SYNTAX :

AGGREGATION {


OUTPUT <OUT_DATASET_REFERENCE_VARIABLE_NAME>;

[CREATE NEW | USE EXISTING] FORMAT <FORMAT_NAME>;

KEYS [<FIELD_NAME>]+;

[<GROUPING_FUNCTION> (<ARGUMENTS>); ]+

[DELETE [<FIELD_NAME>]+ ;]

}

PARAMETERS :


<OUT_DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset reference variable to be used for storing the output data

<FORMAT_NAME> Name of the output format to be created/used


<FIELD_NAME> Field to be used as a key for a grouping function

<GROUPING_FUNCTION> (<ARGUMENTS>) Grouping function to be used. The supported grouping functions are:

Mean : mean(price) – accepts only number columns as parameter

Count : count(country) – accepts any column type as parameter

Sum : sum(value) – accepts only number columns as parameter

Max : max(price) – accepts only number columns as parameter

Min : min(price) – accepts only number columns as parameter

Variance : variance(value) – accepts only number columns as parameter

Standard deviation : stddev(quantity) – accepts only number columns as parameter

Median : median(price) – accepts only number columns as parameter

Weighted mean : wmean(price,quantity) – accepts only number columns as parameters

EX :

AGGREGATION

{

INPUT input1;

OUTPUT output1;

USE EXISTING FORMAT aggDD;

KEYS country;

sum(price);

count(product);

DELETE month,type,quantity,value;

}

4.5.2. Alter

The Alter operation alter the data from one dataset in order to drop or (and) create fields. The output is another dataset. The format of this new dataset can be a new or an existing (compatible) one.

SYNTAX :

ALTER{



[DROP FIELDS [<FIELD_NAME>]+;]

[CREATE FIELDS {


<FIELD_NAME> {

[DESC <FIELD_DESCRIPTION>;]

CAPTION <FIELD_CAPTION>;

MISSINGVALUES [<FIELD_MISSING_VALUE> ]+;

[DEFAULTVALUE <FIELD_DEFAULT_VALUE>;]

TYPE <FIELD_TYPE>;

LENGTH <FIELD_LENGTH>;

[MANDATORY;]

}

}]


}

PARAMETERS :



<FIELD_NAME> Name of the field to be dropped / created


< FIELD_DESCRIPTION > Description of the field

< FIELD_CAPTION > Caption of the field

<FIELD_MISSING_VALUE> Possible missing value (forms a list for a field)

<FIELD_DEFAULT_VALUE>

<FIELD_TYPE> Type of the field (can be one of String, Number, Double or Boolean)

<FIELD_LENGTH> Length of the field

MANDATORY Specify if the field it is mandatory or not (if a field is mandatory it cannot specify the default value)

EX :

ALTER

{

INPUT input1;

OUTPUT output1;

DROP FIELDS field1;

USE EXISTING FORMAT format1;

}


4.5.3. Append

The Append operation appends the data from one dataset to another dataset. In order to do this, the operation copies the first input dataset into the output dataset and then it appends the second input dataset to the output dataset.

If there are some missing fields in the second dataset and CREATE MISSING option is not present the operation will give an exception. If the CREATE MISSING option is present the missing fields will be filled with a default value (not specified). The FORCE CONVERSION option is being used in the case of column type mismatch.

The format of the output for the Append operation is defined as follows:



SYNTAX :

APPEND {

INPUT <FIRST_DATASET_REFERENCE_VARIABLE_NAME>;

INPUT <SECOND_DATASET_REFERENCE_VARIABLE_NAME>;



[CREATE MISSING;] [FORCE CONVERSION;]

}

PARAMETERS :

<FIRST_DATASET_REFERENCE_VARIABLE_NAME> Name of the first input dataset reference variable

<SECOND_DATASET_REFERENCE_VARIABLE_NAME> Name of the second input dataset reference variable



EX :

APPEND

{

INPUT input1;

INPUT input2;

OUTPUT output1;

CREATE MISSING;

FORCE CONVERSION;


}

4.5.4. Copy

The Copy operation copies the input dataset into the output dataset.

The IF NOT EMPTY option implies that the actual copying is being performed if the input dataset is not empty. If the output dataset is not empty, the data is being appended to the output.

The format of the output dataset must be a superset of the input format. The order of the columns may be different.

SYNTAX :

COPY {



[IF NOT EMPTY;]

}

PARAMETERS :


EX :

COPY

{

INPUT input1;

OUTPUT output1;

IF NOT EMPTY;

}

4.5.5. Keysets Template

The Keysets template operation adds a keyset template (together with a set of keysets) for an input dataset. The keysets must present a hierarchical order.

SYNTAX :

KEYSETSTEMPLATE {


[KEYSET <KEYSET_NAME> KEYS [<KEY_COLUMN>]+]+;

KEYSETSTEMPLATE <KEYSET_TEMPLATE_NAME>;

}

PARAMETERS :



<KEYSET_NAME> Name of the keyset

<KEY_COLUMN> Columns that forms the keyset (key columns)

<KEYSET_TEMPLATE_NAME> Name of the keyset template

EX :

KEYSETSTEMPLATE {

INPUT input1;

KEYSET keyset1 KEYS CONTINENT;

KEYSET keyset1 KEYS CONTINENT, COUNTRY;

KEYSETSTEMPLATE keysetTemplate1;

}

4.5.6. Merge The Merge operation performs a join between two input datasets based on a set of keys. The operation can be performed using four different types of join: INNER, LEFT, RIGHT and OUTER. Fields from the first input dataset can be overwritten if the OVERWRITE option is specified. Columns from the second input dataset can also be renamed as specified by the RENAME option. The format of the output for the Merge operation is defined as follows:



SYNTAX :

MERGE {






[INNER | LEFT | RIGHT | OUTER] JOIN;

[ [OVERWRITE | DONT OVERWRITE] EXISTING FIELDS ;

| RENAME {

[<FIELD_NAME> TO <NEW_FIELD_NAME>]+

} ]

}

PARAMETERS :






EX :

MERGE

{

INPUT input1;

INPUT input2;

OUTPUT output1;

USE EXISTING FORMAT countryMerge;

KEYS row_number;

LEFT JOIN;

RENAME { country TO mycountry ; price TO myprice; }

}

4.5.7. Missing Values

The Missing values operation detects the missing dates from the input dataset. Missing rows are inserted with the null values. This operation is partition-based as defined by the keys. The date column can be of type w(week), m(month), q(quarterly) and s(semester).

SYNTAX :

MISSINGVALUES {



KEYS [<KEY_COLUMN>]+;

AUX <TIME_COLUMN>;

}

PARAMETERS :



< KEY_COLUMN > Name of the key column (part of the partition)


< TIME _COLUMN > Name of the date column (the filling is performed based on the values of this column)

EX :

MISSINGVALUES

{

INPUT input1;

OUTPUT output1;

KEYS country, product;

AUX dateColumn;

}

4.5.8. Hierarchical Merge The Hierarchical Merge operation performs a join between many input datasets based on a set of keysets (the join between the datasets is described in the JOIN clause). After the new dataset is created, a keysets template is associated with it, defining the hierarchical structure of the dataset. SYNTAX :

HIERARCHICAL {

[INPUT <DATASET REFERENCE_NAME> KEYSET <KEYSET_NAME> [<COLUMN_NAME>]+ ]+;

JOIN {

[<KEYSET_NAME>.<COLUMN_NAME>=<KEYSET_NAME>.<COLUMN_NAME>]+; }

RENAME {

[<DATASET_REFERENCE_NAME> <COLUMN_NAME> <NEW_COLUMN_NAME>]+;

}

OUTPUT <OUTPUT_FORMAT>< OUT_DATASET_REFERENCE >;

[KEYSETSTEMPLATE <KEYSET_TEMPLATE_NAME>];

}

PARAMETERS :

< DATASET REFERENCE_NAME > Name of the input dataset reference variable

< KEYSET_NAME > Name of the keyset

< COLUMN_NAME > Name of the column(s) that are composing the keyset

< NEW_COLUMN_NAME > The new name of the column

< OUTPUT_FORMAT > The name of the format of the output dataset reference format

< OUT_DATASET_REFERENCE > The name of the output dataset reference format


< KEYSET_TEMPLATE_NAME > The name of the keyset template – all the described keysets are grouped under this name

EX :

HIERARCHICAL {

INPUT DATASET1 KEYSET ContinentalTrade CONTINENT;

INPUT DATASET2 KEYSET CountryTrade COUNTRY, CONTINENTNAME;

INPUT DATASET3 KEYSET CityTrade CITY, COUNTRYNAME;

INPUT DATASET4 KEYSET GroupTrade NGROUP, CONTINENT, CITY;

INPUT DATASET5 KEYSET SubgroupTrade SUBGROUP, NGROUP, NAME;

JOIN {

ContinentalTrade.CONTINENT = CountryTrade.CONTINENTNAME;

CountryTrade.COUNTRY = CityTrade.COUNTRYNAME;

ContinentalTrade.CONTINENT = GroupTrade.CONTINENT;

GroupTrade.NGROUP = SubgroupTrade.NGROUP,

GroupTrade.CITY = SubgroupTrade.INAME;

}

RENAME {

DATASET1 POLICY CONTINENTALPOLICY,

TVALUE CONTINENTALTVALUE;

DATASET2 POLICY COUNTRYPOLICY;

}

OUTPUT hMerge1 output1;

KEYSETSTEMPLATE kst1;

}

4.5.9. Transpose

The Transpose operation allows the possibility to transpose a given dataset from horizontal to vertical and from vertical to horizontal against a given a partition key. In case of the horizontal transposition the output format will determine the value columns to be transposed and the transposition values to be taken into account. The format must be predefined by the user and it must contain besides the partition columns, value columns with the following name convention: <COLUMN_NAME><SEPARATOR><TRANSPOSITION_VALUE> Ex: VALUE_EN


In the case of the vertical transposition, the value columns must be in accordance to the same naming convention above, to allow the application to extract the name of the target value column and the transposition value to be set on the resulting record.

SYNTAX :

TRANSPOSE {

TYPE ["toHorizontal"|”toVertical”]?;

INPUT <INPUT_DATASET>;

OUTPUT <OUTPUT_FORMAT> <OUTPUT_REFERENCE_NAME>;

KEYS [<PARTITION_KEY>]+;

TRKEYS <TRANSPOSITION_KEY>;

FIELDDELIMITER <FIELD_DELIMITER>;

}

PARAMETERS :

<INPUT_DATASET> the input dataset

<OUTPUT_FORMAT> the format of the output

<OUTPUT_REFERENCE_NAME> the name of the output dataset reference

<PARTITION_KEY> name of the partition key column

<TRANSPOSITION_KEY> name of the transposition key column

<FIELD_DELIMITER> delimiter to be used in the name of the transposed value columns

EX :

DATAOPERATION tranposeDataset {

TRANSPOSE {

TYPE "toHorizontal";

INPUT INPUT1;

OUTPUT newFormat1 OUTPUT1;

KEYS col1, col2, col3;

TRKEYS trkey1;

FIELDDELIMITER "_";

}

}

4.5.10. Reorder


The Reorder operation reorders the rows (changes the ROW_NUMBER) from an input dataset and stores them in the output dataset according to the FIELDS option. This is an equivalent to a Sort operation in which the fields are sorted ascending.

The format of the output for the Reorder operation is defined as follows:



SYNTAX :

REORDER {




FIELDS [<FIELD_NAME>]+;

}

PARAMETERS :




<FIELD_NAME> Name of a field to be used for the ordering of the rows

EX :

REORDER

{

INPUT input1;

OUTPUT output1;

CREATE NEW FORMAT countryDsd1;

FIELDS QUANTITY;

}

4.5.11. Select

The Select operation selects rows from an input dataset and writes them to the output dataset according to a specified where condition. The condition must be a valid SQL where clause.

The format of the output dataset is the same as the format of the input dataset.

SYNTAX:


SELECT {



CONDITION <SELECT_CONDITION>;

}

PARAMETERS:


reference variable


<SELECT_CONDITION> SQL where clause to be applied for the operation

EX:

SELECT

{

INPUT input1;

OUTPUT output1;

CONDITION where price > 100 and country='RO';

}

4.5.12. Sort

The Sort operation reorders the rows (changes the ROW_NUMBER) from an input dataset and stores them in the output dataset according to the FIELDS option and the specified sort direction.


SYNTAX:

SORT {



ORDER [<FIELD_NAME> [ASC | DESC] ]+;

}

PARAMETERS:



<FIELD_NAME> Name of a field to be used as a sorting key


EX:

SORT

{

INPUT input1;

OUTPUT input2;

ORDER COUNTRY ASC,VALUE DESC;

}

4.5.13. Split

The Split operation creates separate output datasets starting from an input dataset for a list of partitions based on a set of keys that determine the partitions list. For each partition, an output dataset will be created using the following naming convention:

<INPUT_DATASET_REFERENCE_VARIABLE_NAME>[_<KEY_VALUE>]+

Ex: input1_RO_100, input1_BG_100, input1_RO_200


SYNTAX:

SPLIT {



}

PARAMETERS:



<FIELD_NAME> Name of a field to be used as a key in defining a partition for the split operation

EX:

SPLIT

{

INPUT input1;

KEYS COUNTRY,PRICE ;

}

4.5.14. Statistics

The Statistics operation computes the statistics of the input dataset by partitions. The statistic has the predefined format (Q1, Q3 ...).


SYNTAX:

STATISTICS {


STATISTICS <STATISTIC_REFERENCE_VARIABLE_NAME>;

FIELD <FIELD_NAME>;

[KEYS [<KEY_NAME>]+;]

[MISSINGVALUES (TRUE | FALSE);]

}

PARAMETERS:


<STATISTIC_REFERENCE_VARIABLE_NAME> Name of the statistic dataset reference variable

<FIELD_NAME> Name of the field on which the statistic is being computed

<KEY_NAME> Name of the column that is part of the partition

MISSINGVALUES If TRUE, the missing values (null ones) will be treated as 0.0. Otherwise, the null values will remain the same and will not be part of the computations.

EX:

STATISTICS

{

INPUT input1;

STATISTICS stat1;

FIELD PRICE;

KEYS COUNTRY, PRODUCT ;

}

4.5.15. Update

The Update operation creates an output dataset containing the rows from the first input dataset, on which certain fields are updated with values from the second input dataset, by joining the two input datasets on a set of specified keys.

The format of the output dataset is the same as the format of the first input dataset.

SYNTAX:

UPDATE {






}

PARAMETERS:




<FIELD_NAME> Name of a field to be used as a key for update

EX :

UPDATE

{

INPUT input1;

INPUT input2;

OUTPUT output1;

KEYS country, price;

}

4.5.16. Import

The import operation reads selected columns of data from an external data source and saves them to a dataset. Depending on the parameter selected as the FILETYPE, the import operation can read either from a local file (CSV, FLR, etc.) or from an external database (DB). For file imports, an import path, where the data file is located, must be specified. For database imports, the CONNECTION and STATEMENT properties must be specified.

SYNTAX:

IMPORT

{

OUTPUT <DATASET_NAME> <DATASET_FORMAT>;

[IMPORTPATH <FILE_PATH>;]

(PROPERTIES { [<EXPORT_KEY> <EXPORT_VALUE>]+

}) | ( PROPERTIES_TEMPLATE <EXPORTTEMPLATE_NAME>);

FILETYPE <EXPORTTYPE_NAME>;

[FIELDSETTEMPLATE <FIELDTEMPLATE_NAME>;]

}

PARAMETERS:

<DATASETINPUT_NAME> Name of the dataset to be exported


<EXPORT_KEY> The name of the property

<EXPORT_VALUE> The value of the property

<EXPORTTEMPLATE_NAME> The template used to export the dataset

<EXPORTTYPE_NAME> The type used to export the dataset

<FIELDTEMPLATE_NAME> The field template used to export the dataset

<PUBLISH_NAME> The location where the exported file will be published; this location must be accessible to the application

EX ;

IMPORTDATASET

{

OUTPUT dsImportFromDB1 testDatasetFormat;

PROPERTIES {

CONNECTION "externaldb";

STATEMENT "select_ext_DB";

}

FILETYPE db;

FIELDS {

TEST1;

TEST2;

TEST3;

TEST4;

}

}

4.5.17. Export

The export operation saves an output dataset containing the selected columns from an input dataset. Depending on the parameter selected as the FILETYPE, the export operation can send data either to a local file (CSV, FLR, etc.) or to an external database (DB). The operation can be performed either in a synchronous or asynchronous fashion. For database exports, the CONNECTION and STATEMENT properties must be specified.

SYNTAX:

EXPORT

{

INPUT <DATASETINPUT_NAME>;

(PROPERTIES { [<EXPORT_KEY> <EXPORT_VALUE>]+

}) | ( PROPERTIES_TEMPLATE <EXPORTTEMPLATE_NAME>);


FILETYPE <EXPORTTYPE_NAME>;

[FIELDSETTEMPLATE <FIELDTEMPLATE_NAME>;]

[SYNCHRONOUS (TRUE | FALSE);]

[PUBLISH <PUBLISH_LOCATION>;]

}

PARAMETERS:

<DATASETINPUT_NAME> Name of the dataset to be exported

<EXPORT_KEY> The name of the property

<EXPORT_VALUE> The value of the property

<EXPORTTEMPLATE_NAME> The template used to export the dataset

<EXPORTTYPE_NAME> The type used to export the dataset

<FIELDTEMPLATE_NAME> The field template used to export the dataset

<PUBLISH_NAME> The location where the exported file will be published; this location must be accessible to the application

EX:

DATASETOPERATION fileExport

{

EXPORT

{

INPUT dataset1;

PROPERTIES_TEMPLATE “template1”;

FILETYPE “csv”;

PUBLISH “/outputs/data/exported_data.csv”;

}

}

DATAOPERATION databaseExport

{

EXPORT

{

INPUT INPUT1;

PROPERTIES {

CONNECTION "externalDB";

STATEMENT "insert_ext_DB";

}


FILETYPE db;

FIELDSETTEMPLATE fieldsetCountry;

SYNCHRONOUS FALSE;

}

}

4.5.18. Comext

The Comext Method data operation writes in the designated column the result of the Comext Method outlier algorithm.

SYNTAX :

COMEXT {


FIELD <VALUE_COLUMN>;

[KEYS [<KEY_COLUMN>]+;]

OUTLIER <OUTLIER_COLUMN>;

TRANSPOSE <TRANSPOSE_COLUMN>;

[STATISTICS <DATASET_REFERENCE_STATISTICS>;]

[CONDITION <VALUE_CONDITION>;]

CA <VALUE_THRESHOLD>;

[MISSINGVALUES TRUE|FALSE ]

SEVERITY <VALUE_SEVERITY>;

}

PARAMETERS :


<VALUE_COLUMN> Name of the column that gives the input values to the algorithm

<KEY_COLUMN> Name of a field to be used as a key for computing the partitions

<OUTLIER_COLUMN> Name of a field to be used as a key

<TRANSPOSE_COLUMN> Name of a field to be used as a date / time (transposed column)

<DATASET_REFERENCE_STATISTICS> Name of the statistical reference variable

<VALUE_CONDITION> Expression that defines which observations will be part of the computation of the median

< VALUE_THRESHOLD> Value of the threshold


< VALUE_SEVERITY> Value of the outlier


EX :

DATAOPERATION comextDO {

COMEXT{

INPUT input1;

FIELD PRICE;

OUTLIER outlier_result;

TRANSPOSE calendar_column;

STATISTICS stat1;

CONDITION VALUE>90;

CA 100;

SEVERITY 'comext';

}

}

4.5.19. Food Safety

The Food Safety data operation writes in the designated column the result of the Food Safety Domain outlier algorithm.

SYNTAX :

FOODSAFETY {








INCREASE <VALUE_INCREASE>;

DECREASE <VALUE_DECREASE>;


}

PARAMETERS :









<VALUE_INCREASE> Value of the maximum growth rate between periods

< VALUE_DECREASE> Value of the minimum growth rate between periods


EX :

DATAOPERATION foodSafety {

FOODSAFETY{

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

INCREASE 0.25;

DECREASE 0.995;

SEVERITY 'FOOD_SAFETY';

}

}

4.5.20. Growth Rate

The Growth Rate data operation writes in the designated column the result of the Growth Rate between periods outlier algorithm.

SYNTAX :

GROWTHRATE {










DECREASE <VALUE_DECREASE>;


}

PARAMETERS :








<VALUE_INCREASE> Value of the maximum growth rate between periods

< VALUE_DECREASE> Value of the minimum growth rate between periods


EX :

DATAOPERATION growthRate {

GROWTHRATE{

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

INCREASE 0.25;

DECREASE 0.995;

SEVERITY 'GROWTH_RATE';

}

}

4.5.21. HBHistorical (Hidiroglou Berthelot)

The HB (Hidiroglou Berthelot) historical data operation writes in the designated column the result of the HB historical bounds outlier algorithm.

SYNTAX :


HBHISTORICAL {







[CI <VALUE_CI>;]

[CE <VALUE_CE>;]

[CA <VALUE_CA>;]

[CP <VALUE_CP>;]

EXPONENT <VALUE_EXPONENT>;

}

PARAMETERS :







<VALUE_CI> Value of the imputation coefficient

< VALUE_CE> Value of the exclusion coefficient

< VALUE_CA> Value of the minimum distance multiplication coefficient

< VALUE_CP> The percent of the highest values that are excluded from calculation

< VALUE_EXPONENT> Value of the exponent to be used in the effect calculation

EX :

DATAOPERATION hbHistorical {

HBHISTORICAL {

INPUT input1;

FIELD PRICE;



STATISTICS stat1;


CI 2;

CE 1;

EXPONENT 1;

}

}

4.5.22. LFS

The LFS1/2 data operation writes in the designated column the result of the Labour Force Survey 1/2 outlier algorithm. The difference is that LFS1 uses rates and LFS2 uses ranks. In the definition of the operation only one of the following methods can be specified to be used as modality values: lookup or explicit values through the keyword RATES.

SYNTAX :

(LFS1 | LFS2) {





FIELDS {

[<COLUMN_NAME> -> <MODALITY>;]+

}

[LOOKUP <LOOKUP_NAME>]

[RATES {

<MODALITY> -> <MODALITY_VALUE>;

}]

}

PARAMETERS :






<COLUMN_NAME> Name of the column that has modality

<MODALITY> Modality value


<LOOKUP_NAME> The name of the lookup for rates (LFS1) / ranks (LFS2) – if this is specified, the explicit rates are ignored.

< MODALITY_VALUE> Value of the modality (rates / ranks). This represents values to be used as thresholds in the absence of the above lookup.

EX :

DATAOPERATION Lfs {

LFS1{

INPUT input1;



FIELDS {

Columns1 -> 4;

Columns2 -> 10;

Columns3 -> 20;

}

LOOKUP lookup1;

}

}

4.5.23. Sigma Gap

The Sigma Gap data operation writes in the designated column the result of the Sigma Gap outlier algorithm.

SYNTAX :

SIGMAGAP {







APPROACH <VALUE_APPROACH>;

CM <VALUE_CM>;

[CI <VALUE_CI>;]

[CJ <VALUE_CJ>;]

[CP <VALUE_CP>;]

[UPPER <VALUE_UPPER>;]


[LOWER <VALUE_LOWER>;]



}

PARAMETERS :







<VALUE_APPROACH> Value of the approach (“stddev” for standard deviation and other for difference between two percentiles CI and CJ)

<VALUE_CM> Value of the multiplier coefficient

<VALUE_CI> The first percentile (must be lower than 100)

<VALUE_CJ> The second percentile (must be greater than 0)

< VALUE_CP> Value of the predefined percentage of the excluded values

< VALUE_UPPER> Value of the upper bound

< VALUE_LOWER> Value of the lower bound



EX :

DATAOPERATION sigmaGap {

SIGMAGAP{

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

APPROACH 'stddev';

CM 0.5;

UPPER 120;

LOWER 15;


SEVERITY 'OUTLIER_GAP';

}

}

4.5.24. Urban Method 3

The Urban Method 3 data operation writes in the designated column the result of the Urban Method 3 outlier algorithm.

SYNTAX :

URBANMETHOD3 {






(STATISTICS <DATASET_REFERENCE_STATISTICS>;)?




}

PARAMETERS :








<VALUE_INCREASE> Value of the maximum growth rate


EX :

DATAOPERATION urbanMethod3 {

URBANMETHOD3{


INPUT input1;

FIELD PRICE;



STATISTICS stat1;

INCREASE 0.25;

SEVERITY 'URBAN_METHOD_3';

}

}

4.5.25. Time Series

4.5.25.1. Input Format

For the time series the input format is fixed with name “time_series”. This format can be loaded from the file time_series.fdl. The output is in the column named “outlier”. This processing takes as input only series with time components (year, month, value) according to the tramo documentation. The tramo program and documentation can be found at http://www.bde.es/servicio/software/econome.htm.

4.5.25.2. Output Format

The only output format is the error one.

4.5.25.3. Timeseries operation

The Timeseries data operation writes in the designated column the result of the Tramo run.

SYNTAX :

TIMESERIES {




ERROR <VALUE_ERROR>;

PARAMETER <VALUE_PARAMETER>;

}

PARAMETERS :





<VALUE_ERROR> Name of the error dataset reference variable

<VALUE_PARAMETER> Parameters for the “tramo” to run

The outlier column will contain the outlier value and the type (e.g. Additive (AO) 4.31).

EX :

DATAOPERATION timeSeries {

TIMESERIES{

INPUT timeseries1;

FIELD price;

ERROR error1;

PARAMETER lam=-1,iatip=1,aio=2,va=3.3,noadmiss=1,seats=2;

}

}

4.6. Validation Rules Definition

The validation rules script contains a series of rules that will be applied to the same input dataset. The rules can be horizontal, vertical or rules that do not apply to data coming from inside the input dataset.

4.6.1. No Record Rule

No-Record rules are applied without retrieving data from the input dataset. . A no record rule can either be a computation rule, when the THEN/ELSE keywords are present, or a validation rule, when the ERRMSG keyword is used. If a validation fails, then an error is reported in the error log. The main purpose of the no record rules is to define variables with a global scope, but any function or parameter can be used as well inside the expressions.

SYNTAX:

NORECORD <RULE_NAME> {

[CONDITION < CONDITIONAL_ESLE>;

[ERRMSG <ERROR_MESSAGE> SEVERITY <SEVERITY>

[( [<VARIABLE>]+ )];]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+} ]

| [<ESLE>;]+]?

}

PARAMETERS:

<RULE_NAME> Name of the rule

<CONDITIONAL_ESLE> EDIT SL Expression that returns a Boolean value


<ELSE> EDIT SL Expression

<ERROR_MESSAGE> Message to be displayed in the validation fails

<SEVERITY> Severity of the rule

<VARIABLE> Name of the variable to be reported as part of the Auxiliary Data in the error log

EX :

NORECORD checkNegativeValues

{

VAR STRING TRANSACTION_TYPE_VAR

:= upper(substring(trim(pTRANSACTYP), 1, 2));

VAR BOOLEAN INTRA := false;

}

4.6.2. Horizontal Rule

Horizontal rules are applied to each row in the input dataset. A horizontal rule can either be a computation rule, when the THEN/ELSE keywords are present, or a validation rule, when the ERRMSG keyword is used. If a validation fails, then an error is reported in the error log indicating the ROW_NUMBER of the current row. Additional fields with their current values can be reported as part of the Auxiliary Data field inside the error log.

SYNTAX:

RECORD <RULE_NAME> {

[CONDITION < CONDITIONAL_ESLE>;


[( [<FIELD>]+ )];]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+} ]

| [<ESLE>;]+]?

}

PARAMETERS:




<ERROR_MESSAGE> Message to be displayed in the validation fails


<FIELD> Name of the field or variable to be reported as part of the Auxiliary Data in the error log


EX :

RECORD checkNegativeValues

{

CONDITION price > 0 AND quantity > 0 AND value > 0;

ERRMSG 'Negative values' SEVERITY 'ERROR' (price,quantity,value);

}

4.6.3. Vertical Rule

Vertical rules are applied on blocks of data from the input dataset, which are called partitions. The definition of the rule contains also the definition of the partition, indicating the key columns and the transposed key to be used. A vertical rule can either be a computation rule, when the THEN/ELSE keywords are present, or a validation rule, when the ERRMSG keyword is used. If a validation fails, then an error will be reported in the error log indicating the values of the current partition. Additional fields with their current value for a specific transposition value can be reported as part of the Auxiliary Data field inside the error log.

SYNTAX :

VERTICAL <RULE_NAME> {

[EXPRESSION {

[KEYSET <KEYSET_NAME> | KEYS [<KEY_FIELD >]+ ]+ ;

TRKEYS <TRANSPOSED_KEY_FIELD_NAME>;

CONDITION <CONDITIONAL_ESLE>;


[ ( [<FIELD >]+ ) ]; ]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+}]

]?

}

}

PARAMETERS :


<KEYSET_NAME> Name of a keyset defined at the program level

<KEY_FIELD > Name of a field to be used as a key for the partition

<TRANSPOSED_KEY_FIELD_NAME> Name of the field to be used as a transposed key



<ERROR_MESSAGE> Message to be displayed in case the validation fails



<FIELD > For Vertical Rules a Field Identifier can be either one of the key fields minus the transposed key, the value of a field for a specific transposition value or a variable.

EX :

vertical checkGBQ

{

expression

{

keyset keyset1;

trkeys PRODUCT;

condition country='GB' -> quantity['GAS'] > quantity['OIL'];

errmsg 'Incorrect quantity' severity 'WARNING' (country, quantity[‘OIL’]);

}

}

4.6.4. Hierarchical Rule

Hierarchical rules are applied on data cases, comprised of multiple data shapes grouped under a root node. The definition of the rule contains also the definition of hierarchy, indicating the keysets template defining the hierarchy and the keysets to be involved in the validation rule. A hierarchical rule can either be a computation rule, when the THEN/ELSE keywords are present, or a validation rule, when the ERRMSG keyword is used. If a validation fails, then an error will be reported in the error log indicating the values of the current case. Additional fields with their current value for a specific transposition value can be reported as part of the Auxiliary Data field inside the error log.

SYNTAX :

HIERARCHICAL <RULE_NAME> {

KEYSETSTEMPLATE <TEMPLATE_NAME> [<KEYSET_NAME>]+;

CONDITION <CONDITIONAL_ESLE>;

[ ERRMSG <ERROR_MESSAGE> SEVERITY <SEVERITY>

[ ( [<FIELD_IDENTIFIER>]+ ) ]; ]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+} ] ]?

}

PARAMETERS :



<TEMPLATE_NAME> Name of the keysets template defining the hierarchy

<KEYSET_NAME> Name of the keyset defining a hierarchical level

<FIELD_NAME> Name of a field to be used as a key for the partition


<ERROR_MESSAGE> Message to be displayed in case the validation fails


<FIELD_IDENTIFIER> For Hierarchical Rules a Field Identifier can be either one of the root key fields or the value of a field for a specific transposition value.


EX :

hierarchical rule1

{

KEYSETSTEMPLATE kst1 Country1, City1;

condition NOT isNull(value['COUNTRY1']) ->

value['COUNTRY1'] = sum(value['CITY1'][*][*]);

errmsg 'the sum is not correct1'

severity 'severity' (Country);

}

4.7. Scripting Language Expressions Definition

The SL Expressions are used in order to evaluate validation conditions or to perform computations inside a program. The values from the Vector Map of the current partition are used as Expression Fields in order to evaluate an expression.

4.7.1. Expression Fields

A field can be accessed inside the expression script by using the name of the field as a reference. All the notations are case insensitive (for example if a dataset field is named “COUNTRY” it can be referenced as “country” or “COUNTRY”).

<VALUE_FIELD>

Ex: country, product

In case of a transposition, each value field will contain a list of values corresponding to the values of the transposed column. The values corresponding to the keys of the partition can be used like normal fields as described above. The full list corresponding to all transposition values for a value field can be used inside the script with the following syntax:

<VALUE_FIELD>[*]

Ex: price[*], quantity[*]


Using the same syntax, the list of values in case of a transposition can be passed as an argument to a function.

Ex: sum(price[*]), uniqueInList(product[*])

A single value from the list can be used inside the script with the following syntax:

<VALUE_FIELD>[<TRANSPOSITION_VALUE>]

Ex: price[‘OIL’] , quantity[‘GAS’]

4.7.2. Data Types

There are four types of data: • Boolean • Double • Number • String

4.7.3. Operators

There are three types of operators in the SL Expressions: • Boolean operators, used to evaluate expressions into a true/false result:

o OR o AND o NOT o -> (implication) o = (equals) o <> (not equals) o > (greater than) o < (lower than) o >= (greater than or equal to) o <= (lower than or equal to)

• Computation operators, used to produce a value result following evaluation: o + (plus) o – (minus) o * (multiply) o / (divide)

• Assignment operator, used to assign a value to an operand o := (supports assignment to NULL, value := NULL)

4.7.4. Functions

These following function calls are supported: A • abs(Double) – absolute value • ascii(Char) – returns the ASCII code for a character B • between(Double, Double, Double) – verifies if a number is inside a closed interval • between(Double, Double, Double, Boolean, Boolean) – verifies if a number is inside an

interval allowing the user to specify if the interval is closed or open at each end


• between(String, String, String) – same as above • between(String, String, String, Boolean, Boolean) – same as above C • ceiling(Double) – ceiling for number (Ex: ceiling(3.2) => 4) • chr(Integer) – returns the ASCII character for the ASCII code • concatenate(String…) – concatenate Strings • countMissing(List) – returns the number of null values in the list • count(List) – returns the number of elements in the list E • exp(Double) F • firstIndexOf(String toSearch, String searchIn, Double startingFrom) – first occurance of

the toSearch String in the searchIn String • floor(Double) – floor for number (Ex: ceiling(3.2) => 3) G • getRowCount(datasetReference) – returns the number of rows for the specified dataset

reference

I • in(Boolean, Boolean List) – check to see if a value is inside a list • in(Double, Double List) • in(String, String List) • identicalInList(List) – checks that all elements in a list are identical • isIdentical(List, List, …) – returns a boolean indicating if each list contains identical

elements (elements are identical inside a single list) – Ex: isIdentical(price[*],quantity[*],value[*])

• isUnique(List, List, …) – returns a boolean indicating if the combination of elements from each list for all the index positions are unique– Ex: isUnique(price[*],quantity[*],value[*])

• isTrue(Boolean) – checks if a boolean is true • isNull(Boolean/String/Double) – checks if a value is NULL • inLookup(value1, value2, …, datasetReference, “fieldName1”, “fieldName2”,… ) –

returns a boolean indicating whether or not the value or combination of values is defined for the fieldname or combination of fieldnames in the specified dataset

L • lastIndexOf(String toSearch, String searchIn, Double startingFrom) – last occurance of

the toSearch String in the searchIn String • length(String) – returns the length of the String • like(String, String) – compares two strings in a SQL manner • log(Double) • ln(Double) • ltrim(String) – trim left side of String • lower(String) – switch to lower case


• left(String S, Double N) – returns the first N characters from S

M • max(Double List) – maximum value from a list • min(Double List) – minimum value from a list • mean(Double List) – computes the average value not counting null values • missingMean(Double List) – computes the average value counting null values as zero • mod(Double N, Double n) – N%n N • nvl(Boolean, Boolean) – if first argument value is null return second argument • nvl(Double, Double) • nvl(String, String)

O • occurs(String S, String s) – returns the number of occurences of s in S P • printf(String, String/Double…) – offers the capabilities of the printf method • pow(Double N, Double n) – N**n

R • right(String S, Double N) – returns the last N characters from S • round(Double N, Double n) – round N till n decimals (Ex: round(4.46, 1) => 4.5) • rtrim(String) – trim right side of String

S • str(Double N1, Double N2, Double N3) – Ex: str(30.25, 7, 3)= “_BLANK_30.250“ • strToDouble(String) – convert a String into a Double – return null if String cannot be

converted • substring(String, Double N, Double n) – substring starting from N, counting n

characters • sum(Double List) – sum of elements from list • sqrt(Double) – returns the square root of the value

T • trim(String) – trim String • transcode (“targetField”, lookupReference, “lookupField”, lookupValue) – performs a

lookup based on the specified field and value and returns the value of the target field on the matching row

U • upper(String) – switch to upper case • uniqueInList(List) – checks if a list contains unique values

In all functions related to Strings, the first character in the string has the index value 1.


The behaviour of the isIdentical and isUnique functions is further explained by the following example: C1 C2 C3 A B X A B X A B X isUnique(C1, C2, C3) will return FALSE isIdentical(C1, C2, C3) will return TRUE C1 C2 C3 A B X A B Y A B Z isUnique(C1, C2, C3) will return TRUE isIdentical(C1, C2, C3) will return FALSE C1 C2 C3 A B X A B X A Z W isUnique(C1, C2, C3) will return FALSE isIdentical(C1, C2, C3) will return FALSE

4.7.1. Variables

The Scripting Language allows the definition and usage of variables inside the expressions. Variables can have different scopes, depending on where they are defined as follows:

• Global Variables: Global variables are defined in NORECORD rules. They are available throughout the execution of the entire program after their declaration;

• Local Variables: Local variables are defined in any rule type other than NORECORD. They are available only throughout the execution of the current Block.

The same data types as for fields apply also to variables. A variable can be defined by as follows: SYNTAX :

VAR <DATA_TYPE> <VARIABLE_NAME> [:=<INITIAL_VALUE> ]; PARAMETERS :

<DATA_TYPE> The data type of the variable

<VARIABLE_NAME> The name of the variable

<INITIAL_VALUE> The initial value of the variable EX: VAR STRING STR_VAR := “initial value”;

VAR NUMBER NUM_VAR; When the INITIAL_VALUE is not specified, the variable takes the following default value, depending on its type:

• Boolean - false


• Double – 0 (zero) • Number – 0 (zero) • String – null

4.8. Comments

The user can insert comments inside the Scripting Language. A line from the file will be marked as a comment line by using sharp character (‘#’) in the beginning of the line.

EX: #This is a comment line

4.9. Multi-line Strings

The user can use multi-line strings inside the Scripting Language. The lines will be automatically concatenated and interpreted as a single string.

EX:

DESCRIPTION “This is my description “

“and it spreads on multiple lines “

“because it is very long “;

4.10. New Line Characters

New Line characters can be inserted anywhere inside the script without affecting the behavior of the language. These characters will be ignored by the Scripting Language.

EX :

FORMAT MyFormat {

DESCRIPTION

“This is my format“;

FIELDS {

Year {

CAPTION Year;

MISSINGVALUES 0,

“NONE”, “NULL”, “ZERO”

,”NOTHING”;

TYPE NUMBER;

LENGTH 4;

}

}

}


4.11. Keyword Abbreviations

The user can use the following abbreviations for the keywords in the syntax:

Keyword Abbreviation

DESCRIPTION DESC

MISSINGVALUES MVALS

MANDATORY MNDTR

PARAMETER PARAM

PARAMETERSET PSET

DATAOPERATION DATA

INPUT IN

OUTPUT OUT

PRECONDITION PCOND

CONDITION COND

VALIDATIONSTEP VSTEP

4.12. Reserved Keywords

The following are reserved as Scripting Language keywords and cannot be used otherwise inside the program definition:

A

AND, AGGREGATION, ALTER, APPEND, APPROACH,ASC, AUX

B

BETWEEN, BLOCKNAME, BOOLEAN

C

CAPTION, CONDITION, COND, CONSTANT, COPY, COMEXT, CREATE, CI, CE, CA, CM, CJ, CP, CQU, CQL, CL, CH, CFU, CFL, CONVERSION

D

DATA, DATAOPERATION, DECIMAL_POINT_CHARACTER, DECREASE, DEFAULTVALUE, DEFINITIONS, DESC, DELETE, DROP, DESCRIPTION, DONT, DOUBLE

E

ELSE, EMPTY, ERRMSG, ERROR, ERRORS, ERRORDATA, EXISTING, EXPONENT, EXPORT, EXPRESSION, EXTRACT, EXTRACTION

F

FALSE, FIELD, FIELDS, FIELD_DELIMITER, FIELD_SET_TEMPLATE, FILE_TYPE, FOODSAFETY, FORCE, FORMAT

G


GENERATE_HEADER_LINE, GROWTHRATE

H

HBHISTORICAL, HIERARCHICAL

I

IF, INNER, INCREASE, IN, IDENTICAL, INPUT, IMPORTDATASET, IMPORTPATH

J

JOIN

K

KEYSET, KEYS, KEY, KEYSETS_TEMPLATE

L

LEFT, LENGTH, LOOKUP, LOWER, LFS1, LFS2

M

MANDATORY, MNDTR, MISSING, MISSINGVALUES, MVALS, MERGE

N

NEW, NORECORD, NOT, NUMBER

O

OR, OUTPUT, OUT, OUTLIER, ORDER, OVERWRITE, OUTER

P

PARAMETER, PARAM, PARAMETERSET, PCOND, PRECONDITION, PRNAME, PROGRAM, PROPERTIES, PROPERTIES_TEMPLATE, PSET, PUBLISH

Q

-

R

R, RANKS, RATES, RENAME, RULES, RECORD, REORDER, RIGHT

S

SELECT, SEVERITY, SIGMAGAP, SORT, SPLIT, SPLIT_INTO_QUARTILE, STATISTICS, STRING, STEPS, SYNCHRONOUS

T

TEXT_QUALIFIER, THOUSAND_DELIMITER, TIMESERIES, TRKEYS, TYPE, THEN, TO, TRANSPOSE, TRUE

U

UNIQUE, USE, UPDATE, URBANMETHOD3, UPPER, UPPERLOWER

V

VAR, VERTICAL, VALIDATION, VALIDDATA, VALIDATIONSTE P, VSTEP, VALID

W

-

X

X

Y


-

Z

-

bb2011 (fp) ebb scripting language 01€¦ · the name of the field the description of the field ...

Documents