multiple flat files(csv) to target table in odi12c(12.2.1.0.0)

ODI Multiple Flat Files to Table 2016

1

ODI (12.2.1.0.0)

Multiple CSV to Table via Interface with

dynamically getting of File Name with Status and

File Moved to Archive Folder

Document Made by: Darshankumar Prajapati & Ravindrakumar

Document Version: 1.0

Document Date: 02-Jul-2016

Document Last Updated: 02-Jul-2016


2


3

Purpose: Suppose we have multiple CSV Files, that we have to load in to Single Table via Single

Interface, also File Name must taken Dynamic; also we have to make a table with File Name’s List with

Loaded in to Table Status along with their Record Count.

Prerequisites : It was assumed that before this Tutorial you are able to load a Data from Flat

File to Table via Single Interface in ODI(11.2.1.0.0 or Greater).

Software and Hardware Requirements (Optional): The following is a list of software requirements:

The system should include the following installed products:

Oracle Database 11.2.1.0.0 or Greater

Oracle Data Integrator 12.2.1.0.0

If not done before, start the services and components for Oracle Database 11g.

Created Model: You are going to make this type of Model, so each and every component can be explained in detail in

below (Detailed Technical Steps) section.


4

Detailed Technical Steps:

We have to make one Package under Packages name used here is: Multiple_Csv.

1) Made_List_Of_Files:

Go to Diagram Tab and Select: ODI OS Command.

This will used for Taking All Files Name in One Flat File to achieve Dynamically Getting of File

Name.

Put Below command in Command to Execute Tab:

dir D:\ODI12SOURCE\seha*.CSV /b /o:gn>D:\ODI12SOURCE\FILES.CSV

In case of Linux:

Cd /u01/files/

Ls > FILES.csv

In above command assume that Our Multiple Flat Files Source Location is: D:\ODI12SOURCE\*.*

And We are doing only Listing of all files in to Flat File called FILES.CSV.

Screen Shot of files in My Directory:


5

After executing this you will have these values in FILES.CSV.

2) Get_File_Name:

Now we have to create 1 Interface, which will load Data from above created File (PQ.CSV) to

Table.

For this we need to create one table like below:

CREATE TABLE FILE_LIST

(

FILENAME VARCHAR2(240 BYTE),

STATUS CHAR(1 BYTE),

RECORD_COUNT NUMBER)

Now you will have to create On Interface in which FILES.CSV as a Source and FILE_LIST as

Target.

Screenshot of Interface created:


6

- In Status you can Hardcode ‘N’ for a time.

- We will Update Record_Count Later via separate procedure.

3) File_name

Now we need to create on LOCAL VARIABLE as shown in Below Screen Shots:

This is Refresh Variable:

Step1:


7

Step2:

In this Below query is used:

select filename from file_list where loaded='N'

and rownum<=1

It will just gets file name for which we are going to load data in Target Table.

4) Now we need to have One Mapping which will load Data from Flat File to Table.

In our case we have simply made table same as File, and map each and every column.

Step1:


8

Step2:

Physical diagram Details:


9

This Mapping will simply insert Data from Flat File to our Target Table.

5) Update_loaded_file_status:

Now we need to have One PROCEDURE which will Update File Status which was by default ‘N’ in

step 2, this will set that as ‘Y’, and Will Also put RECORD COUNT in that FILELIST_STATUS table.

Query used to achieve this is:

update file_list set loaded='Y',record_count='<%=odiRef.getPrevStepLog("INSERT_COUNT")%>'

where filename='#FILE_NAME'

In this odiRef.getPrevStepLog("INSERT_COUNT") will simply gets Record Count for Particular Step in

ODI.

Step1:

Step2:


10

Step3:


11

.

6) File Remains to Process:

Now we need to have one Variable which will count How many files are still remains to be proceesed.

Step 1:


12

Step 2:

Query Used:

select count(filename) from file_list where loaded='N'

Step 3:


13

7) Moving_files_to_Archive:

Now we need to Move this processed flat file to Archive Directory.

For this we need to use “ODIFILEMOVE” as below:

Step1:

Step2:

8) File_remains_to_Process:

Now we need to cross check How Many Files are Still Remains to Load.

For this we need to Reuse Variable File_remains_to_Process as EVALUATE VARIBALE as shown below:

So it will be automatically stopped when File_Count NOT > 0.


14

So at the end You have created the below Model.

After running of this Procedure you can check file status in FILELIST_STATUS Table as below:

You can cross check now Files Successfully moved to target directory:


15

You can cross check Record Count in your Final Table in which you have l oaded Data.

Summary:

In this tutorial, you have learned how to:

Verify the Prerequisites.

Create one ODI_OS_COMMAND for just listing of you all flat flies into Single Flat File.

Create a New ODI Model for the Flat File Source to Target Database Table.

Create a New ODI Source Datastore for Use with ODI Interface

Create a New ODI Target Datastore for Use with ODI Interface

Create a Variable for File_remains_to_Process,File_Name.

Create a Mapping which will Load Data from Flat file to Oracle Table.

Create a procedure which will Updates a File Status in Table along with Record Count.

Create a ODI_FILE_MOVE for moving your processed file to ARCHIVE Directory.


16

Hope you have understood which we have developed with the help of this document.

Hope you have learned new things, if you have done something like this, please share it on

[email protected] and [email protected] .

THANK YOU ! KEEP ODIING!!!!

mailto:[email protected]

multiple flat files(csv) to target table in odi12c(12.2.1.0.0)

Data & Analytics