bi apps warehouse

8/7/2019 BI Apps Warehouse

http://slidepdf.com/reader/full/bi-apps-warehouse 1/7

Rittman Mead - Delivered Intelligence

Adding New Facts and Dimensions to the BI Apps Data

Warehouse July 24th, 2008 by Mark Rittman

I¶ve been working with Informatica 8.1.1, the Oracle BI Applications and the DAC (DataWarehouse Administration) for a while now, and over this time I¶ve tried to post a fewtechniques I¶ve picked up whilst putting the tools through their paces. I¶m going to round up thecurrent set of postings now with one on how you add completely new facts and dimensions to theBI Apps data warehouse; this is particularly pertinent as the BI Apps data warehouse only reallycovers a small proportion of the modules available with Oracle E-Business Suite and thereforeit¶s inevitable that you¶re going to want to add data from modules such as Projects into the

warehouse. What¶s good about the Oracle BI Applications is that there¶s a publishedmethodology for doing just that, and if you follow it you can customize your warehouse and still be supported by Oracle.

To take an example, I¶m working with the Sales ± Order Lines subject area in the BIApplications 7.9.5 with my data being sourced from Oracle e-Business Suite 11.5.10. As well asthe standard dimensions of product, customer, time and so on together with my standard SalesOrder Lines and Sales Invoice lines fact tables, I now want to add a new fact table called SalesReturns which I¶ll be analyzing by Product, and by a new dimension called Return Reasons.

The other day I talked about customizing existing mappings through the use of a ³placeholder´

column, X_CUSTOM, that marked a spot in each warehouse table where you could safely startadding new columns. Mappings too have an X_CUSTOM placeholder and if you find the routethis placeholder takes through the mapping, this marks a ³safe path´ through the mapping thatyou can thread your new columns through too. Whereas for customizations of this type you cantake existing Source-Dependent Extract (SDE) and Source-Independent Load (SIL) mappings,copy them to a ³custom´ table and then amend them as needed, if you¶re going to add new factsand dimensions to your warehouse you need to create SDE and SIL mappings from scratch.

To take my example, I need to load a dimension called Return Reason with data from my E-Business Suite database. To do this I start off by creating a new mapping calledSDE_C_ORA_ReturnReasonDimension and add my source table to the mapping together withwhat¶s called a ³Source Qualifier´ transformation. In the screenshot below, the yellow mappingobject is the source qualifier and in it¶s simplest form, it¶s a SELET list of all the columns for a particular table and it¶s this, rather than the source itself, that you extract from. The sourcequalifier lets you customize the SELECT statement so that you can add a filter or otherwisemodify the source data before it¶s processed.



Next to the source qualifier in the mapping is another transformation, this time an ³Expression´transformation. If you¶re used to Oracle Warehouse Builder this is more or less the same, but if you notice in the top row of the transformation is a column (or ³port´ in Informatica terms)called DATASOURCE_NUM_ID. This column is then assigned to a variable called$DATASOURCE_NUM_ID that¶s give a value by the DAC when the mapping is called.DATASOURCE_NUM_ID is one of a number of standard columns that you¶ll find in all OBIApps mappings and it¶s one of two mandatory columns, the other being INTEGRATION_ID, the

business key, that¶s required for OBI Apps staging tables.

The Source-Independent Load that takes data from this staging table and loads it into the ReturnReason dimension looks in this case fairly similar to the SDE mapping. You¶ll notice the sourceat the start (which is the staging table, which happened to be the target in the previous mapping)together with its source qualifier, but then next to it is an example of a transformation, a lookupin this case, that can be shared amongst other mappings in the same way as pluggable mappingsand public transformations in OWB. In fact they¶re better than their OWB equivalents as they¶renot just restricted to PL/SQL (as transformations are) and they update themselves in mappingswhenever the transformation itself is edited (which isn¶t the case with OWB pluggablemappings). In this case, the transformation retrieves the ID of the current ETL run which is then

inserted, along with the ROW_WID (the standard name for a fact or dimension primary keycolumn), the INTEGRATION_ID and the DATASOURCE_NUM_ID into the dimension table.

This lookup transformation is another example of the added value you get with the OBIApplications, in that Oracle (or more precisely Siebel, or Informatica before them) have pre-written a number of standard ETL components that you can drop onto your mappings to, for example, make sure you don¶t add a source row twice if you¶ve had to restart an ETL process.These are things you inevitably end up writing yourself if you¶re putting an OWB project



together, in this case though they¶re written for you and the standard way of constructing OBIApps ETL processes is to use these bits of reusable ETL code.

Once you¶ve picked up the ETL process ID and filtered out any rows that have already been processed for that ETL run ID, there¶s another step where you work out whether the row is either

a new one (and therefore needs to be INSERTed into the dimension table) or an existing one(which therefore needs to be UPDATEd).

Then once you¶ve loaded what¶s called the ³Update Strategy´ transformation you then use this toinsert or update into the actual dimension table.

Now this is one area I think where Informatica is weaker than OWB. In OWB, loading a table isvery simple as you just map the source columns to the target, select either insert, update,insert/update, delete or whatever as the load strategy, and then the data just goes in the table. For facts and dimensions it¶s even cooler as OWB takes care of lookup up the surrogate keys andloads these into the fact table, wheareas Informatica makes you do all this yourself. For inserting



and updating regular tables though, you have to go through these Update Strategytransformations which to me don¶t seem to add anything beyond what OWB does but insteaddoes the same thing in a more convoluted manner. That said though, one thing I will say for Informatica is that the ³fit and finish´ of the product is way beyond OWB ± as it¶s a Windowsapplication rather than Java, it¶s much snappier, more stable and doesn¶t suck all the memory out

of your machine as you¶re doing your work. Coming from a background of teaching OWB andhaving to get students used to the regular crashes, reboots and so on that are a standard feature of OWB, using Informatica over the last month has been quite an eye-opener.

Loading data into the fact table, at least from the SDE (extraction) side, is very similar to thedimension table. The SDE mapping takes data from the source table, adds theDATASOURCE_NUM_ID and stores it along with the INTEGRATION_ID in the fact stagingtable.

Loading from the fact staging table into the fact table itself is a bit more complex. First off youneed to add both the fact staging table, and the two dimension tables to the mapping as sources,then you join them using a single source qualifier transformation. Now the source qualifier transformation to me is one of the major flaws in Informatica; what you use it for is to bringtogether one or more source tables into a single mapping object which you then use as the sourcefor the rest of your mapping. In a way it¶s like an OWB joiner transformation, and in fact the joiner transformations you get in Informatica are actually there for joining data from differentdata sources, as the source qualifier transformation only allows joins across a single sourcesystem.



So far so good, but the flaw for me in this transformation is that it¶s backed up by an SQLstatement, that you can edit and in most cases does get edited so that you can fine-tune the joincondition, add custom columns in and so on. The problem is that this SQL statement has in myexperience been responsible for around 90% of all mapping errors, as the column list in the SQLhas to exactly match the column list in the mapping, it¶s easy to introduce typos, it¶s not thatobvious if the join condition is wrong and so on.



Contrast this with OWB, where you just create input and output groups in a joiner transformation, then drag source columns in to it and just use an SQL expression editor to set the join condition. I¶ve had far less errors occur with OWB¶s way of doing things and I just can¶t seewhy Informatica have provided a transformation type with so much scope for human error.

Anyway, once you¶ve got your source transformation working you can work through the rest of the fact load, again using reusable transformations to retrieve the ETL run ID, get the nextsurrogate key and so on. Going back to the OWB vs. Informatica subject again, one thing that Ifound very good about Informatica is that you¶ve got more or less complete freedom to constructyour ETL process in any way you want, you don¶ t have to worry about whether it¶d make senseas an SQL SELECT statement or whatever.

Now of course this doesn¶t necessarily lead to the most performant of mappings (I couldn¶t helpnoticing that every load took absolutely ages when moving data into the BI Apps datawarehouse, but then again I was loading from E-Business Suite) but as a developer it¶s a prettyexciting environment with a total absence of strange database error messages when you try anddo anything complex. In fact this detachment from the underlying database was in somecircumstances pretty good ± unlike OWB where every move of the mouse seems to trigger adatabase transaction, Informatica only interacts with the database when you save mappings, for example, to the repository ± although I can¶t help thinking that on big data warehouse loads,where you¶ve got an experience OWB and database perform creating the mappings, OWB isgoing to beat Informatica¶s row-based approach through its use of set-based SQL. Still, it¶s nice

to be able to access data from a web service, say, without having to use some mind-bendinglycomplex internal Oracle database feature and I think Informatica have got a nice balance between developer productivity and leverage of the database, especially in the current releasewhere certain ETL functions can be shipped back to the underlying database, in the same waythat the BI Server function-ships calculations back to the underlying database.

The other thing I¶ve kept thinking when working with Informatica is, on the one hand, how muchit¶s like OWB, and on the other, how much it¶s not like Oracle Data Integrator, which is slated as



being the ETL tool that will replace Informatica once Oracle get around to migrating all themappings. ODI is strictly set-based and it¶s mappings are very simple ± just join a number of source tables, perform calculations en-route and load it into a target table, none of these fancyreusable transformations and so on that Informatica uses so widely. Unless Oracle replace all of this advanced ETL functionality with new ODI Knowledge Modules, I¶m not quite sure how

they¶ll make the transition, but then again I¶m no wiser than anyone else how this process isgoing and I guess we¶ll hear more at Open World. For the time being though, that¶s it for me onthe BI Applications; I¶m hoping to get the new Essbase 11 tools up and running over theweekend and then I¶m starting work on some examples for the DW Seminar I¶m running for Oracle in Denmark early in September.

bi apps warehouse

Documents