bing chen, james hanck, patrick hanck, scott hertel, allen...

BingChen,JamesHanck,PatrickHanck,ScottHertel,AllenLissarrague,PaulMédaille

DataIntegrationwithSAP®DataServicesThisE-Biteisprotectedbycopyright.FullLegalNotesandNotesonUsagecanbefoundattheendofthispublication.

SAPPRESSE-Bites

SAPPRESSE-Bitesprovideyouwithahigh-qualityresponsetoyourspecificprojectneed.Ifyou’relookingfordetailedinstructionsonaspecifictask;orifyouneedtobecomefamiliarwithasmall,butcrucialsub-componentofanSAPproduct;orifyouwanttounderstandallthehypearoundproductxyz:SAPPRESSE-Biteshaveyoucovered.AuthoredbythetopprofessionalsintheSAPuniverse,E-BitesprovidetheexcellenceyouknowfromSAPPRESS,inadigestibleelectronicformat,delivered(andconsumed)inafractionofthetime!

AkashKumarPlanViz:ImprovingSAPHANAPerformanceISBN978-1-4932-1300-9|$14.99|75pages

EricDuSAPHANASmartDataStreamingandtheInternetofThingsISBN978-1-4932-1303-0|$9.99|86pages

AronMacDonaldIntegratingSAPHANAandHadoopISBN978-1-4932-1293-4|$12.99|85pages

TheAuthorsofthisE-Bite

BingChen,JamesHanck,PatrickHanck,ScottHertel,AllenLissarrague,andPaulMédaillearethebest-sellingauthorsofSAPDataServices:TheComprehensiveGuide,anSAPPRESSpublication.Learnmoreabouttheauthorsatwww.sap-press.com/sap-data-services_3688/authors/.

http://www.sap-press.com/e-bites

http://www.sap-press.com/3952









http://www.sap-press.com/sap-data-services_3688/authors/

WhatYou’llLearn

DiveintodataintegrationwiththisE-Bite.LearnaboutdatawarehousingscenariostohelpyousolvecommonintegrationchallengesinSAPDataServices.Discoverintegrationstrategiesforthedistributionandretailindustries,andbuilddataflowstounderstandproperdataprovisioningtodatawarehouses.

1IntegrationintoDataWarehouses

1.1DimensionalDataModelOverview

1.2ConformedDimensionsandtheBusMatrix

1.3DimensionalModelDesignPatterns

1.4ProcessingSlowlyChangingDimensions

1.5LoadingFactTables

2IntegrationwithPOSSystems

2.1InquiryCaptureJob

2.2IdentifyDiscountTargetsBatchJob

2.3ProductAvailabilityCheckLookup

2.4Results

3IntegrationwithSAPBW,SAPAPO,andSAPECC

3.1SAPECCExtraction

3.2DeliveryTimeCalculations

3.3SAPAPOInterfaces

3.4SAPECCInterfaces

3.5Results

ThisE-BiteisanexcerptfromSAPDataServicesbyBingChen,JamesHanck,PatrickHanck,ScottHertel,AllenLissarragueandPaulMédaille.

http://rheinwerk-publishing.com/3688

1IntegrationintoDataWarehouses

OneofthemostcommonDataServicesintegrationscenariosisprocessingandmovingdataintoadatawarehouse.Adatawarehouse(DW)isdistinctfromanOnlineTransactionProcessing(OLTP)typeofsysteminthatit’sdesignedandoptimizedforanalyticoperationsonlargesetsofdataversustransactionaloperationsonasinglerecordinanOLTPsystem.ADWoperationcanaggregatebymultipledimensionalattributesovermillionsofrecordsversusmanyprocessesoperatingonasinglerecordinanOLTPscenario.

KimballMethodology

Tofullyunderstandsomeofthetechniquesanddesignpatternsthatyou’llusetointegrateintoacustomDWwithDataServices,youfirstneedsomebackgroundonKimballMethodology.TherehavebeenmanyapproachestoDWdesign,buttheKimballmethodologyisprobablythemostcommonlyacceptedbestpracticeapproachtoday.

Itcombinesspecificdatamodelinganddesignpatternswithaniterativeprocessandadataframeworkcalledabusmatrix.

TheKimballmethodologyisafulllifecyclethatincludesparallelimplementationtracksfortechnicalarchitecture,databasedesign,andBIapplicationsaswellasprojectplanning,management,deployment,andmaintenance.However,forthepurposesofthisE-Bite,thediscussionislimitedtodimensionalmodeling,physicaldesign,anduseofDataServicesfortheExtraction,Transformation,andLoading(ETL)integrationaspectsoftheKimballlifecycle.

ThefollowingarethefundamentalprinciplesoftheKimballapproach:

Focusonbusinessprocess

Dimensionallystructureddatamodels

Iterativedevelopmentinmanageableincrements

Thisapproachensuresthatthefocusremainsondeliveringbusinessvalueinanincrementalapproach.

ForcomprehensivecoverageoftheKimballapproach,seeTheDataWarehouseLifecycleToolkit,byKimballetal.(2nded.,Wiley,2008).

InSection1.1toSection1.5wewilltakeyouthroughdatawarehousingscenariosandstrategiestosolvecommonintegrationchallengeswhenleveragingdimensionalandfactualdatawithDataServices.Wewillalsobuilddataflowstohighlightproperprovisioningofdatatodatawarehouses.

ANoteonSAPBW

ThisresourcespecificallydiscussesKimball-baseddimensionalDWdesignandintegrationsusingDataServices.Thisistypicallyadatawarehouseordatamartwherethedevelopermustintegrateintoacustom-designeddimensionaldatamodelonvarioustypesofdatabases.

SAPBusinessWarehouse(SAPBW),ontheotherhand,implementsmanyoftheconceptsdiscussedinthissection,buttherearespecifictoolsandmethodologiesusedbyDataServicestointegratewithSAPBW,whicharediscussedinSection3.

Now,let’sexplorevarioustechniquesthatareusedtoloaddimensionandfacttablesintoadimensionalDWusingDataServices.

1.1DimensionalDataModelOverview

WhenloadingadimensionalDWmodelwithDataServices,thefundamentaltasksareprocessingandloadingmasteranddescriptivedataintodimensiontables,andloadingtransactions,aggregates,andmeasuresintofacttables.ToeffectivelydesignandimplementDataServicesjobstodothis,youneedtounderstandthepurposeandfunctionalityofthedimensionaldatamodelanditsvariousdesignpatterns.

Adimensionaldatamodel,sometimesknownasastarschema,consistsofafacttablewithmeasureslinkedandkeyedtoasetofdescriptivetablescalleddimensions.Dimensionsdescribehowdataissliced.IntheexampleinFigure1,ordersareseenasafacttablethatyoucandimensionbydate,channel,product,andsoon.

Figure1ExampleStarSchemaModelforOrders

Thedimensionalmodelisadatastructurethatisoptimizedforqueryperformanceandusability.Thedimensionalmodel’skeybenefitsincludethefollowing:

Usability

Consistency

Performance

Extensibilityandflexibility

1.2ConformedDimensionsandtheBusMatrix

Conformeddimensionshaveequalmeaningacrossdifferentfacttables.Thus,inFigure2,thedimensionrecordsthatarelinkedandkeyedwiththeorderfactrecordscanalsobelinkedandkeyedtoinventoryandshipments.Thus,thedate,product,currency,andotherdimensionrecordswillhavethesamemeaningacrossallfacts.

Figure2StarSchemawithConformedDimensions

ApartoftheKimballapproach,atoolthat’sextremelyhelpfulforunderstandingthebusinessprocessesandfacilitatescommunicationandplanning,istheenterpriseDWbusmatrix.AsimplifiedbusmatrixisshowninTable1.

Date Product Salesperson Currency Promo Channel

BusinessProcess

DCInventory

X X X

SalesOrder

X X X X X X

OrderDelivery

X X X X X X

Table1EnterpriseDWBusMatrix

Thebusmatrixcapturestheorganization’sbusinessprocessesanddimensionshorizontallyacrossthetop.Duringplanningdiscussions,thecrucialbusinessprocessescanbecapturedandassociatedwiththedimensionswithwhichtheyshouldbelinked.InthebusmatrixinTable1,theDCinventoryprocessisassociatedwithdate,product,andcurrencydimensions.

Fromtheplanningapproach,keyoutcomesincludethefollowing:

Thefacttablesthatwillrepresentthebusinessprocess

Thegranularityatwhichthosefactswillbecaptured

Afterthebusinessprocessesanddimensionsaredefinedinabusmatrix,youcancreatethedimensionalmodel.Thedimensionalmodelingprocesscanbebrokendownintoaniterativefour-stepprocess:

1. Choosethebusinessprocess.

2. Declarethegrain.Whichfacttablesdoyouhave,andwhatisthegranularityofeach?

3. Identifythedimensions.Whichdimensionsapplytoeachfacttable?

4. Identifythefacts.Whatarethemeasuresonthefacttables?

Thisprocessisrepeatedtoenableanincrementalimplementation.

1.3DimensionalModelDesignPatterns

AtypicaldimensionalstarschemaisshowninFigure3.Therearemanytoolsavailabletocreatelogicalandphysicalmodelsforthedatabaseyou’reusing.

Figure3TypicalStarSchema

Next,we’llbrieflydiscusssomeofthemorecommondesignpatternsappliedindimensionalmodels.

Dimensions

Dimensionsaretablesthatrepresentdescriptivedataaboutfactsandmeasures.Thiscanincludemasterdatasuchascustomerandproduct,dateandtime,orcodedefinitions.It’salsocommontorepresenthierarchiesinthesedimensions,suchasproductcategoriesorgeographicalregions,tofacilitatedrill-downanddrill-upaswellasaggregation.

OneofthebasicrequirementsinaDWenvironmentistopreservethechangehistoryofthesedimensions.Whilefactrecordsaretypicallymuchhigherinvolume,thedimensionscanchangeaswell,suchasacustomerupdatinghisemailaddressoranemployeechangingdepartments.Thesetypesofchangesinadimensiontablearecommonlyreferredtoasslowlychangingdimensions(SCDs)becausethesechangesareusuallymuchlessfrequentthanfactrecordchanges.

FollowingarethemostcommonlyusedtypesorpatternsforSCDs:

Type1Updateorinsertintothedimensiontable,andoverwriteanyexistingrecordwithchanges.Nochangehistoryispreserved.

Type2Keepversionsofallchangesbyinsertingrows.Enddateanyexistingrecord,andcreateanewrecordthatisthecurrentversion.Factrecordsarekeyedtotheappropriateversionofeachdimensionrecord.

Type3Additionalcolumnsareusedtostorepreviousversuscurrentversionsofspecificattributes.Thislimitshowmanyattributescanbetrackedforchangehistory,andtypicallyonlyoneorafewversionscanpracticallybetrackedforeachoftheseattributes.

Type4Twotablesareused,acurrentandahistoricaltable.Anytimeachangeoccurs,theoldrecordismovedtothehistorytable,andthenewrecordisinsertedintothecurrenttable.ThisisverysimilartotheOLTPaudittypeoftables.

ThereareseveralotherSCDtypesaswellashybridapproaches,butwe’llfocusonexploringimplementationoftypes1through4usingDataServices.

SurrogateKeys

Asurrogatekeyisamachine-generatedvaluethathasnobusinessmeaning.Itssolepurposeinadimensionalmodelistouniquelyidentifydimensionrecords.WhenyouimplementSCDs,you’lloftenhavemultipleversionsofasingledimensionentity.Youcan’tusethenaturalorbusinesskeyinthefacttabletoreferencethisdimensionbecauseitwillbeduplicated,soyougeneratesurrogatekeys.ThismeansthattherewillbelookupsandjoinswhenyouprocessthesedimensionandfacttablesinDataServicesaswellaswhenyouquerythestarschema.

We’lllookathowtoprocessandlookupthesesurrogatekeysinDataServicesinSection1.5.

DimensionTableDesignPatterns

PreservingchangehistoryusingSCDsappliestoalltypesofdimensions.Inaddition,therearemanytypesofdimensiondesignpatternsthatcanbeapplied

aswell.

DegenerateDimension

Thisisusedwhenthereisadimension(notameasure)thatdoesn’thavedescriptiveattributes.Often,thisisanumberorIDsuchasasalesordernumberoratransactionIDfromasourcesystem.Inthiscase,itdoesn’tnecessarilymakesensetocreateaseparatedimensiontabletostorethisnumberbecausetherewouldlikelybeasmanydimensionrecordsasfacts,andthereisnothingdescriptivetoreportbywiththisdimension.Inthiscase,theattributeiskeptdirectlyinthefacttable,whichavoidsanunnecessaryjoin.

Role-PlayingDimension

Thisisusedwhenyouhaveasingledimensiontablethatisusedinmany“roles”inafacttable.Acommonexampleisthedatedimension.Youcanhavemanydatesonafacttablerecord(e.g.,orderdateandshipdate),butyoudon’twanttocreatemultiplephysicaldatedimensiontablesthathavetheexactsamedata.Youcancreatemultipledimensionviewsforeachdatethatsharethesameunderlyingtable.

MiniDimension

Thisisadimensiontablethatrepresentssegmentationorbandingofanotherdimension.Thisisoftenusedwhenyouhaveverylargedimensionsthathavefrequentchanges,andyoudon’tneedtotrackeverychangeoneverydimension.Customersegmentationanddemographicsisagoodexample,wheretherearemillionsofcustomerswithmanychangingattributes,andyoudon’tneedtheoverheadoftrackingallthosechanges.Youcancreateaminidimensionthathasgroupsor“bands,”suchasagerangesorregions,anddimensionfactrecordsbythebandsintheminidimension.

JunkDimension

Thisisusedformiscellaneous,low-cardinalityattributesthatdon’thavemuchdescriptivedata,suchasindicatorsandflags.Theymaynotjustifytheirowndimensions,soseveralofthesecanbecombinedintoageneral-purpose“junk”dimension.

Snowflaking

Asnowflakepatterniswhenadimensionisnormalizedintomultipletables.Usuallyyoushouldtrytoavoidthisbecauseitrequiresmorejoinswhenqueryingandcanmakethestarschemamoredifficulttouse.Itcanbeappropriate,however,whenyouhavemultiplelevelsinahierarchy,andthefacttablesaredimensionedatthesevariouslevelsseparately.

Many-ValuedDimension

Amany-valueddimensionrepresentsamany-to-manyrelationshipbetweendimensionandfacttable.Thismustbeusedandqueriedcarefullytoavoiderrorsinaggregation.

FactTableDesignPatterns

Facttablesareaspecializedtypeoftableusedtostoremeasuresandfacts,alongwithallofthesurrogatekeysforthecorrespondingdimensions.Usually,facttablesareverylongandnarrow,withmanyrowsbutrelativelyfewcolumns.Eachrowinthefacttableisessentiallyauniquecombinationofallofthedimensions,andeachfacttableshouldhaveaconsistentgrain.Thismeansthateveryrecordshouldbeeitherthelowestlevelofgrainorbeaggregatedatthesamelevel.

Mostfacttableswillfallintooneofthefollowingthreetypes:

TransactionfactThemostatomicandcommontypeoffacttablewithonerecordpertransaction,suchasorders,point-of-sale(POS)transactions,orinventorymovements.

PeriodicsnapshotAggregatedfactsataconsistenttimeperiod,suchasmonthlyaccountbalancesanddailysales.Newrecordsareaggregatedandinsertedintothefacttableforeachperiod.

AccumulatingsnapshotThisisusedtotrackalong-runningprocess,suchasaninsuranceclaimororderfulfillmentprocess.Youcancreateanewfactrecordwhentheprocessinitiates,andthenupdatedatesandmeasuresonthatfactrecordastheprocesscontinuesovertime.

Othertypesoffacttables:

Fact-lessfactMeasuretheexistenceoroccurrenceofaneventthatdoesn’tnecessarilyhavemeasures,suchasattendance.

ConsolidatedfactCombinemultiplefactsinasingletableatthesamelevelofgrain.Thisisusedtostoremeasuresfrommultipleprocessesinthesamefacttable,suchassalesordersandsalesforecast.

AggregatefactArollupandaggregationoftransactionfactdatathatisusuallyusedforperformanceoptimization.

Next,we’llexplorehowtouseDataServicesinvariousscenariostoprocessandloaddimensionandfacttables.

1.4ProcessingSlowlyChangingDimensions

TheSCDisafoundationalelementofthedimensionalDWasdescribedpreviously.ThissectionwillexplorethemostcommonlyusedtypesofSCDs(types1through4)andhowthesecanbeimplementedinDataServices.

We’llusethefollowingsimpleexampleofastarschemashowninFigure4todemonstratehowtouseDataServicestoprocessandloadSCDsandassociatedfacttablesinatypicalDWscenario.Inthisexample,asalesfacttableisdimensionedbycustomer,employee,product,anddate.

Figure4ExampleOrdersStarSchema

SCDType1

Thisisthemostbasicscenariowhenupdatingadimensiontable.Type1insertsnewrecordsandoverwritesanyexistingrecordswithnewdata.Thistypedoesn’tpreservehistoricaldata.InTable2,weseeanexistingrecordintheDIM_EMPLtablefor“VeronicaMeyer”withacurrentpayrateof“22”.

EMP_SK EMP_NUM EMP_LNAME EMP_FNAME EMP_PAY_RATE

328 2714 Meyer Veronica 22

Table2ExistingEmployeeRecordintheDIM_EMPLTable

Thisemployee’spayratehaschangedfrom22to31.ThenexttimeweloadthisdimensionasSCDType1,weupdatetheexistingrecordwithnewdata.TheresultscanbeseeninTable3whereEMP_PAY_RATEhasbeenupdatedto“31”,andwenolongerhavetheprevioushistoricalvalueof“22”.



Table3UpdatedEmployeeRecordintheDIM_EMPLTable

Figure5illustratesanexampleofhowtoimplementSCDType1inaDataServicesdataflow.

Figure5SCDType1DataFlow

ThefollowingstepsdescribehowtoimplementeachcomponentwhencreatingadataflowtoupdateanSCDType1dimensiontable.

1. Thedatasourceisthe table,whichisqueriedwithnotransformation.

2. The stepisusedtocomparethesourcetabletothetargetdimensiontableusingtheEMP_NUMcolumntocompareEMP_PAY_RATE(Figure6).

Figure6ComparingtheSourceEMPLOYEETabletotheTargetDIM_EMPLTableontheEMP_PAY_RATEColumnOnlyUsingEMP_NUMastheKey

3. The transformthendetermineswhethertoinsertanewrecord,orupdate/overwritetheexistingrecord.InFigure7theMAP_OPERATIONinsertsnewrowsbasedonEMP_NUMandupdatesexistingrowsbasedonchangestoEMP_PAY_RATE.

Figure7MapOperation

4. The transformcreatessurrogatekeysfornewlyinserteddimensionrecords.InFigure8,KEY_GENERATIONisusedtocreateanewsurrogatekeyfortheEMP_SKcolumnandisincrementedby1foreachnewrecord.

Figure8UsingtheKey_GenerationTransformtoGenerateaNewSurrogateKey

SCDType2

SCDType2isprobablythemostcommonlyusedpatterntopreservechangehistoryfordimensiontables.Type2preserveshistorybycreatinganewrecordwiththelatestversionoftheentity,enddatingthepreviousversionoftherecord,andoftenmaintainingacurrentrecordindicatoraswell.InTable4,weseeanexistingrecordintheDIM_EMPLtablefor“VeronicaMeyer”withacurrentpayrateof“22”.

EMP_SK EMP_NUM EMP_LNAME EMP_FNAME EMP_PAY_RATE EMP_START_DATE

328 2714 Meyer Veronica 22 2014.08.25

Table4ExistingEmployeeRecordinDIM_EMPL

Thisemployee’spayratehaschangedfrom22to31on10/16/2014(seeTable5).ThenexttimeweloadthisdimensionasSCDType2,theexistingemployeerecordisenddatedbyupdatingtheEMP_END_DATEwiththecurrentdateand

theEMP_CUR_INDtoN.Thenanewcurrentversionoftheemployeerecordiscreatedwiththenewdata.

EMP_SK EMP_NUM EMP_LNAME EMP_FNAME EMP_PAY_RATE EMP_START_DATE

328 2714 Meyer Veronica 22 2014.08.25

329 2714 Meyer Veronica 31 2014.10.17

Table5EndDatedandNewEmployeeRecordintheDIM_EMPLTable

Thefollowingdataflow(Figure9)illustratesanexampleofhowtoimplementSCDType2inDataServices.

ThisdataflowisverysimilartothepreviousexampleforType1.Thesourceisour table,whichisqueriedwithnotransformation.The

stepisusedtocomparethesourcetabletothetargetdimensiontableusingtheEMP_NUMcolumntocompareEMP_PAY_RATE.The

transformthenmanagestheType2updatesandinsertsbeforeakeyisgeneratedandthetargetdimensiontableisupdated.ThisisspecificallyusedinthisType2scenariotomanagemultipleversionsandrowsoftheemployeerecordovertimeandwouldbeconfiguredasshowninFigure10.


Figure10HistoryPreservingforSCDType2withValidDates,CurrentColumn,andCompareColumn

SCDType3

SCDType3preserveshistorybyusingtwocolumnsforcurrentandpreviousversionsofthedimensionentity.Thisallowsalimitednumberofversionstobetracked,usuallyonlytwo(currentandpreviousororiginal).InTable6,weseeanexistingrecordintheDIM_EMPLtablefor“VeronicaMeyer”withacurrentpayrateof“22”.

EMP_SK EMP_NUM EMP_LNAME EMP_FNAME ORIG_EMP_PAY_RATE


Table6ExistingEmployeeRecordintheDIM_EMPLTable

Thisemployee’spayratehaschangedfrom22to31on10/16/2014.ThenexttimeweloadthisdimensionasSCDType3,weupdatetheCUR_EMP_PAY_RATEandEFFECTIVE_DATEwiththenewpayrateandcurrentdate.TheresultscanbeseeninTable7,whereCUR_EMP_PAY_RATEhasbeenupdatedto“31”,andwepreservetheoriginalpayrateof“22”inORIG_EMP_PAY_RATE.

EMP_SK EMP_NUM EMP_LNAME EMP_FNAME ORIG_EMP_PAY_RATE


Table7UpdatedEmployeeRecordintheDIM_EMPLTable


TheexampleinFigure11illustratesanalternativemethodofcomparingdatasourceandtargetbyusingaqueryjoinoperation,whichissometimesmoreefficientifthesourceandtargettablesareinthesamedatabaseandthecomparisoncanbepusheddown.


Thesourcesareour tableand targetdimensiontable,whicharequeriedwithaleftouterjointodeterminewhethertherecordisneworexisting.Theexistingrecordsareupdatedusingthe transform,whilenewrecordsareinsertedwithsurrogatekeys.

SCDType4

SCDType4isoftenreferredtoasahistorytablebecauseitkeepsatableforcurrentrecordsandaseparatetableforhistoricalrecords.Someorallofthechangescanbekeptinthehistorytable,andsurrogatekeysfrombothtablesarereferencedfromthefacttables.

Thisemployee’spayratechangedfrom22to31on10/16/2014.ThenexttimeweloadthisdimensionasSCDType4,weupdatethecurrentemployeerecordEMP_PAY_RATEinthe table(seeTable8).Wethencreateanewrecordinthe tabletopreservethepreviousversionwiththecurrentdateasCREATE_DATE(seeTable9).



Table8CurrentEmployeeRecordintheDIM_EMPLTable

EMP_SK EMP_NUM EMP_LNAME EMP_FNAME EMP_PAY_RATE CREATE_DATE

472 2714 Meyer Veronica 22 2014.10.16

Table9EndDatedEmployeeRecordintheDIM_EMPL_HISTTable,andNewCurrentRecord



Thesourcesareour tableand targetdimensiontable,whicharequeriedwithaleftouterjointodeterminewhethertherecordisneworexisting.Theexistingrecordsareupdatedusingthe transform,andnewhistoricalrecordsareinsertedintothe table.Newemployeerecordsareinsertedintothe tablewithsurrogatekeys.

Late-ArrivingDimensionData

Normally,youloadthedimensiontablesfirstinastarschemaandthefacttableslastbecausethefacttablerecordsreferencethesurrogatekeysinthedimensiontables.Inreallife,however,dataintegrationsaren’tnecessarilysynchronizedinthecorrectorder.Sometimes,youmayhavefactrecordstoloadwherethecorrespondingdimensionrecordhasn’tyetarrived.

Consideranorderfacttablesituationinwhichtheordersarereadytobeinsertedintothefacttable,butthereisadelayinthelatestproductorcustomerdimensionrecordsbeingavailabletoloadintoyourDW.Ifyoulookupthesurrogatekey,you’llnotfindsomeofthesedimensionrecordsandthefactrecordcan’tbeinsertedwithaforeignkeyconstraint.Thisissometimescalledalate-arrivingdimensionrecord.

Atypicalwaytohandlethissituationistohaveadefaultrecordseededintoeachdimensiontablethathasabusinessmeaningof“unspecified”or“N/A,”andareservedsurrogatekeyvalueof0or1.Thisway,youcaninsertallfacttableseveniftheydon’tyethaveacorrespondingdimensionrecordloaded.Also,whenyouquerythedimensionalmodel,thefactsandmeasureswillcorrectlyaggregateandbedesignatedinthe“unspecified”or“N/A”categoryforanymissingdimensionrecords.

1.5LoadingFactTables

Afteryourdimensiontableshavebeenprocessed,youcanthenloadyourfacttables.Rememberthatafacttableisessentiallyacross-referenceofallthedimensionsinthestarschema,consistingofauniquecombinationofsurrogatekeysaswellasmeasuresandfacts.

SurrogateKeyPipeline

Loadingafacttablerequiresderivingallofthesurrogatekeysforeachfactrecordtobeprocessed.Thisprocesscanbeseenasalogicalpipelinewherethefactrecordgoesthroughasequenceoflookupsagainstthedimensionsusingthenaturalbusinesskeystoobtainthesurrogatekeys.

Thiscantheoreticallybeaccomplishedinajoinoperationwithinthedatabase,butfacttabledatasetsaretypicallyverylarge,andthisapproachoftenwon’tperformaslargesetoperationswhenattemptingtojoin15+tablesinasinglequery.Alternatively,executingaseparatejoinqueryagainstthedatabaseforeachfactrecordincurssubstantialoverheadandisusuallynotthemostefficientwaytoderivesurrogatekeysandloadthefacttable.

ThepreferredwaytoimplementthesurrogatekeypipelineinafacttableloadwithinDataServicesistousethe functioncallwithinaqueryobject.The functionallowsseveraloptionsforcachinglookupdata.Thisallowshighlyefficientlookupsinmemory,aswellasallowingtightcontroloverexecutionplanandcachingoptions.

Thepreferredwaytoimplementthe functionistocreateanewfunctioncallbyright-clickingonthequeryinquestionandchoosingNEWFUNCTIONCALL,asshowninFigure13.

Figure13CreatingaNewFunctionCallintheQueryEditor

Createanewfunctioncallforeachdimensionlookup.SpecifytheLOOKUPTABLEandCACHESPECoptions:

:Cacheinmemory.

:Partialcachinginmemorywithadditionalcachingondemandasnecessary.

:Row-by-rowlookup;nocaching.

Select wheneverpossible(seeFigure14).

Figure14SelectingCacheOptionsforlookup_ext

Next,we’lllookatsomeexampledataflowsintypicalfacttableloadingscenarios.

TransactionFactLoading

Atypicaltransactionfacttabledataflowusingthe functionwithalookupforeachsurrogatekeyisshowninFigure15.

Figure15MultipleCachedSurrogateKeyLookupsintheQueryEditor

Figure16showsasimpletransactionfactloadingdataflowwithsourcesalesdata,oursurrogatekeylookuppipelineasdescribedpreviously,andoursalesfacttableasthetarget.

Figure16SimpleTransactionFactProcessingDataFlow

PeriodicSnapshotandAggregateFactLoading

Aperiodicsnapshotoraggregatefacttabledataflowisverysimilartothetransactionfact,butyouaggregateyourmeasuresbyreportingperiods.ThemappedderivationtoperformthissummationisshowninFigure17.

Figure17AggregationofMeasuresforAggregatedFactTableLoad

AccumulatingSnapshotFactLoading

Loadinganaccumulatingsnapshotrequiresupdatingexistingfactrecordsaswellasinsertingnewrecords.Youcanuseaqueryobjectwithaleftouterjointhatcomparesthesourcefactrecordstothefacttabletodeterminewhethertherecordisneworneedstobeupdated,asshowninFigure18.

Figure18AccumulatingSnapshotFactTableDataFlow

2IntegrationwithPOSSystems

ThenexttwosectionsshowDataServicesjobsbeingappliedtosolveintegrationrequirementsintwodifferentindustries.Let’sfirstexploretheretailindustrywhereintegrationsarebeingleveragedtofacilitatecustomerloyaltysolutionsthatincreaseconsumersales.

Alargeretailerdeterminedthatitsprimarykeyinteractionpointswithconsumerscamewhentheyinquiredaboutaproductonline.Thecompanyfounditcouldrealizeacompetitiveadvantageifitintegrateditsexistingmobilestorefrontandin-storeapplicationswithitspoint-of-sale(POS)datatodeterminewhichinquiriesdidn’tendinconversion.Withtheseinquiries,afollow-uptouchpointwiththeconsumercouldbecreatedtoincreaseconversionofproductinquiriesonlineintosales.

Theretailerdecidedtotieproductinquiry,currentlocation,andproximitytothestoretogenerateaconsumerproduct-specificdiscountthathadaveryshortexpiration.ThesolutionwouldinvolveDataServicesjobsandafunctiontoeffectthefollowing:

Inquirycapture

Storeproximity

Productavailabilitycheck

We’llexplorethesesolutionsinthefollowingsubsections.

2.1InquiryCaptureJob

Thecompanyneedstocapturewhomadetheinquiry,whatproductswerelookedat,andwhattimetheywerelookedat.Thisinformationwillbemergedwithtransactionsfromthecompany’se-commercechanneltodeterminewhethertheinquiryresultedinasaleandtoidentifytheneareststorewithstocktothelocationwheretheinquirywasmade.

Toeffectthecreationoftheinquirycapturejob,thefollowingprimaryobjectsneedtobecreated:

Real-timejob

XSDXMLschemadefinitionfile

XMLSchemaFileFormatobject

DataflowfromsourceXMLmessagetoinquirytable

DataflowtotargetXMLmessage

Real-TimeJob

Createanewreal-timejob,asshowninFigure19.

Figure19CreationofNewReal-TimeJob

Insertadataflowbetweenthe and placeholdersasshowninFigure20.

Figure20Real-TimeJobLayout

XSDXMLSchemaDefinitionFile

Anytimeareal-timejobiscreated,oneXMLmessagesourceandoneXMLmessagetargetmustexistwithinoneortwoofitsdataflows.TocreateanXMLmessagesourceortargetfile,anXMLschemadefinitionfile(XSDfile)mustexist.

AnXSDfiledescribesthelayoutofanXMLdocument.Thisfiletypicallyisdenotedwithan.xsdfileextension.TocreateanXSDfile,youcanuseathird-partyeditororuseamethodthatcreatesatemporarydataflow.Here,weprovidethestepstocreateanXSDfilebyleveragingatemporarydataflow:

1. Dragthe tableontothedataflowworkspaceasbothasourceandatarget.Thenadda andconnectthem(Figure21).

Figure21TemporarilyConnectingTableasSource,Query,andTarget

2. Right-clickQUERYwithintheoutputschemaasshown,andchooseGENERATEXMLSCHEMA(Figure22).

3. AfterthenewXSDfilehasbeensaved,deletethetemporarysource,query,andtargetobjectsfromthedataflow.

Figure22ChoosingGenerateXMLSchemafromtheQueryOutputSchema

XMLSchemaFileFormatObject

Now,you’llusethecreatedXSDfile(s)asthestructureofanXMLSchemaFileFormatobject.

1. AftertheXSDfilehasbeencreated,navigatetotheLOCALOBJECTLIBRARY’sfileFORMATtab,right-click,andchooseNEW•XMLSCHEMA(Figure23).

Figure23ChoosingtoCreateaNewXMLSchema

2. Usingthedialogthatappears,specifythenewlycreatedXSDfilewithintheFILENAMEproperty,andsettheROOTELEMENTNAMEfieldto“Query”(seeFigure24)(orotherifanothermethodwasusedtocreatetheXSDfilethanthemethodspecifiedhere).

3. ClickOK,andtheXMLSchemaFileFormatisreadytobeusedwithinthefirstdataflow.

Figure24SpecifyingtheXMLSchemaFormat

DataFlowfromSourceXMLMessagetoInquiryTable

NowthattheXMLSchemaFormathasbeencreated,itcanbeusedonthefirstdataflow.

1. Dragitontotheworkspacearea.Upondropping,acontextmenuwillappearasshowninFigure25.EnterthenameforthesourcefileintheXMLFILEfield.

Figure25XMLFormatOptionsWhenDroppingontotheDataFlowWorkspace

2. AfterdroppingtheXMLSchemaformatontothedataflowworkspaceandspecifyinganame,thefollowingotherobjectsareadded(asshowninFigure26):

transform(withintheDataIntegratorTransformgroup)

targettable

Figure26DataFlow—XMLMessageSourcetoInquiryTable

DataFlowtoCreateTargetXMLMessage

Intheseconddataflow,atargetXMLmessageiscreated(seeFigure27).Thiswillbeusedtosendaresponsebacktotherequestorofthereal-timeservice.

Figure27TargetXMLMessage

2.2IdentifyDiscountTargetsBatchJob

InquiriesthathavebeenconvertedtosalesneedtohavetheirrepresentativeinquiryrecordsupdatedwiththecorrespondingPOStransactionidentifierthatwascreatedinthee-commercePOSsystem.Toeffectthisupdate,thefollowingprocessesoccur:

Updatinginquirieswiththee-commercePOSidentifier

Identifyingthecloseststorestothesitewheretheonlineinquirywasmade

Identifyingwhetherthecloseststorehastheproductinstock

Thesethreeprocessesareimplementedwithinonejobthathastwodataflows,asshowninFigure28.

Figure28IdentifyingDiscountTargets

UpdatingInquirieswiththeE-CommercePOSIdentifier

Thefirstdataflowinthejobperformsapushed-downupdatetoaddPOStransactionidentifiersfromthee-commercePOSsystemtoidentifythoseinquiriesthathavebeenconvertedtoasalewithoutdiscount.Thematchingisperformedusingtheloyaltyidentifier,theproduct,andaparameterizedtimeintervalinwhichthesalecouldoccur.ThedataflowisshowinFigure29.

Figure29DataFlowThatUpdatestheInquirywithE-CommercePOSIdentifiers

Inquiryrecordsthatareunconverted;thatis,thoserecordswithoutane-commercePOStransactionidentifier,areevaluatedinthenextdataflowasshowninFigure30.

Figure30DataFlowtoDetermineProximateStoreandOn-HandQuantity

DetermineStoreProximityofUnconvertedSalesDataFlow

Usingthelatitudeandlongitudeoftheinquiry,theproximateSTOREIDiscomparedwithstoresintheregion.Todothis,thetwosetsoflatitudeandlongitudearepassedintoa transformthatimplementstheHaversineformula(aformulafirstpublishedintheearly1800stocalculatedistancebetweentwopointsonasphere).

Inthiscase,thesphereisearth,andthetwopointsarethepointlatitudeandlongitudeoftheInternetconnectionfromwhichtheinquirywasmadeandthestoreswithinthatregion.TheimplementationofthatformulaisshowninFigure31.

Figure31InputOptionsofUser-DefinedTransformsImplementingHaversineFormula

InListing1,you’llseethePythoncodeusedforthisformulatocalculatethedistancebetweentwogeographicpositionsthatwillbeusedtodeterminethecloseststoretotheinquiryposition.frommathimportasin,cos,radians,sin,sqrt

defhaversine(lat1,lon1,lat2,lon2):

#convertdecimaldegreestoradians

lon1=float(lon1)

lon2=float(lon2)

lat1=float(lat1)

lat2=float(lat2)

#convertdecimaldegreestoradians

lon1,lat1,lon2,lat2=map(radians,[lon1,lat1,lon2,lat2])

d_lon=lon2-lon1

d_lat=lat2-lat1

a=sin(d_lat/2)**2+cos(lat1)*cos(lat2)*sin(d_lon/2)**2

c=2*asin(sqrt(a))

#6367kmistheradiusoftheEarth

km=6367*c

returnkm

print‘AssignUDTinputrecordcolumnvaluestovariables’

p_lat1=record.GetField(u’LATITUDE’)

p_lat2=record.GetField(u’LATITUDE_1’)

p_long1=record.GetField(u’LONGITUDE’)

p_long2=record.GetField(u’LONGITUDE_1’)

print‘Makecalltodefinedhaversinefunction’

dKm=haversine(p_long1,p_lat1,p_long2,p_lat2)

print‘SetassignresultvaluetoUDToutputcolumn’

record.SetField(u’distanceKm’,unicode(dKm))

Listing1PythonCodetoApplytheHaversineFormula

Afterdeterminingthedistanceofstoreswithintheassociatedregionwithwhichtheinquirywasmade,thestoreIDwiththeleastdistanceisaddedtotheinquiryrecordasshowninFigure32.

Figure32PortionofDataFlowtoCaptureClosestStoretoInquirySite

2.3ProductAvailabilityCheckLookup

Aftertheproximatestoreisfound,thestoreidentifierandproductidentifierarejoinedwiththecurrentstocksnapshottodeterminewhetherthatstorehason-handquantitybecausetheorganizationdoesn’twanttoincentivizediscountswithquickexpirationsforproductsthataren’tinstock.ThisprocessishighlightedinFigure33.Aftercheckingforapositivequantity(withaparameterizedmargin),thecloseststoreandin-stockindicatorareupdatedonthe table.

Figure33ProcesstoPerformIfThereIsOn-HandQuantityoftheProductattheStore

2.4Results

Thiswasasolutionwithaveryaggressivedesignanddevelopmentcycle.DataServicesprovedagainthatit’sabletodelivertherequiredfunctionalityandperformance.Thedatathatisbeingprovisionedisvaluablebeyondtheoriginaloperationalfunctionofencouragingmoresalesandisnowbeingusedandanalyzedfortuningdiscountstoincreasecustomersalesandloyalty.

3IntegrationwithSAPBW,SAPAPO,andSAPECC

Inthissection,wewilllookatthedistributionindustryandhowtouseDataServicestoreduceintegrationtimesbetweenSAPAdvancedPlanningandOptimization(SAPAPO)andSAPECC.

Alargeindustrialdistributioncompanyhadtocalculatethedeliverytimesofproductsbetweeninternalsitesaswellastocustomersitesasoneofitsbusinessprocesses.Therewerethreetypesofdeliverytimes:planneddeliverytime,actualdeliverytime,andaveragedeliverytime.Thedeliverytimeswerederivedfrompurchaseorderdataresidinginthecompany’sSAPECCsystem.Theresultsoftheplanneddeliverytimeandactualdeliverytimeweretobeusedbythecompany’sSAPAdvancedPlanningandOptimizing(SAPAPO)/ProjectSystem(SAPPS)tocalculateaproduct’ssafetystockandpipelineforecast.

Atahighlevel,theneedseemsstraightforwardandeasy,especiallyifviewedbackwards.ThebusinessusersneeddataaccessibleinSAPECC.ThedatacomesfromSAPAPO,whichisprovisionedfromDataServices.ThedatafromDataServicesisderivedfromSAPECC.Initially,dataisextractedfromSAPECC,deliverytimesarecalculatedinDataServices,andthenthedataisloadedintoSAPAPO.SAPAPOrunsaspartofSAPBusinessWarehouse(SAPBW),sotoeffecttheloadintoSAPAPO,weneedtoloaditintotheinstanceofSAPBWthatSAPAPOisrunning.AdditionalcalculationsaremadeinSAPAPOandthenextractedviaDataServicesandloadedintoSAPECC.

WhatseemedstraightforwardandeasywasattemptedbyadifferentExtraction,Transformation,andLoading(ETL)toolset.Thedetailsonthebusinessrequirementsforcalculatingdeliverytimesturnedouttobeverycomplex.Averagedeliverytimeswerecalculatedmonthly.Plannedandactualdeliverytimeswerecalculatedweeklyandforthepreviousweek.Thedeliverytimeswerecalculateddifferentlydependingonwhetherthetransactionwasinternalorexternal,whetherthevendorwasdomesticorforeign,whetherastatisticallyrelevantnumberofdatapointsexisted,andwhethertherewereoutliers.Asaresult,theETLtoolwasunabletointegratesystemsasneeded.Thedeliverytimecalculationtook23hoursandwasunreliable.

ThecompanyhadjustmadeaninvestmenttoimplementDataServices4.2,partlyfordatagovernanceandpartlytotakeoverprocesseslikethisthatnegativelyimpactthebusiness.

AsolutionwascreatedthatleveragesintegrationwithSAPECCthroughextractorsandIDocmessages,andthatleveragesintegrationwithSAPAPOthroughInfoPackagesandBusinessApplicationProgrammingInterface(BAPI)functioncalls.Thefirstjob(Figure34)wassetuptorunonaweeklybasistoextractpurchaseorderdatafromSAPECCandcalculatedeliverytimes.

Figure34DataServicesJobforSAPECCExtractionandCalculation

SAPECCextractorswereusedtoextractthepurchaseorderdata.Whereextractorsweren’tavailableorsetup,suchasmaterialandvendordata,ABAPdataflowswereused.

Dataflowswerecreatedtocalculatedeliverytimesperbusinessrequirements.

AsecondjobwasthencreatedtoloadthecalculationsintoSAPAPO.SAPAPOthenusedthatdatatoruncalculationsonsafetystockandpipelineforecasts.AfterthecalculationstookplaceinSAPAPO,resultswereextractedwithaBAPIfunctioncall.DatawasthenloadedintoSAPECCusingIDocmessages(Figure35).

Figure35DataServicesJobforSAPAPOExtractionandSAPECCLoad

3.1SAPECCExtraction

LeveragingtheSAPECCextractorsandtheirdeltaqueuesallowedforanefficientprocesstoextractthedatachanges.The transformwasusedinconjunctionwiththeextractorstomanagetheinsertsandupdatesintothestagingtables.ThedataflowisshowninFigure36.

Figure36SAPECCExtractor

3.2DeliveryTimeCalculations

Thecalculationsrequiredtodeterminedeliverytimesinvolvedtakingayear’sworthofdataandretrievingaspecifiednumberofrecordsforeachmaterial-locationcombination.Tomakethesecalculationsatthedatalevelandnotindividuallyrowbyrow,wecreatedaseriesofdataflowsthatproducedcalculationsforallscenarios.Thescenariosweredifferentsourcesforinternalversusexternaltransactions,differentvolumesofrecordsfordomesticversusforeign,anddifferentoutlierfactorsforexternaldomesticversusexternalforeignversusinternaltransactions.Ifaminimumvolumeofrecordsdidn’texist,thentheresultswerereplacedwithadefaultvalue.

Afterthecalculationswerecompleted, transformswereusedtoassociatetheappropriateresultswiththematerial-locationcombinationbasedonbusinessrequirementscriteria.Figure37showstheseriesofcasestatementstomaptheresultsforeachgivenscenario.The transformsproceedingthetransformsmapthecalculatedresultsintotheproperoutputfields.

Figure37CalculationOutputDataFlow

3.3SAPAPOInterfaces

Afterthecalculationsarecompleted,thedataneedstobeloadedintoSAPAPOandusedtohelpdeterminethesafetystockandpipeline.TheinterfacefromDataServicesintoSAPAPOusestheInfoPackageobject.AftertheSAPAPOcalculationsaremade,aDataServicesjobwillreadthosecalculatedvaluesfromSAPAPOusingaBAPIfunctioncall.

TousetheInfoPackage,acoupleofsetupswereneeded.ThedatastorecontainingtheconnectioninformationandmetadataissetupasaSAPBWtargetdatastore.DataServicesneededtobesetupasasourcesysteminSAPBW/SAPAPO.Oncesetupandactivated,it’savailableundertheexternalmetadatasources.

Figure38showsthedataflowrequiredtoconsolidatethedataandstructureproperlytoloadintoSAPAPO.

Figure38LoadingDatafromDataServicesintoSAPAPO

ToreaddatafromSAPAPO,wesetupadifferentdatastore,configuredasanSAPBWsourcesystem.WeusedaBAPIfunctioncalltoreaddataintoDataServicesasshowninFigure39.ThefunctioncallisastandardfunctionontheSAPAPO/SAPBWsystem,buttouseitinthisflow,theSAPAPOteamneededtosetuptheplanningareatohavethenecessarydataavailable.Thefunctioncallrequiredinputdatainanestedtable.Tosetupthenestedtablestructureanditerationcorrectly,weconnectedthesourcetablestothefunctionusingan

transform.

Figure39ReadingfromSAPAPOUsingaBAPIFunctionCall

3.4SAPECCInterfaces

ThefinalstepofthisprocesswastoloadtheSAPAPOcalculationdataintoSAPECC.Toaccomplishthis,weusedthe messagetransformastheinterface(Figure40).TheIDocwascreatedandexposedbytheSAPECCteam.Onceexposed,it’savailabletoimportfromtheexternalmetadatasectionoftheSAPECCdatastore.TomatchtheinputstructureneededfortheIDoc,weusedan

transformtostructureanditeratethepayload;andweusedatransformtosettheheaderinformation.Theexpectedvolume

forthisloadingprocesswasexpectedtobe100,000individualmessages,soweoptedforIDocmessagesratherthanIDocfiles.

Figure40LoadingDataintoSAPECCwithIDocMessages

3.5Results

Consistentdatawasanongoingchallengeforseveraldifferentreasons.Theoriginaldesigninthedevelopmentenvironmentusedasubsetofdatathatwaspresentinproduction.Thesmallervolumesofdatahidtheperformanceissueswewouldfindlaterintheinterfaces.TheprimaryinterfacewasseeninsendinglargevolumesofinputdatatotheBAPIfunction.

TheBAPIfunctionrequiredinputintheformofmaterialandlocationkeys.Ifamaterial-locationcombinationwasnotpresentintheSAPAPOsystem,thenanerrorwouldberaised.Westarteddownthepathofsendingonlyvalidmaterial-locationcombinations,butfoundthevolumeofindividualcombinationswas~20millionrecords.SendingthoserecordstothefunctionwouldbogdowntheSAPAPOsystemandeventualraiseanerror.Tryingtothrottlewhatwouldbesentbyaddinga looppreventedthebogerrors,butthetimerequiredtoprocessindividualrequestswasexcessive.Adjustingtheinputs,wefoundthatomittingthematerialandonlysendinglocationkeysgreatlyimprovedtheperformance.TherequestwasbeinghandledmoreinabatchmodeinSAPAPObypullingallmaterialsassociatedwiththatlocation,preventingtheneedtospendresourcesontheDataServicessideforgroupingthematerialandlocationkeys,andnothavingtoprocesseachindividualrecordrequestinSAPAPO.

WealsofoundthatSAPECCextractors,whilesetupforchangedatacapture(CDC),didn’tactuallyproducetheoperationtypes.Thecompany’steamspentseveralcyclestweakingsomeoftheextractors,butultimatelyfoundtheycouldaccomplishthenecessarydataextractionusingABAPdataflowsandpullingdirectlyfromthetables.

Theextractorswerealsocustomizedtofitthecompany’sneeds;asaresult,wefoundmultiplesourcesforthesamedatathatdidn’tmatch.Somecustomizeddatawasn’tcurrent.Namingconventionswerenotconsistentbetweensystems,or,evenworse,theyusedthesamenamefordifferentsetsofdata.Toremedythisintheproject,wespentseveralcyclesmappingthenecessarydata,validatingthatitwasthecorrectdata.Foralargerpictureremedy,thecompany’sdatagovernanceteamislookingintoleveragingSAPInformationSteward’sMetapediatohaveacommondatadictionary.

Startingwiththeoriginalparameterofimprovingperformanceof23hours,wewereabletorunthefullprocessin3hours.ThisincludestheprocessingtimeinSAPAPOandtheadditionofloadingdataintoSAPECC(whichwasn’tpartoftheoriginalprocess).Theresultshelpedjustifythecompany’smovetoone

standardizedETLplatform,providingasystemconsistentandtightlyintegratedwithitsbusinessapplications,ratherthanhavingseveraldifferenttools.

Usage,Service,andLegalNotes

NotesonUsage

ThisE-Biteisprotectedbycopyright.BypurchasingthisE-Bite,youhaveagreedtoacceptandadheretothecopyrights.Youareentitledtousethise-bookforpersonalpurposes.Youmayprintandcopyit,too,butalsoonlyforpersonaluse.Sharinganelectronicorprintedcopywithothers,however,isnotpermitted,neitherasawholenorinparts.Ofcourse,makingthemavailableontheInternetorinacompanynetworkisillegal.

Fordetailedandlegallybindingusageconditions,pleaserefertothesectionLegalNotes.

http://usage_service_notes.html#legal_notes

ServicePages

Thefollowingsectionscontainnotesonhowyoucancontactus.

PraiseandCriticism

WehopethatyouenjoyedreadingthisE-Bite.Ifitmetyourexpectations,pleasedorecommendit.Ifyouthinkthereisroomforimprovement,pleasegetintouchwiththeeditoroftheE-Bite:HareemShafi.

Wewelcomeeverysuggestionforimprovementbut,ofcourse,alsoanypraise!YoucanalsoshareyourreadingexperienceviaTwitter,Facebook,oremail.

TechnicalIssues

Ifyouexperiencetechnicalissueswithyoure-bookore-bookaccountatSAPPRESS,pleasefeelfreetocontactourreaderservice:[email protected].

AboutUsandOurProgram

Thewebsitehttp://www.sap-press.comprovidesdetailedandfirst-handinformationonourcurrentpublishingprogram.Here,youcanalsoeasilyorderallofourbooksande-books.InformationonRheinwerkPublishingInc.andadditionalcontactoptionscanalsobefoundathttp://www.sap-press.com.

mailto:[email protected]

mailto:[email protected]

http://www.sap-press.com

http://www.sap-press.com

LegalNotes

Thissectioncontainsthedetailedandlegallybindingusageconditionsforthise-book.

CopyrightNote

Thispublicationisprotectedbycopyrightinitsentirety.AllusageandexploitationrightsarereservedbytheauthorandRheinwerkPublishing;inparticulartherightofreproductionandtherightofdistribution,beitinprintedorelectronicform.©2016byRheinwerkPublishingInc.,Boston(MA)

YourRightsasaUser

Youareentitledtousethise-bookforpersonalpurposesonly.Inparticular,youmayprintthee-bookforpersonaluseorcopyitaslongasyoustorethiscopyonadevicethatissolelyandpersonallyusedbyyourself.Youarenotentitledtoanyotherusageorexploitation.

Inparticular,itisnotpermittedtoforwardelectronicorprintedcopiestothirdparties.Furthermore,itisnotpermittedtodistributethee-bookontheInternet,inintranets,orinanyotherwayormakeitavailabletothirdparties.Anypublicexhibition,otherpublication,oranyreproductionofthee-bookbeyondpersonaluseareexpresslyprohibited.Theaforementioneddoesnotonlyapplytothee-bookinitsentiretybutalsotopartsthereof(e.g.,charts,pictures,tables,sectionsoftext).Copyrightnotes,brands,andotherlegalreservationsmaynotberemovedfromthee-book.

LimitationofLiability

Regardlessofthecarethathasbeentakenincreatingtexts,figures,andprograms,neitherthepublishernortheauthor,editor,ortranslatorassumeanylegalresponsibilityoranyliabilityforpossibleerrorsandtheirconsequences.

Imprint

ThisE-Biteisapublicationmanycontributedto,specifically:

EditorHareemShafiAcquisitionsEditorKellyGraceWeaverCopyeditorJulieMcNameeLayoutDesignGrahamGearyCoverDesignGrahamGearyProductionE-BookNicoleCarpenterTypesettingE-BookIII-satz,Husby(Germany)

ISBN978-1-4932-1309-2

©2016byRheinwerkPublishingInc.,Boston(MA)1stedition2016Allrightsreserved.Neitherthispublicationnoranypartofitmaybecopiedorreproducedinanyformorbyanymeansortranslatedintoanotherlanguage,withoutthepriorconsentofRheinwerkPublishing,2HeritageDrive,Suite305,Quincy,MA02171.

RheinwerkPublishingmakesnowarrantiesorrepresentationswithrespecttothecontenthereofandspecificallydisclaimsanyimpliedwarrantiesofmerchantabilityorfitnessforanyparticularpurpose.RheinwerkPublishingassumesnoresponsibilityforanyerrorsthatmayappearinthispublication.

“RheinwerkPublishing”andtheRheinwerkPublishinglogoareregisteredtrademarksofRheinwerkVerlagGmbH,Bonn,Germany.SAPPRESSisanimprintofRheinwerkVerlagGmbHandRheinwerkPublishing,Inc.

AllofthescreenshotsandgraphicsreproducedinthisE-Bitearesubjecttocopyright©SAPSE,Dietmar-Hopp-Allee16,69190Walldorf,Germany.

SAP,theSAPlogo,ABAP,Ariba,ASAP,Duet,hybris,SAPAdaptiveServerEnterprise,SAPAdvantageDatabaseServer,SAPAfaria,SAPArchiveLink,SAPBusinessByDesign,SAPBusinessExplorer(SAPBEx),SAPBusinessObjects,SAPBusinessObjectsWebIntelligence,SAPBusinessOne,SAPBusinessObjectsExplorer,SAPBusinessWorkflow,SAPCrystalReports,SAPd-code,SAPEarlyWatch,SAPFiori,SAPGanges,SAPGlobalTradeServices(SAPGTS),SAPGoingLive,SAPHANA,SAPJam,SAPLumira,SAPMaxAttention,SAPMaxDB,SAPNetWeaver,SAPPartnerEdge,SAPPHIRENOW,SAPPowerBuilder,SAPPowerDesigner,SAPR/2,SAPR/3,SAPReplicationServer,SAPSI,SAPSQLAnywhere,SAPStrategicEnterpriseManagement(SAPSEM),SAPStreamWork,SuccessFactors,Sybase,TwoGobySAP,andTheBest-RunBusinessesRunSAPareregisteredorunregisteredtrademarksofSAPSE,Walldorf,Germany.

AllotherproductsmentionedinthisE-Biteareregisteredorunregisteredtrademarksoftheirrespectivecompanies.

TheDocumentArchive

TheDocumentArchivecontainsallfigures,tables,andfootnotes,ifany,foryourconvenience.

Figure1ExampleStarSchemaModelforOrders

Figure2StarSchemawithConformedDimensions

Figure3TypicalStarSchema

Figure4ExampleOrdersStarSchema

Figure6ComparingtheSourceEMPLOYEETabletotheTargetDIM_EMPLTableontheEMP_PAY_RATEColumnOnlyUsingEMP_NUMastheKey

Figure7MapOperation

Figure8UsingtheKey_GenerationTransformtoGenerateaNewSurrogateKey

Figure10HistoryPreservingforSCDType2withValidDates,CurrentColumn,andCompareColumn

Figure13CreatingaNewFunctionCallintheQueryEditor

Figure14SelectingCacheOptionsforlookup_ext

Figure15MultipleCachedSurrogateKeyLookupsintheQueryEditor

Figure16SimpleTransactionFactProcessingDataFlow

Figure17AggregationofMeasuresforAggregatedFactTableLoad

Figure18AccumulatingSnapshotFactTableDataFlow

Figure19CreationofNewReal-TimeJob

Figure20Real-TimeJobLayout

Figure21TemporarilyConnectingTableasSource,Query,andTarget

Figure22ChoosingGenerateXMLSchemafromtheQueryOutputSchema

Figure23ChoosingtoCreateaNewXMLSchema

Figure24SpecifyingtheXMLSchemaFormat

Figure25XMLFormatOptionsWhenDroppingontotheDataFlowWorkspace

Figure26DataFlow—XMLMessageSourcetoInquiryTable

Figure27TargetXMLMessage

Figure28IdentifyingDiscountTargets

Figure29DataFlowThatUpdatestheInquirywithE-CommercePOSIdentifiers

Figure30DataFlowtoDetermineProximateStoreandOn-HandQuantity

Figure31InputOptionsofUser-DefinedTransformsImplementingHaversineFormula

Figure32PortionofDataFlowtoCaptureClosestStoretoInquirySite

Figure33ProcesstoPerformIfThereIsOn-HandQuantityoftheProductattheStore

Figure34DataServicesJobforSAPECCExtractionandCalculation

Figure35DataServicesJobforSAPAPOExtractionandSAPECCLoad

Figure36SAPECCExtractor

Figure37CalculationOutputDataFlow

Figure38LoadingDatafromDataServicesintoSAPAPO

Figure39ReadingfromSAPAPOUsingaBAPIFunctionCall

Figure40LoadingDataintoSAPECCwithIDocMessages

Footnotes

bing chen, james hanck, patrick hanck, scott hertel, allen...

Documents