five tuning tips for your datawarehouse

Five Tuning Tips For YourData WarehouseJeff Moss

My First PresentationYes, my very first presentationFor BIRT SIGFor UKOUGUseful Advice from friends and colleaguesUse graphics where appropriateFind a friendly or familiar face in the audienceImagine your audience is naked!but like Oracle, be careful when combining advice!

Be Careful Combining Advice!Thanks for the opportunity Mark!

AgendaMy backgroundFive tipsPartition for successSqueeze your data with data segment compressionMake the most of your PGA memoryBeware of temporal data affecting the optimizerFind out where your query is atQuestions

My BackgroundIndependent Consultant13 years Oracle experienceBlog: http://oramossoracle.blogspot.com/Focused on warehousing / VLDB since 1998First projectUK Music Sales Data MartProduces BBC Radio 1 Top 40 chart and many more2 billion row sales fact table1 Tb total database sizeCurrently working with Eon UK (Powergen)4Tb Production Warehouse, 8Tb total storageOracle Product Stack

What Is Partitioning ?Partitioning addresses key issues in supporting very large tables and indexes by letting you decompose them into smaller and more manageable pieces called partitions. Oracle Database Concepts Manual, 10gR2

Introduced in Oracle 8.0Numerous improvements sinceSubpartitioning adds another level of decompositionPartitions and Subpartitions are logical containers

Partition To Tablespace MappingPartitions map to tablespacesPartition can only be in One tablespaceTablespace can hold many partitionsHighest granularity is One tablespace per partitionLowest granularity is One tablespace for all the partitionsTablespace volatilityRead / WriteRead OnlyP_JAN_2005P_FEB_2005P_MAR_2005P_APR_2005P_MAY_2005P_JUN_2005P_JUL_2005P_AUG_2005P_SEP_2005P_OCT_2005P_NOV_2005P_DEC_2005T_Q1_2005T_Q2_2005T_Q3_2005T_Q4_2005T_Q1_2006P_JAN_2006P_FEB_2006P_MAR_2006T_Q3_2005Read / WriteRead Only

Why Partition ? - PerformanceImproved query performancePruning or eliminationPartition wise joinsRead only partitionsQuicker checkpointingQuicker backupQuicker recoverybut it depends on mapping of:partition:tablespace:datafileSELECT SUM(sales) FROM part_tabWHERE sales_date BETWEEN 01-JAN-2005 AND 30-JUN-2005Sales Fact Table* Oracle 10gR2 Data Warehousing Manual

Why Partition ? - ManageabilityArchivingUse a rolling window approachALTER TABLE ADD/SPLIT/DROP PARTITIONEasier ETL ProcessingBuild a new dataset in a staging tableAdd indexes and constraintsCollect statisticsThen swap the staging table for a partition on the targetALTER TABLEEXCHANGE PARTITIONEasier MaintenanceTable partition move, e.g. to compress dataLocal Index partition rebuild

Why Partition ? - ScalabilityPartition is generally consistent and predictableAssuming an appropriate partitioning key is usedand data has an even distribution across the keyRead only approachScalable backups - read only tablespaces are ignoredso partitions in those tablespaces are ignoredPruning allows consistent query performance

Why Partition ? - AvailabilityOffline data impact minimiseddepending on granularityQuicker recoveryPruned data not missedEXCHANGE PARTITIONAllows offline buildQuick swap overP_JAN_2005P_FEB_2005P_MAR_2005P_APR_2005P_MAY_2005P_JUN_2005P_JUL_2005P_AUG_2005P_SEP_2005P_OCT_2005P_NOV_2005P_DEC_2005T_Q1_2005T_Q2_2005T_Q3_2005T_Q4_2005T_Q1_2006P_JAN_2006P_FEB_2006P_MAR_2006T_Q3_2005Read / WriteRead Only

Fact Table PartitioningTransaction DateLoad DateEasier ETL ProcessingEach load deals with only 1 partitionNo use to end user queries!Cant prune Full scans!Harder ETL ProcessingBut still uses EXCHANGE PARTITIONUseful to end user queriesAllows full pruning capability

Watch out forPartition exchange and table statistics1Partition stats updatedbut Global stats are NOT!Affects queries accessing multiple partitionsSolutionGather stats on staging table prior to EXCHANGEGather stats on partitioned table using GLOBALJonathan Lewis: Cost-Based Oracle Fundamentals, Chapter 2

Partitioning Feature: Characteristic Reason Matrix

What Is Data Segment Compression ?Compresses data by eliminating intra block repeated column valuesReduces the space required for a segmentbut only if there are appropriate repeats!Self containedLossless algorithm

Where Can Data Segment Compression Be Used ?Can be used with a number of segment typesHeap & Nested TablesRange or List PartitionsMaterialized ViewsCant be used withSubpartitionsHash PartitionsIndexes but they have row level compressionIOTExternal TablesTables that are part of a ClusterLOBs

How Does Segment Compression Work ?Database BlockSymbol TableRow Data Area

Pros & ConsProsSaves spaceReduces LIO / PIOSpeeds up backup/recoveryImproves query response timeTransparentTo readers and writersDecreases time to perform some DML Deletes should be quickerBulk inserts may be quickerConsIncreases CPU loadCan only be used on Direct Path operationsCTASSerial Inserts using INSERT /*+ APPEND */Parallel Inserts (PDML)ALTER TABLEMOVEDirect Path SQL*LoaderIncreases time to perform some DMLBulk inserts may be slowerUpdates are slower

Ordering Your Data For Maximum BenefitsColocate data to maximise compression benefits

For maximum compressionMinimise the total space required by the segmentIdentify most compressable column(s)For optimal accessWe know how the data is to be queriedOrder the data by Access path columns Then the next most compressable column(s)Uniformly distributedColocated

Get Max Compression Order PackagePROCEDURE mgmt_p_get_max_compress_order

Argument Name Type In/Out Default?------------------------------ ----------------------- ------ --------P_TABLE_OWNER VARCHAR2 IN DEFAULTP_TABLE_NAME VARCHAR2 INP_PARTITION_NAME VARCHAR2 IN DEFAULTP_SAMPLE_SIZE NUMBER IN DEFAULTP_PREFIX_COLUMN1 VARCHAR2 IN DEFAULTP_PREFIX_COLUMN2 VARCHAR2 IN DEFAULTP_PREFIX_COLUMN3 VARCHAR2 IN DEFAULT

BEGIN mgmt_p_get_max_compress_order(p_table_owner => AE_MGMT,p_table_name =>BIG_TABLE,p_sample_size =>10000);END:/

Running mgmt_p_get_max_compress_order...----------------------------------------------------------------------------------------------------Table : BIG_TABLESample Size : 10000Unique Run ID: 25012006232119ORDER BY Prefix:----------------------------------------------------------------------------------------------------Creating MASTER Table : TEMP_MASTER_25012006232119Creating COLUMN Table 1: COL1Creating COLUMN Table 2: COL2Creating COLUMN Table 3: COL3----------------------------------------------------------------------------------------------------The output below lists each column in the table and the number of blocks/rows and spaceused when the table data is ordered by only that column, or in the case where a prefixhas been specified, where the table data is ordered by the prefix and then that column.From this one can determine if there is a specific ORDER BY which can be applied toto the data in order to maximise compression within the table whilst, in the case of aa prefix being present, ordering data as efficiently as possible for the most commonaccess path(s).----------------------------------------------------------------------------------------------------NAME COLUMN BLOCKS ROWS SPACE_GB============================== ============================== ============ ============ ========TEMP_COL_001_25012006232119 COL1 290 10000 .0022TEMP_COL_002_25012006232119 COL2 345 10000 .0026TEMP_COL_003_25012006232119 COL3 555 10000 .0042

Data Warehousing SpecificsStar Schema compresses better than NormalizedMore redundant dataFocus onFact Tables and Summaries in Star SchemaTransaction tables in Normalized SchemaPerformance Impact1Space SavingsStar schema: 67%Normalized: 24%Query Elapsed TimesStar schema: 16.5%Normalized: 10%1 - Table Compression in Oracle 9iR2: A Performance Analysis

Things To Watch Out ForDROP COLUMN is awkwardORA-39726: Unsupported add/drop column operation on compressed tablesUncompress the table and try again - still gives ORA-39726!After UPDATEs data is uncompressedPerformance impactRow migrationUse appropriate physical design settingsPCTFREE 0 - pack each blockLarge blocksize - reduce overhead / increase repeats per block

PGA Memory: What For ?SortsStandard sorts [SORT]Buffer [BUFFER]Group By [GROUP BY (SORT)]Connect By [CONNECT-BY (SORT)]Rollup [ROLLUP (SORT)]Window [WINDOW (SORT)]Hash Joins [HASH-JOIN]IndexesMaintenance [IDX MAINTENANCE SOR]Bitmap Merge [BITMAP MERGE]Bitmap Create [BITMAP CREATE]Write Buffers [LOAD WRITE BUFFERS]Serial ProcessPGADedicated ServerCursorsVariablesSort Area[] V$SQL_WORKAREA.OPERATION_TYPE

PGA Memory Management: ManualThe old way of doing things Still available though even in 10g R2ConfiguringALTER SESSION SET WORKAREA_SIZE_POLICY=MANUAL;Initialisation parameter: WORKAREA_SIZE_POLICY=MANUALSet memory parameters yourselfHASH_AREA_SIZESORT_AREA_SIZESORT_AREA_RETAINED_SIZEBITMAP_MERGE_AREA_SIZECREATE_BITMAP_AREA_SIZEOptimal values depend on the type of work1One size does not fit all!1 - Richmond Shee: If Your Memory Serves You Right

PGA Memory Management: AutomaticThe new way from 9i R1Default OFF in 9i R1/R2Enabled by setting at session/instance level: WORKAREA_SIZE_POLICY=AUTOPGA_AGGREGATE_TARGET > 0Default ON since 10g R1Oracle dynamically manages the available memory to suit the workloadBut of course, its not perfect!Joe Seneganik - Advanced Management Of Working Areas In Oracle 9i/10g, presented at UKOUG 2005

Auto PGA Parameters: Pre 10gR2WORKAREA_SIZE_POLICYSet to AUTOPGA_AGGREGATE_TARGETThe target for summed PGA across all processesCan be exceeded if too smallOver Allocation_PGA_MAX_SIZETarget maximum PGA size for a single processDefault is a fixed value of 200MbHidden / Undocumented ParameterUsual caveats apply

Auto PGA Parameters : Pre 10gR2_SMM_MAX_SIZELimit for a single workarea operation for one processDerived DefaultLEAST(5% of PGA_AGGREGATE_TARGET , 50% of _PGA_MAX_SIZE)Hits limit of 100MbWhen PGA_AGGREGATE_TARGET is >= 2000MbAnd _PGA_MAX_SIZE is left at default of 200MbHidden / Undocumented ParameterUsual caveats apply

Auto PGA Parameters : Pre 10gR2_SMM_PX_MAX_SIZELimit for all the parallel slaves of a single workarea operationDerived Default30% of PGA_AGGREGATE_TARGETHidden / Undocumented ParameterUsual caveats applyParallel slaves still limited_SMM_MAX_SIZEImpacts only when PGA_AGGREGATE_TARGET: 3000Mb _PGA_MAX_SIZE = 200Mb_SMM_MAX_SIZE = 100Mb_SMM_PX_MAX_SIZE = 900Mb

10gR2 Improvements_SMM_MAX_SIZE now the driverMore advanced algorithm

_PGA_MAX_SIZE = 2 * _SMM_MAX_SIZEParallel operations_SMM_PX_MAX_SIZE = 50% * PGA_AGGREGATE_TARGET When DOP 5 _smm_px_max_size / DOP is usedJoe Seneganik - Advanced Management Of Working Areas In Oracle 9i/10g, presented at UKOUG 2005

PGA Target Advisorselect trunc(pga_target_for_estimate/1024/1024) pga_target_for_estimate, to_char(pga_target_factor * 100,'999.9') ||'%' pga_target_factor, trunc(bytes_processed/1024/1024) bytes_processed, trunc(estd_extra_bytes_rw/1024/1024) estd_extra_bytes_rw, to_char(estd_pga_cache_hit_percentage,'999') || '%' estd_pga_cache_hit_percentage, estd_overalloc_countfrom v$pga_target_advice/

PGA Target For PGA Tgt Estimated Extra Estimated PGA Estimated Estimate Mb Factor Bytes Processed Bytes Read/Written Cache Hit % Overallocation Count-------------- ------- ---------------- ------------------ --------------- -------------------- 5,376 12.5% 5,884,017 7,279,799 45% 113 10,752 25.0% 5,884,017 3,593,510 62% 8 21,504 50.0% 5,884,017 3,140,993 65% 0 32,256 75.0% 5,884,017 3,104,894 65% 0 43,008 100.0% 5,884,017 2,300,826 72% 0 51,609 120.0% 5,884,017 2,189,160 73% 0 60,211 140.0% 5,884,017 2,189,160 73% 0 68,812 160.0% 5,884,017 2,189,160 73% 0 77,414 180.0% 5,884,017 2,189,160 73% 0 86,016 200.0% 5,884,017 2,189,160 73% 0 129,024 300.0% 5,884,017 2,189,160 73% 0 172,032 400.0% 5,884,017 2,189,160 73% 0 258,048 600.0% 5,884,017 2,189,160 73% 0

Beware Of Temporal Data Affecting The OptimizerSlowly Changing DimensionsCover ranges of timeFrom and To DATE columns define applicabilityNeed BETWEEN operator to retrieve rows for a reporting point in timeSELECT * FROM d_customer WHERE 15/01/2005 BETWEEN valid_from AND valid_toMonth 11st Jan, 2004Month 21st Feb, 2004

Dependent PredicatesWhen multiple predicates exist, individual selectivities are combined using standard probability math1:P1 AND P2S(P1 & P2) = S(P1) * S(P2)P1 OR P2S(P1 | P2) = S(P1) + S(P2) [S(P1) * S(P2)]Only valid if the predicates are independent otherwiseIncorrect selectivity estimateIncorrect cardinality estimatePotentially suboptimal execution planBETWEEN is multiple predicates!Also known as Correlated Columns21 Wolfgang Breitling, Fallacies Of The Cost Based Optimizer2 Jonathan Lewis, Cost-Based Oracle Fundamentals, Chapter 6

Some Test TablesConsider these 3 test tables12 records in an SCD type table

Sheet1

Table 1Table 2Table 3

KeyNon Key AttrFromToKeyNon Key AttrFromToKeyNon Key AttrFromTo

1Jeff01-Jan-200531-Jan-20051Jeff01-Jan-200530-Jun-20051Jeff01-Jan-200531-Dec-2005

2Mark01-Feb-200528-Feb-20052Mark01-Feb-200530-Jun-20052Mark01-Feb-200531-Dec-2005

3Doug01-Mar-200531-Mar-20053Doug01-Mar-200530-Jun-20053Doug01-Mar-200531-Dec-2005

4Niall01-Apr-200530-Apr-20054Niall01-Apr-200530-Jun-20054Niall01-Apr-200531-Dec-2005

5Tom01-May-200531-May-20055Tom01-May-200530-Jun-20055Tom01-May-200531-Dec-2005

6Jonathan01-Jun-200530-Jun-20056Jonthan01-Jun-200530-Jun-20056Jonthan01-Jun-200531-Dec-2005

7Lisa01-Jul-200531-Jul-20057Lisa01-Jul-200531-Dec-20057Lisa01-Jul-200531-Dec-2005

8Cary01-Aug-200531-Aug-20058Cary01-Aug-200531-Dec-20058Cary01-Aug-200531-Dec-2005

9Mogens01-Sep-200530-Sep-20059Mogens01-Sep-200531-Dec-20059Mogens01-Sep-200531-Dec-2005

10Anjo01-Oct-200531-Oct-200510Anjo01-Oct-200531-Dec-200510Anjo01-Oct-200531-Dec-2005

11Larry01-Nov-200530-Nov-200511Larry01-Nov-200531-Dec-200511Larry01-Nov-200531-Dec-2005

12Pete01-Dec-200531-Dec-200512Pete01-Dec-200531-Dec-200512Pete01-Dec-200531-Dec-2005

FromToFromToFromTo

Max Value01-Dec-200531-Dec-2005Max Value01-Dec-200531-Dec-2005Max Value01-Dec-200531-Dec-2005

Min Value01-Jan-200531-Jan-2005Min Value01-Jan-200530-Jun-2005Min Value01-Jan-200531-Dec-2005

Num Distinct1212Num Distinct122Num Distinct121

Num Rows1212Num Rows1212Num Rows1212

Sheet2

Sheet3

Sheet1








6Jonthan01-Jun-200530-Jun-20056Jonathan01-Jun-200530-Jun-20056Jonthan01-Jun-200531-Dec-2005







FromToFromToFromTo





Sheet2

Sheet3

Sheet1








6Jonthan01-Jun-200530-Jun-20056Jonthan01-Jun-200530-Jun-20056Jonathan01-Jun-200531-Dec-2005







FromToFromToFromTo





Sheet2

Sheet3

Optimizer Gets Incorrect Cardinalityselect * from test_1_distinct_tdwhere to_date('09-OCT-2005','DD-MON-YYYY') between from_date and to_date;

KEY NON_KEY_AT FROM_DATE TO_DATE---------- ---------- --------- --------- 1 Jeff 01-JAN-05 31-DEC-05 2 Mark 01-FEB-05 31-DEC-05 3 Doug 01-MAR-05 31-DEC-05 4 Niall 01-APR-05 31-DEC-05 5 Tom 01-MAY-05 31-DEC-05 6 Jonathan 01-JUN-05 31-DEC-05 7 Lisa 01-JUL-05 31-DEC-05 8 Cary 01-AUG-05 31-DEC-05 9 Mogens 01-SEP-05 31-DEC-05 10 Anjo 01-OCT-05 31-DEC-05

10 rows selected.

Execution Plan----------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 11 | 264 | 3 (0)| 00:00:01 ||* 1 | TABLE ACCESS FULL| TEST_1_DISTINCT_TD | 11 | 264 | 3 (0)| 00:00:01 |----------------------------------------------------------------------------------------

Sheet1















FromToFromToFromTo





Sheet2

Sheet3

And Againselect * from test_2_distinct_tdwhere to_date('09-OCT-2005','DD-MON-YYYY') between from_date and to_date;

KEY NON_KEY_AT FROM_DATE TO_DATE---------- ---------- --------- --------- 7 Lisa 01-JUL-05 31-DEC-05 8 Cary 01-AUG-05 31-DEC-05 9 Mogens 01-SEP-05 31-DEC-05 10 Anjo 01-OCT-05 31-DEC-05

4 rows selected.

Execution Plan----------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 11 | 264 | 3 (0)| 00:00:01 ||* 1 | TABLE ACCESS FULL| TEST_2_DISTINCT_TD | 11 | 264 | 3 (0)| 00:00:01 |----------------------------------------------------------------------------------------

Sheet1















FromToFromToFromTo





Sheet2

Sheet3

And Againselect * from test_12_distinct_tdwhere to_date('09-OCT-2005','DD-MON-YYYY') between from_date and to_date;

KEY NON_KEY_AT FROM_DATE TO_DATE---------- ---------- --------- --------- 10 Anjo 01-OCT-05 31-OCT-05

1 row selected.

Execution Plan-----------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |-----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 4 | 96 | 3 (0)| 00:00:01 ||* 1 | TABLE ACCESS FULL| TEST_12_DISTINCT_TD | 4 | 96 | 3 (0)| 00:00:01 |-----------------------------------------------------------------------------------------

Sheet1















FromToFromToFromTo





Sheet2

Sheet3

WorkaroundsIgnore itIf your query still gets the right plan of course!HintsForce the optimizer to do as you tell itStored outlinesAdjust statistics held against the tableAffects any SQL that accesses that objectOptimizer Profile (10g)Offline Optimisation1Dynamic sampling level 4 or aboveSamples single table predicates that reference 2 or more columnsTakes extra time during the parse minimal but often worth it1 - Jonathan Lewis: Cost-Based Oracle Fundamentals, Chapter 2

Dynamic Sampling With A Hintselect /*+ dynamic_sampling(test_1_distinct_td,4) */ * from test_1_distinct_tdwhere to_date('09-OCT-2005','DD-MON-YYYY') between from_date and to_date;

KEY NON_KEY_AT FROM_DATE TO_DATE---------- ---------- --------- --------- 1 Jeff 01-JAN-05 31-DEC-05 2 Mark 01-FEB-05 31-DEC-05 3 Doug 01-MAR-05 31-DEC-05 4 Niall 01-APR-05 31-DEC-05 5 Tom 01-MAY-05 31-DEC-05 6 Jonathan 01-JUN-05 31-DEC-05 7 Lisa 01-JUL-05 31-DEC-05 8 Cary 01-AUG-05 31-DEC-05 9 Mogens 01-SEP-05 31-DEC-05 10 Anjo 01-OCT-05 31-DEC-05

10 rows selected.

Execution Plan----------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10 | 240 | 3 (0)| 00:00:01 ||* 1 | TABLE ACCESS FULL| TEST_1_DISTINCT_TD | 10 | 240 | 3 (0)| 00:00:01 |----------------------------------------------------------------------------------------

Sheet1








6Jonthan01-Jun-200530-Jun-20056Jonthan01-Jun-200530-Jun-20056Jonathan01-Jun-200531-Dec-2005







FromToFromToFromTo





Sheet2

Sheet3

Find Out Where Your Query Is AtData Warehouses are big, big, BIG!Big on rowsBig on disk storageBig on hardwareBig SQL statements issuedLots of data to scan, join and sortMany operationsLong runningSo where is my long running query at ?No solid answers here, just food for thought

A Big Query Execution Plan| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)|--------------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 1 | 124 | | 49722 (10)|| 1 | PX COORDINATOR | | | | | || 2 | PX SEND QC (RANDOM) | :TQ20006 | 1 | 124 | | 49722 (10)|| 3 | HASH JOIN | | 1 | 124 | | 49722 (10)|| 4 | BUFFER SORT | | | | | || 5 | PX RECEIVE | | 207K| 9510K| | 25982 (9)|| 6 | PX SEND BROADCAST | :TQ20000 | 207K| 9510K| | 25982 (9)|| 7 | VIEW | | 207K| 9510K| | 25982 (9)|| 8 | WINDOW SORT | | 207K| 10M| 26M| 25982 (9)|| 9 | MERGE JOIN | | 207K| 10M| | 25976 (9)|| 10 | TABLE ACCESS BY INDEX ROWID| AML_T_ANALYSIS_DATE | 1 | 22 | | 2 (0)|| 11 | INDEX UNIQUE SCAN | AML_I_ANL_PK | 1 | | | 0 (0)|| 12 | SORT AGGREGATE | | 1 | 9 | | || 13 | PX COORDINATOR | | | | | || 14 | PX SEND QC (RANDOM) | :TQ10000 | 1 | 9 | | || 15 | SORT AGGREGATE | | 1 | 9 | | || 16 | PX BLOCK ITERATOR | | 1 | 9 | | 2 (0)|| 17 | TABLE ACCESS FULL | AML_T_ANALYSIS_DATE | 1 | 9 | | 2 (0)|| 18 | FILTER | | | | | || 19 | FILTER | | | | | || 20 | TABLE ACCESS FULL | AML_T_BILLING_ACCOUNT_DIM| 82M| 2371M| | 5457 (5)|| 21 | HASH JOIN | | 18M| 1340M| | 23704 (10)|| 22 | HASH JOIN | | 10M| 500M| | 17005 (11)|| 23 | PX RECEIVE | | 10M| 265M| | 11304 (14)|| 24 | PX SEND HASH | :TQ20003 | 10M| 265M| | 11304 (14)|| 25 | BUFFER SORT | | 1 | 124 | | || 26 | VIEW | AML_V_MD_CUH_SID | 10M| 265M| | 11304 (14)|| 27 | HASH JOIN | | 10M| 337M| | 11304 (14)|| 28 | PX RECEIVE | | 17M| 310M| | 5228 (18)|| 29 | PX SEND HASH | :TQ20001 | 17M| 310M| | 5228 (18)|| 30 | PX BLOCK ITERATOR | | 17M| 310M| | 5228 (18)|| 31 | TABLE ACCESS FULL | AML_T_MEASURE_DIM | 17M| 310M| | 5228 (18)|| 32 | PX RECEIVE | | 34M| 461M| | 5958 (10)|| 33 | PX SEND HASH | :TQ20002 | 34M| 461M| | 5958 (10)|| 34 | PX BLOCK ITERATOR | | 34M| 461M| | 5958 (10)|| 35 | TABLE ACCESS FULL | AML_T_CUSTOMER_DIM | 34M| 461M| | 5958 (10)|| 36 | PX RECEIVE | | 55M| 1212M| | 5562 (3)|| 37 | PX SEND HASH | :TQ20004 | 55M| 1212M| | 5562 (3)|| 38 | PX BLOCK ITERATOR | | 55M| 1212M| | 5562 (3)|| 39 | TABLE ACCESS FULL | AML_T_CUSTOMER_DIM | 55M| 1212M| | 5562 (3)|| 40 | PX RECEIVE | | 94M| 2516M| | 6483 (5)|| 41 | PX SEND HASH | :TQ20005 | 94M| 2516M| | 6483 (5)|| 42 | PX BLOCK ITERATOR | | 94M| 2516M| | 6483 (5)|| 43 | MAT_VIEW ACCESS FULL | AML_M_CD_BAD | 94M| 2516M| | 6483 (5)|SortsAggregationsHash joinsMerge joinsTable scansMaterialized View scansAnalyticsParallel QueryPruningTemp Space Use

V$ Views To The Rescue ?V$SESSION Identify your sessionV$SQL_PLAN Get the execution plan operationsV$SQL_WORKAREA Get all the work areas which will be requiredV$SESSION_LONGOPS Get information on long plan operationsV$SQL_WORKAREA_ACTIVE Get the work area(s) being used right nowV$SESSION

SIDSERIAL#PROGRAMUSERNAMESQL_IDSQL_CHILD_NUMBERSQL_ADDRESSSQL_HASH_VALUEV$SQL_PLAN

SQL_IDCHILD_NUMBERADDRESSHASH_VALUEOPERATIONIDPARENT_IDV$SESSION_LONGOPS

SIDSERIAL#OPNAMETARGETMESSAGESQL_IDSQL_ADDRESSSQL_HASH_VALUEELAPSED_SECONDSV$SQL_WORKAREA_ACTIVE

SQL_ID SQL_HASH_VALUEWORKAREA_ADDRESSOPERATION_IDOPERATION_TYPEPOLICYSIDQCSIDACTIVE_TIMEV$SQL_WORKAREA

SQL_IDCHILD_NUMBERWORKAREA_ADDRESSOPERATION_IDOPERATION_TYPE

Demonstration

ProblemsV$SQL_PLAN BugService Request: 4990863.992 Broken in 10gR1, Works in 10gR2PARENT_ID corruptionCant link rows in this view to their parents as the values are corrupted due to this bugShows up in TEMP TABLE TRANSFORMATION operationsMultiple Work Areas can be activeor NoneSome operations are not shown in Long opsV$SESSION sql_id may not be the executing cursorE.g. for refreshing Materialized View* Test case for bug: http://www.oramoss.demon.co.uk/Code/test_error_v_sql_plan.sql

Questions ?

References: PapersTable Compression in Oracle 9iR2: A Performance AnalysisTable Compression in Oracle 9iR2: An Oracle White PaperFallacies Of The Cost Based Optimizer, Wolfgang BreitlingScaling To Infinity, Partitioning In Oracle Data Warehouses, Tim GormanAdvanced Management Of Working Areas in Oracle 9i/10g, UKOUG 2005, Joze SenegacnikOracle9i Memory Management: Easier Than Ever, Oracle Open World 2002, Sushil Kumar Working with Automatic PGA, Christo KutrovskyOptimising Oracle9i Instance Memory, Ramaswamy, RameshOracle Metalink Note 223730.1: Automatic PGA Memory Managment in 9iOracle Metalink Note 147806.1: Oracle9i New Feature: Automated SQL Execution Memory ManagementOracle Metalink Note 148346.1: Oracle9i Monitoring Automated SQL Execution Memory ManagementMemory Management and Latching Improvements in Oracle Database 9i and 10g , Oracle Open World 2005, Tanel PderIf Your Memory Serves You Right, IOUG Live! 2004, April 2004, Toronto, Canada, Richmond SheeDecision Speed: Table Compression In Action

References: Online Presentation / Codehttp://www.oramoss.demon.co.uk/presentations/fivetuningtipsforyourdatawarehouse.ppthttp://www.oramoss.demon.co.uk/Code/mgmt_p_get_max_compression_order.prchttp://www.oramoss.demon.co.uk/Code/test_dml_performance_delete.sqlhttp://www.oramoss.demon.co.uk/Code/test_dml_performance_insert.sqlhttp://www.oramoss.demon.co.uk/Code/test_dml_performance_update.sqlhttp://www.oramoss.demon.co.uk/Code/test_error_v_sql_plan.sqlhttp://www.oramoss.demon.co.uk/Code/run_big_query.sqlhttp://www.oramoss.demon.co.uk/Code/run_big_query_parallel.sqlhttp://www.oramoss.demon.co.uk/Code/get_query_progress.sql

Oracle Product Stack10gR1 Oracle Database, 10gR2 in Q2 2006Oracle DesignerOracle Enterprise Manager Grid ControlOracle Warehouse Builder

Definition is from 10gR2 Concepts manual

Introduced table and index partitioning in 8.0, many improvements since

Definition doesnt cover sub partitioning which is simply a further level of decomposition. Also it refers only to Tables and Indexes when other objects can also be partitioned.

Partitions and subpartitions are logical concepts essentially they are containers for information, in the same way tables are.Partitions map to tablespaces

1:many relationship between Tablespaces and PartitionsPartitions can only be in one tablespace they cant span tablespacesTablespaces can hold data from numerous partitionsHighest granularity is when each partition is stored in its own tablespace whilst lowest granularity has only 1 tablespace for all the partitions

Tablespace Volatility

Read / Write e.g. for current, open transaction partitions and data which may undergo changesRead Only for older data which is not going to be changed through normal operations

Granularity affects

Ability to use read only if you have only 1 tablespace for all the partitions of a table, then you cant use read only since youll be adding some new data shortlyPerformance, Manageability, Scalability and Availability are all enhanced by partitioning but the granularity of the partition to tablespace mapping can affect those benefits as the next few slides will showImproved Query Performance

Pruning or elimination

Referred to as either pruning or partition eliminationCost Based Optimizer can determine which (sub)partitions to access to process a query/DML operationProcess only the data thats relevant and your query will run quicker potentially much quicker

Partition wise joins

An optimization for joining two tables in parallelMinimises query execution time by reducing the level of data exchanged between parallel execution slavesReduces memory requirementsMinimises data across the interconnect in RAC environments thereby aiding performance

Full partition wise join

Occurs when both tables are partitioned on the same column(s)

Partial Partition wise join

Occurs when only one table is partitionedOracle dynamically repartitions the unpartitioned table in the same manner as the partitioned table

Read only partitionsDepending on partition to tablespace to datafile mapping checkpointing, backup and recovery can be speeded up.If tablespace to partition mapping is very granular, e.g. 1 tablespace per partition, more tablespaces, and by necessity the partitions within those tablespaces, can be put into read only. Less granular implementations will not be as flexible in this sense.

Improves checkpoint performance since tablespaces that are read only do not get visited during a checkpointImproves backup since read only tablespaces need only be backed up once (or twice to be safe!)Improves recovery in circumstances where the granularity of the partition to tablespace mapping allows less tablespace(s) to be recovered and therefore less dataIt is possible that recovery could take longer if the older data is held on storage media which is slower to access in multi tier storage architectures. Essentially, this aspect is all down to the media management strategy. [Suggested to me by Andrew Buesnel, Eon DBA]

ArchivingArchiving is far easier with partitioned tables since you can use the DROP PARTITION operation of the ALTER TABLE command.Often a rolling window approach is used for date range based information.ALTER TABLE ADD PARTITION or if end partition has maximum range value then SPLIT PARTITIONAfter data has reached a certain age it can be rolled off the available data stack and archived using DROP PARTITION.Alternative with a single amorphous table would be to use DELETE DML operation which would be much slower and certainly not scalable.

ETL Processing & Exchange PartitionBuild the data offline out of the way of the current target table let the users carry on using the targetLoad a staging table of the same structure as the target with validated dataAdd constraints RELY NOVALIDATEIndex the staging tableGather statistics on the staging tableThen swap the staging table with the relevant partition of the target table using ALTER TABLE EXCHANGE PARTITIONMaintenance is easierTable Partitions can be moved using ALTER TABLE MOVE PARTITION e.g. to compress the data ready for moving to Read OnlyPartition based processing is generally consistentExceptions to every rule of course ramping up data takeon, adding new feeds, inorganic business growth can all affect thisWarehousing revolves around time so generally a time based key will provide a good partitioning keyRolling Window approach keeps the same amount of data in scope / online therefore performance is consistent

Read only tablespaces allow you to back up only whats in Read / Write scope which is generally consistent therefore scalable

Pruning allows queries to only process in scope data time based partitioning key allows queries to therefore be consistent and scalable

Impact of offline data is minimisedBut it depends on the partition:tablespace mappingLets say we have a media failure in one of the data file for tablespace T_Q1_2005Only partitions P_JAN_2005, P_FEB_2005 and P_MAR_2005 are unavailableIf the mapping were more granular it might only affect one of these partitions

Media Failure may be recovered quicker with a more granular partition:tablespace mapping since less data may need to be retrieved from tape

Queries which dont require data from partitions which are in offline tablespaces will still execute and return results without being affected materially or in terms of performance

ETL Processing can use EXCHANGE PARTITION to build new datasets and swap them in on a (sub)partition by (sub)partition basis.When you swap in an analyzed table for a partition on the partitioned target table you still need to consider doing an analyze of the entire table (with a small estimate percentage) otherwise any queries which use more than 1 partition will use the table level statistics and these may be inaccurate, causing the optimizer to make an ill informed choice.

Gather stats on partitioned table using GLOBALBEGIN DBMS_STATS.GATHER_TABLE_STATS (ownname => 'MY_SCHEMA' ,tabname => 'MY_TABLE' ,granularity => 'GLOBAL' ,cascade => TRUE);END;

I guess the fundamental point to take from this slide is that Partitioning offers significant opportunities to improve many aspects of your system whether it be OLTP or a DSS/Warehouse environment.

Careful planning of the use and implementation of such features is necessary to reap the benefits though.Compresses data by eliminating intra block repeated column valuesWorks at block levelMore repeats = higher compressionTables with high repeats includes those with lots of repeated Foreign Keys, e.g. Fact and Summary Tables within a Data Warehouse environment

Reduces the space required for a segmentSpace saving depends on number of intra block repeats.If the value is too small it wont bother trying to tokenize and compress it since it would not offer any value.

Self contained data to recreate the uncompressed form of all rows in a given compressed block is stored within that block

Lossless after data is compressed it can be uncompressed and nothing is lost unlike lossy algorithms used in for example, media streaming.

Can be used with a number of segment types but there are restrictions:Cant be used on any type of subpartitionCant be used on Hash PartitionsCant be used on Indexes but they have their own type of row based compression for composite indexesCant be used on Index Organized Tables, overflow segments, partitions of overflow segment or mapping table segmentsCant be used on external tablesCant be used on tables that are part of a clusterCant be used with any LOB constructs

Can be set at tablespace levelin which case every object created in that tablespace without specifying the COMPRESSion physical storage attribute will be created with the tablespace settingCompressed block structure similar to uncompressedAddition of Symbol Table to store tokens of repeated values

This is a very simplistic animated slide showing 3 rows being placed in the table.

Firstly, the row with ID 100 comes alongall the values are new and are all placed in the Symbol Table then the appropriate pointers to the tokens are used to insert the row into the data area of the block

Next, row with ID 101 comes alongall of the values are new except for the OUTCOME column which is set to NO as it was for the row with ID 100. The Symbol Table is extended to include the new values onlythen the appropriate pointers to the tokens are used to insert the row into the data area of the block

Finally, the row with ID 102 comes alongall of the values are now known except for the ID column. The Symbol Table is extended to include the new value of 102 onlythen the appropriate pointers to the tokens are used to insert the row into the data area of the block

And so onProsSaves SpaceReducing space required for a given amount of data will reduce blocks readReducing blocks read will likely reduce the amount of LIO and PIO on the systemLess to read (logically or physically) is going to take less time faster queries and backup/recoveryBuffer cache is effectively upsized for freeTransparent to readers and writersUsers dont need to know compression is involved queries are obliviousDecreases time to perform some DMLDeletes should be faster since the compressed rows are smaller and therefore there is less redoSimple test I did on XP based 10gR2 showed 6% less redo but obviously this will differ across individual environments and test circumstancestest_dml_performance_delete.sql in accompanying test suiteInserts may be faster in some circumstances, e.g. if the disk subsystem is IO Bound, there is CPU to spare, the data is highly compressed or a combination thereofConsIncreases CPU LoadThe CPU has to work harder to compress and uncompress the data as needed so CPU load will go upAllegedly twice as much load on CPU but Ive not seen any empirical evidence and no opportunity to test thisMost systems are not in an unlimited IO scenario though so most bulk loads will be IO bound and therefore the CPU loading is unlikely to become the bottleneckGiven that less blocks are to be read/written when compression is used, it is likely that the IO load will reduce and therefore the extra CPU load is offset against the IO benefitsSimple test I did on XP based 10gR2 showed a fall in elapsed time from 80s to 60s when the target was a compressed table.Can only be used on Direct Path OperationsCreate Table As SelectINSERT /*+ APPEND */SQL*Loader Direct Path (DIRECT=TRUE)Parallel INSERT (PDML)Increases time to perform some DMLInserts may be slower in some circumstances, e.g. if the disk subsystem is not IO Bound, there is little CPU to spare, the data is not highly compressed or a combination thereofSimple test I did on XP based 10gR2 showed minimal increase in reduction in elapsed time but each environment is different so testing is advised.test_dml_performance_insert.sql in accompanying test suiteUpdates are slower can be much slower depending on the environment.Simple test I did on XP based 10gR2 showed an increase in elapsed time from 45s to nearly 5 minutes when doing an update of 180K rows in a 5M row tabletest_dml_performance_update.sql in accompanying test suiteHence the recommendation that compression should not be used where Updates are likely!Colocating allows us to maximise the compression benefits.We colocate it by ordering the data but by what key ?

Can decide to go for maximum compressionMinimise space required

or optimal accessorder by that column or columns which we will access via.

..or choose a combination of both.

Can be painful to work out which order it bestso I wrote a simple package to do itPurpose built package developed to determine optimal compression ordering for colocated data.

Freely distributable. No warranty implied or otherwise.

Available from my website at:

http://www.oramoss.demon.co.uk/Code/mgmt_p_get_max_compression_order.prc

In warehousing the Star Schema will compress better than the Normalized schema since the Fact Tables and Summaries which are the ideal candidates for compression in Star Schemas are greater in volume but have more redundancy than the Transaction tables in a Normalized Schema.

Star v 3NF is a discussion for another day!

Query elapsed times against both Star and Normalized schemas are improved although it depends on the query and whether it is CPU or IO bound as to whether it will be improved or made worse.

Normalised:Varies by query with some queries longer and some quickerCPU bound queries are slowerIO Bound queries are quicker

Dropping a column is awkward as it produces an error due to it not being supported even in 10gR2.Workaround with CTAS is cumbersome to say the least but still possible.WorkaroundsCTAS a new table without the undesired column(s) and swap itCreate a view with the required columns on top of the table

One should perhaps ask why a column is being dropped anyway

Once an UPDATE DML operation is performed against a table then any compressed blocks where there is a row which has been updated will no longer be compressed. To recompress the table use ALTER TABLE MOVE statement or the partition equivalent. Updates are slower against compressed tables for tables where updates are anticipated, it is probably better to not use Compression.

Using PCTFREE 0 will ensure that the blocks are packed, whilst using a larger blocksize will also ensure that there is more likelihood of compression factoring out repeated values. These guidelines fit well with Data Warehouses anyway.

Each sort or hash join is different has different memory requirements.If there are 2 sorts and 2 hash joins for a given process it will have 4 different memory profiles for those operations some will require more memory than others.MANUAL management of PGA means that you must set values before a process and those values are used for all operations within that process. A little unrefined and wasteful of memory at times. It also requires that you will know what memory each process is going to need before you run itor more likely that youll just allocate a large amount to cover every eventuality the problem with that is that some sorts perform worse when they have more memory Richmond covers this in his paper.Lets look at the example on the right.

PGA_AGGREGATE_TARGET is set to 3000Mb_PGA_MAX_SIZE is the default of 200Mb_SMM_MAX_SIZE is the default of 100Mb_SMM_PX_MAX_SIZE is the default of 900Mb

Running with 4 parallel slaves means were not hitting the _SMM_PX_MAX_SIZE so were limited by the _SMM_MAX_SIZE.Running with 8 parallel slaves again means were not hitting the _SMM_PX_MAX_SIZE limit were still limited by the _SMM_MAX_SIZE.Once we go to 12 slaves though we see that _SMM_PX_MAX_SIZE is breached 1200Mb v 900Mb and it starts to limit the query with each slave getting only 75Mb instead of the previous 100Mb.

The formula to calculate the DOP where the _SMM_PX_MAX_SIZE will start to have an effect is thus:

DOP > Ceiling(_SMM_PX_MAX_SIZE / _SMM_MAX_SIZE)

An example from my Xp 10gr2 database:

pga_aggregate_target 96468992 ( 92 Mb)_pga_max_size 204800K (200 Mb)_smm_max_size 18841K (18.4 Mb)_smm_px_max_size 47104K (47Mb)

So, we can see everything matchesexcept the _PGA_MAX_SIZE is still 200Mb. This does not occur on HP-UX it works as indicated on the slide.

Not sure why that is maybe my testing, maybe a bug/XP issue.

We can determine whether weve set the PGA_AGGREGATE_TARGET correctly by using some SQL against the v$pga_target_advice tableor we can use EM to use the PGA Aggregate Target Advisor same graphical illustration as the text showing that this PGA is correctly sized.Jonathan Lewis gave a couple of good presentations a the UKOUG 2005 which dealt with using hints

Stored outlines are basically just a stored plan.

Adjusting statistics is finebut it then has far reaching effects on anything which accesses the object in question

Optimizer Profiles are new in 10g run from SQL Tuning task in Enterprise Manager. More on this in an article by Kimberley Floss (http://www.oracle.com/technology/oramag/oracle/04-mar/o24tech_tuning.html)

Dynamic Sampling level 4 will read 32 blocks of data from the table(s) in question to ascertain more accurate statistics which may make the CBO choose a better plan.Can take a little longer but its worth it if the plan is better.For warehousing, the extra time involved is usually inconsequential since the queries are generally longer running than OLTP.

In terms of the number of rowsMulti billion row tables are common

In use of disk storageMultiple Terabytes is common

In scale of hardwareNumerous CPUMultiple IO controllers and IO infrastructure componentsRAIDMore spindles the better for IO performance

In terms of the SQL statements issuedLots of data to scan, join and sortMany operationsLong running

So where is my long running query at ?

wouldnt it be great to know exactly which operation from a given Execution Plan, a query was working on ? It wouldnt necessarily tell you how long it had to go but it would give you an indication of how much of the query has been achieved so far from which you might, with additional knowledge, be able to infer how long things are likely to continue for before the statement completes.Demonstration steps:

Start a SQL*Plus as JEFF/JEFF@G101Start a second SQL*Plus session as JEFF/JEFF@G101Note the SID of the first sessionRun RUN_BIG_QUERY.SQL in the first sessionRun the GET_QUERY_PROGRESS.SQL script in the second session giving it the SID of the first session should show the planRun the get query progress script repeatedly during the life of the query running in the first session to see what happens over its life.Rerun the whole process using RUN_BIG_QUERY_PARALLEL.SQL in session 1 should show parallel query operations as well.Use V$OPEN_CURSOR to get the INSERT cursor handle

five tuning tips for your datawarehouse

Technology