data warehousing & big data at openworld · 2015-05-09 · dw architect, ise information...

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Data Warehousing & Big Data at OpenWorld for your smartphone

Smartphone and tablet apps,

helping you get the most from

this year’s OpenWorld

Access to all the most important information • Presenter profiles • Must-see sessions • Must-attend hands-on labs • Useful links

http://tinyurl.com/kmbsxbu





If you think Partitioning is only for Performance, think again

Matthias Fuchs DW Architect, ISE Information Systems Engineering Hermann Bär Database Product Management, Oracle



Partitioning for Data Management

Concepts and functionality

Some tips and tricks

Partitioning at work

Q&A

1

2

3

4


Oracle Partitioning in Oracle Database 12c

Core functionality Performance Manageability

Oracle 8.0 Range partitioning

Global Range indexes

Static partition pruning Basic maintenance: ADD, DROP, EXCHANGE

Oracle 8i Hash partitioning Range-Hash partitioning

Partition-wise joins

Dynamic partition pruning

Expanded maintenance: MERGE

Oracle 9i List partitioning Global index maintenance

Oracle 9i R2 Range-List partitioning Fast partition SPLIT

Oracle 10g Global Hash indexes Local Index maintenance

Oracle 10g R2 1M partitions per table Multi-dimensional pruning Fast DROP TABLE

Oracle 11g Virtual column based partitioning More composite choices REF partitioning

- Interval partitioning - Partition Advisor - Incremental statistics mgmt

Oracle 11g R2 Hash-Hash partitioning Expanded REF partitioning

“AND” pruning Multi-branch execution

Oracle 12c R1 Interval-REF partitioning - Partition Maintenance on multiple partitions

- Partial local and global indexes - Zone Maps

- Asynchronous global index maintenance for DROP/TRUNCATE

- Online partition MOVE - Cascading TRUNCATE/EXCHANGE

Business Modeling, Performance, and Data Management



• Logical attributes

– Partitioning setup

– Indexing and index maintenance

– Read only (in conjunction with tablespace separation)

• Physical attributes

– Data placement

– Segment properties in general


Nonpartitioned Tables

• Logical table properties

– Columns and data types

– Constraints

– Indexes, …

• Physical table properties

– Table equivalent to segment

– Tablespace

– Compression, [ Logging | nologging ], …

– In-memory

– Properties managed and changed on segment level

Physical and Logical Attributes


Partitioned Tables

• Logical table properties

– Columns and data types

– Constraints

– Partial Indexes, …

– Physical property directives

• Physical [sub]partition properties

– [Sub]partition equivalent to segment

– Tablespace

– Compression, [ Logging | nologging ], …

– In-memory

– Properties managed and changed on segment level


Partition Partition

Table


Partitioned Tables

• Table is metadata-only and directive for future partitions

– No physical segments on table level

– Physical attributes become directive for new partitions, if specified

• Single-level partitioned table – Partitions are equivalent to segments

– Physical attributes are managed and changed on partition level

• Composite-level partitioned tables

– Partitions are metadata only and directive for future subpartitions

– Subpartitions are equivalent to segments



Aspects of Data Management

• Fast population of data

• Fast removal of data

• Fast reorganization of data


Aspects of Data Management

• Fast population of data

– EXCHANGE

– Per-partition direct path load

• Fast removal of data – DROP, TRUNCATE, EXCHANGE

• Fast reorganization of data

– MOVE, SPLIT, MERGE

Addressable with Partition Maintenance Operations


Partition Maintenance Operations on multiple Partitions

• Partition Maintenance on multiple partitions in a single operation

• Full parallelism

• Transparent maintenance of local and global indexes

• Often combined with change of physical properties

– Location, compression

ALTER TABLE orders

MERGE PARTITIONS Jan2009, Feb2009, Mar2009

INTO PARTITION Quarter1_2009 COMPRESS FOR ARCHIVE

HIGH;

…

Jan 2009 Feb 2009 Mar 2009 Apr2009 Nov 2009 Dec 2009

…

Apr2009 Nov 2009 Dec 2009 Quarter 1 2009


Online Partition Move

• Transparent MOVE PARTITION ONLINE operation

• Concurrent DML and Query

• Index maintenance for local and global indexes

Jan2012 Aug2012 Sep2012 Oct2012

…

Nov2012 Dec2012


Asynchronous Global Index Maintenance

• Usable global indexes after DROP and TRUNCATE PARTITION without index maintenance

– Affected partitions are known internally and filtered out at data access time

• DROP and TRUNCATE become fast, metadata-only operations

– Significant speedup and reduced initial resource consumption

• Delayed Global index maintenance

– Deferred maintenance through ALTER INDEX REBUILD|COALESCE

– Automatic cleanup using a scheduled job


Indexing with Oracle Partitioning

• Partial indexes span only some partitions

• Applicable to local and global indexes

• Complementary to full indexing

• Full support of online index maintenance

Global Non-Partitioned Index

Table Partition

Table Partition

Table Partition

Global Partitioned Index

Local Partitioned Index

Partial Global Index

Partial Local Partitioned Index

Partial Global Partitioned Index

Full Indexing

Indexing on

Partial Indexes

Indexing off

No Indexing






Q&A

1

2

3

4


Fast Data Loading with Exchange

• Create table to being exchanged with synopsis prior to table statistics collection

• Table level preference

– INCREMENTAL => TRUE

– INCREMENTAL_LEVEL => TABLE

Optimize incremental statistics management


Online Move for non-interrupted Application Support

• Minimize concurrent DML operations if possible

– Require additional disk space and resources for journaling

– Journal will be applied recursively after initial bulk move

– The larger the journal, the longer the runtime

• Concurrent DML has impact on compression efficiency • Best compression ratio with initial bulk move

Plan for best time window


Index Maintenance for Partition-based Data Management

• Online index maintenance available for both global and local indexes

– Global index maintenance since Oracle 9i, local index maintenance since Oracle 10g

• Decision is always performance versus availability

– Rebuild of index always faster when more than 5%-10% of data are touched

• Consider partial indexing for both old and new data – Not all data has to be indexed to begin with

Choose your optimal strategy


Partition Maintenance Operations

• Issue partition maintenance operations “in batch”

– Every partition DDL creates a new version of the partition metadata

– Leverage new multi-partition operation whenever possible

– Many ADD SUBPARTITION versus single ADD PARTITION

• Use partition extended syntax for bulk loads

– X-lock only on partition that is being locked

– Enables concurrent parallel direct path load from external programs

– Requires clean data

Best Practices - Miscellaneous






Q&A

1

2

3

4

www.ise-informatik.de

Copyright (C) ISE GmbH - All Rights Reserved 24

Enable

eXtreme

Performance. ISE Information Systems Engineering

Established in 1991

IT and Business Consulting for large and medium-sized companies

Main focus:

Oracle Core Technology Database (RAC), Application Server (WebLogic)

Oracle Exadata / Exalogic / Exalytics eXtreme Performance

Oracle Datawarehousing & Business Intelligence

Oracle Exadata Migrations

Performance Analysis & Optimization

Employees: 60

Headquarter in Grafenberg, Branches in Munich and Nuremberg (Germany)



Enable

eXtreme

Performance.

First and only Exastack Technology Center in Germany - Nuremberg

ISE Oracle Technology Center



Enable

eXtreme

Performance.

Partition maintenance operations for data management used for both data

warehousing and OLTP environments

Data warehousing

HCC compression for read-mostly partitions

Move partitions for data storage tiering

OLTP

Purging of old data

Partition Maintenance at work - Examples



Enable

eXtreme

Performance.

Typical data management operations

Establish compression for read-mostly partitions

ALTER TABLE F_EI_POSITIONS MOVE PARTITION DFIPS_201406_R COMPRESS FOR

QUERY HIGH UPDATE INDEXES PARALLEL 4

Tablespace reorganization and storage tiering

ALTER TABLE P_I_POSITIONS MOVE PARTITION DPIPS_201407_R NOCOMPRESS

UPDATE INDEXES PARALLEL 4 TABLESPACE DMDATA

Key design decisions

Parallelism required to meet customers’ SLAs (data maintenance window)

Varying degree based on average partition size

Update indexes mandatory to keep indexes usable

Pure business driver

Online capabilities not mandatory (yet)

Partition Maintenance in Data Warehousing



Enable

eXtreme

Performance.

Business requirement to allow partition compression for every user (schema), but

Not every end user is capable of doing so

Lack of administrative control

Idea

Create a package to manage partition maintenance - but not to execute immediately

Run maintenance on a scheduled base

Scheduled in online mode, runs throughout production hours

Log and monitor the process

Solution

Schema owner decides which tables are compressed

Detailed configuration maintained in configuration table

Table name, compression type, frequency, destination tablespace, etc.

Schema owner gets information per email about the compression state

Self-Service Partition Maintenance



Enable

eXtreme

Performance.

PLSQL package configuration

Implementation started with 10g

11g interval partitioning added

12c plans: interval-reference partitioning, possibly online partition move




Enable

eXtreme

Performance. Self-Service Partition Maintenance

Example configuration entries

Example log entries



Enable

eXtreme

Performance.

Wrapper around fundamental partition maintenance operation

Also checks for index, tablespace etc.

Rebuild index if necessary, check for unusable index


v_sql_str := 'ALTER TABLE '

||a_tab_part_tab(v_tab_part_idx).table_owner

||'.'||a_tab_part_tab(v_tab_part_idx).table_name

||' '||v_admin_str

||' PARTITION '||a_tab_part_tab(v_tab_part_idx).partition_name

||' TABLESPACE '||r_part_admin_master.dest_tablespace

||' '||v_storage||' PCTFREE 0'

||' '||v_update_indexes

||' '||v_compress

||' '||v_tab_parallel;

exec_sql_str(v_sql_str);

/*

|| several tasks added for bitmap indexes

|| auto rebuild indexes

*/



Enable

eXtreme

Performance. Self-Service Partition Maintenance

Impact of newly registered tables for compression (two schemas as example)



Enable

eXtreme

Performance.

Automated script generation for tablespace reorganization and storage

tiering

Script generation only

To be run in offline hours

Business operations

Move data to second diskgroup/storage box

Defragmentation

„Empty“ tablespaces for shrink

Defragmentation and Tablespace Maintenance



Enable

eXtreme

Performance.

Sample processing flow script generation with logging

Defragmentation and Tablespace Maintenance

dbms_output.put_line('rem Move NonPartitioned Tables');

moveNonPartTables;

…

dbms_output.put_line('rem Move Partitioned/SubPartitioned Tables');

movePartTables;

…

dbms_output.put_line('rem Move Lobs');

moveLob;

…

dbms_output.put_line('rem Rebuild NonPartitioned Indexes');

rebuildNonPartInd;

…

dbms_output.put_line('rem Rebuild Partitioned/SubPartitioned Indexes');

rebuildPartInd;



Enable

eXtreme

Performance. Defragmentation and Tablespace Maintenance

Procedure to create SQL scripts for all objects inside a table:

Physically move all partitions

of the table to new tablespace

Modify default tablespace for

future partitions



Enable

eXtreme

Performance.

Oracle has data management challenges, too ;-)

Oracle SOA Suite, Release 11g

SOA stores process data for every process step, so-called „instances“

Many rows in many tables for each instance

After a process is finished, data can (must) be deleted

Deletion must be done with DML

Current data cleanup is inefficient

Delete script, even in parallel, can take many hours

Massive generation of Redo and Undo

Increasing data fragmentation through massive deletes

Data Management in Oracle SOA Suite, Release 11g



Enable

eXtreme

Performance.

Oracle‘s solution: „Eat your own dog food“

Oracle SOA Suite, Release 12c

Out-of-the-box implementation of Partitioning for core tables

Interval-Reference Partitioning

Enforces referential integrity and equipartioning for parent and child tables

Automatic generation of new hierarchical partition tree

Partitioning key contains the state of each flow (open, closed). When the master is checked and

everything is closed, dependent tables with the same partition range can be dropped.

Out-of-the-box data cleanup using drop partition

@soa_exec_interval_verify.sql

-> points out which partition can be dropped

Moving Active, Long Running Instances to a Different Partition

Creating separate partition for long running (new_partition_date)

Attention row movement – with many instances can take while

Data Management with Partitioning in Oracle SOA Suite, Release 12c

http://docs.oracle.com/middleware/1213/soasuite/administer/soa-database-growth-manage.htm#CACIAEAB









Summary

• Partitioning is more than performance

• Data maintenance with Partitioning is powerful and focus for future development

• Hundreds of successful customer use case scenarios

– Just like the ISE use case – thanks Matthias for presenting

• Oracle wants to hear from you

– Interesting use case scenarios and implementations

– Enhancement requests

– [email protected]

mailto:[email protected]


Questions

data warehousing & big data at openworld · 2015-05-09 · dw architect, ise information...

Documents