oracle data archiving - sfougsfoug.org/downloads/dave_moore_oracle_archiving_sfoug.pdfnew...

INTELLIGENCE. INNOVATION. INTEGRITY

Oracle Data ArchivingTaming the Beast

Dave [email protected]


Archiving Defined

Requirements and Solutions

Oracle Archiving Strategies

Oracle Row Removal Options

Oracle Post Archive Operations

10 Minutes on Oracle Performance - Maybe

Agenda


Database Archiving

Database Archiving:The process of removing selected data records from operational databases that are not expected to be referencedagain and storing them in an archive data store where they can be retrieved if needed.

Purge


Archiving All Types of Data

PaperBlueprintsFormsClaims

WordExcelPDFXML

IMSDB2ORACLESYBASESQL ServerIDMS

VSAMProgramsUNIX Files

OutlookLotus NotesAttachments

SoundPicturesVideo


Mandatory Retention Period

Data Archiving and ILM

Create DiscardOperational Reference Archive

Needed for completingbusiness transactions

Needed forreportingor expectedqueries

Needed forcompliance and business protection


Some Sample Regulations Impacting Data Retention


Operational Efficiency

Database Archiving can be undertaken to improve operational efficiency

Large volumes of data can interfere with production operations

– efficiency of transactions– efficiency of utilities: BACKUP/RESTORE, REORG, etc.– Storage

» Gartner: databases copied an average of 6 times!


What Solutions Are Out There?

Keep Data in Operational Database— Problems with authenticity of large amounts of

data over long retention times

Store Data in UNLOAD files (or backups)— Problems with schema change and reading

archived data; using backups poses even more serious problems

Move Data to a Parallel Reference Database— Combines problems of the previous two

Move Data to a Database Archive


Components of aDatabase Archiving Solution

Archive Data Store and Retrieve

Archive DataQuery Access

Archive Administration

Archive Store

Data &Metadata

Production Database

MetadataPoliciesHistory

Recall Database

Captured StructureArchive Policies Data Retention

Data Recall

DataExtract


Archiving Requirements

Policy based archiving: logical selection

Keep data for very long periods of time

Store very large amounts of data in archive

Maintain Archives for ever changing operational systems

Become independent from Applications/DBMS/Systems

Protect authenticity of data

Access data when needed; as needed

Discard data after retention period automatically


Policy based archiving

Why : — Business objects are archived, not files

— Rules for when something is ready can be complex

— Data ready to be archived is distributed over database

Implications:— User must provide policies for when something is to

be archived

How:— Full metadata description of data

— Flexible specification of policy : “WHERE clause”


For Example…

Part NumberTypeDescriptionUnit TypeCostPriceSubstitute Parts

Parts Master isthe parent table to all other tables

PARTS MASTER

Part NumberPO NumberVendor IDQuantity OrderedUnit CostDate OrderedDate Received

ORDER INFO

Part NumberDept. IDCHIT IDQty DisbursedDate Disbursed

DISBURSEMENT

Part NumberBin NumberQty on HandQty on OrderQty Backorder

STORAGE INFO

Part NumberYearQ1 DisbursedQ2 DisbursedQ3 DisbursedQ4 Disbursed

SUMMARY BY QUARTER


Keep Data for a Long Time

Why: retention requirements in decadesImplications:

— Archive will outlive applications/DBMS/systems that generated them

— Archive will outlive people who designed and managed operational systems

— Archive will outlive media we store it on

How:— Unique data store— Application/DBMS/system independence— Metadata independence— Continuous management of storage— Continuous management of archive content


Maintain Archive for Changing Operational Systems

Why : — Metadata changes frequently— Applications are re-engineered periodically

– Change DBMS platform– Change System platform– Replace with new application– Consolidate after merger or acquisition

Implications:— Archive must support multiple variations of an application— Archive must deal with metadata changes

How:— Manage applications as major archive streams having

multiple minor streams with metadata differences— Achieve independence from operating environment


Achieve Metadata Independence

Why : — Operational metadata is inadequate— Operational metadata changes— Operational systems keep only the “current” metadata— Data in archive often does not mirror data in operational

structures

Implications:— Archive must encapsulate metadata— Metadata must be improved

How:— Metadata Capture, Validate, Enhance capabilities— Store structure that encapsulates with data— Keeps multiple versions of metadata


Protect Authenticity of Data

Why : — Potential use in lawsuits/ investigations— Potential use in business analysis

Implications:— Protect from unwanted changes— Show original input— Cannot be managed in operational environment

How:— SQL Access that does not support I/U/D— Do not modify archive data on metadata changes— Encryption as stored— Checksum for detection of sabotage— Limit access to functions— Audit use of functions— Maintain offsite backup copies for restore if sabotaged


Access Data Directly From Archive

Why : — Cannot depend on application environment

Implications:— Full access capability within archive system

How:— Industry standard interface (e.g. JDBC)— LOAD format output for

– For load into a database– May be different from source database

— Requires full and accurate metadata— Ability to review metadata— Ability to function across metadata changes


Discard Function

Why : — Legal exposure for data kept too long

Implications:— Data cannot be kept in archive beyond retention period

— Must be removed with no exposure to forensic software

How:— Policy based discard— System level function

— Tightly controlled and audited

— True “zero out” capability— Discard from backups as well


Database or Archive?

Performance

Space

Compliance

Keep in DB Keep in Archive


Based on Data Availability

Must be Available to App

Must be Available

Must Be Secure

Keep in DB Keep in Archive

Not Needed

Purge


Dave

Oracle ACE

Using Oracle since 1991

Product Author at Neon Enterprise Software

Creator of OracleUtilities.com

Author of “Oracle Utilities” from Rampant Tech Press

Core competencies include performance, utilities and data management



Designed Up Front (Yeah, right)

Determined by Application Owner

Implemented by ____________

Utilize Oracle Features


Rolling Windows

Self Managing

Mostly based on DATE

Utilize DBMS Features — Partitioning

— Transportable Tablespaces– Exchange Partition – Set tablespace read only – Expdp– Copy export file and data file


Rolling Windows via Partitioning

ProbablyNever

Accessed

Rarely Accessed

Heavily Accessed

Cheap as you can get

Not so fastor expensive

Fast, expensive

Data Profile

Storage Profile

Read / WriteRead OnlyRead Only / Compressed

P1 … … P47


Why not use transportable tablespacesor Oracle exports for data retention?


The Problem with Oracle Files

Transportable Tablespaces

Exports

Backups

Export Files &Datafiles Version 16Z

Oracle

Year 2030

ImportTrans Tsp

Year 2007

Not a good method for LT Data Retention


New Partitioning Features in 11G

Interval Partitioning — a true rolling window

REF Partitioning — parent child relationships

— denormalization not necessary

Virtual Column Partitioning

Partition Advisor


Rows Gotta Go


Row Removal Options

SQL DELETE

CTAS / DROP / RENAME

TRUNCATE

Row Marking


Row by Row = SLOW by SLOW

Avoid LOOPS … Use DELETE

Or…

TRUNCATE

Or… CTAS/RENAME

Things to Remember


SQL DELETE

Good for small number of rows

RI handled automatically

Oracle was born to DELETE, better than any PL/SQL that you write.

Issue with Un-indexed Foreign Keys


DELETE Optimization

Work in batches, committing (only when programmatically DELETING)

Use parallel DML (Partitioned tables only)

Drop Indexes before (if possible)

Index FK columns


CTAS

Works well for PURGE, not archivePerfect when you want to keep low percentage of rows in the tableDoesn’t handle RI – no DELETE was issued.Process1. Create table with rows you want to keep2. Drop old table

3. Rename table

4. Recreate indexes

create table new_table unrecoverable as select * from old_table where ...


TRUNCATE

Congratulations if your application lends itself to TRUNCATE without losing new data

What about RI?

May truncate or drop individual partitions


DROP

DROP PARTITION

What would you do before you drop it?

Exchange partition with table

Transportable tablespace.


Design Summary

Create an architecture that lends itself to aging, archiving, deleting

This architecture should compensate for business requirements — For instance, customer orders not accessible after

6 months … or

— top query performance needed for all ‘ACTIVE’accounts … etc

Implement it – THE EASY PART


Post Archive Challenges



“I have successfully deleted 10 billion rows from the table. HoooAhhhh! Performance will be great, space will be available, and I will get credit for optimizing our data warehouse application, saving the company billions of dollars”


… 2 Days Later …



Hmmmmm. It looks like …

- Queries are not any faster . . .- The Select count(*) took the same amount of time . . .- Space was not freed in Oracle (DBA_FREE_SPACE) . . .- Space was not freed in the operating system . . .

WHY NOT ????? Where are the benefits ???


From Swiss to Provolone

After DELETE After Maintenance



Statistics are not fresh

High Water Marks are very high

Space has not been freed within Oracle (if that’s what you want)

Space has not been freed to the OS


Refresh Statistics

Help the optimizer, easy enough

Call dbms_stats, ESTIMATE, 40%

dbms_stats provides many options


Reset High Water Mark (HWM)

DROP or TRUNCATE

Multiple OTHER ways to do this depending on version

In v9 … alter table move tablespace [tsp name]; — Row movement must be enabled

— Tablespace must be a LMT

— Can move into same tablespace

— Will occupy 2X space temporarily

— Must then rebuild indexes

In v10 … alter table shrink space;


Create table, check space

SQL> create table space_example as select * from dba_source;

Table created.

SQL> select count(*) from space_example;

COUNT(*)----------

296463

SQL> exec dbms_space.unused_space(‘DAVE', 'SPACE_EXAMPLE');

Total blocks: 6328Unused blocks: 1Unused bytes: 8192Last Used Block: 55Last Used Block ID: 10377Last Used Ext File ID: 4


Check datafile space

Size Current Poss.FILE_NAME Poss. Size Savings-------------------------------------------------- -------- -------- --------/export/home/ora102/oradata/ora102/qasb001.dbf 29 46 17/export/home/ora102/oradata/ora102/example01.dbf 69 100 31/export/home/ora102/oradata/ora102/qasb002.dbf 41 41 0/export/home/ora102/oradata/ora102/system01.dbf 493 500 7/export/home/ora102/oradata/ora102/sysaux01.dbf 430 430 0/export/home/ora102/oradata/ora102/undotbs01.dbf 91 175 84/export/home/ora102/oradata/ora102/users01.dbf 44 83 39/export/home/ora102/oradata/ora102/test.dbf 51 70 19


Delete rows, check space

SQL> delete from space_example;

296463 rows deleted.

SQL> commit;

SQL> exec dbms_space.unused_space(‘DAVE', 'SPACE_EXAMPLE');


Nothing Changed !


Shrink it, check space

SQL> alter table space_example enable row movement;

SQL> alter table space_example shrink space;

SQL> exec dbms_space.unused_space('BDB', 'SPACE_EXAMPLE');


Space Freed From Table, but still in Oracle


Check space again

Size Current Poss.FILE_NAME Poss. Size Savings-------------------------------------------------- -------- -------- --------/export/home/ora102/oradata/ora102/qasb001.dbf 29 46 17/export/home/ora102/oradata/ora102/example01.dbf 69 100 31/export/home/ora102/oradata/ora102/qasb002.dbf 41 41 0/export/home/ora102/oradata/ora102/system01.dbf 493 500 7/export/home/ora102/oradata/ora102/sysaux01.dbf 430 430 0/export/home/ora102/oradata/ora102/undotbs01.dbf 171 175 4/export/home/ora102/oradata/ora102/users01.dbf 44 83 39/export/home/ora102/oradata/ora102/test.dbf 1 70 69

This datafile should be resized to save 69 MB

SQL> alter database datafile '/export/home/ora102/oradata/ora102/test.dbf' resize 1m;


Free the Space

Space is still reserved for future inserts and updates, just not freed back to the OS Space will not be automatically freed –

confirm by checking DBA_FREE_SPACE

Ways to set it free— drop — truncate— alter table move …— alter table shrink space …


Unindexed Foreign Keys Example

COL1

1 Million Rows PARENT

COL1COL1_PARENT

CHILD

1 Million Rows

SQL> DELETE FROM PARENT WHERE COL1 < 1000;

ON DELETE CASCADE


Before Index

delete from parentwherecol1 < 1000

call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 1 0.01 0.08 2 27 0 0Execute 1 0.90 0.80 4 2208799 6062 999Fetch 0 0.00 0.00 0 0 0 0------- ------ -------- ---------- ---------- ---------- ---------- ----------total 2 0.91 0.88 6 2208826 6062 999

delete from "DAVE"."CHILD"where"COL1_PARENT" = :1



After Index

delete from parentwherecol1 < 1000


delete from "DAVE"."CHILD"where"COL1_PARENT" = :1


SQL> create index prnt_ndx on child(col1_parent);


Unindexed Foreign Keys

Problem is not limited to DELETE statements

Search database for unindexed FK columns

Script is on asktom— Search for unindex.sql


Summary Points

Create sound Archiving strategy based on Oracle technical features as well as business and/or legal requirements Leverage partitioning Move partitions to cheap disk when

appropriate Make partitions ‘read only’ and compressedRemove data via DROP or TRUNCATE if

possible If SQL DELETE, make sure to perform maintenance operations


10 minute Crash Course in Performance Tuning


How do we tune an un-tuned database?


Statspack

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ---------CPU time 122 91.65db file sequential read 1,571 2 1.61db file scattered read 1,174 2 1.59log file sequential read 342 2 1.39control file parallel write 450 2 1.39

-------------------------------------------------------------Wait Events DB/Inst: AKI1/AKI1 Snaps: 5-6

-> s - second-> cs - centisecond - 100th of a second-> ms - millisecond - 1000th of a second-> us - microsecond - 1000000th of a second-> ordered by wait time desc, waits desc (idle events last)

Or AWR in 10G …


OEM


SQL Trace / tkprofcall count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 1 0.00 0.00 0 0 0 0 Execute 500 0.09 0.08 0 1 1007 500 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- ---------- ---------- ---------- ----------total 501 0.09 0.08 0 1 1007 500

Misses in library cache during parse: 0 Optimizer goal: CHOOSE

Parsing user id: 59 (recursive depth: 1) ********************************************************************************

COMMIT

call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 7 0.00 0.00 0 0 0 0 Execute 1553 0.08 0.07 0 0 1550 0 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- ---------- ---------- ---------- ----------total 1560 0.08 0.07 0 0 1550 0


Custom Scripts


Can we agree that tuning a previously tuned database requires different methods than one that has never been tuned?


Cause & Effect

When is the cause useful to know?


When knowledge of the cause may contribute to the cure.


Examples …


Cause and Effect

+ =


Home Entertainment


TV shows black screen …


Traditional

Does TV have power?

Does Receiver have power?

Does DVD player have power?

Does VCR have power?

Does RF speaker work?

Does LF speaker work?

Does RR speaker work?

Does LR speaker work?


Environmental

What Changed?

Cause: Somebody hit the input button on the remote control and changed the media source to VCR.

Remedy: Hit the input button again and change it back to DVD.


Process

Determine what changed.

Determine if changing it back will fix it.


Oracle is Headed in Right Direction

AWR Compare Periods

Configuration Comparison


Oracle Environmental Changes Worth Monitoring

Transaction Loads (# users, # concurrent trans)

Initialization Parameters (v$parameter)

New indexes

Missing indexes

Unused indexes

Execution Plans

New SQL, PL/SQL code

Statistics (new, old, none)

Hardware / Network

More …


Tune Database Take BaselineCompare Snapshots to Baseline

1 2 3

•Wait Events•Hit Ratios•I/O•SQL Tuning

•Perf Metrics •Deltas

Traditional Approach


Tune Database Take BaselineCompare Snapshots to Baseline

1 2 3

•Wait Events•Hit Ratios•I/O•SQL Tuning•Environment

•Perf Metrics•Environment

•Deltas

Recommended Approach


0

50

100

150

200

250

Mon Tue Wed Thu Fri

NIC Card ChangedClick here for more

Avg. Transaction Time (ms)


Become an Oracle Environmentalist!


Dave MooreNEON Enterprise Software

[email protected]

www.neonesoft.com

Oracle Data Archiving�Taming the BeastDatabase Archiving Archiving All Types of DataData Archiving and ILMOperational EfficiencyWhat Solutions Are Out There?Components of a�Database Archiving SolutionArchiving RequirementsPolicy based archiving For Example…Keep Data for a Long TimeMaintain Archive for �Changing Operational SystemsAchieve Metadata IndependenceProtect Authenticity of DataAccess Data Directly From ArchiveDiscard FunctionDatabase or Archive?Based on Data AvailabilityDaveOracle Archiving StrategiesRolling WindowsRolling Windows via PartitioningThe Problem with Oracle FilesNew Partitioning Features in 11GRow Removal OptionsThings to RememberSQL DELETEDELETE Optimization CTASTRUNCATEDROPDesign SummaryFrom Swiss to ProvolonePost Archive ChallengesRefresh StatisticsReset High Water Mark (HWM)Create table, check spaceCheck datafile spaceDelete rows, check spaceShrink it, check spaceCheck space againFree the SpaceUnindexed Foreign Keys ExampleBefore IndexAfter IndexUnindexed Foreign KeysSummary PointsHow do we tune an un-tuned database? Statspack OEM SQL Trace / tkprof Custom Scripts Can we agree that tuning a previously tuned database requires different methods than one that has never been tuned? Cause & Effect��When is the cause useful to know? When knowledge of the cause may contribute to the cure. ��Examples …Cause and EffectHome EntertainmentTV shows black screen …Traditional Environmental ProcessOracle is Headed in Right DirectionOracle Environmental Changes �Worth MonitoringBecome an Oracle Environmentalist! Dave Moore

oracle data archiving - sfougsfoug.org/downloads/dave_moore_oracle_archiving_sfoug.pdfnew...

Documents