perf tuning with-multitenant

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Running your production with MultitenantPerformance & Tuning challenges

Jacques KosticPrincipal Consultant IMS Lausanne

TechEvent 2016-sept-092 09/10/2016

Experience:

• Oracle DBA since 1990• High Availability and Backup & Recovery Architect• SQL and Instance Performance & Tuning• License Audit and Consolidation

Certifications:

• Oracle Certified Master 11g & 12c• Oracle 11g Performance Tuning Certified Expert• Oracle RAC 11g and Grid Infrastructure Administration• Oracle Certified SQL Expert 11g• ITIL Foundation

Teaching Courses at Trivadis:

• Oracle 11g & 12c Grid Infrastructure & RAC• Oracle 11g & 12c Data Guard• Oracle 11g & 12c Performance & Tuning• Oracle 11g & 12c Administration• SQL – PL-SQL

About me

Agenda

TechEvent 2016-sept-093 09/10/2016

1. Customer context

2. Backgrounds

3. New requirements

4. Performance challenges

5. Demo

6. Q/A

TechEvent 2016-sept-094 09/10/2016

Customer Context

Constomer Context

TechEvent 2016-sept-095 09/10/2016

The name will not be disclosed but the most relevantcharacteristics to the project are reported below.

Customer

Environment

Medium size customer from financial sector

Few databases with different workload types

Virtual Private Databases

High level of automation

Consolidation opportunities with the Multitenant Option

TechEvent 2016-sept-096 09/10/2016

Backgrounds

Backgrounds

TechEvent 2016-sept-097 09/10/2016

2013 : Initial setup

– Oracle 11.2.0.3

dNFS

6 Cores per nodes

2 Production RAC databases

DataGuard

4 Cores per nodes

2 Standby RAC databases ADG

Up to 5 test databases

Backgrounds

TechEvent 2016-sept-098 09/10/2016

2015 : First Upgrade

– Oracle 12.1.0.2 with Singletenant

dNFS

6 Cores per nodes


DataGuard

4 Cores per nodes



Backgrounds

TechEvent 2016-sept-099 09/10/2016

Performance challenges after migration to 12c

Execution plan changes

• Lack of data model design

• Stale statistics

Some queries with huge performance deviations

– New 12c features

• SQL Plan directives due to miss estimates

• Adaptive dynamic sampling triggered automatically when parallelism isin the game

• Some bugs…

How we fixed that?

Backgrounds

TechEvent 2016-sept-0910 09/10/2016

SQL > Alter system set optimizer_adaptive_features=FALSE;

System altered.

SQL>

Et voila!

TechEvent 2016-sept-0911 09/10/2016

TechEvent 2016-sept-0912 09/10/2016

New Requirements

New Requirements

TechEvent 2016-sept-0913 09/10/2016

New customer to absorb

Double the size of all databases

Have the same performance in test environments

Same automation procedures

New Requirements

TechEvent 2016-sept-0914 09/10/2016

2016 : Second Upgrade

– Oracle 12.1.0.2 with Multitenant

6 Cores per nodes


DataGuard

6 Cores per nodes



TechEvent 2016-sept-0915 09/10/2016

Performance Challenges


TechEvent 2016-sept-0916 09/10/2016

Production databases

One small OLTP database

One mix OLTP-DWH medium database

–VPD!

– Financial consolidation can be scheduled at any time by endusers

–Average execution time is within one hour

–Parallel executions are used for certain steps


TechEvent 2016-sept-0917 09/10/2016

Test databases

Can be refreshed on demand

–Directly from production

– From any backup

Performance tests before pushing changes into production

Has to be closest to production in term of resource allocation


TechEvent 2016-sept-0918 09/10/2016

What have we done?

Use more SGA as the ODA X5-2 (256 GB per nodes versus 96 GB)

Remove instance caging and introduce a CDB Resource Plan

Limit the PGA by setting pga_aggregate_limit parameter


TechEvent 2016-sept-0919 09/10/2016

Adjust statistics collection methods

Relocate some tablespaces

Optimize Undo for temporary tables by settingTEMP_UNDO_ENABLED parameter

Setup Result Cache

Fine tune some queries


TechEvent 2016-sept-0920 09/10/2016

Result after few days

Customer was happy, jobs are running 55% faster

Refreshes from production are completed 40% faster

Performance in test environment is becoming comparable toproduction

General end user perception was good!


TechEvent 2016-sept-0921 09/10/2016

After one week, a big mess!

Production Instance was very slow

Several jobs where running into the two PDBs

Massive database waits: free buffer waits!!


TechEvent 2016-sept-0922 09/10/2016

Possible causes:

The I/O system is slow.

Waiting for resources, such latches.

The buffer cache is so small and DBWR spends most of it's time cleaning out buffers for server processes.

The buffer cache is so big and they are not enough DBWR processes to free enough buffers in the cache to satisfy requests.


TechEvent 2016-sept-0923 09/10/2016

Limitations with Multitenant

Shared pool

Buffer cache

I/O activity

TechEvent 2016-sept-0924 09/10/2016

Demo

Jacques Kostic

Principal Consultant IMS Lausanne

Tel. +41 79 909 72 63

[email protected]

09/10/2016 TechEvent 2016-sept-0925

Session Feedback – NOW!

TechEvent September 201626 09.09.2016

Please use the Trivadis Events Mobile App to give session feedback

Use "My schedule" if you registered for this session

Otherwise use "Agenda" and the search function

If the mobile App does not work (or if you have a Windows Phone) use your Mobile

Browser

– URL: http://trivadis.quickmobileplatform.eu/

– Username: <your_loginname> (like svv)

– Password: sent by mail...

perf tuning with-multitenant

Technology