exadata performance

5/23/2018 Exadata performance

1/35


2/35

Exadata Performance Debugging

Biswaroop Biswal / Ranga Sarvabhouman


3/35

Agenda

What basic information one must know about Exadata I/O related performance:

- Check if cells are IO bound

- Flash Cache

- Smart Scan

Check if cells are IO bound

Check if db nodes are CPU or memory bound

Check if smart scan works as expected

Check if HCC/Partitioning/DBFS are used

If none is true, go back to database performance tuning and planning


4/35

Check if cells are IO bound?

Differentiate between slowness vs. full utilization

Use OSW iostat and/or CellDisk metrics to compute total HDD and FLASHthroughput (MBPS) and IOPS

Refer to Exadata DBM data sheet for peak numbers:

Watch out for high latency if IOs ever approach peak numbers

High latency does NOTmean slow disks each IO takes long primarily due to time waiting in disk queue

IO latency can be >100ms (note disks are not slow!)

IO latency depends on disk queue length so can be varied based ondifferent workloads

Be aware that max MBPS and max IOPS can not be reachedsimultaneously

How to evaluate mixed workload?

Examine disk utilization - is it close to 100%?

Run calibrate if needed (requires cells being quiesced)
http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/175280.pdfhttp://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/175280.pdf


5/35

What happens when cells are IO bound

Consider SAME (Stripe And Mirror Everywhere)

when any disk or cell is maxed out, performance will be

throttled by that disk/cell even with workload parallelization

Be aware of potential performance disparity between system and

data disks

System disks not only have user data but also have cells own file

systems

System disks may run slower than data disks

More pronounced on High Capacity 2TB drives due to lowerIOPS capacity when compared with High Performance

600GB drives


6/35

Exadata Flash Cache - overview

Know your Flash: Flash storage entities and relationships:


7/35

When/How and what to measure in Flash cache:

When to measure:

- Missing SLA- Poor performance across the environment.

- AWR reports

- Users screaming

How to measure:- V$SYSSTAT/V$SQL/V$SESSION_WAIT/V$SESSION_EVENT and V$SySTEM_EVENT

- Exadata Storage Servers using views

- Exadata Storage Servers using metrics (cellcli and dcli commands)

- Exadata Smart flash log with metrics

What to measure:- Effective use of flash

- Percentage of I/O requests satisfied by flash

- Number of objects kept on flash

- Size of objects kept on flash


8/35

Smart Flash logwhat to look for?

Smart Flash Logging affects log file parallel write time, notlog file synctime.

Check the AWR report for high log file parallel write times; there should bevery few log file parallel write waits > 32 ms, and no waits > 0.5 seconds.This can be verified in the following sections of an AWR report:

Wait Event Histogram

Wait Event Histogram Detail (64 msec to 2 sec)

If a cell has a non-zero value for FL_IO_W_SKIP_BUSY, then this means that

the hard disks which contain the database log files (or their mirroredcopies) are not performing well. This can due to throughput or heavy load onthe database. It can be resolved by expanding your system to more cells.

If a cell has a non-zero value for FL_ACTUAL_OUTLIERS andFL_EFFICIENCY_PERCENTAGE is not 100%, then this means that flash disksare not performing well. This can be due to load or performance issue. If it isload issue then try to reduce the load or replace the flash disk.

Besides hard disk and flash disk performance, there are other factors whichcan affect redo log write latencies:

Examine database nodes to make sure that LGWR is not experiencingscheduling hiccups due to factors such as swapping.

Check whether the network is impacting the performance.


9/35

Smart Scanworks as expected & how to debug

Symptoms In AWR report you would see these 2 wait events

Cell table smart scan

Cell index smart scan

V$viewsv$sysstat and v$cell_state, statistics that you need to look for (also found systemstatitiscs section of AWR report):

Cell physical IO bytes eligible for predicate offload

Cell physical IO bytes saved by storage index

Cell physical IO bytes send directly to DB node to balance CPU usage

Cell physical IO interconnect bytes returned by smart scan

%cell num smart%

PREDIO (v$cell_state)

Oswatcher logs

Reasons Smart scan has less filtering/no filtering

Suboptimal storage index usage

Less/No filtering due to quarantine/ time zone upgrade

Less filtering due to CPU pass through

Other reasons Cell is CPU bound or IO bound

Suboptimal flash usage


10/35

How to identify long running transaction that

causes sub optimal Storage Index usage?

Purpose: Long running transaction prevents min active scn from progressing,

there by causing scans to not use storage index.

Steps Get global min active scn by setting system event to 55703, level 1; the

min active scn is printed in alert log once every 3 minutes. Unset theevent after you get the min active scn. Convert the min scn from hexto decimals.

Use scn_to_timestamp to compare scn from alert log and current_scn,if there is differ by good amount then continue.

Query the X$KTUXE to get the slot number, undo segment number

where the column KTUXESTA is not like INACTIVE. Query v$process to obtain the instance ID, process ID.

Now either you can kill the process or use pstack to obtain moreinformation.


11/35

Less/No Filtering due to quarantines.

Presence of SQL or DB quarantines result in smart scan not being used.

Look at following v$cell_state statistics to see if filtering is not happening

due to quarantines

Smart IO not used to SQL Step or DB Quarantine.

Smart IO not used to Disk Region Quarantine. Use dcli or cellcli on cells to check for quarantines, for eg: , list quarantine

where QuarantineType=Database in cellcli prompt.

Quarantines are removed automatically when cell software is upgraded or

use cellcli drop quarantine to remove manually.


12/35

Less/No filtering due to timezone upgrade; CPU

passthrough

Look at v$cell_state statistics to see if smart scan is nothappening due to timezone upgrade:

Smart IO uses passthru as timezone file is unavailable. OR

Select value from v$sysstat where name = cell num smart

IO sessions using passthru mode due to timezone;Smart scans will take place after timezone file is made available.

Reasons for CPU passthrough (storage is CPU bound) More scan queries is running on the storage server resulting in high

CPU usage.

Suboptimal Storage Index usage results in more physical IO beingperformed. Which results in more filtering on storage server whichincreases CPU usage.

Scans on encrypted tables increases storage CPU usage.


13/35

I/O Resource Management Plans


14/35

I/O Resource Management Plans : Example


15/35



16/35



17/35

Setting the IORM Objective

Available IORM objective settings:

basicIORM does not enforce user-defined plans.

IORM protects against extreme latencies for small I/O requests.

Maximum throughput is maintained.

low_latencyMinimizes latency by limiting the number of concurrent I/O requests

Useful for critical OLTP workloadsPerformance of high-throughput workloads may suffer

high_throughputMaximizes throughput by not limiting concurrent I/O requests

Useful for DSS and data warehouse workloads

Performance of latency-critical workloads may suffer

balancedBalances low disk latency and high throughput

Useful for mixed workloads

autoIORM decides the best objective setting based on active plans and workloads


18/35

Intradatabase Plan : Example


19/35

Interdatabase Plan : Example


20/35

Using Share-Based Allocation in the

Interdatabase Plan

Commencing with Exadata Storage Server software release 11.2.3.1.0, I/O allocations in the

Interdatabase plan can be expressed as shares rather than using the level and allocation attributesshown on the previous page. Each share is a value between 1 and 32, with 1 being the lowest share, and

32 being the highest share. The share value represents the relative importance of each database rather

than specifying an IO allocation percentage.

Share-based allocation is a simplified approach designed to support large numbers of databases. Using

shared-based allocations, an interdatabase plan can support up to 1024 directives.


21/35

Setting Database I/O Utilization Limits

Database Roles


22/35

Category Plan: Example


23/35

IORM and Exadata Storage Server Flash Memory

IORM manages only I/O queues for physical disks.

IORM does not arbitrate requests serviced by Exadata Smart Flash

Cache.

IORM can control whether a database can use Exadata Smart Flash

Cache.

IORM can control whether a database can use Exadata Smart Flash Log.

Exadata Storage Server software release 11.2.2.3, IORM can be used to specify if a

database is allowed to use Exadata Smart Flash Cache. This allows flash cache to be

reserved for the most important databases, which is especially useful in environments

that are used to consolidate multiple databases


24/35

Complete Example


25/35

Complete Example

Exadata Cells


26/35

Exadata Metrics and alerts


27/35

Monitoring Exadata Storage Server with

Metrics


28/35

Monitoring Exadata Cell Metrics

Metric abbreviation :

CL_ (cell) CD_ (cell disk)

GD_ (grid disk)

FC_ (flash cache)

DB_ (database)

CG_ (consumer group)

CT_ (category)

N_ (interconnect network)

_R for read or _W for write.

_SM or _LG to identify small or large I/Os

At the end of the name, there could be _SEC to signify per second or _RQ to signify

per request

CD_IO_RQ_R_SMis the number of requests to read small blocks on a cell disk.

GD_IO_BY_W_LG_SECis the number of MB of large block I/O per second on a

grid disk.

I/O-related metric name :

IO_RQ (number of requests) IO_BY (number of MB)

IO_TM (I/O latency)

IO_WT (I/O wait time)


29/35

Monitoring Exadata Storage Server with Alerts

I l ti F lt ith E d t St S


30/35

Isolating Faults with Exadata Storage Server

Quarantine

In addition to metrics and alerts, when prescribed faults are detected in ExadataStorage Server, a quarantine object is automatically created. By this, the action that

caused the fault can be quarantined, so that the fault can be avoided in the future.

Quarantine reduces the chance of storage server software crashes, and improves

storage availability


31/35

Exadata Storage Server Quarantines

Types of automatic quarantine :

SQL PLAN: Created when the cell crashes while performing Smart Scan for a SQL statement. The SQL Plan

for the SQL statement is quarantined, and Smart Scan is disabled for the SQL plan.

DISK REGION: Created when the cell crashes while performing Smart Scan of a disk region. The 1 MB disk

region being scanned is quarantined and Smart Scan is disabled for the disk region.

Database: Created when the cell detects that a particular database causes instability. Instability detection

is based on the number of SQL Plan Quarantines for a database. Smart Scan is disabled for the database.

Cell Offload: Created when the cell detects that some offload feature has caused instability. Instability

detection is based on the number of database quarantines for a cell. Smart Scan is disabled for all

databases

CellCLI commands to manually manipulate quarantines:

LIST QUARANTINE: To show the quarantines currently on the cell

ALTER QUARANTINE: To set the comment attribute. The comment attribute is the only quarantine attribute

that can be modified. DROP QUARANTINE: To manually remove a quarantine.

CREATE QUARANTINE: To manually create a quarantine object. Manual quarantines are created to proactively

isolate SQL statements that are known to cause problems. Example :

CELLCLI> CREATE QUARANTINE quarantineType="SQLID", sqlid="5xnjp4cutc1s8"


32/35

Choosing the Flash Cache Mode


33/35

Choosing the Flash Cache Mode


34/35

Setting the Flash Cache Mode

Enabling write-back mode

Enabling write-through mode


35/35

Exadata specific system statistics

Gather Exadata specific system statistics:

Enables the optimizer to more accurately cost operations using

actual performance information:

CPU speed IO Performance

Sets multi block read count (MBRC) correctly for Exadata

Requires at least Oracle Database version 11.2.0.2 BP 18 or 11.2.0.3 BP 8

Recommended for all new databases

Test thoroughly before changing existing databases.

Databases with stable good plans do not require a change.

exadata performance

Documents

exadata io related performance

io bound check

event exadata storage

flash size

flash number

database performance

smart flash logging

smart flash logwhat