oracle open world exadata monitoring and management with em12c

64

Upload: kellyn-potvin

Post on 22-Apr-2015

362 views

Category:

Technology


1 download

DESCRIPTION

Oracle Open World 2014 session with Mike Chafin!

TRANSCRIPT

Page 1: Oracle Open World Exadata Monitoring and Management with EM12c
Page 2: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential

EXADATA HEALTH AND RESOURCE USAGE MONITORINGIncluding the Benefits of Administration Tuning Methods

Kellyn Pot’VinConsulting Member of Enterprise Manager Technical TeamStrategic Customer ProgramMike ChafinPrincipal Member of Technical Staff

Oracle Open World 2014

Page 3: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Oracle Confidential 3

Page 4: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 4

Program Agenda

1

2

3

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Page 5: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 5

Program Agenda with Highlight

1

2

3

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Page 6: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 6

Evolution of Methodologies

Documented in the Oracle Documentation Set:

Database 2 Day + Performance Tuning Guide

Oracle Database Performance Tuning Guide

Exadata Performance Tuning White Paper

Each are similar, in that they offer a structural approach to problem solving using the latest tools and technologies.

Page 7: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 7

Tuning Level

What level is the Issue?

• Hardware

• Database

• Schema/Design

• SQL

Page 8: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 8

Important Tuning Concepts

Tuning a database by “DBTime”

DBTime is the term used by Oracle to calculate the time used by active CPU, IO and other foreground processes from the total time elapsed to calculate wait events that are performed within the database.

“Tune for Time or You’re Wasting Time…”

Page 9: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 9

Program Agenda with Highlight

1

2

3

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Page 10: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 10

Tuning Exercise Steps

• Prepare

• Compare

• Review (research, investigate)

• Rule out, Test

• Implement/Resolve

Page 11: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 11

Prepare

Collect hardware and software healthchecks • exachk, cluvfy, emdiag,• Healthchecks• AWR Baseline • Verify and/or set AWR retention period

Copy configuration files• Automate collection and base-lining (EM jobs)

Create AWR, ASH and ADDM reports for research.

Page 12: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 12

Stopping the Insanity Flowchart

Page 13: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 13

The Value of Quick Check Lists

Old School, but VALUABLE.

Page 14: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 14

Program Agenda with Highlight

1

3

2

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Page 15: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 15

Comparison Tools

• Incredible Power in Knowing What’s Changed.

Page 16: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 16

CompareIf there is a problem what has changed? And who might know?

Considerations• Patch levels (everywhere!)• Schema• Tunable OS parameters • Resource Management Plans• Code Changes• ADDM Comparison Report

Page 17: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 17

Compare Configurations- Exadata Level

Page 18: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 18

Compare Configuration- Database Level

• EM Job to compare one ‘reference’ database against one or more other databases

• Job can be scheduled on a repetitive basis, or run ad-hoc

Page 19: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 19

Compare Configurations- Schema Level

Capture Schema baselines

Compare schema’s

• With a baseline

• Between different databases

Synchronize schema’s

Page 20: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Compare ADDM

• Full ADDM analysis across two AWR snapshot periods• Detects causes, measure effects, then correlates them

Causes : Workload changes, Configuration changes Effects : Regressed SQL, Reach resource limits (CPU, I/O, memory, interconnect)

• Makes actionable recommendations along with quantified impact

AWR Snapshot Period 1

AWR Snapshot Period 2

Analysis ReportCompare Period ADDM

SQL Commonality

Regressed SQL

I/O Bound

Undersized SGA

Page 21: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 21

IO Issues

What type of IO Issues are addressed with Exadata features?How can temp tablespace usage be an impact to performance in an Exadata?

• Identify IO Issues- ASH Analytics, ADDM Comparison Report

• Isolate IO Issues- Search SQL, ASH/ADDM Reports,

• Address IO Issues- IO Resource Manager, SQL Tuning Advisor,

Page 22: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 22

Exadata and SQL Monitor

Offload Percent Efficiency Displayed in Report and UI.• Automatically monitors long running SQL• Enabled out-of-the-box with no performance overhead• Monitors each SQL and PL/SQL execution • Shows global PL/SQL and SQL level statistics

• Drill-down to slow SQL for diagnosing unexpected PL/SQL behavior

• Guides tuning efforts

Page 23: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

IO Bound Dashboard in ASH Analytics• Base vs. comparison period• Temp reads/writes specified• Single block read latency

Page 24: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 24

Setting Thresholds

Thresholds with incident rule sets to notify when growth in resource occurs and impacts the environment.

Page 25: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 25

IO Resource Manager

• IO Resource Manager, (IORM) offers ability to maintain consistent performance levels across an engineered system.

• Allocates and limits IO per settings configured

• Simple configuration access via Enterprise Manager

Page 26: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 26

Enterprise Manager Exadata Resource Management

• Exadata plugin specifica access to manage IO Resources for db and across cell nodes.

What IO resources on Exadata are shared vs. on a non-Exadata RAC environment?

Page 27: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 27

IO Resource Management

Page 28: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 28

Monitoring IORM

Page 29: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 29

Monitoring Disk Utilization

Page 30: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 30

Max Cell Disk Limits

Page 31: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 31

Why Setting Limits are Important

• No Hard Limits equate to IO being freely redistributed

• Setting limits ensures that resources are available for critical databases

• Less high priority databases aren’t able to allocate more than necessary

Page 32: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 32

Review

• Errors in the system?

• EM Incident Manager

• Alert Log(s)

• Grid Infrastructure

• ASM

• Database

• OS logs / system logs

Page 33: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 33

AND Rule Out- Hardware

• Compute Node CPU

• I/O Bound / at known limits

• Exadata – Flash cache

• Mixed Workloads creating contention?

• ALL Components up!

Page 34: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 34

Exachk- See Oracle Support Document 1070954.1: “Oracle Exadata Database Machine Exachk or HealthCheck”

Page 35: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 35

Exachk and Healthcheck Utility Checks…

• Incorrect configurations

• Parameter Issues

• Recommendations, steps to correct, along with current value.

See Oracle Support Document 1070954.1: “Oracle Exadata Database Machine Exachk or HealthCheck” and Exadata best practices in Oracle Support Document 757552.1: “Oracle Exadata Best Practices

Page 36: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 36

Automate Exachk via Enterprise ManagerAutomating will populate healthchecks to UI!

Page 37: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 37

Storage Grid OverviewStorage Cell monitoring and administration support

–Cell Home page and performance pages

–Actions supported: Start/stop Cell, verify connectivity, setup SSH

Automatic discovery of Cells

Management by Cell Group

–All cells used by a database automatically placed in a group

–Cell Group level administration operations (batch job monitoring)

Page 38: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 38

More Storage Grid Overview

Page 39: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 39

Hardware- DB Node View

Page 40: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 40

Hardware- Cell Server View

Page 41: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 41

Program Agenda with Highlight

1

4

2

3

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Page 42: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 42

AWR Baselines

• Collection of snapshots used for performance comparisons.

• Baselines are retained within the AWR even after the retention time for the data has been reached.

• Exadata should have a moving and a static baseline in place to capture different workloads.

Page 43: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 43

Details of Baseline Creation in EM

Highlight a snapshot range to use for your static baseline or use a time range to base it from.

Page 44: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 44

AWR Baselines

Page 45: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 45

Completing Baseline Creation in EM

Schedule the job to run in a non-impacting schedule and choose not to purge.

Page 46: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 46

SQL Issues

SQL Tuning – what kind of problem?• Reactive• Transient• Long Running SQL• Proactive• Persistent• Comparative

Page 47: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 47

Proactive SQL Tuning Choices

• SQL Performance Analyzer• SQL Access advisor• ASH and ADDM to prioritize further work• Group operations (DB12)• Create an SLA / baseline• Use SQL Plan Baselines (SPM)• Validate with SQL Performance Analyzer

Page 48: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 48

Reactive SQL Tuning Choices

After all changes, including any environments are ruled out…

• Start with AWR, ADDM and ASH

• Identify outliers / worst performing (Pareto)

• Review top wait events

• SQL Tuning Advisor

• If resource constrained consider resource limiting strategies

Page 49: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 49

Database Performance Page in EM12c

Page 50: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 50

ASH Analytics

Page 51: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 51

ASH Analytics

Requirements for 10g and 11g databases:• Additional PL/SQL packages to be installed into the DBSNMP schema• CONNECT, CREATE TYPE and CREATE PROCEDURE privileges• Submit the job to install as SYS from the UI

View information in real-time in the database

Save a snapshot as HTML (or email the report) for offline analysis• Active report, user can visualize data on multiple dimensions, (or run from command line for text version)

Help videos available: Use Active Session History (ASH) Analyticshttps://apex.oracle.com/pls/apex/f?p=44785%3A24%3A0%3A%3ANO%3A24%3AP24_CONTENT_ID%2CP24_PREV_PAGE%3A5451%2C1

Page 52: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Advisor CentralAll database advisors grouped together on one launch page:

• Advisors, focused on recommendations

• Checkers, focused on diagnostics

Page 53: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 53

SQL Advisors• SQL Tuning Set

– Capture of SQL in the database

– Ideally run after a representative workload has been run

• Access Advisor– Analyses access patterns of SQL in the cache, or from a defined workload, and gives

recommendations on how to (re)structure the database objects

• Performance Analyzer– Analyses before and after images of SQL in a SQL Tuning Set (Testing scenarios)

– Compares results

• Tuning Advisor– Requires Tuning Sets (Collection of SQL from the database)

– Analyses the SQL from a SQL Tuning Set and gives recommendations

Page 54: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 54

AWR Warehouse and Exadata

AWR Warehouse

Scheduler jobs with Agent to Agent push to AWR Warehouse Target, scheduled by EM12c EM Job Service.

Page 55: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 55

Query Across Entire Exadata to Show True Consolidation Value

Adapt Valuable Exadata Specific Queries to AWR Warehouse-

from dba_hist_sys_time_model stm, dba_hist_snapshot s, gv$parameter p, dbsnmp.caw_dbid_mapping m

where stm.stat_name in ('DB CPU','background cpu time')

and LOWER(m.target_name)= '&dbname'

and s.dbid= m.new_dbid and s.snap_id = stm.snap_id

and s.dbid = stm.dbid and s.instance_number = stm.instance_number

and p.name = 'cpu_count' and p.inst_id = s.instance_number)

Page 56: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 56

IO Information Across Databases on Exadata?

16:5916:5917:2917:2917:5917:5918:3018:3019:0019:0019:3019:3020:0020:000

50000

100000

150000

200000

250000

300000

350000

400000

June 10, 2013

l_reads_s read_iops_maxread_iops_direct read_iops_direct_max

20:3020:30

20:5920:59

21:3021:30

22:0022:00

22:3022:30

23:0023:00

23:3023:30

23:5923:59

0:290:29

0

100000

200000

300000

400000

500000

600000

June 10, 2014

l_reads_s read_iops_maxread_iops_direct read_iops_direct_max

Page 57: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 57

Answer Specific IT Questions

Page 58: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 58

Program Agenda with Highlight

1

5

2

3

4

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Page 59: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 59

Making Copies of Configuration Files

Copy and automate copying of essential database files via Enterprise Manager-

• All instance/database init and spfiles• Database password files• ASM init files• ASM password files

Create an Enterprise Manager job to copy these files on a regular basis.

Page 60: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 60

A Review of Hardware Network “Red Herrings”• HA can mask problems, reduced capacity even while

meeting service levels

• Hardware Utilization Issues

• CPU is not what it may seem

• I/O Subsystem (Cell screens)

• Network Congestion (IB screens)

Page 61: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 61

Achieving the tao of Exadata

Exadata is an Engineered System, Powerful and Complex!

• Perform Exachk reviews quarterly, (preferably monthly.)

• Install Enterprise Manager Plug-in for advanced monitoring and enhanced performance impact detection.

• Take advantage of the great performance and tuning features included as part of the diagnostic and tuning packs

• Use IO Resource Manager to deter from impact to critical database systems.

Page 63: Oracle Open World Exadata Monitoring and Management with EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 63

Connect with me via Social Media:

Page 64: Oracle Open World Exadata Monitoring and Management with EM12c