K E L L Y N POT ’VIN

S R . T E C H N I C A L C O N S U L T A N T

EM12c Performance Diagnosis and Tuning Outside the Box

Kellyn Pot’Vin

Westminster, Colorado

Oracle ACE Director, Sr. Technical Specialist at Enkitec

Specialize in performance and management of large enterprise environments.

Board of directors for RMOUG, Director of Training Days and Database Track Lead for KSCOPE 2013

Blog: DBAKevlar.com

Twitter: @DBAKevlar

Performance Diagnostics in EM12c

Simple access to performance, resource usage and demands.

Data collection to investigate performance issues- current, recent and historical.

Capacity planning.

Have the real answer, not assumptions.

Presentation Agenda

Performance Out of the Box with EM12c

Top Activity

SQL Monitor

ASH Analytics

Real-time ADDM

Compare ADDM

Tools at your Disposal

Requires the Diagnostics Pack

Top Activity, “The Grid”

Graphical display of performance usage.

15 second refresh, manual refresh or historical.

When to Worry

Out of the Ordinary Activity, (KNOW YOUR DB!)

Colors outside of green and [some] blue.

Large amounts of blue, (high IO)

Remember that pink, (unknown) red, (concurrency/application) tan, (network) and orange, (commit) in the grid should be investigated.

Brown or black? Run for the hills! (JK)

Here’s our spike, which waits?

Commonly, focus on pink, orange, red and brown for issues.

Network and queuing do have opportunities for tuning, as well.

Green and blue are expected, but also part of problems when over utilized.

We’re in the Red, (Orange, too!)

Inspect High % use.

Note that

the update and execution may be impacting each other.

Session Details

Seeing Red…

Next?

Two sessions are executing

Option to run an AWR or ASH report, (right hand side)

What ASH Tells Us

The Icing on the Cake

Duh, add some memory to the EM12c box!

SQL Monitor for Performance

• Elapsed Time

• SQL_ID, Beginning SQL Text.

• Parallel, Waits and Execution Time

Digging in

• Choose your session, SQL_ID or SQL_Text • Shows active, completed sessions for amount of time chosen. • Shows high level wait events, dbtime, IO usage and duration.

Digging Down

By SQL_ID, we can inspect: • Duration • DB Time • PL/SQL Java time • Wait Activity • Buffer Gets • IO Requests and IO Bytes • If Exadata, Offload Efficiency

Monitoring Procedural Call

All SQL_ID’s called will show, along with duration so it’s simple to pinpoint trouble statements.

SQL Details

• Note that the SQL Statement, along with elapsed time is shown.

• Data sources from Top Activity, not AWR data.

And More Detail

Session info, wait info, cursors and stats.

Added Data

Along with the main stats-

Activity information on the statement.

The execution plan

If there is a SQL Plan or outline in place.

If there have been any tuning advisors run against the statement

And a direct link to SQL Monitoring

How to Use SQL Monitoring

Active Monitoring of database processing.

Investigation of performance.

Save off reports, which provide a graphical image of performance differing from Top Activity or ASH Analytics.

Distinct diagnosis at a session or statement level.

ASH Analytics

Future of Top Activity

Package installation to database.

Always on, non-impact of Top Activity performance data gathering.

More defined, more accurate.

Historical data enhanced over Top Activity historical views.

Pick Your View

Ability to choose timelines by:

Hour Day

Week Month

Calendar Custom

Custom Review Pane

• You can choose to change the overview pane to display data for any amount of time.

• Just click on the pane and drag it to the area you are interested in or extend it to cover the areas you are interested to investigate.

• Choose your filters or view all data and you are ready to go!

Familiar Interface

Similar to Top Activity when in “Activity” mode.

Sql Details View

Pick Your Poison

View data very similar to the SQL and Session data in Top Activity.

All data is sourced by AWR data and dependent on samples and AWR retention/interval info in the respository.

It’s All in the Details

Activity Details

Activity shows wait detail over time.

Processes, including parallel sessions involved during shaded time.

Option to run AWR or ASH report.

The Rest of the Story

For standard SQL- Plan, Plan Control and Tuning History is shown under individual tabs.

SQL Monitor is minimized access to the SQL Monitor view.

Load Map

New Visual Way of Showing Data, Multiple

Ways!

Data Break Down

Display offers incredible diversity in wait, resource usage and other critical event choices.

ASH Analytics – When to Use It

Need the more defined ASH data for EM diagnostics.

Want a second way to present data to less “DBA” centric groups, (load map)

Database level OR session/statement level performance diagnosis.

Dig down deep, present data in numerous formats to get the most complete picture of a complex issue.

Can be used for Real-time or historical analysis.

Real-Time ADDM

Yes, it requires a PL/SQL installation for the view data.

Uses ADDM data for the source.

Always on, low to no impact.

Normal Mode or Emergency Mode when Emergency Monitoring is required.

On Your Mark, Get Set…

This is a recorded ADDM session, beginning from the time you click “Start”.

In Progress Data

Ability to stop and restart.

Findings gathered during progress.

Check progress notifies of any issues.

Finished!

Once finished, verify no failures/errors occurred in the collection.

Use the tabs to investigate findings, activity, hang data and statistics.

The number of findings are shown.

The Findings

Example shows low priority SQL statements using significant db time, but not other issues at this time.

If any issues are found that are high priority, will be listed in red and details below the main pane, (low, medium, high priority levels.)

Activity Tab

Activity Data, but sourced from ADDM.

Similar output to Top Activity and ASH Analytics.

Wait Details

• By highlighting a wait link on the right, you can detail down to the actual wait information for that wait event.

Hanging out

If a database hang situation occurred and the real-time ADDM was used to diagnose, then the HANG DATA tab will show any diagnostic data it has collected during the collection.

Statistics Data:

Last but not Least…

Initialization Parameter data for the database instance.

Any undocumented of non-recommended parameter settings will be identified and listed in the findings section.

Compare Period ADDM

How is it different from Real-Time ADDM?

Ability to compare TWO snapshots in time, side by side of ADDM data.

Compares ADDM snapshots against each other, (dependent on snapshot intervals and retention.)

All comparisons can be saved off or mailed from the console, (mailed through EM12c settings)

Choosing a Comparison Time

Comparison Activity

• Clear comparison from previous day, same time to see performance issue vs. the right hand side snapshot.

• Commonality comparison of the SQL for snapshots being compared. • Note the concurrency, commits and increased application waits.

It’s all in the Details

First tab shows any configuration differences between the two snapshots and what the configuration parameter is.

Findings Summary Detail

Shows comparison increases or decreases in waits.

Lists the percentage of change between each period compared.

Upon highlighting, details data regarding the increase or decrease.

SQL Changes

We can dig down into each of the SQL Statements found to be the highest impacts to the system and diagnose further.

Finding Detail Descriptions

As shown above, the wait on Checkpoints to Tablespace are describe below once you highlight the section in the findings tab.

And for RAC, some waits can be broken down by instance.

Resource Usage: CPU

CPU Usage is viewable by instance and total usage.

If no CPU bound wait issues were seen, its stated by comparison snapshot.

Resource Usage: Memory

• If you note, Memory has a warning alert by the tab to point you to it after the comparison is completed.

• The base and comparison is in red, meaning that Virtual paging was an issue in both snapshots.

• Data is separated by instance in RAC, showing clear usage for better diagnostics.

Resource Usage: IO

I/O is separated by Throughput and Single block read latency.

Again, if there was an issue, a warning would be on the IO tab and the Base and Comparison would show in red instead of green.

Resource Usage: Interconnect

As this is RAC, note that we also have an interconnect tab with data on the speed and performance.

Total vs. rate on throughput is viewed through a radio button choice.

So What Changed?

The graphs show us where we need to focus:

How to Use the Comparison ADDM

Excellent to diagnose “what has changed”.

“Just the Facts” information on a comparison of time.

Dependent upon retention time settings and intervals for AWR.

Historical data, can be set by date, custom, by previous snapshot.

Will move to next snapshot window if mid-snapshot time span is chosen.

EM12c blogs-

Leighton Nelson- http://blogs.griddba.com/

Rob Zoeteweij-http://oemgc.files.wordpress.com/

Gokhan Atil- http://www.gokhanatil.com/

Martin Bach- http://martincarstenbach.wordpress.com

Niall Litchfield- http://orawin.info/blog/

Info for Me!

Company Website: www.enkitec.com

Twitter: @DBAKevlar

RMOUG: www.rmoug.org

Linkedin: Kellyn Potvin and/or Rocky Mountain Oracle User Group

Email: dbakevlar@gmail.com or kpotvin@enkitec.com or TrainingdaysDir@rmoug.org

Blog: https://dbakevlar.com

Reference

Kscope13 features more than 300 educational sessions, full-day symposiums, hands-on training courses, informal networking sessions, and a plethora of chances to increase your technical know-how by learning from the best. • Application Express • ADF and Fusion Dev. • Developer's Toolkit • The Database • Building Better Software • Business Intelligence • Essbase • Planning • Financial close • EPM Reporting • EPM Foundations and Data

Management • EPM Business Content http://kscope13.com/registration