plsql hierarchical profiles for effective tuning

8/20/2019 PLSQL Hierarchical Profiles for Effective Tuning

1/47

Best Practices for InterpretingPL/SQL Hierarchical Profiles for Effective Tun

Avaloq | www.avaloq.com

Switzerland, UK, Luxembourg, Germany, France, Singapore, Hong Kong, Philippines, Austra

CON2082

26 October 2015

Martin Buechi, Lead Software Architect


2/47

Presenter Sharing Passion for Efficient Development of High-Quality DB A

Avaloq: Swiss core banking system vendor

Development started 1993 (Oracle 7), 200 PL/SQL developers

Business logic in 20 M lines of PL/SQL in database

Approach also relevant for small shops with little DB code

Martin Büchi: Architect responsible for Oracle foundation

Oracle PL/SQL Developer of the Year 2009

Other Oracle interests: modularization, ILM, tools

Goals

Share passion. Help you improve your development.

No selling of consulting services or products.

Performance Is Key


3/47

Performance is a key feature of any application

Goals of Performance Profiling

“The universal experience of program

who have been using measurement tobeen that their intuitive guesses fa

Donal

Performance is what users experience:The response time from the click

to the display of the result on the screen.


4/47

Design for performance, then measure performance & tune where ap

Sample tuning process 1. Focus on most valuable business process and biggest resource consu

2. Explore alternatives to achieve same business outcome (requirements

3. Measure response time

4. Tune what takes most time. List improvement options. Implement best

End-to-end performance profile

BrowserNetworkMiddle tier

Database

Overall End-to-End Process and Profile

0 ms 100 ms 2


5/47

Often significant potential

Where to look for performance

Answer questions – get actionable information on important business

How long did PL/SQL take?

Why did it take so long?

Why Profile PL/SQL?

Level Potential # Presentation

Algorithm (e.g., PL/SQL) Very high Very few

Physical data modelling Usually lower Few

SQL Usually lower Very many

Infrastructure, instance Usually lower Many, past very

How can the time be reduced?

Are we done (economically optim


6/47


7/47

Start with dbms_hprof, then use dbms_profiler if reporting on line-level requnclear where time spent inside subprogram, code coverage reporting)

PL/SQL Profilers

Property dbms_profiler dbms_h

Since (and mostly unchanged since) 8.1.5 11gR1

Recording by line + -

Call sequence recording - +

Memory allocation recording - +

Low performance overhead - +

Recording table file


8/47

Package

Installed on all DBs: @$ORACLE_HOME/rdbms/admin/catproc.sql

grant execute on SYS.DBMS_HPROF to ;

Directory

create or replace directory PLSHPROF_DIR as '/tmp';

grant read, write on directory PLSHPROF_DIR to ;

Optional tables (SYS.DBMSHP_%) for reporting

@?/rdbms/admin/dbmshptab.sql

Installation of PL/SQL Hierarchical Profiler


9/47

PL/SQL API

dbms_hprof.start_profiling('PLSHPROF_DIR', 'myprofile.hpf');

/* activity to be profiled */

dbms_hprof.stop_profiling; /* optional, or just terminate se

Built into SQL Developer, TOAD, PL/SQL Developer, etc

SQL Developprofile into tadeletes raw fiusing built-in

Gathering of Profiles


10/47

Scoping

Avoid unnecessary actions (PL/SQL executions) when profiling is activa

Wait events in profile if and only if PL/SQL on stack profile time ≤ elap

– In profile: time in PL/SQL, wait in SQL executed by PL/SQL, Blocking wait in P(dbms_aq.dequeue, dbms_lock.request, dbms_alert.waitany, etc)

– Not in profile: “SQL*Net message from client”, wait in SQL execute directly by

Ideally, turn start / stop with top-level call or on same call stack level

Measure realistically

First vs. following executions

– On DB: PL/SQL code loading, SQL parsing, buffer cache

– In session: package initialization

Taking Useful Profiles


11/47

No out of the box solutions for

Gathering for other session and based on module like dbms_monitor for

No third-party tools for scoping of profiles / no always running recorder l

Build your own

Instrumentation wrapper that calls dbms_application_info.set_* and

Starts / stop PL/SQL hierarchical profiler (and other tools). Unique namesince re-start with same name overwrites. OK to concatenate.

Receives commands from other sessions, e.g., through global context

Raise an SR to support Bug 12883592 : ADDITIONAL FUNCTIONALITYINSPECT THE CURRENT STATE OF OTHER SESSION

Logon trigger as last resort

Build Your Own Advanced Gathering


12/47

Make Gathering & Reporting Accessible to Developers


13/47

create or replace procedure h isprocedure sleep(seconds number) is l_cnt number;begin

if seconds is not null then dbms_lock.sleep(seconds); enselect count(*) into l_cnt from dba_objects, dba_tablesp

end sleep;--procedure call_no_sleep

is begin sleep(null); end call_no_sleep;--procedure call_sleep(seconds number)is begin sleep(seconds); end call_sleep;

begincall_sleep(1); call_sleep(10);call_no_sleep;

end h;

Example with Hierarchy and Skew


14/47

Header P#V PLSHPROF Internal Version 1.0Start P#! PL/SQL Timer Started

...Call P#C PLSQL."SYS"."DBMS_LOCK"::11."SLEEP"#9689ba46Execute μs P#X 1000064 Return P#R

...

Content: Call, Elapsed Time with PL/SQL on Stack, Re

Namespace Object Type Subprogram S


15/47

May not need the most detailed format every time

Format options

Raw or processed (HTML, table, graphical) with or without navigation (d

Individual calls or aggregated (average, median, histogram)

Different perspectives

Look at profiles from multiple angles before taking actions.

A fool with a tool is still a fool

Study & understand output on toy example first

engineer: “efficiently get relevant, actionable information” scientist: “understand everything”

presenter: “nice graphics to explain”

Presentation & Reading Skills to Find Relevant Proble


16/47

Exampleoracle@srv:db > ${ORACLE_BIN}/plshprof -output myprofile mypPLSHPROF: Oracle Database 11g Enterprise Edition Release 11.64bit Production[7 symbols processed][Report written to 'myprofile.html']

oracle@srv:db > ls myprofile*.html

myprofile_2c.html myprofile_2n.html myprofile_fn.htmlmyprofile_md.html myprofile_ms.html myprofile_nsf.htmlmyprofile_pc.html myprofile_td.html myprofile_ts.htmlmyprofile_2f.html myprofile_aggr.html myprofile.htmlmyprofile_mf.html myprofile_nsc.html myprofile_nsp.htmlmyprofile_tc.html myprofile_tf.html

Script to concatenate to single HTML file in appendix.

HTML Output with plshprof


17/47

High SQL percentage?

Problem in SQL

Test system with bad I/O

Bad algorithm in PL/SQL with too many SQL calls

PL/SQL or SQL?


18/47

SQL trace shows

Bad SQL, slow I/O, concurrency waits Number of rows returned processed in SQL

Must start SQL trace first

1 dbms_monitor.session_trace_enable;2 dbms_hprof.start_profiling('PLSHPROF_DIR', 'myprof.hpf');

Bug 22085980 : SETTING A DATABASE EVENT IN A SESSION STOPS THIERARCHICAL PROFILER

PL/SQL Hierarchical Profile and SQL Trace


19/47

Duration = #Execution * #AverageDurationPerExecution fewer calls

Focus on high duration and many executions

Expensive and Called Multiple Times


20/47

May not directly yield biggest benefit, but why so many calls?

Counts: Do We Need to Call a Subprogram n Times?


21/47

Subtree with Much Higher Ind% than Next Line


22/47

Optionally with tool showing source code

Look at breakdown of time in report and examine source code.

Percentage of function as part of subtree not shown.

Breakdown of Children / Parent

Percentage of Descendants (not Subtree)

Percentage of total run


23/47

Very useful if time in some packages split across very many subprogram

Possible option to estimate tuning potential.

Time by Module


24/47

Skew in Subtree Between Callers (Parents)


25/47

Own call analysis tool for individual call and histogram reporting

select * from table(single_call('PLSHPROF_DIR', 'myprofile.h'K.H.H.SLEEP'));

Focused report

${ORACLE_BIN}/plshprof -trace '"K"."H"."H.SLEEP"' -skip 1 -coutput second myprofile.hpf

Skew in Subtree Calls from Same Parent

OCC PARENT SUBTREE FUNCTION DE

1 K.H.H.CALL_SLEEP 2,800,569 42

2 K.H.H.CALL_SLEEP 11,812,431 17

3 K.H.H.CALL_NO_SLEEP 1,762,206 12


26/47

When gathering cannot be properly scoped / focus without distorted

Subtree focus & number of invocations

plshprof -trace '"K"."H"."H.SLEEP"' -skip m -collect n ..

Always three dot-separated arguments between "". See raw trace for ex

– Standalone subprogram H in schema K "K"."H"."H"

– PL/SQL VM "".""."__plsql_vm"

– Anonymous PL/SQL "".""."__anonymous_block"

– Specific SQL "K"."P"."__dyn_sql_exec_line

Limit Reporting to Contiguous Set of Invocations of Su


27/47

Export to Microsoft Excel for Annotation and Processin


28/47

Example

Load into Tables SYS.DBMSHP_%


29/47

Careful: Hierarchy and Skew Lost in Import into Tables


30/47

Sample Domain-Specific Custom Reports for Generat

id type classif cond …

1 5 9090 [doc.bp.class(9089)=101619]…

2 5 5075 [rm$spread_deriv.rel_eam(doc) = '0'] …

create or replace package body k.deriv# isfunction cond$1(

i_doc doc_mgr#.t_doc

) return varchar2 is

1 . G en

er a t e

module function subtree_elapsed_time

deriv# cond$1 973

id type

1 5

2. Execute with HProf, load i

3. Join


31/47

Interactive flame graph (http://www.brendangregg.com/flamegraphs.html)

exec flatten('PLSHPROF_DIR', 'myprofile.hpf', 'myprofile.flaflamegraph.pl myprofile.flat > myprofile.svg

Graphical Presentations and Profiler for Other Langua

http://www.brendangregg.com/flamegraphs.htmlhttp://www.brendangregg.com/flamegraphs.htmlhttp://www.brendangregg.com/flamegraphs.html


32/47

Compact semicolon-separated time-aggregated call stack with durati

Execution count in brackets is our addition

H.H [1] 7

H.H [1];H.H.CALL_SLEEP [2] 4

H.H [1];H.H.CALL_SLEEP [2];H.H.SLEEP [2] 59

H.H [1];H.H.CALL_SLEEP [2];H.H.SLEEP [2];H.__static_sql_exec_line7 [2] 3609411

H.H [1];H.H.CALL_SLEEP [2];H.H.SLEEP [2];DBMS_LOCK.SLEEP [2] 11003530H.H [1];H.H.CALL_NO_SLEEP [1] 1

H.H [1];H.H.CALL_NO_SLEEP [1];H.H.SLEEP [1] 12

H.H [1];H.H.CALL_NO_SLEEP [1];H.H.SLEEP [1];H.__static_sql_exec_line7 [1] 17621

flamegraph.pl Input from Sample Program


33/47

Semantics

y-axis: Call stack, x-axis: width proportional to duration, siblings sorted a

Interactive: Click to zoom / reset zoom, Search

flamegraph.pl Output of Sample Program


34/47

Generate cpuprofile from PL/SQL HProf trace

exec chrome_cpuprofile('PLSHPROF_DIR', 'myprofile.hpf','myprofile.cp

Files too big? Use (sparse / adaptive) sampling

Google Chrome Developer Tools cpuprofile

https://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-execution


35/47

{"head": {"functionName": "(root)", "id": 1,

"children": [{"functionName": "H.H", "hitCount": 7, "id": 2,"children": [{"functionName": "H.H.CALL_NO_SLEEP", "id": 3, ..."functionName": "H.H.SLEEP", "id": 4, ..."functionName": "H.__static_sql", "id": 5, ...

}, ...]

}]

},"startTime": 0, "endTime": 16375218,"samples": [2, 4, 5, 4, ... ],

"timestamps": [0, 1, 5, 1762199, ... ]

Chrome Input from Sample Program

For aggregated times


36/47

Opening .cpuprofile files

https://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-e

Google Chrome Developer Tools cpuprofile

https://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-executionhttps://developers.google.com/web/tools/chrome-devtools/profile/rendering-tools/js-execution


37/47

Semantics

y-axis: Call stack (top down)

x-axis: time

Flame Chart of Sample Program

Many more visualization tools tha

use, e.g., Callgrind / KCachegrind


38/47

Manual

Stare at two reports and compare manually SQL to compare runs loaded into SYS.DBMSHP_% tables

plshprof HTML with two input files

${ORACLE_BIN}/plshprof -output comp h1.hpf h2.hpf

Difference Between Runs (Tuning, Input Variation, etc


39/47

Sample Comparison Overview Report


40/47

Smaller files, but relevant information may be lost requiring re-gather

Syntax

dbms_hprof.start_profiling(…, max_depth => n)

Raw trace file content: Time summed up at last reported levelmax_depth => null max_depth => 2

P#C PLSQL."K"."H"::7."H.CALL_SLEEP" P#C PLSQL."K"."H"::7."H.CALLP#X 1 P#X 2800544P#C PLSQL."K"."H"::7."H.SLEEP" P#R

P#X 23P#C PLSQL."SYS"."DBMS_LOCK"::11."SLEEP"P#X 1000064P#R... /* statis SQL call */ P#RP#X 0P#R

Limiting Call Depth in Gathering

o m i t t e d d e t a i l s


41/47

Compilation options (plsql_%)

PL/SQL hierarchical profiler also works for wrapped and natively compile

With inlining (plsql_optimize_level=3 or pragma inline), calls to inlined suare not visible. Likewise when call to deterministic function is optimized a

Usage notes

Performance verhead low enough to run on production. Ensure enough free disk space and not on a critical file system if full. Ca

maximum file size.

No need for a fast server. Just compare relative times in PL/SQL (not SQPL/SQL).

Bits & Pieces 1/2


42/47

Other usages

Call stack until hang since no stop needed – alternative to oradebug dumerrorstack 1

Actual call tree (of test input only) – vs. complete possible static calls of

References

https://docs.oracle.com/database/121/ADFNS/adfns_profiler.htm#ADFN

Bits & Pieces 2/2

https://docs.oracle.com/database/121/ADFNS/adfns_profiler.htm#ADFNS023https://docs.oracle.com/database/121/ADFNS/adfns_profiler.htm#ADFNS023


43/47

Gathering

dbms_hprof.start_profiling(..., profile_uga | profile_pga =>

Often in combination with dbms_session.get_package_memory_utilizati

What’s in the trace

P#Z 1169723168

P#Z -2146643687

Reporting

plshprof -uga or -pga

dbms_hprof.analyze(..., profile_uga | profile_pga => true)

Memory Allocation Profiling: UGA or PGA

Oracle internal. Not documen


44/47

Conclusions

Performance is a key feature of any application

Oracle-provided tools are sufficient for analysis of PL/SQL hierarchical pexcept for skew analysis and graphical people—for whom this presentat

Understand output on toy example before analyzing real code.

Action items

Profile PL/SQL as part of end-to-end performance analysis.

Integrate gathering and reporting into your developer andkey business user UI.

Conclusions & Action Items


45/47


46/47

Thank you for your attention.

Martin Bü[email protected]


47/47

The KSH, PL/SQL, and Java code for analyzing and transforming PL/SQL Profiler traces as shown in the presentation can be found at

https://drive.google.com/open?id=0B40PcvWHeRrrZW1oRjg1WHFfOEE .

This is sample code. Not production quality code. No warranties.

Sample Code Only
https://drive.google.com/open?id=0B40PcvWHeRrrZW1oRjg1WHFfOEEhttps://drive.google.com/open?id=0B40PcvWHeRrrZW1oRjg1WHFfOEE

plsql hierarchical profiles for effective tuning

Documents