SQL tuning with SPMchange the way you think

Björn Rost

• consultant

• oracle database (HA and performance)

• OS systems (Linux and Solaris)

• automation puppet and ansible

• president of RAC SIG

• ACE Director

• > 42 presentations on 5 continents

BJÖRN

© 2014 Pythian Confidential

ABOUT PYTHIAN

3

11,400 Pythian currently manages more than 11,400 systems.

400+ Pythian currently employs more than 400 people in 200 cities in 35 countries

1997 Pythian was founded in 1997

Global Leader In IT Transformation And Operational Excellence

Unparalleled Expertise

• Top 5% in databases, applications, infrastructure, Big Data, Cloud, Data Science, and DevOps

Unmatched Certifications • 9 Oracle ACEs, 4 Oracle ACE Directors, 1 Oracle ACE Associate • 6 Microsoft MVPs, 1 Microsoft Certified Master • 5 Google Platform Qualified Developers • 1 Cloudera Champion of Big Data • 1 Mongo DB Certified DBA Associate Level • 1 DataStax Certified Partner, 1 MVP

Broad Technical Experience • Oracle, Microsoft, MySQL, Oracle EBS, Hadoop, Cassandra, MongoDB,

virtualization, configuration management, monitoring, trending, and more.

© 2014 Pythian Confidential

SOME OF OUR CLIENTS

4

AGENDA

• optimizer history and motivation

• SPM basics

• process design

• (our) implementation

• extra use cases

MY ORACLE HISTORY

• “installed” Oracle 8 - once

• “ran” two installations of 9i

• “serious” Oracle work not until 10g

RULE BASED OPTIMIZER

• matter of legends - a thing of the past

• not being updated since 7

• unsupported since 10g

RULE BASED OPTIMIZER

SQL> select * from attendees where gender = 'male';

RULE BASED OPTIMIZER

SQL> select * from attendees where gender = 'male';

SQL> select * from attendees where gender = 'male' and country = 'germany';

RULE BASED OPTIMIZER

SQL> select * from attendees where gender = 'male';

SQL> select * from attendees where gender = 'male' and country = 'germany';

If an index on “gender” exists, it will always be used. Even if full scan would be better or “country” had better selectivity

RBO JOINS

• driving table determined by access paths

• trying to avoid FTS on inner table

• nested loop and SMJ only (no hash join)

RULE BASED OPTIMIZER

• developers had to think when writing SQL

• make assumptions about data (in prod)

• but easy for DBAs

• plans were predictable

• and never changed (by themselves)

• blame was on developers

COST BASED OPTIMIZER (CBO)

• introduced in Oracle 7 (1992)

• based on statistics

• compares all possible plans by “cost”

• adapts to new situations

• volume of data

• skewed data and selectivity for different binds

• writing SQL is also really easy now

• don’t bother with join order, indexes etc

CBO PROBLEMS

• executions plans are now a moving target

• plenty of things to “change” and go wrong:

• statistics

• cardinality estimates

• different per variable/literal

• new access paths

• there have to be thresholds that can “flip”

FLIPPING PLANS

• noone calls about CBO finding better plans

• but plenty complain about worse plans

LOSS AVERSION

• Daniel Kahneman

• nobel price in economics

• coin-flip game payouts

• loose $5 vs finding $5

• use worse plan vs use better plan

FLIPPING PLANS

• fix is usually very easily

• some stop/manage gathering stats

• sometimes led to hint-crazyness

• some even want the RBO back!

• or “please freeze our plans”

WHY SPM

• gain plan stability

• flipping SQL execution plans

• rather “static app”

• SPM better in OLTP than DWH

• easy fix

• nasty consequences

• at “uncontrollable” times

• SQL Plan Management can really help!

PROJECT GOALS FOR SPM

• prevent random changes for the worse

• change management for plans (just like code, config …)

• introduce a process for new plans

‣ find and evaluate

‣ review

‣ approve/implement

WHAT IS SPM?

• add one more layer to CBO

• plan “quarantine”

• accept plans/baselines

• prevent new plans from being executed

• but store them anyway

• then review

BASELINES

• stored hints to reproduce a given plan

• examine with display_baseline +OUTLINE

• plan_hash_value to compare

•DBA_SQL_PLAN_BASELINES

BASELINES

select * from table ( dbms_xplan.DISPLAY_SQL_PLAN_BASELINE(

sql_handle=>’SQL_086fba8dc9c2c972', format=>'OUTLINE'));

RULE BASED OPTIMIZEROutline Data from SMB:

/*+ BEGIN_OUTLINE_DATA SWAP_JOIN_INPUTS(@"SEL$27F3751A" "THIS_1_"@"SEL$1") USE_HASH(@"SEL$27F3751A" "THIS_6_"@"SEL$11") USE_HASH(@"SEL$27F3751A" "THIS_1_"@"SEL$1") USE_NL(@"SEL$27F3751A" "THIS_4_"@"SEL$7") USE_NL(@"SEL$27F3751A" "THIS_3_"@"SEL$5") USE_NL(@"SEL$27F3751A" "THIS_2_"@"SEL$3") USE_NL(@"SEL$27F3751A" "THIS_5_"@"SEL$9") LEADING(@"SEL$27F3751A" "THIS_"@"SEL$2" "THIS_5_"@"SEL$9" "THIS_2_"@"SEL$3" "THIS_3_"@"SEL$5" "THIS_4_"@"SEL$7" "THIS_1_"@"SEL$1" "THIS_6_"@"SEL$11") FULL(@"SEL$27F3751A" "THIS_6_"@"SEL$11") FULL(@"SEL$27F3751A" "THIS_1_"@"SEL$1") INDEX_RS_ASC(@"SEL$27F3751A" "THIS_4_"@"SEL$7" ("A"."ID")) INDEX_RS_ASC(@"SEL$27F3751A" "THIS_3_"@"SEL$5" ("B_RECOMMENDATION"."ID")) INDEX_RS_ASC(@"SEL$27F3751A" "THIS_2_"@"SEL$3" ("C_MONTHLY_CREDIT"."ID")) INDEX(@"SEL$27F3751A" "THIS_5_"@"SEL$9" ("D_TARIFF"."ID")) FULL(@"SEL$27F3751A" "THIS_"@"SEL$2") OUTLINE(@"SEL$1") OUTLINE(@"SEL$2") MERGE(@"SEL$1") OUTLINE(@"SEL$58A6D7F6") OUTLINE(@"SEL$3") OUTLINE(@"SEL$4") MERGE(@"SEL$58A6D7F6") MERGE(@"SEL$3") OUTLINE(@"SEL$7237DA6D") OUTLINE(@"SEL$5") OUTLINE(@"SEL$6") OUTLINE(@"SEL$7") MERGE(@"SEL$7237DA6D") MERGE(@"SEL$5") OUTLINE(@"SEL$15E987C1") OUTLINE(@"SEL$8") OUTLINE(@"SEL$9") MERGE(@"SEL$7") MERGE(@"SEL$15E987C1") OUTLINE(@"SEL$096E5AED") OUTLINE(@"SEL$10") MERGE(@"SEL$9") MERGE(@"SEL$096E5AED") OUTLINE(@"SEL$B97648DD") OUTLINE(@"SEL$11") OUTLINE(@"SEL$12") MERGE(@"SEL$B97648DD") MERGE(@"SEL$11") OUTLINE(@"SEL$72AEFE3E") OUTLINE(@"SEL$13") MERGE(@"SEL$72AEFE3E") OUTLINE_LEAF(@"SEL$27F3751A") ALL_ROWS DB_VERSION('11.2.0.4') OPTIMIZER_FEATURES_ENABLE('11.2.0.4') IGNORE_OPTIM_EMBEDDED_HINTS END_OUTLINE_DATA */ --------------------------------------------------------------------------------

BASELINES

Plan hash value: 2175138020

-------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | -------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 221 | 17 (6)| 00:00:01 | | 1 | SORT ORDER BY | | 1 | 221 | 17 (6)| 00:00:01 | |* 2 | HASH JOIN OUTER | | 1 | 221 | 16 (0)| 00:00:01 | |* 3 | HASH JOIN RIGHT OUTER | | 1 | 210 | 13 (0)| 00:00:01 | | 4 | TABLE ACCESS FULL | D_TARIFF | 156 | 2652 | 3 (0)| 00:00:01 | | 5 | NESTED LOOPS OUTER | | 1 | 193 | 10 (0)| 00:00:01 | | 6 | NESTED LOOPS OUTER | | 1 | 184 | 9 (0)| 00:00:01 | | 7 | NESTED LOOPS OUTER | | 1 | 175 | 8 (0)| 00:00:01 | | 8 | NESTED LOOPS OUTER | | 1 | 162 | 7 (0)| 00:00:01 | |* 9 | TABLE ACCESS FULL | PROMO | 1 | 158 | 7 (0)| 00:00:01 | |* 10 | INDEX UNIQUE SCAN | D_TARIFF_PK | 1 | 4 | 0 (0)| 00:00:01 | | 11 | TABLE ACCESS BY INDEX ROWID| C_MONTHLY_CREDIT | 1 | 13 | 1 (0)| 00:00:01 | |* 12 | INDEX UNIQUE SCAN | C_MONTHLY_CREDIT_PK | 1 | | 0 (0)| 00:00:01 | | 13 | TABLE ACCESS BY INDEX ROWID | B_RECOMMENDATION | 1 | 9 | 1 (0)| 00:00:01 | |* 14 | INDEX UNIQUE SCAN | B_RECOMMENDATION_PK | 1 | | 0 (0)| 00:00:01 | | 15 | TABLE ACCESS BY INDEX ROWID | A | 1 | 9 | 1 (0)| 00:00:01 | |* 16 | INDEX UNIQUE SCAN | A_PK | 1 | | 0 (0)| 00:00:01 | | 17 | TABLE ACCESS FULL | PROMO_SIMPLE | 551 | 6061 | 3 (0)| 00:00:01 | --------------------------------------------------------------------------------------------------------------------

BASELINE STATUS• enabled

• defaults to YES

• accepted

• this plan can actually be run

• default YES for 1st one, NO for next

• fixed

• stop collecting new baselines

• default: NO

SQL SELECT * FROM STUFF WHERE COL=?;

execute PLAN!

PARSING

SQL

sql_id

SELECT * FROM STUFF WHERE COL=?;

SELECT * FROM stuff WHERE COL=?;

PARSING

SQL

sql_id

execute ?yes

• look up sql_id in shared pool

• if this exists->soft parse

• more complex:

• adaptive cursor sharing

• bind peeking

• adaptive plan execution

• about 60 more reasons for child cursors

PARSING

SQL

sql_id

hard parseexecute ?yes no

• CBO finds plan

• tries multiple variants

• lowest “cost” plan wins

• cost influenced by:

• access paths (indexes?)

• statistics

PARSING

SQL

sql_id

hard parseexecute ?

?

execute

yes

yes

no

• check if CBO plan exists in SPM

• yes and accepted => execute

• no baseline => add this and accept

PARSING

SQL

sql_id

hard parse

add baseline

execute ?

?

execute execute

yes

yes

no

no

• add baseline (unaccepted)

• execute accepted

• unless it is invalid

PARSING

1. enable use of baselines (if they exist)

2. add baselines

• automatic or manual plan capture

3. evolve baselines

4. delete unused baselines

SPM LIFECYCLE

• bind variables

• cannot manage too many sql_ids

• have same objects in all schemas

• SPM works instance-wide

• also be careful when dropping indexes

PREREQUISITES

STEP 1: ENABLE USE OF SPM

• auto capture

• import from SQL Tuning Set

• import from cursor cache

• export/import from other system

STEP 2: CREATE BASELINES

AUTO CAPTURE

• capture on 2nd execution

• auto-accept if this is first baseline

• add as unaccepted if not

• system level vs session level

• with logon trigger

AUTO CAPTURE

• load from STS

• like AWR

DECLARE l_plans_loaded PLS_INTEGER;BEGIN l_plans_loaded := DBMS_SPM.load_plans_from_sqlset( sqlset_name => 'my_sqlset');END;/

http://www.oracle-base.com/articles/11g/sql-plan-management-11gr1.php

MANUAL IMPORT

• load from cursor cache

• filter on sql_text or sql_id

DECLARE l_plans_loaded PLS_INTEGER;BEGIN l_plans_loaded := DBMS_SPM.load_plans_from_cursor_cache( sql_id => '1fkh93md0802n');END;/

http://www.oracle-base.com/articles/11g/sql-plan-management-11gr1.php

MANUAL IMPORT

• manual gives control over single SQL

• ideal to control few “problem”-SQLs

• systemwide auto will also capture “junk”

• ad-hoc and DWH queries

• oracle internal (EM monitoring) stuff

• both add new unaccepted baselines if at least one exists already

MANUAL VS AUTO

• eventually evaluate unaccepted plans

• dbms_spm.evolve_sql_plan_baseline

• this will run statement

• against real data

• with real bind variables

• report execution time

• can choose to accept or just report

STEP 3: EVOLVE BASELINES

• Automatic SQL Tuning Maintenance Job

• job enabled by default

• but implementation of actions disabled

• also suggests indexes etc

EVOLVE JOB IN 11G

EVOLVE JOB IN 11G

EVOLVE JOB IN 11G

• run job to evaluate all new baselines

• do not actually evolve (set accepted flag)

• just create report and mark baseline

• write status in “description” column

• succesful

• failed

• error

MY OWN EVALUATE SCRIPT

• investigate new baselines

• evolve report

• execution plans before and after

• used for documentation

MY REPORT SCRIPT

create or replace PROCEDURE SPM_REPORT( V_SQL_HANDLE IN VARCHAR2)ASresult_evolve clob;BEGINresult_evolve := DBMS_SPM.evolve_sql_plan_baseline(sql_handle => v_sql_handle, plan_name => null, verify => 'YES', commit => 'NO');dbms_output.put_line(result_evolve);

for i in (select plan_table_output from table (dbms_xplan.display_sql_plan_baseline(sql_handle=> v_sql_handle)))loopdbms_output.put_line (i.plan_table_output);end loop;end;

MY REPORT SCRIPT

------------------------------------------------------- Evolve SQL Plan Baseline Report-------------------------------------------------------

Inputs:------- SQL_HANDLE = SQL_fc3b803fd3b27f5d PLAN_NAME = TIME_LIMIT = DBMS_SPM.AUTO_LIMIT VERIFY = YES COMMIT = NO

MY REPORT OUTPUT

Plan: SQL_PLAN_gsfw07z9v4zux1a7a077c------------------------------------ Plan was verified: Time used 5.422 seconds. Plan passedperformance criterion: 989.19 times better than baseline plan.

Baseline Plan Test Plan Stats Ratio ------------- --------- ----------- Execution Status: COMPLETE COMPLETE Rows Processed: 0 0 Elapsed Time(ms): 2671.4 .275 9714.18 CPU Time(ms): 2680 0 Buffer Gets: 100628 102 986.55 Physical Read Requests: 0 0 Physical Write Requests: 0 0 Physical Read Bytes: 0 0 Physical Write Bytes: 0 0 Executions: 1 1

------------------------------------------------------------------- ReportSummary-------------------------------------------------------------------Number of plans verified: 1Number of plans accepted: 0

MY REPORT OUTPUT

SQL handle: SQL_fc3b803fd3b27f5dSQL text: update SOME.TABLE set STATUSID=:1 , laststatuschanged=:2where id in (:3 , :4 , :5 , :6 , :7 , :8 , :9 , :10 , :11 ,:12 , :13 , :14 , :15 , :16 , :17 , :18 , :19 , :20 , :21 ,:22 , :23 , :24 , :25 , :26 , :27 , :28 , :29 , :30 , :31 ,:32 , :33 , :34 , :35 , :36 , :37 , :38 , :39 , :40 , :41 ,:42 )

MY REPORT OUTPUT

--------------------------------------------------------------------------------Plan name: SQL_PLAN_gsfw07z9v4zux077f8913 Plan id: 125798675Enabled: YES Fixed: NO Accepted: YES Origin: AUTO-CAPTURE--------------------------------------------------------------------------------Plan hash value: 125798675-----------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost | Time | Pstart| Pstop |-----------------------------------------------------------------------------------------------| 0 | UPDATE STATEMENT | | 45 | 945 | 137K | 00:27:29 | | || 1 | UPDATE | MYTABLE | | | | | | || 2 | PARTITION RANGE INLIST| | 45 | 945 | 137K | 00:27:29 |KEY(I) |KEY(I) ||* 3 | TABLE ACCESS FULL | MYTABLE | 45 | 945 | 137K | 00:27:29 |KEY(I) |KEY(I) |-----------------------------------------------------------------------------------------------

MY REPORT OUTPUT

--------------------------------------------------------------------------------Plan name: SQL_PLAN_gsfw07z9v4zux1a7a077c Plan id: 444204924Enabled: YES Fixed: NO Accepted: NO Origin: AUTO-CAPTURE--------------------------------------------------------------------------------Plan hash value: 444204924---------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |---------------------------------------------------------------------------------| 0 | UPDATE STATEMENT | | 45 | 945 | 48 (0)| 00:00:01 || 1 | UPDATE | MYTABLE | | | | || 2 | INLIST ITERATOR | | | | | ||* 3 | INDEX UNIQUE SCAN| MYTAB_PK | 45 | 945 | 47 (0)| 00:00:01 |---------------------------------------------------------------------------------

MY REPORT OUTPUT

➡ review report (in auto-generated ticket)

➡ request approval for action/change

➡ accept this new baseline

➡ leave comment in description column

EVOLVE PROCESS

create or replace PROCEDURE SPM_EVOLVE( V_SQL_HANDLE IN VARCHAR2, V_DESCRIPTION IN VARCHAR2)ASv_alter_result pls_integer;evolve_out clob;BEGINv_alter_result := DBMS_SPM.ALTER_SQL_PLAN_BASELINE ( sql_handle => V_SQL_HANDLE, attribute_name => 'DESCRIPTION', attribute_value => V_DESCRIPTION);DBMS_OUTPUT.PUT_LINE(v_alter_result);evolve_out := DBMS_SPM.EVOLVE_SQL_PLAN_BASELINE ( sql_handle => v_sql_handle, time_limit => DBMS_SPM.AUTO_LIMIT, verify => ‘YES',

COMMIT => 'YES' );DBMS_OUTPUT.PUT_LINE(evolve_out);end;

MANUAL EVOLVE

• new SYS_AUTO_SPM_EVOLVE_TASK

• “Adaptive SQL Plan Management”

• evaluates and auto-accepts plans

• EVOLVE_SQL_PLAN_BASELINE deprecated

• new API: EVOLVE_TASKS

http://www.oracle.com/technetwork/database/bi-datawarehousing/twp-sql-plan-mgmt-12c-1963237.pdf

http://chandlerdba.wordpress.com/2013/12/08/sql-plan-management-12c-dumb-feature/

SPM IN 12C

• delete failed baselines after a while

• to give them chance for re-evaluation

CLEANUP

• new baselines captured automatically

• but gain stability for existing plans

• potentially better plans evaluated automatically

• experience shows only few better plans/week

• ticket is automatically created each time evolve reports succeeds

• DBA reviews this, requests approval and evolve

PROCESS REVIEW

SELECT * FROM TABLE(dbms_xplan.display_cursor('ahmckrb9zn6cn'));

PLAN_TABLE_OUTPUTSQL_ID ahmckrb9zn6cn, child number 0-------------------------------------

PLAN_TABLE_OUTPUT| Id | Operation | Name | Rows | Bytes | Cost (%CPU)|--------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 13 (100)|| 1 | NESTED LOOPS OUTER | | 4 | 264 | 13 (8)||* 2 | HASH JOIN | | 4 | 188 | 7 (15)|| 3 | PARTITION RANGE SINGLE | | 4 | 156 | 3 (0)||* 4 | TABLE ACCESS BY LOCAL INDEX ROWID| TABLE_MONTH | 4 | 156 | 3 (0)||* 5 | INDEX SKIP SCAN | PK_TAB_MONTH | 116K| | 3 (0)|| 6 | TABLE ACCESS FULL | SOMEOTHERTABLE | 503 | 4024 | 3 (0)||* 7 | INDEX RANGE SCAN | SOME_IDX | 1 | 19 | 2 (0)|--------------------------------------------------------------------------------------------

EXAMPLE: OVERRIDE STUPID HINTS

CREATE OR REPLACE VIEW "SOME"."VIEW" ("COL1","COL2","COL3") AS SELECT /*+ use_hash(c, tc) */[...]FROM SOME.TAB1_monat cJOIN SOME.TAB2 tcONtc.col42 = c.col23LEFT JOIN SOME.TAB3 vpONvp.col1 = c.col2

EXAMPLE: OVERRIDE STUPID HINTS

• SQL using some View with HASH JOIN hint

• but this SQL did not perform well with it

• NL would have been much better

• cannot replace view without looking at other SQL

• create new view without hint?

• my solution: set ignore_hints in session

• execute/generate plan and evolve to only accepted

EXAMPLE: OVERRIDE STUPID HINTS

alter session set "_optimizer_ignore_hints" = TRUE;

EXAMPLE: OVERRIDE STUPID HINTS

• export current RBO plans from 9i

• with outlines

• or import with RBO and compatible=9

• import into 11g or 12c database

• this guarantees compatibility

• but new CBO plans will get added

• can slowly review, accept, improve one-by-one

https://blogs.oracle.com/optimizer/entry/upgrading_from_9i_to_11g_and_the_implicit_migration_from_rbo

EXAMPLE: UPGRADE FROM BRO

• DBA_SQL_PLAN_BASELINES

• SQL_HANDLE != SQL_ID

• join via SIGNATURE

• PLAN_NAME != PLAN_HASH_VALUE

• plan is not stored in 11g (and may not reproduce)

• doesn’t help with parsing problems

• e.g. 12c dynamic stats

SOME ANNOYING THINGS

• export prod baselines to test/staging

• find new SQL in dev systems

• import those baselines before SQL goes to production

NEXT GEN IDEAS

• SPM is a very powerful tool

• easy to get started

• flexible to adopt to your needs

SUMMARY

thanks

Björn Rost