risk-free administration for expert database · pdf filecapacity planning using database...
TRANSCRIPT
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Risk-Free Administration for Expert Database Administrators
Kurt Engeleiter, Oracle Jason Lentz, Cerner Bart Gaddis, Cerner
Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• New Performance Management Tools
4 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• New Performance Management Tools
5 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Top Challenges
Database Management
Accurate capacity planning with 20-40% data store growth
40%
Source: From Database Clouds to Big Data: 2013 IOUG Survey On Database Manageability
?
78% Downtime resulting from unverified changes
48% Unpredictable application (SQL) performance
Key Takeaway:
Improve Quality of Service
6 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
DBA Actions That Affect Performance
• Adding or dropping indexes, partitioning tables, adding materialized views
• Gathering optimizer statistics
• Applying patches – one offs, PSUs, CPUs
• Modifying init.ora parameters
• Implementing new features such as Compression, Database In-Memory
• Version or patchset upgrades, e.g.
– 11g -> 12c
– 12.1.0.1 -> 12.1.0.2
7 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Environment Changes That Affect Performance
• Data growth
• User population growth
• Infrastructure changes
– Server upgrades
– Changes in size or type of disk storage
– Migration to new platforms including migration to Cloud
• Database Consolidation
8 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• New Performance Management Tools
9 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Database Replay
Concurrent Replay SPA
SPA Quick Check
• Proactively identifies impact of routine DBA tasks on database performance
Load and throughput testing solution
Designed for detecting and remediating throughout problems
• SQL response time (unit) testing solution
• Designed for identifying and tuning regressed SQLs
• Consolidation and capacity planning solution
Risk Free Performance Management Tools
Lower Risk
Automated Cut Costs
10 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• SPA Quick Check
• SQL Performance Analyzer
• Database Replay
• Concurrent Database Replay
• New Performance Management Tools
11 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
SPA Quick Check Overview • Helps users predict the impact of
specific system changes on SQL workload
• Input is SQL Tuning Set (STS) and a database configuration change
• Automatically executes two trials – pre and post change - and generates report
• Integrated with STS, SQL Plan Baselines, & SQL Tuning Advisor to form an end-to-end solution
12
SQL Plans + Stats
Analysis Report
Compare SQL Performance
Pre-change Trial Post-change Trial
SQL Workload (STS)
SQL Plans + Stats
Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 13
SPA Quick Check
• Optimized for use on prod systems
• Optimal Trial or Explain Plan mode
• Disable multi-executions, full DML execute disabled
Optimized
• Per SQL time limits
• Testing scoped to private session
• Associate with Resource Consumer Group
Controlled
• Context-aware change testing workflows, such as,
• Optimizer gather statistics
• Init.ora parameter changes
Change-Aware
SPA Quick Check
Production Database
Optimized Controlled/S
coped
Change-Aware
Optimal Trial
Mode, no DML
execute
Pre-selected
STS and
default SPA
settings
Per SQL Time Limits,
Limits testing scope to
private session
Context-aware
change testing
DBA
Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
SPA Quick Check Report
1
2
3 4
5
14 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Risk Free Changes with SPA Quick Check
• Statistics management
– Gather statistics in pending mode
– Use SPA Quick Check to determine whether to publish
• Init.ora parameter changes – Current parameter value is used for pre-change
– Select new value
– SPA Quick Check validates new parameter setting
• SQL Tuning Validation
– Validates SQL Profiles
15 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
SPA Quick Check Demo
16 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• SPA Quick Check
• SQL Performance Analyzer
• Database Replay
• Concurrent Database Replay
• New Performance Management Tools
17 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
SQL Performance Analyzer (SPA) Overview • Helps users predict the impact of any
system change on SQL workload
• User selects SQL Tuning Set
• User executes different SQL trials (experiments) of SQL statements performance by test execution or explain plan
• Analysis shows per SQL execution differences
• Integrated with STS, SQL Plan Baselines, & SQL Tuning Advisor to form an end-to-end solution
18 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Risk Free Changes with SPA
• Access structure changes:
– Indexes
– Materialized views
– Parallelism
• Patching
• Upgrades
• Implementing new features such as Compression or Database In-Memory
• Validating new hardware – servers or storage
19 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• SPA Quick Check
• SQL Performance Analyzer
• Database Replay
• Concurrent Database Replay
• New Performance Management Tools
20 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• Database load and performance testing with real production workloads
– Production workload characteristics such as timing, transaction dependency, think time, etc., fully maintained
• Test and measure transaction throughput improvements
• Identify application scalability and concurrency problems on test system before production deployment
• Usage scenarios: platform migration (e.g., Exadata), upgrades, patching, new database features or options, etc.
• 21
Database Replay
Test
Capture Process Replay Analysis & Reporting
Production Replay Driver Clients
Storage Storage
Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Risk Free Changes with Database Replay
• Patching
• Upgrades
• Implementing new features
• Validating new hardware – servers or storage
• Stress testing for capacity planning
22 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• SPA Quick Check
• SQL Performance Analyzer
• Database Replay
• Concurrent Database Replay
• New Performance Management Tools
23 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Capacity Planning: Motivation • User populations and data are continually growing
– How long can my environment handle the workload
• Organizations are pursuing database consolidation to deliver cost savings while increasing business agility
– How can we guarantee service levels between different workloads
– How can we test if one errant application is not affecting the others
• Consolidating workloads and databases has significant challenges
– Applications have different workload profiles – CPU – Memory – Storage – Network
– Will my Multitenant database handle peak workloads and co-exist?
– Is there enough headroom?
24 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• Capture workload on different databases that needs to co-exists concurrently
• Works for schema consolidation and Pluggable Databases
• Use for schema and CDB consolidation
• Identify and remediate inter-application scalability and concurrency problems
• Perform capacity planning through scale up, subsetting, scheduling (time-shifting) of multiple workloads
25
Concurrent DB Replay For Risk Free Consolidation and Capacity Planning
Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Capacity Planning Using Database Replay and Concurrent Database Replay
• Comprehensive scale-up support and what-if scenarios testing
• Scale-up techniques superior to traditional methods
– Zero-scripting approach extended for scale-up
– Scales data and user population
– Realistic data and bindsets
– Flexible, supports custom workload creation
• Capacity Planning Strategies
– Scale-up by scheduling concurrent replays
– Scale-up with multiple PDBs
– Scale-up by workload folding
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
ERP
DW
Replay Captured Workload Align Workload Peaks
CRM
Application Workloads
Capacity Planning Strategies: Peak Workload Testing – Time-Shifting
Peak Workload Testing Through Time-Shifting Exercise worst case scenario where workload peaks line up Evaluate and experience fallout from the safety of a test system
27 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Scale-up with Multiple Duplicate PDBs
Perform scale-up testing to identify possible host bottlenecks when deploying multiple instances of an application
Deploying a multitenant application—or adding a new geographical area to an existing application or line of business
Capacity Planning Strategies: Multiple PDBs
Replay Concurrent Workload Mapping
Test Database
PDB Clones
Sales Workload
28 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Capacity Planning Strategies: Workload Folding
Replay Concurrent Workload Subset
Subset 2
Subset 1
Workload Subsets Sales Workload
Scale-up Through Workload Folding
Perform scale-up testing by subsetting an existing workload capture into 2 & then perform concurrent replay of workload subsets
Suitable for workloads where transactions in subsets are mostly independent 29 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Concurrent Replay Capacity Planning Strategies Summary
Scale-up Strategy Workload Suitability
Concurrent Database Replay Enables consolidation validation
Can be used for all workloads
Multiple identical PDBs Simultaneous workload and data scale-up
Multitenant what-if scenarios
Workload Folding N * Workload scale-up
Suitable for relatively stateless workloads
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
• DBA Challenges
• Risk Free Administration Tools
• SPA Quick Check
• SQL Performance Analyzer
• Database Replay
• Concurrent Database Replay
• New Performance Management Tools
31 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Database Performance Diagnostics Tools
• Automatic Workload Repository – AWR
– Reports about performance and workload data from AWR
• Active Session History – ASH
– Gathers fine-grain data about every active database session every second
• Automatic Database Diagnostics Monitor - ADDM
– Data Analysis and Problem Identification
– Findings and Advise on how best to resolve bottlenecks
• Real-time SQL and Database Operations Monitoring
– Provides in-depth diagnostics about SQL execution at row source level
–
32
Database Performance Hub Provides Unified Monitoring Solution!
Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Database Performance Hub
33
Unified Performance Monitoring
• Single view of DB performance
ADDM, ASH analytics, Real-Time SQL Monitoring, SQL Tuning…
• Switch between ASH analytics, workload view, ADDM findings and SQL monitoring seamlessly
• Supports both real-time & historical mode
• Historical view of SQL Monitoring reports
• Dedicated tab for RAC
Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Performance Hub Report
• New interactive report for analyzing AWR data
• Performance Hub report generated from SQL*Plus
– @$ORACLE_HOME/rdbms/admin/perfhubrpt.sql
OR calling dbms_perf.report_perfhub(….) function
– Single view of DB performance
– ADDM, SQL Tuning, Real-Time SQL Monitoring, ASH Analytics
• Switch between ASH analytics, workload view, ADDM findings and SQL monitoring seamlessly
• Supports both real-time & historical mode
• Historical view of SQL Monitoring reports
34 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Perfhub Demo
35 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
New AWR “Active-HTML” Report
• New AWR report type “active-html” introduced in Oracle Database 12.1.0.2
• Provides best of HTML and Performance Hub Reports
– HTML report contains embedded Performance Hub Report as the last section
– ADDM task finds and recommendations are also presented
– Exadata-aware
• Highly recommended to use AWR “active-html” reports instead of HTML reports
– Combines power for EM navigation and drill down for offline analysis
– Can be saved and mailed like other Active Reports and does not need EM connectivity for viewing
36 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 37
AWR Report Vs Performance Hub Report
Oracle Confidential – Internal
Jason Lentz – Cerner Abilities Center
Bart Gaddis – Cerner Database Architecture
October 26, 2015
Customer Use Case Oracle Real Application Testing
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
40
Cerner today
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
41
Abilities Center – Cerner’s Wind Tunnel
Test Partners / Upgrade Center
CernerWorks
• Release + Tech Production Validation
• Controlled / Monitored Implementation
• Experienced IP Resources Available
IP Development Ability-Proven
Controlled Testing Environment
Hardware Technologies
iBus Platform
Java Services, RCP, Thin Client
Layered Software Technologies
Country Specific Solutions
Millennium Platform
Test Design
• Leverage clinical and domain expertise
• Automate client representative workflows
• Test with role and venue data complexity
Lights On Network
• Collects 7 billion client timers per month
• Built-in instrumentation allows lab/client comparison
• Enables focused enhancements in workflow coverage
Perform-ability 2 Seconds is Too Slow
High Availability 99.99% Uptime
Scalability Regional – National – Global
Affordability Reduce TCO
Reliability 1 Crash is Too Many
Test Measure Analyze
Correct
Quality Dedicated Testing Resources
• Isolated Network
• Secured Environment
• Experienced Associates
• IP Engineering Teams
• 30+ Millennium Domains
• Real Client Databases
• Automated Testing Tools
• SAN Storage
• AIX
• Citrix
• HPUX
• Linux
• MQ
• Oracle
• VMWare
• WebSphere
CAMM Platform
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
42
Why RAT?
• Capture and Replay client-like production workloads • Volume
• Data diversity
• Minimal effort to utilize
• Workload Scale-up and Folding • Test hardware at 10x current volume
• Stress storage and compute nodes
• Integrate/augment with existing workflow automation • Consume replay activity data in real-time
• Measure impact of replay load on automated workflows
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
43
Test Methodology
Controlled Approach
• Measure one change
• Same SGA
• Same replay parameters
Repeatable Replays
• Same number of user calls
• Low divergence
• Consistent duration
Actionable Data
• Confidence in metrics
• Process/parameter changes
• Purchase decisions
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
44
Workload Capture and Replay Procedure
• Two hour capture during peak volume on live client production system
• Database duplicate and recovery to capture SCN (Full copy)
• Database upgrade to 11.2.0.4
• SQL regression tuning
• Preprocess and tune replay clients
• Repeatable Workload Replays • Flashback to guaranteed restore point
• Cache warming - Consolidated replay of entire workload (read only)
• Read-write workload replay
• Standardized data and report collection (Replay, AWR, OS)
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
45
Test Methodology
Controlled Approach
• Measure one change
• Same SGA
• Same replay parameters
Repeatable Replays
• Same number of user calls
• Low divergence
• Consistent duration
Actionable Data
• Confidence in metrics
• Process/parameter changes
• Purchase decisions
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
46
Database Replay Analysis – SQL Perspective
SQL Tuning Sets, primary tool
• Capture SQL during baseline replay and subsequent replays
• SPA (SQL Performance Analyzer) has built in reporting using SQL Tuning Sets
• Find SQL which has regressed or improved
• Find SQL with plan changes
• Relate the change to the impact on workload
• STS comparison - Example Report
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
47
SQL Tuning Sets – Use Cases
Optimizer Related Changes • DBMS_STATS sample size, impact on execution plan changes
• DBMS_STATS histogram methodology (frequency histograms with autosample)
• Optimizer patches, impact on execution plan changes
• Optimizer parameters, impact on execution plan changes
• Discovery of bug related to ORA-7445 on hard parse due to corrupt histogram.
Structural Changes • Validation of Cluster Tables, impact on SQL execution statistics
• Investigation/evaluation of plan changes associated to Cluster Tables
Key Views • DBA_SQLSET_STATEMENTS, DBA_SQLSET_PLANS, DBA_SQLSET_BINDS
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
48
Test Methodology
Controlled Approach
• Measure one change
• Same SGA
• Same replay parameters
Repeatable Replays
• Same number of user calls
• Low divergence
• Consistent duration
Actionable Data
• Confidence in metrics
• Process/parameter changes
• Purchase decisions
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
49
Major Replay Use Cases
Vertical scaling (Server hardware upgrade)
Horizontal scaling (Add RAC instances)
OS tuning (OS version, parameter change)
Notable Results
• IP Fragmentation Thresholds (IPFRAG)
• RESULTS: Increasing all thresholds
decreased packet reassembles and failures
and improved global cache metrics while
remaining within hardware capabilities.
• DETERMINATION: Safe and effective to roll
out to clients.
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
50
Major Replay Use Cases
Vertical scaling (Server hardware upgrade)
Horizontal scaling (Add RAC instances)
OS tuning (OS version, parameter change)
Notable Results
• Migrate to new hosts with more CPU and RAM
• RESULTS: More efficient reads and gets
increased user/application activity while
decreasing waits (concurrency, cluster,
commit).
• DETERMINATION: Improved performance
and scaling point. Benefits outweigh cost.
BRNDEXP 2.1 0714 © 2014 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.
51
Major Replay Use Cases
Vertical scaling (Server hardware upgrade)
Horizontal scaling (Add RAC instances)
OS tuning (OS version, parameter change)
Notable Results
• Add RAC instances, up to 6 wide
• RESULTS: Resource utilization gradually
decreased with every instance added to
cluster.
• DETERMINATION: Adding additional
hardware is an effective solution for
increasing capacity.
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 53 Oracle Confidential – Internal
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 54 Oracle Confidential – Internal