multiple awr reports_parser
TRANSCRIPT
2015© Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
2015 © Trivadis
Multiple AWR Reports Parser
Jacques Kostic
Senior Consultant IMS
7-mar-2015
TechEvent: Multiple AWR Reports Parser
1
2015© Trivadis
AGENDA
1. Customer overview
2. Architecture overview
3. Why such tool?
4. Reports parsing phase
5. Reports analysis phase
6. Conclusions
7. Q&A
7-mar-2015
TechEvent: Multiple AWR Reports Parser
2
2015© Trivadis
7-mar-2015
TechEvent: Multiple AWR Reports Parser
Customer Overview
3
2015© Trivadis
Customer Overview
The name will not be disclosed but the most relevant
characteristics to the project are reported below.
Server and database names are fictive.
Medium size customer of the Public sector.
Data Warehouse environment.
More than 30 databases involved with various sizes up to 900 GB.
Huge amount of processing involving many business entities.
Highly consolidated Oracle infrastructure.
Strong growing factor.
Constant criticality increase.
7-mar-2015
TechEvent: Multiple AWR Reports Parser
4
Customer
Environment
2015© Trivadis
Customer Overview
Goal of the project:
Identify factors that cause important performance deviations during
Data Warehouse load processing
Propose solutions to resolve the issues if identified
Propose cost effective solutions to efficiently absorb load
processing and satisfy new business needs
Deliverables:
Evidences and explanations of deviations during load processing
Possible improvement areas
7-mar-2015
TechEvent: Multiple AWR Reports Parser
5
2015© Trivadis
7-mar-2015
TechEvent: Multiple AWR Reports Parser
Architecture
overview
6
2015© Trivadis
Architecture overview
SAN: VSP Hitachi with Dynamic and Thin Provisioning
Production: OS, Solaris 1 CPU X5460 4 cores, RAM, 64GB
2 * FC 4Gb, 4 * NIC Gb
ZFS file systems in the ZPOOL
9 databases : 11.2.0.3
QAS/DEV: OS, Solaris 1 CPU X5460 4 cores, RAM, 64GB
2 * FC 4Gb, 4 * NIC Gb
ZFS file systems in the ZPOOL
13 databases : 11.2.0.3
VMs: a couple of VMs under VMWare that host some Talend and
Cognos repositories
7-mar-2015
TechEvent: Multiple AWR Reports Parser
7
2015© Trivadis
7-mar-2015
TechEvent: Multiple AWR Reports Parser
Why such tool?
8
2015© Trivadis
Why such tool?
The idea came to me during the kickoff meeting when I realize that I
would not have the expected, level of support, access and time to do the
work.
Key constraints :
Limited internal DBA support
Limited internal BI team support
No access to the system
No OEM access
Requires complete analysis of two consecutive weeks (last and first of next
month)
Very short time to run the project (7 days)
The outcome must focus on key factors with relevant and proven facts.
7-mar-2015
TechEvent: Multiple AWR Reports Parser
9
2015© Trivadis
Why such tool?
It has been agreed that:
System team would send me all graphs for servers and storage.
DBA would send all AWR reports for all databases for the two weeks.
A script was provided to DBAs for reports generation
With 30 minutes AWR window basis, the total number of reports is:
24*2*7*2= 672 reports per database
672 * 22 databases = 14’784 reports!
Solution?
Parse all reports and extract what is relevant to have the full trend and
being able to highlight bottlenecks
7-mar-2015
TechEvent: Multiple AWR Reports Parser
10
2015© Trivadis
7-mar-2015
TechEvent: Multiple AWR Reports Parser
Reports parsing phase
11
2015© Trivadis
Reports parsing phase
During the development phase, my college @ludodba told me that
@flashdba has already build something similar. As I almost completed the
script and some other ideas where coming, I decided to continue spending
some time on the utility. Finally our versions are differing somehow…
Here are some key points and features (at the moment):
Works fine starting from 11g and 12c ready
Most relevant data are captured
Sorting of the output file on the snapshot time
Adjust window time to a given period
Generate cross references for multiple databases impact analysis
Generate events summary of TOP 6 foreground events of all reports
Fast (336 reports parsed in less than 15 seconds)
7-mar-2015
TechEvent: Multiple AWR Reports Parser
12
The tool
2015© Trivadis
Reports parsing phase
7-mar-2015
TechEvent: Multiple AWR Reports Parser
13
Extract information from reports
$> tvd_perl parse_awr.pl -o <output folder> -s <source folder
containing the reports> -t <TimeWindow (30,60,... minutes)>)
$> tvd_perl parse_awr.pl -o Outputs -s WeekOne/prod/dbPRD01 -t 30
Parsing file WeekOne\prod\dbPRD01/awrrpt_1_15454_15455_20141127_08_30_20141127_09_00.txt
Parsing file WeekOne\prod\dbPRD01/awrrpt_1_15390_15391_20141126_00_30_20141126_01_00.txt
Parsing file WeekOne\prod\dbPRD01/awrrpt_1_15411_15412_20141126_11_00_20141126_11_30.txt
…
Parsing file WeekOne\prod\dbPRD01/awrrpt_1_15368_15369_20141125_13_30_20141125_14_00.txt
Parsing file WeekOne\prod\dbPRD01/awrrpt_1_15347_15348_20141125_03_00_20141125_03_30.txt
Parsing file WeekOne\prod\dbPRD01/awrrpt_1_15415_15416_20141126_13_00_20141126_13_30.txt
Processed 336 files
Saving result...
Writing header...
Sorting by snapshot date/time...
Writing data...
Done!
2015© Trivadis
Reports parsing phase
7-mar-2015
TechEvent: Multiple AWR Reports Parser
14
Generate cross references
$> tvd_perl consolidate_awr.pl [-p <filename>] -s <Sourcefolder> -c
<col_number> -o <OutputFile>
$> tvd_perl consolidate_awr.pl -p srvlPRD_dbPRD01.csv
Column:0 -> DB -> String
Column:1 -> INST_NAME -> String
Column:2 -> INST_NUM -> Numeric
Column:3 -> Host -> String
Column:4 -> FileName -> String
Column:5 -> CPUs -> Numeric
Column:6 -> DtSort -> Numeric
Column:7 -> BSnap -> String
Column:8 -> ESnap -> String
Column:9 -> ElapsTime -> Numeric
Column:10 -> DBTime -> Numeric
…
Column:101 -> FGEvent6 -> String
Column:102 -> FGWaits6 -> Numeric
Column:103 -> FGTOut6 -> Numeric
Column:104 -> FGTime6 -> Numeric
Column:105 -> FGEAvg6 -> Numeric
Column:106 -> FGPctDb6 -> Numeric
2015© Trivadis
Reports parsing phase
7-mar-2015
TechEvent: Multiple AWR Reports Parser
15
Generate cross references
$>tvd_perl consolidate_awr.pl -s . -c 10 -o output/dbtime.csv
Collecting columns infos from file srvlPRD_dbPRD01.csv
Processing file srvlPRD_dbPRD01.csv
Collecting columns infos from file srvlPRD_dbPRD02.csv
Processing file srvlPRD_dbPRD02.csv
Collecting columns infos from file srvlPRD_dbPRD03.csv
Processing file srvlPRD_dbPRD03.csv
Collecting columns infos from file srvlPRD_dbPRD04.csv
Processing file srvlPRD_dbPRD04.csv
Writing result in output/dbtime.csv
Done!
dbtime.csv
…Begin Time dbPRD01-DBTime dbPRD02-DBTime dbPRD03-DBTime dbPRD04-DBTime
24/11/2014 00:00 0,29 0,06 0,14 0,34
24/11/2014 00:30 0,22 0,18 0,95 0,46
24/11/2014 01:00 0,11 0,06 0,16 0,28
24/11/2014 01:30 0,24 0,18 0,94 0,33
24/11/2014 02:00 0,1 0,06 0,16 0,3
24/11/2014 02:30 0,22 0,21 1,48 0,28
…
2015© Trivadis
Reports parsing phase
7-mar-2015
TechEvent: Multiple AWR Reports Parser
16
Generate events summary
$> tvd_perl event_rpt.pl -s <Sourcefile> -o <OutputFile>
$> tvd_perl event_rpt.pl -s srvlPRD_dbPRD01.csv –o Events_dbPRD01.csv
Processing file srvlPRD_dbPRD01.csv
Writing result in Events_dbPRD01.csv
Summary of top 6 foreground events
Done!
Event Name Count Time (Min) Min (ms) Max (ms) Avg (ms)Times in
top 6% Occ % Time
db file sequential read 43’629’296 132 0 121 0.000182 224 11.1 25.7
SQL*Net message from client 5’227 126 1’457 1’457 1.457050 1 0.04 24.5
read by other session 3’677’004 93 1 19 0.001533 11 0.54 18.2
…
direct path write temp 18’595 2 4 59 0.009572 13 0.64 0.5
control file parallel write 22’169 2 3 36 0.007308 169 8.38 0.5
library cache: mutex X 8’717 2 5 139 0.019732 51 2.52 0.5
PX Deq: Slave Session Stat 23’319 2 5 8 0.005489 8 0.39 0.4
Events_dbPRD01.csv
2015© Trivadis
Reports parsing phase
All reports consolidation
One file per database and per week
Cross references
Db_time
Db_cpu
Redo size/s
Logical reads/s
Physical reads/s
Physical writes/s
Block changes/s
Host statistics
…
Event summary for each database on top 6 foreground
7-mar-2015
TechEvent: Multiple AWR Reports Parser
17
Extracted data for analysis
2015© Trivadis
7-mar-2015
TechEvent: Multiple AWR Reports Parser
Reports analysis phase
18
2015© Trivadis
Reports analysis phase
Using db_time
For trend analysis we keep only DBPRD2, DBPRD3, DBPRD4 and DBPRD5
7-mar-2015
TechEvent: Multiple AWR Reports Parser
19
Identify significant activity
2015© Trivadis
Reports analysis phase
In green, CPU graphs given by system team, in blue run queue extracted from
AWR reports analysis. We can clearly observe a perfect correlation.
7-mar-2015
TechEvent: Multiple AWR Reports Parser
20
CPU ConsumptionFrom 24 au 30 Nov
From 1 au 7 Dec
2015© Trivadis
Reports analysis phase
In green, CPU graphs given by system team, in blue run queue extracted from
AWR reports analysis. We can clearly observe a perfect correlation.
7-mar-2015
TechEvent: Multiple AWR Reports Parser
21
CPU ConsumptionFrom 24 au 30 Nov
From 1 au 7 Dec
2015© Trivadis
Headline
Text Text
Text
- Text
Text
Text
Text Text
Text
- Text
Text
Text
Text Text
Text
- Text
Text
Text
7-mar-2015
TechEvent: Multiple AWR Reports Parser
22
2015© Trivadis
Headline
Text Text
Text Text Text Text
Text Text
Text Text Text Text
SQL > EXEC DBMS_STATS.GATHER_SCHEMA_STATS
> ownname => user
> estimate_percent. => DBMS_STATS.AUTO_SAMPLE_SIZE)
7-mar-2015
TechEvent: Multiple AWR Reports Parser
23
2015© Trivadis
Headline
Text Text Text Text Text
Text Text Text Text Text
Text Text Text Text Text
Text Text Text Text Text
Text Text Text Text Text
Text Text Text Text Text
Text Text Text Text Text
Text Text Text Text Text
Text Text Text Text Text
Text Text
7-mar-2015
TechEvent: Multiple AWR Reports Parser
24
2015© Trivadis
Questions and answers ...
2015 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
Jacques Kostic
Senior Consultant IMS
7-mar-2015
TechEvent: Multiple AWR Reports Parser
25