beyond monitoring - d1o3t0kvuxq0hm.cloudfront.net · beyond monitoring delete & insert headshot...
TRANSCRIPT
1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Performance Prediction
Using RETIT & Dynatrace App Mon
Beyond Monitoring
Delete &
Insert
Headshot
here
Delete &
Insert
Headshot
here
2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE2
Agenda
Performance Prediction
Use cases, Benefits and examples
Demo
3 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE3
Performance prediction
Take your Continuous delivery to the next level.
4 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Performance Modeling - Why?
A new version of your application is on the way:
• … review realtime application performance.
• … evaluate realtime performance metrics.
• … deep dive into single components of your environment.
Enterprise ApplicationHardwareSoftware
Measurements Measurement Data
6 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Performance Modeling - Why?
Capacity
Planning
Performance
Analysis
Enterprise ApplicationHardwareSoftware
Measurement Measurement Data
Capacity
Management
What happens if…
• … you change yur deployment topology?
• … you migrate to a different hardware environment?
• … the workload changes?
• … you reduce the number of CPU cores?
7 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Performance Modeling - What?
Performance
Model
Resource
Demand
Software
Architecture
Hardware
Environment
Workload
Simulation
Response Time
Throughput
Resource
Utilization
8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
A Brief History of Performance Modeling
Queuing Theory(Erlang, ~1906)
Queuing Networks (Jackson, ~1963)
LayeredQueuing Networks (Neilson / Woodside, ~1995)
Architecture-Level Performance Models (Reussner / Becker ~2009)
w
t
S
λ
Arriving
Jobs
Departing
Jobs
Queue Service
Station
1
4
CPU
HDD
Application Server
Arriving
Transactions
Completed
Transactions
HTTPRequest
[Z=2 s]
User
{m=1200}
WebService
[s=15 ms]
HTTP Server
Thread
{m=50}
Purchase
[s=80 ms]
AppServer
Thread
{m=25}
Browse
[s=20 ms]
(y=1)
(y=0.6) (y=0.4)
User Device
{m=1200}
User Device
{m=1200}
HTTP Server
Machine
HTTP Server
Machine
AppServer
Machine
{m=2}
AppServer
Machine
{m=2}
Repository Model
Resource Environment
System Model
Allocation Model
Usage Model
Pa
llad
io C
om
po
nen
tM
od
el (P
CM
)
Foundation for tools such
as CA Capacity Manager
(was: Hyperformix) or
Sumarian Capacity Planner
Foundation for tools such
as Performance
Assurance‘s ePASA
Foundation for
RETIT solutions
9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Performance Modeling – How?
Knowledge
Generation
Capacity
Planning
Performance
Analysis
Enterprise ApplicationHardwareSoftware
Performance
Model Repository
Measurement
Modeling
Simulation
Measurement Data
Model Parameter
Simulation Results
Capacity
Management
10 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Performance Modeling – When?
OpsDev
Implementation
Operations
1 4.5 10 27.5 50
500
0
100
200
300
400
500
600
Rel
aive
cost
fact
or
(mea
n)*
Fix performance-related bugs earlier
and for less costs by managing
performance knowledge using models.
Source: http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf
*The cost factor for fixing a performance-related problem is normalized in
the different phases relative to the cost of fixing a defect in the requirements phase.
Extrapolate your (load/performance) test
results and reduce labor, machinery and
license costs by reducing the amount tests.
Detect performance change in every version
created in a continuous delivery pipeline without
the need for expensive performance tests.
Provide models along with your
application binaries to simplify
capacity planning activities.
Right-size your environment for seasonal peaks or to
reduce license / operating cost (e.g., reduce the
amount of cloud instances).
11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Dynatrace-Compatible RETIT Solutions
• RETIT Capacity Manager (RCM)
• Performance Modeling Environment
• Connect to any of your Dynatrace deployments to
generate performance models
• “What-If” scenario simulations:
• Hardware changes
• Workload changes
• Software architectural changes
12 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Dynatrace-Compatible RETIT Solutions
• RETIT Continuous Delivery (RCD)
• Plugin for the Jenkins CI server
• Detects performance changes automatically
• Uses Dynatrace data collected in previous test phases
(e.g., acceptance tests) to generate models
• Can evaluate performance for multiple hardware
environments and workloads
13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE13
Agenda
Performance Modeling
Use cases, Benefits and examples
Demo
14 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Use Cases – Design Time Performance Evaluations
Implementation
Operations
Existing
Component 1 Existing
Component …
Existing
Component N
New
Application
Design
Performance-
relevant aspects
known and depicted
in performance models
Structure and
behaviour
defined in design
models
e.g., minimum
response time –
even you assume
that a new
application
consumes no
time
15 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Benefits – Design Time Performance Evaluations
1 4.5 1027.5
50
500
0
100
200
300
400
500
600
Requirements Design Implementation Testing Deployment Operations
Rel
ativ
e co
stfa
cto
r(m
ean
)*
Fix performance problems in your software architecture earlier with a lot less cost than during test, deployment or operations!
Source: http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf
*The cost factor for fixing a performance-related problem is normalized in the
different phases relative to the cost of fixing a defect in the requirements phase.
Implementation
Operations
16 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Real Example - Design Time Perf. Evaluations
• Evaluating Performance in a Service-oriented Architecture (SOA)
Implementation
Operations
• Service-Consumers:
• Process-oriented user interfaces
• Orchestrated by a BPM Engine
• Service-Providers:
• Common data sources and application services
Enterprise Service Bus (ESB)
Business
Dev
Ops
17 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Real Example - Design Time Perf. Evaluations
• Evaluating Performance in a Service-oriented Architecture (SOA)
Implementation
Operations
• Service-Consumers:
• Process-oriented user interfaces
• Orchestrated by a BPM Engine
• Service-Providers:
• Common data sources and application services
• What happens if we introduce/automate new business processes?• Can you achieve the required response time and business process lead
time goals?• How much does it cost to increase the IT system performance to improve
the business process lead times?
Business
Dev
Ops
18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Real Example - Design Time Perf. Evaluations
• Evaluating Performance in a Service-oriented Architecture (SOA)
Implementation
Operations
• Service-Consumers:
• Process-oriented user interfaces
• Orchestrated by a BPM Engine
• Service-Providers:
• Common data sources and application services
Business
Dev
Ops
• Can we achieve the desired performance and process lead
time goals using our existing service-level agreements?
• Which service needs to improve most in order to achieve the
business goals?
• Should we negotiate new SLAs with multiple service proviers
and ask them to improve their performance slidely or with a
few providers but as for more radical changes?
19 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Real Example - Design Time Perf. Evaluations
• Evaluating Performance in a Service-oriented Architecture (SOA)
Implementation
Operations
• Service-Consumers:
• Process-oriented user interfaces
• Orchestrated by a BPM Engine
• Service-Providers:
• Common data sources and application services
• How much additional load will new business processes generate? • Will the existing systems be able to handle the load?• Do we need to increase our capacity?• Who pays for the additional capacity?
Business
Dev
Ops
20 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Real Example - Design Time Perf. Evaluations
• Evaluating Performance in a Service-oriented Architecture (SOA)
Implementation
Operations
• Service-Consumers:
• Process-oriented user interfaces
• Orchestrated by a BPM Engine
• Service-Providers:
• Common data sources and application services
Business
Dev
Ops
APM Data
APM Data
Business
process
descriptions
Software
designs for
new processes
21 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Real Example - Design Time Perf. Evaluations
• Evaluating Performance in a Service-oriented Architecture (SOA)
Implementation
Operations
• Service-Consumers:
• Process-oriented user interfaces
• Orchestrated by a BPM Engine
• Service-Providers:
• Common data sources and application services
Business
Dev
Ops
• You should talk toservice provider XYZ!
• Your workload will increase by X!• You should buy … new servers!
• You can achieve the desiredbusiness process lead times but itwill cost you …!
22 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Real Example - Design Time Perf. Evaluations
Modeling improves the collaboration of all parties involved in thesoftware lifecycle (Business, Dev and Ops)!
Business has a level of granularity (business processes) which easesthe communication with the IT department
Service consumers (Dev) can better estimate the expected responsetimes for new business processes
Service providers (Ops) have early access to workloadinformation when new business processes are released
Implementation
Operations
23 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Use Cases – Model-based Evaluations in CD
Implementation
Operations
Evaluate the performance impact of feature additions and bugfixes
For multiple hardware environments and workloads
Without the need to own corresponding test systems!
Deployment Pipeline in a
Continuous Delivery Process
Model-based Performance
Change Detection
Commit
Stage
Automated
Acceptance
Test
Manual
TestingRelease
Developer
checks in
Notify Developer about Performance Change
24 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Benefits – Model-based Evaluations in CD
Implementation
Operations
Ensure that no version gets released with performance regressions
Leverage cost-benefits of fixing performance problems early in thedevelopment process
Increase the performance awareness of developers by immediate feedback on check-ins
Avoid the need to setup and prepare load/performance testenvironments for each and every project
Leverage your existing acceptance/regression testinginvestments for performance evaluations
25 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• A customer (software vendor) has 5 major enterprise applications (EA):
• 1 CI system per EA (Jenkins)
• 5 customer scenarios (workload/hardware environment combinations) should be tested for each new build
• A small performance test environment costs 10 k € / year
• 250.000 € / year
• RETIT Continuous Delivery (RCD) can realise this scenario withone license per EA!
• Saves the huge investment in test environments
Real Example – Model-based Evaluations in CD
Implementation
Operations
26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Use Cases – Integrating Load Tests w/ Predictions
Implementation
Operations
Setup testenvironment
Define testscenarios
Execute testsAnalyzeresults
Enterprise Applications HardwareSoftware
Measurement Measurement Data
0,00 ms
20,00 ms
40,00 ms
60,00 ms
80,00 ms
100,00 ms
120,00 ms
140,00 ms
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
600 Users 800 Users 1000 Users
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9
Diagrammtitel
Throughput
Response
Time
Resource
Utilization
0,00 ms
20,00 ms
40,00 ms
60,00 ms
80,00 ms
100,00 ms
120,00 ms
140,00 ms
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
600 Users 800 Users 1000 Users
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9
Diagrammtitel
Throughput
Response
Time
Resource
Utilization
0,00 ms
20,00 ms
40,00 ms
60,00 ms
80,00 ms
100,00 ms
120,00 ms
140,00 ms
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
600 Users 800 Users 1000 Users
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9
Diagrammtitel
Throughput
Response
Time
Resource
Utilization
Generate model and
predict performance for
multiple scenarios
27 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Benefits – Integrating Load Tests w/ Predictions
Save costs by reducing the amount of load/performance tests
Increase the coverage of your tests
Evaluate scenarios without buying the corresponding hardware
Easily grow the coverage as the deployment count of yourapplication increases
Implementation
Operations
28 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Save costs by reducing the amount of load/performancetests• Effort for load tests (real customer example, incl. script development, test setup,
execution and result analysis):
• Small scale: 13 person days (PD), medium scale 23 PD, large scale: 41 PD
• Replacing one medium or large scale test by a small scale test with predictionssaves between 8 and 26 PD (assuming 2 PD for predictions)
Real Example – Integrating LT with Predictions
Implementation
Operations
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Savi
ngs
in
per
son
day
s(P
D)
Number of performance tests
PD savings when replacing large scale tests withsmall scale tests and predictions
PD savings when replacing medium scale testswith small scale tests and predictions
29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Use Cases – Model-based Capacity Management
Implementation
Operations
Enterprise ApplicationHardwareSoftware
Measurement Measurement Data
0,00 ms
20,00 ms
40,00 ms
60,00 ms
80,00 ms
100,00 ms
120,00 ms
140,00 ms
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
600 Users 800 Users 1000 Users
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9
Diagrammtitel
Throughput
Response
Time
Resource
Utilization
0,00 ms
20,00 ms
40,00 ms
60,00 ms
80,00 ms
100,00 ms
120,00 ms
140,00 ms
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
600 Users 800 Users 1000 Users
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9
Diagrammtitel
Throughput
Response
Time
Resource
Utilization
0,00 ms
20,00 ms
40,00 ms
60,00 ms
80,00 ms
100,00 ms
120,00 ms
140,00 ms
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
B MRT
B SRT
M MRT
M SRT
P MRT
P SRT
600 Users 800 Users 1000 Users
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9
Diagrammtitel
Throughput
Response
Time
Resource
Utilization
Generate model based on
production data and predict
performance for future scenarios.
What happens if we
migrate to AWS?
How much capacity is required if
we have ten-times more users
during Christmas season?
What happens if we change
our deployment topology?
30 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Right-size your environments to pay only for what you really need
Avoid the need to setup expensive test environments to evaluatechanges
Reduce risk for hardware environment (e.g., cloud) migrations
Reduce the time for capacity management activities
Increased accuracy as the simulations avoid the need forlinear assumptions
Benefits – Model-based Capacity Management
Implementation
Operations
31 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
12:00 12:15 12:30 12:45 13:00 13:15 13:30 13:45 14:00 14:15 14:30 14:45
€/MWh
time
EPEX SPOT INTRADAY AUCTION - 2014-12-11
pi
Smart Grid Capacity Planing for several millionhouseholds
Real Example – Model-based CM
maximum 15 min
Implementation
Operations
32 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE32
Agenda
Performance Modeling
Use cases, Benefits and examples
Demo
33 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
RETIT Solution Demo
• Demo
34 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• What did we learn today?
• Performance modeling…
Summary
Improves cross-
team collaboration
…by integrating multiple
data sources
Increases performance
awareness
… through immediate feedback
during development.
Extends test
coverage
… by allowing you to test more
workloads and hardware
environments.
Saves cost
… by taking the guess-work
out of capacity planning
activities.
36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Backup Evaluation Results
37 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Beispiel – Erhebung der Daten
IBM System X3755M3
4 AMD Opteron 6172
12 cores and 2,1 GHz
256 GB RAM
VMWare ESXi 5.1.0
Benchmark Driver
Faban Harness
16 Virtual CPUs
8 GB RAM
openSuse 12.3 64bit
UI Server
GlassFish AS 4.0
2 CPUs Enabled
10 GB Heap
openSuse 12.2 64bit
Dynatrace / RETIT APM
WS Server
JBoss 7.1.1
6 CPUs Enabled
6 GB Heap
openSuse 12.2 64bit
Dynatrace / RETIT APM
DB Server
PostgreSQL 9.2.7
4 CPUs Enabled
96 GB RAM
openSuse 12.3 64bit
SAR
IBM System X3755M3
4 AMD Opteron 6172
12 cores and 2,1 GHz
256 GB RAM
IBM System X3755M3
4 AMD Opteron 6172
12 cores and 2,1 GHz
256 GB RAM
IBM System X3550
2 Intel Xeon
6 cores and 2,4 GHz
96 GB RAM
38 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Beispiel - Ergebnisse
• Measurement
• Simulations using data collected by dynatrace + SAR
• Simulations using data collected by RETIT APM + SAR
IBM System X3755M3
4 AMD Opteron 6172
12 cores and 2,1 GHz
256 GB RAM
VMWare ESXi 5.1.0
Benchmark Driver
Faban Harness
16 Virtual CPUs
8 GB RAM
openSuse 12.3 64bit
39 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Beispiel - Ergebnisse
UI Server
GlassFish AS 4.0
2 CPU Enabled
10 GB Heap
openSuse 12.2 64bit
WS Server
JBoss 7.1.1
6 CPU Enabled
6 GB Heap
openSuse 12.2 64bit
DB Server
PostgreSQL 9.2.7
4 CPU Enabled
96 GB RAM
openSuse 12.3 64bit
SAR
IBM System X3755M3
4 AMD Opteron 6172
12 cores and 2,1 GHz
256 GB RAM
IBM System X3755M3
4 AMD Opteron 6172
12 cores and 2,1 GHz
256 GB RAM
IBM System X3550
2 Intel Xeon
6 cores and 2,4 GHz
96 GB RAM
RETIT APM RETIT APM
40 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Ergebnisse -> VM zu realer Hardware
Modellgenerierung mit RCM & RETIT APM
41 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Ergebnisse -> VM zu realer Hardware
Modellgenerierung mit RCM & RETIT APM
42 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Änderungen des Nutzerverhaltens:
• Interarrival Anpassen
• Variation der Transaktionsverteilung
• Neue Usage Szenarien anlegen
• z.B. Closed Workloads
Modellgenerierung mit RCM & RETIT APM
43 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Variation der Nutzer, ihrer Verteilung und der Think Time / Arrival Rate
Modellgenerierung mit RCM & RETIT APM
44 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Variation der Nutzer, ihrer Verteilung und der Think Time / Arrival Rate
Modellgenerierung mit RCM & RETIT APM
45 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Beispiele – AMD Opteron 16 Cores -> Meas. Vs. Simulation (SPECjEnterprise2010 – Java EE 5.0 Benchmark)
Höhere Skalierungen Testen
46 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• Beispiele – Intel Xeon 8 Cores (adapted from AMD) -> Meas. Vs. Simulation (SPECjEnterprise2010 –Java EE 5.0 Benchmark)
Höhere Skalierungen Testen
47 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
48 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
49 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE49
50 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE50
51 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE51
52 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Persona Icons – For your use
Recolor these icons as
needed to match
Dynatrace brand
colors. PictureTools/Format/Color
Customer Conversion
Fanatic
for
Digital Business
Owners
Deployment-obsessed
Releasearaptor
for
Development
War Room
Peacemaker
for
Operations
53 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE