performance testing and apm - ncc group · @devopsguys devopsguys devopsguys the benefits of using...
TRANSCRIPT
DevOpsGuys @DevOpsGuys www.devopsguys.com
DevOpsGuys The benefits of using an
APM solution while performance testing
Why Perf testing without APM is like Corona without the Lime…
1
DevOpsGuys @DevOpsGuys www.devopsguys.com
Perf Testing & APM in 30mins
• Why Performance Test?
• What is APM anyway?
• Benefits of APM when Perf Testing
• Q&A
3
DevOpsGuys @DevOpsGuys www.devopsguys.com
Why Performance Test?
1. Ensure that we meet our customer’s
expectations of website response time
– So that they convert and we make £££
2. Ensure that we have sufficient
computing capacity to meet #1 under
load (load testing)
3. Understand how the system reacts when
things go wrong (stress testing)
4
DevOpsGuys @DevOpsGuys www.devopsguys.com
APM Defined
“In the fields of information technology and
systems management, application performance
management (APM), is the monitoring and
managing of performance and availability of
software applications. APM strives to detect
and diagnose application performance
problems to maintain an expected level of
service. APM is the translation of IT metrics into
business meaning (i.e. value).”
6
http://en.wikipedia.org/wiki/Application_Performance_Management
DevOpsGuys @DevOpsGuys www.devopsguys.com
“5 Dimensions of APM*”
1. End user experience monitoring - (Active and passive)
2. Application runtime architecture discovery and modeling
3. User-defined transaction profiling (also called business transaction management)
4. Application component monitoring
5. Reporting & Application data analytics
7
*According to Gartner
DevOpsGuys @DevOpsGuys www.devopsguys.com
CA
Quest HP
OpTier
Opnet
IBM
Compuware
Oracle
Microsoft
SL
Inetco
NetScout
eG Innovations
Progress
ASG
Idera
Visual networks
Info vista
Network Instruments
Knoa
Aternity
Coradient
Correlsense Nastel
Precise
Arctures Technologies
BM
C
Managed Engine
New Relic
AppDynamics
2011 Riverbed
OpTier
2012
AppNeta
Dell
2013
Gartner APM Magic Quadrant
DevOpsGuys @DevOpsGuys www.devopsguys.com
BENEFITS OF LOAD TESTING WITH APM TOOLS
Never leave home without it…
9
DevOpsGuys @DevOpsGuys www.devopsguys.com
Systems Thinking 101
“Systems thinking is a framework for seeing
interrelationships rather than things, for
seeing patterns rather than static snapshots.
It is a set of general principles spanning
fields as diverse as physical and social
sciences, engineering and management.” -
Peter Senge
11
DevOpsGuys @DevOpsGuys www.devopsguys.com
Command & Control vs Systems Thinking
12 http://www.systemsthinking.co.uk/2-1.asp
DevOpsGuys @DevOpsGuys www.devopsguys.com
The Scenario
• E-commerce customer
• New website
• High data-start times on page load
• Poor load testing results
• Initial conversation with client
• 2 week delay while hosting provider scaled out the hardware…
• Problem still not fixed…
13
DevOpsGuys @DevOpsGuys www.devopsguys.com 14
What’s the root cause of the ~1sec Data Start?
Waterfall Diagram of page load times
DevOpsGuys @DevOpsGuys www.devopsguys.com
CASE STUDY #1 56X FASTER IN A DAY…
9am to 9pm. 12hrs. 56x faster!
20
DevOpsGuys @DevOpsGuys www.devopsguys.com
The Scenario
• New website pre-launch testing
• Customer Portal (SaaS solution)
• Start Load testing 9am
• 4 “Test & Fix” cycles in the day
• Final testing completed at 9pm
21
DevOpsGuys @DevOpsGuys www.devopsguys.com
The End Results = 56x Average
Response
Throughput
(Req/sec)
Initial Testing 3024ms 16/min
Final Result 54ms 59/min
56x faster 3.5x throughput
22
Before
3024ms After
54ms
DevOpsGuys @DevOpsGuys www.devopsguys.com
#4
Team Work
“The Enemy is Performance,
not each other…”
24
DevOpsGuys @DevOpsGuys www.devopsguys.com
Summary
1. See the Big Picture (systems thinking)
2. Drill down to the details
3. Faster Iteration = Better Value
4. Stop the “Blame Game”
– aka “There’s your Problem!”
28
DevOpsGuys @DevOpsGuys www.devopsguys.com
DevOpsGuys
Thank You Workshop Session @ 1345 End-to-end testing & APM demo operations | availability | performance | scalability | migration | strategy
29
DevOpsGuys @DevOpsGuys www.devopsguys.com
DevOpsGuys provide development &
operations support (DevOps) services
for your online applications.
You run your business. We run your website.
It’s as simple as that.
What We Do
30
DevOpsGuys @DevOpsGuys www.devopsguys.com
Our service portfolio
• 24 x 7 Application-Centric web operations
• DevOps Transformation & Training
• Continuous Delivery implementation
• Infrastructure Automation & “Infrastructure as Code”
• Application Performance Management
• AppDynamics Implementation & Consultancy
• Web Performance consulting
• Web monitoring
• Load Testing
• Test Automation and scripting
• JetBrains Product Implementation & Consultancy
• Octopus Deploy implementation
31
DevOpsGuys @DevOpsGuys www.devopsguys.com
James Smith (@TheDevMgr)
• 16 years of software
delivery experience.
• Proven track record
– Pizza Hut
– KFC
– Dell
– RAC
– Aviva
– Fujitsu Siemens
– Whitbread
Steve Thair (@TheOpsMgr)
• 25 years of IT Operations
experience.
• Proven track record
– Vodafone
– Credit Suisse Group
– IBM
– BNP Paribas
– Zurich Financial Services
– Totaljobs Group
– TSL Education
32
The Leadership Team
DevOpsGuys @DevOpsGuys www.devopsguys.com
Contact Us Visit us at www.devopsguys.com
Call our team on +44 (0)20 8798 3079
Email our engineers at [email protected]
33