london web perfug_performancefocused_devops_feb2014
TRANSCRIPT
![Page 1: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/1.jpg)
1
BEST PRACTICES ON PERFORMANCE-FOCUSED DEVOPS
Andreas Grabner, Technology Strategist @ Compuware/dynaTrace
![Page 2: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/2.jpg)
![Page 3: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/3.jpg)
Popularity of Topics on Google
![Page 4: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/4.jpg)
Who is doing it? How many successful deployments can they do?
300 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds
![Page 5: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/5.jpg)
More on Amazons Story
75% fewer outages since 2006
90% fewer outage minutes
~0.001% of deployments cause a problem
Instantaneous automatic rollback
Deploying every 11.6s
![Page 6: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/6.jpg)
![Page 7: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/7.jpg)
![Page 8: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/8.jpg)
![Page 9: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/9.jpg)
![Page 10: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/10.jpg)
Testing is Important – and gives Confidence
![Page 11: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/11.jpg)
But are we ready for “The Real” world?
![Page 12: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/12.jpg)
Measure Performance during the game
Ball Possession: 40 : 60Fouls: 0 : 0Score: 0 : 0
Minute 1 - 5
![Page 13: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/13.jpg)
Measure Performance during the game
Minute 6 - 35
Ball Possession: 80 : 20Fouls: 2 : 12Score: 0 : 0
![Page 14: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/14.jpg)
Deep Dive Analysis
![Page 15: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/15.jpg)
Options “To Fix” the situation
![Page 16: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/16.jpg)
Not always a happy ending
Minute 90
Ball Possession: 80 : 20Fouls: 4 : 25Score: 3 : 0
![Page 17: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/17.jpg)
FRUSTRATED FANS!!
17
![Page 18: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/18.jpg)
How does that relate to
Software?
How does that relate to
Software?
![Page 19: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/19.jpg)
From Deploy to …
Deploy Promotion/Event Problems Ops Playbook War Room
Timeline
![Page 20: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/20.jpg)
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
![Page 21: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/21.jpg)
The “War Room” – NOW
Facebook – December 2012
The “War Room” – NOW
Facebook – December 2012
![Page 22: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/22.jpg)
Problem: Attitudes like this don’t help either
Image taken from https://www.scriptrock.com/blog/devops-whats-hype-about/
Shopzilla CIO (in 2010): “… when they get in the war room - the developers and ops teams describe the problem as the enemy, not each other”
![Page 23: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/23.jpg)
Problem: Very “expensive” to work on these issues
~80% of problems
caused by ~20% patterns
YES we know this
80% Dev Time in Bug Fixing
$60B Defect Costs
BUT
![Page 24: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/24.jpg)
![Page 25: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/25.jpg)
![Page 26: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/26.jpg)
Lets start on the frontend - 80/20 rule from Steve
![Page 27: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/27.jpg)
The impact of Architectural Decisions
Metrics: # Visitors, # Requests / User
![Page 28: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/28.jpg)
Bloated Mobile Web Sites (Shopping Sites Dec 2013)
Huge Java Script, HTML and CSS Files
Large Resources take longer to load – especially on slow
connections
Total Size: 2.5MB
Metric: Total Page Size
![Page 29: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/29.jpg)
Bloated Mobile Web Sites Super Bowl Ad Landing Page
434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
Good First Impression and OnLoad Time – but very long Total Load Time
Metrics: Load Time, # Resources, …
![Page 30: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/30.jpg)
Mobile Web Site: JUST Redirects to regular page
Most CSS and JS files are redirected to the www domain
This is a lot of time “wasted” especially on high latency mobile connections
Metric: # HTTP 3xx, 4xx, 5xx
![Page 31: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/31.jpg)
Individual images are very small
But downloading so many resoures in parallel takes its time due to browser
connection limits
52 individual country flags!
Metric: # Images
![Page 32: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/32.jpg)
Misconfigured CDN!
46! HTTP 403 Requests for images on the landing page
Probably a configuration problem with their Amazon AWS CDN
Lots of time “wasted” due to roundtrips that just result in a 403
Metric: # HTTP 4xx
![Page 33: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/33.jpg)
Slow JavaScript
1.1s on my IE 10 to execute magicSpanLinks()
The each loop calls this block of JavaScript for every span node
759 span nodes are processed by the anonymous function
It adds the dynamically generated link and removes the
old span
We can see all the DOM Modifications and how this sums
up in execution time
Metric: Time Spent in JS Execution
![Page 34: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/34.jpg)
Browser-specific Problems
• “I would love to spend money with you – but I cant get through the checkout process”
• Looked at the data from this user to identify the error
HTTP 404 at OrderItemAdd
Metric: HTTP xxx per Browser
![Page 35: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/35.jpg)
Root Cause: A cookie bug in Safari
Invalid Cookie Exception
Impacts ALL Safari Users
• ALL Safari users were impacted that bought a product with special characters in the name
Metric: # Exceptions per User,# Exceptions per Browser
![Page 36: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/36.jpg)
But - then we have to focus on the backend
![Page 37: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/37.jpg)
Deployment Mistakes lead to internal Exceptions
Metric: # of Exception Objects Created
![Page 38: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/38.jpg)
Deployment Mistakes lead to high logging overhead
Metric: Time Spent in critical Methods
![Page 39: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/39.jpg)
Production Deployment leads to Log SYNC Issues
Metric: Time Spent in Sync & Logging# of Log Messages
![Page 40: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/40.jpg)
Top Problem Patterns: Resource Pools
Metric: Connection Pool Usage, Acquisition Time
![Page 41: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/41.jpg)
Long running SQL with Production Data
Metric: Time Spent in SQL Execution
![Page 42: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/42.jpg)
N+1 Query Problem
Metric: # SQL Executions / Request# of “same” SQL Executions
![Page 43: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/43.jpg)
Memory Leaks under Production Load
Still crashes
Problem fixed!Fixed Version Deployed
Metric: Heap Size, Object Churn Rate
![Page 44: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/44.jpg)
List of Metrics we just saw
• Architectural Metrics
– Active Worker Threads
– Time Spent in Tier/Class
– # of Exception Objects Created
– Time Spent in critical Methods
– Time Spent in Sync & Logging
– # of Log Messages
• Database Metrics
– Connection Pool Usage, Acquisition Time
– Time Spent in SQL Execution
– # SQL Executions / Request
– # of “same” SQL Executions
• Memory Metrics
– Heap Size, Object Churn Rate
– # Important Objects on Heap
• Web & Mobile Metrics
– Size and # of Resources
– # of JS Files
– Total Page Size
– # HTTP 3xx, 4xx, 5xx
– Time Spent in JS Execution
– # Crashes
– # Errors
![Page 46: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/46.jpg)
![Page 47: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/47.jpg)
How about this idea?
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test Framework Results Architectural Data
We identified a regresesion
Problem solved
Lets look behind the scenes
Exceptions probably reason for failed tests
Problem fixed but now we have an architectural regression
Problem fixed but now we have an architectural regression
Now we have the functional and architectural confidence
![Page 48: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/48.jpg)
How? Performance Focus in Test Automation
Cross Impact of KPIs
Analyzing All Unit / Performance Tests
Analyze Perf Metrics
Identify Regressions
![Page 49: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/49.jpg)
How? Performance Focus in Test Automation
Embed your Architectural Results in Jenkins
![Page 50: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/50.jpg)
59
Getting control over your weekend again …Getting control over your weekend again …
Enjoy a beer with friends?
Instead of pizza and soda with your colleagues?
![Page 51: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/51.jpg)
IF WE DO ALL THAT
80% Dev Time for Bug Fixing
$60B Costs by Defects
![Page 52: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/52.jpg)
Recommended Book
https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/
![Page 53: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/53.jpg)
FREE Products & More Info
• dynaTrace Enterprise– Full End-to-End Visibility in your Java, .NET, PHP Apps
– Sign up for a 15 Days Free Trial on http://compuwareapm.com
• dynaTrace AJAX Edition– Browser Diagnostics for IE + FF
– Download @ http://ajax.dynatrace.com
• Our Blog: http://apmblog.compuware.com
![Page 55: London web perfug_performancefocused_devops_feb2014](https://reader033.vdocuments.site/reader033/viewer/2022052822/554f5591b4c905524c8b515a/html5/thumbnails/55.jpg)
© 2011 Compuware Corporation — All Rights Reserved
Simply Smarter