application quality gates in continuous delivery: deliver better software faster!
TRANSCRIPT
1 @Dynatrace
Application Quality Metrics for your Pipeline
Andreas (Andi) Grabner - @grabnerandi
Shift-Left Quality
Example of a “Bad” Web Deployment 282! Objects on that page9.68MB Page Size
8.8s Page Load Time
Most objects are images delivered from your main
domain
Very long Connect time (1.8s) to your CDN
Example of a Bad Java/Tomcat Deployment526s to render a financial transaction report
1 SQL running 210s!
Debug Logging with log4j on outdated log4j
library (sync issue)
700 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds
Challenges
Deploy Faster!!
Fail Faster!?
Its not about blind automation of pushing more bad code through a shiny pipeline
Metrics based
Decisions!
Time of D
eployment
Availability dropped to 0%
Usage by Channel? Errors on Devices?
Technical Debt!
80%$60B
Insufficient Focus on Quality
The “War Room”
Facebook – December 2012
20%80%
I learning from
others
4 use cases WHY did it happen? HOW to avoid it! METRICS to guide you.
26 @Dynatrace
27 @Dynatrace
#1don't push
without a plan
28 @Dynatrace
Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
29 @Dynatrace
Key Metrics# ResourcesSize of ResourcesPage Size
30 @Dynatrace
31 @Dynatrace
#2There is no easy "Migration" to Micro(Services)
32 @Dynatrace
26.7s Execution Time 33! Calls to the
same Web Service
171! SQL Queries through LINQ by this Web Service – request
similar data for each call
Architecture Violation: Direct access to DB instead from frontend logic
33 @Dynatrace
Key Metrics# Service Calls# of ThreadsSync and Wait Times# SQL executions# of SAME SQL’s
34 @Dynatrace
35 @Dynatrace
#3don't ASSUME you
know the environment
Distance calculation issues
480km biking in 1 hour!
Solution: Unit Test in Live App reports Geo
Calc Problems
Finding: Only happens on certain
Android versions
3rd party issues
Impact of bad 3rd party calls
38 @Dynatrace
Key Metrics# of functional errors# and Status of 3rd party callsPayload of Calls
12 000 000 $
40 @Dynatrace
#4Thinking Big?
Then Start Small!
41 @DynatraceAvailability dropped to 0%
Load Spike resulted in UnavailabilityAd on air
42 @Dynatrace
Alternative: “GoDaddy goes DevOps”
Response time improved 4x
1h before SuperBowl KickOff
1h after Game ended
43 @Dynatrace
Key Metrics
# Domains
Total Size of Content
44 @Dynatrace
What have we learned so far?
45 @Dynatrace
1. # Resources2. Size of Resources3. Page Size4. # Functional Errors5. 3rd Party calls6. # SQL Executions7. # of SAME SQLs
MetricBased
DecisionsAre Cool
We want to get from here …
To here!
Use these application metrics as additional Quality Gates
Extend your Continuous Integration
12 0 120ms3 1 68ms
Build 20 testPurchase OKtestSearch OK
Build 17 testPurchase OKtestSearch OK
Build 18 testPurchase FAILEDtestSearch OK
Build 19 testPurchase OKtestSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms3 1 68ms
12 5 60ms3 1 68ms
75 0 230ms3 1 68ms
Test & Monitoring Framework Results Architectural Data
We identified a regresesion
Problem solved
Exceptions probably reason for failed testsProblem fixed but now we have an
architectural regressionProblem fixed but now we have an
architectural regressionNow we have the functional and architectural confidence
Let’s look behind the scenes
#1: Analyzing each Test
#2: Metrics for each Test
#3: Detecting Regression based on Measure
Quality-Metrics based Build Status
Pull data into Jenkins, Bamboo ...
Recap!
#1: Pick your App Metrics
# of Service Calls Bytes Sent & Received
# of Worker Threads
# of Worker Threads
# of SQL Calls, # of Same SQLs # of DB
Connections
# of SQL Calls, # of Same SQLs # of DB
Connections
#2: Figure out how to monitor them
#3: Automate it into your Pipeline
#4: Integrate with your Tools
Better Software,
Faster!!
Draw better Unicorns
60 @Dynatrace
Questions and/or DemoSlides: slideshare.net/grabnerandiGet Tools: bit.ly/dttrialYouTube Tutorials: bit.ly/dttutorialsContact Me: [email protected] Me: @grabnerandiRead More: blog.dynatrace.com
61 @Dynatrace
Andreas GrabnerDynatrace Developer Advocate@grabnerandihttp://blog.dynatrace.com