devops pipelines and metrics driven feedback loops

Download DevOps Pipelines and Metrics Driven Feedback Loops

If you can't read please download the document

Post on 15-Apr-2017




3 download

Embed Size (px)


Dynatrace Presentation

Metrics-Driven PipelinesorTesting & Measures in DevOpsAndreas Grabner: @grabnerandi, Slides:

Most screenshots are from Dynatrace AppMon but presented concepts should work with many other tools


The Story started in 2009


The first DevOpsDays held in Ghent, Belgium:




The stuff we did when we were a Start Up and we All wereDevs, Testers and OpsQuote from Andreas Grabner back in 2013 @ DevOps Boston



In case you are a DevOps Virgin I definitely recommend checking out The Phoenix Project (the DevOps Bible) and Continuous Delivery (which is what we actually all want to achieve): Deliverying software faster with great quality and without all potential mistakes that a manual and rigid process brings with it

This inspired many companies which have been talking about their successes!5

Utmost goal: minimize cycle time (= Lead Time)time

feature cycle time

minimizeUsersThis is where youcreate value!

Minimize feature cycle time and


From the DevOps Webinar with Gene & Mark

Mark TomlinsonPerformance Sherpa@mark_on_taskAndi GrabnerPerformance Advocate@grabnerandiGene Kim, CTOResearcher and Author@RealGeneKim

Webinar Recording:

See the full webinar:

High Performers Are 200x

2,555xmore frequent deploymentsfaster lead times than their peersSource: Puppet Labs 2015 State Of DevOps Report:

More Agile3x

24xlower change failure ratefaster Mean Time to RecoverMore Reliable


24 Features in a BoxShip the whole box!Very late feedback

F r u s t r a t i o n !

My blog on this:

1 Feature at a TimeOptimize before Deploy

Immediate Customer Feedback

Continuous Innovation and Optimization

The new way is how we take Pictures Right Now: We see what we are about to build we optimize it on the spot based on best practices we deploy it into production and immediately get feedback from our customers / friends: then we can decide whehter we built the right thing or not. Whether we take another picture of the scene or not or whether we delete some of those that nobody likes10

DevOps Adoption

Resource: 11

700 deployments / YEAR10 + deployments / DAY50 60 deployments / DAYEvery 11.6 SECONDS

Innovators (aka Unicorns): Deliver value at the speed of business

Several companies changed their way they develop and deploy software over the years. Here are some examples (numbers from 2011 2014)Cars: from 2 deployments to 700Flicks: 10+ per DayEtsy: lets every new employee on their first day of employment make a code change and push it through the pipeline in production: THATS the right approach towards required culture changeAmazon: every 11.6s

Remember: these are very small changes which is also a key goal of continuous delivery. The smaller the change the easier it is to deploy, the less risk it has, the easier it is to test and the easier is it to take it out in case it has a problem.


Early Adopters

But not only the hipsters / unicorns have been doing it it is catching on even in enterprises that seem too big. But because they are too big to fail they had to go through a major transformation!Taken from

DevOps @ Targetpresented at Velocity, DOES and more We increased from monthly to 80 deployments per week

only 10 incidents per month

over 96% successful! .


Such as Target ...14

We DeliverHigh Quality Software,Faster and Automated using New Stack

Shift-Left Performance to Reduce Lead Time Adam Auerbach, Sr. Dir DevOps & deploy some of our most critical production workloads on the AWS platform , Rob Alexander, CIO

From 0 to DevOps in 80 daysLessons learnt from shifting an on-prem to a cloud cultureBernd Greifeneder, CTO


Check out the recorded webinar and podcastWebinar:


2 major releases/yearcustomers deploy & operate on-prem26 major releases/year170 prod deployments/dayself-service online sales SaaS & Managed20112016

Die 17

believe in the mission impossible6 monthsmajor/minor release+ intermediate fix-packs+ weeks to months rollout delay

sprint releases (continuous-delivery)1h : code to production

#COMPANY CONFIDENTIAL DO NOT DISTRIBUTE#Perform2015Dynatrace 6.2 verstrkte burn-down phase im letzten 1/3:Ruxit - up/down trend in sprints, ideal wre eine gerade blaue linie, wobei sich rot und grn leicht zeitversetzt berdecken

18 Your Face Data!


A basic key metric for developers should be Did I break the build.This is why we at Dynatrace installed these Pipeline State UFOs that are hooked up with Jenkins to tell engineers how good or bad the current Trunk or Latest Sprint build isKey thing here is that this should not only be applied to the build itself but to metrics across the delivery pipeline: from DevToOps. It should include metrics like the next examples19

Time of DeploymentAvailability dropped to 0%#1: Availability -> Brand Impact


The most basic metric for everyone operating software. Did my last deployment break anything? Is the software still available from those locations where my users are accessing the software? Use Synthetic Monitoring: 20

New Deployment + Mkt PushIncrease # of unhappy users!Decline in Conversion RateOverall increase of Users!#2: User Experience -> ConversionSpikes in FRUSTRATED Users!


Monitoring user experience and impact on conversion rateScreenshot from Dynatrace AppMon & UEM21

#3: Resource Cons -> Cost per Feature4x $$$ to IaaS


Even if the deployment seemed good because all features work and response time is the same as before. If your resource consumption goes up like this the deployment is NOT GOOD. As you are now paying a lot of money for that extra compute powerScreenshot from Dynatrace AppMon22

#4: Performance -> Behavior


Understand user behavior depending on who they are and what they are doing.Screenshot from

Does the behavior change if they have a less optimal user experience?Screenshot from

Seems like users that have a frustrating experience are more likely to click on SupportScreenshot from


Dynatrace Transformation by the numbers23x

170more releasesDeployments / Day31000

60hUnit+Int Tests / hourUI Tests per BuildMore Quality~200

340code commits / dayStories per sprintMore Agile93%Production bugs found by Dev@grabnerandiMore Stability450

99.998%Global EC2 InstancesGlobal Availability


Not every Sprint ends without bruises!


Richard DominguezDeveloper in OperationsPrep SportswearIn 2013 business demanded to go from monthly to daily deployments80% failed!


Unfortunately not every story is a good story. But the bad stories are often not told even though we can learn even more. PrepSportswear failed 80% of their deployments after speading up deployments26

Understanding Code Complexity4 Millions Lines of Monolith CodePartially coded and commented in RussianFrom Monolith to MicroserviceInitial devs no longer with companyWhat to extract withouth breaking it?Shift Left Quality & PerformanceNo automated testing in the pipelineBad builds just made it into productionCross Application ImpactsShared Infrastructure between AppsNo consolidated monitoring strategy


Webinars & eBooks

Scaling an Online Sports Club Search Service


Response Time2016+1) 2-Man Project2)


View more >