Implementing Metrics-Driven DevOpsWhy and How!Andreas Grabner: @grabnerandi, andreas.grabner@dynatrace.com Slides: http://www.slideshare.net/grabnerandiPodcast: https://www.spreaker.com/show/pureperformance

@grabnerandi

@grabnerandi

@grabnerandiAND MANY MORE

@grabnerandihttps://dynatrace.github.io/ufo/

“In Your Face” Data!

@grabnerandi

Time of D

eployment

Availability dropped to 0%

#1: Availability -> Brand Impact

@grabnerandi

New Deployment + Mkt Push

Increase # of unhappy users!

Decline in Conversion Rate

Overall increase of Users!

#2: User Experience -> Conversion

Spikes in FRUSTRATED Users!

@grabnerandi

#3: Resource Cons -> Cost per Feature

@grabnerandi

App with Regular Load supported by

10 ContainersTwice the Load but 48 (=4.8x!) Containers! App doesn’t scale!!

#4: Scalability -> Cost per User

@grabnerandi

#5: Performance -> Behavior

@grabnerandi

@grabnerandi

DevOps @ Targetpresented at Velocity, DOES and more …

http://apmblog.dynatrace.com/2016/07/07/measure-frequent-successful-software-releases/

“We increased from monthly to 80 deployments per week

… only 10 incidents per month …

… over 96% successful! ….”

“We Deliver High Quality Software,Faster and Automated using New Stack“

„Shift-Left Performance to Reduce Lead Time“

Adam Auerbach, Sr. Dir DevOps

https://github.com/capitalone/Hygieia & https://www.spreaker.com/user/pureperformance

“… deploy some of our most critical production workloads on the AWS platform …”, Rob Alexander, CIO

2 major releases/yearcustomers deploy & operate on-prem

26 major releases/year170 prod deployments/dayself-service online sales SaaS & Managed

2011 2016

@grabnerandi

Not only fast delivered but also delivering fast!

-1000ms +2%

Response Time Conversions

-1000ms +10%

+100ms -1%

Why most (will) fail!

@grabnerandi

@grabnerandi

It‘s not about blind automation of pushing more bad code on new stacks through a pipeline

@grabnerandi

It‘s not about blindly adding new features on topof existing withouth measuring its success

@grabnerandi

I learning from

others

@grabnerandi

http://bit.ly/sharepurepath

@grabnerandi

Scaling an Online Sports Club Search Service

2015201420xx

Response Time

2016+

1) 2-Man Project 2) Limited Success

3) Start Expansion

4) Performance Slows Growth Users

5) Potential Decline?

@grabnerandi

Early 2015: Monolith Under Pressure

Can‘t scale vertically endlessly!

May: 2.68s 94.09% CPU Bound

April: 0.52s

@grabnerandi

From Monolith to Services in a Hybrid-Cloud

Front Endto Cloud

Scale Backendin Containers!

@grabnerandi

Go live – 7:00 a.m.

@grabnerandi

Go live – 12:00 p.m.

What Went Wrong?

@grabnerandi

26.7s Load Time5kB Payload

33! Service Calls

99kB - 3kB for each call!

171! Total SQL Count

Architecture ViolationDirect access to DB from frontend service

Single search query end-to-end

@grabnerandi

The fixed end-to-end use case“Re-architect” vs. “Migrate” to Service-Orientation

2.5s (vs 26.7) 5kB Payload

1! (vs 33!) Service Call

5kB (vs 99) Payload!

3! (vs 177) Total SQL Count

@grabnerandi

@grabnerandi

You measure it! from Dev (to) Ops

@grabnerandi

Build 17 testNewsAlert OK

testSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms

1 35 5kb 120ms

Use Case Tests and Monitors Service & App Metrics

Build 26 testNewsAlert OK

testSearch OK

Build 25 testNewsAlert OK

testSearch OK

1 4 1kb 60ms

34 171 104kb 550ms

Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

1 4 1kb 60ms

2 3 10kb 150ms

1 0.6% 3.2s

5 75% 2.5s

Build 35 testNewsAlert -

testSearch OK

- - - -

2 3 10kb 150ms

- - -

8 80% 2.0s

Metrics from and for Dev(to)Ops

Re-architecture into „Services“ + Performance Fixes

Scenario: Monolithic App with 2 Key Features

@grabnerandi

your tool of choice

Baseline Key Metrics

#SQL, #Threads, Bytes Sent, # ConnectionsWPO Metrics, Objects Allocated, ...

@grabnerandihttps://github.com/Dynatrace/Dynatrace-Test-Automation-Sampleshttps://dynatrace.github.io/ufo/

Regression Detected

Fail the build!

@grabnerandi

Dev&Test: Check-In Better Code

Performance: Production Ready Checks! Validate Monitoring

Ops/Biz: Provide Usage and Resource Feedback for next

Sprints

Test / CI: Stop Bad Builds Early

Build & Deliver Apps like the Unicorns!With a Metrics-Driven Pipeline!

@grabnerandi

12:00 a.m – 11:59 p.m.

QuestionsSlides: slideshare.net/grabnerandiGet Tools: bit.ly/dtpersonalWatch: bit.ly/dttutorialsFollow Me: @grabnerandiRead More: blog.dynatrace.comListen: http://bit.ly/pureperfMail: andreas.grabner@dynatrace.com

Andreas GrabnerDynatrace Developer Advocate@grabnerandihttp://blog.dynatrace.com