our devops journey: 6 month waterfalls to 1 hour code deploys

Our DevOps JourneyTransforming 6 Month Waterfalls to 1 Hour Code Deploy

http://dynatrace.com/trial

Anita EnglederDevOps Manager

From the DevOps Webinar with Gene & Mark

Mark TomlinsonPerformance Sherpa

@mark_on_task

Andi GrabnerPerformance Advocate

@grabnerandi

Gene Kim, CTOResearcher and Author

@RealGeneKim

@RealGeneKim

24 “Features in a Box” Ship the whole box!

Very late feedback

F r u s t r a t i o n !

@grabnerandi

@RealGeneKim

Continuous Innovation and Optimization1 “Feature at a Time”

Optimize Before DeployImmediate Customer Feedback

@grabnerandi

From 0 to DevOps in 80 daysLessons learnt from shifting an on-prem to a cloud cultureBernd Greifeneder, CTO


Webinar: http://ow.ly/cEYo305kFEyPodcast: http://bit.ly/pureperf

2 major releases/yearcustomers deploy & operate on-prem

26 major releases/year170 prod deployments/dayself-service online sales SaaS & Managed

2011 2016

7 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

believe in the mission impossible

6 monthsmajor/minor release+ intermediate fix-packs+ weeks to months rollout delay

sprint releases (continuous-delivery)1h : code to production

Dynatrace Transformation by the numbers

23x

170

more releases

Deployments / Day

31000 60hUnit+Int Tests / hour UI Tests per Build

More Quality

~200 340code commits / day Stories per sprint

More Agile

93%Production bugs found by Dev

@grabnerandi

More Stability 450 99.998%Global EC2 Instances Global Availability

DevOps Stories & Practices Learned

•The Role of the DevOps Team•The Role of the Feature Team•The Journey to DevOps•The DevOps Tool Chain

@grabnerandi

Typical Role of DevOps Teams mainly responsible for …

… monitoring Production and creating tickets for engineers… building deployment automation

NOT mainly responsible for …

Role of Dynatrace DevOps Team… monitoring Production and creating tickets for engineers… building deployment automation

owning and promoting tooling that gives engineering teams …

Role of Dynatrace DevOps Team

… VISIBILITY & CONTINUOUS FEEDBACK from commit to deployment and into all deployments

… CONTROL to deploy features/fixes… OWNERSHIP of their own source code for all deployments

acting asEngineers

Role of Dynatrace DevOps Team

Dynatrace Managed/SaaSOrchestration Layer

Dynatrace Pipeline Visualization

Deployment Timeline

Log Overview using Dynatrace Log APIJIRA Integrations

&Product Managers

confidential

Role of Dynatrace Feature Teams

ManualCICDFcontinuousintegration

continuous delivery & deploymentand feedback

Role of Dynatrace Feature Teams

DevOps =

Engineers start thinking like an Ops

before Commit

CDF

develop features &

provide fixes

build & adjust test coverage

implement go-to-

production/migration

plan

add/use proper feedback channel

consume feedback

improve & harden

CI

confidential

DevOpsOur Journey to

CDF & NoOps

Our Journey to CDF & NoOps1

32

4

5

67

Infrastructure/Platform as a Service API call instead fixing hardware

Architects are responsiblefor production too

Same monitoring in every stage! Visibility into every stage/deployment!

Same Deployment Automation in every stage! No manual touches!

Direct Feedback to Engineers!The real experts are primary consumers

Control into engineer’s hand wherever it makes sense.Who pushes the button “deploy”?Provide fast lane to production for

Hotfixes! 1 Hour Code Deploy!

3

45

7

3 Same Deployment Automation in every stage! No manual touches!

Dynatrace ManagedOrchestration Layer

Dynatrace SaaSOrchestration Layer

Orchestration LayerProduct owned by a Feature Team having it’s own CDF Pipeline

4 Same monitoring in every stage! Visibility into every stage/deployment!

Dev Stage Daily Production Stage Acceptance Stage Bi-Daily

4 Same monitoring in every stage! Visibility means also easy access!

https://www.dynatrace.com/meet-davis/

5 Direct Feedback to Engineers! Engineers typically like Logs. But how to get the whole context?

From Log to

Code-level

Thank you!

Food for thoughts: • How is your trust level to your engineering team?

• Do you encourage failure and learn from it?• Where do you currently have bottlenecks from Dev to Ops?

• Can technology help? Can process change help?• Do you have the right monitoring strategy / feedback loop?



Thank you!

7 Provide fast lane to production for Hotfixes!1 Hour Code Deploy

How can I achieve that in case my trunk/master is not yet at any time production ready?

Dev Stage Daily

Continuous Delivery & Feedback

Production Stage Bi-Weekly & On Demand

Acceptance Stage Bi-Daily

Deploy

Develop & Fixing

Acceptance & Performance

& Load Tests & Monitor

Deploy

Fixing

Acceptance & Performance

& Load Tests & Monitor

Deploy

Hotfixing

Monitor

Release

Every 2 weeks version is pushed to next stage = 2 weeks release cycle

Unit+Integration testing & Build



Week 1 Week 2 Week 3 Week 4

CDF Tooling Chain - Dev Stage

Acceptance TestsSelenium/Appium Automated Functional Tests

on UI Level Automated E2E Tests on UI LevelBrowser Compatibility Testing (Chrome, IE, FF, Edge, Safari)

24/7 Deployment ChecksJira Manual Regression TestingAWS, VMware, VirtualBox Test Environments

DevelopGradle + Artifactory build/deployment automationSVN + Git Version ControlEclipse / IntelliJ IDEAQuickbuild + Jenkins Continuous Integration

Unit & Integration Tests (fully automated)Unit Tests +Integration Tests

Quickbuild, Eclipse/IntelliJ IDEA for local execution before commit

Memory/CPU Overhead tests Dynatrace AppMon

Code Quality and Coverage ChecksSonar

Memory leak detectionBullseye, Valgrind

Virus Scan KasperskyOpen Source License compliance checks

Blackduck

Facts/Numbers28.000 Unit Tests + 3.000 integration test executions / hour

Deployment (fully automated)Quickbuild Automated daily deployment of Trunk Builds

using Ansible and Puppet or callingDynatrace Ruxit CloudControl, that uses

AWS Cloud Formation (whole infrastructure as code!)Selenium/Appium

Automated Customer-Like DeploymentDeployment Checks on UI Level

Performance & Load TestsCluster Workload SimulatorEclipse Mem Analyzer (MAT) + Eclipse Thread Dump AnalyzerJava Flightrecorder

MonitoringDynatrace Real-User, Service and

Infrastructure monitoringWebChecks, AWS Monitoring, Log Analytics & Monitoring

Continuous Feedback Tools built by Devs & DevOpsOpsGenie and common HipChat Room to

escalate detected Problems directly to DevelopmentSecurity 24/7 OSSEC Host intrusion monitoring

Deploy

Develop & Fixing


Acceptance & Performance & Load Tests & Monitor

CDF Tooling Chain - Acceptance Stage

Deploy

Fixing


Acceptance & Performance & Load Tests & Monitor

Facts/Numbers (Dev + Acceptance Stage)~700 Automated UI Tests~60 hours UI Test execution per Build ~20 parallel running test sets executed on ~30 execution machines up to ~5 hours per test set~15 different OS (Windows, Linux) Unit & Integration Tests (fully

automated)Unit Tests +Integration Tests


Virus Scan Kaspersky

Deployment (fully automated)Quickbuild Automated deployment of Sprint Builds using Ansible and Puppet or calling Dynatrace Ruxit CloudControl, that uses AWS Cloud Formation

(whole infrastructure as code!)Selenium/Appium


Acceptance TestsSelenium/Appium Automated Functional Tests

on UI Level Automated E2E Tests on UI LevelBrowser Compatibility Testing (Chrome, IE, FF, Edge, Safari)

24/7 Deployment ChecksJira Manual Regression TestingAWS, VMware, VirtualBox Test Environments

FixingGradle + Artifactory build/deployment automationSVN + Git Version ControlEclipse / IntelliJ IDEAQuickbuild + Jenkins Continuous Integration

Performance & Load TestsCluster Workload SimulatorEclipse Mem Analyzer (MAT) + Thread Dump AnalyzerJava Flightrecorder



Continuous Feedback Tools built by Devs & DevOpsOpsGenie and common HipChat Room to escalate detected Problems directly to DevelopmentSecurity 24/7 OSSEC Host intrusion monitoring

CDF Tooling Chain - Production Stage

Deploy

Hotfixing


Monitor

FixingGradle + Artifactory build/deployment automationSVN + Git Version ControlEclipse / IntelliJ IDEAQuickbuild + Jenkins Continuous Integration



Continuous Feedback Tools built by Devs & DevOpsOpsGenie and common HipChat Room to escalate detected Problems directly to DevelopmentSecurity 24/7 OSSEC Host intrusion monitoring

Monthly and on demand Vulnerability Scans (KPMG Linz)Closed Bug Bounty Program at HackerOne

Unit & Integration Tests (fully automated)Unit Tests +Integration Tests


Virus Scan Kaspersky

Deployment (fully automated)Quickbuild Automated deployment of Sprint Builds using Ansible and Puppet or calling Dynatrace Ruxit CloudControl, that uses AWS Cloud Formation

(whole infrastructure as code!)Selenium/Appium


our devops journey: 6 month waterfalls to 1 hour code deploys

Technology