data-driven devops: improve velocity and quality of software delivery with metrics and feedback

35
Copyright © 2016 Splunk Inc. Data-Driven DevOps: Improve Velocity and Quality of Software Delivery with Metrics and Feedback Andi Mann @AndiMann

Upload: splunk

Post on 19-Jan-2017

4.878 views

Category:

Technology


0 download

TRANSCRIPT

Copyright © 2016 Splunk Inc.

Data-Driven DevOps:Improve Velocity and Quality of Software Delivery with Metrics and FeedbackAndi Mann@AndiMann

Abstract (Hidden)Much of the value of DevOps comes from a (renewed) focus on measurement, sharing, and continuous feedback loops. In increasingly complex DevOps workflows and environments, and especially in larger, regulated, or more crystallized organizations, these core concepts become even more critical.

This session will show how, by focusing on 'metrics that matter,' you can provide objective, transparent, and meaningful feedback on DevOps processes to all stakeholders. Learn from real-life examples how to use the data generated throughout application delivery to continuously identify, measure, and improve deployment speed, code quality, process efficiency, outsourcing value, security coverage, audit success, customer satisfaction, and business alignment.

Defining DevOps

INTEGRATION

COLLABORATION

COMMUNICATION

BETWEEN DEV AND OPS

METHODS FOR IMPROVING

CAMS – as close to prescriptive as DevOps gets

Shared Feedback Enables ‘The Three Ways’

Gene Kim, “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.”

DevOps Workflow is Becoming Complex and Opaque

6

Build(Jenkins, Bamboo)

Code(Git,

MS-TFS)

Plan(Jira, Rally)

Test/QA(Cucumber, SonarQube)

Stage(Pivotal,

AWS)

Release(Jenkins, Octopus)

Data Center

Device Data

Engagement Data

Config(Puppet, Ansible)

Monitor(NewRelic, Dynatrace)

Cloud Services Network Services

www/HTTPData

Social Sentiment

Wire Data

Application Data

Continuous Integration (CI) / Continuous Delivery (CD)

Site Reliability Engineering

Business Impact Monitoring

API ServicesSecurity/Compliance

DevOps complexity raises risk of failure● Slower Speed

● Longer MTTR

● Lower Quality

● Reduced Agility

● Poor Visibility

● Hard to Scale

● Increased Waste

● Impaired Collaboration

7

DevOps

From Hype Cycle for Application Services 2015, Gartner Group, July 2015, Betsy Burton, Philip Allega, http://www.gartner.com/document/3096018

WHAT DATA DRIVES YOUR

DEVOPS DECISIONS?

I’m working super hard!!

That’s my stapler.

10

Yeah, but … … what are you

achieving?

I’m gonna need you to come in Sunday.

Gartner’s DevOps ‘Metrics that Matter’

Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati Bandopadhyay

IDC’s DevOps ‘Metrics that Matter’

More DevOps Metrics that Might Matter

Culturee.g.• Retention• Satisfaction• Callouts

Processe.g.• Idea-to-cash• MTTR• Deliver time

Qualitye.g.• Tests passed• Tests failed• Best/worst

Systemse.g.• Throughput• Uptime• Build times

Activitye.g.• Commits• Tests run• Releases

Impacte.g.• Signups• Checkouts• Revenue

From every tool, every process, every component, on-prem or off

One Constant: Machine Data

Common Data Fabric

15

APISDKs UI

Other ToolsEscalation/

Collaboration

Visibility Across the Whole Dev Lifecycle

Plan Code Build Test/QA Stage Release Config Monitor

Common Data Fabric

16

APISDKs UI

Server, Storage. N/W

Server Virtualization

Operating Systems

Infrastructure Applications

Mobile Applications Cloud Services

Other ToolsTicketing/Help

Desk

Custom Applications

Visibility Across the Whole Ops Environment

API Services

Data From Dev and Ops Tools

17

Data From Provisioning and Config

18

Data from Release Servers

19

Data from Infrastructure Systems

20

Data from Database Servers

21

Data from Customer Systems

Finding Your Metrics That Matter

Work from business backwards

Mine realtime machine data

Close the feedback loops

23

Outcomes

Data drives objective feedback loops

VelocityDeliver on time & on budget

IT is delivering on time, on budget

IT and Business Leaders

ImpactDeliver code for business needs

IT is achieving business goals

IT and Business Leaders, Customers, Staff

Show you when you deliver. And when you don’t.

QualityDeliver the quality you promised

We deliver a quality experience for users

Dev and Ops Organizations

Objective data enables continuous improvement

Defect Information

CapacityPlanning

Quality Standards

Enhancement Requests

Integration Requirements

Acceptance Metrics

Service Levels and KPIs

Application Development Test and Acceptance Production

BuildCodePlan Test/QA Stage Release Config Monitor

InfrastructureDependencies

Data clearly identifies ‘waste’

Plan

Develop (UI)

Develop (Db)

Develop (M’ware)

Develop (Backend)

SecurityTest

Monitor

Build(Prod)

Architect

Secure/Comply

DeployAccept

UnitTest

Document

Cap Plan

Train

Feedback

IntegrationTest

Configure

System Test

Launch

CAB

Develop(APIs)

Budget

Build(Dev)

Mgmt/Tooling

W

W

W

W

W

W

W

W

W

16 40 52 35 96 40 48 24 --8 2 5 6 8 2 12

Data ensures transparency between all stakeholders

• Release when ready, not a date!

• Best / worst developers

• Best / worst providers

• Impact of new code on ops

• Impact of new code on biz

Data-driven feedback accelerates velocity

Pivot & improve with Continuous Insights

Product Managers identify new opportunities

Continuously delivered to market

… and Auditors are “happy”

Data-driven feedback improves quality

Code quality scans Static security scans

White BoxDeveloper checks in code

Automated Acceptance Tests

Dynamic Security Scans

Black Box

“Chaos Monkey” tests

Test Fail: Return

Test Fail: Return

X

X

Production

QA Prod Pattern

QA Pattern Library

Test Pass: Promote

Test Pass: Promote to Production

Pattern library used for test and

QA

Data-driven feedback shows business Impact

Fast-feedback loop for actionable commercial insights

Data-driven DevOps allows rapid innovation

BUSINESS DEV/OPS CUSTOMERS

HOW IS OUR:• Security?• Quality?• Stability?• Performance?• Compliance?

HOW IS OUR:• Market Launch?• Feature Usage?• Marketing Changes?• Prioritization?• Customer Sat?

Summary

Data-driven DevOps decisions help you …

Improve Application Velocity

Visibility across silos, tools, and processes

exposes bugs and bottlenecks so you

can remediate, iterate, and innovate

faster.

Improve Application Quality

Track quality across multiple teams,

tools, systems, and service providers, so you can find and fix more issues before

production

Improve Application Impact

Real-time analytics correlates

application delivery with business goals,

so you can drive better experience and iterate faster

Further Reading● Metrics that Matter, http://intellitect.com/devops-metrics/ - Mark

Michaelis, IntelliTect

● Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | TapatiBandopadhyay

● DevOps and the Cost of Downtime: Fortune 1000, IDC

● Blogs.splunk.com

● Splunk.com/DevOps

● Pleasediscuss.com/andimann