Copyright © 2016 Splunk Inc.

Mining Machine Data for ‘Metrics that Matter’ in a DevOps Workflow

Abstract (Hidden)

IT organizations are increasingly using machine data – including in DevOps practices – to get away from ‘vanity metrics’ and instead to generate ‘metrics that matter’. These metrics provide visibility into the delivery of new application code and the business value of DevOps, to both IT and business stakeholders.

Machine data provides DevOps teams and others – including QA, secops, CxOs and LOB leaders – with meaningful and actionable metrics. This allows stakeholders to monitor, measure, manage, and continuously improve the velocity and quality of code throughout the software lifecycle, from dev/test to customer-facing outcomes and business impact.

In this session Andi Mann, chief technology advocate at Splunk, will share core methodologies, interesting case studies, key success factors and ‘gotcha’ moments from real-world experiences with mining machine data to produce ‘metrics that matter’ in a DevOps context.

DevOps is a Culture of Empathy & Sharing

INTEGRATION

COLLABORATION

COMMUNICATION

BETWEEN DEV AND OPSTO DELIVER BETTER SOFTWARE, FASTER

METHODS FOR IMPROVING

Shared Feedback Enables ‘The Three Ways’

Gene Kim, “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.”

Empowered DevOps Teams

Empathy - more than understanding

• Feel your teammates’ pain

• Understand their work and your impact

Empowerment - more than making decisions

• Be responsible in decisions, activities

• Be accountable to your team of teams

DevOps Workflow is Becoming Complex and Opaque

6

Build(Jenkins, Bamboo)

Code(Git,

MS-TFS)

Plan(Jira, Rally)

Test/QA(Cucumber, SonarQube)

Stage(Pivotal,

AWS)

Release(Jenkins, Octopus)

Data Center

Device Data

Engagement Data

Config(Puppet, Ansible)

Monitor(NewRelic, Dynatrace)

Cloud Services Network Services

www/HTTPData

Social Sentiment

Wire Data

Application Data

Continuous Integration (CI) / Continuous Delivery (CD)

Site Reliability Engineering

Business Impact Monitoring

API ServicesSecurity/Compliance

DevOps complexity raises risk of failure● Slower Speed

● Longer MTTR

● Lower Quality

● Reduced Agility

● Poor Visibility

● Hard to Scale

● Increased Waste

● Impaired Collaboration

7

DevOps

From Hype Cycle for Application Services 2015, Gartner Group, July 2015, Betsy Burton, Philip Allega, http://www.gartner.com/document/3096018

From every tool, every process, every component, on-prem or off

The One Constant: Machine Data

Common Data Fabric

9

API

SDKs UI

Other ToolsEscalation/

Collaboration

Visibility Across the Whole Dev Lifecycle

Plan Code Build Test/QA Stage Release Config Monitor

Common Data Fabric

10

API

SDKs UI

Server, Storage. N/W

Server Virtualization

Operating Systems

Infrastructure Applications

Mobile Applications

Cloud Services

Other ToolsTicketing/Help

Desk

Custom Applications

Visibility Across the Whole Ops Environment

API Services

Machine Data From DevOps Tools

11

Provisioning and Config Metrics

12

Machine Data from QA/Pre-Prod/Staging

13

Machine Data from Release Servers

14

Machine Data from Infrastructure Systems

15

Machine Data from Database Servers

16

Machine Data from Customer Systems

What do you measure and why?

I’m working super hard!!

That’s my stapler.

20

Yeah, but … … what are you

achieving?

I’m gonna need you to come in Sunday.

21

Daily Active Users?

Installs?

Downloads?Sales?

DevOps Metrics that Matter

Culturee.g.

• Retention

• Satisfaction

• Callouts

Processe.g.

• Idea-to-cash

• MTTR

• Deliver time

Qualitye.g.

• Tests passed

• Tests failed

• Best/worst

Systemse.g.

• Throughput

• Uptime

• Build times

Activitye.g.

• Commits

• Tests run

• Releases

Impacte.g.

• Signups

• Checkouts

• Revenue

Gartner’s DevOps ‘Metrics that Matter’

Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati Bandopadhyay

IDC’s DevOps ‘Metrics that Matter’

What Are Your ‘Metrics That Matter’?

Finding Your Metrics That Matter

Work from business backwards

Mine realtime machine data

Close the feedback loops

26

Outcomes

Measurement drives Feedback loops

Velocity

Deliver on time & on budget

IT is delivering on

time, on budget

IT and Business Leaders

Impact

Deliver code for business needs

IT is achieving

business goals

IT and Business Leaders, Customers, Staff

Show you when you deliver. And when you don’t.

Quality

Deliver the quality you promised

We deliver a quality

experience for users

Dev and Ops Organizations

Measurement identifies ‘Waste’

Plan

Develop (UI)

Develop (Db)

Develop (M’ware)

Develop (Backend)

SecurityTest

Monitor

Build(Prod)

Architect

Secure/Comply

DeployAccept

UnitTest

Document

Cap Plan

Train

Feedback

IntegrationTest

Configure

System Test

Launch

CAB

Develop(APIs)

Budget

Build(Dev)

Mgmt/Tooling

W

W

W

W

W

W

W

W

W

16 40 52 35 96 40 48 24 --8 2 5 6 8 2 12

Measurement Ensures Transparency

• Release when ready, not a date!

• Best / worst developers

• Best / worst providers

• Impact of new code on ops

• Impact of new code on biz

Measurement Enables Continuous Improvement

Defect Information

CapacityPlanning

Quality Standards

Enhancement Requests

Integration Requirements

Acceptance Metrics

Service Levels and KPIs

Application Development Test and Acceptance Production

BuildCodePlan Test/QA Stage Release Config Monitor

InfrastructureDependencies

Measurement Improves Quality

Code quality scans Static security scans

White BoxDeveloper checks in code

Automated Acceptance Tests

Dynamic Security Scans

Black Box

“Chaos Monkey” tests

Test Fail: Return

Test Fail: Return

X

X

Production

QA Prod Pattern

QA Pattern Library

Test Pass: Promote

Test Pass: Promote to Production

Pattern library used for test and

QA

Measurement Accelerates Velocity

Pivot & improve with Continuous Insights

Product Managers identify new opportunities

Continuously delivered to market

… and Auditors are “happy”

Measurement Aligns Business Impact

Fast-feedback loop for actionable commercial insights

So You Can Innovate at Market Speed

BUSINESS DEV/OPS CUSTOMERS

HOW IS OUR:

• Security?

• Quality?

• Stability?

• Performance?

• Compliance?

HOW IS OUR:

• Market Launch?

• Feature Usage?

• Marketing Changes?

• Prioritization?

• Customer Sat?

Summary

Metrics that Matter Drive Better Feedback Loops

Improve Application Velocity

Visibility across silos, tools, and processes

exposes bugs and bottlenecks so you

can remediate, iterate, and innovate

faster.

Improve Application Quality

Track quality across multiple teams,

tools, systems, and service providers, so you can find and fix more issues before

production

Improve Application Impact

Real-time analytics correlates

application delivery with business goals,

so you can drive better experience and iterate faster

Sources/Additional Reading● splunk.com/DevOps - Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc.

● dev.splunk.com – Resources for developing with or on ther Splunk platform, incl. SDKs, API Docs, guides, etc.

● blogs.splunk.com – Check the ‘DevOps’ and ‘Ansible’ tags for specifics, including how to deploy Spunk w/ Ansible

● splunkbase.splunk.com – Splunk add-ons and applications incl. Ansible Tower App for Splunk and 1000+ more

● DevOps Review 2016: Accelerating Innovation, Computing Research UK, July 2016

● 2016 State of DevOps Report, DevOps Research and Assessment

● The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis

● The Phoenix Project, Gene Kim, Kevin Behr, George Spafford

● Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay

● Metrics that Matter, Mark Michaelis, IntelliTect

● DevOps and the Cost of Downtime: Fortune 1000, IDC

● DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014

38

Copyright © 2016 Splunk Inc.

Thank You!

Andi Mann

Chief Technology Advocate, Splunk

@andimann