making data-driven decisions for better devops...

© 2017 SPLUNK INC.

@andimann @splunk

© 2017 SPLUNK INC.

Andi Mann | Chief Technology Advocate

@AndiMann | [email protected]

Making Data-Driven Decisions for

Better DevOps OutcomesAndi Mann, Splunk

@AndiMann

© 2017 SPLUNK INC.

@andimann @splunk

Metrics and Reporting

The Dev Lifecycle is Complex

API

SDKs UI

Other ToolsEscalation/

Collaboration

No rigid schemas – add in data from any other source.

Plan Code Build Test/QA Stage Release Config Monitor

Common Data Fabric

© 2017 SPLUNK INC.

@andimann @splunk

The Ops Environment is Complex

Other ToolsTicketing/Help Desk

Server, Storage,

Network

Server

Virtualization

Operating

Systems

Infrastructure

Applications

Mobile

Applications Cloud Services

Custom

Applications API Services

Metrics and Reporting

API

SDKs UI

Common Data Fabric

© 2017 SPLUNK INC.

@andimann @splunk

DevOps is Exponentially More Complex

BizOpsStageBuildSecQADevPMOBiz

Plan Code Build Test/QA Stage Release Config Monitor

Server, Storage,

Network

Server

Virtualization

Operating

Systems

Infrastructure

Applications

Mobile

Applications Cloud Services

Custom

Applications API Services

© 2017 SPLUNK INC.

@andimann @splunk

▶ From every tool, process, or component in Dev.

▶ On-premises, in the cloud, or with 3rd party Ops.

▶ Across diverse teams, activities, and services.

One Constant -

Data

© 2017 SPLUNK INC.

@andimann @splunk

BUT WHAT

DATA

DRIVES

GOOD

DECISIONS?

© 2017 SPLUNK INC.

@andimann @splunk

I’m working

super hard!!

That’s my

stapler!

© 2017 SPLUNK INC.

@andimann @splunk

9

Yeah, but …

… what are you

achieving?

I’m gonna

need you to

come in

Sunday.

© 2017 SPLUNK INC.

@andimann @splunk

Users?

Installs?

Downloads?Sales?

© 2017 SPLUNK INC.

@andimann @splunk

Some DevOps Data that Might Matter

Culture

e.g.

• Retention

• Satisfaction

• Callouts

Process

e.g.

• Idea-to-cash

• MTTR

• Deliver time

Quality

e.g.

• Tests passed

• Tests failed

• Best/worst

Systems

e.g.

• Throughput

• Uptime

• Build times

Activity

e.g.

• Commits

• Tests run

• Releases

Impact

e.g.

• Signups

• Checkouts

• Revenue

© 2017 SPLUNK INC.

@andimann @splunk


Specific Data For Each Stakeholder

• time to deliver

• idea to cash

• ROI

• process times

• team efficiency

• unplanned work

• code volume

• commit volume

• release speed

• test volume

• code coverage

• exception counts

• remediation time

• code quality

• access rates

• build speed

• failure rates

• manual builds

• performance

• latency

• scalability

• response time

• uptime/availability

• resource usage

• revenue

• signups

• satisfaction

© 2017 SPLUNK INC.

@andimann @splunk


Shared Data for Multiple Stakeholders

• time to deliver

• idea to cash

• ROI

• process times

• team efficiency

• unplanned work

• code volume

• commit volume

• release speed

• test volume

• code coverage

• exception counts

• remediation time

• code quality

• access rates

• build speed

• failure rates

• manual builds

• performance

• latency

• scalability

• response time

• uptime/availability

• resource usage

• revenue

• signups

• satisfaction

© 2017 SPLUNK INC.

@andimann @splunk

Computing UK’s ‘Metrics that Matter’

Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

© 2017 SPLUNK INC.

@andimann @splunk

Increase App Delivery Velocity

DevOps Teams iterate with continuous insights

Product Managers

identify new

opportunities

Code continuously

delivered to market

Customers

are happy

Auditors

have visibility

© 2017 SPLUNK INC.

@andimann @splunk

Tesco uses Machine Data to Accelerate Development and

Understand CustomersCut Investigation & Resolution time 95%

Reduce Escalations 50%, Accelerate Dev Cycles 30%

Activity Tracking Dashboards with Improved Customer Experience and Reduced Lost Revenue

Operational Analytics with Live Transaction Tracing and End-to-end Infrastructure Insight

© 2017 SPLUNK INC.

@andimann @splunk



© 2017 SPLUNK INC.

@andimann @splunk

Improve Code Quality

Code Quality

Scans

Static Security

Scans

White BoxDevelopers

check in code

Automated

Acceptance Tests

Dynamic Security

Scans

Black Box

“Chaos Monkey”

Tests

Production

QA Prod Pattern

QA Pattern Library

Test Pass:

Promote

Test Pass:

Promote to Production

Pattern library

used for test

and QA

Test Fail:

Return

Test Fail:

Return

© 2017 SPLUNK INC.

▶ Key Customer Benefits

• Increased success rate of deployments

• Ability to detect issues before they affect broad production

• Monitoring deployment process several times per day

Improved DevOps Agility

“It’s like we were working without peripheral vision before and now we have it.”– Robert Gonsalves, Web Operations

MEDIA & ENTERTAINMENT – APPLICATION DELIVERY

© 2017 SPLUNK INC.

@andimann @splunk



© 2017 SPLUNK INC.

@andimann @splunk

Data-Driven Feedback Shows Business Impact

© 2017 SPLUNK INC.

@andimann @splunk

Amaya Gaming Uses Machine Data for Digital Customer Insight

Allows DevOps to ensure quality of releases &

avoid negative impact on service performance.

Analyze which new website features are being

adopted, and how, by end users.

Insight fed back into the development

cycle to improve customer engagement.

© 2017 SPLUNK INC.

@andimann @splunk

But Good Data Is Not Enough

© 2017 SPLUNK INC.

@andimann @splunk

Find The Value In The Data

Planning Development Build Verification Deployment Post-Deploy

100 (0%) 100 (0%) 94.74 (-5.3%) 100 (0%) 100 (0%) 100 (0%)

160 stories 0 in progress 100% success 100% success 364 deploys 0 CFDs

100 stories 95 complete 8.8 MTTB 3.95 MTTT 0.54 success

1 ticket 2.489 MTTR

30 points/dev

94 (-6%)

15.12 days

© 2017 SPLUNK INC.

@andimann @splunk

Find The Failure in The Data

Planning Development Build Verification Deployment Post-Deploy

100 (0%) 100 (0%) 0 (-100%) 100 (0%) 100 (0%) 100 (0%)

160 stories 0 in progress 35% success 100% success 364 deploys 0 CFDs

100 stories 95 complete 8.8 MTTB 3.95 MTTT 0.54 success

1 ticket 2.489 MTTR

30 points/dev

94 (-6%)

15.12 days

© 2017 SPLUNK INC.

@andimann @splunk

Data-Driven Decisions For Better DevOps Outcomes

Improve

Velocity

Reduce the time it

takes to get code

through dev/test to

market—through

faster issue

resolution and

reduced cycle time

Improve

Quality

Real-time visibility

into processes like

code check-in, build,

test, QA to support

continuous

integration and

continuous delivery

“Our devs are now able to

find and fix issues 5-10

times faster.”

“We can monitor all the

automation and handoffs it

takes to deploy 5-10 times

a day.”

Improve

Impact

Instrument customer

engagement and

application usage to

capture critical

business events,

outcomes, and user

behavior

“My code isn’t ready until

it’s Splunk-ready.”

© 2017 SPLUNK INC.

@andimann @splunk

Sources/Additional Reading

● splunk.com/DevOps – Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc.

● dev.splunk.com – Resources for developing with or on the Splunk platform, incl. SDKs, API Docs, guides, etc.

● blogs.splunk.com – Check the ‘DevOps’ tag for specifics, including how to deploy Spunk w/ CI/CD tools

● splunkbase.splunk.com – Splunk add-ons, applications, and TAs for AWS, Jenkins, Ansible, Jira, Puppet, Docker, and 1000+ more

● Vertu Calls On Splunk Enterprise For Smarter DevOps, Splunk Press Release, 2016,

● DevOps Review 2016: Accelerating Innovation, Computing Research UK, July 2016

● 2016 State of DevOps Report, DevOps Research and Assessment

● The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis

● The Phoenix Project, Gene Kim, Kevin Behr, George Spafford

● Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay

● Metrics that Matter, Mark Michaelis, IntelliTect

● DevOps and the Cost of Downtime: Fortune 1000, IDC

● DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014

http://www.splunk.com/DevOps

dev.splunk.com

http://blogs.splunk.com/

http://splunkbase.splunk.com/

http://www.splunk.com/en_us/newsroom/press-releases/2016/vertu-calls-on-splunk-enterprise-for-smarter-devops.html

http://www.computing.co.uk/digital_assets/9965/CTG_DevOps_2016.pdf

http://blogs.splunk.com/tag/2016-state-of-devops-report/

http://itrevolution.com/books/devops-cookbook/

http://itrevolution.com/books/novel/

http://www.gartner.com/document/2753718

http://intellitect.com/devops-metrics/

© 2017 SPLUNK INC.

@andimann @splunk

© 2017 SPLUNK INC.

Andi Mann | Chief Technology Advocate

@AndiMann | [email protected]

Thank You@andimann

[email protected]

making data-driven decisions for better devops...

Documents