making data-driven decisions for better devops...

@andimann @splunk

Andi Mann | Chief Technology Advocate

@AndiMann | amann@splunk.com

Making Data-Driven Decisions for

Better DevOps OutcomesAndi Mann, Splunk

@AndiMann

@andimann @splunk

Metrics and Reporting

The Dev Lifecycle is Complex

SDKs UI

Other ToolsEscalation/

Collaboration

No rigid schemas – add in data from any other source.

Plan Code Build Test/QA Stage Release Config Monitor

Common Data Fabric

@andimann @splunk

The Ops Environment is Complex

Other ToolsTicketing/Help Desk

Server, Storage,

Network

Server

Virtualization

Operating

Systems

Infrastructure

Applications

Mobile

Applications Cloud Services

Custom

Applications API Services

Metrics and Reporting

SDKs UI

Common Data Fabric

@andimann @splunk

DevOps is Exponentially More Complex

BizOpsStageBuildSecQADevPMOBiz

Plan Code Build Test/QA Stage Release Config Monitor

Server, Storage,

Network

Server

Virtualization

Operating

Systems

Infrastructure

Applications

Mobile

Applications Cloud Services

Custom

Applications API Services

@andimann @splunk

▶ From every tool, process, or component in Dev.

▶ On-premises, in the cloud, or with 3rd party Ops.

▶ Across diverse teams, activities, and services.

One Constant -

@andimann @splunk

BUT WHAT

DRIVES

DECISIONS?

@andimann @splunk

I’m working

super hard!!

That’s my

stapler!

@andimann @splunk

Yeah, but …

… what are you

achieving?

I’m gonna

need you to

come in

Sunday.

@andimann @splunk

Users?

Installs?

Downloads?Sales?

@andimann @splunk

Some DevOps Data that Might Matter

Culture

• Retention

• Satisfaction

• Callouts

Process

• Idea-to-cash

• MTTR

• Deliver time

Quality

• Tests passed

• Tests failed

• Best/worst

Systems

• Throughput

• Uptime

• Build times

Activity

• Commits

• Tests run

• Releases

Impact

• Signups

• Checkouts

• Revenue

@andimann @splunk

Specific Data For Each Stakeholder

• time to deliver

• idea to cash

• ROI

• process times

• team efficiency

• unplanned work

• code volume

• commit volume

• release speed

• test volume

• code coverage

• exception counts

• remediation time

• code quality

• access rates

• build speed

• failure rates

• manual builds

• performance

• latency

• scalability

• response time

• uptime/availability

• resource usage

• revenue

• signups

• satisfaction

@andimann @splunk

Shared Data for Multiple Stakeholders

• time to deliver

• idea to cash

• ROI

• process times

• team efficiency

• unplanned work

• code volume

• commit volume

• release speed

• test volume

• code coverage

• code quality

• access rates

• build speed

• failure rates

• manual builds

• performance

• latency

• scalability

• response time

• resource usage

• revenue

• signups

• satisfaction

@andimann @splunk

• time to deliver

• idea to cash

• ROI

• process times

• team efficiency

• unplanned work

• code volume

• commit volume

• release speed

• test volume

• code coverage

• code quality

• access rates

• build speed

• failure rates

• manual builds

• performance

• latency

• scalability

• response time

• resource usage

• revenue

• signups

• satisfaction

@andimann @splunk

• time to deliver

• idea to cash

• ROI

• process times

• team efficiency

• unplanned work

• code volume

• commit volume

• release speed

• test volume

• code coverage

• code quality

• access rates

• build speed

• failure rates

• manual builds

• performance

• latency

• scalability

• response time

• resource usage

• revenue

• signups

• satisfaction

@andimann @splunk

Computing UK’s ‘Metrics that Matter’

Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

@andimann @splunk

Increase App Delivery Velocity

DevOps Teams iterate with continuous insights

Product Managers

identify new

opportunities

Code continuously

delivered to market

Customers

are happy

Auditors

have visibility

@andimann @splunk

Tesco uses Machine Data to Accelerate Development and

Understand CustomersCut Investigation & Resolution time 95%

Reduce Escalations 50%, Accelerate Dev Cycles 30%

Activity Tracking Dashboards with Improved Customer Experience and Reduced Lost Revenue

Operational Analytics with Live Transaction Tracing and End-to-end Infrastructure Insight

@andimann @splunk

Improve Code Quality

Code Quality

Static Security

White BoxDevelopers

check in code

Automated

Acceptance Tests

Dynamic Security

Black Box

“Chaos Monkey”

Production

QA Prod Pattern

QA Pattern Library

Test Pass:

Promote

Test Pass:

Promote to Production

Pattern library

used for test

and QA

Test Fail:

Return

Test Fail:

Return

▶ Key Customer Benefits

• Increased success rate of deployments

• Ability to detect issues before they affect broad production

• Monitoring deployment process several times per day

Improved DevOps Agility

“It’s like we were working without peripheral vision before and now we have it.”– Robert Gonsalves, Web Operations

MEDIA & ENTERTAINMENT – APPLICATION DELIVERY

@andimann @splunk

Data-Driven Feedback Shows Business Impact

@andimann @splunk

Amaya Gaming Uses Machine Data for Digital Customer Insight

Allows DevOps to ensure quality of releases &

avoid negative impact on service performance.

Analyze which new website features are being

adopted, and how, by end users.

Insight fed back into the development

cycle to improve customer engagement.

@andimann @splunk

But Good Data Is Not Enough

@andimann @splunk

Find The Value In The Data

Planning Development Build Verification Deployment Post-Deploy

100 (0%) 100 (0%) 94.74 (-5.3%) 100 (0%) 100 (0%) 100 (0%)

160 stories 0 in progress 100% success 100% success 364 deploys 0 CFDs

100 stories 95 complete 8.8 MTTB 3.95 MTTT 0.54 success

1 ticket 2.489 MTTR

30 points/dev

94 (-6%)

15.12 days

@andimann @splunk

Find The Value In The Visualization

@andimann @splunk

Find The Failure in The Data

Planning Development Build Verification Deployment Post-Deploy

100 (0%) 100 (0%) 0 (-100%) 100 (0%) 100 (0%) 100 (0%)

160 stories 0 in progress 35% success 100% success 364 deploys 0 CFDs

100 stories 95 complete 8.8 MTTB 3.95 MTTT 0.54 success

1 ticket 2.489 MTTR

30 points/dev

94 (-6%)

15.12 days

@andimann @splunk

Find The Failure in the Visualization

@andimann @splunk

How About Now?

@andimann @splunk

How About Now?

@andimann @splunk

How About Now?

@andimann @splunk

Apply Machine Learning to Your Data

@andimann @splunk

ML Lets You Predict (and Prevent) Failure

Summary

@andimann @splunk

Data-Driven Decisions For Better DevOps Outcomes

Improve

Velocity

Reduce the time it

takes to get code

through dev/test to

market—through

faster issue

resolution and

reduced cycle time

Improve

Quality

Real-time visibility

into processes like

code check-in, build,

test, QA to support

continuous

integration and

continuous delivery

“Our devs are now able to

find and fix issues 5-10

times faster.”

“We can monitor all the

automation and handoffs it

takes to deploy 5-10 times

a day.”

Improve

Impact

Instrument customer

engagement and

application usage to

capture critical

business events,

outcomes, and user

behavior

“My code isn’t ready until

it’s Splunk-ready.”

@andimann @splunk

Sources/Additional Reading

● splunk.com/DevOps – Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc.

● dev.splunk.com – Resources for developing with or on the Splunk platform, incl. SDKs, API Docs, guides, etc.

● blogs.splunk.com – Check the ‘DevOps’ tag for specifics, including how to deploy Spunk w/ CI/CD tools

● splunkbase.splunk.com – Splunk add-ons, applications, and TAs for AWS, Jenkins, Ansible, Jira, Puppet, Docker, and 1000+ more

● Vertu Calls On Splunk Enterprise For Smarter DevOps, Splunk Press Release, 2016,

● DevOps Review 2016: Accelerating Innovation, Computing Research UK, July 2016

● 2016 State of DevOps Report, DevOps Research and Assessment

● The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis

● The Phoenix Project, Gene Kim, Kevin Behr, George Spafford

● Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay

● Metrics that Matter, Mark Michaelis, IntelliTect

● DevOps and the Cost of Downtime: Fortune 1000, IDC

● DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014

@andimann @splunk

Andi Mann | Chief Technology Advocate

@AndiMann | amann@splunk.com

Thank You@andimann

amann@splunk.com

making data-driven decisions for better devops...

Documents

agile 2014- metrics driven development and devops

metrics-driven devops: delivering high quality software...

data driven decisions for healthcare presentation

devops pipelines and metrics driven feedback loops

quality driven devops 5 steps to more effective qa ... ·...

data driven decisions

data-driven decisions in software product …

guiding data-driven transportation decisions

international center for leadership in education data -...

metrics driven devops - automate scalability and performance...

docker/devops meetup: metrics-driven continuous performance...

data driven devops as presented at codemash 2018

making data-driven investment decisions

[@indeedeng] logrepo: enabling data-driven decisions

powering data-driven decisions

api-driven devops

data-driven devops with ibm virtual data pipeline and ibm

behavior-driven development (bdd) - devops summit 2016

how verizon innovates through ai-driven devops with...

quality driven devops - turnkey...