data-driven devops: improve velocity and quality of software delivery with metrics and feedback
TRANSCRIPT
Copyright © 2016 Splunk Inc.
Data-Driven DevOps:Improve Velocity and Quality of Software Delivery with Metrics and FeedbackAndi Mann@AndiMann
Abstract (Hidden)Much of the value of DevOps comes from a (renewed) focus on measurement, sharing, and continuous feedback loops. In increasingly complex DevOps workflows and environments, and especially in larger, regulated, or more crystallized organizations, these core concepts become even more critical.
This session will show how, by focusing on 'metrics that matter,' you can provide objective, transparent, and meaningful feedback on DevOps processes to all stakeholders. Learn from real-life examples how to use the data generated throughout application delivery to continuously identify, measure, and improve deployment speed, code quality, process efficiency, outsourcing value, security coverage, audit success, customer satisfaction, and business alignment.
Shared Feedback Enables ‘The Three Ways’
Gene Kim, “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.”
DevOps Workflow is Becoming Complex and Opaque
6
Build(Jenkins, Bamboo)
Code(Git,
MS-TFS)
Plan(Jira, Rally)
Test/QA(Cucumber, SonarQube)
Stage(Pivotal,
AWS)
Release(Jenkins, Octopus)
Data Center
Device Data
Engagement Data
Config(Puppet, Ansible)
Monitor(NewRelic, Dynatrace)
Cloud Services Network Services
www/HTTPData
Social Sentiment
Wire Data
Application Data
Continuous Integration (CI) / Continuous Delivery (CD)
Site Reliability Engineering
Business Impact Monitoring
API ServicesSecurity/Compliance
DevOps complexity raises risk of failure● Slower Speed
● Longer MTTR
● Lower Quality
● Reduced Agility
● Poor Visibility
● Hard to Scale
● Increased Waste
● Impaired Collaboration
7
DevOps
From Hype Cycle for Application Services 2015, Gartner Group, July 2015, Betsy Burton, Philip Allega, http://www.gartner.com/document/3096018
Gartner’s DevOps ‘Metrics that Matter’
Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati Bandopadhyay
More DevOps Metrics that Might Matter
Culturee.g.• Retention• Satisfaction• Callouts
Processe.g.• Idea-to-cash• MTTR• Deliver time
Qualitye.g.• Tests passed• Tests failed• Best/worst
Systemse.g.• Throughput• Uptime• Build times
Activitye.g.• Commits• Tests run• Releases
Impacte.g.• Signups• Checkouts• Revenue
Common Data Fabric
15
APISDKs UI
Other ToolsEscalation/
Collaboration
Visibility Across the Whole Dev Lifecycle
Plan Code Build Test/QA Stage Release Config Monitor
Common Data Fabric
16
APISDKs UI
Server, Storage. N/W
Server Virtualization
Operating Systems
Infrastructure Applications
Mobile Applications Cloud Services
Other ToolsTicketing/Help
Desk
Custom Applications
Visibility Across the Whole Ops Environment
API Services
Finding Your Metrics That Matter
Work from business backwards
Mine realtime machine data
Close the feedback loops
23
Data drives objective feedback loops
VelocityDeliver on time & on budget
IT is delivering on time, on budget
IT and Business Leaders
ImpactDeliver code for business needs
IT is achieving business goals
IT and Business Leaders, Customers, Staff
Show you when you deliver. And when you don’t.
QualityDeliver the quality you promised
We deliver a quality experience for users
Dev and Ops Organizations
Objective data enables continuous improvement
Defect Information
CapacityPlanning
Quality Standards
Enhancement Requests
Integration Requirements
Acceptance Metrics
Service Levels and KPIs
Application Development Test and Acceptance Production
BuildCodePlan Test/QA Stage Release Config Monitor
InfrastructureDependencies
Data clearly identifies ‘waste’
Plan
Develop (UI)
Develop (Db)
Develop (M’ware)
Develop (Backend)
SecurityTest
Monitor
Build(Prod)
Architect
Secure/Comply
DeployAccept
UnitTest
Document
Cap Plan
Train
Feedback
IntegrationTest
Configure
System Test
Launch
CAB
Develop(APIs)
Budget
Build(Dev)
Mgmt/Tooling
W
W
W
W
W
W
W
W
W
16 40 52 35 96 40 48 24 --8 2 5 6 8 2 12
Data ensures transparency between all stakeholders
• Release when ready, not a date!
• Best / worst developers
• Best / worst providers
• Impact of new code on ops
• Impact of new code on biz
Data-driven feedback accelerates velocity
Pivot & improve with Continuous Insights
Product Managers identify new opportunities
Continuously delivered to market
… and Auditors are “happy”
Data-driven feedback improves quality
Code quality scans Static security scans
White BoxDeveloper checks in code
Automated Acceptance Tests
Dynamic Security Scans
Black Box
“Chaos Monkey” tests
Test Fail: Return
Test Fail: Return
X
X
Production
QA Prod Pattern
QA Pattern Library
Test Pass: Promote
Test Pass: Promote to Production
Pattern library used for test and
QA
Fast-feedback loop for actionable commercial insights
Data-driven DevOps allows rapid innovation
BUSINESS DEV/OPS CUSTOMERS
HOW IS OUR:• Security?• Quality?• Stability?• Performance?• Compliance?
HOW IS OUR:• Market Launch?• Feature Usage?• Marketing Changes?• Prioritization?• Customer Sat?
Data-driven DevOps decisions help you …
Improve Application Velocity
Visibility across silos, tools, and processes
exposes bugs and bottlenecks so you
can remediate, iterate, and innovate
faster.
Improve Application Quality
Track quality across multiple teams,
tools, systems, and service providers, so you can find and fix more issues before
production
Improve Application Impact
Real-time analytics correlates
application delivery with business goals,
so you can drive better experience and iterate faster
Further Reading● Metrics that Matter, http://intellitect.com/devops-metrics/ - Mark
Michaelis, IntelliTect
● Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | TapatiBandopadhyay
● DevOps and the Cost of Downtime: Fortune 1000, IDC
● Blogs.splunk.com
● Splunk.com/DevOps
● Pleasediscuss.com/andimann