metrics to power devops

1 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

Metrics to Power DevOpsToward making decisions based on data – not opinion

Sujoy Sen

Sr. Director, Field EngineeringCollabNet, Inc.

Kevin Hancock

Vice PresidentPractice Head – IT Transformation Consulting


Kevin Hancock, Senior Director Worldwide Field Operations

In his ten plus year at CollabNet, Kevin has led successful implementations of CollabNet products and services at some of the world’s largest enterprises.

Kevin works with organizations to transform their globally distributed application development teams into modern, agile organizations, helping to save millions of dollars while accelerating software delivery. Kevin holds a Bachelor of Science degree from Carnegie Mellon University, and a Masters degree from the University of Pittsburgh.

Sujoy Sen, Practice Head – IT Transformation Consulting

Sujoy is part of the leadership team in the Tech Advisory group of TechMahindra. He is heading a practice which helps customers improve the success rate of their critical IT projects. Sujoy is a Certified Six Sigma Black Belt & Manager of Organizational Excellence from American Society for Quality, a PMP, a CISA and an Agile Trainer & Coach.

Sujoy has more than 19 years of experience across various domains and functions. He has been part of Tech Mahindra’s IT & Quality consulting group for the last 10 years.

Today’s Presenters


• Where are we now?

• What’s the problem?

• What do we need?

• How do we get there?

Agenda

4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.

Where are we now?


IT Priorities are changing

McKinsey Global Survey, 2014

• IT is expected to improve business process effectiveness

• IT spending on innovation is expected to increase by 50% and

spending on transaction activities is expected to go down by

15%

• Further current spending on next gen infrastructure is at it

highest in 5 years

• To address the concerns and make effectives use of additional

infrastructure spending, IT should be more agile and responsive

resulting in more innovation

• To enable innovation, a culture of experimentation where IT is

allowed to test new innovations quickly with customer, capture

feedback and continuously improve

• This requires the ability of continuous delivery resulting in better,

faster and cheaper innovation

The focus has shifted from reducing cost to improving the effectiveness of business processes


Agile Upstream

32%*

Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report

*Do standup, burndown/burnup, and

short iterations; product owner; and

constant feedback.

Agile

upstream • Agile Upstream are those practices usually associated with planning, managing and executing a Software project.

• Scrum, Kanban, XP are all used upstream in the process.

• But what about Downstream?


13%†

Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report

†Do upstream plus continuous

integration, continuous delivery, and

test-driven development.

Agile

downstream• Agile Downstream refers to Test

automation (Test-driven Development). Continuous Integration, Continuous Delivery.

• Test automation is a key part of any Continuous Integration process

• Successful Continuous Integration requires Build Management

• Successful Continuous Delivery requires Release Management


• Agile Software Development

• Continuous Integration

• Continuous Delivery

• Visible and accessible Metrics

• Access for ALL Stakeholders (RBAC permitting)

We need an Agile System


Few Companies Have Achieved Enterprise Agility

Plan (Upstream) Deploy (Downstream)

Wo

rkgr

ou

pEn

terp

rise

Perform Upstream Agile: standup, burndown/burnup, and short iterations; product

owner; and constant feedback

13%

<20% A few %

32%Perform Upstream + Downstream

Agile: Perform upstream plus continuous integration, continuous

delivery, and test-driven development

Have Scaled Agile across their organization

Have achieved

Enterprise Agility


What’s the Problem?


• Highly optimized silos or work centers make sense for Waterfall

– Trust each work center to build the most “perfect product” to pass to the next work center

– We know this rarely works

• Agile Enterprises optimize for the System

– Identify and fix bottlenecks

– Only improve a work center if it improves the system

Silos of Data and Automation


• The first way:

– Systems Thinking

• The second way:

– Amplify feedback loops

• The third way:

– Culture of continual experimentation and learning

A model for DevOps Empowerment


The First Way: Systems Thinking

• Focus on value streams

• Think globally

• Understand the system as a whole

• Stop problems early and often

Operations

(customer)

Development

(business)

• Centralised source code and environment configuration.

• Consistent and automated release processes.

• Decreased cycle times, faster release cadence.


• Protect the integrity of the entire system of work, versus completion of tasks

• Expose visual data so everyone can see how their decisions affect the entire system

The second way: Amplify Feedback Loops

Operations

(customer)

Development

(business)

• Find and fix defects, performance and availability issues faster.

• Enable specialised functions such as Ops and InfoSec to collaborate more effectively.

• Increase communication across the entire business.

• Get more work done


• Continual experimentation

• Tasking risks and learning from failure

• Repetition and practice - pre-requisites to mastery

• Create rituals that reward risk taking

• Introduce faults to increase resilience

The Third way: Culture of continual experimentation and learning

Operations

(customer)

Development

(business)


What Do We Need?


Key Metrics that Power DevOps

1. Change Volume2. Lead Time (from dev to deploy)3. Deployment Frequency4. Percentage of Failed Deployments5. Build Count By Day6. File Churn7. Build duration8. Cycle time

Traditional agile metrics have focused here

Enterprise DevOps Metrics require consistency across projects, enterprise-wide

visibility, and the ability to drill to project data and people

Open ALMPlatform


• As an executive, I need to see across projects to determine the relative success of different teams.

Portfolio Dashboards

Any metric is more valuable when it is put into context, and compared with other teams.


• How many user stories and new lines of code are being deployed?

Change Volume

The key is to understand the risk of a given deployment – more user stories and more lines of code adds risk.


• How long does it take for a new requirement/story/task tp go from the time when it hits development to when it successfully gets deployed into production.

Average Age (from development to deployment)

Average age is an important indicator of efficiency in the process – when tracked using value stream mapping, it can help the team to visualize areas in the process which need improvement. Cycle time should reduce as the team gets a better hold of the lifecycle.


• How often is the team deploying new code?

Team Deployments

For an Agile team delivering code in 2 or 4-week sprints, this metric should become predictable, and correlate closely, if not exactly to those sprints.


• On average, how long are builds taking to complete?

Build Duration

This metrics should remain stable over time. Spikes in build duration should be red-flagged and investigated.


• How is the team trending in terms of cycle time, over the lifecycle of a project?

Cycle Time

Technical debt will add to cycle time, but so will general complexity. Most projects start out strong, but over time degrade. Choose the level that is acceptable and seek stability at that level.


Velocity and Estimation

• How are teams doing? Will they finish what is promised?

Velocity will let us know how much work a team can get done, if they are consistent in their estimation and improving; while the burnup will let us know if we are finishing work quicker than it is being added to the queue


How Do We Get There?The Cloud Framework to Power DevOps


Connect Teams, Processes, and Tools


TechM ADOPT Platform with TeamForge

Your data center

Public clouds

Dashboards

Control center for delivery pipelines, with social activity streams and open pluggable API’s

GitEye

Code Search

Code Review

Git + Gerrit

Jenkins

Hudson

Tracker Discussions Wiki Documents Reports File Releases

JIRA

Selenium

PPM

One Platform, Multiple Possibilities

QTP

Platform 4


Tool Snapshot

Project

Dashboard

Agile BacklogsConfiguration

Management

Unit Test / Code

quality tool

integration

Jenkins

integration

Collaboration

Tools

Code Review

Tool

Customized

Reports

Release

Planning

Defect

Tracker

Defect

Tracker/Impediment

Tracker

Infrastructure

Automation

Traceability for

application lifecycle

activities

Task

Tracking /

Managing

Sprint

Platform 4


Scrum

Team 2

Test (T) QA

(Q)

Scrum

Team 1

Dev (D)

T A Prod

Prod (P)

D

Sprint

Build?

Build?

Unit tests

Unit tests

Unit tests

Build?

Continuous Integration

Continuous Testing

Automate

d Tests

Automate

d Tests

Unit tests

Continuous Deployment

Continuous Release & Monitoring

Co

nti

nu

ou

s P

lan

nin

g

Continuous Delivery

Build (B)

Daily DailyDaily/Release

point

• Devops should create a

flow that results in faster

through put time

• Our approach ensures that

we improve the overall

throughput by bringing in

process, tools, automation

and training interventions

where the flow gets stuck

• TechM proposes a holistic

approach to enable Devops

in an organization by

slowing eliminating legacy

practices which don’t

promote collaboration

Focus on Cadence and Orchestration


Simplified process for continuous delivery

Task

to

Package

Idea

to

task

Package

to Deployment

Business

vision

Products

Identification

Product

Roadmap

Product Epics (product

backlog)

Release PlansSprint Plans

(Sprint backlog)Tasks

Design CodeAutomated

Unit test

Automated Code

quality check

Trigger

Integration

Automated

BuildPackage

Check-in to

repository

Deploy to

QA Env*

Manual/ Automated

SIT Testing

Deploy to

UAT Env*

Manual / Automated UAT

Testing

Deploy to Production

Env*

S

E

S EStart End

Automated monitoring in

production

I2T

T2P

P2D

Processes

1

Processes

1


Phased Approach

Isolated Teams

SME Shared

Teams

Continuous

Planning

Continuous

Release and

Monitoring

Continuous

Integration

Continuous

Testing

Continuous

Delivery

Product Teams

Multi Layered

Teams

Multidisciplinary

Project Teams

Demand pipeline

established; Release plans

are developed and

maintained for individual

applications; Projects are

able to establish

traceability across

requirements, design,

development, testing and

defect management is

maintained through out

the lifecycle

Projects are able

demonstrate the

behavior of continuous

feedback; as opposed

to a periodic schedule,

builds are triggered by

every commit to the

single code repository

Project is able to align

itself with enterprise

release cadence;

environments are

provisioned ‘on

demand’; releases

deployed across

multiple environments

including data

migration

Continuous regression

reduces the number of

defects; most

integrated bugs are

fixed regularly to avoid

last minute changes;

test cases reused;

‘non-functional’ code

quality measured and

technical debt

improved

Dev and ops team work

together seamlessly

delivering both projects

and managing

operations. Metrics are

captured to monitor

process performance

and continuous

improvement.

Level 1

Level 2

Level 3

Level 4

Level 5

VALUE

CO

LLA

BO

RA

TIO

N

Capability Model

2

• Our assessment framework is built

on this staged approach. Staged

approach does necessarily mean

that it is a sequence

• TechM would evaluate organization

as-is status against these stages

• As-is assessment would cover

people, process, tools, policies, KPIs

and flow

• As an outcome of the assessment

we would provide the gaps and the

transformation roadmap to cover

these gaps

• The Devops consultant/ Program

manager would work towards

implementing this transformation

roadmap


Transformation Phases


iSPEED tm Methodology

Study Plan Enable and Expand Diffuse

• Review historical data,

documents, tools and processes

• Understand the organization

structure

• Interview key stakeholders

using diagnostic tool

• Prepare business case

• Identify multiple solutions

blueprints

• Prepare transformation

roadmap

• Identify Enterprise Change

Enablement team

• Develop workflows as per

selected blueprint

• Review the workflows with all

stakeholders

• Tune up workflows

• Develop detailed process,

templates, checklists, guidelines

• Modify tool workflows

• Facilitate deployment planning

workshops, prioritize process

activities , prepare deployment

plans

• Enable teams by training and

support

• Identify projects and deploy

• Aggregate data

• Enable Continuous

Improvement

• Quantify benefits and Showcase

projects

• Collate and analyze

performance data

• Benchmark and improve

• Repot

• Establish improvement log

• Continue deployment support

Interview• Interview key

customer

stakeholders

• Understand customer

context

• Abstract key business

drivers

• Enlist the top

priorities

2 to 3 WKS 4 to 6 WKS 8 to 12 WKS OngoingMax 4hrs

Feasibility Decision

• Business case

• Solution blueprint

• Transformation roadmap

• Process Workflows

• Processes, templates, checklist

• Trainings documents

• Project data

• Audit & Management reports

• Management reports

• Improvement log

• Fixed Price

• Fixed Price

• Time and Hire

• Outcome based pricing

• Fixed Price

• Time and Hire

• Outcome based pricing

• Time and Hire

Deliverables

Engagement model


DevOps Transformation Assessment (DOT)

ADCM provides a structured and tested

approach for any organizations Devops

strategy. Outcome of the assessment

provides a roadmap that can be used for

Devops strategy and rollout


DOT Analytics

0

0.5

1

1.5

2

2.5

3

3.5

4

Business ITAlignment

Portfolio planning

Release planning

Capability planning

Capacity planning

Infrastructureplanning

Agile deliveryplanning

Task planning

Value addition

Collaboration

Institutionalization

• Each stage has multiple focus areas.

The depiction here is for continuous

planning

• To arrive at the score, TechM

consultants would speak to various

stakeholders and take them through a

series of questions

• Each focus area is evaluates for

• the value addition that it is

currently bringing,

• The collaboration it facilitates

• Level of implementation that

has been achieved

• Focus areas where today nothing exists

is given a score of zero

• Further for each area a detailed

improvement plan is prepared

Continuous planning


DOT Roadmap Sample

Continuous planning W1 W2 W3 W4 W5 W6 W7

Business IT

alignment

People

Process

Tooling

Flow & integration

Performance & outcomes

Deliverables

Portfolio

Planning

People

Process

Tooling

Flow & integration


Deliverables

Release

Planning

People

Process

Tooling

Flow & integration


Deliverables

Capability

Planning

People

Process

Tooling

Flow & integration


Deliverables

Capacity

Planning

People

Process

Tooling

Flow & integration


Deliverables

Infrastructure planning

People

Process

Tooling

Flow & integration


Deliverables

Agile delivery planning

People

Process

Tooling

Flow & integration


Deliverables

Task

planning

People

Process

Tooling

Flow & integration


Deliverables


Thank You

Contact Details:

Kevin Hancock - [email protected]

Sujoy Sen – Tech [email protected]

mailto:[email protected]

mailto:[email protected]


© 2014 CollabNet, Inc., All rights reserved. CollabNet is a trademark or registered trademark of CollabNet Inc., in the US and other countries. All other trademarks, brand names, or product names belong to their respective holders.

CollabNet, Inc.8000 Marina Blvd., Suite 600Brisbane, CA 94005

www.collab.net

+1-650-228-2500+1-888-778-9793blogs.collab.nettwitter.com/collabnetwww.facebook.com/collabnetwww.linkedin.com/company/collabnet-inc

http://www.collab.net/

blogs.collab.net

twitter.com/collabnet

http://www.facebook.com/collabnet

http://www.linkedin.com/company/collabnet-inc

blogs.collab.net

blogs.collab.net







metrics to power devops

Technology

agile organizations

agile upstreamagile

agile adoption

agile trainer coach

current spending

quality consulting group

business process effectiveness

software delivery