doing analytics right - selecting analytics

Doing Analytics RightPart 1 – Selecting Analytics

Look Whose Talking

@tasktop

• Dave West – Chief Product Officer, Tasktop– Leads product development for Tasktop– Former RUP product mgr and Forrester

Analyst– [email protected] | @davidjwest

• Dr Murray Cantor – Senior Consultant, Cutter Consortium – Trying to improve our industry with

metrics– Former IBM Distinguished Engineer– [email protected] | @murraycantor

mailto:[email protected]


This is the first of a series:

1. Selecting Analytics. Murray Cantor, Dave West.– Aligning the choice of measures with your organization’s efforts and goals

2. Designing and automating analytics. Murray Cantor, Nicole Bryan.– A straightforward method for finding your analytics solution

• The dashboards,

• the required data, and

• an appropriate choice of analytical techniques and statistics to apply to the data.

3. Building the Analytics Environment. Murray Cantor, Nicole Bryan.– The data solution architecture and stack

– How Tasktop can help.

3

http://tasktop.com/webinars

Software is eating the world….

227

4

3

35

200

500K

#1

196,000 2,000,000

Vancouver HQ

Offices in Austin,

Boston, UK

22

Providing some context

Created first software lifecycle bus

2011

Global 500 customers3 OEMs

Created Task Management Category

2009

1000+ customers,3 OEMs

De facto ALM integration for developers

2007

1.5M OSS DLs/month,Majority ISVs

Defined Software Lifecycle Integration

2013

Emerging ALM discipline, new product category

Created first lifecycle data aggregator

2014

Infrastructure for software lifecycle analytics

So…. You have Data, then what…

©2015 Murray Cantor

Metrics are essential for sense and respond loops to

achieve goals

When choosing measures

consider whether

• The measures let you know how

whether you are achieving the

goals?

• You have a way to respond to the

measures?

8

Avoid building dashboards just to use the data


The two key considerations to picking your measures:

9

Mixtures of work efforts

Level of the organization

Work item, artifact

completionStaff member Commits to

Project, product deliveryProject manager, team

leadCommits to

Efficiency, value deliverySenior manager Commits to

Profit, return on

investmentLine of business executive Commits to


The two key considerations to picking your measures:

10



Work item, artifact



leadCommits to


Profit, return on



Kinds of Development Efforts: What is your mix?

11

1. Low innovation/high

certainty

• Detailed understanding

of the requirements

• Well understood code

2. Some innovation/

some uncertainty

• Architecture/Design in

place

• Some discovery required

to have confidence in

requirements

• Some

refactoring/evolution of

design might be required

3. High innovation/High

Uncertainty

• Requirements not fully

understood, some

experimentation might be

required

• May be alternatives in choice

of technology

• No initial design/architecture


The methods landscape

12

Kanban

Lean startup: MVP

Agile, Scrum

Product Development Flow

Systems/Software Engineering

Lean Software

Podular Org.

Liminal Thinking.

Technical Debt Management

Iterative learning: Updating estimates and

plans in the face of evidence

DevOps/Continuous Delivery


Different disciplines apply to different parts of the landscape

13

Lean Manufacturing

Innovation Management

Queuing Theory

Systems Theory

Toyota Management System

Agile Management

Bayesian Nets

Analytics

Quality Control


The different types of efforts requires different sorts of analytics

Descriptive

Bayesian

Descriptive• Counts, percentages

• Averages (means, medians)

• Percentiles

Bayesian• Probabilities

• Risks

• Uncertainties


Descriptive Example: A Value Stream model for routine efforts

15

Control challenges• Random arrival intervals

• Variation of effort to address work items (unlike standardized

manufacturing)


Descriptive example: Cycle times

16

These will be described in

more detail in next webinar


Bayes is the way for development teams and

management to deal with uncertainties

In types II and III development, quantities such as time, cost

to complete, and velocity are not known for certain.

• There is not enough known to make exact predictions

• You need to utilize the actual data you produce sprint by sprint

Bayesian analysis is the centuries old method for rigorously

dealing with with uncertain quantities.

Bayesian analytics allows everyone on the team to learn

together.

17

Attributes of Bayes:

Uncertain quantities are specified probabilities

The probabilities capture both the best/worst estimates and the level of uncertainty

The probabilities/beliefs are updated as information, evidence comes in.

The probability distributions can be “added,” “multiplied,” etc.


Bayesian Example

18

This will be described in more

detail in next webinar


Different Types, Different Analytics (examples)

Type 1 Type 2 Type 3

Goals Efficiency

Efficiency

Efficiency

Timely delivery of

value

Innovation

Organization Continuous delivery

teams

Integrated horizontal

teams

Small expert teams

Work Style Backlog management Scrum Lean Startup/MVP/

Experimentation

Challenges Timeliness vs

utilization

Prioritization

Business/IT

alignment

Feature selection

Pivoting

Analytics Flow control

Cycle times

Cost of delay

Costs of delays

Cycle times

Time/cost

probabilities

Time/cost

probabilities

Value at delivery

19


The two key considerations to choosing your measures:

20



Work item, artifact



leadCommits to


Profit, return on



Different levels, different goals

21

Work item, artifact


Project, product deliveryProject manager,

team leadCommits to


Profit, return on



Analytics useful for aligning goals

For each level to meet its goal, the

leader is dependent on the lower

level.

So, the leader seeks commitments

from that layer. Meeting those

commitments becomes the goal

of the next layer.

Hence the analytics serve to

integrate the organization

22


Goals, feedback loops (examples)

Type 1 Type 2 Type 3

Line of Business

Executive

Profits, returns on assets for lob, mission fulfillment

Dev VP, CIO, … Costs Returns on assets, investment for

division

Meeting cost, schedule commitments

for organization.

Project manager,

team lead

Throughput in the

face of variation of

arrivals, size of

work items

• Throughput

• Productivity

• Meeting cost,

schedule

commitment

for team

Meeting cost,

schedule

commitment for

team

Staff member Productivity = (Completion of work items)/(complexity, difficulty)

23

The details can vary with the enterprise mission


To summarize

There is no one-size fits all choice of

measures

Measures must be part of some

feedback, sense and respond loop

Choice of measures Depends chiefly

on

• Mixture of work

• Level of organization

Much more detail to follow in next

webinars .

24


Two key principles

• Kelvin’s Principle: “To measure

is to know. If you can not

measure it, you can not improve

it”

– Measures are part of control

loops

• The converse principle: “Don’t

bother to measure what you do

not intend to improve”

– Find a small set of measures, not

a long laundry list25


Choosing metrics big picture

Agree on goals

- Depends on the levels and mixture of work

Agree on the how they fit into the loop

1. “How would we know we are achieving the goal”

2.” What response we take?”

Determine the measures needed to answer the questions

- Apply the Einstein test (as simple as possible, but no simpler)

Specify the data needed to answer the questions

Automate collection and staging of the data

26

Today

Later

Integration Maturity Model

This is the first of a series:

1. Selecting Analytics. Murray Cantor, Dave West.– Aligning the choice of measures with your organization’s efforts and goals

2. Designing and automating analytics. Murray Cantor, Nicole Bryan.– A straightforward method for finding your analytics solution

• The dashboards,

• the required data, and

• an appropriate choice of analytical techniques and statistics to apply to the data.

3. Building the Analytics Environment. Murray Cantor, Nicole Bryan.– The data solution architecture and stack

– How Tasktop can help.

29

http://tasktop.com/webinars

Stay in touch

@tasktop

[email protected]

@davidjwest

[email protected]

@murraycantor



@tasktop@cuttertweets