measuring the agile process improvement

11th Central and Eastern European Software Engineering Conference in Russia - CEE-SECR 2015

October 22 - 24, Moscow

Konstantin Savenkov

Measuring the agile process improvement

ContextThere’s no universal solution, everything matters in its

contextBookmate is a B2C content service:

lots of stakeholderslots of external partieslots of urgent stufflots of experimentsrapid team growth

A way different from: MVP, B2B, outsourcing

The goalsSubscribers

New Recurring Reactivated

Traffic Conversion Retention Reactivation

Engi-neer-ing

Marketing Product Sales Business Development Content & Editorial

Stakeholders and engineeringProduct

Marketing

BD & Sales

Content

Engineering

iOS Android Web

Backend

OpsContent Ingestion

Bookmate Publisher

Windows2-weekcycle

How things ideally would happen

ROADMAPS DESIGN DEVELOPMENT QA SHIPPED

PLANNING

How things actually happenINFLATING EFFORT

ROADMAPS DESIGN DEVELOPMENT QA SHIPPED

BACKLOG

NEW STUFF

TECH. DEBT

PLANNING

BUGS

UX

SOFTWARE OPS

URGENTSTUFF

UNCLEAR DESIGN

UNCLEAR TECH

ITERATING

Continuous improvementOverall productivity KPI is clear: shipped/committed ratioWithout actionable metrics, improvement involves too

much emotion and causal attributionThe metrics should be:

end-to-end (any change in KPI reflected by metrics)invariant to story size and story point valueallow for measuring external factors (stakeholders)

What’s out there? stories committed / stories completed technical debt management team velocity story cycle time CAT/QA cycle time estimation accuracy defects per release cycle defects per story point test cases run

DEPEND ON STORY SIZEDEPEND ON STORY POINTA LOT OF GAPS

What do we use

PRODUCTIVITY

Estimated effort on

planned and doneAll available effort

ACCURACY

Estimated effort on

planned and done

Actual effort on

planned and done SHIPABILITY

Actual effort on

planned and doneEffort spent

on plannedPLANABILITY

Effort spent on planned

Actually spent effort

UTILISATION


All available effort

Issues we discover

ACCURACY

Estimated effort on

planned and done

Actual effort on

planned and done SHIPABILITY

Actual effort on

planned and doneEffort spent

on plannedPLANABILITY



UTILISATION



Unexpected vacations, OOO etc

Feasibility of all sort of buffers (management,

hot bugs etc)

Insufficient lookahead

Improper communication

Unclear design

Unclear tech Software bugs

Team cohesionUnclear tech

Design bugs

What’s good about thatAlmost free (given you estimate effort)Don’t depend on story size (measure effort, not stories)Don’t depend on the story point size (it cancels out)Any issue from the messy picture above is reflected by

those metricsEasily projected back to stakeholders to find external bottlenecks, e.g. BUSINESS DEVELOPMENT


(BD)Actually spent effort (BD)

Track stuff that is easily swept under the rugs:

Next batch

QUALITY

Estimated effort onall bugs


TECH DEBT

Estimated effort on

tech. backlogAll available

effort

Be sane about thatIt doesn’t remove bottlenecks and issuesIt just shows themSometimes it’s enough to solve the issueMost of the times it isn’t

measuring the agile process improvement

Data & Analytics