evidence-based management guide...while the answers to these questions are interesting, they do not...

EVIDENCE-BASED MANAGEMENT GUIDE

EBM

How to improve business results by measuring business value and using empirical management

Scrum.org

August 2018

2

©2018 Scrum.Org. Offered for license under the Attribution Share-Alike license of Creative Commons, accessible at http://creativecommons.org/licenses/by-sa/4.0/legalcode and also described in summary form at http://creativecommons.org/licenses/by-sa/4.0/. By utilizing this EBM Guide, you acknowledge and agree that you have read and agree to be bound by the terms of the Attribution Share-Alike license of Creative Commons.

OVERVIEW

Organizations adopting agile product delivery practices can easily lose sight of their real goal of improving the value they deliver, focusing instead on improving activities and outputs but not business outcomes.

Agileisameanstoanend,nottheenditself;thewholepointofadoptingagilepracticesistoimprovebusinessperformance.Whenorganizationslosesightofthis,managersaskquestionsthatseemsensible,butmightcreateunintendedandundesirableconsequences.Someexamplesofsuchquestionsare…

Whiletheanswerstothesequestionsareinteresting,theydonothelpanorganizationimprovethevalueitdelivers,oritsabilitytodelivervalue.Monitoringonlythedirectuseofpracticesdoesnotprovidethebestevidenceoftheireffectiveness;forexample,trackingaDevelopmentTeam'svelocitysaysnothingaboutwhetherthatteamisactuallydeliveringsomethingthatisusefultocustomersorusers.

Withoutmeasuringvalue,thesuccessofanyagileinitiativeisbasedonnothingmorethanintuitionandassumption.Incontrast,theEvidence-BasedManagement(EBM)approachmeasuresvaluedeliveredasevidenceoforganizationalagility,andprovideswaystomeasureandimprovetheabilitytodelivervalue.Thisapproachenablesorganizationstomakerational,fact-baseddecisions,elevatingconversationsfrompreferencesandopinionstoempiricalevidence,logic,andinsight.

3©2018 Scrum.Org. Offered for license under the Attribution Share-Alike license of Creative Commons, accessible at http://creativecommons.org/licenses/by-sa/4.0/legalcode and also described in summary form at http://creativecommons.org/licenses/by-sa/4.0/. By utilizing this EBM Guide, you acknowledge and agree that you have read and agree to be bound by the terms of the Attribution Share-Alike license of Creative Commons.

EVIDENCE-BASED MANAGEMENT

Evidence-Based Management (EBM) is an empirical approach that provides organizations with the ability to measure the value they deliver to customers and the means by which they deliver that value, and to use those measures to guide improvements in both.

Figure 1: EBM consists of four Key Value Areas (KVAs).

EachKVAfocusesonadifferentaspectofeithervalue,ortheabilityoftheorganizationtodelivervalue.OrganizationswithoutstrengthinallfourKVAsmaydelivershort-termvalue,butwillnotbeabletosustainit.Deliveringvalue,happystakeholders,andsatisfiedemployees(CurrentValue)areimportant,butorganizationsmustalsoshowthattheycanmeetmarketdemandwithtimelydelivery(Time-to-Market)whilebeingabletosustaininnovationovertime(AbilitytoInnovate).Continuedinvestmentintheproductisjustifiedbasedonmeasuresofas-yet(UnrealizedValue)thatcouldberealizediftheproductpossessedtherightcapabilities.

4


Current Value (CV) Reveals the value that the product delivers to customers, today

ThegoaloflookingatCVistomaximizethevaluethatanorganizationdeliverstocustomersandstakeholdersatthepresenttime;itconsidersonlywhatexistsrightnow,notthevaluethatmightexistinthefuture.Questionsthatorganizationsneedtocontinuallyre-evaluateforcurrentvalueare:

1. How happy are users and customers today? Is their happiness improving or declining?

2. How happy are your employees? Is their happiness improving or declining? 3. How happy are your investors and other stakeholders? Is their happiness improving

or declining?

AvarietyofthingscanimproveCV:improvingusability,improvingcustomeroruseroutcomes,evencreatingahappierworkplace.LookingatCVfromtheperspectivesofcustomersorusers,aswellasinvestors,isobvious,butconsideringemployeeattitudesrecognizesthatemployeesareultimatelytheproducersofvalue,andhappyemployeesaremoreengagedandproductive.ExampleKeyValueMeasures(KVMs)foreachKVAaredescribedintheAppendix.

Time-to-Market (T2M) Expresses the organization’s ability to quickly deliver new capabilities, services, or products

ThegoaloflookingatTime-to-Marketistominimizetheamountoftimeittakesfortheorganizationtodelivervalue.WithoutactivelymanagingTime-to-Market,theabilitytosustainablydelivervalueinthefutureisunknown.Questionsthatorganizationsneedtocontinuallyre-evaluatefortimetomarketare:

1. How fast can the organization learn from new experiments? 2. How fast can you learn from new information and adapt? 3. How fast can you deliver new value to customers?

AvarietyofthingscanreducetheTime-to-Market:everythingfromremovinginternalcommunicationbottleneckstoimprovingdeliverypipelineautomationtoimprovingapplication


maintainabilityandremovingtechnicaldebt;anythingthatreducestimespentwaitingortimespentperformingwork.

Ability to Innovate (A2I) Expresses the ability of a product development organization to deliver new capabilities that might better meet customer needs

ThegoaloflookingattheA2Iistomaximizetheorganization’sabilitytodelivernewcapabilitiesandinnovativesolutions.Organizationsshouldcontinuallyre-evaluatetheirA2Ibyasking:

1. What prevents the organization from delivering new value? 2. What prevents customers or users from benefiting from that innovation?

Avarietyofthingscanimpedeateamfrombeingabletodelivernewcapabilitiesandvalue:spendingtoomuchtimefixingdefectsorreducingtechnicaldebt,havingtomaintainmultiplecodebranchesorproductversions,acomplexormonolithicapplicationarchitecture,insufficientproduct-likeenvironmentstoteston,lackofoperationalexcellence,poorcodemanagementpractices,lackofdecentralizeddecision-making,inabilitytohireandinspiretalented,passionateteam-members,andsoon.Aslow-valuefeaturesandsystemicimpedimentsaccumulate,morebudgetandtimeisconsumedmaintainingtheproductorovercomingimpediments,reducingitsavailablecapacitytoinnovate.Inaddition,anythingthatpreventsusersorcustomersfrombenefittingfrominnovation,suchashardtoinstallsoftwareorlackofcapabilitiesthatwouldbecompellingenoughtowarrantinstallingthesoftware,willalsoreduceA2I.

Unrealized Value (UV) Suggests the potential future value that could be realized if the organization could perfectly meet the needs of all potential customers

ThegoaloflookingatUnrealizedValueisfortheorganizationtomaximizethevaluethatitrealizesfromtheproductovertime.Questionsthatorganizationsneedtocontinuallyre-evaluateforunrealizedvalueare:

1. Can any additional value be created for our organization in this market or other markets?

2. Is it worth the effort and risk to pursue these untapped opportunities? 3. Should further investments be made to capture additional Unrealized Value?

6


Thesequestionscan’tbecompletelyansweredinisolationfromtheUVofotherproducts;thedecisiontoinvestinoneproductmeansnotinvestinginothers.ConsideringbothCVandUVprovidesorganizationswithawaytobalancepresentandpossiblefuturebenefits.Forexample,aproductmayhavelowCV,becauseitisanearlyversionbeingusedtotestthemarket,butveryhighUV,indicatingthatthereisgreatmarketpotential.InvestingintheproducttotrytoboostCVisprobablywarranted,giventhepotentialreturns,eventhoughtheproductisnotcurrentlyproducinghighCV.

Conversely,aproductwithveryhighCV,largemarketshare,nonearcompetitors,andverysatisfiedcustomersmaynotwarrantmuchnewinvestment;thisistheclassiccashcowproductthatisveryprofitablebutnearingtheendofitsproductinvestmentcycle.

Leading and Lagging Indicators LeadingindicatorsdetectchangesinKVMswithrelativerapidity,enablingfasterresponse,whilelaggingindicatorsmayonlyshowchangesafteralongdelay.Manyindicatorsareneitherintrinsicallyleadingorlagging,butonlybecomeoneortheotherdependingonhowfrequentlytheyaremeasured.Thus,whenrevenueismeasuredeveryday,itisaleadingindicator,butwhenitcanonlybemeasuredmonthlyorlessfrequentlyitbecomesalaggingindicator.

Trueleadingindicatorscanbeusedtoinferlikelyoutcomesreportedbylaggingindicators,sothatbuildandintegrationsuccessisafairpredictorforthelikelystabilityandpredictabilityoftheoverallrelease.

Leadingindicatorsofcustomersatisfactionarehardtoobtain,butusagedatacanserveasaproxy.Transactionabandonratescangiveinsightintosuccessfulcompletionofanactivity,andsimpleusagedatacanatleastinformwhetherfeaturesarebeingused,andcangiveinsightintohowproductsarebeingused.

Measuresofsatisfactionmaybeleadingorlaggingindicators,dependingonhowlongittakestogathertheinformation.Forexample,leadinginformationaboutEmployeeSatisfactioncanbegatheredusingsomethingassimpleasa“Howwasworktoday?”😀😐🙁buttonthattheycanpressastheyleavefortheday.

TruelaggingindicatorslikeRevenueperEmployee,ortheProductRatio,whichmeasurestheprofitabilityoftheproduct,areinfluencedbysomanycontributingfactorsthattheyprovidemostlyonlygeneralideasofhowwelltheproductiscreatingvalue.

Wheninspectingindicators,acommonchallengemightbetodistinguishingbetweencorrelationandcausality.Itrequireshumanjudgmentandbottom-upintelligencetohavemeaningfuldata-drivenconversationtohypothesizeabouttherootcausesbehindshiftsinindicators.Theseconversationsmighttriggernewactionableinsights,hypothesesandexperimentsinserviceofcreatingthedesiredshiftintheindicators.


How to Improve Empirically using EBM “Ifyoudon’tknowwhereyouaregoing,anyroadwillgetyouthere.”–LewisCarrollJuststartingtomeasuringKVAscandrivesomeimprovement,becauseitwillshowyouimmediatelywhereyouhaveimprovementopportunities.Usingamoresystematicapproachcanprovideevenbetterresultsbyenablingorganizationstocontinuouslylearnandimprovethevaluederivedfromsoftwareinvestments.Toproducegenuineandlonglastingimprovements,establishalearningloopasoutlinedbelow.

1. Quantify Value ThefirststepintheEBMlearningloopistoquantifyvalueintheformofKVM’s.TheprocessofdefiningandaligningaroundKVM’sitselfmightbevaluabletoanorganization,becauseitcancreatetransparencyaroundwhatisbeingoptimized.

2. Measure KVMs ThenextstepintheEBMlearningloopistoestablishinitialvaluesorabaselinemeasurefortheKVMsofinterest.Thisstepprovidesaninitialviewoftheviabilityoftheproductandtheabilityoftheorganizationtodelivertheproduct.Itcreatestransparencyaroundrelativestrengthsandweaknesses.

3. Select KVAs to improve Withaclearviewofcurrentorganizationalvalueandanunderstandingofthemeasuresthatrevealit,managersofsoftwareorganizationscannowmakeinformeddecisionsaboutwhichKVAswouldbemostvaluabletochange.Don’ttrytoaffecttoomanyKVAswithinasinglelearningloop.Makingsmallincrementalchangesandthenquicklymeasuringtheresultisbetterthandelayingimprovementandmeasurementbychangingtoomanyfactorsatonce.Changingmanyfactorsatthesametimecanalsoposechallengeswhentryingtoestablishcausalitybetweenoutcomesandactivities.Shortcycleswithfewchangesarethemosteffectivewaytomakesustainableimprovementstoanorganization’soverallagility.

4. Conduct practice experiments to improve targeted KVAs AfterchoosingadesiredKVAtoimprove,select,atmost,afewpracticesthatyouthinkwillimprovetheassociatedKVM(s),andrunanexperiment.Forexample,asoftwareorganizationthatwantstoincreasequalitymightchoosetofocusonreducingtheDefectsKVM.Theymaydecidetotryimplementingtest-firstpracticestoincreasethetestcoverageandthequalityfocusofthedevelopmentteams.

8


5. Evaluate results Oncetheresultsoftheexperimentaremeasured,theyshouldbecomparedwiththeKVMvaluesbeforethechangesweremade(seeFigure4).Iftheexperimentproducedimprovedmeasured,thechangeiskept,otherwiseotherimprovementsaretried.ThelearningloopcontinuesuntiltheKVAreflectthedesiredresults.

Conclusion “Ifyoucan'tmeasureit,youcan'timproveit.”-PeterDruckerEBM’sKeyValueAreasprovideaholisticperspectiveonproductdeliveryperformance.CurrentValueismostimportant,foraproductthatoffersnovaluetoitscustomersoruserswon’tsurvivelong.Customer/userexperienceisonlypartofthepicture;sustainingandimprovingvaluetocustomersisimpossiblewithouthappy,engagedemployees,andwithouthappyinvestors,thoseemployeeswilllackthecapitaltheyneedtomakeimprovements.Rapidlyimprovingthevalueaproductdeliversrequiresfrequentdeliveryofnewvalue,whichmeansimprovingtheTime-to-marketoftheproduct.Thismeansmorethansimplyworkingfaster;infactworkinghardertogofasterisrarelysustainable.Instead,understandingandremovingimpedimentstofasterdeliveryisessentialtodeliveringatfastercycles.FasterTime-to-marketisnotthewholestory.Fastreleasecyclesthatdeliveronlyverysmallimprovementsdolittletorapidlyimprovethevaluedeliveredbyaproduct.Theabilityfortheorganizationtoinnovateisalsodeterminedbyitsabilitytodeliversignificantinnovationineachrelease.Measuringthisabilitygivesorganizationstheinsightstheyneedtobeabletoremovebarriersthatkeepthemstuckinthepast.Improvingorganizationalperformanceisalsoacyclic,iterativeprocess:measuringcurrentconditions,settingperformancegoals,formingsmallimprovementexperimentsthatcanbequicklyrun,andthenmeasuringagaintogaugetheeffect,thenrepeating,continuously.

End Note Evidence-BasedManagementisfreeandofferedinthisGuide.AlthoughimplementingonlypartsofEBMispossible,theresultisnotEvidence-BasedManagement.


Acknowledgement Evidence-BasedManagementwascollaborativelydevelopmentbyScrum.org,theProfessionalScrumTrainerCommunity,theEngagementManagercommunityandKenSchwaberandChristinaSchwaber.

10


Appendix – Example Key Value Measures

Current Value (CV) KVM Measuring:

Revenue per Employee

Gross Revenue / # of employees

This varies significantly by industry. The ratio is a key competitive indicator within an industry.

Product Cost Ratio Total expenses and costs for the product(s)/system(s) being measured, including operational costs compared to revenue.

Employee Satisfaction

Some form of sentiment analysis to help gauge the employee engagement, energy, and enthusiasm.

Engaged employees that know how to maintain, sustain and enhance are one of the most significant assets of an organization.

Customer Satisfaction

Some form of sentiment analysis to help gauge the customer engagement and happiness with the product.

Customer satisfaction reflects sound management, solid soft product and software, and creative, fulfilled employees. Satisfied customers contribute to growth.

Usage Index

Measurement of usage, by feature, help to infer the degree to which customers find the product useful.

Trends over time can help to spot which features are no longer being used, or those that were once used but not used any longer. Measuring time spent using features also helps to inform whether actual usage meets expectations on how long users should be taking with a feature; shorter or longer periods of time may indicate usability issues and improvement opportunities.

Time-to-Market (T2M) KVM Measuring:

Build and integration frequency

The number of integrated and tested builds per time period.

For a team that is releasing frequently or continuously, this measure is superseded by actual release measures, but teams just starting to improve often have to start with stabilizing and improving the build and integration process.

Release Frequency The number of releases per time period, e.g. continuously, daily, weekly, monthly, quarterly, etc.


This reflects the time needed to satisfy the customer with new and competitive products.

Release Stabilization Period

The time spent correcting product problems between the point the developers say it is ready to release and the point where it is actually released to customers.

This measure reflects the impact of poor development practices and underlying design and code base. Stabilization is a drag on competition that grows with time.

Mean Time to Repair

The average amount of time it takes from when an error is detected and when it is fixed.

This reflects how the efficiency of an organization fix an error.

Cycle Time

The amount of time from when work starts on a release until the point where it is actually released.

This measures reflects an organization’s ability to reach its customer.

Lead Time

The amount of time from when an idea is proposed or a hypothesis is formed until a customer can benefit from that idea.

Lead time may vary based on customer and product. It is a contributing factor for customer satisfaction.

Time-to-Learn

The amount of time from when an idea is proposed or a hypothesis is formed until a result is measured.

The ability to learn from actual customer usage and feedback and the ability to adapt to that feedback increases an organization’s business value.

Ability to Innovate (A2I)

KVM Measuring:

Usage Index Measurement of features in the product that are frequently used.

Features that are rarely or never used represent wasted ability to innovate, and often further consume time and effort when they continue to be maintained and tested.

Innovation Rate The percentage of effort or cost spent on new product capabilities, divided by total product effort or cost.

This provides insight into the capacity of the organization to deliver new product capabilities.

12


Defect trends Measurement of change in defects since last measurement.

A defect is anything that reduces the value of the product to a customer, user, or to the organization itself. Defects are generally things that don’t work as intended, which lead to poor customer or user experiences.

On-Product Index The percentage of the total user base that is using the current version of the product.

Low values may indicate that the product is difficult to install or that the upgrade process is complex or risky, or both, or reflect degraded usability, or loss of capabilities between versions, or some combination of all of these.

Installed Version Index The number of versions of a product that are currently being supported.

The higher the value, the more effort the organization has to spend supporting and maintaining older versions of software.

Technical Debt A concept in programming that reflects the extra development and testing work that arises when “quick and dirty” solutions result in later remediation.

Technical debt can be measured using inspection, by developer self-identification, and by code scanning tools.

Production Incident Trends

The number of times the Development Team was interrupted to fix a problem in an installed product.

The number and frequency of Production Incidents are a good indicator of the stability of the product.

Active code branches, time spent merging code between branches

These measures are similar to the Installed Version Index, since different deployed versions usually have separate code branches.

Even within a single release, long-lived code branches usually result in excessive time spent merging changes and integrating the product.

Time spent context-switching

Interruptions reduce team member productivity and reduce the time available for innovation work.

Number of meetings per day per person, and the number of times a day team members are interrupted to help people outside the team can give simple insight into the magnitude of the problem.


Unrealized Value (UV) KVM Measuring:

Market Share

The relative percentage of the market controlled by the product.

Can be measured in terms of users or in terms of revenues. Trends are instructive.

Customer or user satisfaction gap

The difference between a customer or user’s desired experience and their current experience.

Generally obtained by surveys and/or focus groups.

Comparative product strength (weakness)

A comparison between two products made by comparing their respective satisfaction gap scores.

Weaker products may be vulnerable to market share losses.

evidence-based management guide...while the answers to these questions are interesting, they do not...

Documents