metrics driven prioritization

Download Metrics Driven Prioritization

Post on 14-Jun-2015




0 download

Embed Size (px)


Prioritization is one of the toughest aspects of software product-development. In these days of being agile and running lean, we don't want to prioritize our backlog by fiat, politics, or gut-feel. Teams need to be aligned to accomplish core business objectives and need tight feedback loops with the customer and stakeholders in order to achieve those objectices. To accomplish this level of cross-functional product development maturity, we must combine data from a number of sources to quantitatively optimize development. In this talk, I will outline an approach we are using at Neo to help our clients integrate business metics and probabilistic modeling into their prioritization process. Leveraging data from business KPIs, the product development team's own performance metics, and feedback from in-production experiments and A/B tests, I'll illustrate a straightforward yet rigorously quantitative method of prioritizing feature development to focus on achieving business and product outcomes, not simply producing technology outputs.


  • 1. Metrics DrivenPrioritizationEngineering For Product Success #QConSF 2014Sam McAfee, Principal - @sammcafee

2. Innovate or die. 3. Who is this guy? Dot-com boom survivor. Ran a dev shop for 10 years after that. Agile has always been the norm. Read 4 Steps before #LeanStartup. Participated in 3 different startups. All dead. Ran the engineering team. Learned LeanUX from @clevergirl herself. Now a Principal at Neo Innovation ( every word! @sammcafee #qconsf 4. Slides available 5. If you build itIs anyone even going care about this product? 6. Nope. 7. Competing priorities 8. Everyone loves quarterlyplanning meetings 9. A Metrics Driven Approach 10. Pre-requisitesWhat is product success, anyway? 11. A successful product solvesa painful problem for aspecific customer in asizable reachable market. 12. UsableDoes it delight the user?FeasibleCan we build it?12ValuableDoes anyone want it?3 13. Test Your Assumptions Use lightweight personas to test the customer segment. Use problem interviews to test the problem. Use sketches and paper prototypes to test the proposed solution. Use landing pages to test the market. And do all this before you build anything! 14. Cross-functionalThe team includes design, engineering, and product.CollocatedEvery works in the same location.DedicatedThe team only works on one product at a time.Assemble the Team 15. A Metrics Driven Approach 16. Measure the BusinessUse an economic framework to measure theimpacts of your decisions on core KPIs.Measure the TeamsMeasure lead time from concept to cash, and useoutliers to set realistic expectations for delivery.Run ExperimentsYour backlog is a list of experiments you need to runin order to validate your business assumptions. 17. Measure the Business 18. Business Model Canvas 19. Model Your Funnel 20. Build an Economic Framework 21. An Example Product Funnel $20 / month subscription. 10% conversion from landing page. 5% referral rate 10% churn rate (or 90% retention rate) 22. Information Value =Economic impact *uncertainty. 23. An Example Product Funnel Each visit is worth $2.00 Each share is worth $0.10 Customer lifetime value (LTV) is depleted $1 per month, onaverage, per customer. 24. Quantify Cost of Delay 25. Example: Cost of Delay 26. Measure the Teams 27. Most delay in productdevelopment is wait time. 28. Measuring Cycle Time 29. Weighted Shortest Job First 30. Running Experiments 31. The Value of Experiments Each visit is worth $2 (10% of visitors sign up for $20/month). Increasing conversion by 50% is worth $1 per user to us. We have only a 10% chance of being able to increase it by 50%. Historically, only 10% of our experiments on that page succeed. $2 * 1.5 * 0.1 * 0.1 = $0.01 = experiment is only worth a penny!! 32. Prioritizing an experimentrequires data from thefunnel, the team, and anypast experiments. 33. Thank you!@sammcafee 34. Slides available