LEAN ANALYTICStechtalk @ ferret

based on “Lean Analytics” by Croll&Yoskovitz and wikipedia information

LEAN SOFTWARE DEVELOPMENT

The term lean software development originated in a book “Lean Software Development: An Agile Toolkit”, written by Mary Poppendieck and Tom Poppendieck.

LEAN PRINCIPLES

• Focus on the customer and eliminate waste Everything not adding value to the customer is considered to be waste. If some activity could be bypassed or the result could be achieved without it, it is waste.

• Amplify learning The best approach for improving a software development environment is to amplify learning. Instead of adding more documentation or detailed planning, different ideas could be tried by writing code and building. A data driven cycle of hypothesis-validation-implementation should be used to drive innovation and continuously improve the end-to-end process.

• Decide as late as possible Delay decisions as much as possible until they can be made based on facts and not on uncertain assumptions and predictions.

• Deliver as fast as possible The sooner the end product is delivered without major defects, the sooner feedback can be received, and incorporated into the next iteration. The shorter the iterations, the better the learning and communication within the team.

• Empower the team “Find good people and let them do their own job”. People do need something more than just the list of tasks and the assurance that they will not be disturbed during the completion of the tasks. People need motivation. The developers should be given access to the customer; the team leader should provide support and help in difficult situations.

• Build integrity in Understanding the problem domain and solving it at the same time, not sequentially. The information flow should be constant in both directions – from customer to developers and back.

• See the whole “Think big, act small, fail fast; learn rapidly”. Larger software = more part developed by different teams, but lean thinking has to be understood well by all members of a project, before implementing in a concrete, real-life situation.

– Peter Drucker, management consultant

“If you can’t measure something, you can’t manage it.”

LINE IN THE SAND

Measurable target that everyone (incl. executives) agrees to.

GOOD METRICS• Qualitative vs. quantitative

quantitative data answers “what” and “how much”, qualitative data answers “why”. Quantitative data has no emotions.

• Vanity vs. actionable Use metrics you can act on. Vanity metrics might make you feel good, but they don’t let you act. For instance, “total signup” vs. “percent of active users”.

• Exploratory vs. reportableExploratory metrics are speculative, while reporting metrics keep you nearby of normal, day-to-day operations.

• Leading vs. laggingLeading metrics give you predictive understanding of the future, while lagging metrics explain the past. Leading metrics are better because you have time to act on them.

• Correlated vs. causal If 2 metrics change together, they’re correlated, but if one causes another to change, they’re causal. Try to find a causal relationship between smth. you want and smth. you can control.

–Pirate Metrics by Dave McClure that every startup needs to watch

“Acquisition, activation, retention, revenue, and referral - AARRR.”

RIGHT METRIC FOR RIGHT NOW

• Acquisition How do users aware of you? Metrics: traffic, mentions, cost per click, search results, cost of acquisition, open rate.

• Activation Do drive-by visitors subscribe, use etc.? Metrics: signups, completed on boarding process, used service at least once, subscriptions

• Retention Does a one-time user become engaged? Metrics: engagement, time since last visit, daily/monthly active use, churns

• Revenue Do you make money from user activity? Metrics: customer lifetime value, conversion rate, shopping cart size, click-through revenue

• Referral Do users promote you product? Metrics: invites sent, viral coefficient, viral cycle time

ONE METRIC THAT MATTERS

The OMTM is the one number you’re completely focused on above everything else for your current stage.

• It answers the most important question you have you need to identify the riskiest areas of your business, and that’s where the most important question is.

• It forces you to draw a line in the sand and have clear goals after you’ve identified the key problem, you need to set goals.

• It focuses the entire company Use OMTM as a way of focusing you entire company. Display it throughout web dashboards, on TV screens, or regular emails.

• It inspires a culture of experimentation It’s critical to move through the build-measure-learn cycle as quickly and as frequently as possible. To succeed on that, you need to actively encourage experimentation.

When you are focused on retention, you may be looking on churn, and experimenting with pricing, features, improving customer support etc.

the One Metric That Matters changes over time

MAIN STAGES

You can’t just start measuring at once. You have to measure your assumptions in the right order.

• Empathy Go inside target market and sure you’re solving a problem people care about in a way someone will pay for.

• StickinessIt comes from a good product. You need to find out if you can build an acceptable solution to the problem you’ve discovered.

• Virality Once you’ve got a product that’s sticky, you need care about acquisition.

• Revenue You’re giving away free trials, free drinks, or free copies. Now you’re focused on maximising and optimising revenue.

• Scale With revenues coming in, it’s time to move from growing your business to growing you market.

EMPATHY

• Find a problem to fix: - The problem painful enough - Enough people care - They are already trying to solve it

• Talk with people and rate interviews

• Mostly qualitative metrics here. Be honest with yourself.

STICKINESS

• Daily, weekly, monthly active users

• Time to become inactive

• Number of reactivated inactive after email

• Time to spend with feature

7 QUESTIONS TO ASK BEFORE BUILDING A FEATURE

• Why will it make things better? You can’t build a feature without a reason. Ask yourself “Why will it make it better?” and write out a hypothesis.

• Can you measure the effect of the feature? Feature experiments require that you measure the impact of the feature. That impact has to be quantifiable.

• How long will the feature take to build? Time is resource you never get back. If something is going to take too much to build, break it into small parts or test the risk with the prototype first.

• Will the future over complicate things? Complexity kills products. When discussing a feature with your team, pay attention to how it’s being described. “And” is enemy of success.

• How much risk is there in this feature? Building a new feature always comes with some amount of risks - technical risk, user risk, risk of influence to further development etc.

• How innovative is the new feature? Not everything is innovative, but consider innovation when prioritising feature development. Generally, the easiest thing to do rarely have a big impact.

• What do users say they want? Users are important as well as their feedback. But relying on what they say is risky. Be careful about over prioritising based on user input alone.

VIRALITY

• Invitation rate - the number of invites sent divided by the number of users you have

• Acceptance rate - the number of signups or enrolments divided by the number of invites

• Viral coefficient (OMTM) - the number of new customers that each existing customer is able to successfully convertViral = invitation rate x acceptance rate

REVENUE

• QRR(x) - the quarterly recurring revenue for quarter x

• QExpSM(x) - sales and marking expense for the quarter x

• Ratio of inputs to outputs (OMTM) q = [QRR(x)-QRR(x-1)] / QExpSM(x-1) You have problems if q < 0.75

SCALE

• On this stage you already know your product and market. Your metrics now should be focused on the health of your ecosystem and your ability to enter new markets.

• Customer acquisition payback (OMTM) - the customer acquisition cost divided by the customer lifetime value

• Check metrics across channels, regions, and marketing campaigns

• Try to understand if you’re focused on efficiency(try to reduce cost) or differentiation (try to increase margins).

CONCLUSION

• Make Sure Goals are Clearly Understood To prove the value of analytic-focused company, any project needs to have clear goals. Everyone involved in the project needs to be aligned around the goals.

• Make Things Simple to Digest A good metric is the one that’s easy to understand at glance. Metric can be extremely valuable, but used incorrectly they will lead down the wrong path.

• Ensure Transparency If you are going to use data to make decisions, it’s important that you share the data and methodologies.

• Don’t Eliminate your Gut Lean Analytics isn’t about eliminating your gut, it’s about proving your gut right or wrong.

• Ask Good Questions You don’t need to guess, you need to know where to focus. You don’t know all answers, but you should know the right questions to ask.

FUTHER READINGS

• Lean Software Development: An Agile Toolkit by Mary & Tom Poppendieck

• Lean Analytics by A. Croll, B. Yoskovitz

• The Lean Startup by Eric Ries

• Running Lean by Ash Maurya