building a data-driven culture

May 3, 2023Presentation name 1

Building a Data-Driven CultureOlof HoverfältDirector, Sanoma Games September 2, 2015

@Hoverfalt

Sanoma Games is a diverse yet compact and lean unit

B2C & B2B, web & mobile Autonomous unit Key functions “in-house”

Business development

Marketing

Design

Development

Sales

DESIGN

BUSINESS ANALYTICS

UX Business

Being data-driven requires much more than the right tools

http://toolmonger.com/category/85/computer/page/2/

Handheld USB Snake

Scope

...Design Product Marketing Business


Organization supporting lean development

Data-driven culture

Key concepts

Accessible tools

Shared goals

Management fostering self-direction

Key concepts Key concepts

http://theleanstartup.com/

http://www.startupolic.com/launching-your-service-think-about-ux/; https://en.wikipedia.org/wiki/Lean_startup

An MVP is “a version

of a new product

which allows a team

to collect the

maximum amount of

validated learning

about customers with

the least effort"

The MVP approach is useful at all levels of development, from single product feature to new business concepts

Adopt MVP thinking in all development

http://www.yesware.com/blog/how-to-close-more-deals-with-lean-email/

A continuous Build-Measure-Learn cycle enables fast learning but is really sensitive to friction in workflow and cooperation

Ensure all key functions are fully dedicated and the workflow is frictionless




Data-driven culture

Key concepts

Accessible tools

Shared goals



A data-driven culture cannot be created by coercion, but must be built through intrinsic motivation from its visible benefits

Transparency Autonomy Ownership

xxx

xxx

xxx

xxx

xxx

xxx

xxx

xxx

xxx

xxx

Improvement actions

xxx

Revenue

ROS&packag. campaign vol.

Videocampaign vol.

Pricing

Discounts

Campaign volume

Optimization

Direct

Referral

SEO

Ret. visitors

Potential issues

Site problems

Ad blockers

New visitors

Reach

Monetization

Visits

Premiumfill rate

Premium eRPM

Remnant eCPM

Unique visitors

Media sales breakdown by contributing component

Impression success rate

nn%

nn%

nn% mm%

nn%

nn%

nn% lost

Ensure overall understanding of how the business works and of the contribution of and the dependencies between its components. Make deep data available

In order for a team to develop and fine tune an engine, everyone must know how it works

Adoption of a data-driven approach requires for people to see its benefits; how they could kick ass even more at what they do

Image: http://blog.crazyegg.com/lessons/show-benefits/

Make the personal benefits of being data-driven visible.Show how its autonomous pursuit enhances mastery.

Super You!Data-drivenness

You

Ownershipis not about doing what I tell you to do

is not about not doing what I tell you not to dois about doing what I don’t tell you to do

Ensure ownership of benefits, not deliverablesFocus on why, not what




Data-driven culture

Key concepts

Accessible tools

Shared goals



Data-drivenness, at both individual and team level, is a capability that can and should be developed

http://skepdic.com/tencommands.html

+Encourage autonomous reflection but ensure conceptual guidance

Reflection Key concepts

Being data-driven is not only about sensibly answering questions, but about finding them in the first place

It cannot itself be standardized into rules or processes, but it

can be guided by high-level key concepts

A core principle of learning is to be hypothesis-driven. Random experimentation will yield various outcomes but little insight

Make continuous hypothesis-driven split testing the standard approach in all development activities across functions

Hypothesis:[A statement that can be shown true or false]

Design a test and a metric that confirms the hypothesis at a certain pre set limit

Test in a real setting with real usersPay attention also to

what is NOT happening

Confirm or reject the hypothesis based on

the chosen metric and pre set limit

Add the learnings to your body of

knowledge, also the invalidated hypotheses

Build on the knowledge to form the

next hypothesis

Data without insight can be destructive as it may lead development the wrong way look like improvement

DAU(daily active users)

Downloads(i.e. not even activations)

Activity delta(change in user engagement)

Churn(users who quit)

>Vanity metricIndicator of

actual use

>Leading indicator of churn

ActionableLagging indicator

Not actionable

Put a lot of effort into designing KPIs (Key Performance Indicators) that are both sensible and actionable; such that steer development initiatives toward your ultimate business goal

Main KPIs should be static but reflect the current growth phase, while tactical tools can be whatever helps improve the KPIs

High-level (growth-stage-specific) KPIs to continuously optimize

Tactical tools do explore user behavior and to measure impact from change on KPI(s)

http://lanesplitter.jalopnik.com/x-ray-artist-reveals-the-inner-beauty-of-classic-motorc-1698564514




Data-driven culture

Key concepts

Accessible tools

Shared goals



A simple tool that is actually used provides infinitely more value than a sophisticated one that is not

https://xplr.com/xplr-umbrella-dataviz-on-top-of-unsupervised-machine-learning/

>Make sure everyone has easy access to all tools they might need and know how to use them




Data-driven culture

Key concepts

Accessible tools

Shared goals



Goals set at function level can often be unaligned or conflicting, breaking joint effort and leading to sub-optimal performance

UX

Set high level common goals to ensure everyone works and cooperates to optimize the whole as opposed to sub-optimizing their own components

Business Manager goal:Visit eRPM

(effective revenue per session)

Product Manager goal:Avg. DAU(daily active users)




Data-driven culture

Key concepts

Accessible tools

Shared goals



Example:Continuous improvement and A/B testing

A/B testing is about continuous and systematic gradual improvements that over time build increased performance

2.5x+ Improvement

Seemingly small but sustained improvements can have a very large impact on overall performance

49% Improvementmin to max

25% Improvementmean to max

Example:Retention (how well you keep your users)

6.1%Retention D1

Count of users by level

6.1%Retention D1

47%Retention D1

25%Retention D7

6.2%Retention D90

Compare with6.1% D1 Retention

in the previous example

Example:LTV ((Customer) Life Time Value)

User profitability is basically determined by LTV* and CAC*. Being able to even roughly predict LTV can be very insightful.KPIsDAU

Monthly rev.

Retention D1 43 %

Retention D7 24 %

Retention D14 19 %

Retention D30 12 %

Conversion

ARPDAU

ARPPDAU

LTV

CAC

ARPDAU nn € nn €

Extr. tot. days (≤120th) nn nn

LTV nn € nn €

* LTV = (Customer) Life Time Value ; CAC = Customer Acquisition Cost

Example:Liigapörssi (KPIs and tools)

Direct

Org. search

Email

SEM

Display

Other

Our main time unit of comparable comparison is a game “phase”. The key desired user behavior varies within the phase

Free

Trades

1

Pool Free

Trades

2

Pool Free

Trades

6

Pool

Direct

Org. search

Email

SEM

Display

Other

Pay(Revenue)

Join pool(Activation)

Sign up (Acquisition)

Play the whole phase(Retention)

Continue to next phase(Retention)

Sign up or reactivate

(Acquisition)

Desired user behavior

Create pool& invite

(Referral)

Invite friends(Referral)

Make trades(Activation)

Buy trades(Revenue)

Create team (Acquisition)

Phase 1 Phase 2 Phase 3-6Season

AcquisitionActivation

RetentionRevenue

Referral

Acquisition

ReferralRevenue

Retention

Activation Focus area

We use various tools to identify development opportunities, run tests and measure impact on the main high-level KPIs set

Area KPI

Acquisition

Signed up (%)

Created team (%)

CAC (€)

ActivationActive user (%)

In pool (%)

RetentionIn-phase retention (%)

Phase change retention (%)

Revenue

LTV

ARPPU

Conversion

Referral Invited friend (%)

High-level (growth-stage-specific) KPIs to continuously optimize

Tactical tools do explore user behavior and to measure impact from change on KPI(s)

… …

Funnels

Emails

Anything!

Custom Dimensions using hashed user IDs enables user

level analysis, which is VERY powerful

Some tools we are currently using

Tool Domain Sought benefits

Email Improve open rate and CTR

Email (transactional) Identify issues in use and funnels

Funnels Improve user onboarding and payments

Ecommerce Research revenue by product, user type, etc. (Enh. Ecom. Trac.)

Engagement Research anything at the user level!

Engagement Retention, User level progression

User Acquisition Volume and CPI (Cost Per Install)

Landing pages Optimize CTA (Call To Action) conversion

UI heat maps Research behavior and improve UI

KPI dashboard Communication and awareness

Some suggested further reading

* http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version

Slideshare*




Data-driven culture

Key concepts

Accessible tools

Shared goals



@Hoverfalt

May 3, 2023Presentation name 52

building a data-driven culture

Data & Analytics