analytics and optimization
DESCRIPTION
TRANSCRIPT
![Page 1: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/1.jpg)
8 — Analytics
From Code to Product gidgreen.com/course
![Page 2: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/2.jpg)
Lecture 8
• Introduction • Website metrics • Optimization • Competitive intelligence • Surveys • Tools and books
From Code to Product Lecture 8 — Analytics— Slide 2 gidgreen.com/course
![Page 3: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/3.jpg)
Why analytics?
• Quantify success/failure – For yourselves – For investors – Against competition
• Scientific decisions – No blind faith – Fewer arguments – Avoid HiPPO = highest paid person’s opinion
From Code to Product Lecture 8 — Analytics— Slide 3 gidgreen.com/course
![Page 4: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/4.jpg)
Good analytics
• Simple • Few in number • Relevant • Unambiguous • Actionable • Instant (or nearly) • Repeatable
From Code to Product Lecture 8 — Analytics— Slide 4 gidgreen.com/course
![Page 5: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/5.jpg)
AARRR — Metrics for pirates
From Code to Product Lecture 8 — Analytics— Slide 5 gidgreen.com/course
Acquisition Site visit or app download
Activation Registration or usage
Retention Repeat usage
Referral Brings other people
Revenue Generate cash
Dav
e M
cClu
re,
500
Star
tups
![Page 6: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/6.jpg)
Some quotes
“What gets measured, gets managed.” — Peter Drucker
“The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions.”
— Eric Ries, The Lean Startup
From Code to Product Lecture 8 — Analytics— Slide 6 gidgreen.com/course
![Page 7: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/7.jpg)
In-app analytics
• Home rolled or third party • Store usage information locally
– ‘Call home’ when online
• Privacy concerns – Confirmation dialog?
• Complete access to device – But you will be caught!
• Problem: slow iteration
From Code to Product Lecture 8 — Analytics— Slide 7 gidgreen.com/course
![Page 8: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/8.jpg)
Web analytics
• All activity visible to site – Users don’t expect privacy
• Web servers log requests – Also: Javascript solutions
• Page view centric – Other events require integration – Coffee break? – Events not sessions
From Code to Product Lecture 8 — Analytics— Slide 8 gidgreen.com/course
![Page 9: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/9.jpg)
A web server log line
www.websudoku.com 24.186.55.113 [06/May/2012:08:13:02 -0400] "GET / HTTP/1.1” 200 1045 "http://www.google.com/search?q=sudoku”
"Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B179 Safari/7534.48.3" From Code to Product Lecture 8 — Analytics— Slide 9 gidgreen.com/course
![Page 10: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/10.jpg)
Javascript tracking code <script type="text/javascript”> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-1165533-3']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script>
From Code to Product Lecture 8 — Analytics— Slide 10 gidgreen.com/course
![Page 11: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/11.jpg)
Metrics alternatives
From Code to Product Lecture 4 — UI Design— Slide 11 gidgreen.com/course
Server logs Javascript Home-made
Integration None Via HTML Server code
Convenience Download + analyze
Web-based access Up to you
Delay None Up to 24 hours Up to you
Reporting Varies Advanced Up to you
Other events Hard Via API Easy
Data leakage None Total! None
![Page 12: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/12.jpg)
Track web users by…
• IP address – Given for every web request – Good for geography – But: proxies, classrooms, router resets
• Cookies – Track user browser over long term – But: clearing, multi-browsing, first request – Customization of web server
From Code to Product Lecture 8 — Analytics— Slide 12 gidgreen.com/course
![Page 13: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/13.jpg)
Track web users by…
• Log in – Reliable for registered users – But: anonymous users, multiple accounts – Requires custom logging tools
• Solution: combine! – Intelligently tie IPs, cookies and accounts – Example: user registration
• Data always incomplete
From Code to Product Lecture 8 — Analytics— Slide 13 gidgreen.com/course
![Page 14: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/14.jpg)
Lecture 8
• Introduction • Website metrics • Optimization • Competitive intelligence • Surveys • Tools and books
From Code to Product Lecture 8 — Analytics— Slide 14 gidgreen.com/course
![Page 15: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/15.jpg)
Basic website metrics
From Code to Product Lecture 8 — Analytics— Slide 15 gidgreen.com/course
![Page 16: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/16.jpg)
Immediate questions
• When does one visit end? – GA: 30 minutes without activity
• What makes a visitor unique? – GA: Tracking cookie
• How is duration calculated? – GA: Time between first and last pages
• What makes a visitor new? – GA: Never visited your site before
From Code to Product Lecture 8 — Analytics— Slide 16 gidgreen.com/course
![Page 17: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/17.jpg)
Geography
From Code to Product Lecture 6 — BM — Advertising— Slide 17 gidgreen.com/course
![Page 18: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/18.jpg)
Demographics
From Code to Product Lecture 6 — BM — Advertising— Slide 18 gidgreen.com/course
![Page 19: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/19.jpg)
Frequency report
From Code to Product Lecture 8 — Analytics— Slide 19 gidgreen.com/course
![Page 20: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/20.jpg)
Sources of traffic
• Type-in (no referrer) – Includes browser bookmarks
• Search engines – Navigational search = type-in
• Referrals – Website links or social media
• Paid advertising • Email campaigns
From Code to Product Lecture 8 — Analytics— Slide 20 gidgreen.com/course
![Page 21: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/21.jpg)
The multitouch problem
• There’s history before the referrer – Who deserves the credit, e.g. affiliates
• So who gets the credit? – Last click (standard) – First click (unrealistic) – Even split – Split weighted to last
• Real question: what gives best ROI?
From Code to Product Lecture 8 — Analytics— Slide 21 gidgreen.com/course
![Page 22: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/22.jpg)
Search engine queries
From Code to Product Lecture 8 — Analytics— Slide 22 gidgreen.com/course
Also: internal site search
![Page 23: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/23.jpg)
Popular pages
From Code to Product Lecture 8 — Analytics— Slide 23 gidgreen.com/course
![Page 24: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/24.jpg)
Landing/entry pages
From Code to Product Lecture 8 — Analytics— Slide 24 gidgreen.com/course
“You can’t choose your home page” — A. Kaushik
![Page 25: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/25.jpg)
Clickmaps and heatmaps
From Code to Product Lecture 8 — Analytics— Slide 25 gidgreen.com/course
![Page 26: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/26.jpg)
Conversion funnel
From Code to Product Lecture 8 — Analytics— Slide 26 gidgreen.com/course
Sour
ce:
ww
w.s
earc
heng
inej
ourn
al.c
om
![Page 27: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/27.jpg)
Sampling methods
• Popular site => lots of data – Burden to collect, slow to analyze
• Don’t record all events – Choose important pages – Random subset of visitors – Random subset of pageviews
• Sub-sample when analyzing – By page or visitor
From Code to Product Lecture 8 — Analytics— Slide 27 gidgreen.com/course
![Page 28: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/28.jpg)
Staleness due to changes in…
• Content • User familiarity • Search engine rankings • Market • Technology • Cookies
– Long-term analysis would be great!
From Code to Product Lecture 8 — Analytics— Slide 28 gidgreen.com/course
![Page 29: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/29.jpg)
Lecture 8
• Introduction • Website metrics • Optimization • Competitive intelligence • Surveys • Tools and books
From Code to Product Lecture 8 — Analytics— Slide 29 gidgreen.com/course
![Page 30: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/30.jpg)
Optimization
• You don’t know how users behave – Example: show price early on?
• Small changes => big results – But which small changes?
• Use a scientific methodology – Easy to set up – Easy to get report – Statistical significance
From Code to Product Lecture 8 — Analytics— Slide 30 gidgreen.com/course
![Page 31: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/31.jpg)
Wording example
From Code to Product Lecture 8 — Analytics— Slide 31 gidgreen.com/course
Sour
ce:
http
://w
ww
.dus
tinc
urti
s.co
m/
you_
shou
ld_f
ollo
w_m
e_on
_tw
itte
r.ht
ml
![Page 32: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/32.jpg)
A/B testing
• Two parallel variations – Current vs challenger
• Assign randomly and evenly – What about previous visitors? – Repeat requests within a session?
• Set test length in advance – Length of time or number of visits
• Chi-squared (or similar) test
From Code to Product Lecture 8 — Analytics— Slide 32 gidgreen.com/course
![Page 33: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/33.jpg)
Contingency table
Product purchased
Not purchased
9 575
13 563
From Code to Product Lecture 8 — Analytics— Slide 33 gidgreen.com/course
![Page 34: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/34.jpg)
Multivariate testing
From Code to Product Lecture 8 — Analytics— Slide 34 gidgreen.com/course
Sour
ce:
http
://w
ww
.get
elas
tic.
com
/tes
ting
-pa
rt-1
/
![Page 35: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/35.jpg)
Multivariate testing
• Best to use third-party tool • Full factorial vs partial factorial
– Certainty vs efficiency
From Code to Product Lecture 8 — Analytics— Slide 35 gidgreen.com/course
![Page 36: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/36.jpg)
Optimization pitfalls
• Preconception driven – Too many similar tests – Checking before it’s done
• Wrong goal – e.g. started vs completed purchases
• Unfair test – Different time periods – New vs returning users
From Code to Product Lecture 8 — Analytics— Slide 36 gidgreen.com/course
![Page 37: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/37.jpg)
More complex tests
• Non-binary outcomes – Size of purchase, length of stay
• Cohort / longitudinal tests • Whole-site multivariate testing • Pricing
– How to prevent a riot?
• Spot diminishing returns – Focus on registration, payment, etc…
From Code to Product Lecture 8 — Analytics— Slide 37 gidgreen.com/course
![Page 38: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/38.jpg)
Lecture 8
• Introduction • Website metrics • Optimization • Competitive intelligence • Surveys • Tools and books
From Code to Product Lecture 8 — Analytics— Slide 38 gidgreen.com/course
![Page 39: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/39.jpg)
Finding competitors
From Code to Product Lecture 8 — Analytics— Slide 39 gidgreen.com/course
![Page 40: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/40.jpg)
Searches for product
From Code to Product Lecture 8 — Analytics— Slide 40 gidgreen.com/course
![Page 41: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/41.jpg)
But…
From Code to Product Lecture 8 — Analytics— Slide 41 gidgreen.com/course
![Page 42: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/42.jpg)
Ranking for general searches
From Code to Product Lecture 8 — Analytics— Slide 42 gidgreen.com/course
![Page 43: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/43.jpg)
App Store searches
From Code to Product Lecture 8 — Analytics— Slide 43 gidgreen.com/course
![Page 44: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/44.jpg)
Online mentions
From Code to Product Lecture 8 — Analytics— Slide 44 gidgreen.com/course
![Page 45: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/45.jpg)
Website traffic
From Code to Product Lecture 8 — Analytics— Slide 45 gidgreen.com/course
![Page 46: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/46.jpg)
Website traffic
From Code to Product Lecture 8 — Analytics— Slide 46 gidgreen.com/course
![Page 47: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/47.jpg)
Downloads/installs
From Code to Product Lecture 8 — Analytics— Slide 47 gidgreen.com/course
![Page 48: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/48.jpg)
Registrations
From Code to Product Lecture 8 — Analytics— Slide 48 gidgreen.com/course
![Page 49: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/49.jpg)
Revenue
From Code to Product Lecture 8 — Analytics— Slide 49 gidgreen.com/course
Also: UK private companies
![Page 50: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/50.jpg)
Revenue
From Code to Product Lecture 8 — Analytics— Slide 50 gidgreen.com/course
$200k
![Page 51: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/51.jpg)
Lecture 8
• Introduction • Website metrics • Optimization • Competitive intelligence • Surveys • Tools and books
From Code to Product Lecture 8 — Analytics— Slide 51 gidgreen.com/course
![Page 52: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/52.jpg)
Why surveys?
• Customer feedback en masse – Initiated by you (email/web) – Avoid vocal minority
• Understand market – Job descriptions – Size of company – Use of product
• How did you find me?
From Code to Product Lecture 8 — Analytics— Slide 52 gidgreen.com/course
![Page 53: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/53.jpg)
Why surveys?
• Help with strategic decisions – Premium offerings – Major new versions
• Customer satisfaction – Quantify word of mouth
• Understand abandonment – But hard to motivate response
• Open-ended feedback
From Code to Product Lecture 8 — Analytics— Slide 53 gidgreen.com/course
![Page 54: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/54.jpg)
Sources of bias
• Non-response bias – Busy customer ≠ bad customer
• Response bias – Word questions objectively
• Predictions vs facts – Would you pay? How much?
• Snapshot in time – Lots of data vs ongoing data
From Code to Product Lecture 8 — Analytics— Slide 54 gidgreen.com/course
![Page 55: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/55.jpg)
Good survey design
• Keep it short! – Focus on objectives
• Minimize burden on user – Easy questions, especially at start – Multiple choice
• Make it feel anonymous – Social desirability bias
• Free text at end
From Code to Product Lecture 8 — Analytics— Slide 55 gidgreen.com/course
![Page 56: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/56.jpg)
Bad questions
When did you last go online and buy something?
Would you buy our superior product?
Are you willing to pay for things online?
If we created a reliable and bug-free product which had all of the features that you requested in
response to the questions in this survey, would you be willing to pay us $10 per month for it?
What are you looking for?
From Code to Product Lecture 8 — Analytics— Slide 56 gidgreen.com/course
![Page 57: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/57.jpg)
Lecture 8
• Introduction • Website metrics • Optimization • Competitive intelligence • Surveys • Tools and books
From Code to Product Lecture 8 — Analytics— Slide 57 gidgreen.com/course
![Page 58: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/58.jpg)
Analytics tools
From Code to Product Lecture 8 — Analytics— Slide 58 gidgreen.com/course
![Page 59: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/59.jpg)
Other tools
From Code to Product Lecture 8 — Analytics— Slide 59 gidgreen.com/course
![Page 60: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/60.jpg)
Books
From Code to Product Lecture 8 — Analytics— Slide 60 gidgreen.com/course
![Page 61: Analytics and Optimization](https://reader038.vdocuments.site/reader038/viewer/2022110113/540734918d7f72b90a8b4a37/html5/thumbnails/61.jpg)
We didn’t cover…
• Social media analytics – Popularity – Sentiment analysis
• Video analytics – Attention – Embeds
• Content reuse
From Code to Product Lecture 8 — Analytics— Slide 61 gidgreen.com/course