growth hacking and the bandit problem
DESCRIPTION
Growth hackers rely on A/B testing to validate their improvement, but A/B testing is slow and inefficient. How do we make it faster? By solving the bandit problem! Bandit algorithms provide an alternative to A/B testing that is faster and more efficient, allowing more tests in less time. Video of the talk: http://www.youtube.com/watch?v=h7SG_Wo8sPQ Myna: http://mynaweb.com/?utm_source=awareness&utm_medium=slideshare&utm_campaign=Growth%2BHackerTRANSCRIPT
![Page 1: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/1.jpg)
GROWTH HACKING AND
THE BANDIT PROBLEM
Starring Noel Welsh A Production
Friday, 8 February 2013
![Page 2: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/2.jpg)
WHAT DO
GROWTH HACKERS DO?
Friday, 8 February 2013
![Page 3: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/3.jpg)
GROW!Friday, 8 February 2013
![Page 4: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/4.jpg)
Metric(sign-ups, revenue, etc)
Friday, 8 February 2013
![Page 5: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/5.jpg)
Make this happen
Friday, 8 February 2013
![Page 6: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/6.jpg)
HOW?Friday, 8 February 2013
![Page 7: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/7.jpg)
Build
MeasureLearn
Friday, 8 February 2013
![Page 8: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/8.jpg)
Build
Friday, 8 February 2013
![Page 9: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/9.jpg)
Build
Measure
Friday, 8 February 2013
![Page 10: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/10.jpg)
Build
MeasureLearn
Friday, 8 February 2013
![Page 11: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/11.jpg)
FASTER IS BETTER
Friday, 8 February 2013
![Page 12: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/12.jpg)
HOWDO WE MEASURE &
LEARN?
Friday, 8 February 2013
![Page 13: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/13.jpg)
A/BFriday, 8 February 2013
![Page 14: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/14.jpg)
MeasureBuild Learn
Friday, 8 February 2013
![Page 15: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/15.jpg)
Build
MeasureLearn
Friday, 8 February 2013
![Page 16: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/16.jpg)
Build
Measure/Learn
Friday, 8 February 2013
![Page 17: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/17.jpg)
Build Measure / Learn
Friday, 8 February 2013
![Page 18: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/18.jpg)
THE BANDIT PROBLEM(EST 1952 ... OR 1933)
Friday, 8 February 2013
![Page 19: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/19.jpg)
k arms (or variants)
Friday, 8 February 2013
![Page 20: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/20.jpg)
Select a variant
Friday, 8 February 2013
![Page 21: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/21.jpg)
Receive a reward
Friday, 8 February 2013
![Page 22: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/22.jpg)
REPEAT
Friday, 8 February 2013
![Page 23: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/23.jpg)
MAXIMISE TOTAL REWARD
Friday, 8 February 2013
![Page 24: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/24.jpg)
Friday, 8 February 2013
![Page 25: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/25.jpg)
HOW?Friday, 8 February 2013
![Page 26: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/26.jpg)
DISPLAY VARIANT WITH
HIGHEST AVERAGE REWARD
Friday, 8 February 2013
![Page 27: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/27.jpg)
Good Variant Bad Variant
Friday, 8 February 2013
![Page 28: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/28.jpg)
Good Variant Bad Variant
Views: 1Reward: 0Average: 0
Friday, 8 February 2013
![Page 29: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/29.jpg)
Good Variant Bad Variant
Views: 1Reward: 0Average: 0
Views: 1Reward: 1Average: 1
Friday, 8 February 2013
![Page 30: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/30.jpg)
OOPS!Friday, 8 February 2013
![Page 31: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/31.jpg)
Good Variant Bad Variant
Views: 1Reward: 0Average: 0
Views: 1Reward: 1Average: 1
WILL NEVER TRY THE
GOOD VARIANT AGAIN!
Friday, 8 February 2013
![Page 32: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/32.jpg)
BALANCE EXPLOIT
AND EXPLORE
Friday, 8 February 2013
![Page 33: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/33.jpg)
EXPLOITTRY THE VARIANT THAT HAS WORKED BEST IN
THE PAST
Friday, 8 February 2013
![Page 34: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/34.jpg)
EXPLORE TRY OTHER VARIANTS
TO SEE IF THEY’RE BETTER
Friday, 8 February 2013
![Page 35: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/35.jpg)
E-GREEDYFriday, 8 February 2013
![Page 36: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/36.jpg)
10% EXPLORE
90% EXPLOITDon’t like 10% and 90%? Change it.
Friday, 8 February 2013
![Page 37: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/37.jpg)
MORE INTRICATE BALANCING IS POSSIBLE.
SEE UCB-1, THOMPSON SAMPLING, etc.
Friday, 8 February 2013
![Page 38: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/38.jpg)
Friday, 8 February 2013
![Page 39: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/39.jpg)
DOES IT
WORK?
Friday, 8 February 2013
![Page 40: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/40.jpg)
Friday, 8 February 2013
![Page 41: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/41.jpg)
REAL WORLD
CONSIDERATIONS
Friday, 8 February 2013
![Page 42: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/42.jpg)
WORKFLOW
Friday, 8 February 2013
![Page 43: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/43.jpg)
DEFINING
REWARDS
Friday, 8 February 2013
![Page 44: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/44.jpg)
STABLE PREFERENCES
Friday, 8 February 2013
![Page 45: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/45.jpg)
THANK YOU!NOW GO FORTH AND
ENGAGE IN BANDITRY
Friday, 8 February 2013
![Page 46: Growth Hacking and the Bandit Problem](https://reader034.vdocuments.site/reader034/viewer/2022051514/54b3be6c4a7959fc0c8b456d/html5/thumbnails/46.jpg)
CREDITSPhotos and artwork by the following people were used in
this presentation:
Moyan Brenn http://www.flickr.com/photos/aigle_dore/Suicine http://www.flickr.com/photos/bigmikeyeah
ankakay http://www.flickr.com/photos/ankakay/Cdr Aitch http://www.flickr.com/people/hjsouthgate/
StJost http://stjost.deviantart.com/
Friday, 8 February 2013