![Page 1: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/1.jpg)
ABBYA Django app for A/B test
documentation
![Page 2: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/2.jpg)
Who?● Data Scientist at Jimdo● Jimdo
○ DIY website builder○ founded in 2007 in Hamburg, Germany○ > 180 employees in 3 countries (DE, US, JP)○ > 10 million websites
Twitter: @datenheini
![Page 3: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/3.jpg)
Medicine:
● placebo = control group
● drug(s) = test group(s)
A/B Testing Basics
Internet company:
= old version
= new version(s)
![Page 4: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/4.jpg)
A/B Testing Basics
Rules:
● Frame your hypothesis.
● Keep it simple.
● Set a timeframe.
![Page 5: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/5.jpg)
A/B Testing Basics
Best practices:
● Be realistic.
● Grab low-hanging fruits.
● Don’t get frustrated.
![Page 6: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/6.jpg)
A/B Testing Basics
2 minutes of science...
![Page 7: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/7.jpg)
Significance (p-value):
A/B Testing Basics
→ should be 0% (at most 5%)
How often will failed tests lead to positive results?
![Page 8: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/8.jpg)
Statistical power:
A/B Testing Basics
→ should be 100% (at least 80%)
How often will you recognize a successful test?
![Page 9: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/9.jpg)
Evaluation metrics:
● p-value
● statistical power
● effect size
● confidence interval
A/B Testing Basics
![Page 10: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/10.jpg)
Let’s do a little A/B test
And see what we need documentation for.
![Page 11: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/11.jpg)
![Page 12: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/12.jpg)
![Page 13: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/13.jpg)
Why?
Communicating the results
![Page 14: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/14.jpg)
Why?
Persisting the results
→ Knowledge Base
![Page 15: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/15.jpg)
Why?
Avoid duplicated tests
![Page 16: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/16.jpg)
Why?
Everybody has this problem, there needs to be a solution already!
NO!
![Page 17: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/17.jpg)
Why?
![Page 18: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/18.jpg)
What?
![Page 19: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/19.jpg)
What?
Central place for A/B test documentation
![Page 20: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/20.jpg)
What?
Tests and results self-explaining
![Page 21: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/21.jpg)
What?
Keep track of evaluation metrics
![Page 22: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/22.jpg)
What?
Tries to encourage best practices
![Page 23: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/23.jpg)
What?
Productivity gain
![Page 24: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/24.jpg)
What? Productivity gain!
Reference for daily work
Cross-functional effects
Better understanding of customers
![Page 25: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/25.jpg)
How?
![Page 26: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/26.jpg)
How?
CRUD app (create-read-update-delete)
RESTful API (WIP)
Test evaluation logic
![Page 27: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/27.jpg)
Demo time!
![Page 28: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/28.jpg)
It’s your turn!
ABBY is open-source. Have a look:
https://github.com/Jimdo/abby
![Page 29: ABBY - A Django app for A/B test documentation](https://reader034.vdocuments.site/reader034/viewer/2022052412/5583f075d8b42a2a4d8b5103/html5/thumbnails/29.jpg)
Thank you!