a/b testing - how to break things and fail fast... without breaking things
Post on 05-Dec-2014
161 Views
Preview:
DESCRIPTION
TRANSCRIPT
M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2
A/B TestingHOW TO BREAK THINGS AND FAIL FAST…
WITHOUT BREAKING THINGS
About Atlassian
• Products include: JIRA, Confluence, Bitbucket, HipChat…
• Founded in 2002
• Over 35,000 Customers
• Over 700 employees globally
• No sales people
!
!
!
W e h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r.
Think big, act small, fail fast, learn rapidly.
" L E A N S O F T WA R E D E V E LO P M E N T ” BY M A RY A N D TO M P O P P E N D I E C K ( 2 0 0 3 )
”“
Atlassian A/B Testing is cross functional
• Product Management
• Customer Platform
• Customer Advocates
• Internal Systems
!
!
!
C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t :
• Leads
• Product Marketing
• Support
• Bitbucket
• HipChat
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Where do experiment ideas come from?
Anywhere.
What are you are trying to improve?
Blimp Dashboard
JIRA – Ticket
• Pebbles
• Change in copy or CTA presentation
• Rocks
• New page (e.g. no Dev Tools family page)
• Boulders
• Simplified order form.
!
!
!
Types of tests
Examples
Reduce Friction
Increase Visibility
Test drive campaigns
Simplify
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
What are you are trying to improve?
Blimp Dashboard
JIRA – Prioritized Backlog
JIRA – Ticket
JIRA – Ticket
Experiment Illuminati
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Build
Optimizely – Build
Prepare for results.
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
JIRA – Prioritized Backlog
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Start Experiment
Optimizely – Project Code
Magnolia CMS – Include Optimizely Checkbox
Magnolia CMS – Include Optimizely Checkbox
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Results
What are you are trying to improve?
Blimp Dashboard
Optimizely – Results
Confluence – Corresponding Test Page
Confluence – Results Archive
• Ideas are everywhere
• Everything affecting conversion is tested
• No traffic wasted
• Be aggressive
• Regular cadence
• Never run a test that you wouldn’t want to win
Our Test Philosophy
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.
• Involve stakeholders and developers… and get approvals.
Key Take Aways
Thank you!
M A R K H A LVO R S O N • H E A D O F I N T E R AC T I V E • AT L A S S I A N S O F T WA R E • @ h a l v 0 1 1 2
top related