quickbooks connect 2016 - implementing analytic and optimization tools on your website or

Brian Gardner - Director, Product ManagementIlya Izrailevsky - Engineering Manager

Implementing analytic and optimization tools on your website or app

WiFi: QBConnect Password: not required#QBConnect

2

Today’s Speakers

Brian GardnerDirector, Product Management@brianwgardner

Ilya IzrailevskyEngineering Manager@iizrailevsky

#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

Introduction & Overview

Understand the landscape analytic and optimization tools

How to implement an A/B test

Gain the ability to make informed decisions about what in your product is working (or not)

Agenda

5

“The sooner you run an experiment, the sooner you are likely to find a surprise, and the surprise is the market speaking to you, telling you something you didn't know. Several of our businesses here came out of surprises.”

– Scott Cook, Intuit founder & chairman

Intuit’s tradition of experimentation

Deep Customer Empathy

Go Broad to Narrow

Rapid Experiments

with Customers

CustomerDelight


6

What is an A/B test?

BA#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

Why A/B Test?


Discover which experiences delight your customer

Accelerate your learning loop

Make faster, better decisions

with data

8

Who We Serve

9

Introducing:

WASABI


10

First time use test: QuickBooks Self EmployedTest: Complete tax profile first Control: Connect account first

In the test cell, we see:• Significant increase in tax profile completion• Slight increase in categorizing a transaction from a card• No change to connecting a bank• No change to categorizing on the transactions page


11

First time use test: QuickBooks Self EmployedTest: Complete tax profile first Control: Connect account first

In the test cell, we see:• Significant increase in tax profile completion: +20%• Slight increase in categorizing a transaction from a card: +6%• No change to connecting a bank: Flat• No change to categorizing on the transactions page: Flat






Agenda

13

• Marketing vs Product (changing content & UI vs features or back-end)• How much engineering is involved?• Do you have sensitive data?• What platforms and devices do you need to test across?• Cost• Analytics provided / options• Efficient use of traffic• Performance• Ease of use

How do you choose a tool for A/B testing?What are the dimensions you should think about when choosing A solution?


14

Landscape of Popular Options

• WYSIWIG• Minimal Engineering• Marketing focused

• Developer focused• Flexible engineering effort• Own your own data

• Customized• Ongoing maintenance• Own your own data


COMMERCIAL OPEN SOURCE IN-HOUSE

WASABI





Agenda

16

The A/B Testing Lifecycle: Iterate to Awesomeness


Step 1:Plan an experiment

Step 2:Develop a feature to test

Step 3:Instrument your application with Wasabi API Calls

Step 4:Create and run the experiment using the Wasabi UI

Step 5:Deploy and track results quickly

17

• Is this user already assigned for this experiment?

• Does the user qualify for the experiment? (Segmentation Rules)

• Is the user already in a conflicting experiment? (Mutual Exclusion)

• Randomization• Check vs. Sampling Percentage

• Assign to an experience (Bucket)

• Return assignment

How does user assignment actually work?What should I show this user when a WASABI test is active?


19

Architecture: behind the scenes, here’s how it’s built

Web tier

Admin UI API API

Segmentation

Mutual Exclusion

Sampling

Bucket Dice Roll

Application tier

Cassandra MySQLData tier

Authentication

User Directory

Logging

Metrics

Pluggable Components

Bucket assignments

Events / analytics

Kafka

Stream

HDFS SparkVertica S3

Pluggable Components

Wasabi A/B Testing Service






Agenda

21

Best practices and considerations for a good A/B test


Get started! (Don’t let the perfect be the enemy of the good)1

Think about what you’ll learn from a “failed” test in advance.

Be skeptical – don’t take your results at face value• Track all your key metrics – check for unintended consequences

• Think about sources of bias, assumptions: timing, infrastructure, etc.

Don’t stop at one round of experiments!

2

3

4

5

Don’t forget qualitative research as well

22

What Should You Take Away?


Everyone can A/B test! (It’s not just for marketers)

A lot of A/B testing tools in the market. Choose wisely!

Take time to come up with your hypothesis and how you’ll measure it. It can

make or break your experiment.

23

If you’re interested, here’s how you get Wasabi:

https://github.com/intuit/wasabi

Questions?


Contact us: [email protected]





Questions?

Thanks for coming!We’d love your feedback.Before you head out, take a minute to rate this session in the QuickBooks Connect mobile app.

Keep the conversation goingdeveloper.intuit.com/hub/quickbooks-connect

Access Session SlidesEngage with Speakers and Fellow DevelopersGet Questions Answered

quickbooks connect 2016 - implementing analytic and optimization tools on your website or

Software